Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Changyeop Shin, Hyun Ryu, Eun-Seo Cho, Young-Gyu Yoon

Abstract

High-speed optical 3-D fluorescence microscopy is an essential tool for capturing the rapid dynamics of biological systems such as cellular signaling and complex movements. Designing such an optical system is constrained by the inherent trade-off among resolution, speed, and noise which comes from the limited number of photons that can be collected. In this paper, we propose a recursive light propagation network (RLP-Net) that infers the 3-D volume from two adjacent 2-D wide-field fluorescence images via virtual refocusing. Specifically, we propose a recursive inference scheme in which the network progressively predicts the subsequent planes along the axial direction. This recursive inference scheme reflects that the law of physics for the light propagation remains spatially invariant and therefore a fixed function (i.e., a neural network) for a short distance light propagation can be recursively applied for a longer distance light propagation. Experimental results show that the proposed method can faithfully reconstruct the 3-D volume from two planes in terms of both quantitative measures and visual quality. The source code used in the paper is available at https://github.com/NICALab/rlpnet.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_18

SharedIt: https://rdcu.be/cyhUJ

Link to the code repository

https://github.com/NICALab/rlpnet

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper addresses the problem of virtual refocusing in light microscopy, i.e. reconstructing a volumetric 3D image from only a single/two adjacent acquired image planes. Specifically the authors propose to recursively apply an appropriately trained network that propagates the intensity image to the next intensity image along the optical axis. The proposes method uses several loss components that are motivated by the physical properties of light propagation. The authors finally evaluate their method on two datasets and demonstrate the superiority compared to another state-of-the art method for virtual refocusing.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- very well written and clearly described
- sound experiments and superior performance compared to another state-of-the art method
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- physical model of light propagation is not entirely correct
- code not available to the public
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Code was available to the reviewer. Datasets (and code) seems not to be available to the public
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Image formation in fluorescence microscopy is generally incoherent and the acquired images dependent both on the illumination light and the fluorophore distribution (i.e. are not given by a single light field propagating through free space). For both reasons, the assumed model of light propagation including the stated properties in 2.1 are not entirely physically valid. Of course there are clear correlations between adjacent image planes especially for epifluorescence imaging where the PSF essentially “extends into infinity”. But for other modalities that include optical sectioning (confocal, spinning disk, light sheet) the assumed properties of light propagation don’t hold (e.g. for an infinitely thin light sheet, there are no correlations between adjacent image planes due to “light propagation” only correlations from the imaged structures). So might be worth commenting on that.
- In the equation for the cycle loss, how is “back-propagating” by -l implemented? Reversing the two adjacent images I_k, I_k+1 ? More details regarding that would be helpful.
- In table 1, the PSNR of RLP/DEEP-Z for the beads dataset is increasing with the propagation distance, which is a bit counter-intuitive. What is the reason for this?
- Making the code available to the public would increase the impact of the paper
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A generally well written paper with a clear idea and sound execution.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors present a recursive neural network to approximate light propagation and estimate a 3D volume from 2D sections imaged by fluorescence (wide-field) microscopy. The authors derive their network from first principles of light propagation (which is great). The approach is validated using bead samples and images of C elegans specimen and qualitatively and quantitatively compared to a state-of-the-art approach (Deep Z).
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The neural network design is derived in a principled way, which is a great strength of this work compared to many papers I have seen.

The application of this method is very interesting and important for many types of live microscopy experiments.

The methods and the motivation are well explained and discussed. I enjoyed reading this paper.

The validation is sound and the results are convincing to me.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I actually do not find a major weakness here.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I haven’t tried, but I am confident, that we could reproduce the results. The method is well explained and the code is available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I enjoyed reading this paper. Great work!
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper describes an interesting application and it is one of the few papers that use a solid, clear approach when designing the neural network to solve this particular task.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper introduces RLP-Net for 3D virtual refocusing, where two adjacent 2D fluorescence images are adopted for learning light propagation in a recursive manner. Both quantitative and qualitative experiments are provided.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The overall method is clear and technically sound.
- The motivations of recursive light propagation from two 2-D images are provided.
- The ablation study of memory module is well conducted.
- Paper is well organized and easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The comparison is not fair enough. The Deep-Z method is not a strong baseline. Technically speaking, The Deep-Z utilizes only one image for 3-D virtual refocusing while RLP-Net utilizes two images. Thus, it is not so surprising that RLP-Net outperforms Deep-Z by a large margin.
- The cost of RLP-Net is not clear, in terms of (a) the additional one image required in the data acquisition process, and (b) the computational burden of network inference (e.g., running time). Is it really worth in practice to adopt RLP-Net for performance?
- The potential failure case has not been discussed, e.g., when the moving of object is too fast to capture two adjacent images.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Code is provided but has not been carefully checked. I believe it is not difficult to reproduce the results.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please refer to the weaknesses listed above.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall, I lean to recommend acceptance due to the promising method and good writing. However, the authors are encouraged to clarify the cost of RLP-Net, in terms of the data acquisition and network computation.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The reviewers unanimously recommend acceptance. The topic and proposed approach are interesting, the results are promising, and the paper is very well written. One reviewer is concerned about the correctness of the physical model of light propagation, and another about the fairness of the chosen baseline for comparison. These issues, in addition to several others pointed out by the reviewers, need to be clarified in the revision.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

We thank the reviewers for their positive assessment and constructive comments. Below is our response to the major comments.

Image formation in fluorescence microscopy is generally incoherent and the acquired images dependent both on the illumination light and the fluorophore distribution (i.e. are not given by a single light field propagating through free space). For both reasons, the assumed model of light propagation including the stated properties in 2.1 are not entirely physically valid. A: We agree that our light propagation model does not hold for optical sectioning methods, which is why we set our goal as ‘to infer the volume from wide-field fluorescence images.’ To avoid any potential confusion, we will revise the manuscript to clarify that our method is not compatible with optical sectioning methods. However, we would like to rebut the statement of the reviewer that single light field propagating through free space does not suffice to describe fluorescent light propagation. Fluorescent light is indeed incoherent, as the reviewer pointed out, and thus can be described as a 4-D light-field, which allows digital refocusing of light-field microscopy images (Levoy et al, 2006) – albeit at limited resolution due to limited sampling density.

How is “back-propagating” by -l implemented? A: The concatenation order of two input images determines the inference direction as explained in the Section 2.1. For example, the network infers I_{k+2} if the input is (I_k, I_{k+1}) and infers I_{k-1} if the input is (I_{k+1}, I_k)

In table 1, the PSNR of RLP/DEEP-Z for the beads dataset is increasing with the propagation distance, which is a bit counter-intuitive. What is the reason for this? A: We speculate that unexpected change of PSNR as a function of propagation distance stems from the sparsity of the beads in space. Due to the sparsity, some planes include sharply focused beads whereas others don’t, which manifest itself as varying level of difficulties in predicting different planes.

Making the code available to the public would increase the impact of the paper A: We agree with the reviewer’s thought and plan to include the link to our github repository (which is already ready) in the final submission.

Comparison for the RLP-Net (taking two input images) and Deep-Z (taking single input image) is not fair enough as Deep-Z utilizes only one image for 3-D virtual refocusing while RLP-Net utilizes two images. A: We agree that utilizing two images gives an advantage to RLP-Net. To compensate for the disadvantage, we computed Deep-Z output using two input images, independently, and then took the average as the prediction as described in Section 3.

The cost of RLP-Net is not clear, in terms of (a) the additional one image required in the data acquisition process, and (b) the computational burden of network inference (e.g., running time). Is it really worth in practice to adopt RLP-Net for performance? A: The performance of RLP-Net indeed comes with a cost. Presuming a fast axial scanning method such as electrically tunable lens, taking two images that are axially nearby takes twice as long compared to taking one image. RLP-Net took approximately 0.1 second for a single volume inferencing whereas Deep-Z took about 0.08 second for an input image with 128 × 128 pixels. We will clarify this in the revised version of manuscript.

The potential failure case has not been discussed, e.g., when the moving of object is too fast to capture two adjacent images. A: We agree that understanding failure cases is important and will discuss that in the revised version of the manuscript. We would like to mention that, if the sample moves too fast to take two images, it would not be possible to take a 3-D image at all with a conventional method.

back to top

RLP-Net: Recursive Light Propagation Network for 3-D Virtual Refocusing