Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Shuai Yu, Yonghuai Liu, Jiong Zhang, Jianyang Xie, Yalin Zheng, Jiang Liu, Yitian Zhao

Abstract

Optical Coherence Tomography Angiography (OCTA) has been widely used by ophthalmologists for decision-making due to its superiority in providing caplillary details. Many of the OCTA imaging devices used in clinic provide high-quality 2D en face representations while their 3D data quality are largely limited by low signal to-noise ratio and strong projection artifacts, which restrict the performance of depth-resolved 3D analysis. In this paper, we propose a novel 2D-to-3D vessel reconstruction framework based on the 2D en face OCTA images. This framework takes advantage of the detailed 2D OCTA depth map for prediction and thus does not rely on any 3D volumetric data. Based on the data with available vessel depth labels, we first introduce a network with structure constraint blocks to estimate the depth map of blood vessels in other cross-domain en face OCTA data with unavailable labels. Afterwards, a depth adversarial adaptation module is proposed for better unsupervised cross-domain training, since images captured using different devices may suffer from varying image contrast and noise levels. Finally, vessels are reconstructed in 3D space by utilizing the estimated depth map and 2D vascular information. Experimental results demonstrate the effectiveness of our method and its potential to guide subsequent vascular analysis in 3D domain.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_2

SharedIt: https://rdcu.be/cyl9y

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper presents the first attempt to reconstruct the 3D structure of blood vessels from 2D OCTA images by proposing a cross-domain depth estimation network. Then, the 3D vessels are reconstructed by integrating 2D vascular information with the predicted depth maps.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main contribution of this work is estimating 3D vessels from 2D OCTA angiograms based on the depth map estimated given by the proposed depth estimation network.

The results are promising and sufficient to demonstrate the effectiveness of the proposed framework.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

In Section 2.1, Cross-domain Depth estimation network, how to provide accurate ground truth is not described, and its accuracy is thus questionable.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This is the first work to reconstruct 3D vessel structures from 2D OCTA images. The authors are encouraged to provide code and data to establish a new research direction.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The main contribution of this work should be appreciated by MICCAI audience, estimating 3D vessels from 2D OCTA angiograms based on the depth map estimated given by the proposed depth estimation network. It should be informative if the authors can add some more details of how to obtain the ground truth data for the cross-domain depth estimation network.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper presents the first attempt to reconstruct the 3D structure of blood vessels from 2D OCTA images by proposing a cross-domain depth estimation network. The entire frame technically sounds, and the results are promising.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

1
Reviewer confidence

Somewhat confident

Review #2

Please describe the contribution of the paper

A new 3D vessel reconstruction model for OCT angiography.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novelty of the imaging modality and clear demand for a solution to the tackle problem.
- The cross-domain network architecture can have other potential uses.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The decision not to use 3D volumetric data for no apparent reason, and which is present anyway -other than keeping it as 2 ground truth for evaluation-, is surprising.
- Mathematical details are missing
- Execution of experiments is unclear (e.g. lack of repetition, missing stats, etc).
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Poor. In fact, this is one of my major complains for this draft. I found some of the authors responses to the checklist puzzling. They claim to have reported statistics, but statistics other than mean are absent. They claim they describe the study cohort, but I can find demographics, power analysis, etc. They claim that they provide citations to the existing datasets and that no ethical approval is needed, none is correct. I could continue but virtually, none of the information in the checklist is correct, and further replicability of experiments is clearly compromised.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
OVERALL COMMENTS Even if its foundations on previous imaging methods are long establish, angiographic OCT is relatively novel tool (about a decade old). In this sense, most research in this field is naturally justified. Also, the problem of the 3D reconstruction of the vasculature is one interest regardless of the algorithmic decisions on how to proceed. In this topic, literature is still scarce, which justifies the limited introductory review of the state of the art, and further emphasizes the novelty of the topic. In this sense, the departing problem of the paper is excellent. Further, the authors intend to ride the wave of the trendy deep adversarial networks to add another hit -although I found the justification for such decision totally absent-. At this point, the authors got all my attention, but then, as one progresses to the methods, experiments and results the rhythm decay, and the execution is questionable.

SUGGESTIONS TO IMPROVE THE DRAFT
- Replicate your experiments.
- Report statistics correctly; both descriptive, e.g. include standard deviations, confidence interval, power analysis, etc, and inferential, hypethesis testing, etc. Without this, the exemplary results reported are meaningless
- Illustrate reconstructions not only of your method, but also of the other methods for visual comparisons.
- Problem formalization and mathematical details are virtually missing. Yes, for the deep network it is indicated the overall loss function as well as the adversarial loss, but that’s about it. Please provide sufficient mathematical detail for accurate reproducibility of every aspect of your deep model. Further, all the details in section 2.2 are very vague. Please provide the mathematical formalities.
- What is it that you are reporting in Table 1? Are these train or test results? Please report both as they convey different information about your model.
- What type of validity (construct, internal, external, etc) is claimed?
- How is it that an overparameterized model is (apparently) not suffering despite a relatively small dataset? I would blame a massive overfitting but without further details is a difficult guess…
- How were the adversarial images generated is not clear, neither the adversarial attack intended. There is only the indication of split into source and target domains, but little else.
Minor
- The assumption that because one method work well in one domain is likely to work in another (related or not), has always puzzled me… At least, mathematically it has not support whatsoever.
- An opening criticism to previous approaches that it is supposed to justify yours is that they only used the 2D en face angiograms, but then your method proceeds equally. Why?
- Readability of fig 2 is difficult…well, at least it is for me; I reckon I’m just old :(
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Reproducibility and replicability are clearly compromised.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

This paper provides an idea of 3D vessel reconstruction for OCTA images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The attempts of this article are contributory. 3D vessel reconstruction in OCTA images performed by combining imaging and graphics.
2. The experimental result is better than the existing methods.
3. The organization and illustrations of this paper are clear.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Depth estimation from 2D images is limited.
2. The network structure and some parameters still need to be improved.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method seems easy to be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Although the authors proposed that they reconstructed three-dimensional blood vessels from 2D images, there are limitations in doing so because 2D images contain almost no depth information. I recommend that the author attempt the 3D-to-2D method to reconstruct 2D depth images in the future work. reference: Li et al. “Image projection network: 3D to 2D image segmentation in OCTA images,” IEEE Trans. Med. Imaging, vol. 39, no. 11 pp. 3343-3354, 2020. Dataset: https://ieee-dataport.org/open-access/octa-500
2. I believe this work is contributed, especially the three-dimensional blood vessel reconstruction of OCTA combined with graphics. However, judging from the result image, the number of subdivisions of models is not enough, and the result of the triangular mesh result is not as good as that of dense surface point cloud. Hope to give an index analysis of the number of subdivisions and modeling speed.
3. Minor: [Introduction] “…details at capillary-level in a short time, as shown in Fig. 1 (a).” No capillary details in Fig.1(a). [Experiments] Lack of ablation experiments to optimize the network structure. [Experimental Results] Give the variance of the experiment to facilitate readers to understand the robustness of the method.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The author does not specifically analyze and think about the challenges of blood vessel segmentation, but tends to apply and improve the current latest technology. Their method is troublesome for blood vessel segmentation and does not improve much.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The reviewers have divergent opinions on this paper. Some reviewers pointed out that the motivation of not using 3D volumetric data is unclear. Some details of the method are and execution of experiments is unclear. Please explain the motivation of the method and provide details of the methods and experiments in the rebuttal letter.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Author Feedback

Many thanks for your constructive comments. We will respond to the main concerns raised by (meta) reviewers.

Q1: R#2 The motivation of not using 3D volumetric data is unclear. A1: We advocate the use of 3D volumetric data, and we agree that the vessel reconstruction from 3D data would be more promising, despite 3D vessel reconstruction in OCTA imagery is an unexplored task. However, it is not always straightforward to have the required 3D volumetric data in real application. For example, some public datasets such as ROSE[9] does not provide 3D data which makes it impossible to achieve 3D reconstruction of vessels through volumetric data-based methods. In addition, our clinical partner also states the difficulty of 3D data exportation from some OCTA devices, e.g., Heidelberg OCT2, due to restriction from manufacture. To this end, 3D vessel reconstruction from 2D enface image is viable and desirable. Fortunately, Cirrus HD-OCT 5000 provides not only exportable 3D volume data, but also obtains color-coded en face OCTA image (we refer it as a depth map in this paper). These paired data open potential venues to develop 3D vessel reconstruction methods from 2D color-coded image, and the rapid development of domain adaptation approach.

On the other hand, 3D vessel reconstruction from an OCTA volume also faces several challenges. Many of the OCTA imaging devices used in clinic provide high-quality 2D en face representations, while their 3D data quality is largely limited by low signal-to-noise ratio and strong projection artifacts, which restrict the performance of depth-resolved 3D analysis. To our best knowledge, only one study[3] works on vessel reconstruction from 3D data, but this method still suffers from projection and tailing artifacts and may lead to improper vessel structure reconstruction. This is because directly processing of 3D OCTA volume for vessel reconstruction is challenging, due to poor contrast, shadow projection, complex topological structures, and relatively smaller diameters. In summary, the proposed is a potentially powerful tool to reconstruct 3D vessel from 2D en face images when 3D data is not available.

Q2: R#2 Mathematical details are missing in section 2.2. A2: Fig. 3 illustrates the 3D vessel reconstruction via depth map, Sec 2.2 describes the implementation details of centerline point cloud extraction and surface reconstruction. Our work mainly focuses on solving the problem of cross-domain depth map prediction, and 3D reconstruction is based on the depth map as a post-processing step. Due to the space limit, we chose not to provide the mathematical details but the proper references[15-17] at this stage. More details will be added for easy understanding in the revised version.

Q3: R#2 Execution of experiments is unclear. A3: A detailed description of the experimental methodology was given in Sec 3.1, including imaging equipment, image size and physical size, the number of images used for training and testing, the generation of ground truth and the metrics used for testing. However, more information about the implementation details and descriptive statistical (e.g. standard deviations) of our experiments will be added in the final version: the experiments were carried out on a machine with a single NVIDIA GPU GeForce GTX 2080Ti in the PyTorch library. The learning rate was set to 0.0001, the number of training iterations was 200 epochs and the batch size was set to one. All training images were rescaled to 512×512 and random rotation over the range of [-10,10] degrees as well as random horizontal/vertical flipping were employed for data augmentation.

Q4: R#1 How to provide accurate ground truth of depth map is not described. A4: The ground truth of depth map was provided by the Cirrus HD-OCT 5000. The detailed description of OCT-A depth map is described in [A]. We also mentioned it in Sec 1 and Sec 3.1 respectively. [A] Review of Optometry, 2017, 154(3): 36-44.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have explained the reasons of not using 3D volumetric data. That is the required 3D volumetric data in real applications is usually unavailable. As alternative, OCTA volumes are more common in practice and 3D vessel reconstruction from an OCTA volume is challenging as well. The authors have provided some missing details in the rebuttal letter, the AC would suggest the authors included the necessary details in the final version.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have made a good effort at answering most of the important points raised by the reviewiers (primialry R2). Importantly, the paper methodology can easily be improved before the final submission to specify much of the details asked by the different reviewers.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper provides extensive experimental analysis together with ablation studies. However, I also found the problem of 2D->3D segmentation from the depth maps somewhat artificial and don’t find their justification in the rebuttal sufficient. It is also not clear if that is widely available information or only provided by Cirrus (Zeiss) device. They also do not sufficiently address the comments on the experiment results in Table 1. For example, method [26] which is generic domain adaptation algorithm seems to perform essentially equal to the proposed cross-domain depth map prediction. approach.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

back to top

Cross-domain Depth Estimation Network for 3D Vessel Reconstruction in OCT Angiography