Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Yanyu Xu, Xinxing Xu, Lei Jin, Shenghua Gao, Rick Siow Mong Goh, Daniel S. W. Ting, Yong Liu

Abstract

The vessel segmentation in ocular images is a fundamental and important step in the diagnosis of eye-related diseases. Existing vessel segmentation methods require a large-scale ocular images with pixel-level annotations. However, manually annotating masks is a laborious and tedious process. Compared with the traditional pipelines which either annotate the complete training set or several images in full, in this paper, we propose a novel supervision manner, named Partially-Supervised Learning (PSL), which only relies on partial annotations in the form of one patch from each of the few images. Targeting it, we propose an active learning framework with latent MixUp. The active learning strategy is employed to select the most informative patch for further annotation, while the latent MixUp is proposed to learn a proper visual representation of both the annotated and unannotated patches. The experimental results on two types of vessel segmentation datasets (Rose-1 (SVC) dataset for OCTA image, and DRIVE dataset for fundus image) validate the ectiveness of our model. With only 5% annotations on Rose-1 (SVC) and DRIVE dataset, our performance is comparable with the previous methods trained on the whole fully annotated dataset.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87193-2_26

SharedIt: https://rdcu.be/cyhL4

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper presents a method to address the difficult task of annotating manually retinal vessels on fundus and OCTA images, which a prerequisite step in supervised learning techniques. The authors propose a partially-supervised learning approach which consists in annotating only one patch in few images. An active learning framework with a latent Mixup for vessel segmentation is proposed to select the most useful patch to be annotated. In addition, all the rest of unannotated patches in the annotated images and the rest of the unannotated images are used to train the network. The performance of the proposed method was evaluated on two public datasets, DRIVE dataset of fundus images and ROSE-1 (SVC) dataset of OCTA images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea of addressing the problem of manual annotation of retinal vessels is interesting and relevant since public datasets of annotated retinal vessels is limited. The validation protocol is appropriate. Many metrics were computed to evaluate the performance of the proposed method. The results were compared with those of fully-supervised methods and semi-supervised learning methods. Ablation studies were performed to evaluate the effect of budget annotation, active sample selection and the latent Mixup on the segmentation results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

A relevant reference presenting a semi-supervised learning approach with global latent mixing is missing. The work seems to be inspired from ‘Semi-Supervised Medical Image Classification with Global Latent Mixing’ by Prashnna Kumar Gyawali et al. Many choices in the methodology are not justified. For the active sample selection the authors are estimating the uncertainty by computing the distance between the predicted probability and the median probability. The motivation of this choice is missing since there are many others methods for uncertainty quantification such as Monte-Carlo Dropout. The authors said that they compute the new augmented latent representation from their last convolutional layer. It is not clear if they mean the last layer of the UNet which is the sigmoid function or the last layer of the encoder and in this case how the shortcuts connexions of the UNet are considered. This choice should be motivated and evaluated in the ablation study. The results on the OCTA dataset are surprising since their method seems to perform better than a fully-supervised learning. In the ablation study the authors showed that the bigger the annotated ratio, the higher performance the model could achieve.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The pipeline of the proposed model in well described.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

There are many typos in the paper but the most important one is in the definition of Lamda in equation 2. It is stated that Lamda = (alpha, alpha) instead of Lamda = Beta (Alpha, Alpha). The Beta distribution is missing and is well defined in the missed reference indicated above.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method is not novel and most concepts in the methodology are similar to those described in the missed reference.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

They proposed a patch-level supervised method and introduced MixUp with certain changes.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. They provided a semi-supervised scheme for vessel segmentation.
2. Their ablation experiments are quite adequate.
3. The improvement of the method is reasonable.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The innovation of this method is limited, which is an extension of the current technology.
2. The segmentation effect in OCTA images is lower than the current level. (in detailed comments, [1])
3. This method is quite complicated, may not be necessary, and may not be better than the direct training patches using full supervision.
4. They did not solve the practical issues of vessel segmentation, and the effect of capillary segmentation is poor.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method of this paper is more troublesome, it is recommended that the author organize and publish the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Partially-Supervised Learning is an existed name. I suggest another expression, such as Patch-level Partially-Supervised.
2. I don’t know why not conduct full supervision directly on patches, which can also achieve excellent performance. Ex.:
  [1] Giarratano, Y., et al.: Automated segmentation of optical coherence tomography angiography images: Benchmark data and clinically relevant metrics. arXiv:1912.09978 [eess.IV]. https://doi.org/10.7488/ds/2729 (2020)
3. In my opinion, the insufficient training label is a difficulty but not the main challenge in the vessel segmentation task. Only a small number of labels are required for full supervision to get good results of vessel segmentation. However, a small number of labels may not perform well in disease images, such as DR, non-perfusion, etc. These diseases will change the shape of blood vessels and the quality of images. I recommend a dataset containing various diseases to clarify the applicability of the method: https://ieee-dataport.org/open-access/octa-500.
4. A large number of capillaries in the OCTA image have not yet been segmented, as shown in Fig.3 and Fig. 1 in the supplementary material. The author tends to analyze the improvement of the evaluation metrics but ignores the problems of segmentation results. I suggest that the author consider corresponding improvements to the method to overcome this main problem, refer to [1].
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The author does not specifically analyze and think about the challenges of blood vessel segmentation, but tends to apply and improve the current latest technology. Their method is troublesome for blood vessel segmentation and does not improve much.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The proposed method aims to produce accurate vessel segmentation in ocular images, while annotating as little data as possible. It does so by annotating certain image sections of certain images, using a Partially-Supervised Learning method, with an Active Learning Framework and Latent MixUp.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper shows a promising method for obtaining precise vessel segmentation results while requiring low levels of manual annotations.

The manuscript is well documented, the method well described, and it shows a good comparison with a proper selection of competing methods
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

It could be interesting to see the method performance in datasets that present more modern images. The DRIVE data set for example, contains images with a resolution of just 565x584, while modern cameras have provided images with resolutions of 3000x3000 pixels and higher for a decade now.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Good description of the implementation, including parameter values. No code is available, but the datasets are
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Overall the manuscript is detailed, while still being easy to read and engaging to the reader.

The proposed method is quite interesting, and it could really help with the always tedious issue of manually annotating images.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method is novel enough. The potential time savings it offers, while maintaining a high level of accuracy as competing methods, it’s really interesting.

Manuscript is quite detailed and well structured.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper received very mixed reviews. The authors should focus on replying the methodology innovation and experiment designing in rebuttal, as both R1 and R2 raised several issues on these two sections.
- R1 is critic raising the fact that the method is of limited novelty. The motivation of some choices are not justified, e.g., the estimation of the uncertainty by computing the distance between the predicted probability and the median probability, was not clearly stated. Moreover, some unclear statement needs improvement, such as the new augmented latent representation.
- R2 pointed out that the segmentation performance at capillary level is poor, please give some explanations. In addition, please also give a response why not conduct full supervision directly on patches.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

[Q] Novelty (R1, R2) [A] Patch-Level Partially-Supervised Learning: in this work, we mainly focus on using as few annotations as possible to produce satisfactory performance, since manually annotating masks is a laborious and tedious process. With only 5% annotations, our proposed model performance is comparable with the previous methods with full annotations. The novelty of this part is also appreciated and highlighted by R1 and R3. We believe that the main review comments of R2 are biased since he/she was just focusing on the performance of blood vessel segmentation only. He/she ignored our contributions in terms of reduction in annotation cost. We believe both the performance improvement and annotation cost reduction are the challenges of vessel segmentation.

Active Learning Framework: The novelty of this part mainly focuses on the usage of the active strategy to reduce the annotation cost. In [18], Simple Margin learns an SVM on the existing labeled data and chooses as the next instance to query the instance that comes closest to the hyperplane. Similarly, a model is trained on labeled data and chooses the next most uncertain patches, which are closest to the decision boundary. In particular, it computes the distance between the predicted probability and the median probability (p=0.5). The novelty of this part uses a simple but efficient sample selection way. We can also use other sample selection ways, such as using p=0.25, 0.75, entropy using output probability plog(1-p), or Monte-Carlo Dropout. We report the results: Dice MC Dropout 0.750 plog(1-p) 0.750 p=0.25 0.747 p=0.75 0.750 Ours 0.751

Latent MixUp: given the limited annotations, to use the unannotated patches and alleviate the need for annotated data, we design a latent MixUp to learn the augmented representation at the patch level. Compared with SSL work [1], one of the differences between it and partially-supervised learning is that there exist both known and unknown regions in one image. Our proposed MixUp could generate the augmented features at the patch level, while the similar MixUp in [1] generates the features at the image level.

[Q] Implementation Details (R1) [A] In latent MixUp, the last convolutional layer is the sigmoid function not the last layer of the encoder, since the features of this layer contain more high-level and semantic information than the previous layers. If using the last layer of the encoder, the latent MixUp uses both of the last layers of the encoder and decoder do the latent MixUp for the feature consistency. BS is 0.750. Ours is 0.751.

[Q] Performance at capillary level (R2) [A] In partially annotated images, the lack of annotated capillary vessels might worsen the performance at the capillary level. As shown in Tab 2, with the increasing annotated ratio, the increasing performance mainly comes from the capillary vessels. We will show the corresponding examples in the revision. In [2], they use adaptive thresholding as a baseline binarization procedure, a method that takes into account spatial variations in a specified neighborhood of the pixel. As R2 suggested, we use this binarization method over our prediction and show the results: Dice Ours 0.751 Ours + bi 0.752

[Q] Full Supervision. (R2) [A] Full supervision directly on patches used in [1] divides the images into 5 patches and each patch is under a fully supervised setting, which is the upper bound of our model. Actually, we also reported the results of baselines named ‘Label-Only’ in our manuscript, which directly uses the patch-level partial annotation to train the model. The contributions of our method are utilizing the unlabeled data with the help of the active strategy and latent MixUp. The experimental results demonstrate our proposed new framework.

[1] Semi-Supervised Medical Image Classification with Global Latent Mixing. [2] Automated segmentation of optical coherence tomography angiography images: Benchmark data and clinically relevant metrics.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors’ response properly addresses all the criticism, especially those by R1 and 2. I have read the paper twice, and I acknowledge the novelty of this work but lacks some technical details. In addition, the visualization results could be improved. I recommend the acceptance of this paper, but the related information should be incorporated into the final version of the paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors clarified the main novelty of this work is a new vessel segmentation method which achieves similar performance as SOTA with 5% manual annotations. The AC agrees with the authors that reducing the requirement of manual annotations is necessary and important for medical image segmentation tasks.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors’ response somehow (at least partially) addresses the technical novelty issue; however, the performance and the deep-thinking (the real challenge of retinal blood vessel segmentation, especially in real-world applications) issues raised by R2, are the main reasons for me to recommend rejecting this paper. The key challenge comes from the disease cases, good performance can be achieved on healthy/normal cases with very few samples. From the results in Tab.1, the order of these methods is expected to be FSL >PSL>SSL, because reduced information is used for these methods. Also, from Tab.2, the results are as expected, using AC is better, although the improvement is marginal. But the author may consider, manual annotation cost of different patches is different, diversity can improve the performance, especially on tiny vessels and lesion regions, this can be observed from the figure in Tab.2. The most critical problem of this paper is that it doesn’t touch the key challenge of the problem - disease cases. The OCT dataset doesn’t have eye disease cases, and the fundus data set has 7 DR early-stage cases, in which the vessels do not change much compared to normal cases. I do suggest the author play on the dataset suggested by R2(https://ieee-dataport.org/open-access/octa-500. Overall, the paper is self-contained; however, the clinical value is very limited.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

11

back to top

Partial-supervised Learning for Vessel Segmentation in Ocular Images