Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Yanwen Li, Luyang Luo, Huangjing Lin, Hao Chen, Pheng-Ann Heng

Abstract

The novel coronavirus disease 2019 (COVID-19) characterized by atypical pneumonia has caused millions of deaths worldwide. Automatically segmenting lesions from chest Computed Tomography (CT) is a promising way to assist doctors in COVID-19 screening, treatment planning, and follow-up monitoring. However, voxel-wise annotations are extremely expert-demanding and scarce, especially when it comes to novel diseases, while an abundance of unlabeled data could be available. To tackle the challenge of limited annotations, in this paper, we propose an uncertainty-guided dual-consistency learning network (UDC-Net) for semi-supervised COVID-19 lesion segmentation from CT images. Specifically, we present a dual-consistency learning scheme that simultaneously imposes image transformation equivalence and feature perturbation invariance to effectively harness the knowledge from unlabeled data. We then quantify the segmentation uncertainty in two forms and employ them together to guide the consistency regularization for more reliable unsupervised learning. Extensive experiments showed that our proposed UDC-Net improves the fully supervised method by 6.3% in Dice and outperforms other competitive semi-supervised approaches by significant margins, demonstrating high potential in real-world clinical practice.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87196-3_19

SharedIt: https://rdcu.be/cyl1J

Link to the code repository

https://github.com/poiuohke/UDC-Net

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors propose a new method for semisupervised segmentation adopting three key algorithmic ideas: consistency to image transformations, consistency to feature maps transformation, pixel-level uncertainty to reweight supervised part of the loss.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- solid motivation
- a very reasonable and nontrivial combination several known ideas
- excellent validation of the proposed method
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- very low reproducibility
- some important details are missing
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility is poor. There are several open COVID-19 segmentation datasets, the author’s motivation to ignore them is unclear. Here is the list of examples:

(1) Mosmed-1110. It seems to be an excellent dataset for validating semisupervised methods as it contains 50 CTs with segmentation masks and 1000+ without masks. https://github.com/neuro-ml/COVID-19-Triage

(2) Covid segmentation challenge, a MICCAI endorsed event (!). It’s quite large, so some GT masks can be removed in order to evaluate the method. https://covid-segmentation.grand-challenge.org

(3) Medseg - a small but famous dataset. Maybe it can be used as an external test set in a journal version of the paper. http://medicalsegmentation.com/covid19/
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The most important suggestion is to to improve reproducibility by sharing the code and using open datasets in addition to your private one. See the suggestion above, at least datasets 1 and 2 seems to be 100% relevant.
2. Perturbations of feature map are not described. Please specify these transformation or write explicitly that you use the same approach as [21]
2b. It seems that FT-only version of your network is equivalent to CCT [21] (suggesting the answer to the previous comment).
1. To better estimate the real difference between quality metrics of different methods, mean and std values for several runs must be reported.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- an impressive combination of methods for semisupervised learning
- a well-written paper
- solid design of evaluation
- poor reproducibility: both data and code are private
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper tackles the semi-supervised semantic segmentation problem on the Covid-19 CT images. It improves upon the Cross-Consistency Training model (Ouali et al., Cvpr 2020) by enforcing consistency with respect to both image transformations and feature perturbations. It further uses uncertainty estimation to improve the segmentation prediction accuracy.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper compares with a comprehensive list of baselines, and achieves better mean results.
- The proposed dual consistency and uncertainty estimation both seem helpful in terms of improving the accuracy of the predictions.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The aleatoric uncertainty mentioned in Section 2.2 seems to be epistemic. The described method is similar to MC dropout, where the injected perturbations on the input to each decoder can be viewed as the “dropout features.” Then the variance of the outputs from the decoder ensemble is similar to the variance of the Monte Carlo predictions, which quantifies model uncertainty. In other words, I think Eqs. 3 & 4 are two ways to compute epistemic uncertainty.
- In Table 1, it is unclear why authors only show IC + FC + UE (row 6), but not IC + FC + UA?
- None of the results show statistical significance.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors mentioned that they will release the source code, but I did not find any code in the supplementary; thus, it is not entirely clear if the results can be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I think the proposed improvements upon CCT are helpful, but hope the authors can clarify their definition of aleatoric uncertainty, and show statistical significance of all results (or at least the standard deviation).
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

My score is mostly motivated by the good evaluation results.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This work imposes both image transformation equivalence and feature perturbation invariance for semi-supervised COVID-19 lesion segmentation. Moreover, both epistemic uncertainty and aleatoric uncertainty are employed to guide the consistency regularization, outperforming other semi-supervised methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

(1) The paper is well-written; the methodology part is clear and easy to follow.

(2) The paper proposes a dual uncertainty quantification method to measure the epistemic uncertainty and the aleatoric uncertainty for robust learning. These two uncertainties are proven to be complementary.

(3) The experiments are well designed, and the dataset of COVID-19 lesion is new and worthy to be explored.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) The contributions of the work are limited. Both image transformation equivalence and feature perturbation invariance are explored for semi-supervised learning in previous work [1-2]. The paper is like to combine the two consistency for a new application in COVID-19 lesion segmentation.

(2) Latest SSL methods such as DUWM [3] which also discusses dual uncertainty could be listed for comparison.

(3) From ablation studies, there is no comparison of the effectiveness of UE and UA, e.g., how about IC+FC+UA.

Reference [1] Li, X., Yu, L., Chen, H., Fu, C.W., Xing, L., Heng, P.A.: Transformation-consistent self-ensembling model for semi-supervised medical image segmentation. IEEE TNNLS (2020) [2] Ouali, Y., Hudelot, C., Tami, M.: Semi-supervised semantic segmentation with cross-consistency training. In: CVPR. pp. 12674{12684 (2020) [3] Wang Y, Zhang Y, Tian J, et al. Double-Uncertainty Weighted Method for Semi-supervised Learning[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2020: 542-551.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The methodology is clear and can be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

(1) More related work about uncertainty estimation should be included.

(2) More methods should be listed for comparison.

(3) From page 6, the two thresholds used to filter out uncertain samples are 0.34 and 0.12 set in advance and the paper fails to explain how to set the thresholds.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although image consistency and feature consistency are proposed by earlier work, the paper considers the two consistency together and propose an uncertainty quantification for further guiding the consistency regularization. Besides, the application of semi-supervised learning in COVID-19 lesion segmentation is valuable for the community.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors combine several existing methods for semi-supervised segmentation into a new one and apply it to a new application for COVID-19 lesion segmentation. All the reviewers appreciate the reasonability of the method and the better performance compared with several state-of-the-art semi-supervised learning methods. Some details of the implementation are not clear, but the authors promised to release the code. Some minor points should be clarified, such as discussion about the uncertainty estimation method and the result of “IC+FC+UA”.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

We thank the reviewers for appreciating the contribution of our paper and constructive comments for potential improvements. We will fully address the concerns raised by the reviewers in the final version.

back to top

Dual-Consistency Semi-Supervised Learning with Uncertainty Quantification for COVID-19 Lesion Segmentation from CT Images