Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Wen Tang, Han Kang, Ying Cao, Pengxin Yu, Hu Han, Rongguo Zhang, Kuan Chen

Abstract

The Coronavirus Disease 2019 (COVID-19) pandemic has swept the whole world since 2019. Chest computed tomography (CT) plays an important role in clinical diagnosis, management and progression monitoring of COVID-19 patients. In order to decrease the cost of manual segmentation, weakly supervised segmentation methods, such as class activation maps (CAM) based methods, have been applied to achieve COVID-19-related lesion segmentation. Such methods could be used to localize the lesion preliminarily, but it is not precise enough to segment the lesion. In this paper, we propose a double weakly supervised segmentation method to achieve the segmentation of COVID-19 lesions on CT scans. A self-supervised equivalent attention mechanism with neighborhood affinity module is proposed for accurate segmentation. Multi-instance learning is adopted for annotations weaker than image-level. A simple pre-training process is also proved to be effective. We achieve a higher average Dice compared to Unet (0.782 vs 0.601) on COVID-19 lesion segmentation tasks. Codes in this paper will be available at https://github.com/TangWen920812/M-SEAM-NAM.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_25

SharedIt: https://rdcu.be/cyl8l

Link to the code repository

https://github.com/TangWen920812/M-SEAM-NAM

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed a weakly supervised algorithm for learning a neural network model to segment “COVID19 lesions” from CT images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This is a timely submission in the midst of the COVID19 pandemc.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1) The definition of “COVID19 lesions” is unclear. COVID19 may cause pneumonia, pulmonary edema, and many other lung diseases. When the authors talk about segmenting “COVID19 lesions” in the CT images, what are we actually segmenting? 2) The overall prersentation and writing of this paper are poor. For instance, many variables in the Methods section are not clearly defined; Figure 1 doesnt’ really help with understanding the proposed method; there are many grammatical errors. 3) In Table 1, what does dice coefficient mean in negative patients? 4) It’s really hard to digest Figurer 2. What are the ground truth contours?
Please rate the clarity and organization of this paper

Poor
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

No code is attached and the mehtods section is not enough to reproduce this work.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Improve the Methods section: 1) give a high level or algorithmc description of this proposed method; 2) improve Figure 1 and add more captions; 3) clearly every variable that shows in the Methods section.

Proofread and correct grammatical errors.

Define “COVID19 lesions”. Define “double weakly supervised”.

Justify the rationale of spliting the dataset into positive patients and negative patients.

Add captions for Figure 2.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Unclear definition of the problem and overall poor presentation of the methods as well as the experiments.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper
It is hard to single out a contribution of this paper. The authors propose a method called M-SEAM-NAM. M from multi-instance. SEAM from self-supervised Equivalent Attention Mechanism. And NAM from Neighborhood Affinity Module. This combination of techniques enables the segmentation of COVID-19 related lesions on CT axial images without using training masks and not even telling if there is the presence of a lesion in the image (but if there is at least one lesion in a set of images). This enables what they call a doubly weak supervised segmentation. Said that, in comparison to reference [18], the contributions of the paper are
- Use of a neighborhood affinity module (NAM) to enforce spatial correlation of labels
- Use of multiple instance learning to label few(er) slices per case
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Very complex network structure with different modules and a correspondingly complex cost function composed of six terms. Interesting combination of techniques justified by the task at hand.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of description of the baseline. How many samples was the unet trained on?
- Lack of statistical testing. The spread of the dice coefficient is fairly large. It is hard to gauge if the improvements of multi-instance are an artifact or not.
- The presentation could be improved or simplified.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Given the complexity of the paper, the community could benefit from access to the code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It would be interesting to see the combination M-SEAM (no NAM)
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

While SEAM is not new, it merits attention in the MICCAI community given its relatively simplicity and the good results it produces. The network attention module is interesting. It encodes a label regularization in the network. The multiple instance learning is also interesting.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposes to combine (1) multi-instance method (2) Self-supervised Equivalent Attention Mechanism (SEAM) (3) Neighborhood Affinity Module (NAM) for Weakly Supervised Segmentation of COVID-19 lesion.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1.The evaluation result is strong. The Table 1 shows the proposed method outperforms baseline methods and the proposed components postively contribute to the final result.
1. The writing quality and the figures are good.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The authors perform experiments on a private COVID-19 dataset. To demostrate the effectiveness of the proposed method, the authors can perform experiments on other datasets.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors state that codes in this paper will be available online soon. But I found they are using a private COVID-19 dataset.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

In figure 1, I found there are ‘New Better CAM’ and ‘Improved CAM’. I think they are the same thing and should be unified to improve readability.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The major factors are the experiment results.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes to combine multi-instance, self-supervised equivalent attention mechanism, and neighborhood affinity module for weakly supervised segmentation of COVID-19. The algorithm (combination with multi-instance and NAM) is new, and the evaluation results are detailed. Why use “doubly” not “double” in the title.

The strength of the paper is that such a combination is new, but the method looks complicated and some part of the paper writing is not clear. For example, most questions about reviewer 1 might be due to writing or explanation. Suggest the authors make the writing clear, especially by addressing all the weakness and suggested comments.

You have so many losses; need justification. This paper could have been made clearer, and the methodogy contribution is quite of interest.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

Thanks for all the reviewers’ helpful questions and suggestions. We apologize for the misunderstanding writing and we are working on improving the fluency and readability of the whole manuscript according to the comments. Please find our point-to-point response below.

Reviewer #1 Definition of COVID-19 lesions COVID-19 lesions in the manuscript refer to the imaging features of COVID-19 pneumonia including multiple small patchy shadows, interstitial changes appear, multiple ground-glass shadows, infiltrates shadows, and pulmonary consolidation.

Writing of this paper We are working on revising the manuscript and improving the overall presentation. The final version would be reviewed by a native speaker to improve the language quality and avoid grammatical errors.

Dice coefficient in negative patients In negative patients, if no lesions are segmented, the dice coefficient would be 1; otherwise, 0.

Clarity of Figure 2 Figure 2 would be modified, and we will zoom in the lesion regions in Figure 2 to show the ground truth contours clearly.

Improve the Methods section The Methods part would be modified accordingly to better describe the algorithms and network structure. We will give a high level description of our method at the beginning of Method section. We will also improve corresponding figures and clearly define every variable.

Define “double weakly supervised” “Weakly supervised” usually means using image-wise classification label to do segmentation. In our paper, as we use multi-instance learning, several slices could share one label which indicate a more weakly supervised than usual.

Why split dataset into positive and negative We split the dataset to show more detailed performance of each method, as positive cases could be used to show the model performance of lesion segmentation and sensitivity, while negative cases could reflect the model specificity.

Reviewer #2 Lack of description of baseline. The baseline we used in the paper is the SEAM method using ImageNet pretrained model (SEAM from original paper). We will add this description and renew Table 1 in the final version.

Samples number for Unet training The Unet is trained on all 8309 positive slices and 8309 randomly selected negative slices in the training set. We indeed describe this in 3.1 data description and we will make it clearly in final version.

Lack of statistical testing We would use Wilcoxon signed rank test to perform statistical tests and corresponding results would be added into Table 1 if allowed. (i.e. M-SEAM-NAM vs. SEAM-NAM on positive patients, p=0.00033)

Improve presentation A native speaker would help us on improving the language and overall presentation.

The combination M-SEAM We agree that the combination M-SEAM may bring some interesting points and we would perform related experiments in the future.

Reviewer #3 Perform experiments on other datasets Validating our methods on other datasets such as BraTs will be included in our future work.

Readability of Figure 1 We would unify terms used in Figure 1 to improve readability.

Meta-reviewer We have replied reviewers’ comments point-by-point. As for the language, a native speaker would work on improving the overall presentation and readability. For the algorithm description, we would also reorganize the Methods part, justify loss functions and define variables to better illustrate the algorithms. In detail, we would change the “doubly” in the title to “double” according to your suggestion. Seven loss functions are used in the manuscript. The classification loss is used in all network. ERb, ERc, ECR are used in the baseline SEAM methods as described in the original paper. In NAM, three loss functions, NBS, NFS and NFBD are proposed. NBS is used to measure the similarity between background and background, and NFS is used for similarity between foreground and foreground. NFBD is used for measuring of the difference between foreground and background.

back to top

M-SEAM-NAM: Multi-instance Self-supervised Equivalent Attention Mechanism with Neighborhood Affinity Module for Double Weakly Supervised Segmentation of COVID-19