Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Ruoyu Guo, Maurice Pagnucco, Yang Song

Abstract

Deep convolutional neural networks have been highly effective in segmentation tasks. However, high performance often requires large datasets with high-quality annotations, especially for segmentation, which requires precise pixel-wise labelling. The difficulty of generating high-quality datasets often constrains the improvement of research in such areas. To alleviate this issue, we propose a weakly supervised learning method for nuclei segmentation that only requires annotation of the nuclear centroid. To train the segmentation model with point annotations, we first generate boundary and superpixel-based masks as pseudo ground truth labels to train a segmentation network that is enhanced by a mask-guided attention auxiliary network. Then to further improve the accuracy of supervision, we apply Confident Learning to correct the pseudo labels at the pixel-level for a refined training. Our method shows highly competitive performance of cell nuclei segmentation in histopathology images on two public datasets. Our code is available at: https://github.com/RuoyuGuo/MaskGA Net.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87196-3_43

SharedIt: https://rdcu.be/cyl2U

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

In this work, a weakly-supervised learning framework is proposed for nuclei segmentation based on point annotations, in which a mask-guided attention network is designed to reduce the distraction of noisy labels. Besides, a two-stage training strategy is proposed to refine the training of the segmentation network.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea of the mask-guided attention module is interesting, it is relatively novel. By combining the point annotations and the Voronoi labels as an extra guidance, an attention mask is generated to correct the noisy label.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I really appreciate the idea of combining the Voronoi label and the point annotations as a new guidance. It is interesting. While some concerns about this paper are as follows:

(1) The novelty of this work is limited. The authors claim that they design a weakly-supervised learning framework with label correction for nuclei segmentation based on point annotations, and introduce an attention network to reduce the distraction of noisy labels. First, the label correction in the proposed framework is performed by the Confident Learning (CL) algorithm, which is proposed in [10] and also used in some previous works such as [19]. Second, the Voronoi label used in the attention network is also widely-studied in some previous works, e.g., [11,12,17], etc.

(2) In the last sentence of Page 4, ‘‘since the feature maps from each decoder block have different scales and not all levels of features contribute equally to the final output, we equip an aggregation module to the encoder-decoder network.’’ and in this aggregation module, “the feature maps from three channel attention layers are merged by element-wise addition and fed through an upsampling operation to obtain the final predicted probability maps.” As the feature maps are merged by element-wise addition without importance weights, how does the proposed feature aggregation module deal with the unequal feature contribution problem?

(3) For the generated label y_{v}, “positive labels are the point annotations, negative labels are the Voronoi boundaries, and other pixels remain unlabelled.” Based on this annotation rule, y_{v} only has a very small amount of pixel-level labels, and accordingly, when training the attention network with Eq.(1), only a very small number of pixels in the generated attention map are used to optimize the attention network, and most pixels are ignored. In this way, how can we guarantee that the attention network can be well trained and generate good attention map?

(4) When training the proposed framework, data augmentation is used. Is the same data augmentation used for the training of all the other state-of-the-arts?

(5) Better performance is obtained by the proposed framework on the MoNuSeg dataset, while the performance of the proposed framework is worse than other methods on the TNBC dataset. The authors explain that this is because the baseline (Main in Table 2) model is not good. Actually, this baseline is designed by this work, in other words, this designed baseline can not perform well on this task compared to other state-of-the-arts. This makes the proposed framework not so convincing.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Based on the reproducibility checklist filled out by the authors, I think this work is reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

(1) More details and explanations should be added for better clarity. Please refer to item (2), (3) and (4) in the weaknesses section for more details.

(2) Some typos: in the caption of Fig.1, weight -> width; in the first paragraph of section “Methods”, refined -> refine; in the first paragraph of subsection 2.3, maks -> mask
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea of the mask-guided attention proposed in this work is interesting. However, the novelty of this work is slightly limited, and the experiment results are not so convincing. Based on these, I think this work still needs some improvements.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This propose a novel method that can not only mitigate the heavy dependence of precise annotations in nuclei segmentation but also correct the noisy labelling in the generated pseudo ground truth. This method has achieved good performance on two data sets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This article proposes a novel weakly supervised segmentation algorithm.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The performance improvement of the proposed method is weak.
2. Can auther explain the inconsistent results of Yoo et al. [17] in different tables.
3. The author claims that the proposed method has good noise immunity. From Table 2, the proposed method is not much different from the noise immunity of [17].
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

No comments.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

As mentioned above.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I am familiar with this field.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper propose a weakly supervised nuclei segmentation method using point annotations (nuclei centroid). It generate boundary and pseudo based masks to guide and train the network. Since the pseudo masks are probably noisy, the authors deploy Confident Learning to correct pseudo labels for a refined training and better convergence. The authors cooperated state of the art strategies to generate pixel-wise ground truth from point annotation. First they use Euclidean distance transformation to generate Voronoi boundary in which this information will be used to correct the network prediction. Second, they used SLIC superpixel algorithm to generate another type of masks that will be used to train the network using Binary Cross Entropy Function. The method is also accompanied with attention module and feature aggregation for highlighting useful information from the output of each decoder. This work is validated on 2 public datasets and shown an enhanced prediction.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This paper propose a novel weak segmentation method using point annotation. The main novelty lies in the way the authors cooperated different state of the art modules to handle noisy pseudo labels and to transform noisy training into a refined one. Novelty is also seen at the data level, where the authors presented a way to generate pseudo labels from point annotations by relying on state of the art work based on super pixel and by generating as well additional masks using point annotation and Voronoi boundary. The later information is integrated at the level of a loss function (partial pixel-wise) to recognize noisy labels in the output of the attention module, yielding to a refined training and guiding process. The authors also shown how to utilise Confident learning to identify Noisy Labels in super-pixel ground-truth. Hence, we can see that this paper re-visited such Confident Learning approach and showed how to fully exploit it, which can be an assist for other applications and tasks. The clinical scenario presented in this paper is valid and important.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

At the level of experimental setting, it is important to show the performance gain not only based on the mIoU % but also in term of estimating the weak annotation effort versus full annotation effort to strengthen the paper contribution. Also it is important to show some failing cases in the experimental setting and emphasizing on why the attention module or the integrated loss functions fail to cop with. Future directions should be emphasized.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1. The paper clearly describe the proposed method and all the hyper-parameters.
2. The figures related to the architecture and its modules are clear but the captions should be refined to describe better the content of the images.
3. The code will be available online as the author mentioned in their paper.
4. The datasets are described along the experimental protocol clearly and cited.
5. The loss functions are clear and readable.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. If accepted, the authors are encouraged to extend the part related to Confident Learning as a way to prompt this idea and to go more in some details regarding the attention module and may be its formulation to make it clearer and self contain paper.
2. The clinical usability to other applications and task can also be discussed such that the authors can highlight on some modules in their paper and how other work can benefit from it.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. This paper is pleasant to read and easy to follow.
2. The related work is clearly described and the motive behind the work is well highlighted.
3. The method is novel in the way it combines different modules from state of the arts to perform weak segmentation over a challenging task.
4. The experimental setting and comparison is well conducted and shown improvement.
5. The whole idea of the paper on using attention with some extra information that are generated using superpixel and distance transformation to guide and refine the network is novel.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes a weakly supervised network learning, using point annotations, for nuclei segmentation. The reviewers have some questions/concerns about the paper, mainly including:
1. The improvement over the other point-annotation method is small (Table 1). In one of the two datasets, the performance is even worse. The current best performance is still far away from the perfect, and some failure case studies are expected, with some analysis on why the attention modules and label propagation fails.
2. The difference and new contributions of components of the proposed segmentation approach need to be clearly described, such as confident learning [10,19] and Voronoi label [11,12,17] .
3. Performing some quantitative and qualitative analysis on the label noise and how they are overcome are expected. There are quite a few detailed questions raised by reviewers. Please consider to address them in the rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We sincerely thank all reviewers and the Area Chair for your time and comments.

AC, R1-5, R3-1: Performance improvement As noted by R1, our results on TBNC were limited by the baseline model while the proposed label refinement (attention + CL) largely improves the results (Table 2). In fact, we have conducted further investigation after submitting our paper and found that the issue was due to our initial image normalisation (dividing by 255) before applying the segmentation model. By changing the normalisation based on the actual intensity range, IoU increases 0.02-0.03 for TNBC in all experiments. This does not affect the results on MoNuSeg. In addition, [12] involves 4 iterations of refinement and hence can be less efficient than our method. We will provide updated results in the final version.

AC, R1-1: Novelty regarding CL and Voronoi labels Our contribution with CL is that we adopted the method with a different objective, which has not been investigated before. While CL has been used in semantic segmentation in chest X-ray images with a subset of the images affected by label noise and noisy labels generated by dilation, erosion and distortion [19], we apply CL for weakly-supervised segmentation with a significant degree of label noise (superpixels) on all images. In fact, our contributions are mainly to the overall design of the weakly-supervised segmentation model, and the label refinement method with an attention model and CL (last paragraph, page 2). CL alone could not provide very good label refinement and we mitigate this issue with our proposed attention network by: (1) identifying true labels in superpixels to generate the attention map; and, (2) adding constraints at the object level. Voronoi labels have indeed been widely used. However, we use it in a different way by considering Voronoi labels to be true labels, complemented by superpixel labels that provide pixel-wise labels but are inaccurate. This combination of labels enables us to design a different weakly-supervised segmentation network with a different focus on label refinement.

AC: Evaluation of label noise Table 2 shows the comparison of segmentation performance with initial weak labels and after refinement. In addition, we have now performed more analysis by measuring the F1-score of weak labels of the nuclei against real pixel-wise nuclei masks. Initially, the F1-score is 0.73 on MoNuSeg and 0.69 on TNBC; and with Attention + CL, the F1-score increases to 0.83 and 0.76. We will add these results and revise Fig.3 to include a qualitative analysis of label noise in the final version.

R1-2: Feature aggregation Apologies for the imprecise description. Three channel attention blocks are attached after each decoder to weight feature maps before addition, which is followed by two Conv2d layers.

R1-3: Attention map The amount of Voronoi labels is indeed small and hence we design the superpixel labels to offset this issue. On the other hand, the Voronoi labels are highly accurate and can provide reliable ground truth for learning attention maps.

R1-4: Data augmentation Yes, we used the same process.

R3-2: Results of [17] Results in Table 1 were copied from [17]. As the source code is not available and in order to conduct ablation studies, we reimplemented [17] and obtained the results in Table 2. The inconsistency, however, does not affect the purpose of Table 2, and again, one main reason for the inconsistency on TNBC was our image normalisation step, which will be corrected in the final version.

R3-3: Noise immunity We acknowledge that the difference in noise immunity is not much. In Table 2, we mainly wanted to show that we could obtain similar results to [17] without including CL.

R4: Suggestions on comparing annotation efforts, analysis of failure cases, future directions, more method details, and discussion on clinical applications are greatly appreciated and will be addressed in the revised version of this paper.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Based on the reviews and authors’ feedback, some revisions are needed in the paper . For example, the inconsistency of the results in Tables 1 and 2. The justification of the minor improvement, and the differences with the related works.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The idea of using CL for weakly supervised segmentation task (using superpixel labels as noisy label) is a very interesting approach. As far as I know, it is novel (even though CL itself has been proposed for learning with label noise). The rebuttal addressed the concerns regarding novelty, experimental performance, and baselines well. Thus I recommend acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper received one PA(6), one A(7), and one BR(5) by the reviewers. The meta-review asked the authors to comment on the noncompetitive results, failure cases and analysis of the attention modules and label propagation. The meta-review also asked to clarify the contributions and to run qualitative and quantitative analysis on the label noise. The authors replied about the result, mentioning that they fixed the normalisation to improve the result. They also clarified the novelty issue and the label noise evaluation. The rebuttal also provided brief comments on the attention module. I believe the authors addressed most of the issues form the reviews, so I am recommending the paper to be accepted.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

back to top

Learning with Noise: Mask-guided Attention Model for Weakly Supervised Nuclei Segmentation