Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Jinghan Huang, Yiqing Shen, Dinggang Shen, Jing Ke

Abstract

Nuclei segmentation is an indispensable prerequisite for microscope image analyses. However, a successful instance segmentation result is still challenging attributable to the ubiquitous presence of clustered nuclei, as well as the morphological variation among dissimilar phenotype of cells. In this paper, a novel contour-aware 2.5-path decoder network (CA\textsuperscript{2.5}-Net) is proposed for nuclei segmentation in microscope images. In contrast to the regular two-path decoders in many previous contour-aware networks, a shared decoder path is employed when the clustered-edge problem is severe. The range of recognizability difficulty generated by the extra half path also serves as a natural proxy to construct a curriculum-learning model, where training samples are sequenced for a better segmentation performance. Last, in this paper, we publicize two well-annotated privately-owned datasets covering a wide range of difficulty in the nuclei segmentation task, comprising 500 confocal microscopy image patches of deep-sea archaea and drosophila embryos obtained from 2013 to 2020. In the benchmark test of these two own datasets and one open-source set, our model outperforms the state-of-the-art nuclei segmentation approaches by a large margin, evaluated by the metrics of Average Jaccard Index and Dice score. Empirically, the proposed structure triples the training convergence speed in comparison with the competing CIA-net and BRP-net structures in nuclei segmentation.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_43

SharedIt: https://rdcu.be/cymaV

Link to the code repository

N/A

Link to the dataset(s)

https://www.kaggle.com/hjh415/ca25net

Reviews

Review #1

Please describe the contribution of the paper

This paper presents a deep network to segment nuclei that sometimes are touching. The main key ideas are to combine multi-task learning and curriculum learning.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed network has the potential to work well in practice.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1: The method part is written too cumbersome. Eq. 1 to 3 can be expressed by a few words. It is not necessary to use such a space to say such things. I also would like to suggest to revise Eq. 4 and 5. They are too cumbersome while expressing just a few pieces of information.

2: The experimental part is somewhat weak, however not serious. It would be better to evaluate the proposed methods on images with different segmentation difficulties, as in the used dataset some objects are touching, so it may be reasonable to evaluate to what touching extent and also how many objects touching, the proposed method can address.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

very likely
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

1: The method part can be written more clear by removing some mathematical equations. 2: In the ablation study, it may be better to check the role of the curriculum learning.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method itself is worth to reading, but the writing and evaluation parts are perhaps not ready to publish.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper proposes a network for cell instance segmentation. The proposed network is a Unet-like architecture with dense skip connections and multiple decoders trained in a multiple-task manner. The multiple task includes standard segmentation head, boundary prediction head and the overlapping boundary head. The network is trained using curriculum training. In addition, this paper claims to release two microscopy image datasets upon paper acceptance. The experimental results seems promising.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

N/A
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The idea of dense skip connection, multiple task training, and training on overlapping boundary for cell segmentation and curriculum learning, and the truncated smooth loss for cell segmentation are all not new. The novelty is weak.
2. The presentation and organization of the paper are not clear. In Figure 1, it is unclear how this sequence-aware instruction module is operated. What ground truth are used to train the three output heads? There are four paths, but it is unclear how they are connected to the output. For example, DSCM for shared edge decoder is supposed to connect to the shared boundary prediction head, but these green blocks are not connected to the shared boundary ground truth.
3. The released dataset does not look special. It looks like many existing microscopy cell datasets.
Please rate the clarity and organization of this paper

Poor
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

May be reproducible. The authors claim to release data and code upon acceptance.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The idea of dense skip connection, multiple task training, and training on overlapping boundary for cell segmentation and curriculum learning, and the truncated smooth loss for cell segmentation are all not new. The novelty is weak.
2. The presentation and organization of the paper are not clear. In Figure 1, it is unclear how this sequence-aware instruction module is operated. What ground truth are used to train the three output heads? There are four paths, but it is unclear how they are connected to the output. For example, DSCM for shared edge decoder is supposed to connect to the shared boundary prediction head, but these green blocks are not connected to the shared boundary ground truth.
3. The released dataset does not look special. It looks like many existing microscopy cell datasets.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. The idea of dense skip connection, multiple task training, and training on overlapping boundary for cell segmentation and curriculum learning, and the truncated smooth loss for cell segmentation are all not new. The novelty is weak.
2. The presentation and organization of the paper are not clear. In Figure 1, it is unclear how this sequence-aware instruction module is operated. What ground truth are used to train the three output heads? There are four paths, but it is unclear how they are connected to the output. For example, DSCM for shared edge decoder is supposed to connect to the shared boundary prediction head, but these green blocks are not connected to the shared boundary ground truth.
3. The released dataset does not look special. It looks like many existing microscopy cell datasets.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper proposed a novel approach for nuclei instance segmentation, especially focusing on the difficulty of segmenting clustered nuclei. The model has two decoders, one for the normal semantic segmentation task, another for the segmentation of normal edge mask and clustered-edge mask. This work is based on the previous contour-aware networks like DCAN and CIA-Net and make their own innovative improvements. Moreover, this paper introduced curriculum learning into the training of the network, the segmentation difficulty is quantified mainly by the brightness variation. Dense skip connection is also applied into the network to improve the connection between encoder and decoder in U-Net like networks. The experiments were done in a dataset consists of 524 images, the results showed that the proposed approach achieved SOTA performance compared with previous nuclei segmentation methods. Lastly, this work also public an annotated nuclei instance segmentation microscope dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The idea of decomposing instance segmentation task into three sub-tasks is quite innovative. The introduction of clustered-edge mask and normal edge mask segmentation will improve the segmentation of clustered nuclei. The description of contributions is clear and detailed.
2. The paper is well organized and easy to follow.
3. The overall design of experiments (performance comparison and ablation study) is good, briefly explain the effectiveness of each main component of the approach.
4. This paper also public an annotated nuclei instance segmentation microscope dataset, which will benefiting further research of nuclei instance segmentation problem.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The process of how to combine the semantic mask, normal-edge mask and clustered-edge mask into final instance mask is missing.
2. Some ablation study may be missing, for instance, the curriculum learning has no ablation study.
3. The number of methods for comparison is not enough, especially, they didn’t compare with some newly published instance segmentation networks such as Hover-Net[1], Triple U-net[2]. Furthermore, as an instance segmentation model, they didn’t compare with some instance segmentation works like TensorMask, Mask Scoring RCNN.
4. The evaluation metrics are not enough for instance segmentation. Some metrics like PQ (Panoptic Quality) which was used in previous studies should be included.
[1]. Graham, Simon, et al. “Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images.” Medical Image Analysis 58 (2019): 101563. [2]. Zhao, Bingchao, et al. “Triple U-net: Hematoxylin-aware nuclei segmentation with progressive dense feature aggregation.” Medical Image Analysis 65 (2020): 101786.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Very good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please see the weaknesses section.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The concept of decomposing instance segmentation task into three sub-tasks, and the introduction of clustered-edge mask and normal edge mask segmentation.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper proposes a nuclei segmentation framework with some new bench mark datasets to be shared with the community. The reviewers have a few questions/suggestions regarding the paper:
1. The proposed framework consists of some known components such as dense skip connections, multi-task learning, and curriculum learning. The contribution over the existing techniques needs to be clear explained.
2. Some details of the framework are missing. For example, how are the multi-heads in the multi-task learning fused in the training loss and how are they used in the testing to generate the final results?
3. Ablation studies are missing such as: how do individual heads in the multi-task learning contribute to the segmentation? How effectiveness is the curriculum learning?
4. In the experiment, it is expected to show more cases of the various challenges introduced by the paper at the beginning such as overlapping/touching nuclei.
5. More specifications on the new datasets and their differences compared to the existing ones are expected. There are some other issues/comments raised by the reviewers. Please consider to address in the rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Author Feedback

To Rev#1 1)Math Eq Eq.1 defines the instance-level one-single nuclei mask, indispensable to the later define of difficulty score in curriculum learning. Eq2 and Eq3 differentiate well the normal-edge ground truth from that of clustered-edge, which are the core concepts. A presentation by equations better interprets than literate description. 2)Evaluate segmentation difficulty We design a metric namely the difficulty score to evaluate the segmentation difficulty in Eq4-5. Fig5 shows the most challenging scenarios of severely clustered nuclei, highlighted by the red boxes, which is also the strength of our framework. In the less difficult cases, we outperformed the competing methods, in the same pictures. Similar results are too many to fit the limited pages. 3)Ablation study of curriculum learning (Also to Rev#3) Ablation study of curriculum learning is in the last two rows of Tab1, with an observable performance gap. To Rev#2: 1)Framework novelty Our method is not a straight combination of multiple task training, dense skip connection and curriculum learning. Instead, two essential contributions are made. The first is the decomposition of a tough instance segmentation task into sequenced subtasks, in which the clustered-edge segmentation task promisingly tackle the severely clustered-nuclei problem. Second, instead of the routine random shuffling, it is the first attempt at the fine-granularity quantification of nuclei-segmentation difficulty by curriculum learning, and the outcome of sequenced training directly leads to a large margin in performance improvement. 2)Dataset novelty Curated from a large private dataset of 10,000 fluorescence images, this dataset distinguishes itself for its severity in nuclei clustering and variety in segmentation difficulty, also, it is firstly publicized in this paper. The collected drosophila nuclei are highly variated in brightness, challenging for semantic segmentation while the deep-sea archaea images are exceptionally complex in clustering. Annotated by three biologists, it is valuable in a comprehensive evaluation of future segmentation models. Additionally, its variety is applicable to test a curriculum learning mode. 3)Presentation clarity

As already presented in Sec. 2.2, in a SAM module, the output of the shared-edge decoder and normal-edge decoder are applied with a 1×1 convolutional operation respectively, and followed by a weighted aggregation. 2. The ground truth for each task is elaborated in Sec. 2.1 entitled “Ground Truth Preparation”. 3. The suggested connection of green (olive in paper) box and shared boundary is not applicable. The normal-edge (olive box) is segmented prior to the clustered-edge segmentation (grass-green) box. The advised connection to the olive box would break down the hierarchized tasks, and the explanation is elaborated in the last paragraph of Sec. 2.2. To Rev#3: 1)Final mask (Also to Meta.Q2) First, the normal-edge mask and clustered-edge mask are merged straightforwardly. Second, the edges are sharpened to a precise boundary. Finally, a subtraction of the precisely-located-edge mask from the semantic mask is performed. 2)More experiments We have compared with Mask Scoring RCNN but shows a lower overall performance, will be presented. Triple U-Net is for H&E as it leverages RGB channels to generate an H-component image, and more importantly, the clustered edge is not segmented. To Meta: 1) Details of the framework (Q2) The composition of the final training loss is in the last sentence in Sec.4.1 as “The overall loss…” 2)Ablation studies of multi-heads As the effect of normal edge for nuclei segmentation has been universally verified in the literature, we leave out the results of that part to pinpoint the more challenging clustered-edge issue in the final experiment. The observable performance gap between CA2.5-Net and CA2.0-net has been shown in Tab1, which is the ablation study for the multi-task learning. 3)The rest is answered above.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

According to the authors’ feedback, the AC would suggest the authors to: clearly explain and validate the claimed contributions in the paper; clearly define a loss function with its related terms in equations; and summarize ablation studies in a readable format (CA^2.0-Net, CANS^2.5-Net, etc., in Table 1, are only familiar to authors).
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

13

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes a method for nuclei segmentation that combines multi-task learning and curriculum learning and a new dataset to study the task. The rebuttal clarifies the contributions over existing techniques to some extent, and it provides additional technical and experimental details. Although the rebuttal argues for the proposed metric, the final version should include a standard segmentation evaluation methodology for reference, as suggested by R3. Overall, considering the method and the annotated dataset, the paper provides an adequate contribution to the community.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper explores clustered-edge nuclei segmentation with two privately benchmark datasets to be shared with the research community. The decomposing instance segmentation into sequence sub-tasks improved performance over other methods. Given the performance improvement and dataset sharing for advancing the direction of clustered instance segmentation, a decision of accept is recommended.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

back to top

CA^{2.5}-Net Nuclei Segmentation Framework with a Microscopy Cell Benchmark Collection