Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Zipei Zhao, Fengqian Pang, Zhiwen Liu, Chuyang Ye

Abstract

Cell detection in histopathology images is of great value in clinical practice. Convolutional neural networks (CNNs) have been applied to cell detection to improve the detection accuracy, where cell annotations are required for network training. However, due to the variety and large number of cells, complete annotations that include every cell of interest in the training images can be challenging. Usually, incomplete annotations can be achieved, where positive labeling results are carefully examined to ensure their reliability but there can be other positive instances, i.e., cells of interest, that are not included in the annotations. This annotation strategy leads to a lack of knowledge about true negative samples. Most existing methods simply treat instances that are not labeled as positive as truly negative during network training, which can adversely affect the network performance. In this work, to address the problem of incomplete annotations, we formulate the training of detection networks as a positive-unlabeled learning problem. Specifically, the classification loss in network training is revised to take into account incomplete annotations, where the terms corresponding to negative samples are approximated with the true positive samples and the other samples of which the labels are unknown. To evaluate the proposed method, experiments were performed on a publicly available dataset for mitosis detection in breast cancer cells, and the experimental results show that our method improves the performance of cell detection given incomplete annotations for training.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_49

SharedIt: https://rdcu.be/cyma1

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

A positive-unlabeled loss function is developed for well learning a netowrk for cell detection, as positive samples miss-labelling is common for histopathology images. Experimental results bascially verify its effectivenss.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This topic discussed in interesting, given that the expensive cost for cell labeling and there are a large number of cells available. The paper is well written. A completely derivation for the loss function is given
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The experiments should be strengthened. Only Faster RCNN based scheme is considered and the results are few. Other detection networks such as Cascaded RCNN, and the same model by using different backbones should be added, which can more convincingly verify the effectiveness of the proposed solution.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper provides sufficient details about the proposed method, which would be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The differences with ref.[7] should be explained clearer. While the intuitive interpretation of equation 12 also should be given clearer Experiments should be strengthened, given that MICCAI is a top-ranked conference.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Interesting topic, well written paper, clear derivation
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors have proposed a loss function improvement to better account for the problem of unlabelled data in cell/mitosis detection.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Deduction of a suitable loss formulae for the improvement.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Seems a delta from [7] (BDE), and the improvement in both recall and precision is a delta from that one. The results show that there is a significant improvement from the baseline to BDE, however the improvement from BDE to this one is small. Therefore would classify this as a small delta contribution. Still, the authors do show, through crossval, stddev, p that the small difference is still statistically significant. The authors could also have looked for other state of the art works and compare with them.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Enough data is given to reproduce the experiments, and the dataset is also public.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

the text, layout, presentation and contexts in general seemed ok, easy to read and well structured. The experiments were also ok and included crossval with statistical significance included. However, you could include more specific approaches, review, experiment and compare to those.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The relevance of the issue, the and the use of crossvalidation to analyze results stattistically.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

This work applies the positive unlabelled learning loss on the Faster R-CNN loss to deal with the situation of incomplete annotations. The method is verified on a public dataset for mitosis detection in breast cancer cells.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This is probably the first work which attempts to apply PU learning based object detection on mitosis detection in breast cancer cells.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The novelty is limited. Applying PU learning on Faster R-CNN is not new, see: Yang, Y. , K. J. Liang , and L. Carin . “Object Detection as a Positive-Unlabeled Problem.” British Machine Vision Conference 2020.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

There is no code provided.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Rather than just borrow an existing model, it is better to explore some new structures or functions to show the optimality of the model . Besides, all the objects are 32*32 generated bounding boxes, which is not suitable to be categorized as an object detection problems.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Both novelty and experimental design are weak,
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper aims to solve the cell detection in histopathology images with incomplete annotations. The human labeled mitotic cells are true positives, while the rest is not treated as negatives (unlabeled instead).

The reviewers confirm the value of this research problem, but raised some major concerns about the methodologies. For example,
1. Positive-unlabeled learning has been used in object detection in the computer vision community. Rather than applying the method from one application domain to another application domain, what is the new methodological contribution?
2. The comparison with the similar work [7] shows the improvement from the proposed work is very small in Table 1. The differences and advantages of the method compared to [7] need more detailed analysis. In Table 1, the performance varies a lot regarding different training-test dataset splits as well.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

Reviewer 5 (R5) suggests that PU learning has been applied to Faster RCNN in Yang et al., BMVC 2020 and thus the novelty of our work is limited. We would like to thank R5 for providing the reference. However, we respectfully disagree that the novelty of our work is limited. Yang et al., BMVC 2020 simply apply the PU learning strategy developed for classification to the detection problem without noticing the difference between object detection and image classification. In particular, as argued on page 5 (before Eq. 10) in our paper, in classification problems the distribution p(x) of an arbitrary sample is directly approximated using the distribution p(x_u) of unlabeled samples. But this approximation is problematic in object detection, because positively labeled and unlabeled samples originate from the same images. In this approximation, the positively labeled samples are excluded from the approximated distribution and the approximation is biased. We notice that the combination of positively labeled and unlabeled samples can approximate p(x) and revise the PU learning framework for the detection problem, which leads to the novel approximation in Eq. 10. This improvement is confirmed by experimental results (not reported in the current draft). Specifically, with the naive approximation used by Yang et al., BMVC 2020, the average precision is reduced from 0.530 to 0.503, which is much worse than the BDE result (0.516) and close to the baseline result (0.501), and this reduction is consistently observed for each fold; the average recall is 0.770, which is no better than the result achieved with our revised strategy. Therefore, although we have used a PU learning strategy, it is adapted to the detection problem with important contributions and novelty designs. We will include these results and clarify the novelty of our method.

R3 suggests that the improvement from BDE to our method is small. We respectfully disagree. As shown in Table 2, on average our method has improved both precision and recall (ranging from 0 to 1) by more than 0.01. Such improvement is common in papers on object or cell detection (e.g., see AP50 in Table 7 in Kemal Oksuz et al., NeurIPS 2020, Table 3 in Xizhou Zhu et al., ICLR 2021, or Table 4 in Chao Li et al., MedIA 2018). Moreover, the improvement is statistically significant. Also, we believe that our method provides a promising direction for cell detection with incomplete annotations. Even with a relatively simple PU learning strategy, we have already achieved the improvement. Our method can motivate further improvements with more advanced PU learning methods in future work.

Reviewer 1 (R1) suggests explaining the difference between our method and BDE. BDE treats samples in unlabeled areas as negative samples, but their weights are adjusted according to whether they are truly negative. Our method treats samples in unlabeled areas as unknown classes, and the classification term associated with negative samples are approximated with both positive samples and these unlabeled samples using a principled framework. We will better clarify this.

R1 suggests that other detection networks and backbones should be considered. On a related note, Reviewer 3 (R3) suggests that other state-of-the-art works can be compared. We have actually tested different backbones for Faster RCNN, where VGG16, ResNet50, and ResNet101 were considered. For all cases, the proposed method outperforms the baseline method and BDE. Due to page limit, we only reported the results achieved with the VGG16 backbone, as it is associated with the best baseline performance. We will clarify that the results were achieved with VGG16, and the improvement can also be observed for other backbones. Note that our method is agnostic to the network structure. Thus, we expect that it can also be integrated with other detection networks, such as Cascade RCNN. Such integration can be explored in future work.

Other minor issues will also be addressed.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors provided responses to some questions/comments raised by the reviewers. It would be better to describe the differences between the proposed PU-learning and a broad range of PU-learning related works to summarize the technical contributions beyond application domain contributions, rather than limiting the scope to one paper mentioned by the reviewer. The explanation about the significance of the small improvement (precision from 0.516 to 0.530, and recall from 0.759 to 0.771) did not convince the AC. Furthermore, in Table 1, the precision varies within [0.435, 0.567] and the recall varies within [0.670, 0.842], according to different folds. The small improvement is negligible, compared to the variance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Authors argue that Yang et al. (BMVC’2020)apply PU learning for detection without noticing the difference between classification and detection. On the other hand, the submitted paper explicitly explores the detection problem when approximating p(x), leading to improved empirical results. The comparison with [7] in Tables 1 and 2 show statistically significant results, but the rebuttal does not explain the variation in results for different train/test splits. The differences with BDE are clarified in the rebuttal. Except for this last point, the authors provided a good rebuttal, and taking into accounting the initial positive reviews, I recommend the acceptance of the paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes a loss function for Faster R-CNN in the context of cell detection in histopathology images with incomplete annotations. The method obtains modest empirical improvements with respect to [7] on a public dataset of mitosis detection. Although the technical novelty of the proposed approach is limited, the rebuttal adequately argues the differences with the missing reference pointed out by R5, which should be discussed in the text. The empirical improvement is not large in absolute terms, but its statistical significance is assessed. Other reviewer comments should be included in the final version of the paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

Positive-unlabeled Learning for Cell Detection in Histopathology Images with Incomplete Annotations