Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Zuhui Wang, Zhaozheng Yin

Abstract

Recent advances in deep learning have achieved impressive results on microscopy cell counting tasks. The success of deep learning models usually needs sufficient training data with manual annotations, which can be time-consuming and costly. In this paper, we propose an annotation-efficient cell counting approach which injects cell counting networks into an active learning framework. By designing a multi-task learning in the cell counter network model, we leverage unlabeled data for feature representation learning and use deep clustering to group unlabeled data. Rather than labeling every cell in each training image, the deep active learning only suggests the most uncertain, diverse, representative and rare image regions for annotation. Evaluated on four widely used cell counting datasets, our cell counter trained by a small subset of training data suggested by the deep active learning, achieves superior performance compared to state-of-the-arts with full training or other suggestive annotations. Our code is available at https://github.com/cvbmi-research/AnnotationEfficient-CellCounting.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_39

SharedIt: https://rdcu.be/cymav

Link to the code repository

https://github.com/cvbmi-research/AnnotationEfficient-CellCounting

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a cell counting method that requires less amount of labeled data. For this, authors have used active learning and semi-supervised learning. For active learning, authors use prediction inconsistency between ensemble to compute uncertainty and the data with higher uncertain predictions are sampled cluster-wise to improve diversity. The data is further sampled to make sure they are different from training samples. The authors also use semi-supervised learning during the training with consistency loss.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- I like the idea of cluster-wise sampling based on uncertainty and a metric defined Eq (4). The numerator of Eq(4) check the outlier in unlabeled samples where denominator makes sure the the samples are far from training set.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Please see below:
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The dataset used are publicly available. The method can be reproduced if the authors provide code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- It is not clear how data clustering is done. Since the model is trained to predict the cluster label, how is the ground truth of the cluster obtained for training.
- The author have used model checkpoints saved at different epochs as part of an ensemble. If there is a prior art for this technique, authors should cite it. If this is the new technique, proper ablation study should be conducted to prove its usefulness.
- In Section 3, implementation details, how are training data selected for each cell counter? Also, do you need to cluster the unlabeled data when you can obtain the cluster using trained model?
- The performance is better than SOTA method, however, it is not clear if the gain in performance is due to active learning or semi-supervised learning. Please include the result for the case when only active learning is used.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper has technical merit, especially active learning part. Having said that there are shortcomings (please see 7) that need to addressed if this is to be accepted.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The paper proposes to combine different approaches from active learning and semi-supervised learning to reduce the data annotation workload in cell counting problems.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The proposed work models the cell counting problem as a multi-task learning task where the number of counts and distributional information of the data (i.e. The cluster information) is simultaneously learned by the model. The additional output allows the model to incorporate active learning to further reduce the annotation cost.
2. The proposed work clearly states the three challenges in cell counting data annotation and addresses them with batch mode, uncertainty based active learning approaches.
3. The paper proposes an interesting way to evaluate the sample’s uncertainty by accumulating model variance from each epoch of the training.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Some parts of the methodology are not clearly described.
2. The effectiveness of the proposed active sampling method is not well supported by the experiments.
3. The novelty in terms of machine learning methodology does not appear to be significant.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The experiments and methods are clearly described in the paper. They are easy to reproduce even without the source code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. It is not clear how epoch-wised uncertainty can help sample informative data. In my opinion, it takes several epochs before the deep learning model converges. Due to the over-parameterized structure of the deep learning model, predictions of the early epochs can be very random/unstable and do not reflect the true uncertainty of the data sample.
2. The impact of the key parameters N, C and K are not studied. Ablation studies should be conducted to provide insight on how to set them and how sensitive they are.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Please refer to the weaknesses and detailed comments.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper presents an annotation-efficient cell counting approach under an active learning framework. The proposed method can provide accurate cell counting with few suggestive annotations.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper is a novel application of the active learning framework on the cell counting problem. With the proposed method, only some important cell regions in some training images are needed to be labeled to achieve the annotation-efficient cell counting.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Many state-of-the-art active learning methods are not included for evaluation.
2. Some literature about cell count needs to be presented.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Without code being provided, it seems hard to reproduce the results under an active learning framework.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The paper presents an interesting application of active learning on cell counting. The paper is well-organized with good figure illustrations.

Minor Comments: (1) To include more related works about cell counting would benefit the readers to understand the clinical significance of this direction;

(2) To include more active methods to demonstrate the effectiveness of the proposed method;

(3) Ablation study on each loss item should be presented.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A successful application of active learning to the clinical problems.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper receives diverged review ratings. Please the authors address the issues and concerns as listed by the reviewers in their comments in box 4 for weakness and the questions raised in box 7 for detailed inquiries, especially for clarifying the details of the method as inquired by R#1 and R#2.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Author Feedback

R1.1 Clustering In the initial annotation round, the clustering is done by K-means. For the rest annotation rounds, we treat cluster predictions from the previous epoch as ground truth for the next clustering model training (i.e., self-learning by pseudo labeling).

R1.2 Epoch-wise uncertainty The ablation study of the proposed epoch-wise uncertainty, evaluated on four datasets (VGG/MBM/ADI/DCC) by MAE values, shows its effectiveness: w/o-Epoch-Uncertainty: 3.5/6.1/14.5/3.6 w-Epoch-Uncertainty: 2.9/5.7/14.1/3.0

R1.3 Training data selection for each counter and do you cluster unlabeled data when you can obtain the cluster using trained model? With N suggested samples being labeled from one round, we employ the Bootstrapping method to randomly select a subset (n, n<N) of the labeled samples for training each cell counter, described in Section 2.2. Only for the initial annotation round, we need to cluster data by K-means. Yes, for the rest rounds, the cluster labels of unlabeled data are predicted from our trained models.

R1.4 The performance is better than SOTA. Is it due to active learning (AL) or semi-supervised learning (SSL)? The ablation study of AL and SSL on the four datasets shows both of them improve the performance. w/o-AL: 5.9/7.1/17.3/5.4 w/o-SSL: 3.4/6.0/14.4/3.3 AL+SSL(ours): 2.9/5.7/14.1/3.0

R2.1 Predictions of early epochs can be unstable and don’t reflect true uncertainty: The epoch-wise uncertainty is evaluated based on the model predictions of every r epochs, described in Section 2.2. It covers the whole training procedure to find uncertain samples, not just the early few epochs. Moreover, the ablation study of the epoch-wise uncertainty, evaluated on four datasets (VGG/MBM/ADI/DCC) by MAE values, shows its effectiveness: w/o-Epoch-Uncertainty: 3.5/6.1/14.5/3.6 w-Epoch-Uncertainty: 2.9/5.7/14.1/3.0

R2.2 Ablation study of parameters N, C, and K Ablation studies of N (cell counter numbers), C (cluster numbers), and K (sample selection numbers) on four validation datasets are below: N=3(ours): 2.9/5.7/14.1/3.0 N=4: 2.9/5.6/14.1/3.0 N=5: 2.9/5.6/13.9/3.0 C=3: 3.1/5.8/14.1/3.2 C=5(ours): 2.9/5.7/14.1/3.0 C=7: 3.3/6.2/14.6/3.5 K=5: 3.0/5.7/14.1/3.1 K=10(ours): 2.9/5.7/14.1/3.0 K=15: 3.1/5.9/14.4/3.2 We chose parameters by the above MAE metric, listed in implementation details.

R2.3 Effectiveness of the proposed active learning is not well supported by the experiments? The ablation study of AL, on the four datasets by MAE values, shows its effectiveness: w/o-AL (SSL only): 5.9/7.1/17.3/5.4 AL+SSL(ours): 2.9/5.7/14.1/3.0

R2.4 Novelty The novelty includes: a new active learning algorithm with new suggestive metrics; informative image regions, rather than whole images, are selected for annotation; semi-supervised learning using unlabeled data for suggestive metric computation; and multi-tasking learning-based cell counter model.

R3.1 Add more related works about the clinical importance of cell counting Will do. Some examples: (a) tumor cell number of breast cancer is an important biomarker indicative of patients’ prognosis [Shah BIBM’17]; (b) A patient’s health can be evaluated based on the number of red blood cells and white blood cells [Xie MICAAI’15].

R3.2 Add more active learning methods to prove the proposed method’s effectiveness The method we compared [Yang MICCAI’17] is a pioneering work in this area. A latest AL for image segmentation [Li MICCAI’20] is based on this work. We will compare several latest AL methods, such as [Zhao JBHI’21] and [Alshehhi VISIGRAPP’21]. Since these three methods focused on image segmentation, we will think how to revise them to cell counting.

R3.3 Ablation study on each loss term Four loss terms in Eq.1: MSE loss (T1), cluster loss (T2), count loss (T3), and cross-entropy loss (T4). The ablation study of each term on four datasets shows their effectiveness: T1: 4.1/6.9/15.3/4.2 T1+T2: 3.4/6.0/14.4/3.3 T1+T2+T3: 3.1/5.8/14.3/3.2 T1+T2+T3+T4 (ours): 2.9/5.7/14.1/3.0

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper studies the problem of cell counting, which is of apparent practical value for analyzing microscopy images. To solve the problem, a neural network is constructed to make use of both semi-supervised learning and active learning, which appears resonable. The experimental results support the proposed method. The authors’ rebuttal has largely addressed the questions and concerns raised by the reviewers. So I recommend an acceptance to this paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The rebuttal focused on the technical aspects of the paper, Most of the answers provide useful clarifications, but the question about how epoch-wised uncertainty can help sample informative data was not properly answered — instead the rebuttal answers about how MAE is affected with the epoch-wise uncertainty. For the ablation study w.r.t. N,C,K, it is unclear why the paper uses N=3 because that is not option that enables the best results. The promise of adding results from [Shah BIBM’17] and [Xie MICCAI’15] is not satisfactory — why aren’t the results in the paper already? How the proposed method compare with them? The ablation study on the loss terms is well conducted. Even with these problems, given the initial reviews and the clarifications of the rebuttal, I recommend the acceptance of the paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This manuscript combines active learning and semi-supervised learning into a single framework for cell counting in microscopy image data. With only limited annotated training data, the framework can deliver competitive cell counting performance with those methods using full training data. The rebuttal has addressed most concerns from reviewers, such as evaluation of each component of the framework, measurement of effects of important hyperparameters, and clarification of technical contributions.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

12

back to top

Annotation-efficient Cell Counting