Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Jinghan Sun, Dong Wei, Kai Ma, Liansheng Wang, Yefeng Zheng

Abstract

Rare diseases are characterized by low prevalence and are often chronically debilitating or life-threatening. Imaging-based classification of rare diseases is challenging due to the severe shortage in training examples. Few-shot learning (FSL) methods tackle this challenge by extracting generalizable prior knowledge from a large base dataset of common diseases and normal controls, and transferring the knowledge to rare diseases. Yet, most existing methods require the base dataset to be labeled and do not make full use of the precious examples of the rare diseases. To this end, we propose in this work a novel hybrid approach to rare disease classification, featuring two key novelties targeted at the above drawbacks. First, we adopt the unsupervised representation learning (URL) based on self-supervising contrastive loss, whereby to eliminate the overhead in labeling the base dataset. Second, we integrate the URL with pseudo-label supervised classification for effective self-distillation of the knowledge about the rare diseases, composing a hybrid approach taking advantages of both unsupervised and (pseudo-)supervised learning on the base dataset. Experimental results on classification of rare skin lesions show that our hybrid approach substantially outperforms existing FSL methods (including those using fully supervised base dataset) for rare disease classification via effective integration of the URL and pseudo-label driven self-distillation, thus establishing a new state of the art.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_50

SharedIt: https://rdcu.be/cyl6q

Link to the code repository

https://github.com/SunjHan/Hybrid-Representation-Learning-Approach-for-Rare-Disease-Classification

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper proposes an unsupervised representation learning framework for rare disease classification. The proposed framework incorporates contrastive learning, knowledge distillation, and pseudo-label learning.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The motivation of the study is good. The rare disease classification and label-efficient learning are two important areas in medical image analysis.
2. The paper presents extensive experiments which show promising results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The theoritical contribution is incremental. The paper is more like a fusion of a few popular topics. For example, there is no critical extension/adaption to MoCo V1, peusdo-label generation, and knowledge distillation.
2. The technical writing of the problem formulation requires further consideration. In Section 2, the problem formulation is a standard few-shot learning, but the authors have additional (large) unlabeled dataset for URL. I understand it correctly, I think the focus of the paper should be just URL + few-labels fine-tuning for the rare disease. The first sight of the problem setting is kind of misleading.
3. The experimental design requires further discussion. The common evaluation of URL is linear classification protocol [1,3,7]. However, the paper adopts a FSL evaluation protocol for URL, which might not be a fair comparison for either FSL baselines (see below) or URL baselines, as the proposed method is a mixture of two.
4. Following 3., in standard FSL [5,25,28], give a training set with various classes, the model is trained to grasp new knowledge with a few labeled examples (e.g. by meta-learning). However, for the dataset used in this work, there is no such setup. As D_{base} is a large unlabeled set, there is only a 3-class D_{rare}. I highly doubt whether this fit the problem formulation of FSL. I think the authors should make this clarified in the paper.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility seems to be OK.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Following 4.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

My recommendation is based on my comments in 4.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

A new approach for rare skin disease classification is proposed. The model is first trained with unsupervised representation learning on a base dataset. Then the model is fine-tuned on the rare diseases training set to generate pseudo labels to supervise a student with the same architecture.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The perfromance surpass SimCLR and other SOTA models, which is impressive.
2. Well-writen and easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Lack of ablation study for some of the components, for example, if train a network without URL and directly on the D_rare to show the URL works.
2. The classification branch of self-distillation part is very similar to the recent work: Xie, Qizhe, et al. “Self-training with noisy student improves imagenet classification.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. The self-distillation strategy is also similar to cyclic training in this motion pyramid network: Yu, Hanchao, et al. “Motion pyramid networks for accurate and efficient cardiac motion estimation.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2020.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The dataset is public available and the code will be released.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Add more ablation study to show the necessarlity of each component, which will make paper more convincing.
2. Add some citation and discuss the difference with those works
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Interesting work with strong performance. Add discussion with similar work and more ablation study will improve the quality of the work.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper proposes a hybrid distillation strategy to further improve the self-supervised model for rare disease classification. Initial experiments supports that the proposed method outperforms strong self-supervised baselines.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The hybrid distillation strategy for few-shot learning is novel; The proposed method is compared with a comprehensive set of baselines, including the SOTA methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Only the fixed 3 rare diseases were used for evaluation in the experiments. Thus, the evaluation is very limited and the generalization of the proposed method is in doubt. Even with the 7-class skin disease data, the largest 3 classes can be used for D_base and the remaining 4 small classes can be used as simulated rare diseases for another set of evaluation. Evaluations on additional datasets are more desirable.
2. Even with the fixed 3 rare diseases, results with more K values should be reported. Currently, results only with K=1,3,5 were reported, and the trend of decreased gap between the proposed method and the strong baseline MoCo indicates that slightly higher K (e.g., 10) would lead to similar performance between the proposed method and the baseline. Note that results with K=10,20 are often reported in literature, and K=10,20 are also feasible for rare disease classification. If similar performance was observed with higher K values, then what is the advantage of using the proposed hybrid distillation strategy compared to the simple one-stage self-supervised learning?
3. The generalizability is also not clear with respect to CNN backbones. What if using more other ResNets and other non-ResNet backbones?
4. The paper claimed a two-step novel framework, but the first step is the well-known self-supervised framework and should not be claimed novel.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Fine
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please see above comments on weakness.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Due to the limited experiments to support the effectiveness and generalizability of the proposed hybrid distillation strategy.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #4

Please describe the contribution of the paper

The paper proposed a pipeline for the few-shot rare disease classification. It’s the first few-shot learning method for medical image classification using an unsupervised base dataset. The pipeline consists of four parts: unsupervised learning on base set, supervised learning using rare disease set, getting pseudo-label for base set, and knowledge distillation on base set. The method was evaluated on skin lesion dataset, and achieved superior performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The method is relatively novel. Although each step of the pipeline is somewhat using the existing methods, the efficient combination of these steps is novel. The paper writing is clear and concise. The experiment is thorough and indicates the superiority of the proposed method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

All components of the pipeline are not original.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Public dataset and open source codebase.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The paper overall is very good. I only have some minor comments:
1. Some steps frozen f_q, while some steps fine-tuning it. Please explain the logic or show the ablation study.
2. Add a set of experiment to show using the proposed architecture, how much performance boost can be achieved if using supervised base set.
3. fig. 1 (d) f_k’ and f_q’ are flipped.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The overall pipeline is relatively novel. The paper is well-writen. The experiment is quite thorough. The performance is superior.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

7
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper generally received positive feedback from the reviewers, however, some aspects of the work made it short of early acceptance. These concerns that need to be addressed in a rebuttal are:
- theoretical contributions are generally found incremental, and components come from different publications. The authors claim most of them as their novelties.
- the claims of novelty need to be rectified
- the writing of the technical aspects of the paper need improvements
- clarifications with respect to few-shot learning and its direct relevance with the proposed method are properly questioned by one of the reviewers.
- most importantly, the paper needs a proper ablation study
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We thank the (meta-)reviewers for appreciating our work and the constructive comments. Below we summarize the main comments and address them one by one.

C1: Theoretical contributions are generally found incremental, and components come from different publications. The authors claim most of them as their novelties. The claims of novelty need to be rectified (R1/R3/R4) A1: Thanks for suggesting the rectification. For revision, we will be careful to more explicitly acknowledge the existing components (which were already cited), and not to claim them as our novelty. Instead, we will more precisely claim our contributions to be the novel application of unsupervised representation learning (URL) to rare disease classification, and the hybrid distillation combining contrastive and pseudo-label learning. As the reviewers kindly noted, novel and subtle combination of existing concepts for novel applications is important to the research community, especially for medical applications where the adaption of methods typically indicates unique contribution. As R1 acknowledged, both rare disease classification and label-efficient learning are important areas in medical image analysis, and our work presents an effective new approach to the intersecting research area. In addition, R3 & R4 indicated that our efficient combination of the existing components and hybrid distillation strategy are novel. We thus believe that this work should make novel contributions, as validated by its superior performance to a wide variety of SOTA methods.

C2: Writing of the technical aspects of the problem formulation need improvements; clarifications with respect to few-shot learning (FSL) and its direct relevance with the proposed method (R1) A2: Thanks for the insightful suggestions. We acknowledge that our problem setting differs from the conventions of general FSL. However, we would like to reemphasize our rationales behind (already described in the original submission). (1) We follow Li et al. [18] (MICCAI-20) to divide the ISIC 2018 dataset into four common and three rare diseases. Although we can construct various two-way tasks from the rare classes, the three-way protocol more truly reflects the clinical need to distinguish all known rare diseases together. (2) We construct a big query set for the rare diseases rather than repeated construction of small query sets, to better simulate the application scenario where a trained classifier is applied to all incoming test samples. (3) Known types of rare diseases are fixed most of the time, and their recognition constitutes a definite task. Thus, it is reasonable to bridge the few rare-disease samples and the base set for better performance (in contrary to general FSL where the base and novel sets are usually isolated). In this regard, we consider our problem setting a specific and practically useful application of the general FSL. We will highlight the relevance and distinction in the revision.

C3: Most importantly, the paper needs a proper ablation study A3: We agree with the reviewers on the importance of a proper ablation study. In fact, the original submission included ablative experiments for most components of our approach, including (1) training a network without URL and directly on D_rare to show the URL works (R2; Table 1, ‘Training from scratch’), (2) necessity of the hybrid distillation (R2; Table 2), (3) superiority of the adaptive hard pseudo labels (R2; Table S2), and (4) generalization on the 4 conv blocks CNN backbone (R3; Table S4). We will highlight them in the revision. In addition, we have conducted few extra experiments where supervised base set is used with our architecture (R4). The performance is slightly better than SRL-distill but inferior to the proposed hybrid distillation in Table 1, again confirming the advantages of the URL and our framework.

C4: Results with K=10, 20 (R3) A4: As suggested, experiments show that our approach maintains ~1-point leads over MoCo with K=10 & 20.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors provided a clear feedback and addressed most of the concerns. They are advised to addressed the concerns in the final version of the the paper as well.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The technical contribution of this paper is a bit mediocre. It follows the recent success in training unsupervised feature representation learning (contrastive loss) to help improve semi-supervised learning (such as the teacher-student model, but here pseudo-labels are used), like in co-match and self-match, but well integrates this idea into the scenario of rare disease classification. That might be of some interest to the audience in MICCAI. In the revision, the authors should accurately indicate the link and difference between their proposed method and the literature in self-supervised learning, semi-supervised learning, and few-shot learning, and try best to address the comments from Review#3.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

As raised by reviewers, this work presents a combination of existing approaches in a unique framework. While this makes the technical contribution limited, the combination of existing approaches could be useful to address the unique challenges of the medical imaging community. I strongly suggest the authors to smooth their tone on the claimed contributions and properly acknowledge existing work. Furthermore, I also agree with the fact that the proposed setting differs from the few-shot learning scenario, where classes in support and query samples are disjoint from the base classes. Authors should revisit these links. Overall, authors positively addressed reviewer comments and I believe the paper can be accepted at MICCAI’21.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

back to top

Unsupervised Representation Learning Meets Pseudo-Label Supervised Self-Distillation: A New Approach to Rare Disease Classification