Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Siwei Mai, Qian Li, Qi Zhao, Mingchen Gao

Abstract

This project aims to recognize a group of rare retinal diseases, the hereditary macular dystrophies, based on Optical Coherence Tomography (OCT) images, whose primary manifestation is the interruption, disruption, and loss of the layers of the retina. The challenge of using machine learning models to recognize those diseases arises from the limited number of collected images due to their rareness. We formulate the problems caused by lacking labeled data as a Student-Teacher learning task with a discriminative feature space and knowledge distillation (KD). OCT images have large variations due to different types of macular structural changes, capturing devices, and angles. To alleviate such issues, a pipeline of preprocessing is first utilized for image alignment. Tissue images at different angles can be roughly calibrated to a horizontal state for better feature representation. Extensive experiments on our dataset demonstrate the effectiveness of the proposed approach.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_10

SharedIt: https://rdcu.be/cyl9P

Link to the code repository

N/A

Link to the dataset(s)

Cell dataset: https://data.mendeley.com/datasets/rscbjbr9sj/2

BOE dataset: http://people.duke.edu/~sf59/Srinivasan_BOE_2014_dataset.htm

Reviews

Review #1

Please describe the contribution of the paper

The authors implemented a Student-Teacher learning method with a discriminative feature space and knowledge distillation (KD) applied in OCT scans to classify rare retinal diseases related to hereditary macular dystrophies. Finally, the authors presented the effectiveness of the proposed approach reported in accuracy performance metric.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The application of Student-teacher strategy and KD in OCT scans.
- The use of two good free public datasets of OCT
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The authors should include more information about the datasets. The BOE dataset contains scans and disease classification. But, the Cell dataset contains scans and clinical findings classification. Therefore, the control labels (normal set) for the two datasets are not the same!
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The two dataset are free public available.
- The performance metrics and experimental setup is clear.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

An extensive evaluation of the proposed method with other datasets and ocular conditions is needed. Why not use the Farsiu dataset? [https://www.sciencedirect.com/science/article/abs/pii/S016164201300612X] The inclusion of other baseline methods will be a plus for this paper.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The experimental setup is good but the datasets should be explained in details. Normal test for the two dataset cluster different ocular conditions. i.e. in the Boe dataset, the OCT normal label are patients without DME and without AMD… But patients with other ocular condition could be included…. and in the Cell dataset, the OCT normal label only have non-DME, non-drusen and non-CNV.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The paper proposes a student-teacher model for the classification of hereditary retinal diseases using small dataset. The papers also use knowledge distillation to train the student model.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The are many strengths in the paper
- novel application of the student-teacher model for the classification of hereditary retinal diseases
- strong evaluation of the method and comparing it to classical siamese network
- the paper is very well written and organized
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

For the test parameters, authors should use grid search to find the best parameters instead of finding each parameter independently (Fig. 6)

For all figures, especially Fig. 2, consider putting all the sub-captions into the main caption to be more readable

Page 6: Feature Space Representation (grammatical error) to projection => to project
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors should talk more about the specs of their machine used to train the models
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Authors may use deep learning for the image alignment instead of using classical methods
- authors should justify the use of ResNet50 and maybe they should do experiments on other networks in their future extended study
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The paper proposes a novel application of its methods on hereditary retinal diseases classification
- The paper provides a good approach to train a network using small datasets, which is can be of great usefulness practically
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper presents a method for training a multi-class classifier for rare hereditary diseases from OCT b-scans. The difficulty of the setting stems from the limited amount of training data due to the rarity of the conditions. The method consists of a teacher (ResNet-50) and a student model (ResNet-18). The teacher model is pre-trained on an auxiliary dataset and then transferred to the target dataset. Afterwards its outputs are used to train the student network in addition to the labels in the target dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- interesting combination of deep learning methods
- sensible solution for a difficult problem
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- weak analysis
- at times hard to follow
- train-test split [only found that in the reproducibility checklist]) information is missing.
Please rate the clarity and organization of this paper

Poor
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper provides enough information to implement the training pipeline but since data is not provided, fully reproducing the results is not possible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I believe that this submission tackles an important problem. Currently, most of the research in medical deep learning is focused on easily obtainable datasets so many difficult problems, such as classifying rare condition as in this case, are somewhat neglected. The method presented on the paper is well crafted and the results indicate that it works better than the baselines, which are chosen sensibly.

I am leaning towards accepting the paper, but it still has two significant weaknesses:

Firstly, although the results indicate that the proposed method outperforms the baselines, I am not sure about the robustness of those results. There can be a great deal of randomness in the outcomes when working with so little data. I believe that a cross-validation analysis would have been appropriate in this case. Given the small amount of data, the training time should not have been the limiting factor here.

Secondly, the clarity of the paper needs to be improved. The mathematical notation is not decently explained and often confusing. For example what indices i, j and k represent in equations 1 and 2 or what is minimized in equation 2. Also f in equation 2 and H and sigma in equation 3 are not introduced and it is up to the reader to deduce / guess what they stand for. Readers who are not familiar with SNNL or Knowledge Distillation will have a hard time understanding these equations. At times the text can also be hard to follow, since some sentences are strangely constructed. As a non-native English speaker myself, I understand that this can be difficult but that is what grammar- and spell-checking tools are for.

Other remarks: I found that the data was split 0.7/0.15/0.15 between training/validation/test in the reproducibility checklist but that information should be in the paper itself. Also it is important to know how the split was done.

I do not understand why the images had to be resized to 224². This erodes many important details in the images and is not required since the authors are not working with an ImageNet pre-trained model here. The ResNet architecture itself does not require the images to be 224².
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

While the paper still has flaws, I believe that it could be of use to researches and practitioners in the field who are working on similar problems where data is hard to come by. I find that in this case the creativity of the approach and the importance of the problem outweigh the issues I have detailed in 7.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

All the reviewers found the introduced student-teacher method and the paper very interesting and important. The methodological contribution is strong and appropriate for this clinically relevant task.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We sincerely thank all the reviewers and ACs for their valuable comments. We will revise our final version accordingly to improve the clarity of our manuscript. Here we want to respond to a few critical points raised by reviewers.

Q: Reviewer 1, “the control labels of BOE and Cell datasets are not the same” A: The BOE dataset does differ from the Cell dataset in terms of labels, quantity, type of diseases, and quality of images. However, there is no requirement for matched “normal” labels in our algorithm. The “normal” cases in two datasets can be considered as two separate labels. Our algorithm is generalizable to datasets with unmatched labels. The diversity of the two auxiliary datasets is beneficial for better representation learning. We will clarity that in our final version.

back to top

Few-shot Transfer Learning for Hereditary Retinal Diseases Recognition