Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xihe Kuang, Jason Pui Yin Cheung, Xiaowei Ding, Teng Zhang
Abstract
Most deep-learning based magnetic resonance image (MRI) analysis methods require numerous amounts of labelling work manually done by specialists, which is laborious and time-consuming. In this paper, we aim to develop a hybrid-supervised model generation strategy, called SpineGEM, which can economically generate a high-performing deep learning model for the classification of multiple pathologies of lumbar degeneration disease (LDD). A unique self-supervised learning process is adopted to generate a pretrained model, with no pathology labels or human interventions required. The anatomical priori information is explicitly integrated into the self-supervised process, through auto-generated pixel-wise masks (using MRI-SegFlow: a system with unique voting processes for unsupervised deep learning-based segmentation) of vertebral bodies (VBs) and intervertebral discs (IVDs). With finetuning of a small dataset, the model can produce accurate pathology classifications. Our SpineGEM is validated on the Hong Kong Disc Degeneration Cohort (HKDDC) dataset with pathologies including Schneiderman Score, Disc Bulging, Pfirrmann Grading and Schmorl’s Node. Results show that com-pared with training from scratch (n=1280), the model generated through SpineGEM (n=320) can achieve higher classification accuracy with much less supervision (~5% higher on mean-precision and ~4% higher on mean-recall).
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87196-3_14
SharedIt: https://rdcu.be/cyl1E
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper describes a self-supervised strategy to aid the training of a classification model with a small dataset. The method was validated on a dataset of 1,600 lumbar MRIs with four pathologies of lumbar degeneration disease. Performance is shown to be vastly superior to a standard fully-supervised approach on the tasks listed.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Self-supervision for pre-training is cheap and easy to integrate with current existing supervised training
- Strong performance gaiThe experiments can be bettern
- Very well-presented
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The experiments can be better.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Reproducible if the dataset is made public.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Section 4: It would be great if instead of the tables (2 & 3), the authors can plot mean-precision and mean-recall versus n. It would be interesting to see the the range of n that pre-training would help improve.
- It would also be great to see a comparison against other works. For example, [4] also has Pfirrmann Grading classification.
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is well presented but the experiments can be a bit better.
- What is the ranking of this paper in your review stack?
3
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Review #2
- Please describe the contribution of the paper
In this paper, the authors develop a hybrid-supervised strategy called SpineGEM to generate the deep learning model for classification of multiple LDD pathologies with a small training dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The self-supervised pretraining is an innovation on methodology.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The contributions of the paper is not listed in Section 1.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Satisfactory.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The contributions of the paper should be listed in Section 1.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A little bit of innovation on methodology exists in the paper.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Review #3
- Please describe the contribution of the paper
This paper proposes a deep learning-based classification method for multiple pathologies of lumbar degeneration disease. In this work, the author validates the proposed method by using a small dataset and then compares the performance with the existing methods trained on larger datasets.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The main contribution is to present an unsupervised method for the initial segmentation and apply an encoder-decoder structure for the classification of different specific pathologies.
- The author employs a set of random transformation to produce artifacts with different image features.
- The proposed work uses the pre-trained encoder that connects to the classifier, which requires a less manual annotated label for fine-tuning procedures.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The author presents a method using random transformation for representing different pixel intensity values, but it does not cover any types of image noises.
- The author compared a proposed method with conventional supervised learning method, but it would be interesting to see the performance comparison with other state-of-the-art neural networks.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed work could be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The paper would be more matured if the experimental results could be compared with those from state-of-the-arts neural networks.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
My decision on the final score has been made as 7 because this paper presents a novel method for the classification task, and the presented experimental results looks good when a small number of training dataset is used.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
3
- Reviewer confidence
Confident but not absolutely certain
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a self-supervised strategy to aid classification of spine diseases with small datasets. Experimental results demonstrate good performance. The main strength of this paper lies in the self-supervision for pretraining, which requires less annotated label for fine-tuning process. However, there are following concerns requiring to be addressed in the rebuttal. 1. The self-supervised strategy is not new and lacking comparison with other directly unsupervised training methods for demonstrating the value of the proposed method. 2. The experimental comparison with other state-of-the-art methods are not comprehensive (actually I did not find any unsupervised or semi-supervised method for comparison). 3. The public availability of the dataset. 4. Other concerns that reviewers have raised.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
8
Author Feedback
The authors would like to thank the Reviewers for all of your time and effort devoted to the review of our manuscript. The comments are extremely insightful and greatly appreciated. As such, the authors would like to take this opportunity to address each and every concern the Reviewers noted in their review of our submission.
Comments of Meta-Reviews:
Comment 1: The self-supervised pretraining strategy is not new and the comparison with other directly unsupervised training methods is required.
Response: As clarified in Section 1, the idea of self-supervised pretraining strategy is inspired by the published Model Genesis and our novelty is explicitly integrating the anatomical information, including relative locations, sizes, and shapes of multiple tissues, with the segmentation generated via an unsupervised process. To the best of our knowledge, no existing work has achieved the LDD pathology classification with directly unsupervised training methods. We have tested some directly unsupervised training methods, such as k-means and DBSCAN. However, their accuracy was significantly lower (< 30%) than previously reported learning based method, thus we did not include them in the conference paper. We can add the comparison in.
Comment 2: More experimental comparison with other state-of-the-art unsupervised or semi-supervised methods is required.
Response: We fully understand the reviewer wish more comprehensive comparison with other methods. However, the major purpose of the paper is demonstrating that our method can effectively reduce the supervision requirement in training of deep learning model for LDD pathology classification. We have tested other state-of-the-art self-supervised method from A. Jamaludin et al. “Self-supervised learning for spinal MRIs” on our dataset and our method achieved better outcomes. We will include the comparison results in our further extended paper.
Comment 3: The public availability of the dataset.
Response: Currently, the MRI images and pathology labels belong to the University of Hong Kong. We are in the process of obtaining authorization to publicize the dataset, and dataset will be public available thereafter.
Comments of other reviews not included above:
Comment 7 of Review #1: The tables (2&3) can be replaced with the plots of mPer and mRec vs samples’ number (n) to illustrate the range of n that pre-training would help improve.
Response: As shown in Table 1, due to the extremely unbalanced label distribution, 320 (proposed in the paper) is actually the bottom-line of the n to ensure all different types of samples are included in training set, without distorting label distribution. In our experiment, the increasing of n (at most 1280) will not significantly improve the performance. If the paper is accepted, we will add this result back by replacing the tables with plots during the final proof review.
Comment 4&7 of Review #2: Contributions of the paper should be listed in Section 1.
Response: Contributions of the paper is summarized in Section 5, and we will move to Section 1 during the proofing.
Comment 4&7 of Review #3: The image noises should be included in the random transformation, and the proposed method should be compared with other state-of-the-art neural networks.
Response: As clarified in Section 3, our MRIs are obtained with different machines and protocols, which have already contained different kinds of noise. Besides, the purpose of random transformation is to introduce the anatomical artefact for multi-tissues, and noises contain no anatomical information, therefore it is not explicitly included. As clarified in Section 1 our method is a model generation strategy, for pre-training, which can integrate with other neural networks to reduce the supervision requirement, not a new network architecture, thus shall not be compared with other published neural networks.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Due to lack of direct comparison with other existing methods for validating the efficacy of experimental results and insignificant technological contribution, I would like to recommend reject.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
23
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have conducted experimental results on a state of the art method. Please include that experiment in the final version.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
10
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper proposes an interesting approach for the reduction of need for labelled datasets. The rebuttal helps justifying the rationale for the study and the approach with which it was envisioned. The use of anatomical data in conjunction with the presented results justifies its further discussion at MICCAI
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7