Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Hao Guan, Yunbi Liu, Shifu Xiao, Ling Yue, Mingxia Liu

Abstract

Subjective cognitive decline (SCD) is a preclinical phase of Alzheimer’s disease (AD) which occurs before the decits could be detected by cognitive tests. It is highly desired to predict the progress of SCD for possible intervention of AD-related cognitive decline. Many neuroimaging-based methods have been developed for AD diagnosis, but there are few studies devoted to automated progress prediction of SCD due to the limited number of SCD subjects. Even though some studies proposed to transfer models (trained on AD/MCI) to SCD analysis, the signicant domain shift between their data distributions may degrade the prediction performance. To this end, this paper tackles the problem of learning a model from the source data for which can directly generalize to an unseen target domain for SCD prediction. We propose a cost-sensitive meta-learning scheme to simultaneously improve the model generalization and its sensitivity in MRI-based SCD detection. During training, the source domain is divided into virtual meta-train and metatest sets to explicitly simulate the scenario for early-stage detection of AD. Considering the importance of sensitivity for progressive status detection, we further introduce cost-sensitive learning to enhance the metaoptimization process by encouraging the model to gain higher sensitivity for SCD detection with simulated domain shift. Experiments conducted on the large-scale ADNI dataset and a small-scale SCD dataset have demonstrated the effectiveness of the proposed method.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_24

SharedIt: https://rdcu.be/cyl5Y

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper presents a deep architecture to predict the evolution of patients diagnosed with subjective cognitive decline (SCD) toward Alzheimer disease (AD) in MRI. Based on the very limited number of SCD MRI exams, the authors propose an architecture based on the meta-learning framework that was derived to address the problem of domain adaptation in the context of multiple annotated source and unannotated target datasets. The authors also add a cost-sensitive loss term to maximise sensitivity. The source datasets are extracted from the ADNI database including MCI (653) , AD (367) and SCD (16) patients while the target dataset is a private one consisting of 76 SCD patients. The proposed method is compared to alternative architectures.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-the implementation of the meta-learning framework, although it has been previously applied to different medical diagnostic task is original for this application -Attempt to visualize the latent space is interesting
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-The technical soundness of the implementation of the meta-learning framework is questionable. Please see my detailed comment below.
- The experimental study, especially regarding the comparison with SOTA methods, should be further detailed.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors mention that they will release all available code in section2 of the check-list, which is not mentioned in the core paper. They answered ‘yes’ to almost all questions, which is not always correct : the range of hyper-parameters considered and details on how baseline methods were implemented and tuned are not always provided, statistical analysis of results are analysis of situations in which the method failed are not performed
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The paper is well written and clear. I have one major concern regarding the technical correctness of the meta-learning implementation. Please find below some comments and questions.

-The author mention that their architecture relies on the meta-learning paradigm proposed by Li et al in [14]. In this paper, the authors propose an iterative process whose algorithm is described in ‘Algorithm 1’ of their paper. A each iteration 1) the meta-training and meta-test datasets are modified by selecting different combinations of sources datasets and 2) the weight of the CNN are updated based on both backpropagation of the meta-train and meta-test loss function. Also note that a beta parameter is added to weight the different contributions of these two loss terms. Based on the text and on fig 2, it is not clear that your implementation follows this methodology: 1) could you please clarify If you perform this iteration loop to modify the meta source datasets of the meta-train and meta-test sets? If not, I guess your model resumes to a classical CNN model trained with mixed sources data (including PSC patient of the ADNI database) with the addition of a cost-sensitive loss term. 2) could you please further detail how the weights are updated bases on the different loss terms. Fig 2 indicates the update of the meta-test CNN (blue vertical arrow) but what about updating the weight of the meta-train CNN based on the meta-test and cost-sensitive losses. The update process should be clarified.

-Regarding the implementation the authors mention “ For simplicity, the parameters alpha and beta in Eq. (2) are set to 1.”. The authors should discuss this and evaluate the impact of these parameters on the global performance

-Regarding results in Fig 6, If I understand correctly, CSML is the considered model, while CSML-1 and -2 are the same models trained with different sources (cf my remarks above, I guess the meta-train sources are not iteratively mixed) but without the cost-sensitive term. I would say that the performance achieved with CSML-2 and CSML-1 do not differ significantly from that achieved with the global model, thus meaning that the cost-sensitive loss term has a minor impact on the global performance. Please comment.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I am questioning the correctness of the meta-learning implementation. Comparison with the SOTA methods should be improved, as pointed in my comments, to confirm that the proposed method outperforms SOTA methods. The reproducibility check-list does not fully reflect the content of the paper.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors show that by applying a meta-learning framework (meta-train with different domain than target domain, meta-test with target domain, and cost-sensitive loss to improve sensitivity) they are able to predict whether SCD subjects will progress to MCI or remain stable with a better AUC than state of the art methods. They include subjects from ADNI, but also patients from a local hospital with a larger number of SCD subjects than those found in ADNI.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The evaluation of their results was compared against multiple competing methods as well as against multiple state-of-the-art methodologies, obtaining promising results. The inclusion of a cost-sensitive loss enhances the sensitivity of their methodology. The validation dataset includes much more SDC patients than its ADNI counterpart.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I believe the comparison between the proposed method, the competing methods and the variants of the proposed method are somewhat biased towards yielding a positive conclusion regarding the proposed method (e.g., the accuracy of the baseline-2 method is comparable to the proposed method and its variants.)
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors state that they have shared a link to a downloadable version of the new dataset, but I am unable to find it (it could be my mistake, though). Same goes for their code, I don’t know if they will make it available after (if) accepted.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

This work is very interesting and I believe its results are very promising. Howevert, the results of the Baseline-2 method should be further analyzed; I believe it is being harshly evaluated. It achieves almost the same accuracy as the proposed method, a metric of more clinical significance than the AUC values. Maybe you could evaluate whether if by performing a meta-train and meta-test, both with the M domain, so as not to consider it an CSML variant, but including the cost-sensitive loss, the baseline-2 results would also be comparable to your proposed methods in terms of sensitiviy. Moreso, the baseline-2 methodology does take into account the typical restriction that yields the need for a meta-learning scheme, no target samples are available for model training or fine-tuning. Whereas, your methodology does imply the need of target samples available for model training.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The novelty of the methodology, the amount of methods against which it was compared/evaluated, the inclusion of a larger database than typically used, the promising results that they obtained, and the overall good structure/organization/clarity of the manuscript.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

A Cost Sensitive Meta-Learning (CSML) framework for Subjective Cognitive Decline (SCD) progress prediction was presented. The source domain that consisted divided into meta-train and meta-test datasets so as to generalize to an unseen target SCD domain. An ablation analysis was also performed to assess the importance of this learning scheme in the model.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

• Novelty, in that SCD was focused rather than classification of MCI from AD • Clinical relevance of the problem
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

• Minor concern: Was actual longitudinal neuroimaging data at multiple time points used, where SCD would progress into MCI, and the performance of the model on such a data evaluated? If yes, not explicitly evident from the methodology.

• Sensitivity (SEN) metric, on which the model was optimized on was still below state of the art GANs (as evident from Table 2)
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

• Part of the data used is from a publicly available dataset
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Minor Concerns:
1. Please try to include more clinincal details regarding the diagnosis/prognosis of pSCD and sSCD, and the standard norms for it.
2. Please comment on higher specificity (SPE) for the Baseline-2 model as evident in Table 1
3. Was actual longitudinal neuroimaging data at multiple time points used, where SCD would progress into MCI, and the performance of the model on such a data evaluated? If yes, not explicitly evident from the methodology
4. What was the rationale to include Normal Controls (NCs) only in the A source domain, but not in the M or S ? Considering the fact that SCDs are pre-clinical, one would expect that the NC feature distributions would also be closer to SCDs as are MCIs, and should be studied.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

• Novelty of the problem • Further extensive evaluation of the model results using Ablation analysis, Saliency maps and comparisons to similar models and state of the art
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper is written well, and some details need to be supplemented. For example, the experimental comparison with SOTA methods, detail how the weights are updated bases on the different loss terms, clarify this iteration loop to modify the meta source datasets of the meta-train and meta-test sets，and some others. Please revise the paper carefully according to the comments.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We appreciate the ACs and Reviewers for the constructive comments. We are encouraged by the many positive comments about our ‘novelty’ of both the problem and method (R3), ‘original and clinical relevant application’ (R1&R3), and ‘very interesting work and very promising results’ (R2), and ‘well-written’ manuscript (R1&R2&R3&AC). We will address the minor concerns in the following:

Q1: Comparison with SOTA methods (R1,AC) -For the fair comparison, we reproduced the SOTA methods in Table 2 using the same data set as this work. -As suggested, we’ll include more details on implementations of these methods in the final version.

Q2: How to update weights based on different loss terms (R1) -Please note that there is only one CNN in the proposed framework, and there are not the “meta-train CNN” or “meta-test CNN”. In Fig. 2, the CNNs in different colors merely indicate the different update stages of the CNN, not different CNNs. -Let w denote the weight of the CNN. The weight goes through a two-step update. First, the CNN is trained with the meta-train loss on meta-train set, and the weight w is updated according to equation (1). Then, the updated CNN is fed with the meta-test set, and the meta-test loss and cost-sensitive loss are utilized to further update the weight. The training on the meta-test set can be understood as a regularization term which enables the network to gain a generalization ability for SCD prediction on the target domain. -In the final version, we will add more details on the optimization.

Q3: Iteration loop to modify meta source datasets of meta-train and meta-test sets (R1) -The study in [14] has a different problem setting from ours. In [14], there are multiple (typically>4) source domains which share the same label space (and the target domain shares the same label space, too). Thus, their task is to develop a transferable model for entire label space while alleviating the domain shift without accessing the target data for training. Please note that the only difference between their source domains is their covariance shift. -The focus of this work is to predict the future progress of SCD. There are 2 source domains (i.e., M with pMCI&sMCI, and S with pSCD&sSCD) and they do not share the same label space. We aim to develop a train a model on source domains and do not access the target domain (an SCD dataset). Thus, M is utilized as the meta-train set and S is used as the meta-test set. The meta-learning in our framework ensures its fundamental difference from a plain CNN just trained with data from mixed sources. -In the future work, we plan to collect data from more sources to further improve our model.

Q4: Impact of cost-sensitive loss on the global performance & Statistical analysis or results (R1,R2) -We’d like to mention that the task of SCD progress prediction is very challenging, due to the fact that SCD appears at the preclinical stage of Alzheimer’s disease (AD) even without significant objective impairment in the brain. Results reported in Tables 1-2 and Fig. 6 suggest that results of SCD progress prediction are usually worse than those of MCI conversion prediction and AD identification [1,2].
-We add an experiment to evaluate whether the results of CSML and its 2 variants (i.e., CSML-1 and CSML-2) are significantly different via pair-wise t-test. The p-value for results of CSML vs. CSML-1 is 0.029, while that for CSML vs. CSML-2 is 0.033. This suggests that there is significant difference (p<0.05) between CSML and CSML-1/CSML-2. -We will add the results in the final version. [1] DOI:10.1093/brain/awaa137 [2] DOI: 10.1016/j.neuroimage.2019.01.031

Q5: Using longitudinal neuroimaging data at multiple time points or not (R3) -We only used baseline MRIs in this work, without using longitudinal MRI data. -We will point this out in the final version.

back to top

Cost-Sensitive Meta-Learning for Progress Prediction of Subjective Cognitive Decline with Brain Structural MRI