Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Qingyu Zhao, Ehsan Adeli, Kilian M. Pohl

Abstract

Starting from childhood, the human brain restructures and rewires throughout life. Characterizing such complex brain development requires effective analysis of longitudinal and multi-modal neuroimaging data. Here, we propose such an analysis approach named Longitudinal Correlation Analysis (LCA). LCA couples the data of two modalities by first reducing the input from each modality to a latent representation based on autoencoders. A self-supervised strategy then relates the two latent spaces by jointly disentangling two directions, one in each space, such that the longitudinal changes in latent representations along those directions are maximally correlated between modalities. We applied LCA to analyze the longitudinal T1-weighted and diffusion-weighted MRIs of 685 youths from the National Consortium on Alcohol and Neurodevelopment in Adolescence. Unlike existing approaches that focus on either cross-sectional or single-modal modeling, LCA successfully unraveled coupled macrostructural and microstructural brain development from morphological and diffusivity features extracted from the data. A retesting of LCA on raw 3D image volumes of those subjects successfully replicated the findings from the feature-based analysis. Lastly, the developmental effects revealed by LCA were inline with the current understanding of maturational patterns of the adolescent brain.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_38

SharedIt: https://rdcu.be/cyl8y

Link to the code repository

https://github.com/QingyuZhao/LCA

Link to the dataset(s)

https://www.niaaa.nih.gov/national-consortium-alcohol-and-neurodevelopment-adolescence-ncanda

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed a correlation analysis method to compare longitudinal changes of different modality. And it unraveled morphological and diffusivity features coupling during brain development.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Longitudinal changes instead of cross-sectional features were introduced into a deep CCAE models.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Validation of such method to other methods might be questionable. Since for a CCA-like method, we don’t know which is better: 1. have higher correlation between modality. 2. have higher correlation with age which is not in the model. If the choice is age, is it better than if we put age in the model directly or jointly.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The data is public available and code will be released.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The main problem of this paper is the relation ship between the proposed method and age. Is it better if we use age variable directly? And could it provide better information than age.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Basically novelty of the method is considerable, but validation and logic of the paper need to be improved.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper proposed a method called Longitudinal Correlation Analysis (LCA), a self-supervised deep learning approach, which can be applied on MRI data that are longitudinal and multi-modal. The authors applied LCA on the T1- and diffusion-weighted MRIs of the NCANDA study, and the findings suggested coupled brain developmental effects, which was more robust and realistic compared to several other approaches.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A novel formulation that allows utilising longitudinal and multi-modal MRI data, which is scientifically more appropriate for studying brain development compared to approaches that rely on cross-sectional and single MR imaging contrast.
- An original way to utilise repeated measures of individuals, which makes this approach fully self-supervised. The proposed method constructed “pairs of images from two different visits” for each individual, and calculated the difference within the pairs with respect to each modality for training. The full dataset, including those not included in training, was used for evaluation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The DWI part was essentially FA-only. Due to the lack of specificity in capturing the microstructure within crossing fibers, either feature- or image-based analysis that were based on FA maps may fail to reveal developmental effects within those areas.
- The proposed method has limited clinical feasibility. The reported developmental trajectory that was at the group-level, which may not apply to specific individual whose MR imaging contrasts may be complicated by certain types of pathologies (e.g. traumatic brain injury, multiple sclerosis).
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Sufficient details have been provided regarding the models, datasets, and evaluation.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Page 4: Second paragraph, similar to I_x, please formulate the feature vectors with respect to the second-modality scenario.
- Page 5: In “Preprocessing” section, please spell the full name of LT maps.
- Page 5: Were the implementations and evaluations identical applied for DCCA, DCCAE, and LCA? Any difference?
- Page 7: Second paragraph, after Figs. 2(g) + (h), I think there’s a typo for the LSSL vs. LCA, that LCA resulted in lower AIC than LSSL.
Figures:
- Fig. 2: p_FA is more appropriate than p_DWI. Similarly, update throughout the texts, Fig captions, and Fig labels.
- Fig. 3: hard to read the ticks when they overlap with the drawings.
- Fig. 4: for convention, adjust the color scale for saliency by placing 0.25 on top, 0.1 at the bottom, and move the text “saliency” outside of the scale. For 4(e), replace “DWI” with “FA”.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The main reasons that I lean towards acceptance is that this work presents a novel formulation and solid application to a longitudinal and multi-modal dataset, with very interesting findings that were supported with comparisons to other methods (although clarity can be improved). I would like to see methods like this one to be presented, to inspire methodological development in neuroimaging to allow better understanding of brain development across the lifespan.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

The authors proposed an auto-encoder to represent the hidden correlation pattern of longitudinal changes between multimodal features (LCA).
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Their approach performs better on capturing age-related correlations of features in each modality than several cross-sectional or single modal longitudinal analysis methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

a) The major issue is the unfair comparison between LCA and other cross-sectional model. Although these model are designed for cross-sectional data, longitudinal effects are allowed as the inputs and outputs (such as delta x and delta y). Authors should test whether LCA get better results than these approaches using longitudinal data. b) Another issue is the interpretability of LCA model. For instance, the saliency maps of T1 and DWI show distinguished coupled developing brain regions. These regions are expected to be not exact same but at least in the same brain system or neuronal circles. “Complementary aspects” is really a routine explanation.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The detail descriptions of all method steps make it easy to be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

a) Authors should test whether LCA get better results than other approaches all on longitudinal data.

b) The LME fitting effects also should be repeated using bootstrapping procedure.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The unfair comparison between LCA and other models.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Reviewers in their detailed assessments agree on the clarity and the innovative and novel aspects of the paper and thus in prinicple suitability for MICCAI. Whereas the code is available on github, authors are encouraged to also make test datasets available for others to reproduce their results. Data is publicly available, and the paper states that also the code will be released. Reviewers point to weaknesses related to the lack of comparisons of the proposed LCA and other models, questions on clinical feasibility, and also questions on validation.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We thank all reviewers and AC for their insightful comments. We are encouraged that all reviewers agreed on the novelty and clarity of our paper and gave scores in the acceptance range (score 6,7,8). We hope that the concerns are addressed in the following, which uses quotes (“ ”) to cite from the submitted paper.

(R1) Why not directly include age in the model? We explained in the introduction that “Another limitation of deep learning in analyzing development is that they are often formulated as supervised learning models with respect to pseudo-marker of developmental stages, such as age. As suggested in [8], such age-based supervision is sub-optimal as brain development is highly heterogeneous in populations with similar chronological ages.” In line with this observation, our experiments focused on group-level correlation between brain development and age but did not regard age as the ground-truth of the developmental stage for each individual. We noted this in the results: “the variance in pT1 or pDWI at any given age potentially indicates that the brain developmental stage is highly heterogeneous during adolescence, which supports our claim that self-supervised learning is potentially a better approach for quantifying that stage compared to age-guided supervision.” This finding also supports the findings in [8], where factors derived from self-supervised learning are a more accurate marker for quantifying ‘brain age’ than directly using chronological age.

(R1) Is CCA or LCA better? Given that our goal is to reveal “coupled developmental effects across modalities,” LCA is clearly better than CCA. As shown in Fig 3, factors of CCA are irrelevant to brain development, which is not the case for LCA.

(R2) FA cannot model crossing fibers. LCA is a generic model that can be applied to any two types of measurements across any two modalities. We chose FA as it is the most commonly used measurement for DTI analysis. We view modeling crossing fibers as an orthogonal research direction.

(R2) Limited clinical feasibility (e.g. pathology). We note that clinical impact is not limited to studying pathologies. Understanding brain development, even in the healthy population, has been one of the most impactful topics in neuroscience for several decades.

(R2) Inference of LCA is limited to group-level The group-level trajectories were derived from a post-hoc LME analysis only for validation purposes. The proposed LCA can estimate the developmental stage for each individual visit, as shown in Fig 2 (top right) and Fig 4 (top). Therefore, one could easily study individual trajectories and characterize heterogeneity across individuals.

(R4) Missing comparison with other longitudinal approaches Other longitudinal approaches do not fit within our validation purpose. Our method being the first self-supervised method for longitudinal multi-modal MRI analysis highlights its novelty. We stated this in the paper “to the best of our knowledge, there are no unsupervised or self-supervised multi-view methods that also disentangle latent directions from longitudinal data”.

(R4) Complementary regions in the saliency maps should be in the same brain system. Given that LCA analyzed features from the whole brain, enforcing findings to be in certain regions (i.e., brain system) would require additional assumptions (e.g., regional sparseness), which are orthogonal research directions to the proposed self-supervised scheme.

(R4) LME results should use bootstrapping. The paper already presented bootstrapped results “Standard deviation of these statistics was generated based on a bootstrapping procedure”. We will further clarify that the figures show the mean curve of the bootstrapped results.

(AC) Test data should be released. The test set is the entire healthy population from the public NCANDA study.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper has novelty in its approach and methodology for multi-modal longitudinal analysis of MRI brain data. The authors rebuttal is convincing and provide sufficient information to clarify open questions raised by reviews, in particular in view of not using the calendar age but data-derived features which reflect maturation age. Authors should include some items raised by reviewers for their final version of the manuscript. This paper seems to be a good contribution to the MICCAI community.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

All reviewers are positive on this paper. Overall the method dealing with longitudinal multimodal data is novel and the experiments are solid. The rebuttal further clarifies some unclear parts in the submission.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

back to top

Longitudinal Correlation Analysis for Decoding Multi-Modal Brain Development