Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Niharika Shimona D’Souza, Mary Beth Nebel, Deana Crocetti, Joshua Robinson, Stewart Mostofsky, Archana Venkataraman

Abstract

We propose a novel matrix autoencoder to map functional connectomes from resting state fMRI (rs-fMRI) to structural connectomes from Diffusion Tensor Imaging (DTI), as guided by subject-level phenotypic measures. Our specialized autoencoder infers a low dimensional manifold embedding for the rs-fMRI correlation matrices that mimics a canonical outer-product decomposition. The embedding is simultaneously used to reconstruct DTI tractography matrices via a second manifold alignment decoder and to predict inter-subject phenotypic variability via an artificial neural network. We validate our framework on a dataset of 275 healthy individuals from the Human Connectome Project database and on a second clinical dataset consisting of 57 subjects with Autism Spectrum Disorder. We demonstrate that the model reliably recovers structural connectivity patterns across individuals, while robustly extracting predictive and interpretable brain biomarkers in a cross-validated setting. Finally, our framework outperforms several baselines at predicting behavioral phenotypes in both real-world datasets.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_59

SharedIt: https://rdcu.be/cyl85

Link to the code repository

https://github.com/Niharika-SD/Matrix-Autoencoder

Link to the dataset(s)

https://www.humanconnectome.org/study/hcp-young-adult/data-releases

Reviews

Review #1

Please describe the contribution of the paper

The paper present an encoder-decoder architecture to learn association between structural and functional connectomes of subject groups. The encoders are then used to build predictors of clinical and behavioural measures of subjects.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors attempt to learn associations between structural and functional connectome by using an encoder-decoder architectures. The encoders trained on an ensemble of subjects are used to predict clinical and behavioural measures. The authors show that the predictors built with the learned encoders are more accurate on two large datasets.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There is no novelty in the methods: standard encoder-decoder is used to learn association between structural and functional connectome. The method of subnetwork identification is rather crude and I don’t think they corresponds to functional networks or pathways.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The architecture is explained clearly and rather straightforward to implement. No codes are made available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors attempt to learn associations between structural and functional connectome by using an encoder-decoder architectures. The encoders trained on an ensemble of subjects are used to predict clinical and behavioural measures. The authors show that the predictors built with the learned encoders are more accurate on two large datasets.

There is no novelty in the methods: standard encoder-decoder is used to learn association between structural and functional connectome. The method of subnetwork identification is rather crude and I don’t think they corresponds to functional networks or pathways. I presume a sizeable training dataset is still required to avoid overfitting.

The architecture is described clearly and rather straightforward to implement. No codes are made available.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I don’t see any novelty in the methodology. Predictors are not compared thoroughly with existing methods. I doubt methods work on small datasets.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The paper proposed a matrix autoencoder to map functional connectome to structural connectome, as guided by subject-level phenotypic measures.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The method provided a way to simultaneously align functional and structural connectivity and predict the phenotypic measures, and validated through a secondary dataset.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The idea of the method is not quite novel, and the motivation to align functional and structural manifolds is not clear. Is there any specific physiological significance for it, especially in a neuroimaging application? In addition, the prediction performances for fluid intelligence with HCP dataset were not very high in a limited dataset(275 subjects), and the conventional machine learning methods could also attain to this accuracy.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors didn’t provide open-source codes in the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. the detected FC bases were not very concise, which was not interpretable enough to understand the fluid intelligence.
2. I wonder whether the conventional fusion algorithms for structural and functional connectivity may perform better than the current method for phenotype prediction. What is the potential clinical usages for the method since it doesn’t achieve an obviously better accuracy?
3. To align structural and functional conectivity is not to fuse them together, why will it achieve better prediction results than using every single modality?
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The motivation behind the method and the potential usages for the method
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper proposes a novel matrix autoencoder to: (1) reconstruct functional connectivity (2) map functional connectomes from resting-state fMRI (rs-fMRI) to structural connectomes from Diffusion Tensor Imaging (DTI) (3) predict behavioral phenotypes Although the data used in the experiment is limited, the results show that the method is powerful and has excellent potential.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The major strengths of the paper are as follows:
- The paper is well written and easy to follow.
- Simple and intuitive pipeline design that is shown to reliably predict structural connectomes and robustly extract interpretable biomarkers
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The major weaknesses of the paper are as follows:
- Functional and structural connectomes are non structured data where GNNs achieved a great success on this type of data. However, this paper uses 2d FC-NN. One needs some justification for this choice.
- Each connectome has its unique topology which should be preserved when generating the target brain graphs. This paper does not address this issue
- There are many alignment methods which are more recent than the alignment manifold used in this paper. Justification is needed in this choice. Furthermore, a comparison with some state-of-the-art methods (VAE, AVAE) could stronger the manuscript and make the results more convincing.
- Source and generated connectomes have the same size which limits the generezability of the proposed method.
- There are several pioneering works that are overlooked in the introduction. There are several pioneering works that are overlooked in the introduction [1, 2, 3]. What are the limitations of these works to cover by your work?
- Minor weakness: In Fig. 1 (Blue Box): the predicted FC is named Ã not A
[1] Bessadok, A., Mahjoub, M.A., Rekik, I.: Topology-aware generative adversarial network for joint prediction of multiple brain graphs from a single brain graph. International Conference on Medical Image Computing and Computer-Assisted Intervention (2020) 551–561. [2] Zhang, W., Zhan, L., Thompson, P., Wang, Y.: Deep representation learning for multimodal brain networks, Springer (2020) 613–624 [3] Liu, Y., Pan, Y., Yang, W., Ning, Z., Yue, L., Liu, M., Shen, D.: Joint neuroimage synthesis and representation learning for conversion prediction of subjective cognitive decline. International Conference on Medical Image Computing and Computer-Assisted Intervention (2020) 583–592
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors could share their code to encourage the reproducibility of the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I think that the paper might benefit from explaining why they doing the things instead of directly jumping into the how.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- Clarity of writing and formalization.
- Clinically, this framework could increase the amount of information about brain connectivity in a subject with a limited number of scans.
- The method in the paper is novel in brain network prediction results.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This is an interesting work and it has relations to some pioneering work in this field. One goal of this paper is to study the association between brain structural and functional connectome, which is related to the last year’s MICCAI paper (Ref 40). Fig 3 shows the functional bases which are similar to the results using sparse learning or ICA. For example, the patterns of some bases are similar to resting-state networks. The authors need to address the following concerns: 1) If my understanding is correct, part of the model is trying to predict individual SC from FC. Therefore, this paper seems to estimate a one-to-one mapping. Why not use regression directly? If there is no comparison with simple regression, it is difficult to justify the need for the proposed model; 2) if this model enforces a one-to-one mapping between SC and FC, this assumption is wrong. Though the SC is relatively stable using cross-sectional data within a short time window, the FC is not unique. For example, if you use a sliding window or a different length the FC will be different. That is the reason why ref[40] needs to using GAN to handle this potential many-to-one relationship; 3) In the paper the authors used 15 as the number of bases, how this number comes? This is the most important parameter which is similar to the number of components in ICA or dictionaries in sparse learning. This work will be very limited in reproducibility without sufficient discussion of this number.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

11

Author Feedback

We thank the reviewers and meta-reviewer for their careful evaluation of our work. Below, we clarify misunderstandings and address individual concerns.

REVIEWERS 1 & 2: Both reviewers seem to have misunderstood the motivation, novelty, and interpretability of our model, all of which were noted by the Meta Reviewer and Reviewer 3.

To clarify, we have strategically designed an end-to-end neural network for two complementary tasks: 1) to learn a mapping from the rs-fMRI functional connectivity (FC) to the DTI structural connectivity (SC) space, and 2) to jointly predict multiple phenotypic characterizations. Our framework is not “a standard auto-encoder”, as claimed by Reviewer 1. Instead, we have designed 2D FC-NN layers that capture the underlying geometry of the FC connectomes via a structured low rank matrix decomposition. The phenotypic prediction branch ensures the learned 2D FC-NN weights (i.e., brain bases) are simultaneously predictive of the desired clinical phenotype. Thus, unlike conventional deep learning architectures, our framework learns compact and interpretable functional subnetworks, akin to data-driven decompositions such as PCA and ICA. Likewise, the SC decoder and loss is designed to explicitly capture the geometry of the SC matrices to robustly predict individual SC patterns, while also acting as a second regularizer on the FC projection.

Lastly, while Reviewer 2 is correct in that structural-functional alignment vs fusion are different strategies, there is no concrete evidence in the literature than one is better than the other for phenotypic prediction. In fact, our experiments on two datasets of moderate size indicate that our proposed (alignment) method provides better generalization than deep-learning based fusion (BrainNetCNN baseline).

REVIEWER 3: The primary concern is on preserving the connectome topology. By construction, our 2D FC-NN transformations within the encoder-decoder exploit and maintain the positive semi-definite FC matrix structure. Similarly, the SC alignment decoder, coupled with the SC loss function, ensures that the SC topology (which differs from FC) is preserved in the reconstruction. Finally, our framework is agnostic to the parcellation and can be adopted to operate with other atlases. As opposed to the mentioned approaches (GANs, VAEs, GNNs), our framework provides a computationally inexpensive, yet reliable and interpretable alternative to model the interplay between function, structure, and behavior.

META REVIEWER: The meta-reviewer questioned the need for the proposed model in lieu of simple regression between the FC and SC features. We expect that the estimation of a regression model between SC and FC features would be ill posed (>6000 features with roughly 300 samples). In contrast, our framework is constrained to learn a series of linear transformations between the FC correlation matrices and the SC similarity matrices that mimics a structured low-rank decomposition and leverages the geometry of both spaces. The second critique is about the validity of a deterministic (one-to-one) mapping between FC and SC, as compared to the GAN in ref. [40]. In fact, both our approach and [40] use deterministic neural networks to map FC to SC. Like a GAN, our matrix autoencoder can also learn to map different FC matrices to similar SC representations, if needed. In comparison with [40], we offer two key benefits. First, our bases are interpretable. Second, we present a quantitative evaluation of the recovered SC matrices (Figs. 2 main & supplementary) with baseline analyses showing that differences in SC among individuals are preserved. In contrast, a careful read of [40] reveals only qualitative assessments with little evidence (e.g., baseline comparisons) that the complexity of a GAN is required for mapping FC to SC. Lastly, to clarify the question about our choice for the number of bases, we set K=15 using an auxiliary validation set, as stated in Implementation Details.N/A

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

As I mentioned earlier, this is an interesting paper. Though I still have some questions and concerns, I believe this work deserve to be published and other researchers may benefit from this work. So I would like to recommend to accept. Here, I also list my questions/concerns to the authors feedback, and hope they can consider that in their final version or their journal paper. First, if the task is to estimate the relation between FC and SC, the sample should be the FC-SC pair. Since each individual has one SC and multiple FCs, simple regression may be conducted and examined as potential baseline. Second, I agree the AE can be justed to learn different FCs, but it seems that it was not designed in this paper. Third, the authors claimed the bases are more interpretable, but their results did not show the reproducibility of the bases. And I doubt if the bases will be stable if using different initialization settings. In that case, the intepretability of one set of bases is not sufficient. Last, the validation of ref[40] is not qualitative, they have compared to the averaged SC and individual SC. I think the similar strategy was adopted in this paper, too.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

20

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors did well in rebuttal. Most of the questions and concerns raised by reviewers (including meta reviewer) are well addressed.More importantly, the idea of integrating two tasks (mapping FC to SC and predicting phenotypic features) into one framework is appreciated.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper combines existing works to predict diffusion connectivity from functional one with alignment with the aim to predict target scores. Although in the rebuttal the authors have addressed the major concerns raised by the reviewers, they did not justify well the innovative aspects of their methodology. AC scanned through the paper and believes that this paper studies an interesting research problem, but the claim that GNN are more computationally expensive than FCN is devoid of evidence. Node-based convolution is generally less expensive than edge-based convolution and this depends on the size of the deployed networks (FCN or GCN). A few landmark works on multimodal/multiview brain connectivity generation were overlooked. The argument of discarding a whole GCN/GNN literature on brain connectivity is unconvincing. These need to be properly discussed in the Introduction. Also the authors mentioned in their rebuttal that GNNs lack interpretation, whereas there are several works on GNN interpretability and explainability. More convincing empirical evidence or theoretical evidence about the rationale behind the choices made for engineering such framework in the light of recent state-of-the-art is to be provided before submitting to yet another top venue.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

back to top

A Matrix Auto-encoder Framework to Align the Functional and Structural Connectivity Manifolds as Guided by Behavioral Phenotypes