Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Dan Hu, Weiyan Yin, Zhengwang Wu, Liangjun Chen, Li Wang, Weili Lin, Gang Li

Abstract

The difficulty of acquiring resting-state fMRI of early developing children under the same condition leads to a dedicated protocol, i.e., scanning younger infants during sleep and older children during being awake, respectively. However, the obviously different brain activities of sleep and awake states arouse a new challenge of awake-to-sleep connectome prediction/translation, which remains unexplored despite its importance in the longitudinally-consistent delineation of brain functional development. Due to the data scarcity and huge differences between natural images and geometric data (e.g., brain connectome), existing methods tailored for image translation generally fail in predicting functional connectome from awake to sleep. To fill this critical gap, we unprecedentedly propose a novel reference-relation guided autoencoder with deep CCA restriction (R2AE-dCCA) for awake-to-sleep connectome prediction. Specifically, 1) A reference-autoencoder (RAE) is proposed to realize a guided generation from the source domain to the target domain. The limited paired data are thus greatly augmented by including the combinations of all the age-restricted neighboring subjects as the references, while the target-specific pat-tern is fully learned; 2) A relation network is then designed and embedded into RAE, which utilizes the similarity in the source domain to determine the belief-strength of the reference during prediction; 3) To ensure that the learned rela-tion in the source domain can effectively guide the generation in the target do-main, a deep CCA restriction is further employed to maintain the neighboring relation during translation; 4) New validation metrics dedicated for connectome prediction are also proposed. Experimental results showed that our proposed R2AE-dCCA produces better prediction accuracy and well maintains the modu-lar structure of brain functional connectome in comparison with state-of-the-art methods.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_22

SharedIt: https://rdcu.be/cyl35

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed a reference-relation guided auto-encoder with deep CCA restriction to fill the gap of awake-to-sleep connectome prediction for longitudinal study of early brain functional development. In particular, a reference-autoencoder is proposed to realize a guided generation from the source domain to the target domain.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The approach being adopted for the application is new. The authors also proposed two new metrics for experiment valiation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Introduction of the proposed method could be better presented.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducing the results can be challenging.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. What does CCA stand for? The authors need to introduce its full name before using the acronym in the draft.
2. To help readers understand better the approach, it is suggested that the authors prepare a diagram when they explain their approach in the last paragraph of section 1 “Introduction”. This diagram will be very helpful for explaining the different terms, including source domain, target domain, relation network, latent space, etc.
3. Fig. 1 needs a better explanation along with the four main steps currently introduced in the draft. There needs a high-level introduction about Fig. 1 relating to the “Training stage” and “Testing stage” when introducing the four steps.
4. The authors should justify the use of a multilayer perceptron (MLP) neural network for encoding. The MLP is known for losing spatial information in the input image data. If it’s for both embedding and then being able to reconstruct the connectome pattern image, a U-Net type neural network could work better than the MLP.
5. In Table 1, “Metrics” should be moved to the second row.
6. The paper can be improved through validation on a much larger dataset.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Despite the mentioned comments for improvement, this submission does bear some novelty for the application.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

Authors of this paper propose a novel R2AE-dCCA for awake-to-sleep connectome prediction for solving the issue of data scarcity and huge difference between natural images and geometric data.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Augmentation by including the combinations of all the age-restricted neighboring subjects as the references are used and embeddings are fused for each reference.

deepCCA loss is used to maintain the neighboring relation during translation.

Novel validation metrics dedicated for connectome prediction are proposed.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The loss (9) seems contradictory to the motivation of this work.

The testing stage has some issues.

It is unclear why clustering measurements are good for this task.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The details of experimental settings are mostly provided.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The loss (9) has a severe issue by positively adding all terms since (2) and (7) are correlations, so they should be maximized while others need to be minimized. It seems like that uncorrelation are enforced but this is contradictory to the motivation of this work.

Equation (10) in testing stage is very different from what shows in Fig. 1. • Rx should not be applied to x_test and y_Tr • The softmax function sigma needs a list of values as input instead of a single value • The weighting function W is quite ad hoc, which is different from the one used to generate age restricted neighbors in training stage. Why not use the same neighborhood selection strategy for testing?

Authors proposed to use Vin and Min for measuring the quality of connectome prediction. It is unclear why clustering measurements are good for this task since the correlation matrix does not have truly associated clustering labels. For example, authors use K-means with the number of clusters as 10. The question is why 10 is the appropriate value.

Minors: (7) is same as the second term in (6)
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The technical issues of loss function (9) and ad hoc testing stage.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper presents a reference-relation guided autoencoder with deep CCA restriction (R2AE-dCCA) for awake-to-sleep connectome prediction. Experimental results show that the proposed R2AE-dCCA achieves better prediction accuracy and maintains the modular structure of brain functional connectome in comparison with existing methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This work aims at filling the gap of data scarcity and huge differences between natural images and geometric data (e.g., brain connectome). The proposed method is developed based on reference-autoencoder (RAE), relation network and deep CCA. The proposed method is evaluated by Pearson’s correlation coefficient (r) and mean absolute error (MAE) as well as Correlation of top percentile connections, Normalized variation of information (VIn) and mutual information (MIn).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. It is not clear what are the key contributions of this work.
2. The reference format is problematic.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It is not clear whether the dataset used for experiments is publicly available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. It is not clear whether the key contributions of this work are from medical or technical perspective.
2. The reference format is problematic. Some of the references are not presented in a proper way, e.g., [9, 10]. What are the references [23-25]?
3. How is the proposed method compared with the related work [11] experimentally?
4. Can the proposed method be used to predict Sleep-to-Awake Connectome or other applications?
5. To determine the neighbor, why the threshold parameter is set as 30 days?
6. Ablation study should be performed to evaluate the effectiveness of each component and hyperparameter in the proposed method.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think this is a borderline paper. The problem to be addressed sounds interesting.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The topic is interesting and the method is novel. Please address the issues of eq.9-10 raised by Reviewer#2.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We would like to express our deep appreciation to the reviewers’ and AC’s affirmations and insightful comments on our work. The main criticism is from the issues in Eqs (9) and (10), which are actually resulted from two simple typos. By correcting these two typos, the main idea, the framework of our model, and all the mathematical description will be consistent and accurate. As developing an effective model addressing the connectome prediction between different states is of great importance in neuroimaging research community, we hope that the advantages of the proposed R2AE-dCCA will not be shaded by our careless typos, which will be corrected in the final version.

#Q1: R2 thought there is a severe issue in the loss defined in Eq (9) by positively adding all terms. Response: The confusion is led by a typo, while the main idea of our work keeps consistent in the paper and experiments. In Eq. (9), there should be a minus sign before the first item. We did require the predicted and expected connectomes to be maximally correlated and maximize the Pearson’s correlation during the training. By adding a minus sign before the L_corr, the main idea of the loss function will be accurately described.

#Q2: R2 pointed out the issue in Eq. (10) that Rx should not be applied to x_test and y_Tr. Response: The misunderstanding is resulted from a typo. As shown in Fig. 1, R_X will not be applied to x_test and y_Tr. This is a copy and paste typo. The second “R_X(” in Eq. (10) was mis-pasted from the first one. After deleting the second “R_X(”, Eq. (10) correctly matches the ideas shown in Fig. 1.

#Q3: R2 mentioned that the Softmax function should not take a single value as input in Eq. (10). Response: Putting a single value as the input of the Softmax function is a commonly simplified expression to express the computation related to each element in the vector. Actually, in Section 2.1, we have detailedly explained “relation guided fusion” and shown the full expression of Softmax function in Eq. (1). Since Eq. (10) is basically a relation guided fusion, the simplified expression of the Softmax function was leveraged to increase the readability. In the final version, we will change it to the corresponding full expression to avoid confusion.

#Q4: R2 thought the weighting function W in Eq. (10) is ad hoc and suggested to use the same neighborhood selection strategy for training and testing. Response: Applying the same neighborhood selection strategy for training and testing is not feasible because different data were involved in these two different stages. In the training stage, neighborhood selection strategy is specifically proposed to address the large amount of reference-couples obtained from random combination. Restricting the reference-couples within age restricted neighborhood will make the learning efficient and effective. However, in the testing stage, there are no reference-couples but only handful reference data. Choosing the reference data by age will lose the variability of reference and lead to low prediction accuracy. Thus, as we have described in Section 2.1-Testing stage, “with the age of reference connectome being considered into the relation guided fusion, all the connectome y_Tr in the training set are used as reference to avoid the lack of variability”. Therefore, the weighting function W in Eq. (10) is not ad hoc but a deliberate design for different data used in training and testing. We will make the idea more clearly in the final version.

#Q5: R2 questioned why the clustering measurements are good for this task. Response: Modular structure, usually obtained by clustering and validated by clustering measurements, is one of the best ways to validate the performance of connectome prediction. As we have described in Section 2.2, “The modular structure based on graph theory is one of the most important analyses for functional brain networks”, clustering measurements are suitable for the validation of our task.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I think the major questions have been addressed well. The authors should corrrect the typos in the equations 9.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper addresses an interesting problem of awake-to-sleep prediction, which is useful in the study of early brain development. The strengths of this method are a new deep learning architecture for performing CCA, domain adaptation methods for connectivity data. The main weaknesses noted by the reviewers are a potentially confusing methods presentation, an issue with the loss function in Eq. (9), and a potential double-dipping of the testing dataset in Eq. (10). I find the manuscript to be reasonably well presented, and the authors have clarified that there are typos in the two equations, which caused the confusion. Hence, I believe the paper is solid and worth presenting at the MICCAI conference.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The main issue was the concern regarding the Eq 10, and the authors addressed for it by correcting the initial typo and provided more details of the equations in the rebuttal. This has to be clarified in the revised paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

back to top

Reference-Relation Guided Autoencoder with Deep CCA Restriction for Awake-to-Sleep Brain Functional Connectome Prediction