Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Jiadong Yan, Yuzhong Chen, Shimin Yang, Shu Zhang, Mingxin Jiang, Zhongbo Zhao, Tuo Zhang, Yu Zhao, Benjamin Becker, Tianming Liu, Keith Kendrick, Xi Jiang

Abstract

It has been widely demonstrated that complex brain function is mediated by the interaction of multiple concurrent brain functional networks, each of which is spatially distributed across specific brain regions in a temporally dynamic fashion. Therefore, modeling spatio-temporal patterns of those holistic brain functional networks provides a foundation for understanding the brain. Compared to conventional modeling approaches such as correlation, general linear model, and matrix decomposition methods, recent deep learning methodologies have shown a superior performance. However, the existing deep learning models either underutilized both spatial and temporal characteristics of fMRI during model training, or merely focused on modeling only one targeted brain functional network at a time while ignoring holistic ones, resulting in a significant gap in our current understanding of how the brain functions. To bridge this gap, we propose a novel Multi-Head Guided Attention Graph Neural Network (Multi-Head GAGNN) to simultaneously model spatio-temporal patterns of multiple brain functional networks. In Multi-Head GAGNN, the spatial patterns of multiple brain networks are firstly modeled in a multi-head attention graph U-net, and then adopted as guidance for modeling the corresponding temporal patterns of multiple brain networks in a temporal multi-head guided attention network model. Results based on two task fMRI datasets from the public Human Connectome Project demonstrate superior ability and generalizability of Multi-Head GAGNN in simultaneously modeling spatio-temporal patterns of holistic brain functional networks compared to other state-of-the-art models. This study offers a new and powerful tool for helping understand complex brain function.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_53

SharedIt: https://rdcu.be/cyl8P

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors introduced a way to jointly model the spatial and temporal patterns of multiple brain networks. From a 4D fMRI data they generate both spatial and temporal patterns 10 resting state networks (RSN) using an attention mechanism within a U-net architecture.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper is well structure and written in a professional way and results are clear and do a good job of emphasizing better results with proposed idea.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The discussion of the limitation of the work is poorly developed from a deep learning perspective.
- The authors claimed “The averaged spatial similarity of the ten RSNs shows that the proposed AG block achieves the best modeling ability, …”, however, it does not outperform with all RSNs in table 4 as only 4 results were highlighted. Further explanations of tables need to be added in the result section.
- While it is a well written paper the method still ambiguous for me. Why the multi-head attention mechanism is theoretically good to the spatial data? What is the motivation behind using those loss functions, why not other kind of loss?
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The content of the paper meets the list of reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Mathematical notations such as “overlap rate”, “AG block output” … need to be replaced with letters.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is both novel and interesting. Apart from some flaws in the explanation of tables in the result section and the unwell written math symbols in the method section the community stands to benefit from this work.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper proposed a Multi-Head GAGNN to simultaneously model both spatial and temporal patterns of multiple brain networks (10 resting state networks in this paper), and this paper shows superior ability and generalizability in modeling the patterns compared to other state-of-the-art models.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper simultaneously modeled the spatial and temporal patterns of multiple brain networks by a novel multi-Head GAGNN deep learning method. Unlike the previous work focused on modeling only one targeted brain network, ten brain networks are modeled at the same time in this paper, modeling multiple brain networks has high application value in the diagnosis of mental disorders.
2. The generalizability of multi-Head GAGNN is good: the model is trained with emotion t-fMRI and can be applied to motor t-fMRI and have satisfying similarity with the training labels.
3. The proposed multi-Head Attention Graph block shows better modeling ability than other methods (Attention, GCN and CNN).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The temporal pattern result should be compared to the task block design to evaluate the effectiveness of this method.
2. For Table1, the averaged similarity value of the ten RSNs should be provided to better evaluate the modeling capability of the holistic brain networks.
3. By using the emotion t-fMRI to train the model and then apply the motor t-fMRI data to evaluate the generalizability of the proposed Multi-Head GAGNN, the spatial and temporal similarity between models ones and training labels of motor testing data are lower than that of emotion testing data, this is not surprise because the training data is emotion t-fMRI. How about use the motor t-fMRI data as the training dataset and use emotion t-fMRI dataset, can we get the same results?
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of this paper is good. The public dataset was used to training the model and evaluation in this paper. The data processing and modeling methods are also clearly provided. The quantitative evaluation method is used to evaluate the results.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Compare the temporal pattern results and task block design to evaluate the effectiveness of this method.
2. It is more reasonable to use resting state fMRI data as the training dataset to extract the ten most representative resting state networks (RSNs). Moreover, resting state data recording is more user friendly and easier than the task-evoked data recording in clinical use.
3. The relationship among the multi-brain networks (10 RSNs in this paper) may reflect some information about brain activity. In this paper, the relative activity strength information of the 10 RSNs are lost. For motor t-fMRI data, the activity strength of sensorimotor RSN is more dominate than the other ones, but this paper did not extract this information. If relative activity strength can be provided, this will be more useful.
4. More analysis for the spatial and temporal similarity difference between the emotion data testing results and motor data testing results can make this paper better. For example, why the results of RSN5 in table 1 and table 2 are both much lower than the other RSNs?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. This paper proposed a novel Multi-Head Guided Attention Graph Neural Network to modeling spatial and temporal patterns of multiple brain networks, characterization of both spatial and temporal patterns provides a way to understand the intrinsic relationship of brain functional networks.
2. The proposed Multi-Head Guided Attention Graph Neural Network is novel and can modeling multiple brain networks simultaneously.
3. This paper is well organized and the methods and results are clearly represented.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The paper proposed a novel GAGNN network structure (multi-head) to leverage the spatio-temporal features of the fMRI data
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper clearly illustrates the novel network structure, which utlizes both the component of the medical-imaging-classic U-net network and incorporating GAGNN on the top. The network is a decent atempt of incorporating population information.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The paper, similar to many other GAN papers, is potentially an easy vicitim of overfitting. The author should generalize the methodology as well as the trained model to data other than HCP. The high quality HCP data as well as the homogeneity within HCP population together with underlying nature of GAN all pushes the methodology to the edge of overfitting to the data protocol in HCP fMRI.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of the methodology should be good especially with the author willing to provide code. I am more concerned regarding the reproducibility outside the high-quality high-homogeneity HCP dataset and if the methodology can be prompted further.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. It is unclear to me why the author decides to apply the network on the Cartesian image space than the cortical surface space, it would be more intuitive and valuable should the method be applicable to cortical surface.
2. The numerical improvement of the proposed method is not substantially/significantly better than the baseline methods. I think the author should further discuss.
3. I would like to hear the author’s discussion on the over-fitting potentials of the GAN-based network, espcially if the methodlogy could be extended to other research or clinical datasets.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think the paper is novel in term of the proposed neural network design, for the community of fMRI researchers. However the paper does lack any significant neurological novelity and clouded by the high likelihood of self overfitting.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The reviewers appreciated the novel method and improved quantitative performance. They noted a few ambiguities with the method, which the authors should clarify in their final submission.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We appreciate all comments from the reviewers. As suggested by the meta-reviewer, we itemize our responses to major ambiguities with the method as follows and will clarify in the final submission: (1) The potential overfitting problem The modeling performance based on two different task fMRI datasets from HCP indicates the generalizability of our method to different datasets with few overfitting problems. We agree with the reviewer that we could test the model on more research and clinical datasets to further validate the reproducibility of the model as well as to identify abnormal functional networks in patients to help disease diagnosis as real-world applications. (2) Limited performance improvement Although not all of ten RSNs achieve the best performances, the overall similarity of all RSNs significantly outperforms other methods in Table 4. Moreover, the spatial modeling ability of the proposed method achieves considerable improvements of 24% over ST-CNN and 31% over SR in Table 2 and Table 3. (3) Application to the cortical surface space We aim to identify holistic functional brain networks located on both cortical, subcortical, and other brain regions. The RSN #5 in this study is located at the cerebellum and cannot be identified in cortical surface space. We agree with the reviewer that we could test the reproducibility of model performance on cortical functional networks in the cortical surface space in the future. (4) Multi-head attention mechanism to the spatial data Since our aim is to simultaneously model multiple functional networks, therefore we utilize the multi-head attention mechanism to branch the deep model structure in order to model multiple brain networks via extracting the features of 4D fMRI on specific brain regions using attention. (5) Selection of loss function Since we aim to model both spatial and temporal patterns of brain networks, we adopt the widely used spatial overlap rate and Pearson correlation coefficient to define the spatial and temporal pattern similarity, respectively, and further use them as the loss functions in order to consider both spatial and temporal modeling performances. The convergence of the training process indicates the suitability of the loss functions. (6) Use of resting state fMRI In this study, we use two task fMRI datasets in order to validate the generalizability of the model. In the future, we could use resting state fMRI to validate our model and for better clinical applications.

back to top

Multi-Head GAGNN: A Multi-Head Guided Attention Graph Neural Network for Modeling Spatio-Temporal Patterns of Holistic Brain Functional Networks