Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Wonsik Jung, Da-Woon Heo, Eunjin Jeon, Jaein Lee, Heung-Il Suk

Abstract

In recent studies, we have witnessed the applicability of deep learning methods on resting-state functional magnetic resonance image (rs-fMRI) analysis and its use for brain disease diagnosis, e.g., autism spectrum disorder (ASD). However, it still remains challenging to learn discriminative representations from raw BOLD signals or functional connectivity (FC) with a limited number of samples. In this paper, we propose a simple but efficient representation learning method for FC in a self-supervised learning manner. Specifically, we devise a proxy task of estimating the randomly masked seed-based functional networks from the remaining ones in FC, to discover the complex high-level relations among brain regions, which are not directly observable from an input FC. Thanks to the random masking strategy in our proxy task, it also has the effect of augmenting training samples, thus allowing for robust training. With the pretrained feature representation network in a self-supervised manner, we then construct a decision network for the downstream task of ASD diagnosis. In order to validate the effectiveness of our proposed method, we used the ABIDE dataset that collected subjects from multiple sites and our proposed method showed superiority to the comparative methods in various metrics.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87196-3_27

SharedIt: https://rdcu.be/cyl2x

Link to the code repository

N/A

Link to the dataset(s)

Reviews

Review #1

Please describe the contribution of the paper

The study presents the application of a deep learning model for ASD classification.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The stacked denoised autoencoder was compared to autoencoder, denoised autoencoder etc. in the terms of classification performance. The results show the stacked denoised autoencoder is best.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I suggest the authors change the term ‘high-order’ used in the manuscript. Actually, the authors meant ‘high-level’. High-order is used to express the relationships among topographical connectivity in the studies of neurophysiological signal analysis. The revelant papers can be found, where a detailed explanation is given.

The parameters used in the models should be addressed. How did you determine them? Grid search? or else?

A lot of methods beyond AE-category have been used for ASD identificaton. The comaprisons to those methods should be included. The current comparison was limited within the AE-category.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It is able to be reproduced as the proceduce is introduced clearly.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I suggest the authors change the term ‘high-order’ used in the manuscript. Actually, the authors meant ‘high-level’. High-order is used to express the relationships among topographical connectivity in the studies of neurophysiological signal analysis. The revelant papers can be found, where a detailed explanation is given.

The parameters used in the models should be addressed. How did you determine them? Grid search? or else?

A lot of methods beyond AE-category have been used for ASD identificaton. The comaprisons to those methods should be included. The current comparison was limited within the AE-category.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The novelty is very limited.

The comparison is not adequate to demonstrate the superiority of the proposed method.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors proposed a novel method to classify between ASD and TC using random masking of functional connectivity (FC) matrix. They used a stacked auto-encoder with random masks to extract high-order FC representation. The method was tested on ABIDE-I dataset and showed improved classification performance compared to others.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The adopted method has a well-grounded rationale from natural images and was demonstrated effectively in the fMRI autism domain.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The ABIDE-I dataset has 17 sites and thus suffers from significant inter-site variability. The authors need to reduce this effect on the FC matrix perhaps using site harmonization.
2. (Section 3.1): The authors stated, “Further, since we remove the specific ROI connections, it helps enhance the quality of feature representations from the neurophysiological perspectives.” This sentence seems very important and I cannot understand how removing certain FC edges lead to better neuroscientific sense.
3. (Section 4.3): To quantify which FC edges contribute classification, they should report the decrease in performance with masking.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Acceptable
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Summary: The authors proposed a novel method to classify between ASD and TC using random masking of functional connectivity (FC) matrix. They used a stacked auto-encoder with random masks to extract high-order FC representation. The method was tested on ABIDE-I dataset and showed improved classification performance compared to others. Strengths: The adopted method has a well-grounded rationale from natural images and was demonstrated effectively in the fMRI autism domain. Weakness:
1. The ABIDE-I dataset has 17 sites and thus suffers from significant inter-site variability. The authors need to reduce this effect on the FC matrix perhaps using site harmonization.
2. (Section 3.1): The authors stated, “Further, since we remove the specific ROI connections, it helps enhance the quality of feature representations from the neurophysiological perspectives.” This sentence seems very important and I cannot understand how removing certain FC edges lead to better neuroscientific sense.
3. (Section 4.3): To quantify which FC edges contribute classification, they should report the decrease in performance with masking.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Despite the weaknesses, it is a solid paper.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

Method for learning first and higher order connectivity features for classification of ASD.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

large data set interesting approach
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

notation could be improved analysis seems to be falling short a little and is in part confusing
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

okay
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Language needs to be improved “represents sort of the relation between” is too colloquial

Proposed method
- “select a few ROIs”: be more precise please
- “from the input”: at this point I do not know what the input really is.
- “disregarding and remaining”: do you mean discarded?
ROI Connection Masking I found the notation very confusion. After flattening, depending on how it is done, phi might seem like a chaotic 0,1 vector. I think it simplifies and agrees more with standard connectomics notation, if you first mask and then flatten the connectivity matrix. Then you can call the “R element” the node. I also do not think R should be in real number space of dimension 116. R is between 1 and 116, right?

page 5 after equation 7: “is an original complete before masking”: not sure I follow

equation 8: is there a hat on y missing?

Experimental results: “making ratio” guess it is masking ratio

Fig 2: The red ROIs seem incomplete. Your masking matrix shows many more removed connections. Also not sure how this shows me the reconstruction part. Is this a good reconstruction now? Not sure what this figure is supposed to show me.

“We believe that these various connections can be used to explore multifaceted symptoms of ASD via further analysis.” Cool statement, but I’d like to see the reason better explained or this statement better motivated.

Fig. 3: There are clearly four groups in your proposed method. Do they relate to anything? This is, in my opinion, important to explore!
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

see comments
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presents high-order functional connectivity representation methods using a stacked autoencoder with masking to improve classification accuracy. Reviewers found the ROI connection masking, which is the key part of the proposed method, confusing. This should be clarified by addressing reviewers’ questions. Furthermore, neurophysiological perspectives were mentioned as the criteria of ROI selection. This should be discussed well in the rebuttal, with other comments from reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

(R1)Parameter settings: We performed a grid search on the learning rate in 10^{-5,-4,-3,-2}, number of hidden layers in {1,2,3}, mini-batch size in {16,32,48,64,80,96}, and ℓ2regularization in 5x10^{-5,-4,-3}. We also utilized an early stopping strategy to find the best hyperparameters of achieving the highest AUC on the validation set.

(R1)Comparative methods: In this work, we have mainly focused on the effectiveness of the representation strategies using AEs, which have been the most widely studied in rs-fMRI based disease diagnosis. By following the reviewer’s comment, we will add different architectures (RFE-SVM [Chen, 2015], RNN-LSTM [Dvornek, 2017] and DNN [Heinsfeld, 2017]) in our revised version.

(R1)Motivation and technical novelty: Many deep learning methods suffer from an overfitting problem so that they require a large number of training samples. In this paper, we proposed a novel pretext learning mechanism to discover `high-level’ representations of FCs, estimated from rs-fMRI, with a limited number of training samples within the self-supervised learning (SSL) paradigm. We demonstrated its validity by outperforming the comparative methods. To our best knowledge, this is one of the pioneering works for rs-fMRI representation in SSL. Our method can be used in various applications, for which the training samples are rare, not limited to brain disease diagnosis.

(R1)Term change: We will follow the reviewer’s comment to change the term of ‘high-order’ to ‘high-level’ in our revised version.

(R2)Inter-site variability: We have exploited a simple pool approach by aggregating all training samples from multi-site datasets and trained competing models with them. Then, our seed-based network masking strategy in SSL can be regarded as data augmentation, which mitigates site-variability so that our model can represent diverse FC patterns from multiple sites, making it efficient to learn site-invariant and class-discriminative features (Fig. 3). We will further consider exploiting harmonization methods as a preprocessing tool that can explicitly alleviate the effect of inter-site variability in our forthcoming work.

(R2, Meta)Neuroscientific insights: Our proposed SSL method by seed (ROI)-based functional networks (SFNs) masking and their reconstruction from other observed SFNs in an FC is to explicitly discover high-level relations among SFNs as well as among ROIs. By explicitly learning the high-level relations among SFNs, it becomes possible to infer inter-networks relations, which is very challenging or infeasible in the conventional rs-fMRI analysis, and to obtain new neuroscientific insights into functional connectomes. That is, rather than just focusing on representations of inter-region relations in function, our proposed method is beneficial to infer high network-level representations and thus to understand a brain function in a modular manner. We will clarify this in our revised version.

(R2)FC edges contribution in classification: The masking-based performance changes that the reviewer commented on can be obtained by a so-called sensitivity analysis. To explain and analyze the FC edges contribution, we exploited a layer-wise relevance propagation. We then obtained the relevance scores of the input FC edges to the classification output and visualized the most contributed FC edges under different conditions in Fig. 4.

(R3, Meta)ROI connection masking notations: We will change them to be clear in our revised version.

(R3)Reconstruction result: In Fig. 2, we intended to show an example of reconstruction results to validate how well our self-supervised learning reconstructed the masked elements (seed-based functional networks). We will clarify this by revising in a better way in our revised version.

(R3)t-SNE result: It seems the reviewer misunderstood the figure. In Fig. 3, we tried to compare the discriminative power of different learning strategies via t-SNE visualization. We will clarify this in our revision.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The proposed method has novelty and rebuttal helps clarify the motivation and details. Since the definition and importance of high-order (or high-level) is essential, it should be clearly described in the introduction.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Several major concerns have not been well addressed, such as the comparison with SOTA methods with different network architectures. Besides, in the rebuttal, the authors mentioned that “our seed-based network masking strategy in SSL can be regarded as data augmentation, which mitigates site-variability …” This may be incorrect since simply data augmentation cannot reduce the inter-site differences in data distribution.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

16

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Thoug some sentenses may need to be rephrased, I think this paper is very interesting, the method is solid and the results are convincing.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

back to top

Inter-Regional High-level Relation Learning from Functional Connectivity via Self-Supervision