Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Badr Tajini, Hugo Richard, Bertrand Thirion

Abstract

Advances in computational cognitive neuroimaging research are related to the availability of large amounts of labeled brain imaging data, but such data are scarce and expensive to generate. While powerful data generation mechanisms, such as Generative Adversarial Networks (GANs), have been designed in the last decade for computer vision, such improvements have not yet carried over to brain imaging. A likely reason is that GANs training is ill-suited to the noisy, high-dimensional and small-sample data available in functional neuroimaging.In this paper, we introduce Conditional Independent Components Analysis (Conditional ICA): a fast functional Magnetic Resonance Imaging (fMRI) data augmentation technique, that leverages abundant resting-state data to create images by sampling from an ICA decomposition. We then propose a mechanism to condition the generator on classes observed with few samples. We first show that the generative mechanism is successful at synthesizing data indistinguishable from observations, and that it yields gains in classification accuracy in brain decoding problems. In particular it outperforms GANs while being much easier to optimize and interpret. Lastly, Conditional ICA enhances classification accuracy in eight datasets without further parameters tuning.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87196-3_46

SharedIt: https://rdcu.be/cyl2X

Link to the code repository

https://github.com/BTajini/augfmri

Link to the dataset(s)

Public datasets.

Reviews

Review #1

Please describe the contribution of the paper

The authors present a novel method (conditional ICA) able to augment fMRI studies (both at rest and task-related), whose outputs are harder to distinguish from real studies using three different classifiers (linear and non-linear models), that can help in improving the classification power of classifiers (when such output is included in the train stage) more or as much as state-of-the-art methods (GANs and CGANs), and that can generate its outputs much faster than these methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

An overall excellent clarity/organization/structure of the manuscript. A novel methodology (less complex and faster to execute than state-of-the-art methods) A thorough evaluation using multiple datasets
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

None
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors state that the code will be shared, I’m guessing that will happen after (if) accepted.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I really enjoyed reading this manuscrit, the work that you presented is very promising. I would suggest to include (or at least comment on) the results that GANs and CGANs yielded in the other 8 datasets. If you didn’t run that experiment, I would advise you to run it, so that you could more specifically comment on the comparison between those methods and the proposed method. Regarding the supplementary file, even though the results are extremely clear, I would suggest to add the characteristics of the experiment that was conducted to obtain the results shown in Table 6 (appendix), as well as the characteristics of the computer(s) used. Also, again for the supplementary file, I’m not sure why you have tables and figures in the order they are (for example, Table 6 shows up in the manuscript before Table 4).
Please state your overall opinion of the paper

ground-breaking (10)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The computational simplicity of the methodology compared with GANs and GANs, which results in processing times 200 faster. The results of the fake vs real experiment, in which they show that the augmented data being generated is the hardest to discriminate against original data. The results of the classification accuracy experiment, in which they show that by including the augmented data during the train stage, classification tasks yield better results.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper proposes a much simpler and faster approach for generating imaging datasets as opposed to the state-of-the-art GANs. Conditional ICA, or ICA conditioned for multiple specific classes, applied to fMRI brain datasets of resting states and tasks states, was shown to be faster, generated more realistic examples of synthetic images of rest state images, and improved classification accuracy of task state images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Rather than going more complex with some novel deep learning architecture, this paper took a step backward in a sense and showed the power of taking a well-known methodology, ICA, and conditioning it for a very specific imaging problem for which GANs may not be well suited to describe the distribution. Very unique in this solution.
2. There are over 1000 subjects considered in this study, spanning at least 8 different datasets.
3. Testing of this method compared to state-the-art methods was fairly robust for a conference manuscript.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The writing style and organization of the paper is hard to follow. The introduction doesn’t quite setup the justification for using conditional ICA. Your method of choice, “conditional ICA” is not very well distinguished about what makes it “conditional” compared to traditional ICA. The experimental design and results are in the same section together and ought to be split up more clearly. Some edits from a native English writer would be helpful
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Excellent. Above standard. Tested on multiple datasets in a cross validation setting and had independent testing.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please see “major weaknesses” above. Most comments are just about clearly explaining your rationale for choosing conditional ICA and explaining what is “conditional” about it.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- unique solution (not building off some prior network)
- HUGE number of subjects included in this study
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The authors present a data augmentation approach for fMRI data using conditional independent component analysis. They argue that augmentation methods used in other imaging such as GANs are not appropriate in fMRI due to the limited amount of data available and the high dimensionality of the input.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors demonstrate that images produced by other methods such as GANs do not lead to performance increases, likely because of the limitation in training data. The proposed method is able to produce images the yield some performance increases over baseline classifiers. The authors also show that
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The classification improvements seen when using the proposed method are relatively small and there are no statistical tests performed to show significance.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors will share the code and all data is from publicly available sources.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors firs present results on the HCP dataset and then expand to a analysis of eight different datasets. It isn’t clear why the HCP dataset was singled out. It could be cleaner to have a single evaluation using all datasets rather than one with HCP and then a repetition with the others. One minor note is that it seems that HCP is included in the set of eight datasets, so they should not be described as “the other 8 datasets.”

In table 2, the baseline LDA performance is higher than any values in the table other than LDA covariance and conditional ICA. The gains using these two methods are both below 1%, with conditional ICA showing the best performance with a 0.8% improvement over baseline. I have a few points related to this table. First, the improvement of 0.8% contradicts the text which states that the improvement of the proposed model is over 1% for all classifiers. Second, the detail in this table seems to be unnecessary in the main paper since the overall message seems to be just that none of the baseline methods seemed to help while the proposed method achieved modest gains. Finally, it is unclear why the random forest was dropped from this analysis. The justification was that it did not perform as well as the other classifiers, but the MLP performed substantially worse than the other two as well and it was still included.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed model is interesting, but the gains seen when using this augmentation method are all relatively small and statistical tests are not performed to show significance.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #4

Please describe the contribution of the paper

Data Augmentation technique is introduced to generate fMRI data. This technique is based on conditional Independent component analysis. They have compared their work with GANs and utilized eight datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

They have used conditional ICA for augmentation. This is computationally faster in comparison to deep learning GAN networks. This is significantly useful in the health care world because fMRI data are expensive and scarce. It is prominent work in this research area.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In comparison, the authors have mentioned general GANs references of 2014 papers.
2. Authors can compare with reference 29. It is good to compare with the recent work in the fMRI domain.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Public datasets are used.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please compare with the recent work also in the same domain. And if you find that both work are equivalent, you can even point your significant findings, in terms of computation time and others.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It seems significant work. Because of the application of ICA and it makes the work interpretable also and computionally less expensive as compared to GANs.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors present a novel method, named conditional ICA, for data augmentation for resting-state fMRI and task-related fMRI.

The key strengths include: 1) Since fMRI is very expensive, the topic of this paper (data augmentation) may be of great implications in clinical. 2) The novelty of the proposed method is very strong and also this method is computationally faster in comparison to deep learning GAN networks. 3) Evaluations on multiple independent datasets

The key weakness include 1) No statistical analysis to show the significance of the improvements.

Therefore, some feedback from the authors is necessary for the further evaluations.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Author Feedback

We are deeply grateful to the reviewers for their thoughtful feedback. We are pleased that they found our approach strong, clear and novel (R1) and being evaluated by extensive/fairly robust/convincing experiments (R1,R2,R3,R4). The reviewers issue a concern with missing statistical tests that we plan to add to the paper (see Table 1 below). The code will be made publicly available along with a benchmark for reproducibility. The reviews also called for extending our work, by comparing our method as reported by R4 with ref [29], which is a recent work in the same domain, and by performing experiments with GANs and CGANs yielded in the other 8 datasets as suggested by R1. We strongly agree that this research question is worth exploring; yet GANs and CGANs are prohibitively expensive in our setting, and we plan to address such scenarios in future work.

We are addressing the comments of AC2 and R3 below and will incorporate all feedback: To further investigate the significance of differences between the proposed approach and other state-of-the-art methods, we perform a t-test for paired samples on HCP task fMRI data across 5 folds with the same hyperparameters (see Appendix - Table 4). Most notably, the proposed method performs significantly better than other data augmentation techniques. Given the large size of the HCP task data evaluation set, this significance test would demonstrate that the gains are robust. The corresponding results, including the p-values, are shown next:

TABLE 1. Statistical significance results: We display the p-value of the paired t tests comparing the proposed augmentation scheme with each other augmentation, for three different classifiers. The proposed scheme systematically outperforms alternative. Models || LDA || LogR || MLP ____ Original || P ≤ 0.01 || P ≤ 0.05 || P ≤ 0.0001 ICA || P ≤ 0.0001 || P ≤ 0.001 || P ≤ 0.001 COV. || P ≤ 0.05 || P ≤ 0.01 || P ≤ 0.05 ICA + COV. || P ≤ 0.0001 || P ≤ 0.0001 || P ≤ 0.01
GANs || P ≤ 0.001 || P ≤ 0.001 || P ≤ 0.0001 CGANs || P ≤ 0.001 || P ≤ 0.01 || P ≤ 0.0001

More specifically, these results indicate that:

Performance gains with the proposed approach are very stable across folds, which renders them statistically significant.

Our approach achieves the best results on most datasets, demonstrating that the proposed method can generate more realistic examples of synthetic images and may have further impacts on clinical settings.

Additional minor concerns also require clarification and we appreciate the generous comments of the reviewers. Among them, two are mentioned by R3 regarding the HCP dataset and the removal of Random Forest from the analysis. The HCP is a benchmark dataset, and therefore it was decided to assess the augmentation methods on this dataset. Then, as a second step, we investigated whether the best of these methods could be extended to other datasets. With regards to dropping Random Forest, the accuracy is significantly lower than the other classifiers (see Table 2 below), the only exception being where Random Forest outperforms MLP with all augmentation methods (Original, GANs, CGANs) as shown in (Table 2 - main paper).

TABLE 2. Classification accuracy on task HCP dataset. We report the mean accuracy of Random Forest across 5 splits. Models | RF
_________ Original | 0.782
ICA | 0.778 COV. | 0.780
ICA + COV. | 0.780 GANs | 0.780
CGANs | 0,779 Cond. ICA | 0.783

The last concern is pointed out by R2 requesting an explanation of the ”conditional” compared to the traditional ICA. In a nutshell, conditional ICA is a shortcut to express that the model provides class-specific distributions, while leveraging the ICA model. As such, it is not an novel ICA method.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I think the authors have done a good job in addressing the reviewers’ questions. I would like to recommend to accept.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper introduces a simple approach for data augmentation that seems to outperform the popular GAN. This is a timely result, and the authors did a good job of addressing the concerns.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I think this is a very interesting paper and the method is solid. But I also agree that if the authors will extend this to jounal version, they need to provide more statistical analysis, comparasion and validation.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

back to top

Functional Magnetic Resonance Imaging data augmentation through conditional ICA