Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Gia H. Ngo, Minh Nguyen, Nancy F. Chen, Mert R. Sabuncu

Abstract

Most neuroimaging experiments are under-powered, limited by the number of subjects and cognitive processes that an individual study can investigate. Nonetheless, over decades of research, neuroscience has accumulated an extensive wealth of results. It remains a challenge to digest this growing knowledge base and obtain new insights since existing meta-analytic tools are limited to keyword queries. In this work, we propose Text2Brain, a neural network approach for coordinate-based meta-analysis of neuroimaging studies to synthesize brain activation maps from open-ended text queries. Combining a transformer-based text encoder and a 3D image generator, Text2Brain was trained on variable-length text snippets and their corresponding activation maps sampled from 13,000 published neuroimaging studies. We demonstrate that Text2Brain can synthesize anatomically-plausible neural activation patterns from free-form textual descriptions of cognitive concepts. Text2Brain is available at https://braininterpreter.com as a web-based tool for retrieving established priors and generating new hypotheses for neuroscience research.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_57

SharedIt: https://rdcu.be/cyl83

Link to the code repository

https://github.com/sabunculab/text2brain

Link to the dataset(s)

https://github.com/neuroquery/neuroquery_data

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes Text2Brain that allows researchers to retrieve relevant neuroimaging experiments and generate relevant activation maps via free-form text queries using a transformer-3D CNN based approach.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

1) The paper presents an interesting application of BERT+ CNN models to synthesize activation maps from publications of neuroimaging experiments

2) Enabling synthesis of information from existing research involves the construction of domain specific ontologies, a process which is time consuming and involves a lot of effort. The method proposed in this paper overcomes this limitation by using a model to encode domain-specific knowledge during the training process.

3) The paper clearly describes the data collection, training methodology and clear evaluation with well established baselines like Neurosynth and NeuroQuery
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Both Neurosynth and Neuroquery provide a list of publications and keywords that were used to generate the activation map. However, the black box nature of this method – as is the case with all neural network methods – seems to indicate that it may not be possible to retrieve a list of relevant publications for the generated activation map.

The paper does not sufficiently discuss the limitations of the proposed approach.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors mention that the data and source code for training the model will be publicly available upon publication. Further, the text2brain software will be made available via a web interface. I believe this work is reproducible and the work can benefit the community.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The application of scibert + CNN to generate activation maps is an interesting application. However, employing such DNN models comes at the cost of interpretability. For example, NeuroQuery provides a map of terms that were used to expand the query as well as their weights in the brain map. It would help if the authors could include a way (e.g, visualizing the embeddings) to interpret the results in the final version of their tool

It would also help if the authors could elucidate the limitations and pitfalls of the tool. For example, the Neurosynth paper discusses the limitation of the tool in answering queries, verification of plausible activation maps , queries that return incorrect results to name a few.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper proposes a tool that leverages the recent advances in NLP to improve upon the existing meta analysis tools for neuroimaging studies. The paper is well written with clearly specified data collection, training and evaluation procedures. The authors mention that the tool would be made publicly available upon publication. I believe that this tool would greatly benefit the community.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The proposes a novel method named ‘Text2Brain’ for generating activation maps from the free-form text query. Text2Brain consists of a text encoder and a 3D CNN as the image decoder. The experimental results show that the method is powerful and has excellent potential.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The major strengths of the paper are as follows:
- The paper is well written and easy to follow.
- On a novelty level, the paper has a highly original contribution
- On a methodological level, it is a reasonable appropriate method
- The evaluation carefully tests the impact of different parts of their approach and the results show that the method is powerful and has excellent potential
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Technical explanation could be more thorough. Although the paper provides some citations. However, some key steps deserved more explanation.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors mentioned that the model’s source code will be made public upon publication.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Technical explanation could be more thorough. Although the paper provides some citations. However, some key steps deserved more explanation.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is well written, and the method in the paper is novel for generating activation maps from free-form text query.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

This paper present a Text2Brain model which is a neural network approach for coordinate-based meta-analysis of neuroimaging studies to synthesize brain activation maps from from free-form text query.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This work compared the proposed method with other two existing methods, Neurosynth and Neuroquery, by different ways including 1) Predicting activation maps from article titles, 2)Predicting activation maps from contrast descriptions, and 3) Predicting self-generated thought activation map. The experiments are abundant.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In Section 3.1, this paper mentioned “From the dataset of 13000 articles, 1000 articles are randomly sampled as the test set such that the keywords (defined by the articles’ authors) are not included in the training and validation articles. “ However, some keywords of different papers can be the same. Therefore, the testing data may be also included in the training dataset, which will make the testing results better than the actual performance of the model.
2. This paper aimed to synthesize brain activation maps from free-form text query, thus, the more specific the input text, the more accurate the obtained activation map. However, Section 3.1 predict activation maps from article titles which may not include detailed information. For example, some Alzheimer’s disease related articles may not include details of data modality in the titles, but different modality, such as structural MRI and functional MRI, may have different activation map because this brain disease will have different influence on brain function and structure. Therefore, this paper should give more discussion about why using the article titles to predict activation map.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

the code and data can be released after accepted
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. For section 3.2 this paper mentioned “In our analyses, we use the agreement between the IBC and HCP maps as a measure of reliability.”, it is not clear about how to use this kind of agreement in the experiments. In Section 4.2, it seems only HCP maps are used to train the model and the IBC average contrasts is also evaluated by the HCP maps.
2. In section 3.2, this paper used the contrast descriptions from IBC while used the group-average contrast maps from HCP, and there are subtle difference between the IBC and HCP experiments. It is better to discuss if these difference will have influence on the experiments of this paper.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Firstly, I pay attention to whether the topic that the paper aim to study is meaningful. Secondly, I pay attention to if this paper has a comprehensive understanding of the topic including the analysis of problems and advantages and disadvantages of the existing methods. Thirdly, I will pay attention to the method proposed by this paper, if the design logic is clear and if the evaluation methods are reasonable and comprehensive.

Specifically, for this paper, the topic of this paper is meaningful and the summary of related work is good. However, for the method and experiments parts I have some concerns which are listed in block4 and block7.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

8
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

I think this paper focuses on a very interesting and important topic in the field of neuroimaging analysis. All the reviewers have a high evaluation of this paper, including motivation, method, results, and writing. The authors need to address some minor questions raised by Reviewer#1 and #4 in the final version.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

N/A

back to top

Text2Brain: Synthesis of Brain Activation Maps from Free-form Text Query