Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Christian Schiffer, Katrin Amunts, Stefan Harmeling, Timo Dickscheid

Abstract

Cytoarchitecture describes the spatial organization of neuronal cells in the brain, including their arrangement into layers and columns with respect to cell density, orientation, or presence of certain cell types. It allows to segregate the brain into cortical areas and subcortical nuclei, links structure with connectivity and function, and provides a microstructural reference for human brain atlases. Mapping boundaries between areas requires to scan histological sections at microscopic resolution. While recent high-throughput scanners allow to scan a complete human brain in the order of a year, it is practically impossible to delineate regions at the same pace using the established gold standard method. Researchers have recently addressed cytoarchitectonic mapping of cortical regions with deep neural networks, relying on image patches from individual 2D sections for classification. However, the 3D context, which is needed to disambiguate complex or obliquely cut brain regions, is not taken into account. In this work, we combine 2D histology with 3D topology by reformulating the mapping task as a node classification problem on an approximate 3D midsurface mesh through the isocortex. We extract deep features from cortical patches in 2D histological sections which are descriptive of cytoarchitecture, and assign them to the corresponding nodes on the 3D mesh to construct a large attributed graph. By solving the brain mapping problem on this graph using graph neural networks, we obtain significantly improved classification results. The proposed framework lends itself nicely to integration of additional neuroanatomical priors for mapping.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_38

SharedIt: https://rdcu.be/cymau

Link to the code repository

https://jugit.fz-juelich.de/c.schiffer/miccai2021_2d_histology_meets_3d_topology

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

In this work, the authors propose to combine 2D histology with 3D topology for cytoarchitectonic brain mapping using graph neural networks.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The writing is clear and the figure quality is good.
2. The method is comprehensive and the idea of combining 2D histology with 3D topology is novel.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The authors compare results with a base model, which is also designed by authors themselves. But they didn’t perform comparison with existing work.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors agree to release the code but the dataset is not publicly available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors can perform comparison with existing work.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The results and the novelity of the paper.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

3
Reviewer confidence

Somewhat confident

Review #2

Please describe the contribution of the paper

The work proposes a deep learning approach for cytoarchitecture classification which integrates high-resolution 2D texture features with global 3D topology. The method helps to integrate neuroanatomical priors into the mapping problem.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Develop a graphs neural network (GNN) for cytoarchitecture classification which integrates high-resolution 2D texture features with global 3D topology.
2. The model adopts graphs neural network techniques such as GraphSAGE, and GAT.
3. The results show superior performance on GNN over MLP.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In terms of application, It’s seems novel and important but there are poor novelties in proposed GNN models
2. The model mostly adopted from existing GNN such as GraphSAGE, and GAT.
3. The dataset is small, only 7 postmortem human brains corresponding annotations of 113 cytoarchitectonic cortical areas.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Datast used in this work seems private.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Less novelties in the GNN model
2. Larger dataset should be utilized to validate any DL model
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. In terms of application, It’s seems novel and important but there are poor novelties in proposed GNN models
2. Poorly validated
What is the ranking of this paper in your review stack?

7
Number of papers in your stack

7
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The authors propose a graph convolutional neural network approach for cytoarchitectonic brain mapping. 3D reconstruction via rigid alignment followed by mesh creation is done to create graph-representation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Exploring 3D representation of histological brain sections via rough approximation is important since 3D information in brain applications has been shown to trump 2D approaches.
2. Using GNN to take advantage of the mesh representation.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Not generalizable to unseen brains.
2. Sub-graph to whole-graph relationship is not well explained for the GNN networks.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducible
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. It is not entirely clear if the feature encoders are the 2D approaches used prior to the proposed GNN architecture. A clearer description of this section would be beneficial.
2. The GNN architectures all have 3 layers suggesting that the sub-graphs used have 3-hop nodes and when 5-layer is tested, 5-hop information. Comparing SAGE[5] and 7-layer network and potentially deeper networks to find the effect of depth.
3. With both SAGE and GAT, generalizability to unseen brain, though marginal, appears to be with the shallower network while deeper is better for known images. Would this be attributed to the neighborhood? A deeper search on the neighborhood vs depth relationship would be interesting.
4. Prior knowledge is provided to which architecture from Table 2?
5. Comparing to the recent work can be done [1]
References:
1. Schiffer, Christian, et al. “Convolutional Neural Networks for cytoarchitectonic brain mapping at large scale.” arXiv preprint arXiv:2011.12857 (2020).
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. Investigating rough 3D approximation based brain mapping to establish potential direction of future research.
2. Initial ablation studies on depth and width showing interesting results.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

7
Reviewer confidence

Somewhat confident

Review #4

Please describe the contribution of the paper

In this work, the author proposed a graph neural network-based approach to classify continuous brain tissue sections. Graph is based on 3D reconstructed mesh surface. Node features are extracted from 2D tissue slice. 7 brains and 1 unseen brain is used for training and testing.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Idea is intuitive and feasible. Experiment is relatively comprehensive. Improvements have been achieved with proposed method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Result on unseen brain is low making model’s transferability and generalizability questionable.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Code will be made publicly available after publication. Experiment is based on in-house data.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Though the method’s performance on unseen brain is low, I think it is still useful to fill the annotation for a sparsely annotated brain. For this goal, it is desired to show result with different level of annotation sparsity. Current experiment setup did not consider this. Annotation is not sparse and the ratio between annotated and not annotated slice is fixed by 8:2. Contrastive loss is usually used for self-supervised training when sample label is not available. In proposed method, the label of sample is available. In this case, it is more straight forward to use cross entropy loss. what is the advantage and motivation of using contrastive loss here?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Human brain anatomy is of interest to MICCAI society. Though the performance of proposed method may need further improvement, it can still bring some merit to this topic.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

8
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Although this work provides a novel idea on combining 2D histology with 3D topology for cytoarchitectonic brain mapping by graph neural network, applied comprehensive methods and did extensive efforts, there are some concerns from reviewers:
1. The authors applied the graph neural network without any improvement.
2. Data in this work is small and unavailable.
3. The authors should compare their work with other current work, not only these methods they chose.
4. The performance is low on unseen brain, the authors should explain the reason.
5. The authors should explain why they choose contrastive loss function.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

We thank the reviewers for investing their time and expertise into the reviews.

One point of criticism to which we would like to respond, was that a comparison with other state of the art methods would be missing. The state of the art is already represented in our experiments by the “base” model. It implements the method proposed in [21] (please see manuscript for references), and to the best of our knowledge is the currently best performing approach for automated cytoarchitecture classification. The only difference between our implementation (base) and [21] is the final classifier: We use an MLP in place of the linear classifier to better compare the architectures to the GNN approach. We will add a reference to [21] in the respective row of Table 2 to make this clearer. Our decision to use supervised contrastive learning (defining sample similarity based on labels) is also based on the result from [21] that contrastive learning outperforms cross-entropy training for the given task.

It was furthermore noted that the GNN architectures in the proposed approach are not novel. This is correct. Our main contribution is to apply the established GNN architectures GraphSAGE and GAT to the challenging problem of cytoarchitectonic brain mapping, and demonstrate their value for combining 3D spatial context with high-resolution 2D image data for classification. Investigating other architectures or additional model configurations (e.g. layer count, as suggested by the reviewers) is an interesting direction for future research, for which the present study provides an important basis.

Another important point raised by the reviewers is the rather limited generalizability of the models to unseen brains. We agree that the performance on previously unseen brains is still not satisfactory for practical applications. In this respect, the present work confirms findings from previous studies [21,23,24] which also identified transferability to new brains as a difficult problem. However, our approach leads to a threefold improvement in performance over the baseline [21], which we consider an important contribution towards better generalizability.

The reviews also included a critical remark regarding the size of the dataset used in our experiments, on which we would like to comment. Of course, the number of subjects (8 postmortem brains) is small compared to common datasets of in vivo neuroimaging, with hundreds or sometimes thousands of different brains. The acquisition of full stacks of digital whole brain sections from human brains however is incomparably more difficult and time consuming, so that only small numbers of different brains can be expected. In fact, and to the best of our knowledge, the microscopic images and annotations of cytoarchitectonic areas used in our study represent the largest dataset of this kind available today. We also want to point out that models for cytoarchitecture classification do not operate at the level of whole brains, but at the level of cortical image patches (see Fig. 1, left) sampled from different brain sections and regions, which vary considerably in texture, staining, quality and other parameters. Each such image patch is a high-resolution image of 1024-2048 squared pixels. As stated in the paper, the graph of a single brain hemisphere contains on average 1.2 million points, corresponding to the same amount of (potentially overlapping) image patches. One brain constitutes around 3 terabyte of microscopic image data. We are currently working on a cloud infrastructure to make this large dataset publicly available.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

All concerns are well addressed. The novel idea of combining 2D histology with 3D topology for cytoarchitectonic brain mapping is appreciated.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The main contribution of this paper is to apply the GNN architectures GraphSAGE and GAT to a challenging problem of cytoarchitectonic brain mapping. This topic is of great importance in neuroimaging field. Although the dataset is small (8 subjects) and accuracy is not high, due to the difficulties in data acquistion, the data size is actually large. Authors also plan to make this highly valuable dataset public.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes a graph-based learning approach to parcellate the brain surface from very high resolution histology. The used methodology extracts a mid-surface cortical mesh, learns cortical features to define surface nodal data, and learns the cytoarchitecture labeling using a graph network to exploit the geometry of the brain surface. Evaluation is on 7 brains.

One reviewer appreciates exploiting 2D histology in a 3D topology approach.

A second reviewer appreciates the application but questions novelty in terms of GNN.

A third reviewer also highlights the new application and has minor technical questions, but blocks at a missing comparison.

A fourth reviewer appreciates the application but questions low results on an unseen brain.

The general consensus is on a potentially impactful application. The developed tools are mostly existing and combines them in order to leverage 2D histology in a 3D setting. The contribution is on the high resolution parcellation rather than methodological. The submission, however, emphasizes on the methodological novelty, which is less relevant in my opinion since, as pointed out by the reviewers, all components already mostly exist. Data is very small in number of subjects, however this is unreasonable to think they can acquire and label hundreds of such data. This is all well defended by the authors in their rebuttal. In my opinion, the paper should better focus on the provided data since it is really contributing towards a clinical contribution rather than an algorithmic contribution. In its current form, the paper proposes a pipeline, less interesting from a methodological aspect as noted by the reviewer, but which could lead to better brain maps. It constitutes an advancement from 2D methods in histology.

For these reasons, Recommendation is toward Acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

back to top

2D Histology Meets 3D Topology: Cytoarchitectonic Brain Mapping with Graph Neural Networks