Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Fenqiang Zhao, Zhengwang Wu, Li Wang, Weili Lin, Shunren Xia, Gang Li

Abstract

Cortical surface registration and parcellation are two essential steps in neuroimaging analysis. Conventionally, they are performed independently as two tasks, ignoring the inherent connections of these two closely-related tasks. Essentially, both tasks rely on meaningful cortical feature representations, so they can be jointly optimized by learning shared useful cortical features. To this end, we propose a deep learning framework for joint cortical surface registration and parcellation. Specifically, our approach leverages the spherical topology of cortical surfaces and uses a spherical network as the shared encoder to first learn shared features for both tasks. Then we train two task-specific decoders for registration and parcellation, respectively. We further exploit the more explicit connection between them by incorporating the novel parcellation map similarity loss to enforce the region-of-interests’ boundary consistency, thereby providing extra supervision for the registration task. Conversely, parcellation network training also benefits from the registration, which provides a large amount of augmented data by warping one surface with manual parcellation map to another surface, especially when only few manually-labeled surfaces are available. Experiments on a dataset with more than 600 cortical surfaces show that our approach achieves large improvements on both parcellation and registration accuracy (over separately trained networks) and enables training high-quality parcellation and registration models using much fewer labeled data.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_17

SharedIt: https://rdcu.be/cyhPY

Link to the code repository

https://github.com/zhaofenqiang/JointRegAndParc

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

In this paper, the JRP-Net, the deep learning framework for joint cortical surface registration and parcellation is proposed. It consists of the shared encoder, registration decoder, and parcellation decode which are connected by the parcellation map similarity loss. The experimental results with the infant dataset of about 600 cortical surfaces show that the JRP-Net outperforms other conventional methods over the test set.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is clearly written and well organized.
- Good use of figures and illustrations to explain the method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The targets from the paper seem (general) cortical surfaces, but only infant datasets were used.
- Hence, inappropriate comparison to the conventional methods such as FreeSurfer and Spherical Demons which map onto the atlas constructed with adult datasets
- Compared to the base work [14], lack of the method details for extension of the work and fundamental experiments.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- No public availability of the data, code and results, which limits reproducibility and impact of the work.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Apply the proposed method on the NAMIC dataset used in the base work [14] or the LPBA40 dataset (Shattuck et al., 2008)., and show the performance over the conventional methods.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is clearly written, but lack of fundamental experiments with adult datasets to propose a general cortical surface registration and parcellation framework.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

1
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This is a nice paper which builds on Spherical U-net to perform joint segmentation and registration. The paper uses a novel parcellation map similarity loss so that the methods benefit from each other
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The methods are very clear This is one of the first convincing surface deep learning for registration papers I’ve seen Builds on spherical u-net which is a simple and intuitive surface convolutional method Results show good improvement on both problems Convincing ablation study Validated against state-of-the-art traditional surface registration methods showing improvement in sensible overlap metrics and in run-time
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The abstract and first paragraph of this paper overlap with submission 2149 and seem to belong to that? The paper is not appropriately anonymised No details were given on initialization - I assume they must be affine aligned to start with? Description of interpolation proceedure unclear No evidence presented for how plausible the warps are (distortion maps) Figure annotations and captions aren’t always clear
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This paper uses public datasets and the baseline network is publicly available. The description of the methods is mostly clear therefore it should be feasible to replicate
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Overlooking the fairly major oversight that the abstract is a replication of submission 2149 and the fact that these papers are not anonymised. I think this is a really good paper. This makes use of the intuitive surface deep learning framework - spherical u-net - and adapts it from segmentation to segmentation and registration in a mutually beneficial way. It does so with pretty convincing results. The paper is generally well written and the network diagram is excellent. The only two questions I have is 1) over interpolation for M \circ \Phi to the fixed mesh - how is this done? 2) How is the network initialised - assumedly everything is affinely aligned? Related to point 1) the authors state in the results ‘Compared to [16] that directly concatenates F and M as input in a coarse-to-fine manner, our SE+RD architecture fuses high-level features in deep feature space, thus avoiding the time-consuming re- interpolation of deformations in original spherical space while obtaining better results. ‘ Might this be elaborated on as it isn’t clear? In general I find the results pretty convincing. The ablation study is well done and all results are validated against a state-of-the-art comparable traditional method spherical demons. I particularly like the improvements demonstrated in Fig 3 though I had to take a few passes to see them as the arrows are not explained. I would advise using different coloured arrows for different features and/or just directing readers to the parcellation maps as these seem to make the point much more clearly for the second two examples. Note Fig 3 is very different to read unless highly zoomed - I don’t know whether different figures or a better caption would help with that. I couldn’t easily see the blue and red lines. More details on the experiment using a subset of manually labelled examples would help. Does this mean the training data set here is reduced to these examples? It seems unlikely that the method could generalise well with only 10 datasets unless there is something in the initialisation or choice of sub-set that makes it a very simple problem. Certainly segmentation of cortical folding regions following affine alignment can be almost perfectly achieved through memorisation?

Finally it might be nice to comment on the orientation awareness (lack of rotational equivariance) of spherical U-net filters. For some problems this might be an issue but for registration perhaps its an advantage?
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think this is a really good extension of spherical u-net and a well written paper. I’m very slightly concerned that the results shown in fig 2 indicate that initialisation means that the c
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The paper performs joint parcellation and registration in the freesurfer defined spherical space.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The results look interesting. The neural network architecture is novel and interesting.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The cortical parcellations and registration might not be the best idea since for many brain regions, anatomical parcellation might need change in topology e.g. split sulci aligning to single sulci. The topological configurations might be different across subjects. This can lead to error in both in anatomical alignment and the parcellation. The relationship between anatomy and function is indirect and can be studied by aligning antomy first. One if often interested in functional parcellation. This point is not fully addressed. Even if I agree with premise of the paper, joint segmentation and registration has been studied, but it is not clear if cortical parcelation is appropriate for all cortical regions.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper seems to be reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The relationship between anatomical parcellations and topological variability of the sulcal patterns need to be taken into account. Different brain regions have different sulcal variability.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The novelty is limited. The relationship of parcellation and registration is not cleary understood. While most if not all software use registration as a preprocessing before doing the parcellation, the link between the two is different in different parts of the brain. Topology of the parcellation might be inconsistent across subjects. So making registration and parcellation jointly will add bias and preclude the possibility of even studying this question.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes a joint surface registration and surface parcellation approach using a spherical network. Evaluation is on an in-house processed dataset of infant brains.

One reviewer appreciates the clarity of the method, but has doubts on generalization to adult brains.

A second reviewer finds the paper convincing, but raises a concern on the implied warping of spheres with registration maps, initialization, description of the experimental subset of manually labelled brains, and the evaluation of the registration, namely the realism of the distorsion maps. Reviewer indicates copy-paste between this submission, 2146, and 2149, as well as poor anonymity. This would be serious and has been a cause of rejection at MICCAI.

A third reviewer appreciates the use of an existing spherical u-net in the context of joint registration and parcellation, it questions however the novelty, the motivation of using a structural joint registration and parcellation, versus a functional setting, as well as raises concerns on handling the topological variability of brains sulcal patterns.

The three reviewers have divergent views. Two seriously question the motivation of jointly registering and parcellating the brain, due to removing the independence of a structural alignment (registration) and the parcellation (usually intended for functional studies), and to avoiding evaluating the brain on adult brains, which would clear up doubts on the generalization to any cortical surfaces. One contrastingly finds the method highly relevant, but also raises a valid concern on why the distorsion of the registration maps are not evaluated, this is indeed crucial in evaluating a registration method, as maps can find perfect Dices, but be all distorded. There is a claim of overlaps between submission 2146 and 2149, which this AC has no access to, as well as poor anonymity in the paper, which this AC avoided investigating that matter.

For all these reasons, the author should address these valid concerns. Recommendation is towards an invitation for a Rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Author Feedback

We thank all reviewers’ comments and appreciate that the compelling aspects of our work are highlighted by the reviewers: 1) “a really good paper built with a novel and interesting network architecture” (R2, R3); 2) “first convincing surface deep learning for registration paper” (R2); 3) “well done ablation study” shows “good improvements and pretty convincing and interesting results” (R2, R3).

Response to R1’s concern on generalization to adult brains. Our method can be generalized for adult brains, because the bases of our method (the 1-ring convolution kernel, accordingly developed spherical convolution, pooling, and the backbone Spherical U-Net) are independent of datasets and generally effective for both infant and adult brains, which have been validated in [14]. Moreover, since pediatric brains after the rapid development in the first 2 years are similar to adult brains in size, shape and folding (JH Gilmore, Nature Reviews Neuroscience 2018), our dataset including subjects 0-5 years old actually contains adult-like data and thus already partially validates the generalization to adult brains. We will further validate it and clarify this in revision.

We respectfully disagree with R1’s comment “inappropriate comparison to the conventional methods which map onto the atlas constructed with adult datasets”. R1 might have overlooked Sec. 3.1, where we emphasized that we align/map all surfaces to the UNC infant atlas using different methods. Thus the comparison is fair to all methods and appropriate.

R2’s concern on how plausible the warps are. We have provided folded triangles number of surfaces after warp in Sec. 3.2, which can reflect the preservation of surface topology and thus the realism of the warps. Besides, we follow the implementation in Spherical Demons [13] using “scaling and squaring” to guarantee diffeomorphic registration, which is theoretically invertible and has been validated to be effective in preserving topology in our experiment and many other works.

As suggested by R2, more details and references of the initialization, interpolation, experimental subset will be added, and the quality of figures and captions will be further improved.

R2’s concern on anonymity and overlap of abstract and first paragraph with another submission 2149. The two submissions are both properly anonymized, without revealing any author information. Most importantly, they have distinct research topics and technical contributions. We have double checked them using a plagiarism checker tool and the overlap reported is only 1%, which can be found in an anonymized repository https://anonymous.4open.science/r/MICCAI2021_rebuttal/.

R3 questions the motivation of joint parcellation and registration based on cortical structural features while most people are interested in functional features. As pointed by R3, “the relationship between anatomy and function can be studied by aligning anatomy first”, which is exactly one aim of our work and other popular software in the field, e.g., FreeSurfer and Spherical Demons. Our method can provide 10+% registration Dice improvement for initial structural alignment than available tools. The dramatically improved initial structural alignment is of great importance for subsequent functional alignment and studies for better understanding structure-function relationship, a central theme in neuroscience. We will clarify this in revision.

R3’s concern on handling the topological variability of brain sulcal patterns. As we stated in Sec. 3.1, we assign different weights to different structural features (0.75 for coarse feature sulc, 0.25 for fine curv) to ignore the noise and mitigate the effects of topological variability of minor sulci across subjects, thus obtaining more robust registration and partially addressing the topological variability. Fully addressing the local topological variabilities is a promising direction, e.g., by incorporating functional features, but out of the scope of this paper.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have addressed most concerns, notably on generalization on adult brains, which appears reasonable as their shapes remain similar; comparison with atlas-based warps; and minor technical clarifications. The paper contributes to the field of brain surface analysis by providing joint registration and parcellation using recent methodology on learning on spherical representations. I would tend to agree with a reviewer that a quality of the registration warps is crucial to appreciate the validity of registration. Verifying triangle flips and using scaling-and-squaring is insufficient. One advice is to evaluate the Jacobian maps of the deformation fields, verifying it is realistically smooth and close to determinant 1. Concerns on overlap between two submissions have been cleared out. One reviewer have questioned the relevance of jointly registering and parcellating, due to the disparity between structure and function on the brain surface. I would stand in favor of the authors as their method provides a new methodology which could be used to further investigate relations between structure and function.

For these reasons, and its proposed advancement in brain surface analysis, Recommendation is toward Acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposes a novel deep learning method for the joint registration and parcellation of major cortical regions. The rebuttal adequately answers questions from the reviewers including overlaps and anonymity.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Extended from spherical U-net, the paper proposes an algorithm for joint cortical surface registration and parcellation. The algorithm was applied to study an infant dataset with 623 cortical surfaces. It has a convincing ablation study. The work was validated against state-of-the-art traditional surface registration methods showing improvement in sensible overlap metrics and in run-time.

The weakness is some details, such as the initialization, interpolation, are not fully provided in the manuscript. The quality of figures and captions can be further improved.

The authors did a good job in the rebuttal, clearing most of the concerns raised by the reviewers and AC.

There is a concern about the duplicate submission and poor anonymity. The author’s clarification was not convincing enough. Since the AC has no access to submission 2149. We will leave it to program chair for a decision.

An “Accept” recommendation was made to recognize its novelty, solid experimental validation and potential to inspire new work to apply deep neural networks in traditional surface registration work.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

back to top

A Deep Network for Joint Registration and Parcellation of Cortical Surfaces