Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Karthik Gopinath, Christian Desrosiers, Herve Lombaert

Abstract

Commonly-used tools for cortical reconstruction and parcellation, such as FreeSurfer, are central to brain surface analysis but require extensive computation times. This paper proposes SegRecon, a fast learning approach where an integrated end-to-end deep learning method does simultaneously reconstruct and segment cortical surfaces directly from an MRI volume, all in a single step. We train a volume-based neural network to predict, for each voxel, the signed distance to the white-to-grey-matter interface along with its corresponding spherical representation in the registered atlas space. The continuous representation of the spherical coordinates enables our approach to naturally extract an implicit isolevel surface for its reconstruction and obtain the parcel labels from the spherical atlas. We illustrate the advantages of our method with thorough experiments on the MindBoggle dataset. Our parcellation results show more than 4% improvements in average Dice accuracy with respect to FreeSurfer and a drastic speed-up from hours to seconds of computation.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_61

SharedIt: https://rdcu.be/cyl87

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This is a really well written and motivated paper which implements a novel approach for propagating cortical folding based parcellations to new subjects through using a deep learning framework to learn a spherical mapping to a template space. At the same time the method extracts cortical surfaces from images by learning a distance transform summarising the distance from each voxel to the inner cortical boundary. I viewed it as a deep learning implementation of multi-template label fusion which is a nice idea and allows for improving segmentation accuracy by accounting for variability by fusing results from multiple atlases.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Well written and well motivated Good literature review Simple but clever approach to cortical segmentation Implements label fusion to improve precision Significantly increases the speed and accuracy of cortical parcellation Significantly improves the speed of surface mesh extraction (although the accuracy of this is not validated) Uses a publicly available dataset
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Does not (appear to) validate the accuracy of the extracted surface meshes
- Would not translate well to functional parcels or parcellations that are topographically variable (and thus cannot be aligned with registration e.g. Glasser et al Nature 2016)
- Captions of figure 2 is not sufficient to understand the colour coding or purpose of circles
- Some details of how the evaluation was performed is missing
- Arguably limited clinical applications given the field is moving away from summarising cortical metrics within large folding based ROIs
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

As far as I can tell the code is not available but the data set is and there is more than enough details in the paper to replicate the analysis
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

As stated above I think this is a good paper. Its well written, put into clear context with respect to previous literature and solves the problem of cortical segmentation in a simple but effective way, avoiding the need for working with surfaces and surface deep networks at all - which given the fact these are less well developed than 2D/3D deep learning is not a bad thing. There’s no doubt that framing the labelling problem like a multi-template label fusion is a sensible approach. It’s long been known that cortices vary significantly in shape and that improved segmentations can be achieved by fusing the results of propagating multiple atlases which capture this variation. My reservations come around the apparent lack of validation of the surface mesh fitting, which is barely mentioned at all in the results and the discussion - here I am assuming the Hausdorff distance is for ROI boundaries not mesh boundaries? One understands that the regional dice overlap is capturing some of this but it would be good to see the surface fitting on 2D axial slices. Then there is the lack of detail on some of the validation details, for example how is the GCN and random forest prediction models implemented? I also wonder how multi-template fusion with freesurfer would compare? Finally, there isn’t much way of a clinical motivation for this paper therefore it may be useful for the authors to comment on the translational goals and potential applications for the method with examples.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall I like this paper but think that the validation is lacking in some areas. The paper would be much stronger if it was clear that the surface extraction was working really well.

I also have a few reservations around utility for the field as in my opinion cortical folding based atlases and the topological constraints of image registration limit methods in capturing the full topographic variability of cortical organisation and thus brain function.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

Deep learning-based joint surface reconstruction and registration method. Uses freesurfer processed data for training.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Using distance function to represent surface. Reconstruction of surface that is already coregistred to atlas seems novel.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

It is not clear how topology is preserved. Additional topology correction step is shown, but unclear how it is done and if it will be always successful. Why Gaussian smoothing? Will it preserve the details of the cortex? The results are hard to validate. They look visually similar but how well does the boundary match? The cortex is often represented by a genus zero surface.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Seems to be reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It is not clear what happens when the scanner and parameters differ from the training set to test set. What is the effect of bias field, different slice thickness etc. Most importantly what is the effect of pathology? Will this mask abnormal cortical surfaces such as in dysplasia or autism? If yes then the method is inherently unusable. This important question needs to be addressed in full and detail.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall a good paper, but how it will perform in the presence of abnormalities of the cortex is citicall important to evaluate.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

Tis paper proposed a 3D-CNN based method to joint reconstruct and parcellate cortical surfaces, which could save computation time a lot, compared to FreeSurfer.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Using spherical coordinates in loss function instead of ROI labels is interesting and might be easy to transform to other atlases.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Validation of this method is limited. e.g. how accuracy the white/grey matter boundary is?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The data is public available and code will be released if accepted.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The overall idea of this paper is interesting. However, the validation of this paper is only focused on cortical parcellation, but the validation on the reconstruction and tissue segmentation accuracy is really limited. And It’s not that fair to compare the cost time to FreeSurfer, since FS also do other preprocess, such as skull-strip, subcortical labelling.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It may provide a potential tool for FreeSurfer community, if validated better.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposed a 3D-CNN based method to jointly segment and parcellate brain cortex. Major strengths: important topic, clear presentation and plausible method. Major weaknesses:
1. The title and main content seem somehow misleading. The main work in this paper is on segmentation and parcellation of cortex based on U-net. Based on the segmentation (represented by level-set), only the white matter surface is created by smoothing, correction of topology and meshing, which don’t involve any deep learning. It is also unclear how to make sure topological correctness, which is critical in cortical surfaces. Moreover the most important and challenging pial surface is not created in the paper. Since authors only learn segmentation/parcellation, it might not be appropriate to say “learning surface reconstruction”. This also means the comparison with FreeSurfer is not fair, as FreeSurfer does more jobs and provides more meaningful and useful results. The results provided by the proposed method are less uesful, as only white surfaces are created.
2. The main technical contribution is the multi-task U-net. But it is not clear how these taskes can help each other and how to balance them, as different tasks have different difficulties.
3. It seems that the validation has flaws. As pointed by all three reviewers, authors only validated the parcellation accuracy, but didn’t validate the accuracy of segmentation and white matter surface mesh, which requires high accuracy in clinical and scientific studies. Furthermore, MindBoggle dataset is actually obtained by manul edition on FreeSurfer generated results, as they disagree with FreeSurfer in some region definitions in all cases. This means FreeSurfer will likely have low accuracy if using MindBoggle dataset as ground truth.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We thank reviewers and area chairs for their thoughtful comments. We are happy that all three reviewers highlight our work as a clever and a novel approach to perform a joint cortical parcellation and surface reconstruction from MRI volumes. Our paper improves the accuracy and speed of cortical parcellation via a new joint approach that leverages both surface parcellation and reconstruction. However, there seems to be a few misunderstandings that we respond to below individually.

1) Title and content

Our main aim is not to mimic all key functionalities of FreeSurfer but to rather improve the computationally expensive cortical parcellation step. Even though a cortical mesh is produced by our joint approach as a byproduct, our focus is on improving brain parcellations and not cortical reconstructions. In competing spherical methods, parcellation is usually more computationally intensive than reconstruction as it requires additional steps such as inflation.

Our approach provides the fundamentals for extracting the white-matter surface, which could be easily extended to also extract pial surfaces by simply learning a second distance function. However, such extraction of extra surfaces is not the main focus of the paper.

We learn a signed distance function to a cortical surface, which is an implicit representation of the surface. This implicit representation can be used to extract an explicit triangulated mesh, useful, for instance, for the visualization of brain surfaces. However, its exact intricacy remains beyond the scope of this work. Nevertheless, to guarantee a topologically accurate surface, as questioned, we follow a standard mesh extraction as in [12]. We use a proven method proposed by Bazin and Pham [21] to correct the topology defects from spurious distance predictions and guarantee a topology-preserving 3D surface volume (no holes or overlaps).

We compare our method with FreeSurfer and other surface-based networks [14,15,16] that assume the availability of a cortical mesh. Our goal is to validate whether our novel joint strategy would improve parcellation or not. Our results do demonstrate that such a novel joint approach does indeed improve parcellation (Table 2). It also offers a potential to further exploit surface reconstructions, valuable for future work but beyond the scope of this paper.

2) Balancing tasks The loss terms that update the 3D spherical coordinates are independent and balancing them is straightforward. Our experiments indicate no notable effect of individually weighing the distance loss over the angular loss. We thus use equal weighting of the spherical coordinates in l_surf (Eq. 3).

3) Validation of surface reconstruction Our main hypothesis is that we can improve the parcellation task by learning 3D spherical coordinates in a joint parcellation and reconstruction effort. Hence, our study does focus on parcellation (our goal) and considers an extra evaluation on reconstruction (not our goal) as beyond our scope. This joint strategy for parcellation shows a superiority in accuracy and speed (Table 2) and even comes with a potentially usable surface for future work. Nevertheless, as extra information, the reconstruction error in terms of average distances between the predicted and FreeSurfer meshes is found to be 1.313 voxels only. We will clarify this misunderstanding in the final version of the paper.

To summarize, our work answers a new question whether cortical parcellation could be improved with a new joint approach for parcellation and reconstruction. Our results show an improvement in parcellation in terms of accuracy and speed. While reconstructed implicit surfaces are produced by our method and may provide new opportunities for future investigations, they remain beyond the scope of this paper, which is to improve parcellation with a novel joint strategy. In other words, we only show a better parcellation scheme, improved with a joint approach and validated in our results.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The rebuttal is not very convincing.
1. If the focus is on parcellation, the title should be “Learing parcellation ..” rather than “Learning … Surface Reconstruction and Segmentation …”. The current title, abstract and main content still cannot reflect the focus. Meanwhile, it is not “easily extended to also extract pial surfaces by simply learning a second distance function”, considering the known partial volume issues in sulci.
2. It is still unclear why and how different tasks help each other.
3. Even the validation focused on parcellation, the issue on MindBoggle dataset is still not addressed. The region-wise validation result is not presented. Moreover, it is stated that “the reconstruction error in terms of average distances between the predicted and FreeSurfer meshes is found to be 1.313 voxels only”. This error is very large for surface reconstruction, considering that the cortex thickness is typically only 2-3.5 mm/voxels, making real clinical applications of the developed method questionable.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors propose a fast deep-learning based method to jointly reconstruct the brain surface and segment the image. The main criticism of the work is insufficient evaluation of the accuracy of the reconstructed surfaces, and I agree with Reviewer 2 that ensuring that locally abnormal cortical surfaces are not projected to look healthy is extremely important. However, the authors explain in their rebuttal that the purpose of their work is segmentation, and that surface extraction is just a byproduct. Provided that they clarify this important point in the final version, I think the paper is acceptable.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposes a novel method for cortical surface recon and parcellation based on learning the signed distance to the underlying cortex. Given the complexity of this topic, it is understandable that further validations are needed to establish its superiority to existing tools. Overall it’s an interesting method worthwhile for more discussions in the MICCAI conference.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

SegRecon: Learning Joint Brain Surface Reconstruction and Segmentation from Images