Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Liangjun Chen, Zhengwang Wu, Dan Hu, Yuchen Pei, Fenqiang Zhao, Yue Sun, Ya Wang, Weili Lin, Li Wang, Gang Li

Abstract

Longitudinal infant dedicated cerebellum atlases play a fundamental role in characterizing and understanding the dynamic cerebellum development during infancy. However, due to the limited spatial resolution, low tissue contrast, tiny folding structures, and rapid growth of the cerebellum during this stage, it is challenging to build such atlases while preserving clear folding details. Furthermore, the existing atlas construction methods typically independently build discrete atlases based on samples for each age group without considering the within-subject temporal consistency, which is critical for large-scale longitudinal studies. To fill this gap, we propose an age-conditional multi-stage learning framework to construct longitudinally consistent 4D infant cerebellum atlases. Specifically, 1) A joint affine and deformable atlas construction framework is proposed to accurately build temporally continuous atlases based on the entire cohort, and rapidly warp the new images to the atlas space; 2) A longitudinal constraint is employed to enforce the within-subject temporal consistency during atlas building; 3) A Correntropy based regularization loss is further exploited to enhance the robustness of our framework. Our atlases are constructed based on 405 longitudinal scans from 187 healthy infants with age ranging from 6 to 27 months, and are compared to the atlases built by state-of-the-art algorithms. Results demonstrate that our atlases preserve more structural details and fine-grained cerebellum folding patterns, which ensure higher accuracy in subsequent atlas-based registration and segmentation tasks.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_14

SharedIt: https://rdcu.be/cyhPV

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The submission introduces a deep learning framework for constructing a longitudinally consistent 4D infant cerebellum atlas.

The DACN framework of the submission is built from [23] with the extension of explicitly treating longitudinal information and adding an affine atlas rescaling.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Temporal consistency is explicitly encoded into the framework as well as an affine scaling transformation.

The infant literature is still underserved regarding tools and resources, so it is great to see new advances.

The data set consists of 405 scans (187 subjects, 6m-26m), which is a good size data set for infant cohorts.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Overall the paper is well written but there were some decisions that were not clearly explained/justified.

Why did the authors decide to use SyGN as a base of comparison? Many previously published approaches (mentioned in the intro) would have been more appropriate.

Was MSE ever compared to Closs? The authors claim repeatedly that it is not a robust enough measure, but I did not find evidence of it.

A bit more detail about the training/test data set would have been welcome. Which data set is used? Is it a subest of BCP? What is the age distribution of the subjects in the cohort? (Were they regularly acquired?)
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

No information about sharing code/data was mentioned in the submission.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Note, in [23] there are two applications: digits and adult brains. However, nothing is specific to these in the construction of the framework.

It would have been great to discuss how well the framework would do with fewer longitudinal timepoints. What would it converge to?

How is correntropy defined and why was it chosen?

With the affine rescaled atlas, is the affine component computed twice?

What is the ratio of longitudinal samples vs single timepoints that makes this framework work? How would it perform with only single timepoint acquisitions? Exploring the hyperparameter space would be very interesting.

Fig 3: results seem sharper sometimes for ANTS and not the new method. Esp at younger timepoints.

Why were the given timepoints chosen to generate the atlases at?

I was confused about why all scans had to be registered to the oldest timepoint. Isn’t this the job of the network to learn this?

Are the results presented in the submission statistically significant?

Minor: [23] 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Updates to an existing framework are proposed. The experimental comparison could have been done to a more relevant tool / framework. More clearity on some of the above mentioned issues would have helped the appreciation of the submission.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors propose a framework to construct longitudinally consistent infant cerebellum atlases. Their method consists of two networks, a deformable atlas construction network and an affine atlas rescaling network, where the atlas is rescaled affinely to match the input. They address the challenge of having a continuous atlas conditioned by age by adding a longitudinal consistency loss from consecutive time points of the same subject. They demonstrate improved performance over an ANTS-based Atlas.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novel formulation to the challenge of having an age-consistent atlas for infant cerebella. They propose a way of generating an atlas over a continuous age range through an age-conditional generator and a longitudinal consistency term.
- Affine Atlas Rescaling Network allows the atlas to capture cerebellar shape, then affine rescale to match the size of the input.
- Detailed experiments comparing to an ANTs-based atlas, showing improvement in performance.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The longitudinal consistency term is somewhat arbitrary. The authors group consecutive pairs of image, though this seems independent of the difference in time between scans. For example, scans far away (> 6 months) may be expected to have minimal consistency.
- The authors compare to a simple baseline that is likely to perform poorly. There are several prior works, for example, [23], that could have been compared against.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors outline their experimental procedure and parameters in detail. The paper is clearly written and uses existing neural network architectures, so is likely reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

This paper is well written and the challenges of infact cerebellum atlas generation are clearly outlined.

There are a few main areas where I have comments on how to improve the paper.

The first, is I believe the longitudinal consistency loss could be improved. I believe that in this specific dataset, this would not pose an issue, but in the cases where consecutive scans may be spaced for apart, I would expect less consistency than scans close together. However, the localized NCC does not take this into account, and so it may be encouraging incorrect consistency. Furthermore, for subjects without a consecutive scan, this term may overpower the subjects with longitudinal data, eventually rendering ineffective longitudinal consistency.

The second, is the network requires affinely aligned images. While in general this is fine, one of the challenges of infant cerebellum atlas generation outlined by the authors in the introduction is that the requirement of affine alignment will lose shape information. I believe this point should be discussed further, with the authors elaborating on how shape is preserved.

Lastly, I am unsure about the baseline atlas the authors compare against. While comparing with an ANTs atlas has merit, this is an outdated baseline and it was expected that their work would be improved. Furthermore, their test set contains only 12 images, with minimal information about the patient distribution. Thus, it is unclear how well their approach would generalize.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper presents a novel formulation to the challenging problem of infant cerebellum atlas generation. As the cerebellum of the infant changes greatly throughout gestation, an age-conditional approach is needed. The authors address this with a conditional atlas generator based on the input age, and use the voxelmorph framework to learn a diffeomorphic transformation to the input image.

The authors successfully address several challenges with this work, by conditioning based on age, enforcing longitudinal consistency, and preserving shape using the affine atlas rescaling network. Their approach is well-justified, and their experiments and reinforce these choices.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper proposed a novel longitudinally consistent atlas building method designed for infant cerebellum atlas estimation. The proposed method is learning-based that makes use of a nonlinear deformable registration network and an affine registration network. In addition to standard loss functions for cross-sectional atlas building, a longitudinal consistency loss term was introduced using a pair of longitudinal images. The proposed method out-performed a conventional atlas building method (ANTs-based) and the longitudinal loss term increased the performance for 4D infant cerebellem atlas estimation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is well-motivated and clearly written.
- The proposed method nicely tackled two important problems of infant brain atlas estimation and longitudinal atlas estimation.
- The auxiliary affine deformation network would provide a nice tool for an end-to-end analysis on the atlas estimation that is not often included in other atlas estimation frameworks.
- The experiments showed the feasibility and performance of the proposed method properly.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- No major weakness
- The implementation details and computational environments were not included.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Although the paper is clearly written and the methodology is explained very well, the implementation details and the explanation on the dataset are not presented in the paper. This may harm the reproducibility of the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The paper presented a novel method to estimate a longitudinally consistent 4D atlas focused on the development of infant cerebellum. The paper is well-motivated and clearly written. The proposed method is valid and presents interesting results on the longitudinal consistency. The proposed method showed better performance than a conventional atlas estimation method (based on ANTs) and a learning-based approach without the longitudinal consistency term.

I don’t have anything major to comment against the paper. It is interesting that the longitudinal consistency term using a pair of longitudinal scans from an individual infant enhance the performance of the atlas estimation. The term was simple and focused on enhancing the quality of the atlas. It does not aim to explain the longitudinal effect (inter-subject variability).

The proposed method was evaluated properly with manually segmented labels and increased the performance over one standard deviation.

One thing that was not fully validated in the paper is the comparison against the existing cross-sectional learning-based atlas estimation method (cited as [23] in the paper). Since the base architecture resembles the similarity to the method proposed in [23], it would have made a good comparison with the proposed method. It seems that the proposed method without the longitudinal consistency term would have been similar to it, however it would have made a more direct comparison.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is clearly written and tackles important problems in the field. The longitudinal consistency term can be interesting for researchers working on the longitudinal analysis and discuss further.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Reviewers in their detailed assessments agree on the very good clarity organization, and on the innovative and novel aspects of the paper and thus in prinicple suitability for MICCAI. Authors are encouraged to make datasets and code available for others to reproduce their results - reproducibility is an important issue for MICCAI.

Reviewers point to weaknesses related to the lack of implementation details and need for comparisons against existing methods which will have to be addressed. Overall, the paper shows merit but needs significant clarifications and improvements for being ready for MICCAI.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We thank all reviewers’ valuable comments and appreciate that the compelling aspects of our work are highlighted by the reviewers, including the novelty (R1, R2, R3), well-handled a challenging problem (R2, R3), evaluated properly (R2, R3), and good clarity of the paper (R1, R2, R3). We also carefully addressed the raised issues as follows:

Response to Meta-reviewer’s comments:

We will add more details of the implementation and dataset, and make the related dataset and codes available.

Actually, a modified baseline method [23] was compared with our method. We will make this clear in the final version.

R1, R2, and R3’s concern in comparing with [23]. Sorry for the misleading description. In fact, to fairly compare the performance and achieve valid results, we modified [23] by replacing MSE with Closs. To avoid confusion, we will use “[23] + Closs” to replace the original notation “Ours w/o LC” in the revision. To eliminate the influence of infant cerebellum development, we used the trained affine atlas rescaling network (AARN) to affinely rescale the atlases built by “[23] + Closs” for comparisons. Both qualitative and quantitative results (in Fig. 3 and Table 1) consistently show the superior performance of our framework compared to “[23] + Closs”. We will add above clarification in the paper to eliminate the confusion.

R1 and R2’s concern in comparing with SyGN (ANTs). The reason we compared with SyGN (provided by ANTs) is that it is a classical group-wise atlas building method, which considers both shape and appearance and is unbiased toward any specific individual. It is widely validated and publicly available, making it routinely used in many brain atlas building tasks.

R1 and R3’s concern on details of implementation and dataset. We will add the corresponding details in the revision, e.g., the computation platform, the training iteration number, and other important experiment configuration parameters. Meanwhile, we will add figures to show the distribution of longitudinal scans in the Supplementary Material, and add more details of the dataset.

R1 suggested clarifying the Closs. As verified in [26], the Correntropy based loss (Closs) is more robust than MSE in handling noise and outliers, because when facing an outlier/noise, the derivative of MSE increases dramatically, while the derivative of Closs gets closer to zero, which makes it more robust in training. Besides, we found in training deformable atlas construction network (DACN), the values of MSE based regulation loss would arbitrarily increase to infinite during the initial few iterations, which may be caused by outliers. Therefore, we choose Closs to replace MSE to help improve the robustness and training stability. We will clarify the specific Closs definition and the above reasons in the revision.

R1 and R2’s concern on the longitudinal consistency (LC). We agree that this LC term is valid for a dataset with longitudinal scans, and most subjects have to be scanned several times at different ages, which is a popular scheme in large-scale neuroimaging studies. In cerebellum, both deep white matter and middle peduncles appeared myelinated by the end of the 4th postnatal month and preserve shape during brain development (Triulzi et al., 2005). In fact, we can find apparent structural LC from Fig. 1, even between the scans acquired at 6-month and 24-month. Meanwhile, in practice, we find that it is much easier to achieve good results on within-subject registration, due to the existence of LC. Motivated by these, we add a LC term to supervise the within-subject temporal consistency, which can help prompt the within-subject alignment and consequently generate sharper atlases.

R1’s concern on the statistical significance. Compared to the existing methods, our framework achieved statistically significant results in both atlas-based registration and segmentation tasks (p-values < 0.05). We will add the corresponding t-test p-values in the final version.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper has novelty in its approach and methodology and seems an important contribution to longitudinal image analysis of MRI brain structures. The rebuttal clarifies major concerns by reviewers, in particular reviewer 1. Information on sharing code with the community should definitely be included in a final submission would MICCAI decide this paper to be accepted.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Overall, the paper is borderline and leaning to accept. The reviewers were confused about various aspects, including the contribution (longitudinal loss) and appropriate comparisons, and I agree that these could have been done better. Overall, I think the cerebellum atlas is a new application for longitudinal atlas building. The technical contribution is a fairly limited extension to [23] (the LC loss), but taken together with the interesting application I would say the paper will lead to good discussion.

However, the concerns of the reviewers are very valid. The several promises that the authors make in their rebuttal, especially clarifying the naming in the experiments, making it clear that the longitudinal loss is the main aspect being tested. Mimicking this, in the Methods it should be made clear that the main contribution is the LC (and the [Network] subsection is mostly taken from the previous method). These changes will clean up the paper, make the contribution clear, and remove some of the confusion.

There are other reviewer concerns, especially several from R2, which are not addressed but should be. Please add appropriate answers in the camera ready.

Overall, I am proposing acceptance, but this is conditional on the authors making the changes the claim they would in the rebuttal (all text, no new experiments), and ask the program chairs to verify this.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors addressed concerns on comparison. Issues on implementation should be addressable in the final version.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

back to top

Construction of Longitudinally Consistent 4D Infant Cerebellum Atlases based on Deep Learning