Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Jie Wei, Feng Shi, Zhiming Cui, Yongsheng Pan, Yong Xia, Dinggang Shen

Abstract

Accurate and consistent segmentation of longitudinal brain magnetic resonance (MR) images is of great importance in studying brain morphological and functional changes over time. However, current available brain segmentation methods, especially deep learning methods, are mostly trained with cross-sectional brain images that might generate inconsistent results in longitudinal studies. To overcome this limitation, we present a novel coarse-to-fine spatio-temporal constrained deep learning model for consistent longitudinal segmentation based on limited labeled cross-sectional data with semi-supervised learning. Specifically, both segmentation smoothness and temporal consistency are imposed in the loss function. Moreover, brain structural changes over time are summarized as age constraint, to make the model better reflect the trends of longitudinal aging changes. We validate our proposed method on 53 sets of longitudinal T1-weighted brain MR images from ADNI, with an average of 4.5 time-points per subject. Both quantitative and qualitative comparisons with comparison methods demonstrate the superior performance of our proposed method.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87193-2_9

SharedIt: https://rdcu.be/cyhLB

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors presented a novel coarse-to-fine spatio-temporal constrained deep learning model for consistent longitudinal segmentation based on limited labeled cross-sectional data with weakly-supervised learning. In order to obtain a consistent segmentation of intra-subject images scanned at different times, the author first used a coarse-to-fine strategy to individually segment the structures that can change in relate to time, such as the hippocampus. Then a few consistency related losses are applied to each pair of images from the same subject in a weakly supervised way. Results show that the proposed method has increased DICE coefficient score and increased temporal consistency.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- a novel method with supervised learning by using labeled cross-sectional images, as well as weakly-supervised learning by using unlabeled longitudinal images;
- spatial and temporal constraints in the loss function to ensure more consistent, smoother and meaningful segmentation results with time;
- used age-specific longitudinal changes to guide longitudinal segmentation, thus measuring accurate longitudinal changes for different subject groups.
- used temporal consistency as a metric to evaluate the results. Results show a better consistency and reduced variance of volume measurements between scans within a subject.
- writing is good and easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The paper is in general good. It would be better if the authors could compare the proposed method with traditional linear mixture model in terms of the smoothness in the segmentation result. It would also be interesting to get an interpretation of the meaning of the reduced variance in the pipeline.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The process of the experiment and details of the implementation are described very clearly, so the paper should be highly reproducible. However, some details are missing in the description. For example, what is the coefficient a, b, and c in the age-constraint loss? How did they get it?
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- how to get the value of the hyperparameters in age constraint?
- It would be good to see some further analysis of the segmentation result (but perhaps not in this submission). For example, analyzing the group difference between AD, MCI, and NC by the volume information of the segmentation can be conducted in the future.
- The description of L_FC is a bit unclear. What is an image block?
- In the smoothness constraint, if t_i, t_j, and t_k are scan times, why use ½(t_i + t_k) to represent t_j? Wouldn’t a split proportional to the time interval be better?
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper has a reasonable design of pipeline and every step is logical. The evaluation proved the effectiveness of the design. The generated segmentation with time coherence and reduced variance can be useful in longitudinal analysis of subjects.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

In this work, the authors propose a semi supervised learning segmentation method dedicated to longitudinal images series. The network is trained using labeled cross sectional images and also losses that aims at enforcing temporal consistency on unlabeled longitudinal images. Potential temporal morphological deformations are adjusted on longitudinal series using a block matching routine before computing the consistency losses. The network architecture (CFSegNet) used is a cascasde of 2 V-Net, the first one for coarse localization of the ROI, the second for fine grained segmentation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

As far as I know, this is the first work to exploit the specificity of longitudinal images to propose a dedicated semi supervised segmentation method. Longitudinal images need no annotation for the training.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Experiments show that main improvements are obtained on the smoothness of the segmentation/volume, not on the DICE (<1 points improvement). Different experimental setup could reveal if the new loss can improve the DICE when small annotated images are available.

Some implementation
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Most required reproducibility information are provided. The authors should give more details about the “age curve”. Readers cannot know where to find them or how they were estimated.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Comments:

– the DICE improvement is mild (lower than 1 DICE point). It would be worth in an extended version to add more unlabeled longitudinal data to see if better improvement can be achieved. Experiments with a much lower number of annotated data in the trainining would be interesting to reveal the interest of the semi supervised approach.

– block matching is used before applying the SCC losses to adjust for potential morphological change. This block matching step can be done once and for all before training to save computation. More sophisticated registration routine using (for example ANTS, Elastix, Diff Demons..) can be used instead of block matching for better smoothness, invertibility, …

– in paragraph 2.2, it is said that CSegNet and FSegNet share the same parameters. As they don’t work on image of the same resolution and don’t have the same architecture (one image input for CSegNet, 2 for FSegNet), I am not sure this is desirable, I don’t even understand how this is possible (different number of input channels).

– please details how the coefficients of the different losses have been chosen to make sure there is no bias in favor of the proposed method.

Minors:

– The volume change curve are “statistically calculated from populations of ADNI data”. please, be more specific. where does the point in fig. 2 come from and are measured ?

– The authors use the term “weakly” supervised learning but used a set of annotated images (the large majority of the training set). Although the labeled images are only cross sectional, I think the term “semi” supervised learning would be more appropriate.

– As all increasing triplet (ti, tj, tk) are used in the smoothness constraint and an non weighted average is used (the coefficients are independent of ti, tj, tk), L_sc is 0th order: an infinite coefficient for L_SC would implies constant volumes over time.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The work is interesting and has potential but experiments can be improved.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The objective is the accurate and consistent segmentation of longitudinal brain magnetic resonance (MR) images to study brain morphological and functional changes over time. The manuscript describes several stages, methods, and measures. These are: (1) Considers each individual time point for identification of the ROI and segmentation from course to fine. It uses supervised learning based on a V-net. (2) Consistent (longitudinal) segmentation with weakly supervised learning. The components of the segmentation cost are: a. Differences between Dice coefficients b. Differences between binary cross entropy c. Differences between high level features To achieve (a-c) they mention registration followed with block matching. d. Smoothness constraint for volume along time e. Age constraint to allow time related changes The authors validate the method with ADNI data to segment the: left hippocampus, right hippocampus, lateral ventricle, endorhinal. They show improved longitudinal consistency
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper adresses a valid problem of longitudinal studies. That is consistency in segmentation for volumetry along time. The method they use contains several valid parts. Forexample, the segmentation/matching along time, the coarse to fine identification of the structures. Also, the smooth changes of volume along time.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

They improve consistency along time but not necessarily overall performance in each time point. They emphasize on mathematical description and neural network description without giving an overall description of the method. There is not even a diagram describing the method. The reader has to combine the equations and the various description to form the overall picture without receiving help from the text.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

There are many steps and equations to implement. Also, the overall picture could be clearer.It would be an effort to reproduce.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

First in the method, it appears that you have the spatial segmentation. This is followed by temporal adjustment. These two steps could be combined in one for simplicity. It is important to give an overall description of the method with motivation first and draw a diagram particularly considering the multidimensional nature (spatiotemporal) of the data you chose to work with. The temporal consistency uses 5 different terms, that seems to be a lot. You impose temporal consistency so it is no surprise you find that in your results. But, you should also demonstrate that it improves actual performance in individual time points as well.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It addresses many of the issues that need to be dealt with in longitudinal brain segmentation. The data that is analysed is valid. So I suggest accept. But the method is not sufficiently simple and well explained, so I would not recommewnd strong acceptance.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The work addresses a relatively less addressed problem consistent longitudinal segmentation. The reviewers reached consensus on the quality of the work. All reviewers acknowledged the novelty and relevance.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

We sincerely thank all reviewers and ACs for their recognition of the novelty and relevance of this work. We would also like to thank the anonymous reviewers for their insightful and invaluable comments and suggestions. Here are responses to their suggestions and remaining concerns.

Q1: Hyperparameters in age constraint A1: In our experiments, we use totally 431 images from ADNI including 53 longitudinal subjects and 209 individual subjects. We take the age as X-axis and the volume as Y-axis to draw a scatter plot for each ROI. Then we use second-order polynomial curve fitting to get hyperparameters in age constraint.

Q2: Smoothness constraint A2: In out experiments, we use ½(t_i + t_k) to represent t_j for simplicity. As suggested, we will set different weights for t_i and t_k to represent t_j for possible better representation in our extended paper.

Q3: The DICE improvement is mild. You should also demonstrate that it improves actual performance in individual time points as well. A3: For this study, our main goal is to impose temporal consistency for segmentations of longitudinal images, which often leads to less DICE for individual time-points. In this work, we managed to maintain the DICE performance. In future work, we will try to improve the DICE in individual time points as well.

Q4: CSegNet and FSegNet share the same parameters. A4: We would like to clarify that the CSegEet in the first stage shares parameters with the CSegNet in the second stage, and this is true for the FSegNet. But CSegNet and FSegNet do not share parameters.

Q5: Although the labeled images are only cross sectional, I think the term “semi” supervised learning would be more appropriate. A5: As suggested, we will replace the term “weakly” with “semi” in our final paper.

Q6: It is important to give an overall description of the method with motivation first and draw a diagram. A6: Thanks for suggestion. We have given a brief overall description for the method in the second paragraph of the Introduction. However, due to space limit, we just use Fig. 1 as a diagram to explain our method. We then give more detailed description of our method in Section 2.2. As suggested, we will add a diagram that explain our motivation vividly in our extended paper.

Q7: Feature work A7: As suggested, we will make the following improvements in our extended paper: (1) combining the spatial segmentation and temporal adjustment into one step for simplicity; (2) applying block matching step once before training to save computation; (3) giving more experiments to prove which loss is more important and to find more reasonable and effective coefficients of different loss functions; (4) adding more unlabeled longitudinal data to see whether better improvement can be achieved; (5) giving further analysis of the segmentation results; and (6) giving more experiments with a much lower number of annotated data for training to demonstrate the feasibility of the semi-supervised approach.

back to top

Consistent Segmentation of Longitudinal Brain MR Images with Spatio-Temporal Constrained Networks