Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Shuo Wang, Chen Qin, Nicolò Savioli, Chen Chen, Declan P. O’Regan, Stuart Cook, Yike Guo, Daniel Rueckert, Wenjia Bai

Abstract

In cardiac magnetic resonance (CMR) imaging, a 3D high-resolution segmentation of the heart is essential for detailed description of its anatomical structures. However, due to the limit of acquisition duration and respiratory/cardiac motion, stacks of multi-slice 2D images are acquired in clinical routine. The segmentation of these images provides a low-resolution representation of cardiac anatomy, which may contain artefacts caused by motion. Here we propose a novel latent optimisation framework that jointly performs motion correction and super resolution for cardiac image segmentations. Given a low-resolution segmentation as input, the framework accounts for inter-slice motion in cardiac MR imaging and super-resolves the input into a high-resolution segmentation consistent with input. A multi-view loss is incorporated to leverage information from both short-axis view and long-axis view of cardiac imaging. To solve the inverse problem, iterative optimisation is performed in a latent space, which ensures the anatomical plausibility. This alleviates the need of paired low-resolution and high-resolution images for supervised learning. Experiments on two cardiac MR datasets show that the proposed framework achieves high performance, comparable to state-of-the-art super-resolution approaches and with better cross-domain generalisability and anatomical plausibility.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_2

SharedIt: https://rdcu.be/cyl3E

Link to the code repository

https://github.com/shuowang26/SRHeart

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

In this paper, the authors present a novel VAE-based approach to perform segmentation of 3D CMR scans. They use the latent space to optimize over the possible HR geometry. Their model also incorporates the possibility of including single-slice LA scans of the heart (typically done in practice).
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The model presented is very interesting and their approach to combine SAX and LA views can be extremely useful. The validation is extensive and well done.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main limitation of this paper is the lack of description of the downsampling and motion layer steps. How are these incorporated into the optimization? Are they differentiable? What type of motion is used (rigid, affine, non-linear…)
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper should be reproducible given the missing details about the downsampling and motion correction layers. The data is publicly available and training information seems sufficient. Further hyperparameters should be available in the supplementary material, which, unfortunately was not delivered upon review. Assuming the size of the latent space is provided there
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Very little to add. This is an exciting paper which could prove very useful. Moreover, the use of latent optimization can be exploited for other applications although it is not novel from this paper.

One open question is regarding the optimization. Did the authors notice the presence of local minima in the optimization? How big was the latent space? are the results sensitive to it?

With regards to the VAE training, I wonder how well does this method generalize to different heart shapes?

Finally, the paper would greatly benefit from details about the downsampling and, more importantly the motion layer. How were these implemented and how were they incorporated into the gradient descent. Were the motion parameters optimized iteratively with the latent space parameters?
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is a very interesting approach to perform segmentation in 3D. The paper would benefit from added details, specially for the sake of reproducibility. However, in its current form is already interesting.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The paper proposes a method to reconstruct high-resolution 3D cardiac mask volume from low-resolution (along Z-axis) segmentation masks from cine MR data. The method can solve the motion correction and super-resolution tasks jointly via a latent optimization process. The method is validated on two datasets, one private research cohort of high-resolution CMR dataset of 1200 healthy subjects and 200 cases in the UK Biobank dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The method is novel and aims to solve a practical challenge in processing clinical cine MR data for cardiac analytics. The results show good performance of the proposed method comparing with interpolation-based methods and a learning-based interpolation by an enhanced deep residual network (EDSR).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. It shows the performances of the EDSR on different degradation degrees while trained from a single degradation degree. However, I think it would be better and more fair to train it using all degradation degrees and then compare it with the proposed method.
2. The latent optimization stage of the proposed method as a maximum likelihood estimation seems very time-consuming. It would be better to discuss the time complexity and compare it with other approaches, so the readers can better understand the proposed method.
3. It seems difficult to reproduce the work because the training of a good generative model is not trivial, and the high-resolution dataset is rare.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Difficult to reproduce because the training of a good generative model is not trivial, and the high-resolution dataset is rare and hard to collect.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Because the proposed method takes sparse 2D segmentation masks as the input, it is interesting to show how robust the model is to segmentation errors and different population, e.g. diseased subjects. Because the accurate segmentation, especially for diseased cases, is also a challenging task in practice.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The novelty of the method, but some unconvincing experimental settings.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The author proposed a joint motion correction and super-resolution method for heart segmentation by optimizing the latent space parameter to better fit the targeted heart structure.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed method is relatively novel. There are sufficient evaluations on public databases.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

I have a few doubts on the setup of the solution. First, changing the latent parameter in VAE doesn’t guarantee the output shape is plausible, There is a risk that the optimization process may “overfit” to the distorted heart. Second, the description of the motion layer is not clear. How is it regulated, Does it guarantee to lead to an unique solution. Finally, the proposed methods is not compared with other optimization method, such as statistical shape modeling.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Probably reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

First, I would like to point out that the proposed motion correction and super-resolution are done on segmentation masks, therefore would not be able to utilize the image features present in the original MRI images for improvements. Second, I think there should be better therotical analysis on the stability of the latent space parameter optimization and the motion modeling. Third, the evaluation should include other optimization method for segmentation result refinement.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think this paper lack of thorough theoretically analysis of the stability of the proposed method and could benefit from including more evaluation against other optimization based methods.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Reviewers cited the novelty of the proposed method and the overall evaluation as strengths. Some concerns were raised over questions about the optimization methodology, the computational complexity, and specific aspects of the evaluation. Overall, the paper was regarded as among the top papers reviewers by each of the reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We thank all reviewers and the AC for having a consensus on the technical novelty (R1: a novel VAE-based approach; R2: the method is novel; R3: novel) and clinical significance of our work (R1: can be extremely useful; R2: solve a practical challenge). Here we respond to the reviewer’s comments on reproducibility and technical details.

• Reproducibility. We will release the code on Github (https://github.com/shuowang26/SRHeart/) for reproducible research. Although the clinical dataset of high-resolution cardiac segmentation is not publicly available due to data privacy issues, we will release the trained generative model and deployment code to benefit the research community.

• Technical details. Following the suggestions from the reviewers, we will elaborate the details of latent space and network design in the camera-ready manuscript and supplementary materials. The degradation process consists of two differentiable layers: the inter-slice motion layer and the down-sampling layer. Specifically, the motion layer was implemented by adding the slice-wise rigid motion onto the mesh grids, followed by bi-linear interpolation. The down-sampling layer was implemented with mesh grids scaled by the scaling factor and with bi-linear interpolation. The dimension of latent space and other hyperparameters were determined by grid search evaluating the reconstruction quality. As the latent optimization is a non-convex problem, local minima may occur. This can be alleviated by stochastically starting from multiple seeds and exploring other global optimisation techniques.

• Application to diseased subjects. In separate experiments (not reported here), we have validated the proposed method on a number of subjects with cardiac diseases. The results show similar qualitative and quantitative performance compared to those on healthy volunteers.

back to top

Joint Motion Correction and Super Resolution for Cardiac Segmentation via Latent Optimisation