Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Xiang Chen, Yan Xia, Nishant Ravikumar, Alejandro F. Frangi

Abstract

Image registration aims to establish spatial correspondence across pairs, or groups of images, and is a cornerstone of medical image computing and computer-assisted-interventions. Currently, most deep learning-based registration methods assume that the desired deformation fields are globally smooth and continuous, which is not always valid for real-world scenarios, especially in medical image registration (e.g. cardiac imaging and abdominal imaging). Such a global constraint can lead to artefacts and increased errors at discontinuous tissue interfaces. To tackle this issue, we propose a weakly-supervised Deep Discontinuity-preserving Image Registration network (DDIR), to obtain better registration performance and realistic deformation fields. We demonstrate that our method achieves significant improvements in registration accuracy and predicts more realistic deformations, in registration experiments on cardiac magnetic resonance (MR) images from UK Biobank Imaging Study (UKBB), than state-of-the-art approaches.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_5

SharedIt: https://rdcu.be/cyhPL

Link to the code repository

https://github.com/cistib/DDIR

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors propose a deep learning-based approach to preserve discontinuities in the displacement fields calculated during the registration. The authors compared the proposed approach to the state-of-the-art methods in the learning-based nonrigid registration, as well as some old diffeomorphic methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is mostly written well and easily understandable.
- The concept is relatively simple.
- The discontinuity preservation is an important property in the medical image registration and is currently not carefully addressed in the learning-based solutions. It has a significant clinical impact.
- The authors state to make the results reproducible after a potential acceptance.
- The Figures/Tables are clear and confirms the observations.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- In the introduction the authors cite several works related to the discontinuity preservation. However, in the evaluation the authors compare the methods to general diffeomorphic methods that are not dedicated directly to maintain the property. I suggest to add comparison to competitive methods. I even think that using SyN in this experiments is far from being correct since the method is not dedicated to the problem at all. B-Splines-based registration is not a good candidate too. I suggest to compare the method to solutions based on e.g. optimal transport or fluid flow.
- It is not clear what is the main source of the registration improvement. In fact, the authors state that the discontinuity preservation improves the registration results. However, according to the description the method uses the segmentation masks of the corresponding structures. Therefore, the direct correspondence optimization should result in a great alignment of the boundaries. Can the authors comment on this? Why not to to optimize the alignment of the structures directly? The results visible in the Figure 2 (especially the Jacobians) suggest that the direct alignment of the corresponding structures would behave just as well or even better.
- The methods seem to not be directly scalable into other applications. Moreover, it requires the segmentation masks that may be not trivial to acquire for numerous applications. It is a significant limitation that should be emphasized.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

According to the reproducibility checklist and information from the paper, the results should be reproducible after the potential acceptance.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please refer to the strengths and weaknesses of the paper.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method is described well, the paper may have a significant clinical impact (discontinuity preservation is important) and the results should be reproducible. Moreover, compared to other papers in my stack, the paper is more engaging, the experiments are performed more carefully, the concept is more clear, and there is “more novelty”.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper proposes a discontinuity-preserving image registration method for cardiac cine-magnetic resonance (CMR) registration. In contrast to existing deep learning-based methods, the authors propose to predict the partial (disjoint) deformation fields for each targeting anatomical regions, which used four disjoint U-Net to capture the features of each sub-region. The smoothness regularizations are applied on each partial deformation field instead of on the final deformation field to avoid the global smoothing effect. The method is evaluated on one dataset of cardiac cine-magnetic resonance scans, and the results are compared to 2 classical methods and 1 deep learning-based method.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

In contrast to anisotropic diffusion/adaptive regularization methods, the authors address the global regularization problem of deformable image registration using a deep learning-based approach.

Good reproducibility. The authors provided sufficient details about the models, hyperparameter choices, dataset and promised to release the code if the paper got accepted.

A comprehensive and sufficient evaluation is performed. The authors evaluated the proposed method with multiple metrics, including Dice score, Hausdorff Distance (HD) and Jacobian determinant and compared to the state-of-the-art diffeomorphic image registration method.

The formulations are technically sound, and the results are promising (in terms of Dice score and HD).

The paper is well-written and easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Requiring anatomical segmentation label: The proposed method requires anatomical segmentation labels in both training and inference phrase, which hinder its potential in different applications. In addition, the registration accuracy of the proposed method will be subject to the quality of the segmentation.

Exhaustive memory consumption during training: Compare to existing deep learning-based methods [4], the proposed method adopted four U-Net networks for feature extraction, which implies that the GPU memory consumption of it is at least four times higher than [4]. This further limit the clinical potential of the proposed method, i.e., limit to low-resolution input scans.

Some statements regarding globally smooth deformation fields are arguable. For example, “Hence, enforcing deformation fields to be globally smooth can generate unrealistic deformations…”. A deformation field is said to be “unrealistic” or not plausible if it is not smooth (violating the bijective mapping/local invertibility), which contradict this statement. Yet, global smoothing may degrade the registration accuracy (in terms of Dice score).
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Good reproducibility. The authors provided sufficient details about the models, hyperparameter choices, dataset and promised to release the code if the paper got accepted.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Exhaustive memory consumption: While predicting disjoint deformation fields for each targeting region make sense, why not separating a complete deformation field into multiple disjoint deformation fields and applying the smoothness regularization individually? In this case, only one U-Net is needed, and the network still learns to regularize the solution for each sub-region independently.

Requiring anatomical segmentation label: Learning to localize targeting anatomical structures with spatial attention mechanism may helps to remove the need of segmentation labels during the inference phase.

Requiring anatomical segmentation label: Providing runtime analysis and memory consumption will help to determine the clinical value and feasibility of the proposed method.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall, the paper is well-written, and most of the technical details are explained clearly. The proposed method is adequately evaluated. The proposed method has some weaknesses regarding memory consumption and anatomical label supervision during training and inference phrase.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

A network called Deep Discontinuity-preserving Image Registration network (DDIR) is proposed. The idea is to preserve discontinuity while keeping the local smoothness. The input image is segmented into 4 parts and fed into the different networks to generated locally smooth flow fields. The performance is compared with voxelmorph.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Novel idea to keep local smothness while allowing global discontinuity.
2. Paper is well organized and easy to follow.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. One of the concern is about the space resolution. For cardiac MR the resolution in x,y,z axes are very different. For brain image registration or 2D slice registration, the resolution is similar.
2. Motion correction. During the imaging process, the patient is often told to hold the breath but there might still be breathing motion and causing slice mismatch. So generally before registration, the motion correction is necessary. reference: Puyol-Antón, Esther, et al. “Fully automated myocardial strain estimation from cine MRI using convolutional neural networks.” 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, 2018.
3. Concern about the loss function. It is natural to choose NCC loss in the registration task. While in cardiac MR, the image pairs are temporal-related and the pixel intensity does not change a lot in same sequence. So many paper for cardiac MR using MSE loss and achieves higher performance. Reference: Qin, Chen, et al. “Joint learning of motion estimation and segmentation for cardiac MR image sequences.” International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2018. Zheng, Qiao, Hervé Delingette, and Nicholas Ayache. “Explainable cardiac pathology classification on cine MRI with motion characterization by semi-supervised learning of apparent flow.” Medical image analysis 56 (2019): 80-95. Yu, Hanchao, et al. “Foal: Fast online adaptive learning for cardiac motion estimation.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
4. About the dataset, there is another public available dataset called ACDC which is easier to acquire and many papers are based on that.
5. Relationship between cardiac motion estimation and registration. Some references may be missing. It seems these tasks are very similar and closely related. The only difference is 2D or 3D registration. It might be necessary to discuss in a related work session or add them to the comparison session. Reference: Qin, Chen, et al. “Joint learning of motion estimation and segmentation for cardiac MR image sequences.” MICCAI 2018. Zheng, Qiao, Hervé Delingette, and Nicholas Ayache. “Explainable cardiac pathology classification on cine MRI with motion characterization by semi-supervised learning of apparent flow.” Medical image analysis Yu, Hanchao, et al. “Foal: Fast online adaptive learning for cardiac motion estimation.” CVPR 2020. Wang, Liang, et al. “A gradient-based optical-flow cardiac motion estimation method for cine and tagged MR images.” Medical image analysis Yu, Hanchao, et al. “Motion pyramid networks for accurate and efficient cardiac motion estimation.” MICCAI 2020
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The UKBB dataset seems not very easy to acquire, which require application and payment for a fee. There is a ACDC dataset which is easier to acquire. The author agrees to release code and others might use the code to run benchmark on ACDC.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. For the concern of resolution, it would be better to discuss the non-isotropic problem and compare with isotropic problem like brain imaging.
2. For motion correction, it would be great to discuss why the motion correction is not performed here. reference: Puyol-Antón, Esther, et al. “Fully automated myocardial strain estimation from cine MRI using convolutional neural networks.” 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, 2018.
3. For the loss function, it would be great to discuss why not choose MSE as loss function as it is widely used in optical flow and cardiac motion estimation.
4. It would be good to mention ACDC and discuss the choice of datasets. 5.It would be good to discuss relationship between cardiac motion estimation and registration. It seems current work for cardiac motion estimation only consider 2D case, while it could be easily to extend to 3D. Just wondering if you could compare the dice score of center slice in the 3D registration with the 2D case. It would be great to addreferences and compare with those works since they are very similar to the registration task. Missing references: Qin, Chen, et al. “Joint learning of motion estimation and segmentation for cardiac MR image sequences.” MICCAI 2018. Zheng, Qiao, Hervé Delingette, and Nicholas Ayache. “Explainable cardiac pathology classification on cine MRI with motion characterization by semi-supervised learning of apparent flow.” Medical image analysis Yu, Hanchao, et al. “Foal: Fast online adaptive learning for cardiac motion estimation.” CVPR 2020. Wang, Liang, et al. “A gradient-based optical-flow cardiac motion estimation method for cine and tagged MR images.” Medical image analysis Yu, Hanchao, et al. “Motion pyramid networks for accurate and efficient cardiac motion estimation.” MICCAI 2020
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method is quite interesting while it seems not take some important factors into account, like motion correction. Otherwise there will be mismatched slices. Also a very similar problem cardiac motion estimation seems to be ignored. Some references are missing as well.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Overall, the reviewers appreciate the paper, but have several concerns that make the paper quite borderline. I believe the reviews are constructive and can be of help to the authors. The authors are invited to a rebuttal.

Importantly, the authors miss relevant literature, and do not compare to important (esp. classical) literature that tackles the given task, especially discontinuity preserving work. The source of the success of the results is unclear, as R1 notes semi-supervised segmentation has been proposed several times – it should be included in the comparison and the analysis. Even going further, segmentation is required at inference here, which is a significant requirement – how do other methods do given this? Furthermore, the method seems to have several ad-hoc components, which make it very difficult to evaluate the novelty and source of any insight.

I encourage the authors to digest the detailed reviews and address the issues, especially the larger concerns. Moreover, the goal is to improve the paper – if addressing these with further work, I encourage the authors to consider submitting a more improved paper, with higher impact, at the next conference.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Author Feedback

R3 Concern about the image resolution and motion correction Following previous work on cardiac image registration (Ref. 7), we used bicubic interpolation to generate isotropic cine-SAX stacks. UK Biobank data is acquired using a robust and strict protocol. Our contribution is a general discontinuity preserving image registration approach, not specific to cardiac MR registration. Hence, we assume our data have minimal motion artefacts.

R3 Use NCC instead of MSE as the loss function Previous methods do use MSE as the similarity loss. However, none have shown that MSE is better than NCC. We investigated both MSE and NCC initially and found the latter yielded better results.

R3 Why not use ACDC We used ACDC as an independent test set and have evaluated all approaches on the same. DDIR outperformed the rest, achieving ~1.3% improvement over VM-Dice. Registration results on ACDC are included in the revised paper.

R3 References about cardiac motion estimation are missing Our contribution is a general discontinuity preserving image registration approach, not specific to cardiac MR registration. Hence, additional references on cardiac motion estimation are out of scope.

R1 Comparison with traditional methods Source code for traditional discontinuity-preserving registration methods is not available, limiting reproducibility and fair comparison. Hence, we adopt the composition strategy using VM and VM-Dice, to mimic a traditional discontinuity-preserving method (Ref. 13 in the paper). DDIR outperforms the current state-of-the-art. Demons (optical flow-based) reg. approach was evaluated on our test set (included in the revised paper), achieving an avg. Dice score of 71.50%. DDIR and other DL-based methods significantly outperform Demons.

R1 Source of the registration improvement of DDIR There are two main reasons for the improvement in performance: 1/ use of segmentations and multi-channel encoder-decoder provides spatial priors for feature learning; 2/ estimated deformation fields are locally smooth but globally discontinuous - i.e. no global smoothness is enforced. A multi-channel encoder-decoder is used to register the whole image as: 1/ registering the entire image provides additional regularisation, to mitigate the effect of poor regional segmentations; 2/ direct correspondence optimisation of individual structures would make the overall approach heavily reliant on the quality of the segmentation masks.

R1 & R2 Not easy to adapt to other applications, segmentation is a limitation Although DDIR is demonstrated on cardiac MR images, the approach is generic and easily adaptable to other domains (e.g. abdominal images). Requiring segmentation masks is a current limitation that has been emphasised in the revised paper, and will be addressed in future work using an unsupervised approach. However, requiring segmentation masks is a common limitation for most existing discontinuity-preserving registration methods (ref. 6,11,13,14).

R2 The requirement of GPU memory is much larger than VM While we use a 4-channel encoder-decoder, each sub-channel in DDIR is computationally efficient. For example, when training with batchsize 2, the required GPU memory of VM is 1244MB, while DDIR requires 2268MB, less than 2 times of VM. During inference, registering 1 pair of cardiac images requires 348MB and 732MB for VM and DDIR(~2 times of VM) respectively.

R2 Some statements regarding globally smooth deformation fields are arguable What we want to emphasize is that while local smoothness in deformation fields is desirable, global smoothness across the entire image domain does not preserve discontinuities across tissue interfaces. For example, sliding motion at tissue interfaces (lungs) or deformation of multiple structures with different intrinsic material properties (abdominal organs) are by definition discontinuous at boundaries. Our approach addresses this issue to preserve such discontinuities (not afforded by enforcing global smoothness).

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

After taking all the reviews and rebuttal into account, I believe the paper is a minimal contribution to MICCAI, but without substantial flaws for a conference paper. I believe it should be accepted.

There are aspects that I believe the authors should address for this work to become mature – could the need for segmentations be avoided, and comparison to classical approaches by contacting the authors, etc. As it stands, it seems to be a fairly obvious extension of registration methods, and is mostly a tool for a very specific application. There is an opportunity to study what DL networks can do in this space the classical models cannot, how the dependency of masks can be modulated compared to classical methods, etc.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The strength of the paper is that it addresses discontinuity-preserving regularisation, which has not been extensively studied since the advent of DL. The clear disadvantage is that it evaluates registration accuracy through segmentation overlap, while using manual segmentations as input. This was criticised by reviewer 2 but not adequately addressed. Furthermore, no comparison to classic discontinuity-preserving methods were provided (claiming source code is not available), I would argue that e.g. deeds https://ieeexplore.ieee.org/abstract/document/6471238 would serve as a good baseline. Other anatomical regions (abdomen) where mentioned but not addressed. In summary, I think there are too many open questions and inaccuracies at the current stage and do not recommend acceptance. However, this paper could be very valuable when further improved in further work.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

16

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

All the reviewers are interested in the topic and the proposed idea, although the requirement of the segmentation mask is a weak point in this work, the authors addressed the other concerns well.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

back to top

A Deep Discontinuity-Preserving Image Registration Network