Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Rhydian Windsor, Amir Jamaludin, Timor Kadir, Andrew Zisserman
Abstract
This paper explores the use of self-supervised deep learning in medical imaging in cases where two scan modalities are available for the same subject. Specifically, we use a large publicly-available dataset of over 20,000 subjects from the UK Biobank with both whole body Dixon technique magnetic resonance (MR) scans and also dual-energy x-ray absorptiometry (DXA) scans.
We make three contributions:
(i) We introduce a multi-modal image-matching contrastive framework, that is
able to learn to match different-modality scans of the same subject with high accuracy.
(ii) Without any adaption, we show that the correspondences
learnt during this contrastive training step can be used to perform automatic
cross-modal scan registration in a completely unsupervised manner.
(iii) Finally, we use these registrations to transfer segmentation maps
from the DXA scans to the MR scans where they are used to train a network
to segment anatomical regions without requiring any ground-truth MR training examples.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87196-3_9
SharedIt: https://rdcu.be/cyl1z
Link to the code repository
https://github.com/rwindsor1/biobank-self-supervised-alignment
Link to the dataset(s)
Reviews
Review #1
- Please describe the contribution of the paper
a self-supervised learning approach for cross-modality image registration
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Experiments are carried on a very large dataset, one of the largest of its kind. The results are impressive (AUC > 0.99).
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The medical application of the proposed approach is not clear. It also lacks the comparison with other alternative methods.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Results seem reproducible to me, as a lot of additional information is provided in supplement materials.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
-
It’s not clear what’s the medical application of the proposed method. It is not common, to my knowledge, for a patient to have both whole body MR and DXA scans. Even successful matching different modality image pairs does not have real clinical application unless the registration error is lower than some level (i.e., err < 2mm for surgical applications). Additional rationale is needed.
-
Rigid registration for whole body scans. In general, one main challenge of registration between whole body scans is the non-rigid pose change. However, this seems not considered in the paper. Is rigid alignment sufficient in the practical scenario? It needs some justifications.
-
Lack of comparison with classical and/or state-of-the-art. Particularly for cross-modality alignment, I will expect to see the mutual-info registration as the baseline along with 1-2 latest method in this field. Without this comparison, the benefit of using the proposed self-supervised approach is not clear.
-
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The experiment seems reproducible on large dataset. But the comparative results are lacking.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
This paper develops a self-supervised learning approach for different-modality images registration. All the experiments are based on a large publicly-available dataset of over 20,000 subjects from the UK Biobank with both whole body Dixon technique magnetic resonance (MR) scans and also dual-energy x-ray absorptiometry (DXA) scans.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A multi-modal image-matching contrastive framework.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
– Lack of novelty. – Little results are provided.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Fig.2 is confusing.
- Please state your overall opinion of the paper
borderline reject (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Lack of novelty.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This is a simple yet interesting approach for self-supervised multi-modal registration that is demonstrated to work very well on DXA to MRI alignment of UK biobank scans. While contrastive learning is popular and already widely used, the combination of a self-supervised (multimodal) retrieval task with a correspondence search is exciting! The review quality varies reviewer #2 fails to meet MICCAI expectations and did not respond to the ACs request to improve their comments and #3 is missing despite numerous reminders. Yet I can fully support the judgement of reviewer #1 and think the paper could be accepted if a few points are addressed in the rebuttal / final version: a comparison to state-of-the-art hand-crafted multimodal features (e.g. MIND) or a global metric (mutual information) is necessary. The clinical rationale and importance of the application should be clarified (I could imagine a few, e.g. assessing back problems based on muscle and bone information - but the authors should have an idea).
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
4
Author Feedback
Thanks to the reviewers and AC for their comments. We are glad the novelty & strong performance of comparing multimodal spatial embeddings for contrastive pretraining is noted.
The reviewers see two drawbacks currently: -The multimodal registration task’s clinical use is unclear. -More comparison with common registration methods is needed. We will discuss clinical uses of DXA-MRI registration specifically but we’d also like to point out a) the method could be used unmodified for other modalities b) the contrastive retrieval task also has uses, e.g. by using a scan ROI as a query and correlating spatial embeddings we can find similar examples in a large dataset (see Simonyan et al, MICCAI 2011 for a similar approach). Also, network initialization with weights learnt in pretraining is likely to increase performance and data efficiency on downstream clinical tasks, as is often true in self-supervised learning.
Registration Clinical Applications: The main benefit of our method is it allows collation of spatial information from both modalities. This can be exploited in at least two disparate ways: i) Joint features: the extra information provided by both modalities can be used to improve performance in many clinical tasks. For example, DXA is used in spatially-localised body composition analysis. However, MRI has better soft tissue contrast; thus adding MR-derived features could improve fidelity. The benefits of MR for this task are discussed in Borga B J Radiol 2018. As the AC states, another use is for diseases where bone density and tissue type is important. For example, multiple myeloma is diagnosed and monitored by whole spine MRI, however low bone density (assessed by DXA) is a common sign. Thus, local multimodal features are likely to improve assessment. ii) Cross-modal supervision: Registration can be used to extract dense labels from one modality and train a model to predict them using the other, reducing the need for more scans. An example would be synthesising skeletal projections from MR scans using registered DXA targets with pixel-level supervision as an image-to-image translation task. Diagnosing osteoporosis/fracture risk by MR would also be of great clinical benefit. We will add a section to clarify i and ii.
Further comparison to common registration methods: We agree this is useful to understand our method’s efficacy. We compare with two alternatives; i) a rigid mutual information (MI) maximisation method (Matthes et al 2001); ii) MIND (Heinrich et al 2012); a non-rigid correlative approach with state of the art handcrafted features. The keypoint transfer error, defined in the main paper (& summarized below) will be added to the paper.
Method Mean (cm) Median (cm) MI (Rigid) 2.52±1.73 2.21 MIND (Deformable) 10.73±5.15 10.43 Ours (Rigid) 1.32±0.90 1.12 While MI often gives similar results to ours it sometimes fails catastrophically, hence the bigger error. MIND struggles to match image regions, with high error and strange deformations. We believe these methods fail as the DXA is a coronal projection of mostly bony structures, whereas the MRI shows few bony structures, with most visual content coming from soft tissues that can’t be differentiated by DXA. Thus most pixels have no obvious relation between intensities in the two scans. Note that these methods failing indicates the modalities contain disparate information, hence the joint feature idea above.
Rigid vs Non-Rigid: Reviewer 1 asks if rigid registration is appropriate. As we are mainly interested in aligning the torso & both scans are supine, rigidity is mostly valid; our dataset’s only real pose variation is due to legs moving. However our method gives local image descriptors and hence can be used for non-rigid registration by existing methods, eg. diffusion-regularized deformation. We could also allow for pose misalignment during pretraining with an appropriate loss, eg MIL-NCE (Miech et al 2020). We will elucidate in the final draft.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors have addressed the minor issues and added another comparison of conventional metrics as recommended. I think the Mutual Information (rigid) are helpful, but it appears MIND was used in a less than optimal setting (nonrigid and without discrete optimisation) and hence only reduces the initial error to half - so those results could be omitted. Overall, I confirm my positive initial assessment and recommend acceptance of this nice contribution. In future the authors may use public benchmarks such as Learn2Reg to position their work in the context of further SOTA.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
1
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This paper seems to present a straightforward idea for a fairly unique application. It was, unfortunately, not properly reviewed – R2 and 3 gave essentially no reviews, and R1 was borderline acceptable in quality. Overall, there was no clear reason to accept or reject the paper. Doing my best to read these sparse reviews, the primary MR, the rebuttal, and taking a look at the paper, this seems like an interesting paper worthy of discussion. The ideas included are creative, I think, and the contrastive adaptation is interesting for (multi-modal) registration.
It is a bit peculiar that contemporaneous registration methods are not thoroughly tested – like a deep unsupervised method with a multi-modal loss (MI, MIND, etc). The authors address some of this in the rebuttal and while I’m not a fan of accepting a paper given substantial rebuttal experiments, I think in this case it is fitting. Overall, the authors should do a better discussion of current registration approaches, including multi-modal ones, to better complete the paper.
Overall I think the paper should be accepted – I hope the authors build on the limited feedback they got, and improve the motivation, registration literature discussion, and new experiments they showed to both improve the paper and have more productive discussion at the conference.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The main drawbacks pointed out by R1 - clinical motivation and missing baselines - were fully addressed in the rebuttal. A detailed motivation was provided and will be included in the paper. The requested baselines were run, appear sensible and will also be included in the paper. I agree with the original meta-reviewer that R2 is of low quality and should not have a large influence on the final decision. With all points of R1 addressed, I recommend “accept”.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
5