Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Xu Chen, Yanda Meng, Yitian Zhao, Rachel Williams, Srinivasa R. Vallabhaneni, Yalin Zheng
Abstract
Affine registration has recently been formulated using deep learning frameworks to establish spatial correspondences between different images. In this work, we propose a new unsupervised model that investigates two new strategies to tackle fundamental problems related to affine registration. More specifically, the new model 1) has the advantage to explicitly learn specific geometric transformation parameters (e.g. translations, rotation, scaling and shearing); and 2) can effectively understand the context between the images via cross-stitch units allowing feature exchange. The proposed model is evaluated on two two-dimensional X-ray datasets and a three-dimensional CT dataset. Our experimental results show that our model not only outperforms state-of-art approaches and also can predict specific transformation parameters. Our core source code is made available online.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87202-1_3
SharedIt: https://rdcu.be/cyhPJ
Link to the code repository
https://github.com/xuuuuuuchen/PASTA
Link to the dataset(s)
https://medmnist.github.io/#dataset
https://learn2reg.grand-challenge.org/Datasets/
Reviews
Review #1
- Please describe the contribution of the paper
This paper a parameter-specific affine transformation model by explicitly learning all these spatial transformation parameters rather than learning their combinations. Furthermore, cross-stitch units have been developing for multi-task learning, and cross-stitch units to effectively learn an optimal combination of shared representations between image pairs.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Good organization and easy to follow.
- Has the advantage to learn specific geometric transformation parameters explicitly (e.g., translations, rotation, scaling and shearing).
- The authors have done comprehensive evaluation of the proposed method.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Predicting translation, rotation and scaling to form the transformation is a standard way in conventional method. In spite of implement this framework in deep learning based registration method, few novelty is drawn.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Good.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The intuition/motivation/contribution of such work is a bit unclear. It would be better to list the contributions as bullets.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Good organization and easy to follow. But predicting translation, rotation and scaling to form the transformation is a standard way in conventional method. In spite of implement this framework in deep learning based registration method, few novelty is drawn.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Review #2
- Please describe the contribution of the paper
This paper proposes a novel parameter specific affine learning network. Instead of directly predicting the affine transformation matrix, the network learns each parameter separately - translation, rotation, shearing and scaling. The final affine transformation is the composite of each individual transform. In addition, the network also integrates a cross-stitch unit from multi-task learning. Experiments show that by separately predicting affine network parameters the proposed structure outperformed existing networks.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The novelty of the paper would be the proposed parameter specific learning network and the benefit of such a network is that it allows us to retrieve each individual transformation.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the result of the experiment is impressive in the way that authors set up, it was conducted on image pairs between the “same” image, i.e., the target image is just a synthetically affinely transformed moving image. For medical images, this setup typically is not the case. Affine transformation is usually applied as an initial step for registration problems whereas the moving and target image have local deformable changes. It would be interesting to know when local changes exist whether the network is able to learn better than other baseline methods. Also it would be interesting to see if the network is sensitive to noise in the image.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Datasets that have been used in this work are all publicly available. Links are provided. Parameters in the network are explicitly mentioned in the manuscript. Authors have mentioned that the code will be released after the work is published. The reproducibility of the work is greatly appreciated.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Thanks for the nice work proposed by authors. Please see my questions and comments :
- During the initialization, authors setup some maximum ranges for each parameter that later are used to normalize the output. What is the unit for the \lamda of the translation? Is it just for example 0.2 pixels/voxels? If so it seems too small.
- The authors mentioned the order of the composition matters for affine results. Is there any exploration of the order of the composition?
- It may be nice to have some discussion on how sensitive for each parameter network is.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
I would recommend borderline acceptance of the paper given the novel network design and the performance of the network compared against baseline methods, although it would be nice to investigate whether the network is robust to noise or local changes.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #3
- Please describe the contribution of the paper
This paper proposed a novel framework for affine registration to interpret the effect of each type of transformations. The proposed PASTA framework also improved performance on baseline methods. In addition, a novel architecture with cross-stitch units is proposed and proven to be effective in 2D scenario.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Instead of regress transformation matrix A with 6 parameters, the proposed method directly regress rotation, translation, scaling and shear (7 parameters in total) to interpret the effect of each type of transformations. The proposed PASTA framework also improved performance on baseline methods.
- A novel architecture with cross-stitch units is proposed and proven to be effective in 2D scenario.
- Experiments are sufficient and convincing.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
no obvious weakness.
- Please rate the clarity and organization of this paper
Excellent
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Excellent reproducibility with code. Experiments are carried out on public datasets.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Writing:
- Page 1 paragraph 2, “is commonly for” should be “is commonly used for”.
- Please state your overall opinion of the paper
strong accept (9)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper proposed a novel framework for affine registration, instead of regress transformation matrix A with 6 parameters, the proposed method directly regress rotation, translation, scaling and shear (7 parameters in total) to interpret the effect of each type of transformations. The proposed PASTA framework also improved performance on baseline methods. In addition, a novel architecture with cross-stitch units is proposed and proven to be effective in 2D scenario. The paper is well organized, the experiments are sufficient and convincing. Reproducibility is excellent with code and necessary details. Overall I recommend strong accept for this paper.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
4
- Reviewer confidence
Confident but not absolutely certain
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This paper proposes regressing affine transformation parameters for image registration via a deep network. In particular, the authors propose to use an overcomplete parameterization of the affine transform (explicitly predicting translations, shear, and rotations) and combine it with cross-stitch units in the network design. There are several concerns which should be addressed during a rebuttal: 1) The only real registration validation measure appears to be the Dice scores for the Learn2Reg task 2. How does the approach compare to other approaches on Learn2Reg? The other experiments only provide normalized cross correlation measures, but no real registration validation measures (e.g., landmark error, segmentation overlap, etc.). How does the approach compare to a simple optimization based baseline with respect to NCC then? 2) Are results statistically significantly different? 3) What size do the images have? The HandMNIST dataset and the ChestMNIST dataset appear to all be resized to 28x28 according to the MedMNIST paper; are you using higher resolution images? 4) What are your evaluations based on? Are they only trying to recover synthetically created transformations (as suggested by reviewer 2)? If this is the case, what validation strategy was used for Learn2Reg? Also synthetic transformation recovery or truly across different images? Fig. 2 appears to indicate that evaluation is also for synthetic transformations only. 5) What are the intuitions behind improvements using the different affine transformation parameterization and the cross-stitch units? 6) Lastly, 2D/3D in the title suggests that this is a 2D/3D registration approach. Instead it is a registration approach for 2D or 3D registration respectively.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
5
Author Feedback
Thank the meta-reviewer and all the reviewers for the constructive comments. We respond to the comments point by point below.
Meta-review (MR) Q1: The only real registration validation measure appears to be the Dice scores for the Learn2Reg task 2. How does the approach compare to other approaches on Learn2Reg? The other experiments only provide normalized cross correlation measures, but no real registration validation measures (e.g., landmark error, segmentation overlap, etc.). MR Q4: What are your evaluations based on? Are they only trying to recover synthetically created transformations? If this is the case, what validation strategy was used for Learn2Reg? Also synthetic transformation recovery or truly across different images? A: In the current manuscript, we only reported experiment results by aligning synthetical image pairs as we can directly evaluate the errors between the predicted transformation parameters and the synthetical ‘true’ parameters (e.g. Table 2). For the MedMNIST and HandMNIST datasets, no landmarks or segmentation are available. We have used normalized cross-correlation (NCC) as the performance metric. The segmentations are provided with the Learn2Reg dataset. We have evaluated the registration performance by the segmentation overlap as measured by Dice scores. Inspired by Q1 and Q4, we have performed real experiments on the HandMNIST dataset and share the results here. In brief, 44,850 unique pairs were generated by randomly chosen from 300 different images of left hands (a ratio of 60:20:20 for training, validation and testing). The results on the testing set proved the CANet and PASTA would introduce improvement in NCC compared to the methods not using them or without registration (NCC=0.655). Furthermore, we randomly chose 50 pairs of images and annotated the middle finger fingertips. We then evaluated the distance between them in each paired image before and after registration. The average distance is 8.33 pixels before registration, 6.88 for DLIR, 4.46 for GlobalNet, 4.69 for GlobalNet+PASTA and 3.88 for CANet (n=3)+PASTA. We will include these results in the final version for completeness.
MR Q2: Are results statistically significantly different? A: We have performed t-tests, and results are as follows: except DLIR and CANet (n=1) for the HandMNIST and CANet (n=1) for ChestMNIST and 3D lung dataset, all the other networks using PASTA have shown statistically significant improvements than those without PASTA (p<0.001). On the other hand, when PASTA is used, CANet (n=3) performs significantly better than all the other networks (p<0.001) but GlobalNet for the 3D lung dataset. These results confirmed the value of PASTA and the effectiveness of CANet. We will update Table 1 with these results in the revision.
MR Q3: The HandMNIST dataset and the ChestMNIST dataset appear to all be resized to 28x28 according to the MedMNIST paper; are you using higher resolution images? A: We used the original size 64x64 available when we downloaded both the HandMNIST and ChestMNIST.
MR Q5: What are the intuitions behind improvements using the different affine transformation parameterization and the cross-stitch units? A: Compared to existing deep registration methods, our PASTA decoupled the transformation parameters and regressed each of them in a normalized range; thus, it is more effective to optimize. It allows the recovery of all the ‘physical’ transformation parameters, thus more intuitive and explainable. Cross-stitch units help to extract more useful features so that the regression performance is improved accordingly. Note, PASTA is generic and could be compatible with other networks. We will clarify the contributions and list them in bullets in the revision.
MR Q6: 2D/3D in the title suggests a 2D/3D registration approach. Instead it is a registration approach for 2D or 3D registration respectively. A: Thank meta-reviewer #1! We will change the title to avoid confusion about using “2D/3D” term.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
This work proposes a simple approach for affine registration parameterization which can be integrated with CNNs to obtain easier control over the parameter ranges. All three reviewers appreciated this work. However, there were some concerns raised in the review in particular related to the evaluation (which appeared to be based on synthetic deformations only) and statistical significance of the obtained results. The statistical significance of the results were provided in the rebuttal (and would presumably be integrated into a final version). However, evaluations are indeed only based on synthetic deformations, hence this concern remains. It would be useful to include this concern as potential shortcomings of the experimental results in the manuscript.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
While achieving high accuracy on synthetic transformations the paper fails to provide any clinically relevant evaluation. I found two of the reviews rather low-quality and concur with the statement that the novelty is very limited (conventional methods also over-parameterise the estimation of linear transforms). The meta-reviewer asked authors to provide meaningful results (TRE) which would be available for Learn2Reg lung and could at least be computed for the synthetic 2D transforms. The authors did not respond adequately to this request. Evaluating registration simply with NCC is not appropriate (see Rohlfing TMI 2013). I think this submission falls short of MICCAI standards for clinical impact.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
20
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Overall, the paper was appreciated by reviewers and MR, but overall the idea felt simple and underexplored, which I agree with. Affine registration is important, of course, but there are a plethora of datasets to work with, annotations to use for measuring, baselines to run. It does seem like the authors could’ve done a better job on these experimental aspects.
The authors address some of this in the rebuttal – adding more number, statistical tests, etc. This is good, and needs to be in the paper, but it keeps the paper borderline in my view, as all of this should have been done better and more thoroughly during submission, in a problem that is so widely studied.
Given the nature of MICCAI2021, I believe the paper can be accepted and will lead to a good discussion, but I strongly encourage the authors to improve their paper during the camera ready by adding thorough experimental results and discussion mentioned in the rebuttal and otherwise requested by the reviewers.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7