Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Xinwen Liu, Jing Wang, Feng Liu, S. Kevin Zhou

Abstract

Deep neural networks have been extensively studied for undersampled MRI reconstruction. While achieving state-of-the-art performance, they are trained and deployed specifically for one anatomy with limited generalization ability to another anatomy. Rather than building multiple models, a universal model that reconstructs images across different anatomies is highly desirable for efficient deployment and better generalization. Simply mixing images from multiple anatomies for training a single network does not lead to an ideal universal model due to the statistical shift among datasets of various anatomies, the need to retrain from scratch on all datasets with the addition of a new dataset, and the difficulty in dealing with imbalanced sampling when the new dataset is further of a smaller size. In this paper, for the first time, we propose a framework to learn a universal deep neural network for undersampled MRI reconstruction. Specifically, anatomy-specific instance normalization is proposed to compensate for statistical shift and allow easy generalization to new datasets. Moreover, the universal model is trained by distilling knowledge from available independent models to further exploit representations across anatomies. Experimental results show the proposed universal model can reconstruct both brain and knee images with high image quality. Also, it is easy to adapt the trained model to new datasets of smaller size, i.e., abdomen, cardiac and prostate, with little effort and superior performance.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_21

SharedIt: https://rdcu.be/cyhUM

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper presents a universal approach for undersampled MRI reconstruction that can be adapted to different anatomies, by introducing an “Anatomy-SPecific In- stance Normalization” (ASPIN) module that can be swiftly inserted and trained for new incoming anatomies, as well as a model distillation process that allows the model to “absorb” such multi-anatomy images. The method is tested on the FAIR brain and knee datasets, and applied to smaller abdominal, cardiac and prostate datasets. Results show an improvement over separately (independent) and jointly (shared) approaches.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- novel idea that outperforms the state-of-the-art as per challenge leader board https://fastmri.org/leaderboards/challenge/ (2020 for brain and 2019 for knee data)
- computationally efficient, requiring less than 0.1% additional parameters to be trained when adding an ASPIN module
- promising alternative to transfer learning and other domain shifts
- excellent results outperforming state-of-the-art
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- this appears to be a 2D reconstruction method only, no discussion on 3D
- unclear whether slices of the same patient were mixed between training/validation/testing sets since these are randomly drawn
- abdominal, cardiac and prostate data do not have raw k-space data, so only magnitude images are taken and phase is ignored
- no insight provided as to why this approach works better than shared or independent learning - ie where does the improvement happen? Also, what exactly does the MD contribute in this improvement, and why?
- unclear whether adding more training data of the same anatomy type would have equally boosted the performance
- no discussion on whether this type of approach, rather than multi-anatomy, would also work multi-spectral, ie using different MRI protocols but for the same anatomy
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Use of public datasets allows for benchmarks
- graphical model overview and equations for ASPIN function, generalising the IN function, provided
- no code nor pseudo-algorithm provided - would I know how to quickly reimplement this? Probably not straightaway…
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I really like the idea of a multi-purpose reconstruction approach for undersampled data, and am impressed by the results which outperform the challenge leaderboard for the knee and brain data. I am actually intrigued as to where the improvement comes from? I am less certain about the other data sets, especially since k-space will be limited to the magnitude images only, and some shortcuts emulating just single-coil images have been taken. I would like to see more discussion as to why some of the improvements happen, even more so than by how much; and some speculation as to why, instead of adding new anatomies, more image protocols for the same anatomy could be inserted, which might provide even better clinical value.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is some innovative work which I would like to see and discuss at the conference. Results are excellent compared to the FAIR challenge leader board, and even though it is not entirely clear to me what actually makes them so superior, I would like to find out.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper proposed a universal deep neural network with an anatomy-specific instance normalization (ASPIN) and model distillation for undersampled magnetic resonance image reconstruction.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper newly proposed a framework to train a universal network, which can reconstruct images of different anatomies and is generalizable to new anatomies easily. Thus, this could be different from conventional deep-learning-based MR image reconstruction methods that usually need to train an individual model for each dataset separately.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- To demonstrate the effectiveness of the model distillation and ASPIN in reconstructing images of different anatomies, more explicit explanations and rigorous experiments would be needed.
- There is a lack of supported experiments and explanations on how the proposed ASPIN method can capture anatomy-specific knowledge.
- Although quantitative results showed better performance than the baseline (Table 1), it is difficult to see the performance increment in the presented figures (Fig. 2).
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The authors provided details about the proposed models, datasets, and evaluation.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- In Table 1, the difference between the proposed method and ‘w/o MD’ seems to be very minor compared to other methods. This raises questions about the effectiveness of the model distillation in Step (S3). Please clarify it.
- In Section 2.2, the authors proposed ASPIN and stated that is is expected that ASPIN captures anatomy-specific knowledge by learning the separate affiine parameters for each anatomy. However, it is questionable whether simply learning different affine parameters (gamma and beta only) for each anatomy in a layer can be seen as learning “anatomy-specific” knowledge. To support the opinion about the proposed ASPIN, more explicit explanations and rigorous experiments would be needed.
- In Section 3.3, the model that only applies Step (S1, S2, S3) needs to be compared to see the effectiveness of the model distillatiion compared to the pure ‘shared’ model.
- In Results, although quantitative results showed better performance than the baseline (Table 1), it is difficult to see the performance increment in the presented figures (Fig. 2). Please provide figures that can show the effectiveness of the proposed model.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Even though the experiments showed better performance with the proposed network, the overall opinion is “borderline reject” because of the mentioned weaknesses and execution of the idea.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The paper approaches the problem of multiple anatomies MRI reconstruction in setting of using a single universal model. An integrated network of ASPIN and Model Distillation is proposed to overcome domain shift and preserve the knowledge from fully-trained independent models.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The topic is interesting and clinically significant.
2. Experiments were conducted on multiple publicly available datasets.
3. Good writing and easy to follow
4. The idea of combining ASPIN and Model Distillationas for universal recon is interesting.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Statistical significance tests with a correction for multiple comparisons should be conducted.
2. Lacking of comparsions with other methods that are desined for overcoming domain shifts.
3. Experiments were conducted on magnitude-only images. Further verification on real complex data will be needed.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors commit to releasing the code and pre-trained models.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Model Distillation is conducted in the third layer of each CNN-t. Authors may clarify the motivation of choosing the third layer. What if Model Distillation are added in all layers? Will this further improve the performance?
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The rating reflects innovation of the approach for universal MR image reconstructions. Interesting approach, the merits outweight the weaknesses.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This is a solid article with clear novelty. All reviewers and this meta-reviewer agrees on this point. All reviewers raise issues regarding the experimental setup. I believe it is important that authors address and clarify these issues in a rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Author Feedback

Experiments on the effectiveness of ASPIN/MD (R2). We included ablation studies (Sec.3.3) to show the effectiveness of ASPIN/MD in the manuscript. Besides, we have conducted the following experiments, which will be included if space allows: -The learnt ASPIN (beta and gamma) for each anatomy are different but related to the dataset statistics. The cosine similarity between the mean value of 4x undersampled images of each dataset and the learnt beta for each anatomy is up to 0.97. -We used the ASPIN trained on brain images to test cardiac data, and PSNR decreased by 1.4dB. We have done this for each ASPIN and dataset, and the performance always drops. This proves each dataset needs the ASPIN corresponding to its own distribution. Thus, the importance of ASPIN is further validated. We have tried our best to show the effectiveness of ASPIN/MD and would like to invite Reviewer #2 to advise on other experiments.

More explanation on ASPIN/MD (R1,R2). ASPIN: The statistical shift among datasets harms the performance of the “Shared” (Sec. 3.2). Affine parameters in ASPIN capture the statistical representation (mean and variance) of datasets. ASPIN is inserted after the convolution layer for each dataset learning its own scaling and bias parameters. This allows each dataset to scale the feature maps from the antecedent convolutional layer according to its statistics, capturing anatomy-specific feature representation. Dataset-specific normalization for multi-dataset generalization is also in [ChangCVPR2019, LiuTMI2020]. Compared to “Independent”, our method “sees” more training data. With the statistical shift problem solved by ASPIN, the proposed universal model performs better than individual models. MD works by distilling knowledge from well trained individual models. It introduces additional supervision from each model and can be considered as a regularizer, improving the training of the universal model. Without MD, the model does not fully leverage the anatomy-specific knowledge in the single anatomy models.

Experiments only apply to Step (S1, S2, S3) (R2). Sec. 3.3 compares the model applied to a single step. Only applying S1 is “Independent”. Only applying S2 is “w/o MD”. The effectiveness of MD is evaluated by comparing “w/o MD” with the proposed method. It is not feasible to apply only S3, because S3 is trained in conjunction with networks of S1 or S2.

The results are not obvious (R2). Fig.2: The improvement is observed from the ROI and error maps, e.g., the details of cerebri (middle of ROI) in the brain image are largely preserved in the proposed method compared to others. The error maps in the proposed method are much ‘lower’ than the others. Apart from performance improvement, other significant benefits of our method compared to others are the reduced parameter number (Sec. 3.4) and easy generalization to new anatomies. Tab.1 Proposed v.s. w/o MD: the proposed MD does not introduce additional parameters and is easy to implement. The 0.24dB improvement is significant for a single component. MD alone brings around 16% PSNR improvement over the ‘Shared’. We also calculate the p value<0.0001, showing that the proposed outperforms ‘w/o MD’ with a statistical significance.

Minors: -3rd layer for MD as in [15]. The comparison of each layer for MD is in the supplementary material, with the 3rd layer being the best. MD on all layers increases the memory burden, which limits the implementation. (R3) -Please advise which domain shift methods to be compared, since we cannot find other literature on the universal reconstruction problems. (R3) -p value is always < 0.0001, showing the statistical significance of the experiment. (R3) -We did not mix slices between train/val/test. The split is based on volumes. (R1) -We cannot find public abdominal/cardiac/prostate raw data. we will extend to complex-valued data in the future. (R1, R3) -Following [14-28], we perform 2D reconstruction. It is easily extendable to 3D. (R1)

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Authors’ rebuttal addresses lack of explanation in the experimental setup. I suggest including such discussions in the main article. This is a solid article and authors’ rebuttal confirms this point.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The module proposed in the paper for knowledge distillation from different anatomies using instance normalization is a very interesting approach, which may be applicable in other image analysis problems. The reviewers’ concerns seem relatively minor, and the authors address most of these in the rebuttal. I would therefore recommend the acceptance of this work.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes a novel method with outperforming results. The strengths of the paper outweigh the weaknesses.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

Universal Undersampled MRI Reconstruction