Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Ke Wang, Michael Kellman, Christopher M. Sandino, Kevin Zhang, Shreyas S. Vasanawala, Jonathan I. Tamir, Stella X. Yu, Michael Lustig

Abstract

Deep learning (DL) based unrolled reconstructions have shown state-of-the-art performance for under-sampled magnetic resonance imaging (MRI). Similar to compressed sensing, DL can leverage high-dimensional data (e.g. 3D, 2D+time, 3D+time) to further improve performance. However, network size and depth are currently limited by the GPU memory required for backpropagation. Here we use a memory-efficient learning (MEL) framework which favorably trades off memory with a manageable increase in computation during training. Using MEL with multi-dimensional data, we demonstrate improved image reconstruction performance for in-vivo 3D MRI and 2D+time cardiac cine MRI. MEL uses far less GPU memory while marginally increasing the training time, which enables new applications of DL to high-dimensional MRI. Our code is available at https://github.com/mikgroup/MEL_MRI.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_45

SharedIt: https://rdcu.be/cyhV2

Link to the code repository

https://github.com/mikgroup/MEL_MRI

Link to the dataset(s)

http://mridata.org/

Reviews

Review #1

Please describe the contribution of the paper

In this paper, the authors evaluate an existing memory-efficient framework for inverse problems (MEL) on both retrospectively and prospectively undersampled MRI data. Viability of the network is demonstrated on two datasets, knee MRI (in 3D) and cardiac cine MRI (2d + time).
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Paper is well written, and the supplemental material with video for cardiac cine MRI is very nice
- Viability of MEL is demonstrated on two datasets, prospective data undersampling is considered with different matrix size/scanning parameters
- A decent classical baseline (PICS) is considered.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Lack of methodological novelty, since the model essential repeats the model from Kellman et al., “Memory efficient learning for Large-scale Computational Imaging”
- Lack of U-net baseline
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Authors provide sufficient details to reproduce their experiments (with an exception that a private dataset was used)
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- I would cite the fastmri reconstruction challenge as well, as it offers a number of good deep learning baselines to consider
- Generally, I would always include a U-net postprocessing of one of the baselines for MRI reconstruction
- For a good clinical evaluation, one would need to consider how the model generalizes between different organs of interest, different scanners, etc. The evaluation done in the paper is OK, but does not answer these questions.
- Prospective subsampling will give different data than retrospective subsampling. Can you comment on that how your model would generalize?
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The paper is well-written and considers a relevant problem, and the experiments are sufficiently reproducible. A complaint is lack of methodological novelty, since it is an existing framework without modifications being applied
- There is no U-net baseline. I would consider changing my score if such results are added to the paper. Please do consider standardized metric settings (e.g. use the one from FastMRI or whatever). Typically it can be quite hard to beat the U-net.
- Preferably this baseline is computed on public data as well. I appreciate the reason why these datasets are chosen, but there are a few other options (fastMRI, Calgary-Campinas) which can also be run in the 3D setting (especially the latter since in the saved data only a 1D FFT has been taken in the axial direction)
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

-The authors proposed a memory-efficient learning (MEL) framework which favorably trades off memory with a manageable increase in computation during training.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-The authors propose the method, MEL, brings a practical tool for training the large-scale high-dimensional MRI reconstructions with much less GPU memory and is able to achieve improved reconstructed image quality.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

-A more clear outline of the next steps in research would be appropriate.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper is clearly stated, but the reproducibility is not high, and there is no code and data.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

-The training details are less. -A more clear outline of the next steps in research would be appropriate, some discussion but vague. There are already some new work about efficient learning.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The architecture proposed by the author makes it possible to train high-dimensional MRI for reconstruction using only 12GB memory.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This study applies the recently proposed memory efficient learning (MEL) to the model-based deep learning (MoDL) MRI reconstruction to facilitate 3D MoDL or increase the number of unrolled iterations in 2D MoDL. The MEL MoDL method is validated in 3D knee MRI and 2D cardiac cine MRI, demonstrating improved reconstruction performance with marginally increased training time.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Combine MEL and MoDL to break the constraints of GPU memory on the number of unrolled iterations which influence the final reconstruction quality. The proposed method has been validated in prospectively undersampled 3D knee MRI and 2D cardiac cine, showing promising results.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The novelty of this study should be better highlighted. As both MEL and MoDL have been proposed before, this study seems a mere combination of them. Actually, the application of MEL to MRI reconstruction using unrolled neural networks has already been demonstrated in the original MEL paper. The authors should explain explicitly their contributions.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is well explained. And the authors have made efforts to gurantee the reproducibility of this study by submitting the codes.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. In the Abstract section, the authors could explain explicitly how MEL is adopted to improve the reconstruction performance.
2. Please clarify whether the weights in different unrolled iterations are shared or not. If not, the 3D MoDL and 2D MoDL with more unrolls will have much more training parameters and require more training data, which should be discussed.
3. How many unrolls were used in the 3D MEL MoDL, please clarify this in Fig. 4, 6 and Table 1.
4. With MEL, the number of unrolls can be unlimited in theory. Then, how the optimal number of unrolls was determined for 3D or 2D MoDL? Have the authors observed a saturation of the performance with further increasing the number of unrolls?
5. Compared with original MEL, the authors should highlight what is new in this paper.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The raised topic is interesting, but this study seems like a mere combination of previosuly proposed MEL and MoDL. The novelty should be clarified.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper has received two favorable reviews. Please summarize the merits and the critique of the reviewers in a rebuttal addressing the identified weaknesses point-by-point.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We appreciate the reviewers for their thoughtful feedback! We are encouraged that they find our work is well explained (R1,2,3) and brings a practical tool for training large-scale high-dimensional MRI recons with much less GPU memory which positively influences the final reconstruction quality (R2,3). We address reviewer comments below and will incorporate all feedback in the final version.

[Re: novelty and contributions (R1, R3)] Our paper was motivated by (Kellman et al. 2020), which demonstrated proof-of-principle on small computational imaging problems. Here, we apply MEL to technically challenging applications of MRI that have not been previously explored. Additionally, Kellman et al. does not systematically analyze MRI reconstruction tradeoffs, nor do they evaluate the added value of MEL to these applications. In particular, they did not show the benefits of using high-dimensional reconstructions compared to low-dimensional ones. 1) In this work, we first optimized the network architecture and training parameters and showed improved reconstruction performance in terms of both image quality and quantitative metrics (PSNR, SSIM, and FID score) compared to a low-memory network baseline. 2) To our knowledge, our paper is the first one to demonstrate the MEL framework on 2D+time cardiac cine MRI reconstructions. As a result, we can train the 2D+time reconstruction network with a large number of unrolls using less than 4GB GPU memory. In-vivo results from both retrospectively and prospectively undersampled data show improved image quality. 3) One of the key contributions of our paper is that we demonstrate the advantages of high-dimensional DL reconstructions for prospectively undersampled scans without ground truth, which could be potentially deployed in clinical systems. Even though the scanning parameters of the testing sets are different from the training sets, our methods are still able to reconstruct improved image quality with finer textures.

[Re: U-Net baseline (R1)] Based on reviewer feedback, we trained a 3D-UNet for 3D MRI recon, with the following results: 29.55+-1.86 PSNR, 0.7805 +- 0.0392 SSIM, and 60.10 FID score, which were all worse than our proposed 3D method (Table 1). This is consistent with literature that has shown the advantages of 2D unrolled networks (e.g. Hammernick et al. 2017, Aggarwal et al. 2017) over 2D-U-Nets. We will add this result to the supporting information section.

[Re: generalizability of prospective subsampling (R1)] MoDL has been shown to generalize to different sampling patterns at inference time (Aggarwal et al. 2017) due to the use of an explicit data consistency step in the network. Here, we benefit from the same robustness.

[Re: low reproducibility (R2)] We submitted all source code and detailed instructions in the supplementary materials. We will add a link to the public Github repository if the work is accepted.

[Re: Future steps and plans (R2)] Future plans of this project involve the following two directions: 1) Clinical adoptions: In this work, we demonstrate the advantages of high-dimensional MRI reconstruction on clinical prospectively undersampled scans. We will further investigate the performance and robustness w.r.t scanning parameters, scanner vendors as well as anatomies. 2) Higher-dimensional recons: The next steps involve further extending our framework (3D and 2D+time) to 3D+time (e.g. 3D+time DCE) to achieve high-fidelity 3D dynamic reconstructions.

[Re: Performance saturation with the increasing number of unrolls (R3)] Similar to what has been shown in previous works (Aggarwal et al. 2017; Kellman et al. 2020), we did observe saturation. For our 2D+time cardiac cine reconstruction experiments, the image quality difference between 15 unrolls and 10 unrolls is much smaller than the difference between 10 unrolls and 5 unrolls. Therefore, we used 10 unrolls for our experiments, which still requires MEL due to the size of memory per unroll.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposed a memory-efficient learning for MRI image reconstruction. This proposed memory efficient approach is application of MEL on MRI combining with MoDL, applied on 3D knee MRI and 2D cardiac cine MRI. As pointed out the reviewers, the major limitation of this work is the limited novelty and more like an application of MEL method on MRI. This may have some practical impacts on MRI, however, hard to be considered as a novel contribution acceptable by MICCAI.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

11

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper combines to existing technologies to be able to perform DL-based MRI reconstruction in low-memory GPU. They are able to squeeze larger number of unrolls, which improves the reconstruction quality.

As reviewers point out the paper lacks novelty. However, memory efficient algorithms are always interesting for practical purposes.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

11

Meta-review #4

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors evaluated an existing memory-efficient framework for inverse problems (MEL) on both retrospectively and prospectively undersampled MRI data, with both cardiac MRI and knee MRI tested. Memory-efficient Learning is a high relevant topic for MICCAI and the authors have properly addressed the concerns on novelty and 3D variants.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

Memory-efficient Learning for High-dimensional MRI Reconstruction