Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Aleksandr Belov, Joël Stadelmann, Sergey Kastryulin, Dmitry V. Dylov

Abstract

We went below the MRI acceleration factors (a.k.a., k-space undersampling) reported by all published papers that reference the original fastMRI challenge, and then used deep learning based image enhancement methods to compensate for the underresolved images. We thoroughly study the influence of the sampling patterns, the undersampling and the downscaling factors, as well as the recovery models on the final image quality for both the brain and the knee fastMRI benchmarks. The quality of the reconstructed images compares favorably against other methods, yielding an MSE of 0.00114, a PSNR of 29.6 dB, and an SSIM of 0.956 at x16 acceleration factor. More extreme undersampling factors of x32 and x64 are also investigated, holding promise for certain clinical applications such as computer-assisted surgery or radiation planning. We survey 5 expert radiologists to assess 100 pairs of images and show that the recovered undersampled images statistically preserve their diagnostic value.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_25

SharedIt: https://rdcu.be/cyhU9

Link to the code repository

N/A

Link to the dataset(s)

https://fastmri.org/

Reviews

Review #1

Please describe the contribution of the paper

The authors present a comparison of different combinations of Deep-learning based super-resolution for MR image reconstruction. Their method is based in taking images reconstructed with undersampled k-space data to generate high-resolution images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

One major strength of this paper is the completeness of evaluation metrics. These include images for the reader, metrics and numerical evaluation with 5 clinical reviewers.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main limitation is the somewhat reduced set of algortithms tested. Here the authors use two HR reconstruction networks, plus 3 configurations of the latter. The comparison would be more interesting if it included alternative approaches or compared to reconstruction methods from raw k-space data.

Another limitation is the lack of real data to evaluate the results. The current experiment includes only synthetic downsampling.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The description provided in the paper (combined with the appropriate citations of the methods) seems sufficient to reproduce the results. Moreover, the synthetic data used is available online.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

This is an interesting comparison between alternative approaches. For a longer paper it should be extended with validation with real data from prospective experiments and incorporate comparison to other methods.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper provides a comparison between alternate configurations for reconstruction of HR images. The results reported are specific to the networks presented here, but yet worthy of publication. The study is complete and well rounded with different analysis metrics, although it would benefit from real data.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The paper used DL for accelerated MRI reconstruction and evaluated their performance based on expert opinion from 5 radiologists.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main contribution of this paper is the experimental work and with radiologists’ evaluation of the performance of DL for accelerated MRI reconstruction. The authors also claim their best model outperforms the state-of-the-arts.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
I have the following comments:
- The authors did not included the state-of-the-art DL MRI recon methods in their comparison which made the claims somewhat unjustified.
- The simulation of non Cartesian subsampling sampling patterns was very limited and not realistic. This simulation has only limited representation of any real non Cartesian k-space image quality. Therefore the claims on feasibility in extreme k-space undersampling was not justified.
- The radiologist’s assessments were performed using the radial mask. However, as they are not representative as to real MRI data acquisition. I have significant concerns with the validity of any results coming out from this.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility is reasonable since the methods details are provided and datasets are publicly available. However pretrained models were not provided and it is unclear the exact start-of-arts comparison.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I recommend the authors to address my comments in point 4 above,
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is an interesting work and main limitations are 1) experimental work to validate their claims, 2) limited comparison with existing methods.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

2
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This work’s most significant contribution is demonstrating the reconstruction of x16 undersampled MRI data while still achieving high performance across 3 image quality metrics: MSE and PSNR, SSIM.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Study design: overall, this work is well structured. The authors begin with a strong hypothesis, concisely describe methods and present convincing results. Experimenting to observe the difference in performance based on the order of the application of the generators is appreciated.
- Paper presentation: minor grammatical edits aside, the paper is well written. Figures 1, 2 and 3 in particular are pleasing.
- Relevance: reconstructing undersampled MRI data continues to be an area of significant research effort, and any progress in this direction is valuable.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- U-net generator: the U-net generator is not introduced clearly in the text; the reader is forced to infer
- Statistical significance of user study: a deeper understanding of the results of the user study beyond what is presented is greatly desired.
- Results from other planes/anatomies: the performance of the networks on imaging planes/anatomies not encountered in the training dataset is sorely missed.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This work seems to be reproducible - existing network architectures have been leveraged, and critical implementation details have been presented. These are: training and inference time, hardware specifications and implementation framework.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Minor grammatical edits: can be fixed by carefully editing (for example - “31th layer for the…”)
- Converting to journal: the authors are strongly recommended to include experiments that demonstrate the performance of the methods on: (1) coronal and sagittal planes for brain data (2) other anatomies while being trained on brain data (2) other contrasts (3) if possible, combinations of all of the before. It is also advisable to perform a more thorough user survey by recruiting more radiologists; double-blind study where the radiologist is asked to choose between reference image and output image based on diagnostic quality.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- relevant research
- leverages existing network architectures and datasets
- extensive set of experiments to demonstrate performance
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Reviewers saw the examination of extreme acceleration in MR acquisition as interesting and important. The evaluation of the approaches, particularly the involvement of clinical reviewers, was viewed as a strength. Weaknesses cited by the reviewers were that the paper focused on a relatively limited comparison of algorithms and the fact that only synthetically resampled k-space data was used as opposed to real data. One reviewer raised concerns that this resampling was not realistic for MRI acquisition, but this concern was not shared by the other reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We thank reviewers for their valuable feedback and for pointing out the items which we can improve. We will address here two main concerns and will address the minor issues in the camera-ready.

Synthetic sampling. The paradigm of synthetic “acceleration” in MR acquisition is well established in compressed sensing field, where the full k-space is a standard starting point. Given that one also knows the real k-space sampling trajectories that an MR machine can support, this is a valid and a commonly accepted workaround to gauge the performance of image reconstruction algorithms (supported by the related work referenced herein). The strategy to compensate for the artifacts, especially in the extreme undersampling case, by virtue of image space superresolution models is the focus of our work.

Additional algorithms to compare against. We have considered three image-to-image models, three sampling strategies, and two downscaling sequences. We believe this results in a rather dense paper with a large number of experiments as is, especially given the short format of MICCAI paper. We hope the reviewers appreciate the very idea and the proof-of-principle of applying the most popular models to the artefactual data (SRGAN, Pix2Pix, U-Net). Obviously, the super-resolution field is very diverse on its own and considering other, perhaps more powerful, models is a sound way forward, which will be the subject of future work.

back to top

Towards Ultrafast MRI via Extreme k-Space Undersampling and Superresolution