Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Pengfei Guo, Jeya Maria Jose Valanarasu, Puyang Wang, Jinyuan Zhou, Shanshan Jiang, Vishal M. Patel

Abstract

Reconstructing magnetic resonance (MR) images from under-sampled data is a challenging problem due to various artifacts introduced by the under-sampling operation. Recent deep learning-based methods for MR image reconstruction usually leverage a generic auto-encoder architecture which captures low-level features at the initial layers and high-level features at the deeper layers. Such networks focus much on global features which may not be optimal to reconstruct the fully-sampled image. In this paper, we propose an Over-and-Under Complete Convolutional Recurrent Neural Network (OUCR), which consists of an overcomplete and an undercomplete Convolutional Recurrent Neural Network (CRNN). The overcomplete branch gives special attention in learning local structures by restraining the receptive field of the network. Combining it with the undercomplete branch leads to a network which focuses more on low-level features without losing out on the global structures. Extensive experiments on two datasets demonstrate that the proposed method achieves significant improvements over the compressed sensing and popular deep learning-based methods with less number of trainable parameters.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_2

SharedIt: https://rdcu.be/cyhUt

Link to the code repository

https://github.com/guopengf/OUCR

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed an Over-and-Under Complete Convolutional Recurrent Neural Network (OUCR), which consists of an overcomplete and an undercomplete Convolutional Recurrent Neural Network (CRNN) for reconstructing magnetic resonance (MR) images from undersampled k-space data.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper newly combined over complete CRNN with an under complete CRNN for reconstructing undersampled MR images which could be different from conventional deep-learning-based MR image reconstruction methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- To demonstrate the effectiveness of a reconstruction algorithm that combined overcomplete CRNN with an undercomplete CRNN, more explicit explanations and rigorous experiments would be needed.
- In particular, the specific reasons for the increase in performance by combining the overcomplete CRNN and the undercomplete CRNN is unclear.
- The baseline model, i.e., CRNN, is a model that has already been proposed, and simply applying the concept of overcomplete CNN to the model lacks novelty.
- Although quantitative results showed better performance than the baseline (Table 2), it is difficult to see the performance increment in the presented figures (e.g., Fig. 3 and Supp. Fig. 2).
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The authors provided details about the proposed models, datasets, and evaluation. As the authors stated, the reference of the part of the datasets and the code seems be released after the review process.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The specific reasons for the increase in performance by combining the overcomplete CRNN and the undercomplete CRNN is unclear. More explicit explanations and rigorous experiments would be needed. It is very difficult to argue that a new algorithm for MR reconstruction was proposed simply by claiming that both local features and global features were used.
- In Experiments and Results, the propsoed model adds the concept of over complete CNN to the existing CRNN model, but there is no comparison with CRNN in Table 1, Fig. 3, and Fig. 4. When inferred from the results of Table 2, the difference between the existing CRNN and the proposed OUCR model does not seem to be significant.
- Although the quantitative results of the proposed network showed better performance than the baseline CRNN in Table 2, it is difficult to see the performance increment in the presented Supp. Fig. 2 . Especially, the difference between the baseline CRNN and OUCR seems to be very minor. Please provide alternative ways to highlight the difference between the models.
- The authors stated that trainable parameters are less than other models. However, since the number of parameters of the baseline CRNN model is already small, it is difficult to claim it as a novelty.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Even though the experiments showed better performance with the proposed method, the overall opinion is “probably reject” because of the mentioned weaknesses and execution of the idea.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors propose a recurrent neural network architecture using over and undercomplete representations for reconstruction from undersampled k-space data. Results are evaluated for 2 datasets and retrospective undersampled k-space data.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The network architecture follows the successful approach of recurrent neural networks for MRI reconstruction based on gradient unrolling, by including a data consistency layer. The existing architectures are improved by utilising over and undercomplete representations for the network architectures. Which are supposed to concentrate on low/high level features and by combining these, the authors can provide improved reconstruction results.

The methods are tested on 2 datasets and compared to the most common (and successful) network architectures. As such the results are put nicely into context.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The approach of gradient unrolling and recurrent neural networks for MRI reconstruction as such is not novel. As such, the paper proposes an improved network architecture to be utilized in this framework.

Computational details are missing: Memory, reconstruction and training times.

At parts, I found the motivation of under/over complete representations hard to follow and not very clear described.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors stated that codes will be published. The supplementary material includes clear description of the network architecture.

Tests are performed on one public dataset and one that will be referenced after acceptance.

Computation times and resources are not stated.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I found the description of over and undercomplete networks hard to follow at parts and how the recurrent neural network interacts. Maybe the authors could include somewhere in the beginning a high level explanation of the idea. My take away is that: The first (overcomplete) network extracts local features, these are then subsequently merged with high-level (global) features in the undercomplete network to produce the final reconstruction.

Between the encoder and decoder parts, the authors then use a recurrent neural network. This seems to me, that an unrolled algorithm is used twice? For both low and high level features. I am wondering if this could have been combined. At least a motivation for the separation would be helpful.

Finally, the authors omit computational details on computation times, training times, and memory consumption. Please state these for comparison to other methods.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Probably accept: The results are convincing, but the motivation and algorithm structure was at times hard to follow.

Omitted computation resources rise the question that the method might be very compute extensive.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The authors propose combining under and over-complete RNNs to balance attention between global and local features. They demonstrate their method on two datasets and compare it with other methods such as compressed sensing and DL-based reconstructions.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The hybrid model does seek attention to local and global features (fig. 1)
2. The method performs well compared to other methods due to the inclusion of the over-complete architecture, reflected also by the reduction in the number of parameters
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Retrospective undersampling is different from artifacts due to prospective undersampling such as eddy currents. Therefore, most of these results do not account for sudden changes in phase.
2. While the knee data is single coil, an accelerated reconstruction method should include provisions for multi-channel acquisition as is the norm in day to day MR scanning.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have committed to share code upon peer review and have benchmarked their algorithm on a standard dataset and with other published algorithms.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The reconstruction should be tested with prospectively acquired data if possible. If not possible, the authors should comment on how they would strategize to overcome random undersampling of phase encodes (jumping around in k-space) with strategies such as view ordering, including a subsequent phase smoothening constraint in the reconstruction, eddy current compensation, etc.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper provides a proof of the intuitive inclusion of the overcomplete arm that significantly benefits the undercomplete RNN implementations. This work will advance the strategies used in DL based reconstruction. However, demonstrating the reconstruction with prospectively acquired, multi-channel data would be ideal.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Two reviewers scored probably accept / accept, while the first reviewer scored probably reject. This paper proposed an Over-and-Under Complete Convolutional Recurrent Neural Network (OUCR) for MRI. All reviewers recognized the improvements in reconstruction accuracy. But two of them have questions on the motivations and descriptions on the over-complete and under-complete networks. The computational time should be also given. The authors are suggested to carefully answer the questions of reviewers in the rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

We sincerely thank reviewers for their valuable feedback. In what follows we provide clarification to points raised by the reviewers.

Motivation of Using Overcomplete Networks (Meta-reviewer, R1, R2): Most learning-based methods for MR image reconstruction usually use a generic auto-encoder architecture. The main intuition behind this kind of architecture is that at the initial layers, the receptive field of the filters is smaller, so low-level features (e.g., edges) are captured. At deeper layers, the receptive field of the filters is larger, so high-level features (e.g., the interpretation of input) are captured. We argue that using such generic architecture for MR image reconstruction might not be optimal, since low-level vision tasks (e.g., MRI reconstruction) are mainly concerned with extracting descriptions from input rather than the interpretation of input. Fig.1 and supp. Fig.1 visually describe how the receptive field is constrained in overcomplete networks. To have special attention in learning low-level feature structures while not losing out on the global structures, we propose a novel network (OUCR). Detailed ablation study in Table 2 and supp. Fig. 2 also demonstrate that combining with overcomplete CRNN can result in the best performance.

Computational details (Meta-reviewer, R2): All experiments in this study were conducted on a computer with two Nvidia RTX 2080 Ti GPU. Hyperparameters can be found in Page 5, Paragraph 2, Line 12-16. Here, we compare the computational efficiency between the proposed method (OUCR) and PC-RNN [ref.29] (the second best method). During the training, memory usage is 16464 MiB and 21056 MiB for PC-RNN and OUCR. The time of processing one batch data is 0.203 s and 0.256 s for PC-RNN and OUCR. During inference, the time of reconstructing one batch data is 0.041 s and 0.022 s for PC-RNN and OUCR. To summarize, OUCR is slightly computationally expansive during training compared to PC-RNN, since up-sampled feature maps demand more time for back-propagation. However, OUCR is faster than PC-RNN during inference, since OUCR has less parameters and more concise architecture. We will add more computational complexity details in the revised paper.

Comparison with CRNN (R1): We indeed compare the proposed OUCR with CRNN-based method (i.e. PC-RNN). PC-RNN is a SOTA CRNN-based method, which learns the mapping in an iterative way by CRNN from three different scales. In addition, we also conduct detailed ablation study to demonstrate the effectiveness of each designed module of OUCR (i.e. UC-CRNN, OC-CRNN, and RM) in Table 2 and Supp. Fig. 2.

Performance Increment(R1): The reported improvements achieved by OUCR are statistically significant (p < 10-5). The detailed statistical significance test is provided in Supp. Table 1. For visual comparisons in Fig. 3, we can clearly see the reconstruction error is suppressed around brain skull and sulcus (HPKS dataset), and around collateral ligament region (fastMRI dataset) by using OUCR.

Novelty(R1,R2): While many CRNN-based methods have been proposed for MRI reconstruction, to the best of our knowledge, our study is the first attempt to apply the concept of overcomplete architecture to CRNN for MRI reconstruction.

Description of OUCR (R2): Detailed network configuration of OUCR can be found in supp. Table2. The receptive field of OC-CRNN is smaller and that of UC-CRNN is larger. Visual explanation for the receptive field change can be found in supp. Fig. 1.

Acquisition protocol and multi-channel data (R3): To ensure the reproducibility of our work and fair comparison with other methods, we use the same sampling mask function as public available benchmark (fastMRI challenge). The k-space measurements is obtained by simulated acquisition with Cartesian undersampling. The current study is based on the single-coil images for proof-of-concept. The proposed network can be adapted to multi-coil data by expanding the input channels.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposed an over-and-under complete convolution RNN that combines the over-complete and under-complete networks as branches. It was claimed that the combination of these two branches can better model the local and global structures. Results show the effectiveness of this approach. Overall, the reviewers commented on the unclear motivations, missing computational details, and comparisons with baseline. In the rebuttal, the authors answered these questions mostly in a good way, and I think these concerns are mostly clarified, however, it is still a limitation on the motivation and advantages of combining over-complete and under-computed networks in such a proposed way. Overall, the network design of the combined over-complete and under-complete network has some novelties, and the results show improvements over baselines. It can be accepted but the authors are suggested to add these missing details and correct the unclear points commented in the reviews.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This work proposes a new deep learning approach for MR image reconstruction. The reviewers appreciated the improved reconstruction accuracy in comparison to competing methods. However, there were some reviewer concerns regarding the motivation for the approach and questions related to computational time and memory use. Those were answered in the rebuttal by the authors.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Two reviewers raised several concerns, including the motivation of this paper. While the authors attempt to address these concerns, the explanation is still not clear enough. However, the area chair thinks that the paper’s methodology is interesting and seems to perform well. We recommend accepting the paper for publication. Authors should modify the final draft according to the reviewers’ comments.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

back to top

Over-and-Under Complete Convolutional RNN for MRI Reconstruction