Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu

Abstract

The core problem of Magnetic Resonance Imaging (MRI) is the trade off between acceleration and image quality. Image reconstruction and super-resolution are two crucial techniques in Magnetic Resonance Imaging (MRI). Current methods are designed to perform these tasks separately, ignoring the correlations between them. In this work, we propose an end-to-end task transformer network (T$^2$Net) for joint MRI reconstruction and super-resolution, which allows representations and feature transmission to be shared between multiple task to achieve higher-quality, super-resolved and motion-artifacts-free images from highly undersampled and degenerated MRI data. Our framework combines both reconstruction and super-resolution, divided into two sub-branches, whose features are expressed as queries and keys. Specifically, we encourage joint feature learning between the two tasks, thereby transferring accurate task information. We first use two separate CNN branches to extract task-specific features. Then, a task transformer module is designed to embed and synthesize the relevance between the two tasks. Experimental results show that our multi-task model significantly outperforms advanced sequential methods, both quantitatively and qualitatively. Our code will be released.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_30

SharedIt: https://rdcu.be/cyhVd

Link to the code repository

https://github.com/chunmeifeng/T2Net

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper propose a deep learning-based method for joint MRI reconstruction and super-resolution. A two sub-branches network architecture is designed to achieve image reconstruction and super-resolution, respectively. A task transformer module is further introduced for transferring shared features. Experiments have demonstrated the effectiveness of the proposed network.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The problem formulation is clear and reasonable.
2. The results look good.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The technical novelty is somewhat limited since the two branches network can be easily implemented by stacking existing network modules.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think this paper can be reproduced since the authors only use general network modules to build the network.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. What is the motivation for constructing a network structure in a parallel way? Why not use the cascade structure, i.e., reconstruction first, then SR?
2. Why does the Rec branch have a stronger ability to remove artifacts than the SR branch? What is the evidence for this conclusion?
3. Lack of visual analysis and discussion of intermediate results.
4. What is the difference between the proposed task transformer module and the non-local network?
5. Missing references： · Sun, Liyan, et al. “A deep information sharing network for multi-contrast compressed sensing MRI reconstruction.” IEEE Transactions on Image Processing 28.12 (2019): 6141-6153. · Zhao, Xiaole, et al. “Channel splitting network for single MR image super-resolution.” IEEE Transactions on Image Processing 28.11 (2019): 5649-5662. · Cherukuri, Venkateswararao, et al. “Deep MR Brain Image Super-Resolution Using Spatio-Structural Priors.” IEEE Transactions on Image Processing 29 (2019): 1368-1383.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Overall, the proposed method has practical values for the MRI community. The technical novelty is somewhat limited.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The paper proposes a novel method to jointly solve the MRI reconstruction and super-resolution problems. The proposed method contains a task transformer module for transferring shared features. The authors have proved the proposed method has superior results compared to various sequential combinations of state-of-the-art MRI reconstruction and super-resolution models.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The problem is interesting and novel. This is the first work to introduce the transformer framework into multi-task learning for MRI reconstruction and SR.
2. The proposed method is novel. The task transformer works as a bridge to share features among related tasks.
3. The paper is well written and easy to understand.
4. The experiment is well designed and comprehensive. The authors have compared to recent SOTA and they shown the significance of each modolues in the ablation study.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. It would be good if the author can design a sequential baseline that uses both branches and compared it to the proposed method.
2. It would be good if we can have the results of MRI reconstruction and SR of join learning and separated learning, respectively.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors will release the code if the paper is accepted.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

No.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The problem is interesting and the proposed method is novel to me. They have shown the significance of the proposed method in the experiment.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposed an end-to-end task transformer network (T2Net) that combines both reconstruction and super-resolution to transfer shared structure information to the task-specic branch for higher-quality and super-resolved reconstructions.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper newly introduced the transformer framework into multi-task learning for MRI reconstructions and super-resolution which could be different from conventional deep-learning-based MR image reconstruction methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The experimental setup is not clearly described. For example, it is not described which undersampling pattern was used to retrospectively undersampled k-space data and how the MR images were retrospectively downsampled for the super-resolution task.
- The reconstructed MR images presented in the paper seem to be blurred in a variety of degrees (e.g. Fig. 3), but it is not clearly mentioned or discussed how to improve it.
- Since the results were provided with the data where both the acceleration and enlargement were applied, it is hard to see the effectiveness of the two branches (SR, RC branches).
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The authors provided details about the proposed models, datasets, and evaluation. As the authors stated, the reference of the code seems be released after the review process. However, some information about experiment setup is not clearly mentioned such as k-space undersampling pattern and downsampling strategies.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- In SR branch, it seems a simple upsampling layer is applied at the output of the SR breach. In general, an upsampling operation is usually applied before embedding into the network (pre-processing) or is embedded into the middle of the network to extract the features of the high resolution images. Since the features of the high resolution images are hard to be learned if the upsampling operation is located at the last layer, it might be difficult to expect the high quality of the reconstructed images. The authors need to provide more explanations about it.
- In Experimental Setup, it is not described which undersampling pattern was used to retrospectively undersampled k-space data and how the MR images were retrospectively downsampled for super-resolution task. The authors need to provide more detailed information.
- In Results, the reconstructed MR images presented in Fig. 3. seem to be blurred in a variety of degrees. The authors need to discuss the limitation of the proposed methods and how to improve the reconstruction quality in a carefully designed strategy.
- In Results, the authors provided the results with the data where both the acceleration and enlargement were applied. However, in order to see the effectiveness of the two branches, the authors need to provide reconstructed images of each branch and compare the results.
- In Fig. 3, the fully sampled MR images (i.e. reference images) are not provided. The authors need to present fully sampled images to the results.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Even though the experiments showed better performance with the proposed method, the overall opinion is “probably reject” because of the mentioned weaknesses and execution of the idea.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposes a joint solution for MRI reconstruction and super-resolution. While all review comments acknowledge the merit of this paper, they’ve also raised several critical questions (e.g., R4 particularly). In the rebuttal, the authors need address some confusing points in motivating their network architecture and specific choices of designing. Meanwhile, the concerns over comparisons to other methods and validation of the effectiveness of the proposed method should also be considered.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

We sincerely thank the reviewers for their high-quality reviews and constructive feedback on our manuscript. To the best of our knowledge, we are the first to introduce the transformer framework into multi-task learning for MRI reconstruction and SR. We will release all codes and results after publication. We provide point-to-point responses to the comments, which will be integrated into the final version of the paper.

[Q] Results of each branch (R#1/2/3). [A] To demonstrate the effectiveness of each branch, we will add the following PSNR/SSIM results of the Rec branch on the two datasets: IXI 2x (29.2\0.894); IXI 4x (33.1\0.951); clinical 2x (30.2\0.856); clinical 4x (33.0\0.942). The results of the SR branch are already provided in Tab. 1 (as ‘w/o Rec’). The additional results and Tab. 1 demonstrate that our task transformer module can transfer anatomical structure features to the target SR branch, leading to complementary representations. Moreover, we can see that both branches in our method are critical, as each contains task-specific features for the final image restoration.

[Q] Motivation behind parallel architecture (R#1/2). [A] The parallel network architecture is more conducive to multi-task, multi-scale feature fusion than its cascaded counterpart. In contrast to the cascaded structure, the parallel architecture provides multi-level fusion between different blocks/stages, while simultaneously preserving the task-specific representation for each branch. Moreover, we have provided experimental results against various sequential combinations of state-of-the-art MRI Rec and SR models in Tab. 1. These results show that our parallel multi-task model significantly outperforms advanced sequential methods, both quantitatively and qualitatively.

[Q] Strong ability to remove artifacts of Rec branch (R#1). [A] Since the Rec branch are trained under the supervision of xLR, it can effectively remove the artifacts than SR branch. We have clarified this in Sec. 2.1 “Reconstruction Branch” and the input and output of Fig.1. Thanks.

[Q] Difference from the non-local network (R#1). [A] 1. There are three identical input features in a non-local block. In contrast, our task transformer module has three different input features from different task branches. 2. unlike the non-local network, our task transformer module includes relevance embedding, transfer attention and soft attention, which enable related tasks to share visual features. 3. the non-local network aims to capture long-range dependencies. Our task transformer module aims to enable the SR branch to remove artifacts from the Rec branch, i.e., the module is able to transfer related features between branches.

[Q] Upsampling layer (R#3). [A] In our method, the upsampling module is not a single upsampling layer. We utilize the sub-pixel convolutional layer (as pixel-shuffle [1] and one conv layer [2]) to generate the final SR output (see the first sentence after Eq (3)). Additionally, post-upscaling strategies have been demonstrated to be more efficient in terms of computational complexity and achieve higher performance than pre-upscaling SR methods [2]. We will clarify our upsampling module in the updated version of the manuscript. [1] Real-Time Single Image and Video SR Using an Efficient Sub-Pixel CNN, CVPR 2016. [2] Image SR using very deep residual channel attention networks. ECCV 2018.

[Q] Experimental setup (R#3). [A] The undersampling pattern follows 6x Cartesian acceleration and is provided in Sec. 3 “Experimental Results,” line 1. We obtain the LR images according to [3], the details of which are provided in Sec. 2, line 9.

[Q] Blurring in Fig.3 (R#3). [A] The blurring in Fig .3 was caused by compression when we uploaded the figure to the manuscript. We will update Fig.3 with a high-resolution version. Additionally, we can evaluate the reconstructed image through the error map. Our method is clearly robust to artifacts and structural losses in the input.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposes a joint solution for MRI reconstruction and super-resolution. While all review comments acknowledge the merit of this paper, they’ve also raised several critical questions (e.g., R4 particularly). In the rebuttal, the authors addressed some confusing points in motivating their network architecture and specific choices of designing. Meanwhile, the concerns over comparisons to other methods and validation of the effectiveness of the proposed method were also considered.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposed a task transformer network with two branches for reconstruction and SR for joint MRI reconstruction and super-resolution. The two branches are connected by the transformer. The network design has some novelties (especially the introduction of transformer), and the results show improvements. The reviewers have concerns on the parallel architecture, effectiveness of the two branches, etc. The responses clarified these questions. Overall, this work has merits in network design and joint tasks. My remaining suggestions are to try different sampling rates / patterns in k-space, detailed comparisons for the reconstruction results and see if super-resolution task can improve the reconstruction task.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper receives mixed reviews, with R3 being negative. S/he questions about the upsampling layer, experimental setup, and blurry results. The rebuttal answers these questions in a candid fashio, in my opinion, and also, the answers can be included in the final version without pronounced space requirement. I therefore recommend the acceptance of the paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

back to top

Task Transformer Network for Joint MRI Reconstruction and Super-Resolution