Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Chun-Mei Feng, Huazhu Fu, Shuhao Yuan, Yong Xu

Abstract

Super-resolution (SR) plays a crucial role in improving the image quality of magnetic resonance imaging (MRI). MRI produces multi-contrast images and can provide a clear display of soft tissues. However, current super-resolution methods only employ a single contrast, or use a simple multi-contrast fusion mechanism, ignoring the rich relations among different contrasts, which are valuable for improving SR. In this work, we propose a multi-stage integration network (\ie, MINet) for multi-contrast MRI SR, which explicitly models the dependencies between multi-contrast images at different stages to guide image SR. In particular, our MINet first learns a hierarchical feature representation from multiple convolutional stages for each of different-contrast image. Subsequently, we introduce a multi-stage integration module to mine the comprehensive relations between the representations of the multi-contrast images. Specifically, the module matches each representation with all other features, which are integrated in terms of their similarities to obtain an enriched representation. Extensive experiments on fastMRI and real-world clinical datasets demonstrate that 1) our MINet outperforms state-of-the-art multi-contrast SR methods in terms of various metrics and 2) our multi-stage integration module is able to excavate complex interactions among multi-contrast features at different stages, leading to improved target-image quality.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_14

SharedIt: https://rdcu.be/cyhUF

Link to the code repository

https://github.com/chunmeifeng/MINet

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed a multi-stage integration network, namely MINet, for multi-contrast MR super-resolution. Unlike the existing work, the proposed method actively learns the feature representation from the complementary information of multi-contrast images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The feature fusion strategy at different stages is novel, which gives more powerful feature representation. Other existing works on multi-contrast SR mostly focus on the feature fusion at single stage.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The idea of cross-modality attention module is also proposed in [1], where they used the features for MR guided by the features of ultrasound images. [1] Jiao, Jianbo, et al. “Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis.” IEEE Transactions on Medical Imaging 39.12 (2020): 4413-4424.
- In the design of the architecture, there are some limitations. LR T2w image is upsampled by sub-pixel convolution, which requires integer scale factor for upsampling. What happens non-integer upsampling factors such as 1.5? In the multi-stage integration module, all layers should have the same number of channels. What happens if each layer has different number of channels?
- In the experiment, I assume the authors retrospectively downsampled the target modality images. How did they downsample the images?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Lack of explanation on the image dimensions of dataset, information on sensitivity regarding parameter changes, average run time. No standard deviation in table 1. No analysis of situations in which the method failed.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- In the proposed feature attention module, could the authors elaborate more on the difference compared to the previous spatial/channel attention techniques?
- Since the authors synthesize FS-PDWI from PD in fastMRI dataset, please use other general notations instead of T1WI and T2WI in the method section.
- Please provide standard deviation values in Table 1.
- In figure 3, the maximum value of the error map can be adjusted such as 0.1 to emphasize the results of the proposed method.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method is interesting and sounds reasonable. The authors performed extensive experiments to demonstrate the effectiveness of the proposed method.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The paper approaches the problem of MRI super-resolution by leveraging another auxiliary contrast. Multi-Contrast Feature Enhancement and Multi-Contrast Feature Enhancement are proposed to capture more informative features for the target-contrast image restoration.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Experiments were conducted on multiple datasets with different MR contrast types.
2. The paper is well organized and easy to follow.
3. The topic is interesting and clinically significant.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. It remains unclear if the presented results are statistically significantly different. We can observe that the best and second-best results are quite close in some dataset. Statistical significance tests with a correction for multiple comparisons should be conducted.
2. Since authors compare different deep learning-based methods, it is better to show the number of trainable parameters in different methods. This will demostrate that the reported improvement results from the network design.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Authors commit to releasing the code and pre-trained models.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Seems like the proposed method is a 2D approach. Why is there a 3D Conv in Fig.1(b). Please clarify this point. 2.In the section ‘Multi-Contrast Feature Enhancement’, F_Att is pure self-attention mechanism. Calling it as ‘Multi-Contrast …’ is quite confusing for readers.
2. Please add statistically significant test in Table 1.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The rating reflects innovation of the approach for Multi-Contrast SR for MR images. As stated in previous section, demostrating the improvement from network design and statistically significant will make this paper more persuasive.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The paper proposes an approach to fuse multiple contrasts of MRI scans to produce high-resolution MRI image reconstruction from low-resolution counterparts. It outperforms existing approaches solving the same problem.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper tackles an important problem.
- The paper is well-written, nicely explained and easy to follow.
- Proposes an approach that works very well and seems to outperform existing approaches doing the same.
- Comparison seems fair and is performed across datasets demonstrating consistent improvements.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Nothing serious I can think of. :) I added a few suggestions for baselines that the authors could perform if they think it is interesting.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors seem to have promised to release the source code upon acceptance which is important.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Suggestions for extra baselines:

I am curious to see a baseline that totally omits the low-resolution image T2 scan from the mix, and reconstructs a T2 scan from a T1. A CNN trained on {xT1, xT2} pairs. How does it perform when compared to using yT2?

At inference, the authors always assume that xT1 is available for them. I think the authors should emphasize this assumption in the introduction and conclusion explicitly to improve clarity.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is a good, useful contribution to the community. I have no complaints in particular.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper aims to solve super-resolution by using multi-contrast inputs. The contribution comes from integrating complementary information of multi-contrast images effectively. All reviewers are positive to this paper. There are some minor issues though, which should be addressed in the final paper.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We sincerely thank the reviewers for their high-quality reviews and constructive feedback on our manuscript. We provide point-to-point clarifications on the comments, which will be integrated into the final version of the paper.

R1# [Q1] Similar idea to the proposed paper “Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis”. [A1] Thanks for pointing this out. We have carefully read this paper, which focuses on image synthesis. However, there is no paired data in this work, which makes pixel-level constraints impossible. Following [4], we use raw paired MRI data in the k-space to obtain the multi-modal low-resolution (LR) and high-resolution (HR) images. This mimics the real image acquisition process, where an LR MRI image is scanned by reducing the acquisition lines in phase and slicing along the encoding directions. [4] Chen Y, et al. Brain MRI super resolution using 3D deep densely connected neural networks (ISBI 2018).

[Q2] How to solve non-integer upsampling factors, such as 1.5, and different numbers of channels in the multi-stage integration module. [A2] We can use interpolation and bicubic operations to obtain non-integer upsampling factors. Our multi-stage integration module can also be used to address different numbers of channels in our multi-stage integration module.

[Q3] How are images downsampled? [A3] Sorry for the confusion. Following [4], to mimic the real image acquisition process, we downsample the resolution by truncating the outer part of k-space and then applying an inverse FFT to obtain the LR image.

[Q4] Difference from previous spatial/channel attentions. [A4] Existing spatial attention mechanisms typically focus on the scale dimension of the feature, with little uptake of channel dimension information, while recent channel attention mechanisms ignore the scale information. To solve this problem, we propose a novel channel-spatial attention mechanism that contains responses from all dimensions of the feature maps. By doing so, our attention module can extract powerful representations to describe inter- and intra-channel information in continuous channels. This is mentioned on page 4 of our submission.

[Q5] Provide standard deviation values for statistical significance tests. [A5] Following your suggestion, we have provided the standard deviation values as follows: fastMRI \ SMS \ uMR 2x \ 4x 2x \ 4x 2x \ 4x
PSNR\SSIM
1.93 \ 0.06 2.11 \ 0.08 2.12 \ 0.07 1.86 \ 0.09 1.86 \ 0.09 2.32 \ 0.09 1.25 \ 0.03 1.01 \ 0.05 1.41 \ 0.08 1.64 \ 0.08 1.26 \ 0.07 1.46 \ 0.06 1.10 \ 0.04 1.47 \ 0.08 1.27 \ 0.06 1.35 \ 0.06 1.19 \ 0.08 1.32 \ 0.07 1.15 \ 0.05 1.03 \ 0.08 1.10 \ 0.08 1.30 \ 0.05 1.12 \ 0.06 1.27 \ 0.05 1.03 \ 0.04 1.13 \ 0.04 1.05 \ 0.05 1.00 \ 0.05 1.06 \ 0.06 1.16 \ 0.03 1.02 \ 0.02 1.04 \ 0.04 1.05 \ 0.04 1.01 \ 0.02 1.00 \ 0.03 1.11 \ 0.03

R2# [Q1] Statistical significance tests for Tab. 1. and number of trainable parameters. [A1] Please see R1# [A5].

[Q2] Why use a 3D Conv in Fig.1(b)? [A2] Because our F_Att module uses a joint channel and spatial attention, we employ a 3D convolutional layer to generate an attention map by capturing both the channel and spatial features. Please see R1# [A4] for a more detailed description of the design.

[Q3] F_Att as ‘Multi-Contrast …’ is quite confusing for readers. [A3] We call F_Att the channel-spatial attention module. Please see page 4, line 20.

R3# [Q1] Suggestions for extra baselines. [A1] Restoring a T2 image from T1 is different from multi-contrast image enhancement, which belongs to image synthesis. We will do this. Thanks.

[Q2] Emphasize the assumption on the available T1 images in the introduction and conclusion to improve clarity. [A2] We have emphasized our assumption on page 2, in the second paragraph. We will also add it to the conclusion.

back to top

Multi-Contrast MRI Super-Resolution via a Multi-Stage Integration Network