Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Evaggelia Tsiligianni, Matina Zerva, Iman Marivani, Nikos Deligiannis, Lisimachos Kondi

Abstract

In medical image acquisition, hardware limitations and scanning time constraints result in degraded images. Super-resolution (SR) is a post-processing approach aiming to reconstruct a high-resolution image from its low-resolution counterpart. Recent advances in medical image SR include the application of deep neural networks, which can improve image quality at a low computational cost. When dealing with medical data, accuracy is important for discovery and diagnosis, therefore, interpretable neural network models are of significant interest as they enable a theoretical study and increase trustworthiness needed in clinical practice. While several interpretable deep learning designs have been proposed to treat unimodal images, to the best of our knowledge, there is no multimodal SR approach applied for medical images. In this paper, we present an interpretable neural network model that exploits information from multiple modalities to super-resolve an image of a target modality. Experiments with simulated and real MRI data show the performance of the proposed approach in terms of numerical and visual results.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_41

SharedIt: https://rdcu.be/cyhVo

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This work proposed a multimodel convolutional deep unfolding framework for medical image super-resolution. Specifically, they try to utilize multimodel convolution spare coding model to restore high-resolution T2W image using low-resolution T2W and high-resolution T1W image.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is clearly written and easy to follow.
- The proposed framework is technically sound.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some citations are missing, e.g., [1] and [2]
- The comparison with other state-of-the-art methods is not sufficient, only one method is compared.
- There is no detailed analysis and discussion about the results quantitatively and qualitatively
[1]. Zhou, Bo, and S. Kevin Zhou. “DuDoRNet: Learning a dual-domain recurrent network for fast MRI reconstruction with deep T1 prior.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. [2]. Xiang, Lei, et al. “Deep-learning-based multi-modal fusion for fast MR reconstruction.” IEEE Transactions on Biomedical Engineering 66.7 (2018): 2105-2114.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

it’s possible to reproduce this work.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Section 2 and 3 should be brief introduction, the authors spends too much context on these two parts.
2. More comparisons with other related work as listed above are needed.
3. There is no visualization comparison between coISTA and proposed method.
4. The figures in Fig. 2 should be rotated by 90 degrees.
5. Ablation study for adding high-resolution T1W or FLAIR images as complementary information to help recover better T2W image should be conducted.
6. The authors claim the proposed approach has fast inference, however there is no experiments to support this.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I think the paper has merit, it is well written and technically sound. Result section is problematic because the comparison between the proposed approach and other state-of-the-art methods is not sufficient.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

An interpretable image Super-Resolution neural network model exploiting information from diverse modalities has been proposed.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Interpretable Deep Learning modeling for multimodal medical image reconstruction is a novel research topic.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Information about computational cost of the proposed method is missing.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

I think that reproducibility checklist for this paper has been properly fulfilled except for the average runtime for each result which has not been given.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

A learned multimodal convolutional sparse coding (LMCSC) model has been proposed. It is an interpretable neural network model.

The model has been applied to multi-constrast Medical Resonance Imaging obtaining very good results for simulated and real images.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

An interesting approach to a very hard problem has been described. The proposed model seems to have useful clinical application.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors proposed an interpretable neural network model, which exploits information from multiple modalities to super-resolve an image of a target modality. Experiments show the good performance in terms of numerical and visual results.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

-This paper is well organized. -The authors solved a coupled sparse representation problem using an iterative thresholding algorithm. And then unfolded this algorithm into a neural network form. -The authors incorporated this learned multimodal convolutional sparse coding model into a network to reconstruct high-resolution images.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

-The experiment is not sufficient and the details are missing. -The number of article pages needs to be expanded, and the workload is not enough. -Vague discussion about future work.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The details of the paper are not clear enough, and the reproducibility is insufficient.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

-The details of experiment are missing, and less experimental comparison results. -The number of article pages needs to be expanded. -A more clear outline of the next steps in research would be appropriate, some discussion but vague. There are already some good work about interpretable deep learning for medical images.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

-An interpretable neural network model was presented to exploit information from multiple modalities to super-resolve an image of a target modality. -The experimental results are few, and the comparative experiments are not convincing, and there is no ablation study.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed an interpretable deep network by fusing cross-modal medical images for super-resolution. The reviewers mostly feel positive on this work, and scored 6, 9, 6. The reviewers also have some comments on missing references, clarity on details, insufficient comparisons. The authors are suggested to revise the paper according to these comments in the final version.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We would like to thank the reviewers for the positive evaluation of our work and their constructive comments helping tο improve our paper. Next, we will try to reply to the major concerns raised by the reviewers. We will use the notation R#:Q$ to refer to comments included as a reply to Question $ by Reviewer #. We group the major reviewers’ comments as follows:

All reviewers (R1:Q7, R2:Q4, R3:Q4) assert that more reference methods should be included in our experiments for comparison.
A major obstacle while designing our experiments is that neither the code of related methods nor the details of the datasets employed in other works are published. Therefore, due to time limitations, we have chosen to implement an interpretable deep learning model and include it as a reference method, since we believe that this work is mostly related with our approach. However, we agree that the comparison is limited and we will do our best to include more reference methods in our final submission.

Some comments suggest that an ablation study should also be included (R3:Q9). We agree with the suggestion, and we will provide a study concerning the performance of our model for a varying number of stages in our final submission. However, we think that R1 has misunderstood our experimental setting in R1:Q7, where it is asked to include high-resolution (HR) T1W or HR-FLAIR images as complementary information to super-resolve low-resolution (LR) T2W; this is exactly what we have presented. An alternative experiment would be to test our method with LR-T1W or LR-FLAIR images as complementary information. As long as page limitations allow including such an experiment, it will be added in our final submission.

In R1:Q7, the reviewer suggests adding visual results of the reference methods. We will include such results in our final submission exploiting the additional page allowance.

In R3:Q6, the reviewer raises concerns regarding the reproducibility of our work. We believe that we have provided all the information concerning the parameter setting and the initialization details of the proposed neural network so as the interested researcher can reproduce the code. Nevertheless, we plan to make our code publicly available after paper publication.

Some references suggested in R1:Q4 will be definitely included in our final submission.

back to top

Interpretable deep learning for multimodal super-resolution of medical images