Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Zhiyang Lu, Zheng Li, Jun Wang, Jun Shi, Dinggang Shen

Abstract

The thick-slice magnetic resonance (MR) images are structurally blurred in coronal and sagittal views, which causes harm to diagnosis and image post-processing. Deep learning (DL) has shown great potential to reconstruct the desirable high-resolution (HR) thin-slice MR images from these low-resolution (LR) cases, which we refer to as the slice interpolation task. However, since it is generally difficult to sample abundant paired LR-HR MR images, the classical fully supervised DL-based approaches cannot be effectively trained to get robust performance. To this end, we propose a novel Two-stage Self-supervised Cycle-consistency Network (TSCNet) for MR slice interpolation, in which a two-stage SSL strategy is developed for unsupervised network training. Paired LR-HR images are synthesized along the sagittal and coronal directions of input LR images for network pretraining in the first stage SSL, and a cyclic interpolation procedure based on triplet axial slices is then designed in the second stage SSL for further refinement, Sufficient training samples with rich contexts along all directions are exploited as guidance to guarantee the promising interpolation performance. Moreover, a new cycle-consistency constraint is proposed to supervise this cyclic procedure, which can encourage the network to reconstruct realistic HR images. The experimental results on a real MRI dataset indicate that TSCNet achieves superior performance over the conventional and other SSL-based algorithms, and gets competitive visual results compared with the fully supervised algorithm in our slice interpolation task.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_1

SharedIt: https://rdcu.be/cyhUs

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

MR image super resolution is an important research topic, because clinical scans usually have thick slices, where anatomical details can be missing. To address this problem, this work proposes a Two-stage Self-supervised Cycle-consistency Network (TSCNet). TSCNet uses synthetic training image pairs and thus does not require external training data. In addition, a cyclic interpolation procedure is developed. The proposed method was validated on a real MRI dataset. My major concern is about the novelty of the paper, which will be described below.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The presentation of the proposed method is clear.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- There is a lack of novelty compared with existing works.
- The paper also misses some more recent and advanced related works. (see my detailed comments below)
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The implementation details are given, but code and data are not provided.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The idea of self-supervised super-resolution is not new. As described in the introduction, [15] has already proposed this idea of generating synthetic training scans from thick-slice images. In addition, there are several more recent works from the same group that have further improved the methodology, yet they are missing in the literature review, e.g., Zhao et al., MICCAI 2018 and Zhao et al., TMI 2021. The idea of cycle-consistency is not new either. This idea has been extensively used for image generation/synthesis, for example, in CycleGAN. Although the details of the proposed method may be different from existing works, the general ideas of the proposed method are quite similar to existing works, and thus the novelty is limited.
- The authors claim that “due to the difficulty of collecting HR MR images in clinical practice, there are generally few datasets with sufficient paired samples available for model training”. While I agree that few datasets have HR images for training, it may not be difficult in clinical practice to acquire a few HR training scans to train a network model for super-resolution. But for existing datasets where it is not convenient to acquire additional HR training scans, self-supervised super-resolution is very valuable. I would suggest the authors better clarify the motivation of self-supervised super-resolution.
- In section 2.2, it is argued that “since the axial slices are originally sparse, this strategy cannot generate sufficient paired data of the input adjacent slices and the labeled intermediate slice for training”. However, there can be multiple training scans, and when they are combined together, a large number of training slices can be generated. Is it not possible to use multiple training scans? Please clarify.
- In section 3.1, the volumes were downsampled by a factor of two in the slice direction to generate LR images. However, this downsampling factor may not be representative enough. In practice, the slice of LR clinical images can be quite thick. It is common to have a thickness of 3 to 5 mm. In Zhao et al., TMI 2021, a range of downsampling factors (from two to six) were considered, which is much more realistic. Please consider modifying the experimental settings for more realistic simulation.
- The proposed method was compared with [15], but not with the more advanced versions, such as Zhao et al., MICCAI 2018 and Zhao et al., TMI 2021.
- The relative improvement of the proposed method presented in Table 1 is small. Compared with EDSSR, the improvement is about 0.7 (2%) for PSNR and 0.0075 (0.8%) for SSIM. Would this marginal improvement make significant impact in practice? For example, the authors could consider comparing the brain parcellation results obtained from the super-resolved images and see if the marginally improved image quality benefits the image analysis.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

There is a lack of novelty and sufficient review of existing works. The improvement is also marginal.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

6
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The main contribution of this work lies in the proposed two-stage SSL framework, which aims to improve the resolution of MR images without requiring high-resolution images for supervised learning. In addition, the interpolation network is trained with the sparse axial slices in a triplet manner, and a cyclic consistency constraint is proposed for improving the performance.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strengths of this work are listed as follows:
1. A two-stage SSL framework is proposed for the resolution enhancement of MR images. For the first stage, paired LR-HR images are synthesized across the sagittal and coronal directions.
2. For the second stage, sparse axial slices are exploited in a triplet manner.
3. A new cycle-consistency constraint strategy is developed to supervise the second SSL stage.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The first stage of SSL can be further improved by considering axial views. In addition, some experimental details are not clear. For instance, the implementation of the bassline method and ablated versions.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of this work is good. The authors have provides the details regarding network architecture, implementations, etc.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Why not also considering the axial slices in the first stage of SSL? In other words, the LR image can be synthesized from the slices from three views, i.e., axial, sagittal, and coronal. This will provide more samples for training. In particular, the samples from axial views are of better quality, which can improve the performance further.
2. The ablated version “TSCNet with AFS” is not clearly described. Specifically, the authors mentioned that “This algorithm modified the first stage SSL based on the common methods for video frame interpolation …” What are the “common methods” for video frame interpolation? At least, a reference should be provided here.
3. The visual improvement is not significant. It is very hard to identify the advantage of TSCNet over EDSSR and its ablated versions.
4. TSCNet used RDN as its backbone network. How about EDSSR? Is the same backbone network used in EDSSR?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed SSL framework is interesting, especially for the second stage. However, more details should be provided and there is still room for improvement.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposes a self-supervised learning method for reconstruction of thin-slice MRI from thick-slice MRI. The method first synthesizes image pairs on sagittal and coronal directions to pretrain the reconstruction network. It then uses a cycle-consistency loss to fine-tune the network on axial slices. The proposed method outperforms two existing methods for brain MRI slice interpolation and has comparable performance to the fully-supervised upper-bound.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The method is novel. The proposed cycle-consistency loss for slice interpolation is innovative. The idea of reconstructing the (i+1)-th slice from adjacent slices and then applying loss on it is interesting. This loss allows training without ground-truth images, which is highly desirable for real-world clinical applications.
2. The method is compared with a sufficient number of existing methods and shows superior performance both qualitatively (Fig. 3) and quantitatively (Table 1). The ablation study is well-designed and demonstrates the benefit of each component of the algorithm.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The method is validated on only one dataset of brain T1w MRI. The performance on other organs and modalities is unknown. The size of dataset is also small, including only 64 scans.
2. There are a few issues and typos in paper writing, as listed in the constructive comments section.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper uses a public dataset (ADNI). The authors will release code if paper is accepted, according to the checklist. There are also detailed descriptions of algorithm implementation in the paper. Over the reproducibility is good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Page 8, line 1: “Table 1 gives the qualitative results of experiments, …”. It should be “quantitative” results.
2. Please provide more details on data division. How many scans are used for training and testing?
3. In Eq. 1, there are only (X/2)-1 items added in the sum, but the result is divided by (X-2) for calculating average.
4. Page 8, paragraph 1, last sentence: “whereas the cycle-consistency constraint utilized in the second stage of TSCNet can benefit to reconstruct more realistic HR images.” What does “benefit to reconstruct” mean?
5. The font looks different from MICCAI format. Please change if necessary.
6. Adversarial loss is used in the proposed method, which is beneficial for high-resolution reconstruction. Does EDSSR or FSIN use adversarial loss? If not, how would the proposed method compare to a method that also uses adversarial loss?
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper presents a novel method for MRI slice interpolation, with an interesting and creative design of cycle-consistency loss. The experiments are comprehensive and well-designed to show the benefits of proposed method. The results achieved for MRI super-resolution are promising.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper was considered by reviewers to be clear with reasonable evaluation results. A significant concern was raised by Reviewer 1 that highly relevant literature was not cited, which called into question whether the method was novel and provided state of the art performance. It was also noted by multiple reviewers that some aspects of the methodology were not clear, such as the use of axial slices for training. Please address these issues in the rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

We would like to appreciate Reviewer #2 and #3 for the positive comments. The rebuttal to the first comment of Reviewer #1:

Self-supervised learning (SSL) has been widely used for different tasks, whose novelty depends on the specifically designed SSL strategy. The serial works by Zhao et al. mainly design an SSL strategy by generating paired LR-HR patches from the HR axial (X-Y) slices to train a super resolution (SR) network, which is then used to perform SR task on the coronal (X-Z) and sagittal (Y-Z) slices, respectively, to get an HR MR image. While our TSCNet proposes a totally different two-stage SSL strategy for self SR (SSR), which learns to interpolate the intermediate slice from adjacent slices. In the first stage, paired training data are generated by downsampling LR images along both X-axis and Y-axis to pretrain the interpolation network. While in the second stage, another SSL task is developed based on triplet X-Y slices to refine the network with the cycle-consistency constraint. Finally, we conduct interpolation along the Z-axis of LR images with the well-trained network to get HR images. Thus, the contextual information of all three directions is fully used as training guidance, and rich axial contexts are exploited to interpolate realistic X-Y slices with fewer artifacts.

The proposed cycle-consistency constraint is really different from other cycle-consistency ideas. As a typical cycle-consistency method, CycGAN depends on two different data distributions and learns the forward and backward mappings between them. While in our work, we observe that two well interpolated intermediate slices from the triplet X-Y slices can be reversely used to accurately estimate the original central slice. The cycle-consistency constraint is thus proposed to enforce the similarity between the estimated central slice and the original one. It focuses to learn the specific distribution of X-Y slices for better interpolation and optimizes the same interpolation mapping throughout the cyclic procedure. With this constraint, the network can address the domain shift issue and interpolate more realistic X-Y slices.

The serial works of Zhao et al. are published at ISBI2018, MICCAI2018 and TMI2021. We cited the paper in ISBI2018 in our paper as EDSSR. MICCAI2018 addresses SSR and self anti-aliasing (SAA) tasks in steps with two independent networks, which is different from our purpose, and is thus not cited. TMI2021 summarizes the previous two works, which proposes two versions of SMORE (3D) for SSR based on ISBI2018, and a SMORE (2D) for both SSR and SAA based on MICCAI2018. However, it was published in vol. 40, Mar. 2021 after the submission deadline of MICCAI2021. In fact, EDSSR is reused in TMI2021 as the Simplified SMORE (3D), while another SMORE (3D) only adds a data augment technique. Our experimental results show that TSCNet outperforms EDSSR (Simplified SMORE (3D)) without any data augment operation, suggesting its SOTA performance on the single SSR task. The rebuttal to Review #1 and Review #2 about using axial (X-Y) slices:

Although it is feasible to downsample along Z-axis for training data, more samples can be generated from X-axis and Y-axis with better quality. Because the X-Y slices have large between-slice spacing in Z-axis, the paired LR-HR data simulated from this direction will have severe structural discrepancy between the input adjacent slices. They easily misguide the network for coarse interpolation. Thus, we do not conduct the Z-axis downsampling in each stage. Besides, if we add the cycle-consistency constraint in the first stage, the discriminator cannot be well converged without the pretraining on the interpolation network. It is because the quality of interpolated slices is initially quite bad. Thus, we only utilize X-Z and Y-Z slices in the first stage to learn a preliminary but effective interpolation mapping, and add the cycle-consistency constraint in the second stage for refinement.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The proposed method bares many conceptual similarities to previously published work by Zhao et al, as noted by one of the reviewers. Nevertheless, it has some novel elements. In the rebuttal, it is claimed that one of the primary papers was not available because it was published in March of this year. This date referred to the print publication, however. It was electronically published in November of last year (stated on the website). Furthermore, in the rebuttal, the authors fail to acknowledge two other papers from the same group:

ISMORE: An iterative self super-resolution algorithm C Zhao, S Son, Y Kim, JL Prince International Workshop on Simulation and Synthesis in Medical Imaging, 130-139

Applications of a deep learning method for anti-aliasing and super-resolution in MRI C Zhao, M Shao, A Carass, H Li, BE Dewey, LM Ellingsen, J Woo, … Magnetic resonance imaging 64, 132-141

The SASHIIMI paper, in particular, appears to address some of the weaknesses of the EDSSR approach. Not every competing method needs to have a head to head comparison, but important literature needs to be cited and hopefully discussed. Given that relevant literature continues to be overlooked, and that the current evaluation section had weaknesses (no statistical significance, downsampling limited to only 2x, subtle improvements visually), I suggest that this paper be rejected.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

12

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper tackles the problem of interpolating MR slices. The authors propose a two-stage solution. In first stage, they follow Zhao et al. to pre-train the interpolation network. In second stage, they form slice triplet, use the pre-trained network, and adopt cycle-consistency constraint to solve the problem. R1 challenged the novelty of this paper, in comparison to three early papers (ISBI, MICCAI, and TMI recently) from the same group. I think the novelty of this paper mostly comes from the second stage, particularly in the way of combining slice triple with cycle-consistency. In rebuttal the authors also made argument upon the relationship of the mentioned literature. In general, this paper has enough novelty, while core clarification in rebuttal should be incorporated in the final paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This article presents a novel self-supervised super-resolution method that relies on a two-stages approaches including cycle-consistency. One of the major concerns raised by R1 and meta-reviewer was the “novelty” and the referred work. Authors justify well their contribution as regards the SOA methods, specifically about some recent papers. Now these papers (Zhao et al.) and the above referred explanation could be included in the final version of the paper. Unfortunately, other important aspects such as the scaling factor for downsampling and statistical significance of the resutls/improvement are though not really addressed, so it is still unclear if the contribution of this novel SS-SR model has an important impact. Overall, results are slightly better and there is a contribution related to a novel architecture so I would recommend the paper to be accepted.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

back to top

Two-Stage Self-Supervised Cycle-Consistency Network for Reconstruction of Thin-Slice MR Images