Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Hajar Emami, Ming Dong, Siamak P. Nejad-Davarani, Carri K. Glide-Hurst

Abstract

In medical image synthesis, model training could be challenging due to the inconsistencies between images of dierent modalities even with the same patient, typically caused by internal status/tissue changes as dierent modalities are usually obtained at a dierent time. This paper proposes a novel deep learning method, Structure-aware Generative Adversarial Network (SA-GAN), that preserves the shapes and locations of in-consistent structures when generating medical images. SA-GAN is employed to generate synthetic computed tomography (synCT) images from magnetic resonance imaging (MRI) with two parallel streams: the global stream translates the input from the MRI to the CT domain while the local stream automatically segments the inconsistent organs, maintains their locations and shapes in MRI, and translates the organ intensities to CT. Through extensive experiments on a pelvic dataset, we demonstrate that SA-GAN provides clinically acceptable accuracy on both synCTs and organ segmentation and supports MR-only treatment planning in disease sites with internal organ status changes.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_46

SharedIt: https://rdcu.be/cyhV3

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed a novel structure-aware GAN (SA-GAN) model for the generation of syncCT from the corresponding MRI image. This model developed two parallel streams for the generator, in which the global stream translates the source domain MRI image to target domain CT image while the local stream automatically segments the inconsistent organs between these two different domains.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Technical contribution. While the proposed method builds on existing building blocks, I think it is sufficiently novel. In particular, they adopt image style transform to cross-modality mapping framework and address the MR-CT inconsistent issues.
- This paper is clearly written and easy to follow.
- Extensive comparison with chosen baseline
- Extensive ablation study demonstrating the advantage of the proposed contributions.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- lack of details about how the segmentation network is trained, what will happen to the final syncCT if the segmentation result is not accurate?
- not clear how the style transfer is trained?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

it’s possible to reproduce this work
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. how to get the mask for CT? is it similar to MRI mask that generated by a pretrained segmentation network?
2. Are these three style transfers of AdaON_B, AdaON_R, and AdaON_G pretrained? if so, how to pretrain them?
3. please give more detailed descriptions about the fusion layer. how to make sure the boundary between the outs of global and local streams consistent? Is there any failure case?
4. Is there any reason the authors choose this pytorch version 0.2? since it is Pytorch 1.8.1 now.
5. In the experiment section, the author mentioned that ‘SynCTs generated by the CNN, cGAN, and CycleGAN models showed larger errors in the bone regions.’ It’s better to point out these regions in the result in Fig. 3.
6. The idea of style transfer for dealing with the inconsistant regions is very interesting. Use the masked MRI as content and CT image as style, and use three style transfers to generate bladder, rectom and gas separately, However, is it possible that in some slices it has all three organs in MR slice while it has only 2 organs in CT slice. How to solve this problem?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed approach, while built on existing build blocks, is interesting. The experimental evaluation is appropriate in the ablation study but some details about the pretrained MR segmentation network and CT segmentation network is not clear.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The shapes and locations of moving structures like bladder and rectum are inconsistent in CT and MRI acquisition, which may hamper the training of neural networks synthesizing CT images from MRI towards MR-only treatment planning. To minimize the impact of such inconsistency, a structure-aware generative adversarial network synthesizing CT images from MRI was proposed in the paper with a global stream transferring image contrast and a carefully designed local stream controlling the shapes of moving structures.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The way of employing segmentation to fixed the inconsistent between MRI and CT is interesting in that potentially inconsistent areas treated separately with the rest of image.
2. In the local stream, styles of bladder/rectum/gas are transferred from MRI to CT individually with novel AdaOn modules.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The proposed dual-stream SA-GAN requires to manually define inconsistent structures between MRI and CT, while such inconsistence is very complex in real-world settings (such as registration and musculoskeletal system).
2. If the SPM aligns MRI and CT boundaries well, why large areas like bladder cannot be registered correctly? The bladder in CT and MRI seems to be well aligned In Fig. 12 of [21] after registration.
3. Comparisons with baselines are not fair enough, as the training of SA-GAN requires extra segmentation information which does not exist in the optimization of baseline methods.
4. Will there be contrast inconsistence between the edges of bladder/rectum/gas? As they are fused to the rest of the image by simply addition operation.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The proposed SA-GAN and experiment settings is almost clearly stated, except:
1. It would be better if authors could provide SPM registration parameters.
2. The description of AdaON and its difference between AdaIN is not clear enough (In figure, equation, flowchart, etc., not just words).
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Major problems:
1. Comparison is not fair enough. Comparing the SA-GAN with other segmentation-aware methods like cGAN with shape consistency loss used in Ge, Yunhao, et al. “Unpaired Mr to CT Synthesis with Explicit Structural Constrained Adversarial Learning.” 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019.
2. The evaluation process of SA-GAN is quite different from training as label_ct in equation (3) is not available. However, it is not described in this paper.
Minor problems:
1. What is the difference between S’_mr and S_mr in Fig. 2?
2. Why the background of CT image is gray in Fig. 3 while it is black in Fig. 1? Also, the contrast of CT in Fig. 2 is quite different from that in Fig. 3.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The motivation of structure-aware multi-modal synthesis is motivation is strong enough and almost well-written. Although experiments are not very fair, the proposed SA-GAN is interesting and it seems to be promising at MR-only radiation therapy planning as both MRI to CT synthesis and organ-at-risk segmentation are solved at the same time.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposes a strategy to generate SynCT images from corresponding MRI images. The authors proposes a two stream generative adversarial network to generate CT images. Experiments have been done using pelvic dataset to demonstrate the efficacy of the model.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper is well written, relevant details and related works have been discussed nicely. The ablation study section proves the need of the specific modules proposed in the architecture. The paper addresses the important inconsistency issues fro MRI to CT generation and tries to address that specifically.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The figures are not very clear and sometimes it is getting difficult to understand what is going on. The tables and figure captions are also not very descriptive. The paper does a comparison with CycleGAN which probably does not make much sense as CycleGAN does not see paired images and comparing this with a supervised model is probably not fair.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Upon release of the code, the reproducbility can be verified.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Thanks to the authors for addressing an important problem in MRI to CT generation. On the same light of my previous comments, it would be great if you can make the titles of the figures and tables more descriptive. For example, in figure please explain which region of the image readers should look at specifically to see the differences. If possible please zoom that part and add with the figure. Similar thing holds with tables too.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The relevant strength of the paper is discussed before whereas the weakness seems to be less and mostly minor. That leads to my recommendation.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper addresses MR to CT synthesis, especially concerning the changing locations and shapes of certain organs in pelvic data. The authors introduce shape masks into generation of the images. The paper is well received to all reviewers, though they also raise several questions regarding details in the paper. These issues need be incorporated in the final paper.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

-Highlight the regions in the results (R1: point out these regions in the result) (R3: The figures are not very clear) We will draw ROIs and highlight regions with differences for a more detailed comparison with other models in the final version.

-Fusion layer details (R1: how to make sure the boundaries consistent?) (R2: will there be contrast inconsistence between the edges): The boundaries are consistent as the same binary mask is used to generate the inconsistent regions and also to exclude regions from the output of the global stream. The outputs of two streams are combined in the fusion layer through element-wise addition. We observe intensity discontinuity along the boundary between the consistent and inconsistent regions in a few cases. Note that this discontinuity does not impact clinical applications such as dose calculation. While smoothing techniques can be employed to reduce discontinuity, it reduces fidelity of synCT at the same time.

-Training details (R1: segmentation and style transfer training): The reconstruction loss of the global stream and the structure segmentation loss (weighted multi-class cross-entropy) of the segmentation network are jointly minimized with the GAN loss through adversarial training in an end-to-end fashion. In addition, style transfer modules (randomly initialized, no pre-training) are trained separately for bladder, rectum and rectal gas by taking a masked CT style image and a masked MRI from an inconsistent region (the content image) as inputs and minimizing style and content losses.

-R1: what happens if the segmentation result is not accurate: The high DSC shows excellent ability of our segmentation network to localize inconsistent organs. That said, if the segmentation results are not accurate, it will lead to inaccurate regions in synCT.

-R1: how to get the CT masks: As a data pre-processing step, a single physician delineated bladder and rectum volumes on both the CTs and MRIs in the Eclipse Treatment Planning System. Rectal gas was identified by thresholding and applying morphological operators to fill in holes and remove isolated voxels (filling and opening operators, respectively).

-R1: three organs in MR while only 2 organs in CT In slices that have all three organs in MR while there are only 2 organs in CT, SA-GAN generates all three organs based on the input MRI masks because the clinical applications require that the desired synCT should preserve the location, shapes and volumes of the inconsistent organs as in MRI while accurately changing the corresponding image intensities to the real CT.

-R2: (Requires to manually define inconsistent structures),(CT masks in model evaluation) In the training, MRI and CT labels of inconsistent regions are used in Eq. 3 to calculate the modified reconstruction loss. After training, these labels are not required anymore for generating synCT. We used the MRI and CT masks in the test phase solely for the purpose of model evaluation. We will clarify.

-R2: registration with SPM Rigid registration is used as a pre-processing step. Co-registration was performed by using normalized mutual information as the cost-function and by trilinear interpolation. Literature shows that even deformable registration is highly limited in multi-modality workflows with large volume changes and introduces geometrical uncertainties in the pelvic region[26].

-R2: shape consistency loss, R3: comparison with CycleGAN We compared with CycleGAN as it is widely used in medical image generation applications. More importantly, CycleGAN is a potential solution to handle the organ inconsistencies as it does not require paired input images. We will add the discussion with shape consistency loss (Yunhao, et al., ISBI 2019) in the final version.

-R3: The tables and figure captions We will make the tables and figure captions more descriptive in the final version.

back to top

SA-GAN: Structure-Aware GAN for Organ-Preserving Synthetic CT Generation