Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Florian Thamm, Oliver Taubmann, Felix Denzinger, Markus Jürgens, Hendrik Ditt, Andreas Maier

Abstract

By injecting contrast agent during a CT acquisition, the vascular system can be enhanced. This acquisition type is known as CT Angiography (CTA). However, due to typically lower dose levels of CTA scans compared to non-contrast CT acquisitions (NCCT) and the employed reconstruction designed specifically for vessel reconstruction, soft tissue contrast in the brain parenchyma is usually subpar. Hence, an NCCT scan is preferred for the visualization of such tissue. We propose SyNCCT, an approach which synthesizes NCCT images from the CTA domain by removing enhanced vessel structures and improving soft tissue contrast. Contrary to virtual non-contrast (VNC) images based on dual energy scans, which target the physically accurate removal of iodine rather than generating a realistic NCCT with improved gray/white matter separation, our approach only requires a conventional single-energy acquisition. By design, our method integrates prior domain knowledge and employs residual learning as well as a discriminator to achieve perceptual realism. In our data set of patients with ischemic stroke, the absolute differences in automatic ASPECT scoring, which rates early signs of an occlusion in the anterior circulation on a scale from 0 (most severe) to 10 (no signs), was 0.78 +- 0.75 (median of 1) when comparing our SyNCCT to the real NCCT images. Qualitatively, realistic appearance of the images was confirmed by means of a Turing test with a radiologist, who classified 64% of 64 (32 real, 32 generated) images correctly. Two other physicians classified 65% correctly, on average.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_64

SharedIt: https://rdcu.be/cyl9a

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed an interesting application using GAN in ischemic stroke imaging, generating a NCCT from a CTA image. Initial results look promising even though the clinical values are still unknown.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper is generally of a great shape, and well written. Methodology is straight forward and reasonable.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

validations should be improved. more details on method is needed.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Cannot reproduce it without code and data.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. It is quite interesting to see a new application of using GAN
2. Fig. 1 is a little confusing. cannot understand the approach only relying on this figure.
3. It is not clear why VNC -EST is needed to train GAN if this estimation only maps CTA HU to CT HUS. DO you think the network cannot learn this nonlinear HU mapping?
4. Turing test showed some positive signals of fake NCCT. but not good enough. It is more interesting to see how many fake NCCT considered as real, or vice verse.
5. using automated ASPECTS software as a tool to evaluate the performance is not convincing . I do not think current ASPECTS software are accurate enough to be ground truth. It is better to ask radiologists to score ASPECTS on both fake and real NCCT. Lack of convincing results limits the clinical relevance of this paper
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Clinical relevance should be strengthen.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors propose an approach to derive non-contrast CT images from CTA. Three inputs are fed to a 2D Unet: the CTA, the segmentation of the vessels and a first estimate of the synthetic non-contrast CT obtained by applying a function that approximately maps CTA HU to non-contrast CT HU. A discriminator is used to distinguish fake from real non-contrast CTs. The approach is trained/validated on a dataset of 127 patients and tested on 23 patients, all with ischemic strokes. The evaluation consists of a qualitative analysis, an expert validation (Turing test) and a quantitative analysis. In the quantitative analysis, the image synthesis accuracy is evaluated using the generic SSIM and an application-specific score meant to assess infarction severity.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novel application: synthesis of non-contrast CT images from CTA
- Thorough evaluation: qualitative evaluation + expert evaluation using the Turing test + quantitative evaluation using both generic and application-specific metrics
- Paper clear and easy to follow
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Very sparse coverage of the related works, only one paper is mentioned. Even though the authors target a novel application, several approaches have been developed for contrast-enhanced to non-contrast (and vice-versa) or low-dose to high-dose image synthesis.
- The individual contributions of the different inputs have not been evaluated.
- Several methodological points could be better explained, for example how the function that approximately maps CTA HU to non-contrast CT HU was determined, or why predict the difference between CTA and non-contrast CT instead of the non-contrast CT directly.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have been honest when filling the reproductibilty checklist. The authors use a private dataset that cannot be shared but its description is reasonable for an 8-page paper. It seems that no code will be released but the method and implementation are quite well described. The reporting of the experimental results is also reasonable for an 8-page paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Suggestion of papers on non-contrast to contrast-enhanced or single- to dual-energy image synthesis:
  - Gong, E. et al. (2018). Deep learning enables reduced gadolinium dose for contrast-enhanced brain MRI. Journal of Magnetic Resonance Imaging, 48(2), 330–340. https://doi.org/10.1002/jmri.25970
  - Xu, C. et al. (2021). Synthesis of Gadolinium-enhanced Liver Tumors on Nonenhanced Liver MR Images Using Pixel-level Graph Reinforcement Learning. Medical Image Analysis, 69, 101976. https://doi.org/10.1016/j.media.2021.101976
  - Bône, A. et al. (2021). Contrast-enhanced brain MRI synthesis with deep learning: Key input modalities and asymptotic performance. International Symposium on Biomedical Imaging. https://hal.archives-ouvertes.fr/hal-03128023
  - Lee, D. et al (2019). Development of a deep neural network for generating synthetic dual-energy chest x-ray images with single x-ray exposure. Physics in Medicine & Biology, 64(11), 115017. https://doi.org/10.1088/1361-6560/ab1cee
  - Lyu, T. et al. (2021). Estimating dual-energy CT imaging from single-energy CT data with material decomposition convolutional neural network. Medical Image Analysis, 70, 102001. https://doi.org/10.1016/j.media.2021.102001
- The authors should explain how the function that approximately maps CTA HU to non-contrast CT HU was fitted exactly. Is this type of function usual?
- The authors should also explain why they predict the difference between CTA and non-contrast CT instead of the non-contrast CT directly.
- Adding an ablation study assessing the importance of each input would be beneficial.
- In Fig. 3, the authors could add arrows pointing at the contrast-enhanced vessels or “repetitive patterns” mentioned in the text as they are not all obvious to spot.
- It would be good to mention what was the element that the radiologist used to recognise fake images.
- None of the quantitative metrics really allows selecting the best approach while clear differences can be observed visually. This should be discussed.
- I am not sure that the evaluation based on the ASPECTS score really shows that “no artificial stroke patterns were added and existing signs of stroke were correctly carried over to the NCCT domain”. From what I understood, if in the real image lesions were observed in regions A, B and C, the ASPECT score would be 10 - 3 = 7. Then, if in the fake image lesions were observed in regions D, E and F, the ASPECT score would also be 7. The fact that existing lesions are not detected while non-existing lesions appeared would not be captured. I really appreciate the fact that an application-specific metric is used, but its limitations should clearly be stated.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The novel application and thorough evaluation compensate the incomplete literature review, lack of detail in the description of the approach and ambivalent results.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The goal of this paper was to be able to create NCCT images from CTA, using SyNCCT, which uses a single energy level. Their method uses a U-Net and generator-discriminator function.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Well written introduction. It introduced the problem, talked about the related work, and concluded with what was to be explained in the rest of the paper.
2. In the methods section everything is well written and seems reproducible. Equations used are fully explained.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

N/A
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1. The authors detail all the parameters of the U-Net used, causing this work to be highly reproducible. The data are not from an open source database
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Introduction
1. In the second to last sentence of the introduction, “…baseline methods…” was mentioned. It would be helpful to mention the baseline method that the results of will be compared to.
Methods
1. In the preprocessing section (the second sentence), you can delete the word “percentile” since the ‘%’ symbol is used.
2. Instead of using the title “Proposed Method” for section 2.3, maybe use or use something similar to “SyNCCT Method”.
3. Even though the HU is a unit of measurement, it should still be spelled out before using ‘HU’
4. Recommended: Also, in the Estimation section, it may be a good idea to skip the “bed of nails” analogy. It doesn’t really add anything to this section.
5. MSE, SSIM, and GAN need to be defined before using the acronym
Results
1. In the results section, brain extraction was mentioned, however, this was not mentioned in the ‘Methods’ section. If brain extraction did occur, as mentioned in the ‘Results’ section, then it should be mentioned in the methods section, presumably the preprocessing section.
2. Define ‘Dis’
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. Being able to construct a NCCT from a CTA seems as though it would improve the ability for radiologists to detect infarcts, which is clinically relevant.
2. The paper is well written and clearly laced out the details for this method, including giving relevant background that helps the reader understand the problem.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This work presents a novel application of image synthesis which has been thoroughly evaluated. However, the reviewers have highlighted specific points regarding aspects to be clarified in the methodology. Please address these clearly in the rebuttal. Space permitting, the authors are also encouraged to include the references on non-contrast to contrast-enhanced image synthesis to provide a better view of previous works on this topic.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

We highly appreciate the valuable comments and want to thank all reviewers and the MR for the overall positive assessment and the agreement on novelty, clarity, and thoroughness of evaluation of the proposed work. Main points to improve on, as summarized by the MR, were references for the inverted domain transfer (NCCT to CTA) and clarity of specific aspects of the methodology. Also, we want to comment on some minor points. References: We thank R2 for pointing out interesting work related to this topic. We agree discussing related domain transfer problems increases the quality of the paper. Hence, we extended our related work section in this regard considering space constraints. Methodology: Replying to a question by R2, the parameters of the function (Eq.1, VNC-Est.) were determined using a least-squares fit. Poirot et al. follow a similar idea using a LUT; we adapted this concept to a single-energy problem using Eq. 1. R1 inquired about its influence, which relates to R2’s suggestion on performing an ablation study to assess the benefits of the additional channels. We performed a quasi-ablation study since both baselines are simplifications of the proposed method in the most relevant aspects. Mod. Poirot uses a different architecture and has no additional segmentation channel. CycleGANs do not use Eq. 1 and neither a segmentation nor a residual connection. SyNCCT led to quantitatively and qualitatively superior performance within both (Non/GAN) categories, demonstrating the positive impact of the changes we propose. The ambivalence (mentioned by R2) of the quantitative and qualitative results appears only between GAN and Non-GAN methods. Due to the noisy nature of the images and slight registration inaccuracies, distance-based metrics do not properly represent the perception. Therefore, direct supervision led to smooth results, which is a common problem using MSE. As the GAN methods introduce high frequencies, the MAE becomes substantially different. We added this discussion to the paper. Doubts were expressed by R1 that current automated ASPECTS tools are not sufficiently accurate to be ground truth, which in R1’s opinion impacts the clinical significance of our work. Numerous clinical publications, such as Maegerlein et al. (doi.org/10.1148/radiol.2019181228), Li et al. (doi.org/10.1016/j.crad.2019.12.010 ) and Hoefler et al. (doi.org/10.1007/s00234-020-02439-3), have shown that the performance of such tools is comparable to that of a radiologist. The software we used is commercially available and approved for clinical use in the EU. We agree that expert assessments are extremely valuable, which is why we included a Turing test. However, it has to be considered that ASPECTS exhibits a high inter-reader variability, and a reasonable study design for our task would require different readers assessing the real and synthetic NCCTs to avoid bias. This decreases the statistical power of the comparison. Therefore, considering the number of cases available, we preferred to use an automated tool that yields standardized and reproducible results. To address R2’s concern that different sets of affected regions may be responsible for the same final ASPECT score, we added quantitative measures of the coverage between the regions from the SSIM+Dis configuration result with the NCCT regions and found a match of 88% in all regions (SSIM 83%, MSE 83%, MSE+Dis 85%, mod. Poirot 77%, CycleGAN 81%). Thus, the final method does not only perform well on the global ASPECTS score as shown in section 3, but also outperforms the ablations consistently on region-level. Minor points: R1 suggested to publish the FP-rates of the Turing test. Our original submission states the specificity of the experiment and refers to the supplementary material, in which we provide the confusion matrix to accommodate this as well as other requests of that nature. We also followed the suggestions by R3 and R2 (arrows) regarding linguistic improvements to the manuscript.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This work presents a novel application for synthesis of non-contrast CT images from CTA. It is clearly written and the ideas are easy to follow. The authors have well addressed the points raised by the reviewers and these should be easy to include in the paper without the need of an external review. In agreement with the reviewers, I consider this work presents a novel application that is worth of discussion at MICCAI.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This manuscript, as outlined by the reviewers, is very clear and provides an interesting and novel method for the synthesis of non-contrast images from CTA images using U-Nets. The points raised by the reviewers, were properly addressed by the authors in the rebuttal letter, and small improvements can be added to the manuscript based on such replies. The manuscript is in good shape and represents an interesting contribution to MICCAI conference.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have presented an interesting application of GAN for the synthesis of non-contrast CT images from CTA with adequate evaluation. The authors have given more details about the methodology and will extend the related work section. Most of the concerns have been addressed in the rebuttal.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

back to top

SyNCCT: Synthetic Non-Contrast Images of the Brain from Single-Energy Computed Tomography Angiography