Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Yanyun Jiang, Yuanjie Zheng, Weikuan Jia, Sutao Song, Yanhui Ding

Abstract

Contrast-enhanced spectral mammography (CESM) is a valuable tool in the diagnosis and staging of primary breast cancer, for which it has an extremely high predictive value. However, the iodinated contrast media injected during CESM examination can cause adverse reactions, such as allergic reactions, and even cause contra-induced nephropathy. Therefore, iodinated contrast media cannot be used for some patients. To address this problem, we develop a cGAN-based Synthesis Network (cGSNT) that uses mammogram images to synthesize corresponding CESM images. Key points of this study are that cGSNT utilizes the cycle-consistent approach to reduce information loss when converting from high to low tissue contrast images, and the introduction of concatenation layers for dual-view information fusion. The experimental results on paired images of low-energy CESM (mammogram) and recombined CESM demonstrate that the synthetic image is very similar to the real CESM image, and the proposed cGSNT qualitatively and quantitatively outperforms typical non-learning and other popular deep learning methods.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_7

SharedIt: https://rdcu.be/cyl72

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

Contrast-enhanced spectral mammography (CESM) presents high-predictive value in breast cancer. This paper tackles the problem of breast cancer detection when recombined CESM images are not available due to a variety of reasons (e.g., due to allergic reactions of the iodinated contrast). Their solution involves synthesing the recombined CESM images from the low energy mammogram and represents the core of this work.

The authors make use of paired training data of low energy and recombined CESM images (the CESM acquisition produces both) to train a CycleGAN-type of architecture. The implementation is based on DRIT [12] which disentangles content and attribute features from the input images. They introduce a dual view approach to combine a craniocaudal and a medio-lateral oblique views.

They show improvement of the presented method over the competitors in different metrics: MAE, PSNR and SSIM.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors develop a framework based on Cycle GAN with disentangled representations. This approach is the first to synthesize CESM at full resolution in once, since previous attemps use a sliding window across the image [7, 8].
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
My main concerns are with the evaluation:
- Train/validation/test split on the image level instead of the subject level.
- Evaluation performed in a single split of the dataset.
- The strategy to select competitor methods’ hyperparameters is not explained.
Other concerncs regarding the method:
- Some design choices should be better explained/justified.
- The content discriminator training is not explained
Finally, related to the text, the manuscript is not easy to digest and some sentences are a bit hard to follow due to their construction, length or repetitive words.
Please rate the clarity and organization of this paper

Poor
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors will make the code publicly available according to the checklist. Some of the training parameters are written in the text, while others are not made explicit (e.g., learning rate, memory and time requirements, optimizer, number of epochs). But they will probably be available form the code itself. Moreover, the parameters for their competitor models are not available.

To the best knowledge of the reviewer, the dataset is private.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

In my opinion, the methodological part needs better clarification: 1.- Need to explain how the content discriminator works. It’s not clear whether this discriminator is a discriminative (e.g., discriminate view) or an adversarial loss (e.g., such that the content from both views looks similar). 2.- The claim that the content encoder captures structural information and the attribute encoder captures tumor related features is too strong if not shown in the experiments. It would be good to provide some references and/or see the philosophical rationale behind that. 3.- The encoder uses pre-trained weights but it’s not clear whether in the generative model those weights are fine-tune it or not.

Also, more details are needed regarding the training: 1.- One of the requirements is that LE and recombined CSM images are aligned. The registration method may be needed for reproducibility issues. 2.- The authors need to explicitly clarify if the autoencoder is only pretrained using training data or the whole dataset. 3.- It is not clear if all the competitor methods are optimized for the task at hand or uses default values for the hyperparameters. Training of those methods is also valuable to interpret the results. 4.- The role of training and test set is clear. However, the role of the validation set is not explained. The authors should clarify how the validation set was used (e.g., to tune hyperparameters such as the weight losses or early stopping) for completeness.

In the evaluation, the authors show that their method improve the perfomance of other state-of-the-art methods. However, interpreation of those results may be hampered by the some experiment design choices and/or lack of information: 1.- According to the test, the split between train/validation/test is made at the image level and not subject. I think that good practice will split at the subjet level to avoid bias on the metrics reported (e.g., see [8, Sec IV.A.1]) 2.- The evaluation is performed in a single split of the dataset. Different splits may be needed for better generalization assessment. 3.- The qualitative examples shown look great. However, it is not clear whether they come from the training/validation/test sets 4.- The configuration of competitor methods is not explained. Some of the key factors (e.g. hyperparameter tuning) should be stated.

Finally, some minor comments related to the text: 1.- Language is not very clear and some sentences are hard to follow. For example “We performed image synthesis tasks to utilize the proposed model” or “we use the cycle-consistent constraint that argues that the learned mapping functions should be able to bring an image back to its original image”, among others. Hence, the text should be revised properly. 2.- Please, check the quotation marks throughout the text (“”). 3.- In 2.2, second paragraph, line 4, E^a_{CC} should be E^c_{CC} 3.- In 2.2, third paragraph, line 7, “an” should be “a”
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

My recommendation is based in my concerns regarding the evaluations as well as the poor justification/clarification of some of the design and training choices. Moreover, the potential clinical impact is devised but not discussed/shown.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

CESM is often thought of as a poor man’s breast MRI. To perform CESM, imaging is performed after injection with dual-energy digital mammography, which helps provide a low-energy image and a recombined or iodine image that depict enhancing lesions in the breast. Using an enriched dataset of low-energy and recombined CESM breast images, the authors developed a cGAN-based Synthesis Network (cGSNT) that uses mammogram images to synthesize corresponding CESM images. Their cGSNT also reduces the information loss during the image synthesis process, and produced relatively high quality CESM images, without the use of intravenous contrast.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The underlying hypothesis is the raw data acquired from the low energy CESM (essentially a digital mammogram) contains sufficient information to define and differentiate the various soft tissue components that comprise the breast parenchyma. The cGANS accentuated the subtleties between these soft tissue components in order to simulate the combined without contrast agents.

Simulation of CESMs without the need to inject contrast agents is very novel

Table 1 show impressive performance compared to other similar methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

• All three datasets were small: training, validation, and testing; each contained 240, 60, and 92 image sets, respectively. • Authors could have provided more information about the generator network and the about the fusion layer. • Authors do not comment on what type of artifacts were introduced.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper provides moderate details about the models/algorithms, datasets, and evaluation process. I don’t believe it is sufficient to allow for other groups to reproduce this work. Another abstract with a very small dataset.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Many researchers use GAN for data augmentation and there have been recent papers that use CycleGAN to transform contrast CT images into non-contrast images. More recently, CycleGAN was used to transform noncontrast CTA images into contrast CTA images to evaluate thrombus vs lumen in abdominal aortic aneurysms. Body CT uses a wide hounsefield units. For CESM, I think it’s important to assess performance with different types of breast densities.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

CESM is emerging as a viable alternative to contrast-enhanced breast MRI. It may increase access to vascular imaging while reducing examination cost compared to breast MRI. For all CT and MRI exams, intravenous iodinated contrast materials are used to enhance the visualization of tumor neovascularity.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper describes (CycleGAN inspired) network to synthesize contrast-enhanced spectral mammography (CESM) bases upon low-energy images (mammograms). The results (synthesized recombined CESM images) are compared to the actual recombined CESM images based upon data acquired.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strength of this paper is in its potential application for breast screening.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Insufficient details in Results section
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper is appropriately referencing where necessary and is introducing sufficient details describing the core of the method. Therefore, the reproducibility is ensured
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The paper presents interesting results potentially very relevant for breast cancer screening. The paper would greatly benefit from quantitative evaluation of the results. Currently the results are just presented as few examples in a figure. Summary of the results is extremely short and does not allow appropriate understanding. The acronyms are not properly introduced.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper lacks essential descriptions in Results section
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths: *This approach is the first to synthesize CESM at full resolution in once, since previous attemps use a sliding window across the image *Simulation of CESMs without the need to inject contrast agents is very novel *The main strength of this paper is in its potential application for breast screening.

Weaknesses:
- clarity could be improved.
Overall:
- innovation and impact were high
- please see reviewer comments to improve result clarity
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

We sincerely thank the reviewers for the positive and detailed review comments as well as the suggestions for improvement. We will address the points listed below in the final version of the paper.

Methods: We will further supplement the training details of the encoder and content discriminator in the final version. The content encoder and the attribute encoder perform pre-training in the structure shown in Fig. 3(a), using the training dataset. The content discriminator is used to distinguish whether there are tumors in the content vector, and the training data contains the doctor’s diagnosis.

Implementation details: We will further explain the implementation details of the algorithm. In short, we implemented our neural network using PyTorch with four NVIDIA Tesla V100 GPUs with 32GB of video memory to train and evaluate the networks. The learning rates are initialized as 2e-4, followed by decreasing the learning rates by cosine annealing schedule during the training.

Results: We will supplement the visualization results in the final version of the paper. Due to the article’s length limitation, the results may be supplied in the supplementary material.

Some related works have shown that mammographic density is positively associated with the presence of breast cancer; that is, women with denser breasts are more likely to develop breast cancer than women with non-dense (fatty) breasts. e.g., Kim et al., Cancer 2020; Duffy et al., European Journal of Cancer 2017. It would be insightful to carry out the analysis of mammographic density. We will conduct other studies on mammogram density in future work.

Typos: We will fix all these incorrect texts and the acronyms in the updated version of the paper.

Acronyms: Mean absolute error (MAE); Peak signal-to-noise ratio (PSNR); Structural similarity index (SSIM).

back to top

Synthesis of Contrast-enhanced Spectral Mammograms from Low-energy Mammograms Using cGAN-Based Synthesis Network