Paper Info Reviews Meta-review Author Feedback Post-Rebuttal Meta-reviews

# Authors

Sungho Suh, Sojeong Cheon, Dong-Jin Chang, Deukhee Lee, Yong Oh Lee

# Abstract

Synthetic CT images are used in data augmentation methods to tackle small and fragmented training datasets in medical imaging. Three-dimensional conditional generative adversarial networks generate lung nodule synthesis, controlling malignancy and benignancy. However, the synthesis still has limitations, such as spatial discontinuity, background changes, and vast computational cost. We propose a novel CT generation model using attribute-guided generative adversarial networks. The proposed model can generate 2D synthetic slices sequentially with U-Net architecture and bi-directional convolutional long short-term memory for nodule reconstruction and injection. Nodule feature information is considered as input in the latent space in U-Net to generate targeted synthetic nodules. The benchmark with LIDC-IDRI dataset showed that the lung nodule synthesis quality is comparable to 3D generative models in the Visual Turing test with lower computation costs.

SharedIt: https://rdcu.be/cyhVm

# Reviews

### Review #1

• Please describe the contribution of the paper

The paper proposed a 2D UNet plus Conv-LSTM based module to achieve lung module synthesis. The model is able to maintain the spatial consistency as well as the control of the nodule attribute given the nodule-related features.

• Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The use of nodule mask and the way of nodule reconstruction and nodule injection is relatively novel. The method is evaluated on a large dataset, and shown to have better synthesis results than the comparing methods.

• Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The evaluation is weak. The paper didn’t compare some widely used image quality metrics like psnr, ssim etc. The only quantitative evaluation is the reader study. For different methods, the readers score different VOI and the F1-score can’t show the superiority of the proposed method. Test 2 only did only on the proposed method, thus can’t indicate the proposed method can generate nodules that corresponding to the attributes.
2. No ablation study to show the contribution of Conv-LSTM, and the nodule injection.
3. The writing of the method is unclear. Notations are not well defined which makes the paper hard to read.
• Please rate the clarity and organization of this paper

Satisfactory

• Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Public dataset and code.

• Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Add image quality metrcis for the nodule area.
2. Reader Test 1, use same VOIs for all methods.
4. Abalation study on Conv-LSTM and nodule injection.
5. Fig 3, compare proposed method with T-GAN using same image.
6. Clearly define the notations at the beginning of the method section and use the same notation for fig.1.
7. Better explain image x, y. Are they from two separate volume, just one with and one without module? The current description is confusing.

borderline reject (5)

• Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

• What is the ranking of this paper in your review stack?

1

• Number of papers in your stack

5

• Reviewer confidence

Very confident

### Review #2

• Please describe the contribution of the paper

This paper presented an attribute-guided generative adversarial network-based framework for lung nodule synthesis. Specifically, they combined U-Net architecture and bi-directional convolution long short-term memory to build the generator for both nodule reconstruction and injection.

• Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
• the idea of lung nodule synthesis using different attribute setting is interesting。
• Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
• The significance of task and the value to clinical applications is not clear.
• The motivation for applying bi-CLSTM is not explained.
• Ablation study is missing to verify the effectiveness of Bi-CLSTM in the proposed method.
• Please rate the clarity and organization of this paper

Satisfactory

• Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It’s possible to repoduce the result in the paper. no code is provided.

• Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. How does this method ensure the consistency across different slices and for different views?
3. What is the inference time for different methods in Table 1 respectively?
4. Why the dimension of the nodule feature vector is set 12?
5. There are a lot of losses in Eq (1) and (2) for training generator and discriminator. Please give experiments and discussion to explain the effectiveness of each loss.
6. I hope the authors could introduce clearly what is the input for the generator, masked nodule image w/ or w/o mask?

borderline accept (6)

• Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although the point of synthesizing different lung nodule image using attribute-guided generative adversarial network is interesting, I have some confusion about the motivation and real application of the proposed method. There are also some problems in the experiment setting.

• What is the ranking of this paper in your review stack?

2

• Number of papers in your stack

5

• Reviewer confidence

Very confident

### Review #3

• Please describe the contribution of the paper

This paper proposed a lung nodule synthesis method, which could control the reality of the generated nodules in the images with or without nodules. Interestingly, the method is able to synthesize new nodules according to the malignancy scores.

• Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The strength of the paper is the novel generation method for synthesizing diverse and varying malignancy lung nodules.
2. This may be helpful for solving lung nodule-related studies using deep learning. 3. The visual Turing tests are significant in highlighting the confidence of the proposed method.
• Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In Fig. 1, the inputs and outputs for the U-Net and the discriminator should be depicted clearly in the figure.
2. In Section 3.1, the sentence ‘a random number between 3 and 30 mm in the case of y ’ is confused, does it mean the VOIs are with different size in the $d$ dimension and how to train the model using mini-batch training (needs VOIs are with the same sizes)?
• Please rate the clarity and organization of this paper

Good

• Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The total framework can be reproduced, however, the complicated loss functions and their coefficients may be difficult to tune when the method is applied in other datasets. The authors should provide more details on the $lambda$s in the equations as reference.

• Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It is better to refine the Fig. 1 in following aspects: (1) the inputs, outputs and labels should be depicted clearly such as using arrows. (2) where the losses used during training should be given in the figure.

accept (8)

• Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method contains a simple framework, and the proposed losses for preserving structure and reducing artifacts can be verified in the experiments. The experimental results on malignancy score specified visualizations and Turing tests are expressive.

• What is the ranking of this paper in your review stack?

1

• Number of papers in your stack

6

• Reviewer confidence

Confident but not absolutely certain

# Primary Meta-Review

• Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This submission proposes an attribute-guided neural network to synthesize sequential lung nodules based on generative adversarial networks. All reviewers have positive comments on the novelty of the proposed method, but they have concerns about the motivation of the proposed method, and the experiments, like the method evaluation, the ablation study, etc. Please address these concerns in the rebuttal letter and other questions raised by the reviewers if space allows.

• What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

# Post-rebuttal Meta-Reviews

## Meta-review # 1 (Primary)

• Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Most concerns have been addressed in the rebuttal period. In the final version, the authors are encouraged to make updates as stated in the rebuttal letter.

• After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept

• What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

## Meta-review #2

• Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Reviewers’ major concern is the lack of ablation study to justify the effectiveness of some of the proposed modules including Bi-LSTM. Authors responded that they are going to add additional experiments to address this. Unfortunately, this is considered a major change for the final version, which is prohibited based on the MICCAI guidelines. Another major concern is regarding the metrics used for evaluation. Unfortunately, authors’s response comes short in addressing this concern and as a result, the proposed work lacks appropriate and comprehensive evaluation to validate the proposed approach.

• After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject

• What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

## Meta-review #3

• Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Reviewers recognized the overall formulation and method. Major question is on evaluation in supporting the claims, especially on attributes correspondence, and ablation. In rebuttal, authors clarified the motivation of using LSTM, which makes sense. Also, authors explained why metrics like PSNR and SSIM are not included, which I do agree given my previous experience in the same topic. Therefore, in my opinion authors made a good rebuttal to the raised questions, and the overall quality is sufficient for MICCAI.

• After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept

• What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4