Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Sungho Suh, Sojeong Cheon, Dong-Jin Chang, Deukhee Lee, Yong Oh Lee

Abstract

Synthetic CT images are used in data augmentation methods to tackle small and fragmented training datasets in medical imaging. Three-dimensional conditional generative adversarial networks generate lung nodule synthesis, controlling malignancy and benignancy. However, the synthesis still has limitations, such as spatial discontinuity, background changes, and vast computational cost. We propose a novel CT generation model using attribute-guided generative adversarial networks. The proposed model can generate 2D synthetic slices sequentially with U-Net architecture and bi-directional convolutional long short-term memory for nodule reconstruction and injection. Nodule feature information is considered as input in the latent space in U-Net to generate targeted synthetic nodules. The benchmark with LIDC-IDRI dataset showed that the lung nodule synthesis quality is comparable to 3D generative models in the Visual Turing test with lower computation costs.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_39

SharedIt: https://rdcu.be/cyhVm

Link to the code repository

https://github.com/SojeongCheon/LSTM_GAN

Link to the dataset(s)

https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI

Reviews

Review #1

Please describe the contribution of the paper

The paper proposed a 2D UNet plus Conv-LSTM based module to achieve lung module synthesis. The model is able to maintain the spatial consistency as well as the control of the nodule attribute given the nodule-related features.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The use of nodule mask and the way of nodule reconstruction and nodule injection is relatively novel. The method is evaluated on a large dataset, and shown to have better synthesis results than the comparing methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The evaluation is weak. The paper didn’t compare some widely used image quality metrics like psnr, ssim etc. The only quantitative evaluation is the reader study. For different methods, the readers score different VOI and the F1-score can’t show the superiority of the proposed method. Test 2 only did only on the proposed method, thus can’t indicate the proposed method can generate nodules that corresponding to the attributes.
2. No ablation study to show the contribution of Conv-LSTM, and the nodule injection.
3. The writing of the method is unclear. Notations are not well defined which makes the paper hard to read.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Public dataset and code.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Add image quality metrcis for the nodule area.
2. Reader Test 1, use same VOIs for all methods.
3. Reader Test 2, also read other methods.
4. Abalation study on Conv-LSTM and nodule injection.
5. Fig 3, compare proposed method with T-GAN using same image.
6. Clearly define the notations at the beginning of the method section and use the same notation for fig.1.
7. Better explain image x, y. Are they from two separate volume, just one with and one without module? The current description is confusing.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Should add more convicing results.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper presented an attribute-guided generative adversarial network-based framework for lung nodule synthesis. Specifically, they combined U-Net architecture and bi-directional convolution long short-term memory to build the generator for both nodule reconstruction and injection.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- the idea of lung nodule synthesis using different attribute setting is interesting。
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The significance of task and the value to clinical applications is not clear.
- The motivation for applying bi-CLSTM is not explained.
- Ablation study is missing to verify the effectiveness of Bi-CLSTM in the proposed method.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

It’s possible to repoduce the result in the paper. no code is provided.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. How does this method ensure the consistency across different slices and for different views?
2. How to get the nodule location and mask? Does the mask affect the reconstruction results, e.g., mask shape and mask size?
3. What is the inference time for different methods in Table 1 respectively?
4. Why the dimension of the nodule feature vector is set 12?
5. There are a lot of losses in Eq (1) and (2) for training generator and discriminator. Please give experiments and discussion to explain the effectiveness of each loss.
6. I hope the authors could introduce clearly what is the input for the generator, masked nodule image w/ or w/o mask?
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although the point of synthesizing different lung nodule image using attribute-guided generative adversarial network is interesting, I have some confusion about the motivation and real application of the proposed method. There are also some problems in the experiment setting.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper proposed a lung nodule synthesis method, which could control the reality of the generated nodules in the images with or without nodules. Interestingly, the method is able to synthesize new nodules according to the malignancy scores.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The strength of the paper is the novel generation method for synthesizing diverse and varying malignancy lung nodules.
2. This may be helpful for solving lung nodule-related studies using deep learning. 3. The visual Turing tests are significant in highlighting the confidence of the proposed method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In Fig. 1, the inputs and outputs for the U-Net and the discriminator should be depicted clearly in the figure.
2. In Section 3.1, the sentence ‘a random number between 3 and 30 mm in the case of y ’ is confused, does it mean the VOIs are with different size in the $d$ dimension and how to train the model using mini-batch training (needs VOIs are with the same sizes)?
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The total framework can be reproduced, however, the complicated loss functions and their coefficients may be difficult to tune when the method is applied in other datasets. The authors should provide more details on the $lambda$s in the equations as reference.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It is better to refine the Fig. 1 in following aspects: (1) the inputs, outputs and labels should be depicted clearly such as using arrows. (2) where the losses used during training should be given in the figure.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The method contains a simple framework, and the proposed losses for preserving structure and reducing artifacts can be verified in the experiments. The experimental results on malignancy score specified visualizations and Turing tests are expressive.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

6
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This submission proposes an attribute-guided neural network to synthesize sequential lung nodules based on generative adversarial networks. All reviewers have positive comments on the novelty of the proposed method, but they have concerns about the motivation of the proposed method, and the experiments, like the method evaluation, the ablation study, etc. Please address these concerns in the rebuttal letter and other questions raised by the reviewers if space allows.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Author Feedback

Thank you for your valuable comments and corrections. We appreciate that all reviewers have positive comments on the novelty of the proposed method. Here, we answer your critical reviews pointing out that the motivation should be improved, and the experiments need to be supplemented such as the method of evaluation and the ablation studies. As mentioned in the introduction section of the manuscript, medical image datasets with detailed annotation are limited to machine learning society due to the laborious labeling process by the trained expertise and legal issues of sharing publicly private medical information. Even worse, among professional experts, the accuracy of the data annotation can have large inter-and intra- observer variability. Recently, generative adversarial networks (GAN) can be used as a data generation method for training purposes of various tasks, such as medical image segmentation and tumor/nodule classification. However, the preliminary works have limited controllability of the synthetic nodules and require high computational costs due to the 3D U-Net-based network architecture. To overcome the limitations, we proposed a sequential lung nodule synthesis method using attribute-guided GAN. The proposed method has the capability of controlling the semantic features of the synthesized nodules by adding nodule information into the latent space of the proposed U-Net-based networks. The proposed generator was performed by feeding three adjacent slices as input. By learning the spatial and temporal relationships from the image and its two adjacent ones, the limitation of large memory and whole volume requirement in the convolutional 3D U-Net architecture can be released. Next, the reviewers pointed out the proposed method has to be evaluated with widely used image quality metrics such as PSNR and SSIM. The suggestion would be helpful to show the superiority of the proposed method. However, we are concerned that the image quality metrics are measurable only in nodule reconstruction because the ground truth for the synthetic nodule is given. In addition, we should consider the fair comparison about the change of background image. MCGAN, an in-painting method has an advantage in the similarity between the reconstructed image and ground truth, because the proposed method and T-GAN have a tendency in background image changes. However, radiologists indicated that the in-painting method still left the borderline that causes the discontinuity of the background. We will investigate a better way to show the efficiency of the proposed method. For the ablation study to verify the effectiveness of the Bi-CLSTM, we can test two additional architectures under the same training conditions. The first architecture is to change the number of input slices from 3 to 1. This change can show the effectiveness of the sequence of slices with context because the Bi-CLSTM blocks do not preserve the Spatio-temporal correlation of CT slices. The second architecture is to remove the first Bi-CLSTM block in the proposed architecture. If we remove both Bi-CLSTM blocks, the input and output of the proposed method should be changed and it is the same as 2D U-Net architecture. These additional experiments will show the effects of the proposed method in lung nodule injection and reconstruction to CT image. For the camera-ready paper, we will define the notations clearly and describe the input and output dimensions for the generator and discriminator. Furthermore, the source code of the proposed method will be available on the Github page. It would be helpful to reproduce. Once again, thank you for your review.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Most concerns have been addressed in the rebuttal period. In the final version, the authors are encouraged to make updates as stated in the rebuttal letter.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Reviewers’ major concern is the lack of ablation study to justify the effectiveness of some of the proposed modules including Bi-LSTM. Authors responded that they are going to add additional experiments to address this. Unfortunately, this is considered a major change for the final version, which is prohibited based on the MICCAI guidelines. Another major concern is regarding the metrics used for evaluation. Unfortunately, authors’s response comes short in addressing this concern and as a result, the proposed work lacks appropriate and comprehensive evaluation to validate the proposed approach.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Reviewers recognized the overall formulation and method. Major question is on evaluation in supporting the claims, especially on attributes correspondence, and ablation. In rebuttal, authors clarified the motivation of using LSTM, which makes sense. Also, authors explained why metrics like PSNR and SSIM are not included, which I do agree given my previous experience in the same topic. Therefore, in my opinion authors made a good rebuttal to the raised questions, and the overall quality is sufficient for MICCAI.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

back to top

Sequential Lung Nodule Synthesis using Attribute-guided Generative Adversarial Networks