Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Miika Toikkanen, Doyoung Kwon, Minho Lee
Abstract
Intracranial hemorrhage (ICH) is a dangerous condition of bleeding within the skull that calls for rapid and precise diagnosis due to potentially fatal consequences. In this paper, we propose Residual Segmentation with Generative Adversarial Networks (ReSGAN) to accurately localize the hemorrhage from computerized tomography (CT) scans with a GAN-based model. Although convolutional neural networks have shown success in the ICH segmentation task, precise localization remains challenging due to in-balance and scarcity of labeled training data. Synthetic samples from generative models, and aligned templates as reference from brain atlas have been demonstrated to alleviate the issues. We consider synthetic templates as another candidate and solve the problem by directly applying a generative model to segmentation. Our ReSGAN learns a distribution of pseudo-normal brain CT scans, that through residuals, reliably delineates the hemorrhaging areas. We perform experiments on two datasets and compare our model against a well established baseline, that consistently shows significant improvements, therefore demonstrating the validity of our novel method.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87193-2_38
SharedIt: https://rdcu.be/cyhMg
Link to the code repository
https://github.com/miikatoi/ReSGAN-CTICH
Link to the dataset(s)
https://physionet.org/content/ct-ich/1.3.1/
Reviews
Review #1
- Please describe the contribution of the paper
Precise localization of hemorrhage using CT data remains challenging due to in-balance and scarcity of labeled training data. In this paper, Residual Segmentation with Generative Adversarial Networks (ReSGAN) method is proposed to localize the hemorrhage from CT scans. This method learns a distribution of pseudo-normal brain CT scans, that through residuals, reliably delineates the hemorrhaging areas. Experimental results on two datasets shows significant improvement in localization compare to other methods.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper develops an algorithm by viewing normal, and abnormal brain tissue with hemorrhage as semantic classes and manipulates CT scans using CLADE. Further, the ICH delineation is obtained from residual images. Proposed method essentially learns the distribution of pseudo-normal brain CT scans that captures the difference between patient images and corresponding ground truth labels. By comparisons against baseline models, proposed method demonstrated improved segmentation performance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Main weakness of the paper is the lack of comparison with state of the art work in literature. I would suggest to mention novelty of proposed work clearly in introduction section along with presenting more recent method’s results in Table 1.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
One dataset is available online out of the two. Results can be reproduced upon publishing code online.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Major concern is the lack of rigorous comparison in the manuscript. Novelty of the work should be clearly mentioned in introduction section. Few more comments are as follows:
- More architecture detail in Fig.1 should be there in the manuscript such as number of nodes in each layer, convolution kernel sizes, stride sizes etc.
- Why the normal brain is called as pseudo normal? How the thresholding is being performed on the residuals?
- Since paper is based on combination of GauGAN with CLADE. Slight details of these methods should be there in supplementary.
- How the sementic labels being predicted using U-Net?
- How are the regularization parameters (lambda_vgg, lambda_DFM etc.) values being decided? -Grammatical errors and typos need to corrected throughout the manuscript.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Please refer to comments above. Comparison with more recent work would help the manuscript to stand better in terms of novelty of proposed pipeline.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
4
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
This paper addressed the intracranial hemorrhage segmentation problem using GAN based method. In the paper, the authors tried to reconstruct the normal brain mask and the mask with abnormality from the network, and the residual image is used to generate the hemorrhage-related mask. The performance of their method is proved to be better than U-Net by experiments.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The idea is interesting as it uses the brain mask as the guidance to generate synthetic normal brain image and compare with the brain with hemorrhage.
- The performance is good, especially for the SAH judging from the qualitative image.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- For the method with additional UNet, it is also interesting if the authors can experiment on cascaded UNet as one of the baseline. The comparison between ReSGAN with only one UNet seems to be unfair to some degree.
- The authors showed several results for SAH, but only one result from other types. All the hemorrhage seem to be large and obvious. It is hard to tell the real performance for other hemorrhage types, especially tiny ones like SDH and EDH.
- The qualitative result seems to be marginal compared with UNet. It seems with an additional post-processing step, some noisy mask can be removed from UNet and the performance of UNet should be better.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It should be reproducible if the original dataset can be public. It is also not clear why and how the authors removed several cases from the public CT-ICH dataset for their experiment.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Besides the items in the weaknesses, It is better for the authors to compare with other work on the same dataset.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The overall quality of the paper is good and the idea is interesting. However, the overall performance seems to be marginal and the comparison with UNet is somehow not fair. I am glad to change my opinion if the authors can provide more comparison results with original UNet.
- What is the ranking of this paper in your review stack?
3
- Number of papers in your stack
4
- Reviewer confidence
Confident but not absolutely certain
Review #3
- Please describe the contribution of the paper
The authors of “ReSGAN” proposed a novel approach Residual Segmentation with Generative Adversarial Networks (ReSGAN) to segment the hemorrhage from non-contrast CT images. The authors train and test on ~350 examinations of UH and CT-ICH datasets, show the results slight better than the U-Net segmentation results in both Dice score and Hausdorff distance.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper provided a novel formulation about ReSGAN and demonstrated it outperformed the baseline model U-Net on the real datasets.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
If the authors could provide more why not comparing to other state-of-art GAN segmentation methods, it would make paper strong. And for the evaluations, the results are slight better than U-Net, if authors also provided the performance comparison in terms of time and space used, then we could learn whether it is practical useful in clinical feasibility.
The overall experiments in this paper are performed on 2D CT images. If the authors could discuss whether their proposed method can be easily extended to 3D or how it works on 3D image stack, it will make paper stronger.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The steps and parameters are provided in the paper for reproduced the approach.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Overall it looks good to me. If author add the performance comparison in terms of time and space used would be better.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The method is novel but the experiments and evaluations especially comparison is not solid enough to me.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
2
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The paper describes an approach to generate synthetic samples for ICH segmentation. The idea is interesting and has found acceptance amongst the reviewers. However there are certain issues that need further clarification such as motivation, comparison with SOTA, and details on space and time values. The authors are advised to focus on these and other important issues raised by reviewers
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
4
Author Feedback
We thank the reviewers for their constructive and thoughtful feedback. It is pleasing to see that our ideas were generally found to be novel and received positively. The main concern shared by all of the reviewers was the limited comparison to other methods. We address this and the individual concerns as needed here. For the final version of the paper, we will consider all the feedback as best as we can.
Additional Comparison (R1, R2, R3) Our work was motivated by the usage of healthy templates in the Siamese U-Net, Therefore we compared to the baselines from the Siamese U-Net paper. However, we acknowledge that this may be unconvincing, especially since our approach is quite different from the aforementioned models. To this end, as suggested by R2, we implemented a cascaded U-Net to provide additional comparison. Further, we also report the time and space values as recommended by R3. Below we have compiled the dice score (DSC), hausdorff distance (HSD), number of model parameters (p) and the average forward pass time (t) for inference of a single CT-scan slice, for each model on the UH dataset. Even when we include the more fair comparison against the Cascaded U-Net, our model produces far better segmentations. Furthermore, our variant without the learned post-processing is very fast to compute.
Model U-Net Cascaded U-Net Siamese U-Net Ours Ours (learned post-processing) DSC 0.609 0.645 0.661 0.696 0.726 HSD 86.795 53.47 32.473 34.201 25.490 p 1.93M 4.64M 1.93M 4.11M 6.04M t (ms) 8.08 11.20 9.62 6.86 14.94 R1 questions “Why the normal brain is called as pseudo normal? “ In the first paragraph of section 2, we define the templates pseudo-normal, because they cannot consider the structural deformation caused by ICH and therefore should be distinguished from the normal brain.
“How the thresholding is being performed on the residuals?” In the thresholding-paragraph of section 2.2, we explain that a binary thresholding is used, and in the first paragraph of section 3, we specify that the optimal threshold value is found by a coarse-to-fine search on the validation split.
“How the sementic labels being predicted using U-Net?” The semantic critic S takes the same input as the GAN model, but predicts a probability map for each semantic class. The difference to a segmentation network is that this output is only used to provide supervision for the GAN during training.
“How are the regularization parameters (lambda_vgg, lambda_DFM etc.) values being decided?” Lambda_VGG and Lambda_DFM are directly from the official SPADE implementation, but Lambda_CE was found to be sufficient with a coarse search of candidate values. We don’t guarantee these to be optimal.
R2 questions “The authors showed several results for SAH, but only one result from other types. All the hemorrhages seem to be large and obvious. It is hard to tell the real performance for other hemorrhage types, especially tiny ones like SDH and EDH.” We agree. The UH dataset mostly contains SAH (more than 80%) and the samples for the qualitative comparison were not selected with any specific type in mind as we focused on predicting two classes, normal and abnormal. If possible, we will include additional qualitative comparisons from the CT-ICH dataset including SDH and EDH to the supplementary materials.
“The qualitative result seems to be marginal compared with UNet. It seems with an additional post-processing step, some noisy mask can be removed from UNet and the performance of UNet should be better.” Morphological opening has already been applied to the U-Net and the Siamese U-Net output.
“It is also not clear why and how the authors removed several cases from the public CT-ICH dataset for their experiment.” We used all the samples available at https://physionet.org/content/ct-ich/1.3.1/, no cases were removed. Please note that the authors of the dataset only made 75 out of the 82 scans public.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
In the rebuttal the authors have addressed the primary concerns of motivation and comparison with SOTA. They have also provided computation times which adds value to the paper. Overall the idea is interesting and the application is definitely relevant. I would recommend an accept.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
7
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Additional experiments are required to address the concerns raised in the initial review.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
18
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The proposed method is interesting in that it generates synthetic “healthy” CT images from images with lesions. Reviewers mainly raised questions about the evaluation and these seemed to be well addressed in the rebuttal.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
3