Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Sophia J. Wagner, Nadieh Khalili, Raghav Sharma, Melanie Boxberg, Carsten Marr, Walter de Back, Tingying Peng

Abstract

In digital pathology, different staining procedures and scanners cause substantial color variations in whole-slide images (WSIs), especially across different laboratories. These color shifts result in a poor generalization of deep learning-based methods from the training domain to external pathology data. To increase test performance, stain normalization techniques are used to reduce the variance between training and test domain. Alternatively, color augmentation can be applied during training leading to a more robust model without the extra step of color normalization at test time. We propose a novel color augmentation technique, HistAuGAN, that can simulate a wide variety of realistic histology stain colors, thus making neural networks stain-invariant when applied as an augmentation technique in the training process. Based on a generative adversarial network (GAN) for image-to-image translation, our model disentangles the content of the image, i.e., the morphological tissue structure, from the stain color attributes. It can be trained on multiple domains and, therefore, learns to cover different stain colors as well as other domain-specific variations introduced in the slide preparation and imaging process. We demonstrate that HistAuGAN outperforms conventional color augmentation techniques on a classification task on the publicly available dataset Camelyon17 and show that it is able to mitigate present batch effects.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87237-3_25

SharedIt: https://rdcu.be/cymag

Link to the code repository

https://github.com/sophiajw/HistAuGAN

Link to the dataset(s)

https://camelyon17.grand-challenge.org/

Reviews

Review #1

Please describe the contribution of the paper

This paper proposed a novel color augmentation technique, HistAuGAN, that can simulate a wide variety of realistic histology stain colors, thus making neural networks stain-invariant when applied as an augmentation technique in the training process. Based on a generative adversarial network (GAN) for image-to-image translation, our model disentangles the content of the image, i.e., the morphological tissue structure, from the stain color attributes.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

(1) Writing is clear and easy to follow. (2) The experiments are extensive, and the authors illuminate the advantages of the proposed method from multi-views.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) The method is the same as the DRIT++, and the authors claimed that they proposed this technique, which is misleading. (2) Figure 1(b) contains incomplete and misleading information. (3) Eq 1 includes 6 hyperparameters. The following content neither describes the concrete settings nor analyzes them.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This paper is easy to follow.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors should consider and sovle the issues proposed in Weaknesses.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper introduces a mature technique , DRIT++, to solve the batch effects existing in WSIs. Although the method lacks innovation, the experimental results from multi views demonstrate that the method is effectiveness in WSIs.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors propose a novel color augmentation technique, HistAuGAN, that can simulate a wide variety of realistic histology stain colors, thus making neural networks stain-invariant when applied as an augmentation technique in the training process. Based on a generative adversarial network (GAN) for image-to-image translation, the proposed model disentangles the content of the image, i.e., the morphological tissue structure, from the stain color attributes. It can be trained on multiple domains and, therefore, learns to cover different stain colors as well as other domain-specific variations introduced in the slide preparation and imaging process. The authors demonstrate that HistAuGAN outperforms conventional color augmentation techniques on a classification task on the publicly available dataset Camelyon17 and show that it is able to mitigate present batch effects.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

In contrast to previous approaches, the proposed HistAuGAN disentangles the content of a histological image, i.e., the morphological tissue structure, from the stain color attributes, hence preserving the structure while altering the color. Therefore, HistAuGAN can be used as a stain augmentation technique during training of a task-specific convolutional neural network (CNN). According to authors, HistAuGAN is the first GAN-based color augmentation technique that generates realistic histological color variations.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

the quality evaluation by AN expert pathology is a weakness because of the well-known inter-variability among experts.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducibility of the method: yes Public dataset is used
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors could improve their significant results by including at least the expert’s opinion for qualitative evaluation
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Well defined and described proposed method, in a very active research field. Robust experimental framework. Use of public dataset.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper
1. The key problem in scaling a neural network across multiple histopathology labs is identified and analyzed.
2. Both inter-lab and intra-lab variations are mentioned and included in the analysis
3. Attempted to introduce a method to generate an analogous synthetic image in the home domain for each image from the external lab.
4. Effects of the above method on a classification task have been demonstrated.
5. Thorough validation has been presented from both computational and pathologist perspectives.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Novelty observed in the method used to generate images for augmentation for training. This method can help in identifying appropriate variations to be introduced during augmentations.
2. Validation from both computational and pathologist perspective, although might not be novel, is very rare and important.
3. Fig3 plot visualization strongly helps in proving the effectiveness of the proposed method for the given task
4. The evaluation on unseen breast cancer data makes a strong point as well but doesn’t suffice to claim that the method is genralizable.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Baseline methods compared are outdated. There is no comparison to newer methods like Feature aware normalization.
2. Lack of evidence on how it performs on tissue regions that look different because of variation of features within a tissue.
3. Only a binary classification task is demonstrated. The variation and difficulty level varies considerably with pathology and this might not warrant the claim that this method is scalable or generalizable. More efforts are to be directed towards the computational validity of these methods for binary classification with different pathologies considering one at a time. As long as only the classification in the paper is considered, this remark can be ignored.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
1. The magnification level of the images and the pathology information is required, however, this can be obtained from the Camelyon references.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Consider comparing the performance with state of the art. There are methods like Feature Aware normalization, Proscia’s Pathology Deep Learning System which demonstarte ways to combat tissue variations.
2. The performance comparision could be demonstatde on an annotated slide and examples can be shown for High, Medium and Low similarities.
3. Including results on differently looking visual areas is important.
4. Inclusion of snapshots from more exmaples of breast cancer data might be helpful.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. The proposed method is sensible and reproducible.
2. There is no evidence on how well it performs on what’s already out there.
3. Lack of evidence on how it performs on tissue regions that look different because of variation of features within a tissue.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed a novel color augmentation technique, HistAuGAN, that can simulate a wide variety of realistic histology stain colors, thus making neural networks stain-invariant when applied as an augmentation technique in the training process. The strengths of the paper include: 1) clear writing and easy to follow; 2) extensive validation from both computational and pathologist perspectives ; 3) novelty in method. The points should be addressed in rebuttal: 1) comparison with DRIT++ and other state-of-the-art methods; 2) performance on various regions; 3) expert’s opinion for qualitative evaluation.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

Dear Area Chair,

We thank you and the reviewers for their constructive feedback on our manuscript and for giving us the opportunity to clarify the points raised.

Our method is acknowledged by the reviewers as a “novel color augmentation technique” (R1, R2, AC) that can solve the “key problem in scaling a neural network across multiple histopathology labs” (R3). By “simulat[ing] a wide variety of realistic histology stain colors” (R1), it “identifi[es] appropriate variations to be introduced during augmentations” (R3). All reviewers appreciate the structure of the paper and find that the “writing is clear and easy to follow” (R1). They further agree that the experimental evaluations are “extensive” (R1), “significant” (R2), and “thorough from both computational and pathologist perspectives” (R3). Yet they also have concerns and suggestions about i.a) the relationship between HistAuGAN and DRIT++ (Lee et al., 2020) (R1), i.b) its comparison to other state-of-the-art normalization methods (R3), ii) the performance on tissue regions that look different (R3), and iii) the expert’s opinion for qualitative evaluation (R2).

i.a) As stated in the manuscript, HistAuGAN is based on the same network architecture as DRIT++. We would like to clarify that we do not claim in our manuscript that the architecture itself is our idea. Our contribution however is to propose stain color augmentation, in contrast to conventional color normalization techniques. Specifically, we extend DRIT++ by interpolating between domains, i.e. the medical laboratories, to address both inter-lab and intra-lab variations. This increases the color variability of the augmented images by a large margin. We name our augmentation technique HistAuGAN to emphasize its origin in a GAN architecture and its applicability to histological images, which is neither covered by DRIT++ nor by other papers. To address the concerns raised by R1, we changed the caption of Section 2.1 and Figure 1b from “HistAuGAN architecture” to “Model architecture’’, and a formulation in the text to clarify the relationship between HistAuGAN and DRIT++.

(i.b) As shown in the results, using HistAuGAN for training a downstream classification task makes the trained network robust to color variations and improves its performance. Hence, no preprocessing is needed when applying the trained model on external test data. Our solution thus has a large practical advantage compared to normalization methods. We prove this by showing that HistAuGAN scales across multiple domains, while normalization methods like the ones mentioned by R3 require 10 independent trainings between each pair of domains on Camelyon17, which is infeasible in clinical application. Most importantly, we compared HistAuGAN to the best existing color augmentation method, since Tellez et al. (2019) already demonstrated the superiority of color augmentation to existing stain normalization methods including the feature-aware normalization from R3.

(ii) As part of the qualitative evaluation, the reviewers suggested including “results on differently looking visual areas” (R3). In Figure 2, we chose the same patch to assure a fair comparison between the methods. However, Camelyon17 is indeed very diverse and we now include sample patches of various regions with the corresponding HistAuGAN augmentation in the supplementary material that allows for a more extensive qualitative evaluation. (iii) With respect to the qualitative evaluation by an expert pathologist, we are also happy to include “snapshots from more examples of breast cancer data” (R3) with the corresponding “expert’s opinion” (R2) in the supplementary to show the model’s generalisability on untrained tissue.

We hope that we could clarify the concerns of the reviewers and provide a revised manuscript about a “well defined and described proposed method, in a very active research field” (R2) to drive computational pathology research forward in the MICCAI community.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper proposed a novel color augmentation technique, HistAuGAN, that can simulate a wide variety of realistic histology stain colors, thus making neural networks stain-invariant when applied as an augmentation technique in the training process. The authors conducted extensive validation from both computational and pathologist perspectives. The rebuttal sufficiently clarifies the relation with DRIT++ and the qualitative evaluation.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This manuscript introduces a generative adversarial network-based color augmentation method for histology images, by disentangling image content from stain color attributes. The method is scalable to multiple domains (e.g., laboratories) for color augmentation. The major concerns from reviewers are addressed in the rebuttal, such as relationship with DRIT++ and comparison with other state-of-the-art approaches. Although the proposed method might not exhibit strong technical novelty, it seems to be effective in changing color appearance of histology images with preserving the morphological structures. This is very important for histology image analysis. Thus, the manuscript is recommended to be accepted.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The proposed idea has many novel components which will be of interest to the MICCAI community. The rebuttal effectively addresses the concerns raised by reviewers.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

Structure-Preserving Multi-Domain Stain Color Augmentation using Style-Transfer with Disentangled Representations