Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Jia-Ren Chang, Min-Sheng Wu, Wei-Hsiang Yu, Chi-Chung Chen, Cheng-Kung Yang, Yen-Yu Lin, Chao-Yuan Yeh

Abstract

Computational histopathology studies have shown that stain color variations considerably hamper the performance. Stain color variations indicate the slides exhibit greatly different color appearance due to the diversity of chemical stains, staining procedures, and slide scanners. Previous approaches tend to improve model robustness via data augmentation or stain color normalization. However, they still suffer from generalization to new domains with unseen stain colors. In this study, we address the issue of unseen color domain generalization in histopathology images by encouraging the model to adapt varied stain colors. To this end, we propose a novel data augmentation method, stain mix-up, which incorporates the stain colors of unseen domains into training data. Unlike previous mix-up methods employed in computer vision, the proposed method constructs the combination of stain colors without using any label information, hence enabling unsupervised domain generalization. Extensive experiments are conducted and demonstrate that our method is general enough to different tasks and stain methods, including H&E stains for tumor classification and hematological stains for bone marrow cell instance segmentation. The results validate that the proposed stain mix-up can significantly improves the performance on the unseen domains.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87199-4_11

SharedIt: https://rdcu.be/cyl3N

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper proposes a method for domain generalization of machine learning methods for histopathology image analysis based on the “mix-up” data augmentation principle.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strength of the paper is that the proposed method is very simple and elegant, yet shown to be effective for two different applications iin histopathology image analysis.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The authors claim that the. method is “unsupervised domain genralization” but I think that the naming is problematic. The method still requires access to the target domain at training time and the samples have to be labelled with their domain. This makes the “unsupervised” title misleading as there are methods that aim to improve the domain generalization without access new domains at training time.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

One of the datasets used in the paper is from the CAMELYON17 challenge. The results on this dataset should be presented in a way that can be compared to results submitted to the challenge. The authors should state if the submitted method is state-of-the-art w.r.t. to results submitted to CAMELYON17.

The other dataset seems to be private, which limits the overall reproducibility of the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Since this is a methodology paper, I think adding additional classification tasks/datasets will strengthen the paper.

Using a distribution of \alpha that has a peak at 0 (i.e. results in complete transfer of the target stain matrix to the source domain images) will be interesting and I expect it might improve the results.

I would also add a comparison to a baseline that is a combination of normalization and data augmentation as these two techniques can be complementary.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

It is an elegant method that seems to work. Additional experiments will make it more convincing.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The work deals with color issues in tissue analysis that are stained either by mainstream H&E or by specific coloration procedure. Either data augmentation or color normalization have been used so far. The authors tackle the generalization issue to unseen stains by a mix-up data augmentation method including unseen color staining in the training process.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The mix-up strategy never used in histology images according to the authors.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

We hardly understand how to use it in practice;
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Good references and formula
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Abstract : “to adapt varied stain colors” → to adapt to varied stain colors, I guess

Introduction : Research works used color patterns as a core concept to either augment data or homogenize color representations usually by decomposing a stain color matrix. The basic idea is to create a mix-up stain color matrix to improve robustness of training algorithms. Can you explain a bit more the mix-up methods in general computer vision and notably the use of labels in addition to images. The results of the proposed method are on a par with state of the art methods but the authors claim to be the first to propose unsupervised domain generalization in histology images analysis.

Methods :

Do not forget that staining is an absorption phenomena but also a diffusion one that makes sometimes the Beer-Lambert law flawed.

The last paragraph is doubling the same arguments in the introduction. I would rather read about more explanation about the use for different staining for example like ImmuniHistoChemistry IHC staining. I do not really see the unsupervised way in it. As much as new data you have you will mix up the matrix there. So it is purely stain augmentation. How do you use the labels in the source domain and what is this label ?

Experiments :

For the H&E experiments, you have 5 centers. For the Hema dataset, you have only one center right ? What is not clear to me is how you mix up ? What it the source and target domain ? C1 versus C2 to C5. C1 versus HEMA ? Etc.

Your experiments then show a more robust cross-domain interpolation that stabilize the results over different centers and staining procedures which is a very good achievement. In supplemental material, it is shown that random self-perturbation improve robustness. Is it so different than mainstream data augmentation methods?

Sorry for my misunderstanding, but I do not really understand the Hema experiment : how is it carried on and what is the conclusion.

All in all, the work is interesting but according to me deserves to be better explained.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

it is an interesting issue about color in tissue images and variability. Much work have been proposed on the topic. I do not really see how to use this one in pracrice.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors present a augmentation method for histopathological slides. In their approach, images are decomposed into color and density matrix in source and target domain. mixing the color matrices of source and target allows a augmentation that covers the target space and leads to improved results for two challenges: tumor classification and single cell instance segmenation
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors address an important problem with a novel, easy solution that seems to outperform other methods on two tasks.
- The figures illustrate the approach very nicely
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Code and Hema dataset is not publicly available
- Setting the delta parameter is ad hoc
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Code and Hema dataset is not publicly available
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Abstract wordiing can be improved
- Provide Code and Data
- Provide strategies for setting delta parameter
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- A novel, simple method for stain augmentation
- Paper is written clearly and concisely
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

7
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors describe a novel augmentation method for histopathological slides. In their approach, images in both source and target domains are decomposed into stain amount and colour matrices based on a previously proposed stain decomposition method (sparse NMF). Mixing of the colour matrices of source and target allows a ‘mix-up’ augmentation that covers the target space and leads to improved results for two downstream tasks: tumour classification and single-cell instance segmentation. All reviewers think the method is simple and effective, thus recommending paper acceptance. Yet authors could adapt reviewers’ comments in their final submission and also consider releasing the Hema dataset and code to promote reproducibility. In Addition to these comments, I also have a concern that whether the relatively slow speed of Sparse NMF stain decomposition will affect the practical usage of the proposed augmentation method. Since augmentation usually requires real-time performance. Maybe authors can also bring that into the discussion.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We appreciate all the valuable comments from the reviews. In the following, we would like to address the following concerns raised in the reviews: (1) The speed of SNMF stain decomposition will affect the practical usage of the proposed augmentation method (meta-reviewer). (2) The availability of the Hema dataset and the code (R1, R5). (3) The practical usage for unsupervised domain generalization (R1, R3). (4) Random self-perturbation of stain matrix (R3).

For (1), to prevent repeating SNMF decomposition, the stain matrices of images are pre-computed using SNMF before the training procedure. That is, we only compute stain matrices once and use them repeatedly during training. The computational time of SNMF decomposition for for a single image in CAMELYON17 and Hema takes 1.14 and 2.40 seconds, respectively, measured on an Intel Xeon CPU E5-2697 v3.

For (2), we will release our training/test code working on the CAMELYON17 dataset upon formal acceptance. The CAMELYON17 dataset can be obtained from the official website (https://camelyon17.grand-challenge.org/). However, the Hema dataset is not publicly available due to IRB requirements for privacy protection.

For (3), we specify a practical scenario below where the proposed method is practical and applicable. We assume that we have a well-annotated histopathology dataset collected from one medical center. The annotated dataset can be used to train a good model for data in this center. However, the model performance is degraded when applying this model to data collected in other centers due to the substantial color domain gap. Fine-tuning this model is not a feasible solution to this issue because of the lack of annotations from the new data. Our method helps at this moment where we only need the stain matrix of the new coming data. This is why we call our method “unsupervised” because we do not need the labels of new data. The model can be retrained using the original well-annotated histopathology dataset and the stain matrices of the new data. In our experiments, we simulate the described situation, i.e., training the model on C1 and applying it to C2-C5 of the CAMELYON17 dataset as well as training on M1 and applying to M2 of the Hema dataset. The experimental results demonstrate that the proposed method can eliminate domain gaps in an unsupervised way.

For (4), we would like to clarify about the results of random self-perturbation of the stain matrix. The random self-perturbation of a stain matrix is a special case of stain mix-up where the stain matrix is mixed with random noises. It is reasonably expected that self-perturbation of a stain matrix improves robustness as well.

back to top

Stain Mix-up: Unsupervised Domain Generalization for Histopathology Images