Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Jia-Ren Chang, Ching-Yi Lee, Chi-Chung Chen, Joachim Reischl, Talha Qaiser, Chao-Yuan Yeh

Abstract

Understanding of prognosis and mortality is crucial for evaluating the treatment plans for patients. Recent developments of digital pathology and deep learning bring the possibility of predicting survival time using histopathology whole slide images (WSIs). However, most prevalent methods usually rely on a small set of patches sampled from a WSI and are unable to directly learn from an entire WSI. We argue that a small patch set cannot fully represent patients’ survival risks due to the heterogeneity of tumors; moreover, multiple WSIs from one patient need to be evaluated together. In this paper, we propose a Hybrid Aggregation Network (HANet) to adaptively aggregate information from multiple WSIs of one patient for survival analysis. Specifically, we first extract features from WSIs using a convolutional neural network trained in a self-supervised manner, and further aggregate feature maps using two proposed aggregation modules. The self-aggregation module propagates informative features to the entire WSI, and further abstract features to region representations. The WSI-aggregation module fuses all the region representations from different WSIs of one patient to predict patient-level survival risk. We conduct experiments on two WSI datasets that have accompanying survival data, i. e., NLST and TCGA-LUSC. The proposed method achieves state-of-the-art performances with concordance indices of 0.734 for NLST and 0.668 for TCGA-LUSC, outperforming existing approaches.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_70

SharedIt: https://rdcu.be/cyl6N

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper presents a deep learning architecture to aggregate information from multiple WSI of individual patients towards survival analysis. The proposed module consists of a self-aggregation module that propagates global and local information and a WSI-aggregation module that merges information from multiple WSIs. The method had been evaluated on two different publicly available datasets and it has been compared with other methods in the literature.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The topic of the paper is interesting and relevant for MICCAI.
- The presented methodology is novel and the different components of the method well presented.
- The paper presents very solid experimental results, highlighting the advances of the method.
- The authors present comparisons with conventional and deep learning based survival models.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- There is no discussion about the intuition of the selected magnification for the cropping of the patches. Is there a constraint on the used magnification?
- I think that there is no experiment in the paper highlighting the need for the use of multiple WSIs for the survival analysis. The authors should report the performance of their model without the WSI-Aggregation module to highlight the need of multiple WSIs for survival.
- A discussion about computational complexity and memory limitations of the method is missing.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors claim that the code of their method will be publicly available. The authors need to include the type of GPU that they perform their experiments, which is quite important especially for the training of the MoCo algorithm and selecting the batch size.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The authors should include some more details about the training of the compared methods. Did they also use multiple WSIs for their training? In case they did not the presented comparison is not really fair.
- It is not clear from the dataset presentation how many WSIs per patients the authors used for their experiments. This is quite important to prove the soundness of the method.
- In the experimental results section it is not very clear when the authors refer to the self-aggregation module and when to the WSI-aggregation module. The authors should try to be very explicit on which aggregation they refer to.
- Figure 2 is a bit confusing.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The paper tackles a very interesting problem proposing novel ways for using multiple WSIs for survival analysis.
- The method is well presented and easy to follow.
- The experimental setup is solid, including ablation studies and comparisons with other methods.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This study presents a hybrid aggregation network for predicting patients’ survival via whole slide image. The main contribution is the design of self-aggregation module for feature aggregation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The method has novelties in terms of designing self-aggregation module and WSI-aggregation module, when the survival analysis is performed from WSIs.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

This study follows the same idea from the existing technique WSISA, where the main focus is about how to aggregate features from whole slide image. Although the comparison has been performed between the WSISA and the presented method. It is also mentioned that ‘WSISA uses small model for feature extraction’ which degrades the performance. Overall it is not confirmed that the proposed feature aggregation method is better than that used in WSISA.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

For reproducibility, the authors are suggested to make the sample codes public accessible, such that it is reproducible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

It would be more convincing if authors show that the proposed feature aggregation method is better than that used by WSISA, otherwise the value for proposed method is limited.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The presented method novelty and the evaluations.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper
- Proposes an approach based on aggregation of deep learning features from multiple whole slide images of a patient for survival analysis. The aggregation approach uses two aggregation modules, to aggregate local features to image-level feature maps and propagate them to region representations.
- Experimental evaluation using two lung cancer datasets shows performance improvement over previous methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Generates image-level information by adaptively aggregating local features from patches.
- Combines information from multiple images belonging to a patient.
- Achieves better results than previous methods evaluated in the experimental section.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The experimental evaluation is done using lung cancer cases only. The performance of the proposed method for other cancer cases was not evaluated.
- The baseline deep learning methods used in the experimental evaluation are from publications in 2016-2018. The experimental evaluation should also have included more recent works, e.g.: [1] Wulczyn, E., Steiner, D.F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C.H., Chen, P.H.C., Liu, Y. and Stumpe, M.C., 2020. Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One, 15(6), p.e0233678. [2] Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N. and Huang, J., 2020. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65, p.101789.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The datasets used in the experiments are publicly accessible datasets. The paper appears to describe the experimental setup and implementation in sufficient detail. The results could be reproduced with some help from the authors.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

This work targets an important problem. It represents a hybrid aggregation mechanism to combine local information to image level information and to aggregate information from multiple whole slide images for a patient. The results with lung cases show performance improvements over previous methods.

The main weaknesses of the paper are the limited experimental evaluation. The authors should present results with other cancer types. This would help assess the efficacy of the proposed approach beyond lung cancer cases, since different cancer types have diffident survival patterns. The authors should also compare their method against more recent work, such as:

[1] Wulczyn, E., Steiner, D.F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C.H., Chen, P.H.C., Liu, Y. and Stumpe, M.C., 2020. Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One, 15(6), p.e0233678. [2] Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N. and Huang, J., 2020. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65, p.101789.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed method is technically sound and shows performance improvements compared to previous methods. However, the experimental evaluation is limited to lung cancer cases. It is not clear how well the method will do with other cancer types with different survival patterns. Moreover, the deep learning methods used as baseline cases in the experimental evaluation are from works done in 2016-2018. The authors should have compared with more recent work such as: [1] Wulczyn, E., Steiner, D.F., Xu, Z., Sadhwani, A., Wang, H., Flament-Auvigne, I., Mermel, C.H., Chen, P.H.C., Liu, Y. and Stumpe, M.C., 2020. Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One, 15(6), p.e0233678. [2] Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N. and Huang, J., 2020. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65, p.101789.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presents a hybrid aggregation network for survival prediction of cancer patients from whole slide image. It is good to see that authors designed self-aggregation modules in both feature space and WSI level for survival analysis from WSIs of cancer patients. The image-level information is obtained by adaptively aggregating local features from patches. Experimental results have been presented to demonstrate the advances of the proposed method. Consensus from Reviewers on Acceptance, but with raised questions, such as more detailed experiment settings, more experiment results on data other than lung cancer and more comparison with more recent work that mentioned by the reviewers. Please carefully refer to these raised questions while revising the paper further.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

We appreciate all the valuable comments from the reviews. In the following, we would like to address the concerns raised from the reviews: (1) Selected magnification (R2). (2) More experiments on the need of multiple WSIs (R2), feature aggregation method is better than that used by WSISA (R3) and other cancer types (R4). (3) More recent works (R4). (4) A discussion about computational complexity and memory limitations (R2).

For (1), we adopted 10x magnification which is reported in the Section 2.1.

For (2), we appreciate the suggestion from the reviewers. R2 suggests adding one experiment to demonstrate the WSI-aggregation module is important to make a patient-level prediction. We believe that only one WSI can not fully represent the survival status of a patient, and we would like to aggregate all the WSIs of one patient to predict the survival outcome. An ablation study would be added to the supplementary to support this view. R3 wants to know whether the feature aggregation method is better than that used by WSISA. The WSISA used k-means to select important patch features whereas the proposed method used a learnt region representation abstraction. It is reasonably expected that a supervised learnt module can outperform an unsupervised k-means algorithm. R4 would like to know whether the proposed method can be applied to other cancer types. It is our future work and we believe that the proposed method is generic for applying to other cancer types.

For (3), we acknowledge that the reviewer shared recent works and will include these works in the paper.

For (4), the feature extraction from WSIs and the training of self-supervised feature extractor consumed the most computational time and memory cost. It takes several days to train a self-supervised feature extractor using MoCo algorithm on four 1080-ti GPUs. After feature extraction from WSIs, the HANet only takes a few minutes to train. We would like to discuss this issue in the paper.

back to top

Hybrid Aggregation Network for Survival Analysis from Whole Slide Histopathological Images