Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Chong Yin, Siqi Liu, Rui Shao, Pong C. Yuen

Abstract

Liver biopsy image analysis is the gold standard for early diagnosis of non-alcoholic fatty liver disease (NAFLD) worldwide. Deep neural networks offer an effective tool for image analysis. However, when applying deep learning methods to smaller histological image datasets, the model may be distracted by dominant normal tissues and ignore critical tissue alterations that pathologists focus on. In this paper, we propose a selective attention regularization module (SAttenReg) to mimic the diagnosis process of pathologists. Specifically, to explicitly encourage the model to focus on clinically interpretable features (e.g, nuclei and fat droplets), SAttenReg learns the attention map with the regularization of clinically interpretable features. Furthermore, with the different contributions of histological features, the model can selectively focus on different histological features based on the distribution of nuclei in each instance. Experiments conducted on the in-house Liver-NAS and public Biopsy4Grading biopsy image datasets show that our method achieves superior classification performance with promising localization results.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_15

SharedIt: https://rdcu.be/cyl5M

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a selective attention regularization (SAttenReg) module to boost the liver biopsy image diagnosis. The method try to mimic pathothologist’s manner in diagnosis by regulzarizing the clinically relevant features.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed SAttenReg module takes great advantage of the attention mechanism by utilizing both the Grad-CAM results as well as clinically relevant features, e.g., fat droplets and nuclei in liver biopsy images. Extensive evaluation shows obvious better qualitative and quantitative performance
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Good overall.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The overall method description is clear. Would suggest the authors to release the code and data for reproducibility purpose
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Good overall.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. The proposed SAttenReg makes great use of the attention with both Grad-CAM and clinical relevant features.
2. The method description is clear and easy to grasp.
3. Experiental results show the superiority of the the proposed attention module.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

In the manuscript, a method for liver biopsy image analysis is proposed. The method is based on a selective attention regularization. The attention regularization is obtained by using a deep neural network, which takes into account both deep features and clinically interpretable features.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strenght of the paper is the merge of deep features and clinically interpretable features that are combined in an interesting way. The use of not interpretable (deep) features together with interpretable features is an interesting way to train a network, both in terms of performance of the network and in terms of final interpretability of the model.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main weakness of the paper is the absence of discussion about limitations of the method.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

In my opinion, the manuscript contains alot of details to reproduce in a easy way the experiments described in the paper. However more information about Biopsy4Grading dataset could be appreciated. In section 3.1 is not very cleary what “Totally” refers to: Biopsy4Grading dataset and Liver-NAS or only Liver-NAS? Furthermore, a 5-fold cross validation is used for the experiments, but it is not clear if the results were computed on one or more run and then an average of the runs was provided. Finally, no information about the choice of the best epoch, chosen for the computation of the performance.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The authors present a method, based on a selective attention regularization, for liver biopsy image analysis. A deep neural network is used for the selective attention regularization. The network takes into account both deep features and clinically interpretable features. In details clinically interpretable features are related to nuclei and fat droplets. A new loss are introduced, taking into account both deep features and clinically interpretable features. This is a very interesting way to train a deep neural network. The manuscript is well written and easy to read. The discussion about quantitative and qualitative results are very interesting and accurate. However, I have some suggestions for the authors, that could improve the paper:
- more details about Biopsy4Grading dataset would be appreciated. However, all the paragraph related to Dataset in the section 3.1 Experiment settings should be rewritten more clearly;
- in the Traininig details, more information about the training strategy related to the numeber of run, the best epoch chosen for the computation of the performance sould be included;
- some comments on the limitations and on the future works would be appreciated
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The liver biopsy image analysis is a very difficult and interesting problem. In the manuscript an interesting way to combine non interpretable (deep) features and interpretable features is proposed. Furthermore, the paper is fluid to read and well written and organized. In my opinion the paper could be probably accepted with some additions that I have detailed in the previous comments.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper proposes a regularization method using segmentation masks for clinically relevant regions (nuclei and fat). The final task is a classification task, and the related regions for the classification task are given as an auxiliary supervision to the intermediate feature map. Specifically, a 2D attention map is calculated using a linear combination of the feature map, and the model is trained to make the predicted probability map to overlap with the mask of the target regions. Finally, the classification performance is improved.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Well written and easy to follow.
- The method is too simple, and seems to work well.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- One concern related to the novelty. The method is closely related to “Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer”, Sergey Zagoruyko, Nikos Komodakis, ICLR 2017. This paper should be mentioned as a related work, and the authors need to point out the differences.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is quite straightforward, and may not need to publish code/data for reproduction. But, it is always nice to have open-sourced code to reproduce the results.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Overall the proposed method is nice, but there is one concern on the novelty. Please list up the technical novelty with respect to the ICLR paper mentioned in the weakness section.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A little concern on the novelty, but overall the method is nice.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper presents an attention based approach for incorporating clinically interpretable features as well as deep features for liver biopsy image classification. This approach can be useful in incorporating prior clinical knowhow in many other image analysis contexts, in particular when the training datasets may be relatively small. I would like to kindly ask the authors to address the reviewers comments.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

**Reviewer 3 We thank the kind advice on experimental setting part: 1) More description of public dataset Biopsy4Grading [7]

Biopsy4Grading [7] is a public data set. The image tiles (299x299) are extracted from an area of 897x897 px2, and these tiles do not overlap. Overall, they have liver biopsy image tiles related to steatosis (7114), ballooning (15145), and inflammation (9481). A more detailed description can be found in reference [7]. 2) Training details about the number of epoch, the best epoch chosen for computation -We conduct the experiments on two datasets separately. Following the principle of 5-fold cross validation, the dataset is randomly split into 5 groups. In each round, we randomly select 1 group for testing, and the remaining groups for training and validation (train:val:test=7:1:2). Finally, we report the average performance of the five rounds. 3) Limitations and future work

As shown in Table 2, the recognition of lobular inflammation is still a challenging task. The proposed method can enforce the model to pay attention to the clinical features region and improve the interpretability. It seems not enough. This may be caused by the small inter variance and serious imbalance among different histological features. In future work, we will improve the model performance in this direction. Thanks again to the reviewer, and we will give more details about the experimental setting in the final version. ** Reviewer 4 Thanks for making us aware of this relevant previous work (Sergey et al, ICLR 2017). 1) Key difference with the mentioned related paper -The mentioned ICLR paper aims to improve the performance of a student model by mimicking the attention maps of a stronger teacher model. However, for small-scale medical data, it is not always available to have such a stronger teacher model. As analyzed in the introduction part, due to the lack of explicit supervision of attention, it is difficult to determine exactly where the learning model should pay its attention on the complex tissue structures. 2) Contributions and technique novelty Our key contribution is that we propose a selective attention regularization (SAttenReg) method to analyze NAS-related components in liver biopsy images. It explicitly drives the model to focus on clinically interpretable features (e.g., nuclei and fat droplets) to improve the interpretability and reliability of the model. Thanks again to the reviewer for pointing out this relevant paper. We will discuss the method in the final version.

back to top

Focusing on Clinically Interpretable Features: Selective Attention Regularization for Liver Biopsy Image Classification