Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Mingzhou Liu, Fandong Zhang, Xinwei Sun, Yizhou Yu, Yizhou Wang

Abstract

In the early diagnosis of lung cancer, an important step is classifying malignancy/benignity for each lung nodule. For this classification, the nodule’s features (e.g., shape, margin) have traditionally been the main focus. Recently, the contextual features attract increasing attention, due to the complementary information they provide. Clinically, such contextual features refer to the features of nodule’s surrounding structures, such that (together with nodule’s features) they can expose discriminate patterns for the malignant/benign, such as vascular convergence and fissural attachment. To leverage such contextual features, we propose a Context Attention Network (CA-Net) which extracts both nodule’s and contextual features and then effectively fuses them during malignancy/benignity classification. To accurately identify the contextual features that contain structures distorted/attached by the nodule, we take the nodule’s features as a reference via an attention mechanism. Further, we propose a feature fusion module that can adaptively adjust the weights of nodule’s and contextual features across nodules. The utility of our proposed method is demonstrated by a noticeable margin over the 1st place on Data Science Bowl 2017 dataset in Kaggle’s competition

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_3

SharedIt: https://rdcu.be/cyl5v

Link to the code repository

N/A

Link to the dataset(s)

https://www.kaggle.com/c/data-science-bowl-2017

Reviews

Review #1

Please describe the contribution of the paper

In this paper the authors propose a neural network architecture for lung nodule malignancy classification that can capture both nodule specific features as well as relevant features from the surrounding locations of the features, such as pleural Indentation, vascular Convergence and bronchial interruption. Empirical results on the Data Science Bowl 2017 data, show that the proposed framework has improved performance compared to the top competition entries.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Well structured comparison to baselines that supports the effectiveness of the proposed architecture
- Combining context and nodule features for lung nodule classification is an important problem and to the extend of this reviewers’ knowledge hasn’t received dedicated attention from past works
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The authors use only the Kaggle competition dataset in order to validate their work and not the much larger NLST. Empirical results on NLST would significantly strengthen this work.
- The presented visualizations do not add much value since it is hard to understand how accurate they are and to what extend these examples are representative of the behavior of the model. The authors could consider requesting additional annotations from expert radiologists that can annotate the context-features that are relevant and then quantitatively assess whether the network successfully identifies them
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors use a public dataset, however the dataset appears to not be available for download anymore: https://www.kaggle.com/c/data-science-bowl-2017/data In general the authors provide sufficient information to be able to re-implement their approach.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

This is an interesting paper and to the extend of this reviewers’ knowledge novel in that it focuses on separating the nodule specific features that contribute to nodule malignancy vs. the contextual features. The comparison to the baselines is done in a well-structured manner that supports the utility of the proposed CA-net architecture. Unfortunately, the authors employed only the Kaggle competition dataset and not the much larger NLST dataset. The baseline comparison on the NLST dataset would offer much stronger evidence for the utility of the proposed architecture. Moreover, the use of NLST, would have allowed the authors to put their results in context with more recent works, such as “Ardila, D., Kiraly, A.P., Bharadwaj, S. et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med 25, 954–961 (2019)”. The visualizations that are presented in the paper do not provide much added value, since it is hard to assess whether these are representative of the behavior of the model. The authors could consider to gather pixel-level annotations by expert radiologists about the context-features that are relevant for assessing the malignancy of a nodule. In this manner the model attention could be quantitatively assessed.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Interesting and novel work to the extend of this reviewers’ knowledge with well structured comparison to baselines.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper proposed a Context Attention Network (CA-Net) for lung nodule malignancy/benignity classification by leveraging the contexture features from nodule’s surrounding structures together with nodule’s features via an adaptive feature fusion module. The Kaggle DSB2017 dataset is conducted to prove the proposed framework performance and effectiveness for contexture feature extraction.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This paper is well-written and easy to follow. The idea of leveraging the surrounding of the nodule is novel, as it is essential for clinical diagnosis from the radiologist’s point of view. This paper implements this diagnose factor by learning the features automatically via CNN based method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. More details are needed for 3D nodule and surrounding size selection.
2. More discussions are needed about the attention on the surrounding contexture feature for various nodule sizes.
3. More discussion about the pros and cons of the proposed method with future work.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The method is well-written and implementation steps are clearly provided.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Section 2.1: It is not clear how to obtain the size of the cropped region in three dimensions by the nodule size. Such as how to determine the size of the I_N? If the radius is the size, how to guarantee it can be large enough to cover the contextual information?
2. Section 2.1: how to guarantee the detected nodule are accurate and deal with the false positives? Any human annotations are involved?
3. Section 2.2: how to deal with the various sizes of nodule input for feature extraction?
4. Curious about the effectiveness and robustness of this model, especially on small nodules. It seems like the contextual features are easy to capture how about the small one? Additionally, what are the failure cases that cannot accurately diagnosis by utilizing the contexture feature?
5. Some of the font sizes in Fig.2 are too small to read.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. This paper is well written and very easy to follow.
2. The proposed framework is novel by effectively learning the nodule with its surrounding contextual feature, which mimics the clinal diagnosis and fully utilizes the deep learning-based methods for contextual feature extraction rather than the handcrafted feature.
3. The proposed two-path context and nodule attention mechanism is novel by incorporates the nodule with the surroundings effectively learn the essential contextual feature.
4. Experiment results demonstrate the state-of-the-art performance on the DSB2017 dataset, and the illustration shows the contextual features are successfully learned by the proposed framework, which benefits the nodule diagnosis accuracy. One concern is the robustness of the framework to small nodules with fewer contextual features. Therefore, probably accepted is suggested.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposed a CA-Net that accurately captures contextual features of nodules, and achieves state-of-the-art performance on the BSD2017 dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The proposed CA-Net achieved state-of-the-art performance on the Kaggle DSB2017 dataset.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1) The novelty of the proposed contextual attention module is limited; 2) Literature review on contextual modeling is weak; 3) Compared contextual feature caption methods are not state-of-the-art. 4) The effectiveness of the proposed contextual feature caption method didn’t be well demonstrated. Compared contextual feature caption methods, published before 2017, are not state-of-the-art. 5) Ablation study was performed directly on the test set.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Code is unavailable but the idea is easy to reproduce.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

(1) More method details should be provided, such as the size of the images or features at each stage. (2) The motivation of the leaky parameter p_l in the Cancer Prediction Module should be explained. (3) Could the authors explain why the attention highlights look like regular rectangles in Fig.3? Besides, it seems that the proposed method focus on more irregular background regions, instead of the lesions. Is it able to demonstrate the attention ability of the proposed method?
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I am mainly concerned about the limited novelty and weak literature review on contextual modeling, which may not reach the level of this venue.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

6
Reviewer confidence

Very confident

Review #4

Please describe the contribution of the paper

This paper proposes the CA-Net that employs the complementary information between the features of lung nodule features and the features of nodule’s surrounding structures for improving the accuracy of lung cancer prediction. The CA-Net achieves 1st place by a noticeable margin on the Kaggle DSB2017 dataset.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

(1) The idea of employing the complementary information between the lung nodule and its surrounding structures is interesting, which can be transferred to other similar tasks and thus may contribute to the medical image diagnosis community. (2) The ablation study is systematic. (3) The proposed method achieves 1st place by a noticeable margin on the Kaggle DSB2017 dataset.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) Missing the discussion about the impact of nodule detection and Region-Of-Interest pooling on the lung cancer prediction (see following detailed comments 1); (2) The design motivation of Contextual Attention module is unclear (see following detailed comments 2).
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Most key experiment details are included so that it is possible to replicate this work.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. In CA-Net, an off-the-shelf nodule detector [10] is used to obtain nodule’s radius and center coordinates. However, the nodule detector can’t guarantee 100% detection accuracy. The false positive nodules will interfere with the network optimization. Moreover, the Region-Of-Interest pooling can’t guarantee 100% segmentation accuracy. The inaccurate segmentation will influence nodule encoding and surrounding encoding, thus also interfere with the network optimization. How to avoid these negative effects？
2. The design motivation of Contextual Attention module is unclear. What’s the meaning of the attention vector γ? Why compute γ in this way？What is the advantage of XC over XS? In Contextual Attention module, a Squeeze-and-Excitation layer [8] is used as the nodule encoding, and an Encoding Layer [21] is used as the surrounding encoding. Why nodule encoding and surrounding encoding use different layers?
3. Also the performance gains in Table 2 and Table 3 are small. Maybe adding the std and paired t-test for evaluating the performance.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Although this paper is interesting, I still have some comments:
1. In CA-Net, an off-the-shelf nodule detector is used to obtain nodule’s radius and center coordinates. However, the nodule detector can’t guarantee 100% detection accuracy. The false positive nodules will interfere with the network optimization. Moreover, the Region-Of-Interest pooling can’t guarantee 100% segmentation accuracy. The inaccurate segmentation will influence nodule encoding and surrounding encoding, thus also interfere with the network optimization. How to avoid these negative effects？
2. The design motivation of Contextual Attention module is unclear. What’s the meaning of the attention vector γ? Why compute γ in this way？What is the advantage of XC over XS? Why nodule encoding and surrounding encoding use different layers?
3. The performance gains in Table 2 & 3 are small. Maybe adding the std and paired t-test for evaluating the performance.
The authors should address these comments during feedback.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

6
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This work presents a CAD solution for lung cancer prediction. The proposed approach utilizes contextual features such as vascularity and fissure vicinity as attention module within their proposed architecture. The proposed idea and design seems novel. The manuscript is well-written and easy to follow. Furthermore, authors provided a thorough analysis of results and comparison to STOA. Nevertheless, as pointed out by some reviewers, the experimentation is limited to the Data Science Bowl 2017 Kaggle. It would be interesting if authors could extend their experiments with other datasets as well. Furthermore, as pointed out by reviewers, the proposed approach relies on accurate detection and segmentation of nodules. It would be interesting to know the sensitivity of the proposed approach w.r.t. the nodule detection module. I strongly encourage authors to address reviewers’ major concerns in final submission.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

N/A

back to top

CA-Net: Leveraging Contextual Features for Lung Cancer Prediction