Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Tianyi Zhao, Zhaozheng Yin

Abstract

Detecting the airway anomaly can be an essential part to aid the lung disease diagnosis. Since normal human airways share an anatomical structure, we design a graph prototype whose structure follows the normal airway anatomy. Then, we learn the prototype and a graph neural network from a weakly-supervised airway dataset, i.e., only the holistic label is available, indicating if the airway has anomaly or not, but which bronchus node has the anomaly is unknown. During inference, the graph neural network predicts the anomaly score at both the holistic level and node-level of an airway. We initialize the airway anomaly detection problem by creating a large airway dataset with 2589 samples, and our prototype-based graph neural network shows high sensitivity and specificity on this new benchmark dataset. The code is available at https://github.com/tznbz/Airway-Anomaly-Detection-by-Prototype-based-Graph-Neural-Network.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_19

SharedIt: https://rdcu.be/cyl5R

Link to the code repository

https://github.com/tznbz/Airway-Anomaly-Detection-by-Prototype-based-Graph-Neural-Network.git

Link to the dataset(s)

https://github.com/tznbz/Airway-Anomaly-Detection-by-Prototype-based-Graph-Neural-Network.git

Reviews

Review #1

Please describe the contribution of the paper

The author proposes a model on Graph Neural Network. Based on the assumption that airways of normal human share an anatomical structure and abnormal (i.e., anomalies) deviates a lot from the normal cases, the author learn the prototype from the given datasets. Besides, as the collected data only get a holistic label, which means normal or abnormal for the whole lung, instead of specific labels for bronchus node, a weakly-supervised method is adopted. The model detects anomalies by output an anomalousness score.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. This paper propose to combine the GNN with normal prototype model, and initialize the airway anomaly detection problem in the community.
2. The authors offers a dataset which consist of holistic labelled cases.
3. Average results of proposed model are better than baselines and ablation experiments show the effectiveness of prototype.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The evaluation measure is not sufficiently justified, such as directions, levels, length etc.
2. Some explanation on important concepts, such as loss of prototype is not clearly explained.
3. The augmenting on dataset is not justified in the scenario of lung images, by switch, shift and cut.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Not sure. But the description of method is clear.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
About the model:
1. The formula of loss on calculating the prototype model needs to be reformulated. The meaning of summary variable on K is encouraged to demonstrate.
2. The reason of choosing the directions, coordinates, layers as well as length and number of descendants needs to explained. If they are organized by domain experts, references need to be cited.
About the experiments:
1. More results on ablation experiments are encouraged to affiliated in the supplementary materials. As the effectiveness of selected measurements by hand is not shown totally.
2. The effect of augmented datasets and the correctness need to be shown.
3. The paper demonstrates the effectiveness of prototype, but the effectiveness of features selected by hand, i.e., the coordinates is not showed.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper provide an interesting idea of combining the GNN with normal prototype model, and initialize the airway anomaly detection problem in the community. With the graph neural network, the model behaves the best both in the sensitivity and specificity score. The model outputs a lot of normal prototype for normal lungs by co-training. Some detailed comments are:
1. The authors offers a dataset which consist of holistic labelled cases.
2. Average results of proposed model are better than baselines and ablation experiments show the effectiveness of prototype.
3. In section 3.1, the authors concatenate features of images together with several author-defined measurements, such as coordinates, directions, levels, length and number of descendants of the nodes. However, why we choose such measurements are not detailed depicted in the writings.
4. Augmenting on dataset is a universal trick to get more training samples.
5. Ablation experiments are not convictive.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

2
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The paper proposes a prototype-based graph neural network to classify abnormal airway. The input of the algorithm is a hand-crafted feature vector extracted from the segmented and classified airway masks of an existing method. Then, the GNN gives an anomaly score for each node of the airway. In the experiment, the proposed GNN is shown to be better than CNN and the prototype machanism is shown to be beneficial.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Both the problem and the prototype-based graph neural network are relatively novel.
2. The method is weakly supervised, requiring only image-level labels.
3. An airway dataset will be released.
4. The paper is clear to read and the figures are easy to understand.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. In the experiment section, the authors mentioned that 62 normal samples and 23 anomaly samples were collected, then they used data augmentation to generate more samples. It is a good idea to generate synthetic training samples. However, there are 193 testing samples with 22 samples being normal samples, which clearly include synthetic samples too. For medical problems, evaluation should be done on only real samples. Results on synthetic samples may be problematic.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Good. A dataset will be released.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The term “anomaly detection” generally refers to classification of rare outliers (https://en.wikipedia.org/wiki/Anomaly_detection). However, in the proposed dataset, normal airways seems rare. What about the distribution of real patient data? Is normal airways rare? The dataset should reflect real distribution.
2. In Eq 1, how is the neighbor set of a node defined?
3. The method is weakly-supervised, so the authors label all the nodes based on the holistic label during the training. How about labeling the maximum score of the nodes in a sample as the holistic label and ignore other nodes?
4. How about simultaneously optimizing the prototype and GNN?
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Clear presentation. Dataset release. Relatively novel method.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper presents a method to predict an occurring anomaly within a lung CT scan and localizes the anomaly within the scan based on a graph setup that represents the different bronchi of the lung as nodes within the graph and connects them based on anatomical neighborhood. A GCN is used to map the representations of each node, which are based on location and orientation of the corresponding bronchus, to a binary label classifying the bronchus as normal or anomaly. Here, a loss is defined that learns a separation between normal and anomaly samples by minimizing the Euclidian distance of normal samples to a prototype airway (that is learned itself) and maximizing the distance between normal and anomaly samples. At the same time, the prototype is learned in a separate step. These two optimization steps are alternating to successively receive a representation of the prototype and the GCN.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The paper shows a novel method to localize anomalies within a lung CT scan without having a ground truth of the anomaly location during training time. This is achieved by an elegant encoding of the lung anatomy within an graph structure. Following this anatomy described as a tree structure, the usage of a graph for representation seems justified and nicely motivated. The idea of introducing an airway prototype as an anker for the learning of normal representations is interesting and increases interpretability. Additionally, a new dataset is generated for the training of anomaly localisation by extracting describing features for all bronchi based on segmented lung scans and provided to the community. The method is explained understandably.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The loss for training the GCN, which is introduced as a designed loss, has a high similarity to the triplet loss, which is not a problem, but should have been mentioned and compared. Further, within the triplet loss, it is enforced that for an anomaly sample all nodes are seen as anomaly. This is a little counter-intuitive with the idea of anomaly localization. Here, it could be beneficial to introduce the logic (also later explicitly used during inference) that if one node is an anomaly, the whole airway is an anomaly. Like this, the system could also minimize its loss by only giving the nodes as anomaly which are actually deviating. A later visualization shows that the localization of the anomalies seems to work, but still it could be a setup to try to see if performance gets improved. The details of the network setup are not provided. Finally, the method is only evaluated within a short ablation study against a CNN, apparently also not accounting for the class imbalance. It is understandable that the evaluation wants to focus on highlighting the aspects of the method, but the comparison still seems a little narrow.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper provides the used evaluation metrics, but does not explain in detail the setup of the network (layers, etc.). However, the group is going to publish their code as well as the dataset.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

As mentioned above, the setup to set all nodes to anomaly when the airway is abnormal seems a little counter-intuitive with the idea of anomaly localization. It would be great, if the authors could comment on this. Additionally, a better description of the network setup would simplify reproduction of the work. Finally, a more thorough evaluation of the algorithm would be interesting for future work (e.g. comparison against other graph methods).
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea of using a graph based network setup to perform an unsupervised anomaly localization based on a prototype is interesting in my opinion, since the unsupervised character of the approach is important in the data-critical medical domain. Assuming justification by the authors for the raised questions, the idea behind this work could be also interesting for the community.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This submission proposes a prototype-based graph neural network for anomaly detection from a weakly supervised airway dataset. The manuscript is well-organized with a clear presentation. The proposed method is novel and promising. Therefore, this manuscript is recommended for acceptance. Also, the reviewers provide good reviews to further improve the quality of this submission, which are encouraged to be included in the final version if possible.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

N/A

back to top

Airway Anomaly Detection by Graph Neural Network