Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Hui Cui, Ping Xuan, Qiangguo Jin, Mingjun Ding, Butuo Li, Bing Zou, Yiyue Xu, Bingjie Fan, Wanlong Li, Jinming Yu, Linlin Wang, Been-Lirn Duh

Abstract

Lymph node metastasis (LNM) is the most critical prognosis factor in esophageal squamous cell carcinoma (ESCC). Effective and adaptive integration of preoperative CT images and multi-sourced non-imaging clinical factors is a challenging issue. In this work, we propose a graph-based reasoning model to learn new representations from multi-categorical clinical parameters for LNM prediction. Given CT, general, diagnostic, pathological, and hematological clinical information, we firstly propose a graph construction strategy with category-wise contextual attention to embed multi-categorical features as graph node attributes. Secondly, we introduce a co-graph attention layer composed of a conventional graph attention network (con-GAT) and a correlation-based GAT (corr-GAT) to learn new representations. Corr-GAT complements con-GAT by difference-based correlations across image regions in global spectral space. Experimental results of ablation studies and comparison with others over 924 lymph nodes demonstrated improved performance and contributions of our major innovations. Our model has the potential to foster early prognosis and personalized surgery or radiotherapy planning in ESCC patients.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87240-3_63

SharedIt: https://rdcu.be/cyl6D

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a graph neural network architecture to use both imaging and non-imaging feature for a better decision making on lymph node metastasis. This introduces a combination of GAT and correlation based GAT along with a category-wise attention module for better integration of information. The authors showed state of the art performance compared to the existing methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea of using graph NN to correlate non-imaging features with imaging features and improve clinical features is interesting and promising. The co-graph and category wise attention module for more effective fusion of information from different categories is a novel idea. paper is well written and sufficient ablation studies are done
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

None
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

In my opinion the results are reproducible
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

One question that I have is authors tried to correlate the non-imaging features with different regions of images through the way they concatenated the node embedding and created the graph. Did the authors observe any correlations with abnormality regions through some sort of saliency visualization potentially?
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea of the paper is novel, the contribution level is at MICCAI level and the results and experiments support the claims.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper proposed a new co-graph attention layer and category-wise attention to integrate image and non-image (multi-categorical clinical factors) information for LNM prediction.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

(1) This paper is written well, and the logic is clear. (2) Different from other fusion methods, the proposed method is novel for image and non-image features integration.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

(1) some unclear description for the notation. for example, how to get the F_m, each row in F_m for each clinical images is equal?
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

This paper is well written, and is good for reproducibility.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The experiments comparison should be more complete. For example, other fusion methods for image and non-image integration (only concat their features? or other approaches).
2. Are there results for non-imaging clinical data only?
3. Did you try other methods for combining F_ct and f_rp? Since, in the paper, repeat f_rp and concatenate with F_ct, the f_rp is in the patient level and the F_ct is a region in the image.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The novelty for image and non-image fusion.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

A graph-based reasoning model to learn new representations from multi-categorical clinical parameters for LNM prediction was proposed. Also, a co-graph attention layer composed of a conventional graph attention network (con-GAT) and a correlation-based GAT (corr-GAT) to learn new representations was introduced.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. A category-wise attention module to adaptively adjust the weights of imaging, general, diagnostic, pathological and hematological clinical factors when constructing graph node attributes has potential in predicting metastases in CT images.
2. The introduced co-graph attention layer which is composed of a conventional GAT and correlation-based GAT could benefit in improving performance of DL imaging models.
3. The proposed architectures were tested on relatively large dataset (~900 images from ~400 patients).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Sharing the source code would have helped for reproducibility. More importantly, demonstrating the efficacy of the proposed method on some public data? I guess the method should be promising on CT images in general, maybe WSIs, IHC etc? Why not to report an experiment on some publicly available data?
2. It’d be beneficial to report confidence intervals for the ablation studies. Also, maybe adding AUC curves in Appendix.
3. Maybe adding a few CT examples, if possible? More information on the samples, patients demographics, lymph nodes?
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Sharing the source code would have helped for reproducibility. There are no validation on public data so hard to evaluate how reproducible are the results. Maybe testing/training/transfer learning the proposed models on public data (for example, https://wiki.cancerimagingarchive.net/display/Public/CT+Lymph+Nodes) could help the results to be more convincing and reproducible?
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Sharing the source code would have helped for reproducibility. More importantly, demonstrating the efficacy of the proposed method on public data? I guess the method should be promising on CT images in general, maybe WSIs, IHC etc?
2. It’d be beneficial to report confidence intervals for the ablation studies. Also, maybe adding AUC curves in Appendix.
3. Maybe adding a few CT examples, if possible? More information on the samples, patients demographics, lymph nodes?
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Interesting graph-based attention layer composed of a conventional graph attention network (con-GAT) and a correlation-based GAT (corr-GAT) to learn new representations was introduced, could be an interesting architecture to explore more.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The paper proposes a co-graph of imaging and non-imaging parameters, jointly embedded in graph nodes and fed in two graph attention networks that learns new node representations. This is used to predict lymph node metastasis classification. The originality resides in using a category attention modules for the integration of joint information.

One reviewer indicates novelty in using a co-graph and category-wise attention module for an improved fusion of imaging and non-imaging information.

A second reviewer indicates the method as being different from other fusion methods, also highlighting novelty, but also indicates a lack of sufficient comparison with other fusion approaches for imaging and non-imaging, without suggesting references. Perhaps the authors could prepare a response whether to why no other fusion method, beyond its own variants, has been chosen?

A third reviewer also highlights the benefits of the proposed integration of imaging and non-imaging data via a graph attention mechanism, but would have preferred a comparison on a public dataset for reproducibility, and illustrations on CT images.

All three reviewers have a general consensus of appreciated novelty of fusing imaging and non-imaging data via a graph attention mechanism. Two also indicate the concepts of the category-wise attention module as novel. However, one concern to be addressed is the choice of their comparison setting. Improvements could be on improving the comparison with other fusing methods of imaging and non-imaging data.

For these reasons, I believe the paper proposes a contribution to the field with a new method fusing imaging and non-imaging data via a co-graph with a graph attention mechanism, but doubts should be lifted with a response on the motivation of the comparison choices. Recommendation is towards an invitation for a Rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We thank the Area Chair and reviewers R1-R3 for appreciating the novelty and providing constructive comments. We first respond to R2 about the comparison choices (as requested by the Area Chair), followed by responses to other comments from all reviewers.

Comparisons with other fusion models are given in Table 3 and discussed in the first paragraph on Page 8. We chose model [7] because it is a widely used benchmark Radiomics model that integrates imaging features and clinical parameters by multivariable logistic regression. Model [9] is a deep learning model which transforms non-imaging clinical data to an image before fusion with imaging data by CNN. Our model uses a graph neural network to derive new representations in an irregular domain. The comparison can demonstrate the effectiveness of novel feature learning in non-rigid like feature space.

R2 suggested adding results using non-imaging clinical data only. For our dataset, non-imaging clinical data is at the patient level instead of the lymph node (LN) level. Since each patient has 1 to 6 LNs (as noted in Line 6 Section 2) with different class labels, the identical patient-level class label can not be obtained. Thus, we did not perform non-imaging clinical data only experiments.

R3 questioned why not report experiments on a public dataset. As far as we know, there are no public LN prediction datasets with both imaging and non-imaging data. Hence, we did not perform this in the current paper. We thank R3 for providing a public CT LN dataset. Since the dataset has CT images only, we are pursuing a future investigation to pre-train our image encoder using the provided public data.

Other comments:

R3 also asked for more information on samples, patient demographics and LN. In our paper, such statistical information is provided by Section 2, Paragraph 3. We will add a figure with CT samples to Section 2 as requested by R3.

The first line following Eq (1) explains the notation F_m requested by R2. F is explained by the second line on Page 5 and illustrated by Figure 1(c). We confirm that the number of rows in F_m, which is the number of graph nodes, for each clinical image is equal.

R2 asked whether we tried other methods to combine F_ct and F_rp. We acknowledge that we have not tried other approaches because this is beyond the focus of the technical contributions of this paper. However, we appreciate the suggestion and leave this for future explorations.

We appreciate R1’s suggestion to observe correlations between non-imaging features and different regions of images. We will investigate the visualization in future work.

Thank you!

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have clarified concerns on the choice of comparisons, on using non-imaging data as well as other minor misunderstandings. These clarifications should be integrated in the text. The authors proposes to add an illustration on CT, which would be beneficial, perhaps by remodeling the blank spaces in all three tables, but without sacrificing the clarification on the comparisons. This is a realistic touch up in my opinion.

For these reasons, recommendation is toward Acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The reviewers all think positively about this paper. The novelty includes using co-graph for fusion and using categorical attention. The major concern is about the comparison setting and the rebuttal seems to be addressing the concern to a satisfying degree.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have addressed all the minor concerns. The paper is well written and interesting.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

back to top

Co-Graph Attention Reasoning based Imaging and Clinical Features Integration for Lymph Node Metastasis Prediction