Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Xinghua Ma, Gongning Luo, Wei Wang, Kuanquan Wang

Abstract

Coronary artery disease (CAD) has posed a leading threat to the lives of cardiovascular disease patients worldwide for a long time. Therefore, automated diagnosis of CAD has indispensable significance in clinical medicine. However, the complexity of coronary artery plaques that cause CAD makes the automatic detection of coronary artery stenosis in Coronary CT angiography (CCTA) a difficult task. In this paper, we propose a Transformer network (TR-Net) for the automatic detection of significant stenosis (i.e. luminal narrowing > 50%) while practically completing the computer-assisted diagnosis of CAD. The proposed TR-Net introduces a novel Transformer, and tightly combines convolutional layers and Transformer encoders, allowing their advantages to be demonstrated in the task. By analyzing semantic information sequences, TR-Net can fully understand the relationship between image information in each position of a multiplanar reformatted (MPR) image, and accurately detect significant stenosis based on both local and global information. We evaluate our TR-Net on a dataset of 76 patients from different patients annotated by experienced radiologists. Experimental results illustrate that our TR-Net has achieved better results in ACC (0.92), Spec (0.96), PPV (0.84), F1 (0.79) and MCC (0.74) indicators compared with the state-of-the-art methods. The source code is publicly available from the link (https://github.com/XinghuaMa/TR-Net).

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_50

SharedIt: https://rdcu.be/cyhV7

Link to the code repository

https://github.com/XinghuaMa/TR-Net

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper presents a transformer-based network for coronary plaque classification from CCTA data into significant and non-significant stenosis. The authors extract patches along the coronary centerline, extract features using a feed-forward CNN and then feed a transformer network for classification. In-house data annotated by an expert used for training and evaluation. Proposed approach compared to previously published approaches.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main novelty is the use of a transformer network for the classification. This way the authors leverage global information rather than simple classification based on local information along the centerline. Results suggetsed that on the authors dataset, their method performs the best in terms of F1 score.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The clinical value of assessing significant stenosis is somewhat practically limited. Nowadays advanced methods including CT-FFR are used to assess the functional significant of coronary stenosis. These methods requires accurate segmentation. Multiple deep-learning methods were suggested for the segmentation task. From their results, the classification of significant vs non-significant stenosis is quite straightforward. Therefore, the clinical value of the proposed approach is unclear.
2. The transformer architecture is not really novel, though its application for this task is novel and with demonstrated added value, at least on the authors dataset. Therefore additional evalaution on an objective and publicly available dataset is required. For example the Roterdam coronary stensosis assessment dataset.
3. Tehcnical details of the CCTA acquisition and reconstrution protocol are missing.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Moderate. The authors did not disclose their data or at least the technicla parameters of their data. Some of the information about hyper parameters is missing.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Focus the classification task on a clinically-meaningful information such as plaque-type, functionally significant vs functionally non-siginificant coroanry lesions, etc
2. Complete the missing information described in section4 above.
3. Use publicly avialable dataset for comparison with previously published methods.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

See the comments above,
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

The authors proposed a new approach for detection of stenosis in CCTA of coronary arteries. The novelty of the approach is the use of the Transformer network, used in NLP, but to apply it to coronary segmentation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The idea of modelling successively the image information and to classify the sequence analysis is very appealing.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

This paper address an important problem in cardiology/radiology, but the main focus seems to be the basic application of the Transformer network architecture, which is technologically sound, but requires a better validation.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The reproducibility of the paper is feasible.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

See above my comments. The authors should consult with a cardiologist to ensure their proposed system is clinically sound, especially if the goal is the automatic detection of stenosis.

The clinical relevance (first sentence: “serious threat to human health around the world”) should be revised. The wording is very strong, considering other diseases, and could mislead the reader.

The title is confusing, what is a significant stenosis? There is a clinical definition of a stenosis and this should guide the significance of a stenosis. This should be addressed in the paper as well.

The submission contains many strong wording, which makes the reading difficult and often misleading. Some examples: “(…)irreplaceable role(…)” “(…) from more macro perspectives(…) ”

Page 3, first paragraph: The explanation of the Transformer architecture is very weak and superficial. The authors should revise and explain better the use of the Transformer network and its link to NLP.

The comparison with a 2D/3DRCNN is interesting. Have the authors compared their results to the methods of the coronary segmentation challenge? How about a comparison with non-sequential methods such as U-Net, etc?

Why 12 encoders?

Fig. 5 is confusing and the color should match to illustrate the correlation between the detected and the stenosis identified by the radiologist. Moroever, the results shows the sensitivity and specificity of the system, but in this particular application, the detection of the stenosis is much more clinically relevant. The authors should provide more information about the overall detection of the stenosis rather than a technical comparison with RCNN.

A mention should be provided about the provenance of the data (ethics approval, etc).
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The use of RCNN and Transformer network for image segmentation is not new. The idea of modelling the sequence is very appealing, but could have been more effectively performed by validating with clinicians (formal definition of stenosis, find different patterns of plaque formation for instance, etc).
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors present a system that marks precomputed and densely sampled coronary centerline points with a binary classification, which is significant or non-significant stenosis, from CCTA data. To this end a in transformer network is deployed. The results are mostly better than those of other state-of-the-art methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Novel translational application of transformer networks in the medical domain
- Use of global and local cues for detection purposes
- Good use of figures and images. Figure 3 is very good. It concisely shows how the deployed system works.
- Comparison to other state-or-the-art approaches
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The design choices could be better motivated. Why looking at one vessel as a whole via transformer networks is reasonable in this use case at all?
- Evaluation on in house data only
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The code will be made available on the internet. It should be possible to reproduce the trend of the results on other data collections.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Abstract: “complete” -> “completing”
- Introduction
- I don’t agree with the statement the CCTA plays an irreplaceable role in intervention preparation. As of today, most often patients are directly sent to Cathlab after native scan calcium scoring.
- “analyzing coronary plaque” – Is its composition to be analyzed in this work? It should rather be put in relation to what is actually analyzed, which are stenosis grades.
- Figure 1 caption
- The authors may want to avoid qualifiers like “extremely”, etc.
- Its common practice to make (a) and (b) follow the caption like “The HU value of non-calciﬁed plaque is similar to adjacent tissue (a), and types and shapes of plaques are complicated (b).”
- I cannot agree with the following statement “Generally, CCTA scans require time-consuming and laborious image post-processing before being used for diagnosing lesions.” I know of clinicians directly working with the axial slices scrolling back and forth to identify the lesions. Aligning the MPRs with the longitudinal axis of the lesion cannot be considered laborious image post-processing.
- Computer vision techniques to replace manual image post-processing? Is this really the main purpose of automated systems?
- What exactly makes the diagnosis of CAD? From my point of view the overall process of diagnosis is by far not limited to reading CCTA data.
- I would not talk of calcification vs. non-calcification in terms of plaque shapes but rather in terms of plaque composition. The language should be more precise with regards to this.
- State-of-the-art review
- What exactly is the mentioned lengthy manual interaction that is required in most other semi-automated approaches?
- Did Denzinger et al. really predict the FFR value in their experiments?
- Usage of English finite and definite articles could be double-checked.
- Experiments
- Slightly imprecise language: “We extracted the MPR image of the main coronary artery branches in each CCTA scan” Do the authors mean the stack of MPRs associated with straightened curved planar reformations?
- On which level the data is subject to 10-fold cross validation? Is it on patient-, centerline-, or center voxel-level? It could be better explained what makes up the test, training and validation set.
- Could FROC curves be computed?
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

A very relevant clinical scenario is addressed. The proposed method contains true novelty and is thoroughly compared to other state-o-the art approaches where it mostly achieves higher accuracy.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presents a method for detecting and classifying stenosis in CCTA of coronary arteries. The classification result can be either significant or non-significant stenosis. The method uses the transformer network in the coronary segmentation. R1 finds that the novelty comes from using the transformer architecture in this application (e.g., the use of global cue for detection is possible), however the transformer itself is not new. R1 further comments that the clinical value of the proposed method is still not clear as there are already some existing advanced methods, e.g., CT-FFR. And although the authors’ dataset results are good (e.g., the best in terms of F1 score), more evaluations on the publicly available dataset are needed to validate the method and compare with other methods. R2 comments that the application of RCNN and transformer is interesting. But it seems more validations with clinicians are needed. R3 states that the application of the transformer network and the combined use of global and local cues are interesting in this medical domain. But the design choices could have been better described. The method comparison is performed on the authors’ dataset. After reading the comments, my recommendation is given in Q3 and that the authors could consider addressing the concerns on the clinical value of the proposed method, the method evaluation on the in-house dataset, and design choices, on top of the other concerns raised by the reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

Both R2 and R3 directly accepted this paper, and are very positive and rate that the proposed method “is very appealing”, “address an important problem in cardiology/radiology”, “A very relevant clinical scenario is addressed” and “contains true novelty and is thoroughly compared to other state-of-the-art approaches”.

R1 rate that “application for this task is novel”, but he underestimated the clinical value of our method due to some misunderstandings. The main concerns can be summarized as three points: 1) The clinical value; 2) The method evaluation on the in-house dataset; 3) The design choices.

(1) R1 misunderstand the clinical value, because R1 consider that our task and CT-FFR are same:

a) The clinical value of the proposed method is obvious, as proven in many high-level papers, such as (Zreik, TMI 2019), (Tejero-de-Pablos, MICCAI 2019), (Denzinger, MICCAI 2019), etc, in the field of automated stenosis detection tasks. Both R2’s “address an important problem in cardiology” and R3’s “A very relevant clinical scenario is addressed” also can prove the value of this task.

b) The CT-FFR and the proposed method are two totally different tasks. As stated in R1’s “CT-FFR require accurate segmentation”, the procedure of CT-FFR is not “straightforward”. CT-FFR consists of multiple tasks such as coronary tree segmentation, left ventricular mass estimation, microcirculation resistance estimation, simulation based on computational fluid dynamics (CFD), and physiologically plausible assumption, etc. (Taylor, JACC 2013; Tang, JACC: Cardiovascular Imaging 2020), and each task is not well addressed to now. Besides, the accuracy and computational efficiency of CT-FFR is low due to the complex computation procedure, the microcirculation lesions and myocardial scars (Caterina, JACC 2012; Brian S. Ko, JACC 2020). For the above reasons, CT-FFR still has big space to be improved. What’s more, CT-FFR and the proposed method are two parallel and complementary technological tasks. The proposed method is high-efficiency and also has the potential to be used to improve the FFR estimation task in the future.

(2) Currently R1’s “Evaluation on an objective and publicly available dataset” is impractical. “The Rotterdam coronary stenosis assessment dataset” mentioned from R1 is currently the only public dataset suitable for the detection task of coronary artery stenosis. The website that provided the evaluation for the dataset (http://coronary.bigr.nl/stenoses/) is no longer supported, as the notice on the website: “Warning: this challenge website is not supported anymore, and may be taken off-line in the near future”. In the experiment part, we have completed the comprehensive evaluation on various indicators as objectively as possible. In the future, the used data will be accessible to build an open benchmark for the progress of this field.

(3) Thanks for R2 and R3 give the higher score for this paper. Minor revision to highlight the design choices for further readers is practicable, though design choices had been explained clearly especially on Page 3: “To ensure that the model can learn the semantic features of entire coronary artery branches… we introduce Transformer into our method to analyze MPR images from more macro perspectives…Transformer employs an attention mechanism to capture global context information to establish a long-distance dependence on the target….”, and on Page 5: “ a coronary artery branch may have multiple plaques, and each plaque affects the blood flow velocity in the patient’s coronary lumen. Therefore, analyzing the potential relationship between plaques at different locations is very valuable for clinical diagnosis… “. Transformer exactly has the ability of modeling the global context of one vessel, because the multiple self-attention structures in Transformer encoder employ a triplet to obtain the dependencies of semantic sequence in different representation subspaces.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I have read the rebuttal. Most of the concerns have been addressed in the rebuttal. My review is given in Q1.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors present a system based on transformer to classify significant or non-significant stenosis from CCTA data. The reviewers are positive about the practical value of the work, but raised concerns about novelty. The authors have properly address the questions in the rebuttal.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes a novel Transformer-based network for automatic detection of significant stenosis in CCTA data of coronary arteries, by incorporating both local and global information. The paper addresses a relevant problem in cardiology, and results demonstrate improved accuracy compared to other state-of-the-art methods. Major concerns from reviewers regarding evaluation on in-house data instead of public dataset, and the clinical value of the proposed method, have been addressed in the rebuttal.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

10

back to top

Transformer Network for Significant Stenosis Detection in CCTA of Coronary Arteries