Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Kongming Liang, Kai Han, Xiuli Li, Xiaoqing Cheng, Yiming Li, Yizhou Wang, Yizhou Yu

Abstract

Quantitative estimation of the acute ischemic infarct is crucial to improve neurological outcomes of the patients with stroke symptoms. Since the density of lesions is subtle and can be confounded by normal physiologic changes, anatomical asymmetry provides useful information to differentiate the ischemic and healthy brain tissue. In this paper, we propose a symmetry enhanced attention network (SEAN) for acute ischemic infarct segmentation. Our proposed network automatically transforms an input CT image into the standard space where the brain tissue is bilaterally symmetric. The transformed image is further processed by a U-shape network integrated with the proposed symmetry enhanced attention for pixel-wise labelling. The symmetry enhanced attention can efficiently capture context information from the opposite side of the image by estimating long-range dependencies. Experimental results show that the proposed SEAN outperforms some symmetry-based state-of-the-art methods in terms of both dice coefficient and infarct localization.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_41

SharedIt: https://rdcu.be/cyl8B

Link to the code repository

N/A

Link to the dataset(s)

https://github.com/GriffinLiang/AISD

Reviews

Review #1

Please describe the contribution of the paper

Symmetrical brain images are used to identify certain diseases like Alzheimer and infarct from normal physiological changes. After acquiring symmetrical parts of the brain, both parts are compared, to find the existence of the disease. But in reality, due to movement artifacts, symmetrical images are hard to acquire. So this paper proposes [i] a kind of registration technique to acquire symmetrical images, followed by [ii] an attention-based model which captures in-axial and across axial symmetry context to do segmentation of ischemic infarct disease. The paper claims to beat the SOTA in terms of the dice coefficient.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

A two-step disease identification problem with 1) performing registration followed by 2) symmetry enhanced segmentation by training it end-to-end to promote task-specific alignment is a good idea.

The performance of the alignment network (although in an unsupervised setup) is commendable.

The paper is well written

The ablation study is done well.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Some parameter details such as g(.) and h(.) convs are missing.

Novelty is limited because UNet (2d and 3d) based architecture for segmentation and Attention for long-range context modeling are well-established ideas.

Loss function (Dice Loss + Cross-Entropy Loss) details are not explained (Eg: how are the 2 loss terms weighted? Which cross-entropy loss is used - Weighted/Binary/Balanced?).

Although Related work has [3, 11, 15] mentioned, this paper compares the proposed network with [12] and [16] only in quantitative analysis.

Some experimental details like - how were border feature maps handled in SEA network, What is the baseline network in Table 2, are missing.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducible
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Possible improvements in writing:

Citations in Table 1 also would have made it easier to read the paper.

Figure can be more explanatory with notations used in the paper for quicker cross reference.

Middle plane of the input volume is retained as the target image (Section 2.2, 5th line) - the word target image is misleading because it’s usually referred to as ground truth.

Image level and feature level networks can be explained better.

Few other questions need to be addressed:

How is the standard space defined here (page 2, para 2, line 3)?

The dataset used in the experiments already not published? How can one understand if the network is not overfitted etc,.

Symmetry based alignment on cross-axial slices can also be done?

Is the + sign in equation 3 stands for concatenation?

Are the conv operations of theta and phi are 1x1?

How are batches made - patient wise or slice wise? Are slices of a patient’s scan mixed between batches - so treated independently?
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Limited novelty.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors present a deep learning method to quantitative estimate the acute ischemic infarct in CT images of the brain. The method includes an attention module to utilize the symmetry of the images in the estimation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Good use of symmetry properties, both axial and cross-axial, to enhance the information for the estimation.
- Quantitative results show better performance than competing strategies.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The quantitative results are not outstanding, even though the proposed method outperforms other methods.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- The authors plan to share the dataset of the study.
- The description of the method seems sufficiently thorough to be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- It would be interesting to explain some motivation on why using the L1 loss instead of the L2 loss
- Also, it would be nice to know about the clinical advantages of the method: will it save time to radiologist? will it be more precise than a neurologist diagnostic?
- Since the quantitative enhancement provided by the method in Table 1 is not very strong (comparing to competing), it would be interesting to list other advantages (apart from better estimation results) that this method provides with respect to the others.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
- The authors describe a promising method that make use of symmetry. However, the quantitative results are not very significantly improved with respect to other competing strategies evaluated.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This paper proposes a symmetry enhanced attention network (SEAN) for acute ischemic infarct segmentation task. Before an U-shape segmentation network with SEAN, an Alignment Network is added to transforms an input CT image into the standard space. Experimental results on acute ischemic stroke dataset show that the proposed pipeline outperforms state-of-the-art methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The method adopts an unsupervised manner to automatically align the brain images to decrease time complexity.
2. The proposed symmetry enhanced attention module can be easily inserted into the existing network structure to improve the effect of acute ischemic infarct lesion segmentation task.
3. An acute ischemic stroke dataset is newly collected, and will be published for future study.
4. The proposed method has better effect in both dice coefficient and infarct localization.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The paper only compared with baseline methods Unet and HybridUnet, some other works solving acute ischemic infarct segmentation problem should be included.
2. There are open acute ischemic stroke segmentation datasets like ISLES, on which most recent works has reported their accuracy (e.g. [*]). An experiment on open dataset is needed.
3. Some typos or grammatical errors should be carefully checked. U-shape network -> an U-shape network “The we define the structure of the proposed symmetry enhanced attention network”, The -> Then
[*] Kamnitsas, Konstantinos, et al. “Efficient Multi-Scale 3D CNN with fully connected CRF for Accurate Brain Lesion Segmentation.” (2016).
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The proposed module SEAN and Alignment are clearly described, data will be released. Researchers could reproduce the methods and results with some effort.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Conducting experiments on open acute ischemic infarct datasets like ISLES to further prove the effectiveness of the proposed method.
2. Comparing the proposed method with more methods (not limited to symmetry based methods).
3. Fixing some writing typos.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper proposed a novel method for acute ischemic infarct segmentation task, which is easy to follow.
1. A symmetry enhanced attention is applied to improve segmentation result.
2. An alignment network for calibrate an input CT image is faster compared to traditional alignment methods.
3. The paper is generally well written. The overall writing structure is clear and easy to understand.
4. Experiment is insufficient, which is the main weakness of the paper.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors proposed a registration technique to acquire symmetrical images, followed by an attention-based model to captures in- and across-axial symmetry context to segment ischemic infarcts. The proposed method has better performance in both dice coefficient and infarct localization. However, there are some minor questions from reviewers, please address in the rebuttal.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

We thank the reviewers for their valuable comments. We are pleased to note that the reviewers found the proposed usage of symmetry properties is effective (R1, R2, R3) and efficient (R3) for image alignment and feature enhancement. The rest of the rebuttal will focus on addressing the main concerns.

The novelty of symmetry enhanced attention network (SEAN) [R1]. Previous attention-based works compute the response at a position by combining the features at all positions in the same way. Therefore, the extracted features are position-invariant. Instead, SEAN aims to capture the symmetry information and processes the features of the input side and its opposite side using different operations.

Comparison with more related works, such as [3, 11, 15] [R1] and non-symmetry-based methods [R3]. Both [3] and [11] use brain symmetry for patient-level classification instead of lesion segmentation. They model symmetry information by calculating the feature-level differences in the same way as Unet-FT-L1 and HybridUnet-FT-L1 do. We add [15] as one more comparison which achieves 0.525/0.5946/0.5318/0.6742 in Dice/F1/Recall/Precision. For the non-symmetry-based method, we run [17] on AISD using the official implementation and get 0.4126/0.4131/0.6405/0.3049 in Dice/F1/Recall/Precision.

Evaluation on public acute ischemic infarct datasets (e.g. ISLES) [R3]. The data used in ISLES challenges are mainly MRI sequences and CT perfusion (ISLES 2018) while our paper focuses on the segmentation of stroke lesions based on non–contrast CT images. As we mentioned in the introduction, the non-contrast head CT scan is commonly used as the initial imaging because of its wide availability and low acquisition time compared to MRI. To the best of our knowledge, there is no publicly available dataset for acute ischemic infarct segmentation with non–contrast CT images. Since MICCAI is committed to reproducible research, we promise to publish the dataset used in our experiments upon paper acceptance to benefit the community for future research.

Clinical advantages of the proposed method [R2]. Since the density of acute ischemic infarct on non-contrast CT is subtle and can be confounded by normal physiologic changes, the proposed method can support the radiologist perform a more accurate diagnosis. As mentioned in Sec 3.3, the proposed method can also help radiologists to estimate ASPECTS score and speed up diagnosis and treatment.

The quantitative results are not outstanding, even though the proposed method outperforms other methods [R2]. Acute ischemic infarct segmentation on non–contrast CT is a very challenging task. Even using CT Perfusion imaging, the rank-1 method of ISLES2018 can only achieve a dice of 0.5586.

Other details. a) Why use the L1 loss instead of the L2 loss [R1]. We conduct comparison experiments using both L1 and L2 loss. L1 loss is chosen since it achieves better performance. b) Parameter details of g(.) and h(.) [R1]. 1x1 conv with 64 filters. c) Loss function details [R1]. The two loss terms are weighted equally. For CE loss, we set the weights of foreground and background as 5 and 1 respectively. d) Baseline network in Table 2 [R1]. The baseline method is HybridUnet [5] which uses 3D convolutions in the encoding stage. e) Symmetry-based alignment on cross-axial slices can also be done [R1]? The proposed method can perform cross-axial slice alignment. However, the slice spacing of data on AISD is 5mm which is too thick to transform the image effectively. Therefore, we tackle cross-axial symmetry through the proposed symmetry-enhanced attention (SEA). As shown in Fig.2(c), SEA can model long-range dependencies of both local and opposite positions up to T slices apart. Therefore, for a subset of input feature maps, its symmetric region is not needed to be strictly aligned on the same slice but only needs to lie in the T adjacent slices.

[17] Qin, Yao, et al. “Autofocus layer for semantic segmentation.” MICCAI (2018)

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have appropriately addressed all the previous main concerns.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Segmenting stroke lesions directly from non-enhanced static CT scans could have tremendous clinical impact. This work proposes a suitable symmetric attention mechanism to address this problem and improve over state-of-the-art. The initial reviews very relatively positive and the major criticism (not applying their method to MRI or perfusion CT) was well responded to. The authors promise to release their very interesting new dataset. I think this is important and I would in summary recommend acceptance.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The rebuttal addressed most of the concerns. Additional quantitative evaluation experiments are performed such as [15] and [17] which further support the strength of the proposed method. These new results should be included in the camera-ready version. The authors promise to make the data available for research purposes. I am happy with this rebuttal and looking forward to reading the final improved version of this works and the authors’ presentation at the 2021 MICCAI meeting.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

back to top

Symmetry-Enhanced Attention Network for Acute Ischemic Infarct Segmentation with Non–Contrast CT Images