Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Yi Lin, Luyan Liu, Kai Ma, Yefeng Zheng
Abstract
Automated methods for Cobb angle estimation are of high demand for scoliosis assessment. Existing methods typically calculate the Cobb angle from landmark estimation, or simply combine the low-level task (e.g. landmark detection, spine segmentation) with Cobb angle regression task, without fully exploring the benefits from each other. In this study, we propose a novel multi-task framework, named Seg4Reg+, which jointly optimizes the segmentation and regression network. We thoroughly investigate both local and global consistency and knowledge transfer between each other. Specifically, we propose attention regularization loss to class activation map (CAM) from image segmentation pairs to discover additional supervision in regression network, and the CAMs can serve as a region of interest enhancement gate to facilitate segmentation task in turn. Meanwhile, we design a novel training strategy to train the two networks jointly for global optimization. The evaluations performed on the public AASCE Challenge dataset demonstrate the effectiveness of each module and superior performance of our model to the state-of-the-art methods.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87240-3_47
SharedIt: https://rdcu.be/cyl6n
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper presents a novel approach for Cobb angle estimation from spine X-ray images. The key idea is to couple a regression network with a spine segmentation network where the two tasks exchange information in form of class activation maps (regression to segmentation) and attention regularization (segmentation to regression). A thorough ablation study and comparisons with previous work demonstrates good results.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The main strength is the clever design of the multi-task learning framework with a well motivate exchange of information. The triangle consistency between related tasks seems to make sense and is potentially interesting for other areas where two tasks are strongly related.
Another strength is the thorough ablation study that provides valuable insights on the individual contributions of the different components.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The main weaknesses relate to statements made about limitations/shortcomings of previous work and how the presented work lives up to the promise of overcoming these issues. There are also some problems with the mathematical definitions, however, these should be easy to fix.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I believe the methodology can be reasonably well reproduced from the provided description. Difficult to say to what level the results would be fully reproducible given the somewhat complex training framework. Experiments have been carried out on publicly available data, so follow-up work should be able to compare with the presented results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
It may be good to provide precision and recall for the segmentation results, to provide a bigger picture of the positive effect on the segmentation performance. From the visual results, one may think that the proposed method reduces false positives in particular, which could be shown when reporting precision and recall in addition to Dice. JA could be left out as it does not add any additional information on top of Dice (the two are directly related).
In the introduction, the authors state three main shortcomings of previous work: (1) susceptibility to small errors in landmark methods, (2) error accumulation in two-stage frameworks, (3) no global optimum in cascaded networks. The paper, however, lacks in clear evidence that these three shortcomings have been addressed. Due to the lack of a clear failure case analysis, it remains unclear how the proposed methods addresses or solves any of these three. While the overall performance seems to improve, one cannot directly conclude whether the method suffers less from these issues.
For example, I do not believe one can claim that the triangle consistency learning is helping to reach a global optimum. This needs to be rephrased. The learning is using stochastic gradient descent which naturally will converge to a local minimum. It is unclear what the authors refer to here with ‘global optimum’.
Second, the proposed approach may still suffer from the issue of error accumulation. If the segmentation network underperforms and/or the regression network provides inaccurate CAMs, the two networks may amplify the errors of each other. A more detail failure case analysis may provide insights on this.
It seems equation (1) relies on a one-hot representation of the labels y_i but this is unclear from the provided definitions.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Interesting multi-task learning approach with sufficient novelty in the methodology that may be of interest for other problems. Thorough ablation study and good results on publicly available data. Shortcomings can be addressed with minor revision.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
This paper proposes a novel multitasking framework to jointly optimize segmentation and regression networks for the estimation of Cobb angles. It leverages CAMs from segmentation to provide additional supervision to regression and CAMs as ROI enhancement to segmentation in turn. Good results have been reported using the AASCE challenge dataset.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- A novel consistency learning framework that incorporates segmentation and regression with an attention regularization module and ROI enhancement gate to boost the performance.
- A triangle consistency learning is designed for end-to-end training which is detailed in Algorithm 1.
- Outstanding experimental setup and presentation of the results.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The authors claim their method is trained end-to-end while it still requires pre-training.
- The results are not very convincing.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
It’s possible to reproduce the results considering the detailed algorithm, challenge dataset, and cod/model are provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- It’s not clear what’s the intuition of choosing the hybrid loss for the segmentation task.
- “We first pre-train the two networks separately for approximately optimized results to speed up the training process.” It the networks are pre-trained, it is not end-to-end anymore.
- “One branch takes the concatenation of the raw image and its corresponding segmentation mask as input” It doesn’t look consistent with Fig. 1.
- What’s the motivation for such three Cobb angles (PT, MT, TL)?
- The authors should add a discussion on the trustworthiness in directly estimating the Cobb angles.
- The authors mentioned Siamese structure, but the relevant paper is not cited.
- “By combining the ROIE gate, the performance can be further improved by 0.07% (the improvement with the segmentation mask as input is more significant with 0.51% boost in SMAPE)” The improvement is marginal and how the significance is verified?
- It looks like the authors put so much effort to make the model complex but results are on par with SOTA.
- It would be interesting to see visual comparison of the spine segmentation results by the proposed method against the methods reported in Table 2.
- Is there any cases when the proposed method might fail in segmenting and/or estimating the Cobb angles.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Even though the results are not very convincing, the proposed method is interesting and worth sharing with the community.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #3
- Please describe the contribution of the paper
This paper proposed a Seg4Reg+ model for automated Cobb angle estimation. The Seg4Reg+ model incorporates segmentation into regression tasks via an attention regularization module and a region-of-interest enhancement gate to boost the performance of both tasks. The claimed contribution lies on: 1) This paper proposes three limitations of the Seg4Reg model in the Cobb Angle Estimation task. It provides an idea for improving the automated Cobb Angle Estimation model. 2) To optimize the two limitations of the Seg4Reg model, the author proposed an improved scheme named Seg4Reg+. It is a consistent learning framework, incorporating segmentation and regression tasks with an AR module and ROIE gate to boost the performance of both tasks. 3) The experimental results supported the effectiveness of the improved model. It also supported each additional module (AR module /ROIE gate/triangle consistency loss) can improve the model.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
This paper proposed a multi-task framework based on seg4reg to mutually reinforce the spine segmentation and the Cobb angle regression networks. The work has a decent novelty and technical depth. It is interesting the method to use the regression task of an attention regularization (AR) module to extract representative features. The AR module uses the class activation map (CAM) and the Global Average Pooling Layer(GAP) to clearly locate the areas of concern for the spine data. Then input it into the ROIE gate to force the segmentation network to pay more attention to the important area. This improvement not only enables the segmentation task to obtain more accurate position information but also optimizes the regression task. On the other hand, a triangle consistency learning scheme well solves the problem that the regression network will fall into the local optimum. The three losses well balance the split task and the regression task and also add important location information. The most special proposed loss is ARLOSS, which regularizes the activation mapping of the output of the two branches through the mean absolute error to ensure the consistency of the CAM so that the regression network focuses on the spine region. Although the other two LOSS also played an irreplaceable role, they were not proposed for the first time. The experimental design part of this paper can reflect the effectiveness of each part of the model improvement. The experimental design is reasonable and comprehensive, which can well support the theoretical part of the paper. First, Ablation Studies to verify the effectiveness of each module in the proposed Seg4Reg+ approach are designed in the experimental part. This is very necessary. Although there were some minor problems in the ablation experiment (which will be discussed in detail later), they did not affect the overall experimental design idea. Secondly, the effectiveness of ROIE Gate on the segmentation task is also compared and verified. Finally, the regression performance of the whole model is compared with the most advanced methods. Although the model proposed in this paper does not perform optimally in all evaluation indexes, these experiments also fully prove that Seg4Reg+ has a further improvement in regression performance compared with Seg4Reg and other similar models.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
There are no significant grammatical errors and typos in the paper. But this paper is not well-organized and written. In the whole article, there is no introduction of related work, only mentioning the improvement of the SEG4Reg model. This prevents the reader from fully understanding the creative work of the paper and its contribution to the relevant field. In the method part of the paper, the description is not sufficient. For example, CAM and GAP methods used in the regression task are only briefly mentioned without other extensions, which is not conducive to the rigor of the paper as a whole. The analysis and explanation of the experimental part of the paper are not specific enough. In Table 1, by combining the ROIE gate, the performance can be further improved. However, the improvement of the regression results indicates that the regression model can improve the segmentation model in turn, which is not reasonable.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Ok
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The improvement of the Seg4Reg model in this paper is novel and. The experimental results also show the effectiveness of the improvement. However, to make the paper more rigorous and sufficient, I boldly make the following suggestions: 1) Added description of the dataset: The authors used the public dataset of the MICCAI2019 AASCE Challenge – 609 anterior and posterior spinal X-rays. The paper addresses adolescent idiopathic scoliosis (AIS) but does not address the age distribution of patients in the dataset. My suggestion is to try to describe the situation of the data set comprehensively in the paper, which is necessary for the evaluation of the model and experimental results as well as the clinical application. 2) For the elaboration and layout structure of the paper, I suggest that the author should include the relevant work section, introduce the relevant research and the relevant basic model, and highlight the creative work that he has done on this basis. 3) In the method part of the paper, the description is not sufficient. My suggestion is to add some descriptions of key nouns. For example, in the model framework in Figure 1, some explanatory words For CAM and GAP modules can be added. This will help the reader understand the structure of the text and improve the rigor of the narrative. 4) In ablation experiments, I suggest that additional indicators of the segmentation effect be added. Different segmentation effects were obtained through two groups of experiments with and without ROIE gate to verify whether the increase of ROIE modules could indicate the promotion effect of the regression model on the segmentation model.
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The improved model proposed in this paper is novel, and the experimental results also show the effectiveness of the improvement.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
3
- Reviewer confidence
Confident but not absolutely certain
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
There is consensus amongst all reviewers on the quality of this paper, all providing enthusiastic comments on the novelty of the proposed multi-task learning framework used to estimate Cobb angles. The reviewers also appreciate the experimental results, showing the effectiveness of the framework with thorough ablation studies and comparison to state of the art methods
Reviewers note however several modifications and improvements which should be addressed in the revised version. This includes but not limited to an improved background review in the introduction, and added technical details to the description of the method. Reviewers would also like to see some discussion on the limited gain in performance of the method.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
1
Author Feedback
N/A