Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Jiancong Chen, Yingying Zhang, Jingyi Wang, Xiaoxue Zhou, Yihua He, Tong Zhang
Abstract
As an important scan plane, four chamber view is routinely performed in both second trimester perinatal screening and fetal echocardiographic examinations. The biometrics in this plane including cardio-thoracic ratio (CTR) and cardiac axis are usually measured by sonographers for diagnosing congenital heart disease. However, due to the commonly existing artifacts like acoustic shadowing, the traditional manual measurements not only suffer from the low efficiency, but also with the inconsistent results depending on the operators’ skills. In this paper, we present an anchor-free ellipse detection network, namely EllipseNet, which detects the cardiac and thoracic regions in ellipse and automatically calculates the CTR and cardiac axis for fetal cardiac biometrics in 4-chamber view. In particular, we formulate the network that detects the center of each object as points and regresses the ellipses’ parameters simultaneously. We define an intersection-over-union loss to further regulate the regression procedure. We evaluate EllipseNet on clinical echocardiogram dataset with more than 2000 subjects. Experimental results show that the proposed framework outperforms several state-of-the-art methods.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87234-2_21
SharedIt: https://rdcu.be/cyl8g
Link to the code repository
https://git.openi.org.cn/capepoint/EllipseNet
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
This paper presents a one-stage anchor-free detection model for automatic measurement of fetal cardiac biometrics in 4-chamber view in echocardiographic screening. To this end, an FCN network is used for the regression of ellipse parameters. Additionally, several losses specific to the ellipse regression are proposed.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- This paper is well-organized
- The idea of using FCN for the regression of the ellipse is novel
- Extensive experiments demonstrate the effectiveness of the proposed method
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Some key annotations are not well described. For example, how a, b are related to E_a and E_b
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Not reproducible as the code and data are not open sourced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- As the data is not open source. A more detailed description of the dataset is important for the evaluation of the proposed algorithm. If the dataset is not well-annotated, the proposed method may not be that trustworthy.
- How the hyperparameters can also be reported.
- As the authors mentioned that IoU loss is sensitive to the predicted angle. It’s interesting to show how sensitive it is.
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
This paper presents an elegant and simple way for the automatic measurement of fetal cardiac biometrics
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
This paper proposed a new network to detect ellipse in cardiac echocardiography.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The idea is novel.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Many details missing. The illustration of model design is confusing.
- Please rate the clarity and organization of this paper
Poor
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Many details missing on model design.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- Why need a square detection for ellipse localization? Since the model is regressing the several parameters of ellipse, why not regressing the ellipse center altogether?
- Fig.1, what is the structure of the backbone network, what is exactly in each stage?
- Eq. 1, why you choose focal loss? What’s the ratio between Y_{xyc}==1 and Y_{xyc}==0? How do you choose the focal loss parameters?
- Eq.2, what are the loss terms of offset and size?
- What are delta_{a, b, \theta} exactly? How do you performed the L1 loss with them?
- What are the E_a and E_b coming from? The paper said they are “predicted”, how?
- Above eq.7 it says the distance IoU loss is used as objective, but why L_det is still the standard L_{IoU}? What’s the discrepency here?
- Please state your overall opinion of the paper
probably reject (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper provides a new design of ellipse desgin, however the illustration of objective funcion lacks major details. It needs further polish and refinement.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
8
- Reviewer confidence
Very confident
Review #3
- Please describe the contribution of the paper
This paper studies using object detectors on fetal echocardiography. It proposes to use ellipses instead of bounding boxes to fit the target objects: heart and thoracic regions. To further improve the model, a rotated IoU loss is introduced. Experiments on a private dataset demonstrate its effectiveness.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- first attempts to use object detector for calculating cardiothoracic ratio on fetal echocardiography.
- compared to baseline solutions, the proposed approaches give considerably better performance.
- ablation study helps to isolate the impact of each technical component.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-
too many mistakes. In Table 1, in the ablation study part, “only IoU loss”, “w/o IoU loss” and “w/ IoU loss”. I guess some of them are rotated IoU loss. Mistakes in experimental results are not tolerable.
-
private dataset is too small. I understand collecting datasets is not easy. One alternative could be cross-validation. By doing so, the paper will be more convincing.
3.from Fig.2, it seems like the ground truth itself is all ellipses. This fact should lead the dataset to favor ellipse-based detectors. If field practice is doing so, you may clarify this fact in your paper.
-
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
the small dataset may be biased, adding risk to reproduce the results
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Please address my concerns listed above. Other than that, I would suggest the authors do another round of proofreading and use more professional terms such as binary classification instead of one-class classification.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The first study using object detector for calculating cardio-thoracic ratio on fetal echocardiography. Very positive results.
- What is the ranking of this paper in your review stack?
3
- Number of papers in your stack
6
- Reviewer confidence
Very confident
Review #4
- Please describe the contribution of the paper
- The authors propose a one-stage ellipse detection framework named EllipseNet to detect cardiac and thoracic regions and calculate fetal CTR and cardiac axis in 4-chamber view.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The authors are the first to perform cardiac biometrics automatically in fetal echocardiography.
- The proposed one-stage ellipse detection framework is interesting. It integrates recent technologies in object detection, e.g. using CenterNet to find object center, using ellipse rather than bounding box and a rotated intersection over union loss to train framework.
- The result on their data show compelling performance compared other methods.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The authors lack comparison with other methods on public available dataset. The lack of reported performance on public dataset makes comparison difficult, and the proposed method may only be limited to the application fetal cardiac biometrics. Moreover, the referenced paper [9] can predict ellipse bounding box as well. However, there is no comparison with it.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors mentioned in reproducibility that they will not have a link to download the data collected. If the data is not made public accessible, there is a high chance the result may not be reproducible.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The authors may want to at least, evaluate the method on DeepLesion dataset and compare to [9] to prove the contribution of the proposed framework.
- Please state your overall opinion of the paper
borderline reject (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed framework combines several recently proposed technology in object detection, which is somewhat innovative, and the result on their data is compelling. However, the lack of result on public dataset (e.g. Deepleision) makes it difficult to compare with other methods the author mentioned (e.g. [9]).
- What is the ranking of this paper in your review stack?
3
- Number of papers in your stack
4
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Given four inconsistent reviews, you are accordingly invited to submit your rebuttals to address the major comments, especially to: 1) provide a more detailed description of the dataset, including data collection and annotation procedures; 2) provide quantitative analysis on how sensitive the IoU loss is, to the predicted angle; 3) explain why a square detection is still needed for ellipse localization? why not regress all the ellipse parameters altogether/directly? 4) provide more details of Fig.1, such as what is the structure of the backbone network, what is exactly in each stage? 5) explain why to choose focal loss in Eq. 1 and how to choose focal loss parameters. 6) explain what are the loss terms of offset and size in Eq.2. 7) explain why dIoU is not presented in Eq.7, since it is just be introduced above the equation. You may also need to explain the difference between IoU and dIoU in this paragraph/equation. In addition, more experiments (on public datasets) are required by the reviewers; however, these cannot be provided in your rebuttals. The authors may add a comparison to [9] on DeepLesion dataset (suggested by reviewer #6) or report results of cross-validation (suggested by reviewer #5).
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
9
Author Feedback
We thank Reviewers (R2,4,5,6) and Meta-reviewer (RM) for their constructive and valuable feedback. Reviewers rated this work as the first to perform cardiac biometrics automatically in fetal echocardiography with considerably good performance. The main concerns are the network details and reproducibility. We will open-source our codes with trained weights. Please find our point-to-point responses below.
As requested by R2 and RM: the US images are collected and annotated by fetal cardiologists using an offline ellipse annotation tool provided by VGG Image Annotator (https://www.robots.ox.ac.uk/~vgg/software/via/). All our annotations were double-checked by two specialists. More details about the participants and scanners have been described in Section 4.1.
Regarding the IoU and dIoU settings and the sensitivity to predicted angles as requested by R2, R4, and RM, we just use the dIoU metric in this paper and will update all the presentations (Eq. 7) to avoid misunderstanding. Based on our empirical observation, the dIoU loss is essential for the angle regression, especially when the difference of the long and short axes of the ellipse are large.
R4 criticized the missing details in the model design, in fact, our contribution is to propose the EllipseNet within the elegant one-stage detection architecture that was used in both CenterNet and CircleNet, where the backbone is DLA-34 network [14] as stated and referenced in Section 4.2 and Fig.1. The parameters of the focal loss used in this paper are identical to those used in CenterNet and CircleNet. While the focal loss has been extensively proved to be essential for one-stage detection networks. As for the loss terms of the offset and size, we use the L1 loss as described in the paragraph above Eq.1. Regarding delta_{a, b, \theta} in Eq.3, the L1 loss was routinely calculated between the prediction and the ground truth. The idea to introduce such transformations is to constrain the results within the range of [0,1], which is the same setting as studied in Ref[12]. While E_a and E_b are identical to a and b in this paper, we just formulated E_* to represent ellipse.
R5 questioned our ablation studies in terms of the rotated IoU loss, while we used the rotated IoU in all the experiments in this paper and certainly do not have any mistakes in the experimental results. We just use the IoU term for brief presentations. We will add rotated before IoU in all the statements to avoid misunderstanding. Regarding the dataset size, we believe it is comparable to other SOTA automatic biometric works. As for the ellipse annotation, it is a standard procedure in fetal echocardiography examinations and biometrics as stated in Sec 1.
The only concern proposed by R6 is the validation in the public dataset. However, to the best of our knowledge, there is no similar dataset. Regarding the DeepLesion dataset, we found out that its annotations cannot be formulated in ellipses in our experiments. As the cross annotations in Deeplesion are not orthogonal. We tested Ref [9] in our dataset, however, the results are much worse than ours and those presented in Table 1. We guess it is because the network designed in Ref [9] is in favor of recall rate rather than precision.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
As the primary AC, I summarized the major concerns from reviews, however, the authors’ response doesn’t address these problems well, especially on the IoU and dIoU. Since many essential details are missing in the original submission and the response on the key module doesn’t convince me, I recommend rejecting this paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
12
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The reviews have acknowledged the novelty of the presented idea in this paper, as well as its effectiveness. In their rebuttal, the authors have provided many technical answers and details, that seemed to be lacking in the first place.
Therefore I recommend acceptance for this paper.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
2
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors addressed the main points of criticism where, according to my opinion, the major criticism was only the lack of comparison with SOTA methods. Although I am not convinced that the authors could not have done a better job in comparison, the evaluation of the approach [9] in addition to the two original comparisons should be sufficient to publish the manuscript at the MICCAI conference.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
9