Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Tianchen Wang, Zhihe Li, Meiping Huang, Jian Zhuang, Shanshan Bi, Jiawei Zhang, Yiyu Shi, Hongwen Fei, Xiaowei Xu
Abstract
Patent foramen ovale (PFO) is a potential separation between the septum, primum and septum secundum located in the anterosuperior portion of the atrial septum. PFO is one of the main factors causing cryptogenic stroke which is the fifth leading cause of death in the United States. For PFO diagnosis, contrast transthoracic echocardiography (cTTE) is preferred as being a more robust method compared with others. However, the current PFO diagnosis through cTTE is extremely slow as it is proceeded manually by sonographers on echocardiography videos. Currently there is no publicly available dataset for this important topic in the community. In this paper, we present EchoCP, as the first echocardiography dataset in cTTE targeting PFO diagnosis. EchoCP consists of 30 patients with both rest and Valsalva maneuver videos which covers various PFO grades. We further establish an automated baseline method for PFO diagnosis based on the state-of-the-art cardiac chamber segmentation technique, which achieves 0.89 average mean Dice score, but only 0.70/0.67 mean accuracies for PFO diagnosis, leaving large room for improvement. We hope that the challenging EchoCP dataset can stimulate further research and lead to innovative and generic solutions that would have an impact in multiple domains. Our dataset and code are released to the public.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87231-1_49
SharedIt: https://rdcu.be/cyhV6
Link to the code repository
N/A
Link to the dataset(s)
Reviews
Review #1
- Please describe the contribution of the paper
The paper present a new dataset allowing patent forament ovale (PFO) diagnosis to be done from echocardiography. Moreover the authors suggest also a method to automatically evaluate this pathology. The methodology is innovative but the database stay small with only 30 cases with PFO.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The proposed dataset is a new one, dedicated to a specific pathology, then with an important added value in the medical domain. The suggested pipeline for grading the PFO is also well-designed.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The number of cases with PFO in the dataset is very low (only 20 cases), and globally the size of the dataset is small (30 cases). We can expect more cases from a pathology that is relatively common.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
No problem about the reproducibility because the dataset and the code are released to the public. However, it is not clear if the annotations are provided.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The main drawback being the size of the dataset, they must try to increase the number of cases. Moreover, during the discussion of the results, we do not know if the wrong classification cases with VM videos are the same with rest videos. According to the authors, it is possible to consider both videos automatically to improve the diagnosis. Maybe they can also discussed about the possibility to do the diagnosis without the segmentation step (as it is done visually).
- Please state your overall opinion of the paper
strong accept (9)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The database and the methodology try to answer to a real clinical question.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
4
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
The paper introduces a dataset for Patent foramen ovale (PFO) diagnosis in contrast transthoracic echocardiography (cTTE) (common for cryptogenic stroke). The authors also present an automated diagnostic scheme, the first on this dataset, providing a baseline.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Important dataset and initial code released. Highly positive. Alternative echocardiography acquisitions for each of 30 patients, multiple four-chamber annotations per video. This is a great collection of data I hope to use.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
While the automated method is described in some detail, the experimental setup is left out. I only see “we trained the model only with the labeled frames in EchoCP.” Were the patients split into training/validation/test, for example? Any kind of crossfold validation? One cannot hope to compare their novel methods against the proposed baseline without these details.
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
I have small issues with reproducibility
- are the “predefined microbubble patterns” part of the released codebase
- more detail on the experimental setup would be appreciated in the paper.
- is DynUnet the same thing as nnU-Net presented in the cited [7]? I suppose the code released is not ambiguous…
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Small issues:
-cryptogenic stroke which is the fifth leading cause of death in the United States – cryptogenic stroke appears to represent less than half of all stroke, and pfo-correlated stroke is even less, and separately all stroke is a leading cause of death. Could you maybe just say cryptogenic stroke which is a leading cause…
-For data augmentation, even the number of available annotations is limited, we do not proceed with a heavy augmentation – since the number… maybe. Sentence is confusing.
-regions are not correctly detected that results in an intensity peak in LV – detected, which results in…
-For the bad diagnosis examples in rest videos, although…but – the “but” is unneeded.
-the diagnosis accuracy – either “the accuracy of diagnosis” or “the diagnostic accuracy”
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
A novel dataset for an important problem is the major factor in my recommendation.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
5
- Reviewer confidence
Confident but not absolutely certain
Review #3
- Please describe the contribution of the paper
The paper presents a new database in contrast transthoracic echocardiography for patent foramen ovaled diagnosis. Strategies for diagnosis have great interest in the clinical practice due to the complexity of the disease and quality of images. As well, the paper shows preliminary results, which evidence the challenge to improve the methods to diagnosis the disease, an open door that can be explored with this database
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The novelty of this paper is visualized only in the new database, which can be used to explore new strategies to diagnosis patent foramen ovaled. This diagnosis is a challenge, which has to be better worked to improve the precision in the diagnosis of the disease
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
However, the method for the diagnosis isn’t novel and it have to be better explained, with parameters to be reproducible. Some references are old, authors should update the references. The diagnosis can be compared with images of the state of the art, those databases introduced in the paper, namely CAMUS, CETUS, Echonet-dynamic.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Parameters of the method are not presented in the paper. It is problem to reproduce the results
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The diagnosis can be compared with images of the state of the art, those databases introduced in the paper, namely CAMUS, CETUS, Echonet-dynamic, in order to see the importance of the database of contrast transthoracic echocardiography images (cTEE).
- Please state your overall opinion of the paper
probably reject (4)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper must be improved, a comparison with other databases could be useful to evidence the importance of the cTEE images. As well, a more detailed description of the method, parameters, etc., in order to reproduce the results.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The authors present a novel dataset for a challenging clinical problem (assessment of patent foramen ovale), which does not currently exists (at least publicly). Addtionally they propose a method for automatic assessment. The paper is well writen, and reviewers all agree that a new dataset is a worthy contribution. The main concern of the reviewers are the size of the dataset (30 videos) and the lack of comparison or contextualzation with respect to other available datasets for cardiac imaging, e.g. CAMUS or echonetdynamic, which are substantilly larger.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
3
Author Feedback
N/A