Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Josquin Harrison, Marco Lorenzi, Benoit Legghe, Xavier Iriart, Hubert Cochet, Maxime Sermesant

Abstract

Atrial fibrillation (AF) is a complex cardiac disease impact- ing an ever-growing population and increases 6-fold the risk of thrombus formation. However, image based bio-markers to predict thrombosis in presence of AF are not well known. This lack of knowledge comes from the difficulty to analyse and compare the shape of the Left Atrium (LA) as well as the insufficiency of data that limits the complexity of models we can use. Conducting data analysis in cardiology exacerbates the small dataset problem because the heart cycle renders impossible to compare images taken at systole and diastole time. To address these issues, we first propose a graph representation of the LA, to focus on the impact of pulmonary veins (PV) and LA Appendage (LAA) positions, giving a simple object easy to analyse. Secondly, we propose a meta-learning framework for heterogeneous datasets based on the consistent represen- tation of each dataset in a common latent space. We show that such a model is analogous to a meta-classifier, where each dataset is charac- terised by specific projection in a common latent space, while sharing the same separating boundary. We apply this model to the graph represen- tation of the LA and interpret the model to give novel time-dependant bio-markers related to PV and LAA configurations for the prediction of thrombosis.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_52

SharedIt: https://rdcu.be/cyhV9

Link to the code repository

https://github.com/Inria-Asclepios/mcvc

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a graph representation of the Left Atrium (LA), to focus on the impact of pulmonary veins (PV) and LA Appendage (LAA) positions. It proposes a meta-learning framework for heterogeneous datasets based on the consistent representation of each dataset in a common latent space.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Analysis in the latent space
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Shortened state of the art
- Weakness of the comparison
- The multi-task process is not well defined
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Not reproducible
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The paper needs additional language spell checking
2. The state of art is very limited
3. What is the origin of the database used, from which hospital?
4. The use of LDDMM framework needs some details and justifications
5. It is not clear why and how deformetrica software is used
6. The interest of using synthetic database is not clear
7. In the synthetic experiments how is calculated the confidence level
8. Fig 2 needs more analysis
9. The comparison with interpretation algorithms needs more details
10. The multi-task learning process is not clearly defined in this work
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The results still not convincing The authors use existing softwares
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

In this paper, the authors present a technique for graph representation of left atrial (LA) shapes and subsequent interpretable classification. Their technique uses a VAE-like approach to generate a shared latent representation for shapes obtained during both systole and diastole. The presented approach lends itself to the application of “interpretation algorithms” to assess which graph features (i.e. relative position of the nodes) are associated to a specific class (which in this case are two: pre-thrombosis and controls).
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The paper is sufficiently clear and understandable.
2. The approach is somewhat interesting, in particular the idea to use a “multi-head” VAE to predict labels from a common latent space. I also find interesting the results obtained on synthetic data with noisy labels.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Complete lack of literature review: this section is completely absent. It appears that no paper on shape analysis, both in general and applied to atrial shapes, is cited. In reality, there is quite a lot of work on this topic, with several recent studies using deep learning (e.g. Y. Fang et al. “3D deep shape descriptor,” CVPR 2015; Biffi et al. “Explainable Anatomical Shape Analysis through Deep Hierarchical Generative Models,” IEEE TMI 2020). This makes it difficult to assess the authors’ awareness of the field, and how they position their paper in relation to it.
2. Lack of competition in the experimental analysis: no alternative approach for (interpretable) shape classification has been implemented and compared to the proposed one. As a consequence, it’s very difficult to assess the relevance of the proposed approach. Of note, I’m worried that a simple PCA applied to the node coordinates could have provided results similar to the ones obtained by the authors.
3. Limited novelty: it’s mostly the combination of proposed VAE-based architecture, graph-representation for LA shapes and use of “interpretation algorithms” to appear novel rather than the single parts.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors seem to have used a private dataset. No statement about releases of code are included, but some details regarding the architecture of the implemented neural networks and training process are present. As a consequence, the degree of reproducibility is medium-low.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. The proposed approach is interesting in some aspects (including for instance the management of noisy labels), but to properly assess its relevance the authors should perform a thorough comparison (experimental and/or based on discussion) with the state of the art in shape analysis.
2. If the authors see the method only applicable to LA shapes, then it would be interesting to add some analysis regarding the clinical relevance of the features predicted as important. If they instead believe it can be adopted in other domains, performing experiments on other datasets (together with other competing approaches) would add a lot of value to this work.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper presents a somewhat interesting approach to the task at hand, but the complete lack of comparisons with the state of the art makes it extremely difficult to judge its relevance in the field, and more simple approaches (like PCA) could potentially provide similar results.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

The paper presents an interesting strategy of classification of pre-thrombosis in cardiac images, which are difficult to automatically evaluate due to the presence of other anomalies. The method is a meta-learning framework for heterogeneous datasets based on the consistent representation of each dataset in a common latent space, using a supervised framework where the common representation is improved in terms of Kullback-Leibler divergence
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper presents a coherent latent space inherited from the model makes it possible to have deep neural network as encoders while conserving the interpretability of simpler models.
- The paper is well written and easy to follow
- The method is evaluated in synthetic and real images
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The method could be compared with a baseline, or other related methods, in order to evidence the performance.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Code is available, the data can be used and parameters and introduced in the paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The Authors presented a good idea to characterize the Left Atrium, by providing a better interpretation of biomarkers of the disease
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is well written, and the idea it is very interesting.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Somewhat confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Summary: Proposes a graph representation of left atrial shapes, using a meta-learning framework for heterogeneous datasets based on a common latent space representation.

Positives:
- Interesting idea of using a “multi-head” VAE to predict labels from a common latent space.
- Interesting results on synthetic data with noisy labels.
- Conserves the interpretability of the model.
- The paper is well written and easy to follow.
Negatives:
- Literature review is lacking. This could be addressed during a rebuttal stage.
- The multi-task learning process needs to be more clearly defined.
- No baseline comparison (even with PCA). Perhaps explain why not.
- Add some discussion about the clinical relevance of the features predicted as important.
- The information to allow reproducibility should be provided.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Author Feedback

We would like to thank the reviewers for their constructive comments. Here are our responses to the main points.

Regarding the literature review: In previous articles published using deep learning frameworks for LA shape analysis, the main methodological contribution was to tackle the complexity of a shape. Our method proposes a simplified representation of this shape in order to apply fast and interpretable learning frameworks. Our main methodological contribution is in the phase independant classification. Regarding meta-learning and multi-task learning specific literature, we did not go extensively into it as, while following the general ideas (e.g aim for better generalisation, robustness to noise, learning on heterogeneous data), we do not use it for the usual tasks of the domains (e.g. one-shot learning). Due to the page limit, we had to keep the literature review small.

Regarding the learning framework: We believe we did not leave any missing part in the method description, but realise that some additional comments can clarify the process. The principal idea is to generalise a variational classifier, i.e. an encoder representing the input into a latent space by a parameterised Gaussian distributed variable, and a classifier to classify elements of this latent space. The generalisation is achieved by forcing multiple encoders to send their respective dataset into a common latent space. In this case, the classifier serves as a way to constrain all the encoders to represent elements of a shared class in the same space, as the classifier is optimised to find a single boundary between classes. We give the theoretical optimisation-based setting for this task in part 2.3. In terms of optimisation procedure, at each iteration we feed n mini-batches to the n encoders sequentially, and perform an optimisation step after all n mini-batches are passed.

Regarding the baseline: We are not aware of methods able to jointly classify multiple small size datasets. As a consequence we did not provide any head-to-head comparison, but rather we emphasised the relevance of our method. Precisely, in part 4, we attempt to perform classification with a single encoder on subsets individually, as well as on the complete dataset, and show that the first method suffers from mode collapse and poor accuracy and the second method still has lower accuracy in addition to losing the systole/diastole distinction when interpreting the results. Following the reviewers’ comments, we performed PCA followed by logistic regression with cross validation and grid search on the number of principal components. We observed a much lower accuracy both on the complete dataset or individual subsets (0.65 at diastole, 0.71 at systole and 0.66 on the complete dataset). We could add this to the paper if accepted.

Regarding the clinical relevance: While a series of studies have focused on characterizing the LAA shape and size to build risk markers, our initial hypothesis is that morphological phenotypes at risk also include a specific orientation of PVs and LAA. This hypothesis comes from the direct impact those orientations have on the blood flow in the LA, itself being the reason for clot formation. The results we obtained show that we can already reach high classification accuracy with only this information. We believe the novel important features we found are ready to use in a robust and automated CT-based risk marker which we aim to build in future work.

Regarding the reproducibility: We intend to provide code for the model, creation and processing of synthetic data, as well as provenance of the dataset. However we have redacted out this part to preserve anonymity. In addition, all hyper-parameters for our learning procedure are given in part 3 of the paper. Any eventual parameter we do not specify are the ones set by default. The only element we cannot provide is the LA dataset due to privacy policy.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Authors justified the lack of comparison against a baseline (no previous approaches do the same thing), although they have now done the PCA experiment suggested by R2. The literature review will remain limited (space), although the authors may add some details about the learning framework. The authors have explained the clinical relevance of their work in terms of formulating risk markers, and will provide code to enable reproducibility.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper presents an interesting strategy for graph representation of left atrial shapes and pre-thrombosis classification. Although the comparison with the baseline was not addressed well, they have answered other main concerns, such as the results of PCA method.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

11

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The lack of a deep literature review and contextualized baseline detract from the work.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

16

back to top

Phase-independent Latent Representation for Cardiac Shape Analysis