Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Fanwei Kong, Shawn C. Shadden

Abstract

Image-based computer simulation of cardiac function can be used to probe the mechanisms of (patho)physiology, and guide diagnosis and personalized treatment of cardiac diseases. This paradigm requires constructing simulation-ready meshes of cardiac structures from medical image data–a process that has traditionally required significant time and human effort, limiting large-cohort analyses and potential clinical translations. We propose a novel deep learning approach to reconstruct simulation-ready whole heart meshes from volumetric image data. Our approach learns to deform a template mesh to the input image data by predicting displacements of multi-resolution control point grids. We discuss the methods of this approach and demonstrate its application to efficiently create simulation-ready whole heart meshes for computational fluid dynamics simulations of the cardiac flow. Our source code is available at https://github.com/fkong7/HeartFFDNet.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_53

SharedIt: https://rdcu.be/cyhQ8

Link to the code repository

https://github.com/fkong7/HeartFFDNet

Link to the dataset(s)

https://zmiclab.github.io/projects/mmwhs/

https://orcascore.grand-challenge.org/

https://figshare.com/s/2a5de3a2b89a3fb87932

Reviews

Review #1

Please describe the contribution of the paper

This paper proposes a deep + graph learning approach to reconstruct whole heart mesh from 3D volumetric image data. Deep learning is used to extract features from volumetric images, while graph learning is used to regress the control point displacements of B-spline. Segmentation ground truth is used to additionally supervise the right convergence. Validation is conducted on public and simulation data.

Overall, I think this is a good paper, in terms of the presentation and novelty. However, the result section is not convincing.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
Strength:
- This paper is well organized, presented and written.
- The methodology combines a few modules, learning techniques, and losses, if judge from the equations theoretically, each module makes sense and seems contribute.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
Major issues:
- The literature review of related works missed some closely related works. However, I am sorry here that I cannot supply the detailed works, as some of them are from me (to clarify, not only from me). Please carefully check recent years’ MICCAI publication in 2018-2020. You will find a few that should be discussed in your paper.
- The figures in this paper are very close to the figures in [8], even for the colours, which may cause copyright issues. For example, Fig 1 in this paper VS. Fig. 1 in [8]. Fig. 4 in this paper VS. Fig. 3 and 4 in [8].
- 87/15/40+10 is used for the training, validation and testing + evaluation. While no cross validation is conducted. The current results cannot show the stability and robustness of the training of the proposed method.
- For Fig. 2, the authors use simulated data. However, based on my best knowledge, CT usually cannot capture the heart motion very clearly, while MRI are usually used here. Why do the authors not use public MRI data (a few public 4D MRI datasets exist) instead of using simulated CT data?
- There are no quantitative comparisons to other baselines in this paper. The three methods shown in Tab. 1 are ablation studies of the proposed method, rather than a direct comparison to other existing methods. A few methods exist in MICCAI 2018-2020 publications.
- The ablation study is not complete. The authors used quite a few loss functions in the proposed method, however, did not supply a validation to support the authors’ point of using them. Only explaining theoretically is not convincing.
- For Fig. 4, it is hard to see the out-performance of the proposed method over others from only showing qualitative results. The authors can convert all results into same resolution meshes and compare the quantitative results, i.e., distance error
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors supply the code and video demos as attachment. This is very good.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please see item 4.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The result section missed important validation, the current version is not convincing.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

2
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper proposes a deep learning framework that generates simulation-ready whole heart meshes from volumetric image data. Using a U-Net-like architecture as the backbone, features are sampled from the encoding path which are input to a graph convolutional network. The graph convolutional network predicts the displacements of multi-resolution control point grids to deform a template mesh for mesh generation.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. The introduction is well-written, which clearly describes the disadvantages of mesh generation from deep learning segmentation masks.
2. The experimental results look good on the complicated whole heart meshing problem.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Although I understand that the framework is complicated, some critical details are missing in the methodology. For example, in section “Graph Convolution on Control Grid” on p.4, what are f_in and f_out in the context of this paper?
2. This paper seems to be a combination of the graph convolutional network approach from [8] and the B-spline approach from [19], so that the graph convolutional network is used to compute the displacements of the B-spline control points. Nevertheless, as the control point grid is a structured grid in which each control point has 26 spatially dependent neighbors as in 3D images, is it necessary to use graph convolutional network on this problem? Note that the statement on p.4, “therefore, ﬂattening the image feature vectors of a 3D image and use a fully connected layers to predicted control point displacements as proposed in [19] is no longer computationally feasible”, only means that the specific approach in [19] should not be used but does not justify the necessity of using graph convolutional network.
3. It is good to have ablation studies in this paper, but the quantitative (numerical) comparisons with other existing frameworks such as [8] are missing.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Codes are provided as supplementary materials.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. There are no clearly indexed sub-sections.
2. The abbreviation FFD is not explicitly defined.
3. On p.4, second paragraph. Should there be 7, 11, and 17 neighbors at the corner point, edge point, and surface point, respectively?
4. On p.4, section “Graph Convolution on Control Grid”. The important details are missing, for example, what are f_in and f_out in the context of this paper? Are they the coordinates or displacements of the control points? Are they matrices or vectors? What is the size of f_in and does it compatible with the size of matrix L? I understand that the authors cannot list all details in this paper, but more details are necessary.
5. In Fig. 1, how does feature sampling work?
6. The CFD simulation (Fig. 3 and the first paragraph on p.6) is unnecessary. It does not improve the understandability of the paper and does not tell how well the framework performs. The authors can save a page to provide the missing details for the section “Graph Convolution on Control Grid”, or to provide the quantitative results correspond to Fig. 4.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The introduction is well-written but critical details are missing in the methodology. The experiments of comparing different resolutions of control point grids are good, but the quantitative comparisons with existing frameworks are missing.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors propose a neural network to segment CT images of the heart, and directly predict the coefficients of a FFD that are applied template mesh to match that segmentation. This is then tetrahedralised and used for CFD simulations, avoiding tedious manual generation of the computational domain.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-The biggest strength of this paper is the application: they address a current problem that is the automatic generation of patient-specific simulation-ready domains. This is a big real problem that hinders simulation studies in large datasets.
- The experiments are well conducted and reported, the manuscript is easy to read (except the FFD module part, as stated in weaknesses). The figures are insightful and are very helpful for understanding the paper.
- The authors have done an extra effort in the reproducibility aspect by providing the code.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The paper main methodological novelty seems to be in the FFD module, whose explanations are not very clear. My biggest concern there is that the need to use of graph convolution instead of the classical convolution in that submodule is not well justified.
- A minor drawback is that the part in the methodology where they describe the FFD module is not clear enough.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors use public datasets and their code of the NN is included in the datasets, as well as instructions for generating the mesh generation results.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1 – “No prior study has focused on learning to predict surface meshes directly from medical data for the purpose of computational simulations”: While the sentence is true, there are some works that generate 3D surface models from for CT/MRI images, which is in essence what the main part of the proposed methodology does. I am missing a bit the previous literature on the general prediction of surface meshes from medical images for the cardiac, especially in the past using shape models. Examples of previous work predicting shape from images:
- Ecabert et al (2007), “Automatic whole heart segmentation in CT images: method and validation,”
- Attar et al 3D (2019) Cardiac Shape Prediction with Deep NeuralNetworks: Simultaneous Use of Images and Patient Metadata
2- I have trouble understanding why the authors use Graph-convolution to obtain the FFD. The connectivity the authors specify corresponds to the same one as pixels in images, therefore a normal convolution 3x3 would suffice and it is more efficient.

3 – Are the FFD found diffeomorphic?

4 – The description of the Deep FFD module is not very clear, in particular which are the use of image and features and mesh point sampling. What are exactly the point features sampled from the mesh? If they are the coordinates, shouldn’t the parameters of the FFD at the previous resolution level provide this information, without need of sampling?

5 – It is not clear to me how are the segmentation and shape reconstruction tasks in their network interact. What are the test Dice of the segmentation module?

6- In page 5, Generation of 4D meshes for CFD Simulatiuons “Furthermore, as denote by the color maps of vertex IDs, our method consistently deforms the template meshes such that predictions accross different time frames have feature correspondence”. I do not agree that different time frames have point correspondence in CT image, since individual points can not clearly be tracked to the homogeneous wall intensities, low temporal resolution and individual points cannot be correctly tracked, the model mostly follows the contours, and one must be careful since subtle phenomena like LV torsion or akinetic segments due to ischaemia can be lost. Moreover, it is difficult to see and evaluate the color grading in Figure 2. Could the authors specify what they mean?

7– In the CFD simulations, in Figure 3, there doesn’t appear to have deformation in the mesh. Why are the meshes in early-diastole and end-diastole the same?

Minor: 1 - Typo : “control point gird” 2 - The subsection named “Probability Sampling of Images features” in fact refers to sampling of vertex features. 3 - The citation to the tetgen paper needs to be added.

For a future journal publication, please consider:
- A comparison with a manually processed mesh, to evaluate how the smoothing reported affects the simulation would be a good addition to the work.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

I like the fact that the paper addresses a current and important problem which is the preprocessing for numerical simulations, which remains unsolved and is a current bottleneck. Working with a real problem makes it difficult to develop novel ground-breaking methodologies. They successfully combined, and proposed modifications, previous methodologies in a framework that is able to generate simulation ready meshes from images.

My main critique is that the FFD module could be improved (as stated in the comments to the author #2-5), but I think that this is a minor negative point compared to the potential usefulness of this work, and given that the framework itself produces good enough results for the application.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The authors proposed a deep learning-based method to directly construct whole heart mesh from image data. They are trying to solve a important problem which is meaningful in clinic. They tested their methods on a public dataset, and the code and video demos of this paper are attached, which are good! The manuscript is also well organized and presented, though some points still need to be improved (see detailed comments from reviewer). I recommend the authors to try to solve the issues raised by reviewers, such as figure copyright issues, evaluated dataset issue, and the graph-convolution issue for FFD.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

We thank reviewers R1, 2 and 3 for their constructive feedback. [R1,R2]Quantitative validation against existing frameworks: As mentioned in the Intro and by R3, human effort to create patient-specific meshes for simulations remains a critical bottleneck, impeding large-cohort studies and clinical applications of cardiac simulation. Our main objective is to address this problem rather than out-performing the geometric accuracy of the prior whole heart (WH) reconstruction methods, none of which can directly support various simulations. That said, we are able to provide similar accuracy as prior methods. E.g., if you compare our results (Tab 1) with the 4 methods reported in Kong 2021 ([8], [5], [20], [15]), we achieve the best WH HD (26.8mm vs 27.5mm of [20]) and our mean WH Dice (0.88) is similar to [5] (0.888), which won the KiTS19 Challenge. (Note, as mentioned on p.7, such direct comparison is possible since we used the same image pre-processing and training data as in Kong 2021.) We can add this comparison (space permitting or supplementary material if needed).

[R1,R3]Missing related work: We will add Ecabert, Peters 2007 and Attar 2019. These prior methods rely on deforming a pre-defined template or SSM while our method deforms the space and is thus not limited by the chosen template. E.g., we can readily generate meshes of arbitrary subsets of cardiac structures suitable to the application at hand by swapping a new template.

[R2]CFD as not necessary, replace with clearer method description: CFD results are important to show the success of our method in supporting simulations, which is the main purpose. Nevertheless, we will include more details on our graph FFD (see below). We will modify fig 2&3 to include fewer time frames to make more space for methods.

[R1]Usefulness of qualitative comparison: Our main goal is preserving geometric features important for simulations rather than pixel-based geometric accuracy. This can be captured more easily qualitatively in fig4. We produced better vessel in/outlet shapes (important for CFD) than Kong 2021.

[R1]Incomplete ablation studies & lack of cross-validation: We focused on showing the usefulness of our method for simulations within the limited space. Nevertheless, we did these studies (for all cardiac structures) and can add them as supplementary results. Removing L_grid/L_seg, and using 3^3 CNN instead of GCN resulted in lower WH Dice: 0.88 vs 0.84, 0.85, 0.85, respectively. Our method is robust: WH test Dice for 5 models trained w. rand. init & different data splits is 0.878+/-0.002.

[R2,R3]Unclear graph FFD & feature sampling: Some GCN details follow Kong 2021 but we will add more. The dimensions of f_in, f_out and L are d_inN, d_outN and N*N. The mesh feature lengths of intermediate layers match with the numbers in fig 1 (3 for displacements). For feature sampling, briefly, we sampled 16 points on the WH per grid point (cf. fig 1) to project image features to grid points. We sample image features at the FFD-updated sample-point coordinates, compute the expectation of the sampled image features, and concatenate it with the current grid pt features.

[R2,R3]Lack of justification for GCN vs CNN: A GCN decoder is faster and requires far less parameters than a CNN one (1.6M vs 21.8M) as the vertex connections are encoded in the graph Laplacian instead of learned 3^3 filters. We also obtained better WH dice with GCN (0.88 vs 0.85). We can add this info to the revision.

[R1]Figure copyrights: The approach and contents shown in fig 1, 4 are quite different from [8]. While both use a similar CNN backbone, the visual similarity is mostly due to the similar choice of graphical style. We can use a different style choice if desired.

[R1]Why CT not MR: CFD simulations require high-res geometries, which are commonly obtained from time-series CT[11] as cine MR often has poor through-plane res. Cubic spline interpolation on meshes is commonly applied for finer temporal res.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper presents a deep learning framework that generates whole heart meshes from volumetric image data. Overall, this paper is well written, and experiments are well conducted. Additonally, the authors have answered the main concerns, such as the incomplete ablation studies & lack of cross-validation, and other questions well.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The methodology presented is very good, and the results are of very high quality, although for a limited group. This ins an excellent piece of work that will certainly be useful and interesting to the MICCAI community. I personally o not see an issue with copyright as indicated by reviewer 1. For the rest, I believe that reviews comments have been addressed properly by the authors. Perhaps including some more information about validation in sup material s a good idea, time permitting.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have targeted an important problem. The results are convincing with code and video available. The reproducibility is high.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

back to top

Whole Heart Mesh Generation For Image-Based Computational Simulations By Learning Free-From Deformations