Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Deqiang Xiao, Hannah H. Deng, Tianshu Kuang, Lei Ma, Qin Liu, Xu Chen, Chunfeng Lian, Yankun Lang, Daeseung Kim, Jaime Gateno, Steve Guofang Shen, Dinggang Shen, Pew-Thian Yap, James J. Xia

Abstract

Virtual orthognathic surgical planning involves simulating surgical corrections of jaw deformities on 3D facial bony shape models. Due to the lack of necessary guidance, the planning procedure is highly experience-dependent and the planning results are often suboptimal. A reference facial bony shape model representing normal anatomies can provide an objective guidance to improve planning accuracy. Therefore, we propose a self-supervised deep framework to automatically estimate reference facial bony shape models. Our framework is an end-to-end trainable network, consisting of a simulator and a corrector. In the training stage, the simulator maps jaw deformities of a patient bone to a normal bone to generate a simulated deformed bone. The corrector then restores the simulated deformed bone back to normal. In the inference stage, the trained corrector is applied to generate a patient-specific normal-looking reference bone from a real deformed bone. The proposed framework was evaluated using a clinical dataset and compared with a state-of-the-art method that is based on a supervised point-cloud network. Experimental results show that the estimated shape models given by our approach are clinically acceptable and significantly more accurate than that of the competing method.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_45

SharedIt: https://rdcu.be/cyhQ0

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The author proposed a self-supervised learning framework for estimating the bony shapes of orthognatic surgical planning. The author shown in the experiment section for jaw deformity surgical planning, the proposed method outperformed supervised learning based methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Clinical meaning: The proposed self-supervised learning framework can be used for orthognathic surgical planning for patients who need bony structure reconstruction.
- Theoretical aspects: The author introduced self-supervised learning based on CNN decoder-encoder networks trained with a deformed and normal bony structures dataset. The framework consists two networks: (1) the simulator network simulate the deformed normal bony shape based on normal and abnormal shapes. (2) the convertor network tries to reconstruct the normal shape from the deformed normal shape. (3) the similarity between the deformed normal bone and normal bone is controlled through a designed loss function. The method is soundable in theory.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Clinical aspects: (1) Customizable nature of output deformation: many bony corrective surgeries need to provide multiple plans for bone reconstruction. The method proposed in this article still needs to be explored in terms of the customization of bone reconstruction.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The proposed method can be reproduced as the author provided the network details in section 3.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The evluation dataset is too small ()only 24 samples) and the influence of different hyper parameters may need better illustration.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Novelty and clinical significance of the method.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper proposed a self-supervised training framework for reference skull estimation in orthognathic surgical planning. It consists of a simulator network that simulates deformed skulls from normal ones and a corrector network that estimates reference skulls from deformed ones.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This paper is easy to follow. Problem formulation is clear.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. Important experimental details are not clear, e.g. how is the template model chosen and how many number of vertices does it have?
2. Evaluation is incomplete. The paper only compares the estimation result against post-operative result, which does not represents an ideal reference model. More quantitative evaluations like cephalometric analysis or on synthetic data should be performed.
3. Experimental setting of DefNet is not provided, e.g. how many deformed skulls are simulated to train DefNet.
4. Reviews not complete. There are traditional methods like symmetry based methods, statistical shape models etc. that also generates reference skull models.
5. Contribution not strong enough. It seems that the difference between the proposed method and DefNet is mainly the method to simulate deformities. This article does not explain the advantage of the proposed point-net based method over the method used in [4].
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Lack of necessary details to reproduce, e.g. how is template model generated? what is the target size for downsampling data. etc.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Include more reviews, e.g. [4] mentioned several deep point-net based methods, what are they? traditional statistical shape model based methods.
2. Provided more experimental details to improve reproducibility.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Reviews are not complete, lack of necessary implementation details, experimental evaluations are not enough to prove strength of the method.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

1)Author applied the self-supervised learning mechanism to estimate reference bony shapes by developing an end-to-end learning framework that can be directly trained using unpaired datasets; 2) The proposed framework outperforms the related method based on the super-vised learning.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Authors propose a self-supervised deep framework, in which a simulator network maps deformity forms from a patient bone to a normal bone (i.e., simulating the deformed jaw on the normal bone), a corrector network then restores the simulated deformed bone back to normal.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The training data is not very sufficient.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Because the source data is not very common, I think it is not very easy to be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The authors use a self-supervised deep framework to achieve orthognathic surgical planning. It is divided into simulator and corrector. The simulator inputs the normal bone and the deformed bone, and outputs the upper part of the normal and lower part of the deformed bone. The corrector input the output of the simulator, and output the corrected bone shape. The author uses their method to solve the problem of lacking ground truth in 3D shape learning. And their models simplify the training process and improves the estimation accuracy as well.

Page 1 1 Introduction This paragraph is divided into three parts. First, you describe the process of orthognathic surgery. Then you describe the contribution of others in this field. Finally, you explain your own method. There are less job descriptions about yourself and more job descriptions about others. Can it be divided into two parts?

Page 2 Fig.1.(b) Which is moved and which is not? Please make it out.

Page 1 Wang et al. [3] proposed to estimate patient-specific reference bony… There are few descriptions of the relevant work in this field. I hope you can learn more about the work of some other people.

Page 2 To alleviate this issue, a 3D point-cloud netword-based… You should point out who put forward this method. Is it Wang et al. mentioned above or others?

Page 3 Fig.2. Figure 1 uses the “Patient deformed bone”, but figure 2 use “Deformed bone”. They should have the same name. Which one is “Deformed bone” for “Simulated deformed bone” and “Patient deformed bone”? It’s easy to confuse.

Page 3 2 Method You talked about the implementation of the simulator very well, but I have a big question about why you need the simulator. Although it is mentioned later that this is to increase training data, the distance is too far. It should be explained here.

Page 6 Ultimately, a total of 7772(67 × 116)random pairs of deformed-normal… Because the model is a deformation of the jaw, I think the model pays more attention to the jaw part. Can data with different midface but same jaw be regarded as two data?

Page 7 For testing, we acquired the data from another 24 patients with… Has this data been approved?

Page 7 Table 1 shows that our method to estimate the normal jaw is statistically significantly… Please clarify why it is statistically significant? How to get the data of “P < 0.05”?

Page 8 …simplifies the training process and improves the estimation accuracy as well… It is not good to draw conclusions by comparing with only one method. I hope to compare it with other methods. The “simplifies training process” is not reflected in your experiments. Maybe you need to do some comparative experiments on training time.

Page 8 …in solving the problem of lacking ground truth in 3D shape learning. The statement here is a little unclear, I hope to get your explanation. There is incomplete explanation of how the lack of ground truth is trained. How to determine the correct output of the corrector during the training process?

Page 8 References Please try to refer to the literature of recent years. And I hope you can pay attention to the format. Reference 11? Please check it carefully.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The experimental result.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

4
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This is an interesting application with a seemly sound approach proposed for it. However, the reviewer 2 also raised clear concerns regarding to the lack of important details. Therefore i would invite authors to provide a good rebut to these comments. For comments from Reviewer 1, i’d suggest to report statistical tests to show the significance of the gained improvement over the alternative DefNet.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

Lack of experimental details. (Reviewer #2) R: For each of 67 normal bones, we simulated 102 deformed bones based on sparse representation, achieving a total of 6834 paired samples for training DefNet. We will add this information into the final version of our manuscript. To select a template from training surfaces, 51 corresponding landmarks localized on all training surfaces are rigidly aligned. The surface with landmarks that give the smallest distance to the average landmarks is then selected as the template. The number of vertices of each training surface is reduced to 4724 via surface simplification. These pieces of information were already mentioned in the submitted manuscript (Network Training and Inference, Page 5; Experiments and Results, Page 6).

Statistical significance of improvement. (Reviewer #1, Reviewer #3) R: The statistical significance of improvement over DefNet was validated by paired t-tests. All p-values calculated on the four assessing metrics were much smaller than 0.05 for the normal jaw estimation. This information was already mentioned in the manuscript (Experiments and Results, Page 7).

Lack of the literature review. (Reviewer #2, Reviewer #3) R: In the field of CMF reconstruction, traditional methods based on symmetrical mapping (Schmelzeisen et al. 2004, Gellrich et al. 2006) or statistical shape models (Semper-Hogg et al. 2017, Anton et al. 2019) are mainly designed for CMF defects. They are all beyond the scope of the current study, which focuses on congenital jaw deformities. As far as we know, one traditional method in this specific area is based on sparse representation (Wang et al. 2015), we already reviewed this method in the submitted manuscript (Introduction section). DefNet surpasses this sparse representation method by a large margin (Xiao et al. 2021). Therefore, we ignored this traditional method and directly compared with DefNet in the experiments.

Evaluation on synthetic data or cephalometric analysis should be performed. (Reviewer #2) R: The data simulation strategies applied in DefNet and our method are different. It is unfair to compare the two methods using the synthetic data generated by any one of the two simulation strategies. Besides, cephalometric analysis cannot give an accurate assessment on how normal an individual subject looks like, as we already explained in the submitted manuscript (Introduction section).

Explain the advantages of the proposed method over DefNet. (Reviewer #2, Reviewer #3) R: It is almost impossible to acquire enough paired deformed-normal bones from clinical practices directly, thus simulating paired data is necessary for training supervised deep learning models, like DefNet. Data simulation based on sparse representation as done in DefNet is limited in mimicking deformities, and requires a lot of efforts from experts to confirm simulation qualities. Therefore, we proposed a self-supervised deep framework to perform the network training directly on unpaired data. The embedded simulator network has a stronger non-linear representation ability than sparse representation, and is able to accurately synthesize any type of jaw deformities according to patient bones. Generally, compared to DefNet, our framework simplifies the training process and improves prediction accuracy. We already pointed this out in the submitted manuscript (Discussion section).

Will two different normal bones lead to different simulations according to the same patient bone? (Reviewer #3) R: We do not simply combine the deformed jaw from a patient bone with the midface from a normal bone to get the simulated deformed bone. In fact, the proposed simulator network learns the jaw deformity from a patient bone and transfer it to a normal bone. When transferring the same deformity from a patient bone to different normal bones, different simulated deformed bones will be generated, since the deformity varies according to the midface of each normal bone.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The rebuttal adequately addressed all critical comments from reviewer 2, both in sufficiency in statistical analysis, literature review and comparison. I agree with the authors that comparing the two simulation methods is less informative.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

15

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This is an interesting paper proposing a deep simulator framework for orthognathic surgery planning. While the technical contribution is limited and this can be considered as a preliminary proof of concept, the application here is novel and would be of interest to the CAI community. In my point of view, the rebuttal has done a fairly good job in addressing the majority of the criticisms raised by the reviewers.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The rebuttal has addressed most of the concerns raised by the reviewers. Especially, the concerns regarding comparison to DefNet and how it was trained are explained clearly and justified as well. The work is suitable for acceptance at the MICCAI meeting and will generate important discussions during the meeting. The authors should include the explanations provided in the rebuttal in their camera-ready version of the manuscript to improve the work.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

back to top

A self-supervised deep framework for reference bony shape estimation in orthognathic surgical planning