Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Yamid Espinel, Lilian Calvet, Karim Botros, Emmanuel Buc, Christophe Tilmant, Adrien Bartoli

Abstract

Deformable registration is required to achieve laparoscopic augmented reality but still is an open problem. Some of the existing methods reconstruct a preoperative model and register it using anatomical landmarks from a single image. This is not accurate due to depth ambiguities. Other methods require of non-standard devices unadapted to the clinical practice. A reasonable way to improve accuracy is to combine multiple images from a monocular laparoscope. We propose three novel registration methods exploiting information from multiple images. The first two are based on rigidly-related images (MV-B and MV-C) and the third one on non-rigidly-related images (MV-D). We evaluated registration accuracy quantitatively on synthetic and phantom data, and qualitatively on patient data, comparing our results with state of the art methods. Our methods outperforms, reducing the partial visibility and depth ambiguity issues of single-view approaches. We characterise the improvement margin, which may be slight or significant, depending on the scenario.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_63

SharedIt: https://rdcu.be/cyhRi

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors present a 3D (pre-op CT) to 2D (intra-op laparoscopic video) registration algorithm for use in laparoscopic liver surgery. The problem is challenging and remains unsolved. Existing methods typically use single view mono, or single view stereo images and 3D/3D registration. The authors propose 3 novel constraints that utilise multiple view-points, and an algorithm to optimise a non-linear registration. This idea would be relevant and of interest to the CAI community.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The strengths are a well-principled formulation of the 3 novel cost functions, and the algorithm for optimising.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

While I’m excited about the concept, the results do not appear to be a significant improvement. Thankfully, the authors have not over-stated their results, so provide a fair assessment, of how the research is currently progressing.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

While the paper is descriptive, allowing re-implementation, code and data are not provided, as far as I can see.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The paper is very clear.
1. It is not clear how different each view needs to be, or whether the multiple views are feasible in clinical practice, due to the troccar. Can you comment?
2. The danger with multiple views is breathing motion between views. What do you propose to do to minimise these errors.
3. I can’t see where you have stated, or referenced what the desired clinical accuracy would be, without which the reader cannot assess whether the results are approaching the right benchmark, or still a long way off.
Minor typos:
1. Abstract: “Other methods require of non-standard devices unadapted” -> “Other methods require non-standard or additional devices that hinder the clinical workflow.”
2. Abstract: “Our methods outperforms” -> “Our methods outperform single-view approaches by …” etc.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

While, this is a nice idea, unfortunately the results do not appear to be a significant improvement. However, this is such a challenging problem, that I’m not surprised with this fact, and I think it would be good to get people together to discuss such things, and find ways to move forward.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This paper addresses the registration problem between 2D laparoscopy images and 3D CT of the liver. To obtain higher accuracy, this work extends the SV method [1] by using multiple laparoscopy images. The basic idea is to take the average value of multiple laparoscopy images for registration. Both rigid and non-rigid situations are considered. Experiments show the proposed methods have better accuracy than SV.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The problem of 2D-3D registration is important for surgical navigation. The experiments include synthetic, phantom and real patients data, which are sufficient.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

This paper is clear. However, it lacks of many important details. For example, the methods rely heavily on the correspondences between laparoscopy images. But I cannot find the details of how to obtain the correspondences? As mentioned in Section 3.2 you use structure-from-motion (SfM)? But which image feature method do you use (SIFT/SURF/ORB etc.) and which SfM method do you use? For non-rigid cases in Section 3.3 when SfM cannot work, how to build the correspondences?

It looks like the methods require manually annotation on the laparoscopy images during the surgery. It is better to discuss whether this may bring additional cost of the expensive time in the OR since the methods need multiple laparoscopy images.

The proposed methods are mostly based on the SV method in Ref. [1]. The experiments show that MV-B and MV-C is more accurate than SV. But MV-D has almost the same accuracy as SV. Since MV-B and MV-C are rigid and MV-D is non-rigid, and all methods requires annotations on multiple images, I think the method does outperform SV for non-rigid cases.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Many algorithms of the proposed methods are based on existing works, such as the SV method and the rigid-perspective thin-plate spline method. Hence it should be OK to reproduce the methods.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I suggests to add more details of the proposed methods.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The methods are mainly developed based on the SV method. And the main idea of this paper is to warp multiple laparoscopy images to the target one, and use the average value to perform 2D-3D registration with the CT image. This idea seems straight forward. In addition, many details on how to build correspondences and how to estimate the warp functions are missing, especially for the non-rigid MV-D method. Hence, it is difficult to fully evaluate the technical novelty of this paper.

The non-rigid method (MV-D) does not give higher accuracy, and has higher requirement than the SV method.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

This work presents a deformable registration of CT to intra-operative scene using multiple images with three novel registration methods. The authors use both rigidly related images and non-rigidly related images to derive their approaches and evaluate in synthetic, phantom and an in-vivo patient case. The validation is well done, and this presents an interesting contribution to the field of registration.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Beautifully written paper and well positioned problem; clear understanding of the prior art and presents an interesting set of methods for the audience to dig in; clear understanding of the constraints and clinical scenario and makes appropriate use of the clinical field of view; strong mathematical foundation and interesting use of biomechanics; useful figures; strong validation across synthetic, phantom and patient data.
This is a very strong paper.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Lack of quantitative validation for the patient data, some lack of discussion on the potential sources of error, and a few typos.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Pertinent description of the datasets used are captured, although a publicaly available version would be nice. Fig. 2 outlining the pipeline is a bit confusing, and I’m not sure I could reproduce it entirely.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
I thank the authors for their submission in the under-appreciated field of pre-operative to intra-operative registration. Quite frankly, liver surgery augmented reality is hard. The authors have presented a thorough and rigorous presentation of their work in this context. It is a refreshing piece of work in an era of “deep learning madness”. I have a few critiques for this otherwise excellent paper.
- The choice of contours comes from primarily the liver itself. Have the authors considered using non-liver based contours of nearby anatomy (for example vasculature) to enhance their registration?
- Is the choice of an isotropic Neo-Hookean elastic model appropriate for the liver? Can the authors justify their choice in the paper a bit more? I ask because I have concerns of how deviations from this model may propagate into registration errors. How sensitive is the registration to an error in the biomechanical model?
- I find Fig.2 to be important but confusing. It’s hard to keep track of all the different algorithms in this one flow chart. Is there another way to present the additional differences between algorithms?
- What’s the significant of using 10e-3 for the convergence criteria? Did the authors observe any discernible differences in results when comparing other criteria?
- I don’t think I saw a justification for the choice of number of frames used. Why limit to 8? There doesn’t seem to be a trend appearing in the use of more views. Would 30 frames/views be more useful (commonly given laparoscopes are at 30fps)
- The TRE is quite high. Can the authors elaborate on why this may be, and what contributes to this high standard deviation?
- What is the distribution of error like? Is it bimodal? Unimodal? Are there specific cases which cause significantly large error or failure?
- RMS is better to report than mean and standard deviation.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This is an interesting approach that combines image analysis, particle physics, image registration, and augmented reality. The authors have a good choice of baseline, improve on it, and contextualize their work in the clinical scenario. Early results in patient cases are promising. To me, this is exactly the kind of paper that belongs in MICCAI. I look forward to hearing the authors present.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
This MICCAI submission, titled “Using Multiple Images for Deformable 3D-2D Registration of a Preoperative CT in Laparoscopic Liver Surgery” was reviewed by a total of 4 reviewers; one review was withdrawn due to potential conflict. The 3 reviewers varied in their experience/confidence level.

Based on their scoring and my personal readings, I am recommending the decision of Provisional Accept. However, I also offer the following suggestions/questions (mostly taken from the withdrawn review) and ask authors to kindly consider to address them in their future revision:
- The contributions seem over-stated. The title “Using Multiple Images for Registration of Preoperative CT in Liver Surgery” implies that multiple images have not been used in previous works. This is incorrect given there exists much prior art using stereo and multiple images. Some have been cited, others have not e.g. for other abdominal organs and for the liver such as
Pfeiffer et al., Non-Rigid Volume to Surface Registration Using a Data-Driven Biomechanical Model, MICCAI 2020

In the abstract: “Existing methods reconstruct a preoperative model and register it using anatomical landmarks from a single image.” is not correct and it should be removed. I suggest a title modification to reflect this work better: “Using Multiple Images and Contours for Deformable 3D-2D Registration of a Preoperative CT in Laparoscopic Liver Surgery”
- The method is based a lot on [1]: the technical extensions are quite straightforward for the case of multiple images
- Analysis of results statistical significance is not done
- It appears there is no advantage to using multiple views when the organ can deform between views. Thus, if the advantage is only in the case of rigid views, why are the authors not exploiting a 3D reconstruction using SfM or SLAM to constrain better registration with e.g. robust ICP? This is done in almost all previous works to register a pre-op model to rigid laparoscopy views.
- The very large error bars in figure (a) and (b) indicate that the improvement with multiple images for TRE is probably not statistically significant. This needs verifying however.
- The contours appear to have been selected manually which is a significant weakness for clinical use.
- This manuscript is an interesting extension of [1] to multiple images. It seems logical that registration would be improved in the case of rigid views. this appears to have been demonstrated but statistical significance is not verified. The fact that a 3D surface reconstruction is not exploited (sparse or dense) is unusual and a potential weakness. Overall I thought that this work was a reasonable advancement of [1] but my main concern is the lack of improvement in TRE when the liver deforms over time.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

Author Feedback

We thank the reviewers and ACs for their constructive comments and will update the article accordingly.

Reviewer #1: R1: View acquisition owing to trocar. There should be a noticeable difference between the views and some overlap to find correspondences. This is achievable simply by tilting and panning the camera. Views from several trocars may be used.

R1: Breathing between views. In the rigid-view case, one can pause the ventilator, 10 sec are enough. Nevertheless, one of our future goals is to improve MV-D to use non-rigid views and reach similar or better accuracy than MV-C.

R1: Desired clinical accuracy. The TRE should be lower than 1 cm, as the resection margin for e.g. HCC [Zhong et al.]. This means MV-C complies with the precision requirement.

Reviewer #2: R2: Obtention of correspondences. We use SIFT followed by FBDSD [Pizarro et al.] for correspondences and the image warps. These are the same correspondences as used in SfM. We agree that the statement is misleading and will clarify it.

R2, AC: Manual contour annotation. Manual annotation is required. It takes ~5 minutes to mark all the contours in 8 views. The impact on the workflow is thus minimal. We are currently working on automating the process.

Reviewer #3: R3: Usage of other anatomical landmarks. The difficulty is that landmarks must be visible in both the preoperative and the intraoperative data. A more feasible source of constraints is obtained by tracking the surgical tools.

R3: Usage of Neo-Hookean model. We use this model as it can cheaply simulate large deformations [Bender et al.] and has been validated on the hepatic tissue [Chui, Shi, Zaeimdar]. We compensate for possible deviations by alternating shape optimization using the contour vertices fixed and with all the vertices set free [Ozgur et al.]. We will investigate the use of other models in future work.

R3: Improving of pipeline flowchart. We will improve this figure by separating the pipelines for each of the methods, while pointing the similarities and differences between them.

R3: Significance of the convergence criteria. We found the value of 10e-3 experimentally on a variety of synthetic, phantom and patient data. The value is stable over all these situations. A lower value does not lead to a noticeable change of the registration.

R3: Limit of the number of views. We limited the number of views to 8 to keep computation time reasonable. We will investigate if adding more views may improve accuracy. However, using all the video frames would not stabilise the registration, geometrically speaking, as they would be extremely similar to one another.

R3, AC: High TRE. MV-C shows a constant decrease in TRE as we increase the number of views. The standard deviation, while decreasing with the number of views, remains high due to the hidden parts, which are less accurately registered than the visible parts.

R3: Distribution of errors. When analyzing the error distribution for MV-C using 8 views, we see that it is unimodal with positive skewness, following a Burr pdf. Wide-baseline views without overlap and wrong correspondences may be a source of significant errors in registration.

R3: RMS vs mean and STD. The information brought by RMS is already contained by the mean value. STD gives information on the error spread.

Meta-reviewer: AC: Title and abstract correction. Contribution overstating. We agree and will update the title accordingly. We will also include the suggested corrections. We don’t think the contributions are otherwise overstated. For instance, we clearly mention the similar performance between MV-D and SV.

AC: Usage of SfM/SLAM in registration. Using an intraoperative reconstruction obtained through SfM or SLAM is an interesting idea. However, obtaining a reliable intraoperative shape reconstruction is challenging in practice. We plan to evaluate this approach in future work.

back to top

Using Multiple Images and Contours for Deformable 3D-2D Registration of a Preoperative CT in Laparoscopic Liver Surgery