Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Megumi Nakao, Fei Tong, Mitsuhiro Nakamura, Tetsuya Matsuda
Abstract
Shape reconstruction of deformable organs from two-dimensional X-ray images is a key technology for image-guided intervention. In this paper, we propose an image-to-graph convolutional network (IGCN) for deformable shape reconstruction from a single-viewpoint projection image. The IGCN learns relationship between shape/deformation variability and the deep image features based on a deformation mapping scheme. In experiments targeted to the respiratory motion of abdominal organs, we confirmed the proposed framework with a regularized loss function can reconstruct liver shapes from a single digitally reconstructed radiograph with a mean distance error of 3.6mm.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87202-1_25
SharedIt: https://rdcu.be/cyhQn
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The paper is unclear about the contribution. It presents an image-to-graph convolutional network that predicts the 3D shape (mesh) of a deformed organ from two inputs: the undeformed shape of the organ; and a 2D digitally reconstructed radiograph (from a CT of the deformed organ). The pipeline combining a CNN and a GCN is interesting, but described without much detail. The reconstruction results are compared with a previous approach (ref. 13). The novelty is in the 3 loss functions proposed, which, for the described liver experiment, help to improve slightly (in an order < than 10%) the quality of the output when comparing with ref. 13.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
The paper presents a good combination of known elements.
The use of connected meshes that are deformed towards the final shape instead of dealing with images or volumes for post-reconstrucion is an interesting aspect.
The results shown for liver reconstruction are good (organ volume around 90% correct) when the previous works mentioned reach 85%.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
There is no really novel ideas in the submission. The contribution is incremental only: combination of an existing VGG-16 model with known projection steps. The authors build upon their own previous works that are cited anonymously and are thus not accessible.
The data used is in the experiment is very complete. This could be a strength, but it actually makes the challenge less difficult.
The use of a DRR (a radiograph reconstructed from a full 3D CT scan) leaves questions open about the real use of the technique.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The description of the IGCN is given only in high level, leaving many choices unclear for reproduction of the results.
The data used is not publicly available, which could also make reproducibility of the results difficult.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
The problem as I understood is defined as: given a 3D dataset obtained in end-inhalation and a 2D projection image (simulated radiograph) in end-exhalation, predict the 3D shape of the liver in-end exhalation.
The results are compared with Pixel2Mesh (ref. 13) and are better for the studied metrics.
At the end of sec. 2.1, the data augmentation procedure is unclear.
The description of the IGCN in sec 2.2 is only in high-level, being difficult to understand the complete architecture and to reproduce.
2.3: loss functions: First, it should be made clear that this refers to the GCN part of the pipeline (is it? what about the CNN part?). Then, the way the funcions are presented does not include the insight needed to understand why these functions were chosen and not anything else.
In sec. 3, it is unclear what the “existing end-to-end deep learning framework” is. It has not been mentioned before.
Text:
the syntax is overall correct, but in some points the sentences meaning is hard to extract. Ex: “Despite that only very low-contrast textures are confirmed in most parts of the liver,”
Avoid starting sentences by a numeral, such as 0.5.
- Please state your overall opinion of the paper
reject (3)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Lack of detail in the neural network elements.
Due to the lack of detail, perhaps, I could not see what is the novelty in the methodology.
The results are not discussed and no conclusions were made out of them.
The use of very complete data that isn’t available often for clinical practices.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
3
- Reviewer confidence
Confident but not absolutely certain
Review #2
- Please describe the contribution of the paper
This paper presents an image-to-mesh method that reconstructs 3D liver point cloud from a single-view DRR image. The authors propose to construct an initial shape from the training set and deform the predicted shape. In addition to the loss used in P2M[13], the authors propose a mapping loss for mapping useful image feature for deformation prediction.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Unlike previous P2M [13], the authors propose to construct an initial 3D point cloud from training set and they propose to use a IGCN to deform the predicted shape.
- They also propose a mapping loss for mapping useful image feature for deformation prediction.
- The key comparisons to P2M and IGCN with or without mapping loss are included and demonstrated the superior performance.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- I found the motivation of this work ill-posed. The author stated that the generation of liver 3D point cloud is useful for radiotherapy. However, GTV and CTV requires dense segmentation to generate best delineations. Point cloud is sparse and may not be suitable for this task.
- Liver is big organ in DDR and relatively easy to predict. However, to claim the usefulness of this approach in radiotherapy, evaluations on other organs are needed. For example, the author mentioned pancreatic cancer. What about pancreas reconstruction? The segmentation is available in your dataset, and pancreas is hard to delineate from single view DDR. Even with liver, GTV and CTV and PTV relies on the presents of liver tumor as well. What if there is tumor in the liver?
- The author claim it is real-time application. However, there is no analysis on computation time, etc.
- The ground-truth is generated from DMR. I couldn’t find any detail about this method (author removed it for double-blinded review). I assume the best performance of this approach will not exceed DMR. Will DMR be real-time? I think some discussion are needed for it.
- Figure 3, colormap and bar for the Mx and My?
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Paper contains sufficient details to reproduce the results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Please see the detailed comments above
- Please state your overall opinion of the paper
borderline reject (5)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The paper is well-written and the method is interesting. However, I think there is lack of clinical motivation for this approach, especially radiotherapy. The method may obtain reasonable performance on large organ with clear boundary in DRR, such as liver and lung. However, the performance for other important organs, such as pancreas, still remain unclear.
- What is the ranking of this paper in your review stack?
4
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Review #3
- Please describe the contribution of the paper
A new image to graph CNN is proposed here for the shape recon. from a single projection image (composed of GCN and CNN). This topic is key in low-dose intra-operative surgery assistance. Progress in tumor localization also in moving XR images is claimed in a key frame approach using min./max. inhalation. Into a XR image, in the end a point cloud of an organ is fitted accurately.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
-the delta to the state of research is clearly outlined in 2 dedicated paragraphs at the end of the introduction -124 3D CT volumes are used in this study, which is good -figures and schemes help explanation to the reader -this is the first paper, I read in my assignment this year, with statistics properly used until tests
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
-if cross-validation is used in this study, please clearly state so
- Please rate the clarity and organization of this paper
Very Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
method is described very well, so similar reimplementation should be swift
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
-please state clearer, the organs your method is applicable to (e.g. liver) -method is well described, so I give a good rating here -please discuss/speculate in more detail, how tumor localization could profit from your work -please explain meanings of used symbols in all formulas explicitly -some parameter choices seem heuristically/arbitrary (lambdas), could you give a short explanation? -the paper is too long with 9 p.?
- Please state your overall opinion of the paper
strong accept (9)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The best paper in my stack
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
5
- Reviewer confidence
Very confident
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
The application of shape reconstruction from projections is an important clinical application. The idea is interesting and even though it uses a pre-trained networks such as VGG, the contribution of the paper is in combining the neural network architecture with a deformable loss in predicting a shape model.
The authors need to provide more information on the transformation M (learnt by the network). Concerns were raised on experimental evaluation, cross-validation. Related to the author’s claim of “real time application”, computation times and frame rate should be provided.
Another concern was the generalizability of this method to other organs in the author’s application (radiotherapy). How will it perform on pancreas for example?
The authors are invited to pay careful attention to the review comments.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
6
Author Feedback
We are grateful to the Area Chairs and reviewers for taking the time to review our paper. We have grouped the reviewers’ comments into six categories, based on the major concerns.
1) Deformation map and loss function P2M [13] uses the projected point pi of the initial mesh template to capture image features; however, the corresponding features are distant from pi because of the displacement of the target organ (see Fig. 3). P2M relies on convolution to capture distant image features but convolution is not effective for capturing high-resolution features at the ideal, corresponding position qi. This is the insight that motivated the design of our deformation mapping scheme.
The deformation mapping, with the loss function Lmap, is a new concept to overcome the above limitation. IGCN learns the warped position M(pi) to capture effective image features for shape reconstruction. The deformation map s a learnable spatial mapping function determined by the 2D vector field (Mx, My). The color map in Fig. 3 represents the mapping function M, describing the learned displacement in the x and y directions in 2D-projection image coordinates. This scheme was implemented as an extension to the feature extraction scheme of the CNN part.
The total loss, combining the three loss functions, is used to optimize the whole network in IGCN. This optimization process affects both the CNN and GCN parts.
2) Real-time performance We measured the computation time for the whole shape reconstruction process performed in the CNN and GCN layers. The average computation time was 35.4 ms (28 frames per second), demonstrating the real-time performance of the IGCN.
3) Cross-validation We performed shuffle-split 3-fold cross-validation, in which the indices were shuffled, and all cases were configured to be used at least once.
4) Application to other organs The IGCN was designed as a generalized, organ-independent framework. We prepared a 4D-CT image database of five organs and GTV. Although the reconstruction performance for all organs could be investigated, we first targeted liver shapes with partly detectable features (i.e., the diaphragm), to discuss how reconstruction error occurs locally in the cases with and without visual cues.
To apply ICGN to other organs, we are considering two approaches: one aims to reconstruct the 3D shape using the same IGCN pipeline directly. The use of detectable shapes (e.g., liver) as the additional input for the GCN part will be the second, extended solution. This pairwise reconstruction will be technically interesting, and is promising because we have confirmed that pancreatic cancer can be accurately localized from the features of surrounding organs in the published paper [10]. Therefore, we consider liver reconstruction to be an important step toward the future investigation of the reconstruction of totally undetectable organs.
5) Clinical motivation Radiotherapy is applied to liver tumors, which cannot be detected in X-ray images. Therefore, predicting the liver is highly relevant to radiotherapy and was first targeted in this study. A liver tumor can be localized as a relative position inside the liver mesh. Shape reconstruction of the liver is a more difficult reconstruction target than that of the lung because only the diaphragm can be detected in projection images, whereas the large part of the liver shape is not detectable.
6) Pre-computation and output mesh models DMR is not real-time, but DMR is not needed in clinical use. The target shape is directly generated from projection images using the pre-trained network.The previous paper [10] reported that DMR was performed with a Hausdorff distance of 1.1 mm.
One comment pointed out the limitation of the 3D point cloud in radiotherapy planning, but our paper did not claim that a 3D point cloud was useful. As shown in Fig. 4, ICGN generates a mesh structure with a closed surface, which is directly available for the 3D region definition of GTV and CTV.
Post-rebuttal Meta-Reviews
Meta-review # 1 (Primary)
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The authors tackle an important clinical application by performing shape reconstruction from individual projections. The main contribution is in combining the neural network architecture with a deformable loss in predicting a shape model. I am satisfied with the rebuttal and in particular the comment about generalizability. The authors propose an addition input from GCN to solve this problem.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
4
Meta-review #2
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
The paper’s technical novelty is limited, as consistently written by the three reviewers. The application area is interesting and applying the reconstruction technique to the problem at hand is probably the strongest contribution of the paper. However, the idea of using shape reconstruction from projections is not completely new either, making the overall contribution incremental and not so significant. I think the paper would benefit a lot from improvements based on the reviews and clarification/strenghtening of the technical contributions.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Reject
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
14
Meta-review #3
- Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.
Real-time reconstruction of 3D organs from 2D x-rays is potentially clinically valuable. This work presents a novel method to achieve this goal and quantitatively evaluated its performance on the liver. The rebuttal has clearly addressed the main concerns from the reviews.
- After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.
Accept
- What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
8