Back to top List of papers List of papers - by topics Author List
| Paper Info | Reviews | Meta-review | Author Feedback | Post-Rebuttal Meta-reviews |
Authors
Mingyuan Luo, Xin Yang, Xiaoqiong Huang, Yuhao Huang, Yuxin Zou, Xindi Hu, Nishant Ravikumar, Alejandro F. Frangi, Dong Ni
Abstract
3D ultrasound (US) is widely used for its rich diagnostic information. However, it is criticized for its limited field of view. 3D freehand US reconstruction is promising in addressing the problem by providing broad range and freeform scan. The existing deep learning based methods only focus on the basic cases of skill sequences, and the model relies on the training data heavily. The sequences in real clinical practice are a mix of diverse skills and have complex scanning paths. Besides, deep models should adapt themselves to the testing cases with prior knowledge for better robustness, rather than only fit to the training cases. In this paper, we propose a novel approach to sensorless freehand 3D US reconstruction considering the complex skill sequences. Our contribution is three-fold. First, we advance a novel online learning framework by designing a differentiable reconstruction algorithm. It realizes an end-to-end optimization from section sequences to the reconstructed volume. Second, a self-supervised learning method is developed to explore the context information that reconstructed by the testing data itself, promoting the perception of the model. Third, inspired by the effectiveness of shape prior, we also introduce adversarial training to strengthen the learning of anatomical shape prior in the reconstructed volume. By mining the context and structural cues of the testing data, our online learning methods can drive the model to handle complex skill sequences. Experimental results on developmental dysplasia of the hip US and fetal US datasets show that, our proposed method can outperform the start-of-the-art methods regarding the shift errors and path similarities.
Link to paper
DOI: https://doi.org/10.1007/978-3-030-87231-1_20
SharedIt: https://rdcu.be/cyhUL
Link to the code repository
N/A
Link to the dataset(s)
N/A
Reviews
Review #1
- Please describe the contribution of the paper
The authors introduced learning frameworks for the estimation and optimization of their image reconstruction. Using a self-supervised learning scheme for feeding information for the next slices and using the adversarial learning technique for better representation, they could reconstruct better images.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- Embedding and concatenation of several techniques for 3D reconstruction online
- Considering US slices as a sequence of data and use LSTM and Adversarial method for distinguishing real and fack results.
- All these techniques used for relative transformation parameters among all adjacent frames.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- Weak presentation of qualitative results such as figure 5.
- long literature review with no covering of important recent publications and novelties.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Due to using several modules and models, it is difficult to reproduce results.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The explanations of methods are difficult for non-expert readers to understand.
- Some data presentations in manuscript is not useful such as Figure 4 due to small resolution even worse after printing as gray-scale version.
- Literature review can be shorter and better by at least refering readers to previous reviews such as “A review of calibration techniques for freehand 3-D ultrasound systems.” and “Freehand 3-D ultrasound imaging: a systematic review.”
- Abstract is one page while the conclusion is 5 sentences!! not consistent in size and also abstract is hard to grasp all new points of the paper.
- Please state your overall opinion of the paper
accept (8)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
Freehand scanning with US is so challenging and I think the idea of using adversarial generators and LSTM with convolution can be a push for the literature.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
3
- Reviewer confidence
Very confident
Review #2
- Please describe the contribution of the paper
This paper presents an interesting approach for reconstructing 3D US volume by combining several tricks such as shape representation learning and self-supervised learning.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
There are some novelties in this paper, and the empirical results are quite promising.
The paper is well written with little to none grammar errors or typos.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
The presentation of some parts is quite confusing.
- Please rate the clarity and organization of this paper
Satisfactory
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
Codes not provided, and some parts of implementation are not well explained.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
See below for detailed comments.
- What’s the overall procedure of training? Does it rst use MAE and correlation loss to train the ConvLSTM to get estimation of theta, and then use the cross entropy loss to ne tune the ConvLSTM for volume reconstruction?
- During training, what’s the purpose of cross entropy loss? Is the loss used to updating the ConvLSTM component or the component which maps ^ to the reconstructed volume?
- During training, since we have the real volume and the reconstructed volumes, why not use MSE loss instead of cross entropy loss for guiding the reconstruction?
- During testing, what’s the purpose of adversarial loss? Is it for ne tuning the ConvLSTM component?
- How the pre-trained classi er or discriminator is obtained? How is it trained?
- it’s quite unusual to use prior knowledge during testing.
- The experimental results show that the self-supervised learning trick actually do not contribute much to the improvement.
- The authors should give more information about how the reconstructed volume is obtained from the estimated ^
- The de nition of W(Dij) and softmax in (2) is confusing. The softmax of 1=(Dij + epsilon) is just 1. Is this what the authors want?
- For the loss function in (4), the notations of and ^ are abused.
- In the loss Ld in (5), the C(Vp) is not de ned.
- Please state your overall opinion of the paper
borderline accept (6)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
There are some novelties in this paper, and the empirical results are quite promising.
The paper is well written with little to none grammar errors or typos.
The presentation of some parts is quite confusing.
- What is the ranking of this paper in your review stack?
2
- Number of papers in your stack
4
- Reviewer confidence
Very confident
Review #3
- Please describe the contribution of the paper
The authors proposed an online learning framework for ultrasound 3D reconstruction without providing spatial positions for 2D frames. Self-supervised learning and adversarial training were incorporated into the training and test phases, which boosts the performance of the listed evaluation metrics through the ablation study.
- Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1) The proposed methodology experimentally boosted the performance over the previous work, Guo H et al (2020). Online self-supervised learning and online adversarial training are the main contributions. 2) The authors proposed the differentiable reconstruction approximation to enable an end-to-end optimisation for the learning process. 3) The proposed methodology may generalise to multiple ultrasound scan sequences, which may potentially benefit the real clinical scenario of ultrasound scans.
Reference: Guo H et al: Sensorless freehand 3d ultrasound reconstruction via deep contextual learning. In MICCAI 2020.
- Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1) Learning all benchmarking models on only the synthetic data may hinder the way to extend the evaluation from the simulation to the real scenario. 2) Were the baseline models, i.e. CNN [11] and DCL-Net [4], trained and fine-tuned in a rational way? Intuitively, it may be unnatural that DCL-Net with deeper architecture performed bad compared to the proposed one with one layer. 3) The title doesn’t fully reflect the main idea of this paper. 4) Fig. 4 shows still a portion of cases may overfit or diverge by using the iterative optimisation. 5) The clarity of presentation requires some more improvements to warrant publication in MICCAI. Not all the content in the paper seems self-contained, asking the readers a lot to know the specific terminologies with no references. Kindly refer to the more comments in Point 7.
- Please rate the clarity and organization of this paper
Good
- Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
The authors have checked all items in reproducibility checklist. However, from the main text, we are unable to identify:
- Necessary details of the way to simulate training data;
- Link of the dataset;
- A comprehensive list of all used parameters such as batch size.
- How to tune the baseline models;
- Clear definition of the evaluation metrics, such as drift.
- Capacity of the used deep learning models, or memory footprint. If the author can fulfill the reproducibility checklist as promised, it is highly likely that the work can be reproduced.
- Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1) The paper may win if a delicate English polish is done by a native speaker. Some English expressions like “complex scan skill sequences”, “the indicators decline curves”, etc. look vague to me. 2) Suggest prudently defining or formulating the unspecified terminologies, such as relative transform parameter, drift, the distance D_ij, the function C(.). Avoid dual definitions for the symbol L.
- Please state your overall opinion of the paper
Probably accept (7)
- Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
The proposed online learning framework is novel to ultrasound 3d reconstruction. It reinforces the previous work Guo H et al (2020) by adopting other architecture and training strategy. However, there is lack of intuition or theory to explain why the proposed method can outperform. Furthermore, the presentation requires a significant improvement to warrant publication.
- What is the ranking of this paper in your review stack?
1
- Number of papers in your stack
6
- Reviewer confidence
Confident but not absolutely certain
Primary Meta-Review
- Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.
Strengths:
- empirical results are quite promising. *The authors proposed the differentiable reconstruction approximation to enable an end-to-end optimisation for the learning process.
- The proposed methodology may generalise to multiple ultrasound scan sequences, which may potentially benefit the real clinical scenario of ultrasound scans.
Weaknesses: *- long literature review with no covering of important recent publications and novelties. *The title doesn’t fully reflect the main idea of this paper. *Not all the content in the paper seems self-contained,
Overall: *refocusing the introduction on novelty would help
- refocusing presentation to be self-contained is important.
- What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).
2
Author Feedback
N/A