Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Sonia Martinot, Norbert Bus, Maria Vakalopoulou, Charlotte Robert, Eric Deutsch, Nikos Paragios

Abstract

Monte-Carlo simulation of radiotherapy dose remains an extremely time-consuming task, despite being still the most precise tool for radiation transport calculation. To circumvent this issue, deep learning offers promising avenues. In this paper, we extend ConvLSTM to handle 3D data and introduce a 3D recurrent and fully convolutional neural network architecture. Our model’s purpose is to infer a computationally expensive Monte Carlo dose calculation result for VMAT plans with a high number of particles from a sequence of simulations with a low number of particles. We benchmark our framework against other learning methods commonly used for denoising and other medical tasks. Our model outperforms the other methods with regards to several evaluation metrics used to assess the clinical viability of the predictions.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87202-1_48

SharedIt: https://rdcu.be/cyhQ3

Link to the code repository

https://git.io/JcbxD

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

Paper extends ConvLSTM to handle 3D data and introduce a 3D recurrent and fully convolutional neural network architecture, and infer a computationally expensive Monte Carlo dose calculation result for VMAT plans with a high number of particles from a sequence of simulations with a low number of particles.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

Work used MC simulation can asset to reach accurate and fast computation of dose.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Work is experimental and decent in its goal.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Very hard to duplicate.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

1) Motivation of the network topology needs explanation.

2) sensitivity analysis of noise effect.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Work is well-presented and results are convincing.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

1
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The authors describe experiments using long short-term memory (LSTM) for Monte Carlo dose denoising.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors motivate the problem, concisely describe their methods and results, and provide good summary of recent literature.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The use of the LTSM formulation does not seem a good match for the problem domain. Monte Carlo calculations are independent samples without a temporal relationship.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors supply access to data and code, which is a positive point
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

This paper attempts to persuade the reader that a LSTM network is a good idea for MC denoising. It is well understood that denoising helps accelerate MC calculations. It is less clear that DL methods are competitive with classical methods, but they might be competitive and are worthy of study. It is even less clear that the LSTM network is a good idea. In section 3.1, it is argued that MC calculations form a sequential process. But in truth, the calculations are independent and adding them in a sequential fashion is rather artificial. One gets the same result if one changes the order they are added or adds them simultaneously. Thus, LSTM seems a poor fit for this application.

The experimentation is well executed and well presented. However, there are many reasons that experimental results can be better for one network over another that have nothing to do with architecture. Given that LSTM seems not well matched to the application, the bar is raised a little higher.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Although I do not agree with the authors choice of architecture, which is presented as the main contribution, I also do not reject it out of hand. The authors do a decent job presenting their case, and the paper is well written.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

1
Reviewer confidence

Confident but not absolutely certain

Review #3

Please describe the contribution of the paper

While Monte Carlo simulation remain the gold standard approach to simulate dose maps for treatment planning in photon radiotherapy, the computation time are still really important (even when GPU accelerated). This paper presents a neural network-based method for denoising coarse dose maps computed through MC simulations. Such an approach relies on convolutional LSTM extended to 3 dimensions. Quantitative evaluations show that the proposed approach (coarse noisy MC simulation followed by a denoising step) yields excellent results (when compared to a complete MC simulation) while achieving an important gain in terms of computation time.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is well-structured, clear and easy to read. The evaluation protocols are sound and well justified, and the evaluation results are well discussed and provide interesting insights.
- The methodology is justified and the intuition behind the chosen ConvLSTM architecture is well explained. In other papers, it feels as if choosing a deep learning-based approach was the “default” choice. However, there is a need to exploit both spatial and temporal information of the given sequence before inferring the highly sampled dose in this application, hence the choice of ConvLSTM is well justified. Also, the intuition behind the chosen hybrid loss function is well explained and clear.
- An extension of ConvLSTM to 3D is proposed and this allows the network to process medical volumetric sequential data. Each voxel future state can then be predicted from contextual information brough by temporal and spatial features from its own and its neighbors.
- Extensive evaluations on a dataset from different patients and anatomies are presented, and the proposed approach is compared to other learning based denoising methods. Results show an increase in performance while having less trainable parameters.
- All implementation details and parameters are disclosed.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- How is the number of particles from the noisy MC simulation chosen? This was not commented in the paper and seems to be an important aspect of the method. In the evaluation, the input noisy maps are computed from 5x10^8 particles, which I believe takes a large amount of time to compute. Can this number be reduced and how does this affect the performance of the denoising?
- While it is mentioned in the introduction, the gain in terms of computation time was not actually evaluated in the paper. I would recommend adding results about the gain in simulation time by using a denoising approach with respect to a complete MC simulation (and including the time it takes to train the deep learning model).
- In Figure 2, it seems that there is an error: “right to left” is mentioned in the text description but then the figure from top left is described.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Good. Details about implementation, hyperparameters, split and even data are provided. Number of channels and dimensions of each block are disclosed.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- It is important to comment on the required simulation time for the input noisy dose map. Can you reduce the number of particles simulated without affecting the quality of the final denoised map?
- Figure 1 is hard to understand since it presents 3 separate methods. I would recommend to separate the figure in 3 different ones or at least present the proposed approach in a separated figure.
- In section 4, the paper mentions that a complete simulation would take “16k hours”, but this seems to be a lot of time. Is this a typo or perhaps you forgot to mention that it is the time it would take without parallelization?
- I would recommend also on adding a comment on how your approach would be integrated in clinical practice. What kind of training data would be required (patient specific, simulated…) and how it would be obtained? What would be the required Training time?
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The proposed 3D Conv LSTM as a MC denoiser is an interesting and novel approach for reducing the computation time in procedure planning in radiotherapy. The paper is well written, the method is well explained and the implementation details are also presented. The evaluation is almost complete (just lacking evaluation simulation time) and the approach is compared to a benchmark of deep learning based denoiser. Overall, this paper is really good and I recommend to accept it.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

Overall, this is a generally well received paper, highlighting the clarity and strong presentation of the paper. Reviewers also find the choice of the methodology using ConvLSTM to be well justified by the authors, showcasing convincing results with thorough experimentations. Reviewers also appreciate the details on implementation, hyperparameters and data splits.

However, R#2 mentions the choice of LTSM formulation does not seem to be a good match for the problem domain, as Monte Carlo calculations are independent samples without a temporal relationship. Authors should discuss this point, and address several of the improvements mentioned by R#3 in the revised paper.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

Author Feedback

We thank the reviewers and meta-reviewer for the constructive comments and the appreciation of our method.

Reviewer 1&2 argued that LSTMs are not best fitted for this problem because MC calculations are independent. It is true that MC simulations are independent at simulation time. In our paper, we do create a sequence of decreasingly noisy simulations for the model’s input. Assuming that the noisiest simulation in an input sequence requires to simulate 5e8 particles, we create the 5e9 particles simulation by adding 9 other 5e8 particles simulations to the sequence noisy 5e8 particles simulation. Therefore, in an input sequence, the simulations are not independent but rather build on top of each other. Furthermore, the noise in the input sequence exhibits a temporal behavior, as it converges towards 0 as the number of simulated particles increases and that this process depends on the location in the simulation. We believed that ConvLSTMs would be able to learn this dependency, as artificial as it may be. Moreover, for each VMAT plan during training, we were able to create several distinct noisy sequences thanks to the 1000 sub-simulations with 1e8 particles we had to generate to get the ground-truth dose. We argue that this helped the model to better learn the denoising task and generalize. We to investigate other architectures that would be best fitted to the problem and allow to use even noisier simulations as inputs.

Reviewer 3 asked multiple questions and precisions regarding computational time requirements. We did use parallelization to compute the dose maps and the time mentioned in the article refers to the accumulated computation time. To get the ground-truth simulation with 1e11 simulated particles we computed 1000 sub-simulations with 1e8 particles using parallelization. Calculating a single 1e8 particles simulation took 3 to 4 hours depending on the considered patient’s anatomy. The number of particles selected for the noisiest input simulation corresponded to over 20% uncertainty in the PTV. Training a deep learning model required about 5 days on one GPU RTX 2080 Ti. Overall, generating the input simulations and training the model constituted a 90.5% gain of computational time over the MC calculations of ground-truth dose maps. However, the time bottleneck indeed remains in the MC calculations of the noisy simulations. Thus, we intend to assess to what extent the level of noise in the input simulations impacts the model’s performance.

To integrate this method in clinical practice, the required training data would be the patient’s CT and corresponding noisy simulations. The latter noisy MC simulations would be computed using a standard MC engine. The end goal is therefore to reduce the computational load required to compute the noisy simulations as much as possible while generating Monte-Carlo-like simulation at the level of clinical use. This step undoubtedly necessitates to reduce the number of simulated particles for the model’s input.

back to top

High-particle simulation of Monte-Carlo dose distribution with 3D ConvLSTMs