Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Hong Wang, Yuexiang Li, Haimiao Zhang, Jiawei Chen, Kai Ma, Deyu Meng, Yefeng Zheng

Abstract

For the task of metal artifact reduction (MAR), although deep learning (DL)-based methods have achieved promising performances, most of them suffer from two problems: 1) the CT imaging geometry constraint is not fully embedded into the network during training, leaving room for further performance improvement; 2) the model interpretability is lack of sufficient consideration. Against these issues, we propose a novel interpretable dual domain network, termed as InDuDoNet, which combines the advantages of model-driven and data-driven methodologies. Specifically, we build a joint spatial and Radon domain reconstruction model and utilize the proximal gradient technique to design an iterative algorithm for solving it. The optimization algorithm only consists of simple computational operators, which facilitate us to correspondingly unfold iterative steps into network modules and thus improve the interpretablility of the framework. Extensive experiments on synthesized and clinical data show the superiority of our InDuDoNet. Code is available in \url{https://github.com/hongwang01/InDuDoNet}.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_11

SharedIt: https://rdcu.be/cyhUC

Link to the code repository

https://github.com/hongwang01/InDuDoNet

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The authors propose a proximal gradient approach to optimize metal artifact removal from CT images using dual domain (original and sinogram) images taking into account certain geometrical constraints. The proposed approach is then implemented using the unrolled network strategy.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. Improves the metal artifact removal in CT images.
2. Combines domain knowledge with the proximal gradient approach and parameter learning via neural nets.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The proposed approach does not show significantly large improvement on the real CT images (clinic) with average results comparable to the previous dual domain approaches. Including the standard deviation in addition to the mean values is important alongside a t-test to establish significantly improved performance.
2. Beyond outputting both Sn and Xn at each stage of the network, it is not clear what about the algorithm makes it interpretable.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

layer descriptions and learning parameters have been included and can potentially be reproduced.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
1. Both in the abstract and introduction, physical geometric constraints in network training are mentioned without much explanation. A sentence or two explaining this constraint in more detail in the introduction would be helpful in improving the motivation and flow of the paper.
2. The dice coefficients on the downstream segmentation tasks are similar for the different DuDoNet versions. A statistical test to check for significant difference can be performed.
3. What is the parameter size comparison for DuDoNet++ which has comparable performance on large artifact removal?
4. Reinforcement learning approach can be potentially used to automatically learn the optimal number of stages (N).
5. A discussion on why the proposed approach works so much better on small artifact removal than large artifact removal would be useful.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?
1. Effectively combining domain knowledge and parameter learning.
2. Unclear improvement on real world clinical images
3. Beyond block-wise visualization of the sinogram and image, the network interpretability is missing in that, it is not known what features in the images are being used.
What is the ranking of this paper in your review stack?

5
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

The method is a rational agglomeration of precedent work. The authors have utilized extensive experience with this problem, and utilized previous work and built upon it to create an interpretable Dual Domain Network for CT Metal Artifact Reduction (MAR). This is an important problem clinically. Previous MAR methods have not incorporated the CT imaging geometry constraint into the MAR and those methodes that utilize deep learning suffer from the ubiquitous interpretability issue. The loss function employed here has terms in the model space and data space simultaneously. They joint spatial and Radon domain reconstruction using the proximal gradient and sequential quadratice models to create an iterative algorithm. Since the optimization so constructed consists of relatively simple operations, it can be unfolded into a neural network with interpretability. Both synthesized and clinical data validate their approach.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The introduction has a nice review of the relevant literature. The method involves a dual domain network which incorporate intrinsic imaging geometry model constraints and spatial and Radon domain information. The chosen loss function is a dual domain minimization of the data residual in ‘sinogram’ (Radon) space plus a residual in sinogram space with two regularizing functionals with appropriate weight parameters. The normalized sinogram is used. The proximal operator is used in a kind of ‘alternating variables method’ – although not called that, which is a standard for sparse reconstruction problems. The sinogram (normalized) part is updated separately and then the model is updated. In both cases a quadratic approximation based on the previous iteration (or layer in NN) is used. First a Prior-net is used to learn the normalization coefficient for the sinogram normalization. Then using the above unrolled iterative sequential quadratic programming alternating variable approach is used to create the artifact-reduced sinogram and the CT image itself. The depth of the NN is varied (i.e. the number of iterations in the SQP method) in order to determine the optimal length – more depth gives more iterations but may lead to vanishing gradient problems. There is possibly a relationship to over convergence in the iterative procedure, but this is not pursued here. The model verification shows the metal artifacts in the CT image X are gradually removed as the layers in the Net are progressively reached and executed.
Finally the author compares their result with other recognized approaches: Linear interpolation, NMAR, deep learning based approaches CNNMAR a precursor called DuDoNet, also dual domain, . The dual domain approaches consistently out perform other more traditional methods. The author explicitly incorporates the CT geometry in the joint regularization of the sinogram and CT image at each stage of the NN. This method outperforms other SOTA methods in segmentation tasks as well The Dice coefficient is higher for all segmentation tasks except for the ‘Scrum”-presumably “sacrum” is meant - case and in this case it was within 0.0002 of the leader which was also a dual domain method. The author concludes correctly that the experimental and simulation results show the effectiveness of the dual-domain approach for metal artifact reduction. The superior interpretability is also demonstrated by the unrolling o the iterative process. This has clinical utility as well. The supplemental data is very important here. The results are shown in this part, which indicate the superiority of the method. The paper represents a substantial effort.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

There appears to be no guarantee that the learned regularizing functions are in fact convex, which it would seem is a prerequisite condition in order to utilize the proximal operator. The final optimization algorithm for the loss function is an alternating approach which is known to be susceptible to slow or thwarted convergence in some situations. The convexity of the major part of the functional probably saves the approach, although as stated there does not appear to be any guarantee that the learned regularizing functions - not the data misfit - in both domains, would be convex. The English is a bit rough throughout, although the algorithm is explained clearly, and occasional problems with English do not seriously affect the communication of important content related to the models, optimization method, or the results.
The use of the word ‘ablation’ in the section title “Ablation Study” is odd. Level reduction study would be more appropriate, less confusing, and accurate. A discussion of why the ‘alternating variables’ approach was used, i.e. its difficult to convert the true nonlinear optimisation into a DNN and this allows the dual mode optimization to take place. This is a minor point. Note that an iterative method can adaptively stop convergence based on some criteria, whereas the unrolled NN has fixed number of ‘iterations’. Therefore, the methods are not quite equivalent.
Scrum should read sacrum. other misprints or incorrect spellings occur. The supplemental data - at least some of the results - should have been included in the paper itself .
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors will release the code once the paper is accepted for publication. Some data and other code is available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Your paper has good scientific merit. It shows incremental but real improvement over other methods. Perhaps the english was a bit rough though. The analysis is nice. An explicit discussion of the alternating variables approach to minimization you take and why that is important would be good. See also ‘weakness’ section 4 and strengths in section 3 above in this review.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

There was incremental improvement. The analysis was clear. The scientific approach was solid. The author promises to reveal the code in the interests of reproducibility. This paper represents a substantial amount of effort. The improvement over state of the art methods (some of them) seem incremental however.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

This paper proposed a new unrolled deep learning method for metal artifact reduction and CT image reconstruction. The proposed method (InDuDoNet) achieved the SOTA results on the synthesized dataset and clinical database.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This paper is well-written, the idea is interesting and the experimental results are very promising.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

More details and discussions are necessary to clarify the proposed method. See the 7th point below for details.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

the code is claimed will be released.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Although I understood the physics-based unrolled network typically enjoy better interpretability, I have the below concerns:

(1) Does the parameters/weights of the ProxNet are shared in each learning stage/time step? It is necessary to clarify it.

(2) How to update the step size \eta?

(3) What’s the sample complexity for training the proposed method (e.g. the two ProxNets)?

(4) What’s the impact of the number of residual blocks in your ProxNet? I knew from below [1][2], which are very related to the idea presented here, that the number of residual blocks has a big impact on the learnability of such ProxNet in an unrolled network. It was demonstrated that ‘more residual blocks + fewer time step’ works as good as ‘fewer residual blocks + more time steps’. Please clarify this.

[1] Neural proximal gradient descent for compressive imaging, NIPS’2018 [2] Compressive MR Fingerprinting reconstruction with Neural Proximal Gradient iterations, MICCAI’2020
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper is well-written and the proposed method achieved the new SOTA results.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #4

Please describe the contribution of the paper

This paper proposed a joint spatial and Radon domain reconstruction network called InDuDoNet for metal artifact reduction task. The ablation studies suggest superior performance over the ablation methods.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This is a well-written paper. The authors build a data-driven framework while combining with physical constrained as optimization regularization terms. Extensive experiments are conducted on both synthetic dataset and clinical data, which shows superior performance of the proposed method over SOTA methods.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The authors performed a downstream U-net based segmentation task on the real clinical data. Although it still out stands against the competitive methods, the evaluation metric is different from the synthetic experiments, which make the generalization ability unclear.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors claim that the implementation will be released if paper gets accepts. The authors also refer to using released code and model in the ablation study papers.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
- The authors performed extensive ablation studies to show the superiority of the proposed method. The models are trained on synthetic dataset. My comment is that it would make the paper even stronger if a consistent evaluation metrics are selected for both synthetic data and real data. It will then reflect the generalization ability of the proposed method. The presented U-Net downstreaming segmentation task is still useful to show its clinically translation value. However, the dice coefficients are all very close (mostly ranging from 0.93~0.96), which makes it less significant to show the significance of the proposed method.
- There are many hyper parameters to be set in the proposed model (including \beta, \gamma, weight coefficients etc.) It is unclear how sensitive the model’s performance is relevant to these hyper parameters, and thus whether the method can be used for related image synthesis tasks. The authors may want to comment on the transferability of the proposed method.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

This paper is well-written and of high quality. The experiment results are strong. The idea of combining data-driven methods with physical models is of interest to solve related problems.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper proposed a new approach, named InDuDoNet, for CT metal artifact reduction. This new method achieved the SOTA results on both synthesized data and clinical data.

The key strengths include: 1) The new method combines domain knowledge with the proximal gradient approach and parameter learning via neural nets 2) Extensive experiments are conducted on both synthetic data and clinical data, which shows superior performance of the proposed method over SOTA methods.

The key weaknesses include: 1) The final optimization algorithm for the loss function is an alternating approach which is known to be susceptible to slow or thwarted convergence in some situations. The convexity of the major part of the functional probably saves the approach, although as stated there does not appear to be any guarantee that the learned regularizing functions - not the data misfit - in both domains, would be convex.

Based on the above, I suggest the authors provide some feedback to address these.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Author Feedback

Thank all reviewers for their constructive comments. It is pleased to see that R4, R6, and R7 consistently agree on the contribution of this work— the method has superior interpretability (R4, R6, R7) and the experiments are strong (R6, R7). Below we summarize the main comments and address them one by one.

Q1: Discussion of the convexity, convergence, and reason about using the alternating variable method. (R4) A1: [Convexity] In the traditional optimization theory, the convexity of a regularizer is indeed a prerequisite condition for using the proximal operator. However, the concept of proximal mapping was extended to nonconvex functions to some extent in practice (refer to “Hare et al, Computing proximal points of nonconvex functions, MP’2009”). Hence, the convexity of the learned regularizer may not be strictly necessary for using the proximal operator.

[Convergence] Inspired by an alternating variable method, we build the deep unrolling network. Similar to the existing deep unrolling methods, e.g., (ADMMNet, NIPS’2016); (ISTANet, CVPR’2018); (primal-dual, TMI’2018), our framework is not exactly equivalent to an optimization algorithm and thus we cannot either guarantee the theoretical convergence. We will deeply explore this open question in the future.

[Reason] 1) The alternating variable method is simple and achieves some success in previous works [1]. 2) As shown in Page 4, the method only contains simple operators, which are easily unfolded into network modules. 3) Such a deep unrolling framework has shown clear interpretability and achieves great success in other tasks ([28] and (MHFNet, TPAMI’2020)).

Q2: Consistent metric for all data (R7) and statistical t-test/standard deviation (STD) for the downstream task. (R1) A2: We want to kindly clarify that the clean data are not available in the CLINIC-metal dataset, since the paired data (artifact-free and metal-corrupted CT) are extremely hard to obtain in clinical practice. Hence, instead of PSNR/SSIM for synthetic data, we use DC on the downstream clinical task to measure the performance of MAR. For the paired t-test suggested by R1, the P-value is 0.0243 for DuDoNet++ and less than 0.001 for other baselines. All P-values are less than the significance level 0.05 and thus our model performs better than baselines. The STDs for DuDoNet++ and our method are 0.0232 and 0.0208, respectively.

Q3: Details on network parameter (R1 and R6), the update of \eta, the impact of the number of Resblocks T, and sample complexity. (R6) A3: The parameter size of reimplemented DuDoNet++ (25,983,627) is much larger than our network (5,174,936) where the parameters of ProxNet are not shared among different stages. The step size \eta is a learnable parameter, which is updated together with network weights in an end-to-end manner. For T, a similar trend to [NIPS’2018 given by R6] is observed—the average PSNR with different N and T are (N=20, T=2, 42.55); (N=10, T=4, 41.48); (N=4, T=10, 41.06); (N=2, T=20, 40.89), respectively. As we use the same protocol to generate training data, the complexity is similar to DSCMAR [30].

Q4: The impact of weighting coefficients and the transferability of the model. (R7) A4: Our model has been verified to be insensitive to the hyperparameters. By using a simple MSE loss between ground-truth X_gt and the reconstructed image X_N without any weighting terms, our model still achieves the average PSNR/SSIM (41.23/0.9897), outperforming DuDoNet++. The robustness of our model thus increases its transferability for more tasks, like low-dose CT.

Q5: What features make InDuDoNet interpretable? (R1) A5: The interpretability is reflected by the design of the entire framework rather than obtaining some certain image features. The unrolled InDuDoNet benefits from the structured iterative algorithm and the physical imaging model. It’s thus rational to say that InDuDoNet has better interpretability than a pure neural network (both R4 and R6 agreed on this claim).

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors have addressed the main concerns from the reviewers and I believe this paper will benefit the CT field and the new algorithm presented in this paper may be of interests for the MICCAI community at large.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The paper proposes an interesting method based on proximal optimization to combine domain knowledge for CT metal artifact reduction. Reviewers have addressed the reviewers’ concerns. Despite some clarity issues, the paper seems to sufficient novelty and interesting application. We recommend accepting this paper for publication.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Basically, the reviews are positive and consistent. As summarized by the primary AC, the novelty and validation are recognized, and the authors’ response clarifies some details especially on the convexity and convergence. In summary, I agree to accept this paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

1

back to top

InDuDoNet: An Interpretable Dual Domain Network for CT Metal Artifact Reduction