Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Yuanyuan Lyu, Jiajun Fu, Cheng Peng, S. Kevin Zhou

Abstract

Recently, both supervised and unsupervised deep learning methods have been widely applied on the CT metal artifact reduction (MAR) task. Supervised methods such as Dual Domain Network (Du-DoNet) work well on simulation data; however, their performance on clinical data is limited due to domain gap. Unsupervised methods are more generalized, but do not eliminate artifacts completely through the sole processing on the image domain. To combine the advantages of both MAR methods, we propose an unpaired dual-domain network (U-DuDoNet) trained using unpaired data. Unlike the artifact disentanglement network (ADN) that utilizes multiple encoders and decoders for disentangling content from artifact, our U-DuDoNet directly models the artifact generation process through additions in both sinogram and image domains, which is theoretically justified by an additive property associated with metal artifact. Our design includes a self-learned sinogram prior net, which provides guidance for restoring the information in the sinogram domain, and cyclic constraints for artifact reduction and addition on unpaired data. Extensive experiments on simulation data and clinical images demonstrate that our novel framework outperforms the state-of-the-art unpaired approaches.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_29

SharedIt: https://rdcu.be/cyhVc

Link to the code repository

N/A

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

The paper proposed a network for unpaird data, based on a dual-domain network for MAR. More concrete, the unpaired learning is done via cycle loss, with a 2-phase network. The network was conpared to other SOTA on simulated and real data. Furthermore, the authors did an abation study.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The results of the proposed method are promising.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The description of the method is not comprehensive.
Please rate the clarity and organization of this paper

Satisfactory
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

No code is given. The description of the parameters in the experiments are very confusing and not clear.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

I am very puzzled about the derivation of the additive property. as far as i understood, it is based on the assumption, that the artifacts appear mainly due to beam hardening. How far is this assumption valid? There should be also other effects, such as scattering for real data.

The method/network was not very clearly presented. While I see that the framework structure is complicated, it should be also/better described in words what is happening. The reader has a hard time to remember all the abbreviations. Furthermore, it is not clear, why so many losses are computed, and how the sum of all losses is back propagated. Also: it is stated that the P-net is pre-trained. where does it come from? (also not stated in the experiments).

They evaluate all data on supervised and unsupervised models. However, while they motivate the work with a bad generalization when training in simulated data, the paper does not underline this point. Furthermore, they also train the supervised methods on real data. So, if that data is available, what is the advantage of the proposed unsupervised approach? It would also be nice to have some kind of quantitative values for the real data. (Since you are mainly proposing you method for real data…) I would also like an more detailed discussion on the qualitative results.

Where does the metal mask come from? are all SOTA methods getting the same mask?

Unfortunately, it was not clear to me how the networks for the ablation study look. Maybe that could be explained in more detail (or with an additional image?)
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The network itself and the ideas behind it are not well explained. There are too many open questions to the method to present it to MICCAI.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

4
Reviewer confidence

Confident but not absolutely certain

Review #2

Please describe the contribution of the paper

This paper proposed a new dual-domain network (U-DuDoNet) for CT metal artefact removal (MAR). The motivation - decomposition of signal (i.e. additive property) is interesting. Three U-Nets are used to build the whole system. The method is evaluated on both synthetic and clinical datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

see below the 7th point
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

see below the 7th point
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

the code is not available.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The motivation is clear and interesting. Some details should be clarified, see below my concerns.

(1) How easy or hard to trade off the configuration of the loss functions? Do we really need that much regularization (cyclic, adversarial)? Although I understood the motivation for imposing each regularization, it is crucial to show the relationship of each loss versus the reconstruction performance/accuracy. The figure of the cost function (e.g. plot each regularization loss) is helpful to check if the methods under different configurations are converged and really well trained.

(2) I am curious that the Prior network (P-Net) seems to play the most important role in the method, e.g. I see there is a 1.2dB improvement of I^ca in Table 2 when P-Net was plugged in. Is the P-Net supervised trained? If so, the P-Net had already known the knowledge of missing data i.e. the null space of the imaging system (or data acquisition), then the method is not really unsupervised. If so, the contribution of this paper is very limited. I checked some recent unsupervised training for image reconstruction or artefact removal, e.g. see the below [1][2][3]. It is very important in the revision to clarify the links or difference between these related unsupervised reconstruction techniques [1][2][3] and the proposed here.

[1] ‘Unsupervised adversarial image reconstruction’, Pajot et al. 2019 [2] ‘Rare: Image reconstruction using deep priors learned without groundtruth’, Liu et al. 2020 [2] ‘Equivariant Imaging: Learning Beyond the Range Space’, Chen et al. 2021

(3) The notation system could be simplified for easy following. It is better to show the metric and error map in Fig 3,4. It is helpful to report the architectures (layers, Conv kernels, etc) and full training details for both P-Net, S-Net, I-Net and the whole system. Releasing the code and data should be helpful to the community.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea is interesting, the proposed method achieved good results on the synthetic and clinical datasets.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #3

Please describe the contribution of the paper

The authors improved the DuDoNet so that it didn’t require the paired training data (clean and metal artifact corrupted CT). The authors provide the proof of additive property of metal artifacts in CT images which justifies their loss function.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The main strength of this paper is that they adapt the DuDoNet for unpaired training data to improve the robustness of CT metal artifact reduction. Paired clean and metal-artifact corrupted CT is hard to acquire and therefore DuDoNet is mainly dependent on simulation data. This limits generalization to real CT data particularly with strong streaking artifacts. The proposed U-DuDoNet seems to tackle this challenge. It also proposes the novel formulation with cyclic metal artifact reduction constraint and sinogram synthesis which avoids the disentangled latent space learning with the mathematical proof of additive property for metal artifacts. The experimental validation on simulated and clinical CT data is convincing.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The main weakness of this paper is the lack of consideration of scanner geometry in CT reconstruction. Metal artifacts will be different based on CT back projection type (e.g., parallel, fan beam, helical). Previously, DuDoNet uses parallel beam conversion modules to deal with this challenge, but U-DuDoNet will be more sensitive to CT back projection variance in unpaired training data. This also creates the challenge in dealing with metal artifacts in z-direction.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The paper provides the details of tunable parameters for U-DuDoNet training.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The improvement on back projection module would be necessary to take account into the inconsistency in metal artifact behaviors in unpaired data from different CT scanner geometry. 3D extension will be recommended to capture the metal artifacts in z-direction. Providing code and data will be definitely plus if expanded into a journal version.
Please state your overall opinion of the paper

Probably accept (7)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is well written and easy to read. Mathematical derivation and cost function is well defined. The experimental validation on both simulated and real dataset is convincing. This paper is well motivated to tackled the challenge in creating paired training data in clinical CT. The proposed method is novel to adapt the existing DuDoNet with cyclic constraint and sinogram synthesis. The weakness on inversion module can be further improved.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

4
Reviewer confidence

Very confident

Review #4

Please describe the contribution of the paper
1. Proposed a novel framework for unpaired CT image metal artifact reduction.
2. Has quite comprehensive experiments results
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
1. A new framework for unpaired ct image metal artifact reduction.
Specifically, the author try to remove the artifact from both sinogram and ct image domain without paired data, which is quite useful to the clinical ct application. From this point, this paper is interesting.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
1. The main claim of this paper is not very solid. As the authors claims, one contribution of this paper is to prove that the metal artifact is additional. However, this proof is based on a incorrect assumption that “Comparing with metal, the attenuation coefficient of body tissue is almost constant with respect to E”. From my standpoint, the attenuation coefficient is change with to the E. Please give more explanation and reference.
2. The author claim that previous method (ADN) need complicated‘ encoder. However, from table 1, the the running time of proposed method is double of the ADN’s. ADN needs 38ms, while the proposed method need 63.59. From this point, the proposed method is seems like more complicated. Please give more explanation.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The detail of this paper is clear. If the author release the code, it would not difficult to reproduce the result of this paper.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html
Question:
1. Why need two components to represent the artifact. i.e., a_{I} and a_{s}. It seems like not obey the proof equation (3), in which the artifact only represented by one single term.
Suggestions:
1. The author could detail the symbol of figure 2 in legend.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The idea of remove the artifact from both sinogram and ct image domain in a unpaired manner is quite interesting.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

1
Reviewer confidence

Confident but not absolutely certain

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

The work proposes a deep learning method for metal artifact reduction from CT scans. The proposed work has several contributions: 1- the use of unpaired learning, 2- an improvement over SOTA, 3- evaluation on real and simulated data. The majority of the reviewers also acknowledge these contributions. However, the paper has some major weaknesses. Authors are invited for rebuttal to address these comments. Major points are summarized below.

Major More explanation about the additive property should be provided as in its current form it is not clear and convincing (Rev 1 and 5), This is the most important concern raised by two reviewers.

A statistical significance test, for PSNR and SSIM values, should be performed to report the p values. Especially between the proposed method and ADN for image domain CT data. A similar statistical significance analysis should also be performed for the quantitative values presented in Table 2.

The need for an unsupervised method should be better motivated. I understand the motivation of using paired clinical data not being available in real life. However, an unpaired supervised method could be developed as well if clinical data is available.

The authors should clarify the links or difference between related unsupervised reconstruction techniques (Rev2)

More details about the Pnet and how it is trained should be included (Rev 1and 2)

The authors are strongly advised to release the code for the reproducibility of the work.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

Author Feedback

We thank the reviewers and AC for their comments and will revise the paper accordingly.

Additive property (R1, R5) We agree with R1 that beam hardening and scattering both lead to streaks. However, Park’s work (TMI, 2016) shows that a well-modeled beam hardening corrector removes most streaks in a phantom experiment (see Fig. 7). Thus, we build the additive property based on such assumptions.

We agree with R5 that the attenuation coefficient always changes with energy. However, the coefficient of metal changes much rapidly comparing with soft tissues/bones (see Fig.2 in DuDoNet++). Thus, we follow DuDoNet++ and simplify the equations.

Unpaired learning (R1) Supervised learning must have paired ground truth for error back-propagation. Due to inevitable patient movement and breathing between different scans, paired clean and metal-affected images are impossible to obtain. Registration can reduce but not eliminate the mismatch, moreover, the presence of metal artifacts increases the difficulty of registration. Therefore we propose unpaired learning for MAR of real images.

R1 misunderstood our experimental setup on page 6 (Baseline). We train supervised models on simulation data only and test them on simulation and real data.

Comparison with other unsupervised methods (R2)

Liu’s work is related to noise2noise based on the assumption of zero-mean noise while we are not based on noise2noise at all.

Chen’s work achieves self-supervised learning via considering the commutative property of transformation and forward+inverse operators, which we don’t use.

While our framework shares a similar idea with Pajot’s work, that is, unsupervised learning is achieved in cyclic data recovery and degradation process, there have three differences: (i) A predefined degradation operator F would limit the performance when artifact modeling fails. Instead, U-DuDoNet learns to extract real metal artifacts and generate corrupted images via addition. (ii) Pajot’s work only learns from corrupted images, while our model also considers unpaired clean images/sinograms in which data statistics help to remove artifacts. (iii) Our model restores data in dual domains.

P-Net (R2) For training data generation, we randomly choose a metal mask and a clean sinogram (forward project a clean image) and then set the values of a clean sinogram within a metal trace to 0. Thus, the P-net is self-learned.

We respectively disagree with R2 that supervised P-Net leads to limited novelty: (i) The performance of P-Net itself is limited (PSNR/SSIM: 31.29dB/0.737). The extremely low SSIMs are due to strong secondary artifacts and smoothed tissue around the metal. (ii) We do not use the output of P-Net directly but plug P-Net softly in our framework to guide sinogram restoration, see the first term in Eq. (11). Also, Fig.3 in Supp shows that the guided S^a_p can only give a rough estimation. The details and edges with metal trace are well recovered in the restored S^a_se as cyclic unpaired learning and dual-domain learning also play important roles.

Geometry (R4) We agree that geometry affects MAR performance on real data.

(i) To accommodate an arbitrary input image with unknown geometry, U-DuDoNet obtains a sinogram using a simulated fan-beam geometry (320 angles x 320 detectors), which may differ from its real one. But the results show that our MAR method can apply well to a real image.

(ii) As shown by MAR results on multiple real CT images in Supp, U-DoDuNet with a coarse, simulated geometry would handle such geometry differences with a SE-IE architecture.

(iii) MAR in 3D geometry (helical, cone-beam) is still an open question.

Running time Our model is simpler than ADN since FP/FBP layers cost 38.41ms (58.6%) and the network only costs 27.18ms.

Statistical test We did T-Tests per request, which show that U-DuDoNet performs significantly better than ADN and RCN and variant models in ablation with T values > 23.02 and p values < 1e-103.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

Most of the comments were addressed by the authors. However, I encourage the authors to repeat the statistical significance tests for the SSIM values for the proposed vs ADN method. By looking at table 2 the difference is 0.008 and I don’t think the p value would be <0.05. This analysis should be included in the final version. Finally, MICCAI is encouraging reproducible research. Therefore, I hope the authors will make their code available (this is not stated in the rebuttal). Looking forward to reading the final improved version of this works and the authors’ presentation at the MICCAI meeting.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

5

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This paper works on unpaired learning framework for CT metal artifacts removal. The unpaired setting is important for metal artifacts removal since paired training data is hard to be obtained. The reviewers mostly feel positive but have concerns on statistical significance, role of P-Net, comparison with related works, scanner geometry in CT reconstruction, etc. The response clarified these concerns. This work tackles an important task in CT reconstruction under unpaired setting, and shows improvements. It can be accepted. The authors should carefully consider these comments in reviews, and clarified them in the final version. Moreover, the codes should be released for reproducibility.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The proposed methodology brings an improvement to the SOTA and improves the ability to remove artefacts in the process of image reconstruction. The responses of the authors to AC and reviewers is complete and properly addresses all the issues raised. Further, the authors provide additional results based on reviewers requests for quality of the proposed method.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

8

back to top

U-DuDoNet: Unpaired dual-domain network for CT metal artifact reduction