Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Pranjal Sahu, Hailiang Huang, Wei Zhao, Hong Qin

Abstract

Medical image reconstruction algorithms such as Penalized Weighted Least Squares (PWLS) typically rely on a good choice of tuning parameters such as the number of iterations, the strength of regularizar, etc. However, obtaining a good estimate of such parameters is often done using trial and error methods. This process is very time consuming and laborious especially for high resolution images. To solve this problem we propose an interactive framework. We focus on the regularization parameter and train a CNN to imitate its impact on image for varying values. The trained CNN can be used by a human practitioner to tune the regularization strength on-the-fly as per the requirements. Taking the example of Digital Breast Tomosynthesis reconstruction, we demonstrate the feasibility of our approach and also discuss the future applications of this interactive reconstruction approach. We also test the proposed methodology on public Walnut and Lodopab CT reconstruction datasets to show it can be generalized to CT reconstruction as well.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87234-2_6

SharedIt: https://rdcu.be/cyl71

Link to the code repository

https://github.com/PranjalSahu/InteractiveSmoothingDBT

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This manuscript proposed an interactive method to optimize the smoothing parameter in digital breast tomosynthesis reconstruction using CNN.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The modified U-Net incorporates the regularization parameter beta as input to adjust the smoothness of the output image.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

1）The image quality depends on multiple parameters including the smooth parameter, iteration number, noise level, number of sampled data, etc. However, this manuscript only considered the smoothing parameter beta, making it hard to be applied in practical application. 2）The authors didn’t compare the results of the proposed approach with other related methods.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The implementation detailed is clear in the manuscript.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

1）Please consider other parameters such as iteration number, noise level, number of sampled data, etc, in this method, or have a discussion about the impact of these factors on the image quality. 2）Please compare the proposed approach with other related works. 3) In the bottom row of Fig. 5 in supplementary, the last three columns of the U-Net outputs present obvious bright dots-like artifacts. 4) In the middle row of Fig. 5 in supplementary, a higher resolution is suggested.
Please state your overall opinion of the paper

borderline reject (5)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

1）The method didn’t consider other important parameters. 2）The authors didn’t compare the results of the proposed approach with other related methods.
What is the ranking of this paper in your review stack?

6
Number of papers in your stack

3
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

This work introduces a modified UNet trained to mimic the visual characteristics of reconstructed volumes using different smoothing parameter (beta) values. The result was demonstrated on DBT and CT images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.
- The paper is fairly well organized and easy to follow.
- The idea of interactive recon parameter tuning in quasi-real time is a clinically interesting idea
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.
- The paper is limited to small DBT patches, while extension to full sized volumes is yet to be addressed.
- This paper did not compare the CNN generated volumes vs real ones in terms of task-based lesion detectability, which is clinically more relevant.
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance
- Code will be made public
- A private dataset (will some descriptions) & 2 public datasets were used
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Figure 2 shows examples of DBT patches. it would be nice to show some CNN processed entire DBT slices & perform visual analysis on visibility of lesions, presence of artefacts etc. Illustrations of some CT slices can be found in supplemental materials, the same in-depth analysis would also be of interest.

The evaluation of the quality of CNN generated reconstruction volumes remains very subjective in this paper. There were quantitative metrics used in this work, but they are not task-based, therefore have limited value in clinical translation. This paper mentioned masses and calcifications are key radiological signs in breast imaging (there are actually also architectural distortions and asymmetries), but only calcifications were illustrated in DBT patches. Perhaps in a future work, a full investigation of mass & calcification detectability should be performed to compare CNN generated volumes vs real ones.
Please state your overall opinion of the paper

borderline accept (6)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The research in this paper feels very preliminary. The result is not well quantified and the clinical translation is unclear.
What is the ranking of this paper in your review stack?

3
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

The authors propose an interactive smoothing parameter choice for reconstruction in digital breast tomosynthesis. The method is based on the realization that different clinicians prefer different levels of smoothness and sometimes even need more for an proper assessment. The method is tested on a large DBT database as well as on 2 publicly available CT datasets.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

The authors provide the option to choose the level of regularity of obtained reconstructions based on an external input parameter (for smoothness). This is drastically different from most learned reconstructions, where only the quantitatively best reconstruction is provided. The presented approach is based on the idea that reconstructions of different visual quality are needed for different scenarios.

The approach is tested on in-house clinical DBT measurement data and further tested for 2 publicly available CT datasets providing a good overview of the capabilities of the method.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The incorporation of the smoothing parameter in the network (a slightly modified U-Net) is not particularly well motivated from a theoretical view-point. The smoothing parameter is passed through a shallow NN, basically learning a 1D nonlinear function relation between the smoothing parameter and the incorporated multiplicative factor s. At least to me, it is not clear why this resulting multiplicative factor s has the effect of augmenting different regularities.
Please rate the clarity and organization of this paper

Very Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The networks and design choices for the experiments are well described. The DBT dataset is not public, but the 2 other test sets are publicly available and experiments can be reproduced. Authors will publish codes.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The choice to first reconstruct with a classic iterative algorithm (using no regularization) and then incorporate different levels of regularization by a network is interesting. At this point, I might worry that the initial reconstruction without regularization is highly noisy, due to the missing regularization. How do the authors avoid this? Do you employ an early stopping to overcome this.

One question that one might ask is why the authors chose to use PWLS over simple FDK reconstructions. The iterative model-based nature of PWLS should overcome some of the limited-view artefacts more effectively and the network then only needs to incorporate the desired smoothing. In contrast to post-processing from a FDK reconstruction, where limited-view artefacts need to be compensated for as well. Maybe this would be worth discussing shortly.

Finally, there have been recent studies on “learned iterative schemes” for DBT that can improve reconstruction quality considerably: J. Teuwen et al. “Deep learning reconstruction of digital breast tomosynthesis images for accurate breast density and patient-specific radiation dose estimation” Medical Image Analysis, 2021.

This might be interesting to look into for the authors. Nevertheless, the primary problem for such learned approaches are extensive memory requirements for high-resolution reconstructions. As such the presented work here provides a low-cost solution.
Please state your overall opinion of the paper

accept (8)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

Accept: The idea to provide reconstructions with varying regularity depending on the need of the clinician is interesting. The method is well tested on clinical and public experimental data. The whole methodology is not entirely novel, but here executed with clinical relevance, thus I recommend acceptance.
What is the ranking of this paper in your review stack?

2
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

All three reviewers agree that interactive recon parameter tuning in quasi-real-time is a clinically interesting idea, since choosing the different levels of smoothness for reconstruction visual quality can be needed for different scenarios. However how a scalar beta that is input into the shallow NN leads to augmenting reconstruction results is unclear and theoretically not well-motivated. Furthermore, reviewers also expressed concerns about the lack of consideration on other parameters in addition to smoothing parameters and lack of diagnostic evaluation and comparison with related methods.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

3

Author Feedback

Due to space limit, we only address the major comments raised by reviewers (R1, R2, R3, & M).

R1: “method didn’t consider other important parameters.” & “didn’t compare … with other related methods.”

A: The iteration number (another parameter) also impacts the image quality, however, in practice, the smoothing parameter has maximum impact on the detectability of calcifications, hence we considered it. Given a fixed number of iterations, the image noise is a decreasing function of beta. The proposed CNN model can only produce a smoother version of an image from a noisier version, but not vice-versa. We would experiment, in the future, if the model could also generate an earlier (smoother) iteration result from a later (noisier) iteration result. Other parameters such as sampled data (angular range) are fixed in a hardware setting, and we are not focusing on such design choices in this study. It would have been better if the reviewer could suggest a related work to compare because, to the best of our knowledge, there is no current work that tackles this problem in DBT/CT setting. All the current CNN-based de-noising or reconstruction models only output a single image with no control over any parameter, moreover, due to high resolution & high memory requirement, the available learning-based reconstruction approaches for CT are not directly applicable in the DBT domain (refer to R3’s comments). R1 requested to see a higher resolution image (of Fig 5), which is now available here: https://ibb.co/KqktvwV

M & R3: “how a scalar beta that is … leads to … is unclear and theoretically not well-motivated.”

A: We observe that in PWLS reconstruction, the penalty functions such as Huber & Quadratic, essentially pull the values of nearby pixel/voxel towards a common value (while preserving signal ex. in Huber). Nearby pixel variation decreases monotonically as value of beta increases (see Fig 2). We hypothesize that this can be simulated by a non-linear CNN model. The CNN model acts on a non-regularized image & performs a series of convolution-activation operations. The CNN is parameterized by a scalar value beta which modulates the outcome of each convolution layer allowing control on the smoothing strength. Results on three different datasets support our hypothesis and highlight the proposed approach’s limitations (Table 2). In DBT, the influence of voxels outside the cropped patch has a substantial impact for higher beta values which the network isn’t able to capture. This could be improved with full-size input to CNN (Sec. 4). We selected the iteration such that the image with zero regularization is not overly noisy (early-stop). FDK as initial input can be explored in future as it can also reduce the initial reconstruction time.

M & R2: “lack of diagnostic evaluation” & “did not compare … detectability.”

A: We quantify the similarity of the model’s output & PWLS result and have plotted the line profile over several calcifications in Fig. 2. It shows that the model performs well for a wide range of calcification sizes. For further validation, we obtain the mean CNR (co-relates with detectability) of 10 known calcification (0.196mm) in the CIRS phantom for few beta values (beta: 0.101,0.243,0.338,0.406 UNet: 6.43,6.71,6.13,5.88 PWLS (Huber): 7.22,7.86,6.54,5.67).

UNet-based model produces a smoother image compared to PWLS (Fig 2 NPS) and results in lower CNR, however, the trend with varying beta is the same for both, which is useful for determining the peak beta without performing reconstruction again.

R2: “only calcifications were illustrated in DBT …” & “clinical translation is unclear”

A: We shared one DBT mass lesion sample in the supplementary document. Since masses are large, their detectability is not hindered much by noise. The proposed model works on full DBT images by processing them in a patch-by-patch way. Code will be shared on Github (that can help future research on this topic) upon its final acceptance.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

In the rebuttal, authors of this paper have adequately addressed the four major concerns from the reviewers.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

6

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The idea overall is quite intriguing, although simple with limited technical novelty. Sharing the reviewers’ concerns, it is not clear to me technically why the estimation of regularization strength should work as a CNN post-processing: This process assumes (and the rebuttal also states) the regularization to only have a local smoothing effect. However, the regularization in the optimization problem can couple effects across the entire image, through the kernel of the forward problem, e.g. for tomography a streak on the bottom caused by a strong attenuator on the top cannot be removed easily in post-processing by simply considering local effects at the bottom alone. A conventional regularization in the optimization problem can however encourage smoothing on the bottom, while the forward model finding an alternative solution to “explain” the streak with the spot on the top and thereby practically removing the streak. Therefore it is counter-intuitive how a Unet with limited receptive field can capture such effects across the entire image. This feels almost as impossible as tomographic reconstruction itself with a Unet alone.

The authors conclude from Table 2 that the method performs well, but what metric values should be considered as successful and not is hard to define. For example, if one assumes 0.95 to be an acceptable SSIM and PSI, then most of DBT results would fail that threshold. So, without any baseline or upper-bound, it is not clear how to interpret the evaluation results, and one can argue both ways based on them.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

12

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I think the authors have responded to all major comments, except the comemnt about theoretical justification for a scalar beta. This paragraph is a bit murky to me. Given that the paper seems to be written reasonably and the application is of great clinical interest/relevance, I would be inclined to accept this paper.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

4

back to top

Interactive smoothing parameter optimization in DBT Reconstruction using Deep learning