Paper Info Reviews Meta-Review Author Feedback Post-rebuttal Meta-Reviews

Authors

Rosa Maza-Quiroga, Karl Thurnhofer-Hemsi, Domingo López-Rodríguez, Ezequiel López-Rubio

Abstract

In this paper, a novel method to estimate the level of Rician noise in magnetic resonance images is presented. We hypothesize that noiseless images follow Benford’s law, that is, the probability distribution of the first digit of the image values is logarithmic. We show that this is true when we consider the raw acquired image in the frequency domain. Two measures are then used to quantify the (dis)similarity between the actual distribution of the first digits and the more theoretical Benford’s law: the Bhattacharyya coefficient and the Kullback-Leibler divergence. By means of these measures, we show that the amount of noise directly affects the distribution of the first digits, thereby making it deviate from Benford’s law. In addition, in this work, these findings are used to design a method to estimate the amount of Rician noise in an image. The utilization of supervised machine learning techniques (linear regression, polynomial regression, and random forest) allows predicting the parameters of the Rician noise distribution using the dissimilarity between the measured distribution and Benford’s law as the input variable for the regression. In our experiments, testing over magnetic resonance images of 75 individuals from four different repositories, we empirically show that these techniques are able to precisely estimate the noise level present in the test T1 images.

Link to paper

DOI: https://doi.org/10.1007/978-3-030-87231-1_33

SharedIt: https://rdcu.be/cyhVg

Link to the code repository

https://github.com/icai-uma/RicianNoiseEst_3DMRI_BenfordsLaw.git

Link to the dataset(s)

N/A

Reviews

Review #1

Please describe the contribution of the paper

This paper introduces a new method for measuring the noise level for a stationary Rician distributed MRI data using Benford’s law. Unfortunately, the contributions of the article are vague as they are more experimental rather than theoretical.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

I am so sorry to say, but the strengths of the paper are limited to the application of Benford’s law, which was not applied to the noise estimation problem before.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The Authors of the paper do not define the motivation behind their method. Why do they propose yet another technique for a stationary Rician case? I am not sure if estimating the stationary noise level still attracts the community as most of the data are acquired with parallel accelerated techniques such as the SENSE-based methods. In my opinion, the Authors should focus more on the so-called non-stationary (or a spatially variant) case, which seems to be important in current research and clinical studies (see [2-4, 6]).

I am afraid the authors limit their literature review on some selected methodologies leaving aside the essential papers in the field, such as [1-6]. For instance, the PCA method is applied in a non-local scenario leading to really good results for structural MRI data (see [6]).

Also, I am not convinced about adding an artificial noise to already noise-deteriorated MR data.

All in all, I think the methodology presented here is not so novel as the authors used Bendford’s law to describe the data in the Fourier domain and a regression technique to infer about the noise level in the data. Further, mixing Rician and Rayleigh components introduce a possible bias to the estimated noise levels.

Last but not least, the authors do not verify the methodology versus the state-of-the-art.

[1] Aja-Fernández, S., et al. (2009). Noise estimation in single-and multiple-coil magnetic resonance data based on statistical models. Magnetic resonance imaging, 27(10), 1397-1409.

[2] Pieciak, T., et al. (2017). Non-stationary Rician noise estimation in parallel MRI using a single image: a variance-stabilizing approach. IEEE transactions on pattern analysis and machine intelligence, 39(10), 2015-2029

[3] Tabelow, K. et al. (2015). Local estimation of the noise level in MRI using structural adaptation. Medical image analysis, 20(1), 76-86.

[4] Aja-Fernandez, S. et al. (2015). Spatially variant noise estimation in MRI: A homomorphic approach. Medical image analysis, 20(1), 184-197.

[5] Coupé, et al. (2010). Robust Rician noise estimation for MR images. Medical image analysis, 14(4), 483-493.

[6] Manjón, J. V. et al. (2015). MRI noise estimation and denoising using non-local PCA. Medical image analysis, 22(1), 35-47.
Please rate the clarity and organization of this paper

Poor
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The results should be reproducible as no complex operations are made in the paper and the authors exactly provide the databases used in this research.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

“It is already known that MRI noise follows a Rician distribution [7] and that there is around 60% underestimation of the true noise if the noise in MRI is assumed to be Gaussian.” – I am afraid that the sentence is not entirely correct as underestimating the noise level in a Rician distributed signal assuming a Gaussian model strongly depends on the SNR of the signal. It is roughly critical for SNR < 5.

“Although this wavelet model fits better Gaussian noise, it can be adequately modified [11] to estimate the σ parameter in Rician noise.” – I think this is not because of the model but a linear operation on a Gaussian distributed signal.

“Then, for each image, 20 new images with noised values were generated varying the distortion in the range [0; 10) percent of Rician noise, following a uniform continuous distribution” – This is a bit problematic as the Authors do not provide the real amplitude value of the signal; thus, it is not known what the effective SNR of the signal is. For instance, in Fig. 2, the values of 0.0 - 0.09 say nothing about the noise level as we still do not know the amplitude level. The Authors should use the percentage function as presented, for instance, in [5].

“Thus, we accept that a noiseless MRI in the Fourier frequency domain follows Benford’s law.” – I would say the data follows Benford’s law because of the natural characteristics of the foreground regions, nothing more. The more noise you add, the higher discrepancy from Benford’s law we observe; thus, the estimator seems to be more biased.

My suggestion is to start with a synthetic Rician distributed signal with different SNR levels, apply Benford’s law for each SNR separately and try to estimate the noise level for each SNR individually. Once checking the error (in %) for different SNR levels (let say between 5 and 30), one can apply the methodology for the in vivo MRI data. Notice that the more noise you add more affected the estimate you obtain because of Rayleigh distributed signals (i.e., the background signals). Consequently, I would suggest analyzing only foreground signals leaving aside any background regions as they might significantly bias the estimate. Finally, be sure that once you apply a 3D FFT, all the slices are represented by a cube, not by a cuboid.

All in all, the idea is interesting and might be resubmitted to one of the MICCAI Workshops, but it needs to be rethought very carefully to address all issues raised before.
Please state your overall opinion of the paper

reject (3)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The state-of-the-art is incomplete, the novelty of the paper is limited, no compact experimental set-up, and generally, the idea is not convincing as the Authors mix foreground and background pixels together while obtaining Benford’s law behaviour in the Fourier domain.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Very confident

Review #2

Please describe the contribution of the paper

Authors report a method to estimate the level of Rician noise in magnetic resonance images. The cornerstone of the paper is the hypothesis that noiseless images follow Benford’s law, that is, the probability distribution of 1st digit of the image values is logarithmic. The use of supervised machine learning techniques (linear regression, polynomial regression, and random forest) allows predicting the parameters of the Rician noise distribution using the dissimilarity between the measured distribution and Benford’s law as the input variable for the regression.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This was generically interesting new information for me to learn. Perhaps, a new angle at the denoising problem for the MRI crowd at this conference. It should be noted that the application of Benford’s law is not brand new, yet it is dismissed for some reason by the medical imaging community (except for some references which I found below).
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

Related work. Seems like the idea behind the work is in Benford’s law applied to the specific MRI tasks. Being not an expert in this domain of math, I googled that several works reported the use of the approach, Benford’s Law, particularly for medical imaging data [1], and to analytically modelling the effects that a particular PSF might produce on an edge between two tissues [2]. Specifically, [1] demonstrates how Benford-like behavior provides a good fit to the synthetic noise-corrupted data. Another paper that deserves citing is [3], from which the Methods section could benefit as it is “the proposed methodology to estimate Rician noise in 3D MRIs”. Works [4] and [5] report useful SOTA models, which help estimate Rice distribution precisely and robustly.

Similar papers: [1] https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4346748
Quantifying the Partial Volume Effect in PET Using Benford’s Law, IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 54, NO. 5, OCTOBER 2007 [2] Estimation of partial volume mixtures with a confidence measure combining intensity and gradient magnitude information , J Chiverton, K Wells - Medical Image Understanding and Analysis, 2005 [3] http://www.ic.uff.br/~aconci/ABB1410.pdf Estimating the Rician Noise Level in Brain MR Image [4] https://pubmed.ncbi.nlm.nih.gov/20417148/ “Robust Rician noise estimation for MR images” [5] https://link.springer.com/article/10.1186/s12880-019-0407-4 An enhanced adaptive non-local means algorithm for Rician noise reduction in magnetic resonance brain images

Rationale. Although an interesting (and probably fundamental) observation, the paper doesn’t discuss the rationale behind suggesting this approach. One may even suggest that this conference is a wrong venue for the work like this (an applied math conference?). While knowing the level of the noise in an image is important, it’s well-known that the noise distribution in MRI scans is Rician, and so one can assess the noise level just from estimating SNR and then fitting the Rician. Perhaps, the method allows for estimating these values with unprecedented precision, but – why bother if the real fluctuations overwhelm the pattern? (e.g., Fig. 5)
Please rate the clarity and organization of this paper

Good
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

Reproducibility of the submission is questionable due to the absence of any code attached. No supplementary data either.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

Please add a discussion about the rationale behind worrying about the digits distribution. Minor: certain sentenses (e.g., T1 imaging or 3D imaging) make the reader wonder if the method holds for just some cases, or if it is universally applicable to any imaging with Rician distribution. Larger fonts in Fig. 5, please. Consider adding the suggested references. Consider adding references in plots to the benchmark acronyms. The ending of the paper (acknowledgments) seems unfinished.
Please state your overall opinion of the paper

probably reject (4)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

As a non-expert in the Benford’s Law but an expert in general medical imaging denoising, I will resort to additional reading and will partake in the discussion after the rebuttle. I need to see reviews of the other reviewers for this one to put my final score. Meanwhile, I am grading this as borderline reject (5) being confused by 1) rationale, 2) applicability to MICCAI 3) potential lack of clinical impact; and yet – being motivated by the new knowledge that i gained while reviewing this work. I will be happy to reconsider my score after the rebuttle.
What is the ranking of this paper in your review stack?

4
Number of papers in your stack

5
Reviewer confidence

Somewhat confident

Review #3

Please describe the contribution of the paper

The authors propose using Benford’s law to estimate Rician noise in MR images. They perform experiments from four different datasets and estimate noise in T1w images.
Please list the main strengths of the paper; you should write about a novel formulation, an original way to use data, demonstration of clinical feasibility, a novel application, a particularly strong evaluation, or anything else that is a strong aspect of this work. Please provide details, for instance, if a method is novel, explain what aspect is novel and why this is interesting.

This contribution is timely given the focus on deep learning-based denoising to either accelerate MRI or handle other low SNR scenarios. The use of Benford’s law to estimate Rician noise is useful as most algorithms assume a level of noise not learned from the native data set to be denoised.
Please list the main weaknesses of the paper. Please provide details, for instance, if you think a method is not novel, explain why and provide a reference to prior work.

The dynamic range of the coefficients for varying levels of noise is limited (figs 2, 3,4). Introducing a weighting term to expand the range or discussing a potential method to make the coefficients more sensitive to noise level will make this more practically applicable.
Please rate the clarity and organization of this paper

Excellent
Please comment on the reproducibility of the paper. Note, that authors have filled out a reproducibility checklist upon submission. Please be aware that authors are not required to meet all criteria on the checklist - for instance, providing code and data is a plus, but not a requirement for acceptance

The authors have filled out the reproducibility checklist. However, some of the requirements do not apply given the nature of modeling and regression involved in this work.
Please provide detailed and constructive comments for the authors. Please also refer to our Reviewer’s guide on what makes a good review: https://miccai2021.org/en/REVIEWER-GUIDELINES.html

The work has been well thought out and is a current requirement for DL based denoising algorithms. The work might benefit from improving the dynamic range of the coefficients (sensitivity to noise level). At this stage, a discussion on strategies to increase the range is warranted.
Please state your overall opinion of the paper

strong accept (9)
Please justify your recommendation. What were the major factors that led you to your overall score for this paper?

The paper is important work that can expedite and aid multiple other groups interested in the broad area of DL based denoising of MR images with a Rician noise distribution.
What is the ranking of this paper in your review stack?

1
Number of papers in your stack

5
Reviewer confidence

Very confident

Primary Meta-Review

Please provide your assessment of this work, taking into account all reviews. Summarize the key strengths and weaknesses of the paper and justify your recommendation. In case you deviate from the reviewers’ recommendations, explain in detail the reasons why. In case of an invitation for rebuttal, clarify which points are important to address in the rebuttal.

This paper presents a method for measuring the noise level in stationary Rician distributed MRI T1w datasets. The method uses Benford’s law for the Rician noise level estimation with machine learning techniques. R1 mentions that the paper is interesting but has major concerns about the limited literature review, not high level of novelty, and experiment setup. R2 finds the method is generally interesting but mentions that the use of Benford’s law is not entirely new. R2 also has concerns about the method evaluation. R3 finds this work (machine learning-based denoising) is timely and important, and the use of Benford’s law is useful. After reading the comments, my recommendation is given in Q3 and that the authors could consider addressing the concerns on method novelty and evaluation, on top of the other concerns raised by the reviewers.
What is the ranking of this paper in your stack? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

7

Author Feedback

Answer to R1 about the novelty and the methodology employed: Our contribution is the hypothesis that the first digit of the noiseless MRI data in the frequency domain follows a Benford distribution. This mathematical hypothesis is accompanied by a method to actually estimate the noise level depending on how the first digit distribution deviates from Benford’s. Furthermore, the hypothesis and the associated method have been experimentally validated. This methodology to estimate the Rician noise level could be extended to the non-stationary case. We acknowledge that more comparisons could have been implemented, but the focus of this paper is to propose and validate the hypothesis that the first digit of the noiseless transformed MRI values follows Benford’s, and that this allows estimation of the noise level since noisy images deviate from Benford’s. We agree that the difference between Gaussian and Rician noise only gets critical for low values of the SNR, so the sentence should be corrected. On the other hand, our method is also applicable for Gaussian noise. Rician noise was used in the experiments to employ a more realistic noise model than the additive Gaussian noise, but there is no inconvenience in using our approach for Gaussian. Regarding the experimental evaluation, the notation was a bit unclear, i.e. 0.09 means 9% of noise with respect to the amplitude of the signal. Also, the experiments were done only for foreground pixels with very similar results, although we used the whole MR since we wanted to keep the procedure as simple as possible, avoiding preprocessing steps.

Answer to R2 about the rationale, applicability to MICCAI and potential clinical impact: Many of the quality control (QC) procedures in MRI brain imaging depend on the definition of image quality metrics (IQMs) to estimate the extent to which the different artifacts are present in the image and therefore could potentially lead to erroneous results in posterior processing stages, above all, multi-site clinical studies. There are several methods to estimate the amount of noise in an image, assuming in most cases specific statistical distributions of the noise component. Some of the methods to approximate the noise level need a priori information about the noise distribution or require to preprocess the image by segmenting it or by reducing its inhomogeneities. The idea and novelty are to provide a method to estimate the amount of noise, not by assuming the noise distribution, but hypothesizing that the first digit of the noise-free image follows Benford’s law when it is considered in the frequency domain. Thus, we provide an experimental evaluation of this hypothesis and find that noisier images deviate more from this distribution, and the deviation can be quantified by means of the Bhattacharyya coefficient and the Kullback-Leibler divergence. In terms of QC, we could say that this allows us to define two new IQMs for noise estimation. In addition, we propose the use of machine learning techniques to estimate the noise level from these coefficients, and find that, even with standard methods the estimation can be done with high precision. Our proposal was evaluated with datasets that may contain outlying subjects, which are seen in the fluctuations. They were detected, although we preferred to maintain the whole set to show the mean performance of our method and to preserve the reproducibility of the experiments. In addition, we created a public repository with scripts and sample data: https://anonymous.4open.science/r/X-0507 To sum up, our work presents an empirical demonstration of the proposed hypothesis of Benford’s law in the frequency domain and its novel use as a noise estimator with the help of machine learning, providing very promising results. It must be highlighted that more than one IQMs may be required to faithfully evaluate the noise level of an image. Therefore, our proposal is a significant achievement in the search for reliable IQMs.

Post-rebuttal Meta-Reviews

Meta-review # 1 (Primary)

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

I have read the rebuttal. The presented method aims to measure the noise level in stationary Rician MRI data based on Benford’s law and machine learning techniques. The paper is interesting. Although some concerns are addressed in the rebuttal, more works are needed to show the better performance of the method in clinical applications, e.g., non-stationary case.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Reject
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

12

Meta-review #2

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

This work might be more suited to IPMI than MICCAI because it is an interesting idea. Authors propose a method for estimating Rice noise in MRI, which could be plugged in as a component in several different algorithms, or simply reported as part of QC. The authors seem to have addressed concerns about novelty, and justified whether this type of applied mathematics work is appropriate for MICCAI.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

9

Meta-review #3

Please provide your assessment of the paper taking all information into account, including rebuttal. Highlight the key strengths and weaknesses of the paper, clarify how you reconciled contrasting review comments and scores, indicate if concerns were successfully addressed in the rebuttal, and provide a clear justification of your decision. If you disagree with some of the (meta)reviewer statements, you can indicate so in your meta-review. Please make sure that the authors, program chairs, and the public can understand the reason for your decision.

The authors test the hypothesis that Benford’s law can be used to estimate the amount of Rician noise in an MR image (when applied to the Fourier-transformed image). They show through experiments that machine learning techniques can indeed be used to predict the Rician noise parameter, sigma. Despite the reviewers’ reasonable critiques, I think the idea proposed in this paper can generate an interesting discussion and lead to an additional way to assess image quality.
After you have reviewed the rebuttal, please provide your final rating based on all reviews and the authors’ rebuttal.

Accept
What is the rank of this paper among all your rebuttal papers? Use a number between 1 (best paper in your stack) and n (worst paper in your stack of n papers).

2

back to top

Rician noise estimation for 3D Magnetic Resonance Images based on Benford's Law