0
$\begingroup$

Assume the following problem: A deterministic signal $X$ whose magnitude is known to satisfy $0 \leq \Vert X \Vert_2 < \Delta$ for some known constant $\Delta$ is transmitted through a Gaussian Channel with known standard deviation $\sigma$.

It is known that the ideal observer performance on the detection problem (detectability) or ROC-AUC is upper-bounded by the SNR which in this case is at most $\left(\Delta \over \sigma\right)^2$.

My question: What does the SNR tell us about the error in estimating $X$? By Cramér-Rao, it's evident that the variance of an unbiased estimator will be at least $\sigma^2$, but how can we use the fact that the SNR is bounded within a certain interval? It should improve the estimation by introducing bias. I want to provide an exact equation relating the SNR to the error of the estimator.

$\endgroup$
4
  • $\begingroup$ While you're right, Cramér-Rao only holds for unbiased estimation, adding a bias does not inherently reduce variance! Now, I do agree, there's probably something to be won here. $\endgroup$ Aug 8, 2022 at 15:09
  • $\begingroup$ are you really interested in "what can we read from a bounded SNR?" (because, that's just going to bound the variance), or in "what can we read if we know the support of $X$ and the PDF of the noise?". The latter can give you a smaller variance, but it's not what you ask for in the letter. $\endgroup$ Aug 8, 2022 at 15:36
  • $\begingroup$ your SNR formula needs to be squared, by the way! $\endgroup$ Aug 8, 2022 at 15:44
  • $\begingroup$ @MarcusMüller To be honest with you, what I am hoping for is a bound of the type: "If your hypothesis test/ detection AUC is at most A, then your MSE can be as low as B, where B is some function of the SNR". Also good catch on the square :) $\endgroup$
    – Sami
    Aug 8, 2022 at 18:55

1 Answer 1

0
$\begingroup$

Going directly for the variance is hard, because we don't know the pmf/pdf/cdf/moment-generating function/characterstic function/… of your $X$.

What we can do, instead, is going through entropy, and in the end look at the worst case for that. Let's assume $X$ is continuously-valued and has a differentiable cumulative density function $F_X$ and hence a probability density function $f_X$.

  1. The (differential) entropy $h(X)$ (in bits) of $X$ is bounded to $\log_2 \Delta$. \begin{align} h(X):=\ & \int-\log_2(f_X(x) )f(x)\,\mathrm dx\\ =\ &\int\log_2\left(\frac1{f_X(x)}\right) \mathrm dF_X(x)\\ &\text{and we make that log well-defined by}\\ &\text{applying it only where the pdf takes non-}\\ &\text{zero values (outside, the entropy contri-}\\ &\text{buted is 0, anyways) and is bounded (we}\\ &\text{required $X$ to be continously distributed)}\\ =\ &\int_0^{\Delta}\log_2\left(\frac1{f_X(x)}\right) \mathrm dF_X(x)\\ &\text{Now, we realize $\log$ is always a concave}\\ &\text{functions, and thus, }\textbf{Jensen's Inequality}\\ &\text{applies (yeah, measurability is given by $f$}\\ &\text{being the derivative of $F$).}\\ \le\ &\log_2\left( \int_0^\Delta \frac1{f_X(x)}\mathrm dF(x) \right) \tag{☮}\label{le} \\ =\ &\log_2\left( \int_0^\Delta \frac1{f_X(x)} f(x) \mathrm dx \right)\\ =\ &\log_2\left( \int_0^\Delta 1 \mathrm dx \right)\\ =\ &\log_2\Delta \end{align}
  2. Since noise is independent to signal, $h(Y)=h(X+N)=h(X) + h(N)$, with $h(N)=\frac 12\log_2(2\pi e\sigma^2)+\frac12$: $$h_\max(Y) = \frac 12\log_2(2\pi e\sigma^2\Delta^2) +\frac12 .$$

At this point, it's usually good to actually stop – knowing the variance tells us less about our estimator than knowing its entropy - but if we want to make conclusions on the optimal estimator variance:

  1. The weak inequality $\eqref{le}$ becomes an equality for $1/f_X$ being a constant (without proof, but easy to test), so the distribution with the highest possible variance for a random variable $X$ with finite support $[0,\Delta]$ is the uniform distribution $f_X(x) = 1/\Delta, x\in[0,\Delta], f_X(x) = 0\text{ else}$. This makes the maximum likelihood estimator for this worst case easy to derive – and you should be able to read the variance from that.
$\endgroup$
7
  • $\begingroup$ Thanks! I'm not hell-bent on the variance, I'm happy with some other estimate of the error for the estimation problem (e.g. (M)MSE). My main question is (cf. comment above) whether a bound on detection (which depends only on SNR) implies a bound on estimation which involves the SNR. $\endgroup$
    – Sami
    Aug 8, 2022 at 19:00
  • $\begingroup$ Also, regarding the parameterisation of the problem: $X$ is assumed to be a Dirac impulse which we may assume to be parameterised by some other variable $\alpha$, but all that does is to move around the mean of the resulting Gaussian. That said, it may be of interest to consider what an error bound on estimating $\alpha$ may be as opposed to $X$. $\endgroup$
    – Sami
    Aug 8, 2022 at 19:02
  • $\begingroup$ That would only require you to calculate the transformed variable's pdf; this is done using the Jacobi determinant of the transformation and the raw variable. $\endgroup$ Aug 8, 2022 at 19:11
  • $\begingroup$ One thing seems a bit counter-intuitive, although the derivation makes sense: You would expect that conditioning (i.e. the fact that we know that $\Delta^2 \over \sigma^2$ is bounded) should decrease uncertainty, whereas the noise in the channel should increase it. The way it looks, the higher the magnitude of the signal ($\Delta^2$), the higher the uncertainty becomes? $\endgroup$
    – Sami
    Aug 9, 2022 at 10:48
  • $\begingroup$ yeah, sure! The more the signal can vary, the less you know about it! $\endgroup$ Aug 9, 2022 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.