1
$\begingroup$

In Fundamentals Of Statistical Signal Processing: Estimation Theory page 19, Kay mentions a biased mean square error estimator for $\mu$ where the samples $x\sim\text{N}(\mu, \sigma^2)$.

The suggested estimator is $$\hat{\mu}=\frac{a}{N}\sum_{i=0}^{N}x_i$$ where optimal $a$ is given by $$a=\frac{\mu^2}{\mu^2 + \sigma^2/N}$$

He also mentions that those kind of estimators are not practical due to the dependency of the parameter $a$ on the unknown parameters $\mu$ and $\sigma^2$.

Now, I have tried to implement this estimator iteratively, using python/numpy and it provided me with some decent results.

nTests = 100000
unbiasError = np.empty(nTests)
biasError = np.empty(nTests)

for test in range(nTests):
    x = np.random.normal(loc=1, scale=10, size=1000)
    #init
    a = 1
    aHist = 0.5
    flag = 1
    while (np.abs(a - aHist) > 0.00001):
        if flag == 0:
            meanEst = x.mean() * a
        else:
            meanEst = x.mean()
            flag = 0
        varEst = ((x - meanEst)**2).sum() / (x.size - 1)    
        aHist = a
        a = (meanEst**2) / (meanEst**2 + varEst / x.size)
    biasError[test] = meanEst - 1
    unbiasError[test] = x.mean() - 1
print("mean error of the   biased estimator: {:.2f}\nmean error of the unbiased estimator: {:.2f}\n".format(
    np.abs(biasError).mean(), np.abs(unbiasError).mean()))

mean error of the biased estimator: 1.23

mean error of the unbiased estimator: 2.52

I have to admit that it works better on low SNR (For $\sigma^2=10$ the unbiased estimator performs better). Therefore, it is not clear to me if it is completely not practical to use such estimators. Are there clear criteria for the stability and performance of such methods? Is there any known better implementation for such estimator?

.

$\endgroup$
1
  • $\begingroup$ With your iteration, if you try $x=\{1,2\}$ there are two real fixed points: $(\mu,\sigma)=(1,1)$ and $(\mu,\sigma)=(1.40436,0.518294)$. The second seems to be unstable (ie. iteration goes away from there) but the first might not be what you want. The original method has some counterintuitive properties: adding $k$ to every sample, estimating the mean, and then subtracting $k$ changes the result. You ought to have a good rationale for finding this acceptable before using this method. $\endgroup$
    – Dan Piponi
    Jan 16 at 21:46

2 Answers 2

0
$\begingroup$

I doubt the optimal a is ${\mu^2/(\mu^2+\sigma^2/N)}$, it seems the result is unsatisfactory. It's better to use average.

$\endgroup$
4
  • $\begingroup$ Have you tried running the code? Also,the math indicates the optimality. Have you reviewed it? $\endgroup$ Aug 18, 2023 at 13:13
  • $\begingroup$ I can't run python code. I read your code and I think in it varest is not the estimate of variance but mean square error. $\endgroup$
    – c1119
    Aug 19, 2023 at 9:47
  • $\begingroup$ While you are right, we can say it is an approximation or estimator for the true variance. In fact, I have tried replacing it with an unbiased estimation of the variance and got the same results. I have also tried using the unbiased estimation for $\mu$ and got a slightly different estimation which works better in the region left to the intersection and worse to the right of the intersection. So, I keep in mind that there are different methods for estimation of those parameters and they all perform differently but all are legit :) $\endgroup$ Aug 19, 2023 at 13:49
  • $\begingroup$ You can try similarly $\hat{\sigma^2}=\sum_{i}(x_{i}-\mu)^2/(N+2)$. $\endgroup$
    – c1119
    Aug 20, 2023 at 2:21
0
$\begingroup$

I assume that the uncirtenty in the estimation of the parameters $\mu$ and $\sigma^2$ introduces a reduction in the performance but a simple sensitivity test shows that for sufficiently low SNR, the biased estimator performs better. enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.