Need help in understanding minimum variance estimator and CRLB concept using an example

I am reading the book, Fundamentals of Statistical Signal processing, Estimation Theory Volume 1 by Steven M. Kay. In Chapter 2, there is a Fig 2.5 which illustrates which estimator to select based on the CRLB. Basically, the estimator with less variance is preferred since its pdf is more concentrated about the true value. I have certain doubts regarding this statement and the term "inconsistent estimator".

Considering 2 estimators for a parameter $\theta$ found using two estimation technique. Estimation technique A is the ML estimator and B is the ordinary least square having the CRLB denoted as CRLB_A and CRLB_B for the unknown parameter $\theta$ respectively.

Doubt 1: At high SNR, the CRLB_A > CRLB_B. So, should I select CRLB_B at high SNR? For low SNR, it is the reverse.

Doubt 2: Let the $mse(A) = E[(\Delta \theta \Delta \theta^T)] < CRLB_A$

What does this imply? Does it mean that the ML estimator of A is inconsistent?

2 Answers

You may want an estimator with minimun variance because each time you calculate it, it is likely to be closer to its expectation, and in the case that the estimator is unbiased, closer to the true parameter. An estimator is inconsisten if increasing the number of samples of the dataset to calculate it doesn't reduces its variance.

The sample mean estimator of X: $$\bar X = \frac{1}{N}\sum_{i=0}^{N-1}x_i$$ is unbiased and its variance is: $$\mathbb var(X)=\frac{\sigma^2}{N}$$ It is clear that as N increases, the variance gets smaller, so it is consistent.

Regarding question 1, NO. For instance, one of the two estimators can have a higher CRLB always, for both high or low SNR. The criteria for choosing an estimator depends on the problem, the prior knowledge about it, the computation constraints and so on. Normally it is desired a minumn variance and a non-biased estimator, i.e. an efficient estimator. But, if your decision is only based on the CRLB, then choose that one with the smaller one.

Take this 'silly' example: for a dataset you decide that the estimator for the mean is always (arbitrarily) 3.5 even if you know nothing about the data and you do no calculations... It is clear that this estimator has variance 0, because it will be always 3.5, but it is strongly biased, it is randomly biased indeed... is it a good estimator? No, although it has 0 variance. The key point here is that yo can't have an unbiased estimator with a lower variance than its CRLB (if the CRLB exists). If the variance of your estimator is lower than the CRLB, it is biased.

If you have some experience with computer simulation, Matlab or similars, it is a good exercise running Montecarlo simulations and plotting the variance and the CRLB of each realisation versus the SNR for a fixed number of samples, or/and for a fixed SNR plotting the variance and the CRLB of the estimator versus different dataset lengths.

• Thank you for an easy and lucid explanation. 3 quick questions : (1) Can CRLB tend to zero and if so what does that signify? (2) If the estimator is found to be biased, then is its performance inferior? (3) does unbiased estimator also imply efficient estimator? by efficient I understood after reading is when the variance of the estimator reached the CRLB asymptotically. Thank you for your help
– SKM
Oct 27, 2016 at 17:10
• There are biased estimators which have smaller variances. A very readable article is "Rethinking Biased Estimation" by Steven Kay and Yonina Eldar, IEEE Sig Proc magazine vol 25, #3, 2008. Another thing which is interesting is Stein's paradox. These things are a little more advanced - if you are just learning stick with unbiased estimation. Note - you can also derive CRLB for biased estimation. Oct 28, 2016 at 14:09

Fristly, check my answer again, I edited it because the first paragraph wasn´t extrictly right, I hope you notice the difference.

(1) YES, in the example of the sample mean, its variance it is also the CRLB, so if N goes to infinity, the CRLB tends to zero. It means that if you want fewer deviation from the expectation of the estimator, you nedd larger datasets.

(2) In general YES, but... imagine the bias is something you can control, for instance, you are measuring distances with a meter that starts in 1 and not 0. Your estimator is biased, but you can control it. What usually happens is that the bias depends on the parameter in question and things get messy. Normally you will avoid bias and high variance, but these considerations have to be taken carefully.

(3) NO, efficient estimator requires both unbiasness and minimun variance. If the estimator is unbiased and reachs the CRLB asymptotically, then you can say it's asymptotically efficient.