cramer lower bound, MAE, and MSE

Question

I am quite confused, is there any relation between MSE, CRLB and MAE.

Can we test the efficiency of a Maximum likelihood estimator with the MAE, or must we use CRLB?

Also is it true that CRLB is MSE?

Answer 1

The topics you bring up are at the heart of estimation theory. I highly recommend reading Steven Kay's Fundamentals of Statistical Signal Processing: Estimation Theory for a detailed background and plenty of practical examples. On to your question.

It is not uncommon in signal processing to want to estimate an unknown parameter in your data. Perhaps you have a signal that you are trying to detect $s[n]=A$ where you don't know the value of $A$ and you perform a measurement of $N$ values where you believe your measurement to be the sum of your signal and some noise term, $w[n]$:

$$x[n]=s[n]+w[n] \tag{1}$$

If the noise were considered to be drawn from a zero-mean Gaussian distribution, $N(0,\sigma^2)$, then you could propose an estimator of $A$ using purely the measurement $x[n]$. One such estimator may be using only the first sample you collect:

$$\hat{A}=x[0]\tag{2}$$

Alternatively, you may be slightly more clever and use the mean value:

$$\hat{A}=\frac{1}{N}\sum_{n=0}^{N-1}x[n]\tag{3}$$

There is nothing inherently wrong with either of these estimators, though one might give you an answer that is on average "closer" to the true value of $A$. This closeness of an estimate is what we define as an estimator's efficiency. The efficiency can be calculated in many ways, two of which you have mentioned: mean-squared error (MSE) and mean-absolute error (MAE), which are defined respectively for our estimator of $A$ as $$MSE=\mathbb{E}[(\hat{A}-A)^2]\tag{4}$$ and $$MAE=\mathbb{E}[|\hat{A}-A|]\tag{5}$$ where $\mathbb{E}$ is the expectation operator. These two methods are referred to as loss functions. The major difference between these two loss functions are how they penalize error. Note that the error of the $\operatorname{MSE}$ grows quadratically and the $\operatorname{MAE}$ grows linearly as $\hat{A}$ deviates from $A$. This means that for any two estimator we compare (including the maximum likelihood estimator), if estimator 1 on average tends to give estimates farther away from the true value (e.g. outliers) than the estimator 2, the $\operatorname{MSE}$ will penalize estimator 1 more harshly than the $\operatorname{MAE}$ metric. Scientists and engineers could just as easily preferred either loss function over the other, but because the $\operatorname{MSE}$ is continuous over the entire domain of the estimate (for most problems at least), it is more easily differentiated and the math works out more cleanly most of the time. So to answer one of your questions:

Can we test the efficiency of a Maximum likelihood estimator with the MAE?

Yes. You get to choose which loss function you want for comparing the efficiency of two or more estimators. Some loss functions, such as the $\operatorname{MSE}$, are more well respected than others, but you may have a situation where the $\operatorname{MAE}$ is more appropriate. That being said, the $\operatorname{MSE}$ has a close relation to the Cramer-Rao Lower Bound which makes it a good choice in general (read on to find out why).

You mention the Cramer-Rao Lower Bound (CRLB) in your question. Before addressing the specifics of your question, it is worth explaining what the CRLB is. The CRLB is precisely the smallest variance that an unbiased estimator can obtain, given certain conditions (other known/unknown parameters, number of samples in your measurement, etc.) on average. It is calculated via the inverse of the Fisher Information matrix, $I(A)$: $$\operatorname{var}[\hat{A}_{unbiased}]\ge \operatorname{CRLB}=\frac{1}{I(A)}\tag{6}$$ The Fisher information essentially quantifies how much information about your unknown parameter exists in the model of your measurement, where "model" refers to the type of noise you assume $w[n]$ to be and how it relates to your parameters. Intuitively, if there is a lot of information about your unknown parameter, $I(A)$ becomes large and the CRLB becomes very small and there is potential to get a very accurate estimate of $A$. Conversely, if not very much information exists about the parameter, your best-case variance will inevitably be large and your estimate will be poor.

Per Mark Leeds' excellent suggestion: It should be noted that the maximum likelihood estimator (MLE) is derived from the same log-likelihood function that the Fisher information matrix is derived. Because of this, the MLE has a property known as asymptotic normality, in which the CRLB is asymptotically attained as the length of the measurement increases. In addition to this property, the MLE is known for being relatively easy to derive. As Kay states: "For these reasons almost all practical estimators are based on the maximum likelihood principle."

is there any relation between MSE, CRLB and MAE?

Yes, between $\operatorname{MSE}$ the the CRLB (not so much for the $\operatorname{MAE}$). Given the expansion derived here, it can be stated that $$\operatorname{MSE}[\hat{A}]=\operatorname{var}[\hat{A}]+(\mathbb{E}[\hat{A}]-A)^2\tag{7}$$ If we know that our estimator is unbaised, or $$\mathbb{E}[\hat{A}]=A\tag{8}$$ then we can say that $\hat{A}=\hat{A}_{unbiased}$ and $(\mathbb{E}[\hat{A}]-A)^2=0$. Plugging these into $(6)$ and $(7)$, we can state that $$\operatorname{MSE}[\hat{A}] \ge \operatorname{CRLB} \tag{9}$$ If it so happens that your estimator attains the CRLB or $\operatorname{MSE}[\hat{A}_{unbiased}] = \operatorname{CRLB} $, then that estimator is called efficient. By the asymptotic normality property mentioned above, if the estimator $\hat{A}$ is based on the MLE, it will practically attain the CRLB in a mean-squared-error-sense as the signal/measurement length increases.

Also is it true that CRLB is MSE?

To reemphasize, the CRLB is a lower limit for the $\operatorname{MSE}$ only if the estimator is unbiased. They can equal each other, but this is only true for certain situations where the noise model belongs to the exponential family.