3
$\begingroup$

I am quite confused, is there any relation between MSE, CRLB and MAE.

Can we test the efficiency of a Maximum likelihood estimator with the MAE, or must we use CRLB?

Also is it true that CRLB is MSE?

$\endgroup$
4
  • 1
    $\begingroup$ Hi: the CRLB is an LB for the variance of an estimator. The MSE is a metric used to estimate the quality of a model's predictions. It is sometimes unbiased ( depends on model ) and used as an unbiased estimate of the variance of the model's predictions. MAE is another metric that has the same goals as the MSE but it's not unbiased. So, in summary, the CLRB is not really related to MSE and MAE except that the MSE could reach the value of the CLRB in certain cases. Also, for your second question, you should use CRLB not the MAE. But CRLB is tricky to calculate. It's very model dependent. $\endgroup$
    – mark leeds
    Jan 21, 2022 at 14:32
  • $\begingroup$ hello! Thank you for the response, but why should I use MSE/CRLB for testing the efficiency of an ML estimator? $\endgroup$
    – Blobmou
    Jan 21, 2022 at 16:15
  • $\begingroup$ Hi: Maybe you can give the details of the model you are using because I don't know what you mean by MSE/CRLB. The CRLB is a theoretical construct but you can use the expected information to calculate the CRLB for the variance of the ML estimates of the parameters of a particular density. Then, ( assuming mean is zero ) you calculated the expectation of the MSE and see if it is equal to value of the CRLB calculated previously. If it is, then, then the MSE does reach the CRLB. Maybe DSP field looks at things differently but this what is done in statistics. $\endgroup$
    – mark leeds
    Jan 22, 2022 at 2:04
  • $\begingroup$ Check out this link. It's a lot more detailed ( and probably more correct ) than what I said above. google.com/… $\endgroup$
    – mark leeds
    Jan 22, 2022 at 2:13

1 Answer 1

6
$\begingroup$

The topics you bring up are at the heart of estimation theory. I highly recommend reading Steven Kay's Fundamentals of Statistical Signal Processing: Estimation Theory for a detailed background and plenty of practical examples. On to your question.

It is not uncommon in signal processing to want to estimate an unknown parameter in your data. Perhaps you have a signal that you are trying to detect $s[n]=A$ where you don't know the value of $A$ and you perform a measurement of $N$ values where you believe your measurement to be the sum of your signal and some noise term, $w[n]$:

$$x[n]=s[n]+w[n] \tag{1}$$

If the noise were considered to be drawn from a zero-mean Gaussian distribution, $N(0,\sigma^2)$, then you could propose an estimator of $A$ using purely the measurement $x[n]$. One such estimator may be using only the first sample you collect:

$$\hat{A}=x[0]\tag{2}$$

Alternatively, you may be slightly more clever and use the mean value:

$$\hat{A}=\frac{1}{N}\sum_{n=0}^{N-1}x[n]\tag{3}$$

There is nothing inherently wrong with either of these estimators, though one might give you an answer that is on average "closer" to the true value of $A$. This closeness of an estimate is what we define as an estimator's efficiency. The efficiency can be calculated in many ways, two of which you have mentioned: mean-squared error (MSE) and mean-absolute error (MAE), which are defined respectively for our estimator of $A$ as $$MSE=\mathbb{E}[(\hat{A}-A)^2]\tag{4}$$ and $$MAE=\mathbb{E}[|\hat{A}-A|]\tag{5}$$ where $\mathbb{E}$ is the expectation operator. These two methods are referred to as loss functions. The major difference between these two loss functions are how they penalize error. Note that the error of the $\operatorname{MSE}$ grows quadratically and the $\operatorname{MAE}$ grows linearly as $\hat{A}$ deviates from $A$. This means that for any two estimator we compare (including the maximum likelihood estimator), if estimator 1 on average tends to give estimates farther away from the true value (e.g. outliers) than the estimator 2, the $\operatorname{MSE}$ will penalize estimator 1 more harshly than the $\operatorname{MAE}$ metric. Scientists and engineers could just as easily preferred either loss function over the other, but because the $\operatorname{MSE}$ is continuous over the entire domain of the estimate (for most problems at least), it is more easily differentiated and the math works out more cleanly most of the time. So to answer one of your questions:

Can we test the efficiency of a Maximum likelihood estimator with the MAE?

Yes. You get to choose which loss function you want for comparing the efficiency of two or more estimators. Some loss functions, such as the $\operatorname{MSE}$, are more well respected than others, but you may have a situation where the $\operatorname{MAE}$ is more appropriate. That being said, the $\operatorname{MSE}$ has a close relation to the Cramer-Rao Lower Bound which makes it a good choice in general (read on to find out why).

You mention the Cramer-Rao Lower Bound (CRLB) in your question. Before addressing the specifics of your question, it is worth explaining what the CRLB is. The CRLB is precisely the smallest variance that an unbiased estimator can obtain, given certain conditions (other known/unknown parameters, number of samples in your measurement, etc.) on average. It is calculated via the inverse of the Fisher Information matrix, $I(A)$: $$\operatorname{var}[\hat{A}_{unbiased}]\ge \operatorname{CRLB}=\frac{1}{I(A)}\tag{6}$$ The Fisher information essentially quantifies how much information about your unknown parameter exists in the model of your measurement, where "model" refers to the type of noise you assume $w[n]$ to be and how it relates to your parameters. Intuitively, if there is a lot of information about your unknown parameter, $I(A)$ becomes large and the CRLB becomes very small and there is potential to get a very accurate estimate of $A$. Conversely, if not very much information exists about the parameter, your best-case variance will inevitably be large and your estimate will be poor.

Per Mark Leeds' excellent suggestion: It should be noted that the maximum likelihood estimator (MLE) is derived from the same log-likelihood function that the Fisher information matrix is derived. Because of this, the MLE has a property known as asymptotic normality, in which the CRLB is asymptotically attained as the length of the measurement increases. In addition to this property, the MLE is known for being relatively easy to derive. As Kay states: "For these reasons almost all practical estimators are based on the maximum likelihood principle."

is there any relation between MSE, CRLB and MAE?

Yes, between $\operatorname{MSE}$ the the CRLB (not so much for the $\operatorname{MAE}$). Given the expansion derived here, it can be stated that $$\operatorname{MSE}[\hat{A}]=\operatorname{var}[\hat{A}]+(\mathbb{E}[\hat{A}]-A)^2\tag{7}$$ If we know that our estimator is unbaised, or $$\mathbb{E}[\hat{A}]=A\tag{8}$$ then we can say that $\hat{A}=\hat{A}_{unbiased}$ and $(\mathbb{E}[\hat{A}]-A)^2=0$. Plugging these into $(6)$ and $(7)$, we can state that $$\operatorname{MSE}[\hat{A}] \ge \operatorname{CRLB} \tag{9}$$ If it so happens that your estimator attains the CRLB or $\operatorname{MSE}[\hat{A}_{unbiased}] = \operatorname{CRLB} $, then that estimator is called efficient. By the asymptotic normality property mentioned above, if the estimator $\hat{A}$ is based on the MLE, it will practically attain the CRLB in a mean-squared-error-sense as the signal/measurement length increases.

Also is it true that CRLB is MSE?

To reemphasize, the CRLB is a lower limit for the $\operatorname{MSE}$ only if the estimator is unbiased. They can equal each other, but this is only true for certain situations where the noise model belongs to the exponential family.

$\endgroup$
5
  • 1
    $\begingroup$ A very nice explanation and way better than mine. thanks. Maybe you want to add that, asymptotically, the MLE always reaches the CRLB, so the MSE should be asymptotically efficient if its based on an MLE. $\endgroup$
    – mark leeds
    Jan 22, 2022 at 13:13
  • $\begingroup$ @markleeds, Thank you! I have added your suggestion to the answer. $\endgroup$
    – Ash
    Jan 23, 2022 at 7:21
  • $\begingroup$ glad to help to make a great answer even better. $\endgroup$
    – mark leeds
    Jan 23, 2022 at 14:28
  • $\begingroup$ @markleeds Nicely done, both! 👍🏽 $\endgroup$
    – Peter K.
    Jan 23, 2022 at 19:20
  • $\begingroup$ The CRLB can also be derived for biased estimation as well. See the 3rd paragraph in en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound - it is also derived further on in the article. $\endgroup$
    – David
    Jan 24, 2022 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.