Kalman Filter | Difference Between Minimizing the Mean Square Error (MMSE) & Maximizing Likelihood Value in Bayesian Estimation

Question

I am going through data assimilation slides on Multi Sensor Data Fusion by Hugh Durrant Whyte and it mentions:

The Kalman Filter, and indeed any mean-squared-error estimator, computes an estimate which is the conditional mean; an average, rather than a most likely value. (Q: what is the most likely value ?)

I understand what MSQ is, but what does it mean that Kalman Filter estimates mean-square error rather then most likely value? Isnt mean square estimate, the most likely value?

Answer 1

Actually the first section of the notes in the link your provided are about the most likely value in the Bayesian Framework.

So we have a comparison between the Minimum Mean Square Error (MMSE) Estimator and the Maximum a Posterior Estimator.
Both are Bayes Estimator, namely they are a loss function of Posterior Probability:

$$ \hat{\theta} = \arg \min_{a} \int \int l \left( \theta, a \right) p \left( \theta, x \right) d \theta d x $$

Where $ \theta $ is the parameter to be estimated, $ \hat{\theta} $ is the Bayesian estimator, and $ l \left( \cdot, \cdot \right) $ is the loss function. The above integral called the Risk Integral (Bayes Risk).

With the the properties of Bayes Rule it can be shown:

$$\begin{aligned} \arg \min_{a} \int \int l \left( \theta, a \right) p \left( \theta, x \right) d \theta d x & = \arg \min_{a} \int \int l \left( \theta, a \right) p \left( \theta \mid x \right) p \left( x \right) d \theta d x && \text{By Bayes rule} \\ & = \arg \min_{a} \int \left( \int l \left( \theta, a \right) p \left( \theta \mid x \right) d \theta \right) p \left( x \right) d x && \text{Integral is converging hence order can be arbitrary} \\ & = \arg \min_{a} \int l \left( \theta, a \right) \left( \theta \mid x \right) d \theta && \text{Since $ p \left( x \right) $ is positive} \end{aligned}$$

So now, the solution depends on the definition of the loss function $ l \left( \cdot, \cdot \right) $:

• For $ l \left( \theta, a \right) = {\left\| \theta - a \right\|}_{2}^{2} $ we have the MMSE estimator which is given by the conditional expectation $ E \left[ \theta \mid x \right] $. This is what Kalman Filter estimates.
• For $ l \left( \theta, a \right) = {\left\| \theta - a \right\|}_{1} $ we have the Median of the posterior as $ \arg \min_{a} \int \left| \theta - a \right| \left( \theta \mid x \right) d \theta \Rightarrow \int_{- \infty}^{\hat{\theta}} p \left( \theta \mid x \right) d \theta = \int_{\hat{\theta}}^{\infty} p \left( \theta \mid x \right) d \theta $.
• For $ l \left( \theta, a \right) = \begin{cases} 0 & \text{ if } \left| x \right| \leq \delta \\ 1 & \text{ if } \left| x \right| > \delta \end{cases} $ (Hit or Miss Loss) we need to maximize $ \int_{\hat{\theta} - \delta}^{\hat{\theta} + \delta} p\left( \theta \mid x \right) d \theta $ which is maximized by the Mode of the posterior - $ \hat{\theta} = \arg \max_{\theta} p \left( \theta \mid x \right) $ which is known as the MAP Estimator.

As you can see above, different estimators are derived from different loss.

In the case the posterior is Gaussian the Mode, Median and Mean collide (There are other distributions which have this property as well). So in the classic model of the Kalman Filter (Where the Posterior is also Gaussian) the Kalman Filter is actually the MMSE, The Median and the MAP Estimator all in one.

Derivation with More Details

To show full derivation we will assume $ \theta \in \mathbb{R} $ just for simplicity.

The $ {L}_{2} $ Loss

We're after $ \hat{\theta} = \arg \min_{a} \int {\left( a - \theta \right)}^{2} p \left( \theta \mid x \right) d \theta $. Since it is smooth with respect to $ \hat{\theta} $ we can find where the derivative vanishes:

$$\begin{aligned} \frac{d}{d \hat{\theta}} \int {\left( \hat{\theta} - \theta \right)}^{2} p \left( \theta \mid x \right) d \theta & = 0 \\ & = \int \frac{d}{d \hat{\theta}} {\left( \hat{\theta} - \theta \right)}^{2} p \left( \theta \mid x \right) d \theta && \text{Converging integral} \\ & = \int 2 \left( \hat{\theta} - \theta \right) p \left( \theta \mid x \right) d \theta \\ & \Leftrightarrow \hat{\theta} \int p \left( \theta \mid x \right) d \theta \\ & = \int \theta p \left( \theta \mid x \right) d \theta \\ & \Leftrightarrow \hat{\theta} = \int \theta p \left( \theta \mid x \right) d \theta && \text{As $ \int p \left( \theta \mid x \right) d \theta = 1 $} \\ & = E \left[ \theta \mid x \right] \end{aligned}$$

Which is the conditional expectation as required.