I am going through data assimilation slides on Multi Sensor Data Fusion by Hugh Durrant Whyte and it mentions:

The Kalman Filter, and indeed any mean-squared-error estimator, computes an estimate which is the conditional mean; an average, rather than a most likely value. (Q: what is the most likely value ?)

I understand what MSQ is, but what does it mean that Kalman Filter estimates mean-square error rather then most likely value? Isnt mean square estimate, the most likely value?


Actually the first section of the notes in the link your provided are about the most likely value in the Bayesian Framework.

So we have a comparison between the Minimum Mean Square Error (MMSE) Estimator and the Maximum a Posterior Estimator.
Both are Bayes Estimator, namely they are a loss function of Posterior Probability:

$$ \hat{\theta} = \arg \min_{a} \int \int l \left( \theta, a \right) p \left( \theta, x \right) d \theta d x $$

Where $ \theta $ is the parameter to be estimated, $ \hat{\theta} $ is the Bayesian estimator, and $ l \left( \cdot, \cdot \right) $ is the loss function. The above integral called the Risk Integral (Bayes Risk).

With the the properties of Bayes Rule it can be shown:

$$\begin{aligned} \arg \min_{a} \int \int l \left( \theta, a \right) p \left( \theta, x \right) d \theta d x & = \arg \min_{a} \int \int l \left( \theta, a \right) p \left( \theta \mid x \right) p \left( x \right) d \theta d x && \text{By Bayes rule} \\ & = \arg \min_{a} \int \left( \int l \left( \theta, a \right) p \left( \theta \mid x \right) d \theta \right) p \left( x \right) d x && \text{Integral is converging hence order can be arbitrary} \\ & = \arg \min_{a} \int l \left( \theta, a \right) \left( \theta \mid x \right) d \theta && \text{Since $ p \left( x \right) $ is positive} \end{aligned}$$

So now, the solution depends on the definition of the loss function $ l \left( \cdot, \cdot \right) $:

  • For $ l \left( \theta, a \right) = {\left\| \theta - a \right\|}_{2}^{2} $ we have the MMSE estimator which is given by the conditional expectation $ E \left[ \theta \mid x \right] $. This is what Kalman Filter estimates.
  • For $ l \left( \theta, a \right) = {\left\| \theta - a \right\|}_{1} $ we have the Median of the posterior as $ \arg \min_{a} \int \left| \theta - a \right| \left( \theta \mid x \right) d \theta \Rightarrow \int_{- \infty}^{\hat{\theta}} p \left( \theta \mid x \right) d \theta = \int_{\hat{\theta}}^{\infty} p \left( \theta \mid x \right) d \theta $.
  • For $ l \left( \theta, a \right) = \begin{cases} 0 & \text{ if } \left| x \right| \leq \delta \\ 1 & \text{ if } \left| x \right| > \delta \end{cases} $ (Hit or Miss Loss) we need to maximize $ \int_{\hat{\theta} - \delta}^{\hat{\theta} + \delta} p\left( \theta \mid x \right) d \theta $ which is maximized by the Mode of the posterior - $ \hat{\theta} = \arg \max_{\theta} p \left( \theta \mid x \right) $ which is known as the MAP Estimator.

As you can see above, different estimators are derived from different loss.

In the case the posterior is Gaussian the Mode, Median and Mean collide (There are other distributions which have this property as well). So in the classic model of the Kalman Filter (Where the Posterior is also Gaussian) the Kalman Filter is actually the MMSE, The Median and the MAP Estimator all in one.

Derivation with More Details

To show full derivation we will assume $ \theta \in \mathbb{R} $ just for simplicity.

The $ {L}_{2} $ Loss

We're after $ \hat{\theta} = \arg \min_{a} \int {\left( a - \theta \right)}^{2} p \left( \theta \mid x \right) d \theta $. Since it is smooth with respect to $ \hat{\theta} $ we can find where the derivative vanishes:

$$\begin{aligned} \frac{d}{d \hat{\theta}} \int {\left( \hat{\theta} - \theta \right)}^{2} p \left( \theta \mid x \right) d \theta & = 0 \\ & = \int \frac{d}{d \hat{\theta}} {\left( \hat{\theta} - \theta \right)}^{2} p \left( \theta \mid x \right) d \theta && \text{Converging integral} \\ & = \int 2 \left( \hat{\theta} - \theta \right) p \left( \theta \mid x \right) d \theta \\ & \Leftrightarrow \hat{\theta} \int p \left( \theta \mid x \right) d \theta \\ & = \int \theta p \left( \theta \mid x \right) d \theta \\ & \Leftrightarrow \hat{\theta} = \int \theta p \left( \theta \mid x \right) d \theta && \text{As $ \int p \left( \theta \mid x \right) d \theta = 1 $} \\ & = E \left[ \theta \mid x \right] \end{aligned}$$

Which is the conditional expectation as required.

| improve this answer | |
  • $\begingroup$ Good comprehensive answer however I fail to see how is MMSE $ \left( \theta, a \right) = {\left\| \theta - a \right\|}_{2}^{2}$ is given by conditional expectation, cause expectation is just taking average given x. But the square term you are defining is the root mean square loss ${\left\| \theta - a \right\|}_{2}^{2}$ $\endgroup$ – GENIVI-LEARNER Apr 10 at 15:36
  • $\begingroup$ Also if posterior is Gaussian then does it mean the Mean, Mode and Median are all same? I really thought that the values at the tail of the gaussian curve are the Modes as they have low probability but the x's are large compared to the mean. $\endgroup$ – GENIVI-LEARNER Apr 10 at 15:39
  • 1
    $\begingroup$ @GENIVI-LEARNER, I added the derivation for the $ {L}_{2} $ case. Yes, as I wrote above for Gaussian PDF the Mean equals the Median which equals the Mode. Actually for all symmetric distributions the Mean equals to the Median. If you add the property of Uni Modality with the Peak at the symmetric point you get Mean will equal the Median which will equal the Mode. You should read at Wikipedia - Mode. $\endgroup$ – Royi Apr 10 at 16:00
  • 1
    $\begingroup$ @GENIVI-LEARNER, Keep being generous and I will always be happy to try to assist you. If you have more questions, feel free. $\endgroup$ – Royi Apr 10 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.