What and where in the derivation of the Kalman filter is the assumption of Gaussian noise? Why and how does this assumption help?


A Gaussian process is completely specified by its mean and variance. A Kalman Filter updates the process mean which is the state, and its variance. These are the sufficient statistics.

The measurement noise is reduced but the process noise is part of the recursive state history and is tracked.

Your heading question and subsequent paragraph aren't fully consistent. One can use a linear Kalman Filter when the noise isn't Gaussian in many circumstances but it wouldn' be optimal. The derivation assumes Gaussian (or Normal) noise and all deterministic inputs are known, as well as knowing initial values.

  • 1
    $\begingroup$ well, the definability by the first two moments goes for a lot of distributions, so I'm not convinced this is strictly an answer. $\endgroup$ – Marcus Müller Nov 3 '17 at 8:51
  • $\begingroup$ Exponential Family is only family to have finite dimension sufficient statistic, so the choice isn’t that arbitrary. $\endgroup$ – user28715 Nov 3 '17 at 8:59
  • $\begingroup$ The Gaussian is also the distribution with a finite variance that has maximum variance $\endgroup$ – user28715 Nov 3 '17 at 9:04
  • 2
    $\begingroup$ That last point I think hits the spot – the Gaussian actually has maximum entropy, so it's the "worst case still working" distribution. $\endgroup$ – Marcus Müller Nov 3 '17 at 9:34
  • 1
    $\begingroup$ Without actually getting the book, the few online papers by West, specifically invoke Normal distributions. Given the Normal's conjugate distribution is a Normal, the simple derivations I've seen use the conjugate distribution property. He also likes to call it a so-called Kalman filter so, again without hitting the library, its hard to see what is easy o the terms are talking about the same thing, a Classic Kalman Filter. $\endgroup$ – user28715 Nov 3 '17 at 17:53

First of all let us assure that a Kalman filter (estimator) does not only remove Gaussian noise, but can remove (with certain success) any other type of noise as long as it's designed accordingly.

However, what lies at the heart of the standard Kalman filter is the linear estimator; and that linear estimator will be the optimum minimum mean square error estimator, when the noises are Gaussian.

Furthermore the update equations associated with those recursive state estimations are structured on (the expectation of) various matrices associated with state estimation errors and noises on them. The Kalman update matrix equations which are derived based on a solution to yield minimum expected error square sense are most easily obtained, and yield the optimum estimation, when the noise is assumed to be white, Gaussian, uncorrelated, and independent etc. Without these assumptions, such as correlated noise or dependent states, or non Gaussian distributions, the analytic solution to reach minimum error will be much more complex, than the standard form. Have a look at the derivation of a standard linear Kalman estimation filter and see those assumptions and simplifications it does.

However, this is not posing a great disappointment from the Kalman filter applications point of view. Well, because, practically most Kalman filters are designed to be used in environments where the Gaussian, white, uncorrelated noise assumtions are typical, if not always.

Furthermore, the Gaussian noise displays quite many features, that a few are listed below, that make it a suitable choice for certain engineering applications.

  1. Most typical natural (physical) noise sources can be modeled with a Gaussian pdf.
  2. Central Limit Theorem underlines the importance of the Gaussian distribution
  3. Gaussian process has certain properties that aid in the simplified solution of the Kalman filtering equations. Such as for jointly Gaussian random variables, uncorrelatedness implies independence and independence implies uncorrelatedness.
  4. The linear estimator $\hat{Y} = a X + b$ becomes the best (minimum) mean square error ($\mathcal{E} \{ (Y - \hat{Y})^2 \}$) estimate of the R.V. $Y$ in terms of RV $X$, if they are jointly Gaussian. For other distributions, the optimum estimator with minimum mean square error is the conditional expectation of Y given X; $\mathcal{E} \{ Y | x \}$ which is nonlinear.

Finally, and yes; you can derive a set of update relations which constitude an optimal estimator for another given type of noise, correlated, non Gaussian etc. That will be difficult though.

  • $\begingroup$ let's scratch 3, says wikipedia $\endgroup$ – Marcus Müller Nov 3 '17 at 9:35
  • $\begingroup$ The missing word is jointly Gaussian .. $\endgroup$ – Fat32 Nov 3 '17 at 9:46
  • 1
    $\begingroup$ ahhh, yeah, in that case, indeed, we can't end up in one of these corner cases. $\endgroup$ – Marcus Müller Nov 3 '17 at 9:51

Assumption of a Gaussian process allows us to obtain optimality. This uses the facts

  • A linear( or better affine) map takes a Gaussian random variable and maps to another Gaussian random variable.
  • A linear combination of two jointly Gaussian random variables is again a Gaussian random variable.

So we don't have to track the mean and the variance. If we don't assume Gaussianness then there might be arbitrarily better estimators especially nonlinear ones which defeats the purpose of using a Kalman filter. If optimality is not guaranteed then my guess is as good as Rudi's.

In control theory, there are significant objections to modeling the process noise processes as Gaussians but that's another story.

  • 1
    $\begingroup$ Gaussian noise is both the best sort of noise to have and the worst sort of noise to have! :-) It's the best because of the simplicity of dealing with it analytically. It's the worst because who wants real-life noise to have a value close to infinity with non-zero probability!??! $\endgroup$ – Peter K. Nov 3 '17 at 18:43

The Kalman Filter properties allows is to be the best linear estimator (What you call removing noise) for any White Noise (Not only Gaussian White Noise).

The idea of Kalman Filter is estimating the Mean and Covariance of the State Vector at each iteration.
Since optimal linear estimator (For the MMSE criteria) are based on the Mean and Covariance, as long as the Kalman Filter does this correctly it is optimal among the family of linear estimators.
The nice thing is for a linear system model the Kalman Filter does it well for any White Noise.

The thing with the case with Gaussian Noise is that Gaussian Process can be fully represented by its mean and covariance.
Namely once you estimate those you did the best (In the MMSE sense) which is achievable.
So we have something which is the optimal linear estimator to estimate the first two moment s and in the case of Gaussian noise this all that needed -> In the Gaussian case the Kalman Filter is the best among any estimator (Be linear or non linear).

The assumption of Gaussianity is reacquired, in the derivation of the Kalman Filter, in order to keep it the best MMSE estimator regardless of being linear.
This is done by the fact that a Jointly Guassian Vector going through a Linear System will result in a Jointly Gaussian Vector with a simple way to estimate its Mean and Covariance. In the case of Gaussian Process that is all needed.

  • $\begingroup$ Royi: One other thing: DSP people are way better than me as far as understanding the subtleties of the term "white noise" ( i.e: derivative of brownian motion and an abstract quantity that doesn't truly exist ), but, in statistics, one would just express the condition as "as long the noise is iid". I don't know the mathematical implications regarding the "optimal" result but I have a feeling that i.i.d is quite close to "white noise" if not identical. Thanks. $\endgroup$ – mark leeds Aug 21 '18 at 21:47
  • $\begingroup$ Well, by definition any IID Process is White Noise. Actually what we call White Noise in the wide sense only requires being uncorrelated (1st and 2nd moments information) while IID means even higher moments are orthogonal with respect to the expectation operator. So you're not missing anything compared to the DSP guys :-). $\endgroup$ – Royi Aug 21 '18 at 21:51
  • $\begingroup$ gotcha royi. thanks. I actually never knew that iid was a slightly stronger statement than white noise in the wide sense so the wisdom is appreciated. $\endgroup$ – mark leeds Aug 23 '18 at 1:11

If process and measurement noises are zero-mean, uncorrelated, and white, then the Kalman filter is the best linear solution to the above problem. That is, the Kalman filter is the best filter that is a linear combination of the measurements. There may be a nonlinear filter that gives a better solution, but the Kalman filter is the best linear filter. It is often asserted in books and papers that the Kalman filter is not optimal unless the noise is Gaussian. However, as our derivation in this chapter has shown, that is simply untrue. Such statements arise from erroneous interpretations of Kalman filter derivations. Even if the noise is not Gaussian, the Kalman filter is still the optimal linear filter. [Optimal State Estimation by Dan Simon, JW & Sons, NJ, 2006 ]

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.