# Kalman filter with multiple sensors

I have been reading more about EKF and I am a bit confused on how you handle predictions with multiple sensors.

Ex, I have IMU, GPS, Odom and Stereo Camera.

Each can be used to predict location, how do you combine them?

My understanding of EKF fits in the basic framework of predict, act, measure, fuse prediction with sensor.

But it's not clear how multiple sensors that measure the same thing is used.

• Do you have more details about your specific work with KFfor this system? Note that KF is matricial. Inputs are your known readings, outputs your predictions. How these combines will depend on.... how these variables works in a single model. You must have a model to proceed with any KF version. So,... can you share us your model? Feb 16, 2021 at 22:17
• I don't have an explicit model yet. The basic example I am familiar with uses a odom + lidar/landmark which makes sense. I just don't understand how multiple sensors get combined in this framework. Feb 16, 2021 at 23:25
• Every KF uses a State Space model. These are just a set of equations. It is inherently multivariable. Which version of KF have you used before? Feb 17, 2021 at 11:27
• en.wikipedia.org/wiki/Kalman_filter This explains my understanding. For example lets have GPS + IMU + Wheel Odom and forget camera. Both IMU and Odom have to be integrated in some way while the GPS gives a specific location - I understand how to evolve IMU and Odom on their own but not how to fuse the results. Feb 17, 2021 at 19:26

## Summary

You can have only one (or no) primary sensor which is used in the prediction process. All other sensors are secondary and only used for corrections. Each correction takes the current state and a measurement from one of the secondary sensors and outputs the corrected state. You can run corrections as often as you want, typically each time one of the secondary sensors produces a new measurement. There is no need to run corrections at a fixed interval.

The new state is predicted from the model matrices F and B and from the old state x and primary sensor data (or control input) u. $$x_{k+1} = Fx_k + Bu_k$$

The corrected state is calculated from the kalman gain matrix K, observation matrix H and secondary sensor data z. Each secondary sensor has its own observation matrix. $$x_{k,corrected} = x_k + K_k(z_k-H_k x_k)$$

## Detailed version

You have to designate one primary sensor each for orientation and position. Alternatively, you can go without primary sensors if system dynamics allow. The primary sensors should be extremely reliable, accurate in the short term, and support a high update rate. Long term drift is not a concern here.

In a car navigation system, the primary sensors are usually steering angle and odometer, the secondary sensor is GPS. In an attitude and heading indicator for aircraft, the primary sensor is a three-axis gyro and secondary sensors are accelerometer and magnetometer.

There can be an arbitrary number of secondary sensors. Their purpose is to provide long term accuracy, which the primary sensors lack. The secondary sensors have other shortcomings, like unreliability (no GPS in tunnels), poor short term accuracy (apparent gravity vector disturbed by g-forces from maneuvering), or incomplete information (gravity vector provides pitch and bank, but not heading). If you have a sensor that can take both roles - a sensor that is reliable, provides enough information to uniquely deduce orientation or position, and is accurate both in the short and long term, then you don't need a Kalman filter in the first place.

The prediction stage of the Kalman filter runs at a fixed and high rate and continuously updates the current estimates for position and orientation based on the old estimates and measurements from the primary sensors. If you operate the Kalman filter without primary sensors, it turns into a simple model-based extrapolator. Needless to say, extrapolating from the last known GPS position using steering and odometer data is much more accurate than a simple constant-speed extrapolation.

The correction stage of the Kalman filter becomes active whenever a measurement from a secondary sensor arrives. Each correction handles the measurement from only one sensor and updates the current estimates for position and orientation. There can be an arbitrary number of corrections with arbitrary or even irregular update rates (still no GPS in tunnels, therefore no corrections for several minutes).

• This is all well and good but doesn't explain how the update process works with multiple sensor inputs. Let's simplfy it and say you get 3 sensors at each time step each giving a different position, then what? The update process I know only works for one sensor. Feb 17, 2021 at 15:40
• I added a summary to emphasize that only one (or no) sensor is primary and you can have as many secondaries as you want. You can track a car using a constant-speed model (no primary sensor) and then use three different position sensors to correct the model at regular or irregular intervals. (@FourierFlux) Feb 17, 2021 at 18:21