What sensors can be fused using the Kalman Filter framework

Question

I was recently introduced to the concept of Kalman filtering in the context of projectile tracking. A classmate recommended this to me, and what intrigued me most was its ability to fuse different types of information from sensors.

Are there types of measurements that are not compatible for sensor fusion? Can any measurement be fused to better inform the underlying model?

• Example: We take pictures at different time points of a rocket flying through the sky.
• We use some computer vision tool to automate the detection of the rocket's position from the photograph (with some degree of measurement error).
• At a series of frames, the rocket slowly disappears behind a cloud (undetectable) before re-emerging.
  • We know the rocket exists and is positioned somewhere behind this cloud.
  • Is it the (A) "boundary"/border of the cloud that can inform our state space of the rocket,
  • ... or (B) the pixel intensity along the trajectory (suddenly shifting from blue to white) that can inform our state space of the rocket?

Intuitively, I feel both can better inform the trajectory, but they seem to me very different interpretations of the auxiliary information. I am not sure if there are limits of sensor fusion I cannot appreciate as an outsider to this field.

Answer 1

Remark: I will answer this using the Linear framework of the Kalman Filter but the idea is the same.

The Kalman Filter basically propagate and fuses Gaussian Distributions in order to calculate the mean of the state vector, $ \boldsymbol{x} \left[ k \right] $, distribution:

$$ \boldsymbol{x} \left[ k \right] = F \boldsymbol{x} \left[ k - 1 \right] + \boldsymbol{w} \left[ k \right] $$

The Kalman Filter can fuse and use any projection of the state vector which is defined by:

$$ \boldsymbol{z} \left[ k \right] = H \boldsymbol{x} \left[ k \right] + \boldsymbol{v} \left[ k \right] $$

As long as you can define this projection, you can utilize it with the Kalman Filter.
For instance, the location on an image, the acceleration by IMU system, a measurement by a RADAR / LIDAR, a measurement by a SONAR, etc...

Going behind the clouds without being able to see it is usually treated as no measurement. Then we apply only the prediction step of the Kalman Filter with no fusion.

Though, as you wrote, actually not seeing it giving us more data than no data, because we can limit its position to a certain place. You can create such function and use it, but it will be hard and not so informative hence its gain will be minimal.

But indeed the Kalman Filter framework is highly flexible and you can fuse in any sensor with known projection of the state vector.

Answer 2

Are there types of measurements that are not compatible for sensor fusion? Can any measurement be fused to better inform the underlying model?

Any sensor that gives you more information about the state of the system that you're interested in could be useful. The more that a sensor informs you of some tidy, unbiased reading of one or more system states the better. For a linear Kalman filter you'd like the sensor to give you at least some unbiased linear combination of system states. Sensors that give wildly nonlinear responses to the things that they're measuring will lead to difficult filtering problems, but may still be useful if there's no alternative.

• Is it the (A) "boundary"/border of the cloud that can inform our state space of the rocket, ...
• or (B) the pixel intensity along the trajectory (suddenly shifting from blue to white) that can inform our state space of the rocket?

Unless you contort the idea of a Kalman filter beyond all recognition, neither.

I would put it that the vision system decides that the rocket can't be seen (and, thus, no measurement). At this point you'd iterate the prediction step, but you wouldn't feed camera data into the correction step. If you still had other sensor data, you'd use that in the correction step by setting the relevant entries in the measurement matrix (usually $\mathbf H$ in the engineering literature) to zero.

You could have some logic that feeds the boundaries of the cloud to the filter, and constrains the filter to -- in an English translation -- "it's somewhere in here, but we don't know exactly where". Doing this effectively takes you well away from a linear Kalman filter and into a Bayesian or Particle filter. That's going to be fraught with opportunities for error, so it's not something I'd recommend unless you're a new graduate student looking for a PhD topic.