The timing involved in this problem is a little hard to explain, so I'll start with my specifics:
I'm developing a program that visualizes output (a bounding box) from a computer vision model on top of a live camera feed. I want to animate the display at 60 FPS, but the model only provides output at 5-6 FPS. In other words, there are 10-12 frames of animation between every model output.
I'd like to use a Kalman Filter, both to smooth the model output, and to predict the future position of the bounding box. By predicting the position, I can keep the display current and animate on every frame without waiting for the computer vision model to "catch up."
What I'm struggling with is the latency of the computer vision model, and how to update a Kalman filter when there's latency in the measurement.
Here's a timeline, where each
Tn is animation frame
n at 60FPS:
T0: capture camera frame, submit to model for processing
T1: use KF to predict position at T1, display
T2: use KF to predict position at T2, display
T10: model measurement finally comes back. Update KF.
So, the measurement arrives 10 frames after the corresponding prediction. All the KF examples I've seen assume that the prediction and the update are at the same time, rather than offset by latency. To reiterate, I'd be doing the update at
T10, but the measurement in question is from the past! At
I'm planning to keep a buffer of past KF states, and do the update at
T10 using the past KF state from
T0. But is this a bad idea, will I run into weird inconsistencies by going back in time? Is there a better, established approach for this?
My apologies if this has been unclear. The concise version is: how do I update a Kalman Filter, with measurements taken in the past? Thanks so much!