2
$\begingroup$

The timing involved in this problem is a little hard to explain, so I'll start with my specifics:

I'm developing a program that visualizes output (a bounding box) from a computer vision model on top of a live camera feed. I want to animate the display at 60 FPS, but the model only provides output at 5-6 FPS. In other words, there are 10-12 frames of animation between every model output.

I'd like to use a Kalman Filter, both to smooth the model output, and to predict the future position of the bounding box. By predicting the position, I can keep the display current and animate on every frame without waiting for the computer vision model to "catch up."

What I'm struggling with is the latency of the computer vision model, and how to update a Kalman filter when there's latency in the measurement.

Here's a timeline, where each Tn is animation frame n at 60FPS:

T0: capture camera frame, submit to model for processing

T1: use KF to predict position at T1, display

T2: use KF to predict position at T2, display

...

T10: model measurement finally comes back. Update KF.

So, the measurement arrives 10 frames after the corresponding prediction. All the KF examples I've seen assume that the prediction and the update are at the same time, rather than offset by latency. To reiterate, I'd be doing the update at T10, but the measurement in question is from the past! At T0.

I'm planning to keep a buffer of past KF states, and do the update at T10 using the past KF state from T0. But is this a bad idea, will I run into weird inconsistencies by going back in time? Is there a better, established approach for this?

My apologies if this has been unclear. The concise version is: how do I update a Kalman Filter, with measurements taken in the past? Thanks so much!

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ One thing to consider is the use of a discrete time vs continuous time noise model. The continuous time models assume the noise is applied throughout the sampling interval. Thus predicting from $t_0$, to $t_1$ to $t_2$ is the same directly stepping from $t_0$ to $t_2$. The discrete noise models don't have this consistency. If the time steps - see Bar-Shalom & Li "Estimation and Tracking" - Ch on Kinematic models. $\endgroup$
    – David
    Commented Oct 26, 2020 at 14:11

1 Answer 1

Reset to default
1
$\begingroup$

No, this sounds correct. In a sense, you're using a model $$\begin{align}x_k = F_{k-1} x_{k-1} + G_{k-1} u_{k-1}\\ y_k = H_k x_k\end{align}$$

But you're using $$H_k = \begin{cases}H_0 & k \mod 10 = 0 \\ 0 & \mathrm{otherwise}\end{cases}$$

and you're being mindful of the fact that your "$x_k$" is up to ten samples in the past -- so you need to predict $x_{k+1}$, $x_{k+2}$, etc.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.