I am coming from a background in deep learning techniques for computer vision like image classification, action recognition from skeletal data, and semantic segmentation. I have read several papers on Optical Flow, but I'm wondering what the nature of the annotation is for optical flow.

Specifically, supposing we have two images I_1, and I_2, both of the same resolution. For a dense optical flow prediction task, there must be some ground truth image G of the same resolution with annotations at each pixel.

My question is, what are the values of G?

For instance, suppose a pixel in image I_1 at (x_1, y_1) "moves" to pixel (x_2, y_2) in I_2, I can imagine the ground truth being a tuple of values (g_x, g_y). These could be the normalized l_1 distances computed coordinate-wise but I'm not sure if this is what's done in practice, or is it a single regression value per pixel?

I'm assuming pixels that have no movement are given a value of 0. There may even be some "don't train/don't train" pixels, perhaps given a sentinel value of -1.

Any insights on how the labeling of optical flow works much appreciated.


1 Answer 1


See for examples the Middlebury datasets: https://vision.middlebury.edu/flow/data/


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.