What would be the best solution to synchronize data from two inputs. (Camera and Acceleration sensors)

Question

For a video stabilization project, a USB webcam (25 fps) and USB acceleration sensors are being used.

the sensors gives a speed of approximately 50 samples/second.

the derivative of the acceleration aka. 'Jerk' is done in order to localize the start and the end of the shake in the video.

the purpose is to compensate the camera movement in real-time, if it were you how would you :

1. Match the frame (from camera) with the data of the sensor.
2. Calculate the number of pixels to compensate from acceleration sensors data.

Thank you.

Answer 1

This is a common problem that most Multimedia systems faces and solves.

Basically, closest to the source, there should be one entity (call it originator) that must collect both the data simultaneously and then stamps the each observation i.e. frames of video and sample of sensor, a time stamp from the local (best suitable clock). After this, originator distributes data in whichever form required.

Essentially a common reference clock for both sources as seen by originator. Any encoding, processing, and consumption of this data must preserve and interpret these time stamps.

This synchronization is of course only as good the source clock itself; there can be jitter and skew in such a clock used by originator. Either the originator follow clock of any primary source (data or video whichever reliable) and stamp it on the other, or derive a local clock using either reliable crystal or GPS or NTP etc.