# Measure the amount of drift in video and audio frames

I setup a stereo system using 2 GoPro Hero Session Cameras, since I need a larger baseline than the one of the Dual Hero System.

This unfortunately means the two cameras are not genlocked and as a result I expect there to be some drifting between the frames of the two video/audio signals. I am using the GoPro remote to start and stop the videos at the same time, but this is obviously not going to solve the problem.

The nominal camera frame rate and audio sampling rate are 60Hz and 32KHz respectively.

I extracted the two .mp4 and .wav files using the python libraries imageio and scipy, and as expected the number of frames don't match.

import imageio
import scipy.io.wavfile as wavereader
video_frames_l = imageio.get_reader(video_l_path,  'ffmpeg')
video_frames_r = imageio.get_reader(video_r_path,  'ffmpeg')

print "Left num. video frames : %d"%len(video_frames_l) #prints 20922
print "Right num. video frames: %d"%len(video_frames_r) #prints 20914
print "Left num. audio frames : %d"%len(audio_frames_l) #prints 11168768
print "Right num. audio frames: %d"%len(audio_frames_r) #prints 11164672


I'd like to know how can I quantify:

• The synchronization between the two video/audio signals without any post-processing?
• How much drift was there over the whole recording?

And more importantly:

• How can I improve synchronization in post-processing?

During recording I went into a dark silent room and turned lights on and off and clapped my hands a few times to create some signal.

Thanks for any pointers!

the two cameras are not genlocked and as a result I expect there to be some drifting between the frames of the two video/audio signals.

Drifting would imply that there is frame rate jitter. While jitter is unavoidable, I would not expect it to be that large so as to cause sync problems. This jitter would pose a limit to the longest scene that can be shot before starting to notice the de-syncing of the two streams. Without knowing a little bit more about the characteristics of the jitter, it is difficult to get an estimate of that maximum length.

I am not sure if the "sync cable" of the dual Hero produces a common time reference for them or if it is simply a more accurate way of synchronising video start by triggering both cameras at the same time through the port connector. I suspect that it simply triggers them more accurately at the same time.

Therefore, their frame rate must be fairly accurate.

Accurate frame rate would imply a constant "time difference" between the two streams once video capture has started. Therefore, synchronising the two videos by aligning the "sound peaks" from a clicker or a clap should work adequately.

I extracted the two .mp4 and .wav files using the python libraries imageio and scipy, and as expected the number of frames don't match.

They wouldn't because the two cameras still operate as two independent ones. Once you have aligned the two streams, you can then extract videos of some length $l$ from the common starting point and of course in that case the two streams would match in length.

I'd like to know how can I quantify:

• The synchronization between the two video/audio signals without any post processing?
• ...

Synchronisation only makes sense when measuring from a common event.

Bring the two streams into your video editor, measure the time between the two peaks in the sound stream that are produced by the two cameras picking up the sound of the clap. That's your audio delay but only with respect to the start of the recording. Not sure how useful this might be.

If you align the streams, so that they coincide on a frame whose sound clip contains the peak of the clap and then measure their time difference, then that would be your "video" delay. It will be within $\frac{1}{FR}$ of a second, where $FR$ is your frame rate. Again, I am not sure how useful this might be if we assume that the frame rate is fairly accurate.

• ...
• How much drift was there over the whole recording?

Drift is more difficult because you need to capture enough data to reveal its structure. Drift would typically occur at times $\ll T_s$ where $T_s$ is the sampling period. The "trick" is to wait long enough until the jitter errors accumulate so much that they reveal the distortion. For example, in audio, jitter would appear to modulate the captured waveform. To observe this, you need a reference signal (i.e. a sinusoid) and some method of quantifying phase difference (i.e. a mixer or autocorrelation, or other). But, with $T_s$ in the order of tenths of kHz and with jitter expected to occur $\ll T_s$, you would have to accumulate enough errors to make up a "shift" of a few $T_s$ BEFORE you can observe a strong error signal between reference and captured. Without further assumption about jitter, this might well extend into hours of capturing.

Similar considerations apply to video. You can set up the cameras and shoot a very good time reference signal. For example, a rotating radial pattern disc on a brushless hobby RC motor with a common speed controller. In post-processing, align the videos based on their sound streams and then examine the video stream further down the line for any drifts. To do this automatically, you might want to derive a third video stream where each frame is calculated as $I[n+1]-I[n]$ where $I$ is the $n^{th}$ frame from a video stream. This "error" signal should remain constant (and ideally zero) throughout the video.

And more importantly:

• How can I improve synchronization in post-processing?

You cannot.

However, this company seems to be doing a genlock for GoPros

Hope this helps.