# How to synchronize audio and video streaming for not fixed FPS streaming?

It's a 3-stage process.

In the first stage, a video (with audio) is recorded at a fixed 25 FPS rate. In the second stage, it's demuxed in two different channels and the video is processed. In the third stage, the audio and video are again synchronized and muxed together.

The issue is for the third stage, the video after the processing is having varying FPS rate (like 16, 18, 30, 35 FPS). Is there any way to synchronize the audio with the video at original FPS rate. I'm trying to use FFMPEG for this but no success.

• Where is this variation coming from? – A_A Oct 15 '19 at 8:26
• @A_A I am processing the images (changing the color and drawing shapes on the frames) in the second stage, so its taking some time to process – Padfoot Oct 15 '19 at 8:54

Container formats are usually storing video and sound information in chunks that are supposed to be triggered at the same time. In turn, each chunk might further be compressed according to the specifications of a given CODEC.

For example, consider a video at $$FPS=30$$ with its sound sampled at $$Fs = 48kHz$$.

In this example, one frame of the video sequence (one video "sample") should remain on the screen for $$\frac{1}{30}$$ seconds. But, $$\frac{1}{30}$$ seconds is equivalent to $$\frac{48000}{30}=1600$$ audio samples. And the idea here is that you put the frame on the screen and then trigger the sound and by the end of that sound clip you put the next frame up and trigger the next clip.

Or, to be more realistic, you can take the two streams and establish two functions that map audio samples to video samples and the reverse.

For instance, which video frame would be shown to the viewer at sound sample $$n_{audio}=987600$$?

That would be:

$$n_{frame} = \left \lfloor \frac{n_{audio}}{Fs} \cdot FPS \right \rfloor$$

Where $$\left \lfloor \cdot \right \rfloor$$ denotes the floor function.

And in this case:

$$n_{frame} = \left \lfloor \frac{987600}{48000} \cdot 30 \right \rfloor = 617$$

If either of your video frame rate or audio sampling frequency are variable, then the respective parts of the above equations become cumulative sums. So, to find where in time are you within either sequence, you have to accumulate the rates that brought you to that time (first 20 frames at 12FPS, next 40 frames at 20 FPS, ... and so on and similarly for the audio). CODECs will have that information in timestamps or even better, the decoder would be able to return "raw" streams and then you are back to the simple equations from above.

But in this case, unless you operate the CODEC yourself (even an uncompressed CODEC, just to specify which sound chunks match which video frames), it will be very difficult to "pipe" two variable streams to a third party software and let it figure out how to put things together.

The simplistic solution here would be to assume that your video runs at the highest FPS and handle other rates by repeating (or dropping if required) frames. The codec at the end of this will assume that the frames and sound it is passed are at a constant rate and "chop them up" accordingly. But I don't know how suitable this might be for your application (?)

Hope this helps.

• Thanks for the detailed explanation. It really helped me in understanding. I'll try to implement this, but not sure if this will work out for a streaming application. – Padfoot Oct 18 '19 at 3:25
• @padfoot Glad to hear it was useful. The streaming can be handled by the choice of CODEC at the recombination point. All that you would need there is the bit of logic that decides which frames are supposed to be matched with which sound clips and then to emmit the right "record". You don't have to handle the streaming bit per se. All the best. – A_A Oct 18 '19 at 11:08