What happens when we Fast Forward the Video

Question

I've been involved in Image Processing kind of project for the last couple of month, before which i had no experience in Image processing. While filtering the image today i had a thought what exactly happens when we fast forward any movie or a video clip ?

For example, If I have a video whose Frame rate is 30 , this means that 30 frames will be process in a second. and each frame will take 1/30 = 0.03sec to process , If we increase the frame rate or FPS of the video we will get more smooth view of it.

So what exactly happens when we fast forward the video, what i think is (not sure) the audio and the video of any clip must be synchronized so that we can control both accordingly.

So when we forward the video does it skip certain Frames ? if yes, then what about the Audio ? What exactly happens ?

Answer 1

What happens to the video:

• In the most simple implementation, which is suitable for large integer fast forward rates, only 1 frame out of N, where N is the fast forward rate is played. For example, with a normal play rate at 30 FPS, frames 0 to 29 are played in sequence over a duration of one second. In fast forward with a 3x rate, frames 0, 3, 6 ... 87 are played in one second; and frames 1, 2, 4, 5... 88, 89 are discarded. This can be quite computationally expensive as the video decoder has to provide 3 times as many frames per second as in normal play (though the display continues receiving 30 frames per second).
• When the hardware permits it and/or when the fast forward rate is small, the frame rate is simply increased. For example, in fast forward with a 1.1x rate, frames 0 .. 32 are played in one second.
• There are applications when a video stream has to be sped up/down by a small factor (say 5%) while maintaining a target frame rate (for example when a broadcaster wants to speed up a show to make room for twenty extra seconds of ads). In this case, interpolation/resampling methods have to be used - this is the same kind of techniques as those used for telecine conversion.

What happens to the audio:

• In some implementations, the audio is read at a higher sample rate, and decimated to the target sample rate. For example let us consider the example where the original audio is at 44.1kHz, and the fast forward rate is 3x. In 1s, you need to play 3s of the original audio. So you read 44100 x 3 samples from the original audio stream, low-pass filter it with a filter which has a 1/3 normalized cutoff frequency; and keep only every third sample. This gives you 44100 samples of sped-up audio, which are played in a time span of 1s at 44.1kHz. This has the unwanted effect of shifting up the pitch of all sounds, giving voices a "chipmunk effect". For small shift up/rates the effect is acceptable to the common public (though musically trained people might notice a well-known song has been shifted in pitch and is not in its original tonality).
• In some better implementations (eg Apple Quicktime player), a time-stretching algorithm is used to multiply the speed of the input audio without altering the pitch. For playback / scrubbing, the sound quality rarely matters and the time-stretching algorithms are rather crude (you can have a look at the source code of soundtouch).