0
$\begingroup$

I am trying to upsample a video by existing frame interpolation techniques. In the process, I realize that I also need to interpolate the audio signal for the new frames so that the audio signal is in sync with the frames of the new video.

The new upsampled video should have the same frame rate as the original input video, and will be a slowed down version of the input video.

Is there any way I can proceed to do the part of the audio interpolation? If the frame rate is F and the audio sampling rate is Fs, then Fs/F audio samples are associated with each frame of the original video.

Any insight on how to proceed regarding this idea would be helpful.

$\endgroup$
  • 1
    $\begingroup$ Hi just for curiosity, after you interpolate (add new frames) your video source, and change your FPS accordingly, then your video playback time will remain the same ? $\endgroup$ – Fat32 Dec 15 '18 at 14:23
  • 1
    $\begingroup$ No, it increases, as there are more number of frames. $\endgroup$ – Curiosity Dec 15 '18 at 17:09
  • 1
    $\begingroup$ Actually, the frame rate remains the same in the method I've used (as in arxiv.org/abs/1703.07514). But, the number of frames increases. And yes, slow motion and a more upsampled video version is the purpose. Its something I'm working on (rather trying to figure out) and am implementing existing methods just for the interpolation part, but am yet to do anything regarding the audio, and can't seem to get any online resource as to how it can be done. $\endgroup$ – Curiosity Dec 15 '18 at 17:42
  • 1
    $\begingroup$ Do go through your last comment again. If it was an 'A' or 'B' question, I'm sure there should have been an 'or'. I replied in positive, affirming that I indeed want to keep the pitch same, and lengthen its duration, which is basically what you had asked. $\endgroup$ – Curiosity Dec 15 '18 at 21:26
  • 1
    $\begingroup$ I've done the same now. $\endgroup$ – Curiosity Dec 28 '18 at 5:00
2
$\begingroup$

When upsampling the number video frames to allow playing a video in slow motion, rather than for a frame rate increase for smoothness, you will likely need to modify the audio using a time-pitch stretching/shifting algorithm to stretch the audio out (increase the number of samples) for a longer play duration (to match the increase in slow motion video playback time), and to do so without distorting vocal or musical pitches.

If you just interpolate or resample audio without time-pitch correction, you will mess up all the pitch frequencies in the audio.

See this Wikipedia article for a description of several possible time-pitch stretching/scaling algorithms. There are multiple commercial and open source applications and libraries that can accomplish time-pitch modification/correction with varying degrees of quality and artifacts.

$\endgroup$
  • $\begingroup$ sometimes, but not always, the pitch is allowed to darken, as an indicator of the slowing effect... $\endgroup$ – Fat32 Dec 15 '18 at 18:11
  • $\begingroup$ Yes, but for artistic effect, this audio "darkening" likely uses completely different parameters than the video slo-mo rate change. Thus, still requiring time-pitch modification of the audio. $\endgroup$ – hotpaw2 Dec 15 '18 at 18:17
0
$\begingroup$

You just need to resample the audio by the same resampling factor you've applied to the video.

"In sync" is achieved simply by matching rates. If it was in sync before, it will still be after; if you implement that audio resampler yourself, you might have compensate group delay of the resampling filter. That is all.

$\endgroup$
  • $\begingroup$ Suppose, I add 1 frame between every 2 frames of the original video to constitute my new video. What should be the audio resampling factor in that case? $\endgroup$ – Curiosity Dec 15 '18 at 20:22
  • $\begingroup$ The scenario is like this: for every 2 previous frames, I now have 3 frames. So, how to resample the audio signal in this situation? $\endgroup$ – Curiosity Dec 15 '18 at 20:23
  • $\begingroup$ That's not upsampling; why would you do that? If you want to increase your frame rate by 1.5, then only one in three frames would be an "original" frame from the source video, and the other two in three frames would be interpolations. $\endgroup$ – Marcus Müller Dec 15 '18 at 20:39
  • $\begingroup$ Yes, so I add 1 frame between every 2 frames. The 2 frames were already existing, and 1 new frame is constructed from the 2 by using the optical motion. $\endgroup$ – Curiosity Dec 15 '18 at 20:43
  • $\begingroup$ then you don't get "for every 2 previous frames, I now have 3 frames", but "for every 2 previous frames, I now have 4 frames". All that I wrote in my answer still applies. $\endgroup$ – Marcus Müller Dec 15 '18 at 20:54

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.