Map time/scene differences between two similar videos

Question

Goal:

Given two videos that are episodes from series but in different format (one Bluray/one TV for example), map the time differences between the two (which scenes are not in one of the two) and be able to compensate for some subtitles lags between version.

Current state:

I've began something thanks to this question (Map time difference between two similar videos) and started building a python script so now I'm able to transform each frame of each video to a one floating point value.

The problem is now, I dont know how to detect those lags as you can see in the picture. I'm trying some cross correlation maybe with some window but I'm not sure of how I should handle the added scenes in both signals (which means here signal that cannot be correlated).

Questions:

Do you know any algorithm/method that can help me with this process ?

Do you know what the user in this thread (Map time difference between two similar videos) means by "added a Gaussian of that length with amplitude as the phase difference to the mapping (at the appropriate interval)" ?

Thanks for your help !

Answer 1

This was an interesting read and it looks like you're not too far off! I think that what you have left to do is divide one of the signals into intervals (let's call it signal 1) and cross-correlate each segment of signal 1 with the entirety of signal 2. The peak in the cross-correlation will indicate where there is the strongest match for that segment in signal 2. You then calculate the difference in frames between where the segment was in signal 1 and where its strongest match is in signal 2.

The third step in the post you mentioned appears to be Kernel Density Estimation. The author placed a Gaussian at the center of each segment from signal 1. The standard deviation of the Gaussian, I assume is one-sixth the segment length, and the amplitude of the Gaussian is the frame difference. The Gaussian should extend from time 0 to the length of signal 2. He then summed all the Gaussians to obtained a continuous function.

I think you can achieve the same thing if you do the following: once you have the frame difference as a function of time, you convolve that function with a Gaussian whose standard deviation is one-sixth the segment length you used to divide signal 1.

I hope this helped!