I have some audio produced by a buggy recorder, at some point it may start two recording streams, each stream captures chunks of audio and write to a file. The buffer queue is processed asynchronously, you can assume that the chunks will be written randomly.

Since the audio was resampled and encoded (in mp3) and possibly recorded by a different microphone, the samples are not necessarily the same.

A waveform where the repetitions can be visually spotted.

enter image description here

I tried to compute the correlation between chunks of say 1k samples, in longer windows say 20k samples. This was not came up with any obvious matches. Any suggestions on how to refine this approach to identify repeated chunks.

Further visualization

I took one piece of the audio and I computed normalized correlation in blocks of 1000 samples, in a slice of about 70k samples. That gave me this rectangular kaleidoscope image, I suspect that the border of the tiles are positions where the audio switches sources. The high correlation values will give an indication of where pairs of samples that are repeated.

enter image description here

I tried to identify the boundaries by accumulating the differences in the diagonals.

def inspect(x):
    x_gpu = torch.tensor(x, dtype=torch.float16, device='cuda')
    L = 1000; N = len(x)
    x_toeplitz = x_gpu.as_strided((N-L, L), (1,1))
    C = x_toeplitz @ x_toeplitz.T
    d = torch.sqrt(torch.diag(C))
    C /= d[:,None]
    C /= d[None,:]
    return C
def imss(C):
    h,w = C.shape
    while C.shape[0] > 1000:
        C = torch.maximum(C[0:-1:2,:], C[1::2,:])
        C = torch.maximum(C[:,0:-1:2], C[:,1::2])
    ax.imshow(C.cpu().type(torch.float32), extent=(0,w,0,h));
    return ax;

C = inspect(x)

ax = imss(C)
#ax = ax.twinx();
D = torch.zeros_like(C[0,1:])
for i in range(len(C)-1):
    D[:] = torch.add(D, torch.sqrt((C[i+1,1:] - C[i,:-1])**4)/
                     (1e-5 + C[i+1,1:]**2 + C[i,:-1]**2))
plt.plot(D.cpu(), 'r');

What I'd do:

  1. Transform data into a representation that maximizes "similarity" of chunks that are otherwise "alike" but have large Euclidean distance in terms of raw waveforms
  2. Apply similarity measure at each point in time
  3. Extract indices of peak similarities

My transform of choice would be wavelet scattering, which is sparse, robust to noise, time-shift invariant, and stable against time warping deformations, hence qualifying for 1. I'd then apply 2D autocorrelation on first order and maybe second-order coefficients, and experiment with different hyperparameters, especially T.

I would begin on a small sample (like you've shown) where matches are obvious, and optimize above steps to it, then try on an unseen example, and calibrate accordingly. With enough heuristics (thresholding, etc) it should work.

  • $\begingroup$ That makes sense, I would could you give more details on how to apply that. $\endgroup$
    – Bob
    Oct 15 '21 at 12:10
  • $\begingroup$ @Bob 2D autocorrelation done like 1D except instead of points we have columns. Might want to do this in chunks (break up input) since one long sequence can have high autocorrelation by chance; this should be safe if you don't expect the "duplicates" spaced far in time. Might also apply more localized search methods, but that's its own topic - the key idea in my answer is to transform to a representation that eases accurate similarity measures. $\endgroup$ Oct 16 '21 at 0:31
  • 1
    $\begingroup$ Ok, your answer was useful, but it is not something that solves the problem. In part because my question was too broad, sorry. I think we can leave this discussion as is, if at some point in the future I have the time to narrow it down or to solve I update here. Thank you. $\endgroup$
    – Bob
    Oct 18 '21 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.