Identify and remove repeated audio chunks

Question

I have some audio produced by a buggy recorder, at some point it may start two recording streams, each stream captures chunks of audio and write to a file. The buffer queue is processed asynchronously, you can assume that the chunks will be written randomly.

Since the audio was resampled and encoded (in mp3) and possibly recorded by a different microphone, the samples are not necessarily the same.

A waveform where the repetitions can be visually spotted.

I tried to compute the correlation between chunks of say 1k samples, in longer windows say 20k samples. This was not came up with any obvious matches. Any suggestions on how to refine this approach to identify repeated chunks.

Further visualization

I took one piece of the audio and I computed normalized correlation in blocks of 1000 samples, in a slice of about 70k samples. That gave me this rectangular kaleidoscope image, I suspect that the border of the tiles are positions where the audio switches sources. The high correlation values will give an indication of where pairs of samples that are repeated.

I tried to identify the boundaries by accumulating the differences in the diagonals.

def inspect(x):
    x_gpu = torch.tensor(x, dtype=torch.float16, device='cuda')
    L = 1000; N = len(x)
    x_toeplitz = x_gpu.as_strided((N-L, L), (1,1))
    C = x_toeplitz @ x_toeplitz.T
    d = torch.sqrt(torch.diag(C))
    C /= d[:,None]
    C /= d[None,:]
    return C
def imss(C):
    h,w = C.shape
    while C.shape[0] > 1000:
        C = torch.maximum(C[0:-1:2,:], C[1::2,:])
        C = torch.maximum(C[:,0:-1:2], C[:,1::2])
    fig,ax=plt.subplots(figsize=(12,12))
    ax.imshow(C.cpu().type(torch.float32), extent=(0,w,0,h));
    return ax;

C = inspect(x)

ax = imss(C)
#ax = ax.twinx();
D = torch.zeros_like(C[0,1:])
for i in range(len(C)-1):
    D[:] = torch.add(D, torch.sqrt((C[i+1,1:] - C[i,:-1])**4)/
                     (1e-5 + C[i+1,1:]**2 + C[i,:-1]**2))
plt.plot(D.cpu(), 'r');

Answer 1

What I'd do:

1. Transform data into a representation that maximizes "similarity" of chunks that are otherwise "alike" but have large Euclidean distance in terms of raw waveforms
2. Apply similarity measure at each point in time
3. Extract indices of peak similarities

My transform of choice would be wavelet scattering, which is sparse, robust to noise, time-shift invariant, and stable against time warping deformations, hence qualifying for 1. I'd then apply 2D autocorrelation on first order and maybe second-order coefficients, and experiment with different hyperparameters, especially T.

I would begin on a small sample (like you've shown) where matches are obvious, and optimize above steps to it, then try on an unseen example, and calibrate accordingly. With enough heuristics (thresholding, etc) it should work.