# How to allign audio signals after DTW?

Hello I am not a domain expert in signal processing but I need to align two audio signals.

I have seen the following page https://librosa.github.io/librosa_gallery/auto_examples/plot_music_sync

I have implemented DTW and so many other pages. Most of these are using DTW but the problem is that none of them are showing WHAT TO DO AFTER I PERFORM DTW!. DTW gives you a cost matrix and wrapping path (D, wp) but nowhere it is shown how to use these two to align the actual signals.

I have come to a github repo https://github.com/chausies/voice_transmogrifier/blob/master/sync_audio.py

But most of the parts of the code seems difficult to understand. Like why he used variable_phase_vocoder or what close_points is doing. Why to use PchipInterpolator ? In fact why interpolation is done at all in audio?

Please help me in the project. I am new to audio processing.

Thanks

• Can I please ask if this was resolved? – A_A Feb 22 '20 at 9:41
• Not really an answer, but if you just need to align by cutting at the start, syncstart can do that. It uses an fft based correlation of the start to sync two recordings. So no DTW or use of librosa. – Roland Puntaier Feb 19 at 22:02

## 1 Answer

none of them are showing WHAT TO DO AFTER I PERFORM DTW!.

There are two quotes, at least in the page that you have linked that hint at what you do next, these are:

"We assume that you are familiar with the algorithm and focus on the application. Further information about the algorithm can be found in the literature..."

The linked reference is not a bad start for a book on audio signal processing. The second quote is towards the end:

"...you could apply time scale modification algorithms, e.g. speed up the slower signal to the tempo of the faster one."

And this provides a way towards the answer to your secondary question too.

The "cost matrix" that Dynamic Time Warping returns is discrete with units of samples (e.g. match sample-to-sample), or time.

With reference to this diagram from the Librosa documentation:

You can see that this "picture" has two axes: The $$x$$ and $$y$$ axis. On $$x$$ axis, are the time instances that samples occur on one waveform. On the $$y$$ axis, are the time instances that the same samples occur on the second waveform.

If the two waveforms were identical, their samples would be played at the same instances and the cost matrix would describe a straight line which means that sample 0 on $$x$$ is found on sample 0 on $$y$$, sample 1 on $$x$$ is found on 1 in $$y$$ and so on until sample $$N$$.

But, most of the times you apply DTW, you are interested in aligning the two waveforms.

In that case, the "cost matrix" can be interpreted as a speed-up or slow-down factor, so that the samples of waveform $$x$$ catch-up with those of waveform $$y$$.

...why interpolation is done at all in audio?

When you think of the cost matrix as modifying the speed by which samples are reproduced, you start thinking about how to speed up or slow down a waveform.

Especially in audio, there are many different methods by which you can achieve this. But the easiest way to think about it is as if changing the speed by which your "sample pointer" rolls through the waveform.

The most trivial example is this: At the waveform's $$Fs$$ (Sampling frequency), the sample pointer rolls through the waveform at a rate of $$Fs$$ samples per second. To make the recording sound twice as fast, you can replay it at $$2 \cdot Fs$$. But, what happens when, to make the two waveforms catch up with each other, you have to run the "record" at $$1.056 \cdot Fs$$?

When the waveform is played back at normal speed, the sample sequence is $$0, 1, 2, 3, 4, 5, 6, \ldots$$. When the waveform is played back at twice its normal speed, the sample sequence is $$0, 2, 4, 6, 8, 10, \ldots$$.

But for non-integer factors, you would have to "land" in between two samples. For example $$0, 1.056, 2.112, 3.168, \ldots$$.

This is the reason for using interpolation. For example, to work out the sample value at time $$1.056$$ when you know the sample values at $$1$$ and $$2$$, you could use plain simple linear interpolation. This assumes, that when the Analog-to-Digital Converted (ADC) is not sampling the signal, this signal evolves smoothly (as a line) between its values. For high enough $$Fs$$, this assumption might be valid.

So, suppose that your signal is in $$x[n]$$ and you want to work out the signal value somehwere (let's call this $$k$$) in between $$n .. n+1$$. So, $$n < k < n+1$$. Then, $$v = x[n] + (k-n) \cdot (x[n+1] - x[n])$$.

This method is implemented in numpy 's interp() function or even MATLAB's interp1().

Because of this need of having to read in between samples, interpolation is used in such operations as reverberation, chorus, pitch shifting and others. This linear interpolation method will work in general for high enough $$Fs$$ but as the speed-up/slow-down factor becomes bigger and potentially, applied over a longer block of audio, the artifacts become more audible. This is what more complex methods of applying pitch shifting (such as the phase vocoder), are trying to address.

Hope this helps.