How to compare 2 matrices of MFCCs, keeping the time information

Question

I have a corpus of several thousand sounds. Now I pick a sound sample S, and I want to find in my corpus the sound S' that is the most similar to S. The application is for music, for a composition that I am working on :) I found a lot of questions which are all related to speech recognition. The problem here is different as rhythm is the most important info for me, even before timbre. If S' has a different length than S, it must be somewhat of stretched version of it.

The samples are between 0.5 seconds to 2 seconds (approximately). I have tried to compute MFCCs, apply DTW, but I don't get convincing results. I still believe MFCCs are the way to go, however I started to think DTW is not suited for my purpose, because it removes the rhythm information. For example (and if I understood well), DTW would recognize that the sounds "Puuuum pum tchak" and "Pum pum tchaaaak" are the same. But obviously, when it comes to music, they are not, as the timing is crucial.

So, how to compare 2 matrices of MFCCs, keeping the time information? Or is my approach completely wrong?

I was currently looking into procrustes analysis, as I have read that it compares 2 shapes, removing scaling, ... and this sounded good to me.

EDIT

With DTW I might have :

distance("Pum pum tchak", "Pum pum tchaaaak") < distance("Pum pum tchak", "Puuum puuum tchaaak")

in other words, the time wraping doesn't care about the fact that "Puuum puuum tchaaak" is just exactly the same as "Pum pum tchak" but with slower tempo.

Answer 1

You are misunderstanding how DTW works. You can actually adjust the frame deletion/insertion costs to adjust how much temporal mismatch you can tolerate.

We would have: DTW("Puuuum pum tchak", "Puuum pum tchak") < DTW("Puuuum pum tchak", "Pum pum tchk") < DTW("Puuuum pum tchak", "Purr purr purr").

That is to say, if there is a sample with a perfectly good timing, it'll always yield a higher DTW score than a sample with the wrong timing. The trade-off between similar timbre and similar timing can be adjusted by tweaking the DTW alignment costs. One useful thing with DTW is that it finds the optimal alignment between the "query" and the found sample - so you can use the path it founds to timestretch/phase-vocode the result and align it perfectly with the query.

It is not clear what kind of sensitivity on timing your application has; which features matter the most (overall timbre? onomatopoeia which is pronounced?); and what kind of processing you could do to the found clips to align them even better with the query. If rhythm/timing is absolutely essential, you might want to process each sample by an onset detector ; and restrict your search to the set of clips which have the right number of onsets, and the correct inter-onset times.

If you have absolutely no tolerance on differences in rhythm, then you could very well compute the average of pairwise distances between the two clips (padded with zeros if their length does not match). That would be a bit silly, but your application might want that.

Finally, if both your query and your samples can accept the same domain-specific representation (say a beatboxed rhythm and a drum loop can both be transcribed into "bass drum" and "snare drum" events) - it might help to transcribe all clips and to match the transcriptions instead of the audio signals.