2
$\begingroup$

I have big database of audio files, about 1-2 s long. Different people say different words, but during process I done some duplicates which I have to find and remove (database is about 100000 sounds and it is difficult to listen, have some noise, or are similar to human, but are different). Some duplicate audio files cant be shifted by half a second or less during cutting process.

Please help. How can I find real duplicates in my audio database?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Are you asking about how to compare if two "similar" audio files are actually the same word being pronounced? Or are you saying you actually have "bit by bit" duplicates within your database? If you have the latter problem then dsp.stackexchange is not the place to ask, since you question is more about algorithms and fast and practical database processing. But if it is the former, could you clarify your question a bit. "Some duplicate audio files cant be shifted by half a second or less during cutting process", did you mean "... audio files can be..."? $\endgroup$
    – bone
    Commented Feb 13, 2017 at 11:38
  • $\begingroup$ All recording were cut from different long audio files by removing silence between words. Sometimes software cuts an different oisitions, so sound can be shifted some ms. so files are not similar, but recorded sound is similar... $\endgroup$
    – Ernest D
    Commented Feb 14, 2017 at 13:51

2 Answers 2

Reset to default
1
$\begingroup$

Here are three methods of finding duplicates in a database of audio files. If I've understood your question correctly, I think you want the cross-correlation approach.

Looking for identical files - md5 checksum (not a DSP solution I know)

Looking for identical audio samples but time-shifted - cross correlation

Looking for similar sounding audio but different samples - audio fingerprinting.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Happy to improve this answer if the downvoter is willing to explain their reasons. $\endgroup$
    – tobassist
    Commented Feb 20, 2017 at 19:02
1
$\begingroup$

I would suggest to use an alignment algorithm such as DTW (Dynamic Time Warping)

You can check a good Matlab implementation here: http://www.ee.columbia.edu/ln/rosa/matlab/dtw/

You can then sum across the alignment path to get an overall distance value between two files.

You can change the implementation in order to use different features (but I believe that the cosine distance used by Dan Ellis is a good general purpose option.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.