Context: for a research project I have at regular times a batch of wav files with recordings of voice. The hypothesis is that all speakers in a batch are the same, but there might be 1 or a few “others” in it. There are 5 to 10 files in a batch.

My goal is to obtain a similarity matrix for all wav files (speakers) in a batch.

What I got so far: I calculate MFCC (16 values) with an existing tool, I have between 2000 and 4000 frames per wav file. Simply taking the average of MFCCx is not meaningful. Next I perform a GMM clustering on each wav and obtain something like this: (MFCC0 discarded)

Class   Mean (MFCC1)    Mean (MFCC2) …     Proportion
        1       -44             6               0,09
        2        46           -23               0,37
        3        20           -26               0,10
        4        43           -43               0,06
        5        57           -52               0,06
        6        66           -55               0,18
        7       108           -46               0,15

And here I am stuck, what do I do next to get a similarity matrix, can I calculate some “distance” between the different wavs and how?

Voice is natural talk. Recordings are under various channel conditions (sometimes background noise) and can be with varying emotional status (eg laughter, some excitement,..). Speakers are of random nature (man, woman, old, young, …).

I would like to avoid more difficult algorithms for speaker recognition because that is beyond my current knowledge. Given that speakers are random, it will be rare that the other(s) is very similar to the normal speaker, and some error rate is acceptable, so a basic method should be ok.

  • $\begingroup$ Do you have any ground truth information? $\endgroup$
    – user28715
    Commented Aug 23, 2018 at 15:03
  • $\begingroup$ If needed it would be possible to obtain batches that only contain wav's from the same speaker. Would building a baseline model be a necessity? $\endgroup$
    – EddieM
    Commented Aug 23, 2018 at 15:45
  • $\begingroup$ How would you know your performance otherwise? $\endgroup$
    – user28715
    Commented Aug 23, 2018 at 16:02
  • $\begingroup$ I am hoping that a similarity matrix would show clear outliers in case there is another speaker in a batch. If speaker is somewhat similar then I would not notice but some error rate is allowed. $\endgroup$
    – EddieM
    Commented Aug 23, 2018 at 16:32
  • $\begingroup$ The human emotion known as hope is commendable and is a large contributor to our species success but in practice, knowing is usually better than hoping. It’s your project. I’ve given my opinion. I’m not interested in debating this particular opinion. $\endgroup$
    – user28715
    Commented Aug 23, 2018 at 16:47

1 Answer 1


Given MFCC values for some wav’s, how can I easily tell if all speakers are the same?

In general, you would deal with this kind of problem using some form of Clustering.

The fundamental premise behind this idea is that the features of similar data items would have similar values.

How you assess similarity, how you assess "a cluster" and what this means for the particular application you are dealing with are very much context specific issues.

In this particular example, the features are the MFCC coefficients. If these are similar between two frames we would tend to think that the frames contain more or less signals produced from the same source.

The simplest clustering to apply would be k-means clustering but this is not going to help too much here, especially if you don't know how many speakers you might get in a recording. So, long story short, you will most likely gravitate to something like hierarchical clustering.

You don't have to extract the means of each feature to cluster them. Doing so would be equivalent to training and using a minimum distance classifier. The only thing that is needed here to complete that is the "ground truth" knowledge that will tell you which centroids belong to which speakers.

Note here that if you have two speakers talking in a mostly silent room, the "feature space" of the clustering is not necessarily expected to be divided into two distinct "point clouds". It could, for example, contain separate regions for background silence, one speaker talking, the other speaker talking, both speakers talking at the same time. This last class might contain further grading depending on how loudly was each speaker talking while interrupting the other.

Hierarchical clustering allows you to focus on details with varying degrees of "attention", provided that the MFCC clusters the feature space with sufficient accuracy.

The other thing you can do, is a Principal Component Analysis on the MFCC matrix which (hopefully) will return one component for each one of these "major events" happening in the file.

Hope this helps.

EDIT: Forgot to add, if you are looking for the terminology that could return to you some useful results in varying levels of complexity, this is "Speaker Diarisation". See for example here.

  • $\begingroup$ this answer is very on point.. much better than stack overflow $\endgroup$
    – ERJAN
    Commented Jan 24, 2020 at 10:30
  • $\begingroup$ @ERJAN Glad you found this helpful. $\endgroup$
    – A_A
    Commented Jan 24, 2020 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.