Choosing the best measure for machine learning on a Spectrogram

Question

I am working on an application to classify Talas(a piece of music with onsets). For that, I have the spectrogram of a Tala, which I want to compare with the test Tala, or compare 2 Talas.

For this I intend to find the onsets in the spectrogram and then form a classifier based on the frequencies that I get at the onset points in the Tala.

Now as we already know that a spectrogram contains multiple frequencies for a time t , I wish to quantify this as a single value to be used in machine learning, so that I have n values for the n onsets(say), and not multiple frequency values for the same time t.

So here is my question - Given all the frequencies at time t, how can I get a value that represents all these frequencies(like mean, median, etc). What should be the measure I should go ahead with?

Answer 1

I'm only so familiar with your audio terminology but in your spectrogram matrix, you could seek to maximize between-class variance over in-class variance $$r_{ij} = \frac{|\sigma_{ij}^{Talas_1,Talas_2}|^2}{\sum_{c=1}^{2}|\sigma_{ij}^{Talas_c}|^2}$$ , and pick $N$ points in your ratio matrix $r_{ij}$ of that have this maximum value. Unfortunately, this yields a decision surface that may be nonlinear so a neural network is used in the paper below, but you might try to fit a Guassian mixed model first to the $N$ points you pick to see how that works for your application. Here i've just used 2 classes but the ratio can be generalized to multiple classes.

You can also look around in any papers where the goal is to classify non-stationary signals, as they often use time frequency representations for classification.

Reference: Classification of power quality disturbances using time-frequency ambiguity plane and neural networks.

Answer 2

First thing: a spectrogram does not give the "frequencies" at time $t$. It gives amplitudes, measured for a set of linearly spaced frequency bands.

Anyway, there is no "right" answer to your question, because it depends on which musicological aspect you want to focus on, and you need to enlighten us on that. If you want to focus on melody, extracting the fundamental frequency is the most relevant feature, but I suspect this is not the case here. If you want to discriminate different timbres, spectral centroid might be a better way of "summarizing" a slice of spectrogram into a single value - better, but still quite bad.

Indeed I can see several flaws in your methodology:

• Why extract information at time t only (assumedly at a small window centered at t) rather than on the entire segment of sound between onsets?
• Why do you have this constraint that for any onset you have to extract a single real value? Some properties of sound, like timbre, are multidimensional, so you won't be able to entirely capture them in one single value. If we take the example of discriminating tabla sounds (which might or might not relevant to your problem, since you mentioned tala...), we need several features to discriminate them (bayan/dayan discrimination relies on pitch information, while resonant vs non-resonant stroke discrimination relies on timbral features).
• You are completely discarding the rhythmic (time) information. Is that really wanted for your problem?

Tell us more about your problem, there's a good chance it has already been solved, and that you are asking the wrong questions to solve it...

Answer 3

When working with audio signals there are two tricks, first remove the DC from the spectrum and band-limit it to a reasonable size. If you are looking at 1000 point intervals only use a 30 point FFT.

You can try to use ML to get the mean and variance and then use these vectors to evaluate test data. More advanced students might consider the MMI criteria.