0
$\begingroup$

I'm trying to use the STFT as an input to a neural network. After flattening, there are over 4,000 features for a few seconds of audio. Is there a recommended way of summarising these to be a more manageable/useful feature set? All I can think of is using the stats like mean, variance and perhaps the higher order stats, but I can't help but feel there must be some better approach. Even with these stats, would it be better to take them per SFTF in isolation, or calculate them across all segments?

(As an aside, since this is related and someone coming across this question might also be interested, I am also looking into normalisation: https://danielsdiscoveries.wordpress.com/2017/09/29/spectrogram-input-normalisation-for-neural-networks/)

$\endgroup$
2
  • $\begingroup$ You're the one defining the length of your SFTF - maybe you should consider what you're doing in detail when you say you're using the SFTF as input! That thing is more flexible than you seem to realize. Generally, it's pretty impossible to tell you how you should preprocess your data if you don't tell us what kind of thing your overall system should be detecting in the end. $\endgroup$
    – mmmm
    Apr 5 '21 at 23:47
  • $\begingroup$ Don't flatten; 2D conv nets to reduce compute and exploit frequential/temporal dependencies $\endgroup$ Apr 5 '21 at 23:54
0
$\begingroup$

Given a $M \times N$ STFT (spectrogram), use this as the input to a convolutional neural network. Do not flatten the spectrogram. Since your spectrogram will be complex, then you can use the magnitude spectrogram or phase spectrogram or both. However, PyTorch recently released support for complex numbers, so you might be able to train the convolutional neural network using the spectrogram directly.

$\endgroup$
2
  • $\begingroup$ Can I do this with an RNN too? I presume you should be able to be, but I'm just barely starting to dip my toes into CNN/RNNs. I'm trying out neurolab, as it's been a lot simpler to prototype my idea. $\endgroup$
    – moinudin
    Apr 6 '21 at 12:30
  • 1
    $\begingroup$ Sure look up "Multi-dimensiomal RNN". Not sure why a CNN isn't good enough to do the job though. $\endgroup$
    – mhdadk
    Apr 6 '21 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.