1
$\begingroup$

I'm trying to use the STFT as an input to a neural network. After flattening, there are over 4,000 features for a few seconds of audio. Is there a recommended way of summarising these to be a more manageable/useful feature set? All I can think of is using the stats like mean, variance and perhaps the higher order stats, but I can't help but feel there must be some better approach. Even with these stats, would it be better to take them per SFTF in isolation, or calculate them across all segments?

(As an aside, since this is related and someone coming across this question might also be interested, I am also looking into normalisation: https://danielsdiscoveries.wordpress.com/2017/09/29/spectrogram-input-normalisation-for-neural-networks/)

Improve this question
$\endgroup$
2
  • $\begingroup$ You're the one defining the length of your SFTF - maybe you should consider what you're doing in detail when you say you're using the SFTF as input! That thing is more flexible than you seem to realize. Generally, it's pretty impossible to tell you how you should preprocess your data if you don't tell us what kind of thing your overall system should be detecting in the end. $\endgroup$
    – mmmm
    Commented Apr 5, 2021 at 23:47
  • $\begingroup$ Don't flatten; 2D conv nets to reduce compute and exploit frequential/temporal dependencies $\endgroup$
    – OverLordGoldDragon
    Commented Apr 5, 2021 at 23:54

1 Answer 1

Reset to default
3
$\begingroup$

Given a $M \times N$ STFT (spectrogram), use this as the input to a convolutional neural network. Do not flatten the spectrogram. Since your spectrogram will be complex, then you can use the magnitude spectrogram or phase spectrogram or both. However, PyTorch recently released support for complex numbers, so you might be able to train the convolutional neural network using the spectrogram directly.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Can I do this with an RNN too? I presume you should be able to be, but I'm just barely starting to dip my toes into CNN/RNNs. I'm trying out neurolab, as it's been a lot simpler to prototype my idea. $\endgroup$
    – moinudin
    Commented Apr 6, 2021 at 12:30
  • 1
    $\begingroup$ Sure look up "Multi-dimensiomal RNN". Not sure why a CNN isn't good enough to do the job though. $\endgroup$
    – mhdadk
    Commented Apr 6, 2021 at 12:58
  • $\begingroup$ Avoid phase, mag's generally best but complex could work $\endgroup$
    – OverLordGoldDragon
    Commented May 27, 2023 at 3:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.