# Using STFT as an input to a Neural Net

I'm trying to use the STFT as an input to a neural network. After flattening, there are over 4,000 features for a few seconds of audio. Is there a recommended way of summarising these to be a more manageable/useful feature set? All I can think of is using the stats like mean, variance and perhaps the higher order stats, but I can't help but feel there must be some better approach. Even with these stats, would it be better to take them per SFTF in isolation, or calculate them across all segments?

(As an aside, since this is related and someone coming across this question might also be interested, I am also looking into normalisation: https://danielsdiscoveries.wordpress.com/2017/09/29/spectrogram-input-normalisation-for-neural-networks/)

• You're the one defining the length of your SFTF - maybe you should consider what you're doing in detail when you say you're using the SFTF as input! That thing is more flexible than you seem to realize. Generally, it's pretty impossible to tell you how you should preprocess your data if you don't tell us what kind of thing your overall system should be detecting in the end.
– mmmm
Apr 5 '21 at 23:47
• Don't flatten; 2D conv nets to reduce compute and exploit frequential/temporal dependencies Apr 5 '21 at 23:54

Given a $$M \times N$$ STFT (spectrogram), use this as the input to a convolutional neural network. Do not flatten the spectrogram. Since your spectrogram will be complex, then you can use the magnitude spectrogram or phase spectrogram or both. However, PyTorch recently released support for complex numbers, so you might be able to train the convolutional neural network using the spectrogram directly.