# Is it possible to classify signal samples using STFT without (image-based) spectrograms?

I am aiming to conduct a classification-based study using signals collected from various devices. I've researched other approaches which make use of STFT for producing a spectrogram for speech and EEG/ECG signals and classifying the grayscale images.

While this is potentially a preferred approach, some speech-based classifications would make use of MFCC's, for example, from which some n mfcc coefficients would be used per sample as feature columns.

My question here is, in order to potentially speed up my model for a variety of classes and large number of samples - is there a preferred approach to avoid images and instead represent STFT (and/or other features) for signal sample classification?

If you are using a CNN based neural network for classification, my advice would be to use asymmetric strides to reduce the time axis (temporal information), which in return would reduce the required parameters for your classifiers and speed up your classification process.

However, if you want to use the matrix obtained after applying STFT, things might get ugly. Since the matrix you are going to have after applying STFT to signal frames will be complex, it might be a good idea to look into Complex Valued Neural Networks (CVNNs).

A Survey on Complex Values Neural Networks

In general, for speech recognition and acoustic scene classification problems only spectrograms or derivatives of spectrograms are used. These type of features discard the phase information. With CVNNs you can also incorporate the phase information in your classifier.

As an alternative, you can just flatten the magnitude of STFT matrix and use linear layers for classification if you really hate seeing things in 2D. (I have to warn you that doing this in general results in severe overfitting.)

• Thanks for the answer! There's not really a true goal for necessarily needing the STFT, but it was my first task I did which shows my signal coming through clearly (ofc there are other techniques). I'll look at CVNNs and see where I get however. Mar 17 at 20:54
• Regarding CVNNs, I'm looking at the cvnn Python module. However, I'm confused as to how I would represent the STFT for the dataset for each sample I have. I.e., I read in the sample and get the STFT (say from scipy.signal.stft) and this would give me f,t,Zxx where Zxx is the STFT of the signal. However for lots of signals putting all the Zxx (assuming i dont need f and t) into a dataframe for example is going to ramp up memory usage for the loaded dataset array. Thoughts? Mar 18 at 10:45
• In general, this is an issue of batch size. You can just set the batch size to '1', and then feed only one STFT (Zxx matrix).
– Avio
Mar 18 at 21:39

STFT is inherently a 2D representation. Anything else depends on what we mean by "avoid 2D":

1. Transform further to collapse an axis (e.g. MFCC). Then it's pertinent to know what we're giving up for what we gain. Scattering's canonical use case is global averaging in time, providing fully time-shift invariant features, while partly recovering lost information via second order (unlike MFCC); this produces a vector of frequency features. One could do this on STFT, but it wouldn't be time-warp stable.

2. No 2D convs: that's fine, just do 1D along time. This trades flops for params.

3. Need speed: raise hop_size, lower n_fft (but mind loss of info)

4. Subset STFT: transform only frequencies of interest.

Important to note, STFT is not an image in standard sense.

"A preferred approach" is quite too simplistic. What works best will depend a whole lot on the signal and your training data.

State of the art in Audio Event Detection are CRNN's: CNN layers followed by RNN layers. But there's nothing stopping you from feeding each 1-D STFT in turn to a stateful RNN (or LSTM or GRU).

• That is true, I did look into feeding the STFT vectors into an LSTM but it seemed a bit of an unstable approach. To keep it simple I have a dataset of 2s long RF signals with 2MHz sampling rate with each sample corresponding to a specific class. One option was to just pass in the entire files of the samples but that was quite heavy. I'll take a look into CRNNs as well as the other options from other answers and see where it leads for my use case, thanks! Mar 17 at 20:57