Is it possible to classify signal samples using STFT without (image-based) spectrograms?

Question

I am aiming to conduct a classification-based study using signals collected from various devices. I've researched other approaches which make use of STFT for producing a spectrogram for speech and EEG/ECG signals and classifying the grayscale images.

While this is potentially a preferred approach, some speech-based classifications would make use of MFCC's, for example, from which some n mfcc coefficients would be used per sample as feature columns.

My question here is, in order to potentially speed up my model for a variety of classes and large number of samples - is there a preferred approach to avoid images and instead represent STFT (and/or other features) for signal sample classification?

Answer 1

If you are using a CNN based neural network for classification, my advice would be to use asymmetric strides to reduce the time axis (temporal information), which in return would reduce the required parameters for your classifiers and speed up your classification process.

However, if you want to use the matrix obtained after applying STFT, things might get ugly. Since the matrix you are going to have after applying STFT to signal frames will be complex, it might be a good idea to look into Complex Valued Neural Networks (CVNNs).

A Survey on Complex Values Neural Networks

In general, for speech recognition and acoustic scene classification problems only spectrograms or derivatives of spectrograms are used. These type of features discard the phase information. With CVNNs you can also incorporate the phase information in your classifier.

As an alternative, you can just flatten the magnitude of STFT matrix and use linear layers for classification if you really hate seeing things in 2D. (I have to warn you that doing this in general results in severe overfitting.)

Answer 2

STFT is inherently a 2D representation. Anything else depends on what we mean by "avoid 2D":

1. Transform further to collapse an axis (e.g. MFCC). Then it's pertinent to know what we're giving up for what we gain. Scattering's canonical use case is global averaging in time, providing fully time-shift invariant features, while partly recovering lost information via second order (unlike MFCC); this produces a vector of frequency features. One could do this on STFT, but it wouldn't be time-warp stable.

2. No 2D convs: that's fine, just do 1D along time. This trades flops for params.

3. Need speed: raise hop_size, lower n_fft (but mind loss of info)

4. Subset STFT: transform only frequencies of interest.

Important to note, STFT is not an image in standard sense.

Answer 3

"A preferred approach" is quite too simplistic. What works best will depend a whole lot on the signal and your training data.

State of the art in Audio Event Detection are CRNN's: CNN layers followed by RNN layers. But there's nothing stopping you from feeding each 1-D STFT in turn to a stateful RNN (or LSTM or GRU).