Signal Processing Stack Exchange is a question and answer site for practitioners of the art and science of signal, image and video processing. It only takes a minute to sign up.
Sign up to join this community
I like to work with a convolutional neural network in combination with a spectrogram as input.
Assuming the spectrogram has the dimensions $T\times F$ (time$\times$frequency). Is it more natural to see the spectrogram as a 1D image with $F$ channels and the size $T\times 1$ or as a 2D image with the size $T\times F$?
Depending on this I work with 1D convolutions or with 2D convolutions. But I'm not sure whats the more natural way.
Thank you very much
Depends on on what kind of feature(s) you want your CNN trained. With 2D convolution, convolution in the F dimension of a T-F spectrogram will produce something like a real cepstrum, which has proved useful in speech recognition and musical pitch detection.
If you're looking to model timbre (i.e. how a certain type of sounds "looks" in the spectrogram), then presumably you'd need to look at all frequencies at once (i.e. the TxF signal). This is because timbre (and many other features) depend on relations between energies at different frequencies.
To add another example, in sound separation algorithms, some people train CNNs to look at the whole spectrom (TxF). Typically though, these algorithms look only at a "window" of time (e.g. 60 ms). This is necessary to capture temporal information about how sound changes over time. So, you may want to use T x F, but T doesn't cover all frames of the signal, only a segment of it at a time.
