Is this statement correct from DSP aspect?

Question

So, In my report I am comparing the related works, In most of the previous work, researchers have used spectrogram as input to vanilla CNN and classify. Where I am using different handcrafted features from the signal and then feeding it to DNN.

I want to explain why the second method works better than first, for that I am comparing like this

Motivated by prior work, Several AI-based approaches have been proposed to classify sounds using signal analysis. Most of them use short-term magnitude spectrograms transformed from sound data as input to Convolutional neural network (CNN). The drawback of this method is, Since CNN is based only on a spectrogram input, some domain-specific important characteristics might get missed in the feature space. Such as Zero crossing rate(ZCR), Skewness, Shannon Entropy, Fundamental frequency (F0), Formant frequencies etc

I just wanted to know, Is this statement correct from DSP aspect? because I am saying in my statement that spectrogram doesn't contain zero-crossing rate(ZCR), Skewness, Shannon Entropy, Fundamental frequency (F0), Formant frequencies etc

Answer 1

It's almost a matter of philosophy, i.e., difficult to argue hard facts.

On the one hand all the features you mention can be extracted from the raw signals. So in theory the network should be able to learn how to do that if they provide meaningful information for the task at hand. This is what part of the ML community is claiming: feature engineering is dead, let the networks figure out themselves what are the best features, we're only biasing them by what we think are good features, which may be wrong.

On the other hand, if you do have very good reasons to believe that some features are particularly useful, it can actually help the network. It wouldn't need to learn to discover such features first and thus, it could train much faster. This is related to the concept of known operator learning, where you embed fixed operators (that you know in advance should be beneficial to your task) into the network to make it train faster. I'm leaning towards this approach since it often makes sense.

If people used complex-valued STFTs to the input, no data were lost and all the features you mentioned (ZCR, F0, ...) could still be extracted from it. In a spectrogram, the phase is discarded, so one can argue that in fact some information is lost. If you just use your features as input though, it still does not represent the entire signal so some information is still lost and you may have to argue why you think this information is irrelevant.

Here is where it is difficult to argue hard facts and the only thing you may be able to do is show actual training results to make your point.