0
$\begingroup$

MFCCs are ubiquitously extracted for speech processing tasks, but I would like to know how suitable they are for non-speech processing tasks. Intuitively, it is my understanding that MFCCs are powerful features because they are more representative of how the human ear "hears" things. However, for tasks that are seemingly dissimilar from those that would have warranted such a biological adaptation, should MFCCs still be used? For example, in acoustic machine failure prediction, what is the logic behind using MFCCs, if any? Such a domain appears to be one that the human ear shouldn't have any significant advantages in.

Perhaps the human ear has evolved to better identify subtle changes in sound for hunting, or detecting predators, making MFCCs relevant to anomaly detection tasks.

Any thoughts? Should MFCCs be used universally?

$\endgroup$
0
$\begingroup$

The biological foundation for MFCCs are pretty weak. The mel-scale, which the mel-spectrogram and thus MFCC uses, is a rough approximation of how frequency differences are perceived along the frequency axis. The log transform of amplitudes is better than linear, but not really a good model of the ears loudness perception - which is non-uniform both with amplitude and frequency, as well as state-dependent (temporal masking). The choice of hop-lengh is also maybe roughly close to what the human ear has, but vastly simplified - the ear temporal resolution is not uniform either. And no binaural representation either, a critical part of the human auditory system.

The primary reason for using MFCCs is its numerical convenience. The log mel-spectrogram (when computed with appropriate parameters) preserves a large amount of the information in an audio signal - but reduces the feature space considerably and introduces useful structure (time-frequency). Using a 32 ms hop, 512 samples at 16kHz = 32 frames per second, and 64 mel bands one can reduce one second of audio from 16000 dimensions to 64x32=2048. Then when computing the DCT-II to get the MFCC, most of the energy of this ends up in the lower coefficients. This lets us drop the higher coefficients without loosing much information. Typical would be 20 (or 13) coefficients, leading to 20x32=640 dimensions. The information in the different coefficients also tends to be much less correlated than the input spectrogram, an advantage for some models - such as the commonly used GMM-HMM (Gaussian Mixture Model, Hidden Markov Model).

Bottom line is that the feature space with 640 MFCCs per second, instead of 16000 samples per second, is much easier to learn. This dimensional reduction and feature transformations of MFCC is very useful, even if the input is not speech. So one can make OK performing systems for many non-speech tasks using MFCC.

That said with the advent of deep learning and larger amounts of data and compute - MFCC based models are being outperformed by log mel-spectrograms processed by 2d Convolutional Neural Networks, as well as raw waveforms processed by 1d CNNs.

Note also that in the majority of cases, we are still struggling to achieve human level performance. And with supervised learning, our labels are typically provided by humans - a performance roof which the model cannot easily surpass. In this scenario it is not that problematic that our audio reception is limited by loose inspirations from the human auditory system. There are notable exceptions though, such as ultrasound (for bat detection etc) and infrasound (for condition monitoring) - there it is critical to go beyond the frequency range that human hearing has.

$\endgroup$
4
  • $\begingroup$ Hi @jonnor! Thanks for the insightful answer! To you know any references that one could cite that back up your first paragraph? $\endgroup$ Feb 3 at 8:38
  • 1
    $\begingroup$ Hi @verena-haunschmid. For some strong references, I would try to find a text-book on Psychoacoutics or Audiology. I don't have a recommendation for that handy. One more applied book that covers some background on hearing system is Sensory Evaluation of Sound by Nick Zacharov. $\endgroup$
    – Jon Nordby
    Feb 3 at 15:50
  • 1
    $\begingroup$ For loudness the Wikipedia articles on Loudness and Auditory masking are ok starting points. $\endgroup$
    – Jon Nordby
    Feb 3 at 15:52
  • 1
    $\begingroup$ Some of the weaknesses of the Mel scale are usually discussed around other models, such as the Bark scale and Equivalent rectangular bandwidth (ERB) $\endgroup$
    – Jon Nordby
    Feb 3 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.