Resource recommendations to learn audio processing

Question

I'm looking for some good resources to learn audio processing for a machine learning task based on classification of users as either 'COVID-19 positive' or 'COVID-19 negative' based on their cough data. I'll need to extract features from audio files containing cough from a bunch of volunteers before using those features to train an ML model.

I tried looking some books up, but just about all of them seemed to treat this problem either too programmatically or incompletely; skipping over all the theoretical aspects for the same. Books on DSP seemed too generic to cover some important audio processing techniques. So, what I'm looking for are resources that can help me understand audio processing thoroughly, including (but not limited to) the topics mentioned below:

• Feature Extraction
• Audio Data Augmentation
• Audio Data Synthesis

Please note that I am a beginner, and have had no background in DSP (though I have no problem referring some books for this too before narrowing my focus down to audio signal processing if required) or any related field before. Also, the resource you recommend doesn't have to be a book - I'm fine with tutorials, blogs, articles, etc. as long as they are comprehensive enough and don't skip over the relevant theory. Multiple resources encompassing these topics are wholeheartedly welcomed.

Any help is much appreciated.

Answer 1

Here are a few books that might be helpful:

The Scientist and Engineer's Guide to Digital Signal Processing A good resource for you to get familiarized with DSP in general.

Understanding Digital Signal Processing Also a very good intro book with a lot of depth in terms of audio applications.

SPECTRAL AUDIO SIGNAL PROCESSING Slightly more advanced book which covers (in-depth) many topics related to audio signal processing in general.

Speech and Audio Signal Processing: Processing and Perception of Speech and Music Similar to SASP, this book develops some more techniques that might be useful for you.

The three things that you listed are covered to some extent in almost all of the books I mentioned above. Once you have the appropriate data, building the model itself, however, will be your own prerogative using techniques from Pattern Recognition and Machine Learning, for example. However, for building more advanced models you can check out this article: Deep Learning for Audio Signal Processing.

And finally here is an online article that gives a nice rundown for feature extraction of audio data: Audio Feature Extraction.

Answer 2

Julius O. Smith's original book is here:

• Julius O. Smith III,'s Spectral Audio Signal Processing, which is available online.

And for a music background, I'd recommend:

• Bill Sethares' Tuning, Timbre, Spectrum, Scale which has a website here.

Answer 3

I've had your exact learning concerns a few years ago, except I sought features for seizure classification, not audio. Luckily, I found both.

I recommend the following learning path, in order:

1. But what is the Fourier transform? A visual introduction
2. DSP Guide, chapters 7-11: convolution, Discrete Fourier Transform. This is the hardest, yet most important part - do not skip it if you intend to work with signals more than once, it will save tons of time later on. There's also a chapter explicitly on audio, but I've not read it.
3. The Wavelet Tutorial, parts 1 & 2. The idea's to learn time-frequency, not just CWT; the tutorial helps also understand STFT. For any but simplest real-world classification tasks, all 1D methods (FFT, filtering) are utterly useless, so ending studies at convolution and Fourier is self-handicapping.

As for features, the following is a somewhat biased presentation. The current state of the art on several audio classification tasks on several datasets, in context of limited data + no transfer learning (pure deep learning dominates otherwise), is Joint Time-Frequency Scattering, followed by (and it's an extension of) Wavelet Scattering [*], beating MFCC, spectrogram, and other alternatives. For a short explanation of their success, read the following sections in the [*] link: "Invariance, information preservation", "Analysis vs Synthesis", "Fourier Failure", "vs MFCC" - and if you have the patience, "Why shift invariance?", and "Why warp stability & linearization?". An explicit theoretical comparison of both scatterings vs MFCC is given in the following paper (which I first-authored): Differentiable Time-Frequency Scattering on GPU.

These features also have proven performance for synthesis, regression, and generative tasks, though I'm unsure how much is SOTA.

If you search, you'll find implementations of both in Python. Yet, said implementations will be flawed and potentially unacceptably slow. I'll be releasing mine in some near future, which you can be notified of by "Watch"-ing ssqueezepy or "Follow"-ing [*].

Lastly, for maximum performance, it is critical that any feature is paired with an appropriately-configured neural network. To this end, I recommend studying SP from an "information" point of view - if you can conclude that "nothing real-world is bandlimited" is a pointless statement, you're doing good. I intend to publish on this so I won't specify further, but I can recommend relevant articles by me: DFT Coefficients Meaning?, and Does zero-padding distort the spectrum?.