Downsampling audio for use in Machine Learning

Question

I'm trying to use the work (Neural Networks) done in this repo: https://github.com/jtkim-kaist/VAD

It says this:

Note: To apply this toolkit to other speech data, the speech data should be sampled with 16kHz sampling frequency.

I've got speech data at 48khz. I've read in places that reducing sampling rate is a complicated process, you can't just remove every nth datapoint, you have to filter things...

Is this necessary if I only intend to use the data in the Neural Network toolkit provided by the repo I linked? If so, is there an industry standard method for changing sample rate?

I realise that it probably depends on what features are being used. However the feature that is used is this:

MRCG (multi resolution cochleagra) concatenates the cochleagram features at multiple spectrotemporal resolutions

This is a ruddy complicated feature! Lets pretend we're just using a Melspectogram (unless you're willing to answer the question from the perspective of MRCG's).

Neural networks are likely to use features of a Melspectogram that we wouldn't think of. This makes me think it is unwise to train the Neural Net using downsampled speech data unless we intend to predict using 48khz data downsampled to 16khz forever after...

What do you think? Can I use my 48khz data - downsampled with no filtering - with the expectation that the model will work for prediction on real 16khz data?

And then for future readers sake, how about the other way? Say I had an 8khz file, could I increase the sample rate to 16khz without filtering?

Answer 1

Is this necessary if I only intend to use the data in the Neural Network toolkit provided by the repo I linked?

Yes.

Whether or not you are downsampling (instead of just decimating) has nothing to do with classification performance but rather, it is to preserve (as much as possible) the information contained in the signal.

When changing the sample rate of a signal, you need to account for aliasing. Otherwise, the signal will be distorted in such a way that it would be impossible to reconstruct the original signal from the samples.

If so, is there an industry standard method for changing sample rate?

Yes.

The "right" way to do it when downsampling is to first apply an anti-aliasing filter and then decimate the signal but when upsampling, you first upsample and then apply interpolation (which can also be expressed as a filter). Various platforms provide functions to do just that (e.g. Python, MATLAB).

I realise that it probably depends on what features are being used.

To some extent. Perhaps you could simply leave everything unchanged, provided that you adjust certain points in the algorithm that are specifically hardwired for a sampling frequency that is different than the one you are using. However, the bandwidth for speech is limited. If your classifier is also trained on what is NOT speech, having more bandwidth means that you can discriminate more things. If the majority of your dataset is pure speech then having more bandwidth would not make a huge difference.

Can I use my 48khz data - downsampled with no filtering - with the expectation that the model will work for prediction on real 16khz data?

No.

If you downsample without filtering you are very likely to produce distorted signals.

And then for future readers sake, how about the other way? Say I had an 8khz file, could I increase the sample rate to 16khz without filtering?

No.

When you increase the sample rate, you have a more serious problem. You now have to fill "gaps" for which you do not have data. The filtering (or interpolation) stage fills in those gaps but does not add any new information. When you downsample, you are throwing away information. Upsampling cannot replace it.

Perhaps it would be beneficial for you to look up some basics of DSP such as sampling, sampling and reconstruction, aliasing and anti-aliasing filters. Also, to further understand what your features look like, how do they work and how to choose and use them, you might want to look up more information on the Short Time Fourier Transform and Multirate Signal Processing (more generally).

Hope this helps.