audio spectrogram normalization

Question

I am performing language classification from audio signals using mel-spectrograms as my inputs for a ResNet. It works well as long as all of my audio data from different languages is from the same dataset (therefore in the same format). When I train on language A from dataset 1, and validate on language A from dataset 2, the accuracy for that language drops significantly, and instead the network guesses that a lot of the signals from language A are a different language from dataset 2.

This leads me to believe the network is first picking up on features that determine which dataset the audio came from, and then classifying the language as a secondary feature.

I tried reformatting data from each dataset so that it would contain the same amount of information. Dataset 1 (voxforge) consisted of wavfiles and dataset 2 (mozilla common voice) consisted of mp3 data, so I converted dataset 1 to mp3 data that had the same sampling rate and bits per sample as dataset 2.

I’m thinking there must be some other encoding artifact from the datasets that is throwing my network off. Has anyone else had issues with “normalizing” data across datasets or anything about converting encodings so that the information encoded is of equivalent quality?

Answer 1

First of all, yes, what can be achieved by any machine learning tool is only as good as the dataset it was trained on.

The ML can't guess what properties of the high-dimensional data you want it to respect and which not.

So, you'll have to make sure all your data sources are either equal in their probability distributions when it comes to the feature you're interested in (language), essentially.

A relatively easy way of doing so is random sampling, where you categorize all data sources according to languages, and then for each language, pick a subset of recordings so that all sources are equally frequent. You then train with different randomly picked subsets.

Really, this is Bayesian statistics basics; I don't think signal processing will help you here. You can sure still try to preprocess the audio data so that it only consists of relatively recording-method-invariate features (i.e. classical audio features, e.g. Cepstrum coefficients), but then you'd basically be doing what you'd instead wanted to use a neural net for: finding appropriate properties of the signal from which to tell the languages apart, and since the problem would persist, I'm pretty optimistic that the Resnet approach would still large correlate its classification to the data source, since some information would certainly persist through that feature-finding step, and your datasets still aren't even in terms of classification target pdf.

Answer 2

Are there any languages and vocabularies in both data sets? If so, examine the differences between data sets using only those samples common to both data sets. There could be differences in dialects, microphone response, equalization, background noise, room acoustics, HRTF, and etc.