I am performing language classification from audio signals using mel-spectrograms as my inputs for a ResNet. It works well as long as all of my audio data from different languages is from the same dataset (therefore in the same format). When I train on language A from dataset 1, and validate on language A from dataset 2, the accuracy for that language drops significantly, and instead the network guesses that a lot of the signals from language A are a different language from dataset 2.
This leads me to believe the network is first picking up on features that determine which dataset the audio came from, and then classifying the language as a secondary feature.
I tried reformatting data from each dataset so that it would contain the same amount of information. Dataset 1 (voxforge) consisted of wavfiles and dataset 2 (mozilla common voice) consisted of mp3 data, so I converted dataset 1 to mp3 data that had the same sampling rate and bits per sample as dataset 2.
I’m thinking there must be some other encoding artifact from the datasets that is throwing my network off. Has anyone else had issues with “normalizing” data across datasets or anything about converting encodings so that the information encoded is of equivalent quality?