# audio reprocessing for machine learning

I have a bunch of phone interviews where there is large sound volume difference between two voices in some of the interview audios, because they were recorded only on one side of the phone. May I ask what is the preferred procedure for reprocessing these audios so they are fit for downstream machine learning feature extraction and modelling tasks while the characteristics of the audio will not be sacrificed?

After some reading on audio compression, normalisation, and levelling, e.g. here, I have to say I am still not sure what to do next. I have seen simple minmax scaling being suggested for normalising the waveform amplitude. I have also been suggested to normalise the audio to a target amplitude (e.g. normalize in pydub.effects or ffmpeg loudnorm/dynaudnorm), which is loudness normalisation if I am not mistaken. Should I use a compressor or automatic gain controllor? Descript has a auto-leveling feature that seems to be useful, though I can't find any technical info on it.

My current plan, which is the simplest one, is to segment the audio streams to speaker turns, then normalise each speaker turn to (-1, 1) by minmax scaling. How does this sound? Btw any suggestions on declipping? Some of the audios have been clipped. Thanks!

• What is the "downstream machine learning feature extraction and modelling" part supposed to be doing?
– A_A
Apr 21 '20 at 9:21
• Thanks for your reply @A_A. It means the speech feature extraction from waveforms, and then fed into a classifier such as svm, for classification. Apr 21 '20 at 14:02
• Can I please ask if this was resolved?
– A_A
Jun 2 '20 at 9:07

The decision on which method to use to scale the input is very much determined by the objective and therefore what follows the scaling.

A simple linear scaling (whether peak, minmax or other) propagates to the rest of the processing chain as a multiplication. But anything that affects the dynamics of the signal (how quickly it rises and falls) ultimately shapes its content and might affect the classification results. I am assuming here, that this classification is based on some sort of spectral features (?)

The point is to look at the feature space and make sure that it is more or less spread uniformly across its bounds and representative of each class. Any improvement brought about by scaling of the original dataset would be assessed on this basis.

If your original material is something like a talk show, or other occassion where there is clean speech and no overlap of speakers then plain linear scaling would be fine or not even required.

But if you have a heated up discussion or other "quirky" content where people might be coming near the microphone or there are shouts or unpredictable background noise or people talking and overlapping, then that is much more difficult to resolve. A compressor could help up to a point but you may find that you have to provide more training samples to the classifier anyway, so that it learns that "Calm speaker A" and "Shouting speaker D" is the same person (i.e. scaling would not help much there).

...any suggestions on declipping?

In reality, if the audio clipped the information that was to be recorded at that point is entirely lost. And it should be treated as entirely lost.

But, if you have a huge amount of data and not quite but somewhat clipped audio. Maybe it was clipping but not all the time, maybe it was saturating and popping but not entirely hitting the rail and staying there for 5 seconds (for example),....then it might be possible to treat this as a regression type of problem and extrapolate what the clipped part was supposed to look like. This however is easier said than done because there are a lot of parameters that affect the performance of the declipping. For more information see here.

Hope this helps.

• Thanks very much @A_A. That is helpful indeed. I have about 50 audio recordings, they are conversations between two people and there IS fair amount of overlap in speakers. I do have time stamps for each speaker turn though quite a few of these does contain the voice of the other person or other sound e.g. door shut etc. Would you go about scaling the speech linearly per speaker turn? After compression? Apr 22 '20 at 22:58
• I am using speech rhythm features, which capture power distribution, rate and rhythm stability metrics... Apr 22 '20 at 23:02