# Features for audio levels imbalance in films between speeh/music/effects

What would be the key features of an audio track or set of tracks (like 5.1) for identifying film audio tracks that have a clear unbalance between speech levels and sound effects levels (apart from the obvious rms diference between tracks)? And how could they be extracted or calculated?

It's for a machine learning project about automatic identification of such files and ulterior processing/fixing. The imbalance could be on the same track, or in a combination of several ones.

• geez, i dunno. besides loudness, perhaps spectral centroid and maybe short-time effective bandwidth? – robert bristow-johnson Jan 27 '15 at 18:35

I would try a sum and difference calculation between the L and R channels to start. Usually when film audio is mixed, the speech is in both channels in phase while the special effects is out of phase.

So if you do:

X = L+R;
Y = L-R;


X should give you the signal with more speech and Y should give you the other effects. Then if you do an RMS calculation on X and Y, you can find how much are they off by.

To find level mismatches between speech & SFX, you may first need to identify where there is speech and look at loudness & spectral power.

Look at the Harmonic Product Spectrum (HPS) as an inexpensive algorithm to estimate the pitch, that is also immune to various kinds of noise. The pitch of speech has well understood, band-limited, harmonic structure, that make it a good feature to discriminate when you have speech vs non-speech/SFX.

Taking this idea further, you might characterize speech by using formant-like features or traditional speech recognition features like MFCC or PLP.

This really depends largely on what your definition of "mismatch" is. You can certainly determine what the average speech and the average "sound effect" level is. But how would you know what the "right" balance is? The T-Rex stomping down through the jungle clearly needs to be louder than the whisper of the poor humans hiding in the bushes. But by how much? How could you tell that's too loud or too soft?

• In addition, notice also that the definition of mismatch is strongly related to the actual listener (age, cultural background, hearing loss, ...), the reproduction format (stereo, 5.1, binaural, ...), the reproduction system (loudspeakers, headphones, soundbars, ...) and the environment (theater, living room, public transportation, ...). – audionuma May 28 '15 at 5:52
• I am planning on building a weka classifier for my own perception of the relative loudness of these different signals, using known correct or missmatched levels as output class, to classify films or track-sets. Another possible approach for too loud or not classification is to rate the goodness/correctness of the balance between levels in a bounded scale (0 to 5 stars/points, 0 to 10, or similar approach). But for any of these to properly function I need some objective FEATURES from the audio tracks and their relative characteristics. – shirowww Sep 10 '15 at 15:26