I am developping a program, that extracts very expressive frames from a video (meaning: frames, which could summarize a video) for a university seminar. One of my ideas was, to analyze the audio data (e.g.: if it's very loud, there must be sth. happening and so on). I'm using Praat for this analysis. Now, I stumbled upon the pitch, which is supposed to be the fundamental frequency (formant f0). Can anyone tell me what usually happens to the pitch, when somebody talks or when you can hear only music? And do you have any other ideas what the audio data of a film might be able to tell me?
I don't think pitch information is relevant for what you want to do.
The variation of pitch during speech is known as intonation, and can convey emotions, indicate if a sentence is a question etc. However, there is no universal rule as to how pitch variation patterns are mapped to meaning - this is quite language dependent ; and some languages sound "flatter" in terms of pitch contour than others.
While there are some exceptions (say a solo flute performance), music is polyphonic in that several instruments are playing at the same time - and to make things worse some of those instruments might be playing several notes at the same time. As a result, at a given point in time, music is unlikely to contain just a single fundamental frequency ; and when applied to music, Praat's f0 estimator is very likely to yield a noisy and unreliable pitch contour, sometimes jumping from one instrument to the other.
Regarding automatically annotating movies from the audio, here are a few ideas:
- Looking for sharp increases in loudness to detect important events as you suggested.
- Use speech/music discrimination to help detect scene boundaries.
- Train classifiers for common sound effects (explosions, gunshots, punches), and use this to detect action/violence scenes.
- Use speaker diarization techniques (along with some face recognition) to detect all lines spoken by a particular actor.
If the problem that you are actually addresing is audio/speech signal segmentation and distinguishing between the speech and music, see:
I could be wrong on this but Praat seems to be not so usefull for polyphonic music analysis, except if augmented with some additional pre-filtering for the purpose of segmentation: