I am developping a program, that extracts very expressive frames from a video (meaning: frames, which could summarize a video) for a university seminar. One of my ideas was, to analyze the audio data (e.g.: if it's very loud, there must be sth. happening and so on). I'm using Praat for this analysis. Now, I stumbled upon the pitch, which is supposed to be the fundamental frequency (formant f0). Can anyone tell me what usually happens to the pitch, when somebody talks or when you can hear only music? And do you have any other ideas what the audio data of a film might be able to tell me?
2
-
$\begingroup$ Maybe instead of guessing at audio features that might be important, you should have the program randomly pick frames and let humans vote on whether the frame is good for summarizing the video, and then go back and find commonalities between the good frames. $\endgroup$– endolithCommented Jun 13, 2012 at 16:14
-
$\begingroup$ Depends to a significant degree on the music. Voice tends to have a lot of variation in amplitude, whereas music tends to be steadier. But rock music, etc, can have a lot of rapid amplitude variation. I have seen some hints of discrimination of the sort you're looking for with "spectral flatness" (and it's std dev), but nothing I can pin down yet. $\endgroup$– Daniel R HicksCommented Jun 25, 2012 at 3:21
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
I don't think pitch information is relevant for what you want to do.
The variation of pitch during speech is known as intonation, and can convey emotions, indicate if a sentence is a question etc. However, there is no universal rule as to how pitch variation patterns are mapped to meaning - this is quite language dependent ; and some languages sound "flatter" in terms of pitch contour than others.
While there are some exceptions (say a solo flute performance), music is polyphonic in that several instruments are playing at the same time - and to make things worse some of those instruments might be playing several notes at the same time. As a result, at a given point in time, music is unlikely to contain just a single fundamental frequency ; and when applied to music, Praat's f0 estimator is very likely to yield a noisy and unreliable pitch contour, sometimes jumping from one instrument to the other.
Regarding automatically annotating movies from the audio, here are a few ideas:
- Looking for sharp increases in loudness to detect important events as you suggested.
- Use speech/music discrimination to help detect scene boundaries.
- Train classifiers for common sound effects (explosions, gunshots, punches), and use this to detect action/violence scenes.
- Use speaker diarization techniques (along with some face recognition) to detect all lines spoken by a particular actor.
answered Jun 13, 2012 at 9:01
-
$\begingroup$ Can you suggest any software libraries or algorithms for speech/music discrimination? $\endgroup$– swbanditCommented Nov 25, 2014 at 16:48
$\begingroup$
$\endgroup$
If the problem that you are actually addresing is audio/speech signal segmentation and distinguishing between the speech and music, see:
I could be wrong on this but Praat seems to be not so usefull for polyphonic music analysis, except if augmented with some additional pre-filtering for the purpose of segmentation:
