4
$\begingroup$

I am developping a program, that extracts very expressive frames from a video (meaning: frames, which could summarize a video) for a university seminar. One of my ideas was, to analyze the audio data (e.g.: if it's very loud, there must be sth. happening and so on). I'm using Praat for this analysis. Now, I stumbled upon the pitch, which is supposed to be the fundamental frequency (formant f0). Can anyone tell me what usually happens to the pitch, when somebody talks or when you can hear only music? And do you have any other ideas what the audio data of a film might be able to tell me?

$\endgroup$
  • $\begingroup$ Maybe instead of guessing at audio features that might be important, you should have the program randomly pick frames and let humans vote on whether the frame is good for summarizing the video, and then go back and find commonalities between the good frames. $\endgroup$ – endolith Jun 13 '12 at 16:14
  • $\begingroup$ Depends to a significant degree on the music. Voice tends to have a lot of variation in amplitude, whereas music tends to be steadier. But rock music, etc, can have a lot of rapid amplitude variation. I have seen some hints of discrimination of the sort you're looking for with "spectral flatness" (and it's std dev), but nothing I can pin down yet. $\endgroup$ – Daniel R Hicks Jun 25 '12 at 3:21
7
$\begingroup$

I don't think pitch information is relevant for what you want to do.

The variation of pitch during speech is known as intonation, and can convey emotions, indicate if a sentence is a question etc. However, there is no universal rule as to how pitch variation patterns are mapped to meaning - this is quite language dependent ; and some languages sound "flatter" in terms of pitch contour than others.

While there are some exceptions (say a solo flute performance), music is polyphonic in that several instruments are playing at the same time - and to make things worse some of those instruments might be playing several notes at the same time. As a result, at a given point in time, music is unlikely to contain just a single fundamental frequency ; and when applied to music, Praat's f0 estimator is very likely to yield a noisy and unreliable pitch contour, sometimes jumping from one instrument to the other.

Regarding automatically annotating movies from the audio, here are a few ideas:

  • Looking for sharp increases in loudness to detect important events as you suggested.
  • Use speech/music discrimination to help detect scene boundaries.
  • Train classifiers for common sound effects (explosions, gunshots, punches), and use this to detect action/violence scenes.
  • Use speaker diarization techniques (along with some face recognition) to detect all lines spoken by a particular actor.
$\endgroup$
  • $\begingroup$ Can you suggest any software libraries or algorithms for speech/music discrimination? $\endgroup$ – swbandit Nov 25 '14 at 16:48
0
$\begingroup$

If the problem that you are actually addresing is audio/speech signal segmentation and distinguishing between the speech and music, see:

Theodorou T., Mporas I. and Fakotakis N., "Automatic Sound Classification of Radio Broadcast News", Int. J. Sig. Proc., Image Proc. Pattern Recognition, 5(1), 2012

I could be wrong on this but Praat seems to be not so usefull for polyphonic music analysis, except if augmented with some additional pre-filtering for the purpose of segmentation:

Makarand Ramesh, V.; Sahasrabuddhe, H.V.; , "Exploring Data Analysis in Music Using Tool Praat," Emerging Trends in Engineering and Technology, 2008. First International Conference on, pp.508-509

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.