1
$\begingroup$

I am in the process of developing a video game where I would like to sync a character's mouth with an audio clip of their speech that is playing. I found this great website that shows the profiles of different vowel sounds in a frequency spectrum: http://hyperphysics.phy-astr.gsu.edu/hbase/music/vowel.html

After a little (actually, a lot of) effort, I am able to show in real time the spectrum of the audio that is playing.

Spectrum of Audio

The next step I am planning to do is find the 4 largest peaks and try to match it to one of the vowel profiles from the website above. However, I have noticed that the relative power of the frequency domain coefficients is always strong lower than higher. The picture I showed is in silence. During an SSSSSS sound, the spectrum does show peaks at higher frequencies:

enter image description here

If I ignore the first few points in the silence FT and then relatively graph the rest, the spectrum has the same shape. If I ignore a few more, again the lower frequencies are much stronger than the higher frequencies.

I assume that this is the result of the math which I can barely get through to code the basic FFT. Should I be normalizing the result somehow to be able to see peaks in the higher frequencies? What process should I use for this?

Thanks in advance for any help you can provide!


To add info requested in the comments, by "in real time", I mean that every frame (at about 60 frames per seconds), I get a set of 1024 samples (sampled at 44100 Hz) that is centered at the point of audio that is currently playing). I then compute the FT of these samples and draw the spectrum.

$\endgroup$
  • 1
    $\begingroup$ are you plotting a log or linear scale horizontally? $\endgroup$ – danimal Apr 10 '15 at 16:49
  • $\begingroup$ linear scale horizontally. $\endgroup$ – PatrickSharbaugh Apr 10 '15 at 16:50
  • $\begingroup$ have you tried a log scale? $\endgroup$ – danimal Apr 10 '15 at 16:54
  • $\begingroup$ Thank you! That helped make the upper frequencies higher, but it still seems that there is a higher sensitivity to lower frequencies. $\endgroup$ – PatrickSharbaugh Apr 10 '15 at 17:11
  • $\begingroup$ You could try plotting a log scale vertically too in that case $\endgroup$ – danimal Apr 10 '15 at 17:15
1
$\begingroup$

Not sure if your approach is the right path. Anyhow, hearing aid research has produced many "long term average speech spectrum" (LTASS) curves that you can use to whiten the spectrum by dividing it with the LTASS. Have a look at http://www.dydaktyka.cba.pl/AvSpeech.pdf (Byrne et al. 1994 "An international comparison of long‐term average speech spectra" J. Acoust. Soc. Am. 96, 2108) Just don't divide by the LTASS dB values but linearize them first to $10^{LTASS\_in\_dB/20}$ if you are dealing with magnitude frequency spectrum or to $10^{LTASS\_in\_dB/10}$ if you are dealing with power spectrum.

Here's the worldwide average LTASS from Byrne et al. 1994:

Worldwide LTASS

Freq.   Male    Female  Combined
(Hz)    (dB)    (dB)    (dB)
63      38.6    37.0    38.6
80      43.5    36.0    43.5
100     54.4    37.5    54.4
125     57.7    40.1    57.7
160     56.8    53.4    56.8
200     58.2    62.2    60.2
250     59.7    60.9    60.3
315     60.0    58.1    59.0
400     62.4    61.7    62.1
500     62.6    61.7    62.1
630     60.6    60.4    60.5
800     55.7    58.0    56.8
1000    53.1    54.3    53.7
1250    53.7    52.3    53.0
1600    52.3    51.7    52.0
2000    48.7    48.8    48.7
2500    48.9    47.3    48.1
3150    47.0    46.7    46.8
4000    46.0    45.3    45.6
5000    44.4    44.6    44.5
6300    43.3    45.2    44.3
8000    42.4    44.9    43.7
10000   41.9    45.0    43.4
12500   39.8    42.8    41.3
16000   40.4    41.1    40.7
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.