Simplest way of detecting where audio envelopes start and stop

Question

Below is a signal which represents a recording of someone talking. I would like to create a series of smaller audio signals based on this. The idea being to detect when 'important' sound starts and ends and use those for markers to make new snippet of audio. In other words, I would like to use the silence as indicators as to when an audio 'chunk' has started or stopped and make new audio buffers based on this.

So for example, if a person records himself saying

Hi [some silence] My name is Bob [some silence] How are you?

then I would like to make three audio clips from this. One that says Hi, one that says My name is Bob and one that says How are you?.

My initial idea is to run through the audio buffer constantly checking where there are areas of low amplitude. Maybe I could do this by taking the first ten samples, average the values and if the result is low then label it as silent. I would proceed down the buffer by checking the next ten samples. Incrementing along in this way I could detect where envelopes start and stop.

If anyone has any advice on a good, but simple way to do this that would be great. For my purposes the solution can be quite rudimentary.

I'm not a pro at DSP, but understand some basic concepts. Also, I would be doing this programmatically so it would be best to talk about algorithms and digital samples.

Thanks for all the help!

---

EDIT 1

Great responses so far! Just wanted to clarify that this is not on live audio and I will be writing the algorithms myself in C or Objective-C so any solutions that use libraries aren't really an option.

Answer 1

This is the classic problem of speech detection. First thing to do would be to Google the concept. It is widely used in digital communication and there's been a lot of research conducted on the subject and there are good papers out there.

Generally, the more background noise you have to deal with the more elaborate your method of speech detection must be. If you're using recordings taken in a quiet room, you can do it very easily (more later). If you have all sorts of noise while someone is talking (trucks passing by, dogs barking, plates smashing, aliens attacking), you'll have to use something much more clever.

Looking at the waveform you attached, your noise is minimal, so I suggest the following:

1. Extract signal envelope
2. Pick a good threshold
3. Detect places where envelope magnitude exceeds threshold

What does this all mean? An envelope of a signal is a curve that describes its magnitude over time, independently of how its frequency content makes it oscillate (see image below).

Envelope extraction can be done by creating a new signal that contains absolute values of you original signal, e.g. $\{ 1, 45, -6, 2, -43, 2 \ldots \}$ becomes $\{ 1, 45, 6, 2, 43, 2 \ldots \}$, and then low-pass filtering the result. The simplest low-pass filter can be implemented by replacing each sample value by an average of its N neighbors on both sides. The best value of N can be found experimentally and can depend on several things such as your sampling rate.

You can see from the image that is you don't have much noise present, your signal envelope will always be above a certain threshold (loudness level), and you can consider those regions as speech detected regions.

Answer 2

What you really want to do is essentially called as Voice Activity Detection or speech detection.

Basically any pure speech signal (which contains no music) has three parts.

1. The voiced sound - which is basically caused by Vowels
2. The unvoiced sound - which contains consonants.

The characteristic of human sound is such that while a lot of energy is used in voiced sound the real information is contained in consonants. Also, voiced sound is usually lower frequency where as unvoiced sounds are higher frequencies. [To be precise all voiced sound are resonated more or less a constant frequency for a given person which is his/her pitch].

Now, as any system there is noise. The voiced sound is usually quite powerful enough that it can be distinguished visible. When you apply a lower frequency filtering it is possible to collect good magnitude of voiced sounds however, the unvoiced sound (with all the rich information) will get lost.

Coming to the question how to solve it:

The trick lies in the fact that unvoiced sound still come from a resonating source; and inherently restricted over a certain frequency. Where as, the noise is rather uniform. So a simple measure that distinguish all three is "local power" or alternatively but equivalent is to take the windowed auto-correlation.

If you take at a time say 100 samples - and auto correlate itself, if it contains only noise the results will be pretty much zero (this is the property of white noise) where as for speech signal, this magnitude will be observable because the signal still has better structure. This has worked for me in the past.

VAD has been an active research areas- because almost all Mobile phone communications wants to detect non speech part and remove them from encoding. But if they would remove non-voiced speech this would make telephony useless.

The G.729 standard computes VAD based on features like: line spectral frequencies, full-band energy, low-band energy (<1 kHz), and zero-crossing rate.

The GSM standard works as follows: Option 1 computes the SNR in nine bands and applies a threshold to these values. Option 2 calculates different parameters: channel power, voice metrics, and noise power. It then thresholds the voice metrics using a threshold that varies according to the estimated SNR. (from wikipedia)

For more advanced techniques i am listing some references on this subject.

1. Most sited reference: Jongseo Sohn; Nam Soo Kim; Wonyong Sung; "A statistical model-based voice activity detection" Signal Processing Letters, IEEE, Jan 1999, Volume: 6 Issue:1 pp:1-3

2. Most relevant for you: Mark Marzinzik and Birger Kollmeier "Speech Pause Detection for Noise Spectrum Estimation by Tracking Power Envelope Dynamics" IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 10, NO. 2, FEBRUARY 2002 pp.109

3. Ramírez, J.; J. M. Górriz, J. C. Segura (2007). "Voice Activity Detection. Fundamentals and Speech Recognition System Robustness". In M. Grimm and K. Kroschel. Robust Speech Recognition and Understanding. pp. 1–22. ISBN 978-3-902613-08-0.

4. Introductory : Jonathan Kola, Carol Espy-Wilson and Tarun Pruthi "Voice Activity Detection"

Answer 3

I would totally second Jim Clay on his approach, but slightly vary the flavor using envelope:

We know that speech mainly occur around 1-2kHz. Your data sampling is likely to be 44kHz (this depend on your recording device). So what I would do first is a moving average of the squared signal in real time across 10 points, to have an envelope of the signal power. That will induce a delay in the detection, so you want to keep this low.

Then, I would add a calibration phase on your system: ask the user to remain silent, press a button, and record the background noise for let's say 10 seconds. Take the mean or median amplitude of the envelope, multiply by 2 to have a safety, and that would give you the threshold Jim has been talking about, automatically.

If it is not real-time recording, you may find useful to use 0-phase moving average to diminish the annoyance caused by the delay. Tell us if it works for you as it is.

Answer 4

Eric,

If you are truly after something quick and dirty, the first thing you have to get is the envelope, and I would do this simply (in MATLAB) by:

 envelope = abs(hilbert(yourSignal));

At that point, I would simply threshold, and 'voice exists' if you are above a certain threshold.

This is a very simple solution btw, but it might work for you.

Answer 5

I assume that you are dealing with real, not complex signals- if that is not the case, let me know and I can amend the answer.

Power is defined as the square of the signal (i.e. the signal samples multiplied by themselves). You can compare the power to some threshold to determine if there is speech present or not. You would probably need to do some measurements on your recordings to empirically find a good threshold.

If your recordings are "clean" (i.e. not much noise), I would probably go as simple as possible by comparing the instantaneous power (i.e. a single sample) to the threshold. This means that you don't even have to square it if you don't want to, you just need the absolute value and compare it to the square root of the power threshold, which can be precomputed. When you detect speech grab it and some amount of recording before it, to make sure you get all of the speech (maybe 1/10 of a second?). Keep going until you find a prolonged period of no samples that exceed the threshold. Again, the length of the period would need to be determined empirically.

Rinse and repeat.

Answer 6

I have written an activity detector class in Java. It's part of my open-source Java DSP collection. You can use the test program WavSplitter.java to check it out with a WAV file as input.