My problem is that I don't know the energy of the background noise, so I can't just threshold the energy. The processing is done in real time, and I have about 500msec to decide. Ideally, I'd want quiet consonants considered non-silence.
There are a bunch of parameters that you can look at:
- Overall energy
- Short term spectrum: Speech has a fairly distinctive "pink-like" spectrum and noise (which is happening during the non-speech parts) tends to be white if it's electrically dominated or "red" (i.e. low frequency heavy) if it's acoustic background noise or microphone noise
- Amplitude statistics. Most noise signals have a Gaussian distribution, speech is closer to a Laplace distribution
I think a combination of these three should give a fairly robust detection scheme.