Human voice has the same frequencies as other noises, it is not possible to filter it with dumb algorithm. It is possible to filter based on temporal patterns like spectral subtraction, but this way you can only filter static noise, not background speech/bable noise. Pocketsphinx already has spectral subtraction.
To filter noise reliably you need an advanced noise filtering neural network or something like that, see RNNNoise, it is very resource-intensive. Also, it is better to recognize noise sound instead of filtering it since recognizer can figure out things by himself.
One important approach is to reliably filter is to use direction of arrival of the sound, that is a microphone array which can filter noises based on direction. An example of such array is Respeaker or Matrix One. Thats why all smart speakers have microphone arrays as a critical component.
For more accurate recognition it is better to use modern toolkit with neural network like Kaldi, it is also much more accurate with noise filtering and without it.
Further reading for you:
Snips - Snips is a partially proprietary framework, but you can get some nice ideas and hardware advices. It is easy to get a good system using their packages too. Also see Community projects
Make a smart speaker with Respeaker
Zamia Speech the easiest way to run Kaldi on Raspberry Pi