The human voice has some peculiarities that might help. For one, female voice starts at about 200Hz, male lower, so doing a high-pass filter here would help a bit.
Additionally, identify the drone frequencies via a histogram.
Doing all of this in code is a bit cumbersome. Have you considered some audio program like Audacity?