The human voice has some peculiarities that might help. For one, female voice starts at about 200Hz, male lower, so doing a high-pass filter here would help a bit. See also acoustic-features-distinguishing-the-male-and-female-voiceacoustic-features-distinguishing-the-male-and-female-voice.
Additionally, identify the drone frequencies via a histogram.
Doing all of this in code is a bit cumbersome. Have you considered some audio program like Audacity?