I am trying to formalize my understanding of how best to measure the performance of a binary detector. (See previous question here). I had looked into ROC curves for doing this however based on previous research and feedback here I think ROC curves are too simplistic. Here is the problem I face:
I have a microphone that is setup in a park. The mic takes in inputs and an algorithm runs in the background that is supposed to detect the rustle of leaves. (Animal approaching, something in the bushes, etc). It does this by insert time-frequency-detection-scheme here, but that is not the issue. The end result of course is a 'Yes' or 'No', (So a binary detector), indicating whether or not leaves are indeed rustling.
How can/should I characterize this detector's performance, vis-a-vis true positive and false positive rates? (If that is the wrong way to go about it, then what is the right way?)
I understand that to generate a false-positive analysis, requires a false-positive model. In this case, the false-positive model is not something simple like white noise, but spans almost any signal possibility. Therefore, how can we model all those possibilities?
My current solution/thought is to measure a true-positive rate and false-positive rate to characterize detector performance. (For given settings/thresholds).
Thus, for given and fixed detector settings, I test its true-positive rate by rustling leaves (within a specified range), and see if it detects the incidents. This will give me a true-positive rate. (A consistent 'Yes').
However I am stumped as to how to measure the false-positive rate.
I do not understand how to do this, because the mic is outside and subject to anything - cars, people, animals, thunder storms, just life. In other words, just about any combination or permutation of signals imaginable is possible. How then does one test its false positive rate?