5
$\begingroup$

I am trying to formalize my understanding of how best to measure the performance of a binary detector. (See previous question here). I had looked into ROC curves for doing this however based on previous research and feedback here I think ROC curves are too simplistic. Here is the problem I face:

Context:

I have a microphone that is setup in a park. The mic takes in inputs and an algorithm runs in the background that is supposed to detect the rustle of leaves. (Animal approaching, something in the bushes, etc). It does this by insert time-frequency-detection-scheme here, but that is not the issue. The end result of course is a 'Yes' or 'No', (So a binary detector), indicating whether or not leaves are indeed rustling.

Main Question(s):

  • How can/should I characterize this detector's performance, vis-a-vis true positive and false positive rates? (If that is the wrong way to go about it, then what is the right way?)

  • I understand that to generate a false-positive analysis, requires a false-positive model. In this case, the false-positive model is not something simple like white noise, but spans almost any signal possibility. Therefore, how can we model all those possibilities?

Thoughts:

My current solution/thought is to measure a true-positive rate and false-positive rate to characterize detector performance. (For given settings/thresholds).

Thus, for given and fixed detector settings, I test its true-positive rate by rustling leaves (within a specified range), and see if it detects the incidents. This will give me a true-positive rate. (A consistent 'Yes').

However I am stumped as to how to measure the false-positive rate.

I do not understand how to do this, because the mic is outside and subject to anything - cars, people, animals, thunder storms, just life. In other words, just about any combination or permutation of signals imaginable is possible. How then does one test its false positive rate?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ This sounds like a good experimental philosophy/methodology question for stats.stackexchange.com $\endgroup$ – hotpaw2 May 1 '12 at 17:21
  • $\begingroup$ @hotpaw2 Perhaps, although I believe there is a lot of overlap what with the number of detectors we are seeing here. $\endgroup$ – Spacey May 1 '12 at 17:33
3
$\begingroup$

The best test of a detector would be subjecting it to the environment within which it will be "detecting". If I understand correctly, you're asking about synthesizing or otherwise intentionally constructing content for testing for false-positives.

Because you real-world model is highly complex, it's not at all easy to synthesize. I would assume that your true and false positive testing involves listening to leaves shuffling and detecting that characteristic sound, but if I understand correctly you're doing that in a controlled environment without much external interference.

My advice would be to do all testing in a real-world environment with cars, people, and all the things you listed above, and literally spending the time to classify events and non-events yourself. You're essentially collecting training labels for your data, and someone with capability of distinguishing true/false positives/negatives must generate that data, i.e. a human observer. The reason to perform all testing (and not just searching for false-positives) is such environment is that it would affect your true-positive rate, too. Presence of other sounds can actually prevent your detector from discerning between background and desired signal when the desired signal is indeed present.

This procedure may seem lengthy, tedious and time-consuming, but from my experience ti saves you more time as apposed to trying to come up with a complex model with the noise you will experience. I'm not sure that the latter is at all feasible.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hmm, yes, I tend to agree. The real-world 'model' here is so complex and all-encompassing, that empirical labelling/test might be the only way. I suppose the good news is that once the data set is labelled, it will be ripe for some application of machine learning algorithms. ;-) $\endgroup$ – Spacey May 1 '12 at 20:53
  • $\begingroup$ Yes. You then could also post-process data to extract specific sounds that "tripped the alarm" and run consecutive tests with those sounds isolated. $\endgroup$ – Phonon May 1 '12 at 20:55
5
$\begingroup$

on an ROC curve your plotting FP(x-axis) vs TP(y-axis) They are calculated by:

True-positives = # of correctly detected positives/# of actual positives

False-Positives = # of positives that are not true-positives/ number of positives

Now on your ROC curve, because you did not mention incuding this into your question above, you most likely did not include the "coin-flip" line. That is the line y = x (from x = 0-1)

This line represents how well your algorithm performs by just guessing randomly (Yes, No)

If your algorithm is a decent detector, the ROC curve will it will be above the coin-flip line, if it is below this line it is better at being an anti-detector, i.e., when there is no event and would be a poor detector for your application.

I've worked extensively on this for my master's thesis on onset detection. Below is an example of ROC curves plotted against the coin-flip line. Receiver Operating Characteristic Curves for Various Music Onset Detection Algorithms

Improve this answer
$\endgroup$
6
  • $\begingroup$ @Cyberman Thank you for your answer. I understand the ROC curves, but I am not clear on how to generate false-positive data. For example, in your case, false-positive means you detected an onset, when the truth was that their was no onset. However, what types of signals did you test in order to test all possibilities? (Signals from speech, music, noise, car engine etc?) How did you decide that you have accounted for all different types of signals out there? Thanks! $\endgroup$ – Spacey May 1 '12 at 19:32
  • $\begingroup$ @Mohammad, I think we've talked about this in the past. (My username used to be Adam) This research was done on music signals. False-positive means that you're system detected a positive, but in reality it was a false alarm. (due to noise or other issues) $\endgroup$ – CyberMen May 1 '12 at 19:37
  • $\begingroup$ You cannot always account for all signals. You can try testing it on as many as possible, but in reality each detection algorithm is application specific. For example, if you tried to detect a beat in a percussive piece of music, all you would need is an envelope follower. If you tried to detect a singer changing smoothly between two pitches, envelope follower will fail horribly. $\endgroup$ – CyberMen May 1 '12 at 19:39
  • $\begingroup$ I see. In other words, we have to make a list of expected signal types that we expect first, and then simply use those. (Example, in your case, it is only used for music). It seems that in my application, this family of signal types is very open-ended is the problem. $\endgroup$ – Spacey May 1 '12 at 20:04
  • $\begingroup$ correct. You also need a labeled data set that marks where your event occurs so you can test the efficacy of your system. $\endgroup$ – CyberMen May 1 '12 at 20:18
3
$\begingroup$

A false positive analysis requires you to create some kind of model of your system. If a false positive has any sort of measurable effect (method under test says "it's a tree", human says "no, it's a crying baby"), then gather those events in as unbiased a method as possible until your model says they add up to be as statistically significant as your task requires.

If your model is off (whether by complexity, stupidity or criminal intent), then you can end up with stuff like yet another financial melt-down.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks hotpaw2. Therefore if I understand you correctly from this practical stand-point, at some point the false-positive model simply has to encompass all things we expect, (cars, people, thunder, etc), and the task then becomes to simply account for all possibilities as much as we can? $\endgroup$ – Spacey May 1 '12 at 18:57
  • $\begingroup$ You will still need a model (probability distribution of leaves rustling, cars passing, thunderstorms, Mayan end-of-the-world, etc.) in order to estimate a rate. $\endgroup$ – hotpaw2 May 1 '12 at 19:08
  • $\begingroup$ Agreed about the need to model. It seems as though the onus is on the designer to create 1, 2, 100, models for all distinct possibilities, and thus generate 1, 2, or 100 ROC curves accordingly. If some model was missed, (modeled everything except Mayan apocalypse), then cest la vie? $\endgroup$ – Spacey May 1 '12 at 19:20
1
$\begingroup$

I stated this as a comment in your previous thread, but perhaps it will register better as a full-blown response. A binary hypothesis problem of "Yes" and "No" is a gross oversimplification. A true binary hypothesis problem is the following:

Hypothesis 0 (H0) : Noise

Hypothesis 1 (H1) : Noise + Signal

Now, how you model your noise is entirely up to you and your application. If you're looking for a single audio tone, you may be fair in saying that "Noise" would be white Gaussian noise with an unknown variance. Your signal is well defined as a deterministic sinusoid with unknown phase and amplitude.

From this model, you can do a lot of analysis. That is, develop optimal detectors (if they exist), and easily determine Type I and Type II error probabilities etc.

However, it may not be appropriate for what's going on in real life. It's seems as if you're tackling a problem beyond your current scope of understanding. If you don't know even the basics, you're going to be "stepping back" quite a bit. As before, I recommend getting a good book on detection, one that covers through non-White noise, and starting with the basics before getting into something too advanced.

Improve this answer
$\endgroup$
3
  • $\begingroup$ May you please edit the post to focus on this particular question at hand. (eg, how would you model this 'noise', in all its possibilities). Cheers. $\endgroup$ – Spacey May 1 '12 at 18:14
  • $\begingroup$ I'm sorry you don't find my answer useful, and I won't contribute any more answers to your posts. You may delete this one if you'd like to reduce clutter. But notice that every single response to your last thread mirrors mine. You need a model, you need constraints. At the risk of sounding like a broken record, I wish you the best of luck and hopefully you find what you're looking for. Feel free to message me anytime if you'd like more clarification or some good book recommendations. $\endgroup$ – Bryan May 1 '12 at 18:44
  • $\begingroup$ Etiquette and civility are regarded above all other rules on Stack Exchange. Asking questions, answering questions, and asking to clarify answers must all be done politely. Please consider this for future activity on this and all other Stack Exchange sites. $\endgroup$ – Phonon May 1 '12 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.