3
$\begingroup$

In my foray into detections, I am trying to understand how best to generate receiver operating characteristic (ROC) curves, and some questions have come up.

In my studies, I have found out that ROC curve generation is simply the plot of true-positive rate (TPR) versus false-positive rate (FPR). Naturally one wants TPR to be 1 and FPR to be 0. I also understand that each point on the ROC curve, (a point with one TPR and FPR coordinate) corresponds to one set of detector 'settings' (threshold(s)). With one setting fixed, we will compare all positive 'detects' to all actual number of true positives, and true negatives, and thus yield TPR and FPR. My questions are as follows:

  • It seems to me that we must have a labelled data set(s) in order to do this to begin with. That is, we have to manually or otherwise, go through all the data we have, and label them as 'yes' or 'no'. (I realize this might seem obvious but I want to confirm). Is this true?

  • Somewhat related to the above - if in order to generate a true ROC curve that properly captures the detector performance we have to have 'no' data, then it seems to me that we have to have 'no' data that captures almost every possibility that exists in the universe. Clearly this is not feasible, but then how can one claim with any good faith that a detector truly has a certain false alarm rate? This is what confuses me the most.

  • Is it unheard of for detectors to have ROC curves that almost never false alarm, but that have normal looking variance on the y-axis? Put another way: Is it unheard of for detectors to have very little variance on the false-positive axis (x-axis) but a lot more variance on the true-positive axis (y-axis), for a fixed SNR?

I think that just about summarizes my questions, thanks.

EDIT Number 1:

I do not think I understand the feedback.

Theoretical aspect:

How can I statistically 'model' every single possibility in order for me to theoretically characterize it to begin with? For example, lets say that I have a detector that detects the presence of a signal of 10 Hz. The input can be anything under the sun. Maybe its 10Hz in noise. Maybe its 11Hz and no noise. Maybe its 10.5Hz on top of a linear chirp, or a bird song. Maybe its brown noise overlayed with human voice, overlayed with a 60Hz power supply. How does one model that, or any of the myriad combinations therein?

I do not know if I can get the theoretical performance of my detector, because quite simply I do not see how thats possible for realistic scenarios where anything can be your input! This goes back to Point#2, whereby, how can one possibly know of every possible input into the detector? I could be wrong on this so please show me how.

Practical aspect:

Therefore in this sense, I suppose my question is: I already have a detector in place. It takes an input vector, and spits out a decision, 'yes' or 'no'. I understand that I have to label my data sets in order to come up with true positive and false positive rates. In this case it seems to me that the only possibility is to come up with a data set that represents just about anything possible under the sun, and test that against your detector.

What am I not getting here?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ It's unclear whether you are asking about predicting the theoretical performance of a detector given a statistical model or measuring the performance of a detector that has been implemented. If you're doing theoretical analysis, then you use the statistical properties of the detector's input signal and its decision rule to determine what its performance is. As long as your model is valid, then measured performance should agree (within statistical confidence intervals) with what you predicted. $\endgroup$ – Jason R Apr 26 '12 at 15:48
  • $\begingroup$ @JasonR Thank you, I have edited the text. Unfortunately I am not sure I understand what you mean. (See above). $\endgroup$ – Spacey Apr 26 '12 at 21:21
  • 1
    $\begingroup$ It all depends on your model. A classic detection model looks at a binary hypothesis, called H0, and H1, for the null and alternative hypothesis respectively. You run the simulation with just noise, to determine Type I errors (probability of false alarm), and with your signal to determine probability of detection. If the model is more complicated, like for example with narrow band noise or jammers, you need to define it as such and develop the mathematical models. This is oversimplified. I recommend looking into Fundamentals of Statistical Signal Processing, Volume 2: Detection Theory. Cheers $\endgroup$ – Bryan Apr 26 '12 at 21:54
  • $\begingroup$ It is unclear what is being asked here. Could you bold your questions using ** to enclose them like so $\endgroup$ – CyberMen Apr 27 '12 at 18:46
4
$\begingroup$

Question 1) Yes, you must know what the right answer is before-hand in order to judge the receiver's performance. I am doing this right now on a project. I generate data, scramble it, modulate it, add noise and carrier offset, etc., then I give it to the receiver. The bits I started out with are my "yes/no" labels.

Question 2) All such measurements are statistical in nature. Noise is statistical. Jitter is statistical. A priori channel conditions are statistical. Of course the ROC curve is also statistical. How could it be otherwise?

Question 3) I don't quite understand this question. A picture would help.

EDIT: Receivers are not tested in every set of conditions under the sun. First, the designers rely on everyone obeying the FCC requirements for a given band, so in a cellular band they assume that there won't be any TV stations, or radar chirps, and that all other cellular signals will be at specified frequencies and power levels, with specified maximum carrier offsets.

The designers then model "typical" environments (i.e. channels). That implies certain transmission losses due to free-space path loss, antenna gains, multi-path, etc. They then put together various typical and worst-case scenarios and test it under those conditions. For example, a worst case interference scenario would be having adjacent signals at the maximum allowed power.

If the performance is as expected for those selected conditions, the designers assume, through "interpolation", that it will perform acceptably in the other non-tested environments. Conditions can always be worse than the worst-case test scenario of course. That is when the phone drops the call. :-)

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks - about (2): (Please be aware Im very new to ROC curves). That being said, Im very lost on this. You have a vector, that can take any form. How are you generating all your 'false' data that you will put through your detector such that you can measure its false-positive rate? Your input can take any form, so how do you generate all possible forms?? $\endgroup$ – Spacey Apr 26 '12 at 21:24
  • $\begingroup$ @Mohammad You don't generate all forms. As you note- you simply can't do it. Typically you generate "enough" random data sequences and test using those. Think of it like political polling. If you get enough data, it is usually pretty accurate. How do you determine how much is enough? A long time ago I learned a heuristic that you need at least 100 bit errors to get a reasonably accurate BER measurement. $\endgroup$ – Jim Clay Apr 26 '12 at 21:48
  • $\begingroup$ Im sorry, I dont mean to be a wet blanket but I do not understand: Imagine you stick a microphone in a park and have it continually listen to the environment. You have designed it to detect the presence of rustling leaves. (Yes / no). Now one day you want to test its false positive rate. How do you account for sound of: 1) Children playing, 2) Dogs barking, 3) Car horns, 4) Nothing, 5) A UFO landing... and so on. Maybe you make artificial data on all the above and you have a low false alarm rate. Then one day a cat comes and meows into your mic and it 'detects' leaves. Then what? $\endgroup$ – Spacey Apr 26 '12 at 22:12
  • $\begingroup$ (Not trying to be factitious really trying to get this. :-) ) $\endgroup$ – Spacey Apr 26 '12 at 22:13
  • 1
    $\begingroup$ @Mohammad A given ROC curve can only be defined for a given set of conditions. So you could set up different ROC curves for whatever conditions you want to test. White gaussian noise is used a lot as a generic interference source. $\endgroup$ – Jim Clay Apr 27 '12 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.