In my foray into detections, I am trying to understand how best to generate receiver operating characteristic (ROC) curves, and some questions have come up.
In my studies, I have found out that ROC curve generation is simply the plot of true-positive rate (TPR) versus false-positive rate (FPR). Naturally one wants TPR to be 1 and FPR to be 0. I also understand that each point on the ROC curve, (a point with one TPR and FPR coordinate) corresponds to one set of detector 'settings' (threshold(s)). With one setting fixed, we will compare all positive 'detects' to all actual number of true positives, and true negatives, and thus yield TPR and FPR. My questions are as follows:
It seems to me that we must have a labelled data set(s) in order to do this to begin with. That is, we have to manually or otherwise, go through all the data we have, and label them as 'yes' or 'no'. (I realize this might seem obvious but I want to confirm). Is this true?
Somewhat related to the above - if in order to generate a true ROC curve that properly captures the detector performance we have to have 'no' data, then it seems to me that we have to have 'no' data that captures almost every possibility that exists in the universe. Clearly this is not feasible, but then how can one claim with any good faith that a detector truly has a certain false alarm rate? This is what confuses me the most.
Is it unheard of for detectors to have ROC curves that almost never false alarm, but that have normal looking variance on the y-axis? Put another way: Is it unheard of for detectors to have very little variance on the false-positive axis (x-axis) but a lot more variance on the true-positive axis (y-axis), for a fixed SNR?
I think that just about summarizes my questions, thanks.
EDIT Number 1:
I do not think I understand the feedback.
How can I statistically 'model' every single possibility in order for me to theoretically characterize it to begin with? For example, lets say that I have a detector that detects the presence of a signal of 10 Hz. The input can be anything under the sun. Maybe its 10Hz in noise. Maybe its 11Hz and no noise. Maybe its 10.5Hz on top of a linear chirp, or a bird song. Maybe its brown noise overlayed with human voice, overlayed with a 60Hz power supply. How does one model that, or any of the myriad combinations therein?
I do not know if I can get the theoretical performance of my detector, because quite simply I do not see how thats possible for realistic scenarios where anything can be your input! This goes back to Point#2, whereby, how can one possibly know of every possible input into the detector? I could be wrong on this so please show me how.
Therefore in this sense, I suppose my question is: I already have a detector in place. It takes an input vector, and spits out a decision, 'yes' or 'no'. I understand that I have to label my data sets in order to come up with true positive and false positive rates. In this case it seems to me that the only possibility is to come up with a data set that represents just about anything possible under the sun, and test that against your detector.
What am I not getting here?