I have several circle detectors. I want to evaluate their performance in finding the spherical steel markers in X-ray images. The radii of the circles are distributed in a narrow range around 8 px.
It is important that all markers are detected, it is not so important that there are no false positives.
So the first thing that comes to mind is AUC of precision-recall curves as a measure of goodness.
There is however a problem: some detectors return a probability map of some point being a center of the circle and have up to two parameters, while others operate on edge images and return circle center coordinates. Moreover one of the latter ones has many parameters to tweak. In order to have many points on the precision-recall curve for this detector I would have to sample the multidimensional parameter space.
Although technically it is not a problem I have reservations as of conceptual validity of this, since there is some a priori knowledge of reasonable values of those parameters and also they could be optimized by using a training set of images. This renders irrelevant the precision-recall curve points that were obtained with nonsense parameter, but these points nevertheless influence the AUC. Continuing this line of thought leads to a cross-validation approach, which I've never seen being used in articles on circle detection.
What is the correct way of evaluating circle detectors for a given task.
P.S. Also asked this on cross-validated stack exchange, since the question fits somewhere in between that an this community.