The closest resource I have come across is the BSD500 database:
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html
It is a set of natural images that have had the contours drawn by humans. When you download the test images there are also MATLAB files with all the contours.
However, the question is always "what is an edge?" Is it a step in brightness in the local image patch? Is a line an edge? Is it a boundary between semantic objects in a 2D image? The latter is what the above set tests. The best performing approaches for it also consider an abrupt change in texture to be an edge.
I guess the point is the performance of your edge detector is dependent on the application. One could argue that every edge detector is perfect as it perfectly detects edges according to its own definition. But that is just being cheeky.
You could also test some other properties, such as
- Is it invariant to rotation? Does the same line at 0 degrees and X degrees give the same result?
- Is it invariant to scaling? I.e. zooming of the image
- Is it invariant to view point? I.e. warping of the image
- Is it illumination invariant? Does it detect true weak edges as well as strong edges?
I also have a new edge detector that I've been working on. One of recent ideas I have had is to use the Mikolajczyk test set. It consists of eight sets of images with various transformations, including the above. Typically the test set is used to evaluate interest point descriptors.
The process would be
- Detect edges in each image in the set.
- Using the homographies, transform each detection image back to the first image in the set so they can be compared
- Compare the edges between images using one of the measures you mention.
Ideally, if the edge detector is invariant to the various translations, the same edges should be found. This approach might have already been done in the literature but I haven't looked.