I've got a question related to a comparison of image segmentation algorithms. I'd like to compare two algorithm results to each other in step one. In another step I'd like to compare the two algorithms results against a few ground truth segmented images. Unfortunately I got only a few ground truth data results in form of single images. So I can't compare all possible images and parts of an image to the two algorithms.
My first idea on step #1 was that I go down on subpixel area and look up which algorithm is more precise in relation to the original image and discuss it without the ground truth data. I'll take the difference of the pixels on which the difference lasts. If it's too obvious, e.g. if one algorithm fails completely than the other wins automatically.
In the second approach where I've got ground truth data I would measure the absolute difference in pixels of each algorithm on significant parts of an image.
So, what I wanted to know is: 1) Are this correct approaches for comparing segmentation algorithms? 2) How do I measure the difference in step 1? Do I have to measure all segments of each algorithm? 3) What is a good quality criteria? Is it possible to get a quantitative criteria for the two algorithms? 4) Do I have to compare the whole segmented parts of each algorithm to the ground truth data to gain absolutely correct results?
Please let me know if something is unclear or not correctly formulated. Any help, advice or feedback is appreciated! Thank you guys in advance.