Metrics to evaluate the accuracy of image binarization algorithms?

Question

I am working on image binarization for my undergraduate project. The idea is to convert an RGB image to a binary image while retaining maximum features. I presented a comparison of the results of my algorithm and Otsu's algorithm for the project presentation (i.e. the output images of both algorithms, so visual inspection could be done). On visual inspection, Otsu's algorithm results have lesser features when compared to the algorithm I developed.

The professors who reviewed our work said visual inspection is not allowed, I need quantifiable metrics to prove the algorithm I worked on is better. Image binarization is a kind of segmentation, based on the literature review I have done it is different from binary image segmentation, which requires separating object from the background.

My research:

I found this: https://github.com/xuebinqin/Binary-Segmentation-Evaluation-Tool which is used to evaluate binary image segmentation accuracy. This technique uses a ground truth which has the object colored white and the background colored black. The results are compared with the ground truths using various techniques. I also found the Berkeley Segmentation Dataset which has ground truths and scripts to evaluate segmentation accuracy. However, these do not seem to be useful for my purpose because the goal of my project is different. I don't know what metrics I should use for my project.

Are there any such metrics?

Any ideas on this are welcome.

Answer 1

the algorithm I worked on is better.

Whenever you use the word "better" you need to define by what metric: could be all sort of things "runs faster", "less cpu or memory", "more robust against noise", etc. And yes, these do require a quantifiable definition, otherwise you can't rank order them.

In your case you want something that "looks better". That's a perceptual ranking, so it either needs a sufficiently accurate model of human perception OR actual humans. Human perception is incredibly complicated so in most cases actual humans is the only choice.

Fortunately there is whole branch of science that deals with "humans as measurement devices" called "Psychophysics". There is good evidence that you can construct well defined experiments involving human subjects and that these experiments can be meaningful, conclusive and highly reproducible.

Unfortunately it's not easy and tends to be time consuming. There are a lot of aspects of the experiment that need to be carefully controlled. Examples are

1. Size and composition of your subject group (age, gender, ethnicity, cultural & socio-economic background, medical history, etc.)
2. Selection of your test material: number and variety of images
3. Setup of the experiment itself and how it's introduced to the subjects. It is VERY easy to create an unintended bias depending on what you tell people upfront.
4. Questions you ask: "which one looks better?" is a VERY different question from "Which looks more realistic?" This goes back to the definition of "better": are you after realism, aesthetic preference, ease-of-recognition, something else ?
5. Class of answers & scale used for replies: (the four classical scales are nominal, ordinal, interval, or ratio). Two Alternative Force Choice, perhaps ?
6. Statistic and interpretation of the results.

I'm guessing that in your case, the most tricky part will be what exact question to ask and how you select the images. You can always fudge the results by pre-selecting images that support your desired outcome. That happens quite often: not intentional, it's just sub-conscious bias of the experimenter.

The amount of effort you want to apply there depends a fair bit on how "solid" the results need to be. Quick & dirty is fine for casual experimentation and if you want a quick read how things are trending. If you want to use it for a thesis or a paper, you should read up on the fundamentals of psychophysics and document carefully how you manage the different aspects of psychophysical experimentation.

Answer 2

Probably not what you want to hear, but I'd suggest showing the different images to a large number of people (more than 20) and getting them to choose which binary image best represents the color image. Collate the votes. See if other people think your images are better than the alternative.

What you're trying to do is to show that your technique better helps humans see the detail of the color image in a binary image. That's very hard to quantify.