Best parameter to estimate image reconstruction quality?

Question

Context: I am a PhD student in physics and right now I am studying phase retrieval with low illumination

With two different parameters I obtain these two images:

With hight illumination (less noise) and the same parameters I obtain:

The real object is pi. One parameter that I use to estimate the quality of the reconstruction is the Pearson coefficient with the real object.

Problem: I have the same correlation for the first pair of images (with low illumination) because the second one has less noise, but it has a worst resolution.

What is a parameter that describes the fact that, despite noise, the object is more recognizible in the first picture. I cannot take images with other objects, for example to evaluate spatial resolution for different parameters.

Answer 1

Since you say the π symbol will always be in the same place, you don't need to detect/locate it.

You can compare per-pixel. You could calculate a correlation score against a model image.

Since the appearance's magnitude seems to vary, I would recommend first applying a highpass (gradient/edge filter). That leaves edges, which approximate what we consider "recognizable" features.

I've applied a Sobel filter to your pictures. You can see the symbol in the first picture is barely visible, but not visible at all in the second. The first picture will give you some response in a correlation, the second one will not.

These results are below my expectations because you've only provided false-color plots, not source data. The false-color plots don't convert well into grayscale. Turning false-color back into source values is doable but it requires knowledge of the colormap that was used. I will not do that here.

Answer 2

You may take an empirical approach.
Train a model to detect the $\pi$.
This model basically output the probability of a $\pi$ in the input image.

Then give the output of your algorithm to the model and measure its quality by the probability of the detection model.

The procedure should be:

1. Create a data set of good and bad samples and some noise (Like low quality reconstruction). You can generate those with many models for image degradation (Blur, Noise, Sampling, etc...).
2. Train a classifier on the data set.
3. Tune the probability of the classifier.
4. Use it as a measure of quality.

I have done this in a project and it worked really well even if the training data set was synthetic.