1
$\begingroup$

I am matching camera images of projection screens with rendered pages of PDF documents using Pearson's and Spearman's correlation coefficient. When the images (3, 4, 5) are aligned, this works fine in spite of the clear shortcomings (optical aberration in the camera images, the random sample is not independent, uneven illumination, etc.).

In the figure below, Pearson's r between the left and middle images is 0.8395 and Spearman's r is 0.6499. Pearson's r between the left and right images is 0.7139 and Spearman's r is 0.5066.

A camera image (left) and two rendered PDF pages (middle and right) Histograms of the above three images

However, when the content of the rendered PDF page differs slightly from the document in the camera image or when the images (8, 9, 10) are more than slightly misaligned, the correlation coefficient ceases to be useful.

In the figure below, Pearson's r between the left and middle images is 0.771 and Spearman's r is 0.4719. Pearson's r between the left and right images is 0.779 and Spearman's r is 0.4963.

A camera image (left) and two rendered PDF pages (middle and right) Histograms of the above three images

Reducing the images to density estimates (i.e. discarding the pixel positions and comparing histograms) would likely remove valuable information. Non-feature-based image registration seems like a more promising direction. What technique would you recommend to attack this problem?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ why not use features like SURF and do a comparison based on these? $\endgroup$
    – Marcus Müller
    Sep 25, 2018 at 19:02
  • $\begingroup$ Thank you for the suggestion, I am going to try feature-based matching separately. However, some of the camera images have an alarmingly small signal-to-noise ratio, which is going to make feature extraction difficult. Since the images are well-aligned, I though correlation, which expects very little from the input signal, or a related non-feature-based similarity measure might work well. $\endgroup$
    – Witiko
    Sep 26, 2018 at 0:19
  • 1
    $\begingroup$ Total guess as I do not do image processing but it sounds similar to the purpose of doing a cross correlation function in that you compute the correlation at all delay offsets (as often there is a delay mismatch that such an approach would pick up). So in this case could it make sense to compute the correlation with a 2D pixel offset parameter (repeating it for all offsets)? $\endgroup$
    – Dan Boschen
    Sep 26, 2018 at 1:23
  • $\begingroup$ Finding a translation that maximizes the correlation might work nicely! Although perhaps not all the possible offsets, but just a small neighborhood of (0, 0), so that it does not take ages to compute. $\endgroup$
    – Witiko
    Sep 26, 2018 at 10:34
  • $\begingroup$ @MarcusMüller I added results of further experiments as an answer. $\endgroup$
    – Witiko
    Sep 28, 2018 at 14:24

1 Answer 1

Reset to default
1
$\begingroup$

As suggested by Dan Boschen, I maximized the Spearman's correlation coefficient over all possible “small” translations – where small is less than 15 pixels in L1-distance – of the rendered PDF page images. The results were underwhelming, but I suppose this method would work much better if the camera images weren't optically distorted.

In the figure below, Pearson's r between the left and middle images is 0.789 and Spearman's r is 0.479. Pearson's r between the left and right images is 0.8008 and Spearman's r is 0.5094.

Image registration by small translations

As suggested by Marcus Müller, I used a feature detector (ORB, not SURF) to find keypoints in the images. Rather than directly use the feature descriptors for matching, I used them to find a homography that registers the two images.

In the figure below, Pearson's r between the left and middle images is 0.6244 and Spearman's r is 0.4921. Pearson's r between the left and right images is 0.0213 and Spearman's r is 0.036. For my test case, this works remarkably well.

Image registration by feature matching

Improve this answer
$\endgroup$
1
  • $\begingroup$ Nice work! Very interesting. $\endgroup$
    – Dan Boschen
    Dec 27, 2018 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.