I have several collections contaning low-resolution captures of unevenly-lit presentation slides:
and thirty to sixty original high-resolution slide images:
How would you design a similarity measure between the low-resolution captures and the high-resolution slides that would nearly always give the highest score to the correct image pair (or, analogously, a distance measure that approaches zero only for the correct image pair)?
So far I tried a machine-learning solution to get a better feeling for how difficult the problem is. I assigned each image pair a feature vector consisting of:
- various grayscale histogram similarity / distance measures (correlation, intersection, $\chi^2$ distance, and Bhattacharyya distance) applied to both the entire images and the image quadrants,
- Pearson's correlation coefficient between Haralick texture features,
- Hamming distance between various image hashes (aHash, pHash, dHash), and
- Levenshtein distance between the image OCRs normalized by the maximum OCR length.
For the training part of a dataset, I take each feature vector $\vec x$, let $\vec a\vec x + q = 1$ for matching images and $\vec a\vec x + q = -1$ for non-matching images. Then I use linear regression to obtain $\vec a$ and $q$. Finally, for the test part of a dataset, I use $\vec a\vec x+q$ as a scoring function to retrieve a sorted list of high-resolution slides for each low-resolution capture and look at the rank of the correct result.
With a small hand-annotated dataset, this is getting me the average rank of 6, which is better than random draws, but not nearly good enough for an application that would overlay the high-resolution slides over live capture; such an application will require an average rank that approaches 1. There is an opportunity for the application to be smart and only look at a small window of slides around the current slide, but it still needs to correctly guess the initial slide.
As you might expect, the feature vectors are also quite slow to compute (several seconds per a feature vector with a naive implementation on a higher-end quad-core laptop), which makes the solution unsiutable for a real-time application.