Measuring similarity between two images

Question

I have some images (generated from some processing). Some of those images have approximately the same view of the scene with illumination changes. But there are other images which may show the view but with a different viewpoint. I am trying to remove the second set of images mentioned above. I've tried using correlation coefficient as a measure of finding similarity. But most of false image pairs seem to have high correlation coefficient. Then I've tried using mean-square error. On setting the value at 1500(guess), results have improved. Note all the images have similar size. This setting of 1500 changes for different images. So how do I choose this quantity? Or is there a better way to compare such images? (The images are gray-scale)

---

Edit:
I am sorry I was not specific. I have found SURF correspondences between the images and used the transformed images (obtained by projective transform) for matching between the transformed image and part of scene image(the ROI was extracted by transforming coordinates by obtained matrix above).
I have used SSIM as one answer suggested but that metric seems to punish illumination changes which is not what I want.

Answer 1

I don't quite know what you mean by different viewpoints, however, there are a lot of different similarity metrics you could use to compare images. You could try:

• extracting features and comparing the features Like Scale-invariant feature transform (SIFT) features
• you could try dimensionality reduction and comparing that representation, like taking the discrete cosine transform (DCT).
• I know you said you tried mean squared error but you could also try comparing the structural similarity (SSIM).

Edit:

Now that I better understand your question, I believe I found a resource that could be of assistance. Here is a very aptly named paper that is titled "A Novel Algorithm for View and Illumination Invariant Image Matching".