I have some images (generated from some processing). Some of those images have approximately the same view of the scene with illumination changes. But there are other images which may show the view but with a different viewpoint. I am trying to remove the second set of images mentioned above. I've tried using correlation coefficient as a measure of finding similarity. But most of false image pairs seem to have high correlation coefficient. Then I've tried using mean-square error. On setting the value at 1500(guess), results have improved. Note all the images have similar size. This setting of 1500 changes for different images. So how do I choose this quantity? Or is there a better way to compare such images? (The images are gray-scale)
Edit:
I am sorry I was not specific. I have found SURF correspondences between the images and used the transformed images (obtained by projective transform) for matching between the transformed image and part of scene image(the ROI was extracted by transforming coordinates by obtained matrix above).
I have used SSIM as one answer suggested but that metric seems to punish illumination changes which is not what I want.