4
$\begingroup$

I'm trying to measure per-pixel similarities in two images (same array shape and type) using Python.

In many scientific papers (like this one), normalized cross-correlation is used. Here's an image from the ict paper showing the wanted result:

Link to the result image

(b) and (c) are the 2 input images, and (d) is the per-pixel confidence.

I only used OpenCV before to do template matching with normalized cross correlation using cv2.matchTemplate function, but in this case it seems to be a really different use of cross correlation.

Do you know if I can approch this result using Python and image processing libraries (numpy, openCV, sciPy etc...), and the logic behind this use of cross correlation ?

Thanks.

$\endgroup$
  • 1
    $\begingroup$ It's not really too clear exactly what cross-correlation function you're trying to compute. Rather than give the link to the paper, could you write down the function here? OpenCV, numpy and scipy may not have a built-in method to do this, but I'm certain you can write a program using these tools to do what you need. $\endgroup$ – Praveen Jan 9 '16 at 6:46
  • $\begingroup$ Thanks for reply Praveen, the problem is that I don't know this use of cross correlation at all, and there is no formula in the paper(s), they just say ; italic We then compute normalized cross correlation between the static image (b) and the warped dynamic image (c) to produce the per-pixel confidence shown in (d). $\endgroup$ – Paul Parneix Jan 9 '16 at 14:38
  • $\begingroup$ stackoverflow.com/q/6991471/1461210 $\endgroup$ – ali_m Jan 10 '16 at 1:23
5
$\begingroup$

I guess you can compute for each pixel the correlation coefficient between patches centered on this pixel in the two images of interest. Here is an example where I downloaded the figure attached here and tried to compute the correlation in such a way. The output looks different from the one of the article, but it was to be expected since the resolution is very different.

from skimage import io, feature
from scipy import ndimage
import numpy as np

def correlation_coefficient(patch1, patch2):
    product = np.mean((patch1 - patch1.mean()) * (patch2 - patch2.mean()))
    stds = patch1.std() * patch2.std()
    if stds == 0:
        return 0
    else:
        product /= stds
        return product

im = io.imread('faces.jpg', as_grey=True)

im1 = im[16:263, 4:146]
sh_row, sh_col = im1.shape
im2 = im[16:263, 155:155+sh_col]

# Registration of the two images
translation = feature.register_translation(im1, im2, upsample_factor=10)[0]
im2_register = ndimage.shift(im2, translation)

d = 1

correlation = np.zeros_like(im1)

for i in range(d, sh_row - (d + 1)):
    for j in range(d, sh_col - (d + 1)):
        correlation[i, j] = correlation_coefficient(im1[i - d: i + d + 1,
                                                        j - d: j + d + 1],
                                                    im2[i - d: i + d + 1,
                                                        j - d: j + d + 1])

io.imshow(correlation, cmap='gray')
io.show()

Output of the script

The code above is a naive and slow implementation of the correlation, as the two for loops are very slow. For faster execution, you could for example port the script to Cython.

In the article, I think the idea is to measure whether face expressions look similar or not. If a pixel has a large correlation index between two images, it means that the region of the face where this pixel is located does not change much between the images.

If you use this method on good-resolution images, you should increase the patch size for more accurate results (d=2 or 3).

$\endgroup$
  • 1
    $\begingroup$ Merci Emmanuelle ! That's a lot more clear now :). Thanks for this very complete answer, I'll have a look at Cython, it seems to be quite cool and powerfull. $\endgroup$ – Paul Parneix Jan 11 '16 at 6:18
  • $\begingroup$ Thank you very much for this question and clarifying answer of Emmanuelle. In the code above, there are two calculations which I think are related to the co-registration of the Master(b) and Slave(c) images. However in the calculation of NCC map you used the same original images (im1 and im2). Should not one use im1 and im2_register for calculation of the NCC map? With regards Sina $\endgroup$ – Sinooshka Mar 14 at 10:19
0
$\begingroup$

This isn't a complete answer but nowadays Python's skimage module has a bunch of tools for template matching and feature detection:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy