5
$\begingroup$

I'm trying to measure per-pixel similarities in two images (same array shape and type) using Python.

In many scientific papers (like this one), normalized cross-correlation is used. Here's an image from the ict paper showing the wanted result:

Link to the result image

(b) and (c) are the 2 input images, and (d) is the per-pixel confidence.

I only used OpenCV before to do template matching with normalized cross correlation using cv2.matchTemplate function, but in this case it seems to be a really different use of cross correlation.

Do you know if I can approch this result using Python and image processing libraries (numpy, openCV, sciPy etc...), and the logic behind this use of cross correlation ?

Thanks.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ It's not really too clear exactly what cross-correlation function you're trying to compute. Rather than give the link to the paper, could you write down the function here? OpenCV, numpy and scipy may not have a built-in method to do this, but I'm certain you can write a program using these tools to do what you need. $\endgroup$
    – Praveen
    Jan 9, 2016 at 6:46
  • $\begingroup$ Thanks for reply Praveen, the problem is that I don't know this use of cross correlation at all, and there is no formula in the paper(s), they just say ; italic We then compute normalized cross correlation between the static image (b) and the warped dynamic image (c) to produce the per-pixel confidence shown in (d). $\endgroup$
    – Paul Parneix
    Jan 9, 2016 at 14:38
  • $\begingroup$ stackoverflow.com/q/6991471/1461210 $\endgroup$
    – ali_m
    Jan 10, 2016 at 1:23

2 Answers 2

Reset to default
6
$\begingroup$

I guess you can compute for each pixel the correlation coefficient between patches centered on this pixel in the two images of interest. Here is an example where I downloaded the figure attached here and tried to compute the correlation in such a way. The output looks different from the one of the article, but it was to be expected since the resolution is very different.

from skimage import io, feature
from scipy import ndimage
import numpy as np

def correlation_coefficient(patch1, patch2):
    product = np.mean((patch1 - patch1.mean()) * (patch2 - patch2.mean()))
    stds = patch1.std() * patch2.std()
    if stds == 0:
        return 0
    else:
        product /= stds
        return product

im = io.imread('faces.jpg', as_grey=True)

im1 = im[16:263, 4:146]
sh_row, sh_col = im1.shape
im2 = im[16:263, 155:155+sh_col]

# Registration of the two images
translation = feature.register_translation(im1, im2, upsample_factor=10)[0]
im2_register = ndimage.shift(im2, translation)

d = 1

correlation = np.zeros_like(im1)

for i in range(d, sh_row - (d + 1)):
    for j in range(d, sh_col - (d + 1)):
        correlation[i, j] = correlation_coefficient(im1[i - d: i + d + 1,
                                                        j - d: j + d + 1],
                                                    im2[i - d: i + d + 1,
                                                        j - d: j + d + 1])

io.imshow(correlation, cmap='gray')
io.show()

Output of the script

The code above is a naive and slow implementation of the correlation, as the two for loops are very slow. For faster execution, you could for example port the script to Cython.

In the article, I think the idea is to measure whether face expressions look similar or not. If a pixel has a large correlation index between two images, it means that the region of the face where this pixel is located does not change much between the images.

If you use this method on good-resolution images, you should increase the patch size for more accurate results (d=2 or 3).

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Merci Emmanuelle ! That's a lot more clear now :). Thanks for this very complete answer, I'll have a look at Cython, it seems to be quite cool and powerfull. $\endgroup$
    – Paul Parneix
    Jan 11, 2016 at 6:18
  • $\begingroup$ Thank you very much for this question and clarifying answer of Emmanuelle. In the code above, there are two calculations which I think are related to the co-registration of the Master(b) and Slave(c) images. However in the calculation of NCC map you used the same original images (im1 and im2). Should not one use im1 and im2_register for calculation of the NCC map? With regards Sina $\endgroup$
    – Sinooshka
    Mar 14, 2019 at 10:19
  • $\begingroup$ what is the purpose of two lines with translation and im2_register calculated ? How did you come up with numbers in im2 = im[16:263, 155:155+sh_col] ? $\endgroup$
    – Farid Alijani
    Mar 6, 2020 at 14:56
0
$\begingroup$

This isn't a complete answer but nowadays Python's skimage module has a bunch of tools for template matching and feature detection:

Improve this answer
$\endgroup$
2
  • $\begingroup$ Last two links are dead. $\endgroup$
    – jlandercy
    May 30, 2021 at 13:01
  • $\begingroup$ @jlandercy i fixed the dead links. $\endgroup$
    – Trevor Boyd Smith
    May 31, 2021 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.