# How do optical illusions work (photo mosaic) from a signal processing perspective?

Hello fellow investigators

I have two question about optical illusions

1) A photo mosaic is something like this:

What are the signal processing principles behind our eye merging the many tiny images into a large coherent one?

2) Are there etiological explanations of optical illusions in general - by this I mean can it be explained why image inputs get parsed in 'wrong' ways by the eye/brain, why does the human parser get confused; what transfer function is in play?

-
I don't know that I would call such a mosaic an optical illusion. Rather, the mosaic tiles are scaled down such that the eye is no longer able to resolve the details in each subimage. Each mosaic tile is tinted according to the colors present in the desired overall image. This is visually similar to a blurred version of the large image. –  Jason R Jul 20 '12 at 16:43
This is very borderline for us... it might be a better fit at Cognitive Sciences –  Lorem Ipsum Jul 20 '12 at 20:18
@yoda I can't speak for our mods over there, but I think we would probably be willing to take it. –  jonsca Jul 22 '12 at 5:18

What are the signal processing principles behind our eye merging the many tiny images into a large coherent one?

From what I've read (Source 1: Front-End Vision and Multi-Scale Image Analysis, Source 2: Information Visualization), our eyes apply something very similar to a set of gabor filters to the visual inputs we get. So if you, i.e. focus on Mona Lisa's eye, your brain processes gabor filters at the scale roughly the size of the eye, and such a filter removes details at smaller scales.

Are there etiological explanations of optical illusions in general - by this I mean can it be explained why image inputs get parsed in 'wrong' ways by the eye/brain, why does the human parser get confused; what transfer function is in play?

Regarding optical illusions in general: There is tons of research. The two books I've mentioned above quote some of it, but I don't know enough about it to compress it into a good answer here.

But regarding this special case, there's actually a very elegant explanation in Front-End Vision and Multi-Scale Image Analysis, p208, which I'll try to repeat here: Take a "natural" sample image. Split the image into small blocks. Multiply each block with a window function (e.g. a gaussian).

For example, I'll take the "Lena" image in Mathematica:

sample = ColorConvert[ExampleData[{"TestImage", "Lena"}], "Grayscale"];


split it into 16x16 pixel blocks:

blocks = Flatten[Partition[ImageData[sample], {16, 16}, {5, 5}], 1];


multiply each block with a gaussian window:

window = Table[Exp[-(x^2 + y^2)/25], {x, -7.5, 7.5}, {y, -7.5, 7.5}];
blockTimesWindow = Map[window * # &, blocks];


the blocks look something like this:

Next I'll use SVD to find the principal components of these blocks:

{u, s, v} = SingularValueDecomposition[Flatten /@ blockTimesWindow, 10];
Image[0.5 + Partition[#, 16], ImageSize -> 100] & /@ Transpose[v]


and visualize them:

These are the principal components of small parts of my test image. Or in other words: any small part of my test image is (approximately) a weighted sum of these principal components, plus noise. And it's not just this test image. Take any photograph of a natural scene, the result will look very similar.

As you can see, the principal components look suspiciously like gabor filter kernels or gaussian derivatives. That explains why filters like these are a good choice to analyze natural images: in a natural image, any small part is (almost) the weighted sum of a few gabor filter kernels plus noise. And it explains why evolution would choose this kind of low-level processing.

Now your image is not a natural image: Neighboring pixels are not correlated the way they would be in a natural image. But our eyes still use processes that were optimized for natural scenes, where differences at smaller scales don't carry as much important information, so they are filtered out. That explains why the illusion works.

-
Thanks a bundle - that was very useful to me the Gabor and PCA angle :D –  Daniel Bilar Jul 24 '12 at 12:33
p.s. sidenote last time I heard of Gabor was 2002: Gabor hash used in one way physical unclonable functions link –  Daniel Bilar Jul 24 '12 at 13:55

I don't have an answer for part 2. I hope this answers part of your question, I am interested in seeing answers for the second part.

Image mosaics are traditionally made up of many images and then printed so that each image can bee seen up close, but when the viewer is far enough away to see the whole image they blend together. In this case the effect is less the human brain but the image resolution delivered to the brain. In the provided example each image is represented by about an 8 pixel square. The precived effect is that this copy of the Mona Lisa has a lot of noise

I have processed the provided example with an 8x8 pixelize filter to show how each image has the correct average color:

Next I processed the provided example with a 15 px Gaussian blur to simulate noise reduction and shape coalescing. Humans are quite good at sensing a signal through noise.

-
I understand the principle .. thanks for running the filter on the image to illustrate the principle, I appreciate :D –  Daniel Bilar Jul 24 '12 at 12:40