How do optical illusions work (photo mosaic) from a signal processing perspective?

Question

Hello fellow investigators

I have two question about optical illusions

1) A photo mosaic is something like this:

What are the signal processing principles behind our eye merging the many tiny images into a large coherent one?

2) Are there etiological explanations of optical illusions in general - by this I mean can it be explained why image inputs get parsed in 'wrong' ways by the eye/brain, why does the human parser get confused; what transfer function is in play?

Thanks for your time

Answer 1

What are the signal processing principles behind our eye merging the many tiny images into a large coherent one?

From what I've read (Source 1: Front-End Vision and Multi-Scale Image Analysis, Source 2: Information Visualization), our eyes apply something very similar to a set of gabor filters to the visual inputs we get. So if you, i.e. focus on Mona Lisa's eye, your brain processes gabor filters at the scale roughly the size of the eye, and such a filter removes details at smaller scales.

Are there etiological explanations of optical illusions in general - by this I mean can it be explained why image inputs get parsed in 'wrong' ways by the eye/brain, why does the human parser get confused; what transfer function is in play?

Regarding optical illusions in general: There is tons of research. The two books I've mentioned above quote some of it, but I don't know enough about it to compress it into a good answer here.

But regarding this special case, there's actually a very elegant explanation in Front-End Vision and Multi-Scale Image Analysis, p208, which I'll try to repeat here: Take a "natural" sample image. Split the image into small blocks. Multiply each block with a window function (e.g. a gaussian).

For example, I'll take the "Lena" image in Mathematica:

sample = ColorConvert[ExampleData[{"TestImage", "Lena"}], "Grayscale"];

split it into 16x16 pixel blocks:

blocks = Flatten[Partition[ImageData[sample], {16, 16}, {5, 5}], 1];

multiply each block with a gaussian window:

window = Table[Exp[-(x^2 + y^2)/25], {x, -7.5, 7.5}, {y, -7.5, 7.5}];
blockTimesWindow = Map[window * # &, blocks];

the blocks look something like this:

Next I'll use SVD to find the principal components of these blocks:

{u, s, v} = SingularValueDecomposition[Flatten /@ blockTimesWindow, 10];
Image[0.5 + Partition[#, 16], ImageSize -> 100] & /@ Transpose[v]

and visualize them:

These are the principal components of small parts of my test image. Or in other words: any small part of my test image is (approximately) a weighted sum of these principal components, plus noise. And it's not just this test image. Take any photograph of a natural scene, the result will look very similar.

As you can see, the principal components look suspiciously like gabor filter kernels or gaussian derivatives. That explains why filters like these are a good choice to analyze natural images: in a natural image, any small part is (almost) the weighted sum of a few gabor filter kernels plus noise. And it explains why evolution would choose this kind of low-level processing.

Now your image is not a natural image: Neighboring pixels are not correlated the way they would be in a natural image. But our eyes still use processes that were optimized for natural scenes, where differences at smaller scales don't carry as much important information, so they are filtered out. That explains why the illusion works.

Answer 2

I don't have an answer for part 2. I hope this answers part of your question, I am interested in seeing answers for the second part.

Image mosaics are traditionally made up of many images and then printed so that each image can bee seen up close, but when the viewer is far enough away to see the whole image they blend together. In this case the effect is less the human brain but the image resolution delivered to the brain. In the provided example each image is represented by about an 8 pixel square. The precived effect is that this copy of the Mona Lisa has a lot of noise

I have processed the provided example with an 8x8 pixelize filter to show how each image has the correct average color:

Next I processed the provided example with a 15 px Gaussian blur to simulate noise reduction and shape coalescing. Humans are quite good at sensing a signal through noise.