14
$\begingroup$

The human eye is able to detect a square inside the square in the following two images, even though there are no obvious edges and no obvious local changes in the density of dots.

enter image description here

enter image description here

GPT4o believes to see a circle in the first image, Gemini a triangle. So they just guess.

I wonder which computer vision systems, namely CNNs, are able to see the square, and what's going on inside. How do they detect edges where there are no edges? Which local feature of the image gives rise to the virtual edges of the square?

This is a segment of the upper (pseudo-)edge of the square:

enter image description here


Side note: With identical textures (identical levels of order/disorder) the figure is perfectly camouflaged - you can only see it separated from the background when it moves. Watch out here.

$\endgroup$
3
  • $\begingroup$ Just look for few rows and find the angle between adjacent sample row wise. You can do columnwise as well. $\endgroup$
    – Creator
    Commented Jul 1 at 22:22
  • $\begingroup$ @Creator: Angle between what? $\endgroup$ Commented Jul 2 at 7:25
  • $\begingroup$ Angle betwee the lines connecting adjectents dots. It is not one angle, I mean you may consider angels within a area to decide. $\endgroup$
    – Creator
    Commented Jul 2 at 19:41

3 Answers 3

10
$\begingroup$

The name for this image processing task is texture segmentation. It takes into account not only the color of areas, but repetitive surface features, and uses them to divide the image into areas.

I gave it a quick try using FSEG_py which implements the algorithm from Factorization-Based Texture Segmentation. The result is not a perfect mask, but a reasonable start for further processing with e.g. morphological operations:

enter image description here

$\endgroup$
2
  • $\begingroup$ The segmented picture is fascinating. I wonder which histogram gave rise to it, and how it was calculated (pointwise?). Is it easy to grasp? $\endgroup$ Commented Jul 2 at 21:29
  • 1
    $\begingroup$ @Hans-PeterStricker I'm a bit lost on the details of the spectral histogram algorithm, but adding some intermediate saves I got example result from input filter and example histogram channel. Many of the other channels are not as good for separation, so the algorithm then uses multi-dimensional clustering to classify the data points. This is just one approach to texture segmentation, there exist algorithms using e.g. neural networks also. $\endgroup$
    – jpa
    Commented Jul 3 at 6:11
7
$\begingroup$

The perception pipeline is an interesting subject to study. One of my favorite lectures to teach (usually as part of an psychoacoustics course) is the one on illusions and I always sprinkle in lots of visual ones even though there are interesting auditory ones to discuss too. Illusions are a gateway into recognizing and getting guesses at what our perception pipeline might be doing and form conjectures from the observations.

Take your pseudo-edge. I personally do not "really" see this as an edge. However, your square does pop out at me as fairly clearly a square. One phenomenon that illusions illustrate in my mind is the notion that object/shape recognition is important.

The "presumption" of shape is strong. Consider this Kanisza figure:

Three rotated pacmans make us see a triangle

Most people see a triangle shape, though we can argue if it's really there. Our brain favors seeing shapes from mere suggestive evidence. However examples like this certainly complicates the question. Is local edge detection the right fit for model human perception of shape boundaries?

Still the question of object recognition in noise has been studied, and certainly local de-noising is a strategy employed to improve recognition, for example:

Another interesting example is the James Dalmatian and its recognition:

  • van Tonder, G. J., & Ejima, Y. (2000). Bottom–up clues in target finding: Why a Dalmatian may be mistaken for an elephant. Perception, 29(2), 149-157.
$\endgroup$
1
  • $\begingroup$ Thanks, Georg, for the links to Kanisza and the Dalmatian which I both had in mind, too. The difference with both compared to my images is the degree of "realness" of the edges which is a bit higher in my images, isn't it? In both of your cases the object as a whole obviously comes from top-down (as an expectation you may develop or not), in my case at least a bit more strongly from bottom-up. Again, it's a matter of (measurable?) degree. $\endgroup$ Commented Jul 4 at 7:34
3
$\begingroup$

When you see people can "see" it means our vision system is trained to extract features which assist to understand the image.

In this context seeing the different patterns in the arrangement of the dots: Random vs. Organized.

If you'd train an object detector on such train set it will be able to locate the squares.
Basically the training will adapt its feature extraction layers to extract the correct features.
In this case they will be more about the angles and the effect and response to horizontal and vertical filters.

If you have a script to generate such images, it will be no more then few minutes of training to solve this in my opinion.

$\endgroup$
6
  • $\begingroup$ So there will be local feature detectors for "random" and "organized", and edge detectors that react on abrupt changes of this feature - instead of abrupt changes in luminosity or color? $\endgroup$ Commented Jul 2 at 7:27
  • $\begingroup$ Considering that such images also for humans are rather rare "in the wild", it's astonishing that they developed such feature detectors - or is it a "distributed feature" possibly? $\endgroup$ Commented Jul 2 at 7:32
  • $\begingroup$ @Hans-PeterStricker, Indeed. While DL models can rival human vision in many tasks, the holy grail is learning from few samples. This is still where human excel much more than the current models. $\endgroup$
    – Royi
    Commented Jul 2 at 9:42
  • $\begingroup$ @Hans-PeterStricker Yes. $\endgroup$ Commented Jul 2 at 12:20
  • $\begingroup$ @Royi Humans learn from extraordinarily many samples - everything we can perceive as babies. That's why a lot of baby's toys are just generating new stimuli not otherwise found in the environment. With all this background knowledge, as adults, human adults can then "fine-tune" on new patterns with relatively few samples. $\endgroup$ Commented Jul 2 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.