I need to find the squares in an image using OpenCV (no problem in matlab or any other, generally what i expect are some ideas).

Consider the test image below :

enter image description here

I need to find those coloured squares in above image accurately (not the white long strips).

What I have done :

  • I applied the common method (which comes with OpenCV samples), ie find contours in all color planes, approximate it and check for number of elements=4. It works to some extend that, few squares are detected, especially the dark ones.

  • Next step i did was prediction. ie this arrangement is fixed. So, if some are obtained, I can predict remaining ones. It also worked to some further extend. But accuracy was very bad.

But I feel prediction is not a good method here and it doesn't always provide accurate answers as given by the first step.

What I need :

1) Is there any other better methods to detect these squares more accurately? Or multiple methods?

One important point is that, time is not a problem here. Algorithm can be slow, it doesn't matter. But accuracy is the major criteria.

Sometimes, images can be much more blurry.

And one of the major problem I faced is that some squares have almost similar color as that of background (check column 3 first and second squares).

Looking for ideas, thanks in advance


Below is the maximum accurate result I got :

enter image description here

Of course, the result image is resized a little bit.


I have given a much more better solution in my answer below: https://dsp.stackexchange.com/a/7526/818

  • $\begingroup$ is your background always whitish ? $\endgroup$
    – remi
    Oct 5, 2012 at 15:36
  • 1
    $\begingroup$ my idea was to compute the "saturation" and threshold the sat img but using your example it does not work very well (computing saturation as max(R-G,R-B,G-B). The fact that some of the squares look almost like the background make the thing quite hard. If all your images have the same pattern (long white stripes with squares next to them) you should consider finding the most easy bits (such as the really colored squares, or the white stripes) deduce the possible location for other squares and...find a way to check if they are really there or not.Tough but interesting! Can you give more images? $\endgroup$
    – remi
    Oct 5, 2012 at 21:24
  • $\begingroup$ Well, I think this shouldn't have been moved. $\endgroup$
    – Junuxx
    Oct 8, 2012 at 16:25
  • 1
    $\begingroup$ Can you provide more images? Also, what is this thing? $\endgroup$ Oct 8, 2012 at 23:43
  • 2
    $\begingroup$ OP needs to answer some questions. Maybe white background is not necessary. And what's up with light? Is it going to be that bad? These look just unnecessary complexity to me. $\endgroup$ Oct 9, 2012 at 1:54

7 Answers 7


A first attempt using Matlab:

im = imread('squares.jpg');
im2 = rgb2gray(im);

se = strel('disk', 15);

for i = 1:16;
    t = 60+i*5; % try out a range of bw thresholds to see what works best
    labelled = bwlabel(im2>t); % label regions in the BW image
    closed = imclose(labelled, se); % close small regions
    cleared = imclearborder(~closed,4); % clear regions touching the border
    title(['T = ' num2str(t)]);

Results in the following regions:

labelled regions

As you can see, selecting the threshold that results in the highest number of regions (T=120) would already give 7 correct locations, some merged locations, one false positive and two false negatives.

This was a fairly simple attempt but I think it shows that the approach works. Adding some stuff to break up elongated regions, or doing this for each color channel separately are just a couple of the things you could do to improve on this.

It would also help if you provided a few more test images.


I had tried something else to improve my result in question. Below solution is on the assumption that first square(orange) is always detected in step 1. And it is practical due to its high contrast color compare to background. Even the result I showed in question has detected it correctly

Step 1 : Find as many squares possible

I split the image to R,G,B,H,S,V planes and thresholded the image for different threshold values like multiples of 25. For each image, I found squares in it, and put them on a "mask image". I also found average height and width of the square.

mask image (Total 7/12 squares detected) :

mask image

Step 2 : Form a grid of squares

Next I found the centroids of these squares in mask image. Sorted them and found the centroid of first square(orange). From close analysis, we can see the gap between two squares is a square in both horizontal and vertical direction. So this way, I made a grid of squares like below and called it ideal_squares (it is just a name, it doesn't mean this is the output I need):

ideal_squares :

ideal image

Step 3: Remap the ideal_image

Now we have the ideal_squares centroids and original centroids. I found out the correct matches for each original centroid from ideal_centroids (by taking euclidean distance between them). Then I used Scipy interpolate.griddata for interpolation and remapped ideal_image as per the centroid values (it is almost same as warping done in these Q&A : How to remove convexity defects in sudoku square & Image transformation in OpenCV). So below is the output I got :

Output :

Output Image

Step 4 : OR operate above output with mask image from first step

final output

Now you can see all the squares are detected, but with a problem mentioned below :

Problem :

Look at the output of the Step 3, ie remapped image of square grid. Except two central squares, all other squares are clipped. It is a problem associated with this remapping. I am not sure where is the problem, with scipy.interpolate.griddata() or cv2.remap(). I thought the whole image will be warped, but it is not. It warps only image inside the centroids we gave. If I can correct that, the output will be OK.

So if someone knows a good idea for that, most welcome !!!


Step 1: Whatever final binary image you are getting from analyzing in B,G,R,H,S,V plane, in that image do a blob counting algorithm.

Step 2: Find the largest blob on basis of area or contour length. Since your blobs will be mostly parallelogram types so area or contour, any one will do.

Step 3: With the largest blob (since largest blob is the best blob resembling your real world squares) try to find the orientation of the blob...this you can get by a fitting a best fit rectangle OR you can get the corner points...get the slope of the lines joining them (in both horizon and vertical direction).

Step 4: Once you get the two slopes draw two lines running through the axis of the blob. for axis you can average the corner points or you can use the centroid (center of mass)...I would go with average of corner points...

Step 5: Since in each horizontal and vertical direction, spacing is equal (ideally horizontal and vertical spacing are also equal as it comes from your ideal square picture but we will not assume it..) just need to locate the possible centroids of the other parallelograms

BOTTOM LINE: If any one square gets detected perfectly you can make the whole grid. Just keep marking centers at an interval of 2H (H = horizontal width of biggest blob) along the horizontal axis of the biggest blob and at an interval of 2V (V = vertical height of biggest blob) vertically along the vertical axis of the blob.

Some pics to support enter image description here

enter image description here

  • 1
    $\begingroup$ +1 - It would be great if you could implement it. $\endgroup$ Jan 15, 2013 at 19:29
  • 3
    $\begingroup$ @AbidRahmanK That's not what StackExchange is for. Question -> Answer. Otherwise this would end up being a job fair. $\endgroup$ Jul 4, 2016 at 7:28

Note: This method is going to be really slow.

Generate a mask that looks like the contours of a ideal object. Similar to this:

mask of the object

then slide (position,scale,rotation) the mask over the image and match it with the contour of the real image (perhaps blurred a bit to get softer response) to calculate how similar they are, the (position,scale,rotation) with the highest similarity response should be the (position,scale,rotation) of the real object.

The method doesn't mind squares blending into the background or even partial occlusions of the object, since it considers the entire object.

I have personally used this method successfully to track a mouse snout and whiskers, but I had some presumptions like it was close to the last known position etc. But I think you can bring down the search-space by applying some assumptions like: possible sizes of the object in the camera, how far of from the center it can be or rotation <10degrees etc.


this arrangement is fixed

I don't really know what kind of prediction you did before, but have you tried to focus on the white long strips as a root. Then (if 3 columns of squares are equal-sized), you can detect the height of a square (distance between the two strips) and you can detect the max and min area (height and width) in the image.

Then, try to detect the most common colour inside your whole square and set it to a "non-square" area. The rest is supposed to be the squares you are looking for.

  • $\begingroup$ I found 3-4 squares using contour method. Then got height and width of each square. Then checked for gap between detected squares and assumed if gap between them is large enough to hold another square. That is the prediction i made. $\endgroup$ Oct 4, 2012 at 10:05
  • $\begingroup$ Some squares have almost similar color as that of background. So i am afraid, they also will be considered as non-square area as per your method. $\endgroup$ Oct 4, 2012 at 10:06
  • $\begingroup$ Maybe you could try work on every columns to plot a curve where x axis would be the height (in pixel) in the image and the y axis would be the intensity. Then, you could try to find some cutting edges with the derivative form. $\endgroup$
    – Antonin Duroy
    Oct 4, 2012 at 10:25
  • $\begingroup$ that is edge detection itself, right? I tried it,but didn't get good result. $\endgroup$ Oct 4, 2012 at 10:27
  • 1
    $\begingroup$ Yes it is, but you could see by yourself why it fails and maybe isolate some interesting regions in the plot. By the way, if you find some helpful tips to solve your problem, please post them. Good luck for your researches $\endgroup$
    – Antonin Duroy
    Oct 4, 2012 at 11:36

I would suggest using the Hough transform, which is a very robust algorithm for finding simple parametric shapes e.g. lines, circles etc. Detection of lines would be best in your case. You could find the sides of the long white sripes at least; then, with any corner extractor algorithm (Harris or maybe even SIFT or SURF) you could find corners along those lines, even using the fact that the squares are approximately equally spaced.


I tried this problem using opencv, python. Approach involves masking image based on the colors, followed by finding suitable contours.

Code: enter image description herehttps://github.com/rbhambriiit/computer_vision/blob/master/find_color_box

[ Missed 1 box but that should come by tuning the masking function]


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.