# How can I detect buttons in a GUI of an app with pattern detection using python?

I will explain taking as example the GUI of the Calculator app in windows. Given this Calculator application open and focused, I need to find a way of detecting all the buttons.I can only use non-intrusive methods, so things like button id are out of the question. This leads me to image recognition. Or better say image detection, because I don't want a specific image, but a set of images that follow a certain pattern. I know how I can click/right-click/dbl-click/etc using a given button image and image recognition [1]. I don't know how I can do the other way round: scan the GUI and find those areas that meet the requirements of being a button (rectangle, text/icon/graphic labeled etc.). A bigger tackle would be to detect items that don't have the rectangle shape (e.g. icons on a windows desktop)

The nearest thing to what I need is detecting faces in a picture.[2] But I don't know how to apply this in my case. For human face detection, I saw that hundreds of pictures of the face must be used in the Haar cascade generation (don't know how I would do this with only 10-15 button snapshots). If another kind of object has to be detected like an apple, you need to generate Haar cascade for that object again using a lot of images.

Did any of you ever tried to detect buttons, items or whatever in a GUI using only pattern detection? I just need something to tell me "this is an icon/button" so I can grab that region in a snapshot.

[1] I use SikuliX with Python to perform actions on given patterns.

[2] I saw that this is done easily using OpenCV and Haar cascades (in XML format). Creating the Haar cacades requires a little bit of patience and skills.

## migrated from stackoverflow.comFeb 16 '12 at 0:51

This question came from our site for professional and enthusiast programmers.

First, have a look at the squares.py sample provided by OpenCV. It should handle a fair number of button types with some tweaking.

Here is the output I got (with some tweaking) for your Calculator example:

I made the following tweaks to the squares application:

Change this code (starting on line 84):

if(result.total == 4 and
abs(cv.ContourArea(result)) > 1000 and
cv.CheckContourConvexity(result)):


To this:

if(result.total == 4 and
abs(cv.ContourArea(result)) < 1300 and
abs(cv.ContourArea(result)) > 300 and
cv.CheckContourConvexity(result)):


Also, since you don't have to worry about scale or rotation variations, check out using matchTemplate. Also, its corresponding tutorial (in C++) is here.

See my other answer for another example of how template matching works. Also, you might find this answer helpful for detecting the top X matches with matchTemplate.

Hope that helps!

• Thanks a lot for your answer. I'll try your idea although I'm looking for something that can be extended outside this calculator area. This approach would be just great for square buttons. Don't know how it would behave for detecting icons on a windows desktop (as I mentioned in the explanation). But I'll try it and will let everyone know. Thanks again – Radu Enea Feb 15 '12 at 8:13
• How flexible does your program have to be? The more general the problem you try to solve; the more difficult the solution will be to achieve. – mevatron Feb 15 '12 at 14:35
• Well, for the moment it's ok if I stick to the squares. I have bigger problems. Like this one (when I run squares.py; Any idea?): Traceback (most recent call last): File "squares.py", line 144, in <module> on_trackbar(0) File "squares.py", line 126, in on_trackbar drawSquares(img, findSquares4(img, storage)) File "squares.py", line 30, in findSquares4 pyr = cv.CreateImage(sz.width/2, sz.height/2, 8, 3) AttributeError: 'tuple' object has no attribute 'width' – Radu Enea Mar 5 '12 at 16:18
• Solved the issue above. And after a lot of research, I think I have to rephrase the question. – Radu Enea Mar 7 '12 at 16:40

After rephrasing my question I got the answer that I was looking for. That's another example that you have to know what to ask for in order to receive what you need. Here's the page: How can I detect elements of GUI using opencv? Thanks for the help!