9
$\begingroup$

I will explain taking as example the GUI of the Calculator app in windows. Given this Calculator application open and focused, I need to find a way of detecting all the buttons.I can only use non-intrusive methods, so things like button id are out of the question. This leads me to image recognition. Or better say image detection, because I don't want a specific image, but a set of images that follow a certain pattern. I know how I can click/right-click/dbl-click/etc using a given button image and image recognition [1]. I don't know how I can do the other way round: scan the GUI and find those areas that meet the requirements of being a button (rectangle, text/icon/graphic labeled etc.). A bigger tackle would be to detect items that don't have the rectangle shape (e.g. icons on a windows desktop)

The nearest thing to what I need is detecting faces in a picture.[2] But I don't know how to apply this in my case. For human face detection, I saw that hundreds of pictures of the face must be used in the Haar cascade generation (don't know how I would do this with only 10-15 button snapshots). If another kind of object has to be detected like an apple, you need to generate Haar cascade for that object again using a lot of images.

Did any of you ever tried to detect buttons, items or whatever in a GUI using only pattern detection? I just need something to tell me "this is an icon/button" so I can grab that region in a snapshot.

[1] I use SikuliX with Python to perform actions on given patterns.

[2] I saw that this is done easily using OpenCV and Haar cascades (in XML format). Creating the Haar cacades requires a little bit of patience and skills.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

First, have a look at the squares.py sample provided by OpenCV. It should handle a fair number of button types with some tweaking.

Here is the output I got (with some tweaking) for your Calculator example: enter image description here

I made the following tweaks to the squares application:

Change this code (starting on line 84):

if(result.total == 4 and 
   abs(cv.ContourArea(result)) > 1000 and 
   cv.CheckContourConvexity(result)):

To this:

if(result.total == 4 and 
   abs(cv.ContourArea(result)) < 1300 and 
   abs(cv.ContourArea(result)) > 300 and
   cv.CheckContourConvexity(result)):

Also, since you don't have to worry about scale or rotation variations, check out using matchTemplate. Also, its corresponding tutorial (in C++) is here.

See my other answer for another example of how template matching works. Also, you might find this answer helpful for detecting the top X matches with matchTemplate.

Hope that helps!

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks a lot for your answer. I'll try your idea although I'm looking for something that can be extended outside this calculator area. This approach would be just great for square buttons. Don't know how it would behave for detecting icons on a windows desktop (as I mentioned in the explanation). But I'll try it and will let everyone know. Thanks again $\endgroup$
    – Radu Enea
    Commented Feb 15, 2012 at 8:13
  • $\begingroup$ How flexible does your program have to be? The more general the problem you try to solve; the more difficult the solution will be to achieve. $\endgroup$
    – mevatron
    Commented Feb 15, 2012 at 14:35
  • $\begingroup$ Well, for the moment it's ok if I stick to the squares. I have bigger problems. Like this one (when I run squares.py; Any idea?): Traceback (most recent call last): File "squares.py", line 144, in <module> on_trackbar(0) File "squares.py", line 126, in on_trackbar drawSquares(img, findSquares4(img, storage)) File "squares.py", line 30, in findSquares4 pyr = cv.CreateImage(sz.width/2, sz.height/2, 8, 3) AttributeError: 'tuple' object has no attribute 'width' $\endgroup$
    – Radu Enea
    Commented Mar 5, 2012 at 16:18
  • $\begingroup$ Solved the issue above. And after a lot of research, I think I have to rephrase the question. $\endgroup$
    – Radu Enea
    Commented Mar 7, 2012 at 16:40
0
$\begingroup$

After rephrasing my question I got the answer that I was looking for. That's another example that you have to know what to ask for in order to receive what you need. Here's the page: How can I detect elements of GUI using opencv? Thanks for the help!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.