9
$\begingroup$

Given a screenshot of an application, is there any way of finding GUI elements of that application using only opencv? (something like in the image) Desired Outlook image after element detection

I tried to play with blending the image and other effects from the coherence.py sample in the opencv installation folder. Did not succeed to get a viable result.

Has anyone did this? Do you have any idea what I might try?

Thanks

(I'm new to opencv and image processing so please don't avoid obvious explanations if possible) :)

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ This is not in OpenCV, but have a look at Sikuli.org which has exactly the purpose you're looking for $\endgroup$
    – Ivo Flipse
    Commented Mar 8, 2012 at 13:41
  • $\begingroup$ i know about Sikuli. What i needed was a way to detect the areas of interest without actually taking a screen-shot of them in advance. The answer was the MSER approach with specific initialization. Anyway, thanks for the Sikuli tip. $\endgroup$
    – Radu Enea
    Commented Mar 9, 2012 at 14:17
  • $\begingroup$ If I may ask what were you programming? I got here searching for the same question. My intention is to make elements keyboard-clickable when for applications that don't have keyboard shortcuts defined. Did you worked on something similar? $\endgroup$
    – urza.cc
    Commented Mar 15, 2017 at 7:20
  • $\begingroup$ No, it was for a small project of UI test automation. It was supposed to "watch" the user interact with the UI and then generate small scripts to replay the actions. I abandoned it at some point due to Sikuli's lack of reliability on long term. I could get screenshots of the elements in UI, but Sikuli did not manage to find all of them in 100% of cases (probably they were too small or too similar to other elements) $\endgroup$
    – Radu Enea
    Commented Apr 3, 2017 at 10:30

1 Answer 1

Reset to default
10
$\begingroup$

OpenCV's MSER extractor (documented here) might be helpful — the bounding box of local MSER groups would pretty closely match the green rectangles in your mockup.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. Looks like what I'm looking for. I'll give it a try one of these days $\endgroup$
    – Radu Enea
    Commented Mar 8, 2012 at 9:06
  • $\begingroup$ Yap. I did it. I just have to fine tune it now for what I need. Thanks again. $\endgroup$
    – Radu Enea
    Commented Mar 9, 2012 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.