3
$\begingroup$

I am using opencv for finding template images in a video stream. The elements I am trying to find are UI elements of android apps.

Classic template matching is working quite well. But only as long as the scene and the template share the same resolution. My requirement is to have this working for different scene resolutions (different devices).

What I tried so far is:

  1. Rescaling the template at different resolutions in a loop and checking. As soon as my result increases above a certain threshold, I consider it a match. Problem: Not very robust, extremely slow
  2. AKAZE and ORB: Don't really provide the expected results. I don't know if I am missing something, but it doesn't look like those algorithms are made for what I am trying to do. I am getting results like this:

Orb

Any help or ideas are highly appreciated!

A couple of examples:

Scene: scene Template: template

Scene2: scene2 Template2: template2

Scene3: scene3 Template3: template3

$\endgroup$
4
+25
$\begingroup$

If your templates are all based on some kind of text you may use some kind of OCR to match the text itself and not only by features.

Regarding features, you may read: A Comparative Analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK.
Specifically have aloo at the sections:

enter image description here

It seems your feature extractor usually use corners while you need more general purpose features.

enter image description here

enter image description here

It seems SURF and SIFT will be better suit for your case. Since your search should be not care about rotations (At least according to your examples) what you can do is a template matching over the features and not the image. Namely you are after a scaling of the feature location of your template in the image.

$\endgroup$
3
  • $\begingroup$ How can template matching be applied to features? Can you point me to some tutorial or demo implementation? Or some openCV api? $\endgroup$
    – stoefln
    Jun 29 at 21:11
  • $\begingroup$ @stoefln, What I meant is looking at the geometry of the features on the template and search for a constrained match on the image. Since you images are not rotated it should be easier to do. $\endgroup$
    – Royi
    Jun 30 at 3:21
  • $\begingroup$ thanks for trying to help. Could you elaborate on how that could be done? I only have basic knowledge of openCV. $\endgroup$
    – stoefln
    Jun 30 at 12:12
1
$\begingroup$

I believe Haar Cascades(used by Viola-Jones) are inherently scale-invariant. Also severely deprecated by modern Neural Networks, but I know nothing about those. It also doesn't do any OCR - if you need that you would need to run a separate algorithm on the extracted sub-image.

$\endgroup$
0
$\begingroup$

You may want to try something initially supervised.

For example, https://docs.opencv.org/3.4/d1/d2d/classcv_1_1ml_1_1SVM.html

would work if you have data to train it against.

$\endgroup$
-1
$\begingroup$

Why not try to program it by percentage from the beginning? Like this, it's scalable in any resolution. Loop and checking would only generate more work and consume more resources. If there is a way to display the elements by percentage, you get the result in any resolution.

$\endgroup$
2
  • $\begingroup$ I don't understand. Can you elaborate? $\endgroup$
    – stoefln
    Jun 29 at 11:48
  • $\begingroup$ Can you add further information and maybe link to this technique? $\endgroup$
    – tobassist
    Jul 1 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.