3
$\begingroup$

I am truly new to computer vision field, but it is fascinating me! I have now a challenge in my hands and I am seeking for mentors/advisers to give me some follow up.

My project is: From a picture of a video-game cover, search that pic through a video-game cover database, and if there is a very good match the app will return a string with the name of the videogame and the platform.

Problem example:

  1. Take a photo of a cover similar to this:

query image

  1. Cover match to this in the database:

database image

  1. App gives string: "Fifa 12 Playstation 2"

In my preliminary research I found that I should save in my cover database, the name of the game, the platform, the URL for the cover, and the image features (keypoints and descriptors).

I am using SURF features detector/extractor.

There are some concepts that I am still confused about... I am not looking for similarity right? I just need to observe if there are some sort of good keypoints-paired/matched, right? Because in the examples images mentioned before I get "img1 - 1087 features (query image), img2 - 1755 features - 30 % - SIMILARITY - 321/333 inliers/matched" What are the inliers? My calculation of similarity seems wrong to me... I would say that these two images are like 70 % look alike...

ps: I am using Python so...

Results so far:

enter image description here

Thanks for your time and help. Sorry If I could not explain any better my problems/concerns.

$\endgroup$
2
$\begingroup$

I am no expert, but I would like to give some pointers which could help you to solve.

  • Regarding the method you have mentioned/used, is for image registering/stitching. You are interpreting the results wrong. Calculating the percentage of feature match is not the intention of inliers, rather computing corresponding feature points in both images which could be used for image registering/stitching. Inlier is the feature point for which the algorithm has found a corresponding feature point in the other image. Feature points for which the algorithm has not founda match are called outliers. More inliers, better the match. Keep in mind, the algorithm can sometimes match some feature points of completely unrelated images(try running your method on two unrelated images)

  • Can you use the method you have mentioned to solve your problem ?. Yes,You can use number of inliers for each image in the database and decide the one that has the highest number of inliers as the match. You can have a minimum count of inliers to be present, in case the picture of a video-game doesnt exist in your database.

  • Other ways/methods which you could explore to solve this.

    • Explore more on the features you can use. SURF does well for some applications, for some it doesnt work well. You have to choose based on your data (image database). You can check with SIFT, and also color histogram as features.

    • You can check bag of words for image search. (check this)

    • Whenever you add a new game to database, some algorithm(eg - bag of words) need the code book to be recomputed everytime you add a new image to database, so choose the algorithm wisely , coz as the image database gets bigger, the cost of computing increases.

    • Also think in lines where scale of the images differ(i.e image is database and search query image) Some features like SURF/SIFT are scale invariant, if you choose to use any other, keep this in mind.

These points might help you in deciding and I hope I did not complicate your life :)

$\endgroup$
  • $\begingroup$ thanks for your insights :) "Can you use the method you have mentioned to solve your problem ?. Yes,You can use number of inliers for each image in the database and decide the one that has the highest number of inliers as the match. You can have a minimum count of inliers to be present, in case the picture of a video-game doesnt exist in your database. " I am strugling in find that minimum and above what threshold I should consider a image to be a very good match candidate $\endgroup$ – Inês Martins Nov 5 '15 at 9:40
  • $\begingroup$ Yes, it is not a easy task to set thresholds. You can always experiment and set it(not foolproof). You can start doing cross image matching, that is, match an image in database as query image against all the other images in database. This would give you an idea. But the actual usage has other chalenges like scaling , rotation (which SURF can take care of) but not a case where the query image is a cropped version of that in database. Incase you are sure the user always captures the name of the game, you could add the character recognition along the image matching to make the system robust. $\endgroup$ – harshkn Nov 5 '15 at 20:04
  • $\begingroup$ Check chapter 5 in this document(cs.gmu.edu/~kosecka/cs482/…). $\endgroup$ – harshkn Nov 5 '15 at 20:16
  • $\begingroup$ I am already doing what you said, I am matching the query image against all the other images in database and storing in an array the "best" matches and sort by confidence... but this takes to long... How can I speed up the searching? Sorry if it is a stupid question but I do not know much about the best search algorithms $\endgroup$ – Inês Martins Nov 6 '15 at 12:42
  • $\begingroup$ Your methos uses RANSAC or variants of it, it is an iterative algo. It takes time. If the computation time is too large, then Im afraid i have no ideas to improve the speed. You better give a bag of words a try, if you have time. You can look up the Wiki and this code base can help you. github.com/shackenberg/… $\endgroup$ – harshkn Nov 6 '15 at 20:21
0
$\begingroup$

You can try and use multiples approaches to solve the problem. Initially, as preprocessing, you may try to improve the contrast of the image you captured so that the SURF matching performance can be improved. Along with the SURF points which you have used, you can try to determine the text written in the image you captured. You can find some resources here and here.

If you can determine the most significant parts of the text - FIFA 12 and PlayStation - from the image above, if used alongside the result of the SURF comparisions, I think you can successfully complete your project. Let me know whether this works. :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.