0
$\begingroup$

Usually, algorithms as SIFT, SURF and many others provides a set of $k$ keypoints and the associated descriptor in $d$ dimensions (for example, in SIFT each descriptor has $d$=128 dimensions).

So, in order to describe an image we need a matrix $k \times d$ ($k$ descriptor vectors, each one in $d$ dimensions). So far so good.

My question is: how can we describe an image through a single vector?

This could be really useful since we could save a lot of space and because certain algorithms (like LSH) requires a vector as input/query.

In some papers (for example this, section 6.5) this approach is described as "global descriptors".

Up to know, I found only this paper but it doesn't seem so accurate (and it's from 2009, not so new).

APPLICATION:

I'm trying to implement a memoization optimization to the following application: given an image (taken with a smartphone) containing a movie poster, return the internet movie database's score of that movie.

The optimization (so what I'm trying to implement) is about "remembering" (memoize) the input image and the relative result (IMD score) and for each new input image, search "the most similar one" and return the already computed result (avoid the database query) if a certain threshold is reached.

Two "similar images" in this context are two images pictures containing the same poster but from different prospective and/or places.

The actual approach is solving an approximate nearest neighbor through LSH in order to find the most similar image, but in order to do that I need to represent a query (an image) through a vector, and not through a matrix.

$\endgroup$
2
$\begingroup$

There are lots of different ways. You could simply put all the image pixel values into a vector with dimension = width x height. :) What you really want to know is how can we describe an image using a lower-dimensional representation and still caption all the important information.

The same approaches that describe an image patch, e.g. SIFT, apply. Just think of the entire image as the patch. However, the method very much depends on the information in the image and the purpose of the vector. For example, you could use some histograms of the gradients, subsample the frequency spectrum, take a certain number of wavelet coefficients, perform edge detection, use moments of the image, divide it into super-pixels and so on. You might need the representation to have different invariances such brightness, rotation, contrast, phase or scale. Basically any method that gives you a condensed version of the information in an image can be put into a vector.

Research into the right method to use for your particular application is probably in the literature somewhere. As for saving a lot of space, this is what compression is all about. e.g. JPEG discards the smallest coefficients of the DCT of blocks in the image to reduce space.

$\endgroup$
  • $\begingroup$ Thanks to your answer. While you were posting, I was reading this interesting paper where global descriptors are compared with locals ones. The SIFT on the whole image is cited also with many others (especially GIST is designed from the beginning for to be computed on the whole image). $\endgroup$ – user6321 May 26 '16 at 10:49
  • $\begingroup$ I have a related question: dataset that usually use SIFT descriptors (like TexMex, INRIA do they use global SIFT descriptors or local ones? $\endgroup$ – user6321 May 26 '16 at 10:55
  • $\begingroup$ I think the main difference between a global descriptor and a set of local descriptors is translation and rotation. For example, if you used global SIFT on a image of a car the descriptor will depend on where in the image the car is, whereas with bag of words you might get some strong local features like "wheel" that are invariant to position (you just know they are there). So this could be good or bad depending on your application. So keep that in mind - what should your descriptor be invariant to. $\endgroup$ – geometrikal May 26 '16 at 11:02
  • $\begingroup$ @user6321 What is the application? $\endgroup$ – geometrikal May 26 '16 at 11:03
  • $\begingroup$ Answer updated with APPLICATION $\endgroup$ – user6321 May 26 '16 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.