I want to compare the performance of SIFT and SURF against some standard image set.
After a bit of research, I found that ROC curves are one of the best ways to do this task.
In order to plot ROC curve TP rate and FP rate are needed.
Output of SIFT and SURF are detected keypoints. How exactly can I generate ROC curves from those keypoints ?
The ROC: ROC curves are popularly used as performance metric for classification tasks. If the images in your dataset has class labels, then you can employ supervised learning to train a classifier (SVM for example). The dataset is split into training and testing and predicted class score from the classifier for images in the test set are compared to ground truth labels. The ROC curve is acquired by applying a threshold value to the classifier predicted score and obtaining a (TP,FP) value for each threshold to generate the curve.
The features: SIFT and SURF are detectors and image local-patch descriptors (you can also use these without the detector in what is now more successful approach called 'dense sampling' where the keypoints are uniformly sampled in the image).
There are several approaches, but the most popular and recommended for first use is the 'Bag-of-Words' (BoW) model based approach. The reason is that: each image has one label (ignoring multi-class and multi-label scenario for now) and several hundred keypoints and each keypoint has an associated high-dimensional feature vector. The feature for an image at this stage is a matrix and we would like a vector (a point in feature space for each image with its label). The BoW approach does just this job for us.
BoW approach: All the feature vectors for every image in the training set are collated and clustered (ignoring labels and location data) to compute a 'dictionary', using unsupervised learning. The dictionary is a set of cluster centroids, most commonly computed using k-means clustering. The dictionary follows data compression scheme to succinctly represent the data distribution.
Subsequently, each image is 'encoded' using its keypoints and the computed dictionary. Using vector quantization VQ, each keypoint is associated with a dictionary element to compute a occurrence histogram of keypoint associations. The consequence is that each image now has a feature vector which is the normalized occurrence histogram of keypoints assigned to dictionary elements. The dimensionality of this encoded feature vector is the size of the dictionary.
This normalized histogram for each image along with the label is used to training a classifier and then acquire classification results measured in terms of ROC curve. The sequence is:
[Images] -> (SIFT/SURF) -> [SIFT/SURF feature matrix] -> (collate and cluster) -> [dictionary] -> (BoW histogram by VQ encoding) -> [feature vector, label] -> Classifier training (training data) -> Classifier testing (validation/test data) -> ROC curve.
Suggestions: Besides ROC, there are other performance measures available like accuracy, mean average precision, F1 score, etc. Your choice depends on the imageset, label ratio balance, objective of the classifier, etc.
There is plenty of code available online for baseline Bag-of-words model for images. One sample in Matlab using the Caltech101 dataset is at: http://www.vlfeat.org/applications/apps.html#apps.caltech-101
The performance of SIFT and SURF is also provided in the original paper on SURF, so you can compare your results if you choose to work with the same dataset.
