Qualitative Evaluation of Interest-Point-Detectors and Descriptors in Computer Vision

Question

I'm writing on a qualitative evaluation study of interest-point-detectors and descriptors. I've read all Mikolajczyk papers as well as most of the surves from Datta et al. etc. Now I'm implementing an evaluation-tool in OpenCV. I'll take to images. One referred as source- and one as comparison-image.

1.) Detectors: Mikolajczyk uses the repeatability-criterion, correspondence count, matching score and an other metric to evaluate the performance of the detector. I would use repeatability and a correspondence count for matched regions.

2.) Descriptors: Here I would use the widely used Recall and Precision on the matched regions to describe the performance of the descriptor.

My question so far: Are these both good metrics for evaluation?

Now I'm trying to implement this is OpenCV and need a good eye from somebody to tell me if this could do. Here I use SURF-Detector and -Descriptor for testing the metrics. Recall and Precision are not implemented yet, but there is a function calculating this.

#include <opencv.hpp>
#include "opencv2/core/core.hpp"
#include "opencv2/features2d/features2d.hpp"
#include "opencv2/nonfree/features2d.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <stdio.h>
#include <stdlib.h>
#include <iostream>

int startSURF() {

    std::cout << "Starting " << std::endl;
    cv::Mat sourceImage = cv::imread("collection_1/DSCN5205.JPG",
            CV_LOAD_IMAGE_COLOR);
    cv::Mat comparisonImage = cv::imread("collection_1/DSCN5207.JPG",
            CV_LOAD_IMAGE_COLOR);

    if (!sourceImage.data) {
        std::cout << "Source-Image empty" << std::endl;
        return -1;
    } else if (!comparisonImage.data) {
        std::cout << "Comparison-Image empty" << std::endl;
        return -1;
    }

    //Detect keypoint with SURF
    int minHessian = 400;
    cv::Mat sourceMatchedImage, comparisonMatchedImage;
    std::vector<cv::KeyPoint> sourceKeypoints, comparisonKeypoints;

    cv::SurfFeatureDetector surfDetect(minHessian);
    surfDetect.detect(sourceImage, sourceKeypoints);
    surfDetect.detect(comparisonImage, comparisonKeypoints);

    //Calculate the SURF-Descriptor
    cv::SurfDescriptorExtractor surfExtractor;
    surfExtractor.compute(sourceImage, sourceKeypoints, sourceMatchedImage);
    surfExtractor.compute(comparisonImage, comparisonKeypoints,
            comparisonMatchedImage);

    //Flann-Matching   
    cv::FlannBasedMatcher flann;
    std::vector<cv::DMatch> matches;
    flann.match(sourceMatchedImage, comparisonMatchedImage, matches);

    //Repeatability and Correspondence-Counter    
    float repeatability;
    int corrCounter;
    cv::Mat h12;

    std::vector<cv::Point2f> srcK;
    std::vector<cv::Point2f> refK;

    for (int i = 0; i < matches.size(); i++) {
        srcK.push_back(sourceKeypoints[matches[i].queryIdx].pt);
        refK.push_back(comparisonKeypoints[matches[i].queryIdx].pt);
    }

    std::cout << "< Computing homography via RANSAC. Treshold-default is 3" << std::endl;
    h12 = cv::findHomography( srcK,refK, CV_RANSAC, 1 );

    cv::evaluateFeatureDetector(sourceImage, comparisonImage, h12,
            &sourceKeypoints, &comparisonKeypoints, repeatability, corrCounter);

    std::cout << "repeatability = " << repeatability << std::endl;
    std::cout << "correspCount = " << corrCounter << std::endl;
    std::cout << ">" << std::endl;

    std::cout << "Done. " << std::endl;
    return 0;    
}

I'm uncertain if this code works because SURF gets bad repeatability (e.g. 0.00471577) for my testing images with a rotation of almost 45°. Does anybody see a problem with the code?

Is there a way to evaluate the detector without RANSAC? I did not find a yet implemented method for this. Is the default of 3 a good threshold? I could overwrite it but the problem is that a good threshold can only be determined by experimental results. But I need a robust default-value for all detectors.

I think I definitively need the homography. But I never found a way to set the localisation-error ε for the repeatability-measure. Is there a way to set ε?

A lot of questions. I hope I was specific enough but if there is a question please comment and I will do my best to answer in order get help here. I wish you merry Merry Christmas!

[EDIT] Additional Readings: http://www.robots.ox.ac.uk/~vgg/research/affine/

Answer 1

Actually, the more a read your question, the less it makes sense. What I'm trying to answer here are not actually your questions, I'm rather trying to address the problems in your reasoning. They're too long for the comment, and they might set you on the right path.

---

Let's start with detector evaluation and referring to the Mikolajczyk paper.

I would use repeatability and a correspondence count for matched regions.

I would suggest re-reading the Mikolajczyk paper. This statement does not make much sense in the context of the paper.

The repeatability and the correspondence count were designed to be evaluated without involving the descriptors, that is, without performing the matching.

When calculating the repeatability and correspondence number, Mikolajczyk proposes likely matches based on image geometry and feature geometry (e.g. exclusively the feature position and size in the image).

These same measures, but performed on actual matches instead of geometrical estimates produce different measures: the matching score and the nb of correct matches.

These two pairs of measures were both developed for evaluating the quality of the detector (but not the descriptive power of the descriptor). While the first two measures (repeatability and correspondence number) are completely decoupled from the descriptor (e.g. the matching process), that makes them somewhat less reliable. Thus, the other two measures (matching score and number of correct matches) do all detector comparisons using the same descriptor (SIFT), so the results are more reliable while somewhat dependent of the descriptor choice.

---

I'll take two images. One referred as source- and one as comparison-image.
_{^{(I took the liberty of correcting "to images" in "two images". Please correct if wrong)}}

Well, I hope here you mean that you will be using this as a sub-routine to evaluate a dataset. Getting a comparison score between just one pair of images is not very descriptive.

If you are trying to interpret the score for just one pair outside of context, you can not determine what is causing the (both good and bad) results, while it could be any and all of below:

• the type of scene
• any single one of the transformations between images
• some combinations of the transformations
• poor image conditions (e.g. lighting, contrast, camera quality)

That is why Miolajczyk has datasets. Each dataset is dedicated to a single image transformation or varying a single image condition. Each dataset is either "structured" or "textured" (different "types" of scene) -- and contains one dominant planar surface.

The reference image in each dataset is a good quality, frontal image. The second one for each dataset is a small change of the desired type (and represents how the detector reacts to small transformation on this type of image). Further images in the dataset represent more severe transformations of the same type, and are used to examine the detectors invariance under that kind of transformation.

The point of this is that in order to make a good estimate of detector quality, you need to examine the trends this detector exhibits while changing different conditions, one condition at a time.

---

You say two related things:

Is there a way to evaluate the detector without RANSAC?

and then:

I think I definitively need the homography. But I never found a way to set the localisation-error ε for the repeatability-measure. Is there a way to set ε?

Let's start with saying that yes, you definitely need the homography to calculate the repeatability. Note when I said that the repeatability and correspondence count rely on the image geometry -- and this can only be calculated using the homography.

Furthermore, even though I said that the "matching score is performed on actual matches instead of geometrical estimates" -- this is true in principle, but Mikolajczyk still uses some geometrical information to obtain a more relevant result. Thus, yes, you need homography.

For the second part, yes, RANSAC is the best way to estimate a homography between two images of the (same) planar surface. But, that said you should not use only RANSAC estimate directly each time you perform detector evaluation between a pair of images.

What I mean by this is that you should have pre-computed homograpies for every image pair that you plan to use in the testing. Two ways to get pre-computed homographies are:

• Pre-compute them yourself. In that case, you will most probably use RANSAC in this step. (you have some guidelines on this in the Mikolajczyk paper)
• Find datasets with pre-computed correct homographies.

---

Finally, what all my partial answers were leading to is a response to what you're planning to do:

I've read all Mikolajczyk papers (...). Now I'm implementing an evaluation-tool in OpenCV.

Well... why are you re-inventing the wheel? Mikolajczyk produced a testing framework, which is still available online.

This framework includes:

• datasets with homographies provided
• good description of the datasets (e.g. what does each dataset evaluate)
• binaries that do all the "dirty work" from Mikolajczyk: all the region normalization, region rescaling and similar things mentioned in the Mikolajczyk paper. If you read the paper, you should know why they are important.
• a fully implemented function in MATLAB that will calculate the repeatability score and the number of correspondance for any whole dataset (and with minimal modification, will also calculate the matching score and number of correct matches).

In addition to only using the provided MATLAB code, in order to not re-invent the wheel but if you want to avoid MATLAB, you can still:

• tweak the MATLAB code to produce numeric instead of graph output, and then further analyse the numeric output in your language of choice
• use the MATLAB code provided as help and guide in making your own testing environment (since you want to produce the same / similar functionality)

---

And to recap,

My question so far: Are these both good metrics for evaluation?

Well, you were the one that did the research. You should be able to tell us.

From my own personal research, I would say that the Mikolajczyk framework is a good framework for evaluating feature detectors. It was published in 2006 but you can still see recent papers using this framework. It has been accepted by the community as the go-to evaluation framework, and that with most of the community (if not all) using the original code.

Now, please don't get offended by this, but if you are ultimately asking this question, it seems to me like you did not understand the paper(s) you read quite well. I only commented on the "detector evaluation" part in my answer, but I hope it made you realize you still have to put more effort in to understanding the framework you want to work with.

My suggestion is to go back to reading the framework papers for both detector and descriptor evaluation. Possibly ask specific questions about what you do not understand (I'll be glad to answer any new questions about the Mikolajczyk framework).

Now, I understand I did not talk much about "detector evaluation", but I could do some digging and reminding and write something about that if you want as well.

Answer 2

"Are these both good metrics for evaluation?"

It all depends what it is you want to evaluate, ie what you want to use your detectors/descriptors for, and what kind of images you have (with repeated patterns? very similar but distinct objects? inside/outside?).

If you want precise correspondence between 2 images of the exact same things, those are good metrics (that's the kind of images used by Mikolajczyk for the tests). These is useful, for instance, for image stitching & 3D reconstruction.