I am currently doing a research project into the effectiveness of different keypoint detector and feature descriptor combinations for the task of vehicle detection.
At the moment I am using a SVM for simple binary classification of vehicles testing both the bag-of-features model and pyramid match kernels for mapping the set of vectors detected to a single feature space.
I only have a limited dataset of around 5 minutes of video captured from a UAV, of which I have manually extracted the vehicle images and rotated them so that the cars direction is horizontal. Currently I am evaluating the performance of the different detector and descriptor combos using the F-measure (F1 score). However with the aim to find a combo that are invariant to image transformations, scale, rotation, illumination/lighting variations, colour and slight affine transformations as to generalize well and be sufficiently unique to maximise the accuracy of classification, should I also be evaluating the F-measure in different conditions such as scale changes, rotation, perspective rotation, camera yaw, motion blur, and varying static and dynamic lighting conditions.
However with a limited dataset I am finding it difficult to to understand how I should go about this. For example would it be appropriate to rotate my test set by varying degrees and then use the evaluated f-measures to compare the feature detector/descriptor combos accuracy under different rotation?
Any recommendations of how I should approach the evaluation/comparison of the keypoint detector and feature descriptors under these situations would be appreciated.