Database into training and testing images

Question

I am trying to compute the accuracy of the HMAX model. I am using the Face category (containing 435 images) from the Caltech101 database. I split it into x training and y testing. At each time, when x increases, the accuracy also increases. Furthermore, I heard that the number of training should be equal to 80% by comparing it to the tests. So when I split my data into 348 positive training and the rest for positive testing, I got an accuracy that it is smaller than the other smaller splits (when x<348)!!.

Even when I took x=300, I also got a smaller accuracy!!

By the way, I also used the background category and I split it into 50 negative training and 50 negative testing.

Please why I got a smaller accuracy? Please I need your help.

Answer 1

Some of the possible reasons would be:

1. Object recognition accuracy is not really well-defined. Instead accuracies are for particular datasets.
2. What you are working with is called cross-validation, where you split your own dataset into testing and training datasets and then use the trained model to make predictions on the test dataset.
3. Depending on how you do the cross-validation and the final model, the accuracy tests may vary. For example, if (even by a random process) your training set contains well-conforming images and the test dataset contains images which are not even closely conforming to your model, then the accuracy would be low.

As for a solution to your dilemma, you should probably consider looking at the dataset and sampling in such a way that this situation can be avoided. Alternatively you could use a more generalizable algorithm and check for overfitting of the data via cross-validation again.

Answer 2

You should use k fold cross validation to average over the different ways of dividing up the training images.http://en.m.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation

Sometimes a particular split of training and testing images is lucky or unlucky. You can get more reliable estimates of the accuracy with some form of cross validation.