Principal Component Analysis (PCA) on Convolutional Neural Network (CNN) Features

Question

Please, I have a question regarding PCA and features which are extracted from a convolutional layer based on Faster R-CNN features for Instance Search

if we have a test dataset , and we extract all conv features of all images at test dataset called feat. Then we do the following normalization and PCA to all conv features

feats = normalize(feats)
pca = PCA(512 ,whiten=True)
pca.fit(feats)

And then use PCA model for test dataset

# PCA MODEL - use paris for oxford data and vice versa
self.pca = pickle.load(open('_oxford.pkl', 'rb'))

And then applying PCA on the test dataset using the previous pca

print "Applying PCA"
self.db_feats = normalize(self.db_feats)
if self.pooling is 'sum':
self.db_feats = self.pca.transform(self.db_feats)
self.db_feats = normalize(self.db_feats)

What is the idea of using PCA model of another dataset to transform the features of test dataset?

Answer 1

It does appear that the (re)ranking code is using the wrong dataset, i.e. the Oxford model with the Paris images. This question was raised in the following github issue: wrong dataset name #6.

However, the explanation given by the authors is that this is a convention in the literature and they do this to be able to measure their results against prior art.

We basically decided to do this to be consistent with other works in the literature in which PCA was also trained on Oxford and tested on Paris and viceversa. This is explained in the paper. You can find the references there.

I also think that by training on one model (Oxford) and tested on another (Paris) they can get another sense of how their models generalize.

Answer 2

I'm not into details of this specific case but I can see some logic.

A convolution layer can be reformulated as a Matrix Multiplication:

$$ y = W x $$

Let's say we trained on Data Set $ {x}^{1} $ which is big and general.
Namely we expect the trained weights $ {W}^{1} $ to be good enough for almost any other data set.

Let's assume we have another data set $ {x}_{2} $ with its matching weights $ {W}^{2} $.
By our assumption about how good $ {x}^{1} $ generalizes it means $ {W}^{1} \sim {W}^{2} $ with regard to some metric.

If this metric is the Frobenious Norm / Matrix Induced $ {L}_{2} $ Norm then using PCA (Which is basically SVD) makes sense as using the PCA we take the component of $ {W}^{1} $ which generalizes the most and use it for other case.

Answer 3

We are applying something similar like so:

• A CNN is trained on a particular image dataset.
• PCA (or some other transform) is performed on the feature vectors to obtain the main axes of variation.
• The images are inspected to see what the variations actually correspond to.
• Features are calculated using the same network on a much much larger dataset.
• The features are then transformed to principal components and statistics calculated.

A lot of our much larger datasets a time series, so we can graph the change in one principal component over time etc. Work still in progress so no comment yet about how worthwhile it is.

Another thing is that I find most of the feature vector values are close to zero. If nothing else, PCA can reduce the size of the feature vector.