A feature extraction method: Principal Component Analysis (PCA)

Question

I read some articles about PCA and I think the nice way to summarize it is:

It is a way of identifying patterns in data, and expressing the data in such a way as to highlight their similarities and differences.

So in my code, we subtract the initial data from the mean and find principal components, then we multiply the data by a variable called e as for our dimension reduction purposes.

What I don't understand here is that When we want to highlight the similarities and differences, it happens both about noise data, the data which is too big or too small, will be soften. What about the edges in our image? Will the same thing happen to those data? I mean, when we have a change of color from white to black and we don't want to lose such information, will PCA soften the data as well as noise? If yes, this is not what we want, is it? Is there a solution to solve this problem ?

Answer 1

Ok, it sounds like you are trying to do eigenfaces, right?

In that case, you have to think of your face images as points in a very high-dimensional space. For example, if your images are 32x32, then the space has 32 * 32 = 1024 dimensions. Operating in so many dimensions is very difficult, because distances between points become almost meaningless. With so many dimensions any point is very far away from any other point, even if the two points represent the same face.

That's why you want to reduce the dimensionality, and PCA is one way to do it. If you go from 1024 dimensions to, say, 10, then the distances will become more meaningful, which fits in with your idea of "highlight their similarities and differences". However, PCA by itself will not tell you of any patterns in your data, beyond the directions of greatest variance. PCA will project your data into a lower-dimensional space with decorrelated axes, but you have to do more work to actually discover the patterns in the data, such as train a classifier, or run a clustering algorithm.

Now, to understand what happens to the edges and transitions from light to dark, you have to display your eigenvectors as images and look at them. You will see that you don't really preserve the precise edge information, but you do keep course-level information about light and dark areas.

Answer 2

I think a better way to understand what PCA does is to understand what is a good feature. Suppose you are classifying obese people from non-obese people. A good feature (let's call it $f_1$) to use for example might be "body mass index (BMI)" for each person. Another good feature (called $f_2$) to use might be "weight". A third feature $f_3$ to use would be a time-averaged "blood glucose level". However, since BMI is calculated using weight anyways, the features $f_1$ and $f_2$ have some level of correlation between them. However, the weight and the blood glucose level do not have an immediate statistical correlation. They are uncorrelated features. In other words, including features $f_1$ and $f_2$ gives us a gain in information but not as much as including $f1$ and $f3$. If you want to design good features, then you need to "whiten" the data or include as many "uncorrelated" features as possible.

This is where a transformation is useful. Consider a Fourier transform. The basis functions of a Fourier transform are complex exponentials which are orthogonal to each other (uncorrelated). Hence if you transform your feature vector into the Fourier domain you get uncorrelated features. But its not clear which subset to pick.

PCA is a variant of the Karhunen-Loeve transform which use the data itself as a space within which to find basis functions. The eigenvectors of the data space form the basis functions and you can then project onto the eigenvectors with maximum eigenvalues.

As a result you preserve most of the signal energy while losing a little depending on how low dimensional you project.

Now as far as preserving the edges goes, perhaps PCA is not the best basis for this. You might be better in the wavelet domain which is able to preserve discontinuities like edges.