What Is the Significance of a Large Residual When Applying Principal Component Analysis?

Question

I am using Matlab function PCA (principal component analysis) to reduce the dimensionality of a data set with approximately 20 000 observations x 100 dimensions.

After having obtained the principal component coefficients of the data I recreated the input signal in the original coordinate system using the transformation matrix from the PCA function. This yielded a very large residual when comparing with the input signal. I have tried around with different data sets and sizes and it appears to be commonplace to have a large residual. I am not sure yet whether it is due to round-off errors or high SNR in the input data. The dimensionality reduction could of course still be useful, but is this something that one should be cautious about when performing principal component analysis? Or is there another metric that is better to assess the performance of the principal component analysis?

Answer 1

PCA dimensionality reduction seeks for a linear transformation, mapping the data to a lower dimensional space. The linearity comes from the mapping, which is simply a rotation (and translation if you take into account the de-meaning). In the end, it is nothing but a projection of the data.

So if your low dimensional embedding is not reconstructing the data truthfully, then this means that your data is not suitable for such linear mapping, i.e. not linearly separable. You should look into other techniques such as linear discriminant analysis (canonical variates), Kernel-PCA, which applies kernel trick to achieve non-linearity or manifold learning.