I have been reading about Karhunen-Loeve or also known as KL transform and I see that when it is used to reduce dimension the procedure is identical to PCA, that is, for both methods the covariance matrix of the data is constructed and then the eigenvectors are calculated . I would like to know if there is any difference in this aspect that is being overlooked?
For discrete data both are the same - Finding set of orthogonal directions which maximizes the Variance (Energy) of data along them. Sometimes those are called the natural axis of the.
Since we're dealing with variance it is only natural both are calculated from the Contrivance matrix of data.
You may encounter places where KL might be even defined on the correlation matrix.
Anyhow, some nuances when dealing with them:
- Usually when dealing with Continuous Function people use the term KL (Where PCA is preserved to discrete data (Though Mathematicians might work with the PCA in all cases as they have SVD for continuous data as well).
- PCA is used for any discrete data where KL was specifically formed with random variables in mind.
In practice, when one has discrete data and tries to find the directions which maximizes the energy of the data (Dimensionality reduction, compression, etc...) both are the same.