I have been reading about Karhunen-Loeve or also known as KL transform and I see that when it is used to reduce dimension the procedure is identical to PCA, that is, for both methods the covariance matrix of the data is constructed and then the eigenvectors are calculated . I would like to know if there is any difference in this aspect that is being overlooked?

  • 3
    $\begingroup$ This is well known. Recall that PCA is a spectral decomposition of the covariance matrix. In the continuous data case, KL transform is a spectral decomposition of the covariance function. PCA is sometimes called the discrete KL transform. $\endgroup$ – Atul Ingle May 16 '18 at 3:16
  • $\begingroup$ very interesting question! $\endgroup$ – Jonas Schwarz Aug 4 '19 at 8:33

For discrete data both are the same - Finding set of orthogonal directions which maximizes the Variance (Energy) of data along them. Sometimes those are called the natural axis of the.
Since we're dealing with variance it is only natural both are calculated from the covariance matrix of data.

You may encounter places where KL might be even defined on the correlation matrix.

Anyhow, some nuances when dealing with them:

  1. Usually when dealing with Continuous Function people use the term KL. PCA is preserved to discrete data (Though Mathematicians might work with the PCA in all cases as they have SVD for continuous data as well).
  2. PCA is used for any discrete data where KL was specifically formed with random variables in mind.

In practice, when one has discrete data and tries to find the directions which maximizes the energy of the data (Dimensionality reduction, compression, etc...) both are the same.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.