10
$\begingroup$

I have been reading about Karhunen-Loeve (KL) transform. I see that when it is used to reduce dimension the procedure is identical to PCA, that is, for both methods the covariance matrix of the data is constructed, and then the eigenvectors are calculated .

I would like to know if there is any difference in this aspect that is being overlooked?

Improve this question
$\endgroup$
3
  • 4
    $\begingroup$ This is well known. Recall that PCA is a spectral decomposition of the covariance matrix. In the continuous data case, KL transform is a spectral decomposition of the covariance function. PCA is sometimes called the discrete KL transform. $\endgroup$
    – Atul Ingle
    Commented May 16, 2018 at 3:16
  • $\begingroup$ very interesting question! $\endgroup$
    – Jonas Schwarz
    Commented Aug 4, 2019 at 8:33
  • $\begingroup$ @Royi I had already marked it $\endgroup$
    – Roger Figueroa Quintero
    Commented Mar 30, 2022 at 5:23

2 Answers 2

Reset to default
2
$\begingroup$

For discrete data both are the same - Finding set of orthogonal directions which maximizes the Variance (Energy) of data along them. Sometimes those are called the natural axis of the data (Inferred from data).

Since we're dealing with variance it is only natural both are calculated from the covariance matrix of data.

You may encounter places where KL might be even defined on the correlation matrix.

Anyhow, some nuances when dealing with them:

  1. Usually when dealing with Continuous Function people use the term KL. PCA is preserved to discrete data (Though Mathematicians might work with the PCA in all cases as they have SVD for continuous data as well).
  2. PCA is used for any discrete data where KL was specifically formed with random variables in mind.

In practice, when one has discrete data and tries to find the directions which maximizes the energy of the data (Dimensionality reduction, compression, etc...) both are the same.

Improve this answer
$\endgroup$
2
$\begingroup$

See:

Jan J. Gerbrands, On the relationships between SVD, KLT and PCA, Pattern Recognition,Volume 14, Issues 1–6, 1981, Pages 375-381, ISSN 0031-3203,https://doi.org/10.1016/0031-3203(81)90082-0. (https://www.sciencedirect.com/science/article/pii/0031320381900820)

Abstract: In recent literature on digital image processing much attention is devoted to the singular value decomposition (SVD) of a matrix. Many authors refer to the Karhunen-Loeve transform (KLT) and principal components analysis (PCA) while treating the SVD. In this paper we give definitions of the three transforms and investigate their relationships. It is shown that in the context of multivariate statistical analysis and statistical pattern recognition the three transforms are very similar if a specific estimate of the column covariance matrix is used. In the context of two-dimensional image processing this similarity still holds if one single matrix is considered. In that approach the use of the names KLT and PCA is rather inappropriate and confusing. If the matrix is considered to be a realization of a two-dimensional random process, the SVD and the two statistically defined transforms differ substantially.

Keywords: Image processing; Statistical analysis; Statistical pattern recognition; Orthogonal image transforms; Singular value decomposition; Karhunen-Loeve transform; Principal components

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.