# Principal Component Analysis definition

I have just learned about this method, so I am not very familiar with it. As far as I know, Principal Component Anlysis (aka PCA) is used to transform a vector $$x$$ that belongs to a space of $$d$$ dimensions to a vector $$y$$ that belongs to a space of $$p$$ diamensions, where $$p\ll d$$ . The new vector $$y$$ consists of p uncorrelated components. I also know that these $$p$$ components hold the maximum possible energy of the initial vector $$x$$ . Last but not least, since PCA is an orthogonal transformation, it should minimize the error norm (the difference between the initial vector and its projection). Correct me if I am wrong.

So let's say we have a random vector $$x$$ that belong to space $$\mathbb{R}^{d}$$ and we use PCA to project it to a new space of $$p$$ dimensions where $$p\ll d$$. I have come up with three statements. However, I can't find out which one is the correct one:

1. PCA projects vector $$x$$ to a space of $$p$$ dimensions where the projection has minimum energy
2. PCA projects vector $$x$$ to a space of $$p$$ dimensions where the difference between the initial vector and the projection has maximum energy.
3. PCA projects vector $$x$$ to a space which is homeomorphic with $$\mathbb{R}^{p}$$

So which is the correct statement?

• excellent question. i'll be listening in to any answers. – robert bristow-johnson Jun 13 '19 at 18:59
• @robertbristow-johnson Thanks!Do you have any suggestions or thoughts about the correct answer? – MJ13 Jun 13 '19 at 19:01
• no. i've seen a couple of articles on PCA, but i dunno what it's about. – robert bristow-johnson Jun 13 '19 at 19:37
• Sounds like a quizz for some homework. Is this the case? What is a space which is "uniform with"? What is you educated guess so far? – Laurent Duval Jun 13 '19 at 20:03
• Do you have to select one out of the 3 options OR did you create these statements? none seems totally correct to me. If you need a good definition, I can provide this. – makis Jun 13 '19 at 20:08

By projecting a vector x using PCA (on the PCs), you maximize the variance in the reduced space. Initially, the space is not optimal in terms of maximizing the variance.

So:

PCA projects vector 𝑥 to a space of 𝑝 dimensions where the difference between the initial vector and the projection has maximum energy.

(initially the is no maximum variance but after projection the space is optimal in terms of maximizing the variance in the PC defined space)

PS:

Minimizing J1 (the error) corresponds to maximizing the quadratic form e'Se. After demeaning this is e'X'Xe. Set now w = Xe (the projection of the data X). The variance in the projected space is w'w that is equal to the e'X'Xe. Thus, minimizing the error means maximizing varinace

• Doesn't that contrast the fact that since PCA is an orthogonal transformation, it should minimize the error norm? I mean isn't the error norm the difference between the initial vector and the projection? – MJ13 Jun 13 '19 at 20:21
• yes, but maximum energy means minimum error in terms of projection. I wouldn't really choose any of the 3 cause it's not clear but since you have to select, 2 makes more sense to me – makis Jun 13 '19 at 20:23
• Hmm so the fact the we demand maximum energy of this vector concludes that we minimize the error as well? – MJ13 Jun 13 '19 at 20:37
• yes you project onto a space that has maximum variance by reducing the projection error as much as possible – makis Jun 13 '19 at 20:59
• I am just still a bit confused since the difference between the initial vector and the projection is considered to be the error vector...Isn't the energy equal to the norm of this vector? If yes, if we maximize it then we maximize the error as well...So we don't get minimum error but maximum..What am I missing here? – MJ13 Jun 13 '19 at 21:27