Principal Component Analysis definition

Question

I have just learned about this method, so I am not very familiar with it. As far as I know, Principal Component Anlysis (aka PCA) is used to transform a vector $x$ that belongs to a space of $d$ dimensions to a vector $y$ that belongs to a space of $p$ diamensions, where $p\ll d$ . The new vector $y$ consists of p uncorrelated components. I also know that these $p$ components hold the maximum possible energy of the initial vector $x$ . Last but not least, since PCA is an orthogonal transformation, it should minimize the error norm (the difference between the initial vector and its projection). Correct me if I am wrong.

So let's say we have a random vector $x$ that belong to space $\mathbb{R}^{d}$ and we use PCA to project it to a new space of $p$ dimensions where $p\ll d$. I have come up with three statements. However, I can't find out which one is the correct one:

1. PCA projects vector $x$ to a space of $p$ dimensions where the projection has minimum energy
2. PCA projects vector $x$ to a space of $p$ dimensions where the difference between the initial vector and the projection has maximum energy.
3. PCA projects vector $x$ to a space which is homeomorphic with $\mathbb{R}^{p}$

So which is the correct statement?

Answer 1

By projecting a vector x using PCA (on the PCs), you maximize the variance in the reduced space. Initially, the space is not optimal in terms of maximizing the variance.

So:

PCA projects vector 𝑥 to a space of 𝑝 dimensions where the difference between the initial vector and the projection has maximum energy.

(initially the is no maximum variance but after projection the space is optimal in terms of maximizing the variance in the PC defined space)

PS:

Minimizing J1 (the error) corresponds to maximizing the quadratic form e'Se. After demeaning this is e'X'Xe. Set now w = Xe (the projection of the data X). The variance in the projected space is w'w that is equal to the e'X'Xe. Thus, minimizing the error means maximizing varinace