I am new to linear algebra and have this simple question... in least sqaure estimation...the best estimation of the equation $Ax = b$ is $x_{Estimated} = A (A^t A )^{-1} A^t b$...the projection of $b$ on the column space of $A$ is $p = A (A^t A)^{-1} At b$ which is written as $p = P b...P$ is the projection matrix = $A (A^t A)^{-1} A^t$ now using inverse rule $(AB)^{-1} = B^{-1} A^{-1}$ .......$P$ reduces to $I$ ....so is the projection matrix $P$ just the identity matrix $I$ ..?? am i missing out some basic or is my concept not clear? can we write $(A^t A)^{-1} = A^{-1} (A^t)^{-1}$ always??

  • $\begingroup$ thanx Peter for editing...i was missing the superscripts..kind of new to editing with symbols...makes it look proper now...thanx $\endgroup$ – rotating_image Sep 28 '12 at 12:53
  • $\begingroup$ Also note that least-squares solution of $Ax=b$ can be written as $x=A^{+}b$ where $A^{+}$ is a generalized inverse, or pseudo-inverse. Of course $A^{-1}$ is not defined when $A$ is over-determined. To make $A$ square, the problem is sometimes solved using normal equations of the form $A^{\mathbb{T}}Ax=A^{\mathbb{T}}b$, but in near-singular cases, solving by QR decomposition or SVD is more appropriate. SVD works also for rank-deficient cases. $\endgroup$ – Libor Sep 28 '12 at 13:42
  • $\begingroup$ do you mean that if the rank of A (say r) r = n(dimension of x) then the projection matrix P = I ? $\endgroup$ – rotating_image Sep 28 '12 at 13:50
  • $\begingroup$ If $A$ has same rank as dimension of $x$ and $A$ is invertible, then yes. $\endgroup$ – Libor Sep 28 '12 at 14:09

First of all, the minimum norm least square solution is $A^+b$, where $A^+$ is the pseudoinverse. Only when the left inverse $A_L^{-1}$ (or right inverse $A_R^{-1}$) exists, you have $A^+=A_L^{-1}=(A^TA)^{-1}A^T$ (or $A^+=A_R^{-1}=A^T(AA^T)^{-1}$).

Likewise, the projection matrix onto the column space is $P=AA^+$, or $P=AA_R^{-1}=AA^T(AA^T)^{-1}=I$ if $A_R^{-1}$ exists. In other words, if A has full row rank, then the projector $P$ is indeed an identity matrix. If only left inverse exists, then $P=AA_L^{-1}=A(A^TA)^{-1}A^T$ is not an identity matrix.

Finally, $(AB)^{-1}=A^{-1}B^{-1}$ is valid only when $A$ and $B$ are square matrices.


From here:

$$ (AB)(B^{-1}A^{-1}) = A (BB^{-1}) A^{-1} = A A^{-1} = I $$

So $(AB)^{-1} = B^{-1} A^{-1}$ provided $A$ and $B$ are invertible.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.