1
$\begingroup$

I am trying to understand and visualize the concept of a covariance matrix. Suppose I have a matrix:

$A = \begin{pmatrix} 2 & 3 & 4\\ 5 & 5 & 6 \end{pmatrix}$

how do I calculate its covariance matrix, and what is its practical meaning? (All I was able to understand by now is that on the diagonal of the covariance matrix, variances for particular variables are placed, and on the upper and lower fields are the correlations between those variables.)

$\endgroup$
  • 4
    $\begingroup$ Covariance matrices are defined for vectors of random variables, not constants. $\endgroup$ – Dilip Sarwate Apr 16 '13 at 9:41
  • $\begingroup$ Josh130, can you rephrase the question? As Dilip says, there is no real connection between a constant matrix and a covariance matrix (well, the covariance will be zero). Please read the Wikipedia entry and see if that helps. $\endgroup$ – Peter K. Apr 16 '13 at 12:39
3
$\begingroup$

You can't really have a covariance of a matrix. What you can have, is a covariance matrix of a set of vectors. So, if you think of the rows of your matrix A as two vectors in 3D: [2 3 4] and [5 5 6], then the covariance matrix of this set of two vectors is C = A' * A (A transpose times A). Note that if you shuffle the rows of A in a different order, C does not change.

Now that you have C, you understood it correctly. You can also do interesting things with the eigenvalues and eigenvectors of C.

Oops... Totally forgot. Before you do anything, you have to compute the mean vector (column-wise mean of A) and subtract it from every row.

$\endgroup$
  • $\begingroup$ This answer is incorrect. If $A$ is a $2\times 3$ matrix, then $C=A^TA$ is a $3\times 3$ matrix whose entries are the $9$ possible inner products (dot products) of the $3$ columns. Shuffling the rows merely shuffles the order of summation in the inner products, and that's why $C$ does not change. What all this has to do with covariance, a probabilistic concept, is beyond my comprehension. $\endgroup$ – Dilip Sarwate Apr 16 '13 at 16:48
  • 2
    $\begingroup$ @Dilip, If I give you an n x 3 matrix of n 3D vectors, how do you compute the sample covariance? $\endgroup$ – Dima Apr 16 '13 at 17:51
  • 1
    $\begingroup$ What is the definition of sample covariance? Your calculations are meaningless unless you clearly specify what the random variables are, and how you are defining the sample covariance. $\endgroup$ – Dilip Sarwate Apr 16 '13 at 18:03
  • 1
    $\begingroup$ inst.eecs.berkeley.edu/~ee127a/book/login/… Dima seems to be correct. $\endgroup$ – Andrey Rubshtein Jul 15 '13 at 21:30
2
$\begingroup$

Variance is defined as $V(x)=\frac{\sum_{i=1}^n(x_i-\mu)^2}{n}$. Just in case for you, mean $\mu$ is defined as $\mu=\frac{\sum_{i=1}^nx}{n}$. Covariance between two random variables $x$ and $y$ (or columns of a matrix) is defined as $Cov(x,y)=\frac{\sum_{i=1}^n[(x_i-\mu_x)(y_i-\mu_y)]}{n}$ and $Cov(x,x)=V(x)$.

The term covariance matrix may be misleading to you. It is not any sort of a special matrix. It is simply set of variances and covariances between pairs of columns. A position of any element in the covariance matrix corresponds to variance/covariance between a pair of two columns, e.g. a number located in 3rd row and 2nd column in the covariance matrix represents covariance between 3rd and 2nd columns of matrix $\textbf{A}$. $Cov(x,y)=Cov(y,x)$, therefore covariance matrix is symmetric.

If you have a matrix $\textbf{A}=\{\textbf{x}\;\textbf{y}\;\textbf{z}\}$ while $\textbf{x}$, $\textbf{y}$, and $\textbf{z}$ are column vectors of length $n$, covariance matrix can be calculated as follows:

$Cov(A)=\begin{bmatrix} \frac{\sum_{i=1}^n(x_{i}-\mu_x)^2}{n} & \frac{\sum_{i=1}^n(x_{i}-\mu_x)(y_{i}-\mu_y)}{n} & \frac{\sum_{i=1}^n(x_{i}-\mu_x)(z_{i}-\mu_z)}{n} \\ \frac{\sum_{i=1}^n(y_{i}-\mu_y)(x_{i}-\mu_x)}{n} & \frac{\sum_{i=1}^n(y_{i}-\mu_y)^2}{n} & \frac{\sum_{i=1}^n(y_{i}-\mu_y)(z_{i}-\mu_z)}{n} \\ \frac{\sum_{i=1}^n(z_{i}-\mu_z)(x_{i}-\mu_x)}{n} & \frac{\sum_{i=1}^n(z_{i}-\mu_z)(y_{i}-\mu_y)}{n} & \frac{\sum_{i=1}^n(z_{i}-\mu_z)^2}{n}\\ \end{bmatrix}=$

$Cov(A)=\begin{bmatrix} V(x,x) & Cov(x,y) & Cov(x,z) \\ \\ Cov(y,x) & V(y,y) & Cov(y,z) \\ \\ Cov(z,x) & Cov(z,y) & V(z,z) \\ \\ \end{bmatrix}$

$\endgroup$
  • 1
    $\begingroup$ I believe he used his example because he has not yet understood what covariance matrix is. I also believe this will help to understand the concept! Therefore, I provided this explanation. All terms such as $\mu$ with sub-indices correspond to mean of particular columns: $\textbf{x}$, $\textbf{y}$, and $\textbf{z}$ in my example. $\endgroup$ – Celdor Jul 16 '13 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.