Signal Processing Stack Exchange is a question and answer site for practitioners of the art and science of signal, image and video processing. It only takes a minute to sign up.
Sign up to join this community
I am trying to understand and visualize the concept of a covariance matrix. Suppose I have a matrix:
$A = \begin{pmatrix} 2 & 3 & 4\\ 5 & 5 & 6 \end{pmatrix}$
how do I calculate its covariance matrix, and what is its practical meaning? (All I was able to understand by now is that on the diagonal of the covariance matrix, variances for particular variables are placed, and on the upper and lower fields are the correlations between those variables.)
You can't really have a covariance of a matrix. What you can have, is a covariance matrix of a set of vectors. So, if you think of the rows of your matrix A as two vectors in 3D: [2 3 4] and [5 5 6], then the covariance matrix of this set of two vectors is C = A' * A (A transpose times A). Note that if you shuffle the rows of A in a different order, C does not change.
Now that you have C, you understood it correctly. You can also do interesting things with the eigenvalues and eigenvectors of C.
Oops... Totally forgot. Before you do anything, you have to compute the mean vector (column-wise mean of A) and subtract it from every row.
Variance is defined as $V(x)=\frac{\sum_{i=1}^n(x_i-\mu)^2}{n}$. Just in case for you, mean $\mu$ is defined as $\mu=\frac{\sum_{i=1}^nx}{n}$. Covariance between two random variables $x$ and $y$ (or columns of a matrix) is defined as $Cov(x,y)=\frac{\sum_{i=1}^n[(x_i-\mu_x)(y_i-\mu_y)]}{n}$ and $Cov(x,x)=V(x)$.
The term covariance matrix may be misleading to you. It is not any sort of a special matrix. It is simply set of variances and covariances between pairs of columns. A position of any element in the covariance matrix corresponds to variance/covariance between a pair of two columns, e.g. a number located in 3rd row and 2nd column in the covariance matrix represents covariance between 3rd and 2nd columns of matrix $\textbf{A}$. $Cov(x,y)=Cov(y,x)$, therefore covariance matrix is symmetric.
If you have a matrix $\textbf{A}=\{\textbf{x}\;\textbf{y}\;\textbf{z}\}$ while $\textbf{x}$, $\textbf{y}$, and $\textbf{z}$ are column vectors of length $n$, covariance matrix can be calculated as follows:
$Cov(A)=\begin{bmatrix} \frac{\sum_{i=1}^n(x_{i}-\mu_x)^2}{n} & \frac{\sum_{i=1}^n(x_{i}-\mu_x)(y_{i}-\mu_y)}{n} & \frac{\sum_{i=1}^n(x_{i}-\mu_x)(z_{i}-\mu_z)}{n} \\ \frac{\sum_{i=1}^n(y_{i}-\mu_y)(x_{i}-\mu_x)}{n} & \frac{\sum_{i=1}^n(y_{i}-\mu_y)^2}{n} & \frac{\sum_{i=1}^n(y_{i}-\mu_y)(z_{i}-\mu_z)}{n} \\ \frac{\sum_{i=1}^n(z_{i}-\mu_z)(x_{i}-\mu_x)}{n} & \frac{\sum_{i=1}^n(z_{i}-\mu_z)(y_{i}-\mu_y)}{n} & \frac{\sum_{i=1}^n(z_{i}-\mu_z)^2}{n}\\ \end{bmatrix}=$
$Cov(A)=\begin{bmatrix} V(x,x) & Cov(x,y) & Cov(x,z) \\ \\ Cov(y,x) & V(y,y) & Cov(y,z) \\ \\ Cov(z,x) & Cov(z,y) & V(z,z) \\ \\ \end{bmatrix}$
