How intuitively to understand why eigenvectors of the autocorrelation matrix are used, but eigenvectors of the matrix constructed from temporal samples have no sense and aren't used? For example, in detection of a harmonious signal in additive noise.


Some "gut-level" reasons why it is better to work with the autocorrelation matrix instead of a matrix with your observations:

  • If you want to take into account all your observations and you have a lot of data, you'll end up manipulating (inverting, multiplying) fairly large matrices. If you work with the autocorrelation matrix, you "summarize" your data once (in a fairly efficient step requiring just a FFT and an inverse FFT), and from then, you just manipulate your autocorrelation matrix of size $P \times P$ where $P$ is your model order (for example for AR modeling or sinusoidal modeling).
  • With some data it just doesn't work numerically to use the raw observations because you encounter situations in which you have to deal with matrices which are not guaranteed to be positive-definite.

For example, let us consider two approaches to AR model fitting.

Direct usage of the data matrix

The empirical quadratic reconstruction error on your data is:

$$\epsilon = \mathbf{x}^T\mathbf{x} + \mathbf{x}^T\Gamma \mathbf{a} + \mathbf{a}^T\Gamma^T\mathbf{x} + \mathbf{a}^T\Gamma^T \Gamma \mathbf{a} $$

where $\mathbf{a}$ is the vector of AR-coefficients, $\mathbf{x}$ is your observations vector, and $\Gamma$ the matrix with your delayed observations. You need to find the value of $\mathbf{a}$ that minimizes this. After derivation and a bit of shuffling, your solution looks like this:

$$\mathbf{a} = - (\Gamma^T \Gamma)^{-1} \Gamma^T \mathbf{x}$$

And you are screwed because you have absolutely no guarantee that $\Gamma^T \Gamma$ can be inverted. In the process, numerically speaking, you had to deal with fairly large matrix products if you have a long sequence of observations.

Random process view

if you adapt a "random process" angle to the problem, the quantity you have to minimize (the expected value of the error) is:

$$\epsilon = r_x(0) + 2 \mathbf{r} \mathbf{a} + \mathbf{a}^T \mathbf{R} \mathbf{a}$$

And you end up with the more palatable solution:

$$\mathbf{a} = - \mathbf{R}^{-1} \mathbf{r}$$

With a solid guarantee that this will be computable because $\mathbf{R}$ is positive definite!

It looks like your problem is that of sinusoidal modeling (rather than AR modeling). There's a lot of hand-waving here, but what I have said about AR modeling and the hurdles of using the raw data matrix ; also applies to sinusoidal modeling - with eigenvalue decomposition being the problematic operation instead of matrix inversion.

| improve this answer | |

Firstly, eigenvectors and eigenvalues are defined for operators. Correlation is an operation.

Secondly, the eigenvectors of the autocorrelation are particularly interesting because they most efficiently explain the signal's variance in a linear regression. In other words, for a fixed number of vectors, selecting the eigenvectors minimizes the mean squared error where the signal is modeled as a linear sum of the vectors. This technique is referred to as principal component analysis.

If you can expand your notion of a "harmonious" signal, perhaps I can comment further.

| improve this answer | |
  • $\begingroup$ Yes, and may I add, one can also work with the data matrix in principal component analysis. However, this involves singular value decomposition instead. $\endgroup$ – Bryan May 17 '12 at 2:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.