In linear prediction we can say that in case of optimum linear predictor the error with be orthogonal to data. And when we derive minimum mean square error for $\underline{y} = \mathbf{a}\underline{x} + \underline{b}$ case we find following relation

$\xi = \sigma^2_y (1 - \rho_{xy} )^2$

Where $\rho_{xy} $ is correlation coefficient and if we talk about cross correlation $r_{xy}(k,l)$ of zero mean and uncorrelated random variables x and y, then how does this conclude that if $r_{xy}(k,l)$ will be then the signals will be orthogonal?

  • $\begingroup$ What is the relationship between $\rho_{xy}$ and $r_{xy}(k,l)$ ? I'm assuming $\rho$ is the correlation coefficient and $r$ is the cross correlation, but it would be good to spell this out. $\endgroup$ – Peter K. May 6 '16 at 13:12
  • 1
    $\begingroup$ I apologize for ambiguity in the question, I edited it. I hope it is not missing anything now. $\endgroup$ – Tab May 6 '16 at 13:22
  • $\begingroup$ No need to apologise! Thanks for the edit. It's much clearer now. $\endgroup$ – Peter K. May 6 '16 at 13:23

I suppose you mean the cross-correlation at lag zero. Well take an Hilbert space $H$ (i.e. a metric space in which you can define a scalar product $\langle\cdot ,\cdot\rangle$). Then $x,y\in H$ are orthogonal if $\langle x,y\rangle=0$, by definition.
If your Hilbert Space is $L_2(\mathbb{R})$ (the space of real square integrable functions) then the scalar product of $f,g\in L_2(\mathbb{R})$ is defined as
$$ \langle f, g\rangle = \int_{-\infty}^\infty \overline{f(t)}g(t) dt $$ If the signals are assumed to be ergodic, which is actually a costumary assumption, then the cross-correlation is simply a sliding scalar product, you have $$ \text{corr}(f,g)(\tau) = \int_{-\infty}^\infty\overline{f(t)}g(t+\tau) dt $$ hence $$ \text{corr}(f,g)(0) = \langle f ,g \rangle $$ and uncorrelation means orthogonality.
The same applies for discrete time signals in $l_2$

| improve this answer | |
  • 1
    $\begingroup$ Correlation is, more generally, $E[\overline{f(t)}g(t+\tau)]$. The definition you're using can only be used if $f$ and $g$ are ergodic. $\endgroup$ – Peter K. May 6 '16 at 12:32
  • $\begingroup$ yes, thanks for pointing it out! I assumed implicitly that ergodicity was an assumption on the signals $\endgroup$ – LJSilver May 6 '16 at 12:39
  • $\begingroup$ Most people do. I gave this a + 1, but thought we should mention the ergodicity piece. $\endgroup$ – Peter K. May 6 '16 at 12:40
  • 1
    $\begingroup$ I added a piece saying that this holds under ergodicity assumption $\endgroup$ – LJSilver May 7 '16 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.