How is cross-correlation related with orthogonality?

In linear prediction we can say that in case of optimum linear predictor the error with be orthogonal to data. And when we derive minimum mean square error for $\underline{y} = \mathbf{a}\underline{x} + \underline{b}$ case we find following relation

$\xi = \sigma^2_y (1 - \rho_{xy} )^2$

Where $\rho_{xy}$ is correlation coefficient and if we talk about cross correlation $r_{xy}(k,l)$ of zero mean and uncorrelated random variables x and y, then how does this conclude that if $r_{xy}(k,l)$ will be then the signals will be orthogonal?

• What is the relationship between $\rho_{xy}$ and $r_{xy}(k,l)$ ? I'm assuming $\rho$ is the correlation coefficient and $r$ is the cross correlation, but it would be good to spell this out. – Peter K. May 6 '16 at 13:12
• I apologize for ambiguity in the question, I edited it. I hope it is not missing anything now. – Tab May 6 '16 at 13:22
• No need to apologise! Thanks for the edit. It's much clearer now. – Peter K. May 6 '16 at 13:23

I suppose you mean the cross-correlation at lag zero. Well take an Hilbert space $H$ (i.e. a metric space in which you can define a scalar product $\langle\cdot ,\cdot\rangle$). Then $x,y\in H$ are orthogonal if $\langle x,y\rangle=0$, by definition.
If your Hilbert Space is $L_2(\mathbb{R})$ (the space of real square integrable functions) then the scalar product of $f,g\in L_2(\mathbb{R})$ is defined as
$$\langle f, g\rangle = \int_{-\infty}^\infty \overline{f(t)}g(t) dt$$ If the signals are assumed to be ergodic, which is actually a costumary assumption, then the cross-correlation is simply a sliding scalar product, you have $$\text{corr}(f,g)(\tau) = \int_{-\infty}^\infty\overline{f(t)}g(t+\tau) dt$$ hence $$\text{corr}(f,g)(0) = \langle f ,g \rangle$$ and uncorrelation means orthogonality.
The same applies for discrete time signals in $l_2$
• Correlation is, more generally, $E[\overline{f(t)}g(t+\tau)]$. The definition you're using can only be used if $f$ and $g$ are ergodic. – Peter K. May 6 '16 at 12:32