What is the difference between the Weiner-Hopf equation and the normal equation?

In adaptive filters, the development of LMS algorithm typically starts from the Weiner-Hopf equation, while the development of RLS algorithm starts from the normal equation. As I understand, these two equations are the same, and both their solutions is the optimal coefficients that the adaptive filter has to find. Is this true or am I missing something?

• I guess "these" two equations are different. While Wiener-Hopf equations are defined for stochastic signals, the normal equation are defined for deterministic signals. For detailed discussions:please refer to page 541 of Monson H. Hayes book: Statistical digital signal processing and modeling – Oliver Aug 6 '14 at 7:53

1 Answer

Weiner-Hopf equation leads to Wiener filter that is optimal filter. For the case of stationarity in some time span it's the only filter minimizing MSE at its output. LMS and RLS algorithms are the adaptive approaches and they converge to Wiener optimal solution (as you can see from their convegence curves). While LMS is the simple steepest descent method of finding such an optimum, RLS solves Least Squares estimation problem at every step recursively.

LS estimation problem can be treated as a practical approach of Wiener filter. Look:

• $w = \hat{R}^{-1} \cdot \hat{r}$

is the normal equation in matrix form for the LS problem, $\hat{R}$ and $\hat{r}$ are estimations of autocorrelation matrix and cross-correlation vector respectively. So if these estimations is properly done this solution leads us to the Wiener-Hopf eq., i.e. to Wiener filter. But it's only true if estimations are valid. So in general LS solution is not a Wiener solution, it's an approximation.

RLS approach is derived from LS (as it can be viewed from abbreviation):

• $w(k) = S(k) \cdot \hat{r}(k)$,

where $S(k)$ is recursive estimation of $R^{-1}$ and $\hat{r}(k)$ is a recursive estimation of $r$. From the equation above one can see RLS try to solve LS problem at every step.

So the conclusion might be both LMS and RLS try to converge to Wiener solution and Wiener filter theory is in the core of these algorithms. But ways they do this are different.

This is my own understanding, maybe some discrepancies are possible. There is very good reference about adaptive filters: http://books.google.ru/books/about/Adaptive_Filter_Theory.html?id=MdDi_PF7gMsC&redir_esc=y