Basic Questions on Wiener Filtering

I read a lot about Wiener Filters (focusing on discrete time case). I understand the math, but I am quite disconnected from the real life assumptions behind using such a filter. Unfortunately textbooks are just laying out the formulas without putting forward much intuition in it.

The task is to extract x(t) given y(t), where y(t) = x(t) + n(t). Great but first we need to build an appropriate filter. At this point:

1. Are we assuming that we have access to some training samples of x(t) and/or n(t) in addition to y(t)?

2. Or are we just assuming that we have access to statistical properties(auto correlation, cross correlation) of x(t) and/or n(t) in addition to y(t)? If so since x(t),n(t) are not explicitly available, are these statistics computed from other signals available to us with similar statistical properties?

3. Or is it a combination of both, such as having access to some training samples of x(t), but having only access to second order statistics of n(t) (or vice versa)?

4. Is the noise n(t) always assumed to be uncorrelated to x(t) in Wiener filtering, or are the textbooks going for the simplified version? What is the default scenario?

5. Why are all the examples coming up with all-zero filters but not using pole-zero filters? Is it just a matter of simplified textbook treatment or is the problem definition not amenable to using transfer function modelling approach? Yes the first approach is computationally easier, but these days we have significant computing power and I don't think it is a good justification anymore, if that's the only reason.

Looking forward to hear from practitioners.

The task is to filter x(t) when given y(t), where y(t) = x(t) + n(t). Great but first we need to build an appropriate filter. At this point:

No. The task is to filter $$y[n]$$ to achieve $$x[n]$$. Your filter acts on the available signal $$y[n]$$ to get a best estimate of $$x[n]$$ from a noisy observation of it. BTW. discrete-time uses $$y[n]$$ or at least $$y(n)$$ notation, and not $$y(t)$$. In DSP, $$t$$ is reserved for continuous-time.

1) Are we assuming that we have access to some training samples of x(t) and/or n(t) in addition to y(t)?

trainig samples is a term related with adaptivity, pattern recognition, neural networks etc. In the classical Wiener context, you do not need training samples but you need data statistics. See answer 2 for how to obtain the statistics.

2) Or are we just assuming that we have access to statistical properties(auto correlation, cross correlation) of x(t) and/or n(t) in addition to y(t)? If so since x(t),n(t) are not explicitly available, are these statistics computed from other signals available to us with similar statistical properties?

Yes. In the Wiener filter context, you either assume (for theoretical development) or estimate (for practical applcations) the correlations (or Power Spectral Densities) between all those signals. In practice you estimate them from avaiable data by several means, depending on the application. You need data for this; and that's probably what you call training samples.

3) Or is it a combination of both, such as having access to samples of x(t), but having only access to second order statistics of n(t) (or vice versa)?

You do not have access to relevant noise-free $$x[n]$$ by any means; otherwise why filter $$y[n]$$ to get $$x[n]$$? But, in some cases, one can have noise free irrelevant blocks of $$x[n]$$ samples, in addition to the relevant noisy blocks. So you can use those noise free blocks to estimate certain parameters of the process $$x[n]$$, so as to filter out the noisy blocks.

4) Is the noise n(t) always assumed to be uncorrelated to x(t) in Wiener filtering, or are the textbooks going for the simplified version? What is the default scenario?

Yes. The noise $$n(t)$$ (or better $$v[n]$$ in discrete-time context) is almost always modeled as additive, white, zero-mean Gaussian noise uncorrelated with $$x[n]$$. This simplifies all the mathematical development and in typical cases is a good model. But you can relax some of those assumption, if that would yield a better (but more complex to derive) filter...

5) Why are all the examples coming up with all-zero filters but not using pole-zero filters? Is it just a matter of simplified textbook treatment or is the problem definition not amenable to using transfer function modelling approach? Yes the first approach is computationally easier, but these days we have significant computing power and I don't think it is a good justification anymore, if that's the only reason.

IIR filters are good for their computational efficiency (and in Wiener context, they also yield the minimum MSE compared to FIR) but they may suffer from unstability and causality problems, and their design and analysis is more complex in adaptive filtering context. For this reasons, it's simpler and safer to use FIR based filters for adaptive Wiener applications.

• "You do not have access to relevant noise-free x[n] by any means". Of course I was referring to some training data. Isn't the whole point to build the filter on what's currently available and then use it on new dataset? So there must be some training data even in this context. No? – Cowboy Trader Jul 27 '19 at 12:45
• @CowboyTrader Yes that's true as an interpretation. You, design the Wiener the filter based on any availble data, training samples or reasonable assumptions, and then apply it on the actual new data set. The success of the Wiener filter on the new data set is a direct consequence of the resemblance between the new data set and the training set... – Fat32 Jul 27 '19 at 22:45
• Cannot really comment on the statistical point of view, but for DSP work, if an IIR filter is anti-causal, then no lag can make it causal. Also you cannot implement a non-causal filter in real-time. Otherwise, off-line data procesing permits noncausal filters to work as well. So it depends. IIR filters are perfectly stable as long as their quantized poles are inside the unit circle (for causal) but this should always inspected to verify. Whereas for FIR filters you don't even need to think about; as they are guaranteed to be stable for any finite impulse response. – Fat32 Jul 28 '19 at 0:49

In some applications, such as channel equalization, one often uses a training sequence known to the receiver to compute the optimum filter coefficients. During normal operation, the training sequence is replaced by the decisions of the receiver ("decision-directed mode").

In other applications, such as speech denoising, one attempts to estimate the properties of the noise during silence periods. Of course it's crucial to detect such periods of signal absence with great accuracy. The power spectrum of the desired signal can then be estimated by subtracting the estimated noise spectrum from the spectrum of the observed noisy signal (assuming that signal and noise are uncorrelated).

Wiener filtering does not generally assume that signal and noise are uncorrelated. E.g., the non-causal Wiener filter is given by the well-known formula

$$H(z)=\frac{S_{xy}(z)}{S_{yy}(z)}\tag{1}$$

where $$S_{xy}(z)$$ is the cross-power spectral density of the signal and the noisy observation, and $$S_{yy}(z)$$ is the power spectrum of the observation.

Only if noise and desired signal are uncorrelated do we get the intuitively even more pleasing formula

$$H(z)=\frac{S_{xx}(z)}{S_{xx}(z)+S_{nn}(z)}\tag{2}$$

Note that in many practical situations it is reasonable to assume that desired signal and noise are uncorrelated, but the Wiener filter does not assume this a priori.

Many textbooks discuss three cases of the Wiener filter: 1. the non-causal Wiener filter, 2. the causal solution, which is generally IIR, and 3. the (causal) FIR solution. The latter is often emphasized because of its straightforward implementation and the availability of stable adaptive algorithms. Many textbooks also give examples of the IIR solution. One such book that I can recommend is Optimum Signal Processing by S.J. Orfanidis.

• After coming back to this answer I appreciated it even more. Do you think Wiener filtering is still something relevant when everybody is on the deep learning train where nonlinearities and higher order statistics can be captured? – Cowboy Trader Dec 23 '20 at 13:45
• @CowboyTrader: I don't think that the Wiener filter will lose its importance, because it is a simple and efficient solution for a certain class of practically important problems. – Matt L. Dec 23 '20 at 15:13