0
$\begingroup$

In estimating parameters in a discrete time model I've often seen the use of filters applied to the input data, before its applied to least squares processing. I've been told that the filters are useful if one wishes to remove bias or high frequency noise that's not considered a part of the model.

But I'm currently working with a problem where I'm using the Moore Penrose pseudoinverse to calculate the least squares. I'm filtering the input data with a lowpass filter to remove high frequency noise well beyond the model dynamics, but the choice of the filter pole seems to have a significant affect on the outcome of the estimate. And in any case the residues are near zero (they don't seem to be better or worse according to pole selection).

How should one chose to filter input data, and should it affect the estimate?

$\endgroup$
  • $\begingroup$ do you mean LMS adaptive filtering? what parameters are you trying to estimate in a least-squares manner? $\endgroup$ – robert bristow-johnson Nov 5 '15 at 1:15
  • $\begingroup$ @robertbristow-johnson No - least squares. I have a linear parametric model $z = \phi \theta^T$ and I'm estimating the parameter vector $\theta$ using pseudoinverse. $\endgroup$ – docscience Nov 5 '15 at 1:39
  • $\begingroup$ $ z $ is the output and $ \phi $ is the input and you're measuring both and trying to determine the values of the elements of the $ \theta $ matrix? $\endgroup$ – robert bristow-johnson Nov 5 '15 at 1:42
  • $\begingroup$ @robertbristow-johnson yes - basically that's it. So if I filter $z$ and $phi$ with the same filter - but such that it does not step on the dynamics of the model, then I should always get the same $\theta$ regardless of the filter poles - right? $\endgroup$ – docscience Nov 5 '15 at 1:51
  • $\begingroup$ it seems to me that it should be $$ z = \theta^T \phi $$. i would prefer "$ x $" for input and "$ y $" for output and "$ h $" for what goes in-between. can you form the question more rigorously? at least for the sake of those of us that haven't done System Identification since grad school? $\endgroup$ – robert bristow-johnson Nov 5 '15 at 1:56
2
$\begingroup$

I'm not sure what's you model is.

Let's say it is something like:

$$ y = H x + n $$

Now, using the Least Squares model is optimal (In the MSE sense) when $ n $ is AWGN (It is the linear optimal estimator if the noise is white).

So unless the noise in your model is colored, no gain by filtering the data before applying the Least Squares method.

Now, what if the noise isn't IID White Noise?

  1. If it is white just with different statistics, then use the Weighted Least Squares.
  2. If it is colored you should apply Whitening filter on the data.

So, if the Whitening filter is given by $ W $ then you have:

$$ z = W y = W H x + W n = A x + v $$

Now you have classic LS model, where your model matrix is $ A = W H $ and your white noise is $ v = W n $.
Now you can just apply the regular Least Squares.

$\endgroup$
  • $\begingroup$ Hi All: It sounds to me like the kind of filter the OP is referring to is a filter that removes outliers in order that the subsequent model fit is not affected by them. This is tricky because identification is difficult. ( might be better off sending to cross validated if my understanding is correct ) I can't help in terms of how to identify what's an outlier and what's not but my point is that I think that the term "filter" that the OP is using may not have the meaning it has in DSP. He may mean remove outliers in order to obtain a more robust estimate in subsequent model estimation. $\endgroup$ – mark leeds Jul 1 at 23:59
  • $\begingroup$ The OP says "high frequency noise" which suggests Low Pass Filter. So I think most probably he does talk on the classic meaning of filtering. Moreover, he then explicitly says "I'm filtering the input data with a lowpass filter to remove high frequency noise well beyond the model dynamics". So now I'm sure :-). $\endgroup$ – Royi Jul 2 at 4:25
  • $\begingroup$ you're probably right but, at the same time, a lowpass filter might mimic some smoothing operation which could end up being the equivalent of removing outliers. also, what do you mean by white with different statistics ? thanks. $\endgroup$ – mark leeds Jul 3 at 5:48
  • $\begingroup$ In that case I meant if they have, for instance, different standard deviation. $\endgroup$ – Royi Jul 3 at 9:18
  • $\begingroup$ Okay. Gotcha. It seems to me that, to know what $W$ needs to be ( to be a whitening filter ), then one would have to know the elements of the original noise. Is that an assumption when figuring out $W$. Thanks. $\endgroup$ – mark leeds Jul 3 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.