I am currently studying recursive least square (RLS) and as far as I understand, the setup is (for example) that we are given a process with uncertain parameters/noise and we want to adapt parameters of a model of that process online.

To me, that sounds like a Luenberger observer, where the parameters of the underlying model are adapted online, depending on the model quality.

If this is true, a side question would be why this technique isn't usually called an observer, but a filter instead? However, my main question is:

Question: Can RLS also be used instead of a standard low-pass filter?

So, given only a signal from which I want to remove high-frequency components (like noise), can a RLS filter be used to do that as well? And if so, would it have any advantage over normal low-pass filters?


The problem that recursive least squares (RLS) can solve can be formulated as recursively solving for $\hat{\theta}$, such that it is the least squares solution to

$$ \hat{\theta}_n = \arg\min_x \sum_{k=0}^n w[k]\,\|z[k] - \phi[k]^\top x\|_2^2, $$

where $w[k]$ are weights, $z[k]$ and $\phi[k]$ are known and $z[k]$ is assumed to be generated by using $\phi[k]^\top \theta$ with $\theta$ constant and unknown to the user. The recursive part of RLS refers to that the solution $\hat{\theta}_n$, plus additional variables that encode the problem, are used to calculate the solution $\hat{\theta}_{n+1}$. Now, if $\phi[k]$ satisfies a persistence of excitation condition, then over time $\hat{\theta}$ can be shown to converge to the value of $\theta$. A more detailed formulation and derivation can be seen in Mathematics StackExchnage - What's the Idea Behind Recursive Least Squares (RLS)? How Could It Be Derived?

So this formulation initially suggests that it would only be suited for estimating constant variables. However, by using exponentially decaying weights (so weigh old data exponentially less and less) RLS can still behave similar to a low-pass filter. So that the RLS can still reasonably track a slowly varying $\theta$.

It can also be shown that RLS (with constant weights) is equivalent to a Kalman filter for the system with $F_k = I$, $B_k = 0$, $H_k = \phi[k]^\top$, $Q_k=0$ and $R_k=1$. So if you want to filter out noise from a time varying signal and you have a model of how you think that signal should change over time, you might be better of using a Kalman filter using that knowledge. Namely, RLS with exponentially decaying weights will always lag behind a changing signal, similar to a low-pass filter, while a Kalman filter corrects for this using the prediction step.

Finally, regarding RLS being called a filter and not an observer might be because the label observer is I believe usually reserved for estimating properties of dynamical processes, which the constant $\theta$ which RLS tries to estimate is lacking. However, I am often also confused about when to use the term filter, estimator or observer, so difference in meaning might be small.

  • $\begingroup$ I think the term filter is used for when you reach the end of the updating sequence so that $t=T$. Then, the the estimates of $\phi_t$ that resulted at steps $t = 1,\ldots T$ can be thought of as the filtered states of the process.West and Harrison uses that filtered state terminology. It is definitely confusing and I have no idea what observer means. Thanks for great explanation above. $\endgroup$ – mark leeds Nov 1 '19 at 4:30
  • $\begingroup$ @markleeds I think that maybe the terminology used might depend on the field. Namely, my background is in control theory and there the term observer is very common (which as far as I know is equivalent to a state estimator, which requires a model of your system). $\endgroup$ – fibonatic Nov 1 '19 at 11:05
  • $\begingroup$ Thanks for the detailed answer, but I still don't get how to use RLS (or a Kalman filter) as a lowpass without having a model of the process in question. Or did you mean that without a model of the process, it is not possible to use these techniques? My problem is I only have access to $y$ (the system output), not to $u$ (the system input), so I cannot use the standard observer-like structure. $\endgroup$ – SampleTime Nov 1 '19 at 20:40
  • $\begingroup$ @SampleTime the model RLS assumes is basically that the parameter that you are trying to estimate $\theta$ is constant. This is thus equivalent to the model $x[k+1]=x[k]$, so $F_k=I$ and $B_k=0$. If you do have a model, but do no know the input then you might still be able to use a Kalman filter if the input changes slowly over time by adding the input as an extra state. If you even know that the input is of a certain shape, such as a sinusoidal with given frequency, you could even model that as an exogenous system. $\endgroup$ – fibonatic Nov 3 '19 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.