19 votes
Accepted

FIR Filter Design: Window vs Parks McClellan and Least Squares

I agree that the windowing filter design method is not one of the most important design methods anymore, and it might indeed be the case that it is overrepresented in traditional textbooks, probably ...
user avatar
  • 80.4k
9 votes
Accepted

Quadratic Programming with Linear Equality Constraints

Let's solve a more general problem (Least Squares with Linear Equality Constraints): $$ \begin{alignat*}{3} \arg \min_{x} & \quad & \frac{1}{2} \left\| A x - b \right\|_{2}^{2} \\ \text{...
user avatar
  • 41.3k
8 votes
Accepted

What's the Difference Between LMS and Gradient Descent Adaptation?

The LMS algorithm is based on the idea of gradient descent to search for the optimal (minimum error) condition, with a cost function equal to the mean squared error at the filter output. However, it ...
user avatar
  • 23.7k
8 votes
Accepted

Difference between Leaking Factor and Forgetting Factor

The play similar role in those algorithms - the ability to forget the past and adapt to current reality. In the LMS, the classic implementation has $ \alpha = 1 $. Namely the optimal weights at any ...
user avatar
  • 41.3k
7 votes

FIR Filter Design: Window vs Parks McClellan and Least Squares

I'll show here one benefit of a windowed design and a trick to get the same benefit from Parks–McClellan. For half-band, quarter-band etc. filters windowing retains the time-domain zeros of the ...
user avatar
7 votes

FIR Filter Design: Window vs Parks McClellan and Least Squares

Windowed Sinc filters can be adaptively generated on the fly on processors barely powerful enough to run the associated FIR filter. Windowed Sinc filters can be generated in finite bounded time. The ...
user avatar
  • 34k
7 votes
Accepted

Use MATLAB to Restore a Signal from a Given Degraded Signal Using Tikhonov Regularization

The idea is to represent all operation sing Matrices. Once it is done, it is easy to solve the problems as a Least Squares problems. The way to represent Convolution Operation using a Matrix is by ...
user avatar
  • 41.3k
7 votes
Accepted

How to Use the DFT (FFT) to Solve a Least Squares Regularization Problem (Inverse Problem)?

The question really depends on $ f \left( \cdot \right) $. Yet in order to show how to use FFT we can even use 1D signals. Let's rewrite the problem: $$ \hat{x} = \arg \min_{x} \frac{1}{2} \left\| K ...
user avatar
  • 41.3k
7 votes

Solving LASSO ($ {L}_{1} $ Regularized Least Squares) with Gradient Descent

It can easily solved by the Gradient Descent Framework with one adjustment in order to take care of the $ {L}_{1} $ norm term. Since the $ {L}_{1} $ norm isn't smooth you need to use the concept of ...
user avatar
  • 41.3k
7 votes
Accepted

Sequential Form of the Least Squares Estimator for Linear Least Squares Model

Slope from all samples obtained To summarize the question's problem, you want to calculate the slope based on all samples obtained thus far, and as new samples are obtained, update the slope without ...
user avatar
7 votes
Accepted

Questions on the Generalized Tikhonov Regularization

One way to interpret the Tikhonov Regularization is using the Maximum A Posteriori (MAP) framework. Lets' say we have a model of the form: $$ \boldsymbol{y} = H \boldsymbol{x} + \boldsymbol{n} $$ ...
user avatar
  • 41.3k
6 votes
Accepted

Least Angle Regression (LARS) without Matrix Inversion

If you want to solve for single value of $ \lambda $ in the model: $$ \arg \min_{x} \frac{1}{2} {\left\| A x - b \right\|}_{2}^{2} + \lambda {\left\| x \right\|}_{1} $$ Then you can use Coordinate ...
user avatar
  • 41.3k
6 votes

Looking for the Concept About All In One Curve Fitting

You can always augment the matrices to do so. Let's assume the first model is given by: $$ {y}_{1} = {H}_{1} * {\theta}_{1} $$ The second model is given by: $$ {y}_{2} = {H}_{2} * {\theta}_{2} $$ ...
user avatar
  • 41.3k
6 votes
Accepted

What is the Concept of MATLAB Function Polynomial Interpolation?

It is basically an approach choice. Inside the math is identical. Usually, when doing Least Squares curve fitting, you're not looking for the Polynomial coefficients but a scaled version of them. For ...
user avatar
  • 41.3k
6 votes

Least Mean Squares (LMS) Filter Weight Update

That really depends on context, but generally adaptive implies that the calculations are done on-line / on the fly. In some applications, the filter is updated for a while, then the adaptation is ...
user avatar
  • 22.9k
6 votes

Least Squares with Non Zero Mean Noise

Since this is a linear model if you add noise which isn't centered (Non zero mean noise) your estimation will be good up to a bias term. The easy way to do so is to remove the bias from $ y $ and ...
user avatar
  • 41.3k
6 votes

Least Mean Squares (LMS) Filter Weight Update

To expand on what Peter K. has said, if the signals being used by the filter are stationary, then the filter weights or coefficients can be determined and the filter operates as it was designed ...
user avatar
6 votes

What Is the Difference between RLS, LMS and Wiener Filter? When Is One Preferred Over Another?

All three are Estimators / Predictors. All of them try to estimate the coefficients of Linear Filter which minimizes an MMSE Cost Function. The Wiener filter assumes all data is given and sets the ...
user avatar
  • 41.3k
6 votes

Python: Least Squares Support Vector Machine (LS-SVM)

There is a package called FukuML. In their description (Version 0.4.1) they write: Support Vector Machine Primal Hard Margin Support Vector Machine Binary Classification Learning Algorithm Dual ...
user avatar
  • 41.3k
6 votes
Accepted

Why Is Non Linear Least Squares Method from MATLAB and Alglib Gives Different Results on the Same Data?

When you solve Non Linear Least Squares problem of a non convex cost function the end solution (Which is guaranteed to be a Local Minimum) will depend on: Method of Minimization. Method Parameters. ...
user avatar
  • 41.3k
6 votes

Sequential Form of the Least Squares Estimator for Linear Least Squares Model

There are really great answers. I will try to give the Sequential Least Squares approach which generalizes to any Linear Model. Sequential Least Squares Model We're after solving the Linear Least ...
user avatar
  • 41.3k
6 votes
Accepted

Jacobian Computation in Least Squares IIR Filter Design

The Jacobian is not computed numerically but analytically and then just evaluated. The frequency response of the IIR filter is $$H(e^{j\omega})=\frac{b_0+b_1e^{-j\omega}+\ldots+b_Me^{-jM\omega}}{1+...
user avatar
  • 80.4k
6 votes
Accepted

Image Restoration by Solving Constrained Least squares in Frequency Domain (Frequency Domain Filtering)

When dealing with applying a 2D convolution in frequency domain we have to take into account 2 things: Extending the kernel to the dimension of the input data. Dealing with the implicit periodic ...
user avatar
  • 41.3k
6 votes

High Dynamic Range FIR Filters

The problem lies in the formulation of the desired response, and especially in the "don't care" region, which is extremely wide for the chosen filter length. Even though I can't give any ...
user avatar
  • 80.4k
5 votes
Accepted

Why Would Pre Filtering Measurement Data Affect the Least Squares Estimate?

I'm not sure what's you model is. Let's say it is something like: $$ y = H x + n $$ Now, using the Least Squares model is optimal (In the MSE sense) when $ n $ is AWGN (It is the linear optimal ...
user avatar
  • 41.3k
5 votes
Accepted

Deriving the Matrix Inversion Lemma for RLS Equations vs the Woodbury Derivation

It is not clear what are you asking but I will try answer both things. Deriving the Matrix Inversion Lemma The Matrix Inversion Lemma goes as: $$ {\left( A + U C V \right)}^{-1} = {A}^{-1} - {A}^{-...
user avatar
  • 41.3k
5 votes

Zero Phase Filter: Determining Initial Conditions for Forward Backward Filtering

For anyone who is interested, i coincidentally found a paper describing the method implemented in matlab's filtfilt.m. A link to the paper is attached. At least to my understanding matlab's filtfilt.m ...
user avatar
5 votes

Is the Kalman Filter a Best Linear Unbiased Estimator (BLUE) for Heteroscedastic Noise?

Kalman filter is the best linear estimator regardless of stationarity or Gaussianity. Also in the Gaussian case it does not require stationarity (unlike Wiener filter). In the linear Gaussian case ...
user avatar
  • 590
5 votes

The Least Norm Solution of Under Determined Linear System

Least Squares solution is always well defined for Linear System of Equations. In your case, which is under determined it means there are many solutions to the Linear Equations. The Least Squares ...
user avatar
  • 41.3k

Only top scored, non community-wiki answers of a minimum length are eligible