20 votes
Accepted

FIR Filter Design: Window vs Parks McClellan and Least Squares

I agree that the windowing filter design method is not one of the most important design methods anymore, and it might indeed be the case that it is overrepresented in traditional textbooks, probably ...
  • 82.9k
10 votes
Accepted

Estimation of the Amplitude of a Sine / Cosine Wave and Its Independence of the SNR / Amplitude of the Wave

One may have a look at the CRLB of estimating the parameters of a sine wave. The model for signal is given by: $$ x \left[ n \right] = A \cos \left( \omega n + \phi \right) + w \left[ n \right], \; n =...
  • 45.3k
9 votes
Accepted

Difference between Leaking Factor and Forgetting Factor

The play similar role in those algorithms - the ability to forget the past and adapt to current reality. In the LMS, the classic implementation has $ \alpha = 1 $. Namely the optimal weights at any ...
  • 45.3k
9 votes
Accepted

Quadratic Programming with Linear Equality Constraints

Let's solve a more general problem (Least Squares with Linear Equality Constraints): $$ \begin{alignat*}{3} \arg \min_{x} & \quad & \frac{1}{2} \left\| A x - b \right\|_{2}^{2} \\ \text{...
  • 45.3k
8 votes
Accepted

What's the Difference Between LMS and Gradient Descent Adaptation?

The LMS algorithm is based on the idea of gradient descent to search for the optimal (minimum error) condition, with a cost function equal to the mean squared error at the filter output. However, it ...
  • 24.2k
8 votes
Accepted

How to Use the DFT (FFT) to Solve a Least Squares Regularization Problem (Inverse Problem)?

The question really depends on $ f \left( \cdot \right) $. Yet in order to show how to use FFT we can even use 1D signals. Let's rewrite the problem: $$ \hat{x} = \arg \min_{x} \frac{1}{2} \left\| K ...
  • 45.3k
8 votes

Solving LASSO ($ {L}_{1} $ Regularized Least Squares) with Gradient Descent

It can easily solved by the Gradient Descent Framework with one adjustment in order to take care of the $ {L}_{1} $ norm term. Since the $ {L}_{1} $ norm isn't smooth you need to use the concept of ...
  • 45.3k
8 votes
Accepted

Jacobian Computation in Least Squares IIR Filter Design

The Jacobian is not computed numerically but analytically and then just evaluated. The frequency response of the IIR filter is $$H(e^{j\omega})=\frac{b_0+b_1e^{-j\omega}+\ldots+b_Me^{-jM\omega}}{1+...
  • 82.9k
7 votes

FIR Filter Design: Window vs Parks McClellan and Least Squares

I'll show here one benefit of a windowed design and a trick to get the same benefit from Parks–McClellan. For half-band, quarter-band etc. filters windowing retains the time-domain zeros of the ...
7 votes

FIR Filter Design: Window vs Parks McClellan and Least Squares

Windowed Sinc filters can be adaptively generated on the fly on processors barely powerful enough to run the associated FIR filter. Windowed Sinc filters can be generated in finite bounded time. The ...
  • 34.5k
7 votes
Accepted

Use MATLAB to Restore a Signal from a Given Degraded Signal Using Tikhonov Regularization

The idea is to represent all operation sing Matrices. Once it is done, it is easy to solve the problems as a Least Squares problems. The way to represent Convolution Operation using a Matrix is by ...
  • 45.3k
7 votes
Accepted

Least Angle Regression (LARS) without Matrix Inversion

If you want to solve for single value of $ \lambda $ in the model: $$ \arg \min_{x} \frac{1}{2} {\left\| A x - b \right\|}_{2}^{2} + \lambda {\left\| x \right\|}_{1} $$ Then you can use Coordinate ...
  • 45.3k
7 votes

What Is the Difference between RLS, LMS and Wiener Filter? When Is One Preferred Over Another?

All three are Estimators / Predictors. All of them try to estimate the coefficients of Linear Filter which minimizes an MMSE Cost Function. The Wiener filter assumes all data is given and sets the ...
  • 45.3k
7 votes

Python: Least Squares Support Vector Machine (LS-SVM)

There is a package called FukuML. In their description (Version 0.4.1) they write: Support Vector Machine Primal Hard Margin Support Vector Machine Binary Classification Learning Algorithm Dual ...
  • 45.3k
7 votes
Accepted

Why Is Non Linear Least Squares Method from MATLAB and Alglib Gives Different Results on the Same Data?

When you solve Non Linear Least Squares problem of a non convex cost function the end solution (Which is guaranteed to be a Local Minimum) will depend on: Method of Minimization. Method Parameters. ...
  • 45.3k
7 votes
Accepted

Sequential Form of the Least Squares Estimator for Linear Least Squares Model

Slope from all samples obtained To summarize the question's problem, you want to calculate the slope based on all samples obtained thus far, and as new samples are obtained, update the slope without ...
7 votes

Sequential Form of the Least Squares Estimator for Linear Least Squares Model

There are really great answers. I will try to give the Sequential Least Squares approach which generalizes to any Linear Model. Sequential Least Squares Model We're after solving the Linear Least ...
  • 45.3k
7 votes
Accepted

Questions on the Generalized Tikhonov Regularization

One way to interpret the Tikhonov Regularization is using the Maximum A Posteriori (MAP) framework. Lets' say we have a model of the form: $$ \boldsymbol{y} = H \boldsymbol{x} + \boldsymbol{n} $$ ...
  • 45.3k
7 votes

High Dynamic Range FIR Filters

The problem lies in the formulation of the desired response, and especially in the "don't care" region, which is extremely wide for the chosen filter length. Even though I can't give any ...
  • 82.9k
7 votes
Accepted

High Dynamic Range FIR Filters

Like @MattL. and @aconcernedcitizen say, the issue is numerical. Python's scipy.signal.firls uses internally the solver ...
6 votes

Looking for the Concept About All In One Curve Fitting

You can always augment the matrices to do so. Let's assume the first model is given by: $$ {y}_{1} = {H}_{1} * {\theta}_{1} $$ The second model is given by: $$ {y}_{2} = {H}_{2} * {\theta}_{2} $$ ...
  • 45.3k
6 votes
Accepted

What is the Concept of MATLAB Function Polynomial Interpolation?

It is basically an approach choice. Inside the math is identical. Usually, when doing Least Squares curve fitting, you're not looking for the Polynomial coefficients but a scaled version of them. For ...
  • 45.3k
6 votes

Least Mean Squares (LMS) Filter Weight Update

That really depends on context, but generally adaptive implies that the calculations are done on-line / on the fly. In some applications, the filter is updated for a while, then the adaptation is ...
  • 23.7k
6 votes

Least Squares with Non Zero Mean Noise

Since this is a linear model if you add noise which isn't centered (Non zero mean noise) your estimation will be good up to a bias term. The easy way to do so is to remove the bias from $ y $ and ...
  • 45.3k
6 votes

Least Mean Squares (LMS) Filter Weight Update

To expand on what Peter K. has said, if the signals being used by the filter are stationary, then the filter weights or coefficients can be determined and the filter operates as it was designed ...
6 votes
Accepted

Why Would Pre Filtering Measurement Data Affect the Least Squares Estimate?

I'm not sure what's you model is. Let's say it is something like: $$ y = H x + n $$ Now, using the Least Squares model is optimal (In the MSE sense) when $ n $ is AWGN (It is the linear optimal ...
  • 45.3k
6 votes
Accepted

Deriving the Matrix Inversion Lemma for RLS Equations vs the Woodbury Derivation

It is not clear what are you asking but I will try answer both things. Deriving the Matrix Inversion Lemma The Matrix Inversion Lemma goes as: $$ {\left( A + U C V \right)}^{-1} = {A}^{-1} - {A}^{-...
  • 45.3k
6 votes

The Least Norm Solution of Under Determined Linear System

Least Squares solution is always well defined for Linear System of Equations. In your case, which is under determined it means there are many solutions to the Linear Equations. The Least Squares ...
  • 45.3k
6 votes
Accepted

Finding the Best Gaussian Smoothing Kernel to Minimize the Discrepancy Between Two Images

This is closely related to Blind Deconvolution. The only difference is we limit our self to a very specific type of blur kernels. The nice thing about the Gaussian Kernel is being defined by single ...
  • 45.3k
6 votes
Accepted

Why Isn't the ML Estimator (MLE) in MIMO Spatial Multiplexing Obtained by the Least Squares Solution?

You must attention to the written text. It doesn't say the ML isn't optimal, what it says is that the problem isn't regular LS problem but Least squares problem with Constraints. The constraints ...
  • 45.3k

Only top scored, non community-wiki answers of a minimum length are eligible