19
votes
Accepted
FIR Filter Design: Window vs Parks McClellan and Least Squares
I agree that the windowing filter design method is not one of the most important design methods anymore, and it might indeed be the case that it is overrepresented in traditional textbooks, probably ...
- 80.4k
9
votes
Accepted
Quadratic Programming with Linear Equality Constraints
Let's solve a more general problem (Least Squares with Linear Equality Constraints):
$$
\begin{alignat*}{3}
\arg \min_{x} & \quad & \frac{1}{2} \left\| A x - b \right\|_{2}^{2} \\
\text{...
- 41.3k
8
votes
Accepted
What's the Difference Between LMS and Gradient Descent Adaptation?
The LMS algorithm is based on the idea of gradient descent to search for the optimal (minimum error) condition, with a cost function equal to the mean squared error at the filter output. However, it ...
- 23.7k
8
votes
Accepted
Difference between Leaking Factor and Forgetting Factor
The play similar role in those algorithms - the ability to forget the past and adapt to current reality.
In the LMS, the classic implementation has $ \alpha = 1 $.
Namely the optimal weights at any ...
- 41.3k
7
votes
FIR Filter Design: Window vs Parks McClellan and Least Squares
I'll show here one benefit of a windowed design and a trick to get the same benefit from Parks–McClellan.
For half-band, quarter-band etc. filters windowing retains the time-domain zeros of the ...
- 12.5k
7
votes
FIR Filter Design: Window vs Parks McClellan and Least Squares
Windowed Sinc filters can be adaptively generated on the fly on processors barely powerful enough to run the associated FIR filter. Windowed Sinc filters can be generated in finite bounded time.
The ...
- 34k
7
votes
Accepted
Use MATLAB to Restore a Signal from a Given Degraded Signal Using Tikhonov Regularization
The idea is to represent all operation sing Matrices.
Once it is done, it is easy to solve the problems as a Least Squares problems.
The way to represent Convolution Operation using a Matrix is by ...
- 41.3k
7
votes
Accepted
How to Use the DFT (FFT) to Solve a Least Squares Regularization Problem (Inverse Problem)?
The question really depends on $ f \left( \cdot \right) $.
Yet in order to show how to use FFT we can even use 1D signals.
Let's rewrite the problem:
$$ \hat{x} = \arg \min_{x} \frac{1}{2} \left\| K ...
- 41.3k
7
votes
Solving LASSO ($ {L}_{1} $ Regularized Least Squares) with Gradient Descent
It can easily solved by the Gradient Descent Framework with one adjustment in order to take care of the $ {L}_{1} $ norm term.
Since the $ {L}_{1} $ norm isn't smooth you need to use the concept of ...
- 41.3k
7
votes
Accepted
Sequential Form of the Least Squares Estimator for Linear Least Squares Model
Slope from all samples obtained
To summarize the question's problem, you want to calculate the slope based on all samples obtained thus far, and as new samples are obtained, update the slope without ...
- 12.5k
7
votes
Accepted
Questions on the Generalized Tikhonov Regularization
One way to interpret the Tikhonov Regularization is using the Maximum A Posteriori (MAP) framework.
Lets' say we have a model of the form:
$$ \boldsymbol{y} = H \boldsymbol{x} + \boldsymbol{n} $$
...
- 41.3k
6
votes
Accepted
Least Angle Regression (LARS) without Matrix Inversion
If you want to solve for single value of $ \lambda $ in the model:
$$ \arg \min_{x} \frac{1}{2} {\left\| A x - b \right\|}_{2}^{2} + \lambda {\left\| x \right\|}_{1} $$
Then you can use Coordinate ...
- 41.3k
6
votes
Looking for the Concept About All In One Curve Fitting
You can always augment the matrices to do so.
Let's assume the first model is given by:
$$ {y}_{1} = {H}_{1} * {\theta}_{1} $$
The second model is given by:
$$ {y}_{2} = {H}_{2} * {\theta}_{2} $$
...
- 41.3k
6
votes
Accepted
What is the Concept of MATLAB Function Polynomial Interpolation?
It is basically an approach choice.
Inside the math is identical.
Usually, when doing Least Squares curve fitting, you're not looking for the Polynomial coefficients but a scaled version of them.
For ...
- 41.3k
6
votes
Least Mean Squares (LMS) Filter Weight Update
That really depends on context, but generally adaptive implies that the calculations are done on-line / on the fly.
In some applications, the filter is updated for a while, then the adaptation is ...
- 22.9k
6
votes
Least Squares with Non Zero Mean Noise
Since this is a linear model if you add noise which isn't centered (Non zero mean noise) your estimation will be good up to a bias term.
The easy way to do so is to remove the bias from $ y $ and ...
- 41.3k
6
votes
Least Mean Squares (LMS) Filter Weight Update
To expand on what Peter K. has said, if the signals being used by the filter are stationary, then the filter weights or coefficients can be determined and the filter operates as it was designed ...
- 414
6
votes
What Is the Difference between RLS, LMS and Wiener Filter? When Is One Preferred Over Another?
All three are Estimators / Predictors.
All of them try to estimate the coefficients of Linear Filter which minimizes an MMSE Cost Function.
The Wiener filter assumes all data is given and sets the ...
- 41.3k
6
votes
Python: Least Squares Support Vector Machine (LS-SVM)
There is a package called FukuML.
In their description (Version 0.4.1) they write:
Support Vector Machine
Primal Hard Margin Support Vector Machine Binary Classification Learning Algorithm
Dual ...
- 41.3k
6
votes
Accepted
Why Is Non Linear Least Squares Method from MATLAB and Alglib Gives Different Results on the Same Data?
When you solve Non Linear Least Squares problem of a non convex cost function the end solution (Which is guaranteed to be a Local Minimum) will depend on:
Method of Minimization.
Method Parameters.
...
- 41.3k
6
votes
Sequential Form of the Least Squares Estimator for Linear Least Squares Model
There are really great answers.
I will try to give the Sequential Least Squares approach which generalizes to any Linear Model.
Sequential Least Squares Model
We're after solving the Linear Least ...
- 41.3k
6
votes
Accepted
Jacobian Computation in Least Squares IIR Filter Design
The Jacobian is not computed numerically but analytically and then just evaluated. The frequency response of the IIR filter is
$$H(e^{j\omega})=\frac{b_0+b_1e^{-j\omega}+\ldots+b_Me^{-jM\omega}}{1+...
- 80.4k
6
votes
Accepted
Image Restoration by Solving Constrained Least squares in Frequency Domain (Frequency Domain Filtering)
When dealing with applying a 2D convolution in frequency domain we have to take into account 2 things:
Extending the kernel to the dimension of the input data.
Dealing with the implicit periodic ...
- 41.3k
6
votes
High Dynamic Range FIR Filters
The problem lies in the formulation of the desired response, and especially in the "don't care" region, which is extremely wide for the chosen filter length. Even though I can't give any ...
- 80.4k
5
votes
Accepted
Why Would Pre Filtering Measurement Data Affect the Least Squares Estimate?
I'm not sure what's you model is.
Let's say it is something like:
$$ y = H x + n $$
Now, using the Least Squares model is optimal (In the MSE sense) when $ n $ is AWGN (It is the linear optimal ...
- 41.3k
5
votes
Accepted
Deriving the Matrix Inversion Lemma for RLS Equations vs the Woodbury Derivation
It is not clear what are you asking but I will try answer both things.
Deriving the Matrix Inversion Lemma
The Matrix Inversion Lemma goes as:
$$ {\left( A + U C V \right)}^{-1} = {A}^{-1} - {A}^{-...
- 41.3k
5
votes
Zero Phase Filter: Determining Initial Conditions for Forward Backward Filtering
For anyone who is interested, i coincidentally found a paper describing the method implemented in matlab's filtfilt.m. A link to the paper is attached. At least to my understanding matlab's filtfilt.m ...
- 367
5
votes
Is the Kalman Filter a Best Linear Unbiased Estimator (BLUE) for Heteroscedastic Noise?
Kalman filter is the best linear estimator regardless of stationarity or Gaussianity. Also in the Gaussian case it does not require stationarity (unlike Wiener filter). In the linear Gaussian case ...
- 590
5
votes
The Least Norm Solution of Under Determined Linear System
Least Squares solution is always well defined for Linear System of Equations.
In your case, which is under determined it means there are many solutions to the Linear Equations.
The Least Squares ...
- 41.3k
Only top scored, non community-wiki answers of a minimum length are eligible