# How to find the perfect regularization number?

I have an issue!

Assume that we are going to solve $$Ax=b$$ but $$x$$ contains noise.To minimize that noise we can use regularization:

$$x = (A^TA + \lambda I)^{-1}A^Tb$$

Where $$\lambda$$ is a small number $$> 0$$, or sometimes $$\lambda$$ can be huge!

Here are three pictures of one situation where I solve $$Ax=b$$ with noise free and one $$Ax=b$$ with noise and without regularization and one with noise + regularization.

Noise free case: Here we can see that it works smooth. In this case, $$x$$ is a matrix because it contains two signals.

Here we are using 5% noise, that will cause much trouble. No regularization.

Here we are using 5% noise and a large number for regularization.

Question:

Is there a way to determine an optimized regularization parameter $$\lambda$$?

Source: I made a library named Mataveid and it's made for system identification. I have no problem with the noise actually, but I want to find a way to automize the choice of regularization parameter $$\lambda$$.

Approaches can be any of the following:

1. Model the noise that you expect, for ex: gaussian (least squares in Minimum variance unbiased estimator for a linear signal model in presence of gaussian noise). Based on this model try and estimate the noise variance, the regularization term should be close to noise varainace.

2. Deploy machine learning techniques based on signal parameters such as dynamic range, frqeuency, FFT noise floor etc. to estimate noise

3. Follow a recursive approach with adjusting the regularization to minimize a suitable objective, ex: gradient descent etc.

In general your regularization is as good as how close it is to the actual noise or estimation of noise. This goes back to the idea of whether a frequentist approach is better or Bayesian approach. The answers is if the modelling of the parameter is accurate to it's actual behaviour then the Bayesian approach is better but if you model a parameter with a distribution but it is actually not following that closely then Bayesian would lead to suboptimal result and better go for a frequentist approach.

• Thank you! What machine learning technique to you recommend? – Daniel Mårtensson Apr 27 at 11:18
• So if the noise variance is 100, then my $\lambda$ need to be 100 as well? – Daniel Mårtensson Apr 27 at 11:19
• Neural networks are known to be good for such problems as far as I know, I am sure there would be some other simpler ones, neural networks might be an overkill. Regarding noise varainace being 100, think of it this way, what if you knew the noise varainace was 100, then the best you would do to find x is to get a lmmse estimator, the lmmse estimator would have exactly the same expression as you have in your question with $\lambda$ replaced by $\sigma^2$, so yes, you substitute the nosie variance, after all that is the amount of noise that has effected the signal – Dsp guy sam Apr 27 at 11:23
• So the same noise power should be used in estimating the signal.hope that answers your question – Dsp guy sam Apr 27 at 11:24
• Can you show me an example ? – Daniel Mårtensson Apr 27 at 15:40