0
$\begingroup$

2.2 Tikhonov Reqularization

Tikhonov regularization, named for Russian mathematician Andrey Tikhonov, attempts to fix the issue that arises when the least squares method is used with an ill-posed inverse problem by adding an additional constraint on the minimization: $$\min_x\{\Vert Ax-b \Vert ^2_2 + \lambda^2\Vert x\Vert^2_2\}$$ In the new minimization, the additional constraint is the squared norm of $x.$ $\lambda$ is a non-negative constant decided in advance, acting as a weight on the strength of the additional constraint. Some alternate ways to write Tikhonov regularization include $$\bbox[lightgreen]{(A^T A + \lambda^2 I)x=A^{T}b}$$ and $$\min \left\Vert\genfrac{[}{]}{0pt}{}{A}{\lambda I}x - \genfrac{[}{]}{0pt}{}{b}{0}\right\Vert$$

Can someone explain the alternate formulation in green. I am not able to convert this formulation to standard one. I get one Atransposedinverse extra. The second formulation can be easily converted to original formulation, though. Thanks.

$\endgroup$
1
$\begingroup$

Just expand the term to be minimized, take the derivative with respect to $x$ and set it equal to zero:

$$\begin{align}||Ax-b||_2^2+\lambda^2||x||_2^2&=(Ax-b)^T(Ax-b)+\lambda^2x^Tx\\&=x^TA^TAx-2b^TAx+b^Tb+\lambda^2x^Tx\tag{1}\end{align}$$

Taking the derivative w.r.t. $x$ and setting it equal to zero gives

$$2A^TAx-2A^Tb+2\lambda^2x=0\tag{2}$$

from which the given equation follows.

$\endgroup$
  • 1
    $\begingroup$ I didn't notice that there is no min in that formulation unlike others, so effectively it was closed form solution. Thanks Matt. $\endgroup$ – Sukuya Feb 22 '18 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.