2
$\begingroup$

We can write the error function ($E(w)=1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2$) as a linear model using its partial derivatives. Is it possible to do the same thing about the modified error function?

The modified error function is (page 10- pattern recognition by Bishop):

$$E(w)=1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2+\dfrac \lambda2||w||^2$$

$\endgroup$
7
  • 1
    $\begingroup$ Are you essentially asking whether the regularization term is differentiable? $\endgroup$
    – Phonon
    Sep 28, 2012 at 16:29
  • $\begingroup$ @Phonon: That's exactly what I meant. $\endgroup$
    – Gigili
    Sep 28, 2012 at 16:37
  • $\begingroup$ Seems like you should be able to. Remember that you can write $||w||^2$ as $w^Hw$, so the via the chain rule the derivative would be $2w$. Or maybe I'm oversimplifying. $\endgroup$
    – Jason R
    Sep 28, 2012 at 16:58
  • $\begingroup$ @JasonR: I don't think so. IIRC, $||w||^2=w_0^2+w_1^2+\dots+w_m^2$ but I am not sure about $\lambda$. That was my main motivation for asking. $\endgroup$
    – Gigili
    Sep 28, 2012 at 17:06
  • $\begingroup$ Is $\lambda$ a function of $w$? If not, then it just linearly scales the partial derivative of $||w||^2$. Again, maybe I'm just not understanding the problem correctly. $\endgroup$
    – Jason R
    Sep 28, 2012 at 17:32

2 Answers 2

4
$\begingroup$

Yes, you can do that.

$$E(w)=1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2+\dfrac \lambda2||w||^2$$

$$\begin{align} \frac{\partial E(w)}{\partial w} &= \frac{\partial \left( 1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2+\dfrac \lambda2||w||^2 \right)}{\partial w} \\ &= \frac{\partial \left( 1/2(y({\bf x},{\bf w} )-{\bf t})^T(y({\bf x},{\bf w} )-{\bf t})+\dfrac \lambda2{\bf w}^T{\bf w} \right)}{\partial {\bf w}} \\ &= \frac{\partial \left( 1/2(y({\bf x},{\bf w} )-{\bf t})^T(y({\bf x},{\bf w} )-{\bf t})\right)}{\partial {\bf w}}+\frac{\partial \left( \dfrac \lambda2{\bf w}^T{\bf w} \right)}{\partial {\bf w}} \\ &= \left( \frac{\partial y({\bf x},{\bf w} )}{\partial {\bf w} } \right)^T \left( y({\bf x},{\bf w} )-{\bf t} \right) + \lambda{ \bf w }\end{align}$$

$\endgroup$
2
$\begingroup$

@Phonon's answer is absolutely correct, and very worthy of the upvote I gave it. But I had to read it a few times to fully understand it, and I think the reason I was confused is that ${\bf w}$ has a different dimension from ${\bf x}$ and ${\bf t}$. In the book, the dimension of the former is $M$, and of the others is $N$.

As an exercise for myself, I tried to write @Phonon's solution using $\sum$ notation, instead of using vectors, just so I could understand it better. I'm posting the result just in case it's useful to @Gigili, or to anyone else reading this question.

$$ \begin{align} E({\bf w})&=\frac 12\sum_{n=1}^{N}\{y(x_n,{\bf w})-t_n\}^2 +\frac \lambda2||{\bf w}||^2 \\ \Rightarrow \frac{\partial E({\bf w})}{\partial w_i} &= \frac{\partial \displaystyle \left( \frac 12\sum_{n=1}^N\{y(x_n,{\bf w})-t_n\}^2+\frac \lambda2||{\bf w}||^2 \right)}{\partial w_i} \\ &= \left( \frac 12 \sum_{n=1}^N \frac{\partial y(x_n, {\bf w})}{\partial w_i}\cdot 2\big( y(x_n, {\bf w})-t_n\big) \right) + \frac\lambda2(2w_i) \\ &= \left( \sum_{n=1}^N \frac{\partial y(x_n, {\bf w})}{\partial w_i}\big( y(x_n, {\bf w})-t_n\big) \right) + \lambda w_i \end{align} $$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.