# Linear model of the modified error function

We can write the error function ($E(w)=1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2$) as a linear model using its partial derivatives. Is it possible to do the same thing about the modified error function?

The modified error function is (page 10- pattern recognition by Bishop):

$$E(w)=1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2+\dfrac \lambda2||w||^2$$

• Are you essentially asking whether the regularization term is differentiable? Sep 28, 2012 at 16:29
• @Phonon: That's exactly what I meant. Sep 28, 2012 at 16:37
• Seems like you should be able to. Remember that you can write $||w||^2$ as $w^Hw$, so the via the chain rule the derivative would be $2w$. Or maybe I'm oversimplifying. Sep 28, 2012 at 16:58
• @JasonR: I don't think so. IIRC, $||w||^2=w_0^2+w_1^2+\dots+w_m^2$ but I am not sure about $\lambda$. That was my main motivation for asking. Sep 28, 2012 at 17:06
• Is $\lambda$ a function of $w$? If not, then it just linearly scales the partial derivative of $||w||^2$. Again, maybe I'm just not understanding the problem correctly. Sep 28, 2012 at 17:32

$$E(w)=1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2+\dfrac \lambda2||w||^2$$
\begin{align} \frac{\partial E(w)}{\partial w} &= \frac{\partial \left( 1/2\sum_{n=1}^{N}\{y(x_n,w)-t_n\}^2+\dfrac \lambda2||w||^2 \right)}{\partial w} \\ &= \frac{\partial \left( 1/2(y({\bf x},{\bf w} )-{\bf t})^T(y({\bf x},{\bf w} )-{\bf t})+\dfrac \lambda2{\bf w}^T{\bf w} \right)}{\partial {\bf w}} \\ &= \frac{\partial \left( 1/2(y({\bf x},{\bf w} )-{\bf t})^T(y({\bf x},{\bf w} )-{\bf t})\right)}{\partial {\bf w}}+\frac{\partial \left( \dfrac \lambda2{\bf w}^T{\bf w} \right)}{\partial {\bf w}} \\ &= \left( \frac{\partial y({\bf x},{\bf w} )}{\partial {\bf w} } \right)^T \left( y({\bf x},{\bf w} )-{\bf t} \right) + \lambda{ \bf w }\end{align}
@Phonon's answer is absolutely correct, and very worthy of the upvote I gave it. But I had to read it a few times to fully understand it, and I think the reason I was confused is that ${\bf w}$ has a different dimension from ${\bf x}$ and ${\bf t}$. In the book, the dimension of the former is $M$, and of the others is $N$.
As an exercise for myself, I tried to write @Phonon's solution using $\sum$ notation, instead of using vectors, just so I could understand it better. I'm posting the result just in case it's useful to @Gigili, or to anyone else reading this question.
\begin{align} E({\bf w})&=\frac 12\sum_{n=1}^{N}\{y(x_n,{\bf w})-t_n\}^2 +\frac \lambda2||{\bf w}||^2 \\ \Rightarrow \frac{\partial E({\bf w})}{\partial w_i} &= \frac{\partial \displaystyle \left( \frac 12\sum_{n=1}^N\{y(x_n,{\bf w})-t_n\}^2+\frac \lambda2||{\bf w}||^2 \right)}{\partial w_i} \\ &= \left( \frac 12 \sum_{n=1}^N \frac{\partial y(x_n, {\bf w})}{\partial w_i}\cdot 2\big( y(x_n, {\bf w})-t_n\big) \right) + \frac\lambda2(2w_i) \\ &= \left( \sum_{n=1}^N \frac{\partial y(x_n, {\bf w})}{\partial w_i}\big( y(x_n, {\bf w})-t_n\big) \right) + \lambda w_i \end{align}