FIR Networks are a really great kind of Neural Networks. It allows very precise predictions of missing samples for example when ones wants to reconstruct missing signals or adaptive noise cancellations, etc
I'd like to understand how they work and a great paper about it is the one by WAN in 1993 : http://www.cslu.ogi.edu/publications/ps/wan93c.pdf
here is an example of a FIR connection, and the computation of the weight changes knowing weights $b()$ or $a()$, inputs $u(k)$, $x(k)$ and error $e(k)$
It is said in the paper that : as the signal has to be propagated through the network, the resulting error has to be back-propagated through the same network
here is a simple image explanation
so the goal is to find $\delta$(k) for a given neuron to be able to work backwards the gradient (the error)
here is the formula of the weights ($a()$, $b()$) updates: $$\bigtriangleup \bf w \rm ^l _{ij} = - \mu \delta^{l+1} _j (k) \cdot \bf a\rm^l_i(k) \ \ (1)$$
where
$$\delta^l_j(k) = -2e_j(k)f'(s^L_j(k)) \ \ (2)\normalsize \\\ if \ l = L \ (last \ layer) $$ and $$\delta^l_j(k) = f'(s^l_j(k))\cdot\sum_m \bf\delta\rm ^{l+1}_m(k)\cdot\bf w\rm ^{l+1}_{jm} \ \ (3) \normalsize \\\ if \ 1 \leq l \leq L -1$$
$f'$ is the derivate of the activation function. An important point is that $\bf w \rm ^l _{ij}$ and $\bf \delta$ are vectors, not scalars.
page 26 we see : $$\bf w\rm^l_{ij} = [ w^l_{i,j}(0), w^l_{i,j}(1), \ ... \ ,w^l_{i,j}(M^l)]$$
and the vector of delayed activation values is : $$ \bf a\rm ^l _i (k) = [a^l_i(k), a^l_i(k-1), \ ... \ , a^l_i(k - M^{l+1})]$$
So let's take the first step : it is the last layer L (filter b in this example) and we have the error $y(k) = e(k)$, the actual weights $b(0), b(1), b(2)$ and the signal values $\bf a\rm^l_i(k) = [u(k), u(k-1) ...]$
I'm trying to understand fully how to compute each weight change.
using (2) we first compute the weight change $$\bigtriangleup w_{b(2)} = - \mu \delta^{l+1} _j (k) \cdot \bf a\rm^l_i(k)$$ $$\bigtriangleup w_{b(2)} = 2 \mu e_j(k)f'(s^L_j(k)) \cdot u(k-2)$$
so we have the new weight
$$b(2) = b(2) + \bigtriangleup w$$
is this first assertion correct?
and if it is, i have trouble understanding what is the next step:
- do we have to multiply $e(k)$ by the new weigth $b(2)$ to get the error for the next backward step (at weight $b(1)$) ? so for b(1) we use $$\bigtriangleup w_{b(1)} = 2 \mu e_j(k)b(2)f'(s^L_j(k)) \cdot u(k-1)$$
- or do we have to use always the same value of e(k) for the next weight update : $b(1) = b(1) - \mu \delta^l_j(k) \cdot u(k-1)$ ?
for the second FIR filter there is a difference. Here we use eq (3) as it is not the last layer L.
in this simple case of two cascaded FIR filters, i think $(1)$ could be rewrote as:
$$\bigtriangleup \bf w \rm ^l = - \mu \delta^{L} (k) \cdot \bf x\rm(k)$$
where $\delta^{L} _j (k)$ is the previously calculated $\delta$ and $x(k) = [x(k), x(k-1), \ ... \ , x(k-M)]$ and as before we multiply the gradient $\delta$ by the updated weights and go backward to the start.
is it right? or am i wrong (and where)?
Also it would be very nice if ones could possibly put some general pseudo-code to compute the whole FIR network or point to some existing code so i can compare the algorithms.
Thanks for help
Jeff
\huge
markups? It looks awfully terrible and is impossible to read. Can you imagine reading a book written like that? $\endgroup$