1
$\begingroup$

The most-used discrete-time RNN equations used in Deep Learning these days are those of Elman: enter image description here

I have seen two very different continuous-time version of these, with different justifications. The most "natural" to me are the leaky integrator onces (here sigmoid is replaced with tanh, but any activation function would do):

enter image description here

which is justified by saying that taylor expanding this equation with discretization step of 1 gives back the discrete time equations above (see this).

Another version is found on Wikipedia (notice how the weight is outside the activation function!)

enter image description here

and justified by saying

Note that, by the Shannon sampling theorem, discrete time recurrent neural networks can be viewed as continuous-time recurrent neural networks where the differential equations have transformed into equivalent difference equations. This transformation can be thought of as occurring after the post-synaptic node activation functions $y_{i}(t)$ have been low-pass filtered but prior to sampling.

Could someone explain this?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.