# Neural networks in system identification - What type of activation functions?

I made a free software for all operative systems, even Android. It's called Deeplearning2C. It can train a neural network and generate C code and MATLAB-code. C-code for embedded systems and MATLAB-code for simulation.

I tried a example when I trained a neural network with linear data with constraints = nonlinear model in other words. It works very well. Here is the result when I use identity as activation functions. I identification the model from the features (inputs)

$$-y(t-1), -y(t-2), ... , -y(t-n), u(t), u(t-1), ... , u(t-m)$$

And the label (output) $$y(t)$$. This is the regular transfer function or ordinary differential equation identification method.

But the issue I have is that I have a lack of experience of system identification when it comes to neural networks. I'm used to state space models (Subspace identification) and regular parameter estimation (Recursive Least Square). I wrote a library for that too.

My questions for you are that:

1. What type of activation functions should I use?
2. How many layers is "good" enough?
3. How many neurons should I have?
4. How do I know...how to know what to select? I'm seeking practical experience for identification with neural networks.

Here is my software Deeplearning2C. It's using Deeplearning4J as the core, but the user interface is made by me.

Also here is the MATLAB/GNU Octave library for linear system identification with recursive least square and subspace identification. I wrote a C-library too for recursive least square identification as well.

• "when I use no activation function", um, that makes no sense. What do you mean with that? – Marcus Müller Nov 14 '19 at 1:27
• @Marcusmuller i did not use any activation function when I trained a model? Like a scalar 1. – Daniel Mårtensson Nov 14 '19 at 7:03
• Sorry, I don't really understand that. Could you elaborate? What did you use the "1" for? – Marcus Müller Nov 14 '19 at 10:09
• @MarcusMüller It's called identity activation function. – Daniel Mårtensson Nov 14 '19 at 17:11
• ahhh you MULTIPLY the input with 1 – you could have said "identity" :) Anyways, now you've just found a matrix, i.e. a linear function, and with that you can't fulfill the universal approximation theorem. You need a nonlinear activation function. Classically (and basically ALL literature will point that out) that's been $\tanh$, but nowadays we just use RELU, as that works just about as well, but is way easier to calculate. – Marcus Müller Nov 14 '19 at 17:17