# Neural networks in system identification - What type of activation functions?

I made a free software for all operative systems, even Android. It's called Deeplearning2C. It can train a neural network and generate C code and MATLAB-code. C-code for embedded systems and MATLAB-code for simulation.

I tried a example when I trained a neural network with linear data with constraints = nonlinear model in other words. It works very well. Here is the result when I use identity as activation functions.

I identification the model from the features (inputs)

$$-y(t-1), -y(t-2), ... , -y(t-n), u(t), u(t-1), ... , u(t-m)$$

And the label (output) $$y(t)$$. This is the regular transfer function or ordinary differential equation identification method.

But the issue I have is that I have a lack of experience of system identification when it comes to neural networks. I'm used to state space models (Subspace identification) and regular parameter estimation (Recursive Least Square). I wrote a library for that too.

My questions for you are that:

1. What type of activation functions should I use?
2. How many layers is "good" enough?
3. How many neurons should I have?
4. How do I know...how to know what to select? I'm seeking practical experience for identification with neural networks.

Here is my software Deeplearning2C. It's using Deeplearning4J as the core, but the user interface is made by me.

Also here is the MATLAB/GNU Octave library for linear system identification with recursive least square and subspace identification. I wrote a C-library too for recursive least square identification as well.

• "when I use no activation function", um, that makes no sense. What do you mean with that? – Marcus Müller Nov 14 '19 at 1:27
• @Marcusmuller i did not use any activation function when I trained a model? Like a scalar 1. – Daniel Mårtensson Nov 14 '19 at 7:03
• Sorry, I don't really understand that. Could you elaborate? What did you use the "1" for? – Marcus Müller Nov 14 '19 at 10:09
• @MarcusMüller It's called identity activation function. – Daniel Mårtensson Nov 14 '19 at 17:11
• ahhh you MULTIPLY the input with 1 – you could have said "identity" :) Anyways, now you've just found a matrix, i.e. a linear function, and with that you can't fulfill the universal approximation theorem. You need a nonlinear activation function. Classically (and basically ALL literature will point that out) that's been $\tanh$, but nowadays we just use RELU, as that works just about as well, but is way easier to calculate. – Marcus Müller Nov 14 '19 at 17:17

## 1 Answer

Looks like you've done a lot of work on your projects. As @MarcusMüller said, by far the majority of people start with ReLU and go from there. It doesn't have the "vanishing gradient" problem that tanh has for example. All your questions are open ended but common for designing neural networks. There are so many "nobs to turn" to try and make your network be the best it can be.

You could adjust some parameters, train, test, and compare. But then days or weeks have gone by and you'll still be tuning your parameters. One popular method to choosing "good" parameters is called hyper parameter optimization. This blog post does a good job explaining it: https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/. The idea is that you randomly sample your parameters space (for example, number of neurons, a few different activation functions, different learning rates, number of epochs, etc.) and in search for the a good set of parameters. I hope this helps!

• Thank you! Yes, I have done lots of work in system identification. I'm seeking practical applications only and not experimental theory. – Daniel Mårtensson Nov 15 '19 at 21:42