Neural networks in system identification - What type of activation functions?

Question

I made a free software for all operative systems, even Android. It's called Deeplearning2C. It can train a neural network and generate C code and MATLAB-code. C-code for embedded systems and MATLAB-code for simulation.

I tried a example when I trained a neural network with linear data with constraints = nonlinear model in other words. It works very well. Here is the result when I use identity as activation functions.

I identification the model from the features (inputs)

$$-y(t-1), -y(t-2), ... , -y(t-n), u(t), u(t-1), ... , u(t-m)$$

And the label (output) $y(t)$. This is the regular transfer function or ordinary differential equation identification method.

But the issue I have is that I have a lack of experience of system identification when it comes to neural networks. I'm used to state space models (Subspace identification) and regular parameter estimation (Recursive Least Square). I wrote a library for that too.

My questions for you are that:

1. What type of activation functions should I use?
2. How many layers is "good" enough?
3. How many neurons should I have?
4. How do I know...how to know what to select? I'm seeking practical experience for identification with neural networks.

Here is my software Deeplearning2C. It's using Deeplearning4J as the core, but the user interface is made by me.

Also here is the MATLAB/GNU Octave library for linear system identification with recursive least square and subspace identification. I wrote a C-library too for recursive least square identification as well.

Answer 1

Looks like you've done a lot of work on your projects. As @MarcusMüller said, by far the majority of people start with ReLU and go from there. It doesn't have the "vanishing gradient" problem that tanh has for example. All your questions are open ended but common for designing neural networks. There are so many "nobs to turn" to try and make your network be the best it can be.

You could adjust some parameters, train, test, and compare. But then days or weeks have gone by and you'll still be tuning your parameters. One popular method to choosing "good" parameters is called hyper parameter optimization. This blog post does a good job explaining it: https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/. The idea is that you randomly sample your parameters space (for example, number of neurons, a few different activation functions, different learning rates, number of epochs, etc.) and in search for the a good set of parameters. I hope this helps!

Answer 2

I kinda differ with the established answer. The first question is really what is the size of your dataset. In most cases, system identification problems don't have huge amounts of data. Some problems might have a lot of measurements--like IOT sensors, etc. But generally system identification is estimation the form of some unknown function with 100, or 1000, or even 10,000 measurements. So the data is small. My comments below are based upon the assumption of reasonably sized datasets--considering that the image above is of a 1 dimensional scalar valued function.

In a lot of the Physics Informed Neural Networks (PINN) community, researchers work on problems similar to the system identification problem in the OP. So when you have such reasonably sized datasets, then you don't need to worry about mini-batching and stochastic gradient descent. RELUs work well when you have mini-batching and huge amounts of data because they avoid saturation. But in the case of smaller data where the entire dataset can be used in each step of the optimization routine, you don't have to use RELU.

Most papers that I have seen in this space will use a tanh activation function, with L-BFGS optimizer that runs over the entire dataset--no mini-batching. Training the number of nodes and layers is a hyperparameter optimization exercise, but you could probably get away with say 10 layers with 50 neurons max. It is probbably going to be better to start with smaller networks and work up to the max, as bigger networks can give less stable results.

Those are just my thoughts on the OP.