0
$\begingroup$

Suppose an artificial neural network is used to approximate a sine wave (shown in red in the graph below), given the linear input variable $x$ (scaled such that the ANN input is $x_{\rm nn}\in[-1;1]$).

The mean squared error between the target function (sine wave) and the trained ANN output is as shown approximately $4.8\times10^{-3}$, if the ANN structure is 1-8-1 (one input $x_{\rm nn}$, 8 hidden nodes and 1 output node).

Sine wave training data and ANN output

Now suppose the ANN structure is changed to 2-8-1, where the second input is normally distributed random noise. The output is then as shown below. The MSE has increased to approximately $1.2\times10^{-2}$ (using the exact same noise sequence as during training) or slightly larger if a new noise sequence is used when testing the network.

Same sine wave training data, but with additional noise input

The results above were obtained using standard gradient-descent type training with regularisation. All nodes in the hidden layer have $\rm tanh()$ activation functions and the output node is linear, as is standard with function approximation.

What methods are available to automatically remove the second input? That is, training methods allowing the ANN to automatically discover the fact that the second input is noise, and therefore not a useful input. (Therefore excluding all methods where the human programmer must train several different ANN architectures with different input combinations, and evaluate all of them to decide which inputs to use and which to exclude.)

This question extends to larger ANNs where it is difficult to see beforehand which inputs are useful. Ideally, all weights associated with nonsense inputs should automatically be forced to zero.

$\endgroup$
  • $\begingroup$ What is the application? What is the classifier called to do? $\endgroup$ – A_A Sep 24 '18 at 14:58
  • $\begingroup$ @A_A It is a function approximation (or regression) application. The idea is to estimate missing values in the target function; so the red signal in the graph would not be complete, there would be a few unknown values. $\endgroup$ – aslan Sep 24 '18 at 20:03
  • $\begingroup$ I appreciate that but I don't get the reason for changing the structure to 2-8-1. What does the second input represent? Is it now an $f(x,y)$? $\endgroup$ – A_A Sep 24 '18 at 23:32
  • $\begingroup$ The second input is unwanted, which should ideally be rejected by the ANN training algorithm. This is a toy problem to test and understand training methods; in real-world applications there may be 10 inputs of which e.g. only 6 are useful. Removing the 4 nonsense inputs would improve performance, but to identify which are useful and which not, will require testing all possible combinations of inputs by hand. Or maybe there is a training method to perform input selection automatically? $\endgroup$ – aslan Sep 25 '18 at 1:20
0
$\begingroup$

It seems like your model is overfitting because you are not providing it with enough training data.

If I understood correctly, you want the neural network to learn $\hat{y} = \hat{f}(x,n)$ where $n \sim \mathcal{N}(\mu,\sigma)$ is noise and $x$ is in the interval $[-1,1]$ by minimizing $$ \frac{1}{N}\sum_{i=1}^N |y_i - \hat{y_i}|^2 = \frac{1}{N}\sum_{i=1}^N |f(x_i,n_i) - \hat{f}(x_i,n_i)|^2 $$

And the function you are trying to learn is $f(x,n) = sin(\pi x)$, right?.

The network should learn to ignore the noise $n$ given that you provide it enough examples for training. That is, for different values of $x$, your training data should have a lot of examples with different values for $n$, all having the same target $sin(\pi x)$. If you show the network enough examples where the $n$ value is not useful for making the prediction, it should learn to minimize its effect on the output. The more training data you have, the better the network can learn to do this.

$\endgroup$
  • $\begingroup$ Hi: this is an interesting question that I don't know the answer to. hulappa's answer make sense but I wonder if that's the only way. you may want to also look at the lasso because it has the property ( through a regularization approach) that coefficients of a model that are less significant tend to get set to zero. It's not an nn of course but sometimes it's best not to use a screwdriver for every screw or whatever that saying is. $\endgroup$ – mark leeds Dec 1 '18 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.