# Techniques to reject noisy neural network input

Suppose an artificial neural network is used to approximate a sine wave (shown in red in the graph below), given the linear input variable $$x$$ (scaled such that the ANN input is $$x_{\rm nn}\in[-1;1]$$).

The mean squared error between the target function (sine wave) and the trained ANN output is as shown approximately $$4.8\times10^{-3}$$, if the ANN structure is 1-8-1 (one input $$x_{\rm nn}$$, 8 hidden nodes and 1 output node). Now suppose the ANN structure is changed to 2-8-1, where the second input is normally distributed random noise. The output is then as shown below. The MSE has increased to approximately $$1.2\times10^{-2}$$ (using the exact same noise sequence as during training) or slightly larger if a new noise sequence is used when testing the network. The results above were obtained using standard gradient-descent type training with regularisation. All nodes in the hidden layer have $$\rm tanh()$$ activation functions and the output node is linear, as is standard with function approximation.

What methods are available to automatically remove the second input? That is, training methods allowing the ANN to automatically discover the fact that the second input is noise, and therefore not a useful input. (Therefore excluding all methods where the human programmer must train several different ANN architectures with different input combinations, and evaluate all of them to decide which inputs to use and which to exclude.)

This question extends to larger ANNs where it is difficult to see beforehand which inputs are useful. Ideally, all weights associated with nonsense inputs should automatically be forced to zero.

• What is the application? What is the classifier called to do? – A_A Sep 24 '18 at 14:58
• @A_A It is a function approximation (or regression) application. The idea is to estimate missing values in the target function; so the red signal in the graph would not be complete, there would be a few unknown values. – aslan Sep 24 '18 at 20:03
• I appreciate that but I don't get the reason for changing the structure to 2-8-1. What does the second input represent? Is it now an $f(x,y)$? – A_A Sep 24 '18 at 23:32
• The second input is unwanted, which should ideally be rejected by the ANN training algorithm. This is a toy problem to test and understand training methods; in real-world applications there may be 10 inputs of which e.g. only 6 are useful. Removing the 4 nonsense inputs would improve performance, but to identify which are useful and which not, will require testing all possible combinations of inputs by hand. Or maybe there is a training method to perform input selection automatically? – aslan Sep 25 '18 at 1:20
• Your question is a special case of the the general problem of feature selection. I could post this as an answer with a summary of relevant techniques, but I don't have the time and think the wiki article alone can probably get you moving in the right direction. Edit: lol I just noticed this question is very old. I don't know why it showed up on the homepage, maybe some new activity happened here. – jnez71 Aug 24 '20 at 5:02

If I understood correctly, you want the neural network to learn $$\hat{y} = \hat{f}(x,n)$$ where $$n \sim \mathcal{N}(\mu,\sigma)$$ is noise and $$x$$ is in the interval $$[-1,1]$$ by minimizing $$\frac{1}{N}\sum_{i=1}^N |y_i - \hat{y_i}|^2 = \frac{1}{N}\sum_{i=1}^N |f(x_i,n_i) - \hat{f}(x_i,n_i)|^2$$
And the function you are trying to learn is $$f(x,n) = sin(\pi x)$$, right?.
The network should learn to ignore the noise $$n$$ given that you provide it enough examples for training. That is, for different values of $$x$$, your training data should have a lot of examples with different values for $$n$$, all having the same target $$sin(\pi x)$$. If you show the network enough examples where the $$n$$ value is not useful for making the prediction, it should learn to minimize its effect on the output. The more training data you have, the better the network can learn to do this.