# Techniques to reject noisy neural network input

Suppose an artificial neural network is used to approximate a sine wave (shown in red in the graph below), given the linear input variable $$x$$ (scaled such that the ANN input is $$x_{\rm nn}\in[-1;1]$$).

The mean squared error between the target function (sine wave) and the trained ANN output is as shown approximately $$4.8\times10^{-3}$$, if the ANN structure is 1-8-1 (one input $$x_{\rm nn}$$, 8 hidden nodes and 1 output node).

Now suppose the ANN structure is changed to 2-8-1, where the second input is normally distributed random noise. The output is then as shown below. The MSE has increased to approximately $$1.2\times10^{-2}$$ (using the exact same noise sequence as during training) or slightly larger if a new noise sequence is used when testing the network.

The results above were obtained using standard gradient-descent type training with regularisation. All nodes in the hidden layer have $$\rm tanh()$$ activation functions and the output node is linear, as is standard with function approximation.

What methods are available to automatically remove the second input? That is, training methods allowing the ANN to automatically discover the fact that the second input is noise, and therefore not a useful input. (Therefore excluding all methods where the human programmer must train several different ANN architectures with different input combinations, and evaluate all of them to decide which inputs to use and which to exclude.)

This question extends to larger ANNs where it is difficult to see beforehand which inputs are useful. Ideally, all weights associated with nonsense inputs should automatically be forced to zero.

• What is the application? What is the classifier called to do?
– A_A
Sep 24 '18 at 14:58
• @A_A It is a function approximation (or regression) application. The idea is to estimate missing values in the target function; so the red signal in the graph would not be complete, there would be a few unknown values. Sep 24 '18 at 20:03
• I appreciate that but I don't get the reason for changing the structure to 2-8-1. What does the second input represent? Is it now an $f(x,y)$?
– A_A
Sep 24 '18 at 23:32
• The second input is unwanted, which should ideally be rejected by the ANN training algorithm. This is a toy problem to test and understand training methods; in real-world applications there may be 10 inputs of which e.g. only 6 are useful. Removing the 4 nonsense inputs would improve performance, but to identify which are useful and which not, will require testing all possible combinations of inputs by hand. Or maybe there is a training method to perform input selection automatically? Sep 25 '18 at 1:20
• Your question is a special case of the the general problem of feature selection. I could post this as an answer with a summary of relevant techniques, but I don't have the time and think the wiki article alone can probably get you moving in the right direction. Edit: lol I just noticed this question is very old. I don't know why it showed up on the homepage, maybe some new activity happened here. Aug 24 '20 at 5:02

It seems like your model is overfitting because you are not providing it with enough training data.

If I understood correctly, you want the neural network to learn $$\hat{y} = \hat{f}(x,n)$$ where $$n \sim \mathcal{N}(\mu,\sigma)$$ is noise and $$x$$ is in the interval $$[-1,1]$$ by minimizing $$\frac{1}{N}\sum_{i=1}^N |y_i - \hat{y_i}|^2 = \frac{1}{N}\sum_{i=1}^N |f(x_i,n_i) - \hat{f}(x_i,n_i)|^2$$

And the function you are trying to learn is $$f(x,n) = sin(\pi x)$$, right?.

The network should learn to ignore the noise $$n$$ given that you provide it enough examples for training. That is, for different values of $$x$$, your training data should have a lot of examples with different values for $$n$$, all having the same target $$sin(\pi x)$$. If you show the network enough examples where the $$n$$ value is not useful for making the prediction, it should learn to minimize its effect on the output. The more training data you have, the better the network can learn to do this.

• Hi: this is an interesting question that I don't know the answer to. hulappa's answer make sense but I wonder if that's the only way. you may want to also look at the lasso because it has the property ( through a regularization approach) that coefficients of a model that are less significant tend to get set to zero. It's not an nn of course but sometimes it's best not to use a screwdriver for every screw or whatever that saying is. Dec 1 '18 at 22:31
• This answer doesn't really address the problem in any satisfactory way and mostly just restates the details from the OP. Aug 23 '20 at 13:23
• The OP is asking for ANN training methods, which would automatically force the weights associated with the noise inputs to zero. The answer is reminding the OP of the non-trivial fact that gradient descent minimising the MSE on this problem should automatically force the noise to zero, given that there is enough data. Whether this is an acceptable answer depends on the constraints OP has for the problem, that is, whether obtaining more data is possible or not. This paper has quite striking examples of this: proceedings.mlr.press/v80/lehtinen18a.html Aug 31 '20 at 5:58