Techniques to reject noisy neural network input

Question

Suppose an artificial neural network is used to approximate a sine wave (shown in red in the graph below), given the linear input variable $x$ (scaled such that the ANN input is $x_{\rm nn}\in[-1;1]$).

The mean squared error between the target function (sine wave) and the trained ANN output is as shown approximately $4.8\times10^{-3}$, if the ANN structure is 1-8-1 (one input $x_{\rm nn}$, 8 hidden nodes and 1 output node).

Now suppose the ANN structure is changed to 2-8-1, where the second input is normally distributed random noise. The output is then as shown below. The MSE has increased to approximately $1.2\times10^{-2}$ (using the exact same noise sequence as during training) or slightly larger if a new noise sequence is used when testing the network.

The results above were obtained using standard gradient-descent type training with regularisation. All nodes in the hidden layer have $\rm tanh()$ activation functions and the output node is linear, as is standard with function approximation.

What methods are available to automatically remove the second input? That is, training methods allowing the ANN to automatically discover the fact that the second input is noise, and therefore not a useful input. (Therefore excluding all methods where the human programmer must train several different ANN architectures with different input combinations, and evaluate all of them to decide which inputs to use and which to exclude.)

This question extends to larger ANNs where it is difficult to see beforehand which inputs are useful. Ideally, all weights associated with nonsense inputs should automatically be forced to zero.

Answer 1

It seems like your model is overfitting because you are not providing it with enough training data.

If I understood correctly, you want the neural network to learn $\hat{y} = \hat{f}(x,n)$ where $n \sim \mathcal{N}(\mu,\sigma)$ is noise and $x$ is in the interval $[-1,1]$ by minimizing $$ \frac{1}{N}\sum_{i=1}^N |y_i - \hat{y_i}|^2 = \frac{1}{N}\sum_{i=1}^N |f(x_i,n_i) - \hat{f}(x_i,n_i)|^2 $$

And the function you are trying to learn is $f(x,n) = sin(\pi x)$, right?.

The network should learn to ignore the noise $n$ given that you provide it enough examples for training. That is, for different values of $x$, your training data should have a lot of examples with different values for $n$, all having the same target $sin(\pi x)$. If you show the network enough examples where the $n$ value is not useful for making the prediction, it should learn to minimize its effect on the output. The more training data you have, the better the network can learn to do this.