Neural network with much less positive examples

Question

I am developing a medical algorithm working on large amounts of data. However, as in many medical scenarios I have a lot of negative example and only a few positive ones. So any neural network I train is clearly biased towards negative classification. In fact it misclassifies most positive examples and although overall accuracy is OK, false negative rate is sky high.

I wanted to know what is the best way to make the network achieve lower false negative rate, I thought about duplicating the positive examples several times or maybe reducing the negative examples randomly, What do you think?

Answer 1

There are a couple of things you can do to treat the class imbalance.

1. A way is to use an SVM-loss (Hinge-loss) function in the network, where the margin around the decision boundary is maximized. The good part of this: Because this function considers the support vectors only, the data points which do not lie close to the margin will not be taken into account. Of course, with more data, you almost guarantee to change the decision boundary and therefore the support vectors. However, especially if your data is spread around the boundary (if you can sample some positive-like true negatives), then you should get close to the correct minimum. This is an easy way to get around without much effort.

2. You could augment (not just duplicate) the positives more than you augment the negatives, so that the data becomes balanced.

3. The weighting of the samples is already discussed, and I think it is valid.

Answer 2

I found out that I can give a different weight vector for positive and negative examples

for example:

weights = ones(1,size(outputs,2));
weights(outputs(2,:) ==1) = weights(outputs(2,:) ==1)*factor;
train(net,input,output,[],[],weights);

Answer 3

Consider your outputs to be (-1) / (+1) standing for Negative/positive. Now you can consider your classification problem to be function fitting (like usual neural networks) with (-1) / (+1) as possible outputs (You can generalize this for multi-class problems as well, but typically with multidimensional vector outputs).

Now you have some options, some of them you mentioned partially.

1. Training with a single Positive / (+1) sample multiple times, depending on the ratio of observations with (-1) / to (+1).
2. Weighting as you mentioned or better than that, based on the risk of False results on each direction. that is (-1) being considered as (+1) and vise versa.
3. Considering output value of neural network before rounding to be (y), then if you want to see more (+1)'s consider the output to be (+1) if y >= -0.1 or something like that.

I hope this helps.