Using Soft Labels in Classification Models

Question

(Updated)

I am working on a classification project in which I am required to detect a component of a railway switch using time-series data collected from an inductive sensor.

After some signal processing, I have implemented Short Time Fourier Transform (STFT), which yields a feature matrix (51 features in total) and time values corresponding to each STFT window. I labelled the windows by comparing it to the excel file containing my manual labelling.

The points where the components are present have been marked as label 1 and the points where there are no components have been marked as label 0. There are 39263 0s in total, whereas only 71 1s. To cater this problem of unbalanced labels, I used oversampling approach after splitting my data into train and test datasets which making sure that each of them contains at least one 1 label using stratify.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=101, stratify=y, shuffle=True)

oversample = RandomOverSampler(sampling_strategy='minority')
X_train_over, y_train_over = oversample.fit_resample(X_train, y_train)
X_test_over, y_test_over = oversample.fit_resample(X_test, y_test)

So, this is a hard labelling approach (binary labels 0 or 1) that I am using, and it is not yielding good results as the model is over-fitting or becoming over-confident.

The following is a snapshot of the results (Confusion Matrix) obtained from logistic regression:

Now, I am thinking of using a soft labelling approach to enhance the robustness of my model. However, the classification models do not accept float values. What should I do?

I am trying to do something like this:

However, how can I use soft labels for classification? Do I need to clarify more?

Answer 1

You may take one of the following 2 approaches (Which are pretty equivalent):

1. Use a Classifier with Cross Entropy Loss
   The cross entropy loss basically minimizes the differences between 2 distributions. For binary classification we usually have one class with value of 1 and the other as 0. But you may use any values you want for each class. So in case of soft you may use 0.95 for the 1 class and 0.05 for the 0 class.

2. Use a Regressor for the Probability
   This is very similar to the way the logistic regression works. But you may try other regressors as well.

This an answer to your direct question about moving to soft labelling.

Beyond that, your data looks like imbalanced data, it might be the problem more than the classification type. If you shared the features you have in a different question we might have a better idea what to do.