I have a training set of 4000 for ML with 36 features, two-class classifier.

  • Consider that due to some discrepancy feature no. 25 is missing for half of the data. But during data acquisition I know what class it belongs to.
  • I have defined a specified value (-1000) for that missing data. Other feature values are relatively smaller than -1000.
  • If I train the data with the same data set, it might not train the data with high accuracy. Can I somehow extrapolate the data to create the data set that may be present (near-by values) or is it fine to train the model with that data-set? (or any other suggestion).



Labelling a missing value as -1000 (assuming a non-missing value is much smaller in absolute value) will not cause problems with anything based on decision trees/stumps (bagged/boosted). You might also have some success with mixture models, for example a GMM - the learning algorithm will allocate some components to cover the missing data. Same for non-parametric density estimators, nearest neighbours, or SVM with RBF kernels.

Note that in all these cases, even if the model works, there will be a waste of resources ("model space budget") if feature #25 is always present in your testing data. In other words, your model will have learnt how to deal with tricky examples that will never occur when classifying new data! But if feature #25 will sometimes be missing in some of the data you will encounter "in the field", this is the way to go.

Do not expect things to work with algorithms such as neural networks or support vector machines with linear or low order polynomial kernels.

If feature #25 will never be missing "in the field", it might be simpler to just extrapolate feature 25 in your training data. To do so, you can proceed as follows:

Option 1: Nearest neighbor

  • Normalize all features (except #25).
  • For each training example in which feature #25 is missing, search for the nearest training sample, according to features (1 .. 24 and 26 .. 36), for which feature #25 is present.
  • Assign to feature #25 the value taken from the nearest neighbour.

Works best when you have lots of data (but not too much so as to make the search feasible) with kinky distributions.

Option 2: Regression

Use any regression method of your choice to find a relation ship between feature #25 and the other features, on the subset of your dataset in which feature #25 is not missing. For example, a linear regression.

Use the result to predict feature #25 on the remaining training instances.

Works best when your features are "well behaved" for the regression method you plan to use (for example, see the prerequisites for linear regression).

| improve this answer | |
  • $\begingroup$ thanks @pichnettes This answers most of my doubt but there is a slight change that not only feature #25 but some other features may also be missing in different samples. I have a lot of data, if that helps. $\endgroup$ – user50742 Jun 21 '14 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.