I am working in linear classification such as $$y=ax+b.$$ where $x=[x_1...x_n]$. But the method has weakness that is overfitting problem. Could you suggest to me one method or one way to reduce overfitting problem in linear classification (don't consisder other method ex SVM).

  • $\begingroup$ Use a subset of {x1, ..., xn}? regularization? $\endgroup$ Commented May 21, 2014 at 11:25
  • $\begingroup$ @MatthewPlourde: We can use both of them. And If we use regularization term, we must some prior information,right?Could you show both of them $\endgroup$
    – John
    Commented May 21, 2014 at 11:28
  • $\begingroup$ How many classes are you separating the data into? If it's just two, I don't think there could be such thing as overfitting. I may be wrong though. $\endgroup$
    – Phonon
    Commented May 22, 2014 at 9:15
  • $\begingroup$ @John, Could you review my answer? $\endgroup$
    – Royi
    Commented Jul 11, 2023 at 16:40

1 Answer 1


I'd say there are 3 main approaches:

  1. Objective Function
    Add some regularization to the objective function. If the parameters of the model are given by the vector $ \boldsymbol{w} $ then you may add $ \lambda {\left\| \boldsymbol{w} \right\|}_{2}^{2} $ or / and $ \lambda {\left\| \boldsymbol{w} \right\|}_{1} $ as a regularization terms.
  2. Features
    Reduce the number of features by a proper selection of them. Since your model is linear, correlation can be a good place to start with.
  3. Ensemble
    You may create an ensemble of classifiers. Each can be trained on a subset of the data / features. Then merge them by various policies (Majority, Probability, etc...).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.