Signal Processing Stack Exchange is a question and answer site for practitioners of the art and science of signal, image and video processing. It only takes a minute to sign up.
Sign up to join this community
I'm using a set of features extracted from a signal for classifying the data window with KNN algorithm. Since the features have different value ranges, their influence on distance calculation is different when you use euclidean distance in KNN.
To equalize the influence of these features on classification:
What are the advantages of these two approaches over eachother?
Both are reasonable approaches and it is foreseeable that either one could outperform the other empirically. The Euclidean distance assumes the data to be isotropically Gaussian, i.e. it will treat each feature equally. On the other hand, the Mahalanobis distance seeks to measure the correlation between variables and relaxes the assumption of the Euclidean distance, assuming instead an anisotropic Gaussian distribution.
If you know a priori that there is some kind of correlation between your features, then I would suggest using a Mahalanobis distance over Euclidean. Also, note that Z-score feature scaling can mitigate the usefulness of choosing a Mahalanobis distance over Euclidean (less true of min-max normalization though). The major drawback of the Mahalanobis distance is that it requires the inversion of the covariance matrix which can be computationally restrictive depending on the problem.
To complete bjou's answer:
A potential problem with min-max normalization is that it is very sensitive to outliers. For example, if your dataset contain 100 two-dimensional examples; and that the dataset looks like this:
1, 147
8, -252
3, 125
2, -605
...
10000, -100 <- outlier
4, 200
min-max normalization will squash almost all values of the first attribute to 0, at the exception of the last one which will be 1 - and as a result, it is unlikely that the first attribute will have any weight in the classification.
You can instead use a robust variant of min-max normalization which uses the first and third quartile (or 1st and 9th decile) instead of the minimum and maximum.
Another common normalization technique consists in removing the mean and dividing by the standard deviation. You can think of it as an analogue of Mahalanobis distance in which the covariance matrix is constraint to be diagonal. It is somewhat sensitive to outliers to, but not as drastically as min/max.
A last thing worth mentioning: gaussianizing your data (for example with a box-cox transform) is often helpful - whenever you are doing things on data with the euclidean norm there's a gaussianity assumption lurking behind!
