My main reference for machine learning and pattern recognition is Pattern Recognition and Machine Learning by Bishop. But I want a complementary reference to this which has more exercises and explains the mathematical stuff in more details, e.g. formulas and proofs.
Another good book is: - Pattern Recognition by Theodoridis and Koutroumbas
It's about classifiers, features and clustering in the context of pattern recognition.
It contains all the mathematical stuff you want, and backs everything up with references to papers, books, etc. It has no exercises, so for that you need to look somewhere else.
Also see this StackOverflow question on the matter.
Pattern Recognition by Theodoridis and Koutroumbas (Suggested above) has a companion Matlab book which has lot of Matlab exercises and goes along with the main text book. Check it out>> Introduction to Pattern Recognition: A Matlab Approach
The most popular choices seem to be
- Machine Learning: a Probabilistic Perspective by Murphy
- Pattern classification by Duda et all
- The Elements of Statistical Learning by Hastie et all. It is free from Stanford.
- Mining of Massive Datasets, free from Stanford.
- Bayesian Reasoning and Machine Learning, free available online.
- Learning from data. by Abu-Mostafa. It comes with Caltech video lectures
- Information Theory, Inference, and Learning Algorithms by Mackay, free.
- Classification, Parameter Estimation and State Estimation by van der Heijden.
- Computer Vision: Models, Learning, and Inference, by Prince
- Probabilistic Graphical Models by Koller. Has an accompanying course on Coursera.