I am reading bishop book on pattern recognition and machine learning. Going through the introduction section, I am not sure if I get the differences between the two widely used term "test data" and "training data". Is training data the data you will have after the pre-processing stage which is also called feature extraction? Where test data is the original input. Am I right?
Here's the problem.
With an opaque learning algorithm, you have to figure out if your algorithm has really learned something about some deeper structure common to the desired problem area (assuming there is some to be found), or had just learned to recognize some particular inputs and spit out the desired answer only for those inputs (similar to school kids who just "memorize the test" but haven't a clue otherwise). The latter isn't very useful when the training data consists of only a tiny fraction of the desired problem space.
So, to find out, you train your algorithm on one set of data until it spits out the correct answers. Then you change up the input to some different test data that the trained algorithm has never seen before, and see if it can still give you useful answers, or only really worked for the only the stuff in the original training set. That's the test data set.
If you have real-world data, it's common to divide it up into two disjoints sets, and not let the algorithm see the test set during training. Similar to the teacher locking up the quiz questions until the end of the semester.
When you use machine learning algorithms on data sets, you use one part of the data (the training set) to train your algorithm (i.e., feature extraction). Once the training is completed, you'll need to evaluate the performance of the trained algorithm and you do this by applying it to new data, that is, the second part of your original data (the test data).
With this strategy you will be able to evaluate whether your trained algorithm indeed extracts valid features of your data (i.e., it works well on the training data and on the test data), or if it was "overlearned" or "overfitted" (i.e., it performs only well on the training data and bad on the test data).
To answer the OPs questions directly: "Is training data the data you will have after the pre-processing stage which is also called feature extraction?
No, that (i.e. feature extraction) is not what makes a set of data "training data". You will have to extract features in the same was as the training data from your testing data. Here is an example set of steps to distinguish these terms:
- Get raw data. [This set comprises of train/validation/test data].
- Divide the data in train/validation/test splits.
- Use the train data only to fit/classify your data using models such as SVM, neural networks etc.
- Now use test/validation data to check how good of a model you have.
Where test data is the original input. Am I right?
The test data is the data you keep aside while select/learn the parameters of your model. You later use this data to test how good of a model you have. The key assumption is your test data distribution is same as your train data distribution.