12.1. THE MACHINE-LEARNING MODEL
469
Test
Model
Model
Generation
Training Set
Set
Error Rate
Figure 12.3: The training set helps build the model, and the test set validates
it
Generalization
We should remember that the purpose of creating a model or classiﬁer is
not to classify the training set, but to classify the data whose class we do
not know. We want that data to be classiﬁed correctly, but often we have
no way of knowing whether or not the model does so. If the nature of
the data changes over time, for instance, if we are trying to detect spam
emails, then we need to measure the performance over time, as best we
can. For example, in the case of spam emails, we can note the rate of
reports of spam emails that were not classiﬁed as spam.
There is nothing special about the selection of the test data. In fact, we
can repeat the train-then-test process several times using the same data, if we
divide the data into k equal-sized chunks. In turn, we let each chunk be the test
data, and use the remaining k −1 chunks as the training data. This training
architecture is called cross-validation.
Batch Versus On-Line Learning
Often, as in Examples 12.1 and 12.2, we use a batch learning architecture. That
is, the entire training set is available at the beginning of the process, and it is
all used in whatever way the algorithm requires to produce a model once and
for all. The alternative is on-line learning, where the training set arrives in a
stream and, like any stream, cannot be revisited after it is processed. In on-line
learning, we maintain a model at all times. As new training examples arrive,
we may choose to modify the model to account for the new examples. On-line
learning has the advantages that it can
1. Deal with very large training sets, because it does not access more than
one training example at a time.
