13.7. SUMMARY OF CHAPTER 13
553
training set (the training loss) decreases through the training process, the loss
on the test set (the test loss) often behaves diﬀerently. The test loss falls dur-
ing the initial part of the training, and then many hit a minimum and actually
increase after a large number of training iterations, even as the training loss
keeps falling.
Intuitively, the point at when the test loss starts to increase is the point at
which the training process has started learning idiosyncrasies of the training
data rather than a generalizable model. A simple approach to avoiding this
problem is to stop the training when the test loss stops falling. There is, how-
ever, a subtle problem with this approach: we might inadvertently overﬁt to the
test data (rather than to the training data) by stopping training at the point
of minimum test loss. Therefore, the test error no longer is a reliable measure
of the true performance of the model on hitherto unseen inputs.
The usual solution is to use a third subset of inputs, the validation set, to
determine the point at which we stop training. We split the data not just into
training and test sets, but into three groups: training, validation, and test.
Both the validation and test sets are withheld from the training process. When
the loss on the validation set stops decreasing, we stop the training process.
Since the test set has played no role at all in the training process, the test error
remains a reliable indicator of the true performance of the model.
13.6.4
Dataset Augmentation
The accuracy of most machine-learning models increases when we provide ad-
ditional training data. Usually, larger training sets also lead to less overﬁtting.
When the actual training data available is limited, we can often create addi-
tional synthetic training examples by applying transformations or adding noise.
For example, consider the digit-classiﬁcation problem we encountered in
Example 13.7. It is clear that if we rotate an image corresponding to a digit
by a few degrees, it still remains an image of the same digit. We can augment
the training data by systematically applying transformations of this kind. One
way to think of this process is as a way to encode additional domain knowledge
(e.g., a slightly distorted image of a cat is still an image of a cat).
13.7
Summary of Chapter 13
✦Neural Nets: A neural net is a collection of perceptrons (nodes), usually
organized in layers, where the outputs from one layer provide inputs to
the next layers. The ﬁrst (inout) layer takes external inputs, and the
last (output) layer indicates the class of the input. Other layers in the
middle are called hidden layers and generally are trained to recognize
intermediate concepts needed to determine the output.
✦Types of Layers: Many layers are fully connected, meaning that each node
in the layer has all the nodes of the previous layer as inputs. Other layers
