13.6. REGULARIZATION
551
13.6
Regularization
Thus far, we have presented our goal as one of minimizing loss (i.e., prediction
error) on the training set. Gradient descent and stochastic gradient descent
help us achieve this objective.
In practice, the real objective of training is
to minimize the loss on new and hitherto unseen inputs.
Our hope is that
our training set is representative of unknown future inputs, so a low loss on the
training set translates into good performance on new inputs. Unfortunately, the
trained model sometimes learns idiosyncrasies of the training data that allow
it have low training loss, but not generalize well to new inputs – the familiar
problem of overﬁtting.
How can we tell if a model has overﬁt? In general, we split the available
data into a training set and a test set.
We train the model using only the
training-set data, withholding the test set. We then evaluate the performance
of the model on the test set. If the model performs much worse on the test set
than on the training set, we know the model has overﬁt. Assuming data points
are independent of one another, we can pick a fraction of the available data
points at random to form the test set. A common ratio for the training:test
split is 80:20. i.e, 80% of the data for training and 20% for test. We have to
be careful, however: in sequence-learning problems (e.g., modeling time series),
the state of the sequence at any point in time encodes information about the
past. In such cases the ﬁnal piece of the sequence is a better test set.
Overﬁtting is a general problem that aﬀects all machine-learning models.
However, deep neural networks are particularly susceptible to overﬁtting, be-
cause they use many more parameters (weights and biases) than other kinds of
models. Several techniques have been developed to reduce overﬁtting in deep
networks, usually by trading higher training error for better generalization. The
process is referred to as model regularization. In this section we describe some
of the most important regularization methods for deep learning.
13.6.1
Norm Penalties
Gradient descent is not guaranteed to learn parameters (weights and biases)
that reduce the training loss to an absolute minimum. In practice, the pro-
cedure learns parameters that correspond to a local minimum in the training
loss. There are usually many local minima, and some might lead to better gen-
eralization than others. In practice, it has been observed that solutions where
the learned weights have low absolute values tend to generalize better than
solutions with large weights.
We can force gradient descent to favor solutions with low weight values by
adding a term to the loss function. Suppose w is the vector of all the weight
values in the model, and L0 is loss function used by our model. We deﬁne a
new loss function L as follows:
L = L0 + α ∥w∥2
(13.17)
