552
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
The loss function L penalizes large weight values. Here α is a hyperparameter
that trades oﬀbetween minimizing the original loss function L0 and the penalty
associated with the L2-norm of the weights. Instead of the L2-norm, we could
penalize the L1-norm of the weights:
L = L0 + α
X
i
|wi|
(13.18)
In practice, it is observed that the L2-norm penalty works best for most
applications. The L1-norm penalty is useful in some situations calling for model
compression, because it tends to produce models where many of the weights are
zero.
13.6.2
Dropout
Dropout is a technique that reduces overﬁtting by making random changes to
the underlying deep neural network. Recall that when we train using stochastic
gradient descent, at each step we sample at random a minibatch of inputs to
process. When using dropout, we also select at random a certain fraction (say
half) of all the hidden nodes from the network and delete them, along with
any edges connected to them. We then perform forward propagation and back-
propagation for the minibatch using this modiﬁed network, and update the
weights and biases. After processing the minibatch, we restore all the deleted
nodes and edges. When we sample the next minibatch, we delete a diﬀerent
random subset of nodes and repeat the training process.
The fraction of hidden nodes deleted each time is a hyperparameter called
the dropout rate.
When training is complete, and we actually use the full
network, we need to take into account that the full network contains a larger
number of hidden nodes than the networks used for training. We therefore need
to scale the weight on each outgoing edge from a hidden node by the dropout
rate.
Why does dropout reduce overﬁtting? Several hypotheses have been put
forward, but perhaps the most convincing argument is that dropout allows a
single neural network to behave eﬀectively as a collection of neural networks.
Imagine that we have a collection of neural networks, each with a diﬀerent
network topology. Suppose we trained each network independently using the
training data, and used some kind of voting or averaging scheme to create a
higher-level model. Such a scheme would perform better than any of the indi-
vidual networks. The dropout technique simulates this setup without explicitly
creating a collection of neural networks.
13.6.3
Early Stopping
In Section 13.6.1 we suggested iterating through training examples (or mini-
batches) until we reach a local minimum in the loss function. In practice, this
approach leads to overﬁtting. It has been observed that while the loss on the
