13.3. BACKPROPAGATION AND GRADIENT DESCENT
531
Therefore, noting that P
i pi = 1, we have:
l
=
H(p, q)
=
−
X
i
pi log qi
=
−
X
i
pi(yi −log(
X
j
eyj))
=
−
X
i
piyi −log(
X
j
eyj)
X
i
pi
=
−
X
i
piyi −log(
X
j
eyj)
Diﬀerentiating, we get:
∂l
∂yk
=
−pk +
eyk
P
j eyj
=
−pk + µ(yk)
=
qk −pk
Therefore, we end up with the rather neat result:
∇yl = q −p
This combined gradient does not saturate or explode, and leads to good learning
behavior. That observation explains why softmax and cross entropy loss work
so well together in practice.
✷
13.3.4
Iterating Gradient Descent
Given a set of training examples, we run the compute graph in both directions
for each example: forward (to compute the loss) and backwards (to compute
the gradients). We average the loss and gradients across the training set to
compute the average loss and the average gradient for each parameter vector.
At each iteration we update each parameter vector in the direction opposite
to its gradient, so the loss will tend to decrease. Suppose z is a parameter
vector. We set:
z ←z −ηg(z)
Here η is a hyperparameter, the learning rate. We stop gradient descent either
when the loss between successive iterations changes by only a tiny amount (i.e.,
we have reached a local minimum) or after a ﬁxed number of iterations.
It is important to pick the learning rate carefully. Too small a value means
gradient descent might take a very large number of iterations to converge. Too
large a value might cause large oscillations in the parameter values and never
