524
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
pi. Then a key result from information theory is that if we encode messages
using an optimal binary code, the average number of bits per symbol needed to
encode messages is H(p).
Suppose we did not know the symbol probability distribution p when we
design the coding scheme. Instead, we believe that symbols appear following
a diﬀerent probability distribution q. We might ask what the average number
of bits per symbol will be if we use this suboptimal encoding scheme. A well-
known result from information theory states that the average number of bits in
this case is the cross entropy H(p, q), deﬁned as:
H(p, q) = −
n
X
i=1
pi log qi
Note that H(p, p) = H(p), and in general H(p, q) ≥H(p). The diﬀerence
between the cross entropy and the entropy is the average number of additional
bits needed per symbol. It is a reasonable measure of the distance between the
distributions p and q, called the Kullblack-Liebler divergence (KL-divergence)
and denoted D(p∥q):
D(p∥q) = H(p, q) −H(p) =
n
X
i=1
pi log pi
qi
Even though KL-divergence is often regarded as a distance, it is not truly
a distance measure because it is not commutative.
However, it is perfectly
adequate as a loss function for our purposes, since there is in fact an inherent
assymmetry in the situation: p is the ground truth while q is the predicted
output. Notice that minimizing the KL-divergence loss of a model is equivalent
to minimizing the cross-entropy loss, since the term H(p) depends only on the
input and is independent of the model that is learned.
In practice, cross entropy is the most commonly used loss function for clas-
siﬁcation problems. Networks designed for classiﬁcation often use a softmax
activation function in the output layer.
These choices are so common that
many implementations, such as TensorFlow, oﬀer a single function that com-
bines softmax with cross entropy. In addition to convenience, one reason to do
so is that the combined function is more stable numerically, and its derivative
also takes a simple form, as we show in Section 13.3.3.
13.2.10
Exercises for Section 13.2
Exercise 13.2.1: Show that for the logistic sigmoid σ, if y = σ(x), then
dy
dx = y(1 −y)
Exercise 13.2.2: Show that if y = tanh(x) then
dy
dx = 1 −y2
