13.2. DENSE FEEDFORWARD NETWORKS
523
5
4
3
2
1
1
2
3
4
5
5
10
15
20
25
Figure 13.6: Huber Loss (solid line, δ = 1) and Squared Error (dotted line) as
functions of z = y −ˆy
13.2.9
Classiﬁcation Loss
Consider a multiclass classiﬁcation problem with target classes C1, C2, . . . , Cn.
Suppose each point in the training set is of the form (x, p) where x is the input
and p = [p1, p2, . . . , pn] is the output. Here pi gives the probability that the
input x belongs to class Ci, with P
i pi = 1. In many cases, we are certain that
an input belongs to a particular class Ci; in this case pi = 1 and pj = 0 for
i ̸= j. In general, we may interpret pi as our level of certainty that input x
belongs to class Ci, and p as a probability distribution over the target classes.
We design our neural network to produce as output a vector
q = [q1, q2, . . . , qn]
of probabilities, with P
i qi = 1. As before, we interpret q as a probability
distribution over the target classes, with qi denoting the model’s probability
that input x belongs to target class Ci. In Section 13.2.5 we described a sim-
ple method to produce such a probability vector as output: use the softmax
activation function in the output layer of the network.
Since both the labeled output and the model’s output are probability dis-
tributions, it is natural to look for a loss function that quantiﬁes the distance
between two probability distributions. Recall the deﬁnition of entropy from
Section ??. That is, H(p), the entropy of a discrete probability distribution p
is:
H(p) = −
n
X
i=1
pi log pi
Imagine an alphabet of n symbols, and messages using these symbols. Sup-
pose at each location in the message, the probability that symbol i appears is
