530
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
Suppose s = [s1, s2, . . . , sn] is a vector deﬁned by si = yi(1 −yi) (i.e., the
diagonal of the Jacobian matrix). We can express g(v) simply as
g(v) = s ◦g(y)
where a ◦b is the vector resulting from the element-wise product of a and b.5
Now that we have g(v), we can compute g(b) and g(u). We have g(b) =
Jb(v)g(v) and g(u) = Ju(v)g(v). Since
v = u + b
it is straightforward to verify that
Jb(v) = Ju(v) = In
where In is the n × n identity matrix. So we have
g(b) = g(u) = g(v)
We ﬁnally come to the matrix W. Recall that u = Wx. There is a potential
problem here, because all the machinery we have set up works for vectors, while
W is a matrix. But recall from Section 13.2.1 that we assembled the matrix W
from a set of vectors w1, w2, . . . , wn, where wT
i is the ith row of W. The trick
is to consider each of these vectors separately and compute its gradient using
the usual formula.
g(wi) = Jwi(u)g(u)
We know that ui = wT
i x and none of the other uj’s have any dependency on
wi for i ̸= j. Therefore, the Jacobian Jwi(u) is zero everywhere except the ith
column, which is equal to x. Thus we have
g(wi) = g(ui)x
✷
Example 13.5 : As we mentioned, neural networks for classiﬁcation often use
a softmax activation in the ﬁnal layer followed by a cross-entropy loss. We will
now compute the gradient of the combined operator.
Suppose the input to the combined operator is y; let q = µ(y), and let l =
H(p, q), where p represents the true probability vector for the corresponding
training example. We have:
log(qi)
=
log(
eyi
P
j eyj )
=
yi −log(
X
j
eyj)
5This operation sometimes called the Hadamard product, so as not to confuse it with the
more usual dot product, which is the sum of the components of the Hadamard product.
