546
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
Moreover, suppose W is a matrix, and w is the vector obtained by concatenating
the rows of W. Then:
dz
dW
=
dz
dw
dy
dW
=
dy
dw
These conventions also extend naturally to partial derivatives.
We use backpropagation to compute the gradients of the error with respect
to the network parameters. We focus on
de
dW ; the gradients for U and V are
similar, and left as exercises for the reader. It is clear that:
de
dW =
n
X
t=1
det
dW
Focusing on step t, we have:
det
dW = dst
dW
det
dst
We leave it as an exercise to verify that:
det
dst
= V T(yt −ˆyt)
(13.6)
Setting Rt = dst
dW , we note that st = tanh(zt), where zt = Wst−1 + Uxt + b,
we have:
Rt = dst
dzt
dzt
dW
It is straightforward to verify that dst
dzt is the diagonal matrix A deﬁned by:
aij =
(
1 −s2
ti
when i = j
0
otherwise
We should to be careful to note that zt is a function of W both directly and
indirectly, since st−1 also depends on W. So we must express dzt
dW as a sum of
two terms:
dzt
dW = ∂zt
∂W +
∂zt
∂st−1
dst−1
dW
It is easily veriﬁed that:
∂zt
∂st−1
= W T
The form of ∂zt
∂W is a little trickier. It is a matrix B with mostly zero entries,
the nonzero elements being entries from st−1. We leave the computation of B
