528
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
Jx(y) =


∂y1
∂x1
. . .
∂yn
∂x1
...
...
...
∂y1
∂xm
. . .
∂yn
∂xm


We shall make use of the chain rule for derivatives from calculus. If y = g(x)
and z = f(y) = f(g(x)), then the chain rule says:
dz
dx = dz
dy
dy
dx
Also, if z = f(u, v) where u = g(x) and v = h(x), then
dz
dx = ∂z
∂u
du
dx + ∂z
∂v
dv
dx
For functions of vectors, we can restate the chain rule in terms of gradients and
Jacobians. Suppose y = g(x) and z = f(y) = f(g(x)) then:
∇xz = Jx(y)∇yz
If z = f(u, v) where u = g(x) and v = h(x), then
∇xz = Jx(u)∇uz + Jx(v)∇vz
13.3.3
The Backpropagation Algorithm
The goal of the backpropagation algorithm is to compute the gradient of the loss
function with respect to the parameters of the network. Consider the compute
graph from Fig. 13.8. Here the loss function L is the MSE function. We shall
use the notation g(z) to stand for ∇z(L), that is, the gradient of the the loss
function L with respect to some vector z. We already know the gradient of L
with respect to the output y:
g(y) = ∇y(L) = 2(y −ˆy)
We work backwards through the compute graph, applying the chain rule at
each stage. At each point we pick a node all of whose successors have already
been processed. Suppose a is such a node, and suppose it has just one immediate
successor b in the graph (note that in the simple compute graph of Fig. 13.8,
each node has just one immediate successor). Since we have already processed
node b, we have already computed g(b). We can now compute g(a) using the
chain rule:
g(a) = Ja(b)g(b)
In the case where node a has more than one successor node, we use the more
general sum version of the chain rule. That is, g(a) would be the sum of the
above terms for each successor b of a.
