13.3. BACKPROPAGATION AND GRADIENT DESCENT
529
Since we shall need to compute these gradients several times, once for each
iteration of gradient descent, we can avoid repeated computation by adding
additional nodes to the compute graph for backpropagation: one node for each
gradient computation.
In general, the Jacobian Ja(b) is a function of both
a and b, and so the node for g(a) will have arcs to it from the nodes for a,
b and g(b).
Popular frameworks for deep learning (e.g., TensorFlow) know
how to compute the functional expression for the Jacobians and gradients of
commonly used operators such as those that appear in Fig. 13.8. In that case,
the developer needs only to provide the compute graph and the framework
will add the new nodes for backpropagation. Figure 13.9 shows the resulting
compute graph with added gradient nodes.
g( W )
g( v )
g( y)
u
v
y
σ
L
MSE
y
( b)
g
g( u )
W
x
b
Figure 13.9: Compute graph with gradient nodes
Example 13.4 : We shall work out the functional expressions for the gradients
of all the nodes in Fig. 13.8. We already know g(y). So the next node we choose
to process is v.
g(v) = ∇v(L) = Jv(y)∇y(L) = Jv(y)g(y)
We know that y = σ(v). Since σ is an element-wise operator, the Jacobian
Jv(y) takes a particularly simple form. Using the derivative for the logistic
sigmoid function from Section 13.2.2, we see that
∂yi
∂vj
=
(
yi(1 −yi)
if i = j
0
otherwise
The Jacobian is therefore a diagonal matrix:
Jv(y) =


y1(1 −y1)
0
. . .
0
0
y2(1 −y2)
. . .
0
...
...
...
...
0
0
. . .
yn(1 −yn)


