13.3. BACKPROPAGATION AND GRADIENT DESCENT
527
u = Wx
v = u + b
y = σ(v)
L = MSE(y, ˆy)
Each of these steps corresponds to one of the four nodes in the middle row,
in order from the left. The ﬁrst step corresponds to the node with operand
u and operator ×. Here is an example where it must be understood that the
node labeled W is the ﬁrst argument. If necessary, we could label each incoming
edge with a number to indicate its order, but in this case the order should be
obvious, since a column vector x could not multiply a matrix W unless the
matrix happened to have only one row. The second step corresponds to the
node with operator + and operand v. Here, the order of arguments does not
matter, since + on vectors is commutative.
✷
13.3.2
Gradients, Jacobians, and the Chain Rule
The goal of the backpropagation algorithm is to compute the gradient of the
loss function with respect to the parameters of the network. Then, we can
adjust the parameters slightly in the directions that will reduce the loss, and
repeat the process until we reach a selection of parameter values for which little
improvement in the loss is possible. Recall the deﬁnition of the gradient: given a
function f : RN →R from a real-valued vector to a scalar, if x = [x1, x2, . . . , xn]
and y = f(x) then the gradient of y with respect to x, denoted by ∇xy is given
by
∇xy =
h ∂y
∂x1
, ∂y
∂x2
, . . . , ∂y
∂xn
i
Example 13.3 : We could let function f be the loss function, e.g., the squared-
error loss, which we denote L. This loss is a scalar-valued function of the output
y:
L(y) = ∥y −ˆy∥2 =
n
X
i=1
(yi −ˆyi)2
So we can easily write down the gradient of L with respect to y:
∇yL = [2(y1 −ˆy1), (y2 −ˆy2), . . . , 2(yn −ˆyn)] = 2(y −ˆy)
✷
The generalization of the gradient to vector-valued functions is called the
Jacobian.
Suppose we have a function f : Rm →Rn and y = f(x).
The
Jacobian Jx(y) is given by:4
4The Jacobian is sometimes deﬁned as the transpose of our deﬁnition. The formulations
are equivalent. Recall that we are assuming all vectors are column vectors unless transposed,
but we show them as row vectors so they can be written in-line.
