13.3. BACKPROPAGATION AND GRADIENT DESCENT
525
Exercise 13.2.3: Show that tanh(x) = 2σ(2x) −1.
Exercise 13.2.4: Show that σ(x) = 1 −σ(−x).
Exercise 13.2.5: Show that for any vector [v1, v2, . . . , vk], Pk
i=1 µ(vi) = 1.
1
y
x 1
x 2
x 3
x 4
y
3
y
2
1
=1
=0
=1
=0
1
2
3
−4
1
−1
2
3
−2
−2
1 −2
1
1
1
4
1
1
2
2
3
Figure 13.7: Neural net for Exercise 13.2.6
Exercise 13.2.6: In Fig. 13.7 is a neural net with paricular values shown
for all the weights and inputs. Suppose that we use the sigmoid function to
compute outputs of nodes at the ﬁrst layer, and we use softmax to compute the
outputs of the nodes in the output layer.
(a) Compute the values of the outputs for each of the ﬁve nodes.
! (b) Assuming each of the weights and each of the xi’s is a variable, express
the output of the ﬁrst (top) output node in terms of the weights and the
xi’s,
! (c) Find the derivative of your function from part (b) with respect to the
weight on the ﬁrst (top) input to the ﬁrst (top) node in the ﬁrst layer.
13.3
Backpropagation and Gradient Descent
We now turn to the problem of training a deep network. Training a network
means ﬁnding good values for the parameters (weights and thresholds) of the
network. Usually, we shall have access to a training set of labeled input/output
pairs. The training process tries to ﬁnd parameter values that minimize the
average loss on the training set. The hope is that the training set is represen-
tative of the data the model will encounter in the future, and therefore, the
average loss on the training set is a good measure of the average error on all
possible inputs. We must be careful, however; since deep networks have many
