534
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
13.3.6
Exercises for Section 13.3
Exercise 13.3.1: This exercise uses the neural net from Fig. 13.7. However,
assume that the weights on all inputs are variables rather than the constants
shown. Note, however, that some inputs do not feed one of the nodes in the
ﬁrst layer, so these weights are ﬁxed at 0. Assume that the input vector is x,
the output vector is y, and the output of the two nodes in the hidden layer is
the vector z. Also, let the matrix and bias vector connecting x to z be W1 and
b1, while the matrix and bias vector connecting z to y are W2 and b2. Assume
that the activation function at the hidden layer is the hyperbolic tangent, and
the activation function at the output is the identity function (i.e., no change to
the outputs is made). Finally, assume the loss function is mean-squared error,
where ˆy is the true output vector for a given input vector x. Draw the compute
graph that shows how y is computed from x.
Exercise 13.3.2: For the network described in Exercise 13.3.1:
(a) What is Jy(z)?
(b) What is Jz(x)?
(c) Express g(x) in terms of g(tanh (z)).
(d) Express g(x) in terms of the loss function.
(e) Draw the compute graph with gradient computation for the entire net-
work.
13.4
Convolutional Neural Networks
Consider a fully-connected network layer for processing 224 × 224 images, with
each pixel encoded using 3 color values (often called channels – the values of
intensity for red, green, and blue)).8 The number of weight parameters in the
connection layer for each output node is 224 × 224 × 3 = 150, 528. Suppose
we had 224 nodes in the output layer. We would end up with a total of over
33 million parameters! Given that our training set of images is usually of the
order of tens or hundreds of thousands of images, the huge number of parameters
would quickly lead to overﬁtting even using just a single layer.
A Convolutional Neural Network (CNN) greatly reduces the number of pa-
rameters by taking advantage of the properties of image data. CNN’s introduce
two new varieties of network layers: convolutional layers and pooling layers.
8This is the size of most of the images on ImageNet, for example.
