554
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
are pooled, meaning that the nodes of the previous layer are partitioned,
and each node of this layer takes as input only the nodes of one block
of the partition. Convolutional layers are also used, especially in image
processing applications.
✦Convolutional Layers: Convolutional layers can be viewed as if their nodes
were organized into a two-dimensional array of pixels, with each pixel
represented by the same collection of nodes. The weights on corresponding
nodes from diﬀerent pixels must be the same, so they are in eﬀect the same
node, and we need learn only one set of weights for each family of nodes,
one from each pixel.
✦Activation Functions: The output of a node in a neural net is determined
by ﬁrst taking the weighted sum of its inputs, using the weights that are
learned during the process of training the net. An activation function
is then applied to this sum. Common activation functions include the
sigmoid function, the hyperbolic tangent, softmax, and various forms of
linear reciﬁed unit functions.
✦Loss Functions: These measure the diﬀerence between the output of the
net and the correct output according to the training set. Commonly used
loss functions include squared-error loss, Huber loss, classiﬁcation loss,
and cross-entropy loss.
✦Training a Neural Net: We train a neural net by repeatedly computing
the output of the net on training examples and computing the average
loss on the training examples. Weights on the nodes are then adjusted by
propagating the loss backward through the net, using the backpropagation
algorithm.
✦Backpropagation: By choosing our activation functions and loss functions
to be diﬀerentiable, we can compute a deriative of the loss function with
respect to every weight in the network.
Thus, we can determine the
direction in which to adjust each weight to reduce the loss. Using the chain
rule, these directions can be computed layer-by-layer, from the output to
the input.
✦Convolutional Neural Networks: These typically consist of a large num-
ber of convolutional layers, along with pooled layers and fully connected
layers. They are well suited to processing images, where the ﬁrst convolu-
tional layers recognize simple features, such as boundries, and later layers
recognize progressively more complex features.
✦Recurrent Neural Networks: These are designed to recognize sequences,
such as sentences (sequences of words). There is one layer for each position
in the sequence, and the nodes are divided into families, which each have
one node at each layer. The nodes of a family are constrained to have
