Chapter 13
Neural Nets and Deep
Learning
In Sections 12.2 and 12.3 we discussed the design of single “neurons” (percep-
trons). These take a collection of inputs and, based on weights associated with
those inputs, compute a number that, compared with a threshold, determines
whether to output “yes” or “no.” These methods allow us to separate inputs
into two classes, as long as the classes are linearly separable. However, most
problems of interest and importance are not linearly separable. In this chapter,
we shall consider the design of neural nets, which are collections of perceptrons,
or nodes, where the outputs of one rank (or layer of nodes becomes the inputs
to nodes at the next layer. The last layer of nodes produces the outputs of
the entire neural net. The training of neural nets with many layers requires
enormous numbers of training examples, but has proven to be an extremely
powerful technique, referred to as deep learning, when it can be used.
We also consider several specialized forms of neural nets that have proved
useful for special kinds of data. These forms are characterized by requiring that
certain sets of nodes in the network share the same weights. Since learning all
the weights on all the inputs to all the nodes of the network is in general a
hard and time-consuming task, these special forms of network greatly simplify
the process of training the network to recognize the desired class or classes of
inputs. We shall study convolutional neural networks (CNN’s), which are spe-
cially designed to recognize classes of images. We shall also study recurrent neu-
ral networks (RNN’s) and long short-term memory networks (LSTM’s), which
are designed to recognize classes of sequences, such as sentences (sequences of
words).
509
