514
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
particular angle, e.g., if the upper left corner is light and the other eight pixels
dark. Moreover, the algorithm for recognizing an edge of a certain type is the
same, regardless of where in the ﬁeld of vision this little square appears. That
observation justiﬁes the CNN constraint that all the nodes of a layer use the
same weights. In the eye, additional layers combine results from the previous
layers to recognize more and more complex structures: long boundaries, regions
of similar color, and ﬁnally faces and all the familiar objects that we see daily.
We shall have more to say about convolutional neural networks in Sec-
tion 13.4. Moreover, CNN’s are only one example of a kind of neural network
where certain families of nodes are constrained to have the same weights. For
example, in Section 13.5, we shall consider recurrent neural networks and long
short-term memory networks, which are specially adapted to recognizing prop-
erties of sequences, such as sentences (sequences of words).
13.1.4
Design Issues for Neural Nets
Building a neural net to solve a given problem is partially art and partially
science. Before we can begin to train a net by ﬁnding the weights on the inputs
that serve our goals best, we need to make a number of design decisions. These
include answering the following questions:
1. How many hidden layers should we use?
2. How many nodes will there be in each of the hidden layers?
3. In what manner will we interconnect the outputs of one layer to the inputs
of the next layer?
Further, in later sections we shall see that there are other decisions that need
to be made when we train the neural net. These include:
4. What cost function should we minimize to express what weights are best?
5. How should we compute the outputs of each gate as a function of the
inputs? We have suggested that the normal way to compute output is to
take a weighted sum of inputs and compare it to 0. But there are other
computations that serve better in common circumstances.
6. What algorithm do we use to exploit the training examples in order to
optimize the weights?
13.1.5
Exercises for Section 13.1
!! Exercise 13.1.1: Prove that the problem of Example 13.1 cannot be solved
by a perceptron; i.e., the good and bad points are not linearly separable.
