510
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
13.1
Introduction to Neural Nets
We begin the discussion of neural nets with an extended example. After that,
we introduce the general plan of a neural net and some important terminology.
Example 13.1 : The problem we discuss is to learn the concept that “good”
bit-vectors are those that have two consecutive 1’s. Since we want to deal with
only tiny example instances, we shall assume bit-vectors have length four. Our
training examples will thus have the form ([x1, x2, x3, x4], y), where each of the
xi’s are bits, 0 or 1. There are 16 possible training examples, and we shall
assume we are given some subset of these as our training set. Notice that eight
of the possible bit vectors are good – they do have consecutive 1’s, and there
are also eight “bad” examples. For instance, 0111 and 1100 are good; 1001 and
0100 are bad.1
To start, let us look at a neural net that solves this simple problem exactly.
How we might design this net from training examples is the true subject for
discussion, but this net will serve as an example of what we would like to
achieve. The net is shown in Fig. 13.1.
x 1
x 2
x 3
x 4
x 1
x 2
x 3
x 4
0.5
y
1
1
2.5
1.5
1
1
2
0
0
0
1
1
Figure 13.1: A neural net that tells whether a bit-vector has consecutive 1’s
The net has two layers, the ﬁrst consisting of two nodes, and the second
with a single node that produces the output y. Each node is a perceptron,
exactly as was described in Section 12.2. In the ﬁrst layer, the ﬁrst node is
characterized by weight vector [w1, w2, w3, w4] = [1, 2, 1, 0] and threshold 2.5.
Since each input xi is either 0 or 1, we note that the only way to reach a sum
P4
i=1 xiwi as high as 2.5 is if x2 = 1 and at least one of x1 and x3 is also 1.
The output of this node is 1 if and only if the input is one of 1100, 1101, 1110,
1111, 0110, or 0111. That is, it recognizes those bit-vectors that either begin
with two 1’s or have two 1’s in the middle. The only good inputs it does not
1We shall show bit vectors as bit strings in what follows, so we can avoid the commas
between components, each of which is 0 or 1.
