13.1. INTRODUCTION TO NEURAL NETS
513
13.1.2
Interconnections Among Nodes
Neural nets can diﬀer in how the nodes at one layer are connected to nodes
at the layer to its right. The most general case is when each node receives as
inputs the outputs of every node of the previous layer. A layer that receives
all outputs from the previous layer is said to be fully connected. Some other
options for choosing interconnections are:
1. Random. For some m, we pick for each node m nodes from the previous
layer and make those, and only those, be inputs to this node.
2. Pooled. Partition the nodes of one layer into some number of clusters. In
the next layer, which is called a pooled layer, there is one node for each
cluster, and this node has all and only the member of its cluster as inputs.
3. Convolutional.
This approach to interconnection, which we discuss in
more detail in the next section and Section 13.4, views the nodes of each
layer as arranged in a grid, typically two-dimensional. In a convolutional
layer, a node corresponding to coordinates (i, j) receive as inputs the
nodes of the previous layer that have coordinates in some small region
around (i, j). For example, the node (i, j) at one convolutional layer may
have as inputs those nodes from the previous layer that correspond to
coordinates (p, q), where i ≤p ≤i + 2 and j ≤q ≤j + 2 (i.e., the square
of side 3 whose lower-left corner is the point (i, j).
13.1.3
Convolutional Neural Networks
A convolutional neural network, or CNN, contains one or more convolutional
layers. There can also be nonconvolutional layers, such as fully connected layers
and pooled layers. However, there is an important additional constraint: the
weights on the inputs must be the same for all nodes of a single convolutional
layer. More precisely, suppose that each node (i, j) in a convolutional layer
recieves (i + u, j + v) as one of its inputs, where u and v are small constants.
Then there is a weight w associated with u and v (but not with i and j). For
any i and j, the weight on the input to the node for (i, j) coming from the
output of the node i + u, j + v) from the previous layer must be w.
This restriction makes training a CNN much more eﬃcient than training a
general neural net. The reason is that there are many fewer parameters at each
layer, and therefore, many fewer training examples can be used, than if each
node or each layer has its own weights for the training process to discover.
CNN’s have been found extremely useful for tasks such as image recognition.
In fact, the CNN draws inspiration from the way the human eye processes
images. The neurons of the eye are arranged in layers, similarly to the layers
of a neural net. The ﬁrst layer takes inputs that are essentially pixels of the
image, each pixel the result of a sensor in the retina. The nodes of the ﬁrst
layer recognize very simple features, such as edges between light and dark.
Notice that a small square of pixels, say 3-by-3, might exhibit an edge at a
