13.4. CONVOLUTIONAL NEURAL NETWORKS
541
How Many Nodes in a Convolutional Layer?
We have referred to a node of a convolutional layer as a “ﬁlter.” That
ﬁlter may be a single node, or as in Example 13.11, a set of several nodes,
one for each channel. When we train the CNN, we determine weights for
each ﬁlter, so there are relatively few nodes. For instance, in the ﬁrst layer
in Example 13.11, there are 192 nodes, three for each of the 64 ﬁlters.
However, when we apply the trained CNN to an input, we apply each
ﬁlter to every pixel in the input. Thus, in Example 13.11, each of the 64
ﬁlters of the ﬁrst layer is applied to 224 × 224 = 50, 176 pixels. The key
point to remember is that although a CNN has a succinct representation,
the application of the represented neural net to data requires a signiﬁcant
amount of computation. The same, by the way, can be said for the other
specialized forms of neural net that we discuss in Section 13.5.
previous pooling layer and its ﬁlters can represent structures of progressively
larger sizes and complexities. Thus, the number of ﬁlters has been chosen to
double at each convolution layer.
Finally, the eleventh, and last, layer is a fully connected layer. It has 1000
nodes, corresponding to the 1000 images classes we are trying to learn how to
recognize. Being fully connected, each of these 1000 nodes takes all 7 × 7 × 3 =
147 outputs from the 10th layer; the factor 3 is from the fact that all ﬁlters of
the previous layers have three channels.
✷
Designing CNN’s and other deep network architectures is still more art than
science. Over the past few years, however, some rules of thumb have emerged
that are worth keeping in mind:
1. Deep networks that use many convolutional layers, each using many small
ﬁlters, are better than shallow networks that use large ﬁlters.
2. A simple pattern is to use convolutional layers that preserve the spatial
extent of their input by zero padding, and have size reduction done ex-
clusively by pooling layers.
3. Smaller strides work better in practice than larger strides.
4. It’s very useful to have the input size evenly divisible by 2 many times.
13.4.5
Implementation and Training
An examination of Equation 13.1 (generalized so the ﬁlter is f × f rather than
5-by-5) suggests that we can express each entry in the output of the convolution
as the dot product of vectors, followed by a scalar sum. To do so, we must ﬂatten
