13.4. CONVOLUTIONAL NEURAL NETWORKS
537
commonly used. Note that the ﬁlter size speciﬁes only the width and height of
the ﬁlter; the number of channels of the ﬁlter always matches the number of
channels of the input.
The activation map in our example is slightly smaller than the input. In
many cases, it is convenient to have the activation map be of the same size as the
input. We can expand the repsonse by using zero padding: adding additional
rows and columns of zeros to pad out the input. A zero padding of p corresponds
to adding p rows of zeros each to the top and bottom, and p columns to the left
and right, increasing the dimensionality of the input by 2p along both width
and height. A zero padding of 2 in our example would augment the input size
to 228 × 228 and result in an activation map of size 224 × 224, the same size as
the original input image.
The third hyperparameter of interest is stride. In our example, we assumed
that we apply the ﬁlter at every possible point in the input image. We could
think instead of sliding the ﬁlter one pixel at a time along the width and height
of the input, corresponding to a stride s = 1.
Instead, we could slide the
ﬁlter along the width and the height of image two or three pixels at a time,
corresponding to a stride s of 2 or 3. The larger the stride, the smaller the
activation map compared to the input.
Suppose the input is an m× m square of pixels, the output an n× n square,
ﬁlter size is f, stride is s, and zero padding is p. It is easily seen that:
n = (m −f + 2p)/s + 1
(13.3)
In particular, we must be careful to pick hyperparameters such that s evenly
divides m −f + 2p; else we would have an invalid arrangement for the convo-
lutional layer, and most software implementations would throw an exception.
We can intuitively think of a ﬁlter as looking for an image feature, such as
a splotch of color or an edge. Classifying an image usually requires identifying
many features. We therefore use many ﬁlters, ideally one for each useful feature.
During the training of the CNN, we hope that each ﬁlter will learn to identify
one of these features.
Suppose we use k ﬁlters; to keep things simple, we
constrain all ﬁlters to use same size, stride, and zero padding. Then the output
contains k activation maps. The dimensionality of the output layer is therefore
n × n × k, where n is given by Equation 13.3.
The set of k ﬁlters together constitute a convolutional layer. Given input
with d channels, a ﬁlter of size f requires df 2+1 parameters (df 2 weight param-
eters and 1 bias parameter). Therefore, a convolutional layer of k such ﬁlters
uses k(df 2 + 1) parameters.
Example 13.9 : Continuing the ImageNet example, suppose the input consists
of 224×224×3 images, and we use a convolutional layer of 64 ﬁlters, each of size
5, stride 1, and zero padding 2. The size of the output layer is 224 × 224 × 64.
Each ﬁlter needs 3×5×5+1 = 76 parameters (including one for the threshold)
and the convolutional layer contains 64 × 76 = 4864 parameters – orders of
magnitude smaller than the number of parameters for a fully connected layer
with the same input and output sizes.
✷
