13.4. CONVOLUTIONAL NEURAL NETWORKS
539
13.4.3
Pooling Layers
A pooling layer takes as input the output of a convolutional layer and produces
an output with smaller spatial extent. The size reduction is accomplished by
using a pooling function to compute aggregates over small contiguous regions
of the input. For example we might use max pooling over nonoverlapping 2 × 2
regions of the input; in this case there is an output node corresponding to
every nonoverlapping 2 × 2 region of the input, with the output value being the
maximum value among the 4 inputs in the region. The aggregation operates
independently on each channel of the input. The resulting output layer is 25%
the size of the input layer. There are three elements in deﬁning a pooling layer:
1. The pooling function, which is most commonly the max function but could
in theory be any aggregate function, such as average.
2. The size f of each pool, which speciﬁes that each pool uses an f ×f square
of inputs.
3. The stride s between pools, analogous to the stride used in the convolu-
tional layer.
The most common use cases in practice are f = 2 and s = 2, which speciﬁes
nonoverlapping 2 × 2 regions, and f = 3, s = 2, which speciﬁes 3 × 3 regions
with some overlap. Higher values of f lead to too much loss of information in
practice. Note that the pooling operation shrinks the height and width of the
input layer, but preserves the number of channels. It operates independently
on each channel of its input. Note that unlike the convolution layer, it is not
common practice to use zero padding for the max pooling layer.
Pooling is appropriate if we we believe that features are approximately in-
variant to small translations. For example, we might care about the relative
locations of features such as legs or wings and not their exact locations. In
such cases pooling can greatly reduce the size of the hidden layer that forms
the input to the subsequent layers of the network.
Example 13.10 : Suppose we apply max pooling with size = 2 and stride =
2 to the 224 × 224 × 64 output of the convolutional layer from Example 13.9.
The resulting output is of size 112 × 112 × 64.
✷
13.4.4
CNN Architecture
Now that we have seen the building blocks of Convolutional Neural Networks,
we can put them together to build deep networks. A typical CNN alternates
convolutional and pooling layers, ending with one or more fully-connected layers
that produce the ﬁnal output.
Example 13.11 : For instance, Fig. 13.11 is a simple network architecture for
classifying ImageNet images into one of 1000 image classes, loosely inspired by
