13.5. RECURRENT NEURAL NETWORKS
543
(c) Suppose we do not do any zero padding. If the output of one layer is
input to the next layer, after how many layers will there be no output at
all?
Exercise 13.4.2: Repeat Exercise 13.4.1(a) and (c) for the case when there is
a stride of three.
Exercise 13.4.3: Suppose we have the output of an m-by-m convolutional
layer with k ﬁlters, each having d channels. These outputs are fed to a pooling
layer with size f and stride s. How many output values does the pooling layer
produce?
Exercise 13.4.4: For this exercise, assume that inputs are single bits 0 (white)
and 1 (black). Consider a 3-by-3 ﬁlter, whose weights are wij, for 0 ≤i ≤2 and
0 ≤j ≤2, and whose bias is b. Suggest wieghts and bias so that the output of
this ﬁlter would detect the following simple features:
(a) A vertical boundary, where the left column is 0, and the other two columns
are 1.
(b) A diagonal boundary, where only the triangle of three pixels in the upper
right corner are 1.
(c) a corner, in which the 2-by-2 square in the lower right is 0 and the other
pixels are 1.
13.5
Recurrent Neural Networks
Just as CNN’s are a specialized family of neural networks for processing 2-
dimensional image data, recurrent neural networks (RNN’s) are networks spe-
cially designed for processing sequential data. Sequential data naturally arises
in many settings: a sentence is a sequence of words; a video is a sequence of
images; a stock market ticker is a sequence of prices.
Consider a simple example from the ﬁeld of language processing. Each input
is a sentence, modeled as a sequence of words. After processing each preﬁx of
the sequence, we would like to predict the next word in the sentence; the output
at each step is a probability vector across words. Our example suggests two
key design considerations:
1. The output at each point depends on the entire preﬁx of the sentence
until that point, and not just the last word. The network needs to retain
some “memory” of the past.
2. The underlying language model does not change across positions in the
sequence, so we should use the same parameters (weights for each of the
nodes) at each position.
