544
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
s
x
y
V
U
s
x
y
V
U
s
x
y
s
s
x
y
V
U
W
n
n
n
W
V
U
W
W
W
(b) The unrolled RNN of length 
(a) The basic unit of an RNN.
n.
o o o
0
1
1
1
2
2
2
Figure 13.12: RNN architecture
These considerations lead naturally to a recurrent network model, where we
perform the same operation at each step, with the input to each step being
dependent upon the output from prior steps. Figure 13.12 shows the structure
of a typical recurrent neural network. The input is a sequence x1, x2, . . . , xn,
and the output is also a sequence y1, y2, . . . , yn. In our example, each input
xi represents a word, and each output yi is a probability vector for the next
word in the seentence. The input xi is typically encoded as a 1-hot vector –
a vector of length equal to the number of possible words,10 with a 1 in the
position corresponding to the input word and a 0 in all other positions. There
are two important diﬀerence between a general neural network and the RNN:
1. The RNN has inputs at all (or almost all) layers, and not just at the ﬁrst
layer.
2. The weights at each of the ﬁrst n layers are constrained to be the same;
these weights are the matrices U and W in Equation 13.5 below. Thus,
each of the ﬁrst n layers has the same set of nodes, and correspond-
ing nodes from each of the layers share weights (and are thus really the
same node), just as nodes of a CNN representing diﬀerent locations share
weights and are thus really the same node.
At each step t, we have a hidden state vector st that functions as the memory
in which the network encodes information about the preﬁx of the sequence it
has seen. The hidden state at time t is a function of the input at time t and
the hidden state at time t −1:
st = f(Uxt + Wst−1 + b)
(13.5)
10Since there could in principle be an inﬁnite number of words, we might in practice devote
components of the vector only to the most common words or the words that are most impor-
tant in the application at hand. Other words would all be represented by a single additional
component of the vector.
