13.5. RECURRENT NEURAL NETWORKS
545
Here f is a nonlinear activation function such as tanh or sigmoid. U and W are
matrices of weights, and b is a vector of biases. We deﬁne s0 to be a vector of
all zeros. The output at time t is a function of the hidden state at time t, after
being transformed by a parameter matrix V and an activation function g:
yt = g(V st + c)
In our example, g might be the softmax function to ensure that the output is a
valid probability vector.
The RNN in Figure 13.12 has an output at each time step. In some appli-
cations, such as machine translation, we need just a single output at the end of
each sentence. In such cases, the RNN’s single output is further processed by
one or more fully-connected layers to generate the ﬁnal output.
It is simplest to assume that our RNN’s inputs are ﬁxed-length sequences
of length n. In this case, we simply unroll the RNN to contain n time steps.
In practice, many applications have to deal with variable-length sequences e.g.,
sentences of varying lengths. There are two approaches to deal with this situa-
tion:
1. Zero-padding. Fix n to be the longest sequence we process, and pad out
shorter sequences to be length n.
2. Bucketing. Group sequences according to length, and build a separate
RNN for each length.
A combination of these two approaches is used. We can create a bucket for a
small number of diﬀerent lengths, and assign a sequence to the bucket of the
shortest length that is at least as long as the sequence. Then, within a bucket
we use padding for sequences that are shorter than the maximum length for
that bucket.
13.5.1
Training RNN’s
We use backpropagation to train an RNN, just as we would any neural network.
Let us work through an example. Suppose our input consists of sequences of
length n. Our network uses the activation function tanh for the state update,
softmax for the output, and the loss function is cross-entropy. Since the network
has several outputs, one at each time-step, we seek to minimize the total error
e, deﬁned as the sum of the losses ei at each time step i:
e =
n
X
i=1
ei
To simplify notation, we use the following conventions. Suppose x and y are
vectors, and z is a scalar. We deﬁne:
dz
dx
=
∇xz
dy
dx
=
Jx(y)
