13.5. RECURRENT NEURAL NETWORKS
549
s; each of the gate’s entries is between 0 and 1. The Hadamard product11 s ◦g
allows us to selectively pass through certain parts of the state while ﬁltering out
others. Usually, a gate vector is created by a linear combination of the hidden
state and the current input. We then apply a sigmoid function to “squash” its
entries to lie between 0 and 1. In general, an LSTM may use several diﬀerent
kinds of gate vectors, each for a diﬀerent purpose. At time t, we can create a
gate g as follows:
g = σ(Wst−1 + Uxt + b)
Here W and U are weight matrices and b is a bias vector.
At time t, we ﬁrst compute a candidate state update vector ht based on the
previous hidden state and the current input:
ht = tanh(Whst−1 + Uhxt + bh)
(13.10)
Note that W, U, and b with subscript h are two weight matrices and a bias
vector that we learn and use for just the purpose of computing ht for each t.
We also compute two gates, the forget gate ft and the input gate it. The
forget gate determines which aspects of the long-term memory we retain. The
input gate determines which parts of the candidate state update to save into the
long-term memory. These gates are computed using diﬀerent weight matrices
and bias vectors, which also must be learned. We indicate these matrices and
vector with subscripts f and i, respectively.
ft
=
σ(Wfst−1 + Ufxt + bf)
(13.11)
it
=
σ(Wist−1 + Uixt + bi)
(13.12)
We update the long-term memory using the gates and the candidate update
vector as follows:12
ct = ct−1 ◦ft + ht ◦it
(13.13)
Now that we have updated the long-term memory, we need to update the work-
ing memory. We do this in two steps. The ﬁrst step is to create an output gate
ot. The second step is to apply this gate to the long-term memory, followed by
a tanh activation:13
ot
=
σ(Wost−1 + Uoxt + bo)
(13.14)
st
=
tanh(ct ◦ot)
(13.15)
11The Hadamard product of vectors [x1, x2, . . . , xn] and [yq, y2, . . . , yn] is the vector whose
components are the products of the corresponding components of the two argument vectors,
that is, [x1y1, x2y2, . . . , xnyn]. The same operation may be applied to any matrices that have
the same dimensions.
12Technically, an entry of 1 in the forget gate results in retaining the corresponding memory
entry; so the forget gate should really be called the remember gate. Similarly, the input gate
might be better named the save gate. Here we follow the naming convention commonly used
in the literature.
13Once again, the output gate might be better named the focus gate since it focuses the
working memory on certain aspects of the long-term memory.
