548
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
very little contribution from much earlier time steps. This phenomenon is called
the problem of vanishing gradients.
Equation 13.9 results in vanishing gradients because we used the tanh ac-
tivation function for state update. If instead we use other activation functions
such as ReLU, we end up with the product of many matrices with large entries,
resulting in the problem of exploding gradients. Exploding gradients are easier
to handle than vanishing gradients, because we can clip the gradient at each
step to lie within a ﬁxed range. However, the resulting RNN’s still have trouble
learning long-distance associations.
13.5.3
Long Short-Term Memory (LSTM)
The LSTM model is a reﬁnement of the basic RNN model to address the prob-
lem of learning long-distance associations. In the past few years, LSTM has
become popular as the de-facto sequence-learning model, and has been used
with success in many applications. Let us understand the intuition behind the
LSTM model before we describe it formally. The main elements of the LSTM
model are:
1. The ability to forget information by purging it from memory. For example,
when analyzing a text we might want to discard information about a
sentence when it ends. Or when analyzing a the sequence of frames in a
movie, we might want to forget about the location of a scene when the
next scene begins.
2. The ability to save selected information into memory. For example, when
we process product reviews, we might want to save only words expressing
opinions (e.g., excellent, terrible) and ignore other words.
3. The ability to focus only on the aspects of memory that are immediately
relevant. For example, focus only on information about the characters of
the current movie scene, or only on the subject of the sentence currently
being analyzed.
We can implement this focus by using a 2-tier archi-
tecture: a long-term memory that retains information about the entire
processed preﬁx of the sequence, and a working memory that is restricted
to the items of immediate relevance.
The RNN model has a single hidden state vector st at time t.
The LSTM
model adds an additional state vector ct, called the cell state, for each time t.
Intuitively, the hidden state corresponds to working memory and the cell state
corresponds to long-term memory. Both state vectors are of the same length,
and both have entries in the range [−1, 1]. We may imagine the working memory
having most of its entries near zero with only the relevant entries turned “on.”
The architectural ingredient that enables the ability to forget, save, and
focus is the gate. A gate g is just a vector of the same length as a state vector
