550
CHAPTER 13. NEURAL NETS AND DEEP LEARNING
Here, we use subscript o to indicate another pairs of weight matrices and a bias
vector that must be learned.
Finally, the output at time t is computed in exactly the same manner as the
RNN output:
yt = g(V st + d)
(13.16)
where g is an activation function, V is a weight matrix and d is a bias vector.
Equations 13.10 through 13.16 describe the state update operations at a
single time step t of an LSTM. We can think of plain RNN as a special case
of LSTM. When we set the forget gate to all 0’s (so we throw away all prior
long-term memory) and the input gate to all 1’s (save the entire candidate state
update), and the output gate to all 1’s (working memory is same as long-term
memory), we get something that looks very close to an RNN, the only diﬀerence
being an extra tanh factor.
The ability to selectively forget allows LSTM’s to avoid the vanishing-
gradient problem at the expense of introducing many more parameters than
vanilla RNN’s. While we will not provide a rigorous proof here, we note that the
key to avoiding vanishing gradients is the long-term memory update in Equa-
tion 13.13. Several variations of the basic LSTM model have been proposed;
the most common variant is the gated recurrent Unit (GRU) model, which uses
a single state vector instead of two state vectors (long-term and short-term).
A GRU has fewer parameters than an LSTM and might be suitable in some
situations with smaller data sets.
13.5.4
Exercises for Section 13.5
Exercise 13.5.1: In this exercise, you are asked to design the input weights
for one or more nodes of the hidden state of an RNN. The input is a sequence
of bits, 0 or 1 only.14 Note that you can use other nodes to help with the node
requested. Also note that you can apply a transformation to the output of the
node so a “yes” answer has one value and a “no” answer has another.
(a) A node to signal when the input is 1 and the previous input is 0.
! (b) A node to signal when the last three inputs have all been 1.
!! (c) A node to signal when the input is the same as the previous input.
! Exercise 13.5.2: Verify Equation 13.6.
! Exercise 13.5.3: Give the formulas for the gradients de
dU and
de
dV for the gen-
eral RNN of Fig.13.12.
14We should understand that RNN’s, like any neural network, is to be learned from data,
not designed as we are suggesting you do here.
