# Persistence pays off: Paying Attention to
What the LSTM Gating Mechanism Persists

Giancarlo D. Salton  John D. Kelleher
ADAPT Research Centre
Dublin Institute of Technology
Ireland
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Language Models (LMs) are important components in several Natural Language
Processing systems. Recurrent Neural Network LMs composed of LSTM units,
especially those augmented with an external memory, have achieved state-of-
the-art results. However, these models still struggle to process long
sequences which are more likely to contain long-distance dependencies because
of information fading and a bias towards more recent information. In this
paper we demonstrate an effective mechanism for retrieving information in a
memory augmented LSTM LM based on attending to information in memory in
proportion to the number of timesteps the LSTM gating mechanism persisted the
information.

##  1 Introduction

Language Models (LM) are important components in Natural Language Processing
systems including, Statistical Machine Translation and Speech Recognition
Schwenk et al. (2012). An LM is generally used to compute the likelihood of a
sequence of $N$ words appearing in a given language by using the chain rule

$\displaystyle
p(w_{1},\dots,w_{N})=\prod_{t=1}^{N}p(w_{n}|w_{1},\dots,w_{n-1})$ (1)

Recently, Recurrent Neural Networks LMs (RNN-LMs) have became the state-of-
the-art approach to language modelling Józefowicz et al. (2016). However, RNN-
LMs struggle to keep their level of performance as the length of the input
sequence increases, especially if the input contains long-distance
dependencies (LDD).

A typical RNN-LM sequentially propagates forward a context vector that
integrates information about previous inputs to use for the next prediction.
Consequently, the information that is captured at the beginning of a sequence
containing an LDD is likely to have faded from the context by the time the
model spans that dependency. Another problem with such models is that the
context vector may be dominated by more recent information. To address these
limitations, several “memory-augmented” RNN-LMs architectures have been
developed. In general, these models store the hidden states of the RNN in a
memory buffer and then attempt to retrieve relevant information from the
buffer at each timestep (e.g., Tran et al. (2016), Cheng et al. (2016),
Daniluk et al. (2017), Merity et al. (2017), Grave et al. (2017) and Salton et
al. (2017)).

In this paper, we analyse the behaviour of the LSTM units and demonstrate that
an efficient and effective mechanism for a memory augmented LSTM based LM
(LSTM-LM) to retrieve important information from its history is to construct a
representation of the LSTM state history that weights information in
proportion to the number of timesteps the LSTM persisted the information.
Using this strategy reinforces the decisions of the LSTM gating mechanism at
each timestep regarding what is important in a sequence. We demonstrate that
using this simple strategy a memory augmented LSTM-LM can achieve state-of-
the-art results for a single model on the Penn Treebank Marcus et al. (1994)
with fewer parameters than its competitors, an also achieves near state-of-
the-art results on the wikitext2 Merity et al. (2017).

Structure: §2 reviews LSTMs and the gating mechanism; §3 discusses the effect
of uniformly weighting the hidden states of an LSTM; §4 illustrates
persistence of information in an LSTM; §5 describes our memory augmented LSTM-
LM; §6 presents experiments and results; §7 contextualizes our findings; and
§8 contains conclusions.

##  2 LSTMs

LSTM units (_aka._ LSTM cells) and their variants (e.g., GRUs Cho et al.
(2014)) are now a normal building block for neural based NLP systems Bradbury
et al. (2016); Murdoch and Szlam (2017). LSTMs retain and propagate
information through the dynamics of the LSTM memory cell, hidden state, and
gating mechanism (including the _input_ , _forget_ , and _output_ gates). The
LSTM memory cell retains information that is only known by the unit itself and
the hidden state shares information to other LSTM units in the same or any
next layer of the network. This way, the units can decide what to keep in
memory and how much of that information it wants the other units/layers to
know about it. If something is deemed important, the units will both keep it
in memory and let other units/layers to know about it. The gating mechanism
controls the flow of information between the memory cell and the hidden state.
Therefore, the gating mechanism plays an important role on the LSTM hidden
dynamics.

The computations of a standard LSTM unit Gers et al. (2000) (without peephole
connections) involve iterating over the following equations

$\displaystyle\mathbf{\widetilde{c}}_{t}=$ $\displaystyle
tanh(\mathbf{W}\mathbf{x}_{t}+\mathbf{W}\mathbf{h}_{(t-1)}+\mathbf{b})$ (2)
$\displaystyle\mathbf{i}_{t}=$
$\displaystyle\sigma(\mathbf{W}_{ii}\mathbf{x}_{t}+\mathbf{W}_{hi}\mathbf{h}_{(t-1)}+\mathbf{b}_{i})$
(3) $\displaystyle\mathbf{f}_{t}=$
$\displaystyle\sigma(\mathbf{W}_{if}\mathbf{x}_{t}+\mathbf{W}_{hf}\mathbf{h}_{(t-1)}+\mathbf{b}_{f})$
(4) $\displaystyle\mathbf{o}_{t}=$
$\displaystyle\sigma(\mathbf{W}_{io}\mathbf{x}_{t}+\mathbf{W}_{ho}\mathbf{h}_{(t-1)}+\mathbf{b}_{o})$
(5) $\displaystyle\mathbf{c}_{t}=$
$\displaystyle\mathbf{f}_{t}\times\mathbf{c}_{(t-1)}+\mathbf{i}_{t}\times\mathbf{\widetilde{c}_{t}}$
(6) $\displaystyle\mathbf{h}_{t}=$
$\displaystyle\mathbf{o}_{t}\times\tanh(\mathbf{c}_{t})$ (7)

where the weight matrices $\mathbf{W}_{i*}$ are associated to the input; the
weight matrices $\mathbf{W}_{h*}$ are associated with the recurrence; the
vectors $\mathbf{i}_{t}$, $\mathbf{f}_{t}$, $\mathbf{o}_{t}$ are the
activation vectors produced by the input, forget and output gates
respectively; $\mathbf{\widetilde{c}}_{t}$ is the candidate memory cell state;
$\mathbf{c}_{t}$ is the new memory cell state; and $\mathbf{h}_{t}$ is the
output of the unit. Equation 2 will produce a candidate vector
$\mathbf{\widetilde{c}}_{t}$ that contains information extracted from the
input to the LSTM. The input gate (Equation 3) and forget gate (Equation 4)
compute an activation vector to be used to update the memory cell (Equation
6): the candidate vector is multiplied by the input gate to decide how much of
the input is important to the memory cell; and the content of the memory cell
$\mathbf{c}_{(t-1)}$ is multiplied by the forget gate to decide how much the
memory cell will keep from its own content; the results of these two
multiplications are merged to decide what will be remembered in the memory
cell $\mathbf{c}_{t}$ for the next iteration. The output gate (Equation 5)
also calculates an activation vector that is multiplied by the current content
on the memory cell $\mathbf{c}_{t}$ to produce what we call hidden state
$\mathbf{h}_{t}$ (Equation 7). This last step decides how much of the content
in the memory cell $\mathbf{c}_{t}$ will be known on the next timestep (and by
cells in the next layer if it is a multi-layered LSTM or to any layer that may
come next o the network). It is important to notice that all the gates compute
activations in the range $[0,1]$ as they are using a sigmoid activation
function. If the values are close to 0, the gates are closed and if the values
are close to 1, the gates are open. For example, if the forget gate has a
value close to 0 the content of the memory cell is erased; if the forget gate
is close to 1, all the content of the memory cell will be exposed to other
units or layers in the network.

The success of LSTM-RNNs is attributed to their ability to retain information
about the input sequence for several timesteps in their internal memory cell
$\mathbf{c}_{t}$. That information is then made available to the next layer in
the network for the amount of timesteps it is considered relevant to the
current sequence. As pointed by Murdoch and Szlam (2017), each input to an
LSTM makes a contribution to the hidden state of the LSTM and that is
reflected when Equation 6 is iterated. At any given timestep $t$, the cell
state $\mathbf{c}_{t}$ can be decomposed into

$\displaystyle\mathbf{c}_{t}=\sum_{i=1}^{t}(\prod_{j=i+1}^{t}\mathbf{f}_{i})\mathbf{i}_{i}\mathbf{\widetilde{c}}_{i}$
(8)

which, according to the authors, can be interpreted as the contribution at
timestep $t$ to the memory block $\mathbf{c}_{t}$ by a particular past input
at timestep $j$. In that view, the contribution of an input to a given
timestep can be understood as an importance score weighted by the LSTM’s
gating mechanism. Therefore, if something is important to the current context
if should receive a larger importance score and be held in the memory block
for a number timesteps. In addition to retaining information, Murdoch and
Szlam (2017) have also demonstrated that, despite the fact that it is still
difficult to interpret what specific activations in the hidden dynamics of
LSTM units mean, it is possible to extract semantically meaningful rules from
the memory cells to train a powerful classifier that can approximate the
output of the LSTM itself. Moreover, Strobelt et al. (2016) and Karpathy et
al. (2015) have demonstrated that these networks can extract meaningful
attributes from the data into the memory cells. These attributes carry fine
grained information and keep track of attributes such as line lengths, quotes
and brackets.

Although these and other work demonstrate the power of LSTM units and their
gating mechanism, RNN-LMs based on such units (LSTM-LMs) struggle to process
long sequences. In our view, the main reason for this degradation in
performance happens exactly because of the hidden state dynamics of the LSTM
units. Once the information retained in the memory cell $\mathbf{c}_{t}$ is
outdated, the forget gate $\mathbf{f}_{t}$ erases that block enabling the unit
to store fresh data without interference from previous timesteps Gers et al.
(2000, 2003). This behaviour creates a natural bias towards more recent inputs
given that the memory cell has limited capacity to store previous information
and, once the memory cell is saturated, the forget gate will start to drop
information in favour of more recent inputs. Even though the LSTM units can
learn which information it must retain and for how long, the model will
struggle with long sequences that are more likely to contain LDDs and that
saturate the memory cell.

Once a memory cell has been saturated then, although some content has received
a large importance score in past steps, it may be dropped from the memory cell
(because of the inherent limitation of the LSTM’s capacity of storing content)
and will not be available to contribute to the next steps. For example, an
LSTM-LM trained on English may persist the information related to a subject of
a sentence sentence for a number of time steps because the subject is
important but this information may still have faded by the time the verb is
reached. However, by augmenting the network with a memory buffer the
information relating to the subject continues to be accessible so long as the
memory buffer is not reset. This behaviour is an indication of why the memory
augmented models such as the Neural cache model of Grave et al. (2017) and the
Pointer LSTM of Merity et al. (2017) has gained success and achieved state-of-
the-art in LM research. Even though the required content has already faded
from the context, the memory augmentation make it available for subsequent
timesteps.

##  3 The Curious Effectiveness of Uniform Attention

As was noted in Section 1, in recent years a number of extensions to RNN-LMs
have be proposed to overcome this fading of information from memory by adding
a memory buffer (that is used to store the LSTM hidden state at each timestep)
and then at each timestep construct a representation of this history to inform
the current prediction. A variety of relatively sophisticated mechanisms for
retrieving information from the memory buffer have been proposed. In many
cases these retrieval mechanisms include an extra neural network in the RNN-
LMs that at each timestep predicts what elements in the memory buffer should
be retrieved.

Salton et al. (2017) is a recent example that uses an extra neural
network111Similar to that proposed by Bahdanau et al. (2015) and Luong et al.
(2015) for Neural Machine Translation (NMT) to learn what to retrieve from
memory. In this architecture at the end of a timestep the current LSTM hidden
state is added to the memory buffer. Then at the beginning of the next
timestep the additional neural network predicts an attention distribution over
the elements of the memory buffer (i.e., the previous LSTM hidden states).
Using this attention distribution a compact representation of the RNN-LMs
history is constructed by calculating a weighted sum of the elements in the
memory (where the weight of each element is the attention attributed to it by
the RNN). Curiously, although this architecture was successful in terms of
performance the attention mechanism did not work as expected. Instead of
focusing attention for each time step on particular relevant elements in
memory it spread out the attention nearly uniformly across the memory. It
might appear that this architecture was using a strategy of “pay equal
attention to everything in the past”. However, we argue this interpretation
ignores the power of the LSTM gating mechanism.

Our interpretation of the uniform attention mechanism presented by Salton et
al. (2017) is that their _Attentive_ RNN-LM is in fact (indirectly)
reinforcing the decisions of the gating mechanism of the LSTM units and is
retrieving information that is persisted across multiple timesteps. This is
important because it indicates that it may be more fruitful and efficient to
leverage the decisions made by the LSTM gating mechanism (decisions that the
network must make anyway) to drive the retrieval of information from the
memory buffer rather than train a separate neural network. It is worth
emphasising that to date none of the different retrieval mechanisms proposed
in the literature on memory augmented LSTM-LMs have explicitly considered the
behaviour of the LSTM gating mechanism.

##  4 The Persistence of Information

The LSTM gating mechanism will attempt to persist important information for as
long as possible (or until the state is saturated). Figure 1 illustrates the
idea of persistence within an LSTM unit and how information is eventually
dropped. In this illustration, we can see that the information contained in
the top of the LSTM block is persisted across 3 timesteps (timesteps 1, 2, and
3) and the information at the bottom LSTM block is persisted for 2 timesteps
(timesteps 2 and 3). All other content was held for a single timestep.
However, at the moment the model is making a prediction on the last timestep
(in this case, timestep 4), all the old information (including the information
that was persisted) has already vanished from the context.

Figure 1: Illustration of the persistence of information in an LSTM-RNN. At
any given timestep the forget gate may decide to erase the information stored
in the memory cell of the LSTM unit and allow the input gate to decide what to
store in its place. If something is deemed important by the unit, it is held
for more than one timestep until it is regarded as outdated. In addition, if
the information is important, the output gate will let it flow to the next
timesteps and other units in the network. In this hypothetical example,
considering the output gate is open (i.e., values of 1), the information of
the top memory cell is maintained for timesteps 1, 2 and 3, whilst the
information on the bottom memory cell is held on timesteps 2 and 3. All the
other information is held only for a single timestep. Also, notice that we
drop the initial state $\mathbf{h}_{0}$ from this image

We propose that when retrieving/constructing a representation of the LSTM
history from a memory buffer the information held for more than one timestep
should be weighted in proportion to the number of timesteps the LSTM gating
mechanism persisted it across. In the specific situation illustrated in Figure
1 we argue that the information persisted in the top memory cell for 3
timesteps should have a larger during weighting in the representation of the
sequence history compared to the pieces of information that were only held for
a single timestep. This way, we let the gating mechanism of the LSTM determine
what is important about the input and, anything that is persisted for more
than one timestep, will have a greater impact on the final prediction even if
that information has already faded from the current context.

A simple and efficient way to implement this strategy is at each time point to
construct a representation of the history of the RNN-LM that is simply an
average of the LSTM hidden states in the memory buffer. Pieces of information
that the LSTM unit persists for several time steps will have a bigger impact
on this average (simply because they are included multiple times) relative to
items that are not persisted. In effect, this average weights each piece of
information in proportion to the number of time steps the LSTM persisted it
and so an RNN-LM that uses this average as its representation of history pays
attention to what the LSTM gating mechanism persisted.

##  5 Averaging the Outputs

In this work we adapt and simplify the architecture of Salton et al. (2017)
and use a simple average of previous outputs instead of a neural network based
attention mechanism. Our intuition for this modification is that the gating
mechanism of the LSTM is telling us what is important about an input and that
we must find a way to make that information available for long distances in
the future. In fact, Ostmeyer and Cowell (2017) have presented a model that
computes a recurrent weighted average (RWA) over every past hidden state.
However, the authors limit themselves to evaluate the model over simple tasks
and the effectiveness of that model over language modelling is still to be
demonstrated.

Compared to other memory augmented models our architecture is relatively
simple. Figure 2 displays a timestep for our system222Please note that this
figure is inspired by the architecture schematic in Salton et al. (2017). A
multi-layered LSTM-RNN encodes an input at each timestep and the outputs of
the last recurrent layer (i.e., its hidden state) $h_{t}$ (Equation 7) is
added to memory. At each timestep an average of the vectors in the memory
buffer is calculated and concatenated with the $h_{t}$ generated by the
processing of the current input. This concatenated vector is then feed into
the softmax layer which predicts the distribution for the next word in the
sequence.

Figure 2: Illustration of a step of the Average LSTMLM. In this example, the
model receives the third word as input ($\textbf{w}_{3}$) after storing the
previous states ($\textbf{h}_{1}$, $\textbf{h}_{2}$ and the initial state
$\textbf{h}_{0}$) in memory. After producing $\textbf{h}_{3}$, the model
computes the context vector (in this case $\textbf{c}_{3}$) by averaging the
states stored in memory. $\textbf{c}_{3}$ is concatenated to $\textbf{h}_{3}$
before the softmax layer for the prediction of the fourth word
$\textbf{w}_{4}$. Also note that $\mathbf{h}_{3}$ is stored in memory only at
the end of this process and does not participate in the $\textbf{c}_{3}$
calculation.

More formally, the hidden state at timestep $t$, $\mathbf{h}_{t}$, is
calculated for an input $\mathbf{x}_{t}$ as

$\displaystyle\mathbf{h}_{t}=g(\mathbf{x}_{t},\mathbf{h}_{t-1})$ (9)

where $g$ is a LSTM-RNN. Instead of applying a sophisticated attention
mechanism to the hidden states ($\mathbf{h}_{<t}$), we calculate the context
vector $\mathbf{c}_{t}$ as the average of the hidden states in memory:

$\displaystyle\mathbf{c}_{t}=\frac{1}{t}\sum_{i=0}^{t-1}\mathbf{h}_{i}$ (10)

where each $\mathbf{h}_{i}$ is a hidden state stored in the memory
buffer333Please notice that the iteration index for the summation starts at 0
and, thus, the length of the memory buffer is equal to the timestep index $t$.
As we shall explain later, we also included the initial state $\mathbf{h}_{0}$
in the memory buffer and count it to calculate the average.. After this step,
$\mathbf{c}_{t}$ is concatenated to the current hidden state $\mathbf{h}_{t}$
by means of a concatenation layer

$\displaystyle\mathbf{h}_{t}^{\prime}=tanh(\mathbf{W}_{c}[\mathbf{h}_{t};\mathbf{c}_{t}]+\mathbf{b}_{t})$
(11)

and, finally, the modified hidden state $\mathbf{h}_{t}^{\prime}$ is forwarded
to the softmax layer for the next prediction

$\displaystyle
p(w_{t}|w_{<t},x)=softmax(\mathbf{W}_{s}\mathbf{h}_{t}^{\prime}+\mathbf{b})$
(12)

In our experiments with this uniform attention, we found that initialising the
memory with a zero vector $\mathbf{h}_{0}$ and allowing the model to count
this vector as part of the memory when calculating the average444In other
words, the index of the memory starts at timestep 0 instead of timestep 1.
Thus, the memory at any given timestep $t$ will be of length $t+1$. improved
the performance of the model.

##  6 Experiments

To test our intuitions, we evaluate the averaging process of the model using
the PTB dataset using the standard split and pre-processing as in Mikolov et
al. (2010) which consists of 887K, 70K and 78K tokens on the training,
validation and test sets respectively. We also evaluate the model on the
wikitext2 dataset using the standard train, validation and test splits which
consists of around 2M, 217K tokens and 245k tokens respectively.

###  6.1 PTB Setup

Following Salton et al. (2017) we trained a multilayer LSTM-RNN with 2 layers
of 650 units for the PTB experiment. We trained them using Stochastic Gradient
Descent (SGD) with an initial learning rate of 1.0 and we halved the learning
rate at each epoch after 12 epochs. We train the model to minimise the average
negative log probability of the target words until we do not get any
perplexity improvements over the validation set with an early stop counter of
10 epochs. We initialize the weight matrices of the network uniformly in
[$-0.05$, $0.05$] while all biases are initialized to a constant value at
$0.0$ with the exception of the forget gate biases which is initialised at
$1.0$ as suggested by Jozefowicz et al. (2015). We also apply $50\%$ dropout
Srivastava et al. (2014) to the non-recurrent connections and clip the norm of
the gradients, normalized by the mini-batch size of $32$, at $5.0$. We also
tie the weight matrix $\mathbf{W}_{s}$ in Equation 12 to be the embedding
matrix as in Press and Wolf (2016). Thus, the dimensionality of the embeddings
is set to 650.

###  6.2 wikitext2 Setup

For the wikitext2 experiments we trained a multilayer LSTM-RNN with 2 layers
of 1000 units. We also used SGD to minimise the average negative log
probability of the target words with an initial learning rate of 1.0. We
decayed the the learning rate by a factor of 1.15 at each epoch after 14
epochs and we used an early stop counter of 10 epochs. Similarly to the PTB
experiment, we initialize the weight matrices of the network uniformly in
[$-0.05$, $0.05$] while all biases are initialized to a constant value at
$0.0$ with the exception of the forget gate biases which is initialised at
$1.0$. For this model we apply $65\%$ dropout to the non-recurrent connections
and clip the norm of the gradients, normalized by the mini-batch size of $32$,
at $5.0$. Once again, we tie the weight matrix $\mathbf{W}_{s}$ in Equation 12
to be the embedding matrix. Thus, the dimensionality of the embeddings is set
to 1,000.

###  6.3 Data Manipulation and Batch Processing

When training each model, we use all sentences in the respective training set,
but we truncate all sentences longer than 35 words and pad all sentences
shorter than 35 words with a special symbol so all have the same length. We
use a vocabulary size of 10k for the PTB and 33,278 for the wikitext2. Each of
the mini-batches we use for training are then composed of 32 of these
sentences taken from the dataset in sequence.

Contrary to the recent trend in the field, we do not allow successive mini-
batches to sequentially traverse the dataset. We reinitialize the hidden state
of the LSTM-RNN at the beginning of each mini-batch, by setting it to all
zeros. Our motivation for not sequentially traversing the dataset is that
although sequentially traversing has the advantage of allowing the batches to
be processed more efficiently, some dependencies between words may not be
learned if batch traversing is in use as the mini-batch boundaries can split
sentences. We also found that allowing the initial state of all zeros to be
included in the memory when averaging improves the performance of the Average
RNN-LM.

###  6.4 Results

Model | Params | Valid. Set | Test Set
---|---|---|---
Single Models |  |  |
Large Regularized LSTM Zaremba et al. (2015) | 66M | 82.2 | 78.4
Large + BD + WT Press and Wolf (2016) | 51M | 75.8 | 73.2
Neural cache model (size = 500) Grave et al. (2017) | - | - | 72.1
Medium Pointer Sentinel-LSTM Merity et al. (2017) | 21M | 72.4 | 70.9
Attentive LM w/ _combined_ score function Salton et al. (2017) | 14.5M | 72.6 | 70.7
Attentive LM w/ _single_ score function Salton et al. (2017) | 14.5M | 71.7 | 70.1
Averaging RNN-LM | 14.1M | 71.6 | 69.9
Model Averaging |  |  |
2 Medium regularized LSTMs Zaremba et al. (2015) | 40M | 80.6 | 77.0
5 Medium regularized LSTMs Zaremba et al. (2015) | 100M | 76.7 | 73.3
10 Medium regularized LSTMs Zaremba et al. (2015) | 200M | 75.2 | 72.0
2 Large regularized LSTMs Zaremba et al. (2015) | 122M | 76.9 | 73.6
10 Large regularized LSTMs Zaremba et al. (2015) | 660M | 72.8 | 69.5
38 Large regularized LSTMs Zaremba et al. (2015) | 2508M | 71.9 | 68.7
Table 1: Perplexity results over the PTB. Symbols: WT = weight tying Press and Wolf (2016); BD = Bayesian Dropout Gal and Ghahramani (2015). Please note that we could not calculate the number of parameters for some models given missing information in the original publications. Model | Params | Valid. Set | Test Set
---|---|---|---
Zoneout + Variational LSTM Merity et al. (2017) | 20M | 108.7 | 100.9
LSTM-LM Grave et al. (2017) | - | - | 99.3
Variational LSTM Merity et al. (2017) | 20M | 101.7 | 96.3
Neural cache model (size = 100) Grave et al. (2017) | - | - | 81.6
Pointer LSTM (window = 100) Merity et al. (2017) | 21M | 84.8 | 80.8
Averaging RNN-LM | 50M | 74.6 | 71.3
Attentive LM w/ _combined_ score function Salton et al. (2017) | 51M | 74.3 | 70.8
Attentive LM w/ _single_ score function Salton et al. (2017) | 51M | 73.7 | 69.7
Neural cache model (size = 2000) Grave et al. (2017) | - | - | 68.9
Table 2: Perplexity results over the wikitext2. Please note that we could not
calculate the number of parameters for some models given missing information
in the original publications.

Table 1 presents the results in terms of perplexity of the models trained over
the PTB dataset. As we can see, the results obtained by the Averaging RNN-LM
are similar to those obtained by the _Attentive_ RNN-LMs of Salton et al.
(2017) and by an ensemble of 38 LSTM-LMs. Despite the simple method to
retrieve information from the previous timesteps, the Averaging RNN-LM
achieves the same level of performance of more complex models with less
computation overhead.

Table 2 presents the results in terms of perplexity of the models trained over
the wikitext2 dataset. Although the Averaging RNN-LM is still behind the
_Attentive_ RNN-LMs and the Neural cache model of Grave et al. (2017) on this
dataset, the results are encouraging given the simplicity of the Averaging
RNN-LM.

##  7 Discussion

The Averaging LSTM-LM achieves the lowest perplexity for a single model on the
PTB (see Table 1). It does this using less parameters than any of the other
models tested on the PTB555We have considered using the concept of a parameter
budget Collins et al. (2016); Melis et al. (2017) to contextualize the
difference in parameters but given that the difference in terms of total
number parameters between the baseline _Attentive_ RNN-LMs and our model is
relatively small (essentially arising from the fact that we have dropped the
attention neural network component from that architecture) we felt it would
not be appropriate to apply this concept.. Given the similarity of the results
between the _Attentive_ RNN-LMs of Salton et al. (2017) and the Averaging
LSTM-LM it would appear that our hypothesis that the _Attentive_ RNN-LMs was
(indirectly) learning to use the dynamics of the LSTM gating mechanism is
correct. The Averaging LSTM-LM is also very competitive on the PTB compared to
the ensemble methods and in this case the difference in total number of
parameters between the Averaging LSTM-LM and these ensemble models is
significant.

Focusing on the results for the wikitext2 dataset (see Table 2) the Neural
cache model is still the best performing model on this dataset. We are not
able to estimate the number of parameters for the Neural cache model so we
have not included the parameter size of that model in the table. In discussing
the wikietext2 results it is worth noting that the _Attentive_ RNN-LMs of
Salton et al. (2017) and the Averaging LSTM-LM are the only models in Table 2
that reset their memory at each sentence boundary whereas the memory buffers
of other models were allowed to span sentence boundaries. This is an important
difference for the wikitext2 dataset because it is composed off documents
where the sentences are in the correct order. Consequently, by allowing the
memory buffer to span sentence boundaries on the wikitext2 dataset these
models were able to carry forward contextual information from preceding
sentences and as a result they did not suffer from as severe a cold-start
problem at the beginning of each sentence. Given this difference the
competitive performance of the Averaging LSTM-LM is encouraging.

The results for the wikitext2 dataset highlights an interesting trade-off and
design choice for memory augmented LSTM-LMs. One approach is to use a dynamic
length memory buffer which resets at sentence boundaries and uses a simple
mechanism, such as averaging, to construct a representation of the memory to
inform the prediction at each timestep. This is the approach we have proposed
in this paper. This approach has the advantages of simplicity (model size) and
that the memory length can be anchored to landmarks in the history, such as
sentence boundaries. This approach is most appropriate for sentence based NLP
tasks such as sentence based Machine Translation. There is a question,
however, regarding whether this approach will scale to very long sequences
(such as documents) as averaging over long-histories may result in all
histories appearing similar. We have done some initial experiments where we
have permitted the memory buffer to hold longer sequences before being reset
and the performance of the Averaging LSTM-LM dipped. The alternative approach
is to use a larger memory buffer and a more sophisticated retrieval mechanism,
for example the Neural cache model of Grave et al. (2017) is a useful exemplar
for this approach with a memory buffer of 2,000 steps. As the wikitext2
results demonstrate this second approach works well for large datasets where
the sentences are in sequence, the cost of this approach being a more complex
architecture.

##  8 Conclusions

In this paper we have highlighted the power of the LSTM gating mechanism and
argued that the persistence dynamics of this mechanism can provide useful
clues regarding what information is important within a sequence for language
modelling. Previous works have demonstrated that LSTMs are capable of
extracting important features about an input sequence and that the dynamics of
the memory block calculates an importance factor for the contribution of each
previous inputs for a given timestep. However, the dynamics of the gates may
drop the information kept in memory in favour of more recent information. Once
dropped information that may have been held for several timesteps vanishes
from the context and is not available anymore to the model. To deal with this
problem, a variety of “memory augmented” LSTM-LMs have been proposed which
store the previous hidden states of an LSTM in order to make the vanished
information available again to subsequent timesteps. However, to date none of
these memory augmented LSTM-LMs have explicitly used the persistence of
information within an LSTM unit to inform the decision regarding what
information should be retrieved from the memory buffer. We argue that ignoring
the internal dynamics of the LSTM is to overlook a useful and computationally
cheap source of information for memory augmented language modeling.

We believe that attending to the information that an LSTM gating mechanism has
decided is important in an input sequence at a given timestep (and hence has
persisted to a later timestep) is a natural way of deciding what information
will be useful again at a subsequent timestep. Even if the information
contained in the LSTM is replaced or altered later in the process, we argue
that it is relevant to the entire history in proportion to the amount of
timesteps it was held. Informed by this hypothesis, in our work we
demonstrated that a simple average of the previous LSTM hidden states in
memory is an effective mechanism for providing information to the current
timestep about previous inputs.

Admittedly, rating the importance of information in terms of the number of
timesteps the LSTM persisted it for is a relatively simplistic view of the
dynamics of LSTM units and of the complexity of language. Furthermore,
implementing this strategy using an average of past states is also a
relatively blunt way of instantiating this approach. However, as our results
demonstrate this simple approach is effective and we understand this is a
starting point. By drawing attention to the signals implicit in the dynamics
of LSTM units we hope to contribute to the development of more efficient LMs.
At the same time, the fact that the internal dynamics of an LSTM unit may be
used to explicitly signal what is important and what should be retrieved from
a memory buffer may suggest alternative constraints and opportunities that
should be considered in the design of neural units and by doing so contribute
to the development of a new class of units for use in RNN-LMs.

## Acknowledgments

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In International Conference on Learning Representations. volume abs/1409.0473v6.
  * Bradbury et al. (2016) James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016.  Quasi-recurrent neural networks.  arXiv abs/1611.01576.
  * Cheng et al. (2016) Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016.  Long short-term memory-networks for machine reading.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 551–561.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  arXiv abs/1406.1078.
  * Collins et al. (2016) Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. 2016.  Capacity and trainability in recurrent neural networks.  arXiv abs/1611.09913.
  * Daniluk et al. (2017) Michal Daniluk, Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel. 2017\.  Frustratingly Short Attention Spans in Neural Language Modeling.  5th International Conference on Learning Representations (ICLR’2017) .
  * Gal and Ghahramani (2015) Yarin Gal and Zoubin Ghahramani. 2015.  A theoretically grounded application of dropout in recurrent neural networks.
  * Gers et al. (2000) Felix A. Gers, Jürgen A. Schmidhuber, and Fred A. Cummins. 2000.  Learning to forget: Continual prediction with lstm.  Neural Comput. 12(10):2451–2471.  https://doi.org/10.1162/089976600300015015.
  * Gers et al. (2003) Felix A. Gers, Nicol N. Schraudolph, and Jürgen Schmidhuber. 2003.  Learning precise timing with lstm recurrent networks.  J. Mach. Learn. Res. 3:115–143.  https://doi.org/10.1162/153244303768966139.
  * Grave et al. (2017) Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017.  Improving neural language models with a continuous cache.  5th International Conference on Learning Representations (ICLR’2017) .
  * Józefowicz et al. (2016) Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016.  Exploring the limits of language modeling.
  * Jozefowicz et al. (2015) Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015.  An empirical exploration of recurrent network architectures.  In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37. ICML’15, pages 2342–2350.
  * Karpathy et al. (2015) Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015.  Visualizing and understanding recurrent networks.  arXiv abs/1506.02078.  http://arxiv.org/abs/1506.02078.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pages 1412–1421.
  * Marcus et al. (1994) Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994.  The penn treebank: Annotating predicate argument structure.  In Proceedings of the Workshop on Human Language Technology. pages 114–119.
  * Melis et al. (2017) Gábor Melis, Chris Dyer, and Phil Blunsom. 2017.  On the state of the art of evaluation in neural language models.  arXiv abs/1707.05589.
  * Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017.  Pointer sentinel mixture models.  5th International Conference on Learning Representations (ICLR’2017) .
  * Mikolov et al. (2010) Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010.  Recurrent neural network based language model.  In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010. pages 1045–1048.
  * Murdoch and Szlam (2017) W. James Murdoch and Arthur Szlam. 2017.  Automatic rule extraction from long short term memory networks.  arXiv abs/1702.02540.  http://arxiv.org/abs/1702.02540.
  * Ostmeyer and Cowell (2017) Jared Ostmeyer and Lindsay Cowell. 2017.  Machine learning on sequential data using a recurrent weighted average.  arXiv abs/1703.01253.  http://arxiv.org/abs/1703.01253.
  * Press and Wolf (2016) Ofir Press and Lior Wolf. 2016.  Using the output embedding to improve language models.  volume abs/1608.05859.
  * Salton et al. (2017) Giancarlo D. Salton, Robert J. Ross, and John D. Kelleher. 2017.  Attentive language models.  In Proceedings of The 8th International Joint Conference on Natural Language Processing (IJCNLP 2017 ).
  * Schwenk et al. (2012) Holger Schwenk, Anthony Rousseau, and Mohammed Attik. 2012.  Large, pruned or continuous space language models on a gpu for statistical machine translation.  In Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT. pages 11–19.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research 15:1929–1958.  http://jmlr.org/papers/v15/srivastava14a.html.
  * Strobelt et al. (2016) Hendrik Strobelt, Sebastian Gehrmann, Bernd Huber, Hanspeter Pfister, and Alexander M. Rush. 2016.  Visual analysis of hidden state dynamics in recurrent neural networks.  arXiv abs/1606.07461.  http://arxiv.org/abs/1606.07461.
  * Tran et al. (2016) Ke M. Tran, Arianna Bisazza, and Christof Monz. 2016.  Recurrent memory network for language modeling.  arXiv abs/1601.01272.
  * Zaremba et al. (2015) Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015.  Recurrent neural network regularization.  volume abs/1409.2329.


# (Un)paired signal-to-signal translation with 1D conditional GANs

Eric Easthope
University of British Columbia
Department of Electrical and Computer Engineering
Vancouver, British Columbia, Canada

I show that a one-dimensional (1D) conditional generative adversarial network
(cGAN) with an adversarial training architecture is capable of unpaired
signal-to-signal (sig2sig) translation. Using a simplified CycleGAN model with
1D layers and wider convolutional kernels, mirroring WaveGAN to reframe two-
dimensional (2D) image generation as 1D audio generation, I show that
recasting the 2D image-to-image translation task to a 1D signal-to-signal
translation task with deep convolutional GANs is possible without substantial
modification to the conventional U-Net model and adversarial architecture
developed as CycleGAN. With this I show for a small tunable dataset that noisy
test signals unseen by the 1D CycleGAN model and without paired training
transform from the source domain to signals similar to paired test signals in
the translated domain, especially in terms of frequency, and I quantify these
differences in terms of correlation and error.

##  1 Introduction

### 1.1. Background.

The past few years have seen a significant rise in research and public
interest in the use of generative machine learning and artificial intelligence
(ML/AI) models for image-to-image translation tasks.

Perhaps one of the more recognizable models is pix2pix [3], a deep generative
model (DGM) and particularly a deep convolutional generative adversarial
network (DCGAN) [2, 7] that is capable of translating between pairs of high-
resolution images within a learned image data domain. The novelty of pix2pix
laid in its model architecture which combined a deep U-Net generator that
learns to generate mock data samples with a convolutional PatchGAN
discriminator that learns to label regions, “patches,” of inputs as “real”
(sampled data) or “fake” (generated data). Much of the research interest in
pix2pix has centred on image translation tasks but the inherent structure of
the U-Net model does not limit it to images alone. In fact original
developments of U-Net were for semantic segmentation [5].

Research into GANs as they stand within the wider DGM and even wider
generative ML/AI ecosystem have not been limited to images either. Parallel
work on one-dimensional (1D) GANs where time series training data is periodic
[1] has observed that derived models that decompose demonstrated two-
dimensional models into 1D counterparts with a wider learning aperture, which
we set ourselves with the size of convolution kernels, are capable of
generating convincing high-accuracy 1D time series (including audio) from a
learned signal data domain. Wider convolutional apertures are necessary for
models to see and learn the time series periodicity. Others before have taken
the conceptual essence of signal-to-signal translation and adapted its
generator U-Net models for other signal domains; spectrum translation [6]
(spectral/frequency series-to-series), sensor translation [4] (time series-to-
series, 2D), and sound translation [9] (time series-to-series, 1D) to name a
few. [6] focused on the translation of sensor-to-sensor to adapt sensor
conditions for object/material detection. [4] focused on the re-use of sensor
models, translating old sensor measurements to new ones; they used a modified
2D pix2pix-like architecture to do this. [9] focused on timbre-to-timbre
translation in (optionally musical) audio signals, accomplishing this in 1D
but again through what was actually a spectral-to-spectral translation like
[6].

All of these share the idea of using machine learning to translate between
types of windowed signals whether expressed in one or two-dimensional bases.
The takeover of research interest in generic sequence-to-sequence and
equivalently series-to-series translation models and their continued capacity
to surpass many of the performance metrics that were previously maximized by
more specialized convolutional neural networks and the like suggest that some
model generality is possible. WaveGAN [1] showed that a general treatment of
2D convolutional models could make 1D convolutional counterparts suitable to
generate audio with only a change of dimension and data shape. Still, where
this generality starts and stops is unknown and to my knowledge little has
been done to establish the generality of these models across 1D/2D signal
paradigms.

### 1.2. Objective.

This work aims to establish by existence proof that the practice of
transcribing convolutional models between 2D to 1D and possibly other
dimensional configurations has sufficient generality to solve several signal-
to-signal translation problems. To this end I focus first on the 1D unpaired
signal-to-signal translation task.

##  2 Methods & Materials

### 2.1. Model.

In the interests of model re-usability I use a model that is readily available
online: a modified CycleGAN [8] architecture that combines simplified versions
of the generator and discriminator models (using only three U-Net
down/upsampling layers) from the pix2pix [3] architecture replacing 2D layers
with 1D layers and widening convolutional kernels to roughly the square of
their size. Notably CycleGAN uses the same generator and discriminator as
pix2pix, differing in function only by training on paired/unpaired data and
using a different training loss; CycleGAN combines two pix2pix models to learn
signal translations to and from both signal translation domains by minimizing
“roundtrip” error.

This model was trained over the course of a few minutes using the standard
CycleGAN losses, $\lambda=10$, and $\beta_{1}=0.5$ for 100 epochs with a batch
size of one using TensorFlow on an M1 MacBook Pro. Signals and their
translated counterparts were scored using Pearson product-moment correlation
($r$-value) and mean absolute error (MAE).

### 2.2. Dataset.

A mock dataset simulates randomly windowed samples from a bandlimited signal
where we can vary the window length, bandwidth, and maximum phase offset to
study the capacities of the simplified 1D CycleGAN model. I use a small
collection of sixteen tunably (a)synchronous periodic bandlimited signals
sampled reproducibly with a reusable randomness seed; each signal is made up
of multiple sine waves with seed-generated random amplitudes, phase offsets
(up to $2\pi$ radians), and frequency offsets. The (a)synchronicity of
paired/unpaired signals is controllable through the extent of phase offsets. A
sample of dataset elements is shown below.

Figure 1: Sixteen randomly sampled dataset elements used for 1D CycleGAN
training between one-dimensional signal domains (blue/purple lines) with
arbitrary scales.

##  3 Results

Figure 2: Paired signal-to-signal translations by the simplified 1D CycleGAN
architecture against a small unseen four-element test dataset in the time
domain ($r$-values in discussion). Figure 3: Paired signal-to-signal
translations by the simplified 1D CycleGAN architecture against a small unseen
four-element test dataset in the frequency domain after discrete Fast Fourier
Transform. Frequency-wise signals seem to match more than time-wise signals
above ($r$-values in discussion). Figure 4: Signal-to-signal translations
(supplementary anonymized dataset) by the simplified CycleGAN architecture,
which combines elements of the pix2pix architecture with a different training
pattern and loss for unpaired signal data. Differences in reconstruction are
shown at both signal $X$ and signal $Y$ translation domains (compared as
solid/transparent lines).

##  4 Discussion

The simplified 1D CycleGAN model performs reasonably well with the synthetic
dataset despite relatively little training (less than 10 minutes); for all
four test samples (0.21 $<$ r $<$ 0.46 and (0.032 $<$ MAE $<$ 0.037) in the
time domain and (0.71 $<$ r $<$ 0.89) and (0.13 $<$ MAE $<$ 0.15) in the
frequency domain. Yet while the 1D CycleGAN model is trained on unpaired data
it seems to be capable of accurately transforming between paired signals never
seen in training while also conserving much of their frequency content. This
suggests interesting potential for CycleGAN and similar convolutional
architectures to learn translations between (a)synchronous 1D signals.

Further testing with a larger synthetic data schema and several performance
measures is needed to fully validate CycleGAN as a viable option for 1D
signal-to-signal translation in more practical and wild data contexts.

##  5 Conclusion

I showed that a common generative architecture for 2D image-to-image
translation model could be transformed into a 1D model by applying some of the
same concepts as used by [1] to transform the 2D image DCGAN model [7] into a
1D audio model. I showed that CycleGAN [8], which applies the same principles
as [3] but with unpaired training data, performs with a small mock dataset of
tunable periodic signals even without paired training suggesting some
potential for CycleGAN and similar convolutional architectures to learn
translations between (a)synchronous 1D signals.

## References

  * [1] Chris Donahue, Julian McAuley, and Miller Puckette.  Adversarial Audio Synthesis, February 2019.  arXiv:1802.04208 [cs].
  * [2] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Networks, June 2014.  arXiv:1406.2661 [cs, stat].
  * [3] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-Image Translation with Conditional Adversarial Networks, November 2018.  arXiv:1611.07004 [cs].
  * [4] SangYeon Kim, Hyunwoo Lee, Jonghee Han, and Joon-Ho Kim.  Sig2Sig: Signal Translation Networks to Take the Remains of the Past.  In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3620–3624, June 2021.  ISSN: 2379-190X.
  * [5] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully Convolutional Networks for Semantic Segmentation, March 2015.  arXiv:1411.4038 [cs].
  * [6] Cara P. Murphy and John Kerekes.  1D conditional generative adversarial network for spectrum-to-spectrum translation of simulated chemical reflectance signatures.  Journal of Spectral Imaging, 10, June 2021.
  * [7] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, January 2016.  arXiv:1511.06434 [cs].
  * [8] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, August 2020.  arXiv:1703.10593 [cs].