# Improving Replay Sample Selection and Storage for Less Forgetting in
Continual Learning

Daniel Brignac
University of Arizona
Tucson, Arizona
<EMAIL_ADDRESS>Niels Lobo
University of Central Florida
Orlando, Florida
<EMAIL_ADDRESS>Abhijit Mahalanobis
University of Arizona
Tucson, Arizona
<EMAIL_ADDRESS>
###### Abstract

Continual learning seeks to enable deep learners to train on a series of tasks
of unknown length without suffering from the catastrophic forgetting of
previous tasks. One effective solution is replay, which involves storing few
previous experiences in memory and replaying them when learning the current
task. However, there is still room for improvement when it comes to selecting
the most informative samples for storage and determining the optimal number of
samples to be stored. This study aims to address these issues with a novel
comparison of the commonly used reservoir sampling to various alternative
population strategies and providing a novel detailed analysis of how to find
the optimal number of stored samples.

##  1 Introduction

Deep learning has revolutionized the field of computer vision, achieving
human-like capabilities of image understanding and perception. Unlike humans,
however, deep learners consistently struggle to adopt new knowledge while
maintaining performance on previously learned tasks. This is the problem known
as catastrophic forgetting [25] in which a learner’s performance significantly
diminishes as it acquires knowledge for new tasks. This motivates the study of
continual learning [30, 38] to address the problem of catastrophic forgetting.

In continual learning, a learner is presented with a sequence of tasks of
unknown length where the only data available is that of the current task at
hand. As new tasks arrive, we wish to learn each new task while preserving the
knowledge learned from previous tasks. During inference, we may be presented
with data from any of the previously learned tasks, thus the retention of
previous knowledge is imperative when adapting to new tasks.

The inference stage primarily comes in three flavors: task-incremental
learning (task-IL), domain-incremental learning (domain-IL) and class-
incremental learning (class-IL) [39]. Task-IL and domain-IL are generally
considered the easier scenarios as we are either given the task-ID at test
time in task-IL, or we must only solve for the current task at hand in domain-
IL. Class-IL is significantly more challenging as we must infer for all tasks
seen so far without the revealing of a task-ID. Because of this, class-IL has
become a primary focus of recent continual learning works [3, 27, 5, 36].

Replay [29, 32, 4] is a commonly used approach to remedy the problem of
catastrophic forgetting in both class-IL and task-IL. The concept of replay
draws inspiration from Complementary Learning Systems theory of humans which
posits that recent experiences stored in the hippocampus develop connections
to the neocortex that become ingrained over time to eventually be encoded in
long-term memory [24, 13]. As such, when replay is employed in artificial
learners, a small amount of previously learned data is stored in memory to
then be “replayed” during the training of the current task to emphasize the
previously encoded connections of the learner and thus avoid forgetting.

When using replay methods, it is important to have a small canonical set of
exemplars stored in memory that effectively capture the underlying class
distributions of the dataset. Thus, the selection of _which samples to store_
is a non-trivial task. Previous replay methods rely on reservoir sampling [32,
4, 1, 5, 2] to populate memory with past data. As reservoir sampling is a
random sampling method, this could lead to the storage of redundant and
potentially insignificantly informative data points in memory causing replay
methods to not perform to their maximal capability as recently demonstrated in
[37, 43].

There is also no consensus regarding _how many samples should be stored_ to be
used for replay. Naturally, as we store more samples in memory, we expect
performance to increase, however, as this number grows, we start to deviate
from the constraints of continual learning where we seek to store minimal
information. There must exist some small optimal number of samples to be
stored to make maximum use of replay methods.

In this work we study the two questions _which samples to store_ , and _how
many samples should be stored_ in memory by comparing the commonly used
approach of reservoir sampling to three other memory population strategies. We
show through extensive empirical evaluation that reservoir sampling leads to
greater forgetting when compared with more strategic population approaches. We
additionally detail two methods to address the question of how many samples
should be stored based on an analysis of significant eigenvectors and
eigenvalues. We proceed to show that memory populated according to these two
criteria leads to overall more competitiveness and better performance of all
population strategies.

##  2 Related Works

Continual learning methods can generally be grouped into three categories: (i)
regularization methods, (ii) architectural methods and (iii) replay methods.
Regularization methods introduce a new loss term meant to penalize significant
drift from previously learned parameters [19, 22, 17, 41]. Architectural
methods seek to adapt existing network architectures such that various
portions of the model contain global and/or shared knowledge while others
contain task-specific knowledge [26, 44, 35, 33, 27]. Historically,
regularization methods and architectural methods underperform when compared
directly to replay methods further motivating the study of and improvements to
replay.

In replay methods, we are allowed to store some small subset of data in a
memory buffer and selectively “replay” samples from this buffer when training
on the current task to maintain previous task performance. Earlier works
include [23] and [10], both of which use the memory buffer as a constraint on
gradient updates to ensure that loss remains low on the buffer samples. These
gradient-based methods for replay in general show poor performance when
compared to experience replay methods [29, 32, 4, 1, 9, 7, 5]. In experience
replay, it is common practice to store any of the raw data sample, the label,
the logits, or a combination of all three. All the the aforementioned
experience replay methods select the samples to be stored by some random
sampling method, such as reservoir sampling or uniform sampling, leading to
the potential storage of insignificant data.

A number of previous works suggest to either populate the memory buffer with
some fixed arbitrary number of samples as new classes are encountered or to
empirically find optimal buffer size for a desired task performance [1, 31].
These methods for expanding the buffer are rooted in heuristics and thus not
necessarily optimal further motivating the study for optimal growth of the
buffer for learning new tasks.

##  3 Methodology

Algorithm 1 Memory Buffer Population

$\mathcal{T}$, $\mathcal{D}_{t}$, $\mathcal{M}$, buffer size (if not dynamic),
populate $\in$ {_reservoir, herding, GSS, IPM_}, training-strategy

$\mathcal{M}\leftarrow\emptyset$

for $t$ in $\mathcal{T}$ do

for  _minibatch_ in $\mathcal{D}_{t}$ do

if $\mathcal{M}=\emptyset$ then

 _train_($\mathcal{D}_{t}$) according to training-strategy

if populate $\in$ {_resrvoir, GSS_}, then

$\mathcal{M}\leftarrow\text{populate}(minibatch,\text{ buffer size})$

else

 _train_($\mathcal{D}_{t}\cup\mathcal{M}$) according to training-strategy

if populate $\in$ {_resrvoir, GSS_}, then

$\mathcal{M}\leftarrow\text{populate}(minibatch,\text{ buffer size})$

if populate $\in$ {_herding, IPM_}, then

$\mathcal{M}\leftarrow\text{populate}(\mathcal{D}_{t},\text{ buffer size}$)

Consider a sequence of $\mathcal{T}$ tasks where each task has an associated
dataset $\mathcal{D}_{t}=\\{(x_{i}^{t},y_{i}^{t})\\}$ for each
$t\in\\{1,2,...,\mathcal{T}\\}$ and $i=1,...,N_{t}$ where $x_{i}^{t}$ denotes
the $i^{th}$ sample of the $t^{th}$ task, $y_{i}^{t}$ its associated ground
truth label, and $N_{t}$ the number of samples in task $t$. We assume each
task $t$ contains a unique, non-overlapping set of classes drawn from an i.i.d
distribution. In our continual learning formulation, we seek to sequentially
learn each $\mathcal{D}_{t}$ in an offline setting while maintaining
performance on every $\mathcal{D}_{k}$ for $k<t$ by employing a memory buffer
$\mathcal{M}$ which is populated according to a specific population strategy.
For $t=1$, we train only on $\mathcal{D}_{t}$ and for $t>1$, we train on
$\mathcal{D}_{t}\cup\mathcal{M}$.

Since we are primarily concerned with strategic buffer population, the
training method in which we perform continual learning is agnostic of the
memory buffer population. Thus, we can use any training strategy along with
any buffer population strategy studied herein. We give an overview of each
population strategy studied and then present a novel scheme to identify how
many samples per class should be stored in $\mathcal{M}$.

We make a note of the distinction between fixed memory buffers and dynamic
memory buffers. A fixed memory buffer refers to a buffer $\mathcal{M}$ that is
fixed in the amount of data it can hold (_e.g_., a fixed buffer size of 200
can contain a maximum of 200 data samples) while a dynamic buffer may grow in
size as new classes are encountered. We denote the size of the fixed buffer as
$|\mathcal{M}|$. Traditionally, fixed memory buffers are used whenever replay
methods are employed. To the best of our knowledge, we are the first to
consider dynamic memory buffers as discussed in Section 3.5. When the buffer
is not described to be either fixed or dynamic, the method is agnostic of
buffer type. An overview of our described approach is given in Algorithm 1.

###  3.1 Reservoir Sampling

In reservoir sampling [40], we are presented with a stream of data in which we
randomly sample from the stream and store each sample in $\mathcal{M}$. If a
sample is selected to be stored when $\mathcal{M}$ is saturated, we then
randomly replace a sample in $\mathcal{M}$ with the current selected sample to
be stored.

In practice, reservoir sampling tends to favor the storage of samples from
earlier encountered tasks. This causes the minimal storage of samples from
downstream tasks and an overall unbalanced fixed buffer leading to greater
forgetting, particularly when the fixed buffer size is small.

In addition to favoring the storage of earlier encountered task data,
reservoir sampling has no mechanism to differentiate between whether a
selected sample to be stored is informative or redundant. This leads to the
potential storage of insignificant data which in turn can diminish the
network’s previously learned decision boundaries [3]. This motivates the study
of memory buffer population strategies that can always store the next most
informative sample and mitigate forgetting.

###  3.2 The Herding Algorithm

A natural first choice of substitute to reservoir sampling is the herding
algorithm as proposed in iCaRL [29] which is an extension of Welling’s herding
in [42]. Herding seeks to store samples that best represent the sample’s class
mean in feature space. In this sense, the herding algorithm can be thought of
as a greedy mean preserving scheme as each selected sample is the closest to
its learned class mean.

Formally, for a learned class mean $\mu_{c}$ in $D$-dimensional feature space
and a mapping from image space to feature space given by
$\phi:x_{i}^{t,c}\rightarrow\mathbb{R}^{D}$ where $c$ denotes the class label
of image $x_{i}^{t}$, herding seeks to find the sample $x_{i}^{t,c}$ to add to
the class exemplar set $P_{c}$ in memory buffer $\mathcal{M}$ that minimizes
the distance to $\mu_{c}$ as

$\underset{x_{i}^{t,c}}{\operatorname{argmin}}\left|\left|\mu_{c}-\frac{1}{K}\left(\phi(x_{i}^{t,c})+\sum_{j=1}^{K-1}\phi(p_{j})\right)\right|\right|_{2}$
(1)

where $K$ denotes the number of samples to be stored for class $c$ and $p_{j}$
denotes a sample $j$ belonging to $P_{c}$.

Because herding relies on the well learned features for each class to store
samples, herding should be performed at the conclusion of each task. This
makes herding most suitable for the offline continual learning scenario in
which we are allowed multiple iterations through $\mathcal{D}_{t}$ before
moving to the next subsequent task.

###  3.3 Gradient Sample Selection

Gradient sample selection (GSS) is taken from [2] where the objective is to
optimize the loss on current task data while maintaining minimal loss on
previous task data. To ensure that loss remains low on previous tasks, GSS
seeks to maximize the diversity of the buffer $\mathcal{M}$ by selecting
samples whose gradient angle is maximal compared to all other samples
currently stored (_i.e_., maximizing gradient direction variance). This
formulation involves solving a quadratic integer programming of polynomial
complexity w.r.t. the buffer size thus, a greedy formulation of GSS is used
instead.

In the greedy method, GSS maintains a score $r_{i}$ for each sample $i$ in the
buffer based on the maximal cosine similarity of the current sample with the
other samples in the buffer given by

$r_{i}=\underset{i}{\operatorname{max}}\frac{\langle
g_{i},G\rangle}{||g_{i}||_{2}||G||_{2}}$ (2)

where $g_{i}$ and $G$ denote the gradients of the current sample $i$ and the
set of samples stored in $\mathcal{M}$ respectively.

While GSS aims to select and store samples based on maintaining maximum
variance of gradient direction, there is still a randomness component in
determining samples to potentially be replaced. When a sample is selected and
deemed appropriate for replacement, there is no guarantee that this is a least
informative sample, as determined by GSS, and could thus potentially lead to
greater forgetting. In addition, GSS has no mechanism for class balancing as
samples are randomly added and replaced which can further hinder performance.

###  3.4 Iterative Projection and Matching

Iterative projection and matching (IPM) [16] is drawn from active learning in
which we seek to find the most informative data points for training. Here, we
adopt IPM to select the most informative data points for storage in
$\mathcal{M}$ to address the shortcomings with reservoir sampling while also
maintaining a balanced buffer similar to the strategy used in herding.

Let $\boldsymbol{A}_{c}\in\mathbb{R}^{N_{c}\times D}$ denote the matrix of
features $\phi(x_{1}^{t}),...,\phi(x_{N_{c}}^{t})\in\mathbb{R}^{D}$ where
$N_{c}$ denotes the number of samples in class $c$, $D$ the dimension of
features, and $\phi:\mathbb{R}^{H\times W\times C}\rightarrow\mathbb{R}^{D}$ a
mapping from image space to feature space (_i.e_., the learned network). The
$n^{th}$ row in $\boldsymbol{A}_{c}$ is formed by $\phi(x_{n_{c}}^{t})^{T}$.
We seek to reduce $\boldsymbol{A}_{c}$ to some
$\boldsymbol{A}_{R}\in\mathbb{R}^{K\times D}$ where $K$ is the number of
samples to be stored from class $c$.

Let $T\subset\\{1,...,N_{c}\\}$ with $|T|=K$ be the set of selected samples.
Then, we can project the rows of $\boldsymbol{A}_{c}$ onto the span of the
selected rows indexed by $T$. We denote this operation as
$\boldsymbol{\omega}_{T}(\boldsymbol{A}_{c})$. As done in [11], we can now
cast the replay sample selection problem as the optimization problem

$\underset{|T|=K}{\operatorname{argmin}}||\boldsymbol{A}_{c}-\boldsymbol{\omega}_{T}(\boldsymbol{A}_{c})||_{F}^{2}$
(3)

where $||\cdot||_{F}$ is the Frobenius norm. This problem is NP-Hard however
as we must search all subsets $T$ over $\boldsymbol{A}_{c}$ [46]. Thus, we use
the IPM [16] algorithm to approximate 3.

We can express $\boldsymbol{\omega}_{T}(\boldsymbol{A}_{c})$ as a rank-$K$
factorization $\boldsymbol{U}\boldsymbol{V}^{T}$ where
$\boldsymbol{U}\in\mathbb{R}^{N_{c}\times K}$ and
$\boldsymbol{V}^{T}\in\mathbb{R}^{K\times D}$ and modify 3 by recasting it as
two sub-problems [16, 15]

$(u,v)=\underset{u,v}{\operatorname{argmin}}||\boldsymbol{A}_{c}-\boldsymbol{u}\boldsymbol{v}^{T}||_{F}^{2}\text{
s.t. }||\boldsymbol{v}||=1$ (4)
$m^{(1)}=\underset{m}{\operatorname{argmax}}|\boldsymbol{v}^{T}\boldsymbol{\rho}|$
(5)

where $\boldsymbol{\rho}=\phi(x_{n_{c}}^{t})/||\phi(x_{n_{c}}^{t})||_{2}$,
$m^{(1)}$ is the index of the first selected data point, and
$(x_{i}^{t},y_{i}^{t})_{m^{(1)}}$ is the selected point to be stored in
memory.

IPM also relies on the well learned features of each class for best
performance in sample selection and thus must be performed at the conclusion
of a task. This also makes IPM most suitable for the offline continual
learning scenario.

Similar to herding, we again store $|\mathcal{M}|/s$ samples per class for a
balanced and saturated buffer and preserve the most informative samples, as
determined by IPM, by deleting the most recently added samples to each class
when new data is encountered and must be stored to the fixed buffer.

###  3.5 Dynamic Memory Buffers

Traditionally, when replay methods are used, it is common for the buffer size
to be fixed [32, 5, 4]. This fixed buffer size does not take into account the
underlying complexity of the data and more specifically, each class within the
data. To account for such dataset specific complexities, we allow the buffer
to be dynamic, where we add $K$ samples of class $c$ to the buffer using two
algorithm agnostic 111Algorithm agnostic refers to each population strategy as
each strategy will choose different subsets of samples bounded by each studied
dynamic buffer criterion. methods we refer to as intracluster variance and
Kaiser criterion described below. This idea is motivated by providing a
guideline for determining the number of samples needed to represent the data
manifold and the different classes within it as opposed to arbitrarily
choosing a fixed buffer size.

Intracluster Variance. We assume that the number of images necessary for
replay depends on the complexity of the underlying manifold of the data. It is
well known that relations between the data points on the high dimensional
manifold are preserved when the data is embedded into a lower dimension via
norm preserving transformations [6, 12]. For example, the dominant
eigenvectors (or principal components) of the space in which the images lie is
an example of one such transformation. In fact, the number of eigenvectors
needed to represent the data is an indication of the complexity of the
manifold, and therefore provides some indication of the number of images
required for replay. Another method for estimating the complexity of the
manifold is by forming clusters in the data and by observing the change in the
average intracluster variance for each cluster.

Assume that the data is grouped into $K$ clusters, each with $N_{j}$ images.
Specifically, let $x_{i,j}$ for $1\leq i\leq N_{j}$, $1\leq j\leq K$ represent
the $i^{th}$ training image of the $j^{th}$ cluster. The variance of any given
cluster is given by
$\sigma_{j}^{2}=\frac{1}{N_{j}}\sum_{i=1}^{N_{j}}(x_{i,j}-\mu_{j})$ where
$\mu_{j}=\frac{1}{N_{j}}\sum_{i=1}^{N_{j}}x_{i,j}$ is the mean of the
respective cluster. The average variance of all clusters is then simply
$\sigma_{K}^{2}=\frac{1}{K}\sum_{j=1}^{K}\sigma_{j}^{2}$. The premise is that
each cluster represents a local region on the manifold where the data is
concentrated. As $K$ increases, each cluster becomes more and more compact and
their variance $\sigma_{j}^{2}$ for $1\leq j\leq K$ decreases. This causes the
average variance $\sigma_{K}^{2}$ to also decrease as shown in Figure 1(a)
using the CIFAR10, CIFAR100 and Tiny-ImageNet datasets [20, 21] and the
$k$-means clustering algorithm. In fact, the change in variance
$\Delta_{K}=\sigma_{K+1}^{2}-\sigma_{K}^{2}$ also tapers asymptotically as $K$
is allowed to increase. Figure 1(b) shows this behavior and the rate at which
the average variance decreases.

(a)

(b)

Figure 1: (a) Average intracluster variance for varying number of clusters.
(b) Average change in intracluster variance for varying number of clusters.

We observe that increasing the number of clusters beyond the knee of the
curve, 100 for both CIFAR10 and CIFAR100 and 160 for Tiny-ImageNet, provides
diminishing gains in terms of decreasing the variance of each cluster. We
assume that each cluster is as compact as possible, and the cluster mean is a
good representation of the data that lies within the cluster. Therefore, we
choose the number of clusters where the knee of the curve (found using the
Kneedle algorithm [34]) in 1(a) and 1(b) occur as an indication of the number
of representative samples that are required to adequately represent the data
manifold.

With this estimate of the number of samples needed for proper representation
of the underlying data manifold, we populate the buffer with $K$ samples per
class. This allows for the buffer to maintain its class balanced properly and
avoid biased sampling from the buffer when training.

When using instracluster variance, one must have access to the entire dataset
before training begins. This can be seen as unfair in the context of continual
learning since in real world scenarios it would be impossible to view the
entire dataset beforehand. We address this concern by using the Kaiser
criterion below.

Kaiser Criterion. Instead of finding a global number of samples to be kept for
each class as done in intracluster variance, we detail an additional method
based on the number of most useful eigenvectors for each class.

Assume that $\mathbf{x}_{i,c}$ is a $d$ dimensional column vector that
represents the $i^{th}$ training image of class $c$ for $1\leq i\leq N_{c}$.
For each class, we define the data matrix
$\boldsymbol{A}_{c}=[\mathbf{x}_{1,c},...,\mathbf{x}_{1,N_{c}}]$, and compute
the eigenvalues and eigenvectors of $\boldsymbol{A}_{c}\boldsymbol{A}_{c}^{T}$
denoted as $\lambda_{k,c}$ and $\boldsymbol{\phi}_{k,c}$, respectively, for
$k=1,...,d$. Loosely speaking, the rank of this matrix is a proxy for the
number of independent images required to represent the dataset and can thus be
viewed as a reduced version of PCA. This in turn is related to the non-
singular eigenvalues. Therefore, to determine the minimum number of images
required to represent the data, we count the number of non-trivial eigenvalues
(ignoring the ones that are zero or close to it). The Kaiser criterion [18] is
a common method for selecting the useful eigenvectors which states that only
those with eigenvalues greater than 1.0 should be retained for representing
the data.

Dataset | Mean | Min. | Max
---|---|---|---
CIFAR10 | $248\pm 27$ | 215 | 290
CIFAR100 | $50\pm 8$ | 28 | 69
TinyImageNet | $59\pm 9$ | 41 | 94
Table 1: Kaiser Criterion stats for number of samples required for each class.

We note that this method of using raw images to form the data matrix is better
suited than using network features, as that would always require the network
to accurately classify the entire data, which manot not occur if a particular
class is difficult for the network to classify.

We report the statistics of the Kaiser criterion for each dataset in Table 1.
The benefits of using a dynamic buffer with the Kaiser criterion allow for
classes with higher complexity to have more representation in the buffer,
which ultimately leads to higher probability of sampling these more complex
classes from the buffer and further mitigate forgetting.

Additionally, the Kaiser criterion can be used in an offline manner before the
training of each task as the determination for the number of samples to be
stored for each class depends only on a specific class’s data matrix. Thus,
when new task data is available, we can compute the Kaiser criterion for each
individual class by partitioning the task data matrix into class data
matrices.

|  |  | Split-CIFAR10 | Split-CIFAR100 | Split-TinyImageNet
---|---|---|---|---|---
Fixed Buffer Size | Method |  Population Strategy | Class-IL | Class-IL | Class-IL
|  |  | _FAA_ | _FF_ | _FAA_ | _FF_ | _FAA_ | _FF_
200 | ER | Reservoir | $48.39\pm 2.01$ | $60.39\pm 2.26$ | $15.35\pm 0.86$ | $81.20\pm 0.62$ | $8.40\pm 0.16$ | $76.88\pm 0.18$
Herding | $\mathbf{52.32\pm 0.77}$ | $\mathbf{55.17\pm 0.77}$ | $\mathbf{15.94\pm 0.46}$ | $\mathbf{79.60\pm 0.11}$ | $\mathbf{8.81\pm 0.05}$ | $\mathbf{76.61\pm 0.62}$
GSS | $41.45\pm 4.65$ | $68.92\pm 5.57$ | $11.87\pm 0.05$ | $84.34\pm 0.41$ | - | -
IPM | $48.68\pm 1.11$ | $59.44\pm 1.32$ | $15.0\pm 0.26$ | $80.67\pm 0.31$ | $8.55\pm 0.09$ | $77.13\pm 0.44$
DER | Reservoir | $\mathbf{61.17\pm 1.08}$ | $41.27\pm 0.53$ | $24.38\pm 1.46$ | $70.07\pm 1.76$ | $11.15\pm 0.46$ | $74.22\pm 0.65$
Herding | $30.01\pm 1.87$ | $83.12\pm 2.01$ | $9.99\pm 0.23$ | $87.73\pm 0.50$ | $5.7\pm 1.06$ | $71.92\pm 1.87$
GSS | $38.04\pm 6.09$ | $71.95\pm 7.60$ | $12.29\pm 1.70$ | $78.05\pm 0.53$ | - | -
IPM | $60.48\pm 0.34$ | $\mathbf{30.12\pm 1.03}$ | $\mathbf{33.47\pm 1.86}$ | $\mathbf{42.32\pm 1.71}$ | $\mathbf{19.36\pm 1.06}$ | $\mathbf{45.46\pm 0.63}$
GDumb | Reservoir | $29.26\pm 1.18$ | N/A | $4.63\pm 0.49$ | N/A | $2.13\pm 0.28$ | N/A
Herding | $\mathbf{32.16\pm 1.51}$ | N/A | $\mathbf{7.02\pm 0.51}$ | N/A | $\mathbf{3.45\pm 0.30}$ | N/A
GSS | $28.35\pm 1.19$ | N/A | $4.81\pm 0.29$ | N/A | - | N/A
IPM | $31.60\pm 1.83$ | N/A | $6.26\pm 0.14$ | N/A | $2.67\pm 0.36$ | N/A
ER-ACE | Reservoir | $\mathbf{63.32\pm 2.40}$ | $18.62\pm 2.24$ | $28.78\pm 0.66$ | $44.40\pm 1.13$ | $12.82\pm 0.11$ | $48.91\pm 1.65$
Herding | $61.66\pm 0.35$ | $33.04\pm 3.37$ | $\mathbf{29.64\pm 0.29}$ | $62.08\pm 0.27$ | $\mathbf{15.75\pm 0.09}$ | $64.86\pm 1.45$
GSS | $26.25\pm 9.53$ | $\mathbf{1.02\pm 1.01}$ | $7.525\pm 0.13$ | $\mathbf{8.93\pm 0.75}$ | - | -
IPM | $49.41\pm 0.56$ | $16.56\pm 0.09$ | $28.36\pm 0.28$ | $27.17\pm 0.23$ | $15.02\pm 0.53$ | $\mathbf{29.16\pm 0.35}$
500 | ER | Reservoir | $61.07\pm 0.64$ | $44.27\pm 1.10$ | $21.37\pm 1.27$ | $73.55\pm 0.77$ | $10.19\pm 0.20$ | $75.34\pm 0.16$
Herding | $\mathbf{64.09\pm 0.73}$ | $\mathbf{39.26\pm 1.39}$ | $\mathbf{24.25\pm 0.70}$ | $\mathbf{70.46\pm 0.57}$ | $\mathbf{10.38\pm 0.16}$ | $75.50\pm 0.35$
GSS | $61.07\pm 0.64$ | $44.27\pm 1.10$ | $21.37\pm 1.27$ | $73.55\pm 0.77$ | - | -
IPM | $61.08\pm 0.46$ | $44.33\pm 0.68$ | $22.06\pm 0.48$ | $72.80\pm 0.33$ | $10.20\pm 0.17$ | $\mathbf{74.98\pm 0.18}$
DER | Reservoir | $\mathbf{70.07\pm 0.95}$ | $29.5\pm 1.80$ | $34.53\pm 1.68$ | $56.30\pm 1.52$ | $17.15\pm 1.40$ | $66.91\pm 1.81$
Herding | $48.20\pm 2.94$ | $54.64\pm 7.93$ | $13.11\pm 0.51$ | $84.04\pm 0.69$ | $5.32\pm 0.78$ | $67.13\pm 2.02$
GSS | $45.94\pm 6.22$ | $59.09\pm 9.16$ | $16.64\pm 1.69$ | $72.323\pm 5.59$ | - | -
IPM | $65.5\pm 2.68$ | $\mathbf{23.55\pm 5.19}$ | $\mathbf{40.51\pm 0.43}$ | $\mathbf{28.97\pm 1.06}$ | $\mathbf{20.49\pm 0.86}$ | $\mathbf{31.54\pm 2.61}$
GDumb | Reservoir | $\mathbf{43.35\pm 0.55}$ | N/A | $9.85\pm 0.45$ | N/A | $3.6\pm 0.004$ | N/A
Herding | $42.85\pm 0.83$ | N/A | $\mathbf{11.45\pm 0.42}$ | N/A | $\mathbf{4.83\pm 0.19}$ | N/A
GSS | $37.39\pm 1.21$ | N/A | $6.2\pm 0.29$ | N/A | - | N/A
IPM | $42.03\pm 3.05$ | N/A | $9.02\pm 0.67$ | N/A | $3.27\pm 0.40$ | N/A
ER-ACE | Reservoir | $\mathbf{72.15\pm 0.38}$ | $13.18\pm 1.26$ | $\mathbf{37.60\pm 0.15}$ | $38.17\pm 1.16$ | $\mathbf{20.99\pm 0.52}$ | $46.60\pm 0.55$
Herding | $69.75\pm 1.90$ | $23.00\pm 4.32$ | $37.54\pm 0.44$ | $53.08\pm 0.47$ | $20.3\pm 0.36$ | $60.48\pm 0.06$
GSS | $19.54\pm 0.09$ | $\mathbf{0.22\pm 0.11}$ | $8.34\pm 0.09$ | $\mathbf{7.73\pm 0.62}$ | - | -
IPM | $54.43\pm 0.99$ | $14.57\pm 1.17$ | $35.01\pm 0.14$ | $25.50\pm 0.91$ | $18.5\pm 0.42$ | $\mathbf{34.41\pm 8.08}$
5120 | ER | Reservoir | $83.18\pm 1.82$ | $14.33\pm 1.65$ | $50.71\pm 0.27$ | $38.92\pm 0.44$ | $27.36\pm 0.03$ | $54.46\pm 0.69$
Herding | $\mathbf{85.56\pm 0.34}$ | $12.22\pm 0.29$ | $\mathbf{52.93\pm 1.55}$ | $\mathbf{35.21\pm 0.93}$ | $\mathbf{28.43\pm 0.51}$ | $\mathbf{52.12\pm 0.51}$
GSS | $60.35\pm 7.06$ | $43.74\pm 8.69$ | $17.52\pm 0.22$ | $78.74\pm 2.28$ | - | -
IPM | $85.19\pm 0.63$ | $\mathbf{10.97\pm 0.91}$ | $50.75\pm 1.16$ | $36.96\pm 0.60$ | $27.39\pm 0.23$ | $53.43\pm 0.12$
DER | Reservoir | $\mathbf{83.35\pm 0.72}$ | $11.27\pm 0.96$ | $57.22\pm 0.24$ | $22.86\pm 1.51$ | $\mathbf{37.09\pm 0.50}$ | $31.91\pm 1.05$
Herding | $76.21\pm 1.08$ | $25.26\pm 1.63$ | $52.53\pm 0.24$ | $38.70\pm 0.38$ | $5.09\pm 2.59$ | $38.44\pm 5.37$
GSS | $45.86\pm 16.34$ | $54.54\pm 21.02$ | $56.32\pm 1.10$ | $45.05\pm 7.04$ | - | -
IPM | $67.75\pm 0.52$ | $\mathbf{6.25\pm 1.21}$ | $\mathbf{57.30\pm 0.51}$ | $\mathbf{8.85\pm 0.52}$ | $34.33\pm 0.93$ | $\mathbf{7.27\pm 0.98}$
GDumb | Reservoir | $\mathbf{79.89\pm 1.29}$ | N/A | $\mathbf{42.52\pm 0.22}$ | N/A | $\mathbf{21.18\pm 0.06}$ | N/A
Herding | $77.16\pm 0.74$ | N/A | $36.80\pm 0.57$ | N/A | $17.38\pm 0.40$ | N/A
GSS | $70.27\pm 2.29$ | N/A | $19.64\pm 1.40$ | N/A | - | N/A
IPM | $79.58\pm 0.93$ | N/A | $42.31\pm 0.18$ | N/A | $20.99\pm 0.72$ | N/A
ER-ACE | Reservoir | $83.67\pm 0.26$ | $4.93\pm 0.30$ | $57.01\pm 0.27$ | $21.02\pm 0.22$ | $\mathbf{38.68\pm 0.37}$ | $29.41\pm 0.74$
Herding | $\mathbf{85.02\pm 0.86}$ | $11.83\pm 2.08$ | $\mathbf{58.52\pm 0.23}$ | $26.77\pm 0.19$ | $34.66\pm 0.99$ | $41.91\pm 1.08$
GSS | $19.73\pm 0.01$ | $\mathbf{0.05\pm 0.02}$ | $9.20\pm 0.04$ | $\mathbf{4.31\pm 0.71}$ | - | -
IPM | $66.69\pm 0.46$ | $6.29\pm 0.38$ | $53.91\pm 0.43$ | $14.69\pm 0.46$ | $36.28\pm 0.19$ | $\mathbf{18.99\pm 0.27}$
Table 2: Population strategy results tested with various replay based methods
with traditionally used fixed size buffer, averaged across three runs. We do
not report forgetting in GDumb experiments due to the nature of GDumb only
training on the fully populated, balanced buffer. Results for TinyImageNet are
not reported for GSS due to intractable train times.

##  4 Experiments

We investigate the performance of each population strategy described above by
comparing each scheme under a commonly used suite of replay-based training
methods. We test using both a fixed memory buffer with commonly used buffer
sizes and the newly proposed dynamic buffer scheme described in Section 3.5.
Our primary focus is on the offline class-IL setting. We report offline task-
IL results in the supplementary material.

Datasets. We benchmark each population strategy on three commonly used
continual learning datasets: split-CIFAR10, split-CIFAR100 and split-Tiny-
ImageNet [20, 45, 21]. In split-CIFAR10, the CIFAR10 dataset is split into 5
disjoint tasks where each task contains 2 classes. Split-CIFAR100 splits
CIFAR100 into 10 disjoint tasks of 10 classes each. Split-Tiny-ImageNet splits
Tiny-ImageNet into 10 disjoint tasks of 20 classes each. We maintain the same
order in which classes are split across all tested population strategies and
methods.

Compared Methods. Each population strategy studied herein is tested against
four commonly compared methods in continual learning literature, namely ER
[32], DER [4], GDumb [28], and ER-ACE [5]. Each of the aforementioned methods
was state of the art for its time with ER-ACE being the most recently proposed
state of the art method for bench-marking of current replay-based continual
learning research.

All compared methods use some form of experience replay. For each $t>1$, all
methods train on the union of task data and data stored in memory as
$\mathcal{D}_{t}\cup\mathcal{M}$, except in the case of GDumb where we only
populate $\mathcal{M}$ in a balanced manner by performing one iteration
through each $\mathcal{D}_{t}$ and proceed to train solely on $\mathcal{M}$ at
the conclusion of the final task.

Configuration and Hyperparameters. We test each of the buffer population
strategies studied herein with the above described methods using the open-
source codebase, _Mammoth_ , first introduced in [4]. We use the best
configuration for each compared training method when testing performance of
each population strategy with a ResNet18 [14] backbone for fair comparisons.
Exact hyperparmeter configuration can be found in the supplementary material.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: A comparison of the reservoir, herding, and IPM population
strategies paired with DER with a fixed buffer size of 500. Top row
corresponds to Split-CIFAR10 performance and bottom row is Split-CIFAR100. The
columns correspond as follows: left uses reservoir sampling, center uses
herding, and right uses IPM. We do not report GSS results due to all around
inferior performance.

Metrics. We judge performance with the commonly used metrics of final average
accuracy (_FAA_) and final forgetting (_FF_) given by

$FAA=\frac{1}{\mathcal{T}}\sum_{j=1}^{\mathcal{T}}a_{j}^{\mathcal{T}}\\\ $ (6)
$FF=\frac{1}{\mathcal{T}-1}\sum_{j=1}^{\mathcal{T}-1}f_{j}^{\mathcal{T}}\text{
s.t.
}f_{j}^{\mathcal{T}}=\underset{l\in\\{1,...,\mathcal{T}-1\\}}{\operatorname{max}}a_{l}^{l}-a_{j}^{\mathcal{T}}\\\
$ (7)

where $a_{j}^{\mathcal{T}}$ and $f_{j}^{\mathcal{T}}$ are interpreted as the
accuracy and forgetting of task $j$ and the end of training on $\mathcal{T}$
tasks respectively [8]. When judging performance, we seek maximal _FAA_ and
minimal _FF_.

Fixed Buffer Size |  Percentage of Samples Belonging to Each Task ($t_{1}/t_{2}/t_{3}/t_{4}/t_{5}$)
---|---
200 | 23.5% / 19.0% / 16.0% / 22.0% / 19.5%
500 | 23.5% / 15.0% / 20.5% / 22.5% / 18.5%
5120 | 20.0% / 18.5% / 19.53% / 20.43% / 21.54%
Table 3: Percentage of samples in buffer belonging to each task at the end of
training populated via reservoir sampling.

###  4.1 Results

Fixed Buffer. Results for commonly tested fixed buffer sizes are in Table 2.
We first observe that in nearly all cases, reservoir sampling leads to greater
forgetting when compared to the other population strategies, specifically
compared to herding and IPM. We also observe in several trials, that reservoir
sampling also underperforms in _FAA_ compared to herding and IPM. This
behavior can be attributed to both herding and IPM selecting the best samples
from the learned feature space at the conclusion of each task whereas
reservoir sampling simply selects and replaces samples at random.

Another reason for greater forgetting in reservoir sampling is the unbalanced
fixed buffers incurred by the random sampling and replacement of data in the
buffer. We show the percentage of task specific data of the final fixed buffer
when populated via reservoir sampling in Table 3. We can clearly see that, for
smaller buffer sizes, there is a bias to the storage of earlier task data
compared to more recently encountered tasks. In comparison, both herding and
IPM maintain a balanced fixed buffer at all times leading to equal probability
of sampling any task data for batch training. Naturally, as the buffer size
increases, the unbalanced nature of the buffer populated with reservoir
sampling becomes less severe and we see the reservoir population strategy
become more competitive with herding and IPM in both _FAA_ and _FF_.

(a)

(b)

(c)

Figure 3: Final forgetting performance with various final buffer sizes tested
with Split-CIFAR100. Final buffer sizes with a (D) indicate dynamic final
size. In order from left to right are results from reservoir, herding, and IPM
respectively.

We pay particular interest to the scenarios where IPM yields superior _FF_ yet
inferior _FAA_. To analyze why this happens in certain cases, we plot each
population strategy used in conjunction with DER for both split-CIFAR10 and
split-CIFAR100 in Figure 2 (we omit GSS figures due to all around inferior
performance). We observe the interesting behavior where for each $t>1$, IPM
initially has poor current task performance but then proceeds to make
astonishing recoveries for each subsequent task. This indicates that IPM tends
to prioritize performance on the buffer instead of the current task at hand
which in turn leads to lesser forgetting. An interesting observation to make
is that this behavior holds only for DER and ER-ACE (see supplementary
material for additional figures). Both DER and ER-ACE are training schemes
that directly optimize on logits indicating that IPM is best suited for these
types of training schemes.

(a)

(b)

Figure 4: ER-ACE using the GSS population strategy for a fixed buffer with
buffer size 200. (a) Split-CIFAR10 results. (b) Split-CIFAR100 results.

We take note of the relatively low forgetting of the GSS population strategy
when tested with ER-ACE. To investigate why GSS achieves such low forgetting,
we plot the accuracy of each previously learned task as new tasks are learned
in Figure 4. From this, we can infer that GSS’s low forgetting capabilities
when coupled with ER-ACE is caused by the inferior performance on subsequent
tasks after $t=1$ and thus, has nearly nothing to forget. We note the strong
performance of $t=1$ throughout the model’s life, however. This is due to GSS
first populating the buffer with $t=1$ data and thereafter, hardly ever
finding a sample with an appropriate score to replace other samples in the
buffer as described in Section 3.3. Because of this nature, we should not
accept that GSS is the best forgetting performer with coupled with ER-ACE. We
still observe that reservoir sampling mostly does not compare in forgetting to
other strategies when ignoring GSS results.

We observe the all around competitiveness of each population strategy when
tested with GDumb (note, we do not report _FF_ for GDumb as there is no
forgetting to take place since GDumb trains solely on the buffer). Because
GDumb only uses each observed task to populate the memory buffer, it makes
sense that herding and IPM perform roughly the same with reservoir sampling
since herding and IPM depend on the well learned features for population.
Similarly, GSS depends on the gradients of each sample, but because GDumb
takes no gradient steps until the conclusion of the final observed task, GSS
has no proper way to score samples for replacement or not.

Lastly, we make the note that while no single method consistently outperforms
any other, we demonstrate that in many situations, reservoir sampling yields
inferior final forgetting performance. This suggests that when using replay-
based methods in continual learning solutions, reservoir sampling should not
be blindly used as the buffer population strategy of choice, and one should
instead pay careful attention to selection of buffer population algorithm for
the best performance.

Dynamic Buffer. We next perform experiments using dynamic buffers using the
two criteria as described in Section 3.5 and provide tabulated results for
class-IL and task-IL scenarios in the supplementary material. We omit GSS
results using a dynamic buffer due to poor performance with paired with any of
the fixed buffer schemes.

Overall, we observe much of the same trends as seen with fixed buffers.
However, we notice that dynamic buffers seem to benefit most when paired with
reservoir sampling and IPM in Figure 3 (_FAA_ and _FF_ for all datasets are
reported in supplementary material). Because dynamic buffers find the optimal
number of samples for storage, we expect the change in _FF_ to be lower when
we approach that number, which we observe in Figure 3. In general we observe
the Kaiser criterion performing better than intracluster variance. This can be
attributed to the ability to adapt to classs complexity for the Kaiser
criterion, particularly as the number of classes increases in a dataset.

We note the curious performance of DER when coupled with dynamic replay and
give a brief conjecture for why this may be in the supplemental material.

##  5 Conclusions

In this work, we compare the commonly used approach of reservoir sampling for
memory buffer population in replay methods to other greedy sampling methods to
answer the question of _which samples should be stored_ in memory. We show
that reservoir sampling tends to lead to higher forgetting when compared to
methods that use strategic population strategies to select the best data
points for storage. We then address the question of _how many samples should
be stored_ by the formulation of a dynamic buffer populated according to two
criteria based on a dataset’s eigenvectors and eigenvalues. We show that
dynamic buffers lead to more competitive performance for all population
strategies when compared to the arbitrary fixed buffer sizes commonly used in
replay methods.

## References

  * [1] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia.  Online continual learning with maximal interfered retrieval.  In Advances in Neural Information Processing Systems 32, pages 11849–11860. Curran Associates, Inc., 2019.
  * [2] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  Advances in neural information processing systems, 32, 2019.
  * [3] Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara.  On the effectiveness of lipschitz-driven rehearsal in continual learning.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [4] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  Advances in neural information processing systems, 33:15920–15930, 2020.
  * [5] Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky.  New insights on reducing abrupt representation change in online continual learning.  In International Conference on Learning Representations, 2022.
  * [6] Lawrence Cayton.  Algorithms for manifold learning.  Univ. of California at San Diego Tech. Rep, 12(1-17):1, 2005.
  * [7] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin.  Co2l: Contrastive continual learning.  In Proceedings of the IEEE/CVF International conference on computer vision, pages 9516–9525, 2021.
  * [8] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In Proceedings of the European conference on computer vision (ECCV), pages 532–547, 2018.
  * [9] Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Paz.  Using hindsight to anchor past knowledge in continual learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6993–7001, 2021.
  * [10] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-GEM.  In International Conference on Learning Representations, 2019.
  * [11] Ehsan Elhamifar, Guillermo Sapiro, and René Vidal.  See all by looking at a few: Sparse modeling for finding representative objects.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1600–1607, 2012.
  * [12] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan.  Testing the manifold hypothesis.  Journal of the American Mathematical Society, 29(4):983–1049, 2016\.
  * [13] Tyler L Hayes, Giri P Krishnan, Maxim Bazhenov, Hava T Siegelmann, Terrence J Sejnowski, and Christopher Kanan.  Replay in deep learning: Current approaches and missing biological elements.  Neural computation, 33(11):2908–2950, 2021.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Mohsen Joneidi, Saeed Vahidian, Ashkan Esmaeili, Weijia Wang, Nazanin Rahnavard, Bill Lin, and Mubarak Shah.  Select to better learn: Fast and accurate deep learning using data selection from nonlinear manifolds.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [16] Mohsen Joneidi, Alireza Zaeemzadeh, Nazanin Rahnavard, and Mubarak Shah.  Iterative projection and matching: Finding structure-preserving representatives and its application to computer vision.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5409–5418, 2019.
  * [17] Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim.  Less-forgetful learning for domain expansion in deep neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [18] Henry F Kaiser.  The application of electronic computers to factor analysis.  Educational and psychological measurement, 20(1):141–151, 1960\.
  * [19] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  * [20] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [21] Ya Le and Xuan S. Yang.  Tiny imagenet visual recognition challenge.  2015\.
  * [22] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2017.
  * [23] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  Advances in Neural Information Processing Systems, 30, 2017.
  * [24] James L McClelland, Bruce L McNaughton, and Randall C O’Reilly.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.  Psychological review, 102(3):419, 1995.
  * [25] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [26] Nikhil Mehta, Kevin Liang, Vinay Kumar Verma, and Lawrence Carin.  Continual learning using a bayesian nonparametric dictionary of weight factors.  In International Conference on Artificial Intelligence and Statistics, pages 100–108. PMLR, 2021.
  * [27] Quang Pham, Chenghao Liu, and Steven HOI.  Dualnet: Continual learning, fast and slow.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
  * [28] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 524–540. Springer, 2020.
  * [29] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [30] Mark B Ring.  Child: A first step towards continual learning.  In Learning to learn, pages 261–292. Springer, 1998.
  * [31] Anthony V. Robins.  Catastrophic forgetting, rehearsal and pseudorehearsal.  Connect. Sci., 7:123–146, 1995.
  * [32] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [33] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [34] Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan.  Finding a ”kneedle” in a haystack: Detecting knee points in system behavior.  In 2011 31st International Conference on Distributed Computing Systems Workshops, pages 166–171, 2011.
  * [35] Pravendra Singh, Vinay Kumar Verma, Pratik Mazumder, Lawrence Carin, and Piyush Rai.  Calibrating cnns for lifelong learning.  Advances in Neural Information Processing Systems, 33:15579–15590, 2020.
  * [36] Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan.  Exploring example influence in continual learning.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [37] Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias.  Information-theoretic online memory selection for continual learning.  In International Conference on Learning Representations, 2022.
  * [38] Sebastian Thrun and Tom M Mitchell.  Lifelong robot learning.  Robotics and autonomous systems, 15(1-2):25–46, 1995.
  * [39] Gido M Van de Ven and Andreas S Tolias.  Three scenarios for continual learning.  arXiv preprint arXiv:1904.07734, 2019.
  * [40] Jeffrey S Vitter.  Random sampling with a reservoir.  ACM Transactions on Mathematical Software (TOMS), 11(1):37–57, 1985\.
  * [41] Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento.  Continual learning with hypernetworks.  In International Conference on Learning Representations, 2020.
  * [42] Max Welling.  Herding dynamical weights to learn.  In Proceedings of the 26th Annual International Conference on Machine Learning, pages 1121–1128, 2009.
  * [43] Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang.  Online coreset selection for rehearsal-based continual learning.  In International Conference on Learning Representations, 2022.
  * [44] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In International Conference on Learning Representations, 2018.
  * [45] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017.
  * [46] A. Çivril.  Column subset selection problem is ug-hard.  Journal of Computer and System Sciences, 80(4):849–859, 2014.


# Learning ASR-Robust Contextualized Embeddings
for Spoken Language Understanding

###### Abstract

Employing pre-trained language models (LM) to extract contextualized word
representations has achieved state-of-the-art performance on various NLP
tasks. However, applying this technique to noisy transcripts generated by
automatic speech recognizer (ASR) is concerned. Therefore, this paper focuses
on making contextualized representations more ASR-robust. We propose a novel
confusion-aware fine-tuning method to mitigate the impact of ASR errors on
pre-trained LMs. Specifically, we fine-tune LMs to produce similar
representations for acoustically confusable words that are obtained from word
confusion networks (WCNs) produced by ASR. Experiments on multiple benchmark
datasets show that the proposed method significantly improves the performance
of spoken language understanding when performing on ASR transcripts111Code
available at: https://github.com/MiuLab/SpokenVec.

Index Terms—  spoken language understanding, contextualized embedding, ASR
robustness

##  1 Introduction

A spoken language understanding (SLU) module serves an important role in a
spoken dialogue system, which aims at extracting semantic concepts from spoken
utterances and provides structured information for accessing the backend
database. Typical tasks of SLU include intent detection and slot filling.
These two tasks focus on predicting speaker’s intent and extracting semantic
concepts as constraints for the natural language. A movie-related example
utterance “find comedies by James Cameron” shown in Figure 1 has two slot-
value labels and a specific intent for the whole utterance.

Applying deep learning techniques has been shown to boost the performance of
SLU [1, 2, 3, 4]. Most prior work focused on applying understanding models on
_manual_ transcripts, ignoring the errors introduced by automatic speech
recognizers (ASR). Hence, several methods were proposed to address this
problem. Simonnet et al. [5] simulated ASR errors and trained SLU models for
better handling of the errors. The prior work leveraged information from
lattices or word confusion networks [6, 7, 8, 9, 10, 11], and Zhu et al. [12]
applied domain adversarial training for ASR-error adaptation, demonstrating
the importance of incorporating ASR errors for better SLU performance.

Word | _find comedies by james cameron_
---|---
Slot | genre: _comedy_ , director: _James Cameron_
Intent | find_movie
Fig. 1: An annotated utterance example.

Deep contextualized word representations recently have achieved great success
among language understanding tasks [13, 14, 15]. Nevertheless, they may be
less robust to noisy texts, such as the recognized results. In this paper, we
investigate the impact of ASR errors on contextualized embeddings and further
propose a novel confusion-aware fine-tuning method to alleviate this problem.
To our best knowledge, there is no prior work that learned contextualized word
embeddings and considered the errors produced from spoken language for better
robustness. Our contributions are 3-fold:

  * •

This is the first attempt to learn contextualized word embeddings specifically
for spoken language.

  * •

The proposed approach achieves better performance on the benchmark spoken
language understanding tasks.

  * •

The proposed method shows better robustness to ASR errors.

##  2 Learning ASR-Robust Contextualized Embeddings

Preparing datasets for SLU takes a lot of effort. SLU datasets are typically
smaller than NLU datasets since it’s much more labor intensive to collect
labeled spoken utterances. Hence, the goal of this paper is to learn ASR-
robust contextualized embeddings such that the downstream SLU models trained
on manual transcripts can perform well on automatic transcripts, using only
unlabeled spoken utterances to adapt.

To enable the embeddings to adapt ASR errors for improving SLU, our proposed
method consists of three stages: 1) language model pre-training on general
domain corpora $\mathcal{D}_{\text{LM}}$, 2) confusion-aware language model
fine-tuning on the text from the target SLU task, where the text can be either
manual transcripts $\mathcal{D}_{\text{trs}}$ or automatic transcripts
$\mathcal{D}_{\text{asr}}$, and 3) training a language understanding model
with the fine-tuned LM on labeled SLU data $\mathcal{D}_{\text{SLU}}$, which
consists of the manual transcripts $\mathcal{D}_{\text{trs}}$ with their
corresponding labels.

In this paper, we focus on the task of intent detection, which is an
utterance-level multi-class classification problem. More formally, given an
utterance $x=\\{w_{1}^{x},w_{2}^{x},...,w_{|x|}^{x}\\}$, the goal is to
predict its corresponding intent $I_{x}$. The input utterance $x$ can be
either manually transcribed texts, denoted as $x_{\text{trs}}$, or ASR-
recognized results, denoted as $x_{\text{asr}}$. The proposed approach is
detailed below.

###  2.1 Embeddings from Language Model (ELMo)

Peters et al. [13] proposed ELMo to extract context-dependent word embeddings
from a pre-trained LM, and the contextualized embeddings were proved to be
able to improve the performance of downstream NLP tasks. In this paper, we
adopt the same model architecture as in the original work, which consists of a
CNN character encoder and two bidirectional LSTMs [16]. Same strategy of
combining hidden states from different layers is applied [13], which computes
the representation $e_{t}$ for a word $w_{t}^{x}$ in the sentence $x$ as:

$e_{t}=\gamma\sum_{i=0}^{2}{\alpha_{i}\cdot h_{t,i}^{x}},$

where $h_{t,i}^{x}=[\overleftarrow{h_{t,i}^{x}};\overrightarrow{h_{t,i}^{x}}]$
is the concatenation of the $i$-th layer output from both directions at the
time $t$, $\alpha_{i}$ is the weight for the $i$-th layer, and $\gamma$ is a
scaling factor. $\alpha_{i}$ and $\gamma$ are scalar parameters learned along
with downstream tasks. The ELMo model is pre-trained on the general-domain
textual data $\mathcal{D}_{\text{LM}}$.

###  2.2 Language Model Fine-Tuning

One advantage of pre-training a language model is that it can leverage large
amounts of unlabeled text corpora. Usually the data is general such as
Wikipedia. However, the data distribution of the target task may be different
from that used in pre-training, posing a domain mismatch problem. Howard et
al. [17] proposed to fine-tune the pre-trained LM with sentences from the
downstream dataset and showed that it boosts the performance of the downstream
task. Chronopoulou et al. [18] also demonstrated the effectiveness of the
fine-tuning method.

In order to adapt the pre-trained LM to the target data, the fine-tuning
technique is applied. Given an utterance $x=\\{w_{1},w_{2},...,w_{|x|}\\}$,
the bidirectional language modeling loss can be written as:

$\mathcal{L}_{\text{LM}}=\frac{1}{|x|}\sum_{t=1}^{|x|}-\log p(w_{t}\mid
w_{<t})-\log p(w_{t}\mid w_{>t}),$

where $p(w_{t}\mid w_{<t})$ and $p(w_{t}\mid w_{>t})$ are probabilities of
$w_{t}$ predicted by the forward LM and the backward LM respectively.

Language model fine-tuning can be performed on both manual transcription and
recognized results

(a) Supervised confusion extraction
with aligned utterances

(b) Unsupervised confusion extraction using
a WCN generated by ASR

Fig. 2: Illustration of different extraction approaches. $*$ denotes a blank
symbol for alignment purpose.

###  2.3 Confusion-Aware Fine-Tuning

Taking ASR transcripts as inputs may introduce an issue that words in an
utterance may be misrecognized. For instance, fair and fare are acoustically
similar, so an ASR system may fail to distinguish between them, resulting in a
substitution error. Such recognition errors might be recovered by human,
because human are aware of the acoustic confusability of words. However, the
errors may significantly degrade the testing performance when the models are
trained on manual transcripts. In order to enhance the ASR robustness in
contextualized word embeddings, this section integrates the acoustic confusion
into our LM.

We propose a confusion-aware fine-tuning method to mitigate this problem from
pre-trained LMs, which aims at making the LM consider multiple acoustically
confusable words. Let $c=\\{w_{t_{1}}^{x_{1}},w_{t_{2}}^{x_{2}}\\}$ denote an
acoustic confusion, i.e., two words with similar pronunciation in two
different utterances $x_{1}$ and $x_{2}$, and
$C=\\{c_{1},c_{2},\cdots,c_{|C|}\\}$ denote the set of all acoustic confusions
in $x_{1}$ and $x_{2}$. We introduce a new loss term called confusion loss:

$\mathcal{L}_{\text{conf}}=\frac{1}{|C|}\sum_{c\in
C}\sum_{i=0}^{1}{1-\frac{h_{t_{1},i}^{x_{1}}\cdot
h_{t_{2},i}^{x_{2}}}{\left\lVert h_{t_{1},i}^{x_{1}}\right\rVert\left\lVert
h_{t_{2},i}^{x_{2}}\right\rVert}},$

which is the cosine distance between the LM hidden states corresponding to
words. Note that we empirically find that including only the first two layers
in loss computation works the best. Two approaches are designed for extracting
acoustic confusions.

####  2.3.1 Supervised Confusion Extraction

Assuming that both ASR transcripts $x_{\text{asr}}$ and manual transcripts
$x_{\text{trs}}$ of a spoken utterance are accessible, we align
$x_{\text{asr}}$ with $x_{\text{trs}}$ with respect to minimum edit-distance
criterion to extract acoustic confusions as shown in Figure 2(a). By
minimizing $\mathcal{L}_{\text{conf}}$, we directly force the LM to produce
representations for an erroneous word similar to its correct counterpart. This
method is called _supervised confusion extraction_ considering that it
requires the availability of manual transcripts $\mathcal{D}_{\text{trs}}$.

####  2.3.2 Unsupervised Confusion Extraction

Considering the scenario where only audio recording of a spoken utterance is
available, we can apply an ASR on the recording and construct a word confusion
network (WCN). Then a list of n-best hypotheses is generated and aligned using
WCN, and the acoustic confusions can be obtained as depicted in Figure 2(b).

An important advantage of this approach is that it does not require any
labeled utterances or manual transcripts; therefore, we can leverage unlabeled
audio recordings to fine-tune LMs in an unsupervised fashion.

###  2.4 Joint Objective Function for Fine-tuning

In the fine-tuning stage, we minimize the joint objective function including
the LM loss and confusion-aware loss:

$\mathcal{L}_{\text{FT}}=\mathcal{L}_{\text{LM}}+\beta\mathcal{L}_{\text{conf}},$

where $\beta$ is a hyperparameter to balance the contribution of two loss
functions. The procedure enables our model to incorporate not only the target
domain information but the acoustic information for better robustness to ASR
errors.

Fig. 3: Illustration of our SLU model architecture.

###  2.5 Spoken Language Understanding (SLU)

To further build an SLU model that leverages ASR-robust contextualized
embeddings, we employ a biLSTM as our SLU model, where the biLSTM takes
contextualized word embeddings $\\{e_{t}\\}_{t=1}^{|x|}$ as the input, and the
outputs of the last biLSTM layer are max-pooled, linearly transformed and
softmaxed to obtain the predicted probabilities for each class. The overall
architecture is illustrated in Figure 3. During training, we use cross entropy
as the loss function. Weights of the ELMo model are fixed during this stage
except for $\alpha_{i}$ and $\gamma$. The SLU model is trained on
$\mathcal{D}_{\text{SLU}}$ and evaluated on automatic transcripts. The trained
SLU model is expected to achieve better performance on automatic transcripts
due to the integration of ASR-robust contextualized word embeddings.

##  3 Experiments

| train | test | intents | WER
---|---|---|---|---
ATIS | 4478 | 893 | 21 | 15.55%
SmartLights | close | 1765 | 6 | 45.61%
far | 1765 | 6 | 71.02%
Snips | 13084 | 700 | 7 | 45.56%
Table 1: Dataset statistics. Model | ATIS | SmartLights | Snips
---|---|---|---
close | far
Manual | ASR | Manual | ASR | Manual | ASR | Manual | ASR
(a) | Oracle | 96.47 | 94.75 | 90.60 | 74.05 | 82.94 | 56.96 | 96.38 | 91.79
(b) | Context-independent | 93.60 | 90.35 | 95.67 | 65.71 | 95.67 | 44.55 | 96.29 | 72.70
(c) | Pre-trained ELMo | 96.65 | 93.27 | 97.01 | 64.53 | 97.01 | 44.46 | 96.29 | 77.86
(d) | (c) + fine-tune, $\mathcal{L}_{\text{LM}}$ only | 96.91 | 94.27 | 95.91 | 66.33 | 95.66 | 46.22 | 96.38 | 87.74
(e) | (c) + fine-tune, $\mathcal{L}_{\text{FT}}$ (sup-conf) | 96.61 | 95.65 | 95.92 | 67.99 | 95.53 | 46.57 | 97.01 | 88.52
(f) | (c) + fine-tune, $\mathcal{L}_{\text{FT}}$ (unsup-conf) | 97.02 | 95.39 | 95.98 | 67.98 | 95.79 | 47.38 | 97.04 | 89.55
Table 2: Results of intent detection tasks (%). Manual and ASR indicate
evaluating on $x_{\text{trs}}$ and $x_{\text{asr}}$ respectively. close and
far represent different microphone settings. sup-conf stands for supervised
confusion extraction, and unsup-conf stands for unsupervised confusion
extraction. The best numbers for each dataset are marked in bold.

###  3.1 Setup

Three SLU datasets used in the experiments as listed below:

  * •

ATIS (Airline Travel Information Systems) [19, 20, 21] is a benchmark dataset
widely used in language understanding research. The dataset contains audio
recordings of people making flight reservations with corresponding manual
transcripts.

  * •

Snips SmartLights [22] contains spoken commands for smart light assistant. The
dataset comes with two kinds of microphone settings, close field and far
field.

  * •

Snips [23] is a dataset for benchmarking NLU systems. This dataset is larger
than ATIS and Snips SmartLights. We use a commercial text-to-speech system
222https://cloud.google.com/text-to-speech/ to synthesize audio from text
data.

The dataset statistics are shown in Table 1.

For ATIS, we train an ASR system on WSJ [24] using the s5 recipe from Kaldi
[25]. For the other datasets, we use an ASR model released in Kaldi to provide
better recognition results 333https://kaldi-asr.org/models/m1. We use the ASR
system to recognize audio recordings and extract acoustic confusions for fine-
tuning.

###  3.2 Model and Training Details

The pre-trained weights of ELMo from [13] are adopted. The size of
contextualized representations is 1024. Our SLU model has two layers with
300-dimensional hidden states.

In the fine-tuning stage, acoustic confusions that contain stop words are
excluded, and $\beta$ is set to 0.1. We set batch size to 64 and use Adam as
the optimizer [26] with learning rate 0.001 for all stages. We fine-tune ELMo
for 3 epochs and train the SLU model for 50 epochs. The Snips SmartLights
dataset is very small, so we use 10-fold cross validation to evaluate the
models as suggested in [22].

###  3.3 Baselines

We compare our method with two baselines and an oracle system as listed below.

  * •

Context-independent: replaces the contextualized representations with
traditional context-independent word embeddings. The embedding matrix is
initialized randomly.

  * •

Pre-trained ELMo: uses pre-trained ELMo weights without fine-tuning.

  * •

Oracle: trains SLU on $x_{\text{asr}}$ with pre-trained ELMo embeddings.

Note that our models do not utilize the information the oracle system uses,
$x_{\text{asr}}$ paired with labels.

###  3.4 Results

Table 2 shows the experimental results, where the reported numbers are
accuracies averaged over 5 runs. All models are trained on $x_{\text{trs}}$
except for the oracle system, and they all perform great when evaluated on
$x_{\text{trs}}$. Rows (b) and (c) show that ASR errors degrade SLU
performance considerably for both context-independent and context-dependent
embeddings. When testing on $x_{\text{asr}}$, the performance drops 3.38% on
ATIS dataset using pre-trained ELMo embeddings. The results on Snips datasets
show that the performance drops more in higher WER scenarios.

Our proposed method, confusion-aware language model fine-tuning, outperforms
baselines by a large margin on all datasets (rows (d)-(f)), while it maintains
identical performance on $x_{\text{trs}}$. Results in row (d) can be viewed as
an ablation to rows (e) and (f), where we exclude $\mathcal{L}_{\text{conf}}$
from the joint objective. Rows (e) and (f) show that while
$\mathcal{L}_{\text{LM}}$ provides significant improvement alone, adding
$\mathcal{L}_{\text{conf}}$ further boosts performance notably. The results
demonstrate that the proposed method can provide ASR robustness to the SLU
models.

###  3.5 Discussion

The research on SLU has been investigated for several years, and there are two
main branches. The first branch treats SLU as a natural language understanding
(NLU) task, where the prior work trained the models directly on natural
language data without misrecognition [1, 2, 3, 4]. Another branch focuses on
building understanding models with consideration of ASR results. While some
prior work relied on ASR lattices or WCNs to provide richer information to SLU
models [6, 7, 8], the work presented here focuses on using the 1-best results
from ASR. Simonnet et al. proposed an ASR error simulation scheme to train
robust SLU models [5], whereas we dig realistic ASR errors from recognition
results. Shivakumar et al. used WCNs to extract acoustic confusions for fine-
tuning context-independent word embeddings [9, 10]. Our work combines the idea
of using acoustic confusions with language model fine-tuning [17] to obtain
ASR-robust contextualized embeddings.

##  4 Conclusion

This paper proposes a novel confusion-aware language model fine-tuning method
for learning ASR-robust contextualized embeddings. We introduce supervised and
unsupervised methods for extracting acoustic confusions and integrate a
confusion loss that forces LMs to consider acoustically confusable words. The
experiments on SLU demonstrate that our proposed method learns contextualized
embeddings that are robust to ASR errors.

## References

  * [1] K. Yao, B. Peng, Y. Zhang, D. Yu, G. Zweig, and Y. Shi,  “Spoken language understanding using long short-term memory neural networks,”  in SLT, 2014.
  * [2] D. Guo, G. Tur, W.-t. Yih, and G. Zweig,  “Joint semantic utterance classification and slot filling with recursive neural networks,”  in SLT, 2014.
  * [3] G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. Hakkani-Tur, X. He, L. Heck, G. Tur, D. Yu, et al.,  “Using recurrent neural networks for slot filling in spoken language understanding,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 3, pp. 530–539, 2014.
  * [4] C.-W. Goo, G. Gao, Y.-K. Hsu, C.-L. Huo, T.-C. Chen, K.-W. Hsu, and Y.-N. Chen,  “Slot-gated modeling for joint slot filling and intent prediction,”  in NAACL-HLT, 2018.
  * [5] E. Simonnet, S. Ghannay, N. Camelin, and Y. Estève,  “Simulating ASR errors for training SLU systems,”  in LREC, 2018.
  * [6] D. Hakkani-Tür, F. Béchet, G. Riccardi, and G. Tur,  “Beyond asr 1-best: Using word confusion networks in spoken language understanding,”  Computer Speech & Language, vol. 20, no. 4, pp. 495–514, 2006\.
  * [7] G. Tür, A. Deoras, and D. Z. Hakkani-Tür,  “Semantic parsing using word confusion networks with conditional random fields,”  in INTERSPEECH, 2013.
  * [8] F. Ladhak, A. Gandhe, M. Dreyer, L. Mathias, A. Rastrow, and B. Hoffmeister,  “Latticernn: Recurrent neural networks over lattices,”  INTERSPEECH, 2016.
  * [9] P. G. Shivakumar and P. Georgiou,  “Confusion2vec: Towards enriching vector space word representations with representational ambiguities,”  arXiv preprint arXiv:1811.03199, 2018.
  * [10] P. G. Shivakumar, M. Yang, and P. Georgiou,  “Spoken language intent detection using confusion2vec,”  arXiv preprint arXiv:1904.03576, 2019.
  * [11] C.-W. Huang and Y.-N. Chen,  “Adapting pretrained transformer to lattices for spoken language understanding,”  in ASRU, 2019.
  * [12] S. Zhu, O. Lan, and K. Yu,  “Robust spoken language understanding with unsupervised asr-error adaptation,”  in ICASSP, 2018.
  * [13] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer,  “Deep contextualized word representations,”  in NAACL-HLT, 2018.
  * [14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [15] A. Siddhant, A. Goyal, and A. Metallinou,  “Unsupervised transfer learning for spoken language understanding in intelligent agents,”  arXiv preprint arXiv:1811.05370, 2018.
  * [16] S. Hochreiter and J. Schmidhuber,  “Long short-term memory,”  Neural computation, 1997.
  * [17] J. Howard and S. Ruder,  “Universal language model fine-tuning for text classification,”  in ACL, 2018.
  * [18] A. Chronopoulou, C. Baziotis, and A. Potamianos,  “An embarrassingly simple approach for transfer learning from pretrained language models,”  arXiv preprint arXiv:1902.10547, 2019.
  * [19] C. T. Hemphill, J. J. Godfrey, and G. R. Doddington,  “The ATIS spoken language systems pilot corpus,”  in Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990, 1990.
  * [20] D. A. Dahl, M. Bates, M. Brown, W. Fisher, K. Hunicke-Smith, D. Pallett, C. Pao, A. Rudnicky, and E. Shriberg,  “Expanding the scope of the atis task: The atis-3 corpus,”  in HLT, 1994.
  * [21] G. Tur, D. Hakkani-Tür, and L. Heck,  “What is left to be understood in ATIS?,”  in SLT, 2010.
  * [22] A. Saade, A. Coucke, A. Caulier, J. Dureau, A. Ball, T. Bluche, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone, et al.,  “Spoken language understanding on the edge,”  arXiv preprint arXiv:1810.12735, 2018.
  * [23] A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone, T. Lavril, et al.,  “Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces,”  arXiv preprint arXiv:1805.10190, 2018.
  * [24] D. B. Paul and J. M. Baker,  “The design for the wall street journal-based csr corpus,”  in Proc. of the Workshop on Speech and Natural Language, 1992, HLT ’91.
  * [25] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al.,  “The kaldi speech recognition toolkit,”  Tech. Rep., IEEE Signal Processing Society, 2011.
  * [26] D. P. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.