# Continual Learning in Task-Oriented Dialogue Systems

Andrea Madotto∗, Zhaojiang Lin∗, Zhenpeng Zhou†, Seungwhan Moon†,
Paul Crook†, Bing Liu†, Zhou Yu†, Eunjoon Cho†, Zhiguang Wang†
∗The Hong Kong University of Science and Technology
†Facebook
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Work done during internship at Facebook

###### Abstract

Continual learning in task-oriented dialogue systems can allow us to add new
domains and functionalities through time without incurring the high cost of a
whole system retraining. In this paper, we propose a continual learning
benchmark for task-oriented dialogue systems with 37 domains to be learned
continuously in four settings, such as intent recognition, state tracking,
natural language generation, and end-to-end. Moreover, we implement and
compare multiple existing continual learning baselines, and we propose a
simple yet effective architectural method based on residual adapters. Our
experiments demonstrate that the proposed architectural method and a simple
replay-based strategy perform comparably well but they both achieve inferior
performance to the multi-task learning baseline, in where all the data are
shown at once, showing that continual learning in task-oriented dialogue
systems is a challenging task. Furthermore, we reveal several trade-off
between different continual learning methods in term of parameter usage and
memory size, which are important in the design of a task-oriented dialogue
system. The proposed benchmark is released together with several baselines to
promote more research in this direction.

##  1 Introduction

Task-oriented dialogue systems (ToDs) are the core technology of the current
state-of-the-art smart assistant (e.g. Alexa, Siri, Portal etc.). These
systems are either modularized, Natural Language Understanding (NLU), Dialogue
State Tracking (DST), Dialogue Policy (DP) and Natural Language Generation
(NLG), or end-to-end, where a single model implicitly learn how to issue APIs
(i.e., NLU+DST) and system responses (i.e., NLG).

These systems are often updated with new features based on the user needs,
e.g., adding new slots and intents, or even completely new domains. However,
existing dialogue models are trained with the assumption of having a fixed
dataset at the beginning of the training, and they are not designed to add new
domains and functionalities through time, without incurring the high cost of a
whole system retraining. Therefore, the ability to acquire new knowledge
continuously, a.k.a. Continual Learning (CL) Thrun and Pratt (2012), is
crucial in the design of a dialogue system. Figure 1 shows an high-level
intuition of CL in ToDs.

Figure 1: In Continual Learning, the model is trained one dataset at the time.
In this instance, the model is first trained on data from the hotel domain
$D_{Hotel}$ then on the Taxi $D_{Taxi}$ and so on. The parameter of the model
is updated sequentially based on the loss function $L$.

In this setting the main challenge is catastrophic forgetting McCloskey and
Cohen (1989). This phenomena happens since there is a distributional shift
between the tasks in the curriculum which leads to catastrophic forgetting the
previously acquired knowledge. To overcome this challenge three kind of
methods are usually deployed such as loss regularization, for avoiding to
interfere with the previously learned task, rehearsal, which uses an episodic
memory to recall previously learned tasks, and architectural, which adds task-
specific parameters for each learned task. However, architectural methods are
usually not considered as a baseline, especially in sequence-to-sequence
generation tasks Sun et al. (2019), because they usually require a further
step during testing for selecting which parameter to use for the given task.

To the best our knowledge, Continual Learning (CL) in task-oriented dialogue
systems Lee (2017) is mostly unexplored or it has been studied in specific
settings (e.g. NLG Mi et al. (2020)) using only few tasks learned
continuously. Given the importance of the task in the dialogue setting, we
believe that a more comprehensive investigation is required, especially by
comparing multiple settings and baselines. Therefore in this paper:

  1. 1.

we propose a benchmark for Continual Learning in ToDs, with 37 tasks to be
learned continuously on 4 settings,

  2. 2.

we propose a simple yet effective architectural CL method based on residual
adapters Houlsby et al. (2019) that can continuously learn tasks without the
need of a task classifier a testing time, and

  3. 3.

we analyse the trade-off between performance, number-of-parameters, and
episodic memories sizes of the three main categories of CL methods (i.e.,
Regularization, Rehearsal, Architectural).

In Section 2 we introduce the basic concepts and notation used throughout the
paper, for both task-oriented dialogue modelling and continual learning, in
Section 3 we introduce the proposed architectural CL method, in Section 4 we
describe datasets, baselines, evaluation metrics and experimental settings,
and Section 5 we describe the main findings of the paper.

##  2 Background

Figure 2: Example of input-out pairs, for the four settings, INTENT, DST, NLG
and end-to-end (E2E).

###  2.1 Task-Oriented Dialogue Modelling

In this paper, we model task-oriented dialogue systems as a seq2seq generation
task Lei et al. (2018); Byrne et al. (2020) that generates both api-calls and
responses. As shown in Figure 2, the model uses the dialogue history to
generate an api-call, which is the concatenation of the user intent plus the
current dialogue state and then uses its return, which can be empty or the
system speech-act, to generate the system response. This modelling choice is
guided by the existing annotated dialogue datasets, which provide the intent
and the dialogue state of the user at every turn, and the speech-act of the
system; and it allows us to defined four distinct settings for studying
continual learning such as Intent Recognition (INTENT), Dialogue State
Tracking (DST), Natural Language Generation (NLG) and end-to-end (E2E). In the
coming paragraphs, we will formally describe the four setting as different
input-out pairs for a seq2seq model.

#### Data-Formatting

Let us define the dialogue history $H$ as a single sequence of tokens from the
concatenation of the alternating utterances from the user and the system turns
respectively. Without loss of generality, we assume that $H$ has all the
dialogue history without the last system utterance denoted as $S$. For
distinguishing between speakers, we add two special tokens at the beginning of
every utterance i.e., USER: for the user utterance and SYSTEM: for the system
utterance. Then, we define an api-call, denoted by $S_{API}$, as the
concatenation of the api-name, i.e., the user-intent, and its arguments, i.e.,
slot-value pairs from the DST. The following syntax is used:

$X_{API}=\underbrace{\mathbf{I}}_{\text{Intent}}\
(\underbrace{s_{1}=v_{1},\ldots,s_{k}=v_{p}}_{\text{Slot-value pairs}})$ (1)

where $\mathbf{I}$ is an intent or the api-name, $s_{i}$ the slot-name and
$v_{i}$ one of the possible values for the slot $s_{i}$. The return of the
api-call is either an empty string, thus the model uses the dialogue history
to generate a response, or a speech-act, denoted as $S_{OUT}$, in the same
format as the api-call in Equation 1. Similar to the dialogue history, we
define two special tokens API: and OUT: for triggering the model to generate
the api-call and for distinguishing the return of the api from the dialogue
history respectively. Based on this pre-processing, we define the four
settings used in this paper.

Without loss of generality, we define the three modularized setting by their
input-out pairs:

$\begin{array}[]{cc}H\rightarrow\mathbf{I}&\mathrm{(INTENT)}\\\
H\rightarrow\mathbf{I}(s_{1}=v_{1},\ldots,s_{k}=v_{p})&\mathrm{(DST)}\\\
\underbrace{\mathbf{I}(s_{1}=v_{1},\ldots,s_{k}=v_{p})}_{S_{OUT}}\rightarrow
S&\mathrm{(NLG)}\end{array}$

where instead the end-to-end (E2E) setting as the pairs:

$\begin{array}[]{c}H\rightarrow\underbrace{\mathbf{I}(s_{1}=v_{1},\ldots,s_{k}=v_{p})}_{S_{API}}\\\
H+\underbrace{\mathbf{I}(s_{1}=v_{1},\ldots,s_{k}=v_{p})}_{S_{OUT}}\rightarrow
S\end{array}$

Often, $S_{OUT}$ is empty and thus the model is though to map the dialogue
history to the response directly ($H\rightarrow S$). An example of input-out
pairs is shown in Figure 2.

Finally, we define a dialogue dataset as $D_{K}=\\{(X_{i},Y_{i})\\}_{i}^{n}$,
where $(X_{i},Y_{i})$ is a general input-out pair from one of the four
settings in consideration, and $K$ the dialogue domain under consideration
(e.g., Hotel).

#### Model

In this paper, we employ a decoder-only Language Models (e.g. GPT-2), which is
inline with the current state-of-the-art task-oriented dialogue models Peng et
al. (2020); Hosseini-Asl et al. (2020). Then, given the concatenation of the
input $X=\\{x_{0},\dots,x_{n}\\}$ and output $Y=\\{x_{n+1},\dots,x_{m}\\}$
sequences, we compute the conditional language model distribution using the
chain rule of probability Bengio et al. (2003) as:

$p_{\theta}(X|Y)=\prod_{i=0}^{n+m}p_{\theta}(x_{i}|x_{0},\cdots,x_{i-1}).$ (2)

where $\theta$ are the model parameters. The model’s parameters are trained to
minimize the negative log-likelihood over a dataset $D$ of input-out pairs,
which in our case is the data of the four settings. Formally, we define the
loss $\mathcal{L}_{\theta}$ as:

$\mathcal{L}_{\theta}(D)=-\sum_{j}^{|D|}\sum_{i=0}^{n+m}\log
p_{\theta}(x_{i}^{(j)}|x_{0}^{(j)},\cdots,x_{i-1}^{(j)}),$ (3)

where $n+m$ is a maximum sequence length in $D$. At inference time, given the
input sequence $X$, the model parametrized by $\theta$ autoregressively
generate the output sequence $Y$.

###  2.2 Continual Learning

The aim of Continual Learning (CL) is to learn a set of tasks sequentially
without catastrophically forgetting previously learned ones. In task-oriented
dialogue systems, we cast CL as learning a sequence of domains sequentially.
Let us define a curriculum of $T$ domains as the ordered set
$\mathcal{D}=\\{D_{1},\cdots,D_{T}\\}$, where $D_{k}$ is a dataset under the
domain $K$. Moreover, we denote the models’ parameter after learning the task
$K$ with $\theta_{K}$.

Following the recently defined taxonomy for CL Wortsman et al. (2020), we
study the settings in where the task-id is provided during training, but not
during testing 111GNs: Task Given during train, Not inference; shared labels.
Meaning, during training the model is aware of which domain is currently
learning, but during testing, the model is evaluated without specifying the
dialogue domain. This assumption makes our CL setting more challenging but
more realist, since during testing the users do not explicitly specify in
which domain they want to operate.

In this paper, we consider three Continual Learning approaches:
regularization, rehearsal and architectural. In our experiments, we describe
the most commonly used methods for each approach, especially the one known to
work well in language tasks. Thus:

  * •

Regularization methods add a regularization term to the current learned
$\theta_{t}$ for avoiding to interfere with the previously learned once
$\theta_{t-1}$. Formally the loss at task $t$ is:

$L_{\theta_{t}}(D_{t})=L_{\theta_{t}}(D_{t})+\lambda\Omega(\theta_{t}-\theta^{*}_{t-1})^{2}$
(4)

where $\theta^{*}_{t-1}$ are a copy of the previously learned parameters
frozen at this stage. In our experiments, we consider two kind of $\Omega$:
the identity function (L2), and the fisher information matrix Kirkpatrick et
al. (2017) (EWC).

  * •

Rehearsal methods uses an episodic memory $\mathcal{M}$ to store examples from
the previously learned domains, and re-use them while learning new tasks. The
most straightforward methods is to add the content of the memory $\mathcal{M}$
to the current task data $D_{t}$. Following our notation, the model is
optimized using $L_{\theta_{t}}(D_{t}+\mathcal{M})$, and we refer to this
method as REPLAY. Another rehearsal method is to constrain the gradients
updates so that the loss of the samples in memory never increases. More
formally:

$L_{\theta_{t}}(D_{t})\ \text{s.t.}\ L_{\theta_{t}}(\mathcal{M})\leq
L_{\theta_{t-1}}(\mathcal{M})$ (5)

Of this kind, Gradient Episodic Memory (GEM) Lopez-Paz and Ranzato (2017)
compute the gradient constrain via a quadratic programming solver that scales
with the number of parameters of the model. After a first investigation, we
discover that is impractical for large-language model to use GEM, since they
have millions of parameters and the constrain is computed for each batch. To
cope with this computational complexity, Chaudhry et al. (2018) proposed A-GEM
that efficiently compute the gradient constrains while being effective in CL
tasks. Finally, a rehearsal method specific to language is LAMOL Sun et al.
(2019) which instead of storing samples in $\mathcal{M}$, learn a model that
simultaneously learns to solve the tasks and generate training samples.

  * •

Architectural methods add task-specific parameters to an existing base model
for each task. Of this kind, multiple models has been proposed such as
Progressive Net Rusu et al. (2016), Dynamically Expandable Networks (DEN) Yoon
et al. (2017) and Learn-to-Grow Li et al. (2019b). On the other hand, there
are fixed capacity methods, that do not add specific parameters but they learn
parameters-masks Fernando et al. (2017), usually binary Mallya et al. (2018),
to select sub-networks that are task-specific. To the best of our knowledge,
these models have been tested mostly on computer vision tasks, and they can
not easily handle our continual learning settings (i.e., no task-id during
testing).

Figure 3: High-level representation of the AdapterCL.

##  3 AdapterCL

Motivated by the lack of architectural baselines for CL in sequence-to-
sequence modelling, we propose a novel architectural method called AdapterCL.
Our proposed method parameterizes each task using Residual Adapters Houlsby et
al. (2019) and uses an entropy-based classifier to select which adapter to use
at testing time. This method is designed for large pre-trained language
models, e.g. GPT-2, since only the task-specific parameter are trained while
the original weights are left frozen.

#### Residual adapters

are trainable parameters added on top of each transformer layer, which steer
the output distribution of a pre-trained model without modifying its original
weights. An adapter block consists of a Layer Normalization Ba et al. (2016),
followed by a two linear layer Hinton and Zemel (1994) with a residual
connection. Given the hidden representation at layer $l$ denoted as
$H\in\mathbb{R}^{p\times d}$ of a transformer Vaswani et al. (2017), where $d$
is the hidden size and $p$ is the sequence length, the residual adapter
computes:

$\mathrm{Adapter}_{\mu_{i}^{l}}(H)=\mathrm{ReLU}(\mathrm{LN}(x)W_{l}^{E})W_{l}^{D}+H,$
(6)

where $W_{l}^{E}$ and $W_{l}^{D}$ are trainable parameters of dimensions
$d\times b$ and $b\times d$ respectively, and LN$(\cdot)$ denotes the layer
normalization. The bottleneck dimension $b$ is a tunable hyper-parameter that
allows to adjust the capacity of the adapter according to the complexity of
the target task. We define the set of $\mu_{i}=\\{W_{0}^{E}$,
$W_{0}^{D},\cdots,W_{L}^{E}$, $W_{L}^{D}\\}$ as the set of parameters for the
Adapteri for a model with $L$ layers.

To continuously learning new tasks, we first spawn a new Adapter,
parameterized by $\mu$, and then we train its parameters as in Equation 3. For
instance, given the dataset $D_{t}$ and the model with its corresponding
Adapter $\mu_{t}$, the loss is defined as:

$\mathcal{L}_{\mu_{t}}(D_{t})=-\sum_{j}^{|D_{t}|}\sum_{i=0}^{n+m}\log
p_{\mu_{t}}(x_{i}|x_{0},\cdots x_{i-1})$ (7)

Importantly, the loss is optimized over $\mu_{t}$ to guarantee that each task
is independently learned. An high-level representation of AdapterCL is shown
in Figure 3.

#### Perplexity-Based Classifier

In our CL setting the task-id is provided during training, thus each $\mu_{t}$
is optimized over $D_{t}$. During testing, instead, the task-id is not
provided, thus the model has to predict which adapter to use for accomplishing
the task. This step is not required in regularization and rehearsal approaches
since a single set of parameters is optimised during training.

Inspired by Wortsman et al. (2020), we propose to utilize the perplexity of
each adapter over the input $X$ as a measure of uncertainty. Thus, by
selecting the adapter with lowest perplexity, we select the most confident
model to generate the output sequence. The perplexity of an input sequence
$X=x_{0},\cdots,x_{n}$ is defined as:

$\mathrm{PPL}_{\theta}(X)=\sqrt[n]{\prod_{i=1}^{n}\frac{1}{p_{\theta}\left(x_{i}\mid
x_{0},\cdots,x_{i-1}\right)}}$ (8)

Therefore, given the set of adapters parametrized by $\mu_{0},\dots,\mu_{N}$,
each of which trained respectively with $D_{0},\dots,D_{N}$, and an input
sample $X$, we compute:

$\alpha_{t}=\mathrm{PPL}_{\mu_{t}}(X)\ \forall t\in 1,\cdots,N,$ (9)

in where each $\alpha_{t}$ represent the confidence of the adapter $t$ for the
input $X$. The task-id $t$ is thus selected as

$t^{*}=\operatorname*{argmin}{\alpha_{0},\cdots,\alpha_{N}}$ (10)

The perplexity-based selector requires a linear number of forwards with
respect to the number of Adapters (Equation 9), but it has the advantage of
not requiring a further classifier, which itself would suffer from
catastrophic forgetting and would require an episodic memory.

##  4 Experimental Settings

In this sections we describe 1) the datasets used for creating the learning
curriculum, 2) the evaluation metric used to evaluate the different
settings,and 3) the experimental setups.

Name | Train | Valid | Test | Dom. | Intent | Turns
---|---|---|---|---|---|---
TM19 | 4,403 | 551 | 553 | 6 | 112 | 19.97
TM20 | 13,839 | 1,731 | 1,734 | 7 | 128 | 16.92
MWoZ | 7,906 | 1,000 | 1,000 | 5 | 15 | 13.93
SGD | 5,278 | 761 | 1,531 | 19 | 43 | 14.71
Total | 31,426 | 4,043 | 4,818 | 37 | 280 | 16.23

Table 1: Main datasets statistics.

###  4.1 Datasets

To the best of our knowledge, there is no benchmark for CL in dialogue systems
with a high number of tasks to be learned continuously and with multiple
training settings. The closest to our is the work is from Mi et al. (2020),
which continuously learn 5 domains in the NLG settings. In general, NLP
benchmarks for continual learning uses no more that 10 tasks Sun et al.
(2019); d’Autume et al. (2019). Consequently, in this paper we propose a
Continual Learning benchmark by jointly pre-processing four task-oriented
dataset such as Task-Master 2019 (TM19) Byrne et al. (2019), Task-Master 2020
(TM20) Byrne et al. (2019), Schema Guided Dialogue (SGD) Rastogi et al. (2019)
and MultiWoZ Budzianowski et al. (2018). This results in a curriculum of 37
domains to be learned continuously under four setting such as INTENT
classification, DST, NLG, and E2E. This has been possible because the four
datasets provides the speech-act annotation for both the user and the system
turns, and the dialogue-state as well. For avoiding any domain overlapping, we
select only the dialogues with a single domain and we do not merge domains
with similar/same names or semantics. For example, the restaurant domain
appears in TM19, TM20 and MWOZ, with different slot names and values. Thus, we
intentionally keep these data samples separate for modelling scenarios in
which multiple apis are available for the same domain.

Finally, the dataset are pre-processed as in Section 2.1 to form the four
tasks, and the main statistics are shown in Table 1. In appendix, Table 5 we
report the number of sample by domain and setting, where we can notice how
different domains have a much different sample size, from few-hundred samples
(e.g., SGD-travel) to 15K (e.g., TM19-flight). Importantly, since not all the
datasets provides a delexicalized version of the responses, we decide to keep
all the dataset in their plain text form.

|  |  | INTENT | DST | NLG
---|---|---|---|---|---
Method | +Param. | Mem. | Accuracy$\uparrow$ | JGA$\uparrow$ | EER$\downarrow$ | BLEU$\uparrow$
VANILLA | - | $\emptyset$ | 4.08 $\pm$ 1.4 | 4.91 $\pm$ 4.46 | 48.73 $\pm$ 3.81 | 6.38 $\pm$ 0.6
L2 | $|\theta|$ | $\emptyset$ | 3.74 $\pm$ 1.4 | 3.81 $\pm$ 3.44 | 55.68 $\pm$ 7.09 | 5.4 $\pm$ 0.9
EWC | $2|\theta|$ | $\emptyset$ | 3.95 $\pm$ 1.3 | 5.22 $\pm$ 4.46 | 58.2 $\pm$ 3.66 | 5.06 $\pm$ 0.5
AGEM | - | $t|M|$ | 34.04 $\pm$ 6.36 | 6.37 $\pm$ 4.0 | 62.09 $\pm$ 6.88 | 4.54 $\pm$ 0.6
LAMOL | - | $\emptyset$ | 7.49 $\pm$ 6.35 | 4.55 $\pm$ 3.48 | 66.11 $\pm$ 6.97 | 3.0 $\pm$ 0.9
REPLAY | - | $t|M|$ | 81.08 $\pm$ 1.37 | 30.33$\pm$ 1.24 |  17.72 $\pm$ 0.85 |  17.4 $\pm$ 0.68
ADAPT | $t|\mu|$ | $\emptyset$ |  90.46 $\pm$ 0.6 |  35.06 $\pm$ 0.52 | 31.78 $\pm$ 1.28 | 16.76 $\pm$ 0.34
MULTI | - | - | 95.45 $\pm$ 0.1 | 48.9 $\pm$ 0.2 | 12.56 $\pm$ 0.2 | 23.61 $\pm$ 0.1
Table 2: E2E results in term of Intent accuracy, Joint-Goal-Accuracy (JGA),
Slot-Error-Rate (EER) and BLUE. +Param shows the additional number of
parameters per task, and Mem the episodic memory size need per task.

###  4.2 Evaluation Metric

Automatic evaluations for end-to-end task-oriented dialogue systems is
challenging, especially for the response generation. To overcome this issue,
in this paper we use well-defined metric based on the three modularized
settings. In each of the three sub-tasks, we define four metric as following:

  * •

INTENT recognition is evaluated using the accuracy between the generated
intent and the one provided in the gold label.

  * •

DST is evaluated with the Joint Goal Accuracy (JGA) Wu et al. (2019) over the
gold dialogue state.

  * •

NLG is evaluated using both the BLEU score Papineni et al. (2002) and the slot
error rate (EER) Wen et al. (2015) which computed by ratio between the total
number of slots and the values not appearing in the response. In datasets such
as SGD, the slot has binary values e.g., yes or no, and thus we exclude those
from the count as in Kale and Rastogi (2020). In the E2E setting, if the api
output ($X_{out}$) is empty, then we rely on the BLEU score.

Independently from the metric, we compute continual learning specific metrics
such as the average metric thought time (Avg. Metric) as in Lopez-Paz and
Ranzato (2017). We consider access to the test set for each of the $T$ tasks
and after the model finishes learning the task $t_{i}$, we evaluate the model
test performance on all tasks in the curriculum. By doing so, we construct the
matrix $R\in\mathbb{R}^{T\times T}$, where $R_{i,j}$ is the test metric (e.g.,
BLEU, JGA) of the model on task $t_{j}$ after observing the last sample from
task $t_{i}$. Then we define the average accuracy as:

$\mathrm{Avg.Metric}=\frac{1}{T}\sum_{i=1}^{T}R_{T,i}$ (11)

The Avg. Metric score is useful for understanding the learning dynamics
thought time of different baselines. Further metrics such as Backward-Transfer
and Forward-Transfer Lopez-Paz and Ranzato (2017) are available to distinguish
baselines with similar Avg. Metric score, but in this paper we limit our
evaluation to this metric, since there is a large gap among different
baselines. Finally, to evaluate the adapter selection, we use the accuracy
over the gold task-id.

###  4.3 Baselines and Settings

The main aim of this paper is to compare the performance of different CL
approaches and to understand what are the trade-off among them. Therefore,
following the definition provided in Section 2.2, we compare: 1) EWC and L2,
2) A-GEM, LAMOL, and REPLAY, and 3) AdapterCL. Additionally, we provide a
baselines trained on the task continuously, namely VANILLA, without any
regularization or memory, and a multitask baseline (MULTI), which is trained
on all the data in the curriculum at the same time. In L2, EWC, and A-GEM we
tune different $\lambda$ in the range 0.0001 to 100, and in rehearsal based
methods, such as REPLAY and GEM, we keep 50 samples per task, for a total of
1850 sample in $\mathcal{M}$ at the end of the curriculum. This is
particularly important since if we store in memory all the samples of the seen
tasks the model require an high training cost. Arguably, this could be an
option if the per-task sample size is small, but this is not always possible
as in large language models Brown et al. (2020). Therefore the assumption of
minimizing the number of samples in memory is valid and widely used in the CL
literature Mi et al. (2020). Finally, for the AdapterCL, we tune the
bottleneck size $b$ between 10, 50, 100, 200. Interested readers can refer to
appendix for further details. In Continual Learning the model is not able
decide the order of tasks, therefore, we create 5 learning curriculum by
randomly permuting the 37 tasks.

Figure 4: Avg. Metric for the Intent Accuracy and EER in the E2E setting.

##  5 Results & Analysis

The main results in the end-to-end setting are summarized in Table 2, while
the one from the modularized settings in Table 3 in the Appendix. In these
tables we report the Avg. Metric at the end of the curriculum, which is
equivalent to the average test set performance in all the tasks, and the
resources used by each model.

#### Main Results

From these tables, we can observe that: 1) both regularization-based methods
(L2/EWC) and some rehearsal-based methods (AGEM/LAMOL) suffers greatly in this
task, 2) REPLAY and AdapterCL perform comparably well in the Intent and DST
metric, 3) REPLAY works the best in the NLG task, showing that transferring
learning between tasks is needed, and 4) no CL methods can reach the multi-
task baseline, especially in the DST task. Moreover, the adapter selection
accuracy based on Equation 10, is of 95.44$\pm$0.2% in end-to-end,
98.03$\pm$0.1% in Intent Recognition, 98.19$\pm$0.1% in DST, and
93.98$\pm$0.1%.

Although these numbers are meaningful, they do not describe the entire
learning history of the curriculum. To better understand this dynamics, we
plot the Avg. Metric in Equation 11 after each task is learned (i.e. t=T in
the equation). Figure 4 shows the plot for the two of the considered metrics
and all the baselines. From this figure we can better understand how REPLAY
and AdapterCL outperforms other baselines and, interestingly, that LAMOL
performs as well as replay in the first 12 tasks. This because LAMOL learns to
generate training samples instead of using an explicit memory, and thus the
generation become harder when more and more task are shown. This further
strength our motivation for having a benchmark with a long curriculum. In
Appendix, Figure 6 shows the remain two metrics for the E2E setting and Figure
7 show the same plots for the individual modules training.

###  5.1 No Free Lunch

Finally based on the results shown in Table 2, especially based on the
resource used by each methods, we conclude that there is a no free lunch in
term of resources needed for avoiding the catastrophic problem. To elaborate,
in both REPLAY and AdapterCL the resource used grows linearly with the number
of tasks, i.e., in REPLAY the number of samples stored in the episodic memory
grows linearly (i.e., 50 times the number of tasks), and in AdapterCL the
number of parameters grows linearly (i.e., number of adapter parameter times
the number of tasks). Figure 10 in Appendix describe an high-level intuition
of this concept by plotting the number of tasks vs parameter and episodic
memory size.

Therefore, given a resource budget, in term of memory or parameters, different
baselines are preferable. The main advantage of using memory-base (e.g.,
REPLAY) that no parameters are added thus the resulting model is closer to a
multitask baseline, but it comes with the disadvantage of loosing the weights
of the original pre-trained model. This is particularly critical for large
pre-trained LMs which provide a good starting points for fine-tuning new
tasks. On the other hand, the main advantage of parameters isolation methods
(e.g., AdapterCL) is the ability to retaining the original weights and the
ability of controlling which task to trigger given a certain input. The latter
is important in scenarios where just a subset of domains are shown to the user
(e.g. only one particular restaurant api). The main disadvantage, instead, is
the lack of knowledge transfer among tasks, since each dataset is trained in
isolation.

Figure 5: Ablation study on the size of the episodic-memory vs JGA.

###  5.2 Analysis: Episodic Memory Size

In this section, we analyses the effect of increasing the episodic memory size
for the REPLAY method. Trivially, by including all the training sample in the
memory, the model at the last task converges to the multitask baseline. Then,
the question of many how sample to keep per task for avoiding catastrophic
forgetting is important. In light of this, Figure 5 shows the performance of
different Episodic Memory size in the DST task. Here we observe that by
storing only few samples per task (i.e., 10-50) the model still greatly
suffers from catastrophic forgetting, where instead with around 500 samples,
which is equivalent to a total of 18,500 sample in our setting, the
performance are closer to the multitask baseline (i.e., a possible upper
bound). Similar observation are shown for the other two tasks Figure 8 9 in
Appendix.

##  6 Related Work

Continual Learning methods are usually developed and benchmark on computer
visions tasks. Interested readers may refer to Mundt et al. (2020); Parisi et
al. (2019); De Lange et al. (2019) for an overview of existing approaches, and
to Section 2.2 for more details on the three main CL approaches studied in
this paper. On the other hand, Continual Learning is also studied in the Long
Life Learning (LLL) scenario, where a learner continuously accumulate
knowledge and makes use of it in the future Chen and Liu (2018); Liu and Mei
(2020). In this paper, we study the setting in where a series of task are
learned continuously.

#### Continual Learning in NLP

has been explore in both in classification d’Autume et al. (2019); Sprechmann
et al. (2018); Wang et al. (2020) and generation Sun et al. (2019); Hu et al.
(2020) tasks. For instance, Sun et al. (2019); Chuang et al. (2020) proposed
LAMOL, which we use as our baselines, and studied its effectiveness in a
subset of DecaNLP McCann et al. (2018). On the other hand, the work of
d’Autume et al. (2019); Sprechmann et al. (2018) are not suitable for
interactive systems as dialogue systems, since they require a local adaptation
(i.e., a fine-tuning step) during inference. Finally, continual learning has
been used for sentence encoder Liu et al. (2019), composition language
learning Li et al. (2019c) and relation learning Han et al. (2020). However,
these methods are very specific to particular NLP applications.

#### Continual Learning in Dialogue Systems

The very early work on CL for Task-Oriented dialogue is from Lee (2017), where
EWC is used to avoid catastrophic forgetting on three domains learned
sequentially. Continual Learning has been studied in the NLG setting, where a
single model is trained to learn one domain at the time in MWoZ Mi et al.
(2020). The author, used a episodic memory to replay the example in
combination with EWC. In this paper, we compare similar baselines, in a large
benchmark that also include MWoZ and the NLG settings. Continual Learning has
been studied also in the DST setting in Wu et al. (2019) using MWoZ, where
several baselines such as L2, EWC and GEM has been compared. Differently, Li
et al. (2019a) leverage CL for evaluate the quality of chat-bot models and He
et al. (2019) studied the catastrophic forgetting problem in Chit-Chat
systems. Finally, Shuster et al. (2020) showed that by training models on
humans-machine conversations in a open-domain fantasy world game Fan et al.
(2020) the models progressively improve, as measured by automatic metrics and
online engagement scores.

##  7 Conclusion

In this paper, we proposed a benchmark for Continual Learning in task-oriented
dialogue systems, with 37 tasks to be learned continuously on four settings
such as Intent recognition, Dialogue State Tracking, Natural Language
Generation, and end-to-end. Then, we implemented three different Continual
Learning methodologies such as regularization, rehearsal and architectural. In
the latter, we propose a simple yet effective methods based on residual
adapters and uses an entropy-based classifier to select which adapter to use
at testing time. Finally, we analyse the trade-off between performance,
number-of-parameters, and episodic memories size of the evaluated baselines,
unveiling a no-free lunch among this methods.

## References

  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016.  Layer normalization.  _arXiv preprint arXiv:1607.06450_.
  * Bengio et al. (2003) Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003.  A neural probabilistic language model.  _Journal of machine learning research_ , 3(Feb):1137–1155.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gašić. 2018.  Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Byrne et al. (2020) Bill Byrne, Karthik Krishnamoorthi, Saravanan Ganesh, and Mihir Sanjay Kale. 2020\.  Tickettalk: Toward human-level performance with end-to-end, transaction-based dialog systems.  _arXiv preprint arXiv:2012.12458_.
  * Byrne et al. (2019) Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019.  Taskmaster-1:toward a realistic and diverse dialog dataset.  In _2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing_ , Hong Kong.
  * Chaudhry et al. (2018) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2018\.  Efficient lifelong learning with a-gem.  _arXiv preprint arXiv:1812.00420_.
  * Chen and Liu (2018) Zhiyuan Chen and Bing Liu. 2018.  Lifelong machine learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , 12(3):1–207.
  * Chuang et al. (2020) Yung-Sung Chuang, Shang-Yu Su, and Yun-Nung Chen. 2020.  Lifelong language knowledge distillation.  _arXiv preprint arXiv:2010.02123_.
  * d’Autume et al. (2019) Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019\.  Episodic memory in lifelong language learning.  _arXiv preprint arXiv:1906.01076_.
  * De Lange et al. (2019) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2019.  A continual learning survey: Defying forgetting in classification tasks.  _arXiv preprint arXiv:1909.08383_.
  * Fan et al. (2020) Angela Fan, Jack Urbanek, Pratik Ringshia, Emily Dinan, Emma Qian, Siddharth Karamcheti, Shrimai Prabhumoye, Douwe Kiela, Tim Rocktäschel, Arthur Szlam, et al. 2020.  Generating interactive worlds with text.  In _AAAI_ , pages 1693–1700.
  * Fernando et al. (2017) Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. 2017.  Pathnet: Evolution channels gradient descent in super neural networks.  _arXiv preprint arXiv:1701.08734_.
  * Han et al. (2020) Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020.  Continual relation learning via episodic memory activation and reconsolidation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6429–6440.
  * He et al. (2019) Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. 2019.  Mix-review: Alleviate forgetting in the pretrain-finetune framework for neural language generation models.  _arXiv preprint arXiv:1910.07117_.
  * Hinton and Zemel (1994) Geoffrey E Hinton and Richard S Zemel. 1994.  Autoencoders, minimum description length and helmholtz free energy.  In _Advances in neural information processing systems_ , pages 3–10.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A simple language model for task-oriented dialogue.  _arXiv preprint arXiv:2005.00796_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  _arXiv preprint arXiv:1902.00751_.
  * Hu et al. (2020) Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. 2020.  Drinking from a firehose: Continual learning with web-scale natural language.  _arXiv preprint arXiv:2007.09335_.
  * Kale and Rastogi (2020) Mihir Kale and Abhinav Rastogi. 2020.  Few-shot natural language generation by rewriting templates.  _arXiv preprint arXiv:2004.15006_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Lee (2017) Sungjin Lee. 2017.  Toward continual learning for conversational agents.  _arXiv preprint arXiv:1712.09943_.
  * Lei et al. (2018) Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018\.  Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1437–1447.
  * Li et al. (2019a) Lu Li, Zhongheng He, Xiangyang Zhou, and Dianhai Yu. 2019a.  How to evaluate the next system: Automatic dialogue evaluation from the perspective of continual learning.  _arXiv preprint arXiv:1912.04664_.
  * Li et al. (2019b) Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. 2019b.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  _arXiv preprint arXiv:1904.00310_.
  * Li et al. (2019c) Yuanpeng Li, Liang Zhao, Kenneth Church, and Mohamed Elhoseiny. 2019c.  Compositional language continual learning.  In _International Conference on Learning Representations_.
  * Liu and Mei (2020) Bing Liu and Chuhe Mei. 2020.  Lifelong knowledge learning in rule-based dialogue systems.  _arXiv preprint arXiv:2011.09811_.
  * Liu et al. (2019) Tianlin Liu, Lyle Ungar, and João Sedoc. 2019.  Continual learning for sentence representations using conceptors.  _arXiv preprint arXiv:1904.09187_.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  In _Advances in neural information processing systems_ , pages 6467–6476.
  * Mallya et al. (2018) Arun Mallya, Dillon Davis, and Svetlana Lazebnik. 2018.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 67–82.
  * McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018.  The natural language decathlon: Multitask learning as question answering.  _arXiv preprint arXiv:1806.08730_.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier.
  * Mi et al. (2020) Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. 2020.  Continual learning for natural language generation in task-oriented dialog systems.  _arXiv preprint arXiv:2010.00910_.
  * Mundt et al. (2020) Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh. 2020.  A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning.  _arXiv preprint arXiv:2009.01797_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Parisi et al. (2019) German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. 2019.  Continual lifelong learning with neural networks: A review.  _Neural Networks_ , 113:54–71.
  * Peng et al. (2020) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2020.  Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model.  _arXiv preprint arXiv:2005.05298_.
  * Rastogi et al. (2019) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  _arXiv preprint arXiv:1909.05855_.
  * Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_.
  * Shuster et al. (2020) Kurt Shuster, Jack Urbanek, Emily Dinan, Arthur Szlam, and Jason Weston. 2020.  Deploying lifelong open-domain dialogue learning.  _arXiv preprint arXiv:2008.08076_.
  * Sprechmann et al. (2018) Pablo Sprechmann, Siddhant M Jayakumar, Jack W Rae, Alexander Pritzel, Adria Puigdomenech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. 2018.  Memory-based parameter adaptation.  _arXiv preprint arXiv:1802.10542_.
  * Sun et al. (2019) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2019.  Lamol: Language modeling for lifelong language learning.  In _International Conference on Learning Representations_.
  * Thrun and Pratt (2012) Sebastian Thrun and Lorien Pratt. 2012.  _Learning to learn_.  Springer Science & Business Media.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2020) Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime Carbonell. 2020\.  Efficient meta lifelong-learning with limited memory.  _arXiv preprint arXiv:2010.02500_.
  * Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015.  Semantically conditioned lstm-based natural language generation for spoken dialogue systems.  _arXiv preprint arXiv:1508.01745_.
  * Wortsman et al. (2020) Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. 2020.  Supermasks in superposition.  _Advances in Neural Information Processing Systems_ , 33.
  * Wu et al. (2019) Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019.  Transferable multi-domain state generator for task-oriented dialogue systems.  _arXiv preprint arXiv:1905.08743_.
  * Yoon et al. (2017) Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. 2017.  Lifelong learning with dynamically expandable networks.  _arXiv preprint arXiv:1708.01547_.

##  Appendix A Appendices

##  Appendix B Supplemental Material

Figure 6: Avg. Metric for the response generation, i.e., the BLUE and EER.

Figure 7: Modularized results Figure 8: Ablation study on the size of the
episodic-memory vs the intent accuracy.

Figure 9: Ablation study on the size of the episodic-memory vs BLEU/EER.  |  |  | INTENT | DST | NLG
---|---|---|---|---|---
Method | +Parm. | Mem. | Accuracy $\uparrow$ | JGA $\uparrow$ | EER $\downarrow$ | BLEU $\uparrow$
VANILLA | - | $\emptyset$ | 3.27 $\pm$ 0.3 | 5.34 $\pm$ 4.4 | 14.81 $\pm$ 7.7 | 11.06 $\pm$ 2.9
L2 | $|\theta|$ | $\emptyset$ | 3.52 $\pm$ 0.7 | 4.95 $\pm$ 4.4 | 12.93 $\pm$ 5.5 | 11.99 $\pm$ 1.4
EWC | $2|\theta|$ | $\emptyset$ | 3.21 $\pm$ 0.3 | 5.36 $\pm$ 4.3 | 14.2 $\pm$ 6.2 | 11.19 $\pm$ 2.4
AGEM | - | $t|M|$ | 9.74 $\pm$ 2.6 | 5.17 $\pm$ 4.0 | 34.2 $\pm$ 8.6 | 5.51 $\pm$ 2.1
LAMOL | - | $\emptyset$ | 3.73 $\pm$ 1.0 | 4.03 $\pm$ 3.9 | 29.61 $\pm$ 3.1 | 5.42 $\pm$ 1.7
REPLAY | - | $t|M|$ | 76.45 $\pm$ 1.5 |  39.42 $\pm$ 0.2 |  4.95 $\pm$ 1.5 |  21.72 $\pm$ 0.3
ADAPT | $t|\mu|$ | $\emptyset$ |  85.05 $\pm$ 0.6 | 37.9 $\pm$ 0.6 | 14.36 $\pm$ 0.7 | 21.48 $\pm$ 0.2
MULTI | - | - | 87.50 $\pm$ 0.2 | 50.04 $\pm$ 0.1 | 2.84 $\pm$ 0.2 | 26.15 $\pm$ 0.2
Table 3: Results Modularized | INTENT | DST | NLG
---|---|---|---
$|\mathcal{M}|$ | Accuracy $\uparrow$ | JGA$\uparrow$ | EER$\downarrow$ | BLEU $\uparrow$
10 | 57.286$\pm$3.80 | 26.63$\pm$1.26 | 8.44$\pm$0.97 | 18.86$\pm$0.68
50 | 76.446$\pm$1.55 | 39.41$\pm$0.28 | 6.63$\pm$0.53 | 21.11$\pm$0.41
100 | 81.496$\pm$0.86 | 43.13$\pm$0.31 | 5.75$\pm$0.19 | 21.71$\pm$0.25
500 | 85.91$\pm$0.55 | 48.22$\pm$0.53 | 4.96$\pm$0.10 | 22.86$\pm$0.25
ALL | 87.784$\pm$0.16 | 49.97$\pm$0.46 | 4.36$\pm$0.24 | 23.85$\pm$0.12
MULTI | 87.5$\pm$0.1 | 50.04 $\pm$ 0.6 | 3.42$\pm$0.1 | 26.15$\pm$0.1
Table 4: Ablation study over episodic memory size $|\mathcal{M}|$. In the table $|\mathcal{M}|$ represent the number of samples per task kept in memory.  Domains | DST-INTENT | NLG | End-to-End
---|---|---|---
| Train | Dev | Test | Train | Dev | Test | Train | Dev | Test
TM19 movie | 4733 | 584 | 500 | 3010 | 366 | 341 | 12766 | 1632 | 1481
TM19 auto | 3897 | 448 | 522 | 2128 | 223 | 283 | 10918 | 1248 | 1443
TM19 restaurant | 4434 | 568 | 561 | 2582 | 330 | 333 | 12862 | 1669 | 1630
TM19 pizza | 2883 | 381 | 359 | 1326 | 171 | 171 | 8720 | 1145 | 1083
TM19 uber | 4378 | 535 | 525 | 2418 | 290 | 278 | 11331 | 1362 | 1361
TM19 coffee | 2591 | 302 | 335 | 1381 | 151 | 184 | 7429 | 894 | 936
TM20 flight | 15868 | 1974 | 1940 | 10148 | 1272 | 1245 | 36778 | 4579 | 4569
TM20 food-ordering | 3404 | 411 | 431 | 2394 | 277 | 287 | 7838 | 941 | 986
TM20 hotel | 15029 | 1908 | 1960 | 6590 | 842 | 869 | 35022 | 4400 | 4532
TM20 music | 5917 | 764 | 769 | 4196 | 537 | 523 | 13723 | 1773 | 1787
TM20 restaurant | 13738 | 1761 | 1691 | 8356 | 1063 | 994 | 34560 | 4398 | 4297
TM20 sport | 13072 | 1668 | 1654 | 12044 | 1553 | 1542 | 29391 | 3765 | 3723
TM20 movie | 13221 | 1703 | 1567 | 9406 | 1203 | 1093 | 32423 | 4158 | 3881
MWOZ taxi | 1239 | 234 | 194 | 402 | 71 | 56 | 2478 | 468 | 388
MWOZ train | 1452 | 158 | 160 | 563 | 63 | 59 | 2905 | 316 | 320
MWOZ restaurant | 5227 | 243 | 281 | 3333 | 141 | 177 | 10461 | 486 | 563
MWOZ hotel | 2798 | 289 | 385 | 1924 | 194 | 258 | 5602 | 579 | 771
MWOZ attraction | 484 | 43 | 42 | 295 | 27 | 26 | 975 | 86 | 85
sgd restaurants | 2686 | 278 | 616 | 1720 | 166 | 386 | 5756 | 606 | 1354
sgd media | 1411 | 230 | 458 | 988 | 167 | 324 | 3114 | 502 | 1005
sgd events | 4881 | 598 | 989 | 3241 | 389 | 590 | 10555 | 1317 | 2197
sgd music | 1892 | 275 | 556 | 1506 | 224 | 464 | 4040 | 597 | 1215
sgd movies | 1665 | 181 | 52 | 996 | 114 | 44 | 3760 | 420 | 126
sgd flights | 4766 | 1041 | 1756 | 2571 | 627 | 982 | 10429 | 2244 | 3833
sgd ridesharing | 652 | 85 | 187 | 377 | 48 | 107 | 1448 | 188 | 418
sgd rentalcars | 1510 | 250 | 469 | 865 | 153 | 280 | 3277 | 538 | 1009
sgd buses | 1862 | 331 | 653 | 1102 | 218 | 412 | 4050 | 709 | 1393
sgd hotels | 3237 | 394 | 948 | 1997 | 243 | 597 | 6983 | 858 | 2053
sgd services | 3328 | 360 | 926 | 2225 | 230 | 611 | 7262 | 803 | 2016
sgd homes | 2098 | 170 | 533 | 1312 | 96 | 338 | 4519 | 394 | 1158
sgd banks | 1188 | 139 | 293 | 723 | 84 | 181 | 2599 | 319 | 667
sgd calendar | 592 | 115 | 236 | 397 | 65 | 133 | 1313 | 246 | 501
sgd alarm | 212 | 34 | 91 | 221 | 30 | 74 | 580 | 82 | 198
sgd weather | 196 | 32 | 80 | 123 | 23 | 59 | 433 | 70 | 169
sgd travel | 186 | 23 | 48 | 121 | 14 | 30 | 420 | 53 | 106
sgd payment | 227 | 21 | 51 | 143 | 14 | 32 | 497 | 44 | 113
sgd trains | 300 | 73 | 128 | 149 | 43 | 66 | 668 | 158 | 274
Total | 147254 | 18604 | 22946 | 93273 | 11722 | 14429 | 347885 | 44047 | 53641
Table 5: All data-samples used in the experiments Figure 10: No Free Lunch in
CL. This plot the trade-off between number of parameters added per task and
the size of the episodic memory $\mathcal{M}$.