# Cross-lingual Continual Learning

Meryem M’hamdi Xiang Ren Jonathan May
Information Sciences Institute
University of Southern California
{meryem, xiangren<EMAIL_ADDRESS>
###### Abstract

The longstanding goal of multi-lingual learning has been to develop a
universal cross-lingual model that can withstand the changes in multi-lingual
data distributions. There has been a large amount of work to adapt such multi-
lingual models to unseen target languages. However, the majority of work in
this direction focuses on the standard one-hop transfer learning pipeline from
source to target languages, whereas in realistic scenarios, new languages can
be incorporated at any time in a sequential manner. In this paper, we present
a principled Cross-lingual Continual Learning (CCL) evaluation paradigm, where
we analyze different categories of approaches used to continually adapt to
emerging data from different languages. We provide insights into what makes
multilingual sequential learning particularly challenging. To surmount such
challenges, we benchmark a representative set of cross-lingual continual
learning algorithms and analyze their knowledge preservation, accumulation,
and generalization capabilities compared to baselines on carefully curated
datastreams. The implications of this analysis include a recipe for how to
measure and balance different cross-lingual continual learning desiderata,
which go beyond conventional transfer learning.

##  1 Introduction

Figure 1: An overview of CCL: We use an example of a non-stationary datastream
moving from high to low resource languages. Each bold and dashed box
represents either a training or test data instance being fine-tuned or
evaluated on, respectively. To support this problem setup, we evaluate the
cross-lingual capabilities of _continual approaches_. Those capabilities
include knowledge preservation on old languages, accumulation to the current
language, and generalization to unseen languages at each point of the
training. In addition to that, we evaluate model utility at the end of
continual learning.

With more than 7,000 languages spoken around the globe, downstream
applications still lack proper linguistic resources across languages Joshi et
al. (2020), necessitating the use of transfer learning techniques that take
advantage of data that is mismatched to the application. In an effort to
simplify architecture complexity and energy consumption, it is desirable to
unify multi-lingual performance into a single, parameter- and memory-
constrained model, and to allow this model to evolve, learning on multi-
lingual training data as it becomes available without having to pre-train or
fine-tune from scratch. Such is the longstanding goal of language
representation learning. Existing multi-lingual representations such as M-BERT
Devlin et al. (2019) and XLM-R Conneau et al. (2020) are strong pillars in
cross-lingual transfer learning, but if care is not taken when choosing how to
fine-tune them, they can neglect to maximize transfer Ruder et al. (2019) to
new tasks or languages and are subject to forgetting McCloskey and Cohen
(1989), where performance decreases after exposure to new task or language.

Most previous work that attempts to deal with the challenge of transfer
exploitation and forgetting mitigation focuses on the problem of sequentially
learning over different NLP downstream tasks or domains Sun et al. (2020); Han
et al. (2020); Madotto et al. (2021), rather than on language shifts. Indeed,
the current literature for learning over sequences of languages is rather
scarce and is mostly reduced to cross-lingual transfer learning between a pair
of languages Liu et al. (2021); Garcia et al. (2021); Muller et al. (2021);
Pfeiffer et al. (2021); Minixhofer et al. (2022). Liu et al. pre-train a
(parent) language model and then fine-tune it on a downstream task in one of
several different (child) languages. This conflates task and language transfer
and confuses analysis – the interference between the pre-trained language
model ‘task’ and the fine-tuned task, along with the parent and child
languages, cannot be disentangled. Garcia et al. propose an adaptation scheme
to each new language pair independently while retaining the translation
quality on the parent language pairs. Similarly, Muller et al. (2021) and
Pfeiffer et al. (2021) propose lexical and semantic level techniques to adapt
to target languages. However, all these mentioned works still focus on the
‘one-hop’ case, consisting of two steps: (1) training on initial parent
language(s) (pairs), then (2) adapting to new children language(s) (pairs);
the effect of multiple shifts in the datastream is not trivially generalizable
to more than one hop. More recently, Pfeiffer et al. (2022) propose an
approach for language-specific modules based on adapters and evaluate that on
sequential streams of languages. However, they only focus on adapters and two
desiderata of continual learning: interference mitigation and transfer
maximization. We need a more robust and comprehensive fine-grained evaluation
that balances the dynamics between different cross-lingual continual learning
desiderata.

In this paper, we pave the way for a more comprehensive multi-hop continual
learning evaluation that simulates the sequential learning of a single task
over a stream of input from different languages. This evaluation paradigm
requires experimentation over balanced streams of $n$ data scenarios for
$n>2$. Unlike previous work, this paper concretely defines the following
comprehensive goals along with their evaluation metrics as guidelines for
analyzing the cross-lingual capabilities of multilingual sequential training:
knowledge preservation, accumulation, generalization, and model utility as
shown in Figure 1. We apply our test bed to a six-language task-oriented
dialogue benchmark and comprehensively analyze a wide variety of successful
continual learning algorithms. These algorithms are derived from previous
literature investigated in continual learning contexts different from the
cross-lingual context, including (a) model-expansion Pfeiffer et al. (2020b),
(b) regularization Kirkpatrick et al. (2017), (c) memory replay Chaudhry et
al. (2019b), and (d) distillation-based approaches Hinton et al. (2015);
Aguilar et al. (2020). Our findings confirm the need for a multi-hop analysis
and the effectiveness of continual learning algorithms in enhancing knowledge
preservation and accumulation of our multilingual language model. We
additionally demonstrate the robustness of different continual learning
approaches to variations in individual data setup choices that would be
misleading if presented in a traditional manner.

Our main contributions are:  (1) We are the first to explore and analyze
cross-lingual continual fine-tuning111Our code is available at
https://github.com/meryemmhamdi1/x-continuous-learning. across multiple hops
and show the importance of this multi-hop analysis in reaching clearer
conclusions with greater confidence compared to conventional cross-lingual
transfer learning (§4.1).  (2) We demonstrate the aggregated effectiveness of
a range of different continual learning approaches (Figure 1) at reducing
forgetting and improving transfer (§4.3) compared to multilingual sequential
baselines (§4.2). (3) We make concrete recommendations on model design to
balance transfer and final model performance with forgetting (§4.3).  (4) We
show that the order of languages and data set size impacts the knowledge
preservation and accumulation of multi-lingual sequential fine-tuning and
identify the continual learning approaches that are most robust to this
variation (§4.4).  (5) We analyze zero-shot generalization trends and their
correlation with forgetting and show that current continual learning
approaches do not substantially improve the generalization (§4.5).

##  2 Cross-lingual Continual Learning

In this section, we formally define cross-lingual continual learning, describe
its goals and challenges, and introduce the downstream tasks, datastreams, and
evaluation protocols used. Although we are not the first to define or
investigate continual learning for languages, we are, to the best of our
knowledge, the first to define and study cross-lingual continual learning
where continual learning is focused on languages only. Thus, we formally
define cross-lingual continual learning as learning over a set of languages
seen sequentially in multiple hops, which is truer to the term of cross-
lingual and continual learning, respectively. We distinguish that from ‘cross-
lingual cross-task cross-stage continual learning’, which continually learns
over a set of pretraining and downstream tasks sampled from different
languages Liu et al. (2021) and ‘cross-lingual one-hop transfer learning’
Garcia et al. (2021).

###  2.1 Problem Formulation

We define cross-lingual continual learning as the problem of sequentially
fine-tuning a model $\theta$ for a particular downstream task K over a cross-
lingual datastream. In this case, a cross-lingual data stream is made of $N$
labeled and distinct datasets $\mathcal{D}_{1\cdots{N}}$, each one sampled
from a distinct language and consisting of separate train and test portions.
Let $hop_{i}$ be the stage in cross-lingual continual learning where
$\theta_{i}$ is optimized to $\theta_{i+1}$ via exposure to $\mathcal{D}_{i}$.
Let
$\mathscr{L}=\\{\boldsymbol{\ell}_{1},\boldsymbol{\ell}_{2}\cdots\boldsymbol{\ell}_{N}\\}$
be a set of labeled languages, let $\mathfrak{S}(\mathscr{L})$ be the set of
all permutations of $\mathscr{L}$, and without loss of generality let
$p\in\mathfrak{S}(\mathscr{L})$ be one such permutation and
$p[i]\in\mathscr{L}$ be the $i$th language in $p$. The language of
$\mathcal{D}_{i}$ is $p[i]$. Therefore, by default, the number of languages
used is equal to the number of datasets. Let $\mathcal{D}_{<i}$ and
$\mathcal{D}_{>i}$ refer to a sequence of datasets (train or test portions,
depending on context) used in hops from 1 to $i-2$ and $i$ to $N-1$,
respectively; we generalize these terms to $\mathcal{D}_{{\leq}i}$ and
$\mathcal{D}_{{\geq}i}$ by including hop $i-1$ as well at the end or,
respectively, beginning of the sequence.

###  2.2 Goals

We define the goals,222To the best of our knowledge, those goals were never
synthesized for the context of cross-lingual continual learning. necessarily
dependent on each other, for our study of cross-lingual continual learning as
follows (also depicted in Figure 1):

  * •

_Cross-lingual preservation._ This is the ability to retain previous knowledge
of seen languages.

  * •

_Cross-lingual accumulation._ This is the ability to accumulate knowledge
learned from previous languages to benefit from learning on the current
language.

  * •

_Cross-lingual generalization._ This is the ability to generalize uniformly
well to unseen languages, which goes beyond accumulating knowledge up to the
current languages.

  * •

_Model utility._ This is the ability of the fully trained model to perform
equally well in all languages.

In this paper, we wish to understand the relationships between these goals.
Our aim is to come up with a recipe for a more systematic cross-lingual
continual learning. Thus, we need to understand if the goals are aligned with
each other or if maximizing some goals leads to minimizing other goals.

###  2.3 Challenges

Learning sequentially from a non-stationary data distribution (i.e., task
datasets coming from different languages) can impose considerable challenges
on the goals defined earlier:

  * •

_Catastrophic forgetting._ This happens when fine-tuning a model on
$\mathcal{D}_{\geq i}$ leads to a decrease in the performance on
$\mathcal{D}_{<i}$.

  * •

_Negative transfer._ This happens when fine-tuning a model up to
$\mathcal{D}_{\leq i}$ leads to a lower performance on $\mathcal{D}_{i}$ than
training on it alone.

  * •

_Low zero-shot transfer._ This happens when fine-tuning on $\mathcal{D}_{\leq
i}$ gives a lower performance than random on unseen $\mathcal{D}_{>i}$.

  * •

_Low final performance._ This is when fine-tuning on all $\mathcal{D}_{\leq
N}$ gives an uneven performance between languages when tested on
$\mathcal{D}_{\leq N}$ at the end of training.

###  2.4 Downstream Tasks and Datastreams

Here, we describe the downstream tasks and multi-lingual sequential
datastreams used.

#### Downstream Tasks.

We choose task-oriented dialogue parsing as a use case and consider the multi-
lingual task-oriented parsing (MTOP) benchmark Li et al. (2021). Task-oriented
dialogue parsing provides a rich testbed for analysis, as it encompasses two
subtasks: intent classification and slot filling, thus allowing us to test
different task capabilities in cross-lingual continual learning.

#### Datastream Construction.

For a set of $N$ languages $\mathscr{L}$, our study considers a permutation
subset $P\subset\mathfrak{S}(\mathscr{L})$ with the following
properties:333Details of the different language permutations used for the
datastreams can be found in Appendix C.1.

  * •

$|P|=|\mathscr{L}|=N$, i.e. $P$ consists of $N$ permutations, each of which is
a sequence of $N$ datasets in each of the $N$ languages in $\mathscr{L}$.

  * •

$\forall\boldsymbol{\ell}\in\mathscr{L}$, $\forall j\in 1\ldots N$, there
exists some $p\in P$ such that $p[j]=\boldsymbol{\ell}$.

  * •

_H2L_ $\in P$, the permutation from most high-resource to most low-resource
fine-tuning data sets, based on the training split dataset size.

  * •

_L2H_ $\in P$, the reverse of _H2L_.

In our experiments, we use MTOP Li et al. (2021), which is a multi-lingual
task-oriented dialogue dataset that covers six typologically diverse languages
and spans over 11 domains and 117 intents. We chose MTOP since it is the
largest scale dataset available for task-oriented dialogue, and because it
covers languages that have varying amounts of data resources available. We use
only the flat representation of slots (without nesting) to simplify our
evaluation. We use the original data for most experiments. Table 1 shows a
summary of the number of sentences (dialogue utterances) per language and
split.

Lang | ISO | Train | Dev | Test
---|---|---|---|---
English | EN | 15,667 | 2,235 | 4,386
German | DE | 13,424 | 1,815 | 3,549
French | FR | 11,814 | 1,577 | 3,193
Hindi | HI | 11,330 | 2,012 | 2,789
Spanish | ES | 10,934 | 1,527 | 2,998
Thai | TH | 10,759 | 1,671 | 2,765
Table 1: Number of sentences in MTOP per language and split.

###  2.5 Evaluation Protocols

For each language permutation, we train on each dataset in sequence but
continually evaluate on all languages. Let $R$ be some success metric for
evaluating a downstream task $K{}$ and $R_{i,\leq{j}}$ be the evaluation on
the test set for language $\boldsymbol{\ell}{}_{i}$ fine-tuning $K{}$ on
$\mathcal{D}_{\leq j}$. We define the following _meta-metrics_ (which are
inspired, but slightly different from the metrics in Lopez-Paz and Ranzato
(2017) and Chaudhry et al. (2019a)):

  * •

Forgetting (F $\downarrow$). This is the average forgetting _over all
datasets_ (excluding the first dataset) computed as:

$\displaystyle\begin{split}F=\frac{1}{N-1}\sum_{j=2}^{N}{F_{\leq{j}}},\\\
F_{\leq{j}}=\frac{1}{j-1}\sum_{i=1}^{j-1}{F_{i,\leq{j}}},\end{split}$ (1)

where $F_{\leq{j}}$ is the average forgetting that occurred at the point of
training $\mathcal{D}_{j}$. We compute
$F_{i,\leq{j}}=\max_{k\in\left[1,j-1\right]}R_{i,\leq{k}}-R_{i,\leq{j}}$.
$F_{i,\leq{j}}$ is the degree to which performance on $\mathcal{D}_{i}$ has
suffered by continuing to train on $\mathcal{D}_{\leq j}$ instead of stopping
before covering $\mathcal{D}_{j}$.

  * •

Transfer (T $\uparrow$). This is the average forward transfer computed as:

$\displaystyle\begin{split}T=\frac{1}{N-1}\sum_{i=2}^{N}{T_{i}},\\\
T_{i}=R_{i,\leq{i}}-R_{i},\end{split}$ (2)

where $R_{i}$ denotes evaluation of a model fine-tuned only on
$\mathcal{D}_{i}$. Then, $T_{i}$ is the incremental impact of sequential
training on datasets prior to seeing $\mathcal{D}_{i}$. To measure
_generalization to new languages_ , we add a zero-shot transfer
($\textbf{T}^{\textbf{0}}$ $\uparrow$) metric measured as:

$\displaystyle\begin{split}T^{0}=\frac{1}{N-1}\sum_{i=2}^{N}{T^{0}_{i}},\\\
T^{0}_{i}=\frac{1}{i-1}\sum_{j=1}^{i-1}R_{i,\leq{j}}-R^{0}_{i},\end{split}$
(3)

where $T^{0}_{i}$ is the average performance of a model on the forward
transfer to a language $\boldsymbol{\ell}{}_{i}$ after training on
$\mathcal{D}_{<i}$ compared to the random performance $R^{0}_{i}$ before even
fine-tuning on any language (i.e. using fixed pre-trained M-BERT weights and
randomly initialized weights for the output layer).

  * •

Final performance (FP $\uparrow$). This is the average performance after
training on all datasets in the studied stream, computed as:

$FP=\frac{1}{N}\sum_{i=1}^{N}{R_{i,\leq{N}}}.$ (4)

##  3 Methods

Figure 2: A comparison between different variants of model expansion for this
problem setting: either at the side of the input (_Lang-Spec Trans_), the
output (_Lang-Spec Task_), or using adapters (_Lang-Spec Ada_).

For our base model, we use the same M-BERT-based architecture as was used in
Castellucci et al. (2019) and M’hamdi et al. (2021) to jointly learn the
intent classification and slot filling subtasks of MTOP.444More details about
the base model can be found in Appendix B.1. On top of that, we define
baselines, non-continual learning reference models, and continual learning
algorithms.

###  3.1 Baseline & Reference Models

Before delving into continual learning approaches, we consider a simple lower-
bound baseline. In addition to that, we design reference models either trained
from scratch for each new language, in a joint manner, or in a sequential
multi-hop manner. Those are upper-bound non-continual learning models that are
used to assess the performance of different models trained with continual
learning methodologies. Those reference models can be in general superior to
continual learning models but can also be less efficient and not feasible. For
a fair comparison, all models use the same base model architecture and its
loss with no further additions or special optimizations to the architecture.

#### Lower-bound Baseline.

This consists of naive sequential fine-tuning (_Naive Seq FT_), which
sequentially fine-tunes with no continual learning.

#### Non-continual Learning Upper-bound Models.

These are stronger upper-bound models used as reference points of performance.
However, they are either not efficient or prohibitive in the context of cross-
lingual continual learning. Some of them require training from scratch for
each language which is not efficient. Others require having access to all
languages either at the same time or incrementally. Having such access can be
restrictive due to privacy or storage efficiency concerns.

  * •

Language-specific fine-tuning (_Lang-Spec FT_). This trains independent models
on the data set for each language $\boldsymbol{\ell}{}_{i}$ using only
$\mathcal{D}_{i}$.

  * •

Multi-lingual learning (_Multilingual_). This trains one single model jointly
across all data sets $\mathcal{D}_{\leq N}$.

  * •

Incremental joint learning (_Inc Joint_). This incrementally trains adding the
data set for each language in the stream. This consists of the following hops:
1) $\mathcal{D}_{\leq 1}$, 2) $\mathcal{D}_{\leq 2}$, $\cdots$, and N-1)
$\mathcal{D}_{\leq N-1}$. This is the only sequential reference model.

###  3.2 Continual Learning Approaches

To continually fine-tune on different languages, we establish a representative
set of strong approaches555More details about the approaches and their hyper-
parameters can be found in Appendix B and C.2, respectively. spanning the
following categories inspired by previous evaluation paradigms such as Jin et
al. (2022) lifelong language model domain-incremental pertaining. To the best
of our knowledge, we are the first to exhaustively investigate such approaches
for the context of cross-lingual continual learning, whereas different
approaches were investigated separately for different problem definitions.

#### Model Expansion.

We consider the following approaches that add hop-specific parameters, as
shown in Figure 2. We expand on either the input (i.e. M-BERT representations)
or the output side (i.e. task-specific prediction heads). For the former
(_Lang-Spec Trans_), the transformer layers are replicated for each hop while
sharing the prediction heads. To expand on the output side (_Lang-Spec Task_),
we use different prediction heads across hops but share the M-BERT layers. We
additionally consider _Lang-Spec Enc[1-9]_ , which trains M-BERT encoder
layers $\in 1\ldots 9$ in a language-specific manner while sharing the rest.
We also separately add MAD-X adapters Pfeiffer et al. (2020b). We either fine-
tune the adapter layers and freeze the rest of M-BERT (_Lang-Spec Ada(F)_) or
tune them both (_Lang-Spec Ada(T)_).666More details on adapters and how zero-
shot evaluation works for model expansion approaches are in Appendix B.2.

#### Regularization.

We focus on elastic weight consolidation (_EWC_) Kirkpatrick et al. (2017),
which mitigates catastrophic forgetting by reducing the changes in parameters
that are deemed critical to previously seen languages. We use the online
version of EWC (_EWC-Online_) for efficiency.

#### Memory Replay.

We use experience replay (_ER_) Chaudhry et al. (2019b), which alleviates
forgetting by maintaining a fixed-size memory equally balanced between the
different languages and regularly drawing examples from the memory to replay.

#### Distillation-based.

On top of ER, we distill dark knowledge Kariya (2018) from previous model
checkpoints. We explore two variants: logit distillation (_KD-Logit_) Hinton
et al. (2015) and representation distillation (_KD-Rep_) Aguilar et al.
(2020), which optimize the minimum squared error loss on either the output
logits or M-BERT representations between the current and previous models.

##  4 Results & Analysis

In this section, we provide an extensive analysis in the form of different
ablation studies. We ask critical analysis questions that revolve around the
continual learning goals described in §2.2. For §4.2, scores are reported
using accuracy (Acc) and F1-score (F1) for intent classification and slot
filling, respectively. For the remaining sections, all results are reported
for intent classification only, slot-filling results, for which the same
trends are observed, can be found in Appendix D. Bootstrap sampling (over test
data shuffling) is used to compute the average and 95% confidence intervals
(averaged over all language permutations except for §4.4). More details can be
found in Appendix C.3. We also separately repeat key experiments over 3
different seeds and obtain similar findings, which can be found in Appendix E.
We decided to report the results using bootstrap sampling since they have
tighter confidence intervals.

###  4.1 How is a Multi-Hop Analysis Different from its One-Hop Counterpart?

To motivate our cross-lingual continual learning evaluation paradigm, we start
by investigating how a multi-hop analysis is different from a conventional
one-hop transfer learning analysis. Figure 3 shows a comparison between the
two in terms of forgetting (Eq. 1) for different approaches aggregated over
different language permutations. More results for slot filling and other
metrics can be found in Figure 9 in Appendix D.5.

Figure 3:  Comparison between forgetting trends for intent classification
using one-hop (crossed boxplots on the left) and multi-hop analysis (dotted
boxplots on the right), showing the variance over different language
permutations. One-hop analysis exhibits higher variance than its multi-hop
counterpart.

_Lang-Spec Trans_ tends to have the least forgetting and _Naive Seq FT_ the
most, but importantly the variance for the multi-hop analysis is much smaller
than that for the one-hop analysis. Having larger confidence intervals, the
one-hop analysis also tends to be misleading in the sense that certain models
are depicted as having a good performance, while it is not truly the case. For
example, _Naive Seq FT_ , according to the one-hop analysis, shows a range of
forgetting from very little (0.5) to a lot (2.0). So in some circumstances, it
has little forgetting, thus a good performance under the one-hop analysis. But
according to the multi-hop analysis, it clearly has a lot of forgetting with
more confidence. Therefore, the multi-hop analysis leads to a more conclusive
analysis. We conjecture that averaging over more hops and balanced diversified
datastreams is what leads to narrower confidence intervals. This agrees with
the well-known fact that larger sample sizes lead to narrower confidence
intervals Hazra (2017).

###  4.2 Can a Multi-lingual Language Model Learn to Preserve and Accumulate
Knowledge across Different Languages?

Given the conclusiveness of the multi-hop analysis in §4.1, we follow that
type of analysis thereafter. In this section, we investigate how well the
baseline and different non-continual learning reference models learn to
preserve and accumulate knowledge across different languages by looking at the
average over language permutations. Since not all reference models are
sequential, we start by comparing them to the baseline using their final
performances (Eq. 4). The final performance is indicative of how well a single
final model can encapsulate the knowledge across all languages at the end of
training. From Table 2, we notice that _Naive Seq FT_ and _Multilingual_ have
the worst and best final performances, respectively. This suggests that a
multilingual joint model is more beneficial than sequential models. In
practical scenarios, however, we may not have access to all languages at the
same time. Among non-continual learning reference models, _Inc Joint_ is
closest to _Multilingual_ if all data may be preserved. However, this may also
not be the case. In that case, _Inc Joint_ is nearly as good. Training
incrementally and sequentially (_Inc Joint_) is also more beneficial than
fine-tuning on just the language of interest (_Lang-Spec FT_), as the former
exploits cross-lingual transfer capabilities.

Model | Intent Class (Acc) | Slot Filling (F1)
---|---|---
_Naive Seq FT_ | 91.06 $\pm 1.08$ | 69.37 $\pm 1.06$
_Lang-Spec FT_ | 93.40 $\pm 0.08$ | 73.90 $\pm 0.83$
_Inc Joint_ | 94.16 $\pm 0.18$ | 74.88 $\pm 0.38$
_Multilingual_ | 94.25 $\pm 0.07$ | 76.34 $\pm 0.82$

Table 2:  The average final performance across different language permutations
for the baseline compared to reference models. We highlight the best scores in
bold and underline the second best across models.

We focus, thereafter, on _Inc Joint_ 777We do not use _Multilingual_ since it
is non-sequential. Metrics like forgetting are thus always zero, which makes
this model not comparable with other continual learning approaches and
sequential reference models. and compare its forgetting (Eq. 1) and transfer
(Eq. 2) trends to the baseline _Naive Seq FT_ , as shown in Table 3. _Inc
Joint_ exhibits significantly less forgetting, which also causes its final
performance to be higher than _Naive Seq FT_. This suggests that recalling
previously used training data is helpful in knowledge preservation. However,
_Naive Seq FT_ seems to slightly outperform _Inc Joint_ in terms of transfer.
This difference is not statistically significant.888We report the p-values
from pairwise Tukey’s HSD analysis to gain a reliable unified view that
individual t-tests may fail to convey. More explanation can be found in
Appendix C.3. We hypothesize that this could be due to exposing _Inc Joint_ to
all resources from previously seen languages, so it is likely that the data
distribution between all these languages may distract the model from learning
on the new one.

Model | Intent Class (Acc) | Slot Filling(F1)
---|---|---
F $\downarrow$ | T $\uparrow$ | F $\downarrow$ | T $\uparrow$
_Naive Seq FT_ | 2.93 $\pm 1.24$ | 0.68 $\pm 0.14$ | 5.67 $\pm 0.93$ | 1.37 $\pm 0.53$
_Inc Joint_ | 0.11 $\pm 0.10$ | 0.52 $\pm 0.19$ | 0.91 $\pm 0.34$ | 0.83 $\pm 0.77$

Table 3:  Forgetting (F) and transfer (T) performance averaged across
different language permutations for _sequential baseline and reference
models_. We highlight the best models in bold for each subtask and metric.
Figure 4: Correlations between different pairs of metrics: (a) Final
performance versus negative forgetting for the task of intent classification.
The lower the forgetting, the higher the final performance. (b) Final
performance versus transfer for the task of intent classification. As
hypothesized, there is no direct correlation between final performance and
transfer. (c) Transfer versus negative forgetting for the intent
classification task. In general, there is no direct correlation between
transfer and forgetting. (d) Zero-shot generalization versus negative
forgetting for intent classification. Model expansion approaches are
highlighted in shades of green. We zoom over the rest of the models in the
main graph and show an overview of all approaches in the lower right corner
subplot. Mitigating forgetting leads to higher generalization, with the
exception of multi-headed models highlighted in green.

###  4.3 Is Continual Learning Effective in Boosting Knowledge Preservation,
Accumulation, and Model Utility?

To study the effectiveness of continual learning approaches, we compare them
to the baseline using the average over language permutations. We show, in
Figures 4(a) and 4(c), the final performances (Eq. 4) and transfer (Eq. 2) of
different approaches, respectively, versus their negative forgetting (Eq. 1).
In general, we observe that continual learning approaches mitigate forgetting
and improve final performance. They also improve transfer, to some degree,
though gains are mostly not significant compared to _Naive Seq FT_(Appendix
C.3).

From Figure 4(a), we notice that model expansion approaches999We include a
full analysis of the expansion over several subsets of M-BERT components in
Appendix D.2.(_Lang-Spec Trans_ and _Lang-Spec Enc[1-9]_ described previously)
are good at mitigating forgetting and improving the final performance while
_Lang-Spec Task_ is not. M-BERT, when trained in a language-specific manner,
is responsible for encapsulating the cross-lingual representations necessary
for enabling knowledge preservation, whereas changes to the downstream task-
specific layers do not make much of a difference. This implies that in cross-
lingual continual learning more attention should be paid to how to train those
representations in a language-specific manner efficiently. _Lang-Spec Ada(T)_
is one way to do this more efficiently, but its performance still lags behind
other model expansion approaches. _ER_ achieves a performance close to _Lang-
Spec Trans_ and _Lang-Spec Enc[1-9]_ and this suggests that using a portion of
the memory is beneficial.101010An ablation study using different sizes of the
memory is shown in Appendix D.6. It shows that even smaller sizes up to 5% are
still beneficial. We report here the highest memory size as it leads to the
best results.

In the baseline approach, which suffers from the highest forgetting, we also
notice the lowest final performance and transfer in Figures 4(a) and 4(c). As
continual learning approaches reduce forgetting, they also improve the final
performance, and some of them also improve transfer but not to the same
degree. This suggests that the lower the forgetting a model can achieve, the
easier it gets for it to learn a stronger final model. However, there is no
direct correlation between forgetting and transfer. For example, _Lang-Spec
Trans_ is the best model in reducing forgetting but also the worst in terms of
transfer. This could be due to the fact that _Lang-Spec Trans_ exhibits
similar behavior to _Lang-Spec FT_ thus, the transfer of a model, which is the
difference between the performance of that model and that of _Lang-Spec FT_ ,
is almost null. On the other hand, although _Lang-Spec Ada(F)_ has the highest
transfer, it has the lowest final performance and is close to average
forgetting. Although the adapter will not be updated anymore after the model
has been fine-tuned on, we think that the forgetting could be due to the
shared task-specific layer leading to a forgetting closer to Lang-Spec Trans
more than Lang-Spec Ada(T), which also shares M-BERT and tunes it. We show in
Figure 4(b) that there is no direct correlation between final performance and
transfer. This posits that all three metrics need to be studied independently
for a more insightful analysis.

###  4.4 Which Permutations Impose More Challenges on Knowledge Preservation,
Accumulation, and Model Utility?

So far, our analysis has focused on the average over language permutations,
but are the same patterns observed for different language permutations? To
shed light on this, we analyze the performance of different continual learning
algorithms and the baseline in terms of their forgetting (Eq. 1), transfer
(Eq. 2), and final performance (Eq. 4) over _H2L_ and _L2H_ permutations, in
Table 4.111111Full results for slot filling, more language permutations, and a
balanced version of data can be found in Appendix D.3. In general, we observe
that it is more challenging to learn from low to high resource languages.
However, model expansion and memory replay approaches reduce forgetting and
final performance gaps between language permutations. We hypothesize that
_L2H_ being more challenging than _H2L_ could be due to the fine-tuning data
size that is different between languages.

Model | F $\downarrow$ | T $\uparrow$ | FP $\uparrow$
---|---|---|---
_H2L_ | _L2H_ | _H2L_ | _L2H_ | _H2L_ | _L2H_
_Naive Seq FT_ | 1.52 | 5.52 | 0.93 | 0.57 | 92.06 | 88.80
_Lang-Spec Trans_ | 0.40 | 0.62 | 0.59 | 0.03 | 93.86 | 93.37
_Lang-Spec Enc[1-9]_ | 0.60 | 1.05 | 1.00 | 0.63 | 93.75 | 93.15
_Lang-Spec Task_ | 1.53 | 5.53 | 0.84 | 0.38 | 91.93 | 87.68
_Lang-Spec Ada(T)_ | 1.18 | 4.43 | 1.29 | 0.79 | 92.36 | 88.66
_Lang-Spec Ada(F)_ | 0.84 | 1.87 | 3.41 | 2.43 | 91.08 | 89.92
_EWC_ | 1.82 | 5.90 | 0.74 | 0.48 | 91.16 | 88.28
_ER_ | 0.71 | 2.35 | 0.95 | 0.78 | 93.51 | 92.58
_KD-Logit_ | 1.42 | 4.07 | 0.77 | 0.51 | 91.60 | 89.65
_KD-Rep_ | 1.49 | 4.00 | 0.96 | 0.53 | 91.64 | 90.17

Table 4:  Comparison of intent classification for two language permutations.
We highlight in bold the best forgetting (F), highest transfer (T), and final
performance (FP) of accuracy scores among _H2L_ and _L2H_ , whereas the best
and second best scores across approaches for _H2L_ and _L2H_ separately are
underlined and italicized, respectively. We report mean performances for each
metric and language order. All 95% confidence intervals range from $\pm$ 0.01
to $\pm$ 0.04.

To verify this hypothesis, we dig deeper to check if the differences among
fine-tuning data sizes between languages are the main factor by performing an
ablation study on that. Therefore, we use the same amount of fine-tuning and
evaluation resources for each language (9,219 for train, 1,285 for Dev, and
2,299 for test splits) and report the results on _Naive Seq FT_ in Table 5. We
notice that there is still a gap between these two language permutations for
forgetting and final performance. This suggests that the difference in fine-
tuning data size is not what accounts for the differences between the two
language permutations. There are perhaps biases in the pre-training or other
linguistic artifacts that need to be studied in future work.

Model | F $\downarrow$ | T $\uparrow$ | FP $\uparrow$
---|---|---|---
_H2L_ | _L2H_ | _H2L_ | _L2H_ | _H2L_ | _L2H_
Original Data | 1.52 | 5.52 | 0.93 | 0.57 | 92.06 | 88.80
Balanced Data | 1.25 | 5.81 | 0.89 | 0.75 | 89.33 | 85.81

Table 5:  Performance on intent classification comparison between two versions
of the data: original data version and balanced data for _Naive Seq FT_ across
the same permutations as Table 4. We bold the best among _H2L_ and _L2H_ for
each metric.

###  4.5 How do Continual Learning Models Generalize to Unseen Languages?

To analyze the zero-shot transfer to languages unseen during fine-tuning, we
plot the performance of zero-shot transfer (Eq. 3) as a function of negative
forgetting over the average of different language permutations to investigate
any relationships between generalization and preservation. In Figure 4(d), we
infer that most continual learning approaches do not substantially improve
generalization compared to _Naive Seq FT_. In particular, model expansion
approaches (in red) hurt generalization even if they significantly reduce
forgetting. This zero-shot transfer versus interference trade-off is referred
to as the stability-plasticity dilemma Mermillod et al. (2013), where the
weights responsible for improving on new tasks are often responsible for
forgetting previous tasks. Except for model expansion approaches, we notice
that approaches that reduce forgetting also improve generalization compared to
_Naive Seq FT_. Better approaches to balance between the two can be
investigated in future work.

##  5 Related Work

Continual learning for cross-lingual NLP work is under-explored, either
focusing on proposing cross-lingual approaches that indirectly support
continual learning, such as Artetxe et al. (2020), of the transfer-ability of
monolingual models. Other approaches derive a cross-lingual continual learning
problem directly from cross-lingual transfer learning, such as Garcia et al.
(2021); Pfeiffer et al. (2021); Minixhofer et al. (2022), who propose
different lexical and semantic approaches to adapt to new low-resource
languages for different downstream tasks. Similarly, Liu et al. (2021) explore
continual techniques to fine-tune on downstream applications for new languages
while preserving the original cross-lingual ability of the pre-trained model.
Muller et al. (2021) analyze the adaptability and usability of large language
models to unseen and under-studied low-resource languages. However, they all
focus on a one-hop analysis from high to low-resource language pairs or pre-
training to fine-tuning tasks, unlike our work, which analyzes across multiple
hops. More recently, Pfeiffer et al. (2022) propose a new methodology based on
adapters and show that their approach mitigates negative interference between
languages while enabling positive transfer. They use a multi-hop evaluation
paradigm closer to our setup, but they only evaluate with respect to adapters
using interference and transfer and do not analyze other aspects of cross-
lingual continual learning capabilities.

##  6 Conclusion

We formulate the cross-lingual continual learning problem setup. We show that
naive sequential fine-tuning is prone to catastrophic forgetting and has poor
accumulation capabilities sensitive to different language permutations. We
provide the first benchmark to compare the effectiveness of different
continual learning algorithms for the cross-lingual case. We show that
continual learning models improve cross-lingual knowledge preservation, which
also contributes to improving final model performance and, to a lesser degree,
accumulation and generalization. We also discuss the challenges of
sequentially training for certain language permutations. We hope that this
study will encourage more analyses in the same spirit to cover more benchmarks
and datastream setups to gain more insights that go beyond conventional cross-
lingual transfer learning.

## Limitations

#### Application to Other Benchmarks

A central limitation of our work is that the main experiments are based on a
single task-oriented dialogue benchmark. While there are multiple other
natural language understanding benchmarks like XNLI, XQUAD, MLQA, and PAWS-X
Conneau et al. (2018); Artetxe et al. (2020); Lewis et al. (2020); Yang et al.
(2019) that can also be used to back up our claims, we argue that this is
outside the scope of this paper. The main objectives of this paper are to
first come up with a new definition of a cross-lingual continual learning
challenge and then to give an example using a comprehensive and realistic
benchmark like task-oriented dialogue to catalyze more research in that
direction.

#### Choice of Realistic Permutations

For more realistic setups of continual learning, we need to come up with an
approach to define continual learning annotation scenarios of languages.
Rather than using brute force with all possible ways the languages could be
annotated at different stages, a principled way would be more desired. Since
it is hard to tell if there is any logic or pattern in the annotation process
itself and given the sheer amount of realistic scenarios, we chose one
scenario experienced by some of the users: a model is built for a user, then
the user reveals that more languages are desired. We test in our work the
plausibility of continual learning approaches where the sequence moves from
one language to another without repetition of the same language. Working on
scenarios where the data from different languages are integrated as soon as
they are annotated, implying different languages for different hops, is out of
the scope of this paper.

#### Data and Model Size Analysis

In this paper, we pick certain model expansion approach variations to analyze
the effect of model components (one aspect of model size) and two data
distribution scenarios. However, analyzing extensively the effect of the scale
of data and model size is beyond the scope of our work. We agree that
different data sizes can be used, and it is interesting to analyze different
supervision levels, such as using different proportions of the data for each
language and simulating few-shot scenarios. We believe that for low-resource
scenarios, we need to investigate specific approaches to continual learning,
like meta-learning. We plan to investigate that in future work.

#### Application to Other Transformers

Another possible limitation of our work is the restriction of the evaluation
to a base model on top of M-BERT Transformers. With the advent of Transformer-
based encoders as strong pillars for transfer-learning, several Transformers
such as XLM-R have been proposed more recently. Although those models have
been shown to outperform M-BERT on numerous downstream applications,
especially on low-resource languages Conneau et al. (2020), M-BERT is still
largely used due to its reduced number of parameters. In our specific
continual learning challenge, efficiency is a top concern as we are training
in multiple hops and benchmarking on different models. So, M-BERT has been
feasible in our use case. We leave experimenting with other Transformer-based
encoders to future work.

## Acknowledgements

This material is partially based upon work supported in part by the Office of
the Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via Contract No. 2019-19051600007. The views and
conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies, either
expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for governmental
purposes, notwithstanding any copyright annotation therein. We also would like
to extend our thanks to Xisen Yin for insightful discussions on continual
learning and all anonymous reviewers and meta-reviewers for their valuable
feedback.

## References

  * Aguilar et al. (2020) Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. 2020.  Knowledge distillation from internal representations.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7350–7357. AAAI Press.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Biesialska et al. (2020) Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020.  Continual lifelong learning in natural language processing: A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6523–6541, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Castellucci et al. (2019) Giuseppe Castellucci, Valentina Bellomaria, Andrea Favalli, and Raniero Romagnoli. 2019.  Multi-lingual intent detection and slot filling in a joint bert-based model.  _ArXiv preprint_ , abs/1907.02884.
  * Chaudhry et al. (2019a) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019a.  Efficient lifelong learning with A-GEM.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Chaudhry et al. (2019b) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. 2019b.  Continual learning with tiny episodic memories.  _ArXiv preprint_ , abs/1902.10486.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * de Masson d’Autume et al. (2019) Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019.  Episodic memory in lifelong language learning.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13122–13131.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Garcia et al. (2021) Xavier Garcia, Noah Constant, Ankur Parikh, and Orhan Firat. 2021.  Towards continual learning for multilingual machine translation via vocabulary substitution.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1184–1192, Online. Association for Computational Linguistics.
  * Han et al. (2020) Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Jie Zhou, and Maosong Sun. 2020.  More data, more relations, more context and more openness: A review and outlook for relation extraction.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 745–758, Suzhou, China. Association for Computational Linguistics.
  * Hazra (2017) Avijit Hazra. 2017.  Using the confidence interval confidently.  _Journal of Thoracic Disease_ , 9(10):4124–4129.
  * Hinton et al. (2015) Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015.  Distilling the knowledge in a neural network.  _ArXiv preprint_ , abs/1503.02531.
  * Jin et al. (2022) Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022.  Lifelong pretraining: Continually adapting language models to emerging corpora.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4764–4780, Seattle, United States. Association for Computational Linguistics.
  * Joshi et al. (2020) Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020.  The state and fate of linguistic diversity and inclusion in the NLP world.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6282–6293, Online. Association for Computational Linguistics.
  * Kariya (2018) Mahendra Kariya. 2018.  Dark knowledge in neural networks.  Accessed on March 7th, 2023.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13):3521–3526.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Li et al. (2021) Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 2021.  MTOP: A comprehensive multilingual task-oriented semantic parsing benchmark.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2950–2962, Online. Association for Computational Linguistics.
  * Li and Hoiem (2016) Zhizhong Li and Derek Hoiem. 2016.  Learning without forgetting.  In _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV_ , volume 9908 of _Lecture Notes in Computer Science_ , pages 614–629. Springer.
  * Liu et al. (2021) Zihan Liu, Genta Indra Winata, Andrea Madotto, and Pascale Fung. 2021.  Preserving cross-linguality of pre-trained models via continual learning.  In _Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)_ , pages 64–71, Online. Association for Computational Linguistics.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 6467–6476.
  * Madotto et al. (2021) Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, and Zhiguang Wang. 2021.  Continual learning in task-oriented dialogue systems.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7452–7467, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J. Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Gordon H. Bower, editor, _Psychology of Learning and Motivation_ , volume 24, pages 109–165. Academic Press.
  * Mermillod et al. (2013) Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. 2013.  The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects.  _Frontiers in psychology_ , 4:504.
  * M’hamdi et al. (2021) Meryem M’hamdi, Doo Soon Kim, Franck Dernoncourt, Trung Bui, Xiang Ren, and Jonathan May. 2021.  X-METRA-ADA: Cross-lingual meta-transfer learning adaptation to natural language understanding and question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3617–3632, Online. Association for Computational Linguistics.
  * Minixhofer et al. (2022) Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022.  WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3992–4006, Seattle, United States. Association for Computational Linguistics.
  * Muller et al. (2021) Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021.  When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 448–462, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2022) Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022.  Lifting the curse of multilinguality by pre-training modular transformers.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3479–3495, Seattle, United States. Association for Computational Linguistics.
  * Pfeiffer et al. (2020a) Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a.  AdapterHub: A framework for adapting transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 46–54, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2020b) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020b.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673, Online. Association for Computational Linguistics.
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021.  UNKs everywhere: Adapting multilingual language models to new scripts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10186–10203, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ritter et al. (2018) Hippolyt Ritter, Aleksandar Botev, and David Barber. 2018.  Online structured laplace approximations for overcoming catastrophic forgetting.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 3742–3752.
  * Ruder et al. (2019) Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019.  Transfer learning in natural language processing.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials_ , pages 15–18, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017.  Continual learning with deep generative replay.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 2990–2999.
  * Sun et al. (2020) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2020.  LAMOL: language modeling for lifelong language learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * van de Ven et al. (2022) Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. 2022.  Three types of incremental learning.  _Nature Machine Intelligence_ , 4(12):1185–1197.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Yang et al. (2019) Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019.  PAWS-X: A cross-lingual adversarial dataset for paraphrase identification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.
  * Zenke et al. (2017) Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017.  Continual learning through synaptic intelligence.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3987–3995. PMLR.

##  Appendix A More Related Work

Continual learning approaches have found favor, especially among the computer
vision community, including regularization-based Kirkpatrick et al. (2017);
Zenke et al. (2017); Li and Hoiem (2016); Ritter et al. (2018) and memory-
based approaches Shin et al. (2017); Chaudhry et al. (2019b, a). Only
recently, continual learning has started gaining more interest in the NLP
community. Most efforts on continual learning for NLP have focused on
classification tasks and fall into the category of domain or class incremental
continual learning Han et al. (2020). Current approaches often fail to
effectively retain previous knowledge and adapt to new information
simultaneously Biesialska et al. (2020); de Masson d’Autume et al. (2019). New
challenges are formulated to study the problem of continual learning from
different perspectives. Jin et al. (2022) formulate the lifelong learning
pretraining challenge, where pertaining language models continually adapt to
emerging data from new corpora.

Continual learning for cross-lingual NLP is under-explored, either focusing on
proposing cross-lingual approaches that indirectly support continual learning,
such as Artetxe et al. (2020), on the transfer-ability of monolingual models.
Other approaches derive a cross-lingual continual learning problem directly
from cross-lingual transfer learning, such as Garcia et al. (2021), who
propose a lexical approach to adapt to new low-resource languages for machine
translation. Similarly, Pfeiffer et al. (2021) propose lexical-level
adaptation schemes that can be applied to models relying on subword-based
tokenization to adapt them to low-resource languages not covered or whose
scripts are unseen during pre-training. Minixhofer et al. (2022) also propose
adaptations that go beyond the lexical level. Their approach facilitates the
creation of monolingual language models that are transferable to new
languages. Liu et al. (2021) explore continual techniques to fine-tune on
downstream applications for new languages while preserving the original cross-
lingual ability of the pre-trained model. However, they all focus on a one-hop
analysis from high to low-resource language pairs or pre-training to fine-
tuning tasks, unlike our work, which analyzes across multiple hops. Muller et
al. (2021) analyze the adaptability and usability of large language models to
unseen and under-studied low-resource languages. Based on that and depending
on the degree of additional pre-training and fine-tuning required, they
categorize the low-resource languages into easy, intermediate, and hard.
Although this work paves the way for a better understanding of the mechanics
of transferability to low-resource scenarios, they do not study the scenario
where the transferability needs to be performed in multiple hops following a
sequential stream of data. More recently, Pfeiffer et al. (2022) propose a new
methodology for language-specific modules to add additional capacity and deal
with the curse of multilinguality and show that their approach mitigates
negative interference between languages while enabling positive transfer. They
use a continual learning multi-hop evaluation paradigm which is closer to our
setup, but they only evaluate using interference and transfer and only using
one approach based on adapters and do not analyze other aspects of cross-
lingual continual learning capabilities using a holistic approach like our
work.

##  Appendix B More Details about Approaches

In this Section, we provide more details on the base model architecture used
(Section B.1) and different continual learning approaches in Sections B.2,
B.3, B.4, and B.5.

###  B.1 Base Model Architecture

We use the same architecture as in Castellucci et al. (2019) and M’hamdi et
al. (2021) to jointly learn intent classification and slot filling subtasks.
As shown in Figure 5, we leverage features from Transformer Vaswani et al.
(2017) encoder and add classification prediction heads on top of it. More
specifically, a multi-lingual pre-trained model is used to encode the input.
Then, to predict the intent and slot spans, we add task-specific prediction
heads. For intent prediction, this takes the form of a linear layer plus
softmax on top of the $[CLS]$ token representation. For slot filling, we use a
sequence labeling layer in the form of a linear layer plus CRF, respectively.
We use the sum of both intent and CRF-based slot losses to optimize the model
parameters.

Figure 5:  Architecture of task-oriented dialogue base model.

###  B.2 Model Expansion

Model expansion methods, such as _Lang-Spec Trans_ and _Lang-Spec Enc[1-9]_ ,
are fine-tuned for each language with either an entirely or partially
language-specific M-BERT (whole 12 layers in addition to the embeddings or
just the top 8 layers in the case of _Lang-Spec Trans_ and _Lang-Spec
Enc[1-9]_ respectively). When fine-tuning them on a new language, the
previously tuned parameters on the old languages are retained unchanged while
the rest of the parameters that are not language-specific are fine-tuned.
During the evaluation on a particular language, the tuned parameters for that
language are restored and used if the language has been seen in training.
Otherwise, the parameters initialized from M-BERT (before fine-tuning on any
language) are used for zero-shot evaluation.

Adapters consist of downsampling layers followed by upsampling layers inserted
between layers of our Transformer encoder in addition to their invertible
components. We do not add task-specific adapters, as according to our ablation
studies, they didn’t prove beneficial. We add adapter components to every
encoder layer following MAD-X configuration and using their pre-trained
weights.121212obtained from AdapterHub Pfeiffer et al. (2020a)
https://adapterhub.ml/explore/text_lang/ We either fine-tune the weights for
the languages available in AdapterHub or train from scratch for languages for
which there are no pre-training adapter weights. At inference time, we use
adapter layers fine-tuned independently for each language in the datastream.

###  B.3 Online Elastic Weight Consolidation (EWC-Online)

To penalize changes in the parameters crucial to previous languages, we use
EWC, which adds a regularization term to the loss applied only after the first
data set $\mathcal{D}_{i}$ in the language stream is seen. $\forall i\in
2\ldots N$, we compute the total loss as follows:

$\mathcal{L}^{i}_{total}=\mathcal{L}^{i}_{cur}+\lambda\mathcal{L}^{i}_{reg},$
(5)

where $\mathcal{L}_{cur}$ is the usual loss of the downstream task on the
current data $\mathcal{D}_{i}$ and $\mathcal{L}_{reg}$ is the regularization
term, and $\lambda$ is a hyperparameter to control the regularization strength
(which is fixed to 20). For efficiency purposes, we use the online version of
EWC (_EWC-Online_). Following that, our regularization term is computed as,
based on the formulation in van de Ven et al. (2022):

$\mathcal{L}^{i}_{reg}=\sum^{N_{p}}_{j=1}{\tilde{F}^{(i-1)}_{jj}(\theta_{j}-\theta^{k}_{j})^{2}},$
(6)

where $\theta$ are the parameters of the Transformer model in addition to the
downstream prediction heads, $N_{p}$ is the total number of parameters, and
$\tilde{F}^{(i-1)}_{jj}$ is the Fisher information matrix on the last language
just before training on $\mathcal{D}_{i}$. This is computed as the running sum
of the $i^{th}$ diagonal elements of the Fisher Information matrices of
$\mathcal{D}_{j}$, for all $j\in 1\ldots(i-1)$.
$\tilde{F}^{(i)}_{jj}=\gamma\tilde{F}^{(i-1)}_{jj}+F^{i}_{jj}$ and
$\tilde{F}^{1}_{jj}=F^{1}_{jj}$. In practice, $F^{i}$ is simply the gradients
of all parameters flattened into one single matrix.

###  B.4 Experience Replay (ER)

After training for each $\mathcal{D}_{i}$ for all $i\in 1\ldots N$, we
populate the memory with randomly sampled examples from $\mathcal{D}_{i}$. For
each $\mathcal{D}_{i}$ for all $i\in 2\ldots N$, after training for every
$k=100$ mini-batches and optimizing for the current loss separately, the model
randomly samples an equal batch from the memory for each $\mathcal{D}_{j}$
such that $j\in 1\ldots(i-1)$ and replays them using the current model
checkpoint used for training on $\mathcal{D}_{i}$. We retrieve an equal amount
of memory items from each language and at each step and hop. The loss from the
current $\mathcal{D}_{i}$ and the loss on the memory on the $\mathcal{D}_{j}$
are interleaved as the replay on the memory only happens every $k$ steps. This
prioritization of the current language helps make the training more stable
without over-fitting on the small memory from previous languages.

###  B.5 Knowledge Distillation (KD-Logit & KD-Rep)

We use the same strategy explained in §B.4 to select the memory to be replayed
using a knowledge distillation loss. For each $\mathcal{D}_{i}$ for all $i\in
2\ldots N$, after training for every $k=100$ mini-batches, we randomly sample
an equal batch from the memory for each $\mathcal{D}_{j}$ such that $j\in
1\ldots(i-1)$. We also load the model checkpoints for each $hop_{j}$ and use
that model and the memory for $\mathcal{D}_{j}$ to compute either the intent
and slot logits in the case of _KD-Logit_ or the multilingual representations
of M-BERT in the case of _KD-Rep_. We do the same thing using the current
model checkpoint this time. Then, we use the minimum square error loss to
minimize the distance between the intent logits obtained using the previous
and current model checkpoints and do the same thing for slot logits for _KD-
Logit_. Then, we take the same over intent and slot distillation losses across
different languages retrieved from the memory. The same is done for computing
the distillation loss over the multilingual representations in _KD-Rep_.

##  Appendix C Experimental Setup Details

In this Section, we describe the datastreams used (Section C.1) and provide
more implementation details (Section C.2), in addition to bootstrap sampling
and statistical significance specifics (Section C.3).

###  C.1 Datastreams

We use the following datastreams for all our experiments as summarized in
Table 6. The MTOP dataset has been released by Facebook Li et al. (2021) under
Creative Commons Attribution-ShareAlike 4.0 International Public License.

Order 1 | Order 2 | Order 3 | Order 4 | Order 5 | Order 6
---|---|---|---|---|---
English | Thai | Spanish | French | Hindi | German
German | Spanish | Hindi | Thai | English | French
French | Hindi | English | German | Spanish | Thai
Hindi | French | German | English | Thai | Spanish
Spanish | German | Thai | Hindi | French | English
Thai | English | French | Spanish | German | Hindi

Table 6: Simulated language permutations.

###  C.2 Implementation Details

For all experiments, we use M-BERT(bert-base-multilingual-
cased)131313github.com/huggingface/transformers version 3.4.0 pre-trained on
104 languages, including all languages evaluated on in this paper. with 12
layers as our pre-trained Transformer model. We use the dev set to pick the
hyperparameters of the optimizer to be used. We perform a manual search for
the most optimal learning rate over a range $[1e-4$, $3e-4$, $1e-5$, $3e-5]$
for Adam optimizer Kingma and Ba (2015) and finally based on the dev
performance we fix the learning rate to $3e-5$ for all experiments for a fair
comparison. We use $\epsilon=1e-8$, $\beta_{1}=0.9$, $\beta_{2}=0.99$, batch
size of 16, $\gamma=0.1$ for EWC Online, $6000$ memory size for ER and
knowledge distillation. For all experiments, we run for $10$ epochs maximum
and pick the best model based on dev data. We also fix a seed of 42 for the
random initialization of Numpy, random, and torch over all bootstrap
experiments. For additional experiments using multiple seeds, we fix three
seeds. All experiments are run using the same computing infrastructure Pytorch
version 1.7.1, using _one_ Tesla P100 GPU of $16280$ MiB of memory CUDA
version 11.2.

The runtime and the number of parameters depend on the approach and the mode
of training used as shown in Table 7. With the exception of model expansion
and language-specific approaches, all approaches have the same number of
parameters coming from the sum of M-BERT and prediction head parameters.
_Lang-Spec Trans_ has the highest number of parameters, which is six times
more than _Naive Seq FT_ but only requires two times more runtime as only one
$\frac{1}{6}$ part of language-specific M-BERT is updated at each hop for each
whereas the rest is used in evaluation mode only. _Lang-Spec Ada(F)_ has the
smallest number of parameters which is around 24% and takes 2 times less than
the usual runtime of _Naive Seq FT_ (while exhibiting lower forgetting and
higher transfer than _Naive Seq FT_ , as shown in Table 8). Memory replay and
knowledge distillation approaches have more runtime (slightly more than _Lang-
Spec Trans_) as they store and handle memory and compute the replay or
distillation losses interleaved with the main loss, which makes them time-
consuming. What impacts the runtime of ER is much more than just iterating
over a small sampled memory. Its runtime does not only depend on the size of
the memory as much as it depends on the frequency of interleaving happening at
the fine-tuning schedule. After each k minibatch steps, we sample a minibatch
from the memory and fine-tune on it interleaved with the fine-tuning on the
main minibatch. So, that makes the runtime depend on k and not only the size
of the memory. This make its training more time-consuming than if we had to
sample only after each epoch with the same memory size.

Model | Runtime | # Param
---|---|---
_Naive Seq FT_ | 3h16min | 178,081,402
_Lang-Spec FT_ | 5h02min | 1,068,488,412
_Inc Joint_ | 1d22h51min | 178,081,402
_Multilingual_ | 16h45min | 178,081,402
_Lang-Spec Embed_ | 7h46min | 639,123,322
_Lang-Spec Enc[1-3]_ | 7h52min | 284,399,482
_Lang-Spec Enc[4-6]_ | 7h12min | 284,399,482
_Lang-Spec Enc[7-9]_ | 7h8min | 284,399,482
_Lang-Spec Enc[10-12]_ | 7h20min | 284,399,482
_Lang-Spec Enc[1-9]_ | 8h1min | 497,035,642
_Lang-Spec Trans_ | 7h15min | 1,067,348,602
_Lang-Spec Enc[1-12]_ | 7h53min | 603,353,722
_Lang-Spec Enc[1-6]_ | 7h16min | 390,717,562
_Lang-Spec Enc[7-12]_ | 7h10min | 390,717,562
_Lang-Spec Task_ | 6h18min | 179,221,212
_Lang-Spec Ada(T)_ | 4h34min | 222,301,402
_Lang-Spec Ada(F)_ | 1h57min | 44,447,962
_EWC-Online_ | 1d3h17min | 178,081,402
_ER_ | 8h55min | 178,081,402
_KD-Logit_ | 7h23min | 178,081,402
_KD-Rep_ | 8h | 178,081,402

Table 7: Runtime and parameters statistics.

###  C.3 Bootstrap Sampling & Statistical Significance

We run all experiments over one fixed seed of 42. We then use bootstrap
sampling Koehn (2004) to compute the mean and confidence intervals for each of
the metrics described in §2.5 over a single approach. For each language
permutation, and for each $R_{i,\leq{j}}$, representing some performance
metric on language $\boldsymbol{\ell}{}_{i}$ after training on
$\mathcal{D}_{\leq j}$, we sample with replacement 600 sentences from the
testing data over 600 iterations. By using this number of iterations and
sampling sentences, we ensure and also double-check that all sentences in the
test set are covered in the evaluation, ensuring a uniform evaluation across
approaches. Let $x$ be the list of results we get for each iteration
independently. Then, we compute the mean and standard deviation $\bar{x}$ and
$std(x)$ respectively and the 95% confidence interval size $CI$ using the
following equation:

$\displaystyle\begin{split}CI=\frac{1.9639\times std(x)}{\sqrt{600}},\\\
std(x)=\sqrt{\frac{\sum{(x-\bar{x})^{2}}}{600}}.\end{split}$ (7)

This computes $x$ and $CI$ for each language permutation separately. To
aggregate this across different language permutations, we simply take the
average and the standard deviation.

To compute the statistical significance between different approaches, we use
ANOVA and perform a multiple pairwise comparisons analysis using Tukey’s
honestly significant difference (HSD) test141414We use bioinfokit library
https://github.com/reneshbedre/bioinfokit over different language permutations
for each metric.

##  Appendix D More Results & Analysis using Boostrap Sampling

In this Section, we present more results averaged over language orders in
addition to fine-grained results and ablation studies.

###  D.1 Full Average Results

Table 8 shows the full results and confidence intervals for different
continual learning approaches. Compared to intent classification, we observe a
higher forgetting and slightly higher transfer but a lower zero-shot transfer
and final performance in the case of slot filling. This could be due to the
nature of the task of slot filling, which is more challenging to learn. In
general, we can observe the same forgetting, transfer, zero-shot transfer, and
final performance trends between intent classification and slot filling. In
other words, if a model $a$ has higher forgetting of intent classification
than model $b$ then the same thing applies to slot filling. This could be due
to the transfer between intent classification and slot filling that is
maximized when training them jointly. The best model for transfer is _Lang-
Spec Ada(F)_ , which we hypothesize is due to its lightweight adaptation to
the current language, which makes it overfit on that at the cost of a lower
average and final performance overall.

Model | F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$
---|---|---|---|---
Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 2.93 $\pm 1.24$ | 5.67 $\pm 0.93$ | 0.68 $\pm 0.14$ | 1.37 $\pm 0.53$ | 50.24 $\pm 3.43$ | 36.32 $\pm 1.91$ | 91.06 $\pm 1.08$ | 69.37 $\pm 1.06$
_Lang-Spec FT_ |  |  |  |  |  |  | 93.40 $\pm 0.08$ | 73.90 $\pm 0.83$
_Lang-Spec FT_ \+ Ada(T) |  |  |  |  |  |  | 93.04 $\pm 0.09$ | 72.90 $\pm 0.80$
_Lang-Spec FT_ \+ Ada(F) |  |  |  |  |  |  | 88.79 $\pm 0.13$ | 67.46 $\pm 0.89$
_Inc Joint_ | 0.11 $\pm 0.10$ | 0.91 $\pm 0.34$ | 0.52 $\pm 0.19$ | 0.83 $\pm 0.77$ | 50.07 $\pm 2.48$ | 36.39 $\pm 2.60$ | 94.16 $\pm 0.18$ | 74.88 $\pm 0.38$
_Multilingual_ |  |  |  |  |  |  | 94.25 $\pm 0.07$ | 76.34 $\pm 0.82$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.49 $\pm 0.08$ | 1.32 $\pm 0.23$ | 0.23 $\pm 0.21$ | 0.95 $\pm 0.21$ | -0.43 $\pm 0.16$ | 0.42 $\pm 0.06$ | 93.51 $\pm 0.18$ | 74.74 $\pm 0.20$
_Lang-Spec Enc[1-9]_ | 0.78 $\pm 0.15$ | 1.95 $\pm 0.51$ | 0.80 $\pm 0.19$ | 1.44 $\pm 0.71$ | 24.23 $\pm 1.73$ | 12.32 $\pm 1.24$ | 93.50 $\pm 0.21$ | 74.19 $\pm 0.92$
_Lang-Spec Task_ | 2.91 $\pm 1.26$ | 5.26 $\pm 1.01$ | 0.66 $\pm 0.18$ | 1.15 $\pm 1.15$ | 0.10 $\pm 0.25$ | 0.07 $\pm 0.02$ | 90.86 $\pm 1.46$ | 69.41 $\pm 1.57$
_Lang-Spec Ada(T)_ | 2.19 $\pm 1.12$ | 4.23 $\pm 1.26$ | 0.98 $\pm 0.18$ | 2.04 $\pm 0.92$ | 49.35 $\pm 3.64$ | 33.60 $\pm 2.98$ | 91.75 $\pm 1.39$ | 71.13 $\pm 1.68$
_Lang-Spec Ada(F)_ | 1.20 $\pm 0.35$ | 3.35 $\pm 0.85$ | 2.82 $\pm 0.33$ | 3.93 $\pm 0.68$ | 6.52 $\pm 2.16$ | 2.80 $\pm 0.59$ | 90.36 $\pm 0.37$ | 68.55 $\pm 1.10$
Other Continuous Learning Algorithms
_EWC_ | 3.07 $\pm 1.32$ | 5.78 $\pm 1.00$ | 0.73 $\pm 0.12$ | 1.46 $\pm 0.65$ | 50.16 $\pm 3.48$ | 36.31 $\pm 1.94$ | 91.03 $\pm 1.26$ | 69.63 $\pm 1.52$
_ER_ | 1.29 $\pm 0.51$ | 3.06 $\pm 0.59$ | 0.75 $\pm 0.17$ | 1.47 $\pm 0.85$ | 50.71 $\pm 3.55$ | 36.91 $\pm 2.14$ | 93.09 $\pm 0.29$ | 73.00 $\pm 0.52$
_KD-Logit_ | 2.37 $\pm 0.83$ | 5.53 $\pm 0.96$ | 0.62 $\pm 0.15$ | 1.40 $\pm 0.68$ | 50.18 $\pm 3.14$ | 36.25 $\pm 1.91$ | 91.46 $\pm 0.87$ | 69.64 $\pm 1.58$
_KD-Rep_ | 2.29 $\pm 0.80$ | 5.35 $\pm 0.69$ | 0.69 $\pm 0.20$ | 1.43 $\pm 0.59$ | 50.41 $\pm 2.92$ | 36.26 $\pm 1.96$ | 91.69 $\pm 0.71$ | 70.03 $\pm 1.09$

Table 8:  A summary of results for different continual learning approaches
over the average across language order. For each metric and score, we
highlight the best score in bold and underline the second best score.

Model | F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$
---|---|---|---|---
Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
_Naive Seq FT_ | 2.93 $\pm 1.24$ | 5.67 $\pm 0.93$ | 0.68 $\pm 0.14$ | 1.37 $\pm 0.53$ | 50.24 $\pm 3.43$ | 36.32 $\pm 1.91$ | 91.06 $\pm 1.08$ | 69.37 $\pm 1.06$
_Lang-Spec FT_ |  |  |  |  |  |  | 93.40 $\pm 0.08$ | 73.90 $\pm 0.83$
_Lang-Spec Trans_ | 0.49 $\pm 0.08$ | 1.32 $\pm 0.23$ | 0.23 $\pm 0.21$ | 0.95 $\pm 0.21$ | -0.43 $\pm 0.16$ | 0.42 $\pm 0.06$ | 93.51 $\pm 0.18$ | 74.74 $\pm 0.20$
_Lang-Spec Enc[1-12]_ | 0.49 $\pm 0.08$ | 1.30 $\pm 0.16$ | 0.23 $\pm 0.21$ | 0.77 $\pm 0.31$ | -0.31 $\pm 0.18$ | 0.57 $\pm 0.09$ | 93.52 $\pm 0.12$ | 74.51 $\pm 0.25$
_Lang-Spec Embed_ | 3.13 $\pm 1.35$ | 5.88 $\pm 0.95$ | 0.74 $\pm 0.20$ | 1.24 $\pm 0.79$ | 50.67 $\pm 2.98$ | 36.62 $\pm 1.89$ | 90.69 $\pm 1.28$ | 69.59 $\pm 1.23$
_Lang-Spec Enc[1-3]_ | 1.88 $\pm 0.77$ | 4.32 $\pm 0.69$ | 0.77 $\pm 0.19$ | 1.37 $\pm 0.64$ | 52.20 $\pm 3.23$ | 37.42 $\pm 1.99$ | 92.25 $\pm 0.76$ | 71.59 $\pm 1.52$
_Lang-Spec Enc[4-6]_ | 1.47 $\pm 0.65$ | 2.87 $\pm 0.36$ | 0.78 $\pm 0.23$ | 1.61 $\pm 0.45$ | 47.83 $\pm 3.00$ | 34.66 $\pm 1.79$ | 92.71 $\pm 0.65$ | 73.06 $\pm 0.97$
_Lang-Spec Enc[7-9]_ | 1.45 $\pm 0.56$ | 3.02 $\pm 0.52$ | 0.70 $\pm 0.16$ | 1.32 $\pm 0.52$ | 38.33 $\pm 3.00$ | 23.68 $\pm 2.36$ | 92.43 $\pm 0.78$ | 72.28 $\pm 1.05$
_Lang-Spec Enc[10-12]_ | 2.21 $\pm 0.86$ | 4.14 $\pm 0.84$ | 0.47 $\pm 0.24$ | 1.35 $\pm 0.56$ | 41.38 $\pm 2.13$ | 20.04 $\pm 1.89$ | 91.41 $\pm 1.08$ | 71.14 $\pm 1.13$
_Lang-Spec Enc[1-6]_ | 1.27 $\pm 0.67$ | 2.99 $\pm 0.62$ | 0.87 $\pm 0.17$ | 1.64 $\pm 0.65$ | 45.23 $\pm 2.56$ | 31.21 $\pm 2.17$ | 92.92 $\pm 0.52$ | 73.33 $\pm 1.09$
_Lang-Spec Enc[7-12]_ | 1.66 $\pm 0.36$ | 3.37 $\pm 0.69$ | 0.31 $\pm 0.33$ | 0.65 $\pm 0.73$ | 6.04 $\pm 1.13$ | 4.53 $\pm 0.96$ | 91.97 $\pm 0.38$ | 71.63 $\pm 1.15$
_Lang-Spec Enc[1-9]_ | 0.78 $\pm 0.15$ | 1.95 $\pm 0.51$ | 0.80 $\pm 0.19$ | 1.44 $\pm 0.71$ | 24.23 $\pm 1.73$ | 12.32 $\pm 1.24$ | 93.50 $\pm 0.21$ | 74.19 $\pm 0.92$
_Lang-Spec Enc[10-12]_ | 2.21 $\pm 0.86$ | 4.14 $\pm 0.84$ | 0.47 $\pm 0.24$ | 1.35 $\pm 0.56$ | 41.38 $\pm 2.13$ | 20.04 $\pm 1.89$ | 91.41 $\pm 1.08$ | 71.14 $\pm 1.13$

Table 9:  Per group layer analysis: ablation studies of different M-BERT’s
components. Best, second best, and third best scores for each metric are in
bold, underlined, and italicized respectively.

###  D.2 Per M-BERT Components Analysis

Table 9 shows ablation studies for the analysis of M-BERT components following
four different categories: groups of 12 layers with or without embeddings,
groups of 3 layers, 6 layers, and 9 layers at a time trained in a language-
specific manner and the rest shared between languages. We notice that training
the full _Lang-Spec Trans_ and _Lang-Spec Enc[1-12]_ have the best in terms of
forgetting and final performance. Training only the first 8 encoder layers
_Lang-Spec Enc[1-9]_ , excluding embeddings, in a language-specific manner
comes next in terms of a low forgetting and a comparable final performance,
with a relatively better transfer and zero-shot transfer performance. Other
good models reaching a good compromise between transfer, zero-shot transfer,
and forgetting with less language-specific layers are _Lang-Spec Enc[1-3]_ and
_Lang-Spec Enc[1-6]_. _Naive Seq FT_ is comparable to those model-expansion
approaches in terms of zero-shot performance but has lower final performance
and significantly higher forgetting. We also notice the same trend for
language-specific embeddings _Lang-Spec Embed_ , which reaches the second-best
zero-shot transfer performance but with also a high forgetting. This suggests
that language-specific knowledge is less likely to be encoded in the
embeddings and more at the encoder layers. This shows that there is a real
plasticity-stability tradeoff between zero-shot transfer and knowledge
preservation (which we explain in more detail in §4.5).

###  D.3 Full Results on Language Permutations

Full results for all language permutations can be found in Tables 10, 11, and
12. By looking at additional language permutations, _L2H_ (Thai $\rightarrow$
Spanish $\rightarrow$ Hindi $\rightarrow$ French $\rightarrow$ German
$\rightarrow$ English) is still the most challenging one in terms of knowledge
preservation, accumulation, generalization, and model utility. _H2L_ (English
$\rightarrow$ German $\rightarrow$ French $\rightarrow$ Hindi $\rightarrow$
Spanish $\rightarrow$ Thai) is still the easiest to learn. Order 5(Hindi
$\rightarrow$ English $\rightarrow$ Spanish $\rightarrow$ Thai $\rightarrow$
French $\rightarrow$ German) is the second most challenging language
permutation to train. In general, the same trends regarding the more
challenging nature of training for certain language permutations are observed
for both intent classification and slot filling uniformly. Table 13 includes
the results for more language permutations for the balanced data.

Model | _H2L_ | _L2H_
---|---|---
Test Intent Accuracy On
F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$ | F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 1.52 $\pm 0.02$ | 0.93 $\pm 0.02$ | 50.68 $\pm 0.03$ | 92.06 $\pm 0.02$ | 5.52 $\pm 0.04$ | 0.57 $\pm 0.01$ | 44.66 $\pm 0.02$ | 88.80 $\pm 0.02$
_Lang-Spec FT_ |  |  |  | 93.40 $\pm 0.08$ |  |  |  | 93.40 $\pm 0.08$
_Lang-Spec FT_ \+ Ada(T) |  |  |  | 93.04 $\pm 0.09$ |  |  |  | 93.04 $\pm 0.09$
_Lang-Spec FT_ \+ Ada(F) |  |  |  | 88.79 $\pm 0.13$ |  |  |  | 88.79 $\pm 0.13$
_Inc Joint_ | -0.01 $\pm 0.01$ | 0.15 $\pm 0.02$ | 50.32 $\pm 0.03$ | 93.91 $\pm 0.01$ | 0.12 $\pm 0.01$ | 0.63 $\pm 0.01$ | 45.87 $\pm 0.03$ | 94.30 $\pm 0.01$
_Multilingual_ |  |  |  | 94.25 $\pm 0.07$ |  |  |  | 94.25 $\pm 0.07$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.40 $\pm 0.01$ | 0.59 $\pm 0.02$ | -0.48 $\pm 0.00$ | 93.86 $\pm 0.01$ | 0.62 $\pm 0.02$ | 0.03 $\pm 0.01$ | -0.54 $\pm 0.00$ | 93.37 $\pm 0.01$
_Lang-Spec Enc[1-9]_ | 0.60 $\pm 0.01$ | 1.00 $\pm 0.01$ | 22.02 $\pm 0.02$ | 93.75 $\pm 0.02$ | 1.05 $\pm 0.02$ | 0.63 $\pm 0.01$ | 22.50 $\pm 0.01$ | 93.15 $\pm 0.01$
_Lang-Spec Task_ | 1.53 $\pm 0.02$ | 0.84 $\pm 0.01$ | 0.17 $\pm 0.00$ | 91.93 $\pm 0.01$ | 5.53 $\pm 0.04$ | 0.38 $\pm 0.02$ | -0.11 $\pm 0.00$ | 87.68 $\pm 0.02$
_Lang-Spec Ada(T)_ | 1.18 $\pm 0.01$ | 1.29 $\pm 0.01$ | 50.25 $\pm 0.03$ | 92.36 $\pm 0.02$ | 4.43 $\pm 0.04$ | 0.79 $\pm 0.02$ | 42.35 $\pm 0.02$ | 88.66 $\pm 0.02$
_Lang-Spec Ada(F)_ | 0.84 $\pm 0.02$ | 3.41 $\pm 0.02$ | 3.80 $\pm 0.00$ | 91.08 $\pm 0.02$ | 1.87 $\pm 0.05$ | 2.43 $\pm 0.02$ | 9.68 $\pm 0.01$ | 89.92 $\pm 0.02$
Other Continuous Learning Algorithms
_EWC_ | 1.82 $\pm 0.02$ | 0.74 $\pm 0.01$ | 51.13 $\pm 0.03$ | 91.16 $\pm 0.02$ | 5.9 $\pm 0.04$ | 0.48 $\pm 0.02$ | 44.73 $\pm 0.03$ | 88.28 $\pm 0.02$
_ER_ | 0.71 $\pm 0.01$ | 0.95 $\pm 0.02$ | 49.59 $\pm 0.03$ | 93.51 $\pm 0.01$ | 2.35 $\pm 0.03$ | 0.78 $\pm 0.01$ | 44.87 $\pm 0.03$ | 92.58 $\pm 0.02$
_KD-Logit_ | 1.42 $\pm 0.01$ | 0.77 $\pm 0.02$ | 50.79 $\pm 0.03$ | 91.60 $\pm 0.02$ | 4.07 $\pm 0.04$ | 0.51 $\pm 0.01$ | 44.38 $\pm 0.03$ | 89.65 $\pm 0.02$
_KD-Rep_ | 1.49 $\pm 0.01$ | 0.96 $\pm 0.01$ | 51.17 $\pm 0.03$ | 91.64 $\pm 0.02$ | 4.00 $\pm 0.04$ | 0.53 $\pm 0.01$ | 45.11 $\pm 0.02$ | 90.17 $\pm 0.02$
| Test Slot Filling On
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 4.15 $\pm 0.18$ | 0.77 $\pm 0.20$ | 37.03 $\pm 0.05$ | 67.80 $\pm 0.13$ | 7.06 $\pm 0.23$ | 0.77 $\pm 0.17$ | 33.29 $\pm 0.03$ | 68.37 $\pm 0.13$
_Lang-Spec FT_ |  |  |  | 73.90 $\pm 0.83$ |  |  |  | 73.90 $\pm 0.83$
_Lang-Spec FT_ \+ Ada(T) |  |  |  | 72.90 $\pm 0.80$ |  |  |  | 72.90 $\pm 0.80$
_Lang-Spec FT_ \+ Ada(F) |  |  |  | 67.46 $\pm 0.89$ |  |  |  | 67.46 $\pm 0.89$
_Inc Joint_ | 0.78 $\pm 0.11$ | 0.69 $\pm 0.16$ | 37.92 $\pm 0.05$ | 75.14 $\pm 0.13$ | 0.37 $\pm 0.14$ | -0.47 $\pm 0.19$ | 32.75 $\pm 0.03$ | 75.14 $\pm 0.14$
_Multilingual_ |  |  |  | 76.34 $\pm 0.82$ |  |  |  | 76.34 $\pm 0.82$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.99 $\pm 0.11$ | 0.92 $\pm 0.18$ | 0.33 $\pm 0.00$ | 74.88 $\pm 0.13$ | 1.23 $\pm 0.14$ | 0.89 $\pm 0.17$ | 0.39 $\pm 0.00$ | 74.85 $\pm 0.14$
_Lang-Spec Enc[1-9]_ | 2.35 $\pm 0.15$ | 1.79 $\pm 0.18$ | 10.57 $\pm 0.01$ | 72.51 $\pm 0.13$ | 2.03 $\pm 0.15$ | 0.74 $\pm 0.19$ | 12.63 $\pm 0.01$ | 74.01 $\pm 0.14$
_Lang-Spec Task_ | 4.08 $\pm 0.17$ | 1.91 $\pm 0.16$ | 0.06 $\pm 0.00$ | 68.88 $\pm 0.15$ | 7.23 $\pm 0.24$ | -0.67 $\pm 0.19$ | 0.06 $\pm 0.00$ | 66.28 $\pm 0.13$
_Lang-Spec Ada(T)_ | 2.46 $\pm 0.14$ | 2.75 $\pm 0.16$ | 35.05 $\pm 0.05$ | 71.79 $\pm 0.15$ | 6.42 $\pm 0.23$ | 0.40 $\pm 0.17$ | 29.89 $\pm 0.03$ | 67.70 $\pm 0.12$
_Lang-Spec Ada(F)_ | 2.57 $\pm 0.20$ | 4.77 $\pm 0.17$ | 3.34 $\pm 0.00$ | 70.33 $\pm 0.15$ | 5.01 $\pm 0.24$ | 2.70 $\pm 0.20$ | 2.59 $\pm 0.00$ | 67.07 $\pm 0.12$
Other Continuous Learning Algorithms
_EWC_ | 4.22 $\pm 0.20$ | 1.19 $\pm 0.17$ | 37.39 $\pm 0.05$ | 68.33 $\pm 0.13$ | 7.53 $\pm 0.25$ | 0.52 $\pm 0.16$ | 33.25 $\pm 0.03$ | 66.91 $\pm 0.14$
_ER_ | 2.32 $\pm 0.15$ | 1.83 $\pm 0.16$ | 37.50 $\pm 0.05$ | 73.31 $\pm 0.14$ | 3.48 $\pm 0.20$ | 0.44 $\pm 0.19$ | 32.97 $\pm 0.04$ | 72.00 $\pm 0.15$
_KD-Logit_ | 4.42 $\pm 0.18$ | 1.79 $\pm 0.15$ | 37.50 $\pm 0.05$ | 68.13 $\pm 0.14$ | 7.36 $\pm 0.27$ | 0.13 $\pm 0.19$ | 32.86 $\pm 0.04$ | 67.13 $\pm 0.14$
_KD-Rep_ | 4.56 $\pm 0.18$ | 1.61 $\pm 0.15$ | 37.42 $\pm 0.05$ | 68.28 $\pm 0.13$ | 6.65 $\pm 0.28$ | 1.03 $\pm 0.17$ | 32.57 $\pm 0.03$ | 69.03 $\pm 0.13$

Table 10:  Per language permutation view: a pairwise comparison between _H2L_
(English $\rightarrow$ German $\rightarrow$ French $\rightarrow$ Hindi
$\rightarrow$ Spanish $\rightarrow$ Thai) and _L2H_ (Thai $\rightarrow$
Spanish $\rightarrow$ Hindi $\rightarrow$ French $\rightarrow$ German
$\rightarrow$ English). We highlight the best forgetting (lowest), transfer
(highest), zero-shot transfer (highest), and final performance (highest) of
accuracy and f1 scores among those two orders for each approach in bold,
whereas the best scores across approaches for the two orders separately are
underlined.

Model | Spanish $\rightarrow$ Hindi $\rightarrow$ English $\rightarrow$ German $\rightarrow$ Thai $\rightarrow$ French | French $\rightarrow$ Thai $\rightarrow$ German $\rightarrow$ English $\rightarrow$ Hindi $\rightarrow$ Spanish
---|---|---
Test Intent Accuracy On
F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$ | F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 2.62 $\pm 0.03$ | 0.59 $\pm 0.01$ | 52.07 $\pm 0.03$ | 91.49 $\pm 0.02$ | 2.63 $\pm 0.03$ | 0.52 $\pm 0.02$ | 55.0 $\pm 0.02$ | 90.74 $\pm 0.02$
_Lang-Spec FT_ |  |  |  | 93.40 $\pm 0.08$ |  |  |  | 93.40 $\pm 0.08$
_Lang-Spec FT_ \+ Ada(T) |  |  |  | 93.04 $\pm 0.09$ |  |  |  | 93.04 $\pm 0.09$
_Lang-Spec FT_ \+ Ada(F) |  |  |  | 88.79 $\pm 0.13$ |  |  |  | 88.79 $\pm 0.13$
_Inc Joint_ | 0.11 $\pm 0.01$ | 0.47 $\pm 0.01$ | 53.86 $\pm 0.02$ | 94.01 $\pm 0.01$ | 0.25 $\pm 0.01$ | 0.61 $\pm 0.01$ | 50.51 $\pm 0.02$ | 94.09 $\pm 0.01$
_Multilingual_ |  |  |  | 94.25 $\pm 0.07$ |  |  |  | 94.25 $\pm 0.07$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.45 $\pm 0.01$ | 0.05 $\pm 0.02$ | -0.37 $\pm 0.00$ | 93.43 $\pm 0.01$ | 0.51 $\pm 0.01$ | 0.39 $\pm 0.02$ | -0.5 $\pm 0.0$ | 93.63 $\pm 0.01$
_Lang-Spec Enc[1-9]_ | 0.64 $\pm 0.02$ | 0.54 $\pm 0.01$ | 26.32 $\pm 0.02$ | 93.68 $\pm 0.01$ | 0.81 $\pm 0.02$ | 0.82 $\pm 0.02$ | 25.26 $\pm 0.02$ | 93.59 $\pm 0.01$
_Lang-Spec Task_ | 2.23 $\pm 0.03$ | 0.46 $\pm 0.02$ | 0.47 $\pm 0.00$ | 91.73 $\pm 0.02$ | 3.02 $\pm 0.03$ | 0.85 $\pm 0.02$ | -0.07 $\pm 0.0$ | 90.91 $\pm 0.02$
_Lang-Spec Ada(T)_ | 1.36 $\pm 0.02$ | 1.07 $\pm 0.01$ | 50.06 $\pm 0.02$ | 92.70 $\pm 0.02$ | 2.33 $\pm 0.03$ | 0.78 $\pm 0.02$ | 51.96 $\pm 0.02$ | 92.15 $\pm 0.02$
_Lang-Spec Ada(F)_ | 0.82 $\pm 0.02$ | 2.61 $\pm 0.02$ | 5.68 $\pm 0.01$ | 90.34 $\pm 0.02$ | 1.21 $\pm 0.03$ | 2.75 $\pm 0.02$ | 8.84 $\pm 0.01$ | 90.17 $\pm 0.02$
Other Continuous Learning Algorithms
_EWC_ | 2.55 $\pm 0.02$ | 0.87 $\pm 0.01$ | 52.29 $\pm 0.03$ | 92.04 $\pm 0.02$ | 2.57 $\pm 0.03$ | 0.71 $\pm 0.02$ | 54.84 $\pm 0.02$ | 91.67 $\pm 0.02$
_ER_ | 1.27 $\pm 0.02$ | 0.70 $\pm 0.02$ | 54.29 $\pm 0.02$ | 93.08 $\pm 0.01$ | 1.33 $\pm 0.02$ | 0.44 $\pm 0.02$ | 55.05 $\pm 0.03$ | 93.05 $\pm 0.02$
_KD-Logit_ | 2.16 $\pm 0.02$ | 0.54 $\pm 0.02$ | 52.32 $\pm 0.02$ | 92.23 $\pm 0.02$ | 2.18 $\pm 0.03$ | 0.45 $\pm 0.02$ | 53.73 $\pm 0.03$ | 91.84 $\pm 0.02$
_KD-Rep_ | 2.04 $\pm 0.03$ | 0.36 $\pm 0.02$ | 52.06 $\pm 0.03$ | 92.25 $\pm 0.02$ | 2.13 $\pm 0.03$ | 0.65 $\pm 0.01$ | 53.55 $\pm 0.03$ | 92.06 $\pm 0.02$
| Test Slot Filling On
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 5.40 $\pm 0.25$ | 1.95 $\pm 0.17$ | 36.2 $\pm 0.04$ | 70.61 $\pm 0.14$ | 5.5 $\pm 0.19$ | 1.81 $\pm 0.16$ | 38.41 $\pm 0.04$ | 70.30 $\pm 0.15$
_Lang-Spec FT_ |  |  |  | 73.90 $\pm 0.83$ |  |  |  | 73.90 $\pm 0.83$
_Lang-Spec FT_ \+ Ada(T) |  |  |  | 72.90 $\pm 0.80$ |  |  |  | 72.90 $\pm 0.80$
_Lang-Spec FT_ \+ Ada(F) |  |  |  | 67.46 $\pm 0.89$ |  |  |  | 67.46 $\pm 0.89$
_Inc Joint_ | 0.81 $\pm 0.14$ | 1.57 $\pm 0.16$ | 37.46 $\pm 0.05$ | 74.9 $\pm 0.16$ | 1.03 $\pm 0.15$ | 1.72 $\pm 0.17$ | 37.54 $\pm 0.04$ | 75.34 $\pm 0.15$
_Multilingual_ |  |  |  | 76.34 $\pm 0.82$ |  |  |  | 76.34 $\pm 0.82$
Model Expansion Baselines
_Lang-Spec Trans_ | 1.57 $\pm 0.18$ | 1.29 $\pm 0.15$ | 0.49 $\pm 0.00$ | 74.56 $\pm 0.13$ | 1.29 $\pm 0.13$ | 0.60 $\pm 0.17$ | 0.47 $\pm 0.0$ | 74.57 $\pm 0.15$
_Lang-Spec Enc[1-9]_ | 1.80 $\pm 0.19$ | 2.05 $\pm 0.17$ | 13.24 $\pm 0.01$ | 75.2 $\pm 0.16$ | 1.25 $\pm 0.17$ | 0.23 $\pm 0.17$ | 13.57 $\pm 0.01$ | 74.67 $\pm 0.14$
_Lang-Spec Task_ | 4.94 $\pm 0.24$ | 2.20 $\pm 0.16$ | 0.11 $\pm 0.00$ | 71.06 $\pm 0.14$ | 4.77 $\pm 0.22$ | 1.14 $\pm 0.18$ | 0.05 $\pm 0.0$ | 70.63 $\pm 0.14$
_Lang-Spec Ada(T)_ | 3.25 $\pm 0.18$ | 3.26 $\pm 0.16$ | 34.88 $\pm 0.04$ | 72.38 $\pm 0.16$ | 4.31 $\pm 0.21$ | 1.75 $\pm 0.14$ | 35.48 $\pm 0.03$ | 70.39 $\pm 0.13$
_Lang-Spec Ada(F)_ | 2.52 $\pm 0.2$ | 4.03 $\pm 0.18$ | 3.10 $\pm 0.0$ | 68.22 $\pm 0.14$ | 3.06 $\pm 0.2$ | 4.03 $\pm 0.19$ | 3.57 $\pm 0.0$ | 68.67 $\pm 0.14$
Other Continuous Learning Algorithms
_EWC_ | 5.54 $\pm 0.24$ | 1.99 $\pm 0.16$ | 36.34 $\pm 0.04$ | 70.69 $\pm 0.13$ | 5.46 $\pm 0.23$ | 1.07 $\pm 0.18$ | 38.14 $\pm 0.04$ | 70.05 $\pm 0.15$
_ER_ | 3.01 $\pm 0.18$ | 1.98 $\pm 0.16$ | 37.54 $\pm 0.04$ | 72.92 $\pm 0.13$ | 2.77 $\pm 0.18$ | 0.81 $\pm 0.17$ | 38.66 $\pm 0.04$ | 72.82 $\pm 0.14$
_KD-Logit_ | 5.00 $\pm 0.25$ | 2.00 $\pm 0.17$ | 35.67 $\pm 0.04$ | 71.82 $\pm 0.14$ | 5.46 $\pm 0.23$ | 1.52 $\pm 0.17$ | 37.78 $\pm 0.04$ | 69.76 $\pm 0.14$
_KD-Rep_ | 4.84 $\pm 0.22$ | 1.46 $\pm 0.17$ | 35.96 $\pm 0.04$ | 70.71 $\pm 0.16$ | 5.01 $\pm 0.22$ | 0.9 $\pm 0.16$ | 37.25 $\pm 0.04$ | 70.37 $\pm 0.14$

Table 11:  Per language permutation view: a pairwise comparison between Order
3 (Spanish $\rightarrow$ Hindi $\rightarrow$ English $\rightarrow$ German
$\rightarrow$ Thai $\rightarrow$ French) and Order 4 (French $\rightarrow$
Thai $\rightarrow$ German $\rightarrow$ English $\rightarrow$ Hindi
$\rightarrow$ Spanish). We highlight the best forgetting (lowest), transfer
(highest), zero-shot transfer (highest), and final performance (highest) of
accuracy and f1 scores among those two orders for each approach in bold,
whereas the best scores across approaches for the two orders separately are
underlined.

Model | Hindi $\rightarrow$ English $\rightarrow$ Spanish $\rightarrow$ Thai $\rightarrow$ French $\rightarrow$ German | German $\rightarrow$ French $\rightarrow$ Thai $\rightarrow$ Spanish $\rightarrow$ English $\rightarrow$ Hindi
---|---|---
Test Intent Accuracy On
F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$ | F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 2.97 $\pm 0.03$ | 0.75 $\pm 0.01$ | 47.04 $\pm 0.03$ | 91.63 $\pm 0.02$ | 2.32 $\pm 0.02$ | 0.71 $\pm 0.02$ | 51.97 $\pm 0.03$ | 91.63 $\pm 0.02$
_Lang-Spec FT_ |  |  |  | 93.4 $\pm 0.08$ |  |  |  | 93.4 $\pm 0.08$
_Lang-Spec FT_ \+ Ada(T) |  |  |  | 93.04 $\pm 0.09$ |  |  |  | 93.04 $\pm 0.09$
_Lang-Spec FT_ \+ Ada(F) |  |  |  | 88.79 $\pm 0.13$ |  |  |  | 88.79 $\pm 0.13$
_Inc Joint_ | 0.21 $\pm 0.01$ | 0.74 $\pm 0.01$ | 48.41 $\pm 0.03$ | 94.44 $\pm 0.01$ | -0.02 $\pm 0.01$ | 0.54 $\pm 0.02$ | 51.49 $\pm 0.03$ | 94.23 $\pm 0.01$
_Multilingual_ |  |  |  | 94.25 $\pm 0.07$ |  |  |  | 94.25 $\pm 0.07$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.41 $\pm 0.02$ | 0.03 $\pm 0.02$ | -0.57 $\pm 0.0$ | 93.39 $\pm 0.01$ | 0.52 $\pm 0.02$ | 0.29 $\pm 0.02$ | -0.11 $\pm 0.0$ | 93.38 $\pm 0.01$
_Lang-Spec Enc[1-9]_ | 0.80 $\pm 0.02$ | 0.74 $\pm 0.01$ | 23.18 $\pm 0.02$ | 93.35 $\pm 0.01$ | 0.76 $\pm 0.01$ | 1.05 $\pm 0.01$ | 26.12 $\pm 0.02$ | 93.46 $\pm 0.01$
_Lang-Spec Task_ | 2.84 $\pm 0.03$ | 0.67 $\pm 0.01$ | -0.2 $\pm 0.0$ | 91.17 $\pm 0.02$ | 2.32 $\pm 0.02$ | 0.76 $\pm 0.01$ | 0.36 $\pm 0.0$ | 91.7 $\pm 0.02$
_Lang-Spec Ada(T)_ | 2.49 $\pm 0.03$ | 1.05 $\pm 0.01$ | 47.67 $\pm 0.03$ | 92.34 $\pm 0.01$ | 1.35 $\pm 0.02$ | 0.89 $\pm 0.01$ | 53.77 $\pm 0.02$ | 92.30 $\pm 0.02$
_Lang-Spec Ada(F)_ | 1.13 $\pm 0.03$ | 3.09 $\pm 0.02$ | 4.40 $\pm 0.01$ | 90.50 $\pm 0.02$ | 1.32 $\pm 0.02$ | 2.64 $\pm 0.02$ | 6.73 $\pm 0.01$ | 90.15 $\pm 0.02$
Other Continuous Learning Algorithms
_EWC_ | 3.07 $\pm 0.03$ | 0.79 $\pm 0.01$ | 46.44 $\pm 0.03$ | 91.45 $\pm 0.02$ | 2.51 $\pm 0.02$ | 0.81 $\pm 0.01$ | 51.54 $\pm 0.02$ | 91.54 $\pm 0.02$
_ER_ | 1.11 $\pm 0.02$ | 0.72 $\pm 0.01$ | 48.23 $\pm 0.03$ | 93.00 $\pm 0.02$ | 0.98 $\pm 0.02$ | 0.92 $\pm 0.01$ | 52.23 $\pm 0.03$ | 93.32 $\pm 0.01$
_KD-Logit_ | 2.50 $\pm 0.03$ | 0.86 $\pm 0.01$ | 47.96 $\pm 0.03$ | 91.27 $\pm 0.02$ | 1.89 $\pm 0.02$ | 0.59 $\pm 0.02$ | 51.88 $\pm 0.03$ | 92.16 $\pm 0.02$
_KD-Rep_ | 2.24 $\pm 0.03$ | 0.81 $\pm 0.01$ | 48.08 $\pm 0.03$ | 91.89 $\pm 0.02$ | 1.86 $\pm 0.02$ | 0.83 $\pm 0.02$ | 52.51 $\pm 0.03$ | 92.16 $\pm 0.02$
| Test Slot Filling On
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 6.51 $\pm 0.22$ | 1.90 $\pm 0.15$ | 34.53 $\pm 0.04$ | 68.93 $\pm 0.13$ | 5.38 $\pm 0.25$ | 1.00 $\pm 0.18$ | 38.47 $\pm 0.05$ | 70.22 $\pm 0.14$
_Lang-Spec FT_ |  |  |  | 73.9 $\pm 0.83$ |  |  |  | 73.9 $\pm 0.83$
_Lang-Spec FT_ \+ Ada(T) |  |  |  | 72.9 $\pm 0.8$ |  |  |  | 72.9 $\pm 0.8$
_Lang-Spec FT_ \+ Ada(F) |  |  |  | 67.46 $\pm 0.89$ |  |  |  | 67.46 $\pm 0.89$
_Inc Joint_ | 0.99 $\pm 0.15$ | 1.21 $\pm 0.15$ | 32.99 $\pm 0.03$ | 74.45 $\pm 0.16$ | 1.52 $\pm 0.15$ | 0.27 $\pm 0.18$ | 39.69 $\pm 0.05$ | 74.31 $\pm 0.14$
_Multilingual_ |  |  |  | 76.34 $\pm 0.82$ |  |  |  | 76.34 $\pm 0.82$
Model Expansion Baselines
_Lang-Spec Trans_ | 1.65 $\pm 0.17$ | 1.04 $\pm 0.17$ | 0.37 $\pm 0.00$ | 74.51 $\pm 0.14$ | 1.17 $\pm 0.14$ | 0.97 $\pm 0.16$ | 0.47 $\pm 0.00$ | 75.04 $\pm 0.14$
_Lang-Spec Enc[1-9]_ | 1.48 $\pm 0.13$ | 2.18 $\pm 0.17$ | 10.66 $\pm 0.01$ | 75.03 $\pm 0.14$ | 2.77 $\pm 0.18$ | 1.67 $\pm 0.18$ | 13.25 $\pm 0.01$ | 73.73 $\pm 0.14$
_Lang-Spec Task_ | 5.72 $\pm 0.21$ | 2.4 $\pm 0.17$ | 0.06 $\pm 0.00$ | 70.08 $\pm 0.13$ | 4.80 $\pm 0.24$ | -0.04 $\pm 0.18$ | 0.06 $\pm 0.00$ | 69.54 $\pm 0.13$
_Lang-Spec Ada(T)_ | 4.96 $\pm 0.25$ | 2.39 $\pm 0.15$ | 29.17 $\pm 0.03$ | 72.28 $\pm 0.13$ | 3.98 $\pm 0.21$ | 1.69 $\pm 0.16$ | 37.14 $\pm 0.05$ | 72.27 $\pm 0.13$
_Lang-Spec Ada(F)_ | 3.15 $\pm 0.21$ | 4.51 $\pm 0.18$ | 1.90 $\pm 0.00$ | 69.47 $\pm 0.14$ | 3.77 $\pm 0.22$ | 3.54 $\pm 0.16$ | 2.31 $\pm 0.00$ | 67.57 $\pm 0.14$
Other Continuous Learning Algorithms
_EWC_ | 6.38 $\pm 0.23$ | 2.54 $\pm 0.17$ | 34.29 $\pm 0.04$ | 71.25 $\pm 0.14$ | 5.56 $\pm 0.27$ | 1.46 $\pm 0.17$ | 38.44 $\pm 0.05$ | 70.57 $\pm 0.16$
_ER_ | 4.12 $\pm 0.22$ | 2.90 $\pm 0.16$ | 35.45 $\pm 0.04$ | 73.39 $\pm 0.14$ | 2.65 $\pm 0.18$ | 0.83 $\pm 0.17$ | 39.34 $\pm 0.05$ | 73.56 $\pm 0.15$
_KD-Logit_ | 6.03 $\pm 0.27$ | 2.02 $\pm 0.16$ | 35.2 $\pm 0.04$ | 70.70 $\pm 0.14$ | 4.91 $\pm 0.21$ | 0.92 $\pm 0.17$ | 38.49 $\pm 0.05$ | 70.31 $\pm 0.14$
_KD-Rep_ | 5.72 $\pm 0.27$ | 2.6 $\pm 0.15$ | 35.54 $\pm 0.04$ | 71.61 $\pm 0.15$ | 5.35 $\pm 0.21$ | 0.97 $\pm 0.15$ | 38.8 $\pm 0.05$ | 70.15 $\pm 0.13$

Table 12:  Per language permutation view: a pairwise comparison between Order
5(Hindi $\rightarrow$ English $\rightarrow$ Spanish $\rightarrow$ Thai
$\rightarrow$ French $\rightarrow$ German) and Order 6 (German $\rightarrow$
French $\rightarrow$ Thai $\rightarrow$ Spanish $\rightarrow$ English
$\rightarrow$ Hindi). We highlight the best forgetting (lowest), transfer
(highest), zero-shot transfer (highest), and final performance (highest) of
accuracy and f1 scores among those two orders for each approach in bold,
whereas the best scores across approaches for the two orders separately are
underlined.

Model | F $\downarrow$ | T $\uparrow$ | FP $\uparrow$
---|---|---|---
Acc | F1 | Acc | F1 | Acc | F1
Order 1 | 1.25 $\pm 0.02$ | 3.60 $\pm 0.18$ | 0.89 $\pm 0.02$ | 1.76 $\pm 0.17$ | 89.33 $\pm 0.02$ | 65.59 $\pm 0.13$
Order 2 | 5.81 $\pm 0.05$ | 7.89 $\pm 0.28$ | 0.75 $\pm 0.02$ | 0.11 $\pm 0.17$ | 85.81 $\pm 0.02$ | 64.18 $\pm 0.14$
Order 3 | 1.68 $\pm 0.02$ | 4.43 $\pm 0.21$ | 0.77 $\pm 0.02$ | 2.20 $\pm 0.17$ | 89.57 $\pm 0.02$ | 68.88 $\pm 0.14$
Order 4 | 2.70 $\pm 0.04$ | 4.62 $\pm 0.23$ | 0.71 $\pm 0.02$ | 1.22 $\pm 0.17$ | 88.59 $\pm 0.02$ | 68.07 $\pm 0.14$
Order 5 | 1.83 $\pm 0.01$ | 5.74 $\pm 0.24$ | 6.64 $\pm 0.01$ | 4.89 $\pm 0.15$ | 96.00 $\pm 0.01$ | 71.75 $\pm 0.13$
Order 6 | 1.08 $\pm 0.01$ | 4.44 $\pm 0.20$ | 7.09 $\pm 0.01$ | 4.86 $\pm 0.15$ | 96.40 $\pm 0.01$ | 71.81 $\pm 0.13$

Table 13:  Impact of language order across the balanced dataset for _Naive Seq
FT_. The best and second best scores for each language for intent
classification and slot filling independently across approaches are
highlighted in bold and underlined, respectively.

###  D.4 Per Language Analysis

Tables 14, 15, and 16 show the full results for forgetting, transfer, and
zero-shot transfer respectively, across different languages averaged over
different language permutations. We notice that languages like English,
German, French, and Spanish have constantly lower forgetting and higher zero-
shot transfer than languages like Hindi and Thai for both intent
classification and slot filling for _Naive Seq FT_ compared to the reference
model _Inc Joint_ for which the forgetting is low and nearly equal between
different languages. Approaches like _Lang-Spec Trans_ , _Lang-Spec Enc[1-9]_
, _Lang-Spec Ada(F)_ , and to a certain degree _ER_ also reduce that gap. We
also notice that approaches that lower forgetting for a particular language do
so uniformly for all languages. The performance in terms of zero-shot transfer
is significantly lower in the case of Thai.

Model | Test Intent Accuracy On
---|---
German | English | French | Spanish | Hindi | Thai
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 1.52 $\pm 0.12$ | 1.06 $\pm 0.08$ | 1.30 $\pm 0.14$ | 1.49 $\pm 0.13$ | 2.90 $\pm 0.38$ | 5.51 $\pm 1.35$
_Inc Joint_ | 0.31 $\pm 0.05$ | 0.12 $\pm 0.04$ | 0.19 $\pm 0.05$ | 0.15 $\pm 0.04$ | 0.04 $\pm 0.07$ | 0.28 $\pm 0.08$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.36 $\pm 0.06$ | 0.33 $\pm 0.04$ | 0.44 $\pm 0.07$ | 0.34 $\pm 0.06$ | 0.42 $\pm 0.08$ | 0.46 $\pm 0.08$
_Lang-Spec Enc[1-9]_ | 0.54 $\pm 0.07$ | 0.45 $\pm 0.05$ | 0.51 $\pm 0.08$ | 0.59 $\pm 0.06$ | 0.66 $\pm 0.10$ | 0.90 $\pm 0.15$
_Lang-Spec Task_ | 1.22 $\pm 0.12$ | 0.95 $\pm 0.09$ | 1.49 $\pm 0.14$ | 1.37 $\pm 0.12$ | 3.20 $\pm 0.40$ | 5.44 $\pm 1.67$
_Lang-Spec Ada(T)_ | 0.88 $\pm 0.08$ | 0.81 $\pm 0.08$ | 1.16 $\pm 0.12$ | 1.00 $\pm 0.09$ | 1.85 $\pm 0.24$ | 4.23 $\pm 1.15$
_Lang-Spec Ada(F)_ | 0.58 $\pm 0.08$ | 0.61 $\pm 0.08$ | 0.81 $\pm 0.11$ | 0.54 $\pm 0.10$ | 0.86 $\pm 0.11$ | 1.88 $\pm 0.33$
Other Continuous Learning Algorithms
_EWC_ | 1.40 $\pm 0.15$ | 1.00 $\pm 0.08$ | 1.74 $\pm 0.15$ | 1.56 $\pm 0.13$ | 3.26 $\pm 0.37$ | 5.62 $\pm 1.75$
_ER_ | 0.76 $\pm 0.07$ | 0.53 $\pm 0.05$ | 0.87 $\pm 0.08$ | 0.71 $\pm 0.08$ | 1.13 $\pm 0.12$ | 2.19 $\pm 0.22$
_KD-Logit_ | 1.23 $\pm 0.12$ | 0.97 $\pm 0.08$ | 1.47 $\pm 0.12$ | 1.27 $\pm 0.12$ | 2.19 $\pm 0.27$ | 4.41 $\pm 0.75$
_KD-Rep_ | 1.20 $\pm 0.11$ | 0.80 $\pm 0.07$ | 1.45 $\pm 0.11$ | 1.42 $\pm 0.12$ | 2.29 $\pm 0.27$ | 4.02 $\pm 0.63$
| Test Slot Filling On
| German | English | French | Spanish | Hindi | Thai
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 3.64 $\pm 1.31$ | 3.91 $\pm 1.14$ | 2.80 $\pm 0.94$ | 2.94 $\pm 0.94$ | 6.48 $\pm 1.85$ | 8.85 $\pm 3.19$
_Inc Joint_ | 1.21 $\pm 0.85$ | 1.12 $\pm 0.70$ | 0.64 $\pm 0.71$ | 0.96 $\pm 0.62$ | 1.13 $\pm 0.70$ | 0.77 $\pm 0.57$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.90 $\pm 0.71$ | 1.02 $\pm 0.62$ | 1.03 $\pm 0.65$ | 1.21 $\pm 0.74$ | 1.28 $\pm 0.75$ | 1.06 $\pm 0.64$
_Lang-Spec Enc[1-9]_ | 2.03 $\pm 0.93$ | 1.83 $\pm 0.81$ | 1.03 $\pm 0.77$ | 1.31 $\pm 0.69$ | 1.76 $\pm 0.81$ | 2.00 $\pm 0.76$
_Lang-Spec Task_ | 3.32 $\pm 1.29$ | 2.96 $\pm 0.97$ | 2.74 $\pm 0.93$ | 2.76 $\pm 0.89$ | 6.89 $\pm 2.01$ | 8.17 $\pm 3.05$
_Lang-Spec Ada(T)_ | 2.96 $\pm 1.12$ | 3.05 $\pm 0.88$ | 1.49 $\pm 0.76$ | 1.52 $\pm 0.82$ | 4.34 $\pm 1.17$ | 6.84 $\pm 2.26$
_Lang-Spec Ada(F)_ | 1.82 $\pm 0.97$ | 1.85 $\pm 0.88$ | 1.33 $\pm 0.83$ | 1.89 $\pm 0.96$ | 2.72 $\pm 0.99$ | 5.81 $\pm 1.98$
Other Continuous Learning Algorithms
_EWC_ | 3.41 $\pm 1.25$ | 3.90 $\pm 1.24$ | 3.08 $\pm 0.95$ | 3.32 $\pm 0.96$ | 6.29 $\pm 1.86$ | 8.74 $\pm 3.22$
_ER_ | 1.94 $\pm 0.82$ | 2.01 $\pm 0.96$ | 1.60 $\pm 0.76$ | 1.82 $\pm 0.80$ | 3.65 $\pm 1.04$ | 4.73 $\pm 1.18$
_KD-Logit_ | 3.69 $\pm 1.31$ | 3.70 $\pm 1.03$ | 3.10 $\pm 1.01$ | 3.55 $\pm 1.11$ | 5.66 $\pm 1.68$ | 8.05 $\pm 2.68$
_KD-Rep_ | 3.49 $\pm 1.18$ | 3.85 $\pm 1.09$ | 3.13 $\pm 0.95$ | 2.99 $\pm 0.92$ | 5.81 $\pm 1.66$ | 7.93 $\pm 2.18$

Table 14:  CCL per language analysis of forgetting. The best and second best
scores for each language are highlighted in bold and underlined, respectively.

Model | Test Intent Accuracy On
---|---
German | English | French | Hindi | Spanish | Thai
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 0.37 $\pm 0.07$ | 0.30 $\pm 0.06$ | 0.77 $\pm 0.08$ | 1.14 $\pm 0.07$ | 0.64 $\pm 0.09$ | 0.85 $\pm 0.11$
_Inc Joint_ | 0.25 $\pm 0.07$ | 0.04 $\pm 0.06$ | 0.74 $\pm 0.09$ | 1.25 $\pm 0.06$ | 0.27 $\pm 0.12$ | 0.57 $\pm 0.11$
Model Expansion Baselines
_Lang-Spec Trans_ | -0.36 $\pm 0.08$ | -0.07 $\pm 0.06$ | 0.29 $\pm 0.10$ | 0.93 $\pm 0.08$ | 0.12 $\pm 0.10$ | 0.47 $\pm 0.11$
_Lang-Spec Enc[1-9]_ | 0.39 $\pm 0.07$ | 0.28 $\pm 0.05$ | 0.96 $\pm 0.08$ | 1.09 $\pm 0.07$ | 0.80 $\pm 0.11$ | 1.25 $\pm 0.10$
_Lang-Spec Task_ | 0.22 $\pm 0.07$ | 0.12 $\pm 0.06$ | 0.99 $\pm 0.08$ | 1.11 $\pm 0.07$ | 0.69 $\pm 0.10$ | 0.84 $\pm 0.09$
_Lang-Spec Ada(T)_ | 1.38 $\pm 0.07$ | 0.41 $\pm 0.06$ | 1.30 $\pm 0.09$ | 0.93 $\pm 0.11$ | 1.20 $\pm 0.09$ | 0.65 $\pm 0.10$
_Lang-Spec Ada(F)_ | 2.47 $\pm 0.10$ | 1.43 $\pm 0.08$ | 3.03 $\pm 0.11$ | 3.17 $\pm 0.11$ | 2.00 $\pm 0.15$ | 4.84 $\pm 0.33$
Other Continuous Learning Algorithms
_EWC_ | 0.26 $\pm 0.08$ | 0.12 $\pm 0.05$ | 1.13 $\pm 0.07$ | 1.10 $\pm 0.07$ | 0.85 $\pm 0.09$ | 0.92 $\pm 0.11$
_ER_ | 0.27 $\pm 0.08$ | 0.07 $\pm 0.06$ | 1.01 $\pm 0.08$ | 1.16 $\pm 0.07$ | 0.96 $\pm 0.10$ | 1.04 $\pm 0.11$
_KD-Logit_ | 0.16 $\pm 0.08$ | 0.13 $\pm 0.06$ | 0.96 $\pm 0.09$ | 0.96 $\pm 0.07$ | 0.68 $\pm 0.10$ | 0.82 $\pm 0.11$
_KD-Rep_ | 0.12 $\pm 0.08$ | 0.09 $\pm 0.06$ | 0.82 $\pm 0.09$ | 1.30 $\pm 0.07$ | 0.74 $\pm 0.10$ | 1.06 $\pm 0.10$
| Test Slot Filling On
| German | English | French | Spanish | Hindi | Thai
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 1.71 $\pm 0.98$ | 1.24 $\pm 0.71$ | 2.01 $\pm 0.90$ | 0.54 $\pm 0.97$ | 0.20 $\pm 0.91$ | 2.50 $\pm 0.75$
_Inc Joint_ | 1.59 $\pm 0.90$ | -0.17 $\pm 0.89$ | 1.22 $\pm 0.84$ | 1.08 $\pm 0.94$ | -1.10 $\pm 1.04$ | 2.36 $\pm 0.79$
Model Expansion Baselines
_Lang-Spec Trans_ | 1.75 $\pm 0.95$ | 1.37 $\pm 0.80$ | 1.85 $\pm 0.83$ | -0.25 $\pm 0.91$ | -0.67 $\pm 0.93$ | 1.67 $\pm 0.74$
_Lang-Spec Enc[1-9]_ | 1.80 $\pm 0.92$ | 0.45 $\pm 1.05$ | 2.11 $\pm 0.86$ | 0.67 $\pm 0.98$ | 0.51 $\pm 0.88$ | 3.12 $\pm 0.88$
_Lang-Spec Task_ | 2.28 $\pm 1.07$ | -0.27 $\pm 0.86$ | 1.55 $\pm 1.07$ | 0.56 $\pm 1.26$ | 0.44 $\pm 0.94$ | 2.36 $\pm 0.86$
_Lang-Spec Ada(T)_ | 3.24 $\pm 0.94$ | -0.54 $\pm 0.72$ | 1.04 $\pm 0.95$ | 1.59 $\pm 0.94$ | 3.37 $\pm 0.98$ | 3.53 $\pm 0.82$
_Lang-Spec Ada(F)_ | 3.48 $\pm 1.00$ | 3.38 $\pm 0.87$ | 1.46 $\pm 1.00$ | 4.68 $\pm 1.04$ | 2.11 $\pm 1.06$ | 8.48 $\pm 1.27$
Other Continuous Learning Algorithms
_EWC_ | 1.58 $\pm 1.02$ | 0.39 $\pm 0.82$ | 2.11 $\pm 0.87$ | 1.58 $\pm 1.05$ | -0.09 $\pm 0.93$ | 3.19 $\pm 0.73$
_ER_ | 1.97 $\pm 0.93$ | 0.29 $\pm 0.89$ | 2.05 $\pm 0.94$ | 1.38 $\pm 1.04$ | 0.23 $\pm 0.87$ | 2.87 $\pm 0.93$
_KD-Logit_ | 2.20 $\pm 0.98$ | 0.50 $\pm 0.83$ | 2.00 $\pm 0.84$ | 1.35 $\pm 1.00$ | -0.64 $\pm 0.94$ | 2.97 $\pm 0.76$
_KD-Rep_ | 1.90 $\pm 0.88$ | 0.90 $\pm 0.75$ | 2.54 $\pm 0.88$ | 1.01 $\pm 0.91$ | -0.23 $\pm 0.96$ | 2.45 $\pm 0.75$

Table 15:  CCL per language analysis of transfer. The best and second best
scores for each language are highlighted in bold and underlined, respectively.

Model | Test Intent Accuracy On
---|---
German | English | French | Hindi | Spanish | Thai
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 58.53 $\pm 1.49$ | 69.09 $\pm 12.56$ | 60.83 $\pm 3.24$ | 59.42 $\pm 24.92$ | 33.38 $\pm 1.35$ | 20.17 $\pm 1.10$
_Inc Joint_ | 58.48 $\pm 2.13$ | 70.13 $\pm 12.56$ | 61.17 $\pm 2.62$ | 61.18 $\pm 19.86$ | 32.28 $\pm 2.56$ | 17.20 $\pm 0.19$
Model Expansion Baselines
_Lang-Spec Trans_ | -1.42 $\pm 0.00$ | 0.44 $\pm 0.01$ | -0.01 $\pm 0.01$ | -0.95 $\pm 0.01$ | -0.15 $\pm 0.00$ | -0.47 $\pm 0.00$
_Lang-Spec Enc[1-9]_ | 26.17 $\pm 7.44$ | 33.16 $\pm 10.88$ | 25.56 $\pm 7.00$ | 27.21 $\pm 18.32$ | 21.79 $\pm 2.33$ | 11.51 $\pm 0.77$
_Lang-Spec Task_ | -0.25 $\pm 0.12$ | 0.38 $\pm 0.01$ | 0.63 $\pm 0.06$ | -0.66 $\pm 0.02$ | 0.60 $\pm 0.03$ | -0.09 $\pm 0.01$
_Lang-Spec Ada(T)_ | 55.95 $\pm 0.91$ | 67.93 $\pm 14.89$ | 60.21 $\pm 4.16$ | 58.14 $\pm 33.89$ | 36.44 $\pm 4.20$ | 17.40 $\pm 1.10$
_Lang-Spec Ada(F)_ | 5.08 $\pm 0.51$ | 14.37 $\pm 1.06$ | 7.61 $\pm 0.49$ | 6.87 $\pm 1.00$ | 5.50 $\pm 0.90$ | -0.30 $\pm 0.04$
Other Continuous Learning Algorithms
_EWC_ | 58.57 $\pm 1.77$ | 69.39 $\pm 12.59$ | 60.71 $\pm 3.48$ | 58.99 $\pm 24.22$ | 33.59 $\pm 1.40$ | 19.71 $\pm 1.30$
_ER_ | 59.70 $\pm 1.68$ | 70.20 $\pm 13.83$ | 61.32 $\pm 4.05$ | 60.09 $\pm 24.40$ | 33.38 $\pm 1.24$ | 19.57 $\pm 1.48$
_KD-Logit_ | 58.12 $\pm 1.32$ | 68.87 $\pm 12.38$ | 60.85 $\pm 3.45$ | 59.69 $\pm 24.27$ | 33.55 $\pm 1.46$ | 19.99 $\pm 1.19$
_KD-Rep_ | 58.47 $\pm 1.20$ | 68.64 $\pm 12.23$ | 60.96 $\pm 3.56$ | 59.69 $\pm 24.54$ | 34.22 $\pm 1.07$ | 20.49 $\pm 1.00$
| Test Slot Filling On
| German | English | French | Spanish | Hindi | Thai
Shared {Trans, Task} Baselines
_Naive Seq FT_ | 44.25 $\pm 1.16$ | 48.42 $\pm 8.10$ | 47.58 $\pm 1.63$ | 46.60 $\pm 15.31$ | 18.97 $\pm 0.44$ | 12.09 $\pm 0.33$
_Inc Joint_ | 44.73 $\pm 1.68$ | 48.74 $\pm 10.90$ | 47.67 $\pm 2.19$ | 46.98 $\pm 18.10$ | 18.05 $\pm 0.31$ | 12.20 $\pm 0.22$
Model Expansion Baselines
_Lang-Spec Trans_ | 0.45 $\pm 0.00$ | 0.76 $\pm 0.01$ | 0.33 $\pm 0.00$ | 0.83 $\pm 0.01$ | 0.00 $\pm 0.00$ | 0.15 $\pm 0.00$
_Lang-Spec Enc[1-9]_ | 14.81 $\pm 3.81$ | 15.50 $\pm 6.12$ | 16.09 $\pm 4.03$ | 16.11 $\pm 8.84$ | 6.62 $\pm 1.29$ | 4.80 $\pm 0.35$
_Lang-Spec Task_ | 0.07 $\pm 0.00$ | 0.15 $\pm 0.00$ | 0.08 $\pm 0.00$ | 0.04 $\pm 0.00$ | -0.02 $\pm 0.00$ | 0.09 $\pm 0.00$
_Lang-Spec Ada(T)_ | 41.08 $\pm 1.24$ | 44.36 $\pm 18.19$ | 45.26 $\pm 2.44$ | 42.56 $\pm 21.09$ | 17.62 $\pm 1.27$ | 10.72 $\pm 0.13$
_Lang-Spec Ada(F)_ | 4.42 $\pm 0.10$ | 1.12 $\pm 0.04$ | 4.51 $\pm 0.32$ | 4.86 $\pm 0.93$ | 1.80 $\pm 0.03$ | 0.09 $\pm 0.00$
Other Continuous Learning Algorithms
_EWC_ | 44.17 $\pm 1.16$ | 48.52 $\pm 8.21$ | 47.51 $\pm 1.62$ | 46.38 $\pm 15.32$ | 18.94 $\pm 0.42$ | 12.32 $\pm 0.30$
_ER_ | 44.73 $\pm 1.45$ | 49.60 $\pm 9.35$ | 48.17 $\pm 2.22$ | 47.26 $\pm 15.85$ | 19.06 $\pm 0.44$ | 12.62 $\pm 0.24$
_KD-Logit_ | 43.79 $\pm 1.04$ | 48.30 $\pm 8.21$ | 47.31 $\pm 2.05$ | 46.77 $\pm 15.51$ | 18.85 $\pm 0.37$ | 12.49 $\pm 0.22$
_KD-Rep_ | 43.81 $\pm 1.35$ | 48.10 $\pm 7.99$ | 47.38 $\pm 1.85$ | 46.60 $\pm 15.21$ | 18.83 $\pm 0.45$ | 12.82 $\pm 0.26$

Table 16:  CCL per language zero-shot forward transfer. The best and second
best scores for each language for intent classification and slot filling
independently across approaches are highlighted in bold and underlined,
respectively.

###  D.5 More Analysis

Figure 6: Correlations between different pairs of metrics: (a) Final
performance versus negative forgetting for the task of slot filling. The lower
the forgetting, the higher the final performance. (b) Final performance versus
transfer for the task of slot filling. (c) Transfer versus negative forgetting
for slot filling task. (d) Zero-shot generalization versus negative forgetting
for slot filling. Model expansion approaches are highlighted in shades of
green. We zoom over the rest of the models in the main graph and show an
overview of all approaches in the lower right corner subplot. The same trends
observed for intent classification in Figure 4 can be observed here.

Figure 6 plots final performance versus negative forgetting, final performance
versus transfer, transfer versus negative forgetting, and zero-shot transfer
versus negative forgetting for the subtask of slot filling. The same trends
observed for intent classification can also be observed for slot filling.
Figures 7(a) and 7(b) show how _Naive Seq FT_ intent classification accuracy
score and slot filling F1 score, respectively, change for each language
separately after different hops of training. We can see that although the
performance increases as more hops are seen for high-resource Latin-script
languages like English, Spanish, and to some degree French, the same cannot be
said for low-resource languages Thai and Hindi, which also suffer from being
script-isolated.

To analyze the zero-shot generalization to unseen languages, we analyze the
performance of each model across different hops. In other words, we consider
the average performance after seeing from 1 to 5 languages, enabled by the
balanced datastreams we carefully curated 2.4. We can check the performance
after training on each $x$ language(s) from exactly one datastream. Figures
8(a) and 8(b) show a comparison between different approaches across different
hops of training using zero-shot transfer metrics for intent classification
and slot filling, respectively. In general, we can observe that the average
performance of the zero-shot transfer decreases after seeing $n$ languages,
where $n\in[1\ldots 5]$. In this case, after seeing one language, the
performance is equivalent to conventional transfer learning involving two
hops, whereas the performance after seeing $n>=2$ is for multi-hop continual
learning. We notice that as we increase the number of hops, the transfer
capabilities decrease nearly uniformly across most approaches, making the
problem more challenging and different from conventional transfer learning.
Figures 8(c) and 8(d) show the generalization trends for different continual
learning approaches compared to the baselines for intent classification and
slot filling, respectively. We can see that most continual learning approaches
improve in terms of both intent accuracy and slot filling F1 scores over
_Naive Seq FT_ , and the gap increases mainly as more languages are seen
(except at $hop_{4}$). After 5 hops, there is a clear gap between _Naive Seq
FT_ and continual learning approaches on top of them _Lang-Spec Ada(T)_ and
_KD-Logit_. Figure 9 shows more results for multi-hop versus one-hop analysis
for more metrics and tasks. In general, we can observe the same trend, whereby
multi-hop dotted boxplots analysis has smaller confidence intervals than one-
hop crossed boxplots.

(a) Accuracy for intent classification.

(b) F1 score for slot filling.

Figure 7: Comparing cross-lingual generalization of _Naive Seq FT_ across many
hops and different languages for intent classification and slot filling.

(a) Zero-shot transfer of accuracy for intent classification.

(b) Zero-shot transfer of f1 score for slot filling.

(c) Accuracy for intent classification.

(d) F1 score for slot filling.

Figure 8: Measuring cross-lingual generalization to new languages across many
hops for intent classification and slot filling. This is both in terms of
zero-shot transfer metric and plain accuracy and f1 scores.

(a) Forgetting for slot filling.

(b) Transfer for intent classification.

(c) Transfer for slot filling.

(d) Final performance for intent classification.

(e) Final performance for slot filling.

Figure 9:  Comparison between different metrics using one-hop (crossed
boxplots) and multi-hop analysis (dotted boxplots), on the left and right,
respectively, for each approach.

###  D.6 Experience Replay Ablation Studies

Table 17 shows a comparison between the performance of experience replay
variants with different memory sizes ranging from 750 to 6000 instances which
accounts for 5% to 60% of the training data for each language. Although we
notice that forgetting is the lowest and the final performance is the highest
when a memory of 6000 instances is used, the gap is not that significant as
the memory is scaled down. Moreover, differences in transfer are not
correlated with the size of the memory. We notice that ER achieves a
performance that surpasses _Naive Seq FT_ even when using the lowest memory
size. This suggests that even tiny bits of memory are helpful.

Model | F $\downarrow$ | T $\uparrow$ |  $T^{0}$ $\uparrow$ | FP $\uparrow$
---|---|---|---|---
Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1
_Naive Seq FT_ | 2.93 $\pm 1.24$ | 5.67 $\pm 0.93$ | 0.68 $\pm 0.14$ | 1.37 $\pm 0.53$ | 50.24 $\pm 3.43$ | 36.32 $\pm 1.91$ | 91.06 $\pm 1.08$ | 69.37 $\pm 1.06$
_ER_ -750 | 1.97 $\pm 0.73$ | 4.28 $\pm 0.63$ | 0.65 $\pm 0.19$ | 1.46 $\pm 0.59$ | 50.41 $\pm 3.19$ | 36.53 $\pm 1.91$ | 92.10 $\pm 0.68$ | 71.65 $\pm 1.02$
_ER_ -1500 | 1.55 $\pm 0.44$ | 3.88 $\pm 0.42$ | 0.68 $\pm 0.26$ | 1.55 $\pm 0.69$ | 50.83 $\pm 3.38$ | 36.59 $\pm 1.93$ | 92.65 $\pm 0.35$ | 71.68 $\pm 0.71$
_ER_ -3000 | 1.40 $\pm 0.44$ | 3.36 $\pm 0.47$ | 0.70 $\pm 0.25$ | 1.48 $\pm 0.71$ | 51.03 $\pm 3.60$ | 36.77 $\pm 2.06$ | 92.93 $\pm 0.37$ | 72.71 $\pm 0.56$
_ER_ -4500 | 1.43 $\pm 0.58$ | 3.39 $\pm 0.75$ | 0.59 $\pm 0.11$ | 1.44 $\pm 0.38$ | 50.46 $\pm 3.68$ | 36.91 $\pm 2.19$ | 92.73 $\pm 0.72$ | 72.46 $\pm 1.05$
_ER_ -6000 | 1.29 $\pm 0.51$ | 3.06 $\pm 0.59$ | 0.75 $\pm 0.17$ | 1.47 $\pm 0.85$ | 50.71 $\pm 3.55$ | 36.91 $\pm 2.14$ | 93.09 $\pm 0.29$ | 73.00 $\pm 0.52$

Table 17:  Ablation Studies of Experience Replay where we experiment with
different memory sizes per language. For each metric and score, we highlight
the best score in bold and underline the second best score.

##  Appendix E More Results using Multiple Seeds

In this section, we show the results using different seeds for key experiments
in the main paper. We show in Table 18 and 19 the average final performance,
forgetting, and transfer averaged across different language permutations for
the baseline model compared to reference models. We also show in Table 20 the
performance on intent classification comparison between the baseline and
different continual learning algorithms across _H2L_ and _L2H_. Overall, we
notice the same trends and findings observed earlier in Tables 2, 3, and 4.

Model | Acc | F1
---|---|---
_Naive Seq FT_ | 90.40 $\pm 1.53$ | 65.01 $\pm 1.25$
_Lang-Spec FT_ | 93.28 $\pm 0.31$ | 68.93 $\pm 1.17$
_Inc Joint_ | 94.14 $\pm 0.08$ | 71.70 $\pm 0.43$
_Multilingual_ | 94.20 $\pm 0.21$ | 72.23 $\pm 0.99$

Table 18:  The average final performance across different language
permutations for the baseline compared to reference models using multiple
seeds. We highlight the best scores in bold and underline the second best
across models. We notice the same findings as when using bootstrap sampling
but with tighter confidence intervals, as shown in Table 2.

Model | F $\downarrow$ | T $\uparrow$
---|---|---
Acc | F1 | Acc | F1
_Naive Seq FT_ | 3.2 $\pm 1.66$ | 5.47 $\pm 0.87$ | 0.73 $\pm 0.16$ | 2.75 $\pm 0.63$
_Inc Joint_ | -0.1 $\pm 0.01$ | -0.38 $\pm 0.45$ | 0.57 $\pm 0.14$ | 1.73 $\pm 1.05$

Table 19:  Forgetting (F) and transfer (T) performance averaged across
different language permutations for _sequential baseline and reference models_
using different seeds. We highlight the best models in bold. We notice exactly
the same trends as when using bootstrap sampling for our analysis in Table 3.

Model | F $\downarrow$ | T $\uparrow$ | FP $\uparrow$
---|---|---|---
_H2L_ | _L2H_ | _H2L_ | _L2H_ | _H2L_ | _L2H_
_Naive Seq FT_ | 1.37 $\pm 0.14$ | 5.38 $\pm 0.34$ | 0.95 $\pm 0.03$ | 0.56 $\pm 0.07$ | 91.83 $\pm 0.55$ | 88.28 $\pm 0.55$
_Lang-Spec Trans_ | 0.01 $\pm 0.01$ | 0.17 $\pm 0.08$ | 0.57 $\pm 0.06$ | 0.09 $\pm 0.01$ | 93.81 $\pm 0.06$ | 93.27 $\pm 0.10$
_Lang-Spec Task_ | 1.29 $\pm 0.08$ | 5.52 $\pm 0.87$ | 0.88 $\pm 0.12$ | 0.43 $\pm 0.19$ | 92.12 $\pm 0.18$ | 87.20 $\pm 1.76$
_Lang-Spec Ada(T)_ | 0.81 $\pm 0.08$ | 4.17 $\pm 0.30$ | 1.16 $\pm 0.09$ | 0.65 $\pm 0.06$ | 92.53 $\pm 0.22$ | 88.61 $\pm 0.44$
_Lang-Spec Ada(F)_ | 0.38 $\pm 0.09$ | 1.04 $\pm 0.61$ | 3.54 $\pm 0.15$ | 2.34 $\pm 0.11$ | 91.15 $\pm 0.04$ | 90.0 $\pm 0.39$
_EWC_ | 1.35 $\pm 0.24$ | 5.42 $\pm 0.60$ | 0.87 $\pm 0.11$ | 0.71 $\pm 0.12$ | 91.86 $\pm 0.52$ | 88.09 $\pm 0.20$
_ER_ -6000 | 0.69 $\pm 0.14$ | 1.93 $\pm 0.28$ | 0.93 $\pm 0.07$ | 0.72 $\pm 0.14$ | 93.43 $\pm 0.08$ | 92.50 $\pm 0.25$
_KD-Logit_ | 1.33 $\pm 0.11$ | 3.82 $\pm 0.23$ | 0.81 $\pm 0.11$ | 0.54 $\pm 0.07$ | 91.86 $\pm 0.31$ | 89.85 $\pm 0.4$
_KD-Rep_ | 1.37 $\pm 0.1$ | 3.7 $\pm 0.25$ | 0.85 $\pm 0.23$ | 0.52 $\pm 0.13$ | 91.64 $\pm 0.49$ | 89.73 $\pm 0.8$

Table 20:  Performance on intent classification comparison between the
baseline and continual learning algorithms across two language permutations
using multiple seeds. We highlight in bold the lowest forgetting (F), highest
transfer (T), and final performance (FP) of accuracy scores among _H2L_ and
_L2H_ , whereas the best scores across approaches for _H2L_ and _L2H_
separately are underlined. We notice the same trends and findings as Table 4,
where only bootstrap sampling is used to compute the confidence intervals.

##  Appendix F Statistical Significance

We show in Figures 10 and 11 the results for different approaches with a
p-value lower than 0.05 for confidence intervals of 95%, thus rejecting the
null hypothesis that they are drawn from the same distribution. Figures 10(a),
11(a), 10(c), 10(b), 11(a), 10(d), 10(e), and 10(f) show confusion plots of
statistical significance p-values for different metrics (forgetting, transfer,
and final performance) for intent classification and slot filling,
respectively. For example, for forgetting, we notice that improvements or
losses from approaches are statistically significant with $95\%$ confidence
more than $49\%$ and $61\%$ of the time for intent classification and slot
filling. For zero-shot transfer, we notice $60\%$ and $56\%$ of pairwise
comparisons are statistically significant for intent classification and slot
filling. For the final performance, we notice $47\%$ and $49\%$ of pairwise
comparisons are statistically significant for intent classification and slot
filling. For transfer, we notice that improvements or degradation over the
transfer of intent classification are not statistically significant with the
exceptions of _Lang-Spec Trans_ , which is the lowest in terms of transfer,
and _Lang-Spec Ada(F)_ , which exhibits high transfer. The same can be said
for _Lang-Spec Ada(F)_ in slot filling. Overall, model expansion approaches
exhibit the highest statistical significance, whereas _EWC-Online_ and
knowledge distillation are among the lowest. Figures 12 and 13 show the
corresponding statistical significance p-value confusion plots using multiple
seeds. With a few exceptions like _Lang-Spec FT_ \+ Ada(T) and _Lang-Spec FT_
\+ Ada(F), most pairwise p-values that indicate statistical significance
between two models using bootstrap sampling analysis are compliant with
statistical significance computed using multiple seeds.

(a) Forgetting of intent accuracy.

(b) Forgetting of slot filling.

(c) Final performance of intent accuracy.

(d) Final performance of slot filling.

(e) Zero-shot transfer of intent accuracy.

(f) Zero-shot transfer of slot filling.

Figure 10:  P-values for different pairwise comparisons of different continual
learning approaches using Tukey’s honestly significant difference (HSD) test
using bootstrap sampling.

(a) Transfer of intent accuracy.

(b) Transfer of slot filling.

Figure 11:  P-values for different pairwise comparisons of different continual
learning approaches using Tukey’s honestly significant difference (HSD) test
using bootstrap sampling (Cont.).

(a) Forgetting of intent accuracy.

(b) Forgetting of slot filling.

(c) Final performance of intent accuracy.

(d) Final performance of slot filling.

(e) Zero-shot transfer of intent accuracy.

(f) Zero-shot transfer of slot filling.

Figure 12:  P-values for different pairwise comparisons of different continual
learning approaches using Tukey’s honestly significant difference (HSD) test
using different seeds.

(a) Transfer of intent accuracy.

(b) Transfer of slot filling.

Figure 13:  P-values for different pairwise comparisons of different continual
learning approaches using Tukey’s honestly significant difference (HSD) test
using different seeds (Cont.).