# How Interpretable are Reasoning Explanations
from Prompting Large Language Models?

Yeo Wei Jie1, Ranjan Satapathy2, Goh Siow Mong, Rick2, Erik Cambria1
1Nanyang Technological University
2A*STAR, IHPC
<EMAIL_ADDRESS>

###### Abstract

Prompt Engineering has garnered significant attention for enhancing the
performance of large language models across a multitude of tasks. Techniques
such as the Chain-of-Thought not only bolster task performance but also
delineate a clear trajectory of reasoning steps, offering a tangible form of
explanation for the audience. Prior works on interpretability assess the
reasoning chains yielded by Chain-of-Thought solely along a singular axis,
namely faithfulness. We present a comprehensive and multifaceted evaluation of
interpretability, examining not only faithfulness but also robustness and
utility across multiple commonsense reasoning benchmarks. Likewise, our
investigation is not confined to a single prompting technique; it expansively
covers a multitude of prevalent prompting techniques employed in large
language models, thereby ensuring a wide-ranging and exhaustive evaluation. In
addition, we introduce a simple interpretability alignment technique, termed
Self-Entailment-Alignment Chain-of-thought, that yields more than 70%
improvements across multiple dimensions of interpretability. Code is available
at https://github.com/wj210/CoT_interpretability

How Interpretable are Reasoning Explanations
from Prompting Large Language Models?

Yeo Wei Jie1, Ranjan Satapathy2, Goh Siow Mong, Rick2, Erik Cambria1 1Nanyang
Technological University 2A*STAR, IHPC<EMAIL_ADDRESS>

##  1 Introduction

In recent trends, Large Language Models (LLM) have shown impressive
performance across a diverse array of tasks, primarily through extensive
scaling of model size Brown et al. (2020). Techniques such as instruct-tuning
Wei et al. (2021) applied across diverse tasks have empowered LLMs to execute
inference on previously unseen tasks. One attributing factor lies with the
extensive efforts put into innovating new ways of prompting the LLM to better
exploit their knowledge base.

Chain-of-Thought (CoT) Wei et al. (2022) has gathered much attention due to
its simple setup which allows the LLM to generate not only the task output but
also the steps undertaken. In addition to its efficacy in enhancing the
model’s performance, this prompting method concurrently touches on one of the
important aspects of utilizing these models for decision-making:
interpretability.

The assumption is that the reasoning chain preceding the answer illustrates
the model’s thought process, enabling the audience to understand how the
answer is derived. However, such claims though seemingly plausible should be
taken lightly as they may not be faithful to the model’s reasoning process
Jacovi and Goldberg (2020). In this context, plausibility refers to the extent
to which an explanation resonates with and is deemed acceptable by a human
audience. Faithfulness, on the other hand, is characterized by the extent to
which the explanation accurately reflects the model’s decision-making process.

There has been a large number of works that seek to introduce modifications to
CoT, including Self-Consistency Wang et al. (2022b) and Least-to-Most Zhou et
al. (2022). We introduce a simple extension to the list of CoT variants, but
purely with a focus on enhancing interpretability in the reasoning chain. The
approach coined Self-Entailment-Alignment CoT (SEA-CoT) similarly utilizes a
form of consistency between the set of possible outcomes, with an additional
touch of alignment towards desirable explainability qualities. Moreover, we
conduct an extensive investigation into the reasoning explanations by
evaluating under three pivotal axes of interpretability: faithfulness,
robustness, and utility on three commonsense reasoning datasets. These
assessments are implemented across multiple prompting techniques including CoT
and various adaptations of it.

##  2 Motivation

Efforts aimed to enhance faithfulness in NLP take various forms. Extractive
rationalizing model Lei et al. (2016), designed to be faithful, generally
comprises two separate components: explainer and predictor. This design
paradigm conditions the predictor exclusively on text spans extracted by the
explainer, positing that the resultant output, $\hat{y}$ is faithfully aligned
with the extracted text, $\hat{e}$. However, prior studies Wiegreffe et al.
(2020) cautions against such beliefs, identifying limitations in adopting the
explain-then-predict approach. The authors mentioned that such an approach
restricts the focus of the predictor toward the target identified by the
explainer, thereby raising questions about what is being explained.
Conversely, Jacovi et al. Jacovi and Goldberg (2021) highlight concerns
relating to the lack of meaningful insights from multiple text spans.

In accordance, we hypothesize that besides the limitation of narrowing the
predictors’ context, generating the explanation and output using separate
modules could compromise the quality of the explanation. We set up a simple
study, comparing a modular against a single LLM setup on two interpretability
traits, faithfulness, and utility, covered in deeper detail in section 5. We
adopt the PINTO framework Wang et al. (2022a), where the explainer,
$r_{\theta}$ is a frozen pre-trained LLM and the predictor, $f_{\phi}$ is
finetuned on the downstream task, conditioned on both the generated
explanation and context, $\hat{y}=f_{\phi}(x\oplus\hat{e})$, where
$\hat{e}=r_{\theta}(x)$, x is the given context and $\oplus$ is the
concatenation process.

For the single LLM setup, we directly train $f_{\phi}$ to generate both
$\hat{e}$ and $\hat{y}$ jointly. We measure faithfulness by computing the drop
in performance when swapping $\hat{e_{i}}$ with another instance within the
same batch, $\hat{e_{j\neq i}}$ before deriving $\hat{y}|x;\hat{e}$. We use
Leakage-Adjusted Simulatability (LAS) Hase et al. (2020), to measure the
utility of the rationale, a higher score would indicate that $\hat{e}$ is more
useful towards learning $\hat{y}$. The details of LAS are covered in A.7

Figure 1: Faithfulness and Utility scores for joint and modular approach on
two reasoning datasets: CSQA and OBQA.

We conduct experiments on two commonsense reasoning datasets: Commonsense QA
(CSQA) Talmor et al. (2018) and OpenBookQA (OBQA) Mihaylov et al. (2018).
Figure 1 shows that the joint approach scores higher on both accounts of
faithfulness and utility. We hypothesize that a single model is in better
control of aligning its explanation to the resultant outcome. Contrarily, a
model relying on explanations synthesized by an external model may instead
exhibit a diminished correlation between the interdependent variables,
explaining the marginal difference in performance despite being given an
unrelated stimulus.

Notably, this observation resonates well with the recognized capability of
recent LLMs to self-generate text serving diverse objectives. In particular,
LLMs pre-trained on a large amount of text can elucidate their reasoning
processes, assisted with the appropriate prompting format. This preliminary
experiment serves as the main motivation to conduct experiments to scrutinize
the quality of explanations produced by a singular LLM.

##  3 Prompt Techniques

Figure 2: Overview of different prompting techniques to derive the reasoning
chain, to serve as the explanation (boxed with dashed line). [Left to Right]:
Cot, SC-CoT, SEA-CoT, QD and Self-Refine (SR). SC-CoT and SEC-CoT differ in
the explanation selection stage, where the former selects based on maximum
cumulative probability and the latter on two objectives: entailment, $E$, and
overlap, $O$ with an additional forward pass. Each robot figure denotes a
forward pass from the LLM, SR stops when a stopping criteria is encountered or
exceeds the max number of passes. SR requires the most pass, 3 per round.

In this section, we systematically review various ways an LLM, $f_{\phi}$ can
be prompted. These methods primarily differ in how the language model is
queried to derive the final answer. Furthermore, we proposed an approach, SEA-
CoT, aimed at improving the interpretability traits of the reasoning chain to
serve as the explanation for the resultant output. A high-level overview is
shown in Figure 2.

  * •

CoT: Chain-of-thought prompting has shown promising results in encouraging an
LLM to better answer the task by reasoning aloud the steps before arriving at
the final answer. Kojima et al. (2022) has shown that it is possible in the
zero-shot setting simply by appending ‘Let’s think step by step’ at the end of
the instruction.

  * •

Self-Consistent CoT (SC-CoT): Following on, other works like Self-Consistency
Wang et al. (2022b) address the suboptimality of greedy decoding in CoT by
sampling multiple, $N$ paths and choosing the final answer, $\hat{y}^{*}$ via
majority voting. SC-CoT has shown improvements across multiple arithmetics and
commonsense reasoning benchmarks. Since multiple explanations may lead to the
majority answer. We choose the explanation with the highest cumulative
probability. We also experiment with different ways, further discussed in the
ablation section.

  * •

Question decomposition (QD): Zhou et al. (2022) demonstrates that decomposing
a complex problem into more manageable sub-problems significantly facilitates
the problem-solving capability of the model. The model answers each sub-
problem and pieces together the answers to conclude the principal problem. We
treat the sub-question and answers as the target explanation and assess their
interpretability properties.

  * •

Self-Refine (SR): SR Madaan et al. (2023) is a type of iterative process of
prompting the LLM with a set of instructions. The main idea is to instruct the
LLM to continuously provide feedback for its’ own output and refine using the
feedback, the process stops when the feedback deems the output as sufficient
in solving the task at hand. The whole iterative process is achieved by self-
prompting the same language model. There exist other forms of acquiring
feedback, such as querying a trained feedback model or using external factual
knowledge Pan et al. (2023). We choose the approach of querying the same LLM
as we are focused on the explainability of generated outputs from a sole LLM.

##  4 Proposed Approach

Most adaptations on CoT are only aimed at maximizing task performance as
covered in Section 3. Our work is instead focused on enhancing the
interpretability of the presented reasoning chain preceding the task output.
We adapt from SC-CoT, by focusing on the $N$ sequences produced, ranked based
on specified objectives. Instead of picking explanations based on heuristics
such as highest cumulative probability, the reasoning is chosen based on the
maximization of two objectives: entailment and overlapping score between the
supported context $(x\oplus\hat{y})$ and reasoning $\hat{e}$. We posit that a
credible explanation should intrinsically align with the given context it aims
to elucidate Jie et al. (2024); in this scenario, it encompasses both the
question being addressed and the prediction label, measured by the level of
entailment. We additionally maximize the overlap between two sets of key
tokens111The two set of tokens are compared after removing any stopwords to
minimize noise within the context, which we show in later experiments to be
beneficial towards producing higher quality explanations. This simple approach
can be regarded as performing a self-alignment step to pick the most suitable
explanation with the $N$ sequences.

Inspired by works that employ the LLM itself to do self-correction, we do the
same by asking the LLM to rate the entailment level between each own generated
reasoning, $\hat{e}_{i}$ and the joint context, $x\oplus\hat{y}$. We prompt
the LLM with few-shot examples of natural language inference (NLI), $x_{e}$ in
Figure 16 and determine if the hypothesis entails the premise. The final score
to be ranked, $S_{T}$ is a combination of both the probability of entailment,
$S_{e}$, and the IoU score, $S_{o}$.

$\displaystyle S_{e}$
$\displaystyle=p_{e}(f_{\phi}(x\oplus\hat{y},\hat{e_{i}}|x_{e}))$ (1)
$\displaystyle S_{o}$
$\displaystyle=\frac{|\hat{e_{i}}\cap(x\oplus\hat{y})|}{|\hat{e_{i}}\cup(x\oplus\hat{y})|}$
(2) $\displaystyle S_{T}$ $\displaystyle=S_{e}+S_{o}$ (3)

The most interpretable explanation is then chosen via maximizing $S_{T}$. One
caveat is that in the event where $|\hat{y}^{*}|=1$, we fall back to SC-CoT.
However, this can be avoided by trivially setting the number of sequences, $N$
to be higher than the number of possible options.

##  5 Interpretability Qualities

Figure 3: The interpretability qualities measured by different perturbation
tests, to achieve the corresponding goals of an explanation. Goals referenced
from Yeo et al. (2023)

Interpretability is a multifaceted characteristic with multiple desirable
traits concerning various goals of interpretability. Inspired by existing work
on desirable goals of explainable AI Yeo et al. (2023), we assess three
aspects of interpretability: faithfulness, robustness, and utility. We propose
these traits as we believe they are directly linked to achieving such goals,
illustrated in Figure 3. We discuss the connections in further sections. In
accordance, we outline the corresponding evaluations sought out to assess each
trait, shown in Figure 4. These evaluations are primarily conditioned on both
the context and self-generated reasoning chain.

Figure 4: Interpretability test for faithfulness, robustness, and utility. The
reasoning chain is subjected to perturbations: paraphrasing and inserting
mistakes, before re-generating the subsequent output. Counterfactual: the
original question is changed to check if the resultant reasoning accounts for
edits (shaded red). Simulatability: increase in task performance when training
data is augmented with reasoning chain, measured with a student model.

Faithfulness: The concept of faithfulness seeks to gauge the extent to which
the explanation aligns with the underlying decision-making process. Lanham et
al. (2023) conducted a series of tests assessing the faithfulness of reasoning
chains generated using CoT from an LLM. However, the authors only investigated
a single prompting technique, while we conducted extensive experiments
covering multiple prompting approaches. A faithful explanation is crucial as
it fosters trust Cambria et al. (2023) and fairness, ensuring that users can
rely on the explanation to reflect the decision-making process and identify
any potential biases, thereby improving model transparency and understanding
of any causal relationships

Robustness: Robustness seeks to measure how resilient or consistent a given
explanation is under various circumstances. For instance, employing
adversarial attacks on an explanation, as delineated by Chen et al. (2022),
could serve as a mechanism to ascertain whether the model’s decision is
susceptible to diversion or distraction induced by these attacks. A robust
explanation instills confidence and trust in users that the model would behave
appropriately despite noises in the input.

Utility: A largely understudied but important trait, utility is paramount
towards maximizing the information conveyed to the audience. A useful
explanation can allow the discovery of new knowledge to human users such as
understanding the causal relationships or enable more efficient knowledge
distillation between neural models.

###  5.1 Paraphrase

Paraphrasing $\hat{e}$ corresponding to $\hat{y}$ allows us to question the
robustness of the explanation, ie how robust is the explanation against minor
variations, assuming that these variations do not alter the core intent, yet
still enable the model to produce the same outcome? Albeit such a test
concurrently touches on the concept of faithfulness, where similar thought
processes should lead to identical conclusions given the same model Jacovi and
Goldberg (2020). However, for the sake of differentiation, we consider the
primary objective of paraphrasing as an evaluation of robustness in the
following experiments.

###  5.2 Adding mistakes

In contrast to ensuring answer consistency among similar reasoning, inserting
erroneous inputs into an explanation can assess if the reasoning preceding the
output is truly faithful. One would expect the model to change its decision
given an erroneous reasoning chain if it is faithful from the start. We focus
on the alteration in prediction rather than actual task performance, since
incorrect reasoning may potentially correct an erroneous explanation, though
such occurrences are exceedingly rare.

###  5.3 Simulatability

As it is costly to employ humans to assess if a reasoning chain is useful, we
employ forward simulatability as a proxy for utility. We measure
simulatability using LAS in Section 2 (further details in A.7) as it highly
correlates with human judgment. A 220M T5-base Raffel et al. (2020) is
selected as the student model, to measure utility from improvement in
downstream performance. The generated reasoning, $\hat{e}$ is appended to the
input context $x$, which is then used as the final context for predicting the
task label, $\hat{y}=f_{s}(\hat{e}\oplus x)$, where $f_{s}$ refers to the
student model. The student model undergoes fine-tuning with the aid of these
explanations, followed by an evaluation of its performance. A key aspect of
LAS lies with the notion of subtracting a baseline, $M_{s}(f_{s}(x))$ from
$M_{s}(f_{s}(\hat{e}\oplus x))$, where $M_{s}$ is a task scoring function such
as accuracy or F1-score. This is used to assess the benefits gained by adding
$\hat{e}$ into the training process.

###  5.4 Counterfactual reasoning

An alternative method to ascertain faithfulness follows by evaluating whether
an explanation would change when the original question is modified in a
different direction, particularly when directed towards a counterfactual
scenario. Atanasova et al. (2023) shows that an instance of unfaithfulness can
be detected if the counterfactual explanation, $e^{\prime}$ does not
acknowledge the modifications, $c$ in the counterfactual instance
$x_{i}^{\prime}:y^{\prime}$, yet still successfully predicting the
counterfactual label, $y^{\prime}\neq y$. The distinction from Section 5.2 is
that besides detecting signs of unfaithfulness, it also embodies a directed
approach that assesses a model’s capacity to contemplate alternative
scenarios. Conversely, introducing mistakes can be seen as an undirected
measure aimed at gauging the decline in confidence, given an erroneous prior
belief. We deemed an instance of unfaithfulness under the following
conditions:

  1. 1.

$x_{i}^{\prime}=\\{x_{i,1},x_{i,2}...c,...x_{i,L}\\}:y_{i}^{\prime}$

  2. 2.

$\hat{y}=y\land\hat{y^{\prime}}=y^{\prime}$

  3. 3.

$e^{\prime}\cap c=\emptyset$

The first two conditions are prerequisites for assessment, while the third
indicates signs of unfaithfulness.

##  6 Experiments

Datasets: We implement the perturbation experiments across three commonsense
reasoning benchmarks.

  1. 1.

OpenBookQA Mihaylov et al. (2018), which has 4 answer choices for each
question and assesses open-book reasoning capabilities.

  2. 2.

QASC Khot et al. (2020), is an 8-choice multi-hop reasoning dataset, requiring
assembling multiple real-world facts to successfully answer the question.

  3. 3.

StrategyQA Geva et al. (2021) is a binary question dataset structured such
that the model is required to strategize a chain of reasoning steps to derive
the correct answer.

We use only the test set to run the experiments for all perturbations
introduced in Section 5, except LAS, where we employ the LLM to generate
explanations for the training set as well.

Implementation details: We use the 70B Llama-v2 Touvron et al. (2023) from
Meta as the choice of LLM for this experiment. We use a 4-bit quantized
version, via applying the GPTQ technique Frantar et al. (2022) since the full
32-bit model would require extensive resources. The full model implementation
details can be found in Appendix A.2.

Explanation modifications: We perform automatic checks on the modifications to
prevent errors in the experiments. In paraphrase, the modified explanation is
chosen only when the resultant output $\hat{y}|e_{m}$ remains the same, and
the opposite for mistakes insertion. In counterfactual generation. OpenAI’s
GPT3.5 is used for both paraphrasing and mistake insertion and GPT4 for
counterfactuals since the task is much harder as $x^{\prime}$ has to
correspond to an alternate answer choice.

Metric details: We use the percentage of flipped predictions as the
measurement unit for both paraphrased and mistake insertion. For
counterfactual inputs, we only consider $e$ to be unfaithful if the
counterfactual part, $e^{\prime}$ has a zero overlap with modification $c$.
This applies to singular reasoning chains, except QD where we only assessed
each sub-answer. Utility is measured using the LAS score, corresponding to the
increase in performance when supplemented with explanation during training. We
list the prompt templates in Appendix A.1. We compute an aggregate score,
averaging across the four qualities, after normalizing each score between 0
and 1. For paraphrase and counterfactual, we take the complement score, $1-s$,
where $s$ is the original unit.

###  6.1 Results

Figure 5: Interpretability results for the 5 prompting techniques across 3
commonsense reasoning benchmarks. Three axes of interpretability were
assessed. 1) Robustness measured via paraphrasing (Para). 2) Faithfulness is
measured with both counterfactual explanations (CF-UF) and mistake insertion.
3) Utility is represented using simulatability (Simu) of explanation.
Aggregate is the combined average score across the three axes. CF-UF measures
unfaithfulness instead of faithfulness. We take the complement of Para and CF-
UF since a lower score is better.

We show the full experimental results in Figure 5. SEA-CoT surpasses all other
baseline methods based on the average normalized score, notably displaying a
significant difference in OBQA ($>75\%$) over majority of the baselines.
Although SC-CoT is competitive, it still underperforms substantially as
compared to SEA-CoT. We observe that the underperformance of SEA-CoT in the
mistakes criteria can be explained via the relationship between SR’s weak task
performance and high score in mistakes, attributed to a higher likelihood of
altering its output. Whereas, SEA-CoT achieves the highest task performance,
albeit causing a trade-off in this regard. Nonetheless, despite comparable
levels of task performance, SEA-CoT consistently surpasses SC-CoT across other
metrics, indicating that the superior score achieved is still dependent on the
selected reasoning.

Figure 6: StrategyQA example, the reasoning chain produced by SEA-CoT reflects
the important points in the context, making it easier for a learner model to
simulate the answer from the given explanation.

The key distinction between SC-CoT and SEA-CoT is the latter’s self-critique
step, which evaluates how its explanations align with the context and the
intended answer. This approach significantly boosts utility and reduces
unfaithfulness in counterfactual contexts. Higher utility scores support the
hypothesis that context-aligned stimuli enhance the efficiency of learning
signals, facilitating easier training for student models. Looking closer in
Figure 6, where the word "shunned" is mentioned while other baselines used
"would not wear", which does not directly relate to the target question,
causing the model to erroneously infer the wrong label. While CoT successfully
determines the correct answer, it fails to acknowledge the mention of "Amish
cousins", thus exhibiting a tenuous connection to the question.

Unexpectedly, Self-Refine underperforms compared to other baselines, aligning
with Huang et al. (2023) who highlight the drawbacks of self-correction in
reasoning tasks. The primary challenge stems from the intricacy of designing
few-shot examples that can effectively drive successive enhancements over
prior outputs, limiting the potential for self-improvement. SEA-CoT, however,
not only prompts self-assessment but also offers targeted guidance to enhance
reasoning consistency with the relevant context. This simple extension greatly
improves the quality of the explanation, with no downside on performance.

###  6.2 Ablation

Type | P($\downarrow$) | CF-UF ($\downarrow$) | M ($\uparrow$) | S ($\uparrow$)
---|---|---|---|---
Random | 6.1 | 6.44 | 62.17 | 11.87
Max | 1.8 | 6.6 | 61.8 | 12.59
Overlap (O) | 1.56 | 5.04 | 70.83 | 14.88
Entailment (E) | 2.38 | 5.46 | 69.99 | 13.46
O&E (SEA-CoT) | 1.2 | 3.81 | 61.24 | 16.97
Table 1: Ablation over various ways in selecting reasoning steps to serve as
an explanation, implemented on StrategyQA. (O&E) is the proposed SEA-CoT which
uses both components.

This ablation seeks to study the effectiveness of choosing the most suitable
reasoning chain. We break down SEA-CoT’s ranking components and assess each of
them, namely the entailment and keyword overlapping score. We additionally
implement a baseline of SC-CoT that randomly picks from the list of
explanations corresponding to the majority answer. The results from Table 1
demonstrate the efficacy of considering both components of SEA-CoT when
ranking reasoning explanations. Choosing the most probable reasoning step has
shown to not perform well, whereas our approach targeted at enhancing the
important traits of an explanation is simple and yet does not hinder
performance. We also conduct additional studies on different values of N
sequences in Table 3.

###  6.3 Model size

Size | P($\downarrow$) | CF-UF ($\downarrow$) | M ($\uparrow$) | S ($\uparrow$)
---|---|---|---|---
70B | 1.2 | 3.81 | 61.24 | 16.97
13B | 4.1 | 4.38 | 69.62 | 6.16
7B | 3.79 | 7.81 | 70.62 | 15.97
Table 2: Interpretability scores between different model sizes

The scaling laws of model size primarily concern the downstream performance of
LLMs but little is known regarding the influence on interpretability
properties. We replicate the experiments on the StrategyQA dataset with a
focus on SEA-CoT prompting. We present the results in Table 2. The largest
model, 70B generally outperforms the smaller sizes across all metrics while
observing the same phenomenon in mistake insertion, previously discussed in
6.1. The improvement over smaller sizes may also be attributed to the enhanced
accuracy in generating entailment scores for the explanation, analogous to
observing greater performance of larger models in NLI tasks. Llama-13B
surprisingly performs worse than its smaller variant, despite having a bigger
network. More importantly, we note that by using SEA-CoT, even a 7B-sized
model can generate more interpretable reasoning chains than a 70B model with
other baseline prompts.

##  7 Related Works

Natural Language Explanation (NLE): NLE can primarily be categorized as either
abstractive (AE) or extractive (EE). The former is unrestricted by the context
and as such produces more plausible explanations, while the latter is aimed at
ensuring faithfulness. Notably, EE falls short in the realm of plausibility
since humans do not understand spans of text without a full context in view
Gurrapu et al. (2023). Majumder et al. (2021) utilizes a union of both forms,
conditioning the generation of AE on the extracted spans of text while
concurrently grounding the generation on relevant world knowledge. Similar
works include faithfulness through task decomposition Sanyal et al. (2022),
label-specific explanations Kumar and Talukdar (2020). Narang et al. (2020)
demonstrate the possibility of zero-shot explanation generation by pretending
the word explain to the input prompt.

Interpretable CoT: Since its introduction, CoT has garnered interest in the
research community to innovate adaptation of it Chu et al. (2023). Despite CoT
being primarily introduced to facilitate better reasoning skills out of LLMs,
there is much interest to see if these reasoning steps could be used to
explain the model’s thought process. Most of such works primarily investigate
the faithfulness of the reasoning Lanham et al. (2023); Radhakrishnan et al.
(2023); Turpin et al. (2023) or improving the faithfulness in CoT outputs, via
refinement through knowledge retrieval He et al. (2022), symbolic reasoning
Lyu et al. (2023), iterative information selection Creswell and Shanahan
(2022) and factuality calibration Ye and Durrett (2022).

Concurrently, there are works Wang et al. (2023); He et al. (2022); Prasad et
al. (2023) that focus on ascertaining the faithfulness of an explanation to
the presence of factuality. While factuality is an important trait, it is not
a sufficient component to ascertain faithfulness. Non-factual explanations may
still align faithfully with an incorrect answer. Other works concentrate on
semantic correctness Golovneva et al. (2022), regarded closer to plausibility,
which differs from the traits assessed in this study. Our work strives to
conduct a holistic assessment of interpretability across various forms of
prompting techniques used in LLMs, taking into account multiple important
properties that may be of importance to various audiences.

##  8 Conclusion

This work studied multiple ways of assessing the interpretability of an
explanation. Our work is centered around assessing different variants of CoT
and how we can better determine the suitability of the reasoning by-product as
an explanation for the underlying prediction. We also propose a modification
to the SC-CoT framework called SEA-CoT, designed specifically to yield
explanations that better fulfill the objectives of interpretability. Our
proposed framework surpasses the Robustness, Faithfulness, and Utility
dimensions across multiple reasoning benchmarks. In the future, we plan to
extend our work towards instilling interpretability and safety in the training
stages Yang et al. (2023), such as safety alignment in LLM.

##  9 Limitations

Our work only investigates a single LLM - Llama-2 This work could be extended
toward transformers of different structures such as encoder or encoder-
decoder, or larger models, such as GPT3.5/4.0, which due to limiting resources
are restricted to generate modifications instead. A secondary limitation is
the quality of modifications to the original explanation, though we ask the
modifier to check the outcome of the modified inputs (ie output remains the
same when paraphrased), the correctness is nonetheless subjected to the
ability of the modifier. Lastly, this work left out other techniques such as
grounding the LLM’s response via external knowledge, which we note is an
interesting avenue to consider next.

## References

  * Atanasova et al. (2023) Pepa Atanasova, Oana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. 2023.  Faithfulness tests for natural language explanations.  _arXiv preprint arXiv:2305.18029_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cambria et al. (2023) Erik Cambria, Rui Mao, Melvin Chen, Zhaoxia Wang, and Seng-Beng Ho. 2023.  Seven pillars for the future of artificial intelligence.  _IEEE Intelligent Systems_ , 38(06):62–69.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-snli: Natural language inference with natural language explanations.  _Advances in Neural Information Processing Systems_ , 31.
  * Chen et al. (2022) Howard Chen, Jacqueline He, Karthik Narasimhan, and Danqi Chen. 2022.  Can rationalization improve robustness?  _arXiv preprint arXiv:2204.11790_.
  * Chu et al. (2023) Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2023.  A survey of chain of thought reasoning: Advances, frontiers and future.  _arXiv preprint arXiv:2309.15402_.
  * Creswell and Shanahan (2022) Antonia Creswell and Murray Shanahan. 2022.  Faithful reasoning using large language models.  _arXiv preprint arXiv:2208.14271_.
  * Frantar et al. (2022) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  _arXiv preprint arXiv:2210.17323_.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9:346–361.
  * Golovneva et al. (2022) Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022.  Roscoe: A suite of metrics for scoring step-by-step reasoning.  _arXiv preprint arXiv:2212.07919_.
  * Gurrapu et al. (2023) Sai Gurrapu, Ajay Kulkarni, Lifu Huang, Ismini Lourentzou, Laura Freeman, and Feras A Batarseh. 2023.  Rationalization for explainable nlp: A survey.  _arXiv preprint arXiv:2301.08912_.
  * Hase et al. (2020) Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. 2020.  Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language?  _arXiv preprint arXiv:2010.04119_.
  * He et al. (2022) Hangfeng He, Hongming Zhang, and Dan Roth. 2022.  Rethinking with retrieval: Faithful large language model inference.  _arXiv preprint arXiv:2301.00303_.
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023.  Large language models cannot self-correct reasoning yet.  _arXiv preprint arXiv:2310.01798_.
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness?  _arXiv preprint arXiv:2004.03685_.
  * Jacovi and Goldberg (2021) Alon Jacovi and Yoav Goldberg. 2021.  Aligning faithful interpretations with their social attribution.  _Transactions of the Association for Computational Linguistics_ , 9:294–310.
  * Jie et al. (2024) Yeo Wei Jie, Ranjan Satapathy, and Erik Cambria. 2024.  Plausible extractive rationalization through semi-supervised entailment signal.
  * Khot et al. (2020) Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020.  Qasc: A dataset for question answering via sentence composition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8082–8090.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Kumar and Talukdar (2020) Sawan Kumar and Partha Talukdar. 2020.  Nile: Natural language inference with faithful natural language explanations.  _arXiv preprint arXiv:2005.12116_.
  * Lanham et al. (2023) Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. 2023.  Measuring faithfulness in chain-of-thought reasoning.  _arXiv preprint arXiv:2307.13702_.
  * Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.  Rationalizing neural predictions.  _arXiv preprint arXiv:1606.04155_.
  * Lyu et al. (2023) Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023.  Faithful chain-ofthought reasoning.  _arXiv preprint arXiv:2301.13379_.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_.
  * Majumder et al. (2021) Bodhisattwa Prasad Majumder, Oana-Maria Camburu, Thomas Lukasiewicz, and Julian McAuley. 2021.  Knowledge-grounded self-rationalization via extractive and natural language explanations.  _arXiv preprint arXiv:2106.13876_.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_.
  * Narang et al. (2020) Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020.  Wt5?! training text-to-text models to explain their predictions.  _arXiv preprint arXiv:2004.14546_.
  * Pan et al. (2023) Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023.  Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies.  _arXiv preprint arXiv:2308.03188_.
  * Prasad et al. (2023) Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. 2023.  Receval: Evaluating reasoning chains via correctness and informativeness.  _arXiv preprint arXiv:2304.10703_.
  * Radhakrishnan et al. (2023) Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, et al. 2023.  Question decomposition improves the faithfulness of model-generated reasoning.  _arXiv preprint arXiv:2307.11768_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Sanyal et al. (2022) Soumya Sanyal, Harman Singh, and Xiang Ren. 2022.  Fairr: Faithful and robust deductive reasoning over natural language.  _arXiv preprint arXiv:2203.10261_.
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel R Bowman. 2023.  Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting.  _arXiv preprint arXiv:2305.04388_.
  * Wang et al. (2023) Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. 2023.  Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-intensive question answering.  _arXiv preprint arXiv:2308.13259_.
  * Wang et al. (2022a) Peifeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2022a.  Pinto: Faithful language reasoning using prompt-generated rationales.  _arXiv preprint arXiv:2211.01562_.
  * Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Wiegreffe et al. (2020) Sarah Wiegreffe, Ana Marasović, and Noah A Smith. 2020.  Measuring association between labels and free-text rationales.  _arXiv preprint arXiv:2010.12762_.
  * Yang et al. (2023) Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023.  Harnessing the power of llms in practice: A survey on chatgpt and beyond.  _arXiv preprint arXiv:2304.13712_.
  * Ye and Durrett (2022) Xi Ye and Greg Durrett. 2022.  The unreliability of explanations in few-shot prompting for textual reasoning.  _Advances in neural information processing systems_ , 35:30378–30392.
  * Yeo et al. (2023) Wei Jie Yeo, Wihan van der Heever, Rui Mao, Erik Cambria, Ranjan Satapathy, and Gianmarco Mengaldo. 2023.  A comprehensive review on financial explainable ai.  _arXiv preprint arXiv:2309.11960_.
  * Zhou et al. (2022) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022.  Least-to-most prompting enables complex reasoning in large language models.  _arXiv preprint arXiv:2205.10625_.

##  Appendix A Appendix

###  A.1 Perturbation details

We use GPT3.5 to generate paraphrased versions of the reasoning explanation
produced by prompting the LLM, except QD. For QD, we select one subquestion-
answer pair to apply the perturbations to, we paraphrase both chosen question-
answer pairs and only add mistakes to the answer as the focus is on producing
wrong answers and not incomprehensible questions. To convert the question $x$
to a counterfactual instance $x^{\prime}$, we use GPT4 as GPT3.5 frequently
produces nonsensical questions that the available answer options cannot
answer. Furthermore, we subsequently deploy GPT3.5 again to identify the
edited and original portions of $x$, namely the modification $c$. Thus, we end
up with two sets of templates for both paraphrasing and addition of mistakes
(one for QD, one for others) and one set of counterfactual generation. We use
2-shot examples for adding mistakes, 3-shot for counterfactual generation, and
0-shot for paraphrasing. All figures are from Figure 7 to 11

###  A.2 Inference details

As we do not use API for the bulk of the experiments except perturbation
generation and ablation using GPT3-5. We mainly rely on local resources to
conduct inference. We use 4 x A6000 GPU for all experiments, each GPU has 46GB
of VRAM and this gives us a total of 184GB VRAM. A 70B model would require at
least 140GB VRAM, leaving only 44 VRAM left for text generation. Given an
average input size of 1000 (usually longer for prompts such as QD) and a
single batch size of 1, it would require an additional >60 GB VRAM (computed
based on L = 80, H= 64, dim = 8192 for 70B) which makes it infeasible to
implement. Thus, we perform the experiments using a 4-bit quantized version
instead, which is performed using GPTQ on the original Llama-2 70B model. GPTQ
is suitable for quantizing models consisting of billions of parameters. It has
been validated on models up to 176B parameters and shown comparable
performance with 16-bit models. The GPTQ-ed models are readily available on
`huggingface`.

We utilized `text-generation-inference`, an optimized platform for conducting
fast inference on LLMs by Huggingface, to speed up the inference process.
Overall, this allows us to process up to a batch size of 16 across the full
hardware stack.

###  A.3 Hyperparameters

Besides the prompting techniques that use best-of-n preference to select the
final output, we stick to greedy decoding. This leaves SC-CoT and SEA-CoT,
where we set N to 10 and fix temperature and k to $1.0$ and $50$ respectively
while doing sampling. This is only applied during the process of generating
explanations, where we revert to greedy decoding during evaluation across all
prompting techniques. The number of sequences is set to 10 to balance the
computational resources such as RAM and speed. N = 10 is also reported to be
sufficient in Wang et al. (2022b).

###  A.4 Few-shot Prompts

We show the few-shot examples used for the OBQA dataset, highlighting the
differences in the instruction prompt between the various techniques reviewed.
The few-shot examples are similar to Wei et al. (2022), and adjusted when
necessary, depending on the specific prompting methodology.

For Self-Refine, there are three stages of instruction-prompting, where the
second (feedback) and third (refine) stages continue iteratively until the LLM
detects a stopping criterion which ends the cycle, denoted as "Stop refining
the answer.". In the initial generation, the optimal examples are given,
similar to CoT. In the feedback stage, we list scoring criteria which is
focused on improving the interpretability of the reasoning explanation,
instead of focusing on the performance. To simulate various qualities of
output, we include both positive and negative examples. The examples in the
refine stage are similar to the feedback but are instead designed in a
continuous conversion displaying the full process of refining a bad example
into a good one. We limit the number of examples in the refine stage to 3 as
the context length is much longer here. The few-shot example prompts are
displayed from Figure 12 to 15.

###  A.5 Entailment Generation

We designed a separate prompt to be used solely by SEA-CoT, where the LLM is
instructed to self-critique the entailment between its reasoning chain and the
combined context of both the question and the produced answer. We use samples
from the e-SNLI dataset Camburu et al. (2018), we only picked instances
corresponding to either entailment or contradiction and left out the neutral
ones, as the LLM is only instructed to infer if the explanation entails or
contradicts the target context.

The probabilities for the entailment label “yes” are directly used while we
take the complement if generated "no", with the assumption that other tokens
in the vocabulary are negligible. The examples are displayed in Figure 16.

###  A.6 Number of sequences

We carry out additional experiments on increasing N sequences, to see if
increasing the number of options allows the ranking process to select more
interpretable explanations. The results in Table 3, showed that increasing N
has positive effects on the utility of the reasoning steps, while slight
negative effects on the paraphrasing and counterfactual tests. The higher
number of sequences may make it difficult to optimize for each quality
simultaneously, as one explanation may be more faithful but lacks usefulness
in teaching a less technical model to follow its reasoning process.
Nonetheless, this study is promising for context distillation, where we may be
interested in using the generated response of a larger LLM to teach a smaller
model, by using higher N values.

###  A.7 Leakge-Adjusted Simulatability (LAS)

We define the formal definitions of the LAS metric Hase et al. (2020) used in
assessing the utility of an explanation here. LAS is primarily used to measure
the improvement in task performance upon the addition of an explanation,
$\hat{e}$ to the given context, $x$ in producing an outcome,
$\hat{y}|x,\hat{e}$. Most importantly, it accounts for the two cases of
phenomena encountered. The first is when the model can guess the outcome
directly from the input, $x$. In such cases, this renders the explanation,
$\hat{e}$ as a false causal input in producing any improvements on the task
score. The second is when $\hat{e}$ directly leaks the label to the model and
the outcome can be easily guessed without consuming the given question.

The first scenario can be solved by introducing a baseline, $\hat{y}|x$, and
subtracting the task performance, $\mathbbm{1}[\hat{y}_{i}|\hat{e},x_{i}]$
from $\mathbbm{1}[\hat{y}_{i}|x_{i}]$. The second case is accounted by
measuring the performance when the explanation either leaks,
$\mathbbm{1}[\hat{y}_{i}|\hat{e}]=1$ or not,
$\mathbbm{1}[\hat{y}_{i}|\hat{e}]=0$. $\mathbbm{1}$ denotes the event where
outcome is correctly guessed, $\hat{y}=y$.

The overall LAS score regards both scenarios by taking the average of the
subtracted performance in both non-leaking, $LAS_{0}$, and leaking group,
$LAS_{1}$ below.

$\displaystyle
LAS_{0}=\frac{1}{N_{0}}\sum_{i=1}^{N_{0}}(\mathbbm{1}[\hat{y}_{i}|x_{i},\hat{e}_{i}]-\mathbbm{1}[\hat{y}_{i}|x_{i}])$
(4) $\displaystyle
LAS_{1}=\frac{1}{N_{1}}\sum_{i=1}^{N_{1}}(\mathbbm{1}[\hat{y}_{i}|x_{i},\hat{e}_{i}]-\mathbbm{1}[\hat{y}_{i}|x_{i}])$
(5) $\displaystyle LAS=\frac{1}{2}(LAS_{0}+LAS_{1})$ (6)

$N_{0}$ and $N_{1}$ denote the number of non-leaking and leaking encounters.

N | P($\downarrow$) | CF-UF ($\downarrow$) | M ($\uparrow$) | S ($\uparrow$)
---|---|---|---|---
10 | 1.2 | 3.81 | 61.24 | 16.97
30 | 2.01 | 5.98 | 67.77 | 17.2
50 | 1.8 | 6.49 | 68.40 | 18.7
Table 3: Interpretability scores across different numbers of sequences
generated per sample. Figure 7: 0-shot paraphrase template. Input [Underline]
Generated: [highlighted] Figure 8: 2-shot inserting mistake template for all
prompts except QD. Input [Underline] Generated: [highlighted]. Only show 1
example. Figure 9: 2-shot inserting mistake template for QD. Input [Underline]
Generated: [highlighted]. Only show 1 example. Figure 10: 3-shot
counterfactual generation Input [Underline] Generated: [highlighted]. Only
show 1 example. First, identify the next possible answer before editing the
question towards it. Figure 11: 0-shot edit highlighting. Input [Underline]
Generated: [highlighted]. Identify edits corresponding to the original text.
Figure 12: 7-shot prompt used for CoT, SC-CoT and SEA-CoT. There are newlines
between answer choices and each given choice, is opted out to save space.
Figure 13: 7-shot prompt used for QD. We show only 4 examples here, and there
are newlines between each sub-questions and answers, which we similarly leave
out to save space. Figure 14: Prompt for Self-Refine, we show a single example
for the initial generation, the rest is similar in CoT examples. For the
feedback, we include both good and bad examples, both displayed here. We use 7
examples for both initial generation and feedback. Figure 15: Refine stage in
Self-Refine, we show a single example here, where each example demonstrates
the entire refining process from a bad to a good example. Figure 16: NLI
examples for entailment generation for SEA-CoT, used across all datasets.


# Continual Learning for Monolingual End-to-End Automatic Speech Recognition

Steven Vander Eeckt and Hugo Van hamme
KU Leuven
Department Electrical Engineering ESAT-PSI
Kasteelpark Arenberg 10, Bus 2441, B-3001 Leuven Belgium
{steven.vandereeckt<EMAIL_ADDRESS>
###### Abstract

Adapting Automatic Speech Recognition (ASR) models to new domains results in a
deterioration of performance on the original domain(s), a phenomenon called
Catastrophic Forgetting (CF). Even monolingual ASR models cannot be extended
to new accents, dialects, topics, etc. without suffering from CF, making them
unable to be continually enhanced without storing all past data. Fortunately,
Continual Learning (CL) methods, which aim to enable continual adaptation
while overcoming CF, can be used. In this paper, we implement an extensive
number of CL methods for End-to-End ASR and test and compare their ability to
extend a monolingual Hybrid CTC-Transformer model across four new tasks. We
find that the best performing CL method closes the gap between the fine-tuned
model (lower bound) and the model trained jointly on all tasks (upper bound)
by more than $\mathbf{40\%}$, while requiring access to only $\mathbf{0.6\%}$
of the original data.

###### Index Terms:

End-to-End Automatic Speech Recognition, Continual Learning, Monolingual
Speech Recognition

##  I Introduction

Automatic Speech Recognition (ASR) has greatly progressed in recent years,
moving from Hidden Markov Model (HMM) to End-to-End (E2E) models. However,
like the Artificial Neural Networks (ANN) they use, E2E ASR models suffer from
Catastrophic Forgetting (CF) [1] when adapted to new tasks, even for
monolingual tasks: it suffices that the data distributions of the old and new
tasks differ for CF to occur.
Fortunately, many Continual Learning (CL) methods have been proposed in the
image classification community, enabling ANNs to learn continually without
suffering from CF. Following [2], CL methods are categorized into three
groups: i) the regularization-based methods use a regularization loss to train
new tasks such that it does not hurt the performance of previous tasks. [3, 4,
5] do this by estimating the importance of each parameter to previous tasks
and using these importance weights in a weighted L2 regularization to learn
new ones. [6] uses knowledge distillation [7] on the new tasks’ data to
transfer knowledge from the old to the new model; ii) the replay-based methods
store a set of representative samples in a memory to rehearse old tasks when
learning new ones. Most straightforward are [8, 9, 10], which train on the new
task and the memory jointly. Alternatively, [11, 12] focus on gradient
alignment between old and new tasks; iii) the architectural-based methods
increase the model’s capacity when learning new tasks. The latter are not
considered in this paper.
Regarding ASR and, especially, E2E ASR, CL is a very new and unexplored topic.
[13, 14] apply CL to the acoustic model of a HMM-based ASR model. [15]
considers CL for the pre-trained wav2vec2 model [16]. [17] combines a Text-to-
Speech and ASR model to prevent forgetting. [18] applies Learning without
Memorizing [19] to E2E ASR, focusing on a scenario where subsequent tasks are
much smaller than the initial one. Finally, [20] implements four existing CL
methods for E2E ASR. Compared to [20], we implement an extra seven CL methods
for E2E ASR. We test and compare their ability to continually extend and
enhance a monolingual E2E ASR by training it on new data. To make the
experiments more realistic, we run the methods without assuming access to
validation sets of previous tasks to optimize hyper-parameters. Since for many
CL methods, both regularization-based and rehearsal-based, the weight of the
regularization is a crucial hyper-parameter, we propose, based on [2], a
simple and efficient way to determine this weight.

##  II Continual Learning for E2E ASR

We first elaborate on the considered E2E ASR model as well as on the objective
of Continual Learning for E2E ASR.
Model. Our model is the Hybrid CTC/Transformer from [21]. Its loss during
training is computed as:

${\cal L}(X,y;\theta)=c\cdot{\cal L}^{c}(X,y;\theta)+(1-c)\cdot{\cal
L}^{d}(X,y;\theta)$ (1)

where ${\cal L}^{c}(X,y;\theta)$ and ${\cal L}^{d}(X,y;\theta)$ are,
respectively, the CTC and Decoder Cross-Entropy (CE) loss of the model with
parameters $\theta$ on utterance $X$ with ground truth $y$. As in [21], the
weight of CTC for training and decoding is $c=0.3$. No Language Model is used
during decoding. The outputs of the model are 300 word pieces, generated by
the Sentence Piece model [22] on the training data of the first task.
Notation. Denote $f^{c}(X;\theta)\in\mathbb{R}^{L\times o}$ and
$f^{d}(X;\theta)\in\mathbb{R}^{W\times o}$ the CTC and Decoder output,
respectively, of the model with parameters $\theta$, given utterance $X$, with
$L$, $W$ and $o$ the utterance length, output length and number of word
pieces. During training, $f^{d}(X;\theta)$ is conditioned on ground truth $y$.
Problem formulation. Let $D_{1},D_{2},..,D_{T}$ represent the labeled training
datasets of the $T$ tasks. If $\theta^{t}$ are the model’s parameters after
learning $t$ tasks, then the objective of the CL methods is to learn tasks
$1,..,T$ in sequence such that after $T$ tasks, $\theta^{T}$, adapted from
$\theta^{T-1}$, satisfies:

$\theta^{T}=\operatorname*{arg\,min}_{\theta}\sum_{t=1}^{T}\sum_{(X,y)\in
D_{t}}{\cal L}(X,y;\theta)$ (2)

However, when learning task $T$ on $D_{T}$, access to $D_{1},...,D_{T-1}$ is
assumed to be lost (though storing a small number of utterances per task in a
memory is allowed for the rehearsal-based methods), thus $\theta^{T}$ cannot
be directly computed from (2). In addition, we assume that we can no longer
use the previous tasks’ validation sets (to optimize hyper-parameters of the
CL methods). We consider this a more realistic scenario.

##  III Continual Learning Methods

We consider both regularization- and rehearsal-based methods. As E2E ASR
models have a complex architecture and are computationally demanding to train,
we focus on lightweight methods which are easily applicable to any ANN
architecture, and have proven to work well in other domains.

###  III-A Regularization-based Methods

The regularization-based methods compute a regularization loss which is added
to ${\cal L}(X,y;\theta)$ from (1) during training.
Elastic Weight Consolidation (EWC). After training task $t$, EWC [3] computes
the diagonal of the Fisher information matrix, denoted $\Omega^{t}$.
$\Omega_{ii}^{t}$ is considered the importance weight of parameter
$\theta_{i}$ for task $t$. Next, $\Omega^{t}$ is added to $\Omega^{\leq
t}=\Omega^{\leq t-1}+\Omega^{t}$ as in [23], and used in the regularization
loss to learn task $t+1$:

${\cal L}_{ewc}(\theta)=\frac{\lambda}{2}(\theta-\theta^{t})^{T}\Omega^{\leq
t}(\theta-\theta^{t})$ (3)

Since $\Omega^{t}$ is diagonal, (3) reduces to a weighted L2 regularization,
with weight $\Omega^{t}_{ii}$ for parameter $\theta_{i}$.
Memory-Aware Synapses (MAS). MAS [4] works similar as EWC, but computes (the
diagonal of) $\Omega^{t}$ differently. Given that the ASR model has both a CTC
and Decoder output, we compute $\Omega^{t}$ for MAS as follows:

$\Omega_{ii}^{t}=\mathbb{E}_{X\sim
D_{t}}\left[\left|c\frac{\partial\|f^{c}(X;\theta^{t})\|^{2}}{\partial\theta_{i}}+(1-c)\frac{\partial\|f^{d}(X;\theta^{t})\|^{2}}{\partial\theta_{i}}\right|\right]$
(4)

Next, the loss is exactly the same as for EWC in (3).
Continual learning with Sampled Quasi-Newton (CSQN). CSQN [24] was proposed to
extend EWC by considering interactions between parameters. Starting from EWC’s
$\Omega^{t}$, CSQN considers quasi-Newton methods to compute low-rank
approximations of the Hessian of the loss, which are then used as in (3) to
regularize training. We consider both the standard and the reduced version,
which was called BTREE in [24] and which we here denote CSQN-BT.
Learning Without Forgetting (LWF). LWF [6], when learning task $t+1$, uses
knowledge distillation [7] between the old model (with parameters
$\theta^{t}$) as teacher and the current model (with parameters $\theta$) as
student, on the new task’s data:

$\displaystyle{\cal L}_{lwf}$
$\displaystyle(X;\theta)=\lambda\cdot(c\sum_{i=1}^{L}\sum_{j=1}^{o}\frac{f_{i,j}^{c}(X;\theta^{t})}{\gamma}\log\frac{f_{i,j}^{c}(X;\theta)}{\gamma}$
(5)
$\displaystyle+(1-c)\sum_{i=1}^{W}\sum_{j=1}^{o}\frac{f_{i,j}^{d}(X;\theta^{t})}{\gamma}\log\frac{f_{i,j}^{d}(X;\theta)}{\gamma})$

With $\gamma$ called the temperature. In our experiments, $\gamma=1$.

###  III-B Rehearsal-based Methods

The rehearsal-based methods use a small memory of exemplars of previous tasks
to enable CL.
Experience Replay (ER). We consider three variants of ER [8]. In the standard
variant, the mini-batch from the current task is augmented with a mini-batch
sampled from memory and sent through the model to compute the loss. As this
may result in overfitting on the memory, the loss of the mini-batch sampled
from memory may be given a weight $\lambda\in(0,1)$, denoted ER ($\lambda$).
Alternatively, as in [9], the training set and memory can be merged to train
on the resulting set, referred to as BER (Batch-level ER).
Average-Gradient Episodic Memory (A-GEM). Consider $g=\frac{\partial{\cal
L}(X,y;\theta)}{\partial\theta}$ with $(X,y)$ a mini-batch from the current
task. Before $g$ is used to update the model, A-GEM [12] samples a mini-batch
$(\tilde{X},\tilde{y})$ from memory, and computes $g_{ref}=\frac{\partial{\cal
L}(\tilde{X},\tilde{y};\theta)}{\partial\theta}$. If $g$ and $g_{ref}$
interfere, i.e. if $g^{T}g_{ref}<0$, it updates $g$ with $g\leftarrow
g-\frac{g^{T}g_{ref}}{g_{ref}^{T}g_{ref}}g_{ref}$ such that the gradients
align. The resulting gradient is used to update the model. A-GEM is the more
efficient version of GEM (Gradient-Episodic Memory) [11], which was the best
method in [20].
Knowledge Distillation (KD). KD uses the same loss as LWF in (5), not computed
on a mini-batch of the new task, but on a mini-batch sampled from the memory.
Note that this loss is added to (1), so the new task is still learned using
the CE loss.

##  IV Experiments

Experiments were done in ESPnet [25]. For detailed information and more
extensive results, see our repository
111https://github.com/StevenVdEeckt/CGN_CL_Dialect.
Data. We use the Corpus Gesproken Nederlands (CGN) dataset [26], which
contains 900 hours of Dutch speech from both the Netherlands (NL) and Belgium
(VL). We consider all except the more spontaneous speech and, based on the
dialect of the speakers, split the data into four tasks: NL-main, VL-main, NL-
rest, VL-rest (learned in this order). Each task is further split into a
training, validation and test set.
Training. We use the optimizer from [21], with a learning rate of 10.0 for the
first task and 1.0 for subsequent tasks. We allow models to run for 230
epochs, but stop early when the Token Error Rate (TER) at word piece level on
the new task’s validation set has not improved for 10 epochs. As in [27], we
average the last 10 snapshots to obtain a final model.
Determining $\bm{\lambda}$. Many of the CL methods require setting a hyper-
parameter $\lambda$, the weight of the regularization. Based on [2], we
propose a simple and efficient way to determine $\lambda$ for E2E ASR. First,
we consider $\tau^{init}$, the TER (on the new task’s validation set) of the
initial model. Next, we adapt the model for five epochs without
regularization, and compute its TER, obtaining $\tau^{no\\_reg}$. Then, we set
$\lambda$ to a high value and run the model for five epochs with
regularization with weight $\lambda$. We compute the TER and obtain $\tau$. If
$(\tau-\tau^{init})/(\tau^{no\\_reg}-\tau^{init})>a$, i.e. if the gap between
$\tau^{init}$ and $\tau^{no\\_reg}$ is closed for at least $100a\%$, we return
$\lambda$; else, we set $\lambda\leftarrow p\lambda$ with $p\in(0,1)$ and
repeat the process. As such, determining $\lambda$ is done in a fast and
efficient way and does not require access to a validation set of previous
tasks. We determine $\lambda$ only for the first adaptation and then fix it.
In our experiments, we set $a=0.85$ and $p=0.10$. Moreover, for each method,
the initial value of $\lambda$ is a power of 10.
Memory. After learning a task, we sample 500 utterances from the training set
to add to the memory. While sampling uniformly, we only consider utterances
whose output length (i.e. number of word pieces in output) exceeds $0.40\cdot
mean\\_length$, where $mean\\_length$ is the average output length of the
utterances in the training set, to make sure that all of the 500 utterances
contain meaningful sentences.
Baselines. Following baselines are considered: (i) Fine-Tuning (FT): the model
is adapted without CL method (lower bound); (ii) Joint (JT): trained from
scratch on all tasks jointly; (iii) Continued Joint (CJT): adapted from
previous task and trained on current and previous tasks jointly (upper bound).
Metrics. For each method, we report the Average WER (AWER), Backward Transfer
(BWT) and Forward Transfer (FWT) [11], and Coverage (COV) [13]. Assuming $T$
tasks have been learned and $R_{i,j}$ is the WER on task $j$ after learning up
to task $i$, $\text{AWER}=\sum_{i=1}^{T}R_{T,i}$, while the BWT is:

$\text{BWT}=\frac{1}{T-1}\sum_{i=1}^{T-1}-(R_{T,i}-R_{i,i})$ (6)

Note that using this definition, negative BWT indicates forgetting.
Furthermore, we define FWT as:

$\text{FWT}=\frac{1}{T-1}\sum_{i=2}^{T}-(R_{i,i}-R^{FT}_{i,i})$ (7)

where $R^{FT}_{i,j}$ is the WER on task $j$ after learning up to task $i$ with
FT. FWT measures to which extent the model can exploit previously acquired
knowledge to learn new tasks better. Positive FWT indicates better learning
than FT. Finally, COV measures the extent to which the given method closes the
gap between FT (lower bound) and CJT (upper bound) in terms of AWER. It is
$0\%$ when the method performs as poor as FT, and $100\%$ when the method
performs as well as CJT. In addition to AWER, BWT, FWT and COV, we report the
storage requirements (Storage), expressed in an equivalent number of models
(one model requiring 105 MB).
Statistical significance. We use the Wilcoxon signed-rank test on the number
of errors per utterance [28] to test the significance of the results,
considering significance levels $\alpha=0.05$ (∗), $\alpha=0.01$ (∗∗) and
$\alpha=0.001$ (∗∗∗).

##  V Results

Table I shows the results after learning the four tasks in sequence.

TABLE I: Results after learning the four tasks. Significance level refers to improvement over baseline FT. Model | AWER$\downarrow$∗∗∗ | BWT$\uparrow$ | FWT$\uparrow$ | COV$\uparrow$ | Storage
---|---|---|---|---|---
JT | 21.3∗∗∗ | - | - | - | 260.54
CJT | 21.9∗∗∗ | +2.5 | +0.5 | 100.0$\%$ | 261.54
FT | 27.3∗∗∗ | -4.2 | - | 0.0$\%$ | 1.00
EWC | 28.3∗∗∗ | -0.7 | -4.8 | -18.9$\%$ | 2.00
MAS | 28.3∗∗∗ | -1.1 | -4.4 | -18.9$\%$ | 2.00
CSQN | 27.7∗∗∗ | -1.5 | -3.2 | -8.7$\%$ | 32.00
CSQN-BT | 27.8∗∗∗ | -1.7 | -3.2 | -9.8$\%$ | 22.00
LWF | 26.6∗∗∗ | -3.3 | +0.1 | 12.4$\%$ | 1.00
A-GEM | 26.1∗∗∗ | -2.5 | -0.0 | 22.0$\%$ | 3.24
ER | 28.0∗∗∗ | -3.4 | -1.7 | -13.1$\%$ | 3.24
ER ($\lambda$) | 25.8∗∗∗ | -1.9 | -0.3 | 27.2$\%$ | 3.24
BER | 26.4∗∗∗ | -2.8 | -0.2 | 16.7$\%$ | 3.24
KD | 25.0∗∗∗ | -1.2 | +0.0 | 41.7$\%$ | 3.24
TABLE II: WER on Test set and Memory of initial task NL-main after the first adaptation to VL-main. ’Initial’ is the model trained on NL-main, from which the other models are adapted. Model | Test | Memory
---|---|---
Initial | 27.1 | 10.7
FT | 33.0 | 21.8
A-GEM | 31.0 | 2.6
ER | 32.7 | 0.0
ER ($\lambda$) | 30.6 | 0.4
BER | 32.3 | 13.9
KD | 29.4 | 10.5

First, we note that FT indeed suffers from CF, while both JT and CJT are able
learn the tasks well, with the latter reaching a positive BWT and FWT.
Considering the regularization-based methods, we find that the methods
estimating which parameters are important experience difficulties learning the
four tasks. This is especially true for EWC and MAS, both performing worse
than FT. While CSQN and CSQN-BT, by considering interactions between
parameters, perform slightly better, they still underperform FT. We
hypothesize that the poor performance of these methods is due to the tasks
(being very similar) having the same important parameters, which gives the
model two options: either it updates these parameters, resulting in CF of
previous tasks; or it leaves them unchanged, resulting in poor learning of the
new tasks. In this experiment, EWC, MAS and CSQN reduce forgetting, but fail
to learn the new tasks well. While EWC achieves the best BWT of all methods,
it also attains the worst FWT. Note that the performance of EWC is in line
with [20], which also found EWC underperforming FT. Compared to EWC, MAS and
CSQN, LWF performs much (and significantly, with $\alpha=0.001$) better,
achieving the highest FWT (higher than FT). However, its COV is only $12.4\%$,
as it reduces FT’s forgetting (BWT) by only $21\%$.
Comparing LWF to KD, which uses the same regularization but computed on the
memory instead of on the new task’s data, we find that having access to a
memory, even though it is only $0.6\%$ of the original data when learning the
fourth task, yields big improvements (with significance level $\alpha=0.001$).
KD attains a COV of over $40\%$, and learns the new tasks as well as FT, while
reducing the latter’s forgetting by more than $70\%$. It outperforms the other
rehearsal-based methods by a large margin (with significance level
$\alpha=0.001$). A-GEM, while it learns the new tasks well, still suffers from
severe forgetting, reaching a COV of $22\%$. This is again consistent with
[20], which found GEM (of which A-GEM is a more efficient variant)
outperforming LWF, while both improved the performance of FT. Finally, ER
performs worse than FT, as it suffers from CF and is unable to learn the new
task well. Both BER, and especially ER ($\lambda$), perform much better,
reaching a COV of $16.7\%$ and $27.2\%$, respectively.
Table II gives us more insight into how the rehearsal-based methods work. It
shows the WER on the memory and test set of the initial task NL-main of models
adapted to VL-main. For ER, we note that it memorizes the memory completely,
achieving 0.0 WER, and this generalizes very poorly to the test set. ER
($\lambda$) alleviates this, though it still almost perfectly memorizes the
memory set. A-GEM, too, has a very low WER on the memory set. On the other
hand, KD only very slightly improves on the memory set, but it is able to
extract much more ’general’ knowledge from it, limiting the forgetting on the
test set much better than ER, ER ($\lambda$) or A-GEM.

Figure 1: COV after learning the four tasks.

Figure 1 shows the COV after learning each task. We find that after two tasks,
LWF performs as well as A-GEM and ER ($\lambda$). However, as more tasks are
added, the gap between LWF, and A-GEM and ER ($\lambda)$ widens. Moreover,
while BER after two tasks only slightly outperforms FT, its performance,
relative to FT and the other CL methods, improves as more tasks are added.
This is as expected, since BER, learning on the merger of the training set and
the memory, clearly benefits from having a larger memory. Finally, note how
many CL methods’ performance drops when learning NL-rest. That is because NL-
main and NL-rest are very similar, so the CL methods should enable the model
to learn NL-rest, as this will also benefit NL-main, while protecting VL-main.
While this is a realistic scenario when extending a monolingual ASR model, it
turns out to be a very challenging one, especially for EWC, MAS and CSQN,
which, by protecting NL-main’s important parameters, are unable to exploit NL-
rest to further improve these parameters.

###  V-A Increasing vs. Fixed Memory

The rehearsal-based methods from Table I had access to a memory with 500
utterances per task. In practice, it may be more desirable and/or feasible to
have a memory with fixed size, especially as the number of tasks becomes
large. To this end, we fix the size of the memory at 500. Table III shows the
results for A-GEM, ER ($\lambda$) and KD.

TABLE III: Results after learning the four tasks with fixed memory. Significance level refers to deterioration compared to corresponding method from Table I (with increasing memory).  Model | AWER$\downarrow$ | BWT$\uparrow$ | FWT$\uparrow$ | COV$\uparrow$ | Storage
---|---|---|---|---|---
A-GEM | 26.2∗ | -2.8 | -0.0 | 18.9$\%$ | 1.72
ER ($\lambda$) | 25.6 | -1.6 | -0.4 | 29.8$\%$ | 1.72
KD | 25.2∗ | -1.5 | +0.1 | 38.3$\%$ | 1.72

We find that the differences with Table I are small. For A-GEM and KD, we
observe a slight (though significant, with $\alpha=0.05$) deterioration of
AWER. For ER ($\lambda$), on the other hand, there is even a minor
improvement, which, not being statistically significant, is attributed due to
chance. Even for a very small and fixed memory (with only $0.2\%$ of original
training data after four tasks), A-GEM and, in particular, ER ($\lambda$) and
KD are thus highly effective in enabling CL.

###  V-B Storage Requirements

Table I shows the storage requirements of the CL methods to learn the fourth
task. For all methods except JT, which for each new task starts from scratch,
this requires, first of all, the storage of the model itself. In addition, the
rehearsal-based methods require storing an equivalent of 2.24 models, as they
need to store the utterances in the memory.
Compared to the rehearsal-based methods, the regularization-based methods are
more storage efficient (in addition to not requiring data from previous tasks
to be stored in a memory, which may be, due to e.g. privacy concerns, not
always allowed). While LWF requires storing only the previous model, EWC and
MAS, in addition, need to store the importance weights. Compared to the
latter, CSQN and CSQN-BT are less storage efficient, due to the Hessian
approximations.
With an increasing memory, as in Table I, the rehearsal-based methods’ storage
requirements also increase linearly with the number of tasks. However, as we
saw in Table III, this can be overcome by fixing the memory size, with only a
negligible deterioration in performance, enabling A-GEM, and, especially, ER
($\lambda$) and KD, to achieve excellent performance while being very storage
efficient, requiring to store an equivalent of only 1.72 models (independent
of number of tasks). Finally, note how JT and CJT, needing access to all data
the model was ever trained on, require storing an equivalent of 260.54 and
261.54 models, respectively, making them clearly not a practical solution to
overcome CF.

##  VI Conclusion

In this paper, we implemented an extensive number of CL methods, and tested
and compared their ability to extend a monolingual E2E ASR model across four
tasks. Having access to a memory, though very small compared to the original
training set, proved to be very beneficial, as the rehearsal-based methods
generally performed much better than the regularization-based methods. To
assure the former’s storage requirements do not increase with the number of
tasks, the memory size can be fixed with only a negligible degradation in
performance. In general, thus, the rehearsal-based methods seem the best and
most practical way to currently overcome CF in monolingual E2E ASR models; in
particular KD, which closes the gap between the Fine-Tuned (lower bound) and
Continued Joint model (upper bound) for $41.7\%$ and $38.3\%$ while having
access to only $0.6\%$ and $0.2\%$, respectively, of the original data. In
case storing utterances from previous tasks is not allowed, LWF seems to be
the best option, as the other regularization-based methods, which have higher
storage requirements, were unable to improve the Fine-Tuning lower bound.
However, even in case only a very small number of utterances per task can be
stored, it is advised to do so.

## References

  * [1] Michael McCloskey and Neal J. Cohen,  “Catastrophic interference in connectionist networks: The sequential learning problem,”  vol. 24 of Psychology of Learning and Motivation, pp. 109–165. Academic Press, 1989.
  * [2] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars,  “A continual learning survey: Defying forgetting in classification tasks,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 1–1, 2021.
  * [3] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell,  “Overcoming catastrophic forgetting in neural networks,”  Proceedings of the National Academy of Sciences, vol. 114, no. 13, pp. 3521–3526, 2017.
  * [4] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars,  “Memory aware synapses: Learning what (not) to forget,”  in Computer Vision – ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, Eds., Cham, 2018, pp. 144–161, Springer International Publishing.
  * [5] Friedemann Zenke, Ben Poole, and Surya Ganguli,  “Continual learning through synaptic intelligence,”  in Proceedings of the 34th International Conference on Machine Learning - Volume 70. 2017, ICML’17, p. 3987–3995, JMLR.org.
  * [6] Zhizhong Li and Derek Hoiem,  “Learning without forgetting,”  in Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, Eds., Cham, 2016, pp. 614–629, Springer International Publishing.
  * [7] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean,  “Distilling the knowledge in a neural network,”  in NIPS Deep Learning and Representation Learning Workshop, 2015\.
  * [8] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne,  “Experience replay for continual learning,”  in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. 2019, vol. 32, Curran Associates, Inc.
  * [9] Zheda Mai, Hyunwoo J. Kim, Jihwan Jeong, and Scott Sanner,  “Batch-level experience replay with review for continual learning,”  ArXiv, vol. abs/2007.05683, 2020.
  * [10] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato,  “On tiny episodic memories in continual learning,”  arXiv: Learning, 2019.
  * [11] David Lopez-Paz and Marc’Aurelio Ranzato,  “Gradient episodic memory for continual learning,”  in Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2017, NIPS’17, p. 6470–6479, Curran Associates Inc.
  * [12] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny,  “Efficient lifelong learning with a-gem,”  in ICLR, 2019.
  * [13] Brady Houston and Katrin Kirchhoff,  “Continual Learning for Multi-Dialect Acoustic Models,”  in Proc. Interspeech 2020, 2020, pp. 576–580.
  * [14] Samik Sadhu and Hynek Hermansky,  “Continual Learning in Automatic Speech Recognition,”  in Proc. Interspeech 2020, 2020, pp. 1246–1250.
  * [15] Samuel Kessler, Bethan Thomas, and Salah Karout,  “Continual-wav2vec2: an application of continual learning for self-supervised automatic speech recognition,”  arXiv, 2021.
  * [16] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 12449–12460, Curran Associates, Inc.
  * [17] Amin Fazel, Wei Yang, Yulan Liu, Roberto Barra-Chicote, Yixiong Meng, Roland Maas, and Jasha Droppo,  “Synthasr: Unlocking synthetic data for speech recognition,”  CoRR, vol. abs/2106.07803, 2021.
  * [18] Li Fu, Xiaoxiao Li, and Libo Zi,  “Incremental learning for end-to-end automatic speech recognition,”  ArXiv, vol. abs/2005.04288, 2020.
  * [19] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa,  “Learning without memorizing,”  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5133–5141, 2019.
  * [20] Heng-Jui Chang, Hung yi Lee, and Lin shan Lee,  “Towards Lifelong Learning of End-to-End ASR,”  in Proc. Interspeech 2021, 2021, pp. 2551–2555.
  * [21] Shigeki Karita, Nelson Enrique Yalta Soplin, Shinji Watanabe, Marc Delcroix, Atsunori Ogawa, and Tomohiro Nakatani,  “Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration,”  in Proc. Interspeech 2019, 2019, pp. 1408–1412.
  * [22] Taku Kudo and John Richardson,  “SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,”  in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Brussels, Belgium, Nov. 2018, pp. 66–71, Association for Computational Linguistics.
  * [23] Ferenc Huszár,  “On quadratic penalties in elastic weight consolidation,”  ArXiv, vol. abs/1712.03847, 2017.
  * [24] Steven Vander Eeckt and Hugo Van hamme,  “Continual learning with quasi-newton methods,”  TechRxiv, Sep 2021.
  * [25] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai,  “ESPnet: End-to-end speech processing toolkit,”  in Proceedings of Interspeech, 2018, pp. 2207–2211.
  * [26] Nelleke Oostdijk,  “The spoken dutch corpus: Overview and first evaluation,”  Proceedings of LREC-2000, Athens, vol. 2, 01 2000.
  * [27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. 2017, vol. 30, Curran Associates, Inc.
  * [28] Helmer Strik, Catia Cucchiarini, and Judith M. Kessens,  “Comparing the recognition performance of csrs: in search of an adequate metric and statistical significance test,”  in INTERSPEECH, 2000.