# Meta-Learning Online Adaptation of Language Models

Nathan Hu* &Eric Mitchell*
Christopher D. Manning &Chelsea Finn
Stanford University

###### Abstract

Large language models encode surprisingly broad knowledge about the world into
their parameters. However, the knowledge in static language models can fall
out of date, limiting the model’s effective “shelf life.” While online fine-
tuning can reduce this degradation, we find that fine-tuning on a stream of
documents using standard optimizers such as Adam leads to a disappointingly
low level of information uptake. We hypothesize that online fine-tuning does
not sufficiently ‘attend’ to important information. That is, the gradient
signal from important tokens representing factual information is drowned out
by the gradient from inherently noisy tokens, suggesting a dynamic, context-
aware learning rate may be beneficial. To test this hypothesis, we meta-train
a small, autoregressive model to reweight the language modeling loss for each
token during online fine-tuning, with the objective of maximizing the out-of-
date base language model’s ability to answer questions about a document after
a single weighted gradient step. We call this approach Context-aware Meta-
learned Loss Scaling (CaMeLS). Across three different distributions of
documents, our experiments find that fine-tuning on streams of thousands of
documents with CaMeLS substantially improves knowledge retention compared to
standard online fine-tuning. Finally, we find that the meta-learned weights
are general, and that a single reweighting model can be used to enhance the
online adaptation of many LMs.

##  1 Introduction

††* Equal contribution. Correspondence to<EMAIL_ADDRESS><EMAIL_ADDRESS>
Large language models learn impressively broad world knowledge through large-
scale unsupervised pre-training, which they can leverage for a wide variety of
downstream tasks (Brown et al., 2020; Chowdhery et al., 2022; Bubeck et al.,
2023). However, large language models are typically static artifacts, and as
the world changes, the knowledge encoded in their parameters becomes stale.
While retrieval-augmented models are one approach to mitigating the staleness
issue, even very large language models often fail to correctly update their
memorized predictions when presented with counterfactual retrieved information
(Longpre et al., 2021; Li et al., 2022; Si et al., 2023). Moreover, purely
parametric LMs are uniquely suited for edge computing due their compact size
(relative to a large retrieval index) and simplicity of inference (Gerganov,
2023). Recent work has thus considered variants of online fine-tuning on a
stream of documents to efficiently perform direct updates to the knowledge
inside of a large language model (Lazaridou et al., 2021; Jang et al., 2022).

Figure 1: We study the setting of a language model being adapted unsupervised
(without annotation of important tokens) on an online stream of documents, and
being later evaluated on queries (e.g., questions) about those documents.
Downstream inputs are not provided during the adaptation phase, requiring the
model to integrate as much information as possible about the documents.

Ideally, we could simply fine-tune a language model on an online stream of
documents and the information contained in those documents would be readily
available for the model to use for a variety of downstream tasks, for example
answering questions about the information in the documents. In other words,
can a language model simply ‘read the news’ as it occurs by simply fine-tuning
on the stream of news articles? Unfortunately, we find that in this online
adaptation setting, fine-tuning with a well-tuned learning rate leads to a
nearly negligible improvement in a question-answering model’s ability to
answer questions about the stream of documents. We hypothesize that naive
fine-tuning is not effective in the online adaptation setting because the
negative log likelihood (NLL) loss does not accurately reflect the importance
of a token; i.e., tokens containing important factual information may receive
relatively small NLL and therefore a small fine-tuning gradient. For example,
consider the NLL of the word Rishi and Next in the phrase ‘The UK Prime
Minister is Rishi Sunak. Next year…’ for a slightly out-of-date language
model. Because Rishi Sunak was a well-known politician before becoming Prime
Minister, a model may place reasonably high probability mass on his name (even
if other completions are higher probability). On the other hand, ‘Next’ will
invariably receive low probability, because the distribution over the first
word in a sentence is unavoidably high entropy.

As a remedy, we propose Context-aware Meta-learned Loss Scaling (CaMeLS), an
approach to online adaptation that meta-trains an importance-weighting model
to identify the tokens in a document whose gradient yields the most productive
knowledge update to the model being fine-tuned. The importance-weighting model
is trained to re-weight the per-token NLL within the document for a small
proxy LM, such that the proxy LM assigns high probability to the answers to
questions about the document after just a single step on the re-weighted NLL.
During online fine-tuning on a stream of documents, we simply re-weight the
online loss by the importance-weighting model’s output. We find that
intuitively, the importance weighting model learns to up-weight many nouns,
proper nouns, and numbers; however, we find that whether or not a token is up-
weighted depends on the context; a baseline that weights by part of speech
produces inferior performance (Figure 6). Across three online adaptation
benchmark datasets based on streams of news and Wikipedia articles, CaMeLS
substantially improves knowledge acquisition over naive fine-tuning, which
typically provides little to no improvement in question-answering accuracy
after online adaptation. Crucially, the meta-learned importance weights
generalize across models; we meta-train our importance weighting model once
using DistilGPT-2 (Sanh et al., 2019) as the proxy LM and directly use our
weighting model to update GPT-Neo-2.7B, without modification.

Figure 2: Comparing per-token gradient norms when fine-tuning GPT-2-large on
two example passages with typical uniformly-weighted NLL (top row) vs. CaMeLS
(bottom row). CaMeLS sparsifies the gradient, with most tokens having roughly
zero gradient norm, but a significant number (typically key entities, events,
or numbers) being heavily up-weighted. Per-token weights are learned without
token-level annotation of key facts. We transform gradient norm by
$\log(1+\|g\|)$ to enhance readability.

##  2 Related Work

Adapting to new data or task distributions is typically studied in the context
of continual or lifelong learning (Thrun and Mitchell, 1995; Mitchell et al.,
2018). Continual learning in deep networks involves the challenge of
simultaneously avoiding catastrophic forgetting (McCloskey and Cohen, 1989),
the process under which a neural network’s performance on old tasks or data is
dramatically degraded by the process of learning new information, while
maintaining plasticity (Dohare et al., 2022), or the ability to adapt to the
latest change in the data distribution, even after many changes have already
been experienced. While most work in continual learning considers sequences of
supervised data (Kirkpatrick et al., 2016; Lopez-Paz and Ranzato, 2017; Shin
et al., 2017; Chaudhry et al., 2019), some work also studies continual few-
shot (Ren et al., 2021) or unsupervised learning (Rao et al., 2019; Madaan et
al., 2022), which is closer to the setting in this paper. However, these works
typically focus on streams of visual data.

In the context of language models, dynamic, or streaming, language models were
first considered in the context of n-gram language models, combining a cache
of recently-used words to update the predictive probabilities of a tri-gram
model (Kuhn, 1988; Jelinek et al., 1991; Osborne et al., 2014), with later
work describing online EM-based algorithms for efficiently updating n-gram
models (Yogatama et al., 2014). Other studies explore the evolution of
decontextualized word embeddings over time (Kulkarni et al., 2015; Hamilton et
al., 2016) as a result of temporal shifts in the use of language and vector
memories to store recent information when training recurrent neural networks
online (Rei, 2015). More recently, several studies have explored methods for
updating large neural language models, typically through online fine-tuning on
a stream of documents (Lazaridou et al., 2021) with architectural constraints
(Jang et al., 2022) or explicit conditioning on time (Dhingra et al., 2022)
used as strategies to reduce forgetting of old information. Clark et al.
(2022) use meta-learning to reduce the compute requirements of online fine-
tuning. However, recent work suggests that while increasing the size of
language models may largely mitigate the problem of forgetting old information
(Driess et al., 2023), improving the efficiency of acquisition of new
knowledge is still a challenge, and is the focus of the present work. Other
methods for dynamically updating parametric the knowledge in parametric
language models develop specialized techniques, called model editors, designed
to make targeted edits to individual facts (Sinitsin et al., 2020; De Cao et
al., 2021; Mitchell et al., 2021; Hase et al., 2021; Meng et al., 2022) or
behaviors (Mitchell et al., 2022). However, model editors assume human access
to annotations of the tokens or facts that must be updated; in this work, we
emphasize the challenge of learning which tokens in an unlabeled sequence of
documents are important.

##  3 Meta-Learning Improved Online Adaptation of Large Language Models

Given an out-of-date language model and a stream of recent documents, we aim
to update the model such that it effectively answers typical queries about the
documents in the stream. For example, for a document in the stream about the
discovery of a new galaxy close to the Milky Way, typical queries may be
things like ‘Where was the discovery made?’ and ‘Is the new galaxy closer to
us than Andromeda?’, while an atypical query might be ‘How many times do words
with an even number of letters follow words beginning with a vowel in the
document?’ By focusing only on retaining knowledge relevant to the ‘typical’
queries, we avoid the need to completely memorize the documents, making the
problem feasible. In this section, we describe this problem setting more
formally, a then describe an approach to this setting, Context-aware Meta-
learned Loss Scaling.

###  3.1 Unsupervised Online Adaptation

We consider a setting in which an out-of-date language model
$f_{\theta_{base}}$ is updated with an online stream222$D_{\text{test}}$ is
typically an online stream of documents, but an arbitrary ordering over a
static collection of documents is also possible. of recent documents
$D_{\text{test}}=\\{x_{i}\\}$, ultimately producing an updated model
$f_{\theta^{\prime}}$. The updated model $f_{\theta^{\prime}}$ is then
evaluated with a set of queries $Q_{\text{test}}=\\{q_{i}\\}$ with labels
$Y_{\text{test}}=\\{y_{i}\\}$, where the the $i$th query is drawn from a
distribution of queries relating to $i$th document: $q_{i},y_{i}\sim
p(q_{i},y_{i}|x_{i})$. For example, $q_{i}$ may be a question about some
information in document $x_{i}$, and $y_{i}$ the answer to that question
implied by the document. Crucially, when using $D_{\text{test}}$ to update
$f_{\theta_{base}}$, we do not have access to $Q_{\text{test}}$. Thus, our
methodology for updating $f_{\theta_{base}}$ must be broad rather than query
specific. In order to make this problem tractable (i.e., not requiring
complete memorization of the document stream), we assume that we have an
additional corpus of documents $D_{\text{train}}$ and corresponding query
samples $Q_{\text{train}}$ and labels $Y_{\text{train}}$ generated by a
similar generative process to $Q_{\text{test}},Y_{\text{test}}$. This training
set enables learning the types of queries that may be of interest, thereby
informing how we should update our model to maximize the performance on test
queries while minimizing disturbance to its prior knowledge or behaviors. We
next describe an algorithm for leveraging this dataset to more efficiently
update our base model on the test stream of documents $D_{\text{test}}$ than
standard fine-tuning.

###  3.2 CaMeLS: Context-aware Meta-learned Loss Scaling

The goal of CaMeLS is to distill the information in the training documents,
queries, and labels into a parameter $\phi$ that summarizes how best to update
a base model on a document stream to maximize retention of information likely
to be relevant to test queries. CaMeLS accomplishes this goal by training a
_weighting model_ $w_{\phi}$ (a small autoregressive language model) that re-
weights the online negative log likelihood (NLL) loss used in typical online
fine-tuning, focusing on the tokens whose NLL gradient is most useful for
updating a small proxy base model’s knowledge. In other words, the weighting
model is trained to re-weight the NLL loss such that the proxy model correctly
answers questions about the document after performing a single gradient step
on the weighted NLL. The weighting model is trained with an episodic bi-level
optimization, which we explain next in detail (also see Figure 3).

Figure 3: A single step of CaMeLS meta-training. In step 1, the weighting
model produces a set of importance weights over the tokens in a given
document. In step 2, a single gradient step on the weighted NLL produces an
adapted model. In step 3, the weighting model is updated to improve the
adapted base model’s ability to answer questions about the document.

During each episode, a training document-query-label triple $(x,q,y)$ is
sampled from $D_{\text{train}}$ and a locality example from $x_{\text{loc}}$
from $D_{\text{loc}}$. $D_{\text{loc}}$ is a dataset of unlabeled text
representing the distribution over which we want the base model’s behavior to
remain generally unchanged. For all experiments, we use the OpenWebText
dataset (Gokaslan et al., 2019) as $D_{\text{loc}}$. Let
$\theta_{\text{base}}$ denote the parameters of the proxy base model at the
start of the episode. The update to the weighting model involves three steps:
1) computing the weights for the training document, 2) updating a small proxy
base model on the weighted NLL on the training document, and 3)
backpropagating the ‘outer loop’ loss333Performing a single update to the
proxy model is the inner loop and updating the weighting model according to
the updated proxy model’s loss on the query is considered the outer loop of a
bi-level optimization used to train CaMeLS. of the updated proxy model on a
query and label from the training document. These steps are shown in Figure 3.
Let $L(f_{\theta},x,\mathbf{a})$ denote the weighted NLL of $f_{\theta}$ on
document $x$ using weights $\mathbf{a}$. Steps 1 & 2 are described by the
inner loop update rule:

$\theta^{\prime}=\theta_{\text{base}}-\alpha\nabla_{\theta_{\text{base}}}L(f_{\theta_{\text{base}}},x,w_{\phi}(x))$
(1)

The inner loop learning rate $\alpha$ can fixed, sampled, or learned. After
the updated proxy model is computed, we compute an outer loop loss measuring
the effectiveness of the weighted adaptation procedure on the document $x$:

$L_{\text{outer}}=-\log
p_{\theta^{\prime}}(y|q)+c_{\text{loc}}L_{\text{loc}}(\theta_{\text{base}},\theta^{\prime},x_{\text{loc}})$
(2)

In addition to the negative log likelihood of label given the query and
updated base model parameters, the outer loss has a locality term
$L_{\text{loc}}$ which prevents the updated base model parameters from
excessively changing the base model’s behavior. $c_{\text{loc}}$ is set to
$.1$ for all experiments. $L_{\text{loc}}$ is the KL divergence between the
base model before and after adaptation conditioned on the locality input
$x_{\text{loc}}$:

$L_{\text{loc}}(\theta_{\text{base}},\theta^{\prime},x_{\text{loc}})=\\\
\text{KL}\left(p_{\theta_{\text{base}}}(\cdot|x_{\text{loc}})\|p_{\theta^{\prime}}(\cdot|x_{\text{loc}})\right)$
(3)

Finally, we perform a single update to the weighting model’s parameters by
computing the gradient of the outer loop loss with respect to $\phi$. We
optimize $\phi$ with the Adam optimizer, using a learning rate of 1e-5. We
accumulate outer loop gradients over 24 examples (document-query-label
triples).

Dataset | Average text length | Sampled texts per test stream | Tokens per test stream
---|---|---|---
StreamingQA |  $\sim$510 tokens | 1665 articles |  $\sim$850K
SQuAD |  $\sim$150 tokens | 1170 paragraphs |  $\sim$175K
ArchivalQA |  $\sim$80 tokens | 3001 paragraphs |  $\sim$240K
Table 1: Basic statistics of the data in our online document streams. The sample text streams used to evaluate online adaptation vary in length. | SQA | SQuAD | ArchivalQA
---|---|---|---
Split | $N_{x/q}$ | $N_{x}$ | $N_{q}$ | $N_{x}$ | $N_{q}$
Train | 21k | 8.6k | 39.9k | 12.8k | 21.7k
Validation | 1.7k | 1.2k | 5.6k | 3.0k | 5.3k
Test | 5k | 2.1k | 10.6k | 5.0k | 8.7k
QA Train | 40k | - | 40k | - | 12.4k
QA Valid. | 4k | - | 2.1k | - | 3k
Table 2: Number of documents $N_{x}$ and questions $N_{q}$ for each dataset.
Each document in StreamingQA (SQA) corresponds to a single question, while
SQuAD and ArchivalQA contain documents corresponding to multiple questions.

Mitigating train-test shift. The single-step training procedure described
above optimizes for effective knowledge retention for a single document.
However, in our online adaptation setting, we may update for hundreds or
thousands of documents before we evaluate on our downstream queries. In order
to mitigate this train-test shift, we modify CaMeLS with two strategies:
First, we do not use the same base model parameters during each episode of
training. This is done in order to disincentivize the weighting model from
overfitting to a single base model state. For most training episodes, the
starting base model parameters adapted in the inner loop are the final base
model parameters in the previous episode. Every $c_{\text{reset}}=4$ episodes
of training, the starting base model parameters are reset to that of the
original base model. Second, we sample an inner batch of $k$ document-query-
label triples $(x_{1},q_{1},y_{1}),\dots,(x_{k},q_{k},y_{k})$ for each
episode. A sequence of $k$ inner loop updates are performed:

$\displaystyle\theta_{i}=\theta_{i-1}-\alpha\nabla_{\theta}L(f_{\theta_{i-1}},x_{i},w_{\phi}(x_{i}))$
(4)

where $\theta_{0}=\theta_{base}$ and $\theta^{\prime}=\theta_{k}$. The outer
loss is computed as before, except now averaging the query-label loss over the
sampled inner batch. By allowing inner loop updates to accumulate during
adaptation, $\phi$ will learn an updating strategy which preserves the
knowledge of prior updates and maintains the base models ability to be updated
by subsequent updates.

Compute Requirements of CaMeLS. Optimizing bi-level objectives like the one
used by CaMeLS is rather memory and compute-intensive, requiring memory and
compute proportional to the depth of the inner loop (the batch size used for
multiple inner loop updates) and proportional to the size of our base model -
each inner loop step creates an updated copy of the base model parameters in
the computation graph. However, CaMeLS only requires a lightweight base model;
our experiments use DistilGPT-2 as the base model during meta-training, but we
find strong transfer to much larger base models during evaluation. The
weighting model itself is also small; all experiments use DistilGPT-2 as the
weighting model (a MLP with a single hidden state of size 128 is used as the
head to produce token weights). Using the base and weighting models described,
we are able to train weighting models using 6 inner loop steps on a single
NVIDIA A40 GPU. 444Preliminary weight models trained without multiple inner
loop steps and with an outer learning rate of 1e-4 are used in SQuAD and
ArchivalQA experiments.

##  4 Experiments

We now perform experiments aimed at understanding CaMeLS’s behavior in
unsupervised online adaptation. Section 4.3 studies the extent to which
CaMeLS’s importance weights improve knowledge retention in online adaptation.
Section 4.4 qualitatively and quantitatively explores the weights themselves,
suggesting several ablations of CaMeLS that we explore in Section 4.5.
Finally, we examine the forgetting and plasticity dynamics of CaMeLS within
the document stream in Section 4.6.

###  4.1 Datasets

We apply CaMeLS to three question answering datasets with corresponding source
articles. We partition the datasets into 5 splits. Three of these splits
(train, valid, test) are used for training, hyperparameter tuning, and
evaluating the CaMeLS weighting model. In order to fine-tune the initial QA
base models from generic language models, we reserve two more disjoint splits
(in order to prevent reusing questions during initial QA tuning and online
adaptation), labeled QA train and QA valid. Split sizes are shown in Table 2.
At evaluation time, a stream of documents is sampled from the test split. The
number of documents sampled and the text stream length are shown in Table 1.
In the StreamingQA setting, models must adapt to an entire article as opposed
to a selected paragaph, making it our most challenging setting.

StreamingQA (Liška et al., 2022): The StreamingQA dataset contains a
combination of human-written and language model generate questions. Questions
are generated from English WMT news articles published between 2007 and 2020.
Each question is paired with an entire WMT news article. Average article
length is approximately 500 tokens. Question-article pairs from after 2018 are
used for the meta-learning train, validation, and test sets while questions
from before 2015 are used for the two QA pretraining sets.

SQuAD (Rajpurkar et al., 2016): The Stanford Question Answering Dataset
(SQuAD) contains human generated questions from Wikipedia articles. The answer
to each question is a sub span of the article. Each question is paired with a
paragraph of wikipedia text. Average paragraph length is approximately 150
tokens. For our experiments, we use the validation set of SQuAD as our test
split and form our other 4 splits by dividing the train set of SQuAD.

ArchivalQA (Wang et al., 2022): The ArchivalQA dataset contains automatically
generated questions. Questions are generated from articles in the New York
Times Annotated Corpus (Sandhaus, Evan, 2008). The answer to each question is
a sub span of the article. Each question is paired with the paragraph of the
NYT article. The validation set of ArchivalQA is divided to form the 5 splits
we use.

Figure 4: CaMeLS’s meta-learned weights improve knowledge uptake after online
language model adaptation on a stream of data. The F1 score of the base model
before and after adaptation with CaMeLS are computed on questions about the
documents used for adaptation. The relative change in F1 is plotted. Top,
lower left, and lower right show StreamingQA, SQuAD, and ArchivalQA datasets,
respectively. Error bars are standard error over 4 sampled streams of test
data.

###  4.2 Experimental protocol details

We conducted evaluations on two families of autoregressive language models,
the GPT-2 (Radford et al., 2018) and GPT-Neo families (Black et al., 2021). We
note that all models evaluated use the same text tokenization.

For each dataset, we first fine-tune each pretrained model on question-answer
pairs from that dataset. These tuned models represent the static language
models we wish to update and will be referred to as base models. For each
dataset, a single weighting model is trained. The proxy language model used
during weighting model training is DistilGPT-2 fine-tuned on the QA train
split of the respective dataset.

During each evaluation, the base model is updated on a stream of documents
sampled from the test split. The final adapted base model is evaluated on the
questions corresponding to the documents in the sampled stream. In our
experiments, we find that models fine-tuned with uniform weighting benefit
from additional tuning for question answering after adaptation, essentially to
‘re-learn’ the task of question answering. Therefore, we include a phase of QA
fine-tuning after adaptation as well, using the same data as the initial QA
fine-tuning phase. However, base models updated using CaMeLS do not need this
‘QA post-tuning’ (essentially, another benefit of CaMeLS), so we only perform
this post-tuning phase for the uniform models. See Figure 8 for an empirical
evaluation of the effects of this post-tuning.

Figure 5: CaMeLS produces weights that sparsify the fine-tuning gradient.
Middle row shows the raw weights output by CaMeLS. Like Figure 2, top and
bottom rows show raw and weighted gradient norms, respectively.

For each combination of base model and online adaptation strategy, the
learning rate used at test time was chosen via hyper parameter sweep on a
stream of documents sampled from the validation set. The best performing
learning rate on the validation set was then used for test runs.

###  4.3 Fine-tuning with learned weights improves knowledge retention

Our first experiment compares the knowledge retained by online fine-tuning and
CaMeLS for three different data distributions. Figure 4 shows the results.
CaMeLS outperforms online fine-tuning consistently across a range of datasets
and weighting models. Despite the difference in scale between the proxy model
used during weight training and evaluated base models, CaMeLS’s learned
importance weights generalize well to the largest base model we evaluate, GPT-
Neo 2.7B, which is 30 times the size of the proxy model (DistilGPT-2, 82
million parameters) used during training. We find that standard online fine
tuning (uniform weighting) with Adam performs very poorly on online adaption.
Even when using a tuned learning rate and further training for question
answering post adaptation, uniform weighting fails to achieve a significant
relative improvement and can even lead to a decrease in F1 on GPT-Neo 1.3B.
For completeness, Appendix Figure 10 shows the performance of online fine
tuning without QA adaption.

###  4.4 Analysis of learned weights

Figure 6: The distribution of importance weights on the validation split of
StreamingQA, showing both the overall distribution (left) and the
distributions conditioned by part of speech (right). The importance weight
distribution learned by CaMeLS is bimodal, with proper nouns and numbers being
the parts of speech most likely to have high importance weights.

One benefit of CaMeLS over other methods for meta-learning loss functions is
that weights are interpretable. In this section, we provide additional
analysis of CaMeLS’s learned weights. Figure 5 shows the per-token weights for
a sample text, as well as how they combine with the unweighted gradient norms
to produce sparsified per-token gradient norms. Next, we examine the
distribution of weighting model outputs on articles in the validation set of
StreamingQA in Figure 6. As our qualitative evaluations show, we confirm that
the distribution of weights over the entire validation split of StreamingQA is
indeed sparse and bimodal; while nearly 20% of the learned weights are larger
than 200, 79.7% percent of sampled weights are smaller than 1. We thus
interpret the weighting model as acting as a context-aware binary classifier,
determining if a token is informative or uninformative. When binning weights
by part of speech, we find that numbers and proper nouns are most frequently
assigned a high weight. This is consistent with the results of Lazaridou et
al. (2021), which showed that proper nouns and numbers are the parts of speech
where a temporally stale language model’s performance decreases the fastest
over time.

Figure 7: Ablations of CaMeLS. Bimodal Ablation restricts the weighting model
from outputting intermediate values while the POS ablations remove context
dependence, conditioning only on part of speech. While restricting CaMeLS to
output only one of two values only slightly reduces performance, conditioning
only on part of speech, rather than full context, drastically reduces
knowledge retention.

###  4.5 Ablations

In order to verify if context-aware weights are truly necessary to achieving
improved knowledge retention, we now examine several ablations of CaMeLS. In
the Bimodal ablation, outputs of the weighting model are rounded to either the
largest or smallest value in the distribution of importance weights. In the
POS: Resample ablation, the weight of each token is generated by sampling from
the distribution of importance weights on all tokens of the same part of
speech. Finally, in the POS: Mean ablation, each token is weighted by the mean
importance weight assigned to tokens of the same part of speech.

Figure 7 shows the results on the StreamingQA dataset for two different base
models. We observe that ablating the weighting model to only output two values
slightly reduces performance, while still achieving significant F1 improvement
and outperforming uniform weighting. The strong performance of the binary
ablation suggests that a binary decision of whether to train on a given token
is an effective approach to online adaptation, though the full version of
CaMeLS that allows for variation in the weight magnitude still performs best.

In contrast, neither part-of-speech ablation produces effective knowledge
retention, either performing worse than the uniform baseline or failing to
significantly increase F1 score. This result strongly suggests that although
part of speech correlates strongly with learned weights, part of speech alone
is not sufficient to determine when a token contains important information.
Thus, we conclude that context-awareness is indeed helpful for identifying
important tokens in online adaptation.

###  4.6 Forgetting and plasticity

Figure 8:  Base model performance on the downstream question answering task is
evaluated during online adaptation. GPT2-XL is adapted to 3000 articles from
StreamingQA. Base model performance is evaluated every 200 article updates.
Online adaptation via CaMeLS results in a gradual increase in base model
performance. Gains from uniform fine-tuning are only realized after a post-
adaptation QA tuning step.

So far, our evaluations have considered only the QA accuracy at the end of
online adaptation. In this section, we investigate the evolution of learning
during the online adaptation process. Figure 8 compares the exact match
accuracy and F1 score of CaMeLS and standard fine-tuning on the entire set of
evaluation queries after every 200 document updates, using GPT-2 (1.5B) as the
base model. Interestingly, CaMeLS consistently improves along both metrics
during online adaptation, while uniform fine-tuning gradually degrades along
both metrics during the course of fine-tuning. However, by adding a phase of
QA ‘post-tuning’ to the model adapted with standard fine-tuning, we do indeed
recover some improvement over the initial model. On the other hand, we find
that post-tuning does not improve CaMeLS’s performance. Taken together, these
results suggest that the CaMeLS updates do less ‘damage’ to the model’s
representation of the task, making more precise updates to the model’s
knowledge. For standard fine-tuning, knowledge gained during online adaptation
may come at the expense of forgetting the task.

Figure 9: We examine base model performance as a function of updates since
adaptation on the source document. The shaded region represents the interval
in which the source document was presented.

Finally, in Figure 9, we aim to answer the questions how long does the model
remember the answer to a question after observing it? We show the average
improvement in exact match accuracy in relation to the timestep when the model
observes the document containing the information relevant to answering a
question. After the 200 document sequence containing the relevant document,
both methods see a clear average improvement in exact match accuracy,
signifying learning is happening. However, we also note that CaMeLS produces
both a higher initial improvement in EM accuracy as well as a higher
asymptotic improvement in EM accuracy. That is, CaMeLS both improves the
immediate plasticity of the model, integrating knowledge more readily, but
also reduces forgetting, preserving the newly-integrated knowledge for longer.

##  5 Discussion

While large language models are powerful, keeping them up-to-date with an
ever-changing world remains a challenge. In this paper, we consider the
unsupervised online language model adaptation setting, in which a language
model’s knowledge must be updated using a stream of documents, without
annotations of key facts or information. Finding that naive online fine-tuning
provides little retention of knowledge from the document stream, we propose
Context-aware Meta-learned Loss Scaling (CaMeLS), a meta-learning algorithm
that learns an importance weighting model to reweight the per-token loss of
the online data stream. CaMeLS leverages side information of the form of
paired documents and knowledge queries about those documents to identify which
tokens in the documents are most likely to be informative for answering
downstream queries. Empirically, we find that the importance weighting model
learned by CaMeLS consistently improves knowledge retention across three
datasets of documents and questions. Crucially, we find that CaMeLS’s
importance weighting model generalizes across outdated language models,
meaning that an importance weighting model trained on a small proxy language
model (such as DistilGPT-2) can immediately be used to improve online
adaptation of much larger models, like GPT-Neo (2.7B).

Limitations & Future Work. While our experiments suggest that learned
importance weights consistently improve knowledge retention after unsupervised
online adaptation, our study has several limitations. CaMeLS assumes access to
side information in the form of training document, query, and label triples.
This requirement may be onerous in domains where labeling is expensive. Future
work may apply CaMeLS to settings without access to side information queries
and labels, i.e., only a purely unlabeled stream of training documents, using
the temporal structure of the data as the signal for learning. Additionally,
while our experiments show that CaMeLS weights generalize across base models,
we do not explore the extent to which they generalize across data
distributions, which is a valuable direction of study for future work (i.e.,
training and evaluating weighting models across diverse data distributions).
Beyond dataset scale, our experiments study adaptation of base models up to
2.7B parameters, but recent work suggests the continual learning dynamics of
language models changes drastically at extreme scale (100B+ parameters);
future work may increase the scale of the present study by considering larger
weighting models, proxy base models, and base evaluation models. Finally, we
study only question-answering as a downstream task, as it is the most direct
form of knowledge retention assessment. Future work may examine knowledge
retention through alternative downstream tasks that leverage the knowledge in
the document stream more indirectly.

## Acknowledgements

The authors thank Huaxiu Yao for his input at multiple stages of the project.
CF and CM are CIFAR Fellows. EM gratefully acknowledges funding from a Knight-
Hennessy Graduate Fellowship. This research was supported in part by Juniper
Networks.

## References

  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with GPT-4.  ArXiv preprint arXiv:2303.12712.
  * Chaudhry et al. (2019) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019.  Efficient lifelong learning with a-GEM.  In _International Conference on Learning Representations_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Clark et al. (2022) Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Mohammad Norouzi. 2022.  Meta-learning fast weight language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9751–9757, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * De Cao et al. (2021) Nicola De Cao, W. Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  _ArXiv_ , abs/2104.08164.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022.  Time-Aware Language Models as Temporal Knowledge Bases.  _Transactions of the Association for Computational Linguistics_ , 10:257–273.
  * Dohare et al. (2022) Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. 2022.  Continual backprop: Stochastic gradient descent with persistent randomness.
  * Driess et al. (2023) Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. 2023.  Palm-e: An embodied multimodal language model.  In _arXiv preprint arXiv:2303.03378_.
  * Gerganov (2023) Georgi Gerganov. 2023.  llama.cpp.  https://github.com/ggerganov/llama.cpp.
  * Gokaslan et al. (2019) Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. 2019.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Hamilton et al. (2016) William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016.  Diachronic word embeddings reveal statistical laws of semantic change.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1489–1501, Berlin, Germany. Association for Computational Linguistics.
  * Hase et al. (2021) Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021.  Do language models have beliefs? Methods for detecting, updating, and visualizing model beliefs.  Arxiv:2111.13654.
  * Jang et al. (2022) Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo. 2022.  Towards continual knowledge learning of language models.  In _International Conference on Learning Representations_.
  * Jelinek et al. (1991) F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. 1991.  A dynamic language model for speech recognition.  In _Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991_.
  * Kirkpatrick et al. (2016) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114.
  * Kuhn (1988) Roland Kuhn. 1988.  Speech recognition and the frequency of recently used words: A modified Markov model for natural language.  In _Coling Budapest 1988 Volume 1: International Conference on Computational Linguistics_.
  * Kulkarni et al. (2015) Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015.  Statistically significant detection of linguistic change.  In _Proceedings of the 24th International Conference on World Wide Web_ , WWW ’15, page 625–635, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _Advances in Neural Information Processing Systems_.
  * Li et al. (2022) Wei Li, Wenhao Wu, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022.  Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods.  _ArXiv_ , abs/2203.05227.
  * Liška et al. (2022) Adam Liška, Tomáš Kočiský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022.  Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models.
  * Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.  Entity-based knowledge conflicts in question answering.  _arXiv preprint arXiv:2109.05052_.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30.
  * Madaan et al. (2022) Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. 2022.  Representational continuity for unsupervised continual learning.  In _International Conference on Learning Representations_.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J. Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Gordon H. Bower, editor, _Psychology of Learning and Motivation_ , volume 24 of _Psychology of Learning and Motivation_ , pages 109–165. Academic Press.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in GPT.  ArXiv:2202.05262.
  * Mitchell et al. (2021) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021.  Fast model editing at scale.  _CoRR_.
  * Mitchell et al. (2022) Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022.  Memory-based model editing at scale.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 15817–15831. PMLR.
  * Mitchell et al. (2018) T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, B. Yang, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. 2018.  Never-ending learning.  _Commun. ACM_ , 61(5):103–115.
  * Osborne et al. (2014) Miles Osborne, Ashwin Lall, and Benjamin Van Durme. 2014.  Exponential reservoir sampling for streaming language models.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 687–692, Baltimore, Maryland. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018.  Language models are unsupervised multitask learners.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.
  * Rao et al. (2019) Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. 2019.  Continual unsupervised representation learning.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Rei (2015) Marek Rei. 2015.  Online representation learning in recurrent neural language models.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 238–243, Lisbon, Portugal. Association for Computational Linguistics.
  * Ren et al. (2021) Mengye Ren, Michael Louis Iuzzolino, Michael Curtis Mozer, and Richard Zemel. 2021\.  Wandering within a world: Online contextualized few-shot learning.  In _International Conference on Learning Representations_.
  * Sandhaus, Evan (2008) Sandhaus, Evan. 2008.  The new york times annotated corpus.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  In _NeurIPS EMC 2 Workshop_.
  * Shin et al. (2017) Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017.  Continual learning with deep generative replay.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Si et al. (2023) Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2023.  Prompting GPT-3 to be reliable.  In _The Eleventh International Conference on Learning Representations_.
  * Sinitsin et al. (2020) Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. 2020.  Editable neural networks.  In _ICLR_.
  * Thrun and Mitchell (1995) Sebastian Thrun and Tom M. Mitchell. 1995.  Lifelong robot learning.  _Robotics and Autonomous Systems_ , 15(1):25–46.  The Biology and Technology of Intelligent Autonomous Agents.
  * Wang et al. (2022) Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022.  Archivalqa: A large-scale benchmark dataset for open domain question answering over historical news collections.
  * Yogatama et al. (2014) Dani Yogatama, Chong Wang, Bryan R. Routledge, Noah A. Smith, and Eric P. Xing. 2014\.  Dynamic language models for streaming text.  _Transactions of the Association for Computational Linguistics_ , 2:181–192.

## Appendix A: Additional Figures

Below we expand Figure 4 show performance of uniform weighting before post
adaptation tuning for question answering.

Figure 10: Standard fine tuning needs a post adapataion QA step to be
competitive. Top, lower left, and lower right show StreamingQA, SQuAD, and
ArchivalQA datasets, respectively. Error bars are standard error over 4
sampled streams of test data.