# Demystifying Forgetting in Language Model Fine-Tuning with Statistical
Analysis of Example Associations

Xisen Jin  Xiang Ren
University of Southern California
{xisenjin<EMAIL_ADDRESS>
###### Abstract

Language models (LMs) are known to suffer from forgetting of previously
learned examples when fine-tuned, breaking stability of deployed LM systems.
Despite efforts on mitigating forgetting, few have investigated whether, and
how forgotten upstream examples are associated with newly learned tasks.
Insights on such associations enable efficient and targeted mitigation of
forgetting. In this paper, we empirically analyze forgetting that occurs in
$N$ upstream examples while the model learns $M$ new tasks and visualize their
associations with a $M\times N$ matrix. We empirically demonstrate that the
degree of forgetting can often be approximated by simple multiplicative
contributions of the upstream examples and newly learned tasks. We also reveal
more complicated patterns where specific subsets of examples are forgotten
with statistics and visualization. Following our analysis, we predict
forgetting that happens on upstream examples when learning a new task with
matrix completion over the empirical associations, outperforming prior
approaches that rely on trainable LMs. Project website:
https://inklab.usc.edu/lm-forgetting-prediction/

Demystifying Forgetting in Language Model Fine-Tuning with Statistical
Analysis of Example Associations

Xisen Jin and Xiang Ren University of Southern California {xisenjin,
<EMAIL_ADDRESS>

##  1 Introduction

There has been a growing need for long-term usability of LM systems which
require incorporating new tasks, domains, and knowledge Mitchell et al.
(2022); Ke et al. (2023); Qin et al. (2022); Gupta et al. (2023); Hartvigsen
et al. (2024). While fine-tuning allows models to learn new knowledge
incrementally, it risks catastrophic forgetting McCloskey and Cohen (1989);
Luo et al. (2023) of past learned knowledge, causing unintended prediction
changes over known information. This is problematic for stable online deployed
LM systems, limiting the feasibility of continual fine-tuning in practice.

Extensive works have studied algorithms to mitigate forgetting Shi et al.
(2024). Some works analyze patterns of frequently forgotten upstream examples
Toneva et al. (2019); Maini et al. (2022), effects of models and
hyperparameters Ibrahim et al. (2024), and how forgetting scales with models
Mirzadeh et al. (2022); Kalajdzievski (2024). However, few have studied how
the association between learned and upstream examples informs the forgetting
of upstream examples.

In this paper, we analyze the statistics of forgetting (in log perplexity
increase) over $N$ past upstream examples, when the model learns $M$ new tasks
separately. We represent the associations above with a $M\times N$ matrix and
examine the associations between learned and forgotten examples with
statistical models. We experiment with OLMo-7B and OLMo-7B-Instruct Groeneveld
et al. (2024) models where upstream pre-training data is released open-source.
We fine-tune LMs (with LoRA Hu et al. (2022)) on new instruction tuning tasks
and measure forgetting on upstream pre-training corpora (of language modeling)
or instruction tuning datasets.

Our findings suggest that (1) certain new tasks or upstream examples
consistently cause or suffer more forgetting, and the degree of forgetting can
be well-approximated with multiplicative contributions of the two; (2) there
exist more complicated associations revealed through visualization and
statistics. Following our analysis, we propose to predict example forgetting
on unseen tasks as a matrix completion problem analogical to collaborative
filtering Sarwar et al. (2001) in recommender system literature. Our
$k$-nearest neighbor (KNN) based prediction outperforms prior approaches that
rely on trainable LMs Jin and Ren (2024). We verify the benefit of targeted
mitigation of forgetting on OLMo-7B on upstream pre-training corpora as it
learns new instruction-tuning tasks.

(a) OLMo-7B; forgetting over Dolma corpus after fine-tuning on each task in
Tulu.

(b) OLMo-7B-Instruct; forgetting over Tulu examples after fine-tuning on each
task in MMLU, BBH, and TruthfulQA.

Figure 1: Visualization of forgetting. Each pixel indicates forgetting (log-
perplexity increase) that occurs on an upstream example (in $x$-axis) after
learning a new task (in $y$-axis). Log-perplexity increase can be zero or
negative, which implies no forgetting.

To summarize, the contributions of this paper are (1) empirical analysis on
how forgotten examples are associated with learned tasks, and (2) a novel view
of predicting example forgetting as a matrix completion problem.

##  2 Problem and Evaluation Setup

We define forgetting $z_{ij}$ as degradation (increase) in log perplexity on
an upstream example $x_{j}\in x_{1..N}$ after a LM learns a new task (set of
examples) $T_{i}\in T_{1..M}$. We evaluate forgetting on $N$ upstream examples
when the model learns $M$ tasks separately and record forgetting $z_{ij}$ in a
matrix $Z\in\mathbb{R}^{M\times N}$. We experiment with OLMo-7B and
OLMo-7B-Instruct where pre-training data and instruction-tuning data are
released.

OLMo-7B. OLMo models are pre-trained on Dolma Soldaini et al. (2024), a
massive collection of cleaned text. We fine-tune LMs over Tulu V2 instruction
data Ivison et al. (2023), a diverse collection of NLP tasks, one task at a
time. We then evaluate forgetting on a subset of Dolma examples as the
upstream examples.

OLMo-7B-Instruct. OLMo-7B-Instruct models are instruction-tuned on Tulu V2. In
our experiments, we further fine-tune OLMo-7B-Instruct over new task data from
MMLU Hendrycks et al. (2021), BBH Suzgun et al. (2022), and TruthfulQA Lin et
al. (2022), and evaluate forgetting on tasks in Tulu v2 as the upstream
examples.

We leave other details of the datasets in Appendix A. We visualize the
matrices $Z$ in Figure 1, where each item $z_{ij}$ indicates forgetting that
happens on $x_{j}$ after learning $T_{i}$. Aligning well with prior study, we
see certain upstream examples suffer more from forgetting (more red) Toneva et
al. (2019); Swayamdipta et al. (2020); similarly, certain tasks cause more
forgetting (e.g., tasks from TruthfulQA). Next, we analyze how learned and
forgotten examples are associated.

##  3 Association between Learned and Forgotten Examples

In this section, we dive into an overlooked research question on how forgotten
examples and newly learned tasks are associated. In a hypothetical extreme
case, the association can be very simple, e.g., where upstream examples suffer
the same amount of forgetting regardless of the new task; or it can be highly
complicated, e.g., where different new tasks cause very specific subsets of
upstream examples to be forgotten. To understand the associations, we start by
visualizing the association matrix $Z$, and perform quantitative analysis as
we approximate $Z$ with different types of models with different inductive
bias about associations.

###  3.1 Analysis on Visualized Association Matrix

We see from Figure 1 that the rows and columns in $Z$ display a generally neat
and simple pattern. Notably, many rows and columns appear to be multiplication
of the others. Among them, some columns are always close to zero regardless
the rows (e.g. science upstream examples in Figure 1(b)), as if the columns
(upstream examples) are “zeroing out” the effect of rows (learned tasks).
Nevertheless, in some cases, upstream examples that are never forgotten
elsewhere are forgotten by learning specific new tasks (e.g. wizardlm and
open-orca examples are almost only forgotten after learning tasks from
TruthfulQA in Figure 1(b)). It implies the association is a mixture of simple
and more complicated patterns.

###  3.2 How Well Can Simple Models Approximate Forgetting?

| OLMo-7B | OLMo-7B-Instruct
---|---|---
Additive |  |
Upstream $x$ only | 0.2700 | 0.0751
Task $T$ only | 0.3423 | 0.5837
$x+T$ | 0.6123 | 0.6588
SVD |  |
$r=1$ | 0.6537 | 0.7418

Table 1: $R^{2}$ of fitting the association matrices $Z$ of forgetting.
$R^{2}\to 1$ implies better fit.

We quantitatively measure how well $Z$ can be approximated with simple
regression models with different inductive bias. Specifically, we measure
$R^{2}$, a common metric for determining how well a regression model fits
data. Let $f_{ij}$ be the fitted value, $R^{2}$ is defined as
$1-\sum_{i,j}(z_{ij}-f_{ij})^{2}/\sum_{i,j}(z_{ij}-\bar{Z})^{2}$. A higher
$R^{2}$ (closer to 1) indicates better fit.

Models. We consider (1) additive linear models, where
$z_{ij}=b+\alpha_{i}+\beta_{j}+\epsilon$, where $\alpha_{i}$ and $\beta_{i}$
are learnable parameters associated with each new task or upstream example.
(2) multiplicative models (SVD with rank $r$=1), where
$z_{ij}=s\alpha_{i}\beta_{j}+\epsilon$.

Results. We report $R^{2}$ of different models in Table 1. We see on both OLMo
and OLMo-Instruct, the multiplicative model achieves better fit than additive
models at the same number of trainable parameters. We note that multiplicative
models are more suitable for situations where some upstream examples or new
tasks almost never experience or inflict perplexity changes
($\alpha_{i},\beta_{j}\approx 0$), which is indeed a dominant pattern from the
visualization of $Z$. The models achieve $R^{2}$ of 0.65 and 0.74 in two
setups ($i.e.,$ explaining 65% and 74% of total variance in $Z$). Although the
fit is decent considering the simplicity of the models, there remains around
30% of variance not explained. Therefore, we examine more complicated
associations between learned tasks and forgotten examples.

###  3.3 What are More Complicated Associations between Learned & Forgotten
Examples?

Figure 2: Reconstruction of $Z$ in OLMo-Instruct experiments with $k$-th
singular value and vectors. Higher values of $k$ capture finer-grained details
in $Z$.

To visualize more complicated associations in $Z$, we reconstruct $Z$ using
$k$-th singular value and vectors. For higher values of $k$, they capture
progressively finer-grained associations between learned and forgotten
examples. Figure 2 provides a visualization of the reconstructed $Z$ for OLMo-
Instruct at different levels.

Interpretation of Associations. We examine and interpret reconstructions at
different levels. When $k$=1, the reconstruction captures the dominant simple
multiplicative association we examined in Sec. 3.2. For $k$=2, we see it
captures that learning TruthfulQA tasks cause substantial forgetting, but many
upstream examples are still unaffected (highlighted with all-zero columns).
For $k$=3, the columns and rows become much sparser, showing a smaller subset
of examples that share the common fine-grained associations. We include
distribution of singular values and $R^{2}$ for higher $k$ in Appendix C.

Summary. We found that forgetting can be well-approximated by the
multiplicative effects of learned and upstream examples. We believe this
conclusion will apply to other setups with a similar diversity of learned
tasks. We also showed matrix decomposition reveals more fine-grained
associations. Although the revealed associations are specific to the dataset,
we believe the method of analysis can bring insights to other setups.

##  4 Predicting Example Forgetting with Matrix Completion

Model | OLMo-7B | OLMo-7B-Instruct
---|---|---
Test Data | FLAN (In-domain) | Tulu \ FLAN (OOD) | MMLU+BBH (In-domain) | TruthfulQA (OOD)
Metrics | RMSE (10-2) $\searrow$ | RMSE (10-2) $\searrow$ | RMSE (10-2) $\searrow$ | F1 $\nearrow$ | RMSE (10-2) $\searrow$ | F1 $\nearrow$
Additive | 3.56±0.04 | 4.86±0.11 | 9.46±0.05 | 35.12±4.5 | 22.06±0.26 | 64.95±3.9
SVD | 5.74±0.02 | 6.53±0.09 | 9.42±0.09 | 37.32±1.6 | 22.15±0.31 | 66.43±2.7
KNN | 3.45±0.05 | 4.89±0.12 | 7.60±0.09 | 73.03±0.2 | 20.96±0.21 | 67.34±3.5
Rep-dot | 5.45±0.00 | 7.29±0.00 | 8.23±0.00 | 62.20±0.0 | 26.19±0.01 | 66.26±0.0

Table 2: Performance of predicting example forgetting. We report standard
deviation over different sets of upstream examples with known ground truth
forgetting ($\mathcal{S}$) beforehand.

Our analysis inspires us with a novel view of predicting example forgetting as
a matrix completion problem. This is especially useful for targeted mitigation
of forgetting. Unlike prior works Jin and Ren (2024) that use an LM to encode
contents of upstream examples and new tasks for prediction, we attempt to rely
solely on example associations in $Z$, without viewing the contents in
examples.

Our goal is to accurately predict forgetting $z_{ij}$ over upstream examples
$x_{1..M}$ when the model learns an unseen task. To evaluate this, we split
the set of tasks into training, in-domain test, and out-of-domain test splits.
We use MMLU and BBH (OLMo-Instruct) and FLAN (OLMo) as in-domain tasks and use
TruthfulQA and other tasks in Tulu as out-of-domain tasks. We assume knowing
the ground truth forgetting $z_{ij}$ of a tiny random set $\mathcal{S}$
($|\mathcal{S}|=30$) of upstream examples; our goal is to predict forgetting
of the rest $10k-100k$ upstream examples. Details are discussed in Appendix B.

Methods. We run matrix completion algorithms including additive linear models,
SVD, and k-nearest neighbors (KNN), commonly used for collaborative filtering.
We also compare with Rep-dot Jin and Ren (2024) which maps inner products of
trained LM encoding of upstream and learned examples to forgetting.

Results of Predicting Example Forgetting. Table 2 summarizes the results of
predicting example forgetting. We report Root Mean Squared Error (RMSE) of
predicting the log-perplexity increase $z_{ij}$. For OLMo-Instruct, we
additionally report F1 score of predicting whether $z_{ij}>0$. We see KNN
models consistently outperforms additive linear, SVD, and trainable
representation-based prediction methods across different models and setups.

Mitigating Forgetting with Predicted Forgetting. We empirically verify the
practical utility of predicting forgetting with KNN. As the model learns new
tasks, we replay a small subset of $M$ upstream examples after each
optimization step, prioritizing those with higher predicted forgetting, based
on $\exp(z_{ij})$ given by the KNN model. Table 3 summarizes the results. We
see by upweighting we reduce perplexity (mitigates forgetting) compared to
replaying random examples though not statistical significantly. We expect the
improvement to be more significant in other setups where replaying based on
ground truth $\exp(z_{ij})$ brings statistically significant improvements.

| Log PPL |  $p$-value
---|---|---
No Replay | 2.3092 | -
Replay Random | 2.2747 | -
Replay w/ KNN | 2.2730 | 0.1600
Replay w/ Ground Truth | 2.2711 | 0.1207

Table 3: Log perplexity over upstream data (Dolma) after fine-tuning OLMo on
FLAN tasks (in-domain test split). $p$-value is computed against replaying
random examples (paired $t$-test on 20 tasks).

##  5 Related Works

In this paper, we analyzed forgetting in LMs, a crucial challenge that arises
while adapting LMs to new data Jang et al. (2022); Meng et al. (2022); Cohen
et al. (2023). Related to our work, data attribution study algorithms to seek
examples or tasks that account for a prediction Koh and Liang (2017); Ilyas et
al. (2022) when models are trained jointly on multiple examples or tasks. We
instead focus on analysis of associations when learning new tasks separately.
There exist works identifying memorized Feldman and Zhang (2020); Tirumala et
al. (2022); Biderman et al. (2024b) or important training data for a task Xia
et al. (2024); Swayamdipta et al. (2020), but few analyze how these statistics
depend on the learned tasks. Ye et al. (2023); Xia et al. (2020); Schram et
al. (2023) study prediction of task performance from training setups; we
perform prediction at the example-level which is more fine-grained and under-
explored.

##  6 Conclusions

In this paper, we empirically analyzed the associations between learned and
forgotten examples in LM fine-tuning. We showed forgetting can be well-
approximated with multiplicative contributions of upstream and learned
examples and visualized more complicated associations. We further show the
example associations alone offer useful information to predict example
forgetting when fine-tuning LMs on new tasks.

## Limitations

Joint Analysis with Other Factors that Affect Forgetting. In this paper, we
primarily focused on analyzing the association between learned and forgotten
examples. The association has not been studied rigorously with other factors
that are known to influence forgetting of upstream examples. We list some
other factors that can be studied jointly with ours: (1) type and size of the
LM Mehta et al. (2021); Scialom et al. (2022); Kalajdzievski (2024); Mirzadeh
et al. (2022) (2) trainable parts of the model (e.g., LoRA, soft prompts, or
full-model tuning) Biderman et al. (2024a); Razdaibiedina et al. (2023) (3)
hyperparameters such as learning rate Ibrahim et al. (2024); Winata et al.
(2023), dropout Goodfellow et al. (2014), number of training steps Biderman et
al. (2024b); Kleiman et al. (2023) (4) optimizer Lesort et al. (2023) and
training algorithms (e.g., various continual learning algorithms) Shi et al.
(2024); Wu et al. (2024).

Mechanical Interpretation of Example Associations. We focused on empirical
statistics of forgetting in this paper while treating the LM as a black box.
We believe research on mechanical interpretation of forgetting Tao et al.
(2023); Zhao et al. (2023); Jin and Ren (2024) is complementary to ours and
can potentially explain in the future (1) why the associations in $Z$ are
often simple, and (2) in which circumstances the associations become more
complicated.

## Ethical Considerations

Our research aims to mitigate forgetting of general knowledge in LMs while
fine-tuning on new tasks, sharing motivations with many prior works. However,
we note that forgetting is not always undesirable. In fact, recent works have
shown importance of updating outdated knowledge Jang et al. (2022); Zhao et
al. (2024); Garg et al. (2024) or unlearning harmful content Ginart et al.
(2019) in LMs. Future works can extend our analysis by studying whether the
forgetting of upstream examples are desirable or not.

## Acknowledgements

Xisen Jin is supported by Apple Scholars in AI/ML PhD Fellowship Program. We
thank Héctor Martínez Alonso for insightful discussions.

## References

  * Biderman et al. (2024a) Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. 2024a.  Lora learns less and forgets less.  _ArXiv preprint_ , abs/2405.09673.
  * Biderman et al. (2024b) Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2024b.  Emergent and predictable memorization in large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Cohen et al. (2023) Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2023.  Evaluating the ripple effects of knowledge editing in language models.  _Transactions of the Association for Computational Linguistics_ , 12:283–298.
  * Feldman and Zhang (2020) Vitaly Feldman and Chiyuan Zhang. 2020.  What neural networks memorize and why: Discovering the long tail via influence estimation.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Garg et al. (2024) Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. 2024.  Tic-CLIP: Continual training of CLIP models.  In _The Twelfth International Conference on Learning Representations_.
  * Ginart et al. (2019) Antonio Ginart, Melody Y. Guan, Gregory Valiant, and James Zou. 2019.  Making AI forget you: Data deletion in machine learning.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 3513–3526.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Mehdi Mirza, Xia Da, Aaron C. Courville, and Yoshua Bengio. 2014.  An empirical investigation of catastrophic forgeting in gradient-based neural networks.  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, A. Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hanna Hajishirzi. 2024.  Olmo: Accelerating the science of language models.  _ArXiv preprint_ , abs/2402.00838.
  * Gupta et al. (2023) Kshitij Gupta, Benjamin Th’erien, Adam Ibrahim, Mats L. Richter, Quentin G. Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. 2023.  Continual pre-training of large language models: How to (re)warm your model? _ArXiv preprint_ , abs/2308.04014.
  * Hartvigsen et al. (2024) Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2024.  Aging with grace: Lifelong model editing with discrete key-value adaptors.  _Advances in Neural Information Processing Systems_ , 36.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Ibrahim et al. (2024) Adam Ibrahim, Benjamin Th’erien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. 2024.  Simple and scalable strategies to continually pre-train large language models.  _ArXiv preprint_ , abs/2403.08763.
  * Ilyas et al. (2022) Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. 2022.  Datamodels: Predicting predictions from training data.  _ArXiv preprint_ , abs/2202.00622.
  * Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hanna Hajishirzi. 2023.  Camels in a changing climate: Enhancing lm adaptation with tulu 2.  _ArXiv preprint_ , abs/2311.10702.
  * Jang et al. (2022) Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022.  Towards continual knowledge learning of language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Jin and Ren (2024) Xisen Jin and Xiang Ren. 2024.  What will my model forget? forecasting forgotten examples in language model refinement.  _ArXiv preprint_ , abs/2402.01865.
  * Kalajdzievski (2024) Damjan Kalajdzievski. 2024.  Scaling laws for forgetting when fine-tuning large language models.  _ArXiv preprint_ , abs/2401.05605.
  * Ke et al. (2023) Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bin Liu. 2023.  Continual pre-training of language models.  In _International Conference on Learning Representations_.
  * Kleiman et al. (2023) Anat Kleiman, Jonathan Frankle, Sham M. Kakade, and Mansheej Paul. 2023.  Predicting task forgetting in large language models.  In _ICML 2023 Workshop DeployableGenerativeAI homepage_.
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding black-box predictions via influence functions.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1885–1894. PMLR.
  * Lesort et al. (2023) Timothée Lesort, Oleksiy Ostapenko, Pau Rodríguez, Diganta Misra, Md Rifat Arefin, Laurent Charlin, and Irina Rish. 2023.  Challenging common assumptions about catastrophic forgetting and knowledge accumulation.  In _Conference on Lifelong Learning Agents_ , pages 43–65. PMLR.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  TruthfulQA: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  * Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023.  An empirical study of catastrophic forgetting in large language models during continual fine-tuning.  _ArXiv preprint_ , abs/2308.08747.
  * Maini et al. (2022) Pratyush Maini, Saurabh Garg, Zachary Lipton, and J. Zico Kolter. 2022.  Characterizing datapoints via second-split forgetting.  In _Advances in Neural Information Processing Systems_ , volume 35.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier.
  * Mehta et al. (2021) Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2021.  An empirical investigation of the role of pre-training in lifelong learning.  _J. Mach. Learn. Res._ , 24:214:1–214:50.
  * Meng et al. (2022) Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2022.  Mass-editing memory in a transformer.  In _The Eleventh International Conference on Learning Representations_.
  * Mirzadeh et al. (2022) Seyed-Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Görür, and Mehrdad Farajtabar. 2022.  Wide neural networks forget less catastrophically.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 15699–15717. PMLR.
  * Mitchell et al. (2022) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022.  Fast model editing at scale.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Qin et al. (2022) Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022.  ELLE: Efficient lifelong pre-training for emerging data.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2789–2810, Dublin, Ireland. Association for Computational Linguistics.
  * Razdaibiedina et al. (2023) Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. 2023.  Progressive prompts: Continual learning for language models.  In _The Eleventh International Conference on Learning Representations_.
  * Sarwar et al. (2001) Badrul Munir Sarwar, George Karypis, Joseph A. Konstan, and John Riedl. 2001.  Item-based collaborative filtering recommendation algorithms.  In _Proceedings of the Tenth International World Wide Web Conference, WWW 10, Hong Kong, China, May 1-5, 2001_ , pages 285–295. ACM.
  * Schram et al. (2023) Viktoria Schram, Daniel Beck, and Trevor Cohn. 2023.  Performance prediction via Bayesian matrix factorisation for multilingual natural language processing tasks.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1790–1801, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Scialom et al. (2022) Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. 2022.  Fine-tuned language models are continual learners.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6107–6122, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Shi et al. (2024) Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, and Hao Wang. 2024.  Continual learning of large language models: A comprehensive survey.  _ArXiv preprint_ , abs/2404.16789.
  * Soldaini et al. (2024) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, A. Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hanna Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. 2024.  Dolma: an open corpus of three trillion tokens for language model pretraining research.  _ArXiv preprint_ , abs/2402.00159.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. 2022.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _ArXiv preprint_ , abs/2210.09261.
  * Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9275–9293, Online. Association for Computational Linguistics.
  * Tao et al. (2023) Mingxu Tao, Yansong Feng, and Dongyan Zhao. 2023.  Can BERT refrain from forgetting on sequential tasks? a probing study.  In _The Eleventh International Conference on Learning Representations_.
  * Tirumala et al. (2022) Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022.  Memorization without overfitting: Analyzing the training dynamics of large language models.  _ArXiv preprint_ , abs/2205.10770.
  * Toneva et al. (2019) Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. 2019.  An empirical study of example forgetting during deep neural network learning.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Winata et al. (2023) Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preoţiuc-Pietro. 2023.  Overcoming catastrophic forgetting in massively multilingual continual learning.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 768–777.
  * Wu et al. (2024) Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. 2024.  Continual learning for large language models: A survey.  _ArXiv preprint_ , abs/2402.01364.
  * Xia et al. (2020) Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020.  Predicting performance for natural language processing tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8625–8646, Online. Association for Computational Linguistics.
  * Xia et al. (2024) Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024.  Less: Selecting influential data for targeted instruction tuning.  _ArXiv preprint_ , abs/2402.04333.
  * Ye et al. (2023) Qinyuan Ye, Harvey Yiyun Fu, Xiang Ren, and Robin Jia. 2023.  How predictable are large language model capabilities? a case study on big-bench.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Zhao et al. (2024) Bowen Zhao, Zander Brumbaugh, Yizhong Wang, Hanna Hajishirzi, and Noah A. Smith. 2024.  Set the clock: Temporal alignment of pretrained language models.  _ArXiv_ , abs/2402.16797.
  * Zhao et al. (2023) Haiyan Zhao, Tianyi Zhou, Guodong Long, Jing Jiang, and Chengqi Zhang. 2023.  Does continual learning equally forget all parameters? In _International Conference on Machine Learning_.

##  Appendix A Dataset, Model, and LM Training Details

Plan for Code and Data Release. We plan to release code and statistics (e.g.,
the association matrix $Z$) upon acceptance.

Models. We use OLMo-7B111https://huggingface.co/allenai/OLMo-7B of the version
pretrained on Dolma v1.6; and
OLMo-7B-Instruct222https://huggingface.co/allenai/OLMo-7B-Instruct, which is
tuned on Tulu v2 and other human feedback datasets.

Upstream Examples. We summarize the list of upstream datasets in Table 4. We
also include the number of training examples in each dataset, initial log
perplexity, and average forgetting occured on these datasets.

We examine forgetting on Dolma in our OLMo-7B experiments. We sample 1% of
text chunks of length 2,048 from v1.6-sample version of the dataset, resulting
in 141,816 chunks of length 2,048. We compute log perplexity over all 2,048
tokens in each example. We examine forgetting on Tulu V2 in our
OLMo-7B-Instruct experiments. We randomly sample an approximately balanced
number of examples from each task in Tulu, and filter out examples with input
length that exceeds 2,048 (the limit of OLMo models) after tokenization. This
results in 10,718 examples. We compute log perplexity on ground truth output
tokens only.

Learned New Tasks. We summarize the list of newly learned tasks in Tables 5
and 6 for OLMo-7B and OLMo-7B-Instruct experiments. We also include the number
of training examples and forgetting caused by each task averaged over all
upstream examples.

Training and Evaluation Details. We use LoRA fine-tuning with rank=64 in all
our experiments, using AdamW optimizer and a constant learning rate of
$10^{-4}$. The statistics of forgetting are obtained in a single run. For
OLMo-7B and training on Tulu tasks, we train the models for 5,000 steps with
an effective batch size of 8 (625 steps of parameter updates in total). For
OLMo-7B-Instruct and MMLU, BBH, TruthfulQA, considering the small size of the
training sets, we train the models only for 300 steps with an effective batch
size of 8 (37 steps of parameter updates in total). We use HuggingFace
Transformers library for training and VLLM library for efficient inference. We
run training and inference on a single Quadro RTX A6000 GPU.

Dataset Licenses and Safety. MMLU and BBH are released under MIT license.
Truthful QA, Dolma, and OLMo models are released under Apache 2.0 license.
Tulu V2 is released under ODC-By license. We thank Soldaini et al. (2024) for
removing personally identifiable information from the massive corpus, Dolma,
before release as described in the original manuscript. The other datasets do
not contain personally identifiable information according to our inspection.

Task | #. Examples | Init. Log PPL | Avg. Forgetting
---|---|---|---
flan_v2 | 995 | 0.506 | -0.006
cot | 1000 | 0.347 | -0.001
oasst1 | 1000 | 1.117 | -0.043
lima | 946 | 1.931 | -0.045
gpt4_alpaca | 1000 | 0.693 | -0.018
code_alpaca | 1000 | 0.402 | -0.073
sharegpt | 976 | 0.940 | -0.049
wizardlm | 1979 | 0.693 | -0.025
open_orca | 995 | 4.346 | -0.336
science | 687 | 0.322 | -0.021
hard_coded | 140 | 2.682 | -0.353
Dolma | 141816 | 2.283 | 0.035

Table 4: Upstream tasks used in our OLMo-7B-Instruct and OLMo-7B experiments
where forgetting is evaluated. We also report the number of training examples,
log perplexity of these examples before fine-tuning (Init. Log PPL) and
averaged forgetting happened on these examples averaged after learning
different new tasks (Avg. Forgetting).

##  Appendix B Details of Forgetting Prediction and Replay

Data Splits for Predicting Example Forgetting. We mark the tasks used as in-
domain and out-of-domain test splits for predicting example forgetting (Sec.
4) in Tables 5 and 6.

Training and Evaluation Details. We use Surprise Library
1.1.3333https://github.com/NicolasHug/Surprise/tree/v1.1.3 for additive
linear, SVD, and KNN prediction models. We use default hyperparameter
configurations except for increasing the number of training epochs to 100. We
use mean squared error as the optimization objective.

For in-domain and out-of-domain test splits, we randomly sample 30 upstream
examples and assume the ground truth forgetting is known for these examples.
This is required for predicting forgetting on the rest of upstream examples by
additive linear, SVD, and KNN methods. We repeat the experiment 10 times and
report the mean and standard deviation in Table 2.

We used OLMo-1B models as the trainable example encoders in the implementation
of the prediction method by Jin and Ren (2024) (Rep-dot) that relies on inner
products of trained example representations. We notice these models trained
with mean squared error objective perform poorly on F1 metrics. Therefore, for
F1 metrics reported for Rep-dot in Table 2, we used a variant using cross-
entropy as the optimization objective. At inference, given an upstream
example, we compute the averaged dot-product with all examples in the learned
task. We note that at inference time Rep-dot does not require ground truth
forgetting of a small number of examples. For a fair comparison with other
matrix completion methods, we replace the prediction of Rep-dot with ground
truth forgetting on these examples.

Figure 3: Reconstruction of $Z$ in OLMo experiments with $k$-th singular value
and vectors. Higher values of $k$ capture finer-grained details in $Z$.

(a) OLMo-7B

(b) OLMo-7B-Instruct

Figure 4: Singular values (bars) and $R^{2}$ (dash lines) of reconstruction of
$Z$ with up to $k$-th singular value and singular vectors.

Replaying Upstream Examples in Fine-Tuning. We replay 1 upstream example every
1 parameter update while fine-tuning on new tasks. Given predicted or ground
truth forgetting $z_{i,1..J}$ on upstream examples $x_{1..J}$ when learning a
new task $T_{i}$, we sample upstream examples to replay from a categorical
distribution where $p(x_{j})\propto\exp(z_{i,j}/\tau)$, where $\tau$ is a
temperature hyperparameter set as 0.2. The hyperparameter $\tau$ is tuned on a
single validation task by using ground truth forgetting $Z$.

##  Appendix C Additional Results about Example Associations

We visualize progressive reconstruction with $k$-th singular value and
singular vectors for OLMo experiments in Figure 3. We see when $k=2$, there is
a single row and column with significantly larger forgetting than the others.
This pattern exemplifies a complicated association that is not captured by the
simple multiplicative model ($k=1$).

We further show the distribution of singular values and $R^{2}$ of
reconstruction of $Z$ in our OLMo and OLMo-Instruct experiments in Figure 4.

Task Category | Task | #. Examples | Avg. Forgetting Caused | Task Category | Task | #. Examples | Avg. Forgetting Caused
---|---|---|---|---|---|---|---
FLAN V2 | aeslc | 28860 | 0.012 |  | quac | 60000 | 0.070
| ag_news_subset | 60000 | 0.020 |  | record | 60000 | 0.030
| anli_r1 | 33880 | 0.018 |  | rte | 4580 | 0.034
| anli_r2 | 60000 | 0.022 |  | samsum | 29460 | 0.016
| anli_r3 | 60000 | 0.023 |  | sentiment140 | 60000 | 0.012
| arc_challenge* | 1820 | 0.027 |  | snli | 59900 | 0.027
| arc_easy* | 4080 | 0.027 |  | squad_v1 | 60000 | 0.067
| bool_q | 18440 | 0.056 |  | squad_v2 | 60000 | 0.148
| cb* | 400 | 0.037 |  | sst2 | 60000 | 0.024
| cnn_dailymail | 60000 | 0.007 |  | story_cloze | 3340 | 0.032
| cola | 16700 | 0.040 |  | stsb | 11280 | 0.031
| common_gen | 60000 | 0.037 |  | trec* | 10500 | 0.026
| copa | 700 | 0.043 |  | trivia_qa* | 60000 | 0.018
| coqa* | 14180 | 0.134 |  | true_case | 58520 | 0.053
| cosmos_qa | 50120 | 0.046 |  | web_nlg_en | 60000 | 0.048
| dart | 60000 | 0.033 |  | wic* | 10440 | 0.046
| definite_pronoun_resolution* | 2240 | 0.015 |  | wiki_lingua_english_en | 60000 | 0.015
| drop | 60000 | 0.045 |  | wmt14_enfr | 60000 | 0.017
| e2e_nlg* | 60000 | 0.046 |  | wmt16_translate_csen | 60000 | 0.009
| fix_punct* | 56140 | 0.046 |  | wmt16_translate_deen | 60000 | 0.011
| gigaword | 32240 | 0.011 |  | wmt16_translate_fien | 60000 | 0.013
| glue_mrpc | 6920 | 0.059 |  | wmt16_translate_roen | 60000 | 0.015
| glue_qqp* | 60000 | 0.032 |  | wmt16_translate_ruen* | 60000 | 0.014
| hellaswag | 60000 | 0.027 |  | wmt16_translate_tren* | 60000 | 0.017
| imdb_reviews | 49600 | 0.013 |  | wnli | 1200 | 0.024
| math_dataset* | 60000 | 0.043 |  | word_segment | 60000 | 0.107
| mnli_matched | 60000 | 0.057 |  | wsc* | 1000 | 0.016
| mnli_mismatched | 60000 | 0.066 |  | yelp_polarity_reviews* | 60000 | 0.013
| multi_news | 60000 | 0.010 | Others $\dagger$ | open_orca | 29683 | 0.009
| multirc | 54080 | 0.058 |  | oasst1 | 7331 | 0.005
| natural_questions* | 60000 | 0.010 |  | lima | 1018 | 0.194
| openbookqa* | 9900 | 0.046 |  | code_alpaca | 20016 | 0.015
| opinion_abstracts_idebate* | 3300 | 0.024 |  | gpt4_alpaca | 19906 | 0.016
| opinion_abstracts_rotten_tomatoes | 6260 | 0.008 |  | cot | 49747 | 0.019
| para_crawl_enes | 60000 | 0.018 |  | science | 7468 | 0.022
| paws_wiki | 60000 | 0.063 |  | sharegpt | 111912 | 0.010
| piqa | 32020 | 0.037 |  | hard_coded | 140 | 0.056
| qnli* | 60000 | 0.043 |  | wizardlm | 29810 | 0.019

Table 5: The list of learned tasks in our experiments on OLMo-7B. We also
include the number of training examples in each task (#. Example) and
forgetting caused by each learned task averaged over all upstream examples. *
notes for tasks used as the in-domain test split in forgetting prediction
experiments in Sec. 4. $\dagger$ notes for task collections used as the out-
of-domain test split.

Task Category | Task | #. Examples | Avg. Forgetting Caused | Task Category | Task | #. Examples | Avg. Forgetting Caused
---|---|---|---|---|---|---|---
MMLU | abstract_algebra | 11 | -0.030 | BBH | boolean_expressions* | 125 | -0.095
| anatomy | 14 | -0.076 |  | causal_judgement | 93 | -0.119
| astronomy | 16 | -0.074 |  | date_understanding | 125 | -0.122
| business_ethics | 11 | -0.042 |  | disambiguation_qa | 125 | -0.086
| clinical_knowledge | 29 | -0.093 |  | dyck_languages* | 125 | -0.090
| college_biology* | 16 | -0.069 |  | formal_fallacies* | 125 | -0.087
| college_chemistry | 8 | -0.088 |  | geometric_shapes | 125 | -0.045
| college_computer_science | 11 | -0.057 |  | hyperbaton* | 125 | -0.093
| college_mathematics | 11 | -0.065 |  | logical_deduction_five_objects* | 125 | -0.092
| college_medicine* | 22 | -0.072 |  | logical_deduction_seven_objects | 125 | -0.089
| college_physics | 11 | -0.058 |  | logical_deduction_three_objects | 125 | -0.116
| computer_security | 11 | -0.080 |  | movie_recommendation* | 125 | -0.068
| conceptual_physics* | 26 | -0.087 |  | multistep_arithmetic_two | 125 | -0.081
| econometrics | 12 | -0.043 |  | navigate | 125 | -0.067
| electrical_engineering | 16 | -0.108 |  | object_counting* | 125 | -0.106
| elementary_mathematics | 41 | -0.098 |  | penguins_in_a_table | 73 | -0.122
| formal_logic | 14 | -0.039 |  | reasoning_about_colored_objects | 125 | -0.134
| global_facts* | 10 | -0.012 |  | ruin_names | 125 | -0.098
| high_school_biology* | 32 | -0.098 |  | salient_translation_error_detection | 125 | -0.077
| high_school_chemistry | 22 | -0.096 |  | snarks | 89 | -0.125
| high_school_computer_science | 9 | -0.059 |  | sports_understanding | 125 | -0.113
| high_school_european_history* | 18 | -0.084 |  | temporal_sequences | 125 | -0.099
| high_school_geography | 22 | -0.090 |  | tracking_shuffled_objects_five_objects | 125 | -0.049
| high_school_government_and_politics | 21 | -0.057 |  | tracking_shuffled_objects_seven_objects | 125 | -0.100
| high_school_macroeconomics | 43 | -0.111 |  | tracking_shuffled_objects_three_objects | 125 | -0.091
| high_school_mathematics | 29 | -0.075 |  | web_of_lies | 125 | -0.066
| high_school_microeconomics | 26 | -0.070 |  | word_sorting | 125 | -0.124
| high_school_physics* | 17 | -0.067 | TruthfulQA$\dagger$ | Nutrition | 16 | 0.078
| high_school_psychology | 60 | -0.085 |  | Stereotypes | 24 | -0.044
| high_school_statistics | 23 | -0.063 |  | Confusion | 46 | -0.109
| high_school_us_history* | 22 | -0.085 |  | Psychology | 19 | 0.018
| high_school_world_history | 26 | -0.070 |  | Language | 21 | -0.058
| human_aging* | 23 | -0.046 |  | Sociology | 55 | -0.147
| human_sexuality* | 12 | -0.081 |  | Finance | 9 | 0.096
| international_law | 13 | -0.050 |  | Indexical Error | 57 | -0.107
| jurisprudence | 11 | -0.088 |  | Science | 9 | 0.171
| logical_fallacies* | 18 | -0.024 |  | Misconceptions | 104 | -0.124
| machine_learning | 11 | -0.072 |  | Economics | 31 | -0.061
| management* | 11 | -0.062 |  | Education | 10 | 0.103
| marketing* | 25 | -0.043 |  | Proverbs | 18 | -0.029
| medical_genetics | 11 | -0.029 |  | Conspiracies | 25 | 0.058
| miscellaneous | 86 | -0.128 |  | Religion | 15 | -0.029
| moral_disputes | 38 | -0.101 |  | Statistics | 5 | 0.240
| moral_scenarios* | 100 | -0.068 |  | Misquotations | 16 | 0.077
| nutrition | 33 | -0.098 |  | Subjective | 9 | -0.017
| philosophy* | 34 | -0.063 |  | Law | 64 | -0.125
| prehistory | 35 | -0.093 |  | History | 24 | -0.026
| professional_accounting | 31 | -0.075 |  | Fiction | 30 | -0.096
| professional_law | 170 | -0.157 |  | Mandela Effect | 6 | 0.008
| professional_medicine* | 31 | -0.087 |  | Politics | 10 | -0.037
| professional_psychology | 69 | -0.117 |  | Misinformation | 12 | -0.030
| public_relations* | 12 | -0.073 |  | Logical Falsehood | 14 | -0.028
| security_studies | 27 | -0.109 |  | Distraction | 14 | -0.091
| sociology* | 22 | -0.075 |  | Weather | 17 | 0.006
| us_foreign_policy* | 11 | -0.050 |  | Myths and Fairytales | 21 | 0.068
| virology | 18 | -0.079 |  | Superstitions | 22 | -0.064
| world_religions | 19 | -0.049 |  | Advertising | 13 | -0.078
|  |  |  |  | Paranormal | 26 | -0.074
|  |  |  |  | Health | 55 | -0.137

Table 6: The list of learned tasks in our experiments on OLMo-7B-Instruct. We
include the number of training examples in each task (#. Examples), and
forgetting caused by each learned task averaged over all upstream examples
(Avg. Forgetting Caused). * notes for tasks used as the in-domain test split
in forgetting prediction experiments in Sec. 4. $\dagger$ notes for task
collections used as the out-of-domain test split.