Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
Under review as a conference paper at ICLR 2025

PERPLEXED BY PERPLEXITY: PERPLEXITY-BASED
DATA PRUNING WITH SMALL REFERENCE MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

In this work, we investigate whether small language models can determine high-
quality subsets of large-scale text datasets that improve the performance of larger
language models. While existing work has shown that pruning based on the
perplexity of a larger model can yield high-quality data, we investigate whether
smaller models can be used for perplexity-based pruning and how pruning is
affected by the domain composition of the data being pruned. We demonstrate
that for multiple dataset compositions, perplexity-based pruning of pretraining
data can significantly improve downstream task performance: pruning based on
perplexities computed with a 125 million parameter model improves the average
performance on downstream tasks of a 3 billion parameter model by up to 2.04
and achieves up to a 1.45× reduction in pretraining steps to reach commensurate
baseline performance. Furthermore, we demonstrate that such perplexity-based
data pruning also yields downstream performance gains in the over-trained and
data-constrained regimes.

1

INTRODUCTION

A large focus of the machine learning community has been improving the performance of large
language models (LLMs) while reducing their training costs. In this work, we consider how to
improve the quality of an LLM by improving the quality of its pretraining data. Although there
are many techniques to improve data quality, such as augmenting training samples with additional
information (Li et al., 2024; Korbak et al., 2023), in this work we focus on the predominant method
of data pruning: intelligently selecting a high-quality subset of a larger dataset to train on.

Data pruning is commonly used for quality filtering of noisy text data. Simple approaches include
using symbolic rules (Bane et al., 2022; Raffel et al., 2020) or using simple classifiers to determine
high-quality samples (Wenzek et al., 2020). However, in addition to basic quality filtering, more
complex data pruning techniques are also applied to datasets to further improve their quality. Xie et al.
(2023b) perform importance resampling where importance scores are calculated based on feature
similarity to a target text. Tirumala et al. (2023) prune datasets by deduplicating and diversifying data
based on a pretrained language model’s embeddings of the text samples. Xie et al. (2023a) re-weight
domain proportions based on learnability as determined by a smaller proxy model. Marion et al.
(2023) investigate data pruning based on multiple neural heuristics of sample difficulty, ultimately
concluding that the perplexity of a sample under a reference language model is the best pruning
metric.

In this work, we thoroughly investigate the impact that data pruning based on sample perplexity (Mar-
ion et al., 2023) has on LLM pretraining. In particular, we focus on the interplay between pretraining
dataset composition and pruning methodology. We further evaluate perplexity pruning in the over-
trained and data-constrained regimes. We also investigate whether evaluating the quality of data
interventions based on upstream test set perplexity is a sound methodology for gauging downstream
performance. To perform perplexity-based data pruning, we train a small language model on a
random subset of the given pretraining dataset and then evaluate its perplexity on each sample in
the dataset. We then prune the dataset to only include samples within some range of perplexities
(i.e., sub-sample to the highest or lowest perplexity samples). We demonstrate that for two vastly
different pretraining data compositions, a small language model can be used to effectively prune the

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

pretraining dataset of a significantly larger model, leading to significant gains in the final model’s
downstream performance.

Our work differs from previous work on perplexity-based data pruning for LLM pretraining in three
key ways: (i) our emphasis on downstream model quality evaluation, (ii) our exploration of different
pretraining dataset domain compositions, and (iii) our analysis of pruning in non-standard training
regimes. While previous works evaluate the resulting LLM’s quality based on upstream metrics such
as perplexity on the test split of the pretraining dataset, we evaluate data pruning’s impact based on
downstream evaluation benchmarks (e.g. mmlu (Hendrycks et al., 2021), hellaswag(Zellers et al.,
2019), etc.). Evaluating on more meaningful benchmarks enables us to make stronger, more rigorous
conclusions about the impact of perplexity-based data pruning, as we find that some techniques
that significantly improve downstream performance have no, or even adverse, effects on upstream
performance. This difference in metrics enables us to conclude that smaller models can prune
the data for larger models, which was not observed in previous perplexity-basd pruning works.
Secondly, while previous work only investigates pruning on datasets composed of just one domain
(CommonCrawl1), we consider two datasets with different domain compositions: the Pile (Gao et al.,
2020) and Dolma (Soldaini et al., 2024). The Pile is composed of many diverse curated domains,
with only 15.61% of the data being derived from general web-scrapes, while Dolma is a web-scrape
skewed dataset, with 81.31% of its data being derived from the CommonCrawl. We find that
successful pruning techniques vary greatly for different dataset compositions to the point that the best
technique for one dataset composition may degrade performance for a different composition. Finally,
we also evaluate perplexity-based data pruning in the less standard regimes of over-training and
data-constrained training. This investigation provides a broader understanding for when practitioners
should use perplexity pruning for their data.

Contributions Our work makes the following contributions:

• We demonstrate that, across three datasets of varying domain compositions, a small reference
model can effectively prune the pretraining dataset of a significantly larger language model
(30× greater parameters), providing both a significant increase in downstream performance
and decrease in pretraining steps (Table 1 and Figure 1).

• We show that data pruning techniques can be highly sensitive to the domain composition
of the dataset, suggesting the need to evaluate multiple distinct dataset compositions when
conducting data pruning research (Table 1 and Table 4).

• We investigate perplexity-based data pruning in multiple non-standard settings demonstrating
that it can still lead to gains when over-training and when data-constrained (Section 3.4 and
Section 3.5).

• We find that test set perplexity can be a misleading metric for evaluating the efficacy of data
pruning techniques, as interventions that result in significantly higher test set perplexity can
still achieve better performance on downstream tasks (Table 3).

2 PERPLEXITY-BASED DATA PRUNING

We start by training a reference model that will be used to calculate the perplexity of all samples in
our dataset. First, we partition the original dataset into two splits: one for training the reference model
and one for training the final model. After training the reference model on the standard next-token
prediction objective, we compute the reference model’s perplexity on each of the samples in the final
model’s training split. We then prune the final model’s dataset split to a fraction of its original size,
referred to as the selection rate (rs), by selecting samples according to a selection criteria which can
be one of low, medium, or high. In low selection, samples with the lowest perplexity are selected.
In medium selection, we select samples whose perplexity is close to the median perplexity, that is,
samples with perplexity in the [50 − rs
2 ] percentiles of all perplexities. In high selection,
samples with the highest perplexity are selected. After pruning our dataset, we train a final model
using the standard next token prediction objective on the pruned version of the final model training
split. We present a pseudocode for pruning based on perplexity in Algorithm 1.

2 , 50 + rs

1https://data.commoncrawl.org/

2

Under review as a conference paper at ICLR 2025

Algorithm 1: Psuedocode for performing perplexity-based data pruning.
Input: Raw dataset D = {x(i)}M

i=1, where each x(i) is a tokenized text sample;

selection_criteria ∈ {low, medium, high}; selection rate rs ∈ (0, 1); reference
training split size R.

Output: Parameters of final model trained on the perplexity pruned dataset θ∗
Dref, Dtrain ← random_split(D, R)
θref ← random parameter initialization
θ∗
ref ← train(θref, Dref)
P ← {}
for x(i) ∈ Dtrain do
NLLx(i) = 1
|x(i)|
PPLXx(i) = 2NLL
x(i)
P[x(i)] = PPLXx(i)

tj ∈x(i) − log P (tj|t<j; θref)

(cid:80)

final.

end
if selection_criteria == "low" then

min_percentile ← 0.0
max_percentile ← rs

end
else if selection_criteria == "medium" then

min_percentile ← 0.5 − rs
2
max_percentile ← 0.5 + rs
2

end
else if selection_criteria == "high" then

min_percentile ← 1 − rs
max_percentile ← 1.0

end
ˆFP ← empirical CDF of P.values()
Dpruned ← []
for x(i), PPLXx(i) ∈ P do

if min_percentile < ˆFP (PPLXx(i)) < max_percentile then

Dpruned.append(x(i))

end

end
θfinal ← random parameter initialization
θ∗
final ← train(θfinal, Dpruned)
return θ∗

final

We consider the setting in which the reference model is significantly smaller than the final model.
While this assumption is not strictly necessary, we believe that it is the most practically relevant setup,
as it best reflects a data pruning paradigm that would be used for the next generation of LLMs where
the models being trained are larger than any existing models.

3 EXPERIMENTS

3.1 SETUP

Models. All models are based on the MPT family of transformer models (Vaswani et al., 2017;
MosaicML, 2023c). All reference models have 125 million parameters, and we consider final models
with 1 billion and 3 billion parameters.

Data. We consider two datasets in this work. The Pile (Gao et al., 2020) is composed of 22 different
domains that range from general web scrapes to legal text. Dolma (Soldaini et al., 2024) is composed
of 7 different domains and is derived mainly from general web scrapes. We tokenize all datasets
using the GPT-4 tokenizer (OpenAI, 2022).

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Table 1: Average normalized accuracy grouped by task category for both datasets and both final
model sizes. For all datasets and model sizes we find that training on perplexity pruned data
outperforms the baseline. Bold results are within one standard error of the highest score.

Pruning Method

1B Parameters Trained on Pile

No Pruning (Baseline)
High Perplexity Selected

3B Parameters Trained on Pile

No Pruning (Baseline)
High Perplexity Selected

1B Parameters Trained on Dolma

No Pruning (Baseline)
Medium Perplexity Selected
3B Parameters Trained on Dolma

No Pruning (Baseline)
Medium Perplexity Selected

World
Knowl-
edge

Common
Sense
Reason-
ing

Language
Under-
stand-
ing

Symbolic
Prob-
lem
Solving

Reading
Com-
prehen-
sion

Average

15.51
18.18

21.82
25.8

16.48
17.98

23.56
24.19

10.31
12.75

13.09
16.24

12.32
13.03

14.29
16.48

28.11
33.2

39.08
43.32

28.86
31.87

39.57
41.8

3.53
3.36

4.88
2.91

3.58
3.44

4.4
3.3

11.16
10.63

14.28
15.07

7.95
10.41

14.2
13.19

13.73
15.62

18.63
20.67

13.84
15.35

19.2
19.79

Training and hyperparameters. All reference models are trained for a fixed duration of 26 billion
tokens. Unless otherwise specified, all final models are trained to Chinchilla optimal (Hoffmann
et al., 2022), meaning that each final model’s training duration in tokens is 20 times its parameter
count. All models are trained using the decoupled Lion optimizer (Chen et al., 2024) with a cosine
learning rate schedule. All reference models and 1B parameter models are trained with a maximum
learning rate and weight decay of 2e-4 and all 3B models are trained with a maximum learning rate
and weight decay of 1.6e-4. Training is conducted using llm-foundry (MosaicML, 2023b) and
using both Nvidia A100s and H100s. We perform two trials for each experiment.

Evaluation. We evaluate models on 33 different downstream question-answering tasks using the
MosaicML evaluation gauntlet (MosaicML, 2023a). Before averaging the accuracy across tasks, we
normalize the accuracy on each task by the baseline of random guessing Specifically, we normalize
the accuracy of each individual task as an = am−ar
, where am is the accuracy of the model and ar
1−ar
is the expected accuracy of random guessing. We report the average normalized accuracy for each
task category as well as the average normalized accuracy across all task categories. 2. More details
on tasks and task categories are listed in Section 8.

3.2 PERPLEXITY-BASED DATA PRUNING IMPROVES DOWNSTREAM PERFORMANCE

If a certain range of perplexities is a good heuristic for data quality, training on that perplexity-pruned
subset should improve downstream performance. We sweep across pruning selection criteria and
selection rates (Section 7) and find that the best settings are to select high-perplexity samples at a 50%
rate for the Pile and to select medium-perplexity samples at a 50% rate for Dolma. We compare the
most performant pruning settings to baseline models trained on the original datasets without pruning
in Table 1. Across all datasets and model sizes, models pretrained on the perplexity pruned version of
the dataset significantly outperform the baseline model on average. Specifically, perplexity-based
data pruning outperforms the average downstream performance of no pruning for 1B models by
1.89 and 1.51 for the Pile and Dolma respectively, and improves the performance of 3B models by
2.04 and 0.59 for the Pile and Dolma respectively. These results suggest that the perplexity of a
small model provides a strong signal of data quality for a much larger model, as training on the data
selected by the small model leads to significant downstream performance improvements.

2Not to be confused, the random accuracy normalization we use is different from the normalized accuracy
reported by the EleutherAI LM Evaluation Harness, which normalizes based on the Byte-length of the response.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 1: Average normalized task accuracy evaluated intermittently throughout pretraining for
each dataset and model size investigated. Perplexity-based data pruning leads to an improvement in
performance for all intermediate training steps evaluated.

3.3 PERPLEXITY-BASED DATA PRUNING IMPROVES TRAINING EFFICIENCY

Since perplexity-based data pruning improves the final performance of models, we also investigate
how pruned data affects the training dynamics of models. Specifically, we investigate whether
training on perplexity pruned data enables models to achieve the same downstream performance
as models trained on the unpruned data in training fewer steps. We plot the average downstream
performance of partially trained checkpoints from the 1B baseline and perplexity pruned models
in Figure 1. Perplexity pruning outperforms the baseline model for all intermediate pretraining
durations evaluated. Furthermore, perplexity pruned models reach the same average normalized
accuracy as the baseline models in 1.31× and 1.45× fewer steps for Pile 1B and 3B respectively and
in 1.29× and 1.14× fewer steps for Dolma 1B and Dolma 3B respectively. These results demonstrate
that the resulting high-quality data from perplexity-based data pruning enables faster learning which
can be leveraged to achieve the same downstream performance as training on unpruned data with
fewer pretraining steps.

3.4 PERPLEXITY-BASED DATA PRUNING FOR OVER-TRAINED MODELS

A recent trend with LLMs has been to over-train models by training them on more tokens than the
Chinchilla optimal number of tokens (Touvron et al., 2023; Gadre et al., 2024). As our work targets
the data component of LLM pretraining, we investigate the hypothesis that over-training would be
more beneficial for models trained on perplexity pruned datasets as the data is of higher quality. We
test this hypothesis by training a 1B parameter model for 130B tokens, which is 5× the Chinchilla
optimal number of tokens. We evaluate the downstream performance of each over-trained model
in Table 2. The main observation is that while the absolute gain in average downstream normalized
accuracy from perplexity-based data pruning on the Pile is similar for both compute optimal and

5

510152025Token Duration (Billion)46810121416Avg. Normalized Accuracy(a)Pile 1B ParametersBaselineHigh Perplexity Selected1020304050Token Duration (Billion)8101214161820Avg. Normalized Accuracy(b)Pile 3B ParametersBaselineHigh Perplexity Selected510152025Token Duration (Billion)68101214Avg. Normalized Accuracy(c)Dolma 1B ParametersBaselineMedium Perplexity Selected1020304050Token Duration (Billion)8101214161820Avg. Normalized Accuracy(d)Dolma 3B ParametersBaselineMedium Perplexity SelectedUnder review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Table 2: Downstream task performance for Chinchilla Optimal and 5× over-trained data budgets.
The “Improvement Over Baseline” column refers to the gain observed from perplexity pruning as
compared to the baseline trained in the same setting.

Pruning Method

Average

Improvement Over Baseline

1B Parameters Trained on High Perplexity Pile

Chinchilla Optimal
5× Over-Trained

1B Parameters Trained on Medium Perplexity Dolma

Chinchilla Optimal
5× Over-Trained

15.62
18.83

15.35
18.67

1.89
1.74

1.51
0.84

over-trained models, the gain decreases for Dolma when over-training. On the Pile, we find that the
gain from perplexity pruned data is similar in the compute optimal regime and the over-trained regime:
we see a gain in average performance of 1.89 when training compute optimal and a gain of 1.74
when over-training. On Dolma, the gain from perplexity pruned data decreases in the over-trained
regime: we see a gain of 1.51 when training for a compute optimal duration but this decreases to a
gain of 0.84 when over-training. These results show that while the higher quality data resulting from
perplexity-based data pruning does still lead to an improvement in downstream performance in the
over-trained regime, there is not a relative increase in downstream improvement over the baseline
when over-training.

3.5 PERPLEXITY-BASED DATA PRUNING FOR THE DATA CONSTRAINED REGIME

Our experiments so far were conducted in the setting where there exists a sufficient abundance of
data such that even after pruning with the desired selection rate there are enough data points to
fill the desired token budget without requiring any data to be repeated. However, there are many
training settings that do not fall under this data-abundant regime. Consequently, we evaluate how
perplexity-based data pruning performs when the number of tokens is constrained, and pruning
induces a greater number of repetitions of the data. For each dataset, we vary the available data such
that training for a Chinchilla optimal number of tokens requires a different number of repetitions.
Specifically, we investigate data budgets that require {0.5,1,2,4,8} repetitions to reach the
Chinchilla optimal number of tokens3. As each number of repeats refers to the total number of tokens
available, for all pruning experiments the number of repetitions after pruning is actually greater by a
factor of 1
since we prune the available tokens according to rs, the selection rate. Since all models
rs
use a selection rate of 0.5, the models trained on the pruned data see the data for 2× more repetitions.

We plot the average downstream performance as a function of the number of repetitions in Figure 2.
On both the Pile and Dolma, we find that training on perplexity pruned data yields an improvement
for up to two repetitions. These results suggest that perplexity-based data pruning can still provide
performance gains for some degree of data constraint. Furthermore, our results replicate the findings
of Muennighoff et al. (2023) that more than four repetitions yields negligible gains. Specifically,
the baseline model without pruning maintains commensurate performance for up to four repetitions.
Similarly, models trained on perplexity-pruned data maintain commensurate performance for up
to two repetitions through the base data, which corresponds to four repetitions after pruning. That
training on repeated perplexity-pruned data leads to diminishing gains after four repetitions post-
pruning suggests that the higher quality data resulting from pruning does not change the point for
which repeating data yields diminishing improvements in performance.

3.6 UPSTREAM PERPLEXITY IS NOT A RELIABLE EVALUATION METRIC FOR DATA PRUNING

As previous works have used the perplexity of the model on a test split of the pretraining dataset as
an approximation to downstream performance, we wanted to explore how well such perplexity-based

3Repeat=0.5 means that the available number of tokens is twice the training budget, i.e. the data-abundant

setting

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Figure 2: Downstream task performance as a function of available dataset size. The number of
repeats denotes the number of repeats over the raw dataset necessary to achieve the Chinchilla optimal
number of tokens. Training on perplexity pruned data leads to an improvement for up to two repeats
on both the Pile Dolma.

Table 3: Performance as evaluated by perplexity on a test split of the original dataset as well as
average normalized task accuracy for 1 billion parameter final models trained on the Pile. The model
trained on pruned data has worse pretraining test split perplexity even though it significantly improves
average downstream normalized accuracy.

Pruning Method

Test Set Pplx. (↓) Downstream Task Avg. (↑)

1B Parameters Trained on Pile

No Pruning (Baseline)
High Perplexity Selected

1B Parameters Trained on Dolma

No Pruning (Baseline)
Medium Perplexity Selected

7.83
8.51

13.53
14.33

13.73
15.62

13.84
15.35

evaluations agree with downstream performance for data intervention techniques. Pruning performs
an intervention on the dataset, making models trained on the pruned dataset biased estimators of
the original data distribution. Therefore, it is unlikely that the performance on the original data
distribution is a fair evaluation of model quality. We compare the test set perplexity and average
downstream performance for 1 billion parameter models trained on the original and pruned version
of the Pile and Dolma in Table 3. For both the Pile and Dolma, training on perplexity pruned data
significantly worsens perplexity on a test split of the pretraining data, while the average downstream
performance is significantly improved. This result suggests that test set perplexity may not always
be a sound metric for data pruning work and that researchers should instead directly evaluate on
downstream benchmarks.

4 UNDERSTANDING THE EFFECTS OF PERPLEXITY-BASED PRUNING

In this section, we investigate how data pruning works by exploring some of the properties of
perplexity-based pruning.

4.1 HOW ARE REFERENCE PERPLEXITIES DISTRIBUTED

In order to better understand how perplexity-based data pruning works, we investigate the distribution
of the computed reference model perplexities for each dataset. For each dataset, we randomly sample
10% of the calculated perplexities and perform kernel density estimation to estimate the distribution
of log perplexities for a given dataset. We repeat this procedure for the optimal pruned version of the
dataset. We plot the resulting estimates of the log perplexity distribution in Figure 3. We find that
the log perplexity distribution for the Pile is multimodal and asymmetric, while for Dolma and it is
unimodal and symmetric.

7

0.51.02.04.08.0Repeats2930313233343536Avg. Normalized Accuracy(a)Pile 1B ParametersBaselineHigh Perplexity Selected0.51.02.04.08.0Repeats2930313233343536Avg. Normalized Accuracy(b)Dolma 1B ParametersBaselineMedium Perplexity SelectedUnder review as a conference paper at ICLR 2025

Figure 3: Distribution of sample perplexities as evaluated by the reference model for the Pile and
Dolma. We show both the original distribution over the full dataset without pruning as well as the
distribution after applying the optimal perplexity-based data pruning technique for a given dataset.

Figure 4: Proportion of the total dataset each domain makes up before and after pruning. For all
datasets, pruning tends to select more samples from general web domains while leaving out samples
from highly specific domains.

4.2 HOW PRUNING AFFECTS DOMAIN COMPOSITION

We can also interpret the effect that perplexity-based data pruning has on a dataset by examining
how pruning affects each domain’s proportion of the total dataset. We plot the pre and post-pruning
domain compositions for the Pile and Dolma in Figure 4. Interestingly, for all datasets pruning
increases the proportion of data coming from web-scraped domains while decreasing the proportion
of data coming from highly specific technical domains such as code or scientific papers. This trend is
more pronounced in the Pile, where the proportions of Pile-CC and OpenWebText2 nearly double,
while the proportions of domains such as Pubmed Central, ArXiv, and Github are all reduced by at
least a factor of three. Future work should investigate how perplexity-based pruning affects a model’s
performance on downstream tasks that are in the same category as the highly pruned domains.

5 RELATED WORK

Classical methods for pruning text data.
In order to improve the quality of raw web scrapes,
which often contain very noisy samples, pruning via quality filtering has become a common practice.
Simple rules-based methods have been employed to prune datasets by filtering out low-quality
samples according to some hand-crafted heuristic such as whether the text contains prohibited words,
is predominantly English, etc. (Bane et al., 2022; Raffel et al., 2020; Rae et al., 2022; Penedo et al.,
2023). N-gram perplexity-based methods, in which an n-gram model is first trained on a high
quality, curated corpus and then used to score another corpus, have also been applied to filter text

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

012345678Log Perplexity0.00.20.40.60.81.0Density(a)Pile Log Perplexity DistributionNo PruningHigh Perplexity Selected01234567Log Perplexity0.000.250.500.751.001.251.501.752.00Density(b)Dolma Log Perplexity DistributionNo PruningMedium Perplexity Selectedpile-ccpubmed-centralbooks3arxivgithubopenwebtext2freelawstackexchangewikipedia-(en)uspto-backgroundspubmed-abstractsdm-mathematicsgutenberg-(pg-19)opensubtitlesubuntu-irceuroparlyoutubesubtitlesbookcorpus2hackernewsphilpapersnih-exporterenron-emailsDomains0.000.050.100.150.200.250.30Proportion of Dataset (%)(a)Pile Domain CompositionNo PruningHigh Perplexity Selectedccstackc4redditpes2obookswikiDomains0.00.10.20.30.40.50.60.70.8Proportion of Dataset (%)(b)Dolma Domain CompositionNo PruningMedium Perplexity SelectedUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

data (Moore & Lewis, 2010; Axelrod, 2017; Gao, 2021; Laurençon et al., 2022; Muennighoff et al.,
2023). Although our method also uses perplexity to prune data, it does so in a very different manner.
In n-gram perplexity pruning, perplexity is used to estimate whether new text is in distribution as
compared to the currated text the n-gram was trained on, while in our model-based perplexity pruning,
the reference model is trained on the same distribution of text and the perplexity is more akin to an
estimate of the difficulty of an example. In this work, the datasets we leverage already have some
basic rules-based pruning applied, and as such, the method we investigate is largely complementary
to these existing techniques.

Neural network based methods for pruning text data. Recently, there has been much interest
in using neural networks to compute metrics that can be used to intelligently prune datasets. A
common technique in this family of methods is using a model to sample high-quality data from
large datasets based on the sample’s similarity to a curated high-quality corpus that serves as a target
distribution (Feng et al., 2022; Xie et al., 2023b). Xie et al. (2023a) also consider how to use a
small reference model to prune pretraining data for a much larger model, by using a small reference
model to learn the optimal weighting of domain proportions to maximize the "learnability" of the
resulting dataset. Pruning based on the difficulty or loss of a sample has previously been explored
for text data, but the majority of such work focuses on curating data for finetuning (Swayamdipta
et al., 2020; Attendu & Corbeil, 2023; Coleman et al., 2020; Mindermann et al., 2022; Mekala et al.,
2024). Marion et al. (2023), however, investigate multiple model-based sample difficulty heuristics
for pruning pretraining text datasets. Although we use the same method for pruning text pretraining
datasets, our analysis differs substantially as we evaluate model quality based on downstream metrics
and extend our analysis to multiple different dataset compositions which enables us to conclude that
the reference model can be smaller than the final model.

Data pruning on vision tasks. While data pruning is becoming more and more relevant with large
amounts of text data, it has also been extensively applied in the vision domain (Paul et al., 2021;
Toneva et al., 2018; Park et al., 2023). These works often prune data points based on their loss or
gradients during training (Killamsetty et al., 2021; Mirzasoleiman et al., 2020). Model-based methods
have also been leveraged for image data pruning (Fang et al., 2024; Schuhmann et al., 2021). Note
that in the literature, data pruning is also sometimes referred to as coreset selection (Guo et al., 2022).
More recently, Park et al. (2022) show that, somewhat surprisingly, active learning (Castro & Nowak,
2008) based algorithms tend to outperform most data subset selection algorithms. In the context of
contrastive learning, hard-negative mining has been effective as a data pruning method (Kalantidis
et al., 2020; Robinson et al., 2020; Zhang & Stratos, 2021). Recently, Goyal et al. (2024) investigated
scaling laws for training on pruned data in the context of vision models.

6 CONCLUSION

In this work, we conduct an empirical investigation of the impact that perplexity-based data pruning
has on model performance. We demonstrate that small reference models can be used to prune the
data of models with up to 30× more parameters, leading to both significant downstream performance
improvements and increased training efficiency. We then investigate perplexity-based data pruning
in two non-standard settings: the over-trained and data-constrained regimes. We find that for both
settings, training on perplexity pruned data can outperform training on unpruned data, demonstrating
that perplexity-based data pruning is a widely applicable and extensible technique. We also investigate
upstream metrics for evaluating data pruning techniques and provide an example where evaluating
models based on their perplexity on the test split of the pretraining dataset does not align with
evaluating based on downstream model performance. Additionally, we demonstrate that optimal
pruning techniques can vary greatly for different dataset compositions. Although we do not present a
predictive theory for how pruning parameters should be selected for different datasets, we demonstrate
that the optimal pruning parameters for a 1 billion parameter model can successfully transfer to
3 billion parameter models, potentially suggesting that empirically determining the optimal pruning
parameters can be done cheaply. Our work takes a key step towards establishing perplexity-based
data pruning as a primary technique in the modern data researcher’s toolkit.

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

REFERENCES

Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh
Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based
formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019
Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2357–2367, Minneapolis,
Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1245.
URL https://aclanthology.org/N19-1245.

Jean-michel Attendu and Jean-philippe Corbeil. NLU on data diets: Dynamic data subset selection
for NLP classification tasks. In Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim,
Young Jin Kim, Tal Schuster, and Ameeta Agrawal (eds.), Proceedings of The Fourth Workshop on
Simple and Efficient Natural Language Processing (SustaiNLP), pp. 129–146, Toronto, Canada
(Hybrid), July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.sustainlp-1.
9. URL https://aclanthology.org/2023.sustainlp-1.9.

Amittai Axelrod. Cynical selection of language model training data. arXiv preprint arXiv:1709.02279,

2017.

Fred Bane, Celia Soler Uguet, Wiktor Stribi˙zew, and Anna Zaretskaya. A comparison of data filtering
methods for neural machine translation. In Janice Campbell, Stephen Larocca, Jay Marciano,
Konstantin Savenkov, and Alex Yanishevsky (eds.), Proceedings of the 15th Biennial Conference
of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track
and Government Track), pp. 313–325, Orlando, USA, September 2022. Association for Machine
Translation in the Americas. URL https://aclanthology.org/2022.amta-upg.22.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical
commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence,
volume 34, pp. 7432–7439, 2020.

Rui Castro and Robert Nowak. Active learning and sampling. In Foundations and Applications of

Sensor Management, pp. 177–200. Springer, 2008.

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong,
Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms.
Advances in Neural Information Processing Systems, 36, 2024.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina
Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions.
In Jill
Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Lan-
guage Technologies, Volume 1 (Long and Short Papers), pp. 2924–2936, Minneapolis, Min-
nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL
https://aclanthology.org/N19-1300.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
arXiv:1803.05457v1, 2018.

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy
Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for
deep learning. In International Conference on Learning Representations, 2020. URL https:
//openreview.net/forum?id=HJg2b0VYDr.

Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal
Shankar. Data filtering networks. In The Twelfth International Conference on Learning Represen-
tations, 2024. URL https://openreview.net/forum?id=KAk6ngZ09F.

Yukun Feng, Patrick Xia, Benjamin Van Durme, and João Sedoc. Automatic document selection
for efficient encoder pretraining. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

9522–9530, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational
Linguistics. doi: 10.18653/v1/2022.emnlp-main.647. URL https://aclanthology.org/
2022.emnlp-main.647.

Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman,
Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably
with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024.

Leo Gao. An empirical exploration in quality filtering of text data, 2021.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang,
Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb
dataset of diverse text for language modeling, 2020.

Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico Kolter. Scaling
laws for data filtering–data curation cannot be compute agnostic. arXiv preprint arXiv:2404.07177,
2024.

Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selec-
tion in deep learning. In International Conference on Database and Expert Systems Applications,
pp. 181–195. Springer, 2022.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob
In International Confer-
Steinhardt. Measuring massive multitask language understanding.
ence on Learning Representations, 2021. URL https://openreview.net/forum?id=
d7KBjmI3GmQ.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas
Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Au-
relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and
Laurent Sifre. An empirical analysis of compute-optimal large language model training.
In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad-
vances in Neural Information Processing Systems, volume 35, pp. 30016–30030. Curran Asso-
ciates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/
2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf.

Aditi Jha, Sam Havens, Jeremy Dohmann, Alexander Trott, and Jacob Portes. LIMIT: Less is more
for instruction tuning across evaluation paradigms. In NeurIPS 2023 Workshop on Instruction
Tuning and Instruction Following, 2023. URL https://openreview.net/forum?id=
QxtL4Q1enz.

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset
for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun
Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-
IJCNLP), pp. 2567–2577, Hong Kong, China, November 2019. Association for Computational
Linguistics. doi: 10.18653/v1/D19-1259. URL https://aclanthology.org/D19-1259.

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard
negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:
21798–21809, 2020.

Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer.
Grad-match: Gradient matching based data subset selection for efficient deep model training. In
International Conference on Machine Learning, pp. 5464–5474. PMLR, 2021.

Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher L. Buckley,
Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human
preferences. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan
Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023,
23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning
Research, pp. 17506–17533. PMLR, 2023. URL https://proceedings.mlr.press/
v202/korbak23a.html.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral,
Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen,
Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella
Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen,
Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan
Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu,
Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor
Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa
Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic,
Margaret Mitchell, Sasha Luccioni, and Yacine Jernite. The bigscience ROOTS corpus: A 1.6TB
composite multilingual dataset. In Thirty-sixth Conference on Neural Information Processing
Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?
id=UoEw6KigkUn.

Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge.

In
Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
Citeseer, 2012.

Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston,
and Mike Lewis. Self-alignment with instruction backtranslation. In The Twelfth International
Conference on Learning Representations, 2024. URL https://openreview.net/forum?
id=1oijHJBRsT.

Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A
challenge dataset for machine reading comprehension with logical reasoning. In Christian Bessiere
(ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence,
IJCAI 2020, pp. 3622–3628. ijcai.org, 2020. doi: 10.24963/IJCAI.2020/501. URL https:
//doi.org/10.24963/ijcai.2020/501.

Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When
less is more: Investigating data pruning for pretraining LLMs at scale. In NeurIPS Workshop on
Attributing Model Behavior at Scale, 2023. URL https://openreview.net/forum?id=
XUIYn3jo5T.

Dheeraj Mekala, Alex Nguyen, and Jingbo Shang. Smaller language models are capable of selecting
instruction-tuning training data for larger language models. arXiv preprint arXiv:2402.10430,
2024.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct
electricity? a new dataset for open book question answering. In Conference on Empirical Methods
in Natural Language Processing, 2018. URL https://api.semanticscholar.org/
CorpusID:52183757.

Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch,
Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin
Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt.
In
Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato
(eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of
Proceedings of Machine Learning Research, pp. 15630–15649. PMLR, 17–23 Jul 2022. URL
https://proceedings.mlr.press/v162/mindermann22a.html.

Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of
machine learning models. In International Conference on Machine Learning, pp. 6950–6960.
PMLR, 2020.

Robert C. Moore and William Lewis. Intelligent selection of language model training data. In Jan
Hajiˇc, Sandra Carberry, Stephen Clark, and Joakim Nivre (eds.), Proceedings of the ACL 2010 Con-
ference Short Papers, pp. 220–224, Uppsala, Sweden, July 2010. Association for Computational
Linguistics. URL https://aclanthology.org/P10-2041.

MosaicML.

Llm evaluation scores, 2023a.

URL https://www.mosaicml.com/

llm-evaluation.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

MosaicML. Llm foundry. https://github.com/mosaicml/llm-foundry, 2023b.

MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023c.

URL https://www.databricks.com/blog/mpt-7b.

Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra
Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language
models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
https://openreview.net/forum?id=j5BuTrEj35.

OpenAI. Tiktoken: A fast bpe tokeniser for use with openai’s models. https://github.com/

openai/tiktoken, 2022.

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi,
Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset:
Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.),
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 1525–1534, Berlin, Germany, August 2016. Association for Computational
Linguistics. doi: 10.18653/v1/P16-1144. URL https://aclanthology.org/P16-1144.

Dongmin Park, Dimitris Papailiopoulos, and Kangwook Lee. Active learning is a strong baseline for

data subset selection. In Has it Trained Yet? NeurIPS 2022 Workshop, 2022.

Dongmin Park, Seola Choi, Doyoung Kim, Hwanjun Song, and Jae-Gil Lee. Robust data pruning
under label noise via maximizing re-labeling accuracy. arXiv preprint arXiv:2311.01002, 2023.

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding
important examples early in training. Advances in Neural Information Processing Systems, 34:
20596–20607, 2021.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli,
Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb
dataset for falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh
Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
URL https://openreview.net/forum?id=kM5eGcdCzq.

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John
Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan,
Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks,
Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron
Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu,
Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen
Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro,
Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch,
Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux,
Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume,
Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas,
Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger,
Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol
Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu,
and Geoffrey Irving. Scaling language models: Methods, analysis and insights from training
gopher, 2022.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena,
Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified
text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL
http://jmlr.org/papers/v21/20-074.html.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions
for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings
of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392,
Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/
D16-1264. URL https://aclanthology.org/D16-1264.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with

hard negative samples. arXiv preprint arXiv:2010.04592, 2020.

Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives:
An evaluation of commonsense causal reasoning. In Logical Formalizations of Commonsense
Reasoning, Papers from the 2011 AAAI Spring Symposium, Technical Report SS-11-06, Stanford,
California, USA, March 21-23, 2011. AAAI, 2011. URL http://www.aaai.org/ocs/
index.php/SSS/SSS11/paper/view/2418.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An
adversarial winograd schema challenge at scale. In AAAI, pp. 8732–8740, 2020. URL https:
//aaai.org/ojs/index.php/AAAI/article/view/6399.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis,
Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of
clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur,
Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh
Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas
Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle
Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke
Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge,
and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining
Research. arXiv preprint, 2024.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam
Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska,
Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W.
Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda
Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan
Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La,
Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna
Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes,
Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut
Erdem, Ayla Karaka¸s, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski,
Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk
Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine
Stinson, Cedrick Argueta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta
Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt, Christopher D
Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel,
Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman,
Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle
Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David
Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz
Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho
Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad
Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà,
Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan
Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar,
Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra,
Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio
Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-Ari, Hana
Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar,
Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee
Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon
Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon,
Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason
Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy
Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg
Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones,
Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth,
Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel,
Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar,
Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin,
Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble,
Ludwig Schmidt, Luheng He, Luis Oliveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten
Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika,
Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn,
Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás
Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew
Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał
Sw˛edrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac
Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini,
Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover,
Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas
Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah
Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans,
Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah
Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut,
Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing
Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon
Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe
Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne
Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan
Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou,
Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz,
Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey,
Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan,
Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane
Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath,
Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-
Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon,
Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer
Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu,
Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan,
Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg,
Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera
Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu,
Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout
Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh,
Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen,
Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang,
Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating
the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN
2835-8856. URL https://openreview.net/forum?id=uyTL5Bvosj.

Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A.
Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training
dynamics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the
2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9275–
9293, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.
emnlp-main.746. URL https://aclanthology.org/2020.emnlp-main.746.

Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S. Morcos. D4: Improving LLM
pretraining via document de-duplication and diversification. In Thirty-seventh Conference on
Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
//openreview.net/forum?id=CG0L2PFrb1.

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and
Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning.
arXiv preprint arXiv:1812.05159, 2018.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand
Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language
models, 2023.

Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need.
In I. Guyon, U. Von
Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad-
vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.,
URL https://proceedings.neurips.cc/paper_files/paper/2017/
2017.
file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.

Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán,
Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets
from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri,
Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani,
Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the
Twelfth Language Resources and Evaluation Conference, pp. 4003–4012, Marseille, France, May
2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https:
//aclanthology.org/2020.lrec-1.494.

Thom Wolfe, Lewis Tunstall, and Patrick von Platen. Jeopardy dataset on hugging face hub. https:

//huggingface.co/datasets/jeopardy, 2022.

Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V
Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model
pretraining. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL
https://openreview.net/forum?id=lXuByUeHhd.

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language
models via importance resampling. In Thirty-seventh Conference on Neural Information Processing
Systems, 2023b. URL https://openreview.net/forum?id=uPSQv0leAu.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine
really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics, 2019.

Wenzheng Zhang and Karl Stratos. Understanding hard negatives in noise contrastive estimation.

arXiv preprint arXiv:2104.06245, 2021.

7 FULL DATA PRUNING SETTINGS SWEEP

In this section, we report the results of sweeping over different perplexity-based pruning setting
configurations. In particular, for each dataset, we first sweep over the selection criteria to determine
where from the distribution of perplexities samples should be selected. Then, using the best selection
criteria, we sweep the selection rate to determine how much we should prune.

Setup. We use the same training and evaluation setup as detailed in Section 3.1. We only perform
the sweep over pruning settings for 1 billion parameter final models for computational budget reasons;
however, we find that the best selection criteria at the 1 billion parameter scale also confers a
performance improvement at the 3 billion parameter scale, as detailed in 3.2.

16

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Table 4: Results from sweeping different selection criteria. We report the average normalized
accuracy for each task grouping as well as across all tasks. While high perplexity selection is optimal
for the Pile, medium perplexity selection is optimal for Dolma. Bold results are within one standard
error of the highest normalized accuracy.

Pruning Method

1B Parameters Trained on Pile

No Pruning (Baseline)
Low Perplexity Selected
Medium Perplexity Selected
High Perplexity Selected

1B Parameters Trained on Dolma

No Pruning (Baseline)
Low Perplexity Selected
Medium Perplexity Selected
High Perplexity Selected

World
Knowl-
edge

Common
Sense
Reason-
ing

Language
Under-
stand-
ing

Symbolic
Prob-
lem
Solving

Reading
Com-
prehen-
sion

Average

15.51
11.14
16.12
18.18

16.48
16.13
17.98
16.65

10.31
5.76
9.01
12.75

12.32
10.1
13.03
13.12

28.11
18.66
28.1
33.2

28.86
27.28
31.87
31.14

3.53
3.54
3.41
3.36

3.58
3.45
3.44
3.15

11.16
8.72
10.86
10.63

7.95
7.85
10.41
8.55

13.73
9.56
13.5
15.62

13.84
12.96
15.35
14.52

Table 5: Results from sweeping different selection rates. We report the average normalized accuracy
for each task grouping as well as across all tasks. Bold results are within one standard error of the
highest normalized accuracy.

Pruning Method

1B Parameters Trained on Pile

25% Selection Rate
50% Selection Rate
75% Selection Rate

1B Parameters Trained on Dolma

25% Selection Rate
50% Selection Rate
75% Selection Rate

World
Knowl-
edge

Common
Sense
Reason-
ing

Language
Under-
stand-
ing

Symbolic
Prob-
lem
Solving

Reading
Com-
prehen-
sion

Average

18.21
18.18
17.08

17.94
17.98
18.2

12.88
12.75
10.11

12.16
13.03
11.78

34.44
33.2
31.37

31.63
31.87
29.96

3.73
3.36
3.81

3.58
3.44
3.32

9.44
10.63
9.02

8.91
10.41
10.82

15.74
15.62
14.28

14.85
15.35
14.82

7.1 FINDING THE BEST SELECTION CRITERIA

For each dataset, we first sweep the selection criteria while keeping the selection rate fixed at 50%. We
report the performance of each selection criteria in Table 4. We find that on the Pile high perplexity
selection works the best and on Dolma medium perplexity selection works the best, improving the
average downstream performance by 1.89 and 1.51 respectively. An important observation from the
sweep is that the best selection criteria from one dataset does not transfer to another dataset and may
actually degrade performance compared to the baseline. Although medium-perplexity selection is
the best method on Dolma, selecting medium-perplexity samples on the Pile leads to a decrease in
the average downstream performance of 0.23 as compared to not performing pruning. These results
inform us that high and medium perplexity selection are the optimal selection criteria for the Pile
and Dolma respectively, and that the optimal selection criteria does not necessarily transfer between
datasets with different domain compositions.

17

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

7.2 FINDING THE BEST SELECTION RATE

Using the optimal selection criteria that we found for each dataset, we next investigate the best
selection rate for each dataset. We investigate three different selection rates: 25%, 50%, and 75%. We
present the results for each selection rate in Table 5. On the Pile, we find that there is no significant
difference in downstream performance for selection rates of 25% and 50%; on Dolma we find that a
selection rate of 50% achieves the best average downstream performance. For simplicity, we chose
to conduct the rest of the experiments in the paper using a selection rate of 50% on both datasets.
Furthermore, we find that all the selection rates tested outperform the baseline of no data pruning as
measured by average downstream performance. This suggests that the selection criteria has a greater
impact on the performance of a pruning configuration than the selection rate.

8 DETAILED EVALUATION SETUP

Jha et al. (2023) also use the MosaicML evaluation gauntlet to perform evaluations in their work. As
such, with explicit permission from the authors, we exactly reproduce their text describing the tasks
and tasks categories in the evaluation gauntlet. The following is from Section D of their paper:

The World Knowledge category includes the following datasets:

• Jeopardy (2,117 questions that are a custom subset of the dataset originally obtained from

Wolfe et al. (2022))

• MMLU (14,042 four-choice multiple choice questions distributed across 57 categories

Hendrycks et al. (2021)

• BIG-bench wikidata (20,321 questions regarding factual

information pulled from

wikipedia) Srivastava et al. (2023)

• ARC easy (2,376 easy multiple choice middle school science questions) Clark et al. (2018)
• ARC challenge (1,172 hard multiple choice science questions) Clark et al. (2018)
• BIG-bench: misconceptions (219 true or false questions regarding common misconceptions)

Srivastava et al. (2023)

The Commonsense Reasoning category loosely assesses a model’s ability to do basic reasoning
tasks that require commonsense knowledge of objects, their properties, and their behavior. It includes
the following datasets:

• BIG-bench Strategy QA (2,289 very eclectic yes/no questions on a wide range of common-

sense subjects e.g “Can fish get Tonsilitis?”)Srivastava et al. (2023)

• BIG-bench Strange Stories (174 short stories followed by questions about the charac-

ters)Srivastava et al. (2023)

• BIG-bench Novel Concepts (32 find-the-common-concept problems)Srivastava et al. (2023)
• COPA (100 cause/effect multiple choice questions) Roemmele et al. (2011)
• PIQA (1,838 commonsense physical intuition 2-choice questions) Bisk et al. (2020)
• OpenBook QA (500 questions that rely on basic physical and scientific intuition about

common objects and entities) Mihaylov et al. (2018).

Language Understanding tasks evaluate the model’s ability to understand the structure and properties
of languages, and include the following datasets:

• LAMBADA (6,153 passages take from books - we use the formatting adopted by OpenAI’s

version)Paperno et al. (2016)

• HellaSwag (10,042 multiple choice scenarios in which the model is prompted with a scenario
and choose the most likely conclusion to the scenario from four possible options)Zellers
et al. (2019)

• Winograd Schema Challenge (273 scenarios in which the model must use semantics to
correctly resolve the anaphora in a sentence. The Eval Gauntlet uses the partial evaluation
technique technique introduced in Trinh & Le (2019)) Levesque et al. (2012)

18

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

• Winogrande (1,267 scenarios in which two possible beginnings of a sentence are presented

along with a single ending) Sakaguchi et al. (2020)

• BIG-bench language identification (10,000 questions on multiple choice language identifica-

tion) Srivastava et al. (2023)

• BIG-bench conceptual combinations (103 questions using made up words) Srivastava et al.

(2023)

• BIG-bench conlang translation (164 example problems in which the model is given transla-
tions of simple sentences between English and some fake constructed language) Srivastava
et al. (2023)

Symbolic problem solving tasks test the model’s ability to solve a diverse range of symbolic tasks
including arithmetic, logical reasoning, algorithms, and algebra. These datasets include:

• BIG-bench elementary math QA (38,160 four-choice multiple choice arithmetic word

problems) Srivastava et al. (2023)

• BIG-bench dyck languages (1000 complete-the-sequence questions) Srivastava et al. (2023)
• BIG-bench algorithms (1,320 questions) Srivastava et al. (2023)
• BIG-bench logical deduction (1500 four-choice multiple choice questions relating to relative

ordering of objects) Srivastava et al. (2023)

• BIG-bench operators (210 questions involving mathematical operators) Srivastava et al.

(2023)

• BIG-bench repeat copy logic (32 samples in which the model is required to follow some

instructions for copying words/symbols)

• Simple arithmetic with spaces (1000 arithmetic problems consisting of up to 3 operations

and using numbers of up to 3 digits, developed by MosaicML)

• Simple arithmetic without spaces (1000 arithmetic problems consisting of up to 3 operations

and using numbers of up to 3 digits, developed by MosaicML)

• Math QA (2,983 four-choice multiple choice math word problems) Amini et al. (2019)
• LogiQA (651 four-logical word problems) Liu et al. (2020)

The Reading comprehension benchmarks test a model’s ability to answer questions based on the
information in a passage of text. The datasets include:

• BIG-bench Understanding fables (189 short stories) Srivastava et al. (2023)
• Pubmed QA Labeled (1000 hand-labeled medical documents followed by a related question

for which the model must respond yes/no/maybe) Jin et al. (2019)

• SQuAD (10,570 short documents followed by a related question. The model is expected to

output the exact correct answer) Rajpurkar et al. (2016)

• BoolQ (3,270 short passages on a diverse range of subjects followed by a yes/no questions)

Clark et al. (2019)

8.1 EVALUATION PROCEDURE

To compute model performance on the above datasets, the Eval Gauntlet uses one of the following
three ICL metrics for each dataset (from MosaicML’s composer library).

1. InContextLearningQAAccuracy: This metric uses the query, the corresponding correct
answer and a list of alternative answers to measure a model’s prediction. If the model’s
response conditioned on the query starts with either the correct answer or with one of the
alternative answers, it is considered correct. This is used for question-answering tasks such
as TriviaQA.

2. InContextLearningLMAccuracy: This metric tests a model’s ability to output a precise set
of tokens. A model’s output conditioned on a given query is judged to be correct only if the
model’s highest probability tokens match the correct sequence of tokens. This is used for
language modeling tasks such as LAMBADA.

19

Under review as a conference paper at ICLR 2025

3. InContextLearningMultipleChoiceAccuracy: This metric is used for testing a model’s ability
to answer multiple choice questions accurately. It compares the respective perplexity of the
query prepended to each of the possible choices, according to the model. If the query-choice
pair with the lowest per token perplexity is indeed the correct choice, then the model’s
output is judged to be correct. This is used for multiple choice tasks such as HellaSwag,
Winograd etc.

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

20

