Scaling Laws for Downstream Task Performance in Machine Translation
Under review as a conference paper at ICLR 2025

SCALING LAWS FOR DOWNSTREAM TASK
PERFORMANCE IN MACHINE TRANSLATION

Anonymous authors
Paper under double-blind review

ABSTRACT

Scaling laws provide important insights that can guide the design of large language
models (LLMs). Existing work has primarily focused on studying scaling laws for
pretraining (upstream) loss. However, in transfer learning settings, in which LLMs
are pretrained on an unsupervised dataset and then ﬁnetuned on a downstream
task, we often also care about the downstream performance. In this work, we
study the scaling behavior in a transfer learning setting, where LLMs are ﬁnetuned
for machine translation tasks. Speciﬁcally, we investigate how the choice of the
pretraining data and its size affect downstream performance (translation quality)
as judged by: downstream cross-entropy and translation quality metrics such
as BLEU and COMET scores. Our experiments indicate that the size of the
ﬁnetuning dataset and the distribution alignment between the pretraining and
downstream data signiﬁcantly inﬂuence the scaling behavior. With sufﬁcient
alignment, both downstream cross-entropy and translation quality scores improve
monotonically with more pretraining data. In such cases, we show that it is possible
to predict the downstream translation quality metrics with good accuracy using
a log-law. However, there are cases where moderate misalignment causes the
downstream translation scores to ﬂuctuate or get worse with more pretraining,
whereas downstream cross-entropy monotonically improves. By analyzing these,
we provide new practical insights for choosing appropriate pretraining data.

1

INTRODUCTION

Scaling laws quantify the relationship between a model’s performance and key design factors such
as the size of the training data or the model’s architecture. In the context of LLMs, these laws offer
valuable guidance for model development, resource allocation, and selection of appropriate training
data. Extensive research has focused on scaling laws for upstream perplexity or cross-entropy loss
(i.e., evaluated on pretraining data), demonstrating that these quantities can be well predicted using
power laws (Kaplan et al., 2020; Hoffmann et al., 2022; Gordon et al., 2021; Hernandez et al., 2022;
Fernandes et al., 2023; Henighan et al., 2020; Johnson et al., 2018). However, in practice, LLMs
often undergo transfer learning–they are ﬁrst pretrained on unsupervised data and then ﬁnetuned
for speciﬁc downstream1 tasks such as coding or translation. The question of whether scaling laws
can be used to predict downstream task performance is critical (OpenAI, 2024), yet remains largely
unanswered (Hernandez et al., 2021; Tay et al., 2021). Here, the term task performance refers to
metrics that measure task-related quantities such as accuracy and translation scores like BLEU,
ROUGE, or COMET, which are different from next-token prediction metrics such as cross-entropy.

In this work, we study scaling laws for transfer learning and focus on machine translation tasks.
Speciﬁcally, we look into the relation between the pretraining dataset size and the downstream task
performance after ﬁnetuning on the task. We ﬁnd that, in addition to the ﬁnetuning data size and the
choice of the performance metric, this relation fundamentally depends on the alignment between
the pretraining data and the downstream task (based on the translation alignment score provided in
Section 3). While similar observations have been made in different contexts in the transfer learning
literature (Tamkin et al., 2020; Agostinelli et al., 2022), our work provides new insights and concrete
scaling laws for the downstream performance in machine translation.

1We use the term downstream to refer to the ﬁnetuning task or metrics computed on it, and the term upstream

to refer to the metrics computed on the pretraining dataset.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

We carry out systematic experiments in which we pretrain LLMs on multilingual unsupervised datasets
and then ﬁnetune them on several machine translation tasks. Across the experiments, we vary the type
of pretraining data (to control the degree of distribution alignment with the downstream task) and the
ﬁnetuning data size. We study the following metrics: downstream BLEU score (Papineni et al., 2002),
downstream ROUGE score (Lin, 2004), downstream COMET score (Rei et al., 2020; Stewart et al.,
2020; Rei et al., 2022)2, and downstream cross-entropy. We ﬁnd that in settings where the distributions
are well-aligned, both the translation scores and downstream cross-entropy improve monotonically
with more pretraining (see Figure 1, orange curves). In these settings, we demonstrate that the
translation scores (e.g., BLEU, ROUGE, and COMET) can be well predicted using the following
p ))β, where Dp denotes the size of the pretraining data, and A, α, β
log-law: f (Dp) = (log(A · Dα
are the coefﬁcients to be ﬁt. We further propose a power-law L(Dp) = E + A
for the downstream
Dα
p
cross-entropy as the pretraining data scales – echoing similar laws developed for the upstream cross-
entropy as a function of the pretraining dataset size (Kaplan et al., 2020; Hoffmann et al., 2022) and
downstream cross-entropy as a function of the ﬁnetuning dataset size (Hernandez et al., 2021).

However, when distributions are not sufﬁciently aligned and the ﬁnetuning data size is relatively
small, we ﬁnd that there are cases where the translation scores exhibit an unclear, non-monotonic
behavior, whereas the downstream cross-entropy still improves monotonically following a power-law.
This observation suggests that using cross-entropy as a proxy for task-related metrics like BLEU,
ROUGE, or COMET scores may lead to critical misjudgments in practice if used to make decisions
about the “relevance” of the pretraining data for the downstream task or the required size of the
pretraining data for the target downstream performance. Finally, our empirical studies suggest that
pretraining brings little to no improvement on the translation quality when the ﬁnetuning (translation)
dataset is already large enough, complementing the ﬁndings of Hernandez et al. (2021).

Our contributions and main ﬁndings can be summarized as:

• We carry out systematic experiments on 770-million and 3-billion encoder-decoder
T5 (Raffel et al., 2020) models to study how downstream performance, measured by
downstream cross-entropy and translation scores, scales with the pretraining dataset
size. For pretraining, we experiment with different subsets of the Multilingual C4 (MC4)
dataset (Raffel et al., 2020), including English (en), German (de), French (fr), and Romanian
(ro). For ﬁnetuning, we study the following translation tasks: WMT-17 en-de (Bojar et al.,
2017), WMT-15 en-fr (Bojar et al., 2014), and WMT-16 en-ro (Bojar et al., 2016).

• We observe that, when the distributions of the pretraining and downstream tasks are well-
aligned, both the translation scores and downstream cross-entropy improve monotonically
with more pretraining (Figure 1, orange curves). For BLEU, ROUGE, and COMET scores,
we propose a new log scaling law and show that it has good predictive accuracy.

• When the distributions are not sufﬁciently aligned and the ﬁnetuning data size is relatively
small, translation scores ﬂuctuate or even get worse with more pretraining–losing the
monotonic scaling behavior (Figure 1, red curves). In these same settings, we ﬁnd that the
downstream cross-entropy still scales monotonically according to a power-law.

• We argue that the value of pretraining data for translation tasks should be evaluated using
downstream translation scores like BLEU, ROUGE, and COMET score and propose a prac-
tical guide for such an assessment by leveraging the proposed scaling law for these scores.
• We show that the proposed log scaling law generalizes to tasks beyond translation, with
experiments on SuperGLUE (Wang et al., 2019) tasks, which covers question answering,
reasoning, reading comprehension, and textual entailment.

2 RELATED WORK

Scaling laws for transformers. Scaling laws for LLMs have attracted signiﬁcant attention as they
can inform the decisions about key design choices such as model size and the type and size of the
pretraining data (Kaplan et al., 2020; Hoffmann et al., 2022; Hernandez et al., 2021). Most of the
pioneering work has focused on how upstream cross-entropy loss or perplexity scales with more

2In the rest of the paper, we will drop “downstream” when we refer to the downstream translation scores

such as BLEU, ROUGE, and COMET.

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

pretraining data, larger models, or longer training (Kaplan et al., 2020; Hoffmann et al., 2022). Follow-
up works have analyzed scaling behavior of translation models (Ghorbani et al., 2021; Zhuocheng
et al., 2023; Gordon et al., 2021; Fernandes et al., 2023; Bansal et al., 2022; Zhang et al., 2022),
studied theoretical foundation behind scaling laws (Sharma & Kaplan, 2020; Hutter, 2021; Bahri
et al., 2021), or extended the laws to the vision models (Zhai et al., 2022; Jain et al., 2023). Closest
to our work, Hernandez et al. (2021) have analyzed transfer learning but with a focus on how the
cross-entropy loss behaves as the ﬁnetuning data scales. Unlike our work, their scaling law describes
the relation between the size of a (ﬁnetuning) dataset and the cross-entropy loss on the same dataset
– making this closer to the standard scaling laws in the literature since the ﬁnetuning loss and the
ﬁnetuning dataset are computed over samples from the same distribution. On the other hand, we
propose scaling laws for the downstream metrics on the ﬁnetuning dataset as the pretraining data
scales – switching the focus to an “out-of-distribution” analysis. The only work we are aware of that
proposed scaling laws for the downstream task performance as a function of pretraining dataset size
(Sun et al., 2017) has focused on classiﬁcation tasks in the vision domain and used smaller models.

Transferability metrics and value of pretraining. While it may be commonly suggested that
pretraining data improves both upstream and downstream performance, this rule has been challenged
in the vision domain. Zoph et al. (2020); He et al. (2019); Shen et al. (2019); Ghiasi et al. (2018);
Mikami et al. (2022) have demonstrated that pretraining can sometimes have no effect on the
downstream task performance and sometimes it can even hurt the performance. We make similar
observations in the language domain with extensive experiments on machine translation tasks and
identify cases where (a) adding more pretraining data hurts the downstream task performance
when pretraining data is not aligned enough with the task and (b) pretraining does not improve
the downstream task performance noticeably when the ﬁnetuning dataset is large enough. Our
observations about the importance of “aligned” pretraining data are also supported by recent work
on machine translation (Alves et al., 2024; Xu et al., 2024) trying to keep the pretraining data as
multilingual as possible instead of being heavily English-centric (Stap et al., 2024; Li et al., 2024).
Another related line of work is on transferability metrics (Tamkin et al., 2020; Chiang & Lee, 2022;
Ibrahim et al., 2022; Agostinelli et al., 2022; Nguyen et al., 2020; You et al., 2021; Dai et al., 2019;
Huang et al., 2022; Ibrahim et al., 2022; Tran et al., 2019; Bao et al., 2019; Van Asch & Daelemans,
2010; Plank & Van Noord, 2011), which are efﬁcient heuristics used to select the most appropriate
source models or pretraining data for a given target task. We note that transferability metrics are
designed to solve ranking problems, different from scaling laws. For example, these metrics answer
questions such as given a pool of source models (or pretraining datasets), which source model (or
pretraining dataset) is the best to ﬁnetune on for a given target task. These metrics are not designed to
predict the performance of the model when key quantities (e.g., pretraining data size) are scaled.

3 SCALING LAWS FOR TRANSFER LEARNING

In this section, we present our proposed scaling laws for translation quality metrics (e.g., BLEU,
ROUGE, and COMET scores) and downstream cross-entropy. We also introduce an alignment score
for translation tasks, discuss when the proposed scaling laws apply, and provide practical guidance
for assessing the value of a pretraining dataset for a given target downstream translation task. The
experimental results will be later discussed in Section 5.

3.1 A SCALING LAW FOR TRANSLATION QUALITY METRICS

Different from cross-entropy and perplexity, which follow a power-law scaling behavior (Kaplan
et al., 2020; Hoffmann et al., 2022), we ﬁnd out that translation scores, such as BLEU and COMET,
scale closer to a log-law, as evident from Figures 1,2, 3, and 4. Therefore, we propose the following
scaling law for translation scores3 as a function of the pretraining dataset size Dp:

f (Dp) = (log(A · Dα

p ))β,

(1)

where A, α, and β are coefﬁcients to be ﬁt. We notice that these coefﬁcients depend on how aligned
the pretraining dataset with the target downstream task (translation from language 1 to language

3In Appendix B, we show that the same law also applies to other tasks, including question answering,

reasoning, reading comprehension, and textual entailment.

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

2) and how large the ﬁnetuning (translation) dataset is. With extensive experiments across several
translation tasks and multilingual pretrained models, we demonstrate that the law in (1) indeed well
describes translation quality scaling, with a small prediction error which we quantify in Appendix C.3.

3.2 TRANSLATION ALIGNMENT SCORE

It is nontrivial to deﬁne a general alignment score that could be used for any pair of pretraining data
and downstream task since it is an open research question what makes a pretraining data more aligned
with (or relevant to) a particular task. Therefore, we focus on a more controlled setting and deﬁne an
alignment score for translation tasks that captures the language overlap between the pretraining data
and the translation task. We note that there might be alternative deﬁnitions of translation alignment.
We propose one that measures what percentage of the languages in the translation task is present in
the pretraining data in a balanced way.

Deﬁnition 1 (Translation Alignment Score). We use the following score to measure alignment
between a multilingual pretraining data D and a translation task T (Lsource, Ldest):
A(D, T (Lsource, Ldest)) = PLsource · PLdest + 0.7 · PLsource + 0.8 · PLdest
(2)
where D is the pretraining data mixture, T (Lsource, Ldest) is the translation task from Lsource to Ldest,
PLsource is percentage of Lsource in D, and PLdest is percentage of Ldest in D.

For instance, for an en-to-fr translation task, a pretraining data mixture with 50% en and 50% fr data
would yield an alignment score of A(D, T (en, fr)) = 0.5 · 0.5 + 0.7 · 0.5 + 0.8 · 0.5 = 1.
Likewise, a pretraining data mixture with 100% en would have an alignment score of
A(D, T (en, fr)) = 1 · 0 + 0.7 · 1 + 0.8 · 0 = 0.7.

3.3

IS CROSS-ENTROPY LOSS ALWAYS A GOOD METRIC?

We compare the downstream cross-entropy loss and the translation scores empirically as prior work
made the assumption that upstream or downstream cross-entropy loss is a good indicator for a model’s
downstream task performance. Following the well-understood scaling behavior of the upstream
cross-entropy loss as a function of the pretraining dataset size (Kaplan et al., 2020; Hoffmann et al.,
2022), we show that the same scaling law can also describe the downstream cross-entropy loss as

L(Dp) = E +

A
Dα
p

,

(3)

where E, A, and α are the coefﬁcients to be optimized. In Section 5, we report BLEU score and
cross-entropy together for a direct comparison and discover several cases where the two metrics do not
correlate well. We provide similar results for COMET score in Appendix C.1. These results support
some of the ﬁndings of Ghorbani et al. (2021) suggesting inconsistency between the translation quality
scores and the cross-entropy, but also shows that the exponential relationship between BLEU score
and cross-entropy advocated by Gordon et al. (2021) does not always hold. More speciﬁcally, our
empirical results show that while cross-entropy loss always monotonically decreases (with appropriate
learning rate) as the pretraining dataset size increases, translation score may show a non-monotonic
trend when the pretraining data is not sufﬁciently aligned with the task. For instance, in Figure 1, we
show the scaling behavior of translation scores like BLEU, ROUGE, and COMET and cross entropy
as the size of a more aligned (A = 1) and a less aligned (A = 0.7) pretraining data increases. The
ﬁrst three plots show that increasing the less aligned data’s size sometimes hurts the translation scores
(more detailed results with full description of datasets and tasks will be in Sections 4 and 5). Even
though they may initially follow the law in (1) for smaller pretraining dataset sizes, the scaling law
breaks for larger data for the “less aligned” pretraining data. However, if we were to look at only
the cross-entropy loss in Figure 1-(right), we would conclude that both the more aligned and less
aligned data bring noticeable improvements to the model and they both are worth being added into
the pretraining mixture – which would be a poor decision.

A remotely related study on the mismatch between the task-related metrics and the cross-entropy
(McKenzie et al., 2023) looked at how the downstream task performance changes as the model grows
and suggested that LLMs may show worse task performance with increased model size but, similar
to our ﬁndings, this is not captured by the monotonically decreasing cross-entropy loss.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 1: Scaling behavior of BLEU, ROUGE, COMET, and cross-entropy when the pretraining and
downstream data are aligned with A = 1 (orange) and A = 0.7 (red). Task: en-to-fr translation.

3.4 WHEN DO SCALING LAWS FALL SHORT IN TRANSFER LEARNING?

While the cross-entropy loss always follows a monotonically decreasing trend which can be captured
by the scaling law in (3), we do not always see a monotonic increase in the translation scores when
increasing the pretraining dataset size (See Figure 1 (red curves) for an example on English-to-French
translation task.). We observe that this only happens when the pretraining dataset is not sufﬁciently
aligned with the translation task – which results in low translation scores overall compared to models
that were pretrained in other datasets. For the pretrained models that lead to high translation scores
after ﬁnetuning, we consistently see that the translation scores increase monotonically and can be
well described with the scaling law in (1). Therefore, whether the scaling law could ﬁt the empirical
translation scores or not could be a good ﬁrst-check in assessing the value of pretraining data for
the downstream translation task. We elaborate more on this in the next section.

3.5 A GUIDE FOR PRETRAINING DATA VALUATION

Finally, combining our ﬁndings on the scaling behavior of translation scores, we propose the following
guide for assessing the value of pretraining dataset for a target downstream task:

1. Given a pretraining dataset, pretrain as long as possible under the given computational and
time constraints4. Periodically choose pretraining checkpoints, ﬁnetune on them, and record
the downstream performance metric (we recommend BLEU, ROUGE, or COMET scores
over cross-entropy due to the discussion in Section 3.4).

2. Since the law in (1) has three coefﬁcients to be ﬁt, once we have 3 pairs of (number of
pretraining tokens seen, translation score), we try to ﬁnd the optimal coefﬁcients and move
onto one of the following steps:
(a) If the translation scores have a non-monotonic behavior (e.g., red curves in Figure 1),
we cannot ﬁt the scaling law. Since the non-monotonic behavior could be an
indication of misalignment (following the discussion in Section 3.4), we expect worse
performance with more pretraining data. Therefore, we recommend checking the score
of the best available ﬁnetuned checkpoint and comparing it to the performance of the
non-pretrained model trained on the downstream task directly. This would indicate
how much value we can get from this pretraining dataset.

(b) If the scaling law ﬁts well (e.g., orange curves in Figure 1), then we make the initial
prediction for the translation score as we increase the pretraining dataset size. If we are
not satisﬁed with the predicted score, then we conclude that it is not worth pretraining
on this dataset. If the predicted score is high enough, then we keep pretraining until
we reach the target score. If the scaling law breaks at any point, we conclude that the
pretraining dataset is not sufﬁciently aligned with the downstream task and pretraining
further may not be beneﬁcial.

4 EXPERIMENTAL SETUP

In the experiments, we ﬁrst pretrain a model without doing more than one pass over any of the
examples. Then, we ﬁnetune selected checkpoints of the pretrained model. Naturally, there is a

4We avoid repeating sequences as repetitions may complicate the scaling behavior (Hernandez et al., 2022;
Muennighoff et al., 2023; Tirumala et al., 2023). This means as pretraining goes on, we effectively pretrain each
checkpoint on a “larger dataset”.

5

Under review as a conference paper at ICLR 2025

one-to-one mapping between the checkpoint number and the number of pretraining tokens seen. This
way, we collect pairs of (number of pretraining tokens, translation score) and (number of pretraining
tokens, downstream cross-entropy loss) to analyze them with the proposed scaling laws in (1) and (3).
All the plots are on a log-log scale. We present BLEU results in this section and provide COMET
results in Appendix C.1. The observations and conclusions are similar in both scores.

Model. We use the 3-billion encoder-decoder T5 model with 24 encoder layers, 24 decoder layers,
embedding dimension 1024, and 32 heads with dimension 128. We note that this is the same model
as T5-3B in Abnar et al. (2022). In Appendix C, we also provide results with the 770-million encoder-
decoder T5 model. This model corresponds to T5-Large in Raffel et al. (2020). We share more details
about the architectures in Appendix A. For encoding the text as WordPiece tokens (Sennrich et al.,
2016; Kudo, 2018), we use SentencePiece (Kudo & Richardson, 2018) trained with a vocabulary of
size 250, 112 that covers all the languages in the MC4 dataset (Raffel et al., 2020).

Datasets. We use the English (en), German (de), French (fr), and Romanian (ro) portions of the
MC4 dataset. We experiment with both pretraining on these languages individually as well as mixing
pairs of languages. In Figure 2, we present results for the models pretrained on (left) a mixture
of 50% en-MC4 + 50% de-MC4, (center) a mixture of 50% en-MC4 + 50% fr-MC4, and (right)
a mixture of 50% en-MC4 + 50% ro-MC4 – meaning that 50% of one pretraining batch is sampled
from en-MC4 and the other 50% is sampled from the other language. Notice that all the pretraining
data-translation task pairs in Figure 2 has an alignment score of A = 1. In Figure 3, we show results
for the models pretrained only on en-MC4, corresponding to an alignment score of A = 0.7. In
Figure 4, in addition to these, we also present results for the models pretrained on a mixture of 30%
en-MC4 + 70%-fr and a mixture of 70% en-MC4 + 30%-fr as well as models pretrained only on
de-MC4, only on fr-MC4, and only on ro-MC4. We ﬁnetune the pretrained models on WMT-17
en-de with 3B tokens (Bojar et al., 2017), WMT-15 en-fr with 21B tokens (Bojar et al., 2014), and
WMT-16 en-ro with 312M tokens (Bojar et al., 2016), separately. To understand the effect of the
ﬁnetuning data size on scaling, we sometimes use a smaller randomly sampled portion from these
translation datasets and indicate the number of tokens used in the plots.

In Appendix B, we provide additional experimental results to demonstrate that the proposed scaling
law is applicable to tasks beyond translation as well. For this, we analyze models pretrained on
en-MC4 and ﬁnetuned on SuperGLUE (Wang et al., 2019), which includes several classes of
tasks such as question answering (BoolQ, MultiRC), reasoning (COPA), reading comprehension
(ReCoRD), and textual entailment (RTE).

Hyperparameters. During pretraining, we use a batch size of 256 and a sequence length of 512 for
1, 000, 000 steps except for the ro-MC4 pretraining. For ro-MC4, we pretrain for 510, 000 steps since
otherwise, we would need to do repetitions over the sequences. Following Raffel et al. (2020), we use
, where n is the current pretraining step
an “inverse square root” learning rate schedule,

1√

max(n,k)

and k = 104. We do a grid search for the base learning rate from {0.05, 0.1, 0.5, 1.0, 2.0, 5.0} and
pick the best one for each pretrained model based on upstream cross entropy. We perform full-weight
ﬁnetuning. During ﬁnetuning, again following Raffel et al. (2020), we use a batch size of 128 and
a sequence length of 512 for 300 steps. We use a constant learning rate by selecting the best from
{0.001, 0.005, 0.01, 0.05, 0.1}. In both stages, we use AdaFactor optimizer (Shazeer & Stern, 2018).

Optimizing the scaling law coefﬁcients. To ﬁt the coefﬁcients in the scaling laws in (1) and
(3), similar to Hoffmann et al. (2022), we use the Huber loss (Huber, 1992) and the L-BFGS
algorithm (Nocedal, 1980) to estimate the scaling law robustly in the presence of outliers. For the
Huber loss, we use δ = 0.1 for the translation scores and δ = 1e−3 for the downstream cross-entropy
loss. We select the best ﬁt among a grid of initializations and report the prediction error computed
via the Huber loss in Appendix C.3. To optimize the coefﬁcients, we use the ﬁrst four data points
that require the smallest amount of pretraining data and leave the remaining data points as held-out
data to evaluate the accuracy of the laws. We note that, ideally, three points should be enough since
both laws have three coefﬁcients to be optimized for. However, adding more points improves the ﬁt
by making the optimization more robust to outliers. We provide more details about how to optimize
the scaling law coefﬁcients in Appendix A.2. We refer the reader to Appendix C.3 for the list of
optimized coefﬁcients and the prediction errors for each law we present in the next section.

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

p))β. (left) WMT-17
Figure 2: (top) BLEU score vs pretraining dataset size: f (Dp) = (log(A · Dα
en-to-de translation task. Pretraining dataset has 50% en-MC4 + 50% de-MC4. Dotted, dashed, and
solid blue curves correspond to the ﬁtted scaling laws for different ﬁnetuning dataset sizes, Df = 6M ,
Df = 31M , Df = 3B tokens, respectively. (center) WMT-15 en-to-fr translation task. Pretraining
dataset has 50% en-MC4 and 50% fr-MC4. Dotted, dashed, and solid orange curves correspond to
the ﬁtted scaling laws for different ﬁnetuning dataset sizes, Df = 42M , Df = 210M , Df = 21B
tokens, respectively. (right) WMT-16 en-to-ro translation task. Pretraining dataset has 50% en-MC4
+ 50% ro-MC4. Dotted, dashed, and solid green curves correspond to the ﬁtted scaling laws for
different ﬁnetuning dataset sizes, Df = 625K, Df = 3M , Df = 312M tokens, respectively.
(bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: L(Dp) = E + A
.
Dα
p
Same models as the top row. For all the plots, the markers are the actual experimental results and
the black horizontal curves correspond to the non-pretrained model directly trained on the task
dataset. The ﬁnetuning dataset size increases in the order of dotted-dashed-solid for all the
curves including the black horizontal lines. Note that all the plots have alignment score of A = 1.

5 RESULTS AND ANALYSIS

In Figure 2, we analyze the models that are pretrained on different portions of (left) a mixture of
50% en-MC4 + 50% de-MC4 (A = 1), (center) a mixture of 50% en-MC4 + 50% fr-MC4 (A = 1),
and (right) a mixture of 50% en-MC4 + 50% ro-MC4 (A = 1). These models are then ﬁnetuned on
different portions of (left) en-de, (center) en-fr, and (right) en-ro translation datasets. In the top row,
we report the BLEU score and, in the bottom row, we report the downstream cross-entropy loss. The
dotted, dashed, and solid lines correspond to the scaling laws in (1) and (3) for different ﬁnetuning
dataset sizes Df . The black lines correspond to “non-pretrained” models (randomly initialized) that
are directly trained on different portions of the ﬁnetuning dataset. In all cases, the scaling laws ﬁt well
to the empirical results (the markers) with prediction error at most 0.061 for the BLEU score (δ = 0.1)
and 5.95e − 12 for the downstream cross-entropy (δ = 1e − 3) (see Appendix C.3 for more details).
As expected, as the ﬁnetuning dataset size increases (e.g., going in the order of dotted-dashed-solid
lines), the BLEU score increases and the cross-entropy loss decreases smoothly and monotonically.
Similarly, as the pretraining dataset size Dp increases (along the x-axis), we see improvements in both
metrics. Notice that the improvements by an increase in the pretraining dataset size is more effective
for smaller ﬁnetuning datasets. When the ﬁnetuning dataset is large enough (e.g., solid lines), BLEU
score is more or less constant regardless of the pretraining dataset size. In fact, we see little to no
improvement of pretraining compared to the non-pretrained models (black lines) when the ﬁnetuning
dataset is large. This implies that, for these tasks, there is no need to pretrain the models when
the ﬁnetuning dataset is large enough (We note that typically supervised ﬁnetuning data is not
as widely available as unsupervised data due to its cost – hence pretraining on unsupervised

7

Under review as a conference paper at ICLR 2025

data is important in practice.). Luckily, we can correctly predict whether this is going to be the
case (i.e., whether the available ﬁnetuning data is enough to eliminate pretraining altogether)
with the use of scaling laws.

In Figure 3, we change the pretraining dataset to 100% en-MC4 in all plots, giving an alignment
score of A = 0.7. Intuitively, we expect this dataset to be less aligned with the translation tasks than
the multilingual pairs in Figure 2 since it does not include one of the languages in the translation
tasks. Indeed, we see smaller BLEU score and higher cross-entropy loss in general for the same
ﬁnetuning dataset size. Most of the conclusions from Figure 2 carry over to the results in Figure 3. For
instance, the pretraining data matters less when the ﬁnetuning dataset is large enough. One noticeable
difference is in the BLEU scores for the en-fr translation task (center). We see that, for Df = 42M
and Df = 210M , the scaling law for BLEU score actually breaks once the pretraining dataset size
passes a threshold while the cross-entropy loss scales as expected. This is counter-intuitive because
the BLEU score sometimes decreases for larger pretraining dataset. Notice that this break in scaling
law does not happen in en-de or en-ro translation tasks as the scaling law ﬁts well to the pretraining
data with prediction error at most 0.025 for these tasks (δ = 0.1). To better investigate this, in
Figure 4, we take a closer look at some less aligned pretraining datasets due to the choice of language.

Figure 3: Same as Figure 2, except that the pretraining dataset is 100% en-MC4 for all plots, i.e., the
alignment score is A = 0.7.
In Figure 4-(left), we provide the scaling laws for en-de translation task where the pretraining datasets
are 100% en-MC4 (A = 0.7, same as Figure 3-(left)), 50% en-MC4 and 50% de-MC4 (A = 1,
same as Figure 2-(left)), 100% de-MC4 (A = 0.8), 100% fr-MC4 (A = 0, less aligned), and 100%
ro-MC4 (A = 0, less aligned). Notice that the last two pretraining datasets are expected to be the least
aligned with the translation task since the translation pair does not include these languages. We see
that, despite this, the scaling laws consistently ﬁt well for both the BLEU score and the cross-entropy
loss. However, this is not always the case for the en-fr translation task. In Figure 4-(right), we provide
the scaling laws for the en-fr translation task where the pretraining datasets are different mixtures
of en-MC4 and fr-MC4 datasets. We also include the “less aligned” pretraining datasets such as
100% de-MC4 (A = 0) and 100% ro-MC4 (A = 0). Surprisingly, we see that the scaling law for the
BLEU score breaks after some point for the only-English (100% en-MC4, A = 0.7), only-German
(100% de-MC4, A = 0), and only-Romanian (100% ro-MC4, A = 0) pretraining datasets while
the cross-entropy loss always follows the scaling law in (3). Interestingly, we do not observe such a
break in the BLEU score scaling for the only-French (100% fr-MC4, A = 0.8) pretraining dataset –
hinting that not including French data in pretraining leads to poor scaling in the en-fr translation task
but not including English does not have such an effect. We also notice that the BLEU score is the
lowest for these three pretraining datasets where scaling breaks. This suggests that the scaling law
in (1) works well for the BLEU score as long as the pretraining dataset has the promise to give
rise to a good performance. However, when the scaling law does not ﬁt well, we may suspect

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Figure 4: Comparison of scaling behavior for different pretraining datasets. (top) BLEU score vs
p))β. (left) WMT-17 en-de translation task. (right)
pretraining dataset size: f (Dp) = (log(A · Dα
(bottom) Cross-entropy (CE) validation loss vs pretraining
WMT-15 en-fr translation task.
dataset size: L(Dp) = E + A
. Same as the top row but for CE loss instead of BLEU score.
Dα
p
For all the plots, the markers are the actual experimental results and the black horizontal curves
correspond to the non-pretrained model directly trained on the task dataset.

the BLEU score to be low overall. Therefore, whether we can ﬁt the scaling law for the BLEU
score seems to give a good indication about the degree of alignment between the pretraining
data and the particular translation task.

Remark 1. We observe another interesting phenomenon in Figure 4. For both en-de and en-fr tasks,
100% en-MC4 leads to signiﬁcantly worse BLEU score and downstream cross-entropy than the more
aligned 50% en-MC4 + 50% de/fr-MC4 balanced datasets, respectively. However, de-MC4 and
fr-MC4 perform almost as well as the balanced datasets in en-de and en-fr tasks. We believe this is
because, in these translation tasks, the model generates text in German/French (not English), and
de/fr-MC4 pretraining is more helpful than en-MC4. We leave further investigation to future work.

We also highlight that we cannot make any strong conclusion about the degree of alignment of
the pretraining dataset with the task by only looking at the downstream cross-entropy loss because
of the inconsistency with the BLEU score, a task-related metric, observed in the en-fr plots in
Figures 3 and 4. This is a counter-example for the claim by Gordon et al. (2021) that the two
metrics have an exponential relation. To better demonstrate this, in Figure 5, we provide a BLEU
score vs. downstream cross-entropy log-log plot for en-de and en-fr translation tasks, respectively.
While the two metrics indeed seem correlated in Figure 5-(left) on the en-de task, we observe a
somewhat arbitrary relation for the en-fr task in Figure 5-(right) in some cases – which clearly cannot
be explained with an exponential relation. This suggests that downstream cross-entropy is not
always a good indicator for BLEU score and raises the question whether the existing scaling
laws for cross-entropy are actually useful predictors for models’ downstream behavior.

All the observations on BLEU score presented in this section carry over to COMET score as well
(see Figure 1 and Appendix C.1).

Remark 2. To better understand the root cause of the non-monotonic behavior of the BLEU score
when the alignment is not sufﬁcient (i.e., when the BLEU score ﬂuctuates with more pretraining

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Figure 5: BLEU score vs. downstream cross-entropy loss. (left) For en-de translation task, we see
a consistent correlation between the two metrics for all the pretraining datasets. This supports the
ﬁndings of Gordon et al. (2021). (right) For en-fr translation task, the two metrics usually show
an arbitrary relation. Sometimes, the BLEU score increases while the cross-entropy also increases.
Unlike the en-de results (left), the exponential relation in (Gordon et al., 2021) is not observed here.

data), we revisit its deﬁnition. Recall that the common form of BLEU score is as follows

BLEU = brevity-penalty ·

(cid:33)1/4

precisioni

,

(cid:32) 4
(cid:89)

i=1

(4)

where precisionn refers to the precision of n-grams, and the second term is the geometric mean of the
precision when n is varied from 1 to 4. In all the experiments, we observe brevity-penalty = 1, i.e.,
the non-monotonic behavior comes from the precision term, not the brevity penalty.

5.1 OTHER TASKS

In Appendix B, we show that the proposed log scaling law is not only applicable to the translation
scores/tasks but also to metrics on question answering, reasoning, reading comprehension, and textual
entailment tasks within SuperGLUE (Wang et al., 2019). Our results demonstrate that the same
scaling law captures the scaling of these metrics as the pretraining data grows.

6 DISCUSSION AND CONCLUSION

We study the scaling behavior of the downstream performance in machine translation as the pretraining
data grows and propose scaling laws for both downstream cross-entropy and translation quality
metrics. We demonstrate through extensive experiments that the scaling behavior is signiﬁcantly
inﬂuenced by (1) the degree of alignment between the pretraining and the downstream data and (2)
the ﬁnetuning dataset size. In favorable cases where the distributions are sufﬁciently aligned, we show
that downstream translation quality, measured by translation scores, can be accurately predicted using
a log scaling law. However, with less alignment, there are cases where translation scores ﬂuctuate
unpredictably whereas downstream cross-entropy improves monotonically. We also observe that
when the ﬁnetuning dataset size is sufﬁciently large, pretraining has little to no value. Our ﬁndings
highlight the importance of studying downstream performance metrics and not making decisions
solely based on cross-entropy (whether upstream or downstream).

Limitations. Our work goes beyond cross-entropy loss to understand and predict the downstream
model performance at scale. While the proposed laws ﬁt the empirical data well and predict the
translation scores at scale successfully when there is sufﬁcient alignment, there are cases where these
scores do not scale monotonically. Our work identiﬁes many such cases; however, as mentioned in
Remark 1, a more linguistic approach into alignment in translation could provide better understanding.

Reproducibility Statement. We used publicly available datasets and models, and speciﬁed their
versions with proper citations in Section 4 and Appendix A. We provided details on the training
procedure and hyperparameters for both pretraining and ﬁnetuning stages.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of
large scale pre-training. In International Conference on Learning Representations, 2022. URL
https://openreview.net/forum?id=V3C8p78sDa.

Andrea Agostinelli, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. Transferability metrics
for selecting source model ensembles. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 7936–7946, 2022.

Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin
Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José
G. C. de Souza, and Andre Martins. Tower: An open multilingual large language model for
In First Conference on Language Modeling, 2024. URL https:
translation-related tasks.
//openreview.net/forum?id=EHPns3hVkj.

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural

scaling laws. arXiv preprint arXiv:2102.06701, 2021.

Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur,
and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International
Conference on Machine Learning, pp. 1466–1482. PMLR, 2022.

Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas.
In 2019 IEEE

An information-theoretic approach to transferability in task transfer learning.
international conference on image processing (ICIP), pp. 2309–2313. IEEE, 2019.

Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan

Szpektor. The second PASCAL recognising textual entailment challenge. 2006.

Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The

ﬁfth PASCAL recognizing textual entailment challenge. 2009.

Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias
Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri,
Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia
Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference
In Proceedings of the First Conference on Machine Translation, pp.
on machine translation.
131–198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL
http://www.aclweb.org/anthology/W/W16/W16-2301.

Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian
Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri,
Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 conference on
machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation,
Volume 2: Shared Task Papers, pp. 169–214, Copenhagen, Denmark, September 2017. Association
for Computational Linguistics. URL http://www.aclweb.org/anthology/W17-4717.

Ondˇrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes
Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014
workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical
machine translation, pp. 12–58, 2014.

Cheng-Han Chiang and Hung-yi Lee. On the transferability of pre-trained language models: A
study from artiﬁcial datasets. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence,
volume 36, pp. 10518–10525, 2022.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina
Toutanova. BoolQ: Exploring the surprising difﬁculty of natural yes/no questions. In Proceedings
of NAACL-HLT 2019, 2019.

Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment
In Machine learning challenges. evaluating predictive uncertainty, visual object

challenge.
classiﬁcation, and recognising tectual entailment, pp. 177–190. Springer, 2006.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. Using similarity measures to select
pretraining data for ner. In Proceedings of the 2019 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long
and Short Papers), pp. 1460–1470, 2019.

Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank:
Investigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn
und Bedeutung 23. Data can be found at https://github.com/mcdm/CommitmentBank/.

Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws

for multilingual neural machine translation. arXiv preprint arXiv:2302.09650, 2023.

Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolutional

networks. Advances in neural information processing systems, 31, 2018.

Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia,
Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International
Conference on Learning Representations, 2021.

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing
textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment
and paraphrasing, pp. 1–9. Association for Computational Linguistics, 2007.

Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural
machine translation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-
tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language
Processing, pp. 5915–5922, Online and Punta Cana, Dominican Republic, November 2021.
Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.478. URL https:
//aclanthology.org/2021.emnlp-main.478.

Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of

the IEEE/CVF International Conference on Computer Vision, pp. 4918–4927, 2019.

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo
Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative
modeling. arXiv preprint arXiv:2010.14701, 2020.

Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer.

arXiv preprint arXiv:2102.01293, 2021.

Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson
Elhage, Zac Hatﬁeld-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability
of learning from repeated data. arXiv preprint arXiv:2205.10487, 2022.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.
Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.

Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy trans-
ferability estimation. In International Conference on Machine Learning, pp. 9201–9225. PMLR,
2022.

Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology

and distribution, pp. 492–518. Springer, 1992.

Marcus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074, 2021.

Shibal Ibrahim, Natalia Ponomareva, and Rahul Mazumder. Newer is not always better: Rethinking
transferability metrics, their peculiarities, stability and performance. In Joint European Conference
on Machine Learning and Knowledge Discovery in Databases, pp. 693–709. Springer, 2022.

Achin Jain, Gurumurthy Swaminathan, Paolo Favaro, Hao Yang, Avinash Ravichandran, Hrayr
Harutyunyan, Alessandro Achille, Onkar Dabeer, Bernt Schiele, Ashwin Swaminathan, et al. A
meta-learning approach to predicting performance and data requirements. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3623–3632, 2023.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Mark Johnson, Peter Anderson, Mark Dras, and Mark Steedman. Predicting accuracy on large
datasets from smaller pilot data. In Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics (Volume 2: Short Papers), pp. 450–455, 2018.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott
Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361, 2020.

Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking
In
beyond the surface: A challenge set for reading comprehension over multiple sentences.
Proceedings of the 2018 Conference of the North American Chapter of the Association for Com-
putational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 252–262,
2018.

Taku Kudo. Subword regularization: Improving neural network translation models with multiple sub-
word candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pp. 66–75, 2018.

Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword

tokenizer and detokenizer for neural text processing. EMNLP 2018, pp. 66, 2018.

Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Cheng, and Jiajun Chen. Eliciting the translation ability
of large language models via multilingual ﬁnetuning with translation instructions. Transactions of
the Association for Computational Linguistics, 12:576–592, 2024.

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization

branches out, pp. 74–81, 2004.

Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu,
Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. Inverse scaling: When bigger isn’t
better. arXiv preprint arXiv:2306.09479, 2023.

Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi
Maeda, and Kohei Hayashi. A scaling law for syn2real transfer: How much is your pre-training
effective? In Joint European Conference on Machine Learning and Knowledge Discovery in
Databases, pp. 477–492. Springer, 2022.

Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra
Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language
models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
https://openreview.net/forum?id=j5BuTrEj35.

Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure
to evaluate transferability of learned representations. In International Conference on Machine
Learning, pp. 7294–7305. PMLR, 2020.

Jorge Nocedal. Updating quasi-newton matrices with limited storage. Mathematics of computation,

35(151):773–782, 1980.

OpenAI.

Learning to reason with llms.

learning-to-reason-with-llms/, 2024.

In https://openai.com/index/

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association
for Computational Linguistics, pp. 311–318, 2002.

Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: The word-in-context dataset for
evaluating context-sensitive meaning representations. In Proceedings of NAACL-HLT, 2019.

Barbara Plank and Gertjan Van Noord. Effective measures of domain similarity for parsing. In
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human
Language Technologies, pp. 1566–1576, 2011.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a uniﬁed text-to-text
transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Comet: A neural framework for mt
evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pp. 2685–2702, 2020.

Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova,
Alon Lavie, Luisa Coheur, and André FT Martins. Comet-22: Unbabel-ist 2022 submission for the
metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp.
578–585, 2022.

Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives:
An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with
subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pp. 1715–1725, 2016.

Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold.

arXiv preprint arXiv:2004.10802, 2020.

Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost.

In International Conference on Machine Learning, pp. 4596–4604. PMLR, 2018.

Zhiqiang Shen, Zhuang Liu, Jianguo Li, Yu-Gang Jiang, Yurong Chen, and Xiangyang Xue. Object
detection from scratch with deep supervision. IEEE transactions on pattern analysis and machine
intelligence, 42(2):398–412, 2019.

David Stap, Eva Hasler, Bill Byrne, Christof Monz, and Ke Tran. The ﬁne-tuning paradox: Boost-
ing translation quality without sacriﬁcing LLM abilities. In Lun-Wei Ku, Andre Martins, and
Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com-
putational Linguistics (Volume 1: Long Papers), pp. 6189–6206, Bangkok, Thailand, August
2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.336. URL
https://aclanthology.org/2024.acl-long.336.

Craig Stewart, Ricardo Rei, Catarina Farinha, and Alon Lavie. Comet-deploying a new state-of-the-art

mt evaluation metric in production. In AMTA (2), pp. 78–109, 2020.

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable
effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on
computer vision, pp. 843–852, 2017.

Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating transferability
in pretrained language models. In Findings of the Association for Computational Linguistics:
EMNLP 2020, pp. 1393–1401, 2020.

Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan
Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efﬁciently: Insights from
pretraining and ﬁnetuning transformers. In International Conference on Learning Representations,
2021.

Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari S Morcos. D4: Improving llm
pretraining via document de-duplication and diversiﬁcation. In Thirty-seventh Conference on
Neural Information Processing Systems Datasets and Benchmarks Track, 2023.

Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classiﬁ-
cation tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
1395–1405, 2019.

Vincent Van Asch and Walter Daelemans. Using domain similarity for performance estimation. In
Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pp.
31–36, 2010.

14

Under review as a conference paper at ICLR 2025

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer
Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language
understanding systems. arXiv preprint 1905.00537, 2019.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine
translation: Boosting translation performance of large language models. In The Twelfth Interna-
tional Conference on Learning Representations, 2024. URL https://openreview.net/
forum?id=farT6XXntP.

Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of
pre-trained models for transfer learning. In International Conference on Machine Learning, pp.
12133–12143. PMLR, 2021.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
12104–12113, 2022.

Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, and Orhan
Firat. Examining scaling and transfer of language model architectures for machine translation. In
International Conference on Machine Learning, pp. 26176–26192. PMLR, 2022.

Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.
ReCoRD: Bridging the gap between human and machine commonsense reading comprehension.
arXiv preprint 1810.12885, 2018.

Zhang Zhuocheng, Shuhao Gu, Min Zhang, and Yang Feng. Scaling law for document neural
machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023,
pp. 8290–8303, 2023.

Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le.
Rethinking pre-training and self-training. Advances in neural information processing systems, 33:
3833–3845, 2020.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A ADDITIONAL EXPERIMENTAL DETAILS

For the T5-3B experiments, pretraining for 1M steps takes 15-20 hours and ﬁnetuning takes 5-7 hours
on an 8x8 TPU. For the sake of anonymity, we are unable to provide further information on compute
speciﬁcations at this time, but we will add details upon acceptance.

A.1 MODEL ARCHITECTURES

We provide the architecture details of the T5-3B and T5-770M models in Tables 1 and 2. These
models were initially introduced by Raffel et al. (2020).

Table 1: T5-3B (Raffel et al., 2020) architecture details.

Embedding Dimension
Number of Heads
Number of Encoder Layers
Number of Decoder Layers
Head Dimension
MLP Dimension

1024
32
24
24
128
16384

Table 2: T5-770M (Raffel et al., 2020) architecture details.

Embedding Dimension
Number of Heads
Number of Encoder Layers
Number of Decoder Layers
Head Dimension
MLP Dimension

1024
16
24
24
64
2816

A.2 OPTIMIZING THE SCALING LAW COEFFICIENTS

In this section, we provide more details on how we optimize the coefﬁcients of the scaling laws.
Following Hoffmann et al. (2022), we use the Huber loss (Huber, 1992) to minimize overﬁtting to
the outliers. Huber loss is particularly useful to suppress the effect of the outlier data points in the
optimization problem. More speciﬁcally, if the data point with value r is predicted by the law as ˆr,
the loss for that data point would be

(cid:96)δ(r, ˆr) =

(cid:26) 1

2 (r − ˆr)2
δ · (|r − ˆr| − 1

for |r − ˆr| ≤ δ,

2 δ) otherwise.

(5)

Due to the numerical range difference between the COMET/BLEU score (between 0 and 100) and
the downstream cross-entropy typically taking much smaller values, we use δ = 0.1 for BLEU score
law in (1) and δ = 1e − 3 for the downstream cross-entropy law in (3).

For optimization, we use the L-BFGS algorithm (Nocedal, 1980). Speciﬁcally, for COMET/BLEU
score law in (1), we solve

min
E,A,α,β

(cid:88)

Data point i

(cid:96)δ(log fi, log ˆf (Dpi)),

(6)

where Dpi is the pretraining dataset size and fi is the COMET/BLEU score for the data point i, and
ˆf (·) is the approximation for the optimal law f (·). Similarly, for the downstream cross-entropy loss
law in (3), we solve

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

min
E,A,α

(cid:88)

Data point i

(cid:96)δ(log Li, log ˆL(Dpi)),

(7)

where Dpi is the pretraining dataset size and Li is the downstream cross-entropy loss for the data
point i, and ˆL(·) is the approximation for the optimal law L(·).

B SUPERGLUE EXPERIMENTS

Figure 6 demonstrates how SuperGLUE (Wang et al., 2019) task metrics such as Boolean Ques-
tions (BoolQ) (Clark et al., 2019), CommitmentBank (CB) (De Marneffe et al., 2019), Choice of
Plausible Alternatives (COPA) (Roemmele et al., 2011), Multi-Sentence Reading Comprehension
(MultiRC) (Khashabi et al., 2018), Recognizing Textual Entailment (RTE) (Dagan et al., 2006;
Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009), Reading Comprehen-
sion with Commonsense Reasoning Dataset (ReCoRD) (Zhang et al., 2018), and Word-in-Context
(WiC) (Pilehvar & Camacho-Collados, 2019) scale as the pretraining data grows. For these exper-
iments, we use T5-3B model pretrained on en-MC4 data (same as Section 5). For ﬁnetuning on
SuperGLUE, we use a batch size of 128 and a sequence length of 512 for 300 steps. We use a constant
learning rate by selecting the best from {0.001, 0.005, 0.01, 0.05, 0.1, 0.5}.
p))β, that was demonstrated to
The results indicate that the same scaling law, f (Dp) = (log(A · Dα
ﬁt well to translation scores in Section 5 also captures the scaling of question answering (BoolQ,
MultiRC), reasoning (COPA), reading comprehension (ReCoRD), and textual entailment (RTE) tasks,
as well.

C ADDITIONAL EXPERIMENTAL RESULTS

In this section, we provide additional experimental results that we had to skip in the main body due
to the page limit.

C.1 RESULTS WITH COMET SCORES

We extend our experimental evaluation to COMET score, which we had to skip in the main body
due to the page limit. In Figure 7, we provide the COMET scores for the models previously used
in Figures 2 and 3 for BLEU score and cross-entropy. Similar to BLEU score, the law given in (1)
well describes the scaling behavior of COMET score, when there is sufﬁcient alignment between the
pretraining and dowsntream data (Figure 7-(top)). When the alignment is not sufﬁcient (Figure 7-
(bottom)), again similar to the BLEU score, COMET score ﬂuctuates and sometimes gets worse with
more pretraining.

C.2 RESULTS ON T5-770M

In Figures 8 and 9, we present results similar to Figures 2 and 3 in Section 5, but for T5-770M
instead of T5-3B. In general, we observe a similar trend. The proposed scaling laws describe the
downstream behavior well when the pretraining and downstream data are aligned. Similar to the
results in T5-3B in the main body of the paper, in Figure 9-(top, right), we observe a break in the
scaling law when the pretraining dataset is 100% en-MC4 and the task is en-fr translation – suggesting
the same misalignment for this pretraining data and task that was also observed in Section 5 on the
larger T5-3B model.

C.3 OPTIMIZED COEFFICIENTS AND PREDICTION ERRORS OF THE SCALING LAWS

In Tables 3, 4, 5, and 6, we provide the optimized coefﬁcients for the scaling laws plotted in Figures 2
and 3 together with the prediction error.

17

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure 6: SuperGLUE scores vs pretraining dataset size: f (Dp) = (log(A · Dα
p))β. Pretraining
dataset is en-MC4 and ﬁnetuning dataset is SuperGLUE. For all the plots, the markers are the actual
experimental results and the black horizontal curves correspond to the non-pretrained model directly
trained on the SuperGLUE dataset.

18

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Figure 7: (top) COMET score results for the 50%-50% balanced experiments in Figure 2:
f (Dp) = (log(A · Dα
p))β. (left) WMT-17 en-to-de translation task. Pretraining dataset has 50%
en-MC4 + 50% de-MC4. Dotted, dashed, and solid blue curves correspond to the ﬁtted scaling
laws for different ﬁnetuning dataset sizes, Df = 6M , Df = 31M , Df = 3B tokens, respectively.
(right) WMT-15 en-to-fr translation task. Pretraining dataset has 50% en-MC4 and 50% fr-MC4.
Dotted, dashed, and solid orange curves correspond to the ﬁtted scaling laws for different ﬁnetuning
dataset sizes, Df = 42M , Df = 210M , Df = 21B tokens, respectively. (bottom) COMET score
results for the 100% en-MC4 pretraining experiments in Figure 3: Same as the top row, except
that the pretraining dataset is 100% en-MC4. For all the plots, the markers are the actual experimental
results and the black horizontal curves correspond to the non-pretrained model directly trained on
the task dataset. The ﬁnetuning dataset size increases in the order of dotted-dashed-solid for all
the curves including the black horizontal lines.

Table 3: The coefﬁcients for the BLEU score law f (Dp) = (log(A · Dα
p ))β for the results in Figure 2-
(top). For the BLEU score laws, we use δ = 0.1 for the Huber Loss. We report log A instead of A
since A typically takes very small and very large values.

Pretraining Dataset

Finetuning Dataset

Finetuning Dataset Size

log A

α

β

Prediction Error

50% en + 50% de-MC4
50% en + 50% de-MC4
50% en + 50% de-MC4

50% en + 50% fr-MC4
50% en + 50% fr-MC4
50% en + 50% fr-MC4

50% en + 50% ro-MC4
50% en + 50% ro-MC4
50% en + 50% ro-MC4

WMT-17 en-de
WMT-17 en-de
WMT-17 en-de

WMT-15 en-fr
WMT-15 en-fr
WMT-15 en-fr

WMT-16 en-ro
WMT-16 en-ro
WMT-16 en-ro

−180.75
−1.68 × 103
−1.64 × 108
−1.82 × 104
−2.33 × 104
5.08 × 103

−36.02
−0.115.03
−1.82 × 104

9.00
84.04
9.91 × 106
8.98 × 102
1.21 × 103
4.61 × 108

1.77
5.69
9.04 × 102

0.75
0.49
0.19

0.42
0.40
0.16

1.28
0.89
0.40

0.034
0.050
0.048

0.061
0.013
0.005

0.042
0.015
0.015

6M
31M
3B

42M
210M
21B

625K
3M
312M

19

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

p))β. (left) WMT-17
Figure 8: (top) BLEU score vs pretraining dataset size: f (Dp) = (log(A · Dα
en-to-de translation task. Pretraining dataset has 50% en-MC4 + 50% de-MC4. Dotted and dashed
blue curves correspond to the ﬁtted scaling laws for different ﬁnetuning dataset sizes, Df = 6M
and Df = 31M tokens, respectively. (right) WMT-15 en-to-fr translation task. Pretraining dataset
has 50% en-MC4 and 50% fr-MC4. Dotted and dashed orange curves correspond to the ﬁtted
scaling laws for different ﬁnetuning dataset sizes, Df = 42M and Df = 210M tokens, respectively.
(bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: L(Dp) = E + A
.
Dα
p
Same models as the top row. For all the plots, the markers are the actual experimental results and the
black horizontal curves correspond to the non-pretrained model directly trained on the task dataset.
The ﬁnetuning dataset size increases in the order of dotted-dashed for all the curves including
the black horizontal lines.

Table 4: The coefﬁcients for the downstream cross-entropy law L(Dp) = E + A
Dα
p
Figure 2-(bottom). For the downstream cross-entropy laws, we use δ = 10−5 for the Huber Loss.

for the results in

Pretraining Dataset

Finetuning Dataset

Finetuning Dataset Size

50% en + 50% de-MC4
50% en + 50% de-MC4
50% en + 50% de-MC4

50% en + 50% fr-MC4
50% en + 50% fr-MC4
50% en + 50% fr-MC4

50% en + 50% ro-MC4
50% en + 50% ro-MC4
50% en + 50% ro-MC4

WMT-17 en-de
WMT-17 en-de
WMT-17 en-de

WMT-15 en-fr
WMT-15 en-fr
WMT-15 en-fr

WMT-16 en-ro
WMT-16 en-ro
WMT-16 en-ro

6M
31M
3B

42M
210M
21B

625K
3M
312M

E
3.21 × 10−5
3.28 × 10−5
2.24 × 10−5
2.72 × 10−5
2.57 × 10−5
1.11 × 10−7
2.45 × 10−5
2.62 × 10−5
2.08 × 10−5

A

35.45
4.70 × 102
2.56 × 10−2
2.01 × 106
1.75 × 107
3.41 × 10−5

0.49
2.40
3.94

α

0.64
0.78
0.36

1.18
1.30
1.82 × 10−2

0.41
0.49
0.53

Prediction Error
1.36 × 10−12
3.17 × 10−12
5.76 × 10−14
7.52 × 10−13
2.24 × 10−13
5.20 × 10−14
3.61 × 10−12
2.19 × 10−12
5.95 × 10−12

20

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

p))β. (left) WMT-
Figure 9: (top) BLEU score vs pretraining dataset size: f (Dp) = (log(A · Dα
17 en-to-de translation task. Dotted and dashed red curves correspond to the ﬁtted scaling laws
for different ﬁnetuning dataset sizes, Df = 6M and Df = 31M tokens, respectively. (right)
WMT-15 en-to-fr translation task. Dotted and dashed red curves correspond to the ﬁtted scaling
laws for different ﬁnetuning dataset sizes, Df = 42M and Df = 210M tokens, respectively.
(bottom) Cross-entropy (CE) validation loss vs pretraining dataset size: L(Dp) = E + A
.
Dα
p
Same models as the top row. For all the plots, the markers are the actual experimental results and the
black horizontal curves correspond to the non-pretrained model directly trained on the task dataset.
The ﬁnetuning dataset size increases in the order of dotted-dashed for all the curves including
the black horizontal lines.

Table 5: The coefﬁcients for the BLEU score law f (Dp) = (log(A · Dα
p ))β for the results in Figure 3-
(top). For the BLEU score laws, we use δ = 0.1 for the Huber Loss. We report log A instead of A
since A typically takes very small and very large values.

Pretraining Dataset

Finetuning Dataset

Finetuning Dataset Size

log A

α

β

Prediction Error

100% en-MC4
100% en-MC4
100% en-MC4

100% en-MC4
100% en-MC4
100% en-MC4

100% en-MC4
100% en-MC4
100% en-MC4

WMT-17 en-de
WMT-17 en-de
WMT-17 en-de

WMT-15 en-fr
WMT-15 en-fr
WMT-15 en-fr

WMT-16 en-ro
WMT-16 en-ro
WMT-16 en-ro

6M
31M
3B

42M
210M
21B

625K
3M
312M

−1.88
−1.81 × 104
1.02 × 10−7

1.00
−6.38 × 107
204.81

−10.54
−40.41
3.61

0.15
896.12
104.92
2.57 × 10−5
3.43 × 106
3.80 × 1014

0.55
2.11
8.17 × 105

3.30
0.28
0.42
1.11 × 104
0.20
9.97 × 10−3

1.12
0.79
0.19

0.014
0.006
0.015

0.042
0.034
0.004

0.008
0.025
0.018

21

Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Table 6: The coefﬁcients for the downstream cross-entropy law L(Dp) = E + A
Dα
p
Figure 3-(bottom). For the downstream cross-entropy laws, we use δ = 10−5 for the Huber Loss.

for the results in

Pretraining Dataset

Finetuning Dataset

Finetuning Dataset Size

100% en-MC4
100% en-MC4
100% en-MC4

100% en-MC4
100% en-MC4
100% en-MC4

100% en-MC4
100% en-MC4
100% en-MC4

WMT-17 en-de
WMT-17 en-de
WMT-17 en-de

WMT-15 en-fr
WMT-15 en-fr
WMT-15 en-fr

WMT-16 en-ro
WMT-16 en-ro
WMT-16 en-ro

6M
31M
3B

42M
210M
21B

625K
3M
312M

E
3.22 × 10−13
3.24 × 10−5
2.24 × 10−5
3.49 × 10−5
4.24 × 10−5
1.26 × 10−7
5.79 × 10−12
1.78 × 10−12
5.85 × 10−5

A
3.18 × 10−3
5.20 × 10−3
2.56 × 10−2
1.05 × 10−2
19.39
2.59 × 10−5
1.03 × 10−3
9.98 × 10−4
1.37 × 103

α

0.15
0.20
0.36

0.25
0.66
4.81 × 10−3
7.76 × 10−2
8.33 × 10−2
0.88

Prediction Error
5.79 × 10−12
9.25 × 10−13
5.76 × 10−14
3.63 × 10−13
5.40 × 10−13
3.63 × 10−14
5.56 × 10−12
8.23 × 10−12
3.05 × 10−13

22

