Predictable behavior from scaling advanced AI systems is an extremely desirable property.
Although a well-established literature exists on how pretraining performance scales, the literature on how particular downstream capabilities scale is significantly muddier.
In this work, we take a step back and ask: why has predicting specific downstream capabilities with scale remained elusive?
While many factors are certainly responsible, we identify a new factor that makes modeling scaling behavior on widely used multiple-choice question-answering benchmarks challenging.
Using five model families and twelve well-established multiple-choice benchmarks, we show that downstream performance is computed from negative log likelihoods via a sequence of transformations that progressively degrade the statistical relationship between performance and scale.
We then reveal the mechanism causing this degradation: downstream metrics require comparing the correct choice against a small number of specific incorrect choices, meaning accurately predicting downstream capabilities requires predicting not just how probability mass concentrates on the correct choice with scale, but also how probability mass fluctuates on specific incorrect choices with scale.
We empirically study how probability mass on the correct choice co-varies with probability mass on incorrect choices with increasing compute, suggesting that scaling laws for incorrect choices might be achievable.
Our work also explains why pretraining scaling laws are commonly regarded as more predictable than downstream capabilities and contributes towards establishing scaling-predictable evaluations of frontier AI models.

§ INTRODUCTION

Predictable scaling behavior of frontier AI systems such as GPT-4 [OpenAI, 2024, OpenAI et al., 2024], Claude [Anthropic, 2024] and Gemini [Team et al., 2023, Reid et al., 2024] is crucial for anticipating their capabilities and informing key decisions around their development and deployment [Anthropic, 2023, OpenAI, 2023, Dragan et al., 2024].
While scaling laws describing relationships between parameters, data, compute, and pretraining loss are well-established [Hestness et al., 2017, Rosenfeld et al., 2019, Henighan et al., 2020, Kaplan et al., 2020, Gordon et al., 2021, Hernandez et al., 2021, Jones, 2021, Zhai et al., 2022, Hoffmann et al., 2022, Clark et al., 2022, Neumann and Gros, 2022, Hernandez et al., 2022, Maloney et al., 2022, Sardana and Frankle, 2023, Muennighoff et al., 2024, Besiroglu et al., 2024], the literature is less conclusive concerning predicting specific downstream capabilities with scale.
For instance, prior work has observed that performance on standard natural language processing (NLP) benchmarks can exhibit emergent abilities [Brown et al., 2020, Ganguli et al., 2022, Srivastava et al., 2022, Wei et al., 2022] where performance changes unpredictably with scale, with further work demonstrating that such unpredictable changes might at times be artifacts of researchers' analyses, i.e., choices of metrics and lack of resolution [Srivastava et al., 2022, Schaeffer et al., 2023, Hu et al., 2024].
More recently, Du et al., 2024 claim that downstream capabilities can be predicted, but only after the pretraining cross-entropy loss falls below a certain threshold, and Gadre et al., 2024 claim that while performance on individual tasks can be difficult to predict, aggregating results across dozens of diverse benchmarks yields clearer scaling trends.
In this work, we take a step back and ask: why has predicting specific downstream capabilities with scale remained elusive?

Multiple-choice benchmark accuracy is computed from negative log-likelihoods via a sequence of transformations that degrades predictability.
Computing Accuracy begins with computing the negative log-likelihoods of each choice, then negating and exponentiating each to obtain the probability of each choice (A). Choices are then restricted to a set of available choices by masking invalid continuations, and renormalizing to obtain relative probability mass on each choice (B).
Lastly, the model's choice is defined as $\argmax_{i} \{p^{\text{Choices}}(\text{Available Choice}_i) \}$, and  is 1 if and only if the model's choice is the correct choice (C).

While many factors are certainly responsible, we identify a new factor that makes modeling the scaling behavior on widely used multiple-choice question-answering benchmarks challenging.
We demonstrate that common multiple-choice metrics like Accuracy, Brier Score, and Probability Correct are computed from raw model outputs (log probabilities) via a sequence of transformations that progressively degrades the statistical relationship between those metrics and the scaling parameters (parameters, data, compute).
The cause is that these metrics rely on a direct comparison between the ground truth output and a small set of specific incorrect outputs.
As a result, accurately predicting downstream performance requires modeling not only the concentration of probability mass on the correct output with increasing compute, but also modeling the fluctuations of probability mass on particular incorrect alternatives, which (to date) is a necessary but unaddressed step.
We then empirically study how probability mass on incorrect choices fluctuates with increasing compute.
Our findings help explain the apparent unpredictability of individual downstream metrics and the greater robustness of pretraining loss scaling laws, which do not depend on specific incorrect choices.
More broadly, we argue that a precise understanding of the factors affecting downstream performance is essential for designing evaluations that can reliably track the progression of frontier AI capabilities.

§ METHODOLOGY: DATA FOR STUDYING SCALING OF DOWNSTREAM CAPABILITIES

To study how downstream capabilities on specific tasks change with scale for different model families, we generated per-sample scores from a large number of model families and multiple-choice NLP benchmarks. To ensure the computed scores were consistent with prior work, we used EleutherAI's Language Model (LM) Evaluation Harness [Gao et al., 2023] rather than implementing our own evaluations.

Model Families

Because our goal was to explore the scaling behavior of evaluations with increasing compute, we chose to evaluate model families with dense combinations of parameter counts and token counts. This includes the following families (additional details in App. <ref>):

* Pythia [Biderman et al., 2023]: The Pythia family contains 8 models from 70M to 12B parameters trained on the Pile [Gao et al., 2020] for 300B tokens.
We use 8 checkpoints per size of the non-deduplicated variants.

* Cerebras-GPT [Dey et al., 2023]: The Cerebras-GPT family contains 7 models ranging from 111M to 13B parameters.
The models were trained on the Pile [Gao et al., 2020] for different durations as part of a scaling study with $\sim 20 \times$ tokens to parameters in a “Chinchilla”-optimal manner [Hoffmann et al., 2022].

* OLMo [Groeneveld et al., 2024]: The OLMo family contains a 1B parameter model trained for 3T tokens and two 7B parameter models trained for 2T-2.5T tokens. We selected 7 checkpoints for 1B (spanning 84B[OLMo 1B checkpoints below 84B tokens were unfortunately accidentally lost by their creators.] to 3T tokens) and 7 checkpoints for 7B (spanning 4B to 2.4T tokens).

* INCITE [AI, 2023]: The INCITE family contains 3B and 7B parameter models, trained on 0.8T and 1T tokens of RedPajama-v1[Computer, 2023]. The 3B model has only a single checkpoint, so we excluded it. We found this family to be a slight outlier from other families, which we speculate is because its pretraining data were contaminated by benchmarks [Elazar et al., 2023].

* LLM360 [Liu et al., 2023]:
LLM360 includes two 7B parameter LLMs trained on 1.3T and 1.4T tokens. We selected 13 checkpoints of Amber spaced approximately logarithmically.

NLP Benchmarks We evaluated the above model families on widely-used multiple-choice benchmarks for assessing comprehension, reasoning, and world knowledge: AI2 Reasoning Challenge (ARC) Easy and Hard [Clark et al., 2018], HellaSwag [Zellers et al., 2019], MathQA [Amini et al., 2019], MCTACO [Zhou et al., 2019], MMLU [Hendrycks et al., 2020], OpenbookQA [Mihaylov et al., 2018], PIQA [Bisk et al., 2020], RACE [Lai et al., 2017], SciQ [Welbl et al., 2017], SIQA [Sap et al., 2019],  WinoGrande [Keisuke et al., 2019] and XWinoGrad En [Muennighoff et al., 2023]. For MMLU, we analyzed each of the 57 subjects (e.g., Abstract Algebra) independently. For each benchmark, we used default evaluation settings from the LM Evaluation Harness [Gao et al., 2023].

Performance Metrics We used three common multiple-choice metrics [Srivastava et al., 2022, Schaeffer et al., 2023, Du et al., 2024]: ,  [Brier, 1950], and probability mass on the correct choice relative to the available choices.

Compute Budget Calculations

Following prior work [Kaplan et al., 2020], we approximated[This approximation neglects FLOP costs associated with attention calculations over sequence length; however, such operations are negligible so long as $d_{model} >> n_{ctx} / 12$, and this approximation is therefore standard in most language model scaling law analyses.] the pretraining compute $C$ (in terms of training FLOP) of a given model checkpoint as a function of the parameter count (excluding the embedding layer) $N$ and the amount of training data seen in tokens $D$:

\begin{equation*}
C = C(N, D) \; \approx \; 6 \, N\, D
\end{equation*}

§ WHAT MAKES PREDICTING DOWNSTREAM PERFORMANCE DIFFICULT?

Performance on multiple choice benchmarks is commonly presented as ,  [Schaeffer et al., 2023], or probability mass on the correct choice out of the available choices
[Du et al., 2024]. These quantities are computed via a sequence of transformations that begins with the negative log-likelihood of the correct choice on this particular benchmark sample as some function $f(\cdot, \cdot)$ of compute:

\begin{equation}
\mathcal{L}_{\theta}^{\text{Vocab}}(\text{Correct Choice}) = f(\text{Compute}, \text{Benchmark Datum})  \label{eqn:loss}
\end{equation}

Two details are critical. Firstly, this negative log-likelihood is not computed in expectation over a corpus; it is specific to this particular singular datum in the benchmark. All the scores we discuss are per-datum. Secondly, this negative log-likelihood is computed over the vocabulary of the model. One can then compute the probability mass of the correct choice, again with respect to the vocabulary:

\begin{equation}
p_{\theta}^{\text{Vocab}}(\text{Correct Choice}) = \exp \big(-\mathcal{L}_{\theta}^{\text{Vocab}}(\text{Correct Choice}) \big) \label{eqn:p_correct_vocab}
\end{equation}

Next, probabilities are restricted to the set of available choices $\{\text{Available Choice}_i \}_{i}^{|\text{Available Choices}|}$ by masking invalid continuations and normalizing again with respect to this set:

\begin{align}
p_{\theta}^{\text{Choices}}(\text{Correct Choice}) &\defeq \frac{p_{\theta}^{\text{Vocab}}(\text{Correct Choice})}{\sum_{i} \, p_{\theta}^{\text{Vocab}}(\text{Available Choice}_i)} \label{eqn:p_correct_choices}
\end{align}

We emphasize the support over the token space of the model versus over the set of available choices in the benchmark's question because, as we will show, this crucially affects predictability.
Finally, one uses the choices-normalized probability masses to compute standard downstream metrics:

\begin{equation}
\texttt{Accuracy}_{\theta} \defeq  \mathbbm{1} \Big( \text{Correct Choice} == \argmax_{i} \Big\{ p_{\theta}^{\text{Choices}}(\text{Available Choice}_i) \Big\} \Big) \label{eqn:accuracy}
\end{equation}

\begin{equation}
\texttt{Brier Score}_{\theta} \defeq \sum_{i} \Big(\mathbbm{1}(\text{Available Choice}_i == \text{Correct Choice}) - p_{\theta}^{\text{Choices}}(\text{Available Choice}_i) \Big)^2 \label{eqn:brier}
\end{equation}

where $\mathbbm{1}(\cdot)$ is an indicator variable. We demonstrate that this sequence of transformations degrades how predictable performance is with scale before identifying the underlying mechanism.

Distributions of score-compute correlations and their corresponding complementary cumulative distribution functions. Left: For each benchmark, model family, performance metric, and correlation metric, one can compute how scores correlate with compute. This yields a distribution (over samples) of score-compute correlations. Note: the uniform (yellow) distribution is small but non-zero everywhere. Right: To easily extract what fraction of samples in a benchmark have score-compute correlations above any given threshold, we convert the probability distributions to complementary cumulative distribution functions, i.e., $1$ minus the (empirical) cumulative distribution function (CDF). Top: Schematic idealized distributions. Bottom: Real data on ARC Challenge [Clark et al., 2018].

Multiple-choice metrics like  and  are computed via a sequence of transformations that degrades correlations between performance scores and pretraining compute. (A) Initially, scores under $\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ and compute are highly correlated. Transforming $\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ into $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ has no effect for rank correlations. (B) Transforming $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ into $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ decorrelates scores from compute. (C) Transforming $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ into  minorly decreases score-compute correlations. (D) Transforming $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ into  more substantially decorrelates scores from compute. Benchmark: ARC Challenge [Clark et al., 2018]. Correlation: Spearman. Results are consistent across benchmarks and all three correlation metrics;
for more, see App. <ref>.

To quantify how this sequence of transformations affects predictability of performance, we measured how per-sample scores correlate with pretraining compute, and then studied how the distribution (over samples) of correlation values shifted as one transitions from loglikelihoods to $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ to $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ to  or .
Specifically, for each combination of $(\text{model family, benchmark, performance metric, correlation metric})$, we computed a correlation value for each sample in the benchmark between pretraining compute and scores.
This yielded a distribution (over samples) of correlation values for the combination (Fig. <ref> left).
Visualizing the distribution of correlations for the combination told us what fraction of samples in the benchmark yielded scores that are correlated, uncorrelated or anticorrelated with compute (Fig. <ref> right).
We used three standard correlation metrics - Pearson, Kendall [Kendall, 1938] and Spearman [Spearman, 1961]) - and found consistent results.

All four statistics of score-compute correlation distributions demonstrate that transforming $\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ into  causes score-compute correlations to deteriorate.
We find a consistent trend across benchmarks and model families for three correlation metrics (Spearman, Pearson and Kendall) and for four statistics of correlation distributions (mean, median, the area under the survival function, and negative Wasserstein distance from perfect correlation or perfect anti-correlation) that the sequence of transformations degrades score-compute correlations, as shown by the right-to-left $\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$-to-$p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$-to-$p_{\theta}^{\text{Choices}}(\text{Correct Choice})$-or--to- vertical stripes. See App. Figs. <ref>, <ref>, <ref> for other correlation metrics and other score-compute correlation distribution statistics.

We present ARC Challenge [Clark et al., 2018] as an illustrative benchmark to demonstrate how the sequence of transformations affects the distribution of score-compute correlations, but note that all other benchmarks exhibited similar patterns (App. <ref>). We visualized the distributions via their complementary (empirical) cumulative distribution functions (complementary CDFs) (App. <ref>):

\begin{equation}
\hat{S}(c) \defeq \frac{1}{N} \sum_{n=1}^{N} \mathbbm{1}\{C_n > c\}
\label{eqn:empirical_survival_function},
\end{equation}

where $N$ is the number of data in the benchmark and
$C_n$ is the correlation (over the models in the model family) between compute and scores on the $n$-th datum in the benchmark. For a given threshold $c$, the complementary CDF $\hat{S}(c)$ returns the fraction of the benchmark's samples with score-compute correlations greater than the threshold $c$ (Fig. <ref>A). Beginning with log likelihoods, approximately 90% of samples exhibit score-compute correlations $> 0.75$, regardless of the model family (Fig. <ref>A). Transforming negative log-likelihoods into probability masses $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ does not affect the distribution of score-compute correlations for Spearman and Kendall.
However, transforming $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ into $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ decreases the distribution of score-compute correlations (Fig. <ref>B), with only 40% of samples having score-compute correlations $> 0.75$.
Transforming $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ into  has little-to-no effect (Fig. <ref>C) but transforming into  (Fig. <ref>D) furthers decreases score-compute correlations.
To quantitatively test whether these transformations indeed decrease the correlation between scores and compute, we measured four statistics of these score-compute correlation distributions: the mean, the median, the area under the complementary CDF and the negative[We chose the negative Wasserstein distance for consistency with the other statistics: higher values correspond to higher correlations between scores and compute.] of the minimum of two Wasserstein distances: between the empirical correlation distribution and an ideal distribution of all correlations $=1$, and between the empirical distribution and an ideal distribution of all correlations $=-1$. Across the four summary statistics, for most benchmarks and for most model families, we discovered a consistent ordering of metrics of the score-compute correlation distributions (Fig. <ref>):

\begin{align*}
&\text{Corr}\big(\text{Compute}, \textcolor{red}{\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice})} \big) \\
&\quad \geq \; \text{Corr}\big(\text{Compute}, \textcolor{red!50!white}{p_{\theta}^{\text{Vocab}}(\text{Correct Choice})}\big)\\
&\quad  > \; \text{Corr}\big(\text{Compute}, \textcolor{blue!50!white}{p_{\theta}^{\text{Choices}}(\text{Correct Choice})}\big)\\
&\quad \geq \; \text{Corr}\big(\text{Compute}, \textcolor{blue!75!white}{\texttt{Brier Score}}\big)\\
&\quad  > \; \text{Corr}\big(\text{Compute}, \textcolor{blue!90!black}{\texttt{Accuracy}}\big)\\
\end{align*}

§ PROBABILITY MASSES ON INCORRECT CHOICES CAUSE UNPREDICTABILITY

What is the mechanism that degrades how correlated scores are with compute? All three metrics with degraded correlations - $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$, , and  - depend not just on how the model's probability mass $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ concentrates on the correct choice as compute increases, but also depend on how the model's probability mass fluctuates on incorrect available choices $\{ p_{\theta}^{\text{Vocab}}(\text{Incorrect Choice}) \}_{\text{Incorrect Choices}}$ as compute increases. As an example, suppose $p_{\theta}^{Vocab}(\text{Correct Choice}) = 0.4$ on a 4-way multiple-choice question; what is the accuracy? Spreading the remaining mass uniformly on the incorrect choices will make  $=1$, whereas concentrating mass on a single incorrect choice will make  $=0$.

Predictability deteriorates because of probability mass fluctuating on specific incorrect choices with scale. Left: Transitioning from $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ to $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ demonstrates that $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ contains little information about $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ and vice versa; loosely speaking, any value of one can map to any value of the other. Center: While $p_{\theta}^{\text{Choices}}(\text{Correct Choice}) > 0.5$ must yield $\texttt{Accuracy} = 1$, for any $p_{\theta}^{\text{Choices}}(\text{Correct Choice}) < 0.5$, knowing $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ contains little information about $\texttt{Accuracy}$ and vice versa. Right:  is more predictable from $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ than , but still quite variable. Three benchmarks shown: MathQA [Amini et al., 2019], MMLU Conceptual Physics [Hendrycks et al., 2020], SciQ [Welbl et al., 2017].

To demonstrate how drastically the probability mass placed on incorrect choices can alter performance, we visualized the relations between pairs of metrics immediately preceding and following a given transformation (Fig. <ref>). For negative log-likelihood of the correct choice and $p_{\theta}^{Vocab}(\text{Correct Choice})$ (not pictured), we observed a clean correspondence between performance under the metric and compute: one can reliably map a given value of these metrics to compute, and vice versa. In contrast, once performance is evaluated using a metric that is a function of the incorrect choices - $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$,  or  - nearly any value of a score under one metric can map to any value of $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ or $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ respectively (Fig. <ref>), breaking the chain along which one can cleanly infer compute from an observed metric. We can see that , a metric meant to produce more continuous scores [Schaeffer et al., 2023], is less variable than  provided a known $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$, but it cannot recover information about $p_{\theta}^{\text{Vocab}}(\text{Correct Choice})$ that is lost when shifting to $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$. We next show that this is because of the additional information regarding the underdetermined values of $p_{\theta}^{\text{Choices}}(\text{Incorrect Choice})$ for each incorrect choice.

Probability mass on the correct choices and the incorrect choices are correlated, but can fluctuate substantially. Probability mass on correct choices and incorrect choices positively covaries and typically increases with compute. However, the spread is large: for any given value of $p^{\text{Vocab}}_{\theta} (\text{Correct Choice})$, the mass on incorrect choices can vary by many orders of magnitude.

§ SCALING BEHAVIOR OF PROBABILITY MASS ON INCORRECT CHOICES

In order to accurately predict performance on multiple-choice question-answering benchmarks, one must predict not just how probability mass concentrates on correct choices with scale, but also how probability mass fluctuates on incorrect choices with scale. For metrics like $\texttt{Accuracy}$, these predictions must be made for each sample because knowing the average mass (across many data) placed on incorrect choices says little about how much mass is placed on any single incorrect choice for a single sample. We conclude by providing preliminary evidence that achieving such a feat might be possible. Specifically, we test how probability masses on correct choices and probability masses on incorrect choices covary with increasing compute (Fig. <ref>). Multiple benchmarks display strong positive relationships between mass on correct choices and mass on incorrect choices, suggesting that fitting per-sample scaling trends for each incorrect choice might be possible; doing so would enable predicting changepoints in metrics like  or . However, whether per-benchmark per-sample per-choice scaling trends can be fit and accurately extrapolated is unclear since the spread varies by several orders of magnitude. We leave this challenge to future work.

§ DISCUSSION, RELATED WORK AND FUTURE DIRECTIONS

This work identifies a factor that induces unpredictability in multiple-choice assessments of frontier AI models, as well as the underlying mechanism: probability mass on incorrect choices.

Our results have implications for the design of future evaluations for frontier AI models that are reliably predictable with scaling. We hope that our work will be extended to further the science of scaling-predictable evaluation of AI systems, especially for complex and important model capabilities.
We note several future directions for extension of our work and we hope that the community also adopts our framing to further improve scaling-predictable evaluations.

Related Work

For a comprehensive exposition of related work, please see App. <ref>.

Takeaway #1: Think through your metrics!

If one cares about scaling-predictable evaluations, then one needs to think through how their evaluations transform raw model outputs into useful signals to know what to expect.

Takeaway #2: Continuous metrics are insufficient to guarantee predictable changes.

As shown by $p_{\theta}^{\text{Choices}}(\text{Correct Choice})$ & , even “continuous” metrics can be unpredictable,
e.g., if the metric weighs correct behavior against specific incorrect behaviors.

Takeaway #3: Recommended scaling-predictable metrics for pretraining practitioners.

Pretraining practitioners seeking scaling-predictable signals for capabilities should perhaps focus on \(p_{\theta}^\textit{Vocab}(\text{Correct Choice})\) on relevant benchmarks. Scores under this metric provide smoother scaling trends and are arguably more interpretable than the pretraining loss.

Takeaway #4: Evaluations should be reshaped based on intended desiderata.

Too often, we take evaluations as frozen static objects, but evaluations should be adapted to pertinent goals. For instance, if the goal is to predict capabilities with scale, evaluations should be designed or adapted to be scaling-predictable.

Direction 1: Beyond Multiple Choice Benchmarks

Our study is restricted to benchmarks evaluated via loglikelihood-based multiple-choice formats. While we believe this is inherently valuable due to the usefulness and prevalence of such tasks, this limits the application of our findings. We hope that our discoveries and proposed mechanisms may be used to inform the study of predictable and reliable evaluation writ large, and that future work should explore the extent to which our findings can be generalized to more complex capabilities. Our findings corroborate those of Lyu et al., 2024, who find that multiple-choice answer scores often diverge from generative evaluations. Consequently, a particularly important direction for further study is to investigate generative evaluations, which may contain similar transformations distancing performance from the observed loss.

Direction 2: Predicting Benchmark Performance A Priori

Our work provides an explanation why multiple-choice benchmark performance is not easily predictable for metrics such as  and , as observed in the literature [Du et al., 2024]. However, our analyses assume access to entire model families' scores across several orders of magnitude of pretraining FLOPs, and do not employ backtesting, as sensibly recommended by Owen, 2024. A predictive model should be able to identify change points well in advance on standard metrics like  or .

[Achiam et al., 2023]
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.

[AI, 2023]
T. AI.
Releasing 3b and 7b redpajama-incite family of models including base, instruction-tuned & chat models.
<https://www.together.ai/blog/redpajama-models-v1>, 2023.
Accessed: 2024-05-19.

[Amini et al., 2019]
A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi.
MathQA: Towards interpretable math word problem solving with operation-based formalisms.
In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357–2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
URL <https://aclanthology.org/N19-1245>.

[Anthropic, 2023]
Anthropic's responsible scaling policy.
<https://www.anthropic.com/news/anthropics-responsible-scaling-policy>, 2023.
Accessed: 2024-05-19.

[Anthropic, 2024]
Introducing the next generation of claude.
<https://www.anthropic.com/news/claude-3-family>, 2024.
Accessed: 2024-05-19.

[Beeching et al., 2023]
E. Beeching, C. Fourrier, N. Habib, S. Han, N. Lambert, N. Rajani, O. Sanseviero, L. Tunstall, and T. Wolf.
Open llm leaderboard.
<https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard>, 2023.

[Besiroglu et al., 2024]
T. Besiroglu, E. Erdil, M. Barnett, and J. You.
Chinchilla scaling: A replication attempt, 2024.

[Biderman et al., 2023]
S. Biderman, U. S. Prashanth, L. Sutawika, H. Schoelkopf, Q. Anthony, S. Purohit, and E. Raff.
Emergent and predictable memorization in large language models, 2023a.

[Biderman et al., 2023]
S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal.
Pythia: A suite for analyzing large language models across training and scaling, 2023b.

[Biderman et al., 2024]
S. Biderman, H. Schoelkopf, L. Sutawika, L. Gao, J. Tow, B. Abbasi, A. F. Aji, P. S. Ammanamanchi, S. Black, J. Clive, A. DiPofi, J. Etxaniz, B. Fattori, J. Z. Forde, C. Foster, M. Jaiswal, W. Y. Lee, H. Li, C. Lovering, N. Muennighoff, E. Pavlick, J. Phang, A. Skowron, S. Tan, X. Tang, K. A. Wang, G. I. Winata, F. Yvon, and A. Zou.
Lessons from the trenches on reproducible evaluation of language models.
arXiv preprint, 2024.

[Bisk et al., 2020]
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi.
Piqa: Reasoning about physical commonsense in natural language.

[Bowman, 2023]
S. R. Bowman.
Eight things to know about large language models, 2023.

[Brier, 1950]
G. W. Brier.
Verification of forecasts expressed in terms of probability.
Monthly weather review, 780 (1):0 1–3, 1950.

[Brown et al., 2020]
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.
Language models are few-shot learners.
Advances in neural information processing systems, 33:0 1877–1901, 2020.

[Caballero et al., 2023]
E. Caballero, K. Gupta, I. Rish, and D. Krueger.
Broken neural scaling laws, 2023.

[Clark et al., 2022]
A. Clark, D. De Las Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud, et al.
Unified scaling laws for routed language models.
In International Conference on Machine Learning, pages 4057–4086. PMLR, 2022.

[Clark et al., 2018]
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
arXiv preprint arXiv:1803.05457, 2018.

[Computer, 2023]
T. Computer.
Redpajama: An open source recipe to reproduce llama training dataset, 2023.
URL <https://github.com/togethercomputer/RedPajama-Data>.

[contributors, 2023]
W. contributors.
Survival function, 2023.
URL <https://en.wikipedia.org/wiki/Survival_function>.
[Online; accessed 22-May-2024].

[DeepSeek-AI et al., 2024]
DeepSeek-AI, :, X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, H. Gao, K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo, G. Hao, Z. Hao, Y. He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y. Li, Y. K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu, X. Liu, X. Liu, Y. Liu, H. Lu, S. Lu, F. Luo, S. Ma, X. Nie, T. Pei, Y. Piao, J. Qiu, H. Qu, T. Ren, Z. Ren, C. Ruan, Z. Sha, Z. Shao, J. Song, X. Su, J. Sun, Y. Sun, M. Tang, B. Wang, P. Wang, S. Wang, Y. Wang, Y. Wang, T. Wu, Y. Wu, X. Xie, Z. Xie, Z. Xie, Y. Xiong, H. Xu, R. X. Xu, Y. Xu, D. Yang, Y. You, S. Yu, X. Yu, B. Zhang, H. Zhang, L. Zhang, L. Zhang, M. Zhang, M. Zhang, W. Zhang, Y. Zhang, C. Zhao, Y. Zhao, S. Zhou, S. Zhou, Q. Zhu, and Y. Zou.
Deepseek llm: Scaling open-source language models with longtermism, 2024.

[Dey et al., 2023]
N. Dey, G. Gosal, Zhiming, Chen, H. Khachane, W. Marshall, R. Pathria, M. Tom, and J. Hestness.
Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023.

[Dragan et al., 2024]
A. Dragan, H. King, and A. Dafoe.
Introducing the frontier safety framework.
<https://deepmind.google/discover/blog/introducing-the-frontier-safety-framework/>, 2024.
Accessed: 2024-05-19.

[Du et al., 2024]
Z. Du, A. Zeng, Y. Dong, and J. Tang.
Understanding emergent abilities of language models from the loss perspective, 2024.

[Elazar et al., 2023]
Y. Elazar, A. Bhagia, I. H. Magnusson, A. Ravichander, D. Schwenk, A. Suhr, E. P. Walsh, D. Groeneveld, L. Soldaini, S. Singh, et al.
What's in my big data?
In The Twelfth International Conference on Learning Representations, 2023.

[Gadre et al., 2024]
S. Y. Gadre, G. Smyrnis, V. Shankar, S. Gururangan, M. Wortsman, R. Shao, J. Mercat, A. Fang, J. Li, S. Keh, R. Xin, M. Nezhurina, I. Vasiljevic, J. Jitsev, A. G. Dimakis, G. Ilharco, S. Song, T. Kollar, Y. Carmon, A. Dave, R. Heckel, N. Muennighoff, and L. Schmidt.
Language models scale reliably with over-training and on downstream tasks, 2024.

[Ganguli et al., 2022]
D. Ganguli, D. Hernandez, L. Lovitt, A. Askell, Y. Bai, A. Chen, T. Conerly, N. Dassarma, D. Drain, N. Elhage, et al.
Predictability and surprise in large generative models.
In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764, 2022.

[Gao et al., 2020]
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy.
The pile: An 800gb dataset of diverse text for language modeling, 2020.

[Gao et al., 2022]
L. Gao, J. Schulman, and J. Hilton.
Scaling laws for reward model overoptimization, 2022.

[Gao et al., 2023]
L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac'h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou.
A framework for few-shot language model evaluation, 12 2023.
URL <https://zenodo.org/records/10256836>.

[Gordon et al., 2021]
M. A. Gordon, K. Duh, and J. Kaplan.
Data and parameter scaling laws for neural machine translation.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915–5922, 2021.

[Groeneveld et al., 2024]
D. Groeneveld, I. Beltagy, P. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. H. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. R. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, L. Soldaini, N. A. Smith, and H. Hajishirzi.
Olmo: Accelerating the science of language models, 2024.

[Hendrycks et al., 2020]
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.
Measuring massive multitask language understanding.
arXiv preprint arXiv:2009.03300, 2020.

[Hendrycks et al., 2021]
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt.
Measuring mathematical problem solving with the math dataset.

[Henighan et al., 2020]
T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al.
Scaling laws for autoregressive generative modeling.
arXiv preprint arXiv:2010.14701, 2020.

[Hernandez et al., 2021]
D. Hernandez, J. Kaplan, T. Henighan, and S. McCandlish.
Scaling laws for transfer.
arXiv preprint arXiv:2102.01293, 2021.

[Hernandez et al., 2022]
D. Hernandez, T. Brown, T. Conerly, N. DasSarma, D. Drain, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, T. Henighan, T. Hume, et al.
Scaling laws and interpretability of learning from repeated data.
arXiv preprint arXiv:2205.10487, 2022.

[Hestness et al., 2017]
J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou.
Deep learning scaling is predictable, empirically.
arXiv preprint arXiv:1712.00409, 2017.

[Hoffmann et al., 2022]
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al.
Training compute-optimal large language models.
arXiv preprint arXiv:2203.15556, 2022.

[Hu et al., 2024]
S. Hu, X. Liu, X. Han, X. Zhang, C. He, W. Zhao, Y. Lin, N. Ding, Z. Ou, G. Zeng, Z. Liu, and M. Sun.
Predicting emergent abilities with infinite resolution evaluation, 2024.

[Huang et al., 2024]
Y. Huang, J. Zhang, Z. Shan, and J. He.
Compression represents intelligence linearly, 2024.

[Isik et al., 2024]
B. Isik, N. Ponomareva, H. Hazimeh, D. Paparas, S. Vassilvitskii, and S. Koyejo.
Scaling laws for downstream task performance of large language models, 2024.

[Jones, 2021]
A. L. Jones.
Scaling scaling laws with board games.
arXiv preprint arXiv:2104.03113, 2021.

[Joshi et al., 2017]
M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer.
triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension.
arXiv e-prints, art. arXiv:1705.03551, 2017.

[Kaplan et al., 2020]
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei.
Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361, 2020.

[Keisuke et al., 2019]
S. Keisuke, L. Ronan, B. Chandra, and C. Yejin.
Winogrande: An adversarial winograd schema challenge at scale.

[Kendall, 1938]
M. G. Kendall.
A new measure of rank correlation.
Biometrika, 300 (1/2):0 81–93, 1938.

[Kleinbaum and Klein, 2012]
D. G. Kleinbaum and M. Klein.
Survival Analysis: A Self-Learning Text.
Springer, 3 edition, 2012.
ISBN 978-1441966452.

[Kwiatkowski et al., 2019]
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov.
Natural questions: a benchmark for question answering research.
Transactions of the Association of Computational Linguistics, 2019.

[Lai et al., 2017]
G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy.
RACE: Large-scale ReAding comprehension dataset from examinations.
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics.
URL <https://aclanthology.org/D17-1082>.

[Liu et al., 2023]
Z. Liu, A. Qiao, W. Neiswanger, H. Wang, B. Tan, T. Tao, J. Li, Y. Wang, S. Sun, O. Pangarkar, R. Fan, Y. Gu, V. Miller, Y. Zhuang, G. He, H. Li, F. Koto, L. Tang, N. Ranjan, Z. Shen, X. Ren, R. Iriondo, C. Mu, Z. Hu, M. Schulze, P. Nakov, T. Baldwin, and E. P. Xing.
Llm360: Towards fully transparent open-source llms, 2023.

[Lyu et al., 2024]
C. Lyu, M. Wu, and A. F. Aji.
Beyond probabilities: Unveiling the misalignment in evaluating large language models.
arXiv preprint arXiv:2402.13887, 2024.

[Maloney et al., 2022]
A. Maloney, D. A. Roberts, and J. Sully.
A solvable model of neural scaling laws.
arXiv preprint arXiv:2210.16859, 2022.

[McCandlish et al., 2018]
S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team.
An empirical model of large-batch training, 2018.

[McKenzie et al., 2022]
I. McKenzie, A. Lyzhov, A. Parrish, A. Prabhu, A. Mueller, N. Kim, S. Bowman, and E. Perez.
The inverse scaling prize, 2022.
URL <https://github.com/inverse-scaling/prize>.

[Mihaylov et al., 2018]
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal.
Can a suit of armor conduct electricity? a new dataset for open book question answering.
In EMNLP, 2018.

[Muckatira et al., 2024]
S. Muckatira, V. Deshpande, V. Lialin, and A. Rumshisky.
Emergent abilities in reduced-scale generative language models, 2024.

[Muennighoff et al., 2023]
N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, X. Tang, D. Radev, A. F. Aji, K. Almubarak, S. Albanie, Z. Alyafeai, A. Webson, E. Raff, and C. Raffel.
Crosslingual generalization through multitask finetuning, 2023.

[Muennighoff et al., 2024]
N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel.
Scaling data-constrained language models.
Advances in Neural Information Processing Systems, 36, 2024.

[Neumann and Gros, 2022]
O. Neumann and C. Gros.
Scaling laws for a multi-agent reinforcement learning model.
arXiv preprint arXiv:2210.00849, 2022.

[OpenAI, 2023]
Openai’s approach to frontier risk.
<https://openai.com/global-affairs/our-approach-to-frontier-risk/>, 2023.
Accessed: 2024-05-19.

[OpenAI, 2024]
Hello gpt-4o.
<https://openai.com/index/hello-gpt-4o/>, 2024.
Accessed: 2024-05-16.

[OpenAI et al., 2024]
OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. Button, T. Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey,
W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Łukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O'Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita,
A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan,
W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, and B. Zoph.
Gpt-4 technical report, 2024.

[Owen, 2024]
D. Owen.
How predictable is language model benchmark performance?, 2024.

[Reid et al., 2024]
M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
arXiv preprint arXiv:2403.05530, 2024.

[Rosenfeld et al., 2019]
J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit.
A constructive prediction of the generalization error across scales.
In International Conference on Learning Representations, 2019.

[Ruan et al., 2024]
Y. Ruan, C. J. Maddison, and T. Hashimoto.
Observational scaling laws and the predictability of language model performance, 2024.

[Sap et al., 2019]
M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi.
Social iqa: Commonsense reasoning about social interactions.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019a.

[Sap et al., 2019]
M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi.
Socialiqa: Commonsense reasoning about social interactions, 2019b.

[Sardana and Frankle, 2023]
N. Sardana and J. Frankle.
Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2023.

[Schaeffer et al., 2023]
R. Schaeffer, B. Miranda, and S. Koyejo.
Are emergent abilities of large language models a mirage?
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 55565–55581. Curran Associates, Inc., 2023.
URL <https://proceedings.neurips.cc/paper_files/paper/2023/file/adc98a266f45005c403b8311ca7e8bd7-Paper-Conference.pdf>.

[Spearman, 1961]
C. Spearman.
The proof and measurement of association between two things.

[Srivastava et al., 2022]
A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
arXiv preprint arXiv:2206.04615, 2022.

[Team et al., 2023]
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.
Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.

[Wei et al., 2022]
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al.
Emergent abilities of large language models.
arXiv preprint arXiv:2206.07682, 2022.

[Welbl et al., 2017]
J. Welbl, N. F. Liu, and M. Gardner.
Crowdsourcing multiple choice science questions.
In L. Derczynski, W. Xu, A. Ritter, and T. Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics.
URL <https://aclanthology.org/W17-4413>.

[Zellers et al., 2019]
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi.
Hellaswag: Can a machine really finish your sentence?
arXiv preprint arXiv:1905.07830, 2019.

[Zhai et al., 2022]
X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer.
Scaling vision transformers.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104–12113, 2022.

[Zhou et al., 2019]
B. Zhou, D. Khashabi, Q. Ning, and D. Roth.
“going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding.
In K. Inui, J. Jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363–3369, Hong Kong, China, Nov. 2019. Association for Computational Linguistics.
URL <https://aclanthology.org/D19-1332>.

§ RELATED WORK

Language Model Evaluation

The capabilities of AI models are typically evaluated using constructed datasets to assess performance on a specific task, acting as a proxy for some real-world usage scenario. However, performing robust and reliable evaluations is a challenge, with many potential pitfalls and unsolved problems [Biderman et al., 2024]. For example, we might prefer to ask models open-ended questions and evaluate their answers in natural language, but it then often becomes difficult to robustly score the resulting model outputs, especially for partial correctness. For this reason, it is common practice for evaluation benchmarks to simplify their scoring via approximations, such as extracting a sub-string from free-form outputs heuristically [Joshi et al., 2017, Kwiatkowski et al., 2019, Hendrycks et al., 2021] and checking that it matches a specific gold target string, or casting a task to a multiple-choice format, in which a closed set of correct and incorrect answers is known, and the model's answer is determined by selecting the most likely option among these strings. For more details on the precise procedures typically used for multiple choice elsewhere in the literature, see Biderman et al., 2024. We believe that the multiple-choice format is valuable, due to its flexibility, popularity and relevance [Brown et al., 2020, Beeching et al., 2023, Biderman et al., 2024], but we discuss its limitations in Section <ref>.

Scaling Laws

Many neural networks exhibit power-law scaling of the pretraining loss as a function of the amount of compute, data, or parameters used for training [Hestness et al., 2017, Brown et al., 2020, Hoffmann et al., 2022].
These neural scaling laws demonstrate that the pretraining loss can be highly predictable as a function of these fundamental inputs, which has a number of practical applications: Scaling laws fit to smaller training runs can be used to predict the pretraining loss of a much larger training run, and can be used to determine effective hyperparameters [McCandlish et al., 2018, DeepSeek-AI et al., 2024], or the optimal allocation of dataset and model size for a given compute budget [Hoffmann et al., 2022, Muennighoff et al., 2024, Dey et al., 2023, Sardana and Frankle, 2023, Besiroglu et al., 2024]. In some cases, such laws can be used to predict performance of a larger model in a particular domain, such as coding [Achiam et al., 2023]. The existence of scaling laws turns deep learning into a predictable science at the macro level by providing a simple recipe for improving model quality and de-risking returns on increasing investment into scale [Ganguli et al., 2022, Bowman, 2023].

Emergent Abilities

Language models have been observed to exhibit apparent emergent abilities—behaviors on downstream task performance that cannot be predicted from smaller scales [Wei et al., 2022, Srivastava et al., 2022].
Emergence appears not to be simply a product of training compute or model size, but is also dependent on other factors such as dataset composition [Muckatira et al., 2024, Wei et al., 2022].
Schaeffer et al., 2023 find that some emergent phenomena can be a “mirage” arising due to choices made by researchers such as the use of discontinuous metrics and insufficient resolution. However, Du et al., 2024 note that for many tasks, emergence remains despite the use of continuous metrics. Additionally, discontinuous metrics have been argued to often be the most reflective of real-world usefulness, so emergence in these hard metrics is important. Hu et al., 2024 found that for generative evaluations, infinite resolution can be achieved but requires significant compute and that generated answer be verifiable.

Predicting Downstream Task Performance

Although predicting macroscopic pretraining loss is useful, a far more useful goal is to predict the scaling of model performance on particular downstream tasks or domains. If this was possible, then model developers could tune their datasets and training procedures in a more fine-grained way before launching computationally intensive training runs. Model performance on a particular downstream task is typically correlated with compute, albeit with a few exceptions [McKenzie et al., 2022, Huang et al., 2024]. However, despite attempts to fit scaling laws to values other than loss, including benchmark scores [Gadre et al., 2024, Isik et al., 2024], model memorization [Biderman et al., 2023], or reward [Gao et al., 2022], these downstream performance metrics are usually more noisy or require more compute to fit accurately. Owen, 2024 and Gadre et al., 2024 both find that while aggregate benchmark performance with more compute can be predicted, the scaling behaviour of individual tasks can be noisy. Additionally, Owen, 2024, Du et al., 2024 and Gadre et al., 2024 claim that predicting scaling behavior on a task without access to models exhibiting better-than-random performance (i.e., “before emergence occurs”) cannot be done reliably. Concurrently to our work, Ruan et al., 2024 propose Observational Scaling Laws by mapping model capabilities from compute to a shared low-dimensional space of capabilities across model families before predicting performance on novel tasks.
Our goal in this work is to investigate the comparative unpredictability of individual downstream performance scores, and advise how to create more scaling-predictable evaluations that are closely coupled with real-world use-cases.

§ DEFINITION OF SURVIVAL FUNCTION

The survival function $S_X(x)$ – also known as the reliability function, the tail distribution, or the complementary cumulative distribution function – gives the probability that a random variable $X$ exceeds a certain value $x$ [Kleinbaum and Klein, 2012, contributors, 2023]:

\begin{equation}
S_X(x) \defeq Pr[X > x] = \int_{x}^{\infty} f_X(x') \, dx' = 1 - F_X(x)
\label{eqn:survival_function}
\end{equation}

where $F_X(x) = Pr[X \leq x]$ is the cumulative distribution function (CDF) and $f_X(x)$ is the probability density function (pdf) or probability mass function (pmf) of the random variable $X$.
The CDF $F_X(x)$ gives the probability that the random variable $X$ is at most $x$, while the survival function $S_X(x)$ gives the probability that $X$ exceeds $x$.

When the true distribution of $X$ is unknown, we can use the empirical CDF (ECDF) $\hat{F}_X(x)$ and the empirical survival function (ESF) $\hat{S}_X(x)$:

\begin{equation}
\hat{S}_X(x) \defeq \frac{1}{n} \sum_{i=1}^{n} 1\{x_i > x\} = 1 - \hat{F}_X(x)
\label{eqn:empirical_survival_function}
\end{equation}

where $n$ is the number of observations,
$x_i$ is the realized value of the random variable $X$ for observation $i$,
and $1\{x_i > x\}$ is the indicator function.
The empirical survival function $\hat{S}_X(x)$ specifies the fraction of observations for which the sampled random variable $X$ exceeds $x$.

§ COMPUTE RESOURCES FOR EXPERIMENTS

Experiments were done across a wide family of model families and sizes.
The GPUs we used for medium-sized models (7B parameters and above) used a single A100s with 80GB of vRAM.
For smaller models ($\leq$8B) we used A100s with 80GB of vRAM,
Quadro RTX 8000 with 48GB of vRAM, or
RTX A4000 with 16GB of vRAM.
For 70B parameter models, we used at least 2 A100 GPUs with 80GB of vRAM.

§ ADDITIONAL MODEL FAMILY DETAILS

Here we provide further experimental details regarding our selection of model families.

* Pythia [Biderman et al., 2023]: We consider two “families” for Pythia in our experiments. Pythia (Parameter Scaling) refers to the use of fully-trained checkpoints from 9 different model sizes (all model sizes documented in Biderman et al. (2023), as well as a 14M parameter model trained later by the authors). Pythia-12B (Data Scaling) refers to the use of 8 checkpoints across training for the Pythia-12B model, namely having seen 2M, 64M, 2B, 6B, 20B, 60B, 200B, and 300B tokens in training.

* Cerebras-GPT [Dey et al., 2023]: Cerebras (Parameter and Data Scaling) refers to our use of 1 checkpoint per model in the Cerebras-GPT family, each fully trained for differing quantities of data as documented by the model creators, for 7 checkpoints in total.

* OLMo [Groeneveld et al., 2024]: OLMo (7B Data Scaling) refers to the use of 7 checkpoints for OLMo-7B across training, namely, checkpoints having seen 4B, 44B, 133B, 442B, 885B, 1.5T, and 2.4T tokens.

* INCITE [AI, 2023]: INCITE-7B (Data Scaling) considers 6 checkpoints over training for the 7B parameter model, having seen 240B, 280B, 400B, 500B, 700B, and 1T tokens.

* LLM360 [Liu et al., 2023]: LLM360 Amber (Data Scaling) considers 13 checkpoints of the Amber model, having seen 0B, 3.5B, 7B, 10.5B, 17.5B, 31.5B, 49B, 87.5B, 147B, 252B, 430B, 738B, and 1.26T tokens.

§ BROADER IMPACT

This paper contributes to a better understanding of the predictability of large language models (LLMs), which can have both positive and negative societal impacts.
On the positive side, by making LLM benchmarks more predictable, this research can help society anticipate and plan for potential challenges associated with their development and deployment. This increased predictability can facilitate proactive measures to mitigate risks and ensure the responsible use of AI technologies.

However, the increased predictability of LLMs could theoretically be exploited by malicious actors to accelerate the development of AI systems designed for malicious purposes.
We also stress the importance of proactive risk assessment and the implementation of safeguards to prevent the misuse of AI technologies.

§ SCORE-COMPUTE CORRELATION DISTRIBUTIONS' STATISTICS

§.§ Pearson Correlations

Statistics for empirical distributions of correlations between scores and compute for all benchmarks and model families. These correlation values were computed with Pearson correlation and are consistent with the main text's results computed with Spearman correlation (Fig. <ref>): The sequence of transformations from $\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice}) \rightarrow p_{\theta}^{\text{Vocab}}(\text{Correct Choice}) \rightarrow p_{\theta}^{\text{Choices}}(\text{Correct Choice}) \rightarrow \text{Accuracy}$ degrades predictability.

§.§ Spearman Correlations

Statistics for empirical distributions of correlations between scores and compute for all benchmarks and model families. These correlation values were computed with Spearman correlation. The sequence of transformations from $\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice}) \rightarrow p_{\theta}^{\text{Vocab}}(\text{Correct Choice}) \rightarrow p_{\theta}^{\text{Choices}}(\text{Correct Choice}) \rightarrow \text{Accuracy}$ degrades predictability.

§.§ Kendall Correlations

Statistics for empirical distributions of correlations between scores and compute for all benchmarks and model families. These correlation values were computed with Kendall correlation and are consistent with the main text's results computed with Spearman correlation (Fig. <ref>): The sequence of transformations from $\log p_{\theta}^{\text{Vocab}}(\text{Correct Choice}) \rightarrow p_{\theta}^{\text{Vocab}}(\text{Correct Choice}) \rightarrow p_{\theta}^{\text{Choices}}(\text{Correct Choice}) \rightarrow \text{Accuracy}$ degrades predictability.

§ PER-BENCHMARK SCORE-COMPUTE CORRELATION DISTRIBUTIONS

§.§ NLP Benchmark: ARC Challenge [Clark et al., 2018]

ARC Challenge: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: ARC Easy [Clark et al., 2018]

ARC Easy: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: HellaSwag [Zellers et al., 2019]

HellaSwag: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MathQA [Amini et al., 2019]

HellaSwag: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MC TACO [Zhou et al., 2019]

MC TACO: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Abstract Algebra [Hendrycks et al., 2020]

MMLU Abstract Algebra: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Anatomy [Hendrycks et al., 2020]

MMLU Anatomy: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Astronomy [Hendrycks et al., 2020]

MMLU Astronomy: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Business Ethics [Hendrycks et al., 2020]

MMLU Business Ethics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Clinical Knowledge [Hendrycks et al., 2020]

MMLU Clinical Knowledge: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU College Biology [Hendrycks et al., 2020]

MMLU College Biology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU College Chemistry [Hendrycks et al., 2020]

MMLU College Chemistry: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU College Computer Science [Hendrycks et al., 2020]

MMLU College Computer Science: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU College Mathematics [Hendrycks et al., 2020]

MMLU College Mathematics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU College Medicine [Hendrycks et al., 2020]

MMLU College Medicine: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU College Physics [Hendrycks et al., 2020]

MMLU College Physics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Computer Security [Hendrycks et al., 2020]

MMLU Computer Security: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Conceptual Physics [Hendrycks et al., 2020]

MMLU Conceptual Physics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Econometrics [Hendrycks et al., 2020]

MMLU Econometrics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Electrical Engineering [Hendrycks et al., 2020]

MMLU Electrical Engineering: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Elementary Mathematics [Hendrycks et al., 2020]

MMLU Elementary Mathematics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Formal Logic [Hendrycks et al., 2020]

MMLU Formal Logic: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Global Facts [Hendrycks et al., 2020]

MMLU Global Facts: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Biology [Hendrycks et al., 2020]

MMLU High School Biology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Chemistry [Hendrycks et al., 2020]

MMLU High School Chemistry: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Computer Science [Hendrycks et al., 2020]

MMLU High School Computer Science: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Chemistry [Hendrycks et al., 2020]

MMLU High School Chemistry: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School European History [Hendrycks et al., 2020]

MMLU High School European History: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Geography [Hendrycks et al., 2020]

MMLU High School Geography: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Government & Politics [Hendrycks et al., 2020]

MMLU High School Government & Politics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Macroeconomics [Hendrycks et al., 2020]

MMLU High School Macroeconomics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Mathematics [Hendrycks et al., 2020]

MMLU High School Mathematics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Microeconomics [Hendrycks et al., 2020]

MMLU High School Microeconomics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Physics [Hendrycks et al., 2020]

MMLU High School Physics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Psychology [Hendrycks et al., 2020]

MMLU High School Psychology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School Statistics [Hendrycks et al., 2020]

MMLU High School Statistics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School US History [Hendrycks et al., 2020]

MMLU High School US History: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU High School World History [Hendrycks et al., 2020]

MMLU High School World History: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Human Aging [Hendrycks et al., 2020]

MMLU Human Aging: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Human Sexuality [Hendrycks et al., 2020]

MMLU Human Sexuality: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU International Law [Hendrycks et al., 2020]

MMLU International Law: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Jurisprudence [Hendrycks et al., 2020]

MMLU Jurisprudence: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Logical Fallacies [Hendrycks et al., 2020]

MMLU Logical Fallacies: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Machine Learning [Hendrycks et al., 2020]

MMLU Machine Learning: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Management [Hendrycks et al., 2020]

MMLU Management: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Marketing [Hendrycks et al., 2020]

MMLU Marketing: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Medical Genetics [Hendrycks et al., 2020]

MMLU Medical Genetics: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Miscellaneous [Hendrycks et al., 2020]

MMLU Miscellaneous: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Moral Disputes [Hendrycks et al., 2020]

MMLU Moral Disputes: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Moral Scenarios [Hendrycks et al., 2020]

MMLU Moral Scenarios: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Nutrition [Hendrycks et al., 2020]

MMLU Nutrition: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Philosophy [Hendrycks et al., 2020]

MMLU Philosophy: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Prehistory [Hendrycks et al., 2020]

MMLU Prehistory: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Professional Accounting [Hendrycks et al., 2020]

MMLU Professional Accounting: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Professional Law [Hendrycks et al., 2020]

MMLU Professional Law: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Professional Medicine [Hendrycks et al., 2020]

MMLU Professional Medicine: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Professional Psychology [Hendrycks et al., 2020]

MMLU Professional Psychology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Public Relations [Hendrycks et al., 2020]

MMLU Public Relations: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Security Studies [Hendrycks et al., 2020]

MMLU Security Studies: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Sociology [Hendrycks et al., 2020]

MMLU Sociology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU US Foreign Policy [Hendrycks et al., 2020]

MMLU US Foreign Policy: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU Virology [Hendrycks et al., 2020]

MMLU Virology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: MMLU World Religions [Hendrycks et al., 2020]

MMLU World Religions: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: OpenBookQA [Mihaylov et al., 2018]

OpenBookQA: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: PIQA [Bisk et al., 2020]

PIQA: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: RACE [Lai et al., 2017]

RACE: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: SciQ [Welbl et al., 2017]

SciQ: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: Social IQA [Sap et al., 2019]

Social IQA: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: Winogrande [Keisuke et al., 2019]

Social IQA: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.

§.§ NLP Benchmark: XWinograd English [Muennighoff et al., 2023]

XWinograd English: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.