# AutoBencher: Creating Salient, Novel, Difficult Datasets for Language Models

Xiang Lisa Li1, Evan Zheran Liu2, Percy Liang1, Tatsunori Hashimoto1
Stanford University1, Imbue2
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Evaluation is critical for assessing capabilities, tracking scientific
progress, and informing model selection. In this paper, we present three
desiderata for a good benchmark for language models: (i) salience (e.g.,
knowledge about World War II is more salient than a random day in history),
(ii) novelty (i.e., the benchmark reveals new trends in model rankings not
shown by previous benchmarks), and (iii) difficulty (i.e., the benchmark
should be difficult for existing models, leaving headroom for future
improvement). We operationalize these three desiderata and cast benchmark
creation as a search problem, that of finding benchmarks that that satisfy all
three desiderata. To tackle this search problem, we present AutoBencher, which
uses a language model to automatically search for datasets that meet the three
desiderata. AutoBencher uses privileged information (e.g. relevant documents)
to construct reliable datasets, and adaptivity with reranking to optimize for
the search objective. We use AutoBencher to create datasets for math,
multilingual, and knowledge-intensive question answering. The scalability of
AutoBencher allows it to test fine-grained categories and tail knowledge,
creating datasets that are on average 27% more novel and 22% more difficult
than existing benchmarks. A closer investigation of our constructed datasets
shows that we can identify specific gaps in LM knowledge in language models
that are not captured by existing benchmarks, such as Gemini Pro performing
much worse on question answering about the Permian Extinction and Fordism,
while OpenAGI-7B performing surprisingly well on QA about COVID-19.111Code is
available at https://github.com/XiangLi1999/AutoBencher.

##  1 Introduction

Evaluation is crucial for informing model selection and guiding model
development, and it is especially challenging to evaluate language models.
Prior works have constructed high-quality benchmarks, guided by two
desiderata: (1) _salience_ : the benchmark should test practically important
capabilities, for example, the capability to solve math problems is examined
by GSM8K [6] and capability to avoid answering misleading questions is
examined by TruthfulQA [17]. (2) _difficulty_ : the benchmark should yield low
accuracy with existing models. For example, GPQA [28] carefully constructs
questions that are challenging even for domain experts. In this work, we
identify an additional goal of (3) _novelty_. More recently, observational
studies of existing benchmarks show that model performance on existing
benchmarks are highly correlated [18, 27, 20, 5], and thus there is an
increasing need for benchmark novelty. Novelty measures the degree to which a
benchmark reveals previously unknown performance trends in model rankings,
such as unexpected model strengths or weaknesses on specific domains.

In order to measure salience, difficulty, and novelty of a benchmark, we
operationalize these three properties: (1) We define salience as a binary
variable that captures whether a benchmark on a specific topic (e.g.,
COVID-19) is important. (2) The difficulty of a benchmark is determined by the
lowest error rate achievable by existing models. (3) We calculate the novelty
of a benchmark as the predictability of model performance on the benchmark
based on performance on other benchmarks. Formalizing the three desiderata
allows us to take an optimization approach to constructing benchmarks. Instead
of viewing benchmarking as the task of constructing a dataset that represents
some task, we cast it as a way to find particular test set that fulfill our
three desiderata. This turns benchmarking into an optimization problem, where
the objective is made explicit.

To approximately solve this search problem and automatically create new
salient, difficult, and novel benchmarks, we propose AutoBencher. As shown in
Figure 1, given a broad domain (e.g., history), AutoBencher proposes candidate
evaluation topics (e.g., World War II) and then constructs a small dataset for
each topic using privileged information (e.g., relevant Wikipedia articles).
These datasets are scored by our metrics and used as part of a local search
algorithm to find a benchmark with improved difficulty. We leverage the
scalability of AutoBencher to identify and select topics that jointly maximize
a weighted sum of difficulty and novelty, subject to a salience constraint
specified by the user.

We use AutoBencher to create datasets in 5 domains: math, history, science,
economics, and multilingualism, altogether producing around 3700 examples.
These benchmarks reveal novel trends not shown by prior benchmarks (§ 6.2):
For example, we find that while Gemini Pro is one of the strongest models on
existing history benchmarks, it performs quite poorly on AutoBencher-
discovered topics of Permian Extinction and Fordism, performing even worse
than some 7B models, such as Mistral-7B. Meanwhile, we also find that 7B
models (e.g., Mistral-7B and OpenAGI-7B), which typically exhibit lower
accuracy due to their smaller size, perform quite well on the topic of
COVID-19, even achieving accuracy on par with GPT-4 Turbo. Additionally, our
datasets increase difficulty by 22% compared with human-constructed
benchmarks, such as MMLU and XOR QA (§ 6.1).222We will release the dataset and
code.

Figure 1:  (Left) Models ranked by accuracy on existing datasets and
AutoBencher datasets. Existing datasets show roughly the same performance
trends, while AutoBencher discovers tests that induce novel rankings. (Right)
Given a domain (e.g., history), AutoBencher creates datasets that are salient,
difficult, and novel. It achieves this by searching through evaluation topics,
scoring each based on these desiderata, and selecting the best one. These
datasets are automatically created by providing privileged information to a
language model.

##  2 Related Work

#### Benchmarking Language Models

A large number of benchmarks have been constructed to measure different skills
of language models. To give a few examples of common LLM benchmarks, MMLU
measures the understanding of academic subjects [11], GSM8K measures
mathematical reasoning [6], and Winogrande measures common sense reasoning
[30]. Researchers have also grouped the benchmarks to create leaderboards that
rank LMs’ overall capabilities, such as HELM [16] and Open LLM Leaderboard
[3]. Additionally, researchers also carefully subsample existing benchmarks to
obtain a smaller and more efficient benchmark with similar model accuracy
[20]. Prior works of LLM-as-Judge incorporate language models as part of the
evaluation pipeline to automatically judge the quality of model-generated
responses [8, 39, 9]. Our work further automates the evaluation pipeline by
using LM to propose evaluation topics and generate questions.

The most similar work to ours is Li et al. [15], which uses LMs to brainstorm
questions for instruction following tasks. However, their goal is very
different from ours: they aim to find a minimal set of questions to reproduce
the same LM ranking achieved by evaluating a large number of examples; our
goal is to search for new benchmark topics that break the performance trends
of existing benchmarks.

#### Adaptive Datasets

In AutoBencher, one important desideratum we optimize for is novelty. Prior
works have also constructed datasets adaptively to search for difficult
questions [22, 12, 29, 38, 7]. Most of these works have generated test cases
with human annotators, whereas we use language models to automate the search,
saving extensive human effort. Similar to AutoBencher, Perez et al. [26]
searches for prompts that induce harmful behaviors in language models using
another language model. However, they focus on instance-level failures in
safety, whereas we focus on finding systematic failures on specific topics.

##  3 A Metric-Driven Framework of Benchmark Creation

We now discuss our three desiderata of _salience_ , _difficulty_ , and
_novelty_ and formally define them as quantitative metrics that can be
directly optimized.

#### Preliminaries.

Let $c\in\mathcal{C}$ be a evaluation topic. We define a dataset
$\mathcal{D}_{c}=\\{(x_{i},y_{i})\\}_{i}$ as a set of question-answer pairs
$(x_{i},y_{i})$ that evaluate mastery of the topic $c$. As an example, topics
$\mathcal{C}$ could include “the Industrial Revolution”, “World War II”,
“Ancient Civilization”, etc. In this work, we will generate the datasets
$\mathcal{D}_{c}$ from a fixed generator $p(\mathcal{D}_{c}\mid c)$ and focus
on optimizing the set of topics $c$ to change the novelty and difficulty of a
dataset.

Let $\mathcal{M}=\\{\texttt{LM}_{m}\\}_{m=1}^{M}$ denote the set of $M$
existing models to evaluate. We denote the accuracy of model
$\texttt{LM}_{m}\in\mathcal{M}$ on benchmark $\mathcal{D}_{c}$ as
$\texttt{acc}(\texttt{LM}_{m},\mathcal{D}_{c})$. We define the _accuracy
vector_
$V_{c}=[\texttt{acc}(\texttt{LM}_{1},\mathcal{D}_{c}),\cdots,\texttt{acc}(\texttt{LM}_{M},\mathcal{D}_{c})]$
of a benchmark $\mathcal{D}_{c}$ as the accuracy of all models on the
benchmark $\mathcal{D}_{c}$.

#### Salience.

Recall that salience measures the practical importance of a topic $c$. First,
we assume a set of salient topics $\mathcal{S}$ specified by the user, and we
define salience as a binary variable, such that $\textsc{salience}(c)=1$ if
$c\in\mathcal{S}$ and $\textsc{salience}(c)=0$ otherwise. For example, we may
define salient topics $\mathcal{S}$ to be the set of topics with the number of
corresponding Wikipedia page views exceeding a threshold.

#### Novelty.

Recall that novelty measures how much new information a dataset reveals about
existing models over existing benchmarks. Therefore, we formalize
$\textsc{Novelty}(\mathcal{D}_{c};\mathbf{D}_{\text{prev}};\mathcal{M})$ as a
function of the benchmark in question $\mathcal{D}_{c}$, prior benchmarks
$\mathbf{D}_{\text{prev}}:=\\{\mathcal{D}_{1}\ldots\mathcal{D}_{N}\\}$, and
the existing set of models $\mathcal{M}$. Intuitively, the results on a new
benchmark reveal new information if model performance on the new benchmark
vastly differ from the trends on prior benchmarks (e.g., if a normally low-
performing model outperforms all other models on the new benchmark).

To quantify this, we first find how much information is explainable by the
accuracy on existing datasets $\mathbf{D}_{\text{prev}}$, by performing a
regression from $V_{\text{prev}}:=\left[V_{1}\cdots
V_{N}\right]\in\mathbb{R}^{N\times M}$ to predict $V_{c}\in\mathbb{R}^{1\times
M}$,

$\hat{V}_{c}:=w^{*\top}V_{\text{prev}}+b\text{~{}~{}and~{}~{}}w^{*}=\operatorname*{arg\,min}_{w}\bigg{|}\bigg{|}w^{\top}V_{\text{prev}}+b-V_{c}\bigg{|}\bigg{|}_{2}^{2}.$

We then compute the rank correlation between the predicted accuracy
$\hat{V}_{c}$ and the ground truth accuracy as
$\textsc{RankCorr}(V_{c},\hat{V}_{c})$ as a predictability measure for the new
dataset. Formally,

$\displaystyle\textsc{Novelty}(\mathcal{D}_{c},\mathbf{D}_{\text{prev}},\mathcal{M})=1-\textsc{RankCorr}(\hat{V}_{c},V_{c}).$

If the new accuracy vector $V_{c}$ is spanned by the existing accuracy
vectors, $\textsc{RankCorr}(V_{c},\hat{V}_{c})$ will be close to 1, resulting
in low novelty. On the other hand, if the new accuracy vector $V_{c}$
discovers some new patterns in model performance such as an orthogonal
direction, $\textsc{RankCorr}(V_{c},\hat{V}_{c})$ will be low, resulting in
high novelty.

Take Figure 1 (left) as an example, let ARC be the existing datasets
$\mathbf{D}_{\text{prev}}$. To compute the novelty of the MMLU dataset (column
2), we perform a regression from the accuracy vector of ARC to predict the
accuracy vector of MMLU. Since they have the same ranking, model rankings on
MMLU is perfectly predictable from ARC, yielding a RankCorr of $1$ and a
Novelty of $0$. To compute the novelty of the QA: COVID-19 dataset (column 3),
we also predict its accuracy vector from ARC. However, since the accuracy
vector of COVID-19 datasets shows abrupt changes in model performance, it is
not spanned by the accuracy vector of the ARC dataset. Therefore, its
performance is not predictable by the existing dataset, yielding a low
RankCorr and a high Novelty.

#### Difficulty.

A benchmark’s difficulty is determined directly by a model’s error rate.
Ideally, a benchmark should leave sufficient headroom above the best current
error rate to enable tracking future progress. We formalize the difficulty of
a benchmark as the remaining gap between the best model’s accuracy and optimal
accuracy:
$\textsc{Difficulty}(\mathcal{D}_{c},\mathcal{M})=1-\max_{m}\texttt{acc}(\texttt{LM}_{i},\mathcal{D}_{c})$.

#### Separability.

Besides the three main desiderata, we also include the separability of model
accuracy as an additional regularizer. Separability measures the amount of
separation between different model accuracies; In this paper, we use
separability to complement novelty, ensuring that all the model performance
trends revealed by a benchmark is robust. We formalize the separation on
benchmark $\mathcal{D}_{c}$ between the accuracy of models $\mathcal{M}$ as
the mean absolute deviation
$\textsc{Sep}(\mathcal{D}_{c},\mathcal{M})=\text{mean}(|V_{c}-\text{mean}(V_{c})|)$.

#### Existing Benchmarks Case Study.

We now analyze several existing benchmarks under these desiderata: MMLU [11]
contains salient topics on academic subjects; it is sufficiently difficult
with the best model accuracy of 86% and has good separability to distinguish
existing models. However, the benchmark lacks novelty, as it’s highly
correlated with prior benchmarks like ARC, with a rank correlation of 94%.

Datasets in the Inverse Scaling Prize [21] is novel but many lack salience.
For example, pattern matching suppression instructs the LM to continue a
sequence by violating the current pattern, and Memo trap instructs the LM to
continue a famous quote with a different ending. These tasks reveal novel
performance patterns (i.e., larger models perform worse), but are much less
salient than datasets like the knowledge-intensive MMLU.

##  4 AutoBencher

Our goal is to find a sufficiently salient evaluation topic $c$ that maximizes
a linear combination of novelty and difficulty. We directly maximize novelty
and difficulty subject to a constraint on salience, because salience is
operationalized as a binary variable. Specifically, we aim to solve the
following constrained optimization problem:

maximize
$\displaystyle\mathcal{J}(c)=\textsc{Novelty}(\mathcal{D}_{c};\mathbf{D}_{\text{prev}},\mathcal{M})+\beta_{1}~{}\textsc{difficulty}(\mathcal{D}_{c};\mathcal{M})+\beta_{2}~{}~{}\textsc{Sep}(\mathcal{D}_{c};\mathcal{M})$
(1) subject to $\displaystyle c\in\mathcal{S}.$

#### Solving the optimization problem.

Identifying solutions to this search problem is difficult due to the
complexity of performing a discrete search over three potentially conflicting
criteria (salience, difficulty, and novelty) and we cannot intuitively judge
which topics might reveal novel performance trends a priori. Therefore, we
propose AutoBencher, which automates the search with a language model, denoted
as $\texttt{LM}_{\text{gen}}$, and outputs the best discovered topic and its
dataset.

One natural, naive design is to perform a random search, where we prompt
$\texttt{LM}_{\text{gen}}$ to generate a diverse set of topics $c$, prompt
$\texttt{LM}_{\text{gen}}$ to generate a dataset of (question, answer) pairs
for each topic, and then select the best topic according to the objective
function $\mathcal{J}(c)$.

However, this design suffers from two issues: (1) _Example correctness_ :
Since we use $\texttt{LM}_{\text{gen}}$ to construct the dataset, the
generated answers might be incorrect due to model hallucination. (2) _Example
difficulty_ : The difficulty of the generated questions is upper-bounded by
the capabilities of $\texttt{LM}_{\text{gen}}$ and hence cannot be used to
evaluate stronger models. (3) _Topic difficulty_ : empirically, we observe
that $\texttt{LM}_{\text{gen}}$ tends to propose well-known, and
insufficiently difficult topics.

We now propose some techniques to address these issues: We first augment
$\texttt{LM}_{\text{gen}}$ with privileged information to improve the
correctness and difficulty of the generated datasets (§ 4.1). Next, we propose
adaptive search, which uses the trajectory of past generated benchmarks to
improve topic difficulty (§ 4.2). Finally, we present the full pseudocode and
present how each metric gets optimized in this pipeline (§ 4.3).

###  4.1 Generating Datasets with Privileged Information

Figure 2:  How the model $\texttt{LM}_{\text{gen}}$ uses privileged
information to create (question, answer) examples.

To improve the difficulty of the generated questions and the correctness of
the generated answers, we augment $\texttt{LM}_{\text{gen}}$ with privileged
information that we do not provide to the LMs that we are evaluating (which we
will refer to as the test-taker LMs to avoid confusion). This privileged
information ensures answer correctness by grounding the generated answers in a
reliable source and creates an information imbalance between the example
creation model $\texttt{LM}_{\text{gen}}$ and test-taker LMs, which simplifies
the task for $\texttt{LM}_{\text{gen}}$ and enables it to create questions
that are more difficult than possible with its base capabilities. We next
detail the privileged information we provide in three domains: knowledge
intensive, multilingual, and mathematics. Figure 2 illustrates how this
information is used in each domain.

#### Knowledge-intensive.

We augment $\texttt{LM}_{\text{gen}}$ with a set of relevant documents (e.g.,
Wikipedia). Specifically, to create a knowledge-intensive question on topic
$c$, we first retrieve the set of most relevant articles by querying $c$ in
the Wikipedia Search API. Then, we prompt $\texttt{LM}_{\text{gen}}$ to
jointly generate (question, answer) pairs conditioned on the retrieved
articles such that the answer can be verified by the document and the question
is answerable _without_ the document. We enforce this by checking the lexical
overlap between the answers and the retrieved articles.

#### Multilingual.

We augment $\texttt{LM}_{\text{gen}}$ with a translation system. Since models
tend to have better reasoning capabilities in English than in other languages,
we generate (question, answer) pairs in other languages by first generating
the example in English via the knowledge-intensive question procedure above.
Then, we translate the question and answer to the target language.

#### Mathematics.

We augment $\texttt{LM}_{\text{gen}}$ with Python math libraries (e.g., sympy,
scipy, numpy). To ensure that the answers are correct, we prompt
$\texttt{LM}_{\text{gen}}$ to generate questions along with Python code to
compute their answers and use the execution result as the answer.

###  4.2 Proposing Topics with Adaptive Search

When we use $\texttt{LM}_{\text{gen}}$ to propose evaluation topics, a key
challenge is that $\texttt{LM}_{\text{gen}}$ does not have information about
what topics might be difficult for LMs. To address this, we propose adaptive
search, an iterative approach that collects accuracy information in each
iteration to inform topic proposals in subsequent iterations. We keep track of
a trajectory $\mathcal{H}$, represented as a sequence of (topic, accuracy)
pairs. As we run more iterations, $\mathcal{H}$ accumulates more (topic,
accuracy) pairs, and forms a better belief about what topics are likely be
difficult (see Figure 1 right).

Specifically, we follow lines 1 to 7 in Algorithm 1. In each iteration,
AutoBencher proposes $K$ topics conditioned on the trajectory $\mathcal{H}$
collected from previous iterations (line 3), where we specifically prompt to
ask for difficult topics (i.e., low accuracy). We filter out non-salient
topics (line 4) and instantiate each remaining topic as a dataset via the
procedure described in § 4.1 (line 5). Then, we compute the accuracy of a
test-taker model $\texttt{LM}_{\text{test}}$ on each dataset as a measure of
difficulty (line 6). Finally, we feed all proposed (topic, accuracy) pairs to
subsequent iterations to increase difficulty (lines 7). We choose to only
evaluate the accuracy of a single strong test-taker model, rather than
evaluating all models $\mathcal{M}$ for computational efficiency.

Our adaptive search procedure does not take Novelty or Sep into account, since
these two quantities also require expensively evaluating all models
$\mathcal{M}$. Instead, we optimize these desiderata in a final re-ranking
step, which considers all of the metrics together.

Require: $\mathcal{M}$, $\texttt{LM}_{\text{gen}}$,
$\texttt{LM}_{\text{test}}$, domain $d$, max iterations $N$, hyperparameter
$\beta_{1}$, $\beta_{2}$ 1:Initialize previously-proposed topics
$\mathcal{H}=\varnothing$ 2:for maximimum number of iteration $N$ times do 3:
Propose topics conditioned on prev. topics
$c_{1},\ldots,c_{K}\sim\texttt{LM}_{\text{gen}}(\cdot\mid\mathcal{H})$ 4:
Filter out to keep only the salient topics with $c\in\mathcal{S}$ 5: Generate
small dataset $\mathcal{D}_{c_{k}}$ for each remaining topic by prompting
$\texttt{LM}_{\text{gen}}$ 6: Compute the test-taker model accuracy on each
dataset $\texttt{acc}(\texttt{LM}_{\text{test}},\mathcal{D}_{c_{k}})$ 7:
Update previously proposed topics
$\mathcal{H}=\mathcal{H}\cup\\{(c_{k},\texttt{acc}(\texttt{LM}_{\text{test}},\mathcal{D}_{c_{k}}))\\}$
8:Extract set of all proposed topics
$\mathcal{P}=\\{c:(c,\texttt{acc}(\texttt{LM}_{\text{test}},\mathcal{D}_{c}))\in\mathcal{H}\\}$
9:Compute the search objective $\mathcal{J}(c)$ on all proposed topics
$c\in\mathcal{P}$ 10:Select the topic with the highest objective value
$c^{*}=\arg\max_{c\in\mathcal{P}}\mathcal{J}(c)$ 11:Generate large dataset
$\mathcal{D}_{c^{*}}$ by prompting $\texttt{LM}_{\text{gen}}$ on topic $c^{*}$
12:return Chosen topic $c^{*}$ and corresponding dataset $\mathcal{D}_{c^{*}}$

Algorithm 1 AutoBencher

###  4.3 Metric Attribution

Overall, the entire algorithm behind AutoBencher is summarized in Algorithm 1.
In lines 1 to 7, we perform the adaptive search described above. Afterward, we
re-rank the proposed topics via the full search objective $\mathcal{J}(c)$: We
compute the objective for each proposed topic (line 9) and construct a dataset
on the topic with the highest objective value (lines 10–12).

This algorithm optimizes for all the desiderata of novelty, salience, and
difficulty. salience is met via the filtering step in line 4, which filters
out all non-salient topics. Difficulty is approximately optimized via the
adaptive search procedure (lines 1–7) that steers toward producing more
difficult topics. Sep and Novelty are optimized in the final re-ranking step
(lines 9–10), where we compute and select based on the full objective function
$\mathcal{J}$.

##  5 Experimental Setup

###  5.1 Baselines and Metrics

We evaluate AutoBencher on five domains: mathematics, multilingual, and
knowledge-intensive QA on history, economy, and science.

#### Baselines.

We compare benchmarks generated by AutoBencher with human constructed
benchmarks, which we group collectively under HumanBench. For knowledge-
intensive domains, we compare with the subject datasets in MMLU [11]. These
datasets consist of multiple-choice questions on academic subjects ranging
from high school to professional level. We compare with 4 history subjects
(e.g., high school world history), 4 economy subjects (e.g., econometrics),
and 7 science subjects (e.g., college physics). See the complete list in
Appendix C.

For mathematics, we compare with $7$ datasets from the Mathematics Dataset
[31]333https://github.com/google-deepmind/mathematics_dataset, which covers
basic math capabilities: algebra, arithmetic, calculus, probability,
comparison, measurement, numbers. For multilingualism, we compare with XOR QA
[2], a multilingual question-answering dataset covering 7 diverse languages.
We compare with the test set, split by language into 7 datasets.

#### Models.

For this paper, the set of models we evaluate is
$\mathcal{M}=\\{$gpt-4-turbo-2024-04-09, gpt-3.5-turbo-0613,
claude-3-sonnet-20240229, claude-3-opus-20240229, claude-2.0,
Mixtral-8x7B-Instruct-v0.1, Mistral-7B-Instruct-v0.1, gemini-pro,
OpenAGI-7B-v0.1, vicuna-7b-v1.5, Llama-2-7b-chat-hf, Xwin-Math-7B-V1.0,
WizardMath-7B-V1.0, gpt-neo-2.7B, alpaca-7b, zephyr-7b-beta,
openchat-3.5-0106$\\}$ These models are designed to cover three categories:
the strongest closed models, strong open models and strong 7B models.

#### Metrics.

We evaluate on the four metrics: Novelty (Novel), Sep (Sep), Difficulty
(Diff), and salience as defined in § 3. For calculating Novelty, we set
$\mathbf{D}_{\text{prev}}$ as the aggregate of all datasets in HumanBench.

###  5.2 AutoBencher Hyperparameters and Costs

Hyperparameters. AutoBencher uses gpt-4-0125-preview [24] as
$\texttt{LM}_{\text{gen}}$ to propose topics and generate the datasets. For
the test-taker model $\texttt{LM}_{\text{test}}$, we select gpt-3.5-turbo-0613
[23], Mixtral-8x7B and Mistral-7B, to cover different levels of model
accuracies. In AutoBencher, we perform $N=8$ iterations of adaptive search,
each proposing $k=5$ topics. We generate $|\mathcal{D}_{c}|=50$ examples per
topic. In total, we explore 2000 questions across 40 topics in one round of
AutoBencher.

Costs. Each run of the AutoBencher agent uses around 750K tokens, which costs
around $15. Among them, 43K tokens are used for proposing topics, 576K tokens
are used for constructing datasets, and 147K for evaluation.

##  6 Main Results

We find that AutoBencher successfully constructs datasets that satisfy the
three desiderata. We first report the novelty and difficulty scores in § 6.1.
Then we provide the list of discovered topics and qualitative examples of
questions generated by AutoBencher in § 6.2. Finally, we conduct a human study
that verifies the salience, novelty, and difficulty of the discovered topics
in § 6.3.

###  6.1 Novelty and Difficulty

Recall that our definition of novelty measures the rank correlation between
models’ accuracies on one dataset with their accuracies on all other datasets.
A lower correlation indicates more novel performance trends. We find that
datasets constructed by AutoBencher are significantly more novel than existing
human-constructed datasets, reducing the rank correlation by 27%. Moreover,
AutoBencher datasets also exhibit 22% greater difficulty (Diff) and higher
separation (Sep) between models, increasing the accuracy gaps between existing
models by 1%, on average. These improvements hold across all domains, as shown
in Table 1.

We evaluate the impact of adaptive search on novelty and difficulty by
ablating it in AutoBench-AS. Rather than conditioning on the (topic, accuracy)
pairs of previously proposed topics, we simply prompt
$\texttt{LM}_{\text{gen}}$ to propose salient, difficult, and diverse topics.
Table 1 (top) shows that AutoBench-AS obtains lower novelty and difficulty
scores than full AutoBencher, but still outperforms the human-constructed
datasets in all metrics. This is likely because adaptive search only affects
the quality of the proposal distribution, and AutoBench-AS still accounts for
novelty and difficulty via final re-ranking on the objective function.

Table 1:  Comparison between AutoBencher and prior human-constructed datasets
(HumanBench) on novelty, separation (Sep), and difficulty (Diff). Higher
numbers are better for all metrics. AutoBencher constructs datasets that are
significantly more novel and difficult over human-constructed datasets.
Ablating the adaptive search component impacts all metrics, particularly
difficulty.

| History | Economy | Science
---|---|---|---
| Novel | Sep | Diff | Novel | Sep | Diff | Novel | Sep | Diff
HumanBench | 0.05 | 0.031 | 0.103 | 0.13 | 0.011 | 0.056 | 0.22 | 0.020 | 0.4
AutoBench-AS | $0.24\pm 0.07$ | 0.037 | 0.257 | $0.34\pm 0.06$ | 0.021 | 0.206 | $0.35\pm 0.12$ | 0.024 | 0.144
AutoBench | $\mathbf{0.39\pm 0.10}$ | 0.042 | 0.440 | $\mathbf{0.43\pm 0.10}$ | 0.026 | 0.321 | $\mathbf{0.39\pm 0.06}$ | 0.031 | 0.475

| Multilingual | Math
---|---|---
| Novel | Sep | Diff | Novel | Sep | Diff
HumanBench | 0.24 | 0.043 | 0.606 | 0.24 | 0.178 | 0.386
AutoBench | $\mathbf{0.57\pm 0.07}$ | 0.047 | 0.113 | $\mathbf{0.84\pm 0.1}$ | 0.122 | 0.514

###  6.2 Qualitative Examples

To qualitatively analyze the results of AutoBencher, we list the topics that
are selected by AutoBencher along with the number of Wikipedia page views of
that topic and $3$ random (question, answer) pairs from each constructed
dataset in Table 3. We find that the generated questions and answers are
generally of high quality: they are primarily factually correct and self-
contained, though some questions are too open-ended or have dubious answers,
e.g., “What is the significance of blockchain technology in environmental
science?” Below, we also discuss the novel performance trends AutoBencher
reveals in four topics. See Appendix D for the full ranking details.

Fordism is an important economic concept on mass production. Gemini Pro drops
from an average rank of $6$ on existing economics datasets to $16$. In
contrast, OpenChat-3.5 performs quite well. It jumps from an average economics
rank of 8 to 2 on this dataset, nearly matching Claude 3 Sonnet.

Secret Society is a historical topic, and exhibits better-than-expected
GPT-3.5 Turbo accuracy, rising from rank $7$ to $3$. Gemini Pro performs worse
than expected, dropping from rank $6$ to $16$.

Economic impact of COVID-19 reveals much better-than-expected accuracy by
Mistral-7B and OpenAGI-7B, which nearly match GPT-4 Turbo, perhaps due to the
wealth of COVID-19 data.

Permian extinction is the earth’s most severe known extinction event. Gemini
Pro and Claude-2.0 both perform much worse than expected, dropping ranks by
$7$ and $4$ respectively.

General trends. Overall, AutoBencher reveals two general trends: First, small
7B models can perform quite well on select topics, matching the accuracies of
much larger models, and even at times, the best models. Second, Gemini Pro
frequently performs much worse on AutoBencher-selected topics than on existing
datasets.

Table 2: We find that the human-generated datasets on these discovered
evaluation topics are also novel. This confirms that the discovered topics
indeed reveal novel model performance.

| Economy | Science | History
---|---|---|---
| Novel | Sep | Diff | Novel | Sep | Diff | Novel | Sep | Diff
HumanBench | 0.13 | 0.011 | 0.056 | 0.22 | 0.020 | 0.400 | 0.05 | 0.031 | 0.103
AutoBench | $0.43\pm 0.1$ | 0.026 | 0.321 | $0.39\pm 0.06$ | 0.031 | 0.475 | $0.39\pm 0.1$ | 0.042 | 0.440
Human Study | $0.34\pm 0.06$ | 0.042 | 0.130 | $0.39\pm 0.06$ | 0.057 | 0.268 | $0.17\pm 0.04$ | 0.034 | 0.269

Table 3:  Discovered topics (labeled with their Wikipedia page view count) and three (question, answer) pairs randomly drawn from the datasets constructed by AutoBencher. Topic |  Qualitative Examples of (question, answer)
---|---
[History] Secret society (1.2M) |  Question: Who was assassinated in 1963? Answer: John F. Kennedy
Question: Who compiled ’The Cyclopædia of Fraternities’ in 1899? Answer:
Albert Clark Stevens
Question: Which university’s historian offered a three-pronged definition of
secret societies? Answer: University of Idaho
[History] Space Race (1.4M) |  Question: What was the first artificial satellite? Answer: Sputnik 1
Question: Which astronaut was the first American to orbit the Earth? Answer:
John Glenn
Question: What action did President Eisenhower take in response to the Soviet
space lead? Answer: Recommended the establishment of NASA
[Science] Environmental science (0.9M) |  Question: What legislation was established in Mesopotamia to address deforestation? Answer: the first legislation limiting deforestation
Question: How does artificial intelligence contribute to wildlife
conservation? Answer: by predicting the movement of animal populations and
protecting the habitats of wildlife
Question: What is the significance of blockchain technology in environmental
science? Answer: monitors and regulates global fisheries
[Science] Late Permian extinction event (1.6M) |  Question: What is the 13C / 12C ratio of buried sedimentary organic matter below normal? Answer: 2.0 to 2.5% below normal
Question: What was the fate of ammonoids during the Permian-Triassic
extinction event? Answer: suffered a selective extinction pulse 10 million
years before the main event
Question: What phenomenon drastically increased competition and accelerated
biotic recovery in the Anisian? Answer: niche crowding
[Economy] Fordism (0.5M) |  Question: What predominates over manufacturing in the current economic landscape? Answer: Service industries
Question: What was a byproduct of Ford substantially increasing his workers’
wages? Answer: Giving them the means to become customers
Question: What major advantage did Fordism bring to industrial production?
Answer: It cut down on the manpower necessary for the factory to operate
[Economy] Economic impact of COVID-19 (41M) |  Question: What was the impact of the pandemic on educational systems? Answer: many governments temporarily closed educational institutions
Question: What was the increase in percentage of white collar workers working
remotely from before the pandemic to May 2020? Answer: From 6% to 65%
Question: By what percentage did the income generated from work across the
world drop in the first nine months of 2020? Answer: 10.7 percent
[Math] Probability of two events |  Question: What is the probability of selecting an odd number from the set 1, 3, 5, 7, 9 and then selecting a number divisible by 2 from the set 2, 4, 6, 8? Answer: 1.0
Question: In a lottery where you pick a number from 1 to 100, what is the
probability of guessing the number correctly two times in a row? Answer:
0.0001
Question: What is the probability of rolling a 1 on a six-sided die and then
flipping tails on a coin? Answer: 0.08333
[Math] Order of operations |  Question: What is 72 + 18 - (6 * 5)? Answer: 60
Question: What is 88 + 12 - (56 / 7)? Answer: 92.0
Question: What is (80 - 40) / (10 / 2)? Answer: 8.0

###  6.3 Human Study for Robustness

We have shown that AutoBencher can identify salient topics such as the Permian
extinction where capable models fail. However, this does not prove that the
dataset we construct is salient, or that the topic itself is what is causing
the model to fail. For example, the optimization process of AutoBencher may
have discovered specific, adversarial interactions between
$\texttt{LM}_{\text{gen}}$ and the test-taker model, or there may be subtle
flaws in the dataset itself. To rule out these issues, we perform a
verification study where humans generate the dataset given only the topic
category, and show that the same trends appear with human generated datasets.

Specifically, we gave one of the authors the discovered topics and access to
Wikipedia and asked them to generate a QA dataset on the given topic. We
report the novelty and difficulty metrics of the human-generated dataset in
Table 2. We find that the human generated datasets on these topics are also
more novel than the existing datasets in each domain, improving novelty by
$16\%$. Also, the human constructed dataset on the discovered topics attains
better difficulty and separability scores than existing datasets on average,
though the gaps are smaller here. Overall, these results show that our
identified novel failures are robust to dataset construction approaches (e.g.,
by AutoBencher, or by human) and AutoBencher is a promising way to find
salient, difficult, and novel model failures.

##  7 Conclusion and Future Works

In this paper, we present a metrics-driven approach towards constructing new
datasets. Given three desiderata of salience, difficulty, and novelty, we
operationalize them as metrics and cast benchmark construction as a discrete
search problem optimizing for these metrics. To tackle this search, we propose
AutoBencher, which uses a language model to search for datasets that meet
these criteria.

The paper focuses on the capabilities of LLMs such as knowledge-intensive QA,
mathematics, and multilingualism. Future works could apply AutoBencher to
broader domains (e.g., safety, efficiency) to search for novel model behavior
patterns. Additionally, we restrict the proposed topics to fall under a domain
for the scope of this paper, and future works could explore this design space
and extend AutoBencher to propose creative tasks.

## References

  * Anthropic [2024] Anthropic.  Claude 3 haiku: our fastest model yet.  https://www.anthropic.com/news/claude-3-haiku, 2024.
  * Asai et al. [2021] Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi.  XOR QA: Cross-lingual open-retrieval question answering.  In _NAACL-HLT_ , 2021.
  * Beeching et al. [2023] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * Black et al. [2022] Sid Black, Stella Rose Biderman, Eric Hallahan, Quentin G. Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, M. Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, J. Tow, Ben Wang, and Samuel Weinbach.  GPT-NeoX-20B: An open-source autoregressive language model.  _arXiv_ , 2022.
  * Burnell et al. [2023] Ryan Burnell, Han Hao, Andrew R. A. Conway, and Jose Hernandez Orallo.  Revealing the structure of language model capabilities, 2023.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Dinan et al. [2019] Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston.  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4537–4546, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1461.  URL https://aclanthology.org/D19-1461.
  * Dubois et al. [2023] Yann Dubois, Tatsunori Hashimoto, and Percy Liang.  Evaluating self-supervised learning via risk decomposition.  In _International Conference on Machine Learning (ICML)_ , 2023.
  * Fu et al. [2023] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu.  Gptscore: Evaluate as you desire.  _arXiv preprint arXiv:2302.04166_ , 2023.
  * Ge et al. [2023] Yingqiang Ge, Wenyue Hua, Kai Mei, Jianchao Ji, Juntao Tan, Shuyuan Xu, Zelong Li, and Yongfeng Zhang.  Openagi: When llm meets domain experts, 2023.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Jia and Liang [2017] Robin Jia and Percy Liang.  Adversarial examples for evaluating reading comprehension systems.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2017\.
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * Jiang et al. [2024] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts, 2024.
  * Li et al. [2024] Xiang Li, Yunshi Lan, and Chao Yang.  Treeeval: Benchmark-free evaluation of large language models through tree planning.  _arXiv preprint arXiv:2402.13125_ , 2024.
  * Liang et al. [2022] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, D. Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji, O. Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, S. Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ , 2022.
  * Lin et al. [2021] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_ , 2021.
  * Liu et al. [2023] Nelson F. Liu, Tony Lee, Robin Jia, and Percy Liang.  Do question answering modeling improvements hold across benchmarks?  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13186–13218, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.736.  URL https://aclanthology.org/2023.acl-long.736.
  * Luo et al. [2023] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023.
  * Maia Polo et al. [2024] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin.  tinybenchmarks: evaluating llms with fewer examples.  _arXiv preprint arXiv:2402.14992_ , 2024.
  * McKenzie et al. [2022] Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez.  The inverse scaling prize, 2022.  URL https://github.com/inverse-scaling/prize.
  * Nie et al. [2020] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, J. Weston, and Douwe Kiela.  Adversarial nli: A new benchmark for natural language understanding.  In _Association for Computational Linguistics (ACL)_ , 2020.
  * OpenAI [2022] OpenAI.  Introducing ChatGPT.  https://openai.com/blog/chatgpt, 2022.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Peng et al. [2023] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4, 2023.
  * Perez et al. [2022] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.  Red teaming language models with language models.  _arXiv preprint arXiv:2202.03286_ , 2022.
  * Perlitz et al. [2024] Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen.  Efficient benchmarking of language models, 2024.
  * Rein et al. [2023] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.  Gpqa: A graduate-level google-proof q&a benchmark, 2023.
  * Ribeiro et al. [2020] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Association for Computational Linguistics (ACL)_ , pages 4902–4912, 2020.
  * Sakaguchi et al. [2019] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  _arXiv preprint arXiv:1907.10641_ , 2019.
  * Saxton et al. [2019] David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli.  Analysing mathematical reasoning abilities of neural models.  _ArXiv_ , abs/1904.01557, 2019.  URL https://api.semanticscholar.org/CorpusID:85504763.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Team et al. [2024] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, HyunJeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd Al Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross McIlroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Iinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang, Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Kępa, Yomna Eldawy, Jiawern Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang, Richie Feng, Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg, Yifan He, Oleksii Duzhyi, Anton Älgmyr, Timothée Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François-Xavier Aubet, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana-Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Styrc, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih-Wei "Louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O’Neill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Prateek Kolhar, Chris Gorgolewski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolò Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Ähdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojciech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Bražinskas, Andrei Sozanschi, Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Kärrman, Paweł Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan, Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar, Lara Tumeh, Eyal Ben-David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikołaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J. Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Santamaria-Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Põder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucińska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang, Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhania, Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani, Omar Alhadlaq, Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng, Ben Bariach, Laura Weidinger, Tu Vu, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals.  Gemini: A family of highly capable multimodal models, 2024.
  * Team [2023] Xwin-Math Team.  Xwin-math, 11 2023.  URL https://github.com/Xwin-LM/Xwin-LM/Xwin-Math.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _arXiv_ , 2023.
  * Tunstall et al. [2023] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf.  Zephyr: Direct distillation of lm alignment, 2023.
  * Wang et al. [2023] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu.  Openchat: Advancing open-source language models with mixed-quality data.  _arXiv preprint arXiv:2309.11235_ , 2023.
  * Xu et al. [2020] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang.  Adversarial domain adaptation with domain mixup.  In _Association for the Advancement of Artificial Intelligence (AAAI)_ , volume 34, pages 6502–6509, 2020.
  * Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging LLM-as-a-judge with MT-bench and chatbot arena.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=uccHPGDlao.

##  Appendix A Limitation

Recall that in AutoBencher, the evaluation datasets are fully model generated.
While these datasets serve as good proposal distributions for finding novel
patterns, we believe that the novel patterns should be confirmed and verified
by humans before making model development decisions based on these findings.

Additionally, we are using GPT-4 as the $\texttt{LM}_{\text{gen}}$, which
could potentially bias in favor of models in the same families such as
GPT-3.5-turbo. However, empirically, we find that this is not obvious, as
AutoBencher generated dataset very often rates Claude-3 models the highest.

##  Appendix B Broader Impact

Automatic benchmark creation via AutoBencher has several potential negative
impacts if used improperly. Primarily, since these created benchmarks can help
identify research directions to work on or weaknesses in existing models,
adequate care needs to be taken to ensure that the benchmark results are
indeed meaningful. Since the benchmark creation is automated, there is
potential for weird or insignificant results to arise, and users must not
blindly trust these results, but manually quality-check them before drawing
significant conclusions. Additionally, we cannot let automatic benchmark
creation prevent humans from investing more thought and effort into human data
curation, which can serve complementary functions.

##  Appendix C More details on Experimental Setup

Recall in § 5.1, we compare AutoBencher with human-generated benchmarks as
baseline. Here is the detailed HumanBench for each domain:

For history, we compare with $4$ history subjects high school world history,
prehistory, high school european history, high school us history.

For economy, we compare with $4$ subjects: high school microeconomics,
econometrics, high school macroeconomics, marketing.

For science, we compare with $7$ subjects: high school physics, college
physics, college chemistry, high school chemistry, high school biology,
college biology, astronomy.

For the LMs $\texttt{LM}\in\mathcal{M}$ that we evaluate. We list their
sources with proper citations in Appendix G.

To compare the response of a LM to the dataset label, we use a language model
(i.e., gpt-4-0125-preview) to judge the correctness of the model-generated
response, and output reasons for the judgment.

##  Appendix D Rank Analysis

We report the models’ ranking and their respective accuracies on AutoBencher
datasets in Table 6, Table 5. We highlight the models that perform worse than
expected (in red), and the models that perform better than expected (in
green).

We also provide the ranking results of our human study in Table 4.

Table 4:  The model ranking results of the human study. We highlight the very significant novel trends. We use red to label models that perform worse than expected, and green to label models that perform better than expected. | History | Science | Economy
---|---|---|---
| pred | gold | avg | pred | gold | avg | pred | gold | avg
claude-3-opus-20240229 | 1 | 3 | 2 | 2 | 2 | 1 | 5 | 5 | 1
gpt-4-turbo-2024-04-09 | 2 | 2 | 1 | 1 | 1 | 3 | 1 | 1 | 2
claude-3-sonnet-20240229 | 3 | 1 | 3 | 4 | 3 | 2 | 7 | 2 | 3
claude-2.0 | 5 | 4 | 4 | 3 | 4 | 4 | 4 | 3 | 4
Mixtral-8x7B-Instruct-v0.1 | 4 | 9 | 5 | 5 | 6 | 5 | 6 | 6 | 5
gemini-pro | 6 | 8 | 6 | 6 | 5 | 7 | 3 | 15 | 6
gpt-3.5-turbo-0613 | 7 | 7 | 7 | 7 | 7 | 6 | 2 | 8 | 7
openchat-3.5-0106 | 8 | 5 | 8 | 8 | 8 | 8 | 10 | 7 | 8
zephyr-7b-beta | 10 | 11 | 10 | 10 | 10 | 9 | 9 | 12 | 9
OpenAGI-7B-v0.1 | 9 | 6 | 9 | 9 | 9 | 10 | 8 | 4 | 10
Mistral-7B-Instruct-v0.1 | 12 | 16 | 11 | 11 | 11 | 11 | 12 | 13 | 11
vicuna-7b-v1.5 | 15 | 14 | 12 | 12 | 12 | 12 | 11 | 10 | 12
Llama-2-7b-chat-hf | 16 | 15 | 15 | 14 | 13 | 14 | 13 | 11 | 13
Xwin-Math-7B-V1.0 | 14 | 10 | 14 | 15 | 14 | 13 | 15 | 16 | 14
WizardMath-7B-V1.0 | 13 | 13 | 13 | 16 | 16 | 15 | 14 | 14 | 15
alpaca-7b | 11 | 12 | 16 | 13 | 15 | 17 | 16 | 9 | 16
gpt-neo-2.7B | 17 | 17 | 17 | 17 | 17 | 16 | 17 | 17 | 17
Table 5:  The model ranking results of the datasets constructed by AutoBencher. We highlight the very significant novel trends. We use red to label models that perform worse than expected, and green to label models that perform better than expected. | History | Economy | Science
---|---|---|---
| pred | gold | avg | pred | gold | avg | pred | gold | avg
claude-3-opus-20240229 | 1 | 2 | 2 | 2 | 3 | 1 | 1 | 2 | 1
gpt-4-turbo-2024-04-09 | 2 | 1 | 1 | 1 | 4 | 2 | 4 | 3 | 3
claude-3-sonnet-20240229 | 4 | 4 | 3 | 10 | 2 | 3 | 2 | 1 | 2
claude-2.0 | 5 | 6 | 4 | 4 | 5 | 4 | 3 | 7 | 4
Mixtral-8x7B-Instruct-v0.1 | 3 | 7 | 5 | 6 | 12 | 5 | 5 | 5 | 5
gemini-pro | 6 | 16 | 6 | 5 | 15 | 6 | 7 | 14 | 7
gpt-3.5-turbo-0613 | 7 | 3 | 7 | 3 | 7 | 7 | 8 | 6 | 6
openchat-3.5-0106 | 8 | 9 | 8 | 8 | 1 | 8 | 14 | 8 | 8
zephyr-7b-beta | 11 | 11 | 10 | 9 | 14 | 9 | 10 | 13 | 9
OpenAGI-7B-v0.1 | 9 | 5 | 9 | 7 | 9 | 10 | 6 | 4 | 10
Mistral-7B-Instruct-v0.1 | 12 | 10 | 11 | 11 | 8 | 11 | 13 | 12 | 11
vicuna-7b-v1.5 | 15 | 12 | 12 | 12 | 6 | 12 | 11 | 9 | 12
Llama-2-7b-chat-hf | 14 | 13 | 15 | 13 | 11 | 13 | 12 | 10 | 14
Xwin-Math-7B-V1.0 | 16 | 14 | 14 | 14 | 16 | 14 | 16 | 16 | 13
WizardMath-7B-V1.0 | 13 | 15 | 13 | 15 | 10 | 15 | 15 | 15 | 15
alpaca-7b | 10 | 8 | 16 | 16 | 13 | 16 | 9 | 11 | 17
gpt-neo-2.7B | 17 | 17 | 17 | 17 | 17 | 17 | 17 | 17 | 16
Table 6:  LMs’ accuracy on datasets constructed by AutoBencher. | History | Economy | Science
---|---|---|---
Models | AutoBench | MMLU | AutoBench | MMLU | AutoBench | MMLU
claude-3-opus-20240229 | 0.51 | 0.93 | 0.64 | 0.88 | 0.50 | 0.81
gpt-4-turbo-2024-04-09 | 0.53 | 0.93 | 0.62 | 0.85 | 0.50 | 0.69
claude-3-sonnet-20240229 | 0.42 | 0.88 | 0.67 | 0.78 | 0.50 | 0.71
claude-2.0 | 0.42 | 0.85 | 0.62 | 0.78 | 0.42 | 0.68
Mixtral-8x7B-Instruct-v0.1 | 0.40 | 0.85 | 0.55 | 0.76 | 0.43 | 0.68
gemini-pro | 0.28 | 0.84 | 0.48 | 0.75 | 0.26 | 0.60
gpt-3.5-turbo-0613 | 0.51 | 0.82 | 0.60 | 0.72 | 0.42 | 0.63
openchat-3.5-0106 | 0.40 | 0.79 | 0.67 | 0.69 | 0.41 | 0.58
zephyr-7b-beta | 0.35 | 0.72 | 0.48 | 0.66 | 0.30 | 0.58
OpenAGI-7B-v0.1 | 0.42 | 0.77 | 0.55 | 0.66 | 0.44 | 0.57
Mistral-7B-Instruct-v0.1 | 0.37 | 0.66 | 0.57 | 0.56 | 0.35 | 0.48
vicuna-7b-v1.5 | 0.35 | 0.64 | 0.60 | 0.52 | 0.38 | 0.42
Llama-2-7b-chat-hf | 0.33 | 0.57 | 0.55 | 0.50 | 0.37 | 0.38
Xwin-Math-7B-V1.0 | 0.33 | 0.58 | 0.38 | 0.45 | 0.17 | 0.39
WizardMath-7B-V1.0 | 0.30 | 0.59 | 0.55 | 0.44 | 0.20 | 0.37
alpaca-7b | 0.40 | 0.37 | 0.55 | 0.35 | 0.35 | 0.28
gpt-neo-2.7B | 0.26 | 0.25 | 0.26 | 0.27 | 0.09 | 0.31

##  Appendix E AutoBencher Search Trajectory

In order to analyze AutoBencher, we provide intermediate search results of the
AutoBencher. Figure 3, Figure 5 and Figure 4 show the search trajectory of
AutoBencher for history, economy, and science domains. Specifically, we report
the evaluation topics that were explored and their respective accuracy as a
Star plot.

Figure 3:  Search trajectories of AutoBencher (history) with different
$\texttt{LM}_{\text{test}}$. It shows the evaluation topics that are explored
and their respective accuracy as a star plot.

Figure 4:  Search trajectories of AutoBencher (science) with different
$\texttt{LM}_{\text{test}}$. It shows the evaluation topics that are explored
and their respective accuracy.

Figure 5:  Search trajectories of AutoBencher (economy) with different
$\texttt{LM}_{\text{test}}$. It shows the evaluation topics that are explored
and their respective accuracy.

Figure 6:  The histogram of accuracy for all topics explored in a AutoBencher
run. The three rows are economy, science, and history respectively.

##  Appendix F More Results on Separation and Headroom

In Figure 7, we show the Pareto frontier of the two difficulty metrics: Sep
and Difficulty. Each orange stars represent datasets constructed by
AutoBencher, and each blue dot represents an MMLU subject. Datasets
constructed by AutoBencher are mostly at the Pareto frontier, outperforming
MMLU subjects in both metrics.

Figure 7:  We show the Pareto frontier of the two difficulty metrics: Sep and
Difficulty. Each orange stars represent datasets constructed by AutoBencher,
and each blue dot represents an MMLU subject. Datasets constructed by
AutoBencher are mostly at the Pareto frontier, outperforming MMLU subjects in
both metrics.

##  Appendix G Asset License

We list the license associated with each model below:

  * •

gpt-4-turbo-2024-04-09 [24]: Proprietary

  * •

gpt-3.5-turbo-0613 [23]: Proprietary

  * •

claude-3-sonnet-20240229 [1]: Proprietary

  * •

claude-3-opus-20240229 [1]: Proprietary

  * •

claude-2.0: Proprietary

  * •

Mixtral-8x7B-Instruct-v0.1 [14]: Apache 2.0

  * •

Mistral-7B-Instruct-v0.1 [13]: Apache 2.0

  * •

gemini-pro [33]: Proprietary,

  * •

OpenAGI-7B-v0.1 [10]: Apache-2.0

  * •

vicuna-7b-v1.5 [25]: Llama2,

  * •

Llama-2-7b-chat-hf [35]: Llama2,

  * •

Xwin-Math-7B-V1.0 [34]: Llama2,

  * •

WizardMath-7B-V1.0 [19]: Llama2,

  * •

gpt-neo-2.7B [4]: MIT

  * •

alpaca-7b [32]: CC by NC 4.0

  * •

zephyr-7b-beta [36]: MIT

  * •

openchat-3.5-0106 [37]: Apache-2.0

##  Appendix H Our License

We release our work under an MIT license.

##  Appendix I Details of Human Study

Recall in § 6.3, we conduct a human study to verify the trends found by
AutoBencher still holds for the human-constructed dataset. For this human
study, the instruction is to generate a set of question-answer pairs given a
topic $c$ (e.g., Fordism). The annotator may use resources from Wikipedia
(e.g., Wikipedia articles on Fordism), and also other linked Wikipedia pages.
The annotator should generate roughly $50$ questions per topic, and the
questions should be challenging. Additionally, each question should be
answerable by a domain expert. The generated answer for each question should
be correct, and concise. If the question is open-ended, the answer should then
cover as many correct responses as possible.