WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
Under review as a conference paper at ICLR 2025

WILDBENCH: BENCHMARKING LLMS WITH
CHALLENGING TASKS FROM REAL USERS IN THE WILD

Anonymous authors
Paper under double-blind review

ABSTRACT

We introduce WildBench, an automated evaluation framework designed to bench-
mark large language models (LLMs) using challenging, real-world user queries.
WILDBENCH consists of 1,024 examples carefully selected from over one million
human-chatbot conversation logs. For automated evaluation with WILDBENCH,
we have developed two metrics, WB-Reward and WB-Score, which are computable
using advanced LLMs such as GPT-4-turbo. WILDBENCH evaluation uses task-
specific checklists to evaluate model outputs systematically and provides structured
explanations that justify the scores and comparisons, resulting in more reliable
and interpretable automatic judgments. WB-Reward employs fine-grained pair-
wise comparisons between model responses, generating five potential outcomes:
much better, slightly better, slightly worse, much worse, or a tie. Unlike previous
evaluations that employed a single baseline model, we selected three baseline mod-
els at varying performance levels to ensure a comprehensive pairwise evaluation.
Additionally, we propose a simple method to mitigate length bias by converting
outcomes of “slightly better/worse” to “tie” if the winner’s response exceeds the
loser’s by more than K characters. WB-Score evaluates the quality of model
outputs individually, making it a fast and cost-efficient evaluation metric. WILD-
BENCH results demonstrate a strong correlation with the human-voted Elo ratings
from Chatbot Arena on hard tasks. Specifically, WB-Reward achieves a Pearson
correlation of 0.98 with top-ranking models. Additionally, WB-Score reaches 0.95,
surpassing both ArenaHard’s 0.91 and AlpacaEval2.0’s 0.89 for length-controlled
win rates, as well as the 0.87 for regular win rates.

1

INTRODUCTION

Large language models (LLMs) have become integral to a wide range of real-world applications
due to their strong generalization capabilities across diverse tasks. However, effectively evaluating
their performance remains a challenging problem, particularly when striving for an automated and
cost-effective solution. Traditional benchmarking datasets like MMLU (Li et al., 2023a) focus
primarily on assessing the reasoning abilities of LLMs using multiple-choice questions, which fall
short in evaluating the more open-ended problems that real-world users pose. Chatbot Arena (Chiang
et al., 2024) provides an online platform where human preferences are collected to judge pairs of
model outputs, subsequently ranking LLMs using Elo ratings. While this human-based evaluation
method offers valuable insights into user preferences, it has notable limitations, such as high labor
costs, the inability to deliver real-time results, a lack of data transparency, and the challenge of fairly
evaluating all models with the same data.

Several automated benchmarks such as AlpacaEval (Li et al., 2023b), MT-bench (Zheng et al., 2024),
and ArenaHard (Li et al., 2024) employ advanced LLMs like GPT-4-Turbo to assess the quality of
model responses. Comparative analyses of these benchmarks are presented in Table 1 and Figure 3.
These existing benchmarks exhibit significant shortcomings in task composition and skill coverage,
particularly in mirroring the natural distribution of real-world user tasks. MT-bench, comprising
only 80 hand-crafted examples, lacks sufficient breadth for a comprehensive evaluation. Meanwhile,
AlpacaEval, with 805 tasks derived from multiple alignment datasets, includes relatively simple tasks,
such as “What is the capital of Australia?” and suffers from low task diversity; for instance, over 20
tasks redundantly assess recipe generation skills (e.g., “can you provide a recipe for ...?”). We show a
few examples in Figure 1 to illustrate the differences between AlpacaEval and our WILDBENCH.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Example tasks sampled from AlpacaEval (Li et al., 2023b) and WILDBENCH. Tasks
in WILDBENCH are more diverse and challenging, which are collected from real users in the wild.
Complex real-user tasks usually have multiple constraints and require higher-order reasoning skills,
which are well represented in WILDBENCH.

AlpacaEval mostly focuses on information-seeking tasks, containing merely 6% coding and 3%
mathematics tasks. Conversely, ArenaHard, sampling 500 tasks from ChatbotArena, displays an
excessive concentration on coding and debugging tasks, accounting for over 57% of its content. Most
existing benchmarks do not sufficiently challenge the models with the varied and unexpected nature
of user inquiries in practical settings, thus limiting their overall effectiveness in providing a holistic
evaluation. This issue highlights the necessity for more comprehensive benchmarks that can better
simulate the wide range of tasks from real users.

In this paper, we introduce WILDBENCH, an automated evaluation framework designed for assessing
LLMs using complex tasks from real-world users. The examples in WILDBENCH are periodically
updated, with the current version (V2) comprising 1,024 tasks carefully curated from real user-chatbot
dialogs provided by the AI2’s WildChat project (Zhao et al., 2024). We engage multiple advanced
LLMs to process a filtered selection from WildChat, tasking them with the analysis of the requisite
knowledge and skills for each task and subsequently labeling the difficulty level. Tasks considered as
easy by all models are excluded. We ensure the distribution of tasks mirrors the original WildChat
data, such that the task distribution of WILDBENCH is still natural (Figure 3). Additionally, all
finalized tasks undergo manual review. Further details are provided in Section 2.

As shown in Figure 1, WILDBENCH presents a significantly harder challenge due to the complexity,
depth, and realism of the tasks involved. WILDBENCH is sourced from real-world user interactions
and has been carefully curated to ensure diversity and challenge. The tasks in WILDBENCH typically
demand higher-order reasoning, such as writing and/or debugging code with specific constraints,
creative writing with multiple constraints on the style and content, or designing a software system
with complex requirements. These tasks often require critical thinking, creativity, and technical
expertise, making WILDBENCH substantially more challenging than AlpacaEval, where simpler,
factual, or surface-level tasks dominate.

WILDBENCH evaluation is illustrated in Figure 4. To design a reliable automatic evaluation, we
employ two key designs for using LLMs as judges. Drawing inspiration from how humans evaluate
responses to open-ended questions, we develop task-specific checklists. These checklists guide LLMs
in generating consistent and reliable judgments, with each checklist comprising questions focused on
specific criteria. Similar to the zero-shot Chain-of-Thoughts (CoT) prompting (Kojima et al., 2022),
we prompt LLMs to provide step-by-step, structured analyses of each LLM response. This method
encourages a detailed, fine-grained evaluation process, culminating in a well-justified final decision.

We employ two primary metrics: WB-Reward for pairwise comparisons and WB-Score for individual
scoring. WB-Reward is based on pairwise comparisons between LLMs, with five possible outcomes:
“A is much/slightly better/worse than B” or “Tie.” Notably, we used three baseline models to compare
with each testing model instead of using a single baseline model, as most prior works do. This
approach provides a more comprehensive assessment based on different levels of model performance.
WB-Score measures the quality of each model’s generation individually, offering a quicker and
more cost-effective evaluation. To mitigate the bias towards longer outputs, a common issue in
LLM-as-a-judge evaluations (Dubois et al., 2024), we introduced a simple length-penalty method,
converting slight wins/losses to ties when the winner’s output is significantly longer than the loser’s.

2

What is the capital of Australia?What is some cool music from the 1920s?How do I wrap a present neatly?Can you write code?~20 recipe generation tasks AlpacaEvalPlease provide me python code to go through a directory and its subdirectories and delete images that are not horizontal.hey can you write an essay on the impact of the G20 summit on the global economy, trade, development and the role of young people in shaping the future of the world, it has to have more than 1200 words. Write it beau>ful and poe>c. Use extensive vocabulary. Use a lot of factual and empirical data. Use some, ancient indian historical references.I want to create an open source, highly realistic and grounded text-based business simulation game that is played in the terminal, with a large range of different features that make the game as realistic a simulation as possible. In light of this the game should not have set values for anything because that is unrealistic - real life isn’t like that; the sim should be as close to reality as possible. I will host it on Github. Please create a FULL, COMPLETE file structure for the game’s Github repo.Diverse tasks from real users! 123Under review as a conference paper at ICLR 2025

Both metrics have demonstrated strong correlations with human judgments, evidenced by a Pearson
correlation of 0.98 for WB-Reward and 0.95 for WB-Score against the human-voted Elo rating from
Chatbot Arena on the top-ranking models. These scores significantly surpass other benchmarks,
such as ArenaHard(Li et al., 2024)’s 0.91 and AlpacaEval2.0’s 0.87 (0.89 for the length-controlled
version) (Li et al., 2023b; Dubois et al., 2024), validating WILDBENCH’s effectiveness and alignment
with human-based evaluation. More details are shown in Table 3 in Section 4.

2 WILDBENCH DATA CURATION

In this section, we describe the data curation process for the tasks used to evaluate LLMs in WILD-
BENCH . Our goal is to ensure that the selected tasks not only represent real-world use cases but are
also challenging enough to distinguish the varying capabilities of LLMs.

Table 1: Statistical comparison of LLM alignment benchmarks. Length are in characters.

Dataset

MT-Bench
AlpacaEval
ArenaHard

#Tasks #Turns ChatHistory QueryLen PromptLen RealUser TaskTag

Evaluation

80
805
500

2
1
1

¸Dynamic
Ø
Ø

¸Static

202.2
164.9
406.4

978.5

Dynamic
164.9
406.4

3402.1

Ø
Ø
¸

¸¸

¸
Ø
Ø

¸

Score
Pair (ref=1)
Pair (ref=1)

Score+Pair (ref=3)

WILDBENCH

1,024

≤5

Figure 2: Distribution of query lengths in AlpacaEval, ArenaHard, and WildBench.

2.1 MINING CHALLENGING TASKS FROM WILDCHAT

We sourced tasks from the WildChat dataset (Zhao et al., 2024), which comprises one million
human-chatbot conversations from real users. This dataset is particularly suited for conversion into an
evaluation benchmark because it contains a diverse array of tasks that users expect LLMs to perform,
such as writing assistance, coding, mathematics, data analysis, role playing, and planning.

Basic filtering. To control the quality and diversity of the selected tasks, we applied several filtering
steps. First, we removed user queries that were either too short (less than 10 tokens) or excessively
long (more than 3,000 tokens). We also excluded conversations with more than five user-chatbot
turns to maintain focus and coherence in the tasks, as conversations exceeding five turns tend to
contain multiple topics. Furthermore, we focused on English data and filtered out non-English tasks.
Since our focus is more on evaluating the capabilities of LLMs rather than content moderation, we
also removed toxic conversations. To ensure task diversity, we used sentence embeddings from
SentenceBERT (Reimers & Gurevych, 2019) to calculate the cosine similarity between queries,
discarding those with a high similarity score above 0.9. The threshold is determined by manual
inspection. Lastly, to further enhance task diversity, we used a diverse user pool by retaining only the
last conversation for each unique device, thus removing tasks from the same user that might require
similar underlying skills.

Difficulty annotation. To identify challenging tasks that can distinguish the performance of different
LLMs, we used GPT-4-Turbo (OpenAI, 2023), Claude-3-Sonnet, and Opus (Anthropic, 2024) to
analyze the required background knowledge and reasoning capabilities for each task. These models
assigned a difficulty rating on a five-point scale (from “very easy” to “very hard”). Tasks rated as
“very easy” or “easy” by all models were excluded. From the remaining pool, we randomly sampled
1,500 tasks to ensure that the distribution of task categories is similar to the original dataset.

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 3: Distribution of task categories in AlpacaEval, ArenaHard, and WildBench.

Human annotation. To improve the quality of selected tasks, human annotation was used for quality
control. We first used GPT-4-Turbo to summarize the intent of each query. These summaries were
then used to help human reviewers remove nonsensical tasks. Finally, we retained 1,024 tasks for
WILDBENCH. We also manually reviewed the tasks to ensure that they were challenging and diverse,
covering a wide range of task categories. For the checklist questions, we verified that they were clear,
interpretable, and relevant to the evaluation of LLM responses.

Dynamic updates and data leakage prevention. WILDBENCH is designed to be a dynamic
benchmark that is updated regularly to reflect new types of user interactions. In fact, we have already
released two versions of the benchmark (V1 in 2024 March and V2 in 2024 May), with similar
curation process but on different iterations of WildChat data. To prevent potential data leakage for
LLMs that use WildChat as part of their training or alignment, we coordinated with the WildChat
team to ensure that the tasks we sample will not be publicly available in the WildChat dataset.

2.2 WILDBENCH STATISTICS

To better understand the composition of our evaluation, we analyze basic statistics and task categories.

Basic statistics. Table 1 compares the statistics of WILDBENCH to existing benchmarks AlpacaE-
val (Li et al., 2023b; Dubois et al., 2024), MT-Bench (Zheng et al., 2024), and ArenaHard (Li et al.,
2024). Among these benchmarks, only ArenaHard and WILDBENCH are sourced from user queries in
the wild (“RealUser”), rather than being curated by experts or through crowdsourcing. The difference
between ArenaHard and our WildBench is that our data distribution aligns with real users’ task
categories, rather than overly focusing on coding and debugging as ArenaHard does.

Long-context tasks. WILDBENCH includes conversation histories of up to four turns per conversa-
tion, reflecting complex and extended user interactions that are facilitated by recent advancements in
LLMs, with over 20% of conversations having more than two or more turns as shown in Figure 8. Ad-
ditionally, as shown in Figure 2, WILDBENCH has longer query lengths, attributable to the extensive
context provided by real user interactions captured in the dataset. This is because that GPT-4-Turbo,
one of the chatbots behind WildChat, supports up to 128K context tokens and 4K output tokens.
This capability exemplifies the importance of a dynamic, in-the-wild benchmark: as models evolve,
they unlock new user applications. Thanks to these realistic user activities, WILDBENCH is a more
suitable benchmark for testing the long-context problem solving abilities of LLMs.

Task categories. To enable a fine-grained analysis of LLM capabilities across varied tasks, we
categorize the tasks into 12 categories based on previous analysis of ShareGPT queries (Ouyang et al.,
2023) and our intent annotation of the tasks. Detailed descriptions about the 12 task categories are
shown in Appendix A. The distribution of the task categories is shown in Figure 3. In this figure, we
also compare to AlpacaEval and ArenaHard. Notably, WILDBENCH is more balanced compared to
AlpacaEval and ArenaHard, which have over 50% of their tasks in Information seeking and Coding
& Debugging categories, respectively.

3 AUTOMATIC EVALUATION WITH WILDBENCH

In this section, we introduce the evaluation process of LLMs using WILDBENCH. We first explain
how we generate a checklist for each test query to enhance interpretability and reduce evaluation

4

Information seekingCoding & DebuggingAlpacaEval (805)ArenaHard (500)🌟WildBench (1024)Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 4: Evaluation framework for WILDBENCH. There are two metrics: WB-Score for individual
evaluation and WB-Reward for pairwise evaluation. The checklist is used to guide the evaluation
process. The length penalty is used to mitigate the length bias. WB-Reward and WB-Score both have
strong correlations with human-based ranking of LLMs on Chatbot Arena.

ambiguity in WILDBENCH. Then, we introduce two automatic metrics: WILDBENCH-Score and
WILDBENCH-Reward. Finally, we discuss how we mitigate the length bias in the evaluation process.

3.1

INSTANCE-SPECIFIC CHECKLISTS

Powerful LLMs have been widely used as judges to evaluate the quality of LLM outputs in many
automatic evaluation methods, such as AlpacaEval (Li et al., 2023b). However, even asking humans
to judge which of the given two model outputs is better can be subjective and ambiguous. Moreover,
such judgements provide limited information about the quality of the models. Without a constant,
interpretable, and comprehensive evaluation standard, the results can be noisy and hard to interpret.

To address this issue, we generate a checklist for each test query in WILDBENCH to comprehensively
evaluate the responses of different models. The checklist consists of 5-10 questions that are designed
to be interpretable and easy to verify. We combine the responses of GPT-4-Turbo and Claude-3-Opus
to finalize the checklists, thereby mitigating the bias of using a single LLM as the evaluator. These
checklists have been manually reviewed and are used as part of the prompts for LLM judges to
evaluate the responses of different models. An example of the checklist can be found in Figure 4.
Taking the G20 example in Figure 1, here is a subset of checklist questions for the task:

Example checklist for the G20 task example in Figure 1.

¸ Does the essay contain more than 1200 words as requested by the user?
¸ Is the language of the essay beautiful and poetic, incorporating extensive vocabulary as

specified?

¸ Does the essay include a significant amount of factual and empirical data related to the

impact of the G20 summit on the global economy, trade, and development?

¸ Are there references to the role of young people in shaping the future of the world within

the context of the G20 summit?

¸ Does the essay include ancient Indian historical references as requested by the user?
¸ Is the essay structured in a clear and logical manner, facilitating an easy understanding

of the discussed topics?

3.2 PAIRWISE EVALUATION WITH WB-REWARD METRIC

WB-Reward is based on pairwise evaluation, which uses a GPT-4-Turbo judge to compare the
responses of two LLMs to determine which one performs better on a given task, using a structured
checklist to guide the comparison. This metric provides straightforward comparisons among models
and the intermediate outcomes of win/lose rates are easy to interpret.

5

IndividualLLM A’s responseLLM B’s responsejson_output = {  "analysis of A": "[analysis of Response A]",  "analysis of B": "[analysis of Response B]",  "reason of A=B": "[where Response A and B perform equally]",  "reason of A>B": "[where Response A is better than B]",  "reason of B>A": "[where Response B is better than A]",  "choice": "[A++ or A+ or A=B or B+ or B++]"} A++ means A is muchbetter, A+means A is slightlybetter,...🌟WB-Reward   Model X vs Y (Baseline) +1 when X>>Y; +0.5 when X>Y;-1 when X<<Y;-0.5 when X<Y; 0 when X=Y;w/ Length Penalty  Baseline Models è LLM response📝Checklistjson_output = {  "strengths": "[analysis for the strengths]",  "weaknesses": "[analysis for the weaknesses]",  "score": "[1~10]"} Score 5~6: The response is fair but has some issues (e.g., factual errors, hallucinations, missing key information); ...GPT-4T  Haiku Llama-2-70BPairwiseWB-Score💯👤Query💬History👤Query💬HistoryChecklist📝📝Example Task (history + query)👤User:I want a formula that will ﬁnd the last matching value in sheet named Requisition that matches the value in cell B1 of my current sheet and return the value from the row in column B ….🤖AI: ….  👤USER: the formula does not appear to be ﬁnding the last value in column A; 🤖AI: ….  👤USER: you provided the exact same formula, is there an alternative formula >> Coding & Debugging, Data AnalysisChecklist(a list of questions and criteria for eval)1⃣Does the alternative formula provided correctly address the user's need to ﬁnd the last matching value in a speciﬁed column and return a corresponding value from another column? 2⃣Is the alternative formula syntactically correct and compatible with spreadsheet software such as Microsoft Excel or Google Sheets? ...Correlation w/ ChatbotArena Elo(Pearson; Top; Hard-En-240520)WB-Score🦁WB-Reward🦁ArenaHardAE2-LCAE20.8650.8920.9090.9550.984Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Step-by-step evaluation process. In Figure 4, we detail the step-by-step evaluation process for
pairwise comparison. First, we provide a chain of evaluation questions to guide the LLM judge to
analyze the user query and the conversation history. The LLM then evaluates the two responses and
also analyze where and why one is better than the other. Finally, we ask the LLM to make a final
judgment on which response is better and why. This method is inspired by the evaluation process in
human evaluation, where human judges are asked to provide detailed feedback on the quality of the
responses before making a final decision. The full evaluation prompt can be found at Appendix D

WB-Reward metric. To compute the WB-Reward for a test model X against a baseline model Y, we
assign rewards based on the comparison result: +1 if X is much better than Y, +0.5 if X is slightly
better than Y, 0 for a tie, -0.5 for X is slightly worse than Y, and -1 for X is much worse than Y.

Baseline LLMs for pairwise evaluation. Using a single baseline model for pairwise evaluation
can lead to noisy and biased evaluations. To mitigate this issue, we use three baseline models
(GPT-4-Turbo-0429, Claude-3-Haiku, and Llama-2-70B-chat (Touvron et al., 2023)) to compute the
rewards for each model. Our metric WB-Reward (Mix) is the average of the rewards from these three
baselines on 1024 examples, providing a more robust performance evaluation on WILDBENCH.

Mitigating length bias with a margin for ties. Previous studies have shown that LLM judges tend
to prefer longer responses (Dubois et al., 2024). To mitigate this bias, we propose a simple and
intuitive length penalty method. If the winning response is longer than the losing one by a certain
threshold (K characters), we convert Slightly Win/Slightly Lose to a Tie. K can be customized via our
leaderboard web-page for personalized configuration. Setting K = ∞ will disable the length penalty.
We designed this feature to support a more personalized and flexible leaderboard. For example, users
who prefer shorter and more concise outputs can set a smaller K if they do not prioritize correlating
perfectly with the general human-based model rankings on ChatbotArena. This choice allows for a
customized leaderboard experience depending on user preferences.

3.3

INDIVIDUAL EVALUATION WITH WB-SCORE METRIC

Although pairwise evaluation provides a direct comparison between LLMs, it is usually more
expensive and time-consuming than grading each individual LLM generation. To individually
evaluate the performance of each model on WILDBENCH, we prompt GPT-4-Turbo to assign a score
from 1 to 10 for each model’s response. The full evaluation prompt can be found at Appendix E.

Score definition. To ensure a stable and consistent evaluation, we ask GPT-4-Turbo to evaluate the
quality of each response based on the checklist and provide detailed strengths and weakness of each
output before giving a score from 1 to 10. The scores are defined as follows:

• Score 1–2: The response is very poor and does not make sense at all.
• Score 3–4: The response is poor and does not help the user solve the problem meaningfully.
• Score 5–6: The response is fair but has issues (e.g., factual errors, hallucinations, missing key information).
• Score 7–8: The response is good but could be improved.
• Score 9–10: The response is perfect and provides helpful information to solve the problem.

Score rescaling. The WILDBENCH-Score is calculated as the average of the scores on all examples
tested, where each score is first subtracted by 5 and then multiplied by 2 (i.e., S′ = (S − 5) × 2). A
score of 5 represents a borderline acceptable response, so this rescaling can help to better differentiate
the performance of models that can effectively solve the tasks.

4 RESULTS & ANALYSIS

We analyze the performance of different models on WILDBENCH. We first present the leader-
board analysis, then examine the length bias issue in the evaluation process, and finally discuss the
correlation between WILDBENCH-Score and ChatbotArena Elo rating.

Leaderboard features. In Table 2, we present a subset of the results from our live leaderboard demo.
For the most up-to-date results and more interactive features, such as customizing length penalties and
viewing the detailed task-wise performance of each model, please refer to our live leaderboard. Our
live leaderboard also supports exploring data and comparing model outputs side by side to understand
the strengths and weaknesses of each model.

6

Under review as a conference paper at ICLR 2025

Table 2: Evaluation results (subset) of LLMs using WILDBENCH and other benchmarks. Please
refer to Figure 6-7 and demo website to view and interact with the full results.

Model names

WB-Reward (no length penalty) WB-
Mix ◎GPT4T ◎Haiku ◎Llama2 Score

Yi-1.5-34B-Chat

GPT-4o-0513 (cid:181) 35.7
1
◎ GPT-4-Turbo-0409 (cid:181) 34.6
2
GPT-4-Turbo-0125 (cid:181) 29.9
3
Gemini-1.5-Pro (cid:181) 27.8
4
Llama-3-70B-Inst
21
5
Claude 3 Opus (cid:181) 20.1
6
Gemini-1.5-Flash (cid:181) 17.4
7
8
16.8
10 Llama3-Inst-8B-SimPO 14
Claude 3 Sonnet (cid:181)
7.2
13
14
4.4
Qwen1.5-72B-Chat
Command-R-Plus (cid:181)
0.4
17
◎ Claude 3 Haiku (cid:181) -8.5
Mistral-Large (cid:181) -10.5
-11.9
-14.6
Command-R (cid:181) -16
-18.8
-21.6
-24.3
-25
Tulu-2-dpo-70b -25.4

Mixtral-8x7B-Inst
DBRX Inst
Yi-1.5-6B-Chat
Mistral-7B-Inst-v0.2

StarlingLM-7B-beta
Llama-3-8B-Inst

20
21
23
24
25
26
27
29
30
32

33
34
35
36
38
39
40

◎ Llama-2-70B-chat
Qwen1.5-7B-Chat

-26.8
-27
Phi-3-medium-128k -33.3
GPT-3.5-turbo-0125 -33.5
-48
-57
-74.1

Llama-2-7B-chat
Gemma-7B-it
Gemma-2B-it

1.5
0
-4.4
-4.4
-19
-20.4
-16.6
-18.3
-22.5
-31.6
-34.8
-36.3

-46.9
-48.1
-48.7
-49.8
-48.4
-53.4
-57.3
-55
-58.1
-59.3

-56.9
-57.7
-66.4
-66.3
-71.8
-78.4
-87.8

46.3
45.3
38.8
37.9
31.9
34.3
26.3
24.1
18.9
19.4
13.1
7.4

0
-4
-5
-9.7
-12.7
-13.5
-16.3
-19.9
-22.4
-20.3

-23.6
-23
-30
-30
-44.6
-55.8
-73.6

59.3
58.4
55.2
50
50.2
46.3
42.5
44.5
45.7
33.9
34.7
30.2

21.4
20.5
18
15.7
13.1
10.4
8.7
2.1
5.5
3.3

0
-0.2
-3.6
-4.1
-27.8
-36.8
-60.8

65.3
64.7
63.3
55.7
60.4
63.1
53.1
57.8
53.9
55.5
56.5
51.4

50.4
54.2
46.8
45.7
45.7
47.8
48.9
39.6
43.4
45.2

39.2
40
42.1
42.1
27.6
23.9
6.2

Arena Arena- AlpacaEval2

Elo

1293
1251
1239
-
1213
1232
-
-
-
1187
1143
1155

1169
1158
1111
1144
1106
1114
1106
-
1071
1099

1070
1059
-
1105
1012
1047
980

Hard

LC WR

-
82.6
78.0
-
41.1
60.4
-
-
33.8
46.8
36.1
33.1

41.5
37.7
23.0
20.6
17.0
23.4
23.9
-
-
15.0

11.6
-
-
23.3
4.6
7.5
3.0

57.5
55.0
-
-
34.4
40.5
-
-
44.7
34.9
36.6
-

-
32.7
-
22.9
-
23.7
25.4
-
17.1
21.2

14.7
14.7
-
-
5.4
10.4
5.4

51.3
46.1
-
-
33.2
29.1
-
-
40.5
25.6
26.5
-

-
21.4
-
22.6
-
18.3
18.4
-
14.7
16.0

13.9
11.8
-
-
5.0
6.9
3.4

By using three baseline models of varying performance levels (GPT-4-Turbo > Claude 3 Haiku >
Llama-2-70B-chat), we observe that the tested models can be naturally grouped into three tiers based
on their performance. Tier 1 models outperform Claude 3 Haiku, Tier 2 models outperform Llama-2-
70B-chat but are worse than Claude 3 Haiku, and Tier 3 models are worse than Llama-2-70B-chat.

4.1 LEADERBOARD ANALYSIS

Where are the gaps between models? A unique feature of the WILDBENCH leaderboard is the
ability to compare models across different task categories, which enables us to identify the strengths
and weaknesses of each model on different types of tasks. In Figure 5, we select a set of popular
models for analysis: Llama-3-8B-Inst (Meta, 2023), Llama-3-8B-Inst-SimPO (Meng et al., 2024b),
Yi-1.5-34B-chat (AI et al., 2024), Llama-3-70B-Inst, GPT-4-Turbo-0409, and Claude 3 Opus. We
show their performance in WB-Score across five task categories (merged from the 12 categories shown
in Figure 3). Larger models like GPT-4-Turbo-0409 and Claude 3 Opus perform well across all task
categories, while open LLMs like Llama-3-8B-Inst and Yi-1.5-34B-chat show weaker performance
on coding and math-related tasks.

Will an 8B model outperform a 70B model? On the AlpacaEval-2.0 leaderboard, Llama-3-8B-
Inst-SimPO (LC=44.7%) significantly outperforms Llama-3-70B-Inst (LC=34.4%) (Meng et al.,
2024a), which is surprising and differs from our results. As shown in both Table 2 and Figure 5, our
results indicate that Llama-3-8B-Inst-SimPO is generally still worse than Yi-34B-chat and Llama-3-
70B-Inst. However, on information-seeking and creative tasks, Llama-3-8B-Inst-SimPO performs
comparably to Llama-3-70B-Inst. Thus, we believe AlpacaEval’s evaluation results underestimate
the performance of Llama-3-70B-Inst due to task selection bias in addition to the weakness of their
evaluation prompting method. While the performance of Llama-3-8B-Inst-SimPO is not as good as it
seems on AlpacaEval-2.0, it is indeed the best 8B model in our evaluation and outperforms some
other larger models. Interestingly, Llama-3-8B-Inst-SimPO consistently improves the performance of
Llama-3-8B-Inst on all task categories, resulting in a similar shape on the radar plot in Figure 5.

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

Table 3: Correlation with Chatbot ArenaElo Elo (Hard-En-240520) of alignment benchmarks.

Metric

ArenaElo (Hard-En)

P-Cortop
1.000

P-Corall
1.000

S-Corall K-Corall
1.000
1.000

Arena-Hard
AlpacaEval2-LC
AlpacaEval2

WB-Score
WB-Rewardmix
∞
WB-Rewardmix
500

0.909
0.892
0.865

0.955
0.984
0.984

0.925
0.951
0.952

0.940
0.973
0.976

0.965
0.924
0.960

0.943
0.978
0.974

0.890
0.818
0.868

0.846
0.912
0.912

Metric

∞

∞

Avg Length
WB-Rewardllama
WB-Rewardgpt4t
WB-Rewardhaiku
WB-Rewardllama
500
WB-Rewardgpt4t
500
WB-Rewardhaiku
500

∞

P-Cortop
0.472

P-Corall
0.554

S-Corall
0.376

0.976
0.974
0.985

0.977
0.992
0.973

0.965
0.961
0.974

0.969
0.973
0.976

0.965
0.965
0.982

0.961
0.969
0.974

Are longer responses always better? WILD-
BENCH is robust to length bias. For example,
Llama-2-70B-chat and Llama-3-70B-Inst have
similar output lengths (2,965 vs 2,983 chars),
yet Llama-3-70B-Inst ranks 5th while Llama-2-
70B-chat ranks 33rd on the leaderboard of 40
models. Additionally, Yi-1.5-6B’s output length
is the 4th longest among the 40 models (3,322
characters), but it ranks 29th on the leaderboard.
This suggests that the WILDBENCH evaluation
is not biased towards longer responses, with re-
sponse quality being the most important factor
in the evaluation process. Additionally, we use
a length penalty to ensure that longer responses
are not always favored, and users can customize
the length penalty to adjust the trade-off be-
tween response length and quality according to
their needs. This feature is available on our live
leaderboard and is illustrated in Figure 6.

4.2 CORRELATION TO HUMAN JUDGMENT

Figure 5: Performance breakdown by task category
of 6 models on WILDBENCH.

To analyze how well WILDBENCH evaluation correlates with human judgment, we compare our
results to the ChatbotArena Elo rating generated by large-scale online human evaluations. Focusing
on hard prompts, we use the Elo ratings from the Hard-English version released on May 20, 2024.

We compare our WB-Reward and WB-Score with three other metrics: AlpacaEval winrate (WR),
length-controlled winrate (LC), and ArenaHard scores. We use three correlation metrics: Pearson
correlation (P-Cor), Spearman correlation (S-Cor), and Kendall’s tau correlation (K-Cor). To ensure
a fair comparison, we consider all models that have all four metrics available in Table 2, which results
in 14 models. To distinguish the top-performing models, we also consider the top 6 models, denoting
their correlation metrics as P-Cortop, and P-Corall respectively. The reason why we care about the
correlation on top-ranking models is that models released in the future are likely to compete with
the top models, so the Pearson correlation in this range is more important from the perspective of
predicting the future application of a metric. The analysis results are shown in Table 3.

Both WB-Reward and WB-Score show strong correlations with the human-based Elo rating, par-
ticularly for the top-performing models, achieving the best correlation among all other automatic
metrics. Among using different baseline models for pairwise evaluation, we find that using Haiku as
the baseline model yields the best correlation. These results suggest that the WILDBENCH evaluation
correlates well with human judgment in ranking model performance as an automatic metric.

4.3 ABLATION STUDIES AND DISCUSSIONS.

Checklists. In our ablation study on the impact of checklists, we compared model performance
with and without checklists by removing the associated parts from the prompt templates. The
results indicate that incorporating checklists improves the final correlation with human preferences.

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Reasoning & PlanningCreativeTasksCoding&DebuggingInfo SeekingMath&  DataUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Specifically, the WB-Score without checklists achieves a Pearson correlation of 0.905 (for all models),
which is lower than the 0.925 correlation achieved when using checklists.

Length penalties. We experimented with different K (100, 200, 500, 1000, inf) in the length penalty
method. We found that K = 500 is the best choice, as it achieves the highest correlation with human
judgments. This result suggests that the length penalty method is effective in mitigating the length
bias in LLM evaluations.

Do multiple LLMs as judges help? How much do multiple LLMs help? We experimented with
using GPT-4, Claude 3 Opus, and Mistral-Large as LLM judges. Our experiments revealed that
these LLM judges produced very similar results, thereby exerting minimal influence on the final
relative ranking of LLMs. Considering to reduce the cost of evaluation and faster turnaround time,
we recommend using a single LLM as a judge in practice. In the future versions, we will explore
more efficient ways to use multiple LLMs as judges, for example, by using different judge LLMs for
different tasks that are best suited to their strengths.

Data distribution. How do we explain that WildBench has a different distribution compared to
ChatbotArena’s platform but still shows a strong correlation, even better than ArenaHard? The
objective of WildBench is to evaluate LLMs on challenging tasks from real users. The ArenaElo
we use for comparison is derived from the hard-English split in ChatbotArena, where human users
submit tasks and vote. Thus, both WildBench and ChatbotArena aim to address the same goal. While
it is practically impossible to match the exact distribution of users and tasks between the two—given
that WildChat users are anonymous and ChatbotArena does not publicize its data—both are sourced
from real users on the web. Consequently, this represents the best possible approach for correlating
our LLM ratings with human-based ratings.

Two complementary metrics: WB-Reward & WB-Score. Both metrics use checklists and a
CoT-style prompt for evaluation, utilizing the same testing data. The key differences are in their
methodologies: WB-Score: Evaluates each model’s outputs individually on a scale of 1-10, with
detailed explanations for each score (see Appendix); WB-Reward: Compares a model’s outputs
to those of three baseline models at different performance levels for a comprehensive evaluation.
Pairwise evaluations can be coarse, but using three baseline models and refined pairwise choices
(e.g., much better or slightly better) mitigates this. WB-Score provides a universal score comparable
across models using the same evaluation templates and checklists. Additionally, WB-Score is cheaper
and faster to run (10 minutes, $5) compared to WB-Reward, which requires 3-4 times the cost due
to multiple baselines. Both metrics have their strengths and weaknesses. We use both to build our
official leaderboard, allowing users to choose the most suitable metrics for their experiments.

5 RELATED WORKS

Close-ended benchmarks. Close-ended benchmarks typically consist of multiple-choice questions
and have been widely used to evaluate LLMs authors (2022). For example, MMLU (Hendrycks et al.,
2020) includes multi-choice questions across various subject areas. Its variants include CMMLU (Li
et al., 2023a) for Chinese, KMMLU (Son et al., 2024) for Korean, and MMLU-Pro (Wang et al.,
2024) for more challenging evaluation. GPQA (Rein et al., 2023) is another close-ended benchmark
designed to be challenging even for humans with internet access. Specialized benchmarks with
ground-truth answers, such as GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021), also
fall into this category. While these benchmarks focus on close-form answers, our work evaluates
LLMs’ ability to generate free-form responses and engage in conversations with users.

Expert-curated and crowdsourced data. Several open-ended generation benchmarks rely on data
curated by human experts or crowdsourcing workers. For instance, MT-Bench (Zheng et al., 2024)
manually creates examples for predefined categories. AlpacaEval (Li et al., 2023b) is based on
author-written examples (Dubois et al., 2023; Taori et al., 2023; Wang et al., 2022), which primarily
consists of simple instructions such as rewriting tasks.

In-the-wild data. A key feature of our work is that its underlying data is sourced from real-world
use cases, ensuring alignment with actual LLM use cases. Notable benchmarks using real-world data
include ChatbotArena (Zheng et al., 2024; Chiang et al., 2024), where users input their questions
and choose the better response from two LLMs. However, ChatbotArena relies on extensive human
feedback. WildVision (Lu et al., 2024) is a similar project but designed for vision language models.

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

ArenaHard (Li et al., 2024) is another work that selects user queries from ChatbotArena to construct
a benchmark for automatic evaluation.

Evaluation methods. Evaluating open-ended generation poses challenges due to the lack of a single
valid ground truth. Human evaluation, though reliable, is expensive and time-consuming. To reduce
costs and enable fast evaluation, powerful LLMs are often used as judges, as seen in benchmarks like
MT-Bench, AlpacaEval, ArenaHard, and our own. Evaluation methods include single-system grading,
which assigns scores to individual outputs, and pairwise comparisons, which compare outputs of two
systems to compute win rates. Pairwise comparisons, while more expensive, can highlight subtle
differences across systems (Zheng et al., 2024). To mitigate self-selection bias where an LLM prefers
its own outputs (Panickssery et al., 2024), we use checklists generated from multiple LLMs, similar
to InfoBench (Qin et al., 2024). In addition, we ask LLM judges generate structured explanations
that enable human verification for further calibration, inspired by Just-Eval (Lin et al., 2023). There
are also local evaluators that can be used to evaluate LLMs with our WILDBENCH with open-weight
LLMs, such as TIGERScore (Jiang et al., 2023) and Prometheus (Kim et al., 2024).

Data leakage prevention. Publicly available benchmarks risk contamination from LLMs trained on
such data. GPQA includes a special string to help LLM developers filter out its data (Rein et al., 2023),
yet indirect leakage through cited examples remains possible. To mitigate this, we reserve a subset
of WildChat that is never released publicly, which keeps its expert-curated evaluation data private.
However, WILDBENCH provides a public validation set and details the benchmark construction
process for greater transparency.

Other dimensions for evaluation. While our focus is on evaluating LLM capabilities, other
evaluation dimensions, such as safety (Mazeika et al., 2024; Jiang et al., 2024), fairness (Gallegos
et al., 2024), logical reasoning (Lin et al., 2024), agentic planning (Liu et al., 2023; Mialon et al.,
2023; Lin et al., 2022), and hallucination detection (Min et al., 2023; Mishra et al., 2024; Hong et al.,
2024), are equally important.

6 CONCLUSION AND FUTURE DIRECTIONS

In this work, we introduced WILDBENCH, a benchmark designed to evaluate LLMs using real-
world user queries. An important feature of WILDBENCH data is the nature of in-the-wild user
queries with natural task distribution. To evaluate LLM performance using the collected data, we
introduced a CoT-like LLM-as-judge method to improve the interpretability of evaluations and reduce
ambiguity. We also incorporated a length penalty method to mitigate the length bias in LLM-as-judge
evaluations. Experiments show that our primary metrics, WB-Reward and WB-Score, have very
strong correlations with human judgments, surpassing existing evaluations.

We present extensive experiments and analyses, showcasing the performance of a wide range of 40
LLMs, including both proprietary and public ones, on the WILDBENCH benchmark. By providing a
detailed breakdown of scores across different task categories, WILDBENCH offers insights on the
strengths and weaknesses of different models. By introducing WILDBENCH, we aim to provide
a realistic, dynamic, and contamination-resilient evaluation framework that accurately reflects the
capabilities of LLMs. We will actively maintain the project for continually evaluating new LLMs
with unseen tasks over time.

REFERENCES

01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li,
Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin
Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu,
Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and
Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.

Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.
com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.
pdf, 2024.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

The BigBench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of
language models. ArXiv, abs/2206.04615, 2022. URL https://api.semanticscholar.
org/CorpusID:263625818.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng
Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena:
An open platform for evaluating llms by human preference, 2024.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher
Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint
arXiv:2110.14168, 2021.

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin,
Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that
learn from human feedback. arXiv preprint arXiv:2305.14387, 2023.

Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled

alpacaeval: A simple way to debias automatic evaluators, 2024.

Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernon-
court, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and fairness in large language models:
A survey, 2024.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
arXiv preprint

Jacob Steinhardt. Measuring massive multitask language understanding.
arXiv:2009.03300, 2020.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,
and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv
preprint arXiv:2103.03874, 2021.

Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie, Yu Zhao, Laura Perez-
Beltrachini, Max Ryabinin, Xuanli He, Clémentine Fourrier, and Pasquale Minervini. The hal-
lucinations leaderboard – an open effort to measure hallucinations in large language models,
2024.

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore:
Towards building explainable metric for all text generation tasks. Transactions on Machine
Learning Research, 2023.

Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar
Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. Wildteaming at scale:
From in-the-wild jailbreaks to (adversarially) safer language models. ArXiv, abs/2406.18510, 2024.
URL https://api.semanticscholar.org/CorpusID:270738096.

Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham
Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language
model specialized in evaluating other language models. arXiv preprint arXiv:2405.01535, 2024.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large
language models are zero-shot reasoners. ArXiv, abs/2205.11916, 2022. URL https://api.
semanticscholar.org/CorpusID:249017743.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy
Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint
arXiv:2306.09212, 2023a.

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion
Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL
https://lmsys.org/blog/2024-04-19-arena-hard/.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy
Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following
models. https://github.com/tatsu-lab/alpaca_eval, 2023b.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On
grounded planning for embodied tasks with language models. ArXiv, abs/2209.00465, 2022. URL
https://api.semanticscholar.org/CorpusID:251979509.

Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Raghavi
Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethink-
ing alignment via in-context learning. ArXiv, abs/2312.01552, 2023. URL https://api.
semanticscholar.org/CorpusID:265608902.

Bill Yuchen Lin, Ronan Le Bras, and Yejin Choi. Zebralogic: Benchmarking the logical rea-
soning ability of language models, 2024. URL https://hf.co/spaces/allenai/
ZebraLogicBench-Leaderboard.

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang
Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui
Zhang, Shengqi Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie
Tang. Agentbench: Evaluating llms as agents. ArXiv, abs/2308.03688, 2023. URL https:
//api.semanticscholar.org/CorpusID:260682249.

Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin.
Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint
arXiv:2406.11069, 2024.

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee,
Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standard-
ized evaluation framework for automated red teaming and robust refusal, 2024.

Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a
reference-free reward. 2024a. URL https://api.semanticscholar.org/CorpusID:
269983560.

Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-

free reward, 2024b.

Meta.

Introducing Meta Llama 3: The most capable openly available LLM to date. https:

//ai.meta.com/blog/meta-llama-3/, 2023.

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann André LeCun, and Thomas
Scialom. Gaia: a benchmark for general ai assistants. ArXiv, abs/2311.12983, 2023. URL
https://api.semanticscholar.org/CorpusID:265351664.

Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer,
Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual
precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023.

Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov,
and Hannaneh Hajishirzi. Fine-grained hallucination detection and editing for language models,
2024.

OpenAI. Gpt-4 technical report, 2023.

Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang
Zhu, Heng Ji, and Jiawei Han. The shifted and the overlooked: A task-oriented investigation
of user-GPT interactions. In The 2023 Conference on Empirical Methods in Natural Language
Processing, 2023. URL https://openreview.net/forum?id=qS1ip2dGH0.

Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own

generations, 2024.

Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng
Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Evaluating instruction following ability in
large language models, 2024.

12

Under review as a conference paper at ICLR 2025

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks,

2019.

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani,
Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark,
2023.

Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi,
Cheonbok Park, Kang Min Yoo, and Stella Biderman. Kmmlu: Measuring massive multitask
language understanding in korean. arXiv preprint arXiv:2402.11548, 2024.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy
Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.
https://github.com/tatsu-lab/stanford_alpaca, 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris-
tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,
Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn,
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee,
Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi,
Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen
Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic,
Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models,
2023.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions.
arXiv preprint arXiv:2212.10560, 2022.

Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming
Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi
Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language
understanding benchmark, 2024.

Wenting Zhao, Xiang Ren, John Frederick Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild-
chat: 1m chatgpt interaction logs in the wild. 2024. URL https://api.semanticscholar.
org/CorpusID:269390491.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang,
Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and
chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

Appendix

A TASK CATEGORIES

In Section 2.2 we mentioned that tasks are categorized into 12 categories to enable fine-grained
analysis of LLM capabilities. The definition of these task categories are as follows.

• Information seeking - Users ask for specific information or facts about various topics.
• Reasoning - Queries require logical thinking, problem-solving, or processing of complex ideas.
• Planning - Users need assistance in creating plans or strategies for activities and projects.
• Editing - Involves editing, rephrasing, proofreading, or other tasks related to the composition of

general written content.

• Coding & Debugging - Users seek help with writing, reviewing, or fixing code in programming.
• Math - Queries related to mathematical concepts, problems, and calculations.
• Role playing - Users engage in scenarios requiring ChatGPT to adopt a character or persona.
• Data Analysis - Requests involve interpreting data, statistics, or performing analytical tasks.
• Creative Writing - Users seek assistance with crafting stories, poems, or other creative texts.
• Advice seeking - Users ask for recommendations or guidance on various personal or professional

issues.

• Brainstorming - Involves generating ideas, creative thinking, or exploring possibilities.
• Others - Any queries that do not fit into the above categories or are of a miscellaneous nature.

We consolidate the original categories into five major groups for easier task-wise analysis. Specifically,
we combine “Information seeking” and “Advice seeking” into “Info Seeking”; “Math” and “Data
Analysis” into “Math & Data”; and “Reasoning” and “Planning” into “Reasoning & Planning.” The
remaining types are grouped under “Creative Tasks.” These consolidated groups are illustrated in
Figure 5.

Please note that the following links are anonymous for double-blind
review, which we will update after the review process. The supple-
mentary zip file contains the source code for the evaluation scripts,
the leaderboard, and the data.

Figure 8: Distribution of the
number of turns in WildBench.

B MORE INFORMATION ON WILDBENCH DATA

The distribution of the number of turns in WILDBENCH can be found
in Figure 8. The dataset documentation, metadata, and the public sub-
set of WILDBENCH can be found at https://huggingface.
co/datasets/anonymous/WildBench/viewer/v2. We
release the data under AI2’s ImpACT license as a low-risk artifact,
and we bear all responsibility in case of rights violations. We will
ensure that the dataset will be available for a long time and maintain
the data by continuously updating it.

C MORE INFORMATION ON WILDBENCH EVALUATION

Our evaluation results on the public subset of WILDBENCH can be reproduced using evaluation scripts
available at https://github.com/anonymous/WildBench/. We have included generation
script for each model under the folder https://github.com/anonymous/WildBench/
tree/main/scripts, and the scripts for evaluating generations can be found at https://
github.com/anonymous/WildBench/tree/main/evaluation.

D PROMPT TEMPLATE FOR PAIRWISE EVALUATION METRIC WB-REWARD

The prompt template for pairwise evaluation is shown below. It can be divided into three sections:
the first section provides the high-level instruction, the task to be tested, and two model outputs; the

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

second section specifies the checklist and the rules; and the last section instructs the LLM judge to
follow the step-by-step evaluation process as detailed in Section 3.2

# Instruction
You are an expert evaluator. Your task is to evaluate the quality of
(cid:44)→

the responses generated by two AI models. We will provide you with
the user query and a pair of AI-generated responses (Response A and
B). You should first read the user query and the conversation
history carefully for analyzing the task, and then evaluate the
quality of the responses based on and rules provided below.

(cid:44)→

(cid:44)→

(cid:44)→

(cid:44)→

# Conversation between User and AI

## History
<|begin_of_history|>
{$history}
<|end_of_history|>

## Current User Query
<|begin_of_query|>
{$user_query}
<|end_of_query|>

## Response A
<|begin_of_response_A|>
{$candidate_A}
<|end_of_response_A|>

## Response B
<|begin_of_response_B|>
{$candidate_B}
<|end_of_response_B|>

# Evaluation

## Checklist

<|begin_of_checklist|>
{$checklist}
<|end_of_checklist|>

Please use this checklist to guide your evaluation, but do not limit
(cid:44)→

your assessment to the checklist.

## Rules

You should compare the above two responses based on your analysis of
(cid:44)→

the user queries and the conversation history. You should first
write down your analysis and the checklist that you used for the
evaluation, and then provide your assessment according to the
checklist. There are five choices to give your final assessment:
["A++", "A+", "A=B", "B+", "B++"], which correspond to the
following meanings:

(cid:44)→

(cid:44)→

(cid:44)→

(cid:44)→

(cid:44)→

- `A++`: Response A is much better than Response B.
- `A+`: Response A is only slightly better than Response B.
- `A=B`: Response A and B are of the same quality. Please use this
(cid:44)→
- `B+`: Response B is only slightly better than Response A.
- `B++`: Response B is much better than Response A.

choice sparingly.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

## Output Format
First, please output your analysis for each model response, and then
(cid:44)→

summarize your assessment to three aspects: "reason A=B", "reason
A>B", and "reason B>A", and finally make your choice for the final
assessment.

(cid:44)→

(cid:44)→

filling in the placeholders in []:

Please provide your evaluation results in the following json format by
(cid:44)→
```
{

"analysis of A": "[analysis of Response A]",
"analysis of B": "[analysis of Response B]",
"reason of A=B": "[where Response A and B perform equally well]",
"reason of A>B": "[where Response A is better than Response B]",
"reason of B>A": "[where Response B is better than Response A]",
"choice": "[A++ or A+ or A=B or B+ or B++]",

}
```

E PROMPT TEMPLATE FOR INDIVIDUAL EVALUATION METRIC WB-SCORE

The prompt template for individual evaluation is shown below. It can be similarly divided into three
sections: the first section provides the high-level instruction, the task to be tested, and the model
output; the second section specifies the checklist and the rules; and the last section instructs the LLM
judge to follow the step-by-step evaluation process as detailed in Section 3.3.

# Instruction

the responses generated by AI models.

You are an expert evaluator. Your task is to evaluate the quality of
(cid:44)→
We will provide you with the user query and an AI-generated responses.
You should first read the user query and the conversation history
(cid:44)→

carefully for analyzing the task, and then evaluate the quality of
the responses based on and rules provided below.

(cid:44)→

# Conversation between User and AI

## History
<|begin_of_history|>

{$history}

<|end_of_history|>

## Current User Query
<|begin_of_query|>

{$user_query}

<|end_of_query|>

## AI Response
<|begin_of_response|>

{$model_output}

<|end_of_response|>

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

# Evaluation

## Checklist

<|begin_of_checklist|>

{$checklist}

<|end_of_checklist|>

Please use this checklist to guide your evaluation, but do not limit
(cid:44)→

your assessment to the checklist.

## Rules

user queries and the conversation history.

You should compare the above response based on your analysis of the
(cid:44)→
You should first write down your analysis and the checklist that you
(cid:44)→

used for the evaluation, and then provide your assessment according
to the checklist.

(cid:44)→
The scores are in the range of 1~10, where 1 means the response is very
(cid:44)→
Here are more detailed criteria for the scores:

poor and 10 means the response is perfect.

in a meaningful way.

- Score 1~2: The response is very poor and does not make sense at all.
- Score 3~4: The response is poor and does help user solve the problem
(cid:44)→
- Score 5~6: The response is fair but has some issues (e.g., factual
(cid:44)→
- Score 7~8: The response is good enough but could be improved in some
(cid:44)→
- Score 9~10: The response is perfect and provides helpful information
(cid:44)→

errors, hallucinations, missing key information).

that can help user solve the problem.

ways.

## Output Format
First, please output your analysis for the model response, and then
(cid:44)→

summarize your assessment to two aspects: "strengths" and
"weaknesses"; Finally, please write down your rating for the
assessment.

(cid:44)→

(cid:44)→

filling in the placeholders in []:

Please provide your evaluation results in the following json format by
(cid:44)→
```
{

"strengths": "[analysis for the strengths of the response]",
"weaknesses": "[analysis for the weaknesses of the response]",
"score": "[1~10]"

}
```

F FULL WILDBENCH LEADERBOARD

The full WILDBENCH leaderboard as of Jun 5, 2024 can be found in Figure 6; The updated leader-
board as of Sept 1, 2024 can be found in Figure 7. Note that we used a new metric named WB-Elo
that is based on merging WB-Reward and WB-Score to a collection of pairwise comparisons and
perform Elo rating updates on top of existing LMSYS Elo rating, thus we can have a faster and more
stable leaderboard update. You can view and interact with the latest results on our leaderboard on our
website at https://huggingface.co/spaces/anonymous/WildBench

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

Figure 6: Leaderboard of WildBench (2024 Jun 5th)

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Figure 7: Leaderboard of WildBench (2024 Sept 1st)
19

