TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs
Under review as a conference paper at ICLR 2025

TMGBENCH: A SYSTEMATIC GAME BENCHMARK
FOR EVALUATING STRATEGIC REASONING ABILITIES
OF LLMS

Anonymous authors
Paper under double-blind review

ABSTRACT

The rapid advancement of large language models (LLMs) has accelerated their
application in reasoning, with strategic reasoning drawing increasing attention.
To evaluate the strategic reasoning capabilities of LLMs, game theory, with its
concise structure, has become the preferred approach for many researchers. How-
ever, current research typically focuses on a limited selection of games, resulting
in low coverage of game types. Additionally, classic game scenarios carry risks
of data leakage, and the benchmarks used often lack extensibility, rendering them
inadequate for evaluating state-of-the-art models. To address these challenges, we
propose TMGBENCH, a benchmark characterized by comprehensive game type
coverage, novel and diverse scenarios, and flexible game organization. Specifi-
cally, we incorporate all 144 game types summarized by the Robinson-Goforth
topology of 2×2 games, which are constructed as classic games in our benchmark.
Furthermore, we employ synthetic data generation techniques to create diverse,
higher-quality game scenarios through topic guidance and human inspection for
each classic game, which we refer to as story-based games. Lastly, to provide a
sustainable evaluation framework adaptable to increasingly powerful LLMs, we
treat the aforementioned games as atomic units and organize them into more com-
plex forms through sequential, parallel, and nested structures. We conducted a
comprehensive evaluation of mainstream LLMs, covering tests on rational rea-
soning, reasoning robustness, Theory-of-Mind capabilities, and reasoning in com-
plex game forms. The results revealed that LLMs still have flaws in the accuracy
and consistency of strategic reasoning processes, and their levels of mastery over
Theory-of-Mind also vary. Additionally, o1-mini, the latest reasoning model from
OpenAI, was also evaluated across the sequential, parallel, and nested game struc-
tures and reached accuracy rates of 66.6%, 60.0%, and 70.0%, respectively, high-
lighting the challenges posed by TMGBENCH.

1

INTRODUCTION

The rapid advancement of large language models (LLMs) has reshaped the paradigm of artificial
intelligence, achieving breakthroughs across various domains (Zhao et al., 2023; Huang & Chang,
2022; Lewkowycz et al., 2022; Huang et al., 2022; Paranjape et al., 2023). These achievements
are largely attributed to LLMs’ ability to assimilate vast amounts of knowledge during training,
emerging with the capacity to organize information at a coarse level and link knowledge at a fine-
grained level through their internal representations (Min et al., 2023; Zhao et al., 2023). These core
capabilities have driven the success of LLMs in numerous reasoning tasks, including mathematical
reasoning (Hendrycks et al., 2021; Zhang et al., 2023), commonsense reasoning (Sap et al., 2019;
Bisk et al., 2020), logical reasoning (Lei et al., 2023), and strategic reasoning (Lor`e & Heydari,
2023; Duan et al., 2024). Among these, strategic reasoning has attracted considerable attention due
to its multi-agent nature and close association with social intelligence (Gandhi et al., 2023).

Strategic reasoning refers to the cognitive process of anticipating, planning, and responding to
others’ actions to achieve specific objectives within competitive or cooperative contexts (Zhang
et al., 2024a). Consequently, game scenarios—naturally involving both cooperation and com-
petition—have intuitively become a fertile ground for studying LLMs’ strategic reasoning abili-

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

ties (Brookins & DeBacker, 2023). In particular, researchers have engaged LLMs in game-playing,
analyzing their decision-making behaviors and evaluating their strategic intelligence in such scenar-
ios (Duan et al., 2024). The Prisoner’s Dilemma, as one of the most classic game theory scenarios,
has been extensively studied in this context (Herr et al., 2024). Additionally, other traditional games
such as the Battle of the Sexes (Kreps, 1990), the Stag Hunt (Carlsson & Van Damme, 1993), and
the Dictator Game (Forsythe et al., 1994) have also drawn significant attention. These studies pro-
vide initial insights into the strategic reasoning capabilities of LLMs (Horton, 2023; Brookins &
DeBacker, 2023; Phelps & Russell, 2023; Akata et al., 2023; Li et al., 2023; Aher et al., 2022).

However, current research has three major limitations, hindering a comprehensive, robust, and sus-
tainable evaluation of LLMs’ strategic reasoning capabilities: (1) Limited coverage of game types:
Most studies focus on a handful of classic games without considering the full diversity of game
structures. (2) Potential risk of game scenario leakage: Classic game scenarios are likely to be
present in the training corpus, raising concerns over data leakage. (3) Poor extensibility of game
forms: Existing studies primarily focus on a narrow range of game forms, which may no longer
suffice to challenge high-performing LLMs such as o1-mini from OpenAI.

To address the above issues, we introduce TMGBENCH, a benchmark that encompasses a com-
prehensive range of game types, features synthesized game scenarios, and supports scalable and
reorganizable game forms. Specifically, to address the first issue, we include all 144 game types de-
fined by the Robinson-Goforth topology of 2x2 games (Robinson & Goforth, 2005). This topology
encompasses a variety of game structures based on different numerical payoff matrices, including
but not limited to classic games like the Prisoner’s Dilemma(§2.2). To address the second issue, we
employ synthetic data generation techniques to create five different story-based games for each clas-
sic game. In essence, a story-based game is a contextual framing counterpart of its corresponding
classic game, sharing the same structure but differing in context (Lor`e & Heydari, 2023). To ensure
high-quality data synthesis, we introduce two additional steps: topic control and human inspection.
We first define a set of topics commonly associated with cooperation and competition, such as busi-
ness and law, to guide the data generation process. Then, to ensure that the synthesized games meet
the required game structures and are easily understandable, we conduct rigorous human inspection
(§2.3). To address the third issue, we propose three forms for expanding and organizing games:
sequential, parallel, and nested. Using the above constructed games as atomic units, we reorganize
them into these complex forms to assess the strategic reasoning of LLMs. The sequential and par-
allel forms evaluate the model’s capacity for sequential and parallel decision-making, respectively,
while the nested form explores the LLMs’ multi-layered strategic reasoning abilities (§2.4).

Based on TMGBENCH, we conduct comprehensive analyses and evaluations of current mainstream
LLMs (§3), including assessments of rational reasoning, reasoning robustness, Theory-of-Mind
(ToM) capabilities, and reasoning in complex game forms, leading to the following key findings:

(1) Advanced LLMs like gpt-4o demonstrate strong strategic reasoning, with over 80% accuracy, but
struggle to generalize across contexts and scenarios. Models like claude-3-5-sonnet further reveal
this inconsistency, with performance variability marked by coefficients of variation nearing 0.5.

(2) Though GPT models often perform well, their reasoning inconsistency on certain task sub-types
is marked by an ‘asymmetric pattern’ as reported, which is the main cause of the statistical biases.

(3) Several top-tier LLMs demonstrate stable first-order ToM abilities, with some effectively uti-
lizing second-order ToM for comparable tasks. In contrast, models such as Llama-3.1-70B appear
restricted to first-order reasoning.

(4) Complex-form games that are derived from atomic units in TMGBENCH present considerable
challenges for LLMs, including those with strong reasoning abilities like o1-mini from OpenAI,
which often struggle as the number of games increases.

2 TMGBENCH

2.1 BENCHMARK OVERVIEW

TMGBENCH is a benchmark designed to evaluate the strategic reasoning capabilities of LLMs in
game-theoretic scenarios, illustrated by Figure 1. It comprehensively covers 144 types of games

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Figure 1: An concept map of TMGBENCH.
The data preparation of the benchmark in-
cludes 3 ingredients: Robinson-Goforth topol-
ogy, game structure and contextual framing.
The evaluation of the benchmark embraces
several prompting methods (including ToM
promptings) to elicit strategic reasoning pro-
cess of LLMs.

Figure 2: We design several complex forms of
strategic reasoning tasks using TMGBENCH.
which include:
(1) sequential form, where
LLMs are required to respond to multiple game
tasks in a row, with history of previous tasks; (2)
parallel form, where LLMs are required to re-
sponse multiple game tasks simultaneously; (3)
nested form, where LLMs are required to re-
sponse a set of interlinked game tasks (in our
settings, we relate to them as pre-game and
core-game). Games in the complex forms
can be selected with different game structures
and various contexts.

(see §2.2), with each type containing multiple instances (in each instance, there are two players and
each player can choose between two strategies, resulting in four possible situations), which can be
categorized into classic and story-based settings. Notably, the story-based instances are produced
using synthetic data generation techniques and are grounded in real-life themes, effectively mitigat-
ing the issue of data leakage (see §2.3). Furthermore, each game in TMGBENCH can be treated as
an atomic unit, and multiple atomic games can be structured in a more complex task with parallel,
sequential, or nested form (see §2.4). These complex scenarios effectively facilitate the evaluation
of advanced LLMs’ abilities in parallel, sequential, and multi-layered decision-making. To precisely
evaluate the reasoning abilities of LLMs, we use their performance in inferring the optimal strategy
combination, i.e., the Nash equilibrium, as the evaluation criterion. Additionally, the designed eval-
uation metrics provide a fine-grained assessment of the robustness and self-consistency of LLMs’
strategic reasoning abilities (see §2.5).

2.2 GAME TOPOLOGY

Although previous research has explored LLMs’ reasoning abilities within the context of game the-
ory, existing studies have primarily focused on a few well-known games, such as the Prisoner’s
Dilemma, Battle of the Sexes, and Stag Hunt (Brookins & DeBacker, 2023; Phelps & Russell, 2023;
Guo, 2023). However, these studies cover a limited game types, resulting in incomplete evaluations.
Thereby, a broader variety of games is urgently needed to conduct a systematic assessment of LLMs.

To address this, we incorporate 144 game types (we later refer to a type as an equivalence class)
based on the Robinson-Goforth topology of 2×2 games (Robinson & Goforth, 2005). Classic games
like the Prisoner’s Dilemma belong to one of the equivalence classes within this topology. Specif-
ically, the topology of 2×2 games elegantly illustrates the relationships among strictly ordinal 2×2
games, each with a unique payoff structure, leading to different dominant strategies, Nash equilibria,
and reasoning approaches (more details in Appendix C.1). We categorize all the 144 games with
numerical payoffs from the original topology into the classic setting tasks. Due to space constraints,
we provide an introduction to the Robinson-Goforth topology in Appendix C.2.

3

Robinson-Goforth TopologyPdB1B2A14313A22124Game Structure(e.g. Prisoner’s Dilemma, PD)Sarahcoffee shop ownerMarkcoffee chain entrepreneurKeep pricesLower pricesMaintain operationsExpand businessContextualFramingDirect AnswerChain of ThoughtFirst-order ToMSecond-order ToMLLM’s answerstandard answerDIFFData PreparationEvaluationIt will be the best if Sarah keeps prices steady, I can maximize profits by maintain operations then ...Keeps prices steady would be better, and Mark must be reluctant to expand his business ...```pythonanswer= …```Python-style required answerBusinessPoliticsTransportationGPT-4oAssistedSequentialParallelNestedpre-gamegame1game2game3game1game2game3NEcore-gameNEif:pre-gamethen:core-gamenew NE of  the pre-game:ConditionGame PairUnder review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 3: Demonstration of the inconsistency
heat map. Each of the grids is divided into 4
quarter-grids, indicating the 4 situations. By
subtracting the standard map from the practi-
cal map element-wise, we get the inconsistency
map, where blue colours indicate positive dif-
ference and red colours indicate negative dif-
ference. The deeper the colour means the larger
the difference between the LLM’s response and
the standard answer.

Figure 4: Axisymmetry in heat maps can be
illustrated by the left sub-figure, where the
standard heat map exhibits perfect axisymme-
try across the counter-diagonal.
In contrast,
LLMs’ responses tend to demonstrate quasi-
axisymmetry, as shown by the right sub-figure.
Certain pairs of positions fail to align precisely
when reflected across the axis and may exhibit
discrepancies, deviating from the ideal sym-
metric pattern.

2.3 CONTEXTUAL FRAMING

Relying on the Robinson-Goforth topology, we can systematically construct all types of classic
setting tasks. However, this alone is insufficient, as games often take place in diverse real-life
contexts, involving different topics, types of participants and their preferences. Such contextual
framing of games introduces new challenges for LLMs (Lor`e & Heydari, 2023).

To further explore LLMs’ strategic reasoning capabilities in real-world scenarios, we use classic
games as seed data and employ synthetic data generation techniques, leveraging GPT-4o to construct
story-based games. Specifically, in story-based games, we replace the pure game information of
classic games with real-life scenarios, covering topics such as business, law and transportation.
Additionally, the two players are substituted with characters representing broader semantics (e.g.,
people, animals, organizations, and even nations), and the payoff values are transformed from pure
numbers into specific states or rewards relevant to the characters. For each classic game, we generate
5 corresponding story-based games.

To ensure high-quality data generation, we undertake the following steps: First, we use GPT-4o
to synthesize the contextual data. Second, we design precise prompts to ensure the generated data
adhere to the given game structures. Third, we select topics from real-life scenarios where strategic
interactions are common, guiding the data generation process. Finally, we conduct rigorous human
reviews to ensure the data’s quality and diversity.

Details on the data generation process, prompts, human review procedures, and topic distribution of
the data can be found in Appendix D.

2.4 COMPLEX FORMS

The 2×2 games in the topology represent a highly condensed game structure. However, in real
life, we often encounter more complex game forms, such as making continuous decisions, making
multiple decisions simultaneously, or considering the impacts of one decision on another.

To evaluate LLMs’ strategic reasoning abilities with more constraints, we treat the aforementioned
individual games as atomic games and expand them in three forms: sequential, parallel, and nested.
The organization of these forms is illustrated in Figure 2. Specifically, in the sequential form, we
randomly sample multiple games from the story-based games, requiring the LLM to make decisions
sequentially. Only if the LLM provides correct answers for all games is it considered to have made
correct decisions. In the parallel form, the LLM is given multiple randomly sampled games and
must make decisions simultaneously. Similarly, the LLM is deemed to have made correct decisions
only if it solves all games correctly. In the nested form, we randomly sample two games, desig-
nated as the pre-game and the core-game, where the core-game holds greater importance.
The decisions made by the LLM in the pre-game affect the strategy space in the core-game.
Thus, the LLM is judged to have made correct decisions only if it demonstrates forward-looking

4

Practical MapStandard MapInconsistency MapvUnder review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

reasoning by choosing a sub-optimal solution in the pre-game to achieve the optimal solution in
the core-game. We demonstrate a template to generate an nested form game in Appendix F.2.

Theoretically, using these atomic games, we can expand the framework to generate infinitely many
increasingly complex game forms, thereby providing a continuous benchmark for evaluating the
performance of more advanced LLMs.

2.5 EVALUATION METRICS

As explained in Section 2.2, our benchmark is perfectly suitable to display in a 12x12 square table,
each grid representing one of the 144 equivalence classes. In the evaluation process we conduct
repetitive tests in every data point of each equivalence class. Each test starts with the input of the
setting (classic/story-based) and the question, and ends with LLM’s response containing a list of
choices corresponding to multiple choices or no choice (when the given list is empty).

Notation. For notation, we assign Freqi,j,o as the frequency of the o-th choice happening to be in
the tests of the grid at i-th row, j-th column, where the 1, 2, 3 and 4-th choice correspond to the
upper-left, upper-right, lower-left and lower-right quarter-grid respectively.

Inconsistency Heat Map. According to conclusions of the Robinson-Goforth topology (Robinson
& Goforth, 2005), we convert the standard answer of each equivalence class into a heat map named
the standard heat map, with the coloured quarter-grid to be the choice in the standard answer. Sim-
ilarly, as for practical result provided by LLMs, we set the value of Freqi,j,o as the colour depth
of each quarter grid, which builds up the practical heat map. Naturally, we subtract the standard
heat map from the practical heat map in an element-wise manner to get the inconsistency heat map,
which is a standardised tool for our evaluation, shown in Figure 3.

Inconsistency Degree. In order to display the quantified performance of LLMs, we extract inconsis-
tency degree from a map, which helps reveal the gap between LLMs’ response and standard answer,
and it is defined as

ID =

1
144

12
(cid:88)

12
(cid:88)

i=1

j=1

1
4

4
(cid:88)

o=1

∆Freq2

i,j,o

where ∆Freqi,j,o indicates the the difference (between the LLM’s answer and the standard answer)
of frequency of the o-th choice at i-th row, j-th column.

Bias Degree. Owing to the symmetric property of the topology framework of 2×2 matrix games, the
distribution of answers over the heat map has axial symmetry by the counter-diagonal (Figure 4).
Motivated by this elegant property, we set up another metric to evaluate the bias degree of LLMs’
answers, which we expect robuster LLMs to display lower degrees of bias. The bias degree reflects
the stability and symmetry of LLMs’ strategy, and it is defined as

BD =

1
144

12
(cid:88)

12
(cid:88)

i=1

j=1

1
4

4
(cid:88)

o=1

(Freqi,j,o − Freqj,i,refo

)2

where the meaning of refo is the index of choice o’s counterpart considering the reflection operation
by the counter-diagonal, and we have the mapping relation: {1, 2, 3, 4} (cid:55)→ {4, 2, 3, 1}. (e.g. ref1 =
4 means that the reflection counterpart of choice 1 is choice 4, vice versa)

Perfect Accuracy Rate. In addition to the metrics mentioned above, we also set up a more rigorous
metric named perfect accuracy rate, which ignores the partially correct answer and only considers
perfectly correct answer in each test, and it is defined as

PAR =

1
144

12
(cid:88)

12
(cid:88)

i=1

j=1

1
T

T
(cid:88)

t=1

I{rspt,i,j = stdi,j}

which means that we count only if the response perfectly matches the standard answer, where T
represents the number of times we invoke a LLM to response on a certain game task.

Metrics with Subscript. As a matter of fact, within the topology, different equivalence classes have
different number of Nash equilibria (ranging from {0, 1, 2}), leading to a discrepancy in reasoning

5

Under review as a conference paper at ICLR 2025

Table 1: Overall statistics of LLMs’ performance on classic setting tasks. The up arrow(↑) means
the larger value indicates better performance, while the down arrow(↓) means the smaller value
indicates better performance. All values are expressed as percentages.

Family

Model

GPT

Claude

Llama

gpt-4o
gpt-4o-mini
gpt-3.5-turbo

claude-3-5-sonnet
claude-3-haiku

Llama-3.1-70B
Llama-3.1-8B

PAR(↑)

Metric / Prompting
ID(↓)

BD(↓)

DA

52.08
14.93
30.21

59.38
24.31

13.02
18.75

CoT

80.38
74.02
34.38

79.69
40.28

54.29
22.63

DA

16.81
27.15
27.64

14.79
39.58

36.15
38.49

CoT

3.78
4.38
17.87

7.13
25.17

15.32
31.19

DA

28.49
48.59
50.15

27.76
72.22

40.71
81.32

CoT

7.79
8.29
30.19

14.34
44.10

26.63
47.64

Qwen

Qwen2-72B

43.06

46.21

26.30

19.94

35.59

29.29

Figure 5: Radar charts of the 9 sub-metrics of 8 LLMs’ performance, comparing the DA prompting
(left side) and the CoT prompting (right side). AntiID and AntiBD are derived from ID and BD
while higher values indicate better performances (in order to consistent with PAR).1

difficulty, therefore we propose metrics with subscript that represents for different types of equiva-
lence groups (we refer them to 0-task, 1-task, 2-task respectively), which we refer to as sub-metrics.
Therefore we have IDn, BDn, PARn(n = 0, 1, 2) which means the inconsistency degree, the bias
degree, and the perfect accuracy rate across all equivalence classes that have n equilibra.

3 ANALYSIS

3.1 OVERVIEW OF LLMS’ PERFORMANCE

Overall, we select several SOTA models according to Open LLM Leaderboard (Fourrier
et al., 2024) and conduct extensive experiments on TMGBENCH. These models include GPT
(gpt-4o-2024-05-13, gpt-4o-mini-2024-07-18, gpt-3.5-turbo-0125), Claude (claude-3-5-
sonnet-20240620, claude-3-haiku-20240307), Llama (Llama-3.1-8B, Llama-3.1-70B), and
Qwen (Qwen2-72B). We perform 4 independent tests on each data point, covering both the clas-
sic setting and the story-based setting (thus we conduct 2,880 tests to generally evaluate a certain
model). During the evaluation, we set the temperature of the tested LLMs to 0 or near 0, ensuring
the lowest degree of uncertainty and enhancing the faithfulness of our evaluation. More details of
the evaluation process are provided in Appendix D.1.

Games in TMGBENCH are not easy for most LLMs. First we overall evaluate how well LLMs
can behave on the classic setting tasks of our benchmark, to assess their basic capability of strategic

1AntiBD = 1 −

√

BD, AntiID = 1 −

√

ID

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

PAR1AntiBD1AntiID2PAR2AntiBD2AntiID0PAR0AntiBD0AntiID10.250.500.75DAPAR1AntiBD1AntiID2PAR2AntiBD2AntiID0PAR0AntiBD0AntiID10.250.500.75CoTgpt-4ogpt-4o-minigpt-3.5-turboclaude-3-5-sonnetclaude-3-haikuLlama-3.1-70BLlama-3.1-8BQwen-2-72BUnder review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

(a) PARn(↑)

(b) IDn(↓)

(c) BDn(↓)

Figure 6: Comparison of LLMs’ performance under the classic setting (indicated by ‘C-’ label, in
opaque colour) and the story-based setting (indicated by ‘S-’ label, in semi-opaque colour with error
bar), where the length of the bars represent the value of metrics, and the error bars represent the
standard deviation over all 5 data points of the story-based setting tasks.

reasoning. We initially adopt two basic prompting methods: Direct Answer (DA) prompting and
Chain-of-Thought (CoT, (Wei et al., 2022)) prompting, which represent shallower, faster thinking
patterns and deeper, slower thinking patterns, respectively.

As seen from Table 1, gpt-4o, gpt-4o-mini and claude-3-5-sonnet are more capable compared to
other models, with a high overall accuracy rate (around 80%) and low inconsistency and low bias
score (around 5%). Specifically, as shown in Figure 5 formed by 9 sub-metrics, gpt-4o performs
the best on 1-tasks, gpt-4o-mini beats others on 2-tasks, and claude-3-5-sonnet are relately better at
0-tasks. Moreover, comparing the performance of employing DA prompting and CoT prompting,
we find that CoT prompting almost provides comprehensive improvement but few exceptions like
the PAR2 of Llama-3.1-70B.

Despite the excellent performance of the top-tier models (gpt-4o and claude-3-5-sonnet), other mod-
els often do not exhibit robust performance across all 3 different types of tasks. The inconsistency
degree and bias degree in these models can be more than double or triple those of the top-performing
models. This indicates that from a systematic point of view, even classic setting tasks from TMG-
BENCH are challenging for most LLMs.

LLMs’ performance is vulnerable across various narratives. At the theoretical level, we consider
classic setting tasks and story-based tasks to be fundamentally the same problems within the domain
of game theory. However, this conclusion appears not transferable to LLMs at the practical level. For
LLMs, the complexity and nuance of story-based tasks introduce unique challenges, where LLMs
are required to be robust in understanding and reasoning concurrently.

In Figure 6, we compare the performance of LLMs using CoT prompting, which is more robust
according to previous analysis. The figure reveals the vulnerable performance of LLMs on tasks in
story-based setting (corresponding to various narratives), marked by two primary characteristics:

(1) The advanced models, specifically gpt-4o, gpt-4o-mini and claude-3-5-sonnet, exhibit signifi-
cant performance degradation. Notably, gpt-4o demonstrates a broad under-performance across the
board, while gpt-4o-mini experiences the most pronounced decline in performance on 2-task sce-
narios, where its S-PAR2 metric falls to less than one-third of its C-PAR2 counterpart. Similarly,
claude-3-5-sonnet shows the largest performance drop in 0-task, with its S-PAR0 metric reduced to
less than one-fourth of C-PAR0, and its S-ID0 metric exceeding four times that of C-ID0.

(2) The performance of certain localities exhibits significant fluctuations. A particularly notable
degradation occurs in the PAR scores for 0-task and 2-task scenarios handled by claude-3-5-sonnet,
where the coefficients of variation cv (defined as cv = σ
µ , with σ representing the standard devi-
ation and µ the mean) approach 0.5. These eminent values of cv suggest a lack of robustness in
performance across different narratives.

7

0.00.20.40.60.81.0Value of PARn()gpt-4ogpt-4o-minigpt-3.5-turboclaude-3-5-sonnetclaude-3-haikuLlama-3.1-70BLlama-3.1-8BQwen2-72BModelsC-PAR0C-PAR1C-PAR2S-PAR0S-PAR1S-PAR20.00.20.40.60.81.0Value of IDn()C-ID0C-ID1C-ID2S-ID0S-ID1S-ID20.00.20.40.60.81.0Value of BDn()C-BD0C-BD1C-BD2S-BD0S-BD1S-BD2Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 2: Performance of LLMs using different ToM compared to CoT. Text in red color indicates the
performance gets better and text in blue color indicates the performance gets worse (both compared
to CoT). Bold text means the best performance across the three prompting methods. Grey areas
mean an LLM is good at using some kind(s) of ToM. All values are expressed as percentages.

Model

Prompting

PAR0(↑)

0-Task
ID0(↓) BD0(↓)

PAR1(↑)

1-Task
ID1(↓) BD1(↓)

PAR2(↑)

2-Task
ID2(↓) BD2(↓)

gpt-4o

gpt-4o-mini

gpt-3.5-turbo

claude-3-5-sonnet

claude-3-haiku

Llama-3.1-70B

Llama-3.1-8B

Qwen2-72B

CoT
FoToM
SoToM

CoT
FoToM
SoToM

CoT
FoToM

CoT
FoToM
SoToM

CoT
FoToM

CoT
FoToM
SoToM

CoT
FoToM

CoT
FoToM

34.72
43.06
31.94

25.00
25.00
18.06

0.00
0.00

86.11
68.06
47.22

0.00
0.00

8.33
2.78
23.61

0.00
0.00

20.83
0.00

13.37
9.46
9.81

15.62
19.53
26.56

19.44
21.44

4.25
7.73
21.35

40.28
33.33

22.47
30.82
21.27

27.34
22.14

29.25
36.46

14.41
9.81
10.68

23.94
19.53
26.22

29.69
22.83

20.23
16.06
28.99

47.22
37.50

26.43
35.59
28.73

46.09
59.20

32.20
35.07

92.36
95.14
91.67

72.45
99.54
98.84

41.67
54.40

88.89
92.13
90.05

49.07
47.22

65.59
49.54
60.42

25.77
27.55

50.78
45.14

1.58
0.72
1.45

5.08
0.03
0.19

17.55
19.30

4.72
2.56
4.05

22.45
22.22

13.43
18.68
14.09

32.90
31.97

19.35
26.92

6.76
4.14
6.00

11.09
5.08
5.38

30.95
42.52

11.68
7.74
14.38

44.91
48.61

27.16
27.49
23.70

47.17
67.18

28.73
49.54

54.17
50.00
52.78

70.83
47.22
68.06

25.00
0.00

18.06
47.22
33.33

27.78
11.11

25.00
69.44
12.50

26.39
15.28

44.44
11.11

7.38
8.94
7.99

7.97
10.59
5.38

18.23
37.85

24.48
15.10
14.93

26.39
43.06

19.53
6.08
24.05

24.74
33.64

14.15
37.50

7.38
8.59
8.16

7.69
10.59
5.38

26.13
59.20

24.48
15.10
14.93

36.11
56.94

23.70
22.74
25.26

52.00
65.49

29.77
49.13

3.2 FINDINGS OF LLMS’ BEHAVIOURS

LLMs demonstrate first/second-order ToM abilities.
In tasks across all equivalence classes,
1-tasks have the lowest reasoning difficulty because at least one player has a dominant strategy,
which means the player can make an unconditionally optimal decision regardless of the counter-
part’s choice. In such cases, once a player (denoted as A) can make this unconditionally optimal
decision, their counterpart (B) can, using first-order Theory-of-Mind (ToM), easily determine the
best response for themselves (B).

This insight motivated us to apply FoToM prompting to LLMs, representing the First-order Theory-
of-Mind thinking, to aid in solving these tasks. As seen in Table 2, top-tier models like gpt-4o show
improvement in both 0-tasks and 1-tasks when utilizing FoToM. Model claude-3-5-sonnet improves
on 1-tasks and 2-tasks, and gpt-4o-mini displays a significant surge in performance on 1-tasks and so
does Llama-3.1-70B on 2-tasks. However, for models like Llama-3.1-8B and Qwen2-72B, FoToM
does not seem to provide any prominent advantage and may even result in worse performance.
Notably, no LLM achieves overall improvement across all task categories by merely using first-
order ToM, and 0-tasks appear to be the most challenging for LLMs to solve.

Furthermore, we wondered if LLMs display some ability to use first-order ToM could also be capable
of second-order ToM. According to Liddle & Nettle (2006), higher-order ToMs are generally more
difficult to master than first-order ToM. Thus we selected only advanced models that demonstrated
proficiency in first-order ToM to attempt solving specific tasks using Second-order Theory-of-Mind
(SoToM) prompting. As seen in Table 2, models like gpt-4o, gpt-4o-mini and claude-3-5-sonnet
show consistent performance when applying second-order ToM to tasks they are already capable of
solving better with first-order ToM. However, the improvements from using SoToM generally do not
exceed those achieved with first-order ToM. In addition, Llama-3.1-70B’s underperformance with
SoToM suggests that possessing first-order ToM capabilities does not necessarily imply proficiency
with second-order ToM. The prompts used for FoToM and SoToM are provided in Appendix D.2.

Certain behavioural pattern contributes to poor performance. Based on the analysis from the
previous sections, it is encouraging to note that top-tier LLMs demonstrate high accuracy and low
inconsistency when solving 1-task scenarios, regardless of the prompting used (CoT, FoToM, or
SoToM). However, their performance declines significantly when addressing other types of tasks.

8

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Figure 7: Inconsistency heat map of GPT se-
ries models using different prompting methods.
The yellow boxes and green boxes represent
the 0-task areas in the topological framework.

Figure 8: Top LLMs’ performance on the
games in complex forms of three types. Ow-
ing to the expensive inference cost, we run 20
times for each configuration.

For the advanced GPT series models, it is particularly noteworthy that they perform the worst on 0-
tasks out of all types. Apart from the low PAR and high ID on 0-tasks compared to 1-tasks, the bias
degree also doubles (for gpt-4o) or even several times higher (for gpt-4o-mini). Surprisingly, as il-
lustrated in Figure 7, these models display a similar answering pattern that appears non-coincidental.
Within the topological framework, there are two square areas representing 0-tasks (enclosed in yel-
low boxes and green boxes), which should theoretically be symmetric across the counter-diagonal.
The standard heat map of these two areas is entirely blank, reflecting no existing equilibrium, so the
two areas of the inconsistency heat maps just reflect the distribution of LLMs’ practical responses.

Under closer inspection, it becomes evident that the models exhibit a consistent pattern when ad-
dressing 0-tasks. In yellow-box areas, their answers tend to emphasize the upper-right and lower-left
quarter-grids, whereas in green-box areas, their answers tend to emphasize the upper-left and lower-
right quarter-grids. This pattern appears to be the primary cause of the high bias degree. However,
it introduces a strong asymmetry along the counter-
the phenomenon is quite counter-intuitive:
diagonal. In other words, simply swapping the id of two players and their actions, which does not
alter the fundamental game structure, leads the LLMs to identify different Nash equilibria. Never-
theless, it is quite strange for them to provide such uniform “wrong answers” within each box, while
the answers across the two boxes are entirely asymmetric.

To testify that this is not due to the position bias in the prompts (refer to the FoToM prompting and
SoToM prompting in Appendix D.2), we design the reFoToM prompting and the reSoToM prompting
(refer to the reFoToM prompting and reSoToM prompting in Appendix D.2) which swap the order of
the players happens in the FoToM prompting and the SoToM prompting respectively. The results in
Appendix E.1 imply that such ‘asymmetric inconsistency pattern’ is not strong related to the orders
in the prompt. We demonstrate two typical examples of this phenomenon in Appendix E.2.

Complex forms bring more challenging tasks. To verify that TMGBENCH can be extended to
harder tasks which may better align with complicated scenarios from the reality, we run the test
on the three complex forms we mention in Section 2.4, to assess the performance of two strongest
LLMs (o1-mini and gpt-4o) in complex strategic reasoning.

We setup the test by dividing it into several types: (1) in sequential form and parallel form, we set
the variable of number of the games from the set {3, 5, 10}; (2) in nested form, we just use some
2-folds nested games (due to the high verification cost when the number increases).

As seen from Figure 8, the top-tier model gpt-4o has a dramatically low accuracy rate in either
sequential or parallel games, even the strongest reasoning model o1-mini still failed at times; when
the number of the games increase, their performances both drop, which is consistent with intuition.
As for the games of nested form, two models’ performances are relatively reasonable, while it is fair
to infer that if we increase the number of layers of the games that in the nested structures, it will
present a great challenge for LLMs. The overall accuracy rates of o1-mini over the three forms are
66.6%, 60.0% and 70.0% respectively, while gpt-4o performs worse, with accuracy rates reaching
only 50.0%, 35.0% and 70.0% respectively.

9

CoTFoToMSoToMgpt-4o-minigpt-4osequentialparallelnestedForm048121620Accuracy Countgpt-4o, 3-lengtho1-mini, 3-lengthgpt-4o, 5-lengtho1-mini, 5-lengthgpt-4o, 10-lengtho1-mini, 10-lengthgpt-4o, 2-folds nestedo1-mini, 2-folds nestedUnder review as a conference paper at ICLR 2025

4 RELATED WORK

Strategical Reasoning of LLMs. Large language models have made notable breakthroughs in rea-
soning tasks, such as mathematical, causal, and commonsense reasoning, enabling their increasing
use in complex tasks that support human decision-making (Imani et al., 2023; Kıcıman et al., 2023;
Zhao et al., 2024). This progress has sparked a growing interest in studying their strategic reasoning
capabilities (Zhang et al., 2024a). Game theory, with its highly abstract representation of real-world
strategic scenarios, has garnered significant attention from researchers (Duan et al., 2024; Huang
et al., 2024). The prisoner’s dilemma, as one of the most classical games, has been widely used to
evaluate the strategic reasoning abilities of LLMs (Brookins & DeBacker, 2023; Guo, 2023; Akata
et al., 2023; Phelps & Russell, 2023; Xu et al., 2023). In addition, several well-known game theory
scenarios, such as the Dictator Game (Horton, 2023; Fan et al., 2023; Brookins & DeBacker, 2023),
the Ultimatum Game (Aher et al., 2022), the Public Goods Game (Li et al., 2023) and the Battle
of the Sexes (Akata et al., 2023), have been employed to evaluate LLMs’ capabilities. However,
current studies often focus on individual games, resulting in incomplete assessments and less ro-
bust conclusions. To address this, we propose TMGBENCH, a benchmark for evaluating LLMs by
2×2 games, where its atomic games can be further organized using sequential, parallel, and nested
formats to provide an in-depth evaluation of the SOTA models gpt-4o and o1-mini.

Theory-of-Mind of LLMs. Theory-of-Mind (ToM) refers to the ability to understand and infer
human mental states (Premack & Woodruff, 1978). Due to the multi-player nature of game theory,
players’ ability to reason about the “minds” of other participants is crucial. Existing research has
initiated discussions on whether machines possess ToM capabilities. For instance, Kosinski (2023)
suggested that ToM might emerge spontaneously in LLMs, as demonstrated through assessments
using false-belief tasks. However, (Ullman, 2023) argued that such successes are fragile, easily
disrupted by minor perturbations that would not affect an entity genuinely possessing ToM. Never-
theless, many researchers propose enhancing LLMs’ strategic reasoning abilities by incorporating
ToM. Guo et al. (2023) designed the Suspicion-Agent, which integrates a ToM-aware planning ap-
proach that leverages higher-order ToM capabilities, considering not only what the opponent might
do (first-order ToM) but also what the opponent believes the Suspicion-Agent will do (second-order
ToM). Additionally, Yim et al. (2024) introduced a ToM planning method in the Guandan poker
game, Liu et al. (2024) proposed an intention-guided mechanism, Xu et al. (2023) developed Prob-
abilistic Graphical Modeling, and Zhang et al. (2024b) introduced K-Level-Reasoning, all utilizing
ToM to enhance LLMs’ strategic reasoning. Given the broad application of ToM, this paper lever-
ages TMGBENCH to comprehensively evaluate LLMs’ ability to employ first-order and second-
order ToM reasoning techniques for strategic reasoning.

5 DISCUSSION

Limitations. Our TMGBENCH focuses on a very specific area within the vast domain of game the-
ory, highlighting the fact that there is still a significant portion of game theory that lacks systematic
exploration. While it may be infeasible or even impossible to develop a framework that is suitable
for all types of games, we hope that benchmarks like TMGBENCH can help identify issues such as
inherent imbalances and the non-robustness of LLMs’ strategic reasoning abilities.

Conclusion. In this work, we introduce TMGBENCH, a benchmark for systematically evaluating
the strategic reasoning abilities of LLMs by 2x2 matrix games. Based on Robinson-Goforth topol-
ogy, we develop the classic setting tasks, and introduce various narratives based on story contexts
generated by GPT-4o. By utilizing TMGBENCH, we can identify current flaws in LLMs’ perfor-
mance on these tasks, such as low accuracy rates and unstable inconsistency and bias degrees, even
though the task difficulty is relatively moderate compared to many others. Additionally, when em-
ploying prompts to elicit their Theory-of-Mind thinkings on these tasks, some LLMs show improved
performance, indicating that LLMs can, to some extent, master ToM and apply it in their reason-
ing processes. However, possessing first-order ToM abilities does not necessarily mean that LLMs
will excel at mastering higher-order ToM. Furthermore, based on TMGBENCH, we introduce more
forms of complex strategic reasoning tasks and pose a new challenge for LLMs.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Gati Aher, RosaI. Arriaga, and Adam Tauman Kalai. Using large language models to simulate multi-
ple humans and replicate human subject studies. In International Conference on Machine Learn-
ing, 2022. URL https://api.semanticscholar.org/CorpusID:251719353.

Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz.
Playing repeated games with large language models. ArXiv preprint, abs/2305.16867, 2023. URL
https://arxiv.org/abs/2305.16867.

Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about
In The Thirty-Fourth AAAI Conference on Artifi-
physical commonsense in natural language.
cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence
Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelli-
gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432–7439. AAAI Press, 2020.
URL https://aaai.org/ojs/index.php/AAAI/article/view/6239.

Philip Brookins and Jason Matthew DeBacker. Playing games with gpt: What can we learn about a

large language model from canonical strategic games? Available at SSRN 4493398, 2023.

Hans Carlsson and Eric Van Damme. 12 equilibrium selection in stag hunt games. Frontiers of game

theory, pp. 237, 1993.

Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-
Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. Gtbench: Uncovering the strategic reasoning
limitations of llms via game-theoretic evaluations. ArXiv preprint, abs/2402.12348, 2024. URL
https://arxiv.org/abs/2402.12348.

Benjamin Edelman, Michael Ostrovsky, and Michael Schwarz. Internet advertising and the gener-
alized second-price auction: Selling billions of dollars worth of keywords. American economic
review, 97(1):242–259, 2007.

Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. Can large language models serve as rational
players in game theory? a systematic analysis. ArXiv preprint, abs/2312.05488, 2023. URL
https://arxiv.org/abs/2312.05488.

Robert Forsythe, Joel L Horowitz, Nathan E Savin, and Martin Sefton. Fairness in simple bargaining

experiments. Games and Economic behavior, 6(3):347–369, 1994.

Cl´ementine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open
llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/
open_llm_leaderboard, 2024.

Kanishk Gandhi, Dorsa Sadigh, and Noah Goodman. Strategic reasoning with language models. In

NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.

Herbert Gintis. The bounds of reason: Game theory and the unification of the behavioral sciences-

revised edition. Princeton university press, 2014.

Fulin Guo. Gpt in game theory experiments. ArXiv preprint, abs/2305.05516, 2023. URL https:

//arxiv.org/abs/2305.05516.

Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. Suspicion-
agent: Playing imperfect information games with theory of mind aware gpt-4. ArXiv preprint,
abs/2309.17277, 2023. URL https://arxiv.org/abs/2309.17277.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song,
and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv
preprint, abs/2103.03874, 2021. URL https://arxiv.org/abs/2103.03874.

Nathan Herr, Fernando Acero, Roberta Raileanu, Mar´ıa P´erez-Ortiz, and Zhibin Li. Are large lan-
guage models strategic decision makers? a study of performance and bias in two-player non-
zero-sum games. ArXiv preprint, abs/2407.04467, 2024. URL https://arxiv.org/abs/
2407.04467.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

John J Horton. Large language models as simulated economic agents: What can we learn from

homo silicus? Technical report, National Bureau of Economic Research, 2023.

Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenx-
iang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. How far are we on the decision-
making of llms? evaluating llms’ gaming ability in multi-agent environments. ArXiv preprint,
abs/2403.11807, 2024. URL https://arxiv.org/abs/2403.11807.

Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey.
ArXiv preprint, abs/2212.10403, 2022. URL https://arxiv.org/abs/2212.10403.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng,
Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied rea-
soning through planning with language models. ArXiv preprint, abs/2207.05608, 2022. URL
https://arxiv.org/abs/2207.05608.

Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large
language models. ArXiv preprint, abs/2303.05398, 2023. URL https://arxiv.org/abs/
2303.05398.

Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language
models: Opening a new frontier for causality. ArXiv preprint, abs/2305.00050, 2023. URL
https://arxiv.org/abs/2305.00050.

Michal Kosinski. Theory of mind might have spontaneously emerged in large language models.
ArXiv preprint, abs/2302.02083, 2023. URL https://arxiv.org/abs/2302.02083.

David M Kreps. Game theory and economic modelling. Oxford University Press, 1990.

Bin Lei, Chunhua Liao, Caiwen Ding, et al. Boosting logical reasoning in large language models
through a new framework: The graph of thought. ArXiv preprint, abs/2308.08614, 2023. URL
https://arxiv.org/abs/2308.08614.

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra-
masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative
reasoning problems with language models. Advances in Neural Information Processing Systems,
35:3843–3857, 2022.

Jiatong Li, Rui Li, and Qi Liu. Beyond static datasets: A deep interaction approach to llm evaluation.
ArXiv preprint, abs/2309.04369, 2023. URL https://arxiv.org/abs/2309.04369.

Bethany Liddle and Daniel Nettle. Higher-order theory of mind and social competence in school-age

children. Journal of Cultural and Evolutionary Psychology, 4(3-4):231–244, 2006.

Ziyi Liu, Abhishek Anand, Pei Zhou, Jen-tse Huang, and Jieyu Zhao.

Interintent: Investigating
social intelligence of llms via intention understanding in an interactive game context. ArXiv
preprint, abs/2406.12203, 2024. URL https://arxiv.org/abs/2406.12203.

Nunzio Lor`e and Babak Heydari. Strategic behavior of large language models: Game structure
vs. contextual framing. ArXiv preprint, abs/2309.05898, 2023. URL https://arxiv.org/
abs/2309.05898.

Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz,
Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via
large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1–40, 2023.

Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and
Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models.
ArXiv preprint, abs/2303.09014, 2023. URL https://arxiv.org/abs/2303.09014.

Steve Phelps and Yvan I. Russell. The machine psychology of cooperation: Can gpt mod-
els operationalise prompts for altruism, cooperation, competitiveness and selfishness in eco-
ArXiv preprint, 2023. URL https://api.semanticscholar.org/
nomic games?
CorpusID:258685424.

12

Under review as a conference paper at ICLR 2025

David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and

brain sciences, 1(4):515–526, 1978.

David Robinson and David Goforth. The topology of the 2x2 games: a new periodic table, volume 3.

Psychology Press, 2005.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com-
In Proceedings of the 2019 Conference on
monsense reasoning about social interactions.
Empirical Methods in Natural Language Processing and the 9th International Joint Confer-
ence on Natural Language Processing (EMNLP-IJCNLP), pp. 4463–4473, Hong Kong, China,
2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1454. URL https:
//aclanthology.org/D19-1454.

George Tsebelis. Nested Games: Rational Choice in Comparative Politics. University of California

Press, 1990.

Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. ArXiv

preprint, abs/2302.08399, 2023. URL https://arxiv.org/abs/2302.08399.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny
Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in
neural information processing systems, 35:24824–24837, 2022.

Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See-Kiong Ng, and
Jiashi Feng. Magic: Investigation of large language model powered multi-agent in cognition,
adaptability, rationality and collaboration. In ICLR 2024 Workshop on Large Language Model
(LLM) Agents, 2023.

Yauwai Yim, Chunkit Chan, Tianyu Shi, Zheye Deng, Wei Fan, Tianshi Zheng, and Yangqiu Song.
Evaluating and enhancing llms agent based on theory of mind in guandan: A multi-player co-
operative game under imperfect information. ArXiv preprint, abs/2408.02559, 2024. URL
https://arxiv.org/abs/2408.02559.

Sarah J Zhang, Samuel Florin, Ariel N Lee, Eamon Niknafs, Andrei Marginean, Annie Wang,
Keith Tyser, Zad Chin, Yann Hicke, Nikhil Singh, et al. Exploring the mit mathematics and
eecs curriculum using large language models. ArXiv preprint, abs/2306.08997, 2023. URL
https://arxiv.org/abs/2306.08997.

Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu,
Ting Song, Man Lan, and Furu Wei. Llm as a mastermind: A survey of strategic reasoning with
large language models. ArXiv preprint, abs/2404.01230, 2024a. URL https://arxiv.org/
abs/2404.01230.

Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Man Lan, and Furu Wei. K-level
reasoning with large language models. ArXiv preprint, abs/2402.01521, 2024b. URL https:
//arxiv.org/abs/2402.01521.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min,
Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. ArXiv
preprint, abs/2303.18223, 2023. URL https://arxiv.org/abs/2303.18223.

Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for

large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

A BRIEF INTRODUCTION TO STRATEGIC REASONING

Definition. Strategic reasoning (Gandhi et al., 2023; Zhang et al., 2024a) is a unique and sophisti-
cated form of reasoning that focuses on making optimal decisions in multi-agent environments. It
involves carefully selecting strategies by anticipating the actions of others and understanding how
one’s choices will influence their responses.

Distinction. What sets strategic reasoning apart is its dynamic nature and the inherent uncertainty
of adversarial actions. Unlike other reasoning paradigms (commen sense reasoning, symbolic rea-
soning, casual reasoning, etc.), it demands a deep comprehension of ever-changing contexts and the
ability to make rational, forward-thinking decisions based on the anticipated behaviors of others.

Example.
In online advertising auctions (Edelman et al., 2007), advertisers compete for adver-
tisement placements by bidding on specific audiences or keywords. Success depends on strategic
reasoning, such as allocating budgets effectively, predicting competitors’ bids, and targeting audi-
ences where competition is lower. Advertisers must also optimize their advertisement quality to
reduce costs while maintaining visibility. Since auctions are dynamic and often follow a second-
price model (where the winner pays just above the second-highest bid), advertisers continuously
adjust their strategies to balance cost and competitiveness. This interplay of decisions makes ad-
vertising auctions a prime example of strategic reasoning in real-world applications. Considering
scenarios where strategic reasoning can be applied with LLMs, fields such as societal simulation,
economic simulation, game theory, and gaming (Zhang et al., 2024a) are prominent areas that often
require this capability.

Significance. Strategic reasoning is a cornerstone for enabling intelligent systems to operate effec-
tively in complex, multi-agent environments. In the context of LLMs, equipping them with strategic
reasoning capabilities extends their potential beyond static information retrieval or pattern recogni-
tion tasks. It allows LLMs to simulate realistic decision-making processes, navigate dynamic social
or economic systems, and collaborate or compete with other agents. This is particularly crucial
in applications such as policy design, automated negotiations, and multi-agent simulations, where
understanding and anticipating others’ behavior is essential for success. By fostering LLMs with
strategic reasoning, we are able to bridge the gap between artificial intelligence and human-like
adaptive decision-making, paving the way for more socially aware, context-sensitive, and intelligent
systems that can tackle real-world challenges with greater precision and impact.

B BASIC THINGS ABOUT GAME THEORY

In this section, we discuss two fundamental concepts in game theory: dominant strategy and Nash
equilibrium.

A dominant strategy is one that always provides a player with a payoff at least as high as any other
strategy, regardless of the actions of other players. In other words, if a player has a dominant strategy,
they will consistently choose it, as it either maximizes their payoff or does not reduce it, irrespective
of the strategies chosen by others.

Nash equilibrium refers to a set of strategies, one for each player, where no player can benefit
by unilaterally changing their strategy. At a Nash equilibrium, each player’s strategy is the best
response to the strategies of the other players. This means that if all players are following their Nash
equilibrium strategies, no one has an incentive to deviate from their current strategy. It represents a
stable state in the game where players’ strategies are mutually optimal.

In many games, the dominant strategy equilibrium and Nash equilibrium may coincide, but not
always. A dominant strategy equilibrium is a specific type of Nash equilibrium where each player
has a strategy that is optimal regardless of others’ strategies. However, in many cases, dominant
strategies may not exist, requiring Nash equilibria to be identified through analysis and computation.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

(a) Most Famous Games

(b) Details in a Grid

Figure 9: The topology of the normal-form game system, which is presented by a square consisting
of 12×12 grids. Figure 9a displays the position of the most famous games in the topology. In each
grid, there are specific details of the game, which is shown in Figure 9b.

C 2×2 MATRIX GAME

C.1 DEFINITION

A normal-form game, commonly referred to as a 2×2 matrix game when involving two players each
with two strategies, is a fundamental concept in game theory for representing strategic interactions.
In this form, the game is depicted as a matrix, clearly outlining the players’ strategies and corre-
sponding payoffs. A typical 2×2 matrix game is structured as shown in Table 3.

Table 3: The form of typical 2×2 matrix games.

Player A: Strategy 1
Player A: Strategy 2

Player B: Strategy 1
(a, w)
(c, y)

Player B: Strategy 2
(b, x)
(d, z)

In this matrix, each cell represents the payoffs for both player A and player B, based on their chosen
strategies. For instance, if player A selects strategy 1 and player B selects strategy 2, player A
receives a payoff of a, while player B receives a payoff of w.

C.2 TOPOLOGY

Game theory research often concentrates on the Prisoner’s Dilemma and a few other symmetric
games, even though most potential games are asymmetric, and many ordinal games involve ties.

The findings on the topology of ordinal normal-form games (Robinson & Goforth, 2005) provide an
elegant framework for systematically studying these games, encompassing all equivalence classes in
an ordinal sense (where “ordinal” refers to the ranking of payoffs rather than their specific values).

In this topological framework, as depicted in Figure 9, well-known games such as the Prisoner’s
Dilemma, Stag Hunt, Battle of the Sexes, and Chicken are all symmetric and situated on the counter-
diagonal of a 12×12 grid. The remaining games are located in the other grids, each with a corre-
sponding “sister game” that can be derived by reflecting across the counter-diagonal. A pair of sister
games are identical when the roles of the two players are reversed.

Within each grid, basic information about the games in the equivalence classes is provided, including
the family name and abbreviation, the payoff matrix, and the order graph, which illustrates the
incentives for the row/column player to unilaterally change their choice for a higher payoff.

15

ChickenChBattle of the SexesBoSHero(BoSsw)HrDelight-PureDpDelight-MixedDmPrisoner’sPdDilemmaSStag HuntAneAssuranceneAswAssuranceswHmHarmony-MixedHpHarmony-PureNcNo ConflictPdColumn payoffs43Row payoffs132124Layer id: 1Prisoner’s DilemmaRow id: 1Column id: 1(2, 2)(1, 4)(4, 1)(3, 3)Under review as a conference paper at ICLR 2025

These 144 equivalence classes include 18 games with no equilibrium, 18 games with exactly two
equilibria, and 108 games with a single equilibrium. Their distribution within the topology is sym-
metric across the counter-diagonal.

Figure 10: The distribution of games with 0, 1, or 2 Nash equilibria (a) is depicted according to the
topology. Grids in grey indicate games with only 1 Nash equilibrium, while white grids represent
games with no Nash equilibrium. Grids in other colours represent games with exactly 2 Nash equi-
libria. Text in blue/red indicates that the column/row player has a dominant strategy in the game,
while white text signifies that both players have dominant strategies. In contrast, black text indicates
that neither player has a dominant strategy.

C.3 SOLUTION STRUCTURE

As previously mentioned, all games in the topological framework can be categorized into three
distinct groups based on the number of Nash equilibria. If we consider Nash equilibrium as the
solution to finding stable strategy combinations, Figure 10 illustrates the structure of these solutions.

In games with exactly one Nash equilibrium, at least one player (either the column player, row
player, or both) has a dominant strategy, meaning they do not need to consider the other player’s
choice. These games are represented by grey or black grids.

Conversely, games with either 0 or 2 Nash equilibria share the characteristic that neither player has
an unconditionally optimal choice, meaning no dominant strategies exist. However, in games with
no Nash equilibrium (white grids), at least one player always has an incentive to unilaterally change
their choice, regardless of the situation. In contrast, games with two Nash equilibria (orange, blue,
or green grids) feature two stable strategy combinations.

Additionally, from a symmetry perspective, two sister games that are symmetric across the counter-
diagonal belong to the same category and have identical Nash equilibria.

D MORE INFORMATION ABOUT OUR TMGBENCH

D.1 GENERATION PIPELINE

In our study, we design an efficient dataset generation pipeline that leverages GPT-4o as the core
to produce the entire dataset, with rigorous human quality reviews incorporated. The pipeline is
organized into three carefully designed stages:

Classic Game Construction. Based on the topology of 2×2 games, we first introduce game de-
scriptions for the payoff matrices of 144 game types, resulting in 144 classic games. An example of
a classic game is shown below, which mirrors the structure of the Prisoner’s Dilemma. These 144
classic games will serve as seed games, with their inherent game structures generalized into more
diverse, story-based games.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

1 NE0 NE1 NE2 NEs1 NE1 NE1 NE1 NE1 NE2 NEs1 NE0 NE1 NE1 NE1 NE1 NEUnder review as a conference paper at ICLR 2025

Example of classic game: classic/111

[Scenario]
Player A and Player B are playing a game. Either of them has two choices, namely A1,
A2/B1, B2. The payoff matrix of their different choice combinations is given below (larger
number means higher payoff):

| B2

| A \ B | B1
|
|-------|-------|-------|
| 1 \ 4 | 3 \ 3 |
| A1
| 2 \ 2 | 4 \ 1 |
| A2

Both Player A and Player B are targeting maximizing their own payoff.
[/Scenario]

Story-based Game Generation. The aforementioned classic games offer a highly condensed math-
ematical representation of diverse game scenarios. However, in the real world, games often occur in
complex social contexts involving various themes. To capture this complexity, we further designed
story-based games, incorporating richer entities and more intricate game scenarios.

Specifically, we used synthetic data generation techniques and crafted detailed prompts to set the
construction constraints for generating high-quality story-based games. Additionally, to enhance
the realism of our game scenarios, we manually defined several thematic categories to guide the
data synthesis process (see §D.3). Both the prompt constraints and thematic categories ensure
the generated content aligns with the intended structure and thematic elements. An example of a
generated story-based game is shown below, which follows the same game structure as the Pris-
oner’s Dilemma and is presented within a new narrative context. As such, the story-based game
story-based/111 0 serves as a counterpart to the classic game classic/111. For each
classic game, we generate five corresponding story-based games. The data synthesis prompt is as
follows. The red text are the placeholders for the variables of the generation code, where ”domain”
indicates the topic we random-choose for the task, and ”matrix str” indicates the payoff matrix de-
rived from the game structure we enumerate.

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

Story-based Game Generation Prompt

Please generate a game theory short story with the following requirements:
- Specific topic: {domain}
- There are two characters who may be in a situation of ”cooperation” or ”competition”;
- Each character has 2 choices, and the combinations of their choices form 4 different sce-
narios;
- In these 4 scenarios, the two characters face different benefits/losses, which can be ab-
stracted as different rewards they can obtain or different states they can achieve in each
scenario;
- They each have a preference relationship for these rewards/states. We use numbers to
represent the degree of preference, with 4 representing the most preferred and 1 the least
preferred (i.e., preference degree 4>3>2>1);
- The payoff matrices for both characters can be abstracted and represented in one matrix,
where A and B represent two characters and their choices are A1, A2/B1, B2. The respon-
dent matrix is shown as below (the story you generate should have the same payoff structure
as it):
{matrix str}
Now please design a story that includes:
- Characters
- Each character’s choices
- Characters’ preferences for different scenarios
- Story description
Response format:
[Characters]
{

“A”: “...”, “B”: “...”

}
[/Characters]
[Choices]
{

“A1”: “...”, “A2”: “...”, “B1”: “...”, “B2”: “...”

}
[/Choices]
[Preferences]
Characters’ preferences for different scenarios (4 most preferred, 1 least preferred):
{

“A”: {

4: “...”, 3: “...”, 2: “...”, 1: “...”

},
“B”: {

4: “...”, 3: “...”, 2: “...”, 1: “...”

}

}
[/Preferences]
[Payoff Matrix]
...
[/Payoff Matrix]
[Scenario]
...(to detailedly describe the situation, including the information of characters, choices, pref-
erences and payoffs on different choice combinations)
[/Scenario]

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

Example of story-based task: story-based/111 0.txt

[Characters]
{

“A”: “Sarah, a small coffee shop owner”,
“B”: “Mark, owner of a large coffee chain”

}
[/Characters]
[Choices]
{

“A1”: “Keep prices steady”,
“A2”: “Lower prices”,
“B1”: “Maintain current operations”,
“B2”: “Expand aggressively in Sarah’s area”

}
[/Choices]
[Preferences]
Characters’ preferences for different scenarios (4 most preferred, 1 least preferred):
{

“A”: {

4: “Lower prices while Mark expands (survive and gain market share)”,
3: “Keep prices steady while Mark maintains operations (stable coexistence)”,
2: “Lower prices while Mark maintains operations (unnecessary price war)”,
1: “Keep prices steady while Mark expands (lose customers)”

},
“B”: {

4: “Maintain operations while Sarah keeps prices steady (maximize profits)”,
3: “Expand while Sarah keeps prices steady (gain market share)”,
2: “Maintain operations while Sarah lowers prices (status quo)”,
1: “Expand while Sarah lowers prices (costly competition)”

}

}
[/Preferences]
[Payoff Matrix]

| B2

| A \ B | B1
|
|-------|-------|-------|
| 1 \ 4 | 3 \ 3 |
| A1
| 2 \ 2 | 4 \ 1 |
| A2

[/Payoff Matrix]
[Scenario]
In a bustling city, Sarah runs a cozy, independent coffee shop that has been a local favorite
for years.
Recently, Mark’s large coffee chain has been eyeing expansion into Sarah’s neighborhood,
threatening her business. Sarah faces a critical decision: she can either keep her prices steady
(A1) or lower them (A2) to attract more customers. Meanwhile, Mark must decide whether
to maintain his current operations (B1) or aggressively expand into Sarah’s area (B2). If
Sarah keeps her prices steady and Mark maintains his current operations (A1, B1), Sarah
struggles to compete (1) while Mark enjoys maximum profits (4). If Sarah lowers her prices
and Mark stays put (A2, B1), both experience moderate success (2, 2) as Sarah attracts some
new customers without directly competing with Mark. Should Mark decide to expand while
Sarah keeps prices steady (A1, B2), both would face challenges but could coexist (3, 3) as
Sarah retains loyal customers and Mark gains new ones. However, if Sarah lowers her prices
as Mark expands (A2, B2), Sarah might survive and even gain market share (4), but Mark
would face costly competition (1).
Their decisions will shape the local coffee market and determine the fate of Sarah’s beloved
shop.
[/Scenario]

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

Quality Verification. To ensure coherence and internal consistency in the generated games, we
implement a multi-step generation strategy, incorporating meticulous human review. First, GPT-4o
generates an initial draft of the story, which is then reviewed by a human for any inconsistencies or
logical flaws. If the draft fails this review, GPT-4o is prompted to identify the problematic sections
and apply a self-correction mechanism.

During the self-correction phase, GPT-4o analyzes the story for inconsistencies and revises the
flawed sections. The revised version undergoes another round of human review. This iterative
refinement process continues until the story meets the required quality standards.

If, after several rounds of regeneration, the story still contains significant issues or fails to meet the
criteria, we may reject the output entirely. In such cases, the process is restarted from scratch with a
new draft to ensure a fresh approach and to avoid perpetuating prior errors.

D.2 REASONING PROMPT USED

In this section, we present the prompts used by various reasoning methods. Notably, when invoking
o1-mini to give response, we only use DA prompting, since the model are reported to perform
reasoning internally and user should avoid‘ prompting like chain-of-thought.

DA prompting

System: You are a spectator, and you should answer question based on given senario.
User:
{task description} {task question}
Only give a block of python-style code containing your answer without any process. e.g.
“‘python
answer = [(“Ax”, “By”)] # list-type
”’

CoT prompting

System: You are a spectator, and you should answer question based on given senario.
User:
{task description} {task question}
Think step by step, and finally give a block of python-style code containing your answer.
e.g.
“‘python
answer = [(“Ax”, “By”)] # list-type
”’

FoToM prompting

System: You are a spectator, and you should answer question based on given senario.
User:
{task description} {task question}
From A’s perspective, try to figure out B’s action and make choice. Then from B’s perspec-
tive try to figure out A’s action and make choice. Finally as a spectator, give a block of
python-style code containing your answer. e.g.
“‘python
answer = [(“Ax”, “By”)] # list-type
”’

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

SoToM prompting

System: You are a spectator, and you should answer question based on given senario.
User:
{task description} {task question}
From A’s perspective, try to figure out B’s action, note that he may also reason based on
your information or reasoning. Then from B’s perspective try to figure out A’s action, note
that he may also reason based on your information or reasoning. Finally as a spectator, give
a block of python-style code containing your answer. e.g.
“‘python
answer = [(“Ax”, “By”)] # list-type
”’

reFoToM prompting

System: You are a spectator, and you should answer question based on given senario.
User:
{task description} {task question}
From B’s perspective, try to figure out A’s action and make choice. Then from A’s perspec-
tive try to figure out B’s action and make choice. Finally as a spectator, give a block of
python-style code containing your answer. e.g.
“‘python
answer = [(“Ax”, “By”)] # list-type
”’

reSoToM prompting

System: You are a spectator, and you should answer question based on given senario.
User:
{task description} {task question}
From B’s perspective, try to figure out A’s action, note that he may also reason based on
your information or reasoning. Then from A’s perspective try to figure out B’s action, note
that he may also reason based on your information or reasoning. Finally as a spectator, give
a block of python-style code containing your answer. e.g.
“‘python
answer = [(“Ax”, “By”)] # list-type
”’

D.3 BENCHMARK DIVERSITY

Our dataset is characterized by the diverse contexts encapsulated within the story-based tasks, a
diversity that manifests across several dimensions.

Firstly, we have identified 20 distinct topics derived from everyday life scenarios where coopera-
tion and competition are likely to occur. These topics align with situations commonly depicted in
various game families. The distribution of story-based games across these 20 topics is visualized in
Figure 11a.

The topics encompass a broad spectrum of fields, including Business, Ecology, Sports, Technology,
Health Care, Politics, and more. Notably, Business constitutes the largest proportion of the dataset at
11.1%, while the remaining topics are more evenly distributed, with percentages generally ranging
from approximately 1.4% to 7.9%.

Given the nature of these long-text reasoning tasks, the scenarios within our story-based games
typically range from 200 to 450 words in length. As illustrated in Figure 11b, over 90% of scenario
lengths fall within the 250 to 400-word interval. Additionally, we provide a scatter plot of scenario
lengths by topic to further demonstrate the diversity of our generated dataset.

21

Under review as a conference paper at ICLR 2025

(a) The topic distribution of story-based games.

(b) Cumulative distribution of lengths by ratio and scatter plot of lengths by topic.

Figure 11: Statistical distribution of story-based games over 20 topics.

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

11.1%7.9%7.4%6.7%6.1%6.0%5.4%5.3%5.0%4.9%4.7%4.7%4.5%4.0%3.9%3.6%3.2%2.6%1.5%1.4%BusinessEcologySportsArtTechnologySociologyHealth CarePoliticsMilitary StrategyTransportationEducationPsychologyEngineeringSpace ExplorationInternational RelationsBiologyLawEmploymentInterpersonal InteractionEconomics2002503003504004500.000.250.500.751.00Cumulative Ratio200250300350400450Task Length (`Scenario` Part)ArtBiologyBusinessEcologyEconomicsEducationEmploymentEngineeringHealth CareInternational RelationsInterpersonal InteractionLawMilitary StrategyPoliticsPsychologySociologySpace ExplorationSportsTechnologyTransportationCategoryUnder review as a conference paper at ICLR 2025

Figure 12: Inconsistency heat map of GPT series models using reFoToM and reSoToM prompting.

Table 4: The significance degree of top-tier GPT models performance. The larger value indicates the
higher significance of the peculiar answering pattern. Near-zero value means no particular pattern.
All values are expressed as percentages.

Model

gpt-4o

CoT

FoToM ReFoToM SoToM ReSoToM

gpt-4o-mini

5.56

13.89

9.38

26.74

8.33

20.49

4.51

32.64

6.25

35.42

E ADDITIONAL RESULTS

E.1 ASYMMETRIC INCONSISTENCY PATTERN

We show in Figure 12 that GPT series models still display similar pattern when using reFoToM and
reSoToM prompting. Yellow-box areas and green-box areas display an asymmetric inconsistency
pattern.

In order to further quantify how significant does the results display such pattern, we also propose a
metric named significance degree which confined in [0, 1] and it is defined as

SD =

1
18

(cid:88)

i,j

I{#NE(i, j) = 0} ·

1
4

(Freqi,j,1 + Freqi,j,4 − Freqi,j,2 − Freqi,j,3) ∗ S(i, j)

where #NE(i, j) represents the number of Nash equilibria of the tasks of grid (i, j), and S(i, j) is
decided by the area that the grid (i, j) belongs to, having value 1 given the grid in the green area and
value -1 given the grid in the yellow area.

We present the statistical results of LLMs’ performance in Table 4, which show that the SD values
for using ReFoToM are similar to those for FoToM, and the values for ReSoToM are close to those
for SoToM.

Additionally, the results indicate that employing ToM helps gpt-4o reduce the asymmetric incon-
sistency pattern, while it conversely makes gpt-4o-mini more ’stubborn’ in maintaining this pattern.
Furthermore, higher-order ToM appears to have a stronger effect than first-order ToM.

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

reFoToMreSoToMgpt-4o-minigpt-4oUnder review as a conference paper at ICLR 2025

E.2 CASES OF FAILURE

In this section, we demonstrate two examples of the real response from gpt-4o-mini, which derived
from a pair of symmetric data points (the grid (row 3, column 4) and the grid (row 9, column 10))
that theoretically have the same answer and similar reasoning pattern. We use the red coloured texts
to indicate the mistaken parts of the reasoning processes.

row 3 column 4

To determine the most likely choice combinations for Players A and B, we can analyze the
payoff matrix step by step.
1. **Identify the Payoffs**: The payoff matrix is as follows:
| B2

| A \ B | B1
|
|-------|-------|-------|
| 3 \ 1 | 1 \ 4 |
| A1
| 2 \ 3 | 4 \ 2 |
| A2

2. **Player A’s Choices**:
- If Player B chooses B1:

- A1 gives Player A a payoff of 3.
- A2 gives Player A a payoff of 2.

- If Player B chooses B2:

- A1 gives Player A a payoff of 1.
- A2 gives Player A a payoff of 4.

From Player A’s perspective:

- Against B1, A1 (3) is better than A2 (2).
- Against B2, A2 (4) is better than A1 (1).

3. **Player B’s Choices**:
- If Player A chooses A1:

- B1 gives Player B a payoff of 1.
- B2 gives Player B a payoff of 4.

- If Player A chooses A2:

- B1 gives Player B a payoff of 3.
- B2 gives Player B a payoff of 2.

From Player B’s perspective:

- Against A1, B2 (4) is better than B1 (1).
- Against A2, B1 (3) is better than B2 (2).

4. **Best Responses**:

- If Player A chooses A1, Player B will choose B2 (since 4 >1).
- If Player A chooses A2, Player B will choose B1 (since 3 >2).
- If Player B chooses B1, Player A will choose A1 (since 3 >2).
- If Player B chooses B2, Player A will choose A2 (since 4 >1).

5. **Nash Equilibria**:

- The combination (A1, B2) gives payoffs (3, 4).
- The combination (A2, B1) gives payoffs (2, 3).

6. **Conclusion**:

- The most likely choice combinations are (A1, B2) and (A2, B1). Since both combina-

tions are viable and lead to different outcomes, we conclude that both are possible.
Thus, the final answer in the required format is:
“‘python
answer = [(“A1”, “B2”), (“A2”, “B1”)] # list-type
”’

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

row 9 column 10

To determine the most likely choice combinations for Players A and B, we can analyze the
payoff matrix step by step.
1. **Identify the Payoffs**:

- For Player A:

- If A chooses A1:

- Against B1: Payoff = 2
- Against B2: Payoff = 4

- If A chooses A2:

- Against B1: Payoff = 3
- Against B2: Payoff = 1

- For Player B:

- If B chooses B1:

- Against A1: Payoff = 4
- Against A2: Payoff = 2

- If B chooses B2:

- Against A1: Payoff = 1
- Against A2: Payoff = 3

2. **Best Responses**:

- If Player B chooses B1:

- Player A’s best response is A2 (3 >2).

- If Player B chooses B2:

- Player A’s best response is A1 (4 >1).

- If Player A chooses A1:

- Player B’s best response is B1 (4 >1).

- If Player A chooses A2:

- Player B’s best response is B2 (3 >2).

3. **Nash Equilibria**:

- A Nash Equilibrium occurs when both players are choosing their best responses to each

other.

- From the analysis:

- (A1, B1) is a Nash Equilibrium because A’s best response to B1 is A2, but B’s best

response to A1 is B1.

- (A2, B2) is also a Nash Equilibrium because A’s best response to B2 is A1, but B’s

best response to A2 is B2.
4. **Conclusion**:

- The most likely choice combinations are (A1, B1) and (A2, B2) since both are Nash

Equilibria.
Now, we can represent this in the required Python-style code:
“‘python
answer = [(“A1”, “B1”), (“A2”, “B2”)] # list-type
”’

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

25

Under review as a conference paper at ICLR 2025

F ABOUT COMPLEX FORM GAME

F.1 EXAMPLE OF DIFFERENT COMPLEX FORM

In this section, we use visualized example (referred to Figure 13) to illustrate different kinds of
complex forms.

Figure 13: An example of how we build sequential, parallel, and nested game from some of the
atomic games in TMGBENCH.

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

FootballFootball(3, 2)BalletBallet(2, 3)③Prisoner’s DilemmaCooperateDefectCooperate(3, 3)(0, 5)Defect(5, 0)(1, 1)Description: Two prisoners must independently decide whether to cooperate or betray (defect). Betrayal offers a higher payoff if the other cooperates, but mutual betrayal leads to the worst outcome for both.Stag HuntHunt StagHunt HareHunt Stag(4, 4)(0, 3)Hunt Hare(3,0)(3, 3)Description: Two players can hunt a stag together (requiring cooperation) or hunt a hare individually (a safer but less rewarding choice). Trust is essential to maximize the payoff.Battle of the SexesFootballBalletFootball(3, 2)(0, 0)Ballet(0, 0) (2, 3)Description: A couple prefers to do an activity together but have different preferences. One prefers a football game, while the other prefers ballet. Coordination is key, but each prefers their favored activity.NENENENENEThree Atomic GamesSequentialDefectDefect(1, 1)Prisoner’s DilemmaCooperateDefectCooperate(3, 3)(0, 5)Defect(5, 0)(1, 1)Stag HuntHunt StagHunt HareHunt Stag(4, 4)(0, 3)Hunt Hare(3,0)(3, 3)Battle of the SexesFootballBalletFootball(3, 2)(0, 0)Ballet(0, 0) (2, 3)Hunt StagHunt Stag(4, 4)Hunt HareHunt Hare(3, 3)①②In sequential games, we designed different types of atomic games to evaluate whether LLMs can perform strategy reasoning stably without being influenced by historical game information.ParallelIn parallel games, we designed different types of atomic games to evaluate whether LLMs can simultaneously consider different types of game scenarios and make correct strategic reasoning.Prisoner’s DilemmaCooperateDefectCooperate(3, 3)(0, 5)Defect(5, 0)(1, 1)Stag HuntHunt StagHunt HareHunt Stag(4, 4)(0, 3)Hunt Hare(3,0)(3, 3)Battle of the SexesFootballBalletFootball(3, 2)(0, 0)Ballet(0, 0) (2, 3)Input all three gamesBattle of the SexesFootballFootball(3, 2)Battle of the SexesBalletBallet(2, 3)Prisoner’s DilemmaDefectDefect(1, 1)Stag HuntHunt StagHunt Stag(4, 4)Stag HuntHunt HareHunt Hare(3, 3)Output three answers simultaneouslyNestedPrisoner’s DilemmaCooperateDefectCooperate(3, 3)(0, 5)Defect(5, 0)(1, 1)Stag HuntHunt StagHunt HareHunt Stag(4, 4)(0, 3)Hunt Hare(3,0)(3, 3)Pre-gameCore-gameInput two gamesPrisoner’s DilemmaCooperateCooperate(3, 3)Defect(5, 0)Stag HuntHunt StagHunt HareHunt Stag(4, 4)(0, 3)Hunt Hare(3,0)(3, 3)Pre-gameCore-gamePrisoner’s DilemmaDefectCooperate(0, 5)Defect(1, 1)Stag HuntHunt StagHunt HareHunt Stag(4, 4)(0, 3)Hunt Hare(3,0)(3, 3)Pre-gameCore-gameHunt StagHunt Stag(4, 4)Hunt HareHunt Hare(3, 3)CooperateDefect(5, 0)DefectDefect(1, 1)Scenario 1: If (Hunt Stag, Hunt Stag) is chosen in the pre-game, it leads to only being able to choose (Cooperate, Cooperate) and (Defect, Cooperate) in the core-game, which means the Nash equilibrium point (Defect, Defect) cannot be selected in the core-game. Therefore, choosing (Hunt Stag, Hunt Stag) in the pre-game is an incorrect strategy.Scenario 2: If (Hunt Hare, Hunt Hare) is chosen in the pre-game, then (Cooperate, Defect) and (Defect, Defect) can be chosen in the core-game, which allows the LLM to select the Nash equilibrium point (Defect, Defect) in the core-game. Therefore, choosing (Hunt Hare, Hunt Hare) in the pre-game is a correct strategy.Scenario 1Scenario 2In nested games, we designed two inner-linked atomic games to evaluate if LLMs can achieve optimial payoff by applying strategic reasoning with some restrictions.Under review as a conference paper at ICLR 2025

F.2 NESTED FORM GAME

In this section, we demonstrate the template we use for generating nested form games. The text in
red are the placeholders for the variables of the generation codes.

template of a kind of nested form game

<Pre-Game >
{pre game}
<Core-Game >
{core game}
[Question]
Player A and B are facing the two games, the pre-game and the core-game.
Note that their final goal is to maximize own payoff first in the core Game, then in the
pre-game.
Additionally, {restricted player} is attached with an restriction that if the situation of the
pre-game is {restricted situation}, then he can not choose action {restricted choice}.
What is/are the most possible choice combination(s) of the pre-game ultimately? (when all
choice combinations have equal possibility, the answer should contain nothing)
[/Question]

After a nested form game is generated through our template, we still need to check if the Nash
If the set of Nash
equilibria of the pre-game changes after the restriction from the core game.
equilibria does change, then we use this as a piece of data to evaluate LLMs, observing if they can
observe such a violation of original NEs’ structure.

F.3 SIGNIFICANCE OF ATOMIC GAMES AND COMPLEX-FORM GAMES

Our evaluation of complex-form games serves as a test of whether LLMs can solve real-world prob-
lems with higher complexity, rather than merely solving a single atomic game in isolation. The
atomic games in TMGBENCH represent the primary components of complex real-world social sce-
narios (Gintis, 2014). In addition to the typical combinations found in temporal contexts (i.e., se-
quential games) or spatial contexts (i.e., parallel games), which require reasoning and independent
decision-making, Tsebelis (1990) introduced a concept known as nested games, where two or more
games are inner-linked. This type of game composition often arises in real-world domains such as
politics and economics, where decisions made in one arena can influence or constrain decisions in
another.

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

27

