# Human and Machine Intelligence in $n$-Person Games with Partial
Knowledge111I would like to thank LOFT 2022 conference (Groningen)
participants for the inspiring talks. I also would like to thank the audience
in M-BEES Conference at Maastricht University and Stony Brook Game Theory
Conference for their valuable comments.

Mehmet S. Ismail Department of Political Economy, King’s College London,
London, WC2R 2LS, UK<EMAIL_ADDRESS>
(Revised: 4 August, 2023; First version: 9 December, 2022)

###### Abstract

In this note, I introduce a new framework called $n$-person games with partial
knowledge, in which players have only limited knowledge about the aspects of
the game—including actions, outcomes, and other players. For example, playing
an actual game of chess is a game of partial knowledge. To analyze these
games, I introduce a set of new concepts and mechanisms for measuring the
intelligence of players, with a focus on the interplay between human- and
machine-based decision-making. Specifically, I introduce two main concepts:
firstly, the Game Intelligence (GI) mechanism, which quantifies a player’s
demonstrated intelligence in a game by considering not only the game’s outcome
but also the “mistakes” made during the game according to the reference
machine’s intelligence. Secondly, I define gaming-proofness, a practical and
computational concept of strategy-proofness. The GI mechanism provides a
practicable way to assess players and can potentially be applied to a wide
range of games, from chess and backgammon to AI systems. To illustrate the GI
mechanism, I apply it to an extensive dataset comprising over a million moves
made by top chess Grandmasters. JEL: C72, D80, C88

Keywords: Human intelligence, machine intelligence, $n$-person partial
knowledge games, general-sum games, strategyproof mechanisms

##  1 Introduction

Games present a compelling framework for examining human and machine
intelligence, characterized by interactions and decision-making under
uncertainty. The exploration of intelligence measurement in games has
attracted considerable attention in the fields of computer science and
psychology, yet remains relatively unexplored in economics. The emergence of
artificial intelligence (AI) has fostered the development of cutting-edge
techniques for evaluating game outcomes. Nevertheless, the interplay between
human- and machine-based decision-making in games poses a formidable
challenge, and devising functional mechanisms to measure intelligence while
accounting for this interplay is an active research area.

In this paper, I introduce a novel set of mechanisms for measuring
intelligence in $n$-person ($n\geq 1$) games with partial knowledge, focusing
on the interplay between human- and machine-based decision-making. In this
framework, player(s) have limited knowledge about the aspects of the game,
including actions, outcomes, and other players. For example, playing an actual
game of chess represents a two-person zero-sum game with partial knowledge,
even though chess is, technically, a game of perfect information. In this
novel framework, I introduce two main concepts: firstly, the Game Intelligence
(GI) mechanism, which quantifies a player’s demonstrated intelligence in a
game by considering not only the game’s outcome but also the “mistakes” made
during the game according to the reference machine’s intelligence. Secondly, I
define gaming-proofness, a practical and computational concept of strategy-
proofness. I design the GI mechanism to offer a practicable mechanism for
assessing players’ intelligence while factoring in “human-optimal” and
“machine-optimal” moves in games.

To develop the GI mechanism, I begin by defining the Game Point Loss (GPL)
metric, which measures a player’s expected loss during a game based on their
moves and the reference machine’s stochastic evaluations of these moves. I
employ the GPL metric in the computation of the GI mechanism, which then
serves to assess players’ intelligence. To evaluate the desirability of
various mechanisms, I introduce the concept of “gaming-proofness,” a property
of a mechanism that is resistant to manipulation by players from a subjective
and computational standpoint. A mechanism is called gaming-proof if a human
player cannot strategically play a suboptimal move to increase their
intelligence score as determined by the mechanism. I show that the expected GI
score is gaming-proof if the human player’s evaluation of the machine’s
deviation from a human-optimal move is worse than their own evaluation of the
deviation. In other words, a human player cannot increase their expected GI
score by playing a suboptimal move, provided that their own evaluation of the
position is superior to what they perceive as the machine’s evaluation.
Gaming-proofness might fail if the human player has access to a “better”
machine or has a greater “intelligence” or computational power than the
machine that is used to calculate the human’s GI score.

Intelligence is a complex construct that cannot be perfectly captured by the
outcome of a single game, even in games like chess. A high score in a game
does not necessarily indicate high intelligence, as chance may significantly
impact the outcome. For instance, in backgammon, a player might play a poor
move but still maintain a positive probability of winning the game.
Furthermore, “machine-optimal” moves may not always be optimal from a human
standpoint, and conversely, “human-optimal” moves may not be machine-
optimal.222Top chess players frequently employ “suboptimal” moves to increase
their probability of winning a game. As a result, assessing intelligence in
games necessitates an examination of the interplay between human- and machine-
based intelligence.

The accuracy of measuring intelligence in games increases as the number of
games played grows. However, the degree of intelligence that can be assessed
is dependent on the game in question. In games where every outcome provides
the same payoff to all players, there is no room for errors, and measuring
intelligence becomes unproductive. Conversely, in games where each decision is
crucial, intelligence can be measured with higher accuracy.

Intelligence is a relative concept, and a player’s Game Intelligence—as
implied by the name—hinges on the specific game being played. While it is
possible to compare GI across various games, such comparisons may not always
be meaningful. For example, in games with identical outcomes (e.g.,
win/draw/loss), comparisons would be more insightful. However, even in
win/draw/loss games, comparisons may not be flawless, as each game features
distinct characteristics that affect the level of intelligence necessary for
successful play.

In section 2, I apply the GI mechanism to a selection of top-level chess
games. By analyzing human versus human chess games, I demonstrate the
practicality of the GI concept and its ability to assess and compare players
in real-world scenarios. In section 3, I review relevant literature on the
subject. In section 4, I present the game-theoretic framework and establish
the main results. Finally, in section 7, I conclude the paper.

###  1.1 Ranking players and teams in other sports and games

The Game Intelligence (GI) mechanism has broad applicability and can be used
to measure players’ and teams’ intelligence across a wide range of $n$-person
games, including sports, board games, video games, as well as individual
decision problems. By normalizing the rewards and points players receive, the
GI mechanism can also facilitate comparisons and rankings among players in
different sports. Nevertheless, caution is warranted when implementing the GI
mechanism in sports or games where AI systems have not yet attained a level of
intelligence sufficient for measuring human intelligence. As AI progresses in
playing and evaluating such sports, the effectiveness of the GI mechanism will
correspondingly improve.

It is important to note that a GI mechanism based on common performance
indicators, such as the “number of shots” in basketball, would not be gaming-
proof, as a team could manipulate this metric to inflate statistics.
Consequently, developing a human-level AI system for sports like football
(i.e, soccer) or basketball is an essential prerequisite. In tennis, the GI
mechanism could be used to determine the long-term game intelligence of
established players, while in Go, it could serve to rank players, especially
considering the frequent use of handicaps to counter the first-mover
advantage. In backgammon, the GI mechanism could—and should—replace the well-
established error-rate metric, which is not gaming-proof in part because it
does not take the human-optimal moves into account.

By integrating the strengths of both human- and machine-optimal decision-
making, the GI mechanism offers the prospect of delivering a more accurate
assessment of a team’s or player’s intelligence compared to other metrics in
various sports and games.

##  2 An overview of the theory and the data

In this section, I present the basic theory applied to more than one million
moves analyzed by top chess AI (Stockfish) of some of world’s top chess
grandmasters. The dataset includes all world chess championships since 1898
and many elite tournaments.

This section is self-contained: it only introduces a minimal notation for the
theory, and it assumes no familiarity about chess except the following
information. Chess is a two-person game in which players (White and Black)
alternate taking turns and making moves until a player wins or the game is
drawn. Chess moves can be analyzed by powerful AI systems, usually called
engines, some of which are way stronger than the best human player.

###  2.1 The GI score and GPL

Let $EPL_{i}(j)$ denote the expected point loss for the $j$th move of player
$i\in\\{1,2\\}$ in a chess game. The win probability of the active player
(White or Black) according to an engine such as Stockfish before making a move
is denoted by $w$, and the win probability of the same player after making a
move is denoted by $w^{\prime}$.333Ideally, the win/draw/loss probabilities of
the best move and the actual move should be measured based on the same engine
settings. The win/draw/loss probabilities might also be based on actual human
data e.g. from Lichess.org. See also Maia engine [26] whose aim is to play
human-like moves. Similarly, $d$ denotes the probability of a draw before the
move, and $d^{\prime}$ represents the probability of a draw after the move.
The number of points a player receives for winning the game is denoted by
$r_{w}$, and the number of points for a draw is denoted by $r_{d}$. The
expected loss of move $j$ for player 1 (White) can be calculated using the
following formula:

$EPL_{1}(\text{move }j)=r_{w}\times w+r_{d}\times d-r_{w}\times
w^{\prime}-r_{d}\times d^{\prime}.$

In simpler terms, the expected point loss of a move is the difference between
the points a player expects to win with the best engine move and the points
the player expects to receive with the actual move. It is important to note
that the expected point loss may be negative, as it depends on the particular
engine and its settings. Additionally, the win probability before and after a
move can differ as engine evaluation may change before or after the move due
to the limitations of the engines.

Subsequently, the game point loss (GPL) of each player can be computed as the
sum of the expected point losses for each of their moves during the game. The
GPL of White, denoted as $GPL_{1}$, is given by the sum of the expected point
losses for all of White’s moves in the game:

$GPL_{1}=EPL_{1}(\text{move }1)+EPL_{1}(\text{move
}2)+\cdots+EPL_{1}(\text{last move}).$

Similarly, the GPL of Black, denoted as $GPL_{2}$, is the sum of the expected
point losses for all of Black’s moves in the game:

$GPL_{2}=EPL_{2}(\text{move }1)+EPL_{2}(\text{move
}2)+\cdots+EPL_{2}(\text{last move}).$

It is worth noting that typically White and Black make an equal number of
moves, but there may be cases where one player makes one more or less move
than the other.

Next, I define the game intelligence of each player. Let $r_{i}$ denote the
points that player $i$ receives from the game, where $r_{i}$ can be 1 if
player $i$ wins, 0.5 if the game ends in a draw, and 0 if player $i$ loses.
Then, the game intelligence (GI) of player $i$ is defined as the difference
between the points received by player $i$ and their GPL, i.e.,

$GI_{i}=r_{i}-GPL_{i}.$

The GI measures each player’s demonstrated intelligence in the game, taking
into account the the human element (as reflected by the outcome of the game)
and the engine’s evaluations of the moves the player made (as reflected in the
GPL). A higher GI signifies higher intelligence by the player, while a lower
GI indicates the contrary.

###  2.2 World Chess Championships

Player | GI | $\text{SD}_{GI}$ | $\text{GI}_{W}$ | $\text{GI}_{B}$ | GPL | $\text{GPL}_{W}$ | $\text{GPL}_{B}$ |  $\\#$Games |  $\\#$Moves
---|---|---|---|---|---|---|---|---|---
Carlsen | 0.17 | 0.58 | 0.14 | 0.19 | 0.41 | 0.46 | 0.36 | 56 | 2958
Anand | 0.09 | 0.63 | 0.19 | 0 | 0.4 | 0.37 | 0.43 | 88 | 3323
Kramnik | 0 | 0.78 | 0 | 0 | 0.52 | 0.58 | 0.47 | 65 | 2488
Topalov | 0 | 0.70 | 0.15 | -0.16 | 0.46 | 0.39 | 0.53 | 23 | 1200
Kasparov | 0 | 0.78 | 0.14 | -0.14 | 0.52 | 0.46 | 0.58 | 197 | 7076
Karpov | -0.07 | 0.76 | 0.05 | -0.19 | 0.58 | 0.53 | 0.63 | 194 | 7929
Liren | -0.11 | 0.70 | -0.15 | -0.07 | 0.61 | 0.79 | 0.43 | 14 | 672
Capablanca | -0.15 | 0.83 | -0.21 | -0.09 | 0.66 | 0.75 | 0.57 | 48 | 1921
Fischer | -0.24 | 0.90 | -0.09 | -0.36 | 0.86 | 0.76 | 0.95 | 20 | 866
Petrosian | -0.37 | 0.79 | -0.34 | -0.4 | 0.88 | 0.92 | 0.84 | 69 | 3079
Table 1: Summary statistics of world chess champions 1886–2023. Means of GI,
GPL, and the standard deviation (SD) of GI are presented.

Table 7 offers a snapshot of the performance of world chess champions from
1886 to 2023, focusing on their GI scores and GPLs. The data, derived from an
analysis of approximately 87,000 moves from world championship games using one
of the strongest chess AIs, Stockfish, highlights some intriguing patterns and
comparisons.

Magnus Carlsen stands out with the highest average GI of 0.17, signifying
that, according to our model, he demonstrated superior game intelligence,
notably achieving this despite committing a comparable number of or more
inaccuracies than Anand, who has the second highest average GI. This suggests
that Carlsen’s playing style, while not always following the AI’s ’optimal’
move, has a propensity to elicit more mistakes from his opponents.

Figure 1 visually compares the GI score distributions of Carlsen and Garry
Kasparov, two of the most frequently compared chess champions. One factor
potentially influencing the observed differences in their distributions is the
smaller number of world championship games played by Carlsen, which inherently
affects the spread of his GI scores. However, as shown in Figure 2 discussed
in the subsequent subsection, the GI distributions for Carlsen and Kasparov
become significantly more similar when the dataset is expanded to include a
greater number of games

Figure 1: Distribution of GI scores of Carlsen and Kasparov

The table also differentiates GIs when players had either the White ($GI_{W}$)
or Black ($GI_{B}$) pieces. The majority of players, excluding Carlsen, show
better performance when playing as White, aligning with the commonly held view
that playing as White offers a slight advantage. Notably, Carlsen is the only
player who has a positive $GI_{B}$, solidifying his reputation as an
exceptional player when assigned the black pieces.

Furthermore, the table provides an understanding of the variability of a
player’s performance, as measured by the standard deviation of the GI. For
instance, Anand, despite having the second-highest average GI (0.09), shows a
larger spread in his performance, as indicated by his higher standard
deviation compared to Carlsen’s. While Carlsen’s GI is not statistically
significantly different from Anand’s GI, the magnitude of the difference,
especially for $GI_{B}$, is huge.

A comparison of GPL values reveals that both Anand and Carlsen made moves most
consistent with AI-suggested optimal moves, signifying their precision in game
play. Although Anand and Topalov outperformed Carlsen when playing as White,
Carlsen showcased exceptional skill as Black, recording the lowest GPL.

###  2.3 The full dataset: Super grandmaster games

Player | GI | $\text{SD}_{GI}$ | $\text{GI}_{W}$ | $\text{GI}_{B}$ | GPL | $\text{GPL}_{W}$ | $\text{GPL}_{B}$ |  $\\#$Games |  $\\#$Moves
---|---|---|---|---|---|---|---|---|---
Fischer | -0.02 | 0.84 | 0.05 | -0.08 | 0.75 | 0.73 | 0.76 | 640 | 26,669
Carlsen | -0.03 | 0.90 | 0.00 | -0.05 | 0.66 | 0.70 | 0.63 | 1,187 | 57,422
Kasparov | -0.03 | 0.86 | 0.03 | -0.09 | 0.71 | 0.70 | 0.71 | 1,188 | 46,103
Karjakin | -0.12 | 0.80 | 0.00 | -0.25 | 0.66 | 0.62 | 0.70 | 493 | 23,388
Aronian | -0.13 | 0.78 | -0.10 | -0.16 | 0.67 | 0.70 | 0.63 | 641 | 30,264
Kramnik | -0.13 | 0.83 | -0.10 | -0.16 | 0.74 | 0.79 | 0.70 | 1,250 | 56,758
Anand | -0.14 | 0.94 | -0.12 | -0.17 | 0.75 | 0.78 | 0.72 | 1,529 | 64,173
Giri | -0.19 | 0.79 | -0.16 | -0.22 | 0.70 | 0.73 | 0.68 | 440 | 19,963
Leko | -0.19 | 0.78 | -0.13 | -0.26 | 0.72 | 0.71 | 0.73 | 443 | 20,170
Mamedyarov | -0.20 | 0.89 | -0.03 | -0.37 | 0.72 | 0.63 | 0.81 | 437 | 18,230
Karpov | -0.21 | 0.87 | -0.16 | -0.26 | 0.87 | 0.89 | 0.83 | 1,448 | 72,154
Table 2: Summary statistics of ‘super’ grandmasters. Means of GI, GPL, and the
standard deviation (SD) of GI are presented.

Table 8 takes a broader perspective, summarizing the GI score and GPL for
‘super’ grandmasters. This dataset, which includes Stockfish analysis of over
one million moves from predominantly elite chess tournaments, reveals that
some of the top players have a negative average GI. This outcome might be
attributable to the inclusion of games from the early stages of their careers,
where their play may not have been as refined. Moreover, it is worth noting
that their median GI scores, as shown in Table 9, are all positive. This
discrepancy between the average and median values suggests that these players’
GI scores might be skewed by a relatively few number of games with
exceptionally low values.

Fischer, Carlsen, and Kasparov stand out with the highest average GI scores,
which are closely matched, suggesting a comparable level of demonstrated game
intelligence according to our model. Their GI scores are statistically and
practically indistinguishable. However, their GPLs tell a different story,
with Carlsen’s score (0.66) being lower than both Kasparov’s (0.71) and
Fischer’s (0.75), suggesting that Carlsen’s moves were closer to the AI’s
optimal choice. The distinct differences in their GPLs are statistically
significant, as detailed in Table 5. Intriguingly, Fischer manages to secure
the highest GI score, despite his relatively high GPL. This suggests that even
though Fischer’s moves may not always coincide with the AI’s top choice, they
seem to provoke a higher frequency of errors from his opponents. This unique
aspect of Fischer’s playing style contributes positively to his overall GI
score.

Figure 2 provides a visual comparison of the GI score distributions for
Fischer, Carlsen, and Kasparov. Carlsen and Kasparov show remarkably similar
distributions. However, Fischer’s distribution has a visibly different shape,
which could be due to the fact that he stopped played chess immediately after
becoming a world champion.

Figure 2: Distribution of GI scores of Fischer, Carlsen, and Kasparov

The GI mechanism, by taking into account both the final score and the chess
engine’s intelligence reflected in its evaluations, provides a means to assess
and compare the demonstrated intelligence of players in a tournament. Such
comparisons yield insights that a simple win-loss-draw analysis might miss,
highlighting the nuanced complexity of chess and the varied playing styles of
its top players.

##  3 Related works

The present study aims to focus on actual play in games, instead of complete
plans of action (i.e., strategies). This approach is inspired by what can be
called as “Bonanno’s program” (see, e.g., [4] and more recently [5]), which
advocates a shift of emphasis towards describing real play in games. While
this paper draws inspiration from Bonanno’s program, its framework is
distinct, designed specifically to examine behavior in partial knowledge
settings.

Under partial knowledge games, players do not have complete
knowledge/information about the aspects of the game, including other players,
strategies, game tree, and outcomes. Although this study does not explicitly
aim to model unawareness in the way it has been modeled, it does allow for the
possibility that players may be—and usually are—unaware of certain
aforementioned aspects of the game. For example, a player may be unaware of
certain moves in a chess game. In that respect, I was inspired by the
burgeoning body of research on unawareness that began with [10]. Early studies
on unawareness, such as those by [10] and [15], were not specifically based on
game theory (though the latter study considers a multi-agent setting).
However, game theory has been increasingly used in this field. For a review of
the literature on unawareness, see [32].

In addition, there are game theoretical frameworks, some of which were in part
inspired by chess, that focus on equilibrium concepts and/or strategies [21,
14, 20]. In contrast to these solution concepts and frameworks, the present
study does not assume equilibrium, strategy, or rational play.

#### Backgammon and chess

As a former professional backgammon player and an amateur chess player, I was
inspired by the prior work of computer scientists and programmers who
developed AI systems (called engines) for playing backgammon, such as TD-
Gammon [36] and GNU Backgammon, and chess such as Deep Blue [8] and Stockfish.
Such engines allow players to evaluate the quality of their moves and their
performance via metrics such as “error rate” in backgammon, and “average
centipawn loss” and “total pawn loss” [1] in chess. However, existing methods
based solely on engine evaluations do not account for human-optimal moves and
are limited by their reliance on deterministic pawn-based or average-based
metrics that can be manipulated. A well-known critique of pawn-based metrics
is that a one-pawn unit is a different object in different positions: for
example, in a completely winning position losing a pawn unit is very different
than losing a pawn unit in a drawn position.

A considerable body of empirical research has used chess engines to assess the
pawn-based quality of moves. While a comprehensive review of this literature
is beyond the scope of this note, some selected works in the fields of
computer science and economics include those by [13, 29, 35, 23, 22,
39].444The majority of these studies used the Stockfish chess engine, which
only recently announced probabilistic evaluations in its latest version,
released in December 2022. Chess can serve as a valuable framework for
examining cognitive performance in various environments. For instance, a
recent study by [23] discovered a decline in performance when players competed
remotely compared to in-person. Their empirical analysis took into account
several controls, including the pawn evaluation before the move, the player’s
Elo rating, the remaining time, and the complexity of the position.
Furthermore, they excluded the first 15 moves from their analysis, as top
players typically prepare these opening moves in advance.

#### Human and machine intelligence

For over a century, psychometric tests have been widely utilized in psychology
to evaluate an individual’s cognitive abilities [34]. These assessments
typically measure skills such as verbal, mathematical, and spatial reasoning.
For an in-depth exploration of intelligence measurement in both human and
artificial agents, readers are referred to [3, 12, 6, 31, 16].

Measuring machine intelligence is an important aspect of AI research, with
various well-known theories and frameworks. The Turing Test [37] is a classic
method for evaluating intelligence through natural language conversations,
while the Universal Intelligence Measure [24] offers a formal definition of
intelligence and a quantitative measure based on Kolmogorov complexity and
AIXI [19]. Additionally, some alternatives have been proposed, such as the
C-test and the anytime universal intelligence test, which aim to assess
various aspects of intelligence [18, 17].

While aforementioned theoretical works define machine intelligence in terms of
average performance (i.e., rewards) across various environments, my definition
of game intelligence for a player encompasses not only rewards but also the
evaluation of the player’s actions by the reference machine’s intelligence.
The novelty of the GI measure is that it combines these two elements in a
unique way that has a clear interpretation, where players in a game can be
ranked based on their GI score. Like many intelligence tests, GI can be
applied to both humans and machines. Furthermore, I introduce the concept of
gaming-proofness for intelligence test mechanisms as a means of comparing and
contrasting different types of mechanisms and argue that such mechanisms
should be gaming-proof. I show that measurements based solely on the outcomes
of games or the machine evaluations are not gaming-proof.

#### Strategyproofness and game-proofness

The concept of gaming-proofness proposed in this paper is related to the
concept of strategyproofness studied in social choice, mechanism design, and
sports. The literature on strategy-proofness includes [9, 11, 25, 2, 7, 28].
Gaming-proofness differs from the well-known notions of strategy-proofness in
that gaming-proofness uses the notion of play rather than a strategy profile
as an object of analysis. In essence, a mechanism is considered gaming-proof
if it is very difficult, if not impossible, for a player to gain an advantage
by manipulating the mechanism, even though it may be theoretically possible to
manipulate the mechanism with greater computing power or intelligence than the
machine used to measure the player’s intelligence.

#### Bounded rationality

As mentioned above, the primary aim of this paper is to assess player
intelligence in $n$-person games using game outcomes and evaluations from
machines. While players may exhibit varying degrees of rationality, my focus
is not on classifying them as rational, boundedly rational, or irrational.
Instead, I concentrate on evaluating their intelligence given their
information and cognitive constraints. Bounded rationality has been a topic of
study for a long time, but the concept has no single definition [33].
Rubinstein [30] provides a comprehensive overview of the subject in his
textbook. In this paper, the players could be considered “boundedly rational.”
They have incomplete knowledge of important game aspects, such as the game
tree and possible outcomes, and their limited foresight prevents them from
performing full backward induction. Despite these limitations, some players in
my setup can also be considered “rational” in some sense. They make the best
decisions they can, given their information, environment, time, and
computational constraints. They assign probability distributions to the
uncertainty they face, although not in a fully Bayesian way, as they do not
account for the machine’s decision-making process. The players cannot reliably
predict the machine’s choices and therefore do not assign probability
distributions to them.

##  4 The theoretical framework

###  4.1 Games with partial knowledge

Table 3 introduces the basic notation I use in this paper. Let
$G=(N,H,X,I,\bar{A},R,M)$ be a finite extensive form game with partial
knowledge. In terms of its structure, it is ‘close’ to an extensive form game
with imperfect information but it does not contain strategy profiles and
instead assumes a limited knowledge of the aspects of the game rather than
common knowledge as I discuss below.

Name | Notation | Element
---|---|---
Players | $N=\\{1,2,\dots,n\\}$ | $i$
Information sets | $H$ | $h$
Nodes/positions | $X$ | $x$
Terminal nodes | $Z$ | $z$
Player function | $I:X\to N$ |
Actions at node $x$ | $A_{i}(x)$ | $a_{i}(x)$
Plays | $\bar{A}$ | $\bar{a}$
Outcomes/rewards | $R=\\{r_{1},r_{2},\dots,r_{m}\\}$ | $r_{j}\in\mathbb{R}$
Player $i$’s reward from $\bar{a}$ | $R_{i}(\bar{a})$ |
Probability distribution over $R$ | $\Delta R$ | $p$
Machines | $\mathcal{M}$ | $M$
$i$’th component of machine $M$ | $M_{i}:X\to\Delta R$ |
Machine-optimal action | $a^{*}_{i}(x)$ |
$i$’th component of mechanism $\mu$ | $\mu_{i}:\bar{A}\times\mathcal{M}\to\mathbb{R}$ |
Partial knowledge game | $G=(N,H,X,I,\bar{A},R,M)$ |
Table 3: Notation for extensive form games with partial knowledge

$N=\\{1,2,...,n\\}$ denotes the finite set of players where $n\geq 1$, $X$ a
finite game tree, $x_{0}$ the root of the game tree, $h\in H$ an information
set, $Z\subset X$ the set of terminal nodes, $I$ the player function,
$R_{i}:Z\rightarrow\mathbb{R}$ the payoff or “reward” function of player $i\in
N$.555Reward function could be a (von Neumann-Morgenstern [38]) utility
function if it is observable or measurable in the context of the game being
played. In that case, the risk attitude of the players would be captured in
these utilities. $I:H\to\mathbb{N}$ denotes the player function, where $I(h)$
gives the “active” player who moves at information set $h$, and $A_{i}(h)$ the
set of possibly mixed actions (i.e., behavior strategies) at information set
$h$. With a slight abuse of notation, $A_{i}(x)$ denotes the set of actions at
node $x$, which may belong to an information set $h$. (For a standard textbook
on game theory, see, e.g., [27].)

For a node $x_{m}\in X$, the path of play between $x_{0}$ and $x_{m}$ is
denoted by $[x_{m}]=\\{x_{0},x_{1},x_{2},...,x_{m}\\}$, where for every
$j=0,1,...,m-1$, $x_{j+1}$ is an immediate successor of $x_{j}$. For a node
$x^{\prime}\in X$, a play between $x_{0}$ and $x^{\prime}$ is denoted by
$\bar{a}$, which is a sequence of actions such that for every node $x$ in path
of play $[x^{\prime}]$, $a_{i}(x)$ leads to a node in $[x^{\prime}]$, and for
every node $y^{\prime}\in[z]\setminus\\{x_{0}\\}$, there exists a node
$y\in[x^{\prime}]$ and an action $a_{i}(y)$ that leads to node $y^{\prime}$. A
play $\bar{a}$ is called a full play and denoted by $\boldsymbol{\bar{a}}$
when $x^{\prime}$ is a terminal node. Let
$\bar{a}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l_{i}}_{i})$ be player $i$’s play,
which is the sequence of actions taken by player $i$ in play $\bar{a}$, where
$a^{1}_{i}$ is the first action taken by $i$, $a^{2}_{i}$ is the second
action, and $a^{l_{i}}_{i}$ is the last action for some natural number
$l_{i}>0$. Let $\boldsymbol{\bar{A}}$ denote the set of all full plays and
$\bar{A}$ the set of all plays. Let $\boldsymbol{\bar{a}}\setminus
a^{\prime}_{i}(x_{k})\in\bar{A}$ denote an alternative play, which is a full
play such that, holding everything else in sequence $\boldsymbol{\bar{a}}$
fixed prior to playing action $a_{i}(x_{k})$, player $i$ plays action
$a^{\prime}_{i}(x_{k})$ instead of action $a^{k}_{i}(x_{k})$ at $x_{k}$, and
then the play potentially diverges from the original full play
$\boldsymbol{\bar{a}}$.

The finite set of outcomes or rewards in the game is denoted by
$R=\\{r_{1},r_{2},...,r_{m^{\prime}}\\}$, where $r_{j}\in\mathbb{R}$ is a
typical element, with $m^{\prime}$ being the total number of outcomes. For
example, in chess, there are three possible outcomes: win, draw, and loss,
with their corresponding values being 1, 1/2, and 0, respectively. However,
the value of an outcome is not necessarily common among the players. With
slight abuse of notation, the function $R_{i}(\bar{a})$ denotes player $i$’s
outcome resulting from play $\bar{a}$, as each play uniquely determines a
terminal node, and each terminal node is associated with a unique play. Let
$\Delta R$ denote the set of all probability distributions over the set of
outcomes $R$ and $p\in\Delta R$ be a probability distribution: that is,
$\sum^{m}_{j=1}p(r_{j})=1$, and for all $j$, $p(r_{j})\geq 0$. A player may
have a probability distribution over a proper subset $R^{\prime}$ of the set
of outcomes $R$, partly because the player might not know the complete set of
outcomes. With a slight abuse of notation, this scenario is captured by a
probability distribution $p\in\Delta R$ such that none of the elements in
$R\setminus R^{\prime}$ are in the support of $p$.

#### Machines

I define a stochastic machine (i.e., an AI system) $M_{i}$, $i\in N$, as a
function $M_{i}:X\rightarrow\Delta R$. In simple terms, a machine yields a
probability distribution, $M_{i}(a_{i}(x))$, over a subset of outcomes for
node $x\in X$ in a game. Let $M$ denote a profile of machines. For example, a
chess engine is a machine that takes in a position and outputs a probability
distribution of the position. Note that a machine is an abstract term that can
refer to a human player (denoted as $M^{h}_{i}$) in a game. This can be
interpreted as follows: For any action $a_{i}(x)$ at node (i.e., position)
$x\in X$, a human player has a prior (i.e., probability distribution)
$M^{h}_{i}(a_{i}(x))\in\Delta R$.

In the context of games where a specific game-theoretic solution concept is
well-defined, the solution profile may be interpreted as a machine.666Note
that classical solution concepts such as Nash equilibrium are not in general
applicable to extensive form games with partial knowledge. For example, in a
standard extensive form game, a Nash equilibrium would induce a probability
distribution in every information set. These induced distributions would then
define a machine in that specific game.

######  Definition 1 (Machine-optimal actions).

An action $a^{*}_{i}(x)$ is called machine-optimal with respect to machine
$M_{i}$ if it satisfies the following condition:

$a^{*}_{i}(x)\in\arg\max_{a_{i}(x)\in
A_{i}(x)}\sum^{m}_{j=1}r_{j}M_{i}(a_{i}(x))(r_{j}).$

In simple words, a machine-optimal action at a node is the action with the
highest evaluation according to machine $M_{i}$. A machine-optimal action is
also known as the “best move” in a given position. A human-optimal action is
simply an action that is machine-optimal with respect to $M^{h}_{i}$. In other
words, a human-optimal action gives the highest subjective evaluation at a
position with respect to their prior.

#### Partial knowledge

In this paper, I do not assume that players have common knowledge of each
other’s priors, nor do I assume perfect recall. Specifically, given an
information set $h\in H$, player $i=I(h)$ knows that it is their turn to
choose an action, but may not know all the elements of the information set
$h$, nor the set of all available actions, $A_{i}(h)$. Players also do not
necessarily know their own or other players’ future available actions, or the
outcome of every play. As noted above, a player might not know the complete
set of possible outcomes in a position.

Players’ knowledge about the game, including actions, outcomes, and other
players, is captured by their subjective “evaluation.” Formally, player $i$’s
evaluation of action $a_{i}(x)$ at node $x$ is defined as:

$\displaystyle
EV_{i}(a_{i}(x),M^{h}_{i})=\sum^{m}_{j=1}r_{j}M^{h}_{i}(a_{i}(x))(r_{j}).$

###  4.2 Mechanisms

Let $\mathcal{M}$ be the set of all stochastic machines. In a game $G$, a
(stochastic) mechanism is a profile $\mu$ of functions whose $i$th component
is defined as $\mu_{i}:\bar{A}\times\mathcal{M}\rightarrow\mathbb{R}$. For
every machine $M$, every player $i$, and every play $\bar{a}$, the function
gives a probability distribution over the outcomes in the game. The
interpretation of $\mu_{i}$ is that it is a combined measure of human and the
machine intelligence of a play. Put differently, $\mu_{i}(\bar{a},M)$ gives an
intelligence score to player $i$ for play $\bar{a}$ given reference machine
$M$. One example of a stochastic mechanism is the GI mechanism.

#### Practicable mechanisms

I next introduce the concepts of “expected point loss” and “game point loss”
(GPL).777Note that in a previous version entitled “Tiebreak mechanisms with
stochastic AIs,” game point loss was called “total expected loss” (TEL).

######  Definition 2 (Expected point loss).

Let $M_{i}(a^{*}_{i}(x_{k}))$ be a machine’s evaluation of the best action for
player $i$ at position $x_{k}$ and $M_{i}(a^{k}_{i}(x_{k}))$ be machine’s
evaluation of $i$’s actual action. Then, the expected point loss of action
$a^{k}_{i}(x_{k})$ is defined as

$EPL_{i}(a^{k}_{i}(x_{k}))=\sum^{m}_{j=1}r_{j}M_{i}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}M_{i}(a^{k}_{i}(x_{k}))(r_{j}).$

The EPL of action $a^{k}_{i}(x_{k})$ is the difference between the evaluation
of the best move $a^{*}_{i}(x_{k})$ for player $i$ at node $x_{k}$ and the
evaluation of player $i$’s actual move $a^{k}_{i}(x_{k})$.

######  Definition 3 (Game point loss).

Let $\bar{a}\in\bar{A}$ be a play and $a^{j}_{i}$ be player $i$’s action at
position $x_{k}$ in play $\bar{a}$, where
$\bar{a}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l_{i}}_{i})$ for some $l_{i}$. Then,
player $i$’s game point loss (GPL) is defined as,

$GPL_{i}(\bar{a})=\sum_{k=1}^{l_{i}}\Bigl{(}\sum^{m}_{j=1}r_{j}M_{i}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}M_{i}(a^{k}_{i}(x_{k}))(r_{j})\Bigl{)}.$

The GPL of a player is the sum of the expected point losses of all their moves
in a play $\bar{a}$. From these two metrics, I next define the “game
intelligence” mechanism.

######  Definition 4 (Game Intelligence).

Let $\boldsymbol{\bar{a}}\in\boldsymbol{\bar{A}}$ be a full play. Then, the
game intelligence (GI) of $\bar{a}$ is defined as

$GI_{i}(\boldsymbol{\bar{a}})=R_{i}(\boldsymbol{\bar{a}})-GPL_{i}(\boldsymbol{\bar{a}}).$

The GI of a full play $v$ is defined as the difference between a player’s
actual game point, denoted as $R_{i}(\boldsymbol{\bar{a}})$, and their GPL.

Next, I introduce the concept of the expected Game Intelligence (EGI) score
for plays.

######  Definition 5 (Expected Game Intelligence).

Let $\bar{a}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l}_{i})$ be player $i$’s play
where $a^{l}_{i}$ is human-optimal at node $x_{l}$. Then, the expected game
intelligence (EGI) of $\bar{a}_{i}$ is defined as

$EGI_{i}(\bar{a}_{i})=EV_{i}(a^{l}_{i},M^{h}_{i})-GPL_{i}(\bar{a}_{i})$
$=\sum^{m}_{j=1}r_{j}M^{h}_{i}(a^{l}_{i}(x_{l}))(r_{j})-\sum_{k=1}^{l}\Bigl{(}\sum^{m}_{j=1}r_{j}M_{i}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}M_{i}(a^{k}_{i}(x_{k}))(r_{j})\Bigl{)}.$

The intuition behind the EGI score is that it replaces the actual outcome
$R_{i}(\bar{a})$ with the expected outcome of the game, according to player
$i$’s prior $M^{h}_{i}(a^{l}_{i}(x_{l}))$.

##  5 Theoretical results

###  5.1 Maximal intelligence

One of the first questions that comes to mind is whether there exists a play
with the “highest” intelligence score.

######  Definition 6 (Maximal intelligence score).

A full play $\boldsymbol{\bar{a}}$ is called maximally intelligent or said to
have a maximal intelligence score for machine $M$ with respect to mechanism
$\mu$ if for every player $i$ and for every alternative play
$\boldsymbol{\bar{a}}\setminus\hat{a}_{i}(x_{k})$ that deviates from action
$a^{k}_{i}$ in $\boldsymbol{\bar{a}}$,

$\mu_{i}(\boldsymbol{\bar{a}},M)\geq\mu_{i}(\boldsymbol{\bar{a}}\setminus\hat{a}_{i}(x_{k}),M).$

######  Proposition 1 (Existence of maximally intelligent play).

In every $n$-person game with partial knowledge, for every mechanism $\mu$ and
every machine $M$, there exists a full play $\bar{a}\in\bar{A}$ that has the
optimal intelligence score for machine $M$ with respect to mechanism $\mu$.

###### Proof.

The proof strategy is to apply a standard backward induction argument. Let $G$
be a finite $n$-person game with partial knowledge. I prove that for every
mechanism $\mu$, for every machine $M$, there exists a maximally intelligent
full play $\bar{a}$ in this game.

Step 1: First, because the game is finite, every full play in the game has a
last (non-terminal) node. Since there are no subsequent non-terminal nodes,
there exists a “maximally intelligent” action that has the highest
intelligence score according to mechanism $\mu$ at every such node.

Step 2: Consider a second-to-last (non-terminal) node of the game. By step 1,
note that for every action taken in the current node, the game has a maximally
intelligent action in the next node. I can now use backward induction logic to
show that the second-to-last node has a maximally intelligent action. Step 2a:
Consider each possible action that a player can take in the second-to-last
node, and assume that every subsequent player will play their maximally
intelligent action in the next node. Step 2b: Use the maximally intelligent
actions from the subsequent nodes to determine the payoff that each player
would receive if they took a particular action in the second-to-last node.

Repeat step 2 for each predecessor node of the game, using the maximally
intelligent actions from the subsequent nodes to determine the maximally
intelligent actions for each node. By the end of this finite process, the full
play that consists of maximally intelligent actions at every node starting
from the root must be maximally intelligent. Therefore, every $n$-person game
with partial knowledge has a maximally intelligent full play. ∎

While it is theoretically possible to find a maximally intelligent full play
in games, this approach is not in general practical for human vs. human real-
world games. This is because the existence result assumes the knowledge of the
machine using which the intelligence score is calculated. An access to such a
machine is not generally possible in real-world games.

###  5.2 Dynamic consistency of machines

Another interesting question is whether a machine’s evaluations should be
“dynamically consistent.”

######  Definition 7.

A machine $M$ is called dynamically consistent in game $G$ if for every player
$i$, for every node $x_{k}\in X$, and every action $a_{i}(x_{k})$ that leads
to node $x_{k+1}$, $M_{i}(a_{i}(x_{k}))=M_{i}(x_{k+1})$. A machine $M$ is
called dynamically consistent if it is dynamically consistent in every game.

Dynamic consistency requires that a machine’s evaluation of any action should
not change after the action has been played. However, it is neither a
necessary nor a sufficient condition for a “good” machine. For instance, a
machine could mistakenly evaluate each position in a chess game as a win for
White and still be dynamically consistent. On the other hand, Stockfish is an
example of a machine that achieves superhuman performance despite not being
dynamically consistent. That being said, if a machine is dynamically
consistent, computing the GI can be simplified as shown in the following
proposition.

######  Proposition 2 (GI and dynamic consistency).

Let $G$ be a sequential two-person constants-sum game with partial knowledge.

  1. 1.

For every full play $\boldsymbol{\bar{a}}\in\boldsymbol{\bar{A}}$, where
$l_{1}=l_{2}$ and $x_{2l_{1}-1}$ is the terminal node,

$GI_{1}(\boldsymbol{\bar{a}})=GI_{2}(\boldsymbol{\bar{a}})+R_{1}(\boldsymbol{\bar{a}})-R_{2}(\boldsymbol{\bar{a}})-\sum^{m}_{j=1}r_{j}M_{1}(x_{0})(r_{j})+\sum^{m}_{j=1}r_{j}M_{1}(x_{2l_{1}-1})(r_{j})$
(5.1)

  2. 2.

For every play $\boldsymbol{\bar{a}}\in\bar{A}$, where $l_{1}\neq l_{2}$ and
$x_{l_{1}+l_{2}-1}$ is the terminal node,

$GI_{1}(\boldsymbol{\bar{a}})=GI_{2}(\boldsymbol{\bar{a}})+2R_{1}(\boldsymbol{\bar{a}})-\sum^{m}_{j=1}r_{j}M_{1}(x_{0})(r_{j})+\sum^{m}_{j=1}r_{j}M_{2}(x_{l_{1}+l_{2}})(r_{j})$
(5.2)

###### Proof.

Recall that
$GI_{i}(\boldsymbol{\bar{a}})=R_{i}(\boldsymbol{\bar{a}})-GPL_{i}(\boldsymbol{\bar{a}})$.
Assume that $l_{1}=l_{2}$, and let $x_{2l_{1}-1}$ be the terminal node. Then,

$GI_{1}(\boldsymbol{\bar{a}})=R_{1}(\boldsymbol{\bar{a}})-\sum_{k=1}^{l_{1}}\Bigl{(}\sum^{m}_{j=1}r_{j}M_{1}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}M_{1}(a^{k}_{i}(x_{k}))(r_{j})\Bigl{)}.$

and

$GI_{2}(\boldsymbol{\bar{a}})=R_{2}(\boldsymbol{\bar{a}})-\sum_{k=1}^{l_{2}}\Bigl{(}\sum^{m}_{j=1}r_{j}M_{2}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}M_{2}(a^{k}_{i}(x_{k}))(r_{j})\Bigl{)}.$

Thus,

$GI_{1}(\boldsymbol{\bar{a}})=GI_{2}(\boldsymbol{\bar{a}})+R_{1}(\boldsymbol{\bar{a}})-R_{2}(\boldsymbol{\bar{a}})$
$+\sum_{k=1}^{l_{2}}\Bigl{(}\sum^{m}_{j=1}r_{j}M_{2}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}M_{2}(a^{k}_{i}(x_{k}))(r_{j})\Bigl{)}$
$-\sum_{k=1}^{l_{1}}\Bigl{(}\sum^{m}_{j=1}r_{j}M_{1}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}M_{1}(a^{k}_{i}(x_{k}))(r_{j})\Bigl{)}.$

Dynamic consistentcy of $M$ and constant-sum property of $G$ imply that for
every $i\neq i^{\prime}$,

$\sum^{m}_{j=1}r_{j}M_{i}(a^{k}_{i}(x_{k}))(r_{j})+\sum^{m}_{j=1}r_{j}M_{i^{\prime}}(a^{*}_{i}(x_{k+1}))(r_{j})=R_{1}(\boldsymbol{\bar{a}})+R_{2}(\boldsymbol{\bar{a}}).$

Thus, $GI_{1}(\boldsymbol{\bar{a}})$ simplifies to Equation 5.1. Equation 5.2
can be obtained analogously. ∎

Dynamic consistency, which requires that a machine’s evaluation of any action
should not change after the action has been played, is not a desirable
property in the real world. In fact, any machine with finitely limited
computing power must be dynamically inconsistent, since it is always possible
to design a game whose solution requires a computing power greater than that
of the machine. If the machine were dynamically consistent in that game, then
its evaluation at the root of the game would be “arbitrary” because it cannot
search through all the nodes.

###  5.3 Consistency of mechanisms

I next define a notion of consistency for mechanisms.

######  Definition 8 (Consistency).

Let $G$ be a game. A mechanism $\mu$ is called consistent if for every player
$i$ and every pair of plays $\boldsymbol{\bar{a}}$ and $\boldsymbol{\bar{b}}$
in $\boldsymbol{\bar{A}}$ the following three conditions are satisfied.

  1. 1.

$R_{i}(\boldsymbol{\bar{a}})>R_{i}(\boldsymbol{\bar{b}})$ and
$GPL_{i}(\boldsymbol{\bar{a}})<GPL_{i}(\boldsymbol{\bar{b}})$ implies that
$\mu_{i}(\boldsymbol{\bar{a}})>\mu_{i}(\boldsymbol{\bar{b}})$.

  2. 2.

Assume $GPL_{i}(\boldsymbol{\bar{a}})=GPL_{i}(\boldsymbol{\bar{b}})$.
$R_{i}(\boldsymbol{\bar{a}})\geq R_{i}(\boldsymbol{\bar{b}})$ if and only if
$\mu_{i}(\boldsymbol{\bar{a}})\geq\mu_{i}(\boldsymbol{\bar{b}})$.

  3. 3.

Assume $R_{i}(\boldsymbol{\bar{a}})=R_{i}(\boldsymbol{\bar{b}})$.
$GPL_{i}(\boldsymbol{\bar{a}})\leq GPL_{i}(\boldsymbol{\bar{b}})$ if and only
if $\mu_{i}(\boldsymbol{\bar{a}})\geq\mu_{i}(\boldsymbol{\bar{b}})$.

To put it simply, a mechanism is consistent if the value it assigns to a full
play contradicts neither the outcome of the full play nor the GPL evaluation
of the machine that is used to calculate the value of the mechanism.

The three conditions in the definition of consistency specify how the
mechanism should behave in different situations. For instance, if a play
results in worse evaluation for both the human and the machine compared to
another play, then the intelligence score of the latter must be greater than
the intelligence score of the former according to the mechanism. Conditions
(2) and (3) apply a comparative statics logic: if one of the evaluations
(either the human evaluation or the machine evaluation) is held fixed, then
the mechanism should be consistent with the other evaluation.

There is only one case where the concept of consistency does not apply. This
is when there are two plays, $\boldsymbol{\bar{a}}$ and
$\boldsymbol{\bar{b}}$, where $\boldsymbol{\bar{a}}$ leads to a strictly worse
evaluation for the machine than $\boldsymbol{\bar{b}}$, and
$\boldsymbol{\bar{b}}$ leads to a strictly worse evaluation for the human than
$\boldsymbol{\bar{a}}$. In this case, consistency does not place any
restrictions on the intelligence score of these plays. The reason is that
sometimes a move that is optimal for a human may not be optimal for a machine,
and vice versa. Both humans and machines can be wrong at times, which is why
the definition of consistency was formulated as such. I next show that GI
mechanism is consistent.

######  Proposition 3.

Let $\mu$ be the GI mechanism. Then, any mechanism of the form $\alpha
R_{i}(\boldsymbol{\bar{a}})-\beta GPL_{i}(\boldsymbol{\bar{a}})+\delta$, where
$\alpha>0$, $\beta>0$, and $\delta\in\mathbb{R}$, is consistent.

###### Proof.

Let $\boldsymbol{\bar{a}}$ and $\boldsymbol{\bar{b}}$ in
$\boldsymbol{\bar{A}}$ be a pair of plays. First, assume that
$R(\boldsymbol{\bar{a}})\geq R(\boldsymbol{\bar{b}})$ and
$GPL_{i}(\boldsymbol{\bar{a}})\leq GPL_{i}(\boldsymbol{\bar{b}})$. It implies
that

$R_{i}(\boldsymbol{\bar{a}})-GPL_{i}(\boldsymbol{\bar{a}})\geq
GI(\boldsymbol{\bar{b}})=R_{i}(\boldsymbol{\bar{b}})-GPL_{i}(\boldsymbol{\bar{b}}).$

Thus, $GI(\boldsymbol{\bar{a}})\geq GI(\boldsymbol{\bar{b}})$.

Second, assume that
$GPL_{i}(\boldsymbol{\bar{a}})=GPL_{i}(\boldsymbol{\bar{b}})$. Then,
$GI(\boldsymbol{\bar{a}})-GI(\boldsymbol{\bar{b}})$ if and only if

$R_{i}(\boldsymbol{\bar{a}})-GPL_{i}(\boldsymbol{\bar{a}})-R_{i}(\boldsymbol{\bar{b}})+GPL_{i}(\boldsymbol{\bar{b}}).$
$=R_{i}(\boldsymbol{\bar{a}})-R_{i}(\boldsymbol{\bar{b}}).$

Thus, $GI(\boldsymbol{\bar{a}})\geq GI(\boldsymbol{\bar{b}})$ if and only if
$R_{i}(\boldsymbol{\bar{a}})\geq R_{i}(\boldsymbol{\bar{b}})$.

Third, assume that $R(\boldsymbol{\bar{a}})=R(\boldsymbol{\bar{b}})$. Then,
$GI(\boldsymbol{\bar{a}})-GI(\boldsymbol{\bar{b}})$ if and only if

$GPL_{i}(\boldsymbol{\bar{b}})-GPL_{i}(\boldsymbol{\bar{a}}).$

Therefore, $GI(\boldsymbol{\bar{a}})\geq GI(\boldsymbol{\bar{b}})$ if and only
if $GPL_{i}(\boldsymbol{\bar{b}})\geq GPL_{i}(\boldsymbol{\bar{a}})$.

For analogous reasons as above, the mechanism given by $\alpha
R_{i}(\boldsymbol{\bar{a}})-\beta GPL_{i}(\boldsymbol{\bar{a}})+\delta$, where
$\alpha>0$, $\beta>0$, and $\delta\in\mathbb{R}$ is a consistent mechanism. ∎

While the metrics such as GPL, AGPL, and Accuracy are useful and informative,
it is easy to see that they are not consistent in part because they do not
depend on the outcome of a game. Analogously, the intelligence measurements
discussed in section 3 would not be consistent.

###  5.4 Gaming-proofness

Let $\hat{M}^{h}_{i}:X\rightarrow\Delta R$ denote a hypothetical machine where
$\hat{M}^{h}_{i}(x)$ is the probability distribution that human player $i$
believes machine $M_{i}$ assigns at node $x$.

######  Definition 9 (Gaming-proofness).

Let $\bar{a}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l-1}_{i},a^{l}_{i})$ be player
$i$’s play in which $a^{l}_{i}$ is a human-optimal action at node $x_{l}$ and
$\bar{a}^{\prime}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l-1}_{i},a^{l^{\prime}}_{i})$
be another play such that $a^{l^{\prime}}_{i}\neq a^{l}_{i}$. A mechanism
$\mu$ in a game $G$ is called gaming-proof for player $i$ at node $x_{l}$ if
for every $\bar{a}^{\prime}_{i}$,
$\mu_{i}(\bar{a}_{i},\hat{M}^{h}_{i})\geq\mu_{i}(\bar{a}^{\prime}_{i},\hat{M}^{h}_{i})$.
A mechanism $\mu$ is called gaming-proof if it is gaming-proof for every
player $i$ at every node $x$ such that $I(x)=i$.

In simple words, a gaming-proof mechanism ensures that no player can increase
their intelligence score by intentionally making a suboptimal move, as viewed
from their own perspective.

######  Proposition 4.

Let $\bar{a}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l-1}_{i},a^{l}_{i})$ be player
$i$’s play in which $a^{l}_{i}$ is a human-optimal action at node $x_{l}$ and
$\bar{a}^{\prime}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l-1}_{i},a^{l^{\prime}}_{i})$
be another play where $a^{l^{\prime}}_{i}\neq a^{l}_{i}$. EGI mechanism is
gaming-proof for player $i$ at node $x_{l}$, if, for every
$a^{l^{\prime}}_{i}$, the following inequality holds:

$EV_{i}(a^{l}_{i},M^{h}_{i})-EV_{i}(a^{l^{\prime}}_{i},M^{h}_{i})\geq
EV_{i}(a^{l^{\prime}}_{i},\hat{M}_{i})-EV_{i}(a^{l}_{i},\hat{M}_{i}).$

###### Proof.

By definition, GI mechanism is gaming-proof for player $i$ at node $x_{l}$ if
$EV_{i}(a^{l}_{i}(x_{l}),M^{h}_{i})\geq
EV_{i}(a^{l^{\prime}}_{i}(x_{l}),M^{h}_{i})$ implies that
$EGI(\bar{a}_{i})\geq EGI(\bar{a}^{\prime}_{i})$. The latter inequality
implies that

$EV_{i}(a^{l}_{i},M^{h}_{i})-GPL_{i}(\bar{a}_{i},\hat{M}_{i})\geq
EV_{i}(a^{l^{\prime}}_{i},M^{h}_{i})-GPL_{i}(\bar{a}^{\prime}_{i},\hat{M}_{i})$

or, more explicitly,

$\displaystyle
EV_{i}(a^{l}_{i},M^{h}_{i})-\sum_{k=1}^{l}\left(\sum^{m}_{j=1}r_{j}\hat{M}_{i}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}\hat{M}_{i}(a^{k}_{i}(x_{k}))(r_{j})\right)$
$\displaystyle\geq$ $\displaystyle
EV_{i}(a^{l^{\prime}}_{i},M^{h}_{i})-\sum_{k=1}^{l^{\prime}}\left(\sum^{m}_{j=1}r_{j}\hat{M}_{i}(a^{*}_{i}(x_{k}))(r_{j})-\sum^{m}_{j=1}r_{j}\hat{M}_{i}(a^{k}_{i}(x_{k}))(r_{j})\right).$

Observe that GPL of player $i$ before node $x_{l}$ is the same in the
calculation of $EGI(\bar{a}_{i})$ and $EGI(\bar{a}^{\prime}_{i})$. Therefore,
the inequality can be written as

$EV_{i}(a^{l}_{i},M^{h}_{i})-EV_{i}(a^{*}_{i},\hat{M}_{i})+EV_{i}(a^{l}_{i},\hat{M}_{i})\geq
EV_{i}(a^{l^{\prime}}_{i},M^{h}_{i})-EV_{i}(a^{*}_{i},\hat{M}_{i})+EV_{i}(a^{l^{\prime}}_{i},\hat{M}_{i}).$

This simplifies to

$EV_{i}(a^{l}_{i},M^{h}_{i})-EV_{i}(a^{l^{\prime}}_{i},M^{h}_{i})\geq
EV_{i}(a^{l^{\prime}}_{i},\hat{M}_{i})-EV_{i}(a^{l}_{i},\hat{M}_{i}),$

as desired. ∎

The intuition behind Proposition 4 centers on the relationship between the
human player’s evaluation of a particular game position and their belief of
the machine’s evaluation. Specifically, the human player’s evaluation of the
machine’s deviation from a human-optimal move must be worse than or equal to
their own evaluation of the deviation.

In other words, a human player cannot strategically play a suboptimal move to
increase their intelligence score, provided that their own evaluation of the
position is superior to what they perceive as the machine’s evaluation. This
premise holds true only if the human’s judgment is consistently more accurate
than their own approximation of the machine’s judgment.

##  6 Empirical Results

###  6.1 Data Gathering and chess AI analysis

The dataset at hand consists of approximately 1.2 million individual moves,
spread across approximately 15,000 games played by world-class chess
Grandmasters. The games were procured in Portable Game Notation (PGN) format
from a website called pgnmentor.com. The data set includes games from world
championships, elite tournaments, and individual matches played by top
Grandmasters. A subset, roughly 18% of the games, were evaluated by Stockfish
16, the state-of-the-art chess engine as of July 2023, on personal computers.
The analysis process involved annotating every position in these games using
the Stockfish centipawn evaluations. To calculate the GPL of the players in
these games, I used the win-draw-loss percentages from Stockfish vs. Stockfish
games as a function of centipawn loss.888I will later discuss how to calculate
GPL by estimating win-draw-loss probabilities from human vs. human games and
Elo ratings of the players.

I connected this vast database of PGNs to the Lichess API, which gave me the
ability to probe the Lichess Masters Database. Although this database is not
available to the public, Lichess allows for searching opening positions to
identify matches between top-tier players. The Lichess Masters Database
includes a multitude of games that have already been analyzed by Stockfish.

To validate the results, I cross-referenced and compared the statistics of
around 87,000 positions from world championship games evaluated by Lichess,
with the games I had personally analyzed using Stockfish 16. The results
closely mirrored those reported in section 2. In order to conserve
computational resources, I decided to continue using Lichess’s Stockfish
centipawn evaluations for the combined dataset, with the exception of those
files which had missing evaluations or pertained mainly to the games of Bobby
Fischer, which were not part of the Lichess database.

The process of cleaning the data involved segregating non-classical games,
such as rapid, blitz, online, blindfold, simul, and exhibition matches, from
the classical ones. Additionally, games played by Fischer, Carlsen, and
Kasparov where their Elo rating was below 2500 were omitted as this rating is
generally accepted as the threshold for grandmaster level play. I expanded the
dataset by downloading games from all significant tournaments, as well as
matches played by the top 20 players as per all-time Elo rankings.

I have written a script to isolate the first 20 moves of each game, then used
the Lichess Masters Database API to match these sequences with existing
records. I downloaded and separated those PGN files that were annotated, at
least partially, by Stockfish centipawn evaluations. I made up for the missing
evaluations (about 200) by running them through Stockfish 16 with a depth
setting of 20.

Interestingly, there were very few Bobby Fischer games in the Masters
Database. Therefore, I took the initiative to analyze as many of his classical
games as possible using Stockfish 16 with a depth of 20. The games of Fischer
and the corrected games are the only entries in my dataset that do not rely on
Lichess’s Stockfish centipawn evaluations.

Once the dataset grew to cover more than a million moves, I decided it was a
logical stopping point. This allowed me to avoid overusing computational
resources as well as the resources provided by Lichess, which is funded by
user donations.

###  6.2 Statistical Analysis

To understand the performance of these Grandmasters, I conducted a statistical
test known as the one-sided Mann-Whitney U test. The results of these tests
are presented in two tables.

| Carlsen | Anand | Kramnik | Topalov | Kasparov | Karpov | Fischer
---|---|---|---|---|---|---|---
Carlsen | - | 0.14 | 0.24 | 0.05 | 0.02 | 0.02 | 0.03
Anand | 0.35 | - | 0.66 | 0.21 | 0.14 | 0.13 | 0.11
Kramnik | 0.53 | 0.66 | - | 0.12 | 0.06 | 0.05 | 0.06
Topalov | 0.16 | 0.18 | 0.08 | - | 0.61 | 0.64 | 0.35
Kasparov | 0.07 | 0.10 | 0.04 | 0.53 | - | 0.46 | 0.19
Karpov | 0.08 | 0.11 | 0.05 | 0.57 | 0.51 | - | 0.19
Fischer | 0.01 | 0.02 | 0.01 | 0.08 | 0.07 | 0.08 | -
Table 4: P-values obtained from one-sided Mann-Whitney U tests. The values to
the right of the diagonal compare the $GI_{B}$ scores, while those to the left
compare the black player’s $GPL_{B}$ in world championships. Higher p-values
imply that we cannot reject the null hypothesis that the row player’s scores
are greater than the column player’s scores.

Table 4 displays the p-values resulting from the tests conducted on world
championships. For example, looking at the first row, the p-value of 0.14 in
the Carlsen-Anand comparison suggests that, when Carlsen is the black player,
we cannot confidently say that his $GPL_{B}$ score exceeds Anand’s score.

Similarly, a value of 0.24 in the Carlsen-Kramnik comparison implies that
there’s an insufficient statistical basis to assert that Carlsen’s $GI_{B}$
score is higher than Kramnik’s. These p-values provide a measure of the
statistical likelihood that a given player outperforms another, given our null
hypothesis.

| Carlsen | Kasparov | Fischer | Kramnik | Anand | Karpov | Topalov
---|---|---|---|---|---|---|---
Carlsen | - | 0.39 | 0.70 | 0.00 | 0.00 | 0.00 | 0.00
Kasparov | 0.09 | - | 0.73 | 0.00 | 0.00 | 0.00 | 0.00
Fischer | 0.00 | 0.00 | - | 0.00 | 0.00 | 0.00 | 0.00
Kramnik | 0.01 | 0.18 | 0.97 | - | 0.28 | 0.01 | 0.00
Anand | 0.00 | 0.07 | 0.93 | 0.30 | - | 0.03 | 0.00
Karpov | 0.00 | 0.00 | 0.02 | 0.00 | 0.00 | - | 0.00
Topalov | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.11 | -
Table 5: P-values obtained from one-sided Mann-Whitney U tests. Values to the
right of the diagonal are comparing GI scores, while those to the left are
comparing GPL scores. A higher p-value indicates we cannot reject the null
hypothesis that the row player’s values are greater than those of the column
player.

Table 5 presents a similar analysis for grandmasters (GMs). For instance, the
p-value of 0.39 in the Carlsen-Kasparov comparison tells us that we cannot
reject the hypothesis that Carlsen’s GI score is greater than Kasparov’s.
Similarly, a value of 0.70 in the Carlsen-Fischer comparison suggests a high
likelihood that Carlsen’s GPL score exceeds that of Fischer.

In essence, these tables provide an indication of the relative performance of
these Grandmasters, with a focus on comparing their gameplay when playing as
black (which is generally considered the more difficult side to play). The
results presented here underline the complexity of chess, and the nuances that
come with analyzing player performance based on gameplay data.

###  6.3 Extension: Refined Calculation of GI Score Using Human Win-Draw-Loss
Estimates

To have a more nuanced understanding of chess players’ performance, I have
calculated the GI score and GPL using estimated human win-draw-loss
probabilities rather than relying solely on engine vs. engine win-draw-loss
probabilities. This method leverages the well-known Elo rating system, which
quantifies players’ skill levels.

#### Elo Ratings as a Human Benchmark

The Elo rating system is used to rank players based on their past performance.
Given two players with Elo ratings $E_{i}$ and $E_{j}$, the expected score for
player $i$, represented by $ES_{i}$, $i\in\\{W,B\\}$ is computed using the
following formula:

$ES_{i}=\frac{1}{1+10^{(E_{j}-E_{i})/400}}.$

This formula allows the integration of win-draw-loss probabilities generated
by engines with probabilities based on estimated human performance. As a
result, the calculated probabilities are more refined, considering the actual
Elo ratings of the players and the centipawn evaluation of the position.

Let $c$ be the centipawn evaluation of the position.

$ES_{W}(c)=\begin{cases}ES_{W}+(1-ES_{W})\left(\frac{2}{1+e^{-0.00368208c}}-1\right)&\text{if
}c\geq 0\\\
1-ES_{B}-(1-ES_{B})\left(\frac{2}{1+e^{-0.00368208(-c)}}-1\right)&\text{if
}c<0.\end{cases}$

Expected score of Black is simply defined as follows: $ES_{B}(c)=1-ES_{W}(c)$.

I have adapted this formula from Lichess’s human win-draw-loss formula, which
does not include the expected score as a function of Elo ratings of the
players.

#### Application to Norway Chess 2023

Table 6 shows the application of this method to the Norway Chess 2023
tournament. The GI and GPL scores are calculated with respect to the estimated
human performance, providing a more relatable alternative to scores based
solely on the win-draw-loss probabilities derived from engine vs. engine
games.

Rank | Player | GI Score | GPL | TPR
---|---|---|---|---
1 | Nakamura | 3.36 | 0.29 | 2885
2 | So | 2.76 | 0.25 | 2798
3 | Caruana | 2.34 | 0.35 | 2842
4 | Giri | 2.28 | 0.30 | 2797
5 | Gukesh | 2.19 | 0.31 | 2801
6 | Mamedyarov | 1.39 | 0.35 | 2756
7 | Carlsen | 0.30 | 0.41 | 2699
8 | Abdusattorov | -0.95 | 0.49 | 2668
9 | Firouzja | -1.68 | 0.63 | 2707
10 | Tari | -2.47 | 0.55 | 2589
Table 6: Norway Chess 2023 results based on human win-draw-loss probabilities
that take into account the Elo ratings of the players. TPR refers to
tournament performance rating

This method not only enriches the understanding of player performance by
integrating human factors but also enhances the relevance and interpretability
of the metrics used. For more technical details on the methodology, please
refer to section 4.2.

###  6.4 Potential Limitations

The process of analyzing chess games through computational methods, while
powerful, has inherent limitations that need to be acknowledged. The following
sections detail some of the primary challenges and constraints associated with
the methodology employed in this study.

####  6.4.1 Incompleteness of the Dataset

While the dataset of world championship games is complete, including all games
played in world championships, the larger dataset that covers other games is
not as exhaustive. Specifically, approximately 83% of Fischer’s, 78% of
Kasparov’s, and 63% of Carlsen’s classical games are included.

Except for Fischer’s games, all of these were annotated with Stockfish
centipawn evaluations from the Lichess Masters Database. The choice of these
games was influenced by their perceived importance for inclusion and Stockfish
analysis in the Masters Database, which requires significant computational
resources. This selection process may introduce certain biases, as human users
decide which games to analyze or include in the Lichess database. Therefore,
the incompleteness and potential bias in the dataset must be considered when
interpreting the results.

####  6.4.2 Variability in Analysis

The games in the Lichess Masters Database have been analyzed at different
points in time, using different versions of Stockfish, and at varying depths.
Although I independently analyzed more than 100k moves and found the results
to be similar to those obtained by Lichess, this variability can still impact
the findings.

While it’s true that evaluations may change as engines become more
sophisticated, the analysis of human moves seems to remain relatively
consistent. This stability might be attributed to the fact that even older
engines were considerably stronger than the best human players. Nevertheless,
the lack of uniformity in analysis methods across the dataset could still be
seen as a limitation.

####  6.4.3 Ignoring Time in Evaluations

The engine centipawn evaluations employed in this study do not take into
account the remaining time on the clock when evaluating positions. Time is a
crucial element in both human and machine decision-making in chess,
influencing the choices players make during a game.

As more advanced engines are developed that incorporate time into their
evaluations, computing the GI score with respect to these engines would likely
present a more accurate picture of players’ decisions. However, the vast
majority of PGN files, especially older ones, do not report the time of each
move. This absence of time data restricts the ability to include this vital
factor in the analysis, potentially limiting the insights that can be drawn
regarding human decision-making under time constraints.

The limitations outlined above offer important context for the findings of
this study. While the methods employed provide valuable insights into chess
strategy and player performance, these constraints should be considered when
interpreting the results. Future research could explore ways to mitigate these
limitations, such as incorporating time in evaluations or expanding the
dataset to include a more comprehensive selection of games. By continuing to
refine the methodology, further progress can be made in understanding the
complex dynamics of chess from both human and computational perspectives.

##  7 Conclusions

In this paper, I have introduced a new framework for evaluating the
intelligence of players in $n$-person general-sum games with partial knowledge
that takes into account the interplay between human- and machine-based
decision-making. The Game Intelligence (GI) mechanism provides a practicable
way to assess the demonstrated intelligence of players in a game and can
potentially be applied to a wide range of games, from backgammon and poker to
sports like football, basketball, and tennis. Nonetheless, a limitation of the
GI methodology resides in its measurement of a player’s intelligence relative
to a machine’s intelligence (or vice versa). As such, it may not yet offer an
accurate assessment of individual or collective intelligence in specific
games, such as basketball, wherein no existing AI system attains human-level
intelligence.

To conclude, this paper’s potential applications extend beyond competitive
games and sports. The GI mechanism can also be used to evaluate the
intelligence of AI systems and organizations. As AI systems continue to
advance, the GI mechanism will only become more effective in providing a
reliable measure of intelligence in many $n$-person decision-making
environments.

## References

  * [1] Nejat Anbarci and Mehmet S Ismail.  AI-powered tiebreak mechanisms: An application to chess.  arXiv preprint arXiv:2210.08289, 2022.
  * [2] Haris Aziz, Florian Brandl, Felix Brandt, and Markus Brill.  On the tradeoff between efficiency and strategyproofness.  Games and Economic Behavior, 110:1–18, 2018.
  * [3] David J. Bartholomew.  Measuring Intelligence: Facts and Fallacies.  Cambridge University Press, 2004.
  * [4] Giacomo Bonanno.  A dynamic epistemic characterization of backward induction without counterfactuals.  Games and Economic Behavior, 78:31–43, March 2013.
  * [5] Giacomo Bonanno.  Rational Play in Extensive-Form Games.  Games, 13(6):72, October 2022.
  * [6] Nick Bostrom.  Superintelligence: Paths, Dangers, Strategies.  Oxford University Press, 2014.
  * [7] Steven J Brams, Mehmet S Ismail, D Marc Kilgour, and Walter Stromquist.  Catch-Up: A Rule That Makes Service Sports More Competitive.  The American Mathematical Monthly, 125(9):771–796, 2018.
  * [8] Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu.  Deep Blue.  Artificial Intelligence, 134(1-2):57–83, 2002.
  * [9] Edith Elkind and Helger Lipmaa.  Hybrid Voting Protocols and Hardness of Manipulation.  In Proceedings of the 16th International Conference on Algorithms and Computation, ISAAC’05, page 206–215, Berlin, Heidelberg, 2005\. Springer-Verlag.
  * [10] Ronald Fagin and Joseph Y. Halpern.  Belief, awareness, and limited reasoning.  Artificial Intelligence, 34(1):39–76, 1987.
  * [11] Piotr Faliszewski and Ariel D. Procaccia.  AI’s War on Manipulation: Are We Winning?  AI Magazine, 31(4):53–64, Sep. 2010.
  * [12] Ben Goertzel and Cassio Pennachin.  Artificial General Intelligence.  Springer, 2007.
  * [13] Matej Guid and Ivan Bratko.  Computer analysis of world chess champions.  ICGA Journal, 29(2):65–73, 2006.
  * [14] Joseph Y. Halpern and Leandro C. Rêgo.  Extensive games with possibly unaware players.  Mathematical Social Sciences, 70:42–58, 2014.
  * [15] Aviad Heifetz, Martin Meier, and Burkhard C. Schipper.  Interactive unawareness.  Journal of Economic Theory, 130(1):78–94, 2006.
  * [16] José Hernández-Orallo.  The Measure of All Minds: Evaluating Natural and Artificial Intelligence.  Cambridge University Press, 2017.
  * [17] José Hernández-Orallo and David L Dowe.  Measuring universal intelligence: Towards an anytime intelligence test.  Artificial Intelligence, 174(18):1508–1539, 2010.
  * [18] José Hernández-Orallo and Neus Minaya-Collado.  A Formal Definition of Intelligence Based on an Intensional Variant of Algorithmic Complexity.  In Proceedings of International Symposium of Engineering of Intelligent Systems (EIS98), pages 146–163, 1998.
  * [19] Marcus Hutter.  Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic.  SprinSpringer Berlin, Heidelberg, 2005.
  * [20] Mehmet S. Ismail.  One for All, All for One—Von Neumann, Wald, Rawls, and Pareto.  In Proceedings of the 21st ACM Conference on Economics and Computation, pages 763–764, Virtual Event Hungary, July 2020.
  * [21] Philippe Jehiel and Dov Samet.  Valuation equilibrium.  Theoretical Economics, 2(2):163–185, 2007.
  * [22] Steffen Könn, Juan Palacios, and Nico Pestel.  Indoor Air Quality and Cognitive Performance.  Management Science, 2023.
  * [23] Steffen Künn, Christian Seel, and Dainis Zegners.  Cognitive Performance in Remote Work: Evidence from Professional Chess.  The Economic Journal, 132(643):1218–1232, 2022.
  * [24] Shane Legg and Marcus Hutter.  Universal Intelligence: A Definition of Machine Intelligence.  Minds and Machines, 17:391–444, 2007.
  * [25] Shengwu Li.  Obviously Strategy-Proof Mechanisms.  American Economic Review, 107(11):3257–87, November 2017.
  * [26] Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson.  Aligning Superhuman AI with Human Behavior: Chess as a Model System.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1677–1687, Virtual Event CA USA, August 2020. ACM.
  * [27] Martin J Osborne and Ariel Rubinstein.  A Course in Game Theory.  MIT Press, Cambridge, MA, 1994.
  * [28] Marc Pauly.  Can strategizing in round-robin subtournaments be avoided?  Social Choice and Welfare, 43(1):29–46, 2013.
  * [29] Kenneth Regan and Guy Haworth.  Intrinsic Chess Ratings.  Proceedings of the AAAI Conference on Artificial Intelligence, 25(1):834–839, 2011.
  * [30] Ariel Rubinstein.  Modeling Bounded Rationality.  MIT Press, 1998.
  * [31] Tom Schaul, Julian Togelius, and Jürgen Schmidhuber.  Measuring Intelligence Through Games.  arXiv preprint arXiv:1109.1314, 2011.
  * [32] Burkhard C Schipper.  Unawareness—A gentle introduction to both the literature and the special issue.  Mathematical Social Sciences, 70:1–9, 2014.
  * [33] Herbert A. Simon.  A Behavioral Model of Rational Choice.  The Quarterly Journal of Economics, 69(1):99–118, 1955.
  * [34] Charles Spearman.  “General Intelligence,” Objectively Determined and Measured.  The American Journal of Psychology, 15(2):201–292, 1904.
  * [35] Anthony Strittmatter, Uwe Sunde, and Dainis Zegners.  Life cycle patterns of cognitive performance over the long run.  Proceedings of the National Academy of Sciences, 117(44):27255–27261, November 2020.
  * [36] Gerald Tesauro.  Temporal difference learning and TD-Gammon.  Communications of the ACM, 38(3):58–68, 1995.
  * [37] Alan M. Turing.  Computing Machinery and Intelligence.  Mind, 59(236):433–460, 1950.  Publisher: Oxford University Press, Mind Association.
  * [38] J. von Neumann and O. Morgenstern.  Theory of Games and Economic Behavior.  Princeton: Princeton University Press, 1953, 3rd edition, 1944.
  * [39] Dainis Zegners, Uwe Sunde, and Anthony Strittmatter.  Decisions and Performance Under Bounded Rationality: A Computational Benchmarking Approach.  arXiv preprint arXiv:2005.12638, 2020.

## Appendix

###  7.1 Other useful mechanisms

First, I define the aggregate game intelligence.

######  Definition 10 (Aggregate Game Intelligence).

Let $\boldsymbol{\bar{a}^{1},\bar{a}^{2},...,\bar{a}^{K}}$ where
$\boldsymbol{\bar{a}^{k}}\in\boldsymbol{\bar{A}}$ be a sequence of plays in
each of which $i$ is a player. Player $i$’s aggregate game intelligence (aGI)
is defined as

$\sum_{k=1}^{K}GI_{i}(\boldsymbol{\bar{a}^{k}}).$

The aGI is the sum of a player’s GI over a sequence of plays in which they
participated. This mechanism can be especially useful in games that involve
exogenous chance moves such as backgammon. In such games, aGI is obtained
after players repeatedly play the game from the same position. For example, in
backgammon, this is known as a rollout. Rollouts used to be a standard
practice to “evaluate” a position before strong computer programs emerged.
Now, rollouts are typically performed by an AI playing against itself.999In
some games, it may not be possible or feasible to play the game from the same
position over and over again. In that case, the following metric would be
useful. Generalized game intelligence (gGI) can be defined as
$gGI_{i}(\bar{a})=\alpha R_{i}(\bar{a})-\beta GPL_{i}(\bar{a})$, where
$(\alpha,\beta)\in[0,1]^{2}$. This metric could be useful when a game’s
outcome is too volatile, in which case $\alpha$ and $\beta$ can be calibrated.

I now introduce the concept of Average Game Point Loss (AGPL), which will be
referenced later.

######  Definition 11 (Average GPL).

Let $\bar{a}\in\bar{A}$ be a play and $a^{j}_{i}$ be player $i$’s action at
position $x_{k}$ in play $\bar{a}$, where
$\bar{a}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l_{i}}_{i})$ for some $l_{i}$. Then,
player $i$’s average game point loss (AGPL) is defined as,

$AGPL_{i}(\bar{a})=\frac{1}{l_{i}}GPL_{i}(\bar{a}).$

Next, I define three additional metrics based on GI that allow for the
comparison of intelligence scores between different players across various
games.101010As noted earlier, it is impossible to make a perfect comparison of
game intelligence across different games. Each of these three metrics
possesses distinct strengths and weaknesses when making such comparisons.

######  Definition 12 (Relative Game Intelligence).

Let $GI_{max}$ and $GI_{min}$ denote the maximum and the minimum GI that
player $i$ can attain in game $G$, respectively, with respect to reference
machine $M$. Then, player $i$’s relative game intelligence (RGI) given play
$\bar{a}\in\bar{A}$ is defined as follows:

$RGI_{i}(\bar{a})=\frac{GI_{i}(\bar{a})-GI_{min}}{GI_{max}-GI_{min}}.$

The RGI measures a player’s intelligence in a game relative to the maximum
possible intelligence score determined by reference machine $M$.

######  Definition 13 (RGI Percentile).

Let $\\{RGI_{1},RGI_{2},...,RGI_{m}\\}$ be the set of all RGIs for a
population of $m\geq 2$ players who played game $G$ and
$|\\{j\leavevmode\nobreak\ |\leavevmode\nobreak\ RGI_{j}\geq RGI_{i}\\}|$ be
the number of players whose RGIs are greater than or equal to $RGI_{i}$. The
RGI percentile $RGIP_{i}$ for player $i$, $i\leq m$, is defined as follows.

$RGIP_{i}=\frac{100(m-|\\{j\leavevmode\nobreak\ |\leavevmode\nobreak\
RGI_{j}\geq RGI_{i}\\}|)}{m}.$

The Relative Game Intelligence Percentile (RGIP), is a metric that represents
a player’s standing within a given population based on their RGI. It provides
a way to compare a player’s game intelligence against others who participated
in the same game. Additionally, RGIP can be utilized to compare the relative
game intelligence of players across different games. However, as noted
earlier, such comparisons should be approached with caution and taken with a
grain of salt.

######  Definition 14 (Accuracy).

Let $\bar{a}\in\bar{A}$ be a play and $a^{j}_{i}$ be player $i$’s action at
position $x_{k}$ in play $\bar{a}$, where
$\bar{a}_{i}=(a^{1}_{i},a^{2}_{i},...,a^{l_{i}}_{i})$ for some $l_{i}$. Then,
player $i$’s accuracy is defined as,

$Accuracy_{i}(\bar{a})=\frac{1}{l_{i}}\sum_{k=1}^{l_{i}}\Bigl{(}\frac{\sum^{m}_{j=1}r_{j}M_{i}(a^{k}_{i}(x_{k}))(r_{j})}{\sum^{m}_{j=1}r_{j}M_{i}(a^{*}_{i}(x_{k}))(r_{j})}\Bigl{)}.$

The accuracy metric is a measure of a player’s intelligence in a game based on
how often they make the best move according to the machine’s evaluation.
Specifically, it is the average ratio of the actual expected game points
obtained by the player to the maximum expected game points that could have
been obtained by always playing the machine’s best move at each node. This
metric is calculated for each move a player makes in a play, and then averaged
over all the moves in the play to give a measure of the player’s overall
accuracy. A higher accuracy score indicates that the player made moves that
were closer to optimal, according to the machine’s evaluation.

###  7.2 Tables

Player | GI | $\text{SD}_{GI}$ | $\text{GI}_{W}$ | $\text{GI}_{B}$ | GPL | $\text{GPL}_{W}$ | $\text{GPL}_{B}$ |  $\\#$Games |  $\\#$Moves
---|---|---|---|---|---|---|---|---|---
Carlsen | 0.17 | 0.58 | 0.14 | 0.19 | 0.41 | 0.46 | 0.36 | 56 | 2958
Anand | 0.09 | 0.63 | 0.19 | 0 | 0.4 | 0.37 | 0.43 | 88 | 3323
Kramnik | 0 | 0.78 | 0 | 0 | 0.52 | 0.58 | 0.47 | 65 | 2488
Topalov | 0 | 0.70 | 0.15 | -0.16 | 0.46 | 0.39 | 0.53 | 23 | 1200
Kasparov | 0 | 0.78 | 0.14 | -0.14 | 0.52 | 0.46 | 0.58 | 197 | 7076
Karpov | -0.07 | 0.76 | 0.05 | -0.19 | 0.58 | 0.53 | 0.63 | 194 | 7929
Liren | -0.11 | 0.70 | -0.15 | -0.07 | 0.61 | 0.79 | 0.43 | 14 | 672
Capablanca | -0.15 | 0.83 | -0.21 | -0.09 | 0.66 | 0.75 | 0.57 | 48 | 1921
Fischer | -0.24 | 0.90 | -0.09 | -0.36 | 0.86 | 0.76 | 0.95 | 20 | 866
Petrosian | -0.37 | 0.79 | -0.34 | -0.4 | 0.88 | 0.92 | 0.84 | 69 | 3079
Smyslov | -0.37 | 0.90 | -0.31 | -0.44 | 0.89 | 0.91 | 0.87 | 77 | 3437
Alekhine | -0.42 | 0.96 | -0.22 | -0.62 | 0.99 | 0.87 | 1.11 | 140 | 6267
Spassky | -0.47 | 0.93 | -0.35 | -0.6 | 0.94 | 0.88 | 1 | 67 | 3064
Botvinnik | -0.47 | 0.92 | -0.31 | -0.64 | 0.98 | 0.89 | 1.07 | 169 | 7818
Lasker | -0.54 | 0.92 | -0.44 | -0.65 | 1.19 | 1.15 | 1.23 | 102 | 4835
Tal | -0.63 | 0.89 | -0.49 | -0.76 | 1.11 | 1.11 | 1.12 | 42 | 1906
Euwe | -0.72 | 1.14 | -0.54 | -0.87 | 1.13 | 1.12 | 1.13 | 63 | 2709
Steinitz | -0.8 | 1.11 | -0.48 | -1.12 | 1.3 | 1.07 | 1.52 | 115 | 4769
Table 7: Summary statistics of world chess champions 1886–2023. Means of GI, GPL, and the standard deviation (SD) of GI are presented. Player | GI | $\text{SD}_{GI}$ | $\text{GI}_{W}$ | $\text{GI}_{B}$ | GPL | $\text{GPL}_{W}$ | $\text{GPL}_{B}$ |  $\\#$Games |  $\\#$Moves
---|---|---|---|---|---|---|---|---|---
Fischer | -0.02 | 0.84 | 0.05 | -0.08 | 0.75 | 0.73 | 0.76 | 640 | 26,669
Carlsen | -0.03 | 0.90 | 0.00 | -0.05 | 0.66 | 0.70 | 0.63 | 1,187 | 57,422
Kasparov | -0.03 | 0.86 | 0.03 | -0.09 | 0.71 | 0.70 | 0.71 | 1,188 | 46,103
Karjakin | -0.12 | 0.80 | 0.00 | -0.25 | 0.66 | 0.62 | 0.70 | 493 | 23,388
Aronian | -0.13 | 0.78 | -0.10 | -0.16 | 0.67 | 0.70 | 0.63 | 641 | 30,264
Kramnik | -0.13 | 0.83 | -0.10 | -0.16 | 0.74 | 0.79 | 0.70 | 1,250 | 56,758
Anand | -0.14 | 0.94 | -0.12 | -0.17 | 0.75 | 0.78 | 0.72 | 1,529 | 64,173
Giri | -0.19 | 0.79 | -0.16 | -0.22 | 0.70 | 0.73 | 0.68 | 440 | 19,963
Leko | -0.19 | 0.78 | -0.13 | -0.26 | 0.72 | 0.71 | 0.73 | 443 | 20,170
Mamedyarov | -0.20 | 0.89 | -0.03 | -0.37 | 0.72 | 0.63 | 0.81 | 437 | 18,230
Karpov | -0.21 | 0.87 | -0.16 | -0.26 | 0.87 | 0.89 | 0.83 | 1,448 | 72,154
Svidler | -0.21 | 0.89 | -0.10 | -0.33 | 0.71 | 0.67 | 0.75 | 453 | 17,484
Ivanchuk | -0.25 | 0.80 | -0.20 | -0.29 | 0.79 | 0.78 | 0.81 | 718 | 29,450
Nakamura | -0.26 | 0.85 | -0.09 | -0.44 | 0.79 | 0.68 | 0.91 | 467 | 21,807
Gelfand | -0.27 | 0.79 | -0.23 | -0.31 | 0.79 | 0.81 | 0.78 | 559 | 23,236
Caruana | -0.28 | 0.92 | -0.22 | -0.35 | 0.85 | 0.83 | 0.87 | 559 | 26,851
Grischuk | -0.28 | 0.81 | -0.15 | -0.41 | 0.81 | 0.79 | 0.82 | 401 | 18,496
Adams | -0.32 | 0.80 | -0.19 | -0.46 | 0.82 | 0.77 | 0.86 | 452 | 20,105
Topalov | -0.39 | 0.83 | -0.24 | -0.53 | 0.90 | 0.80 | 0.98 | 695 | 30,015
Shirov | -0.40 | 0.97 | -0.33 | -0.48 | 0.90 | 0.90 | 0.91 | 540 | 23,051
Short | -0.52 | 0.96 | -0.36 | -0.73 | 0.99 | 0.90 | 1.12 | 400 | 18,479
Table 8: Summary statistics of ‘super’ grandmasters. Means of GI, GPL, and the standard deviation (SD) of GI are presented. Player | GI | $\text{GI}_{W}$ | $\text{GI}_{B}$ | GPL | $\text{GPL}_{W}$ | $\text{GPL}_{B}$
---|---|---|---|---|---|---
Carlsen | 0.23 | 0.25 | 0.20 | 0.47 | 0.49 | 0.45
Kasparov | 0.21 | 0.27 | 0.18 | 0.54 | 0.49 | 0.58
Fischer | 0.20 | 0.28 | 0.14 | 0.61 | 0.61 | 0.62
Kramnik | 0.16 | 0.17 | 0.15 | 0.55 | 0.57 | 0.51
Karjakin | 0.15 | 0.26 | 0.02 | 0.48 | 0.44 | 0.51
Anand | 0.14 | 0.15 | 0.13 | 0.57 | 0.58 | 0.54
Aronian | 0.12 | 0.19 | 0.05 | 0.52 | 0.50 | 0.54
Giri | 0.09 | 0.11 | 0.08 | 0.54 | 0.57 | 0.53
Grischuk | 0.08 | 0.15 | -0.18 | 0.62 | 0.54 | 0.67
Mamedyarov | 0.07 | 0.30 | -0.18 | 0.54 | 0.39 | 0.67
Karpov | 0.06 | 0.10 | 0.00 | 0.68 | 0.69 | 0.65
Leko | 0.04 | 0.06 | 0.02 | 0.55 | 0.56 | 0.55
Nakamura | 0.02 | 0.23 | -0.19 | 0.62 | 0.48 | 0.75
Gelfand | 0.01 | 0.05 | -0.03 | 0.59 | 0.70 | 0.56
Svidler | -0.01 | 0.12 | -0.21 | 0.56 | 0.47 | 0.64
Caruana | -0.02 | 0.05 | -0.10 | 0.62 | 0.61 | 0.63
Ivanchuk | -0.04 | -0.01 | -0.11 | 0.66 | 0.64 | 0.68
Adams | -0.11 | 0.10 | -0.34 | 0.66 | 0.58 | 0.76
Topalov | -0.18 | 0.08 | -0.43 | 0.74 | 0.60 | 0.82
Short | -0.35 | -0.21 | -0.75 | 0.87 | 0.81 | 1.02
Shirov | -0.37 | -0.19 | -0.50 | 0.75 | 0.80 | 0.72
Table 9: Median values of the metrics of super granmasters Player | GI | $\text{GI}_{W}$ | $\text{GI}_{B}$ | GPL | $\text{GPL}_{W}$ | $\text{GPL}_{B}$
---|---|---|---|---|---|---
Carlsen | 0.35 | 0.40 | 0.33 | 0.20 | 0.23 | 0.20
Anand | 0.32 | 0.33 | 0.30 | 0.19 | 0.19 | 0.32
Kramnik | 0.26 | 0.16 | 0.32 | 0.31 | 0.35 | 0.18
Capablanca | 0.22 | 0.19 | 0.22 | 0.36 | 0.40 | 0.30
Topalov | 0.19 | 0.35 | -0.33 | 0.32 | 0.30 | 0.39
Kasparov | 0.18 | 0.30 | 0.08 | 0.39 | 0.33 | 0.43
Karpov | 0.14 | 0.18 | 0.07 | 0.39 | 0.40 | 0.39
Liren | -0.03 | -0.17 | 0.08 | 0.43 | 0.45 | 0.42
Fischer | -0.12 | -0.03 | -0.38 | 0.69 | 0.57 | 0.88
Alekhine | -0.18 | -0.04 | -0.57 | 0.77 | 0.65 | 0.84
Petrosian | -0.22 | -0.22 | -0.22 | 0.71 | 0.77 | 0.69
Spassky | -0.23 | -0.09 | -0.65 | 0.73 | 0.67 | 0.84
Botvinnik | -0.33 | -0.06 | -0.52 | 0.88 | 0.74 | 0.92
Smyslov | -0.33 | 0.10 | -0.51 | 0.78 | 0.58 | 0.87
Lasker | -0.44 | -0.20 | -0.65 | 1.12 | 1.04 | 1.21
Tal | -0.59 | -0.30 | -0.75 | 1.08 | 1.04 | 1.13
Euwe | -0.68 | -0.33 | -0.87 | 1.01 | 0.61 | 1.02
Steinitz | -0.78 | -0.33 | -1.02 | 1.08 | 0.97 | 1.21
Table 10: Median values of the metrics of World Chess Champions