Do LLM Agents  Have Regret? A Case Study in Online Learning and Games
Under review as a conference paper at ICLR 2025

DO LLM AGENTS HAVE REGRET?
A CASE STUDY IN ONLINE LEARNING AND GAMES

Anonymous authors
Paper under double-blind review

ABSTRACT

Large language models (LLMs) have been increasingly employed for (interac-
tive) decision-making, via the development of LLM-based autonomous agents.
Despite their emerging successes, the performance of LLM agents in decision-
making has not been fully investigated through quantitative metrics, especially
in the multi-agent setting when they interact with each other, a typical scenario
in real-world LLM-agent applications. To better understand the limits of LLM
agents in these interactive environments, we propose to study their interactions in
benchmark decision-making settings in online learning and game theory, through
the performance metric of regret. We first empirically study the no-regret behav-
iors of LLMs in canonical non-stochastic online learning problems, as well as
the emergence of equilibria when LLM agents interact through playing repeated
games. We then provide some theoretical insights into the sublinear regret growth
in the cases we observed, under certain assumptions on the supervised pre-training
and the rationality model of human decision-makers who generate the data. No-
tably, we also identify (simple) cases where advanced LLMs such as GPT-4 fail
to be no-regret. To further promote the no-regret behaviors, we propose a novel
unsupervised training loss of regret-loss, which, in contrast to the supervised pre-
training loss, does not require the labels of (optimal) actions. Finally, we establish
the statistical guarantee of generalization bound for regret-loss minimization, and
more importantly, the optimization guarantee that minimizing such a loss may au-
tomatically lead to known no-regret learning algorithms, when single-layer self-
attention models are used. Our further experiments demonstrate the effectiveness
of our regret-loss, especially in addressing the above “regrettable” cases.

1

INTRODUCTION

Large language models (LLMs) have recently exhibited remarkable emerging capabilities (Bubeck
et al., 2023; Achiam et al., 2023; Wei et al., 2022b; Yao et al., 2023a). As a consequence, a bur-
geoning body of work has been investigating the employment of LLMs as central controllers for (in-
teractive) decision-making, through the construction of LLM-based autonomous agents (Hao et al.,
2023; Shen et al., 2023; Yao et al., 2023b; Shinn et al., 2023; Wang et al., 2023d; Significant Gravi-
tas, 2023). Specifically, the LLM agent interacts with the (physical) world in a dynamic/sequential
way: it uses LLMs as an oracle for reasoning and planning, then acts in the environment based on
the reasoning/planning and the feedback it perceives over time. LLM agent has achieved impressive
successes in embodied AI (Ahn et al., 2022; Huang et al., 2022a; Wang et al., 2023a), natural science
(Wu et al., 2023a; Swan et al., 2023), and social science (Park et al., 2022; 2023) applications.
Besides being dynamic, another increasingly captivating feature of LLM-based decision-making is
the involvement of strategic interactions, oftentimes among multiple LLM agents. For example, it
has been continually reported that the reasoning capability of LLMs can be improved by interacting
with each other through negotiation and/or debate games (Fu et al., 2023; Du et al., 2023); LLM
agents have now been widely used to simulate the strategic behaviors for social and economic stud-
ies, to understand the emerging behaviors in interactive social systems (Aher et al., 2023; Park et al.,
2023). Moreover, LLMs have also exhibited remarkable potential in solving various games (Bakhtin
et al., 2022; Mukobi et al., 2023), and in fact, a rapidly expanding literature has employed repeated
games as a fundamental benchmark to understand the strategic behaviors of LLMs (Brookins &
DeBacker, 2023; Akata et al., 2023; Fan et al., 2023). These exciting empirical successes call for a
rigorous examination and understanding through a theoretical lens of decision-making.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

Regret, on the other hand, has been a core metric in (online) decision-making. It measures how
“sorry” the decision-maker is, in retrospect, not to have followed the best prediction in hindsight
(Shalev-Shwartz, 2012). It provides not only a sensible way to evaluate the sophistication level of
online decision-makers, but also a quantitative way to measure their robustness against arbitrary (and
possibly adversarial) environments. More importantly, it inherently offers a connection to modeling
and analyzing strategic behaviors: the long-run interaction of no-regret learners leads to certain
equilibrium when they repeatedly play games (Cesa-Bianchi & Lugosi, 2006). In fact, no-regret
learning has served as a natural model for predicting and explaining human behaviors in strategic
decision-making, with experimental evidence (Erev & Roth, 1998; Nekipelov et al., 2015; Balseiro
& Gur, 2019). It has thus been posited as an important model of “rational behavior” in playing
games (Blum et al., 2008; Roughgarden, 2015). Hence, it is natural to ask:

Can we examine and better understand the online and strategic decision-making
behaviors of LLMs through the lens of regret?
Acknowledging that LLM(-agents) are extremely complicated to analyze, to gain some insights into
the question, we focus on benchmark decision-making settings: online learning with convex (linear)
loss functions, and playing repeated games. We defer a detailed literature review to Appendix A,
and summarize our contributions as follows.
Contributions. First, we carefully examine the performance of several representative pre-trained
LLMs in the aforementioned benchmark online decision-making settings, in terms of regret. We
observe that LLM agents can achieve regret sublinear in time in (non-stochastic) online learning
settings, where the loss functions change over time either arbitrarily (and even adversarially), or by
following some patterns with bounded variation, and in playing both representative and randomly
generated repeated games. For the latter, equilibria will emerge as the long-term behavior of the
multi-LLM interactions. Second, we provide some theoretical insights into the sublinear regret ob-
served in most online learning instances, based on some hypothetical model of the human decision-
makers who generate the data, and certain assumptions on the supervised pre-training procedure, a
common practice in training large models for decision-making. In particular, we make a connection
of the pre-trained LLMs to the known no-regret algorithm of follow-the-perturbed-leader (FTPL)
under such assumptions. Third, we also identify (simple) cases where advanced LLMs such as
GPT-4 fail to be no-regret. We thus propose a novel unsupervised training loss, regret-loss, which,
in contrast to the supervised pre-training loss, does not require the labels of (optimal) actions. We
then establish both statistical and optimization guarantees for regret-loss minimization, which, in
particular, show that minimizing such a loss can automatically lead to the known no-regret learning
algorithm of follow-the-regularized leader (FTRL), under single-layer self-attention parameteriza-
tion. Our further experiments demonstrate the effectiveness of our new loss, especially in addressing
the above “regrettable” cases. With the fast development of LLMs, we emphasize that our goal is not
to assert whether (current) LLMs are no-regret learners or not, especially given both the positive and
negative observations above. Instead, our hope is to introduce and inspire more rigorous metrics and
principles into the current evaluation and development of LLM agents, for online and multi-agent
strategic decision-making.

2 PRELIMINARIES
Notation. For a finite set
R
OOOd×d and Id×d as the d
×
integer d, we define [d] =
{
Rd
B(p, R,

0
}

, we use ∆(

) to denote the simplex over

. We denote R+ =

x

|

{

x

x

(cid:80)

S

S

S

∈

}
(cid:17)

≥

⊆

−
(cid:16)

∥ · ∥

) :=

x

∈

. For p
R

Rd, R > 0 and C

}
∥ ≤
exi
i∈[d] exi

1, 2, . . . , d
p
x
| ∥
{
Rd, define Softmax(x) =

∈
. We define 000d and 111d as the d-dimensional all-zero and all-one vector, respectively, and
d-dimensional zero matrix and identity matrix, respectively. For a positive
Rd being a convex set, define
x
p
. For any
∥
−
∥p to denote its
is true,
E

∈
and ProjC,∥·∥(p) = arg minx∈C ∥
Rn, we use
v
∥

∈
denoting the L2-norm by default. We define 1(

) = 1 if some event
) = 0 otherwise. For a random variable X, we use supp(X) to denote its support.

Lp-norm, with
and 1(
E
2.1 ONLINE LEARNING & GAMES
Online learning. We consider the online learning setting where an agent interacts with the envi-
ronment for T rounds, by iteratively making decisions based on the feedback she receives. Specif-
Π for some bounded domain
ically, at each time step t, the agent chooses her decision policy πt ∈
Π, and after her commitment to πt, a bounded loss function ft : Π
B, B] for some constant
→
B > 0 is chosen by the environment, potentially in an adversarial fashion. The agent thus incurs a
loss of ft(πt), and will update her decision to πt+1 using the feedback. We focus on the most basic

. For a vector v

[
−

v
∥

i∈[d]

E

∥

2

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

A

A

A

πt(
·

, i.e., Π = ∆(
.
|A|

for some loss vector ℓt ∈

Rd that may change over time, where d :=

setting where the agent chooses actions from a finite set
every round, which is also referred to as
the Experts Problem (Littlestone & Warmuth, 1994; Hazan, 2016), without loss of much generality
(c.f. Appendix B.5 for a discussion). In this case, Π becomes the simplex over
),
ℓt, πt⟩
and ft(πt) =
⟨
[T ], the agent may receive either the full vector ℓt, or only the realized loss ℓtat (we
At time step t
∈
), as
sometimes also interchangeably write it as ℓt(at)), the atth element of ℓt, for some at ∼
feedback, which will be referred to as online learning with full-information feedback, and that with
bandit feedback, respectively. The latter is also referred to as the adversarial/non-stochastic bandit
problem in the multi-armed bandit (MAB) literature. Note that hereafter, we will by default refer
to this setting that does not make any assumptions on the loss sequence (ℓt)t∈[T ] simply as online
learning. Moreover, if the loss functions change over time (usually with certain bounded variation),
we will refer to it as non-stationary online learning for short, whose bandit-feedback version is also
referred to as the non-stationary bandit problem.
Repeated games. The online learning setting above has an intimate connection to game the-
, where N is the number of
ory. Consider a normal-form game
An and rn :
B, B] are the action set and the payoff function
players,
of player n, respectively. The N players repeatedly play the game for T rounds, each player n
). The joint action
maintains a strategy πn,t ∈
at = (a1,t,
rn(at)
}n∈[N ]. From a single-
{
· · ·
player’s (e.g., player n’s) perspective, she encounters an online learning problem with (expected)
loss function ℓt :=
n denotes the index for all the
players other than player n. We will refer to it as the game setting for short, and use the terms of
“agent” and “player” interchangeably hereafter. The key difference between online learning and re-
peated games is in their interaction dynamics: online learning involves an agent facing a potentially
adversarial, changing environment (or sequence of loss functions), while in repeated games, agents
interact by playing the same game repeatedly, which might be less adversarial when they follow
specific learning algorithms.
2.2 PERFORMANCE METRIC: REGRET
We now introduce regret, the core performance metric used in online learning and games. For a given
algorithm A , let πA ,t denote the decision policy of the agent at time t generated by A . Then, the
regret, which is the difference between the accumulated (expected) loss incurred by implementing
A and that incurred by the best-in-hindsight fixed decision, can be defined as

Ea−n,t∼π−n,t[rn(
, a−n,t)] at time t, where
·

An) at time t, and takes action an,t ∼

N,
⟨
A1 × · · · × AN →

, aN,t) determines the payoff of each player at time t,

{An}n∈[N ],
[
−

rn}n∈[N ]⟩
{

πn,t(
·

∆(

−

−

=

G

T
(cid:88)

T
(cid:88)

−

∼

t=1

t=1

(cid:80)T

ℓt, π

t=1⟨

t=1⟨

inf
π∈Π

ft(π).

RegretA

ft(πA ,t)

(cid:0)(ft)t∈[T ]

(cid:1) :=
In the Experts Problem, the definition is instantiated as RegretA ((ℓt)t∈[T ]) := (cid:80)T
ℓt, πA ,t⟩ −
inf π∈Π
. With bandit-feedback, a commonly used metric may also take further expecta-
⟩
tion for RegretA , over the randomness of the policies (πA ,t)t∈[T ]. An algorithm A is referred to as
being no-regret, if max(ft)t∈[T ] RegretA ((ft)t∈[T ])
o(T ), i.e., the worse-case regret grows sub-
linearly in T . Known no-regret algorithms include follow-the-regularized-leader (Shalev-Shwartz
& Singer, 2007), follow-the-perturbed-leader (Kalai & Vempala, 2005) (See Appendix B.4 for more
details).
In non-stationary online learning, one may also use the metric of dynamic regret (Zinkevich, 2003),
where the comparator in the definition also changes over time, as the best decision policy at each
time t: D-RegretA ((ft)t∈[T ]) := (cid:80)T
t=1 inf π∈Π ft(π), which is a stronger notion
−
than RegretA ((ft)t∈[T ]) in that RegretA ((ft)t∈[T ])
D-RegretA ((ft)t∈[T ]).
3 DO PRE-TRAINED LLMS HAVE REGRET? EXPERIMENTAL VALIDATION
In this section, we explore the no-regret behaviors of representative LLMs (i.e., mainly GPT-4 Turbo
and GPT-4, together with GPT-3.5 Turbo, Mixtral-8x7b-instruct, and Llama-3-70B-instruct), in the
context of online learning and games. All experiments with LLMs are conducted using the public
OpenAI (Openai, 2023) or LLM Engine (LLM Engine, 2023) Python API. We provided some hy-
pothetical intuition as to why pre-trained LLM might be no-regret in Appendix C.1, which will be
made concrete next.
Interaction protocol. To enable the sequential interaction with LLMs, we first describe the setup
and objective of our experimental study. At each round, we incorporate the entire history of loss vec-
tors of past interactions into our prompts, as concatenated texts, and ask the LLM agent to determine
a policy that guides the decision-making for the next round. Note that since we hope to evaluate the

t=1 ft(πA ,t)

(cid:80)T

≤

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

sophistication level of pre-trained LLMs through online learning or games, we only provide simple
prompts that she should utilize the history information, without providing explicit rules of how to
make use of the history information, nor asking her to minimize regret (in any sense). A detailed
description and an ablation study of the prompts are deferred to Appendix C.8, and an illustration of
the protocol for playing repeated games is given in Figure C.1.
3.1 FRAMEWORK FOR SUBLINEAR REGRET BEHAVIOR VALIDATION
Before delving into the results, we note that to the best of our knowledge, we are not aware of
any principled framework for validating sublinear growth of the regret with finite-time experimental
data. Therefore, we propose two frameworks below to rigorously validate the no-regret behavior
of algorithms over a finite T , which might be of independent interest. More details are deferred to
Appendix C.3.
Trend-checking framework. We propose a statistical hypothesis test aligned with our objectives:
(cid:1) /t(cid:1)
H0 : The sequence (cid:0)RegretA
(cid:1) /t(cid:1)
H1 : The sequence (cid:0)RegretA
(cid:1) /t approaches zero (or a negative value) as t goes
Ideally, one should check if RegretA
to infinity. With a finite T value, testing these hypotheses provides a method to quantify this –
whether we reject H0 offers a way to measure it. To this end, one needs to count the number
(cid:1) /(t + 1) > 0, for which we use Proposition 1
(cid:0)(fτ )τ ∈[t+1]
of RegretA
below. We will report the p-value of H0, denoted as ptrend, as the output of this framework.
Proposition 1. (p-value of the null hypothesis). Define the event
(cid:0)(fτ )τ ∈[t]
t

t∈[T ] does not exhibit a decreasing pattern
t∈[T ] shows a decreasing pattern.

(cid:0)(fτ )τ ∈[t]
(cid:0)(fτ )τ ∈[t]
(cid:0)(fτ )τ ∈[t]

(cid:0)(fτ )τ ∈[t+1]
t + 1

> 0 for t = 1, . . . , T is at least s ≥

(cid:0)(fτ )τ ∈[t]

T − 1
2

RegretA

The number of

E(s, T ) :=

(cid:1) /t

RegretA

RegretA

−

(cid:41)

(cid:40)

−

(cid:1)

(cid:1)

.

.

≤

−
t

(cid:1)(cid:1)(cid:9)

(cid:18)T

1
2T −1

(s, T ))

(cid:80)T −1
t=s

(cid:0)(fτ )τ ∈[t]

(cid:0)(fτ )τ ∈[t]

(cid:0)(fτ )τ ∈[t]

(cid:0)(fτ )τ ∈[t]

Under the assumption that the null hypothesis H0 holds, the probability of this event happening is
(cid:19)
1
bounded as PH0 (
E
Regression-based framework. We propose an alternative approach by fitting the data with re-
gression. In particular, one can use the data (cid:8)(cid:0)t, log RegretA
t∈[T ] to fit a function
g(t) = β0 log t + β1, where the estimate of β0, i.e., (cid:98)β0, satisfying (cid:98)β0 < 1 may be used to indicate the
(cid:1) over time. While being simple,
no-regret behavior, i.e., the sublinear growth of RegretA
(cid:1) as
it cannot be directly used when RegretA
10 if this happens. We define preg as the p-value of the regression parameter (cid:98)β0, and will report

(cid:1) < 0. Hence, we set log RegretA

−
the pair of ( (cid:98)β0, preg) as the output of this framework.
3.2 RESULTS: ONLINE LEARNING
We now present the experimental results of pre-trained LLMs in online learning in: 1) (arbitrar-
ily) changing environments, 2) non-stationary environments, and 3) bandit-feedback environments.
Results for 2) and 3) are deferred to Appendices C.4 and C.5.
Changing environments. We first consider the setting with (arbitrarily) changing environments,
which are instantiated as follows: 1) Randomly-generated loss sequences. At every timestep,
d
we generate a random loss vector ℓt ∼
xi, yi ∼
xi, yi}
i=1[min
{
{
×
(µµµd, I) with clipping to [0, 10] to ensure boundedness of the loss, where
}i∈[d] or ℓt ∼ N
Unif(0, 10)
Unif([0, 10]d). Note that we use this as a way to systematically generate potentially arbitrary
µµµd ∼
loss sequences, and also note that our regret was defined for each realization of the random loss
vectors (instead of their expectations as in the definition of regret in stochastic bandit problems),
which can be arbitrarily different across timesteps. 2) Loss sequences with certain trends. Although
many real-world environments may change, they often change following certain patterns. There-
fore, we consider two representative trends, the linear and periodic (sinusoid) trend. We sample
a, b
T + a for the linear trend and ℓt = 5(1 + sin(at + b))
for the periodic trend. In the experiments, we choose d = 2. The average regret (over multiple
randomly generated instances) performance is presented in Figure 3.11, where we compare GPT-4
with well-known no-regret algorithms, FTRL with entropy regularization and FTPL with Gaussian
perturbations (with tuned parameters). It is seen that these pre-trained LLMs can achieve sublinear
regret in a large portion of the instances, and have sometimes even lower regret values than baselines.
1We emphasize that the error bars in the figures are not associated with the randomness/variance of the

Unif([0, 10]d) and let ℓt = (b

xi, yi}
{

]) for

Unif(

, max

a) t

−

∼

algorithms/LLM-agents, but with the randomness/variance of the generation of environment instances.

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 3.1: Regret of pre-trained LLMs for online learning with full-information feedback. Notably,
both commercial and open-source LLMs can achieve sublinear regret as validated by our frameworks
and the comparison with FTRL/FTPL, though the performances of weaker models of GPT-3.5 and
open-source ones are worse. Interestingly, the GPT-4 model can even outperform well-known no-
regret learning algorithms, FTRL and FTPL.

Figure 3.2: Regret of pre-trained LLMs for online learning with full-information feedback, with
longer horizons of T = 100 and T = 200. In most cases, the LLMs can achieve sublinear regret as
validated by our frameworks and the comparison with FTRL/FTPL, though the performances of the
weaker models of GPT-3.5 is worse.

Behavioral patterns of LLMs. To understand how LLMs make decisions at each time step, we
provided example outputs of LLMs explaining how they generate their policies in Appendix C.10.
We find LLMs tend to use the history of the reward vectors by looking at their sum/average, and
tend to introduce randomization in decision-making. These are known to be the keys to achieving
no-regret behaviors in online learning (Hazan, 2016; Cesa-Bianchi & Lugosi, 2006).

Longer-horizon results. We also test the robustness and scalability of our empirical findings in
more challenging environments. We extend the problem horizon to T = 100 for the two settings
where loss vectors are generated in a stationary way (i.e., Uniform and Gaussian), and T = 200 for
the other two more challenging, non-stationary settings (i.e., Linear-trend and Sine-trend). Note that
since in each round, we need to feed all the previous history to the LLMs, the API costs in fact scale
quadratically with respect to the horizon T . Therefore, we replace GPT-4 by its cheaper (and more
recent) version of GPT-4o. To further scale to even longer-horizon cases T = 500, we summarize
the history to reduce the prompt length by providing the LLM with the summation of history loss
associated with each action. Similar summary-based input was also used in the concurrent work
Krishnamurthy et al. (2024), where both the averaged reward and the action selection count of each
action were summarized for the (i.i.d.) stochastic bandit setting. The corresponding results are
provided in Figure 3.2 and Table 1, where the LLMs still exhibit no-regret behaviors as validated by
our frameworks and the comparison with FTRL/FTPL.

(ptrend, (cid:98)βo, preg)

GPT-4o

FTRL

FTPL

Uniform
Gaussian
Linear-trend
Sine-trend

(0.0, 0.85, 0.0)
(0.0, 0.86, 0.0)
(0.02, 0.83, 0.5)
(0.09, 0.28, 0.0)

(0.0, 0.6, 0.0)
(0.0, 0.64, 0.0)
(0.02, 0.76, 0.1)
(0.01, 0.24, 0.0)

(0.0, 0.52, 0.0)
(0.0, 0.68, 0.0)
(0.01, 0.79, 0.0)
(0.01, 0.26, 0.0)

Table 1: Experiments on the longer-horizon setting of T = 500, where the GPT-4o model still
exhibits can achieve sublinear regret as validated by our frameworks and the comparison with
FTRL/FTPL.

3.3 RESULTS: MULTI-PLAYER REPEATED GAMES
We now consider the setting when multiple LLMs make online strategic decisions in a shared envi-
ronment repeatedly. Specifically, at each round, the loss vectors each agent receives are determined

5

0510152025Iterations01020304050RegretUniformGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.65,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.01,0.92,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.58,0.97,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.0,0.94,0.0)FTRLFTPL0510152025Iterations01020304050607080RegretGaussianGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.63,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.01,0.92,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.01,0.95,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.0,0.91,0.0)FTRLFTPL0510152025Iterations020406080RegretLinear-trendGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.55,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.72,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.08,0.99,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.0,0.89,0.0)FTRLFTPL0510152025Iterations−100102030RegretSine-trendGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.35,0.03)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.95,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.27,0.84,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.0,0.99,0.0)FTRLFTPL020406080100Iterations020406080100120140RegretUniformGPT-4o:(ptrend,ˆβ0,preg)=(0.0,0.47,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.11,0.9,0.0)FTRLFTPL020406080100Iterations0255075100125150175RegretGaussianGPT-4o:(ptrend,ˆβ0,preg)=(0.0,0.56,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.01,0.97,0.0)FTRLFTPL050100150200Iterations050100150200RegretLinear-trendGPT-4o:(ptrend,ˆβ0,preg)=(0.0,0.8,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.88,0.0)FTRLFTPL050100150200Iterations−50−250255075100125RegretSine-trendGPT-4o:(ptrend,ˆβ0,preg)=(0.04,-2.53,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.2,0.53,0.0)FTRLFTPLUnder review as a conference paper at ICLR 2025

Figure 3.3: Regret of pre-trained LLMs for repeated games of different sizes, n most cases, both
commercial and open-source LLMs can achieve sublinear regret as validated by our frameworks
and the comparison with FTRL/FTPL. We report the regret of one agent for ease of presentation.

Figure 3.4: (left) Regret of GPT-4 (Turbo) under the canonical counterexample for FTL (Hazan,
2016, Chapter 5). (mid, right) Failure of GPT-4 (Turbo) on two scenarios with regrettable behaviors,
while Transformers trained by our new regret-loss (N = 1) in Section 5 can achieve sublinear regret.

by both her payoff matrix and the strategies of all other agents. Note that the payoff matrix is not
directly revealed to the LLM agent, but she has to make decisions in a completely online fashion
based on the payoff vector marginalized by the opponents’ strategies (See Figure C.1 for a prompt
example). This is a typical scenario in learning in (repeated) games (Fudenberg & Levine, 1998).
Representative games. We first test LLMs on 6 representative general-sum games (win-win, pris-
oner’s dilemma, unfair, cyclic, biased, and second best) studied in Robinson & Goforth (2005) (c.f.
Appendix B.6). For each type of the game, we conduct 20 repeated experiments.
Randomly generated games. To further validate the no-regret behavior of LLMs, we also test
on 50 randomly generated three-player general-sum games, and 50 randomly generated four-player
general-sum games, where each entry of the payoff matrix is sampled randomly from Unif([0, 10]).
These are larger and more challenging settings than the structured and representative ones above.
We summarize the experimental results in Figure 3.3, which are similar to the above in the online
setting: for all types of games, pre-trained LLMs achieve sublinear regret, which is often lower than
that obtained by FTRL/FTPL for most games. We provide six instances of three-player general-sum
games and six instances of four-player general-sum games in Figure C.4 and Figure C.5, respec-
tively. Occasionally, GPT-4 even provides a negative regret value.

3.4 PRE-TRAINED LLM AGENTS CAN STILL HAVE REGRET
It seems tempting to conclude that pre-trained LLMs are indeed no-regret in both online learning
and playing repeated games. However, is this capability universal? We show that the no-regret
property can break for LLM agents if the loss vectors are generated in a more adversarial way.
Canonical counterexamples for follow-the-leader. To begin with, we consider well-known two
examples that follow-the-leader (FTL) algorithm (Shalev-Shwartz, 2012) suffers from linear regret.

Example 1: ℓ1(1) = 5, ℓ1(2) = 0 and ℓt(2

t%2) = 10, ℓt(1 + t%2) = 0 for t

2 (Hazan, 2016).

−

≥

t%2) = 10, ℓt(1 + t%2) = 0 for 1

c and ℓt(1) = 10, ℓt(2) = 0 for

T (= 500), for some integer c satisfying 0 < c < T (Feder et al., 1992).

t

≤

≤

Example 2: ℓt(2
t
c + 1

−

≤

≤

Here, % denotes the modulo operation. Interestingly, for Example 1, GPT-4 agent can easily identify
the pattern for the loss sequence that the optimal action alternates, thus accurately predicting the loss
it will receive and achieving low regret in Figure 3.4. For Example 2, the GPT-4 agent with raw

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

0510152025Iterations051015202530RegretTwo-playergamesGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.63,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.9,0.0)FTRLFTPL0510152025Iterations0510152025RegretThree-playergamesGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.69,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.9,0.0)FTRLFTPL0510152025Iterations05101520RegretFour-playergamesGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.8,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.93,0.0)FTRLFTPL0510152025Iterations−20−1001020RegretAlternatinglossGPT-4:(ptrend,ˆβ0,preg)=(0.0,-2.89,0.0)GPT-4-Turbo:(ptrend,ˆβ0,preg)=(0.0,-3.86,0.0)0510152025Iterations020406080100120RegretNoisyalternatinglossSingle-layer:(ptrend,ˆβ0,preg)=(0.27,0.75,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.27,0.55,0.0)FTRL:(ptrend,ˆβ0,preg)=(0.42,0.98,0.0)GPT-4:(ptrend,ˆβ0,preg)=(1.0,1.96,0.0)GPT-4Turbo:(ptrend,ˆβ0,preg)=(1.0,2.0,0.0)0510152025Iterations050100150200RegretAdaptivelossSingle-layer:(ptrend,ˆβ0,preg)=(0.27,0.75,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.08,0.03,0.68)FTRL:(ptrend,ˆβ0,preg)=(0.42,0.51,0.0)GPT-4:(ptrend,ˆβ0,preg)=(1.0,1.13,0.0)GPT-4Turbo:(ptrend,ˆβ0,preg)=(0.97,1.08,0.0)Under review as a conference paper at ICLR 2025

×

history also provides an impressively lower (negative) regret than FTRL and FTPL (Figure C.6).
The GPT-4 agent with summarized history suffers from much larger regret than FTRL and FTPL.
We refer the detailed comparison between using raw history and summarized history to Figure C.6.
In summary, the GPT-4 agent can predict such worst-case sequences well, and does not fail in the
same way as FTL, which is known to suffer from a lack of randomness in decisions.
Additionally, the results on Example 2 also imply that summary-based history input can perform
worse than the raw-history one in the adversarial setting we consider, while the former was claimed
to be the key in succeeding in the i.i.d. stochastic bandit setting (Krishnamurthy et al., 2024). The
10−157 under a one-sided
regret values of the two cases differ significantly, with a p-value of 1.2
independent t-test. These results further illustrate the fundamental differences between the settings
considered in Krishnamurthy et al. (2024) and ours.
Inspired by the above, we design a new loss sequence that
Noisy alternating loss sequence.
is similar but less predictable by adding some noise to the canonical counterexample. Specifi-
cally, we construct the following (simple) loss sequence with 2 actions such that ℓt(1 + t%2) =
min(25/t, 10), ℓt(2
∈
Adaptive loss sequence. We also develop a simpler but more adaptive loss sequence that takes
the full power of the adversary in our online learning setup. After the GPT-4 agent provides πt,
arg maxi πti) = 0. We understand for such
we choose ℓt with ℓt(arg maxi πti) = 10 and ℓt(3
adaptive adversaries, it is preferable to use regret metrics other than the external regret, i.e., the
policy regret (Arora et al., 2012a; Cesa-Bianchi et al., 2013), as the comparator loss sequences may
also change in this case. We consider this setting mainly to understand the limit of LLMs in terms
of the same regret metric.
We also report the average regret over 20 repeated experiments for the later two settings using
GPT-4 and more advanced GPT-4 Turbo in Figure 3.4, where we cannot reject the hypothesis that
GPT-4 (Turbo) has linear-regret by either our trend-checking or regression-based framework. These
observations have thus motivated us to design new approaches to further promote the no-regret
behaviors of the models, with additional training, as to be detailed in Section 5. Before it, we first
provide some theoretical insights into the observed sublinear regret behaviors.

Unif([9, 10]) for t

t%2)

[25].

−

∼

−

4 WHY DO PRE-TRAINED LLMS (NOT) HAVE REGRET? A HYPOTHETICAL

MODEL AND SOME THEORETICAL INSIGHTS

We now provide some plausible explanations about the observed no-regret behavior of pre-trained
LLMs, which are hypothetical by nature, since to the best of our knowledge, the details of pre-
training these popular LLMs (e.g., GPT-3.5 Turbo and GPT-4), regarding data distribution, training
algorithm, etc., have not been revealed. We instead make the explanations based on some common
assumptions in the literature for modeling human behaviors, and the recent literature on understand-
ing LLMs and Transformers.

4.1 A (HUMAN) DECISION-MAKING MODEL: QUANTAL RESPONSE
A seminal model for human decision-making behavior is the quantal response model, which as-
sumes that humans are often imperfect decision-makers, and their bounded rationality can be mod-
eled through unseen latent variables that influence the decision-making process (McFadden, 1976;
McKelvey & Palfrey, 1995), for which we refer the formal definition and introduction to Section D.2.
However, the traditional quantal response formulation primarily focused on scenarios with a single
loss vector. In online decision-making, given the history information, the decision-maker at each
time t is faced with multiple loss vectors. Hence, we adopt the following generalization to model
the decision-making behavior in this setting.
Definition 4.1 (Quantal response against multiple losses). Given a set of losses (ℓi)i∈[t], a noise
Pnoise, and ηt > 0, the generalized quantal response against (ℓi)i∈[t] is defined as
distribution ϵ

∼

P ηt

quantal

(cid:0)a (cid:12)

(cid:12) (ℓi)i∈[t]

(cid:1) := P ηt

quantal

(cid:32)

(cid:12)
(cid:12)
(cid:12)
(cid:12)

a

t
(cid:88)

i=1

(cid:33)

ℓi

= P

(cid:18)

a ∈ arg min

a′∈A

(cid:19)

z(a′)

, where z = ηtϵ +

t
(cid:88)

i=1

ℓi.

In simpler terms, the generalized quantal response is defined as the standard quantal response against
the summation of the losses. Such a model has been investigated in the learning-in-games and be-
havioral economics literature (see Section D.2 for more details). Such a definition is also aligned
with our empirical findings on LLMs’ behavioral patterns in Section 3.2: i) evaluating the summa-
tion/average; ii) introducing randomization in decision-making. To gain more insights into these

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Under review as a conference paper at ICLR 2025

empirical findings, we next analyze a case where pre-training under certain canonical assumptions
provably leads to the quantal response behaviors and further yields no-regret guarantees.

· · ·

, xN ) and then generates (xN +1,

4.2 CASE STUDY: PRE-TRAINING UNDER CANONICAL DATA DISTRIBUTION
Pre-training of LLMs is predominantly based on next-token prediction. When applying LLMs
to sequential decision-making, the model receives the context of the decision-making task as
N+
(x1, x2,
∈
and N < M , where each xi ∈ V
is the fi-
nite token set. This process can be conceptualized as predicting the optimal action in the form of the
next token prediction (Yao et al., 2023b; Shinn et al., 2023; Liu et al., 2023a;e). Meanwhile, large
models are often (pre-)trained under several fixed/stationary environments (Laskin et al., 2023; Lin
et al., 2024; Lee et al., 2023; Reed et al., 2022), which may limit their ability to handle arbitrary/non-
stationary/adversarial loss sequences in our online learning setup. Thus, it is natural to ask: Is it
possible to have no-regret behaviors emerging as a consequence of this (optimal) action prediction,
under only a fixed pre-training distribution of the environments?

, xM ) encoding the action for some N, M

represents one natural language token for i

[M ], and

· · ·

∈

V

Here we analyze a standard pre-training objective on a token
[T ],
sequence distribution x1:Nt+1 ∼
∈
which is the expected log-likelihood maximization for next-
token prediction over Θ, the parameter space of the LLM:
Nt+1
(cid:88)

for given t

P text
t

log LLMθ (xj | x1:j−1) ,

(4.1)

E

max
θ∈Θ

∼P text
t

j=1

x1:0) = LLMθ (x1).

there are latent variables

Figure 4.1: Comparison of GPT-
4 with the generalized QR model,
where the model can very well
capture the behavior of the GPT-4
agent for examples in Section 3.2.

x1:Nt+1
where we define LLMθ (x1 |
For the pre-training distribution, we model it as follows: there
exists a latent variable z, representing the loss for the underly-
ing static decision-making problem. The pre-training dataset,
however, only contains partial observations x1:Nt (a natural
language representation of ℓ1:t) of z due to imperfect data col-
lection, which could be attributed to the fact that z is private to
the data-generator (human), representing the actual intention
of the human/data-generator. Hence, LLM will only be pre-
trained with partial and noisy information about z. Meanwhile, we assume that some high-quality
action label xNt+1:Nt+1 (a natural language representation of a) w.r.t. the underlying loss vector z is
also available in the dataset, which could come from user surveys, personal blogs, or data annotation.
We formalize such an assumption:
Assumption 1 (Pre-training distribution). Given T
∈
∈
N+,
[Nt+1], N0 = 0,
such
that P(z, ℓ1:t, x1:Nt+1 ) = P(z, ℓ1:t)P(x1:Nt |
z), and P text
:=
t
P(x1:Nt+1 ) = (cid:82)
xNi−1+1:Ni }i∈[t] encode the
{
context, i.e., information for ℓ1:t, and the user will decode action a from xNt+1:Nt+1.
To further understand our assumption, we provide an example in Section D.3, showing how a nat-
ural text corpus may satisfy it. Similar assumptions that suppose the existence of such latent vari-
ables in generating the pre-training datasets have also been made recently in Lee et al. (2023);
Lin et al. (2024); Liu et al. (2023e), for understanding the in-context decision-making behaviors
of LLMs/Transformers through posterior sampling, for which we defer a detailed comparison to
z is modeled as Gaus-
Section D.8. In particular, we show in Theorem 4.1 that if the noise, i.e., ℓi −
sian distributions and xNt+1:Nt+1 encodes the optimal action for z, the pre-trained LLM provably
recovers the prominent human behavior model in Section 4.1, the quantal response model.
Theorem 4.1 (Informal: Emergence of no-regret behavior). Suppose Assumption 1 holds with both
the prior distribution on z and the likelihood on
}i∈[t] being Gaussian, and xNt+1:Nt+1 en-
ℓi |
{
codes the optimal action for z. Then, as long as the function class of LLMθ is expressive enough,
with θ⋆ being a maximizer of Equation (4.1), the behavior of LLMθ⋆ follows Definition 4.1. Further-
more, the use of LLMθ⋆ can achieve no (dynamic) regret for (non-stationary) online learning with
full-information/bandit feedback for arbitrary loss vectors (with bounded variations).

N+,
, Nt
ℓ1:t)P(xNt+1:Nt+1 |
P(z, ℓ1:t, x1:Nt+1)dℓ1:tdz. Intuitively, tokens

(z, ℓ1:t), N1,

[T ], Nt+1

(x1:Nt+1)

· · ·

ℓ1:t

∈

∈

(cid:82)

z

t

z

The formal statement and proof are deferred to Section D.6. The significance of our results lies in
that even when pre-training is conducted solely with loss vectors generated from stationary distri-
butions (ℓ1:t are i.i.d. conditioned on z), it can still enable the emergence of no-regret behavior in

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

0510152025Iterations010203040Regret/DynamicregretExamplesinSec.3.2GPT-4regret:(ptrend,ˆβ0,preg)=(0.0,0.46,0.0)GPT-4dynamicregret:(ptrend,ˆβ0,preg)=(0.0,0.89,0.0)GeneralizedQRregret:(ptrend,ˆβ0,preg)=(0.0,0.51,0.0)GeneralizedQRdynamicregret:(ptrend,ˆβ0,preg)=(0.0,0.92,0.0)Under review as a conference paper at ICLR 2025

online learning against potentially adversarial losses. Key in the proof is an interesting connection
of pre-trained LLM models to the online learning algorithm of FTPL. Furthermore, Assumption 1
can be relaxed to better match the actual LLMs’ pre-training data distributions from diverse sources
(c.f. Section D.7), and the prior distribution of z could also be replaced by a general distribution
(c.f. Theorem D.2). Finally, we point out its implications for playing games in Section D.6.1.

How well can our hypothetical model class predict actual LLMs’ behaviors? To further verify
T −1
our theoretically-justified model in Theorem 4.1, we propose to estimate the parameters of
t=0
in Definition 4.1 using the interaction data with actual LLMs, and use the estimated model to predict
LLMs’ behaviors on some test set. In Figure 4.1, we show the averaged regret for the LLMs and
our estimated model, where the generalized quantal response can very well capture the behavior
of the LLM agent for all problem instances in Section 3.2, on which the LLMs oftentimes achieve
sublinear regret, justifying the applicability of our hypothetical model and assumptions.

ηt}
{

Finally, we acknowledge that for existing pre-trained LLMs like GPT-4, the canonical assumptions
above, though may be further relaxed (c.f. Remark D.3), may not hold in general. More importantly,
the supervision labels, i.e., the optimal action given z, may be sometimes imperfect or unavailable
during the dataset collection. These caveats motivate the study in our next section.

5 PROVABLY PROMOTING NO-REGRET BEHAVIOR BY A NEW LOSS
In light of the observations in Section 3, we ask the question:

Is there a way to enhance the no-regret property of the models without (optimal) action labels?
To address this question, we propose to train models with a new unsupervised learning loss that
naturally provides no-regret behaviors. We will particularly focus on the Transformer architecture
(Vaswani et al., 2017) under this new loss, a common architecture used in most existing LLMs.

5.1 A NEW UNSUPERVISED TRAINING LOSS: Regret-Loss
Intuitively, our new training loss is designed to enforce the trained models to minimize regret under
an arbitrary sequence of loss vectors. Specifically, we define the training loss as

RegretLLMθ

(θ) := max
ℓ1,...,ℓT

(cid:0)(ℓt)t∈[T ]
L
[T ]. As discussed in Kirschner et al. (2023), directly minimizing the
where
max regret can be computationally challenging, except for superficially simple problems. More-
over, Equation (5.1) is not necessarily differentiable with respect to the parameter θ, if it does not
satisfy the condition of Danskin’s Theorem (Danskin, 1966); or even if it is differentiable (i.e., the
maximizer of (ℓt)t∈[T ] is unique), computation of derivatives can be challenging since we need to
calculate arg max(ℓt)t∈[T ] RegretLLMθ ((ℓt)t∈[T ]) while we have inf in the definition of regret. There-
fore, we provide a general class of surrogate losses to approximate Equation (5.1):

B for t

ℓt∥
∥

(5.1)

≤

∈

∞

(cid:1)

(θ, k, N ) := E

L

(cid:34) (cid:80)

j∈[N ] h(RegretLLMθ ((ℓ(j)
(cid:80)

t )t∈[T ]))f (RegretLLMθ ((ℓ(j)

t )t∈[T ]), k)

(cid:35)

,

(5.2)

j∈[N ] f (RegretLLMθ ((ℓ(j)

t )t∈[T ]), k)

→

→

N+, h : R

∈
R+ is a continuous function for each k

N+, N
where k
∈
, k) : R
h′, and f (
·

∈
1(R1 > R2)+1(R1 = R2), where we use the convention of

R+ is a continuous function, with continuous derivative
f (R1,k)
f (R2,k) =
0 = 0. These conditions on h, f
∞·
will be assumed throughout the paper. Examples of such an f include f (x, k) = xk and exp(kx).
We will sample N trajectories of loss sequences (ℓ(j)
t )t∈[T ],j∈[N ] from some continuous probability
B, B]T ×N (without other additional statistical assumptions), and the
distribution supported on [
expectation in Equation (5.2) is thus taken with respect to this distribution. In Section E.2, we prove
that under certain regularity conditions of f and h, we have

N+, satisfying limk→∞

∞·

−

(cid:18)

(cid:19)

lim
N,k→∞ L

(θ, k, N ) = h

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])
(cid:18)

,

(cid:19)

and the uniform convergence of

(θ, k, N ):

−
(cid:12)
(cid:12)
(cid:12) = 0, where Θ is a compact set of the model parameters. Hence, one can expect that
(θ, k, N )
L
minimizing the loss function in Equation (5.2) with large enough k and N may promote the trained
models to have a small regret value. We will hereafter refer to Equation (5.2) as the regret-loss2.

L

lim
N,k→∞

supθ∈Θ

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

(cid:12)
(cid:12)
(cid:12)h

2We can also define dynamic-regret-loss, and Section 5.2 can be generalized to this case (c.f. Remark E.1).

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Under review as a conference paper at ICLR 2025

5.2 GENERALIZATION AND REGRET GUARANTEES OF REGRET-LOSS MINIMIZATION
We first establish a statistical guarantee under general parameterizations of LLMθ that is Lipschitz
with respect to θ, including the Transformer-based models as used in GPT-4 and most existing LLMs
(see Proposition 2 for an example with a formal statement). This guarantee focuses on their gener-
alization ability when trained to minimize the empirical regret loss (c.f. Equation (E.3)), denoted
(θ, k, N, NT ), by replacing the expectation E in Equation (5.2) with the empirical mean using
as (cid:98)
L
NT samples. We denote (cid:98)θk,N,NT ∈
(θ, k, N, NT ), and present the generalization
(θ, k, N ) (c.f. Section E.2), we
guarantee in Theorem E.1. Thanks to the uniform convergence of
further obtain the following theorem on the regret guarantee of LLM(cid:98)θk,N,NT
Theorem 5.1. (Regret). Suppose3 for any k
supermodular function (i.e., log f (R1, k1)
R1 ≥

N+, h, f (
, k) are non-decreasing, and log f is a
∈
·
log f (R2, k2) for
log f (R1, k2)
k2). Then, with high probably, we have

arg minθ∈Θ (cid:98)
L

R2 and k1 ≥

log f (R2, k1)

−

−

≥

L

:

(cid:33)

(cid:18)

(cid:0)(ℓt)t∈[T ]

(cid:1)

≤ h

inf
θ∈Θ

max
∥ℓt∥∞≤B

RegretLLMθ

(cid:19)

(cid:0)(ℓt)t∈[T ]

(cid:1)

+ (cid:101)O





(cid:115)



 .

dθ
NT

h

lim
N→∞

lim
k→∞

max
∥ℓt∥∞≤B

RegretLLM

(cid:98)θk,N,NT

(cid:32)

We defer the proof of the theorem to Section E.4. Therefore, if additionally, the model parameteri-
zation (i.e., Transformers) can realize a no-regret algorithm (as to be shown next), then Theorem 5.1
means that with a large enough NT , the learned LLM(cid:98)θk,N,NT
becomes a no-regret learner, i.e.,
(cid:1) = o(T ). Finally, as a consequence, it is folklore that when multiple
RegretLLM (cid:98)θk,N,NT
such LLMs interact, a coarse correlated equilibrium will emerge in the long-term (c.f. Corollary 1).

(cid:0)(ℓt)t∈[T ]

t
(cid:88)

i=1

5.3 REGRET-LOSS TRAINED TRANSFORMERS CAN BE ONLINE LEARNING ALGORITHMS
Despite the generality of the previous results, one cannot use an infinitely large N and k in prac-
tice. Hence, we now provide results when N is finite, for the architecture of Transformer models
(Vaswani et al., 2017). We focus on single-layer (linear) self-attention models, as in most recent
theoretical studies of Transformers (Ahn et al., 2023; Zhang et al., 2023a; Mahankali et al., 2023),
and N = 1. Note that in this case, the choice of f (and thus k) is not relevant. Thus, throughout this
[T ] as realizations of
subsection, we drop superscript (j) in Equation (5.2). We sample ℓt for t
some random variable Z, where we assume that Z is symmetric about zero, and Var(Z) = Σ
0.
We consider the single-layer linear self-attention model as follows, for which we can show that the
global optimizer of our regret-loss can automatically lead to a no-regret learning algorithm:

≻

∈

g(Zt; V, K, Q, vc, kc, qc) =

⊺
(V ℓi + vc) ((Kℓi + kc)

(Qc + qc)) .

·

(5.3)

∥

(cid:17)

−

(cid:80)T

∥ · ∥

⊺
ℓ1ℓ
2

t=1 ℓt∥

2RΠΣ−1E

Theorem 5.2. Consider the policy space Π = B(0, RΠ,
) for some RΠ > 0. The configu-
ration of a single-layer linear self-attention model in Equation (5.3) (V, K, Q, vc, kc, qc) such that
(cid:16)
K ⊺(Qc + qc) = vc = 000d and V =
Σ−1 is a global optimal solu-
tion of Equation (5.2) with N = 1, h(x) = x2. Moreover, every global optimal configuration of
Equation (5.2) within the parameterization class of Equation (5.3) has the same output function g.
Additionally, if Σ is a diagonal matrix, then plugging any global optimal configuration into Equa-
tion (5.3), and projecting the output with ProjΠ,∥·∥ is equivalent to FTRL with an L2-regularizer.
Theorem 5.2 not only shows the capacity of self-attention models: it can realize online learning
algorithms, but also shows, more importantly, that minimizing our new regret-loss may automati-
cally produce it. In particular, one does not need to hard-code the parameters of the Transformer to
implement no-regret algorithms. Under single-layer self-attention parameterization (with a softmax
operation), we can also show that a stationary point of the loss function (Equation (5.2)) can lead to
the FTRL algorithm. We defer this result to Section E.5. Finally, it is also possible to generalize the
results to FTRL with entropy (instead of the L2) regularizer (c.f. Section E.9).
5.4 EXPERIMENTAL RESULTS FOR REGRET-LOSS TRAINED TRANSFORMERS
We now provide experimental results for minimizing our regret-loss under the Transformer models,
and evaluate in the following environments: 1) randomly-generated loss sequences (Figure E.3);
2) loss sequences with certain trends (Figure E.4); 3) repeated games (Figure E.5); and 4) coun-
terexamples for pre-trained LLMs to be regrettable (Figure 3.4). Training setup can be found in
Section E.11.1. We also provide an ablation study for training on Equation (5.2) in Section E.12.

Finally, we provide discussions on the limitations and future directions in Section F.
3Note that these conditions on h, f are in addition to those specified after Equation (5.2).

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Jacob Abernethy, Chansoo Lee, Abhinav Sinha, and Ambuj Tewari. Online linear optimization via

smoothing. In Conference on Learning Theory, pp. 807–823. PMLR, 2014.

Jacob D Abernethy, Chansoo Lee, and Ambuj Tewari. Fighting bandits with a new kind of smooth-

ness. Advances in Neural Information Processing Systems, 28, 2015.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate
multiple humans and replicate human subject studies. In International Conference on Machine
Learning, pp. 337–371. PMLR, 2023.

Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to imple-
ment preconditioned gradient descent for in-context learning. Advanced in Neural Information
Processing Systems, 2023.

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea
Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say:
Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.

Mohammad Ahsanullah, Valery B Nevzorov, and Mohammad Shakil. An introduction to order

statistics, volume 8. Springer, 2013.

Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz.
Playing repeated games with large language models. arXiv preprint arXiv:2305.16867, 2023.

Ekin Aky¨urek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning
algorithm is in-context learning? investigations with linear models. International Conference on
Learning Representations, 2023.

Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David
Wingate. Out of one, many: Using language models to simulate human samples. Political Anal-
ysis, 31(3):337–351, 2023.

Raman Arora, Ofer Dekel, and Ambuj Tewari. Online bandit learning against an adaptive adversary:
from regret to policy regret. In Proceedings of the 29th International Coference on International
Conference on Machine Learning, pp. 1747–1754, 2012a.

Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a meta-

algorithm and applications. Theory of computing, 8(1):121–164, 2012b.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multi-

armed bandit problem. SIAM journal on computing, 32(1):48–77, 2002.

Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians:
Provable in-context learning with in-context algorithm selection. Advanced in Neural Information
Processing Systems, 2023.

Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew
Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by com-
bining language models with strategic reasoning. Science, 378(6624):1067–1074, 2022.

Santiago R Balseiro and Yonatan Gur. Learning in repeated auctions with budgets: Regret mini-

mization and equilibrium. Management Science, 65(9):3952–3968, 2019.

Claude Berge. Topological spaces: Including a treatment of multi-valued functions, vector spaces

and convexity. Oliver & Boyd, 1877.

Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed-bandit problem with non-

stationary rewards. Advances in neural information processing systems, 27, 2014.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Avrim Blum and Yishay Mansour. From external to internal regret. Journal of Machine Learning

Research, 8(6), 2007.

Avrim Blum, MohammadTaghi Hajiaghayi, Katrina Ligett, and Aaron Roth. Regret minimization
and the price of total anarchy. In Proceedings of the fortieth annual ACM symposium on Theory
of computing, pp. 373–382, 2008.

Philip Brookins and Jason Matthew DeBacker. Playing games with GPT: What can we learn about

a large language model from canonical strategic games? Available at SSRN 4493398, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.

S´ebastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-

armed bandit problems. Foundations and Trends® in Machine Learning, 5(1):1–122, 2012.

S´ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka-
mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general
intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.

Colin F Camerer. Behavioral game theory: Experiments in strategic interaction. Princeton Univer-

sity Press, 2011.

Nicolo Cesa-Bianchi and G´abor Lugosi. Prediction, Learning, and Games. Cambridge University

Press, 2006.

Nicolo Cesa-Bianchi, Philip M Long, and Manfred K Warmuth. Worst-case quadratic loss bounds
IEEE Transactions on Neural Net-

for prediction using linear functions and gradient descent.
works, 7(3):604–619, 1996.

Nicolo Cesa-Bianchi, Ofer Dekel, and Ohad Shamir. Online learning with switching costs and other

adaptive adversaries. Advances in Neural Information Processing Systems, 26, 2013.

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and
Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. Inter-
national Conference on Learning Representations, 2024.

Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan,
Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and
exploring emergent behaviors in agents. International Conference on Learning Representations,
2024.

Yiting Chen, Tracy Xiao Liu, You Shan, and Songfa Zhong. The emergence of economic rationality

of gpt. Proceedings of the National Academy of Sciences, 120(51):e2316205120, 2023.

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can
GPT learn in-context? language models secretly perform gradient descent as meta-optimizers.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Associa-
tion for Computational Linguistics: ACL 2023, pp. 4005–4019, Toronto, Canada, July 2023.
Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-acl.247. URL
https://aclanthology.org/2023.findings-acl.247.

John M Danskin. The theory of max-min, with applications. SIAM Journal on Applied Mathematics,

14(4):641–664, 1966.

Constantinos Daskalakis, Maxwell Fishelson, and Noah Golowich. Near-optimal no-regret learning
in general games. Advances in Neural Information Processing Systems, 34:27604–27616, 2021.

Jingying Ding, Yifan Feng, and Ying Rong. Myopic quantal response policy: Thompson sampling

meets behavioral economics. arXiv preprint arXiv:2207.01028, 2022.

Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter,
Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi-
modal language model. International Conference on Machine Learning, 2023.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch.

Improv-
ing factuality and reasoning in language models through multiagent debate. arXiv preprint
arXiv:2305.14325, 2023.

Christoph Engel, Max RP Grossmann, and Axel Ockenfels.

Integrating machine behavior into

human subject experiments: A user-friendly toolkit and illustrations. Available at SSRN, 2023.

Ido Erev and Alvin E Roth. Predicting how people play games: Reinforcement learning in experi-
mental games with unique, mixed strategy equilibria. American Economic Review, pp. 848–881,
1998.

Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. Can large language models serve as rational

players in game theory? a systematic analysis. arXiv preprint arXiv:2312.05488, 2023.

Meir Feder, Neri Merhav, and Michael Gutman. Universal prediction of individual sequences. IEEE

Transactions on Information Theory, 38(4):1258–1270, 1992.

Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an

application to boosting. Journal of computer and system sciences, 55(1):119–139, 1997.

Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with

self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142, 2023.

Drew Fudenberg and David M Kreps. Learning mixed equilibria. Games and Economic Behavior,

5(3):320–367, 1993.

Drew Fudenberg and David K Levine. The theory of learning in games, volume 2. MIT Press, 1998.

Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game

theory and reinforcement learning. arXiv preprint arXiv:1704.00805, 2017.

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn
in-context? a case study of simple function classes. Advances in Neural Information Processing
Systems, 35:30583–30598, 2022.

Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris
Papailiopoulos. Looped transformers as programmable computers. International Conference on
Machine Learning, 2023.

Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning
with language model is planning with world model.
In Houda Bouamor, Juan Pino, and Ka-
lika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language
Processing, pp. 8154–8173, Singapore, December 2023. Association for Computational Linguis-
tics. doi: 10.18653/v1/2023.emnlp-main.507. URL https://aclanthology.org/2023.
emnlp-main.507.

Elad Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization,

2(3-4):157–325, 2016.

Josef Hofbauer and William H Sandholm. On the global convergence of stochastic fictitious play.

Econometrica, 70(6):2265–2294, 2002.

Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven
Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming
for multi-agent collaborative framework. nternational Conference on Learning Representations,
2024.

John J Horton. Large language models as simulated economic agents: What can we learn from

homo silicus? Technical report, National Bureau of Economic Research, 2023.

Arnljot Hoyland and Marvin Rausand. System reliability theory: models and statistical methods.

John Wiley & Sons, 2009.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot
planners: Extracting actionable knowledge for embodied agents. In International Conference on
Machine Learning, pp. 9118–9147. PMLR, 2022a.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan
Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through
planning with language models. arXiv preprint arXiv:2207.05608, 2022b.

Hui Jiang. A latent space theory for emergent abilities in large language models. arXiv preprint

arXiv:2304.09960, 2023.

Adam Kalai and Santosh Vempala. Efficient algorithms for online decision problems. Journal of

Computer and System Sciences, 71(3):291–307, 2005.

Mikołaj J Kasprzak, Ryan Giordano, and Tamara Broderick. How good is your gaussian approxima-
tion of the posterior? finite-sample computable error bounds for a variety of useful divergences.
arXiv preprint arXiv:2209.14992, 2022.

Johannes Kirschner, Alireza Bakhtiari, Kushagra Chandak, Volodymyr Tkachuk, and Csaba Szepes-
vari. Regret minimization via saddle point optimization. In Thirty-seventh Conference on Neural
Information Processing Systems, 2023.

Akshay Krishnamurthy, Keegan Harris, Dylan J Foster, Cyril Zhang, and Aleksandrs Slivkins. Can

large language models explore in-context? arXiv preprint arXiv:2403.15371, 2024.

Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald,
DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, et al. In-context reinforcement learning
with algorithm distillation. International Conference on Learning Representations, 2023.

Tor Lattimore and Csaba Szepesv´ari. Bandit algorithms. Cambridge University Press, 2020.

Jonathan N Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma
Brunskill. Supervised pretraining can learn in-context reinforcement learning. Neural Information
Processing Systems, 2023.

Chao Li, Xing Su, Chao Fan, Haoying Han, Cong Xue, and Chunmo Zheng. Quantifying the impact
of large language models on collective opinion dynamics. arXiv preprint arXiv:2308.03313,
2023a.

Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.
Camel: Communicative agents for” mind” exploration of large scale language model society.
Neural Information Processing Systems, 2023b.

Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model

based evaluations. arXiv preprint arXiv:2307.02762, 2023c.

Siyu Li, Jin Yang, and Kui Zhao. Are you in a masquerade? exploring the behavior and im-
arXiv preprint

pact of large language model driven social bots in online social networks.
arXiv:2307.10337, 2023d.

Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers
as algorithms: Generalization and stability in in-context learning. International Conference on
Machine Learning, 2023e.

Zifan Li and Ambuj Tewari. Beyond the hazard rate: More perturbation algorithms for adversarial

multi-armed bandits. J. Mach. Learn. Res., 18:183–1, 2017.

Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng
Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-
agent debate. arXiv preprint arXiv:2305.19118, 2023.

Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforce-
ment learning via supervised pretraining. International Conference on Learning Representations,
2024.

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Nick Littlestone and Manfred K Warmuth. The weighted majority algorithm.

Information and

computation, 108(2):212–261, 1994.

Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with

feedback. arXiv preprint arXiv:2302.02676, 3, 2023a.

Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, and Yan Zhang.
recommender? a preliminary study. arXiv preprint arXiv:2304.10149, 2023b.

Is chatgpt a good

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg
evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023c.

Yueyang Liu, Benjamin Van Roy, and Kuang Xu. Nonstationary bandit learning via predictive
sampling. In International Conference on Artificial Intelligence and Statistics, pp. 6215–6244.
PMLR, 2023d.

Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. Reason
for future, act for now: A principled architecture for autonomous llm agents. In NeurIPS 2023
Foundation Models for Decision Making Workshop, 2023e.

LLM Engine. Llm engine, 2023. URL https://llm-engine.scale.com.

Nunzio Lor`e and Babak Heydari. Strategic behavior of large language models: Game structure vs.

contextual framing. arXiv preprint arXiv:2309.05898, 2023.

Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably
the optimal in-context learner with one layer of linear self-attention. International Conference on
Learning Representations, 2023.

Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Bas¸ar. Model-free
non-stationary RL: Near-optimal regret and applications in multi-agent RL and inventory control.
arXiv preprint arXiv:2010.03161, 2020.

Daniel L McFadden. Quantal choice analaysis: A survey. Annals of Economic and Social Measure-

ment, Volume 5, number 4, pp. 363–390, 1976.

Richard D McKelvey and Thomas R Palfrey. Quantal response equilibria for normal form games.

Games and economic behavior, 10(1):6–38, 1995.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke
Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In
Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference
on Empirical Methods in Natural Language Processing, pp. 11048–11064, Abu Dhabi, United
Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/
2022.emnlp-main.759. URL https://aclanthology.org/2022.emnlp-main.759.

Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse
arXiv preprint

Clifton. Welfare diplomacy: Benchmarking language model cooperation.
arXiv:2310.08901, 2023.

Denis Nekipelov, Vasilis Syrgkanis, and Eva Tardos. Econometrics for learning agents. In ACM

Conference on Economics and Computation, pp. 1–18, 2015.

Openai. Gpt-4 technical report. 2023.

Ian Osband, Daniel Russo, and Benjamin Van Roy.

(more) efficient reinforcement learning via

posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.

Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S
Bernstein. Social simulacra: Creating populated prototypes for social computing systems.
In
Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp.
1–18, 2022.

15

Under review as a conference paper at ICLR 2025

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and
Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceed-
ings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23,
New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701320. doi:
10.1145/3586183.3606763. URL https://doi.org/10.1145/3586183.3606763.

Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu,
arXiv preprint

Communicative agents for software development.

and Maosong Sun.
arXiv:2307.07924, 2023.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio G´omez Colmenarejo, Alexander Novikov,
Gabriel Barth-maron, Mai Gim´enez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Ec-
cles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol
Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine
ISSN 2835-8856. URL https://openreview.net/forum?
Learning Research, 2022.
id=1ikK0kHjvj. Featured Certification, Outstanding Certification.

David Robinson and David Goforth. The topology of the 2x2 games: a new periodic table, volume 3.

Psychology Press, 2005.

Tim Roughgarden. Intrinsic robustness of the price of anarchy. Journal of the ACM (JACM), 62(5):

1–42, 2015.

Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard,
Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. Peer: A collabora-
tive language model. International Conference on Learning Representations, 2023.

Shai Shalev-Shwartz. Online learning: Theory, algorithms, and applications. Hebrew University,

2007.

Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends®

in Machine Learning, 4(2):107–194, 2012.

Shai Shalev-Shwartz and Yoram Singer. A primal-dual perspective of online learning algorithms.

Machine Learning, 69:115–142, 2007.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging-
gpt: Solving AI tasks with chatgpt and its friends in huggingface. Neural Information Processing
Systems, 2023.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Re-
flexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on
Neural Information Processing Systems, 2023.

Significant Gravitas.

Autogpt,

2023.

URL

https://github.com/

Significant-Gravitas/AutoGPT.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam
Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri`a Garriga-Alonso, et al. Beyond the
imitation game: Quantifying and extrapolating the capabilities of language models. Transactions
on Machine Learning Research, 2023.

Melanie Swan, Takashi Kido, Eric Roland, and Renato P dos Santos. Math agents: Computational
infrastructure, mathematical embedding, and genomics. arXiv preprint arXiv:2307.02502, 2023.

Chen Feng Tsai, Xiaochen Zhou, Sierra S Liu, Jing Li, Mo Yu, and Hongyuan Mei. Can large
language models play text games well? current state-of-the-art and open questions. arXiv preprint
arXiv:2304.02868, 2023.

Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kamb-
hampati. Planbench: An extensible benchmark for evaluating large language models on planning
In Thirty-seventh Conference on Neural Information Processing
and reasoning about change.
Systems Datasets and Benchmarks Track, 2023.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa-
tion processing systems, 30, 2017.

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, Jo˜ao Sacramento, Alexander Mordv-
intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient
descent. In International Conference on Machine Learning, pp. 35151–35174. PMLR, 2023.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cam-

bridge university press, 2019.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan,
and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.
arXiv preprint arXiv:2305.16291, 2023a.

Xinyi Wang, Wanrong Zhu, and William Yang Wang. Large language models are implicitly topic
models: Explaining and finding good demonstrations for in-context learning. International Con-
ference on Machine Learning 2023 Workshop ES-FoMO, 2023b.

Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Xiao-
jiang Huang, Yanbin Lu, and Yingzhen Yang. Recmind: Large language model powered agent
for recommendation. arXiv preprint arXiv:2308.14296, 2023c.

Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and
select: Interactive planning with large language models enables open-world multi-task agents.
Advances in neural information processing systems, 2023d.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge:
An optimal black-box approach. In Conference on learning theory, pp. 4300–4354. PMLR, 2021.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo-
gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language
models. Transactions on Machine Learning Research, 2022a.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny
Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in
Neural Information Processing Systems, 35:24824–24837, 2022b.

Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen
Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen. A survey on large language models
for recommendation, 2024. URL https://arxiv.org/abs/2305.19860.

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li,
Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-
agent conversation framework. arXiv preprint arXiv:2308.08155, 2023a.

Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. Smartplay: A benchmark for llms as

intelligent agents. arXiv preprint arXiv:2310.01557, 2023b.

Fanzeng Xia, Hao Liu, Yisong Yue, and Tongxin Li. Beyond numeric awards: In-context dueling

bandits with llm agents. arXiv preprint arXiv:2407.01887, 2024.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context
learning as implicit bayesian inference. International Conference on Learning Representations,
2022.

Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining inter-consistency of large
language models collaboration: An in-depth analysis via debate. In Houda Bouamor, Juan Pino,
and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP
2023, pp. 7572–7590, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.508. URL https://aclanthology.org/2023.
findings-emnlp.508.

17

Under review as a conference paper at ICLR 2025

Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu.
Exploring large language models for communication games: An empirical study on werewolf.
arXiv preprint arXiv:2309.04658, 2023a.

Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. Language agents with reinforcement learning

for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023b.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik
Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances
in Neural Information Processing Systems, 2023a.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models. International Conference on Learn-
ing Representations, 2023b.

H Peyton Young. Strategic learning and its limits. OUP Oxford, 2004.

Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tian-
min Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language
models. International Conference on Learning Representations, 2024.

Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context.

arXiv preprint arXiv:2306.09927, 2023a.

Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context
learning learn? bayesian model averaging, parameterization, and generalization. arXiv preprint
arXiv:2305.19420, 2023b.

Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Com-
peteai: Understanding the competition behaviors in large language model-based agents. arXiv
preprint arXiv:2310.17512, 2023.

Julian Zimmert and Yevgeny Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial

bandits. The Journal of Machine Learning Research, 22(1):1310–1358, 2021.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In

International Conference on Machine Learning, pp. 928–936, 2003.

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

18

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Supplementary Materials for
“Do LLM Agents Have Regret? A Case Study in Online Learning and Games”

CONTENTS

1 Introduction

2 Preliminaries

2.1 Online Learning & Games

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.2 Performance Metric: Regret

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 Do Pre-Trained LLMs Have Regret? Experimental Validation

3.1 Framework for Sublinear Regret Behavior Validation . . . . . . . . . . . . . . . .

3.2 Results: Online Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3 Results: Multi-Player Repeated Games . . . . . . . . . . . . . . . . . . . . . . . .

3.4 Pre-Trained LLM Agents Can Still Have Regret . . . . . . . . . . . . . . . . . . .

4 Why Do Pre-Trained LLMs (Not) Have Regret? A Hypothetical Model and Some The-

oretical Insights

4.1 A (Human) Decision-Making Model: Quantal Response

. . . . . . . . . . . . . .

4.2 Case Study: Pre-Training under Canonical Data Distribution . . . . . . . . . . . .

5 Provably Promoting No-Regret Behavior by a New Loss

5.1 A New Unsupervised Training Loss: Regret-Loss . . . . .

. . . . .

. . . . .

. . .

5.2 Generalization and Regret Guarantees of Regret-Loss Minimization . . . . . . . .

5.3 Regret-Loss Trained Transformers Can be Online Learning Algorithms

. . . . . .

5.4 Experimental Results for Regret-Loss Trained Transformers

. . . . . . . . . . . .

A Related Work

A.1 Comparison with Concurrent Work Krishnamurthy et al. (2024)

. . . . . . . . . .

B Deferred Background

B.1 Notation .

.

.

.

.

.

.

.

.

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.2 Additional Definitions for Appendix . . . . . . . . . . . . . . . . . . . . . . . . .

B.3 In-Context Learning .

.

.

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.4 Online Learning Algorithms

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B.5 Why Focusing on Linear Loss Function? . . . . . . . . . . . . . . . . . . . . . . .

B.6 Six Representative General-Sum Games . . . . . . . . . . . . . . . . . . . . . . .

C Deferred Results and Proofs in Section 3

C.1 Intuition why pre-trained language models may exhibit no-regret behavior . . . . .

19

1

2

2

3

3

4

4

5

6

7

7

8

9

9

10

10

10

22

23

25

25

25

26

26

27

28

28

28

Under review as a conference paper at ICLR 2025

C.2 Visualization of Interaction Protocols

. . . . . . . . . . . . . . . . . . . . . . . .

C.3 Frameworks for No-Regret Behavior Validation . . . . . . . . . . . . . . . . . . .

C.4 Deferred Experiments for Non-stationary Environments in Section 3.2 . . . . . . .

C.5 Deferred Experiments for Bandit-feedback Environments in Section 3.2 . . . . . .

C.6 Additional Figures for Section 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . .

C.7 Additional Results for Section 3.4 . . . . . . . . . . . . . . . . . . . . . . . . . .

C.8 Ablation Study on Prompts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.9 Results for GPT-4 Turbo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.10 LLM Agents’ Explanation on Their Output Policies . . . . . . . . . . . . . . . . .

C.11 Case Studies on Real-world Applications

. . . . . . . . . . . . . . . . . . . . . .

C.11.1 Sequential Recommendation . . . . . . . . . . . . . . . . . . . . . . . . .

C.11.2 Interactive Negotiation . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D Deferred Results and Proofs in Section 4

D.1 Pre-Trained LLMs Have Similar Regret as Humans (Who Generate Data)

. . . . .

D.2 Background and Motivations for (Generalized) Quantal Response

. . . . . . . . .

D.3 The Example Instantiating Assumption 1

. . . . . . . . . . . . . . . . . . . . . .

D.4 Alignment of Assumption 1 with Quantal Response . . . . . . . . . . . . . . . . .

D.5 Relationship between FTPL and Definition 4.1 . . . . . . . . . . . . . . . . . . .

D.6 Formal Statement and Proof of Theorem 4.1 . . . . . . . . . . . . . . . . . . . . .

D.6.1 Implications of Theorem 4.1 for Repeated Games . . . . . . . . . . . . . .

D.7 Extending Theorem 4.1 with Relaxed Assumptions . . . . . . . . . . . . . . . . .

D.7.1 Relaxation under More General Data Distributions . . . . . . . . . . . . .

D.7.2 Relaxation under Decision-Irrelevant Pre-Training Data

. . . . . . . . . .

D.8 Comparison with Lee et al. (2023); Lin et al. (2024); Liu et al. (2023e) . . . . . . .

D.9 Details of Estimating the Parameters of Our Hypothetical Model

. . . . . . . . . .

E Deferred Results and Proofs in Section 5

E.1 Basic Lemmas .

.

.

.

.

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.2 Deferred Proof for the Arguments in Section 5.1 . . . . . . . . . . . . . . . . . . .

E.3 Definition of the Empirical Loss Function . . . . . . . . . . . . . . . . . . . . . .

E.4 Deferred Proofs of Theorem E.1 and Theorem 5.1 . . . . . . . . . . . . . . . . . .

E.5 Detailed Explanation of Optimizing Equation (5.2) with Single-layer Self-attention
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Model

.

.

.

.

.

.

.

.

.

.

.

E.6 Deferred Proof of Theorem E.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.7 Deferred Proof of Theorem 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.8 Empirical Validation of Theorem E.2 and Theorem 5.2 . . . . . . . . . . . . . . .

E.8.1 Empirical Validation of Theorem E.2 . . . . . . . . . . . . . . . . . . . .

E.8.2 Empirical Validation of Theorem 5.2 . . . . . . . . . . . . . . . . . . . . .

20

30

30

31

33

34

35

36

39

39

41

41

41

44

44

45

46

46

47

47

51

51

51

53

53

54

54

54

54

59

59

63

63

66

71

71

71

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

E.9 Discussions on the Production of FTRL with Entropy Regularization . . . . . . . .

E.9.1 Numerical Analysis of Step 2 and Step 4 . . . . . . . . . . . . . . . . . .

E.9.2 Empirical Validation .

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.10 Comparison with Ahn et al. (2023); Zhang et al. (2023a); Mahankali et al. (2023) .

E.11 Details of Section 5.4 .

.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

E.11.1 Training Details of Section 5.4 . . . . . . . . . . . . . . . . . . . . . . . .

E.12 Ablation Study on Training Equation (5.2) . . . . . . . . . . . . . . . . . . . . . .

F Limitations and Concluding Remarks

71

75

76

76

76

79

79

82

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

21

Under review as a conference paper at ICLR 2025

A RELATED WORK

LLM(-agent) for decision-making. The impressive capability of LLMs for reasoning (Bubeck
et al., 2023; Achiam et al., 2023; Wei et al., 2022b;a; Srivastava et al., 2023; Yao et al., 2023a) has
inspired a growing line of research on LLM for (interactive) decision-making, i.e., an LLM-based
autonomous agent interacts with the environment by taking actions repeatedly/sequentially, based
on the feedback it perceives. Some promises have been shown from a planning perspective (Hao
et al., 2023; Valmeekam et al., 2023; Huang et al., 2022b; Shen et al., 2023).
In particular, for
embodied AI applications, e.g., robotics, LLMs have achieved impressive performance when used
as the controller for decision-making (Ahn et al., 2022; Yao et al., 2023b; Shinn et al., 2023; Wang
et al., 2023d; Driess et al., 2023; Significant Gravitas, 2023). However, the performance of decision-
making has not been rigorously characterized via the regret metric in these works. Very recently, Liu
et al. (2023e) has proposed a principled architecture for LLM-agent, with provable regret guarantees
in stationary and stochastic decision-making environments, under the Bayesian adaptive Markov
decision processes framework. In contrast, our work focuses on online learning and game-theoretic
settings, in potentially adversarial and non-stationary environments. Moreover, (first part of) our
work focuses on evaluating the intelligence level of LLM per se in decision-making (in terms of the
regret metric), while Liu et al. (2023e) focused on developing a new architecture that uses LLM as
an oracle for reasoning, together with memory and specific planning/acting subroutines, to achieve
sublinear (Bayesian) regret, in stationary and stochastic environments.

LLMs in multi-agent environments. The interaction of multiple LLM agents has garnered sig-
nificant attention lately. For example, Fu et al. (2023) showed that LLMs can autonomously improve
each other in a negotiation game by playing and criticizing each other. Similarly, (Du et al., 2023;
Liang et al., 2023; Xiong et al., 2023; Chan et al., 2024; Li et al., 2023c) showed that multi-LLM
debate can improve the reasoning and evaluation capabilities of the LLMs. Qian et al. (2023);
Schick et al. (2023); Wu et al. (2023a) demonstrated the potential of multi-LLM interactions and
collaboration in software development, writing, and problem-solving, respectively. Zhang et al.
(2024) exhibited a similar potential in embodied cooperative environments. More formally, multi-
LLM interactions have also been investigated under a game-theoretic framework, to characterize the
strategic decision-making of LLM agents. Bakhtin et al. (2022); Mukobi et al. (2023) and Xu et al.
(2023b;a) have demonstrated the promise of LLMs in playing Diplomacy and WereWolf games,
respectively, which are both language-based games with a mixture of competitive and cooperative
agents. Note that these works utilized LLM to solve a specific rather than a general game. Related
to our work, Brookins & DeBacker (2023); Akata et al. (2023); Lor`e & Heydari (2023); Brookins
& DeBacker (2023); Fan et al. (2023) have also used (repeated) matrix games as a benchmark to
evaluate the reasoning capability and rationality of LLM agents. In contrast to our work, these em-
pirical studies have not formally investigated LLM agents using the metric of regret, nor through the
lenses of online learning and equilibrium-computation, which are all fundamental in modeling and
analyzing strategic multi-agent interactions. Moreover, our work also provides theoretical results to
explain and further enhance the no-regret property of LLM agents.

LLMs & Human/Social behavior. LLMs have also been used to simulate the behavior of hu-
man beings, for social science and economics studies (Engel et al., 2023). The extent of LLMs
simulating human behavior has been claimed as a way to evaluate the level of its intelligence in a
controlled environment (Aher et al., 2023; Tsai et al., 2023). For example, Li et al. (2023b); Hong
et al. (2024); Zhao et al. (2023) showed that by specifying different “roles” to LLM agents, certain
collaborative/competitive behaviors can emerge. Argyle et al. (2023) showed that LLMs can emulate
response distributions from diverse human subgroups, illustrating their adaptability. Horton (2023)
argued that an LLM, as a computational model of humans, can be used as homo economicus when
given endowments, information, preferences, etc., to gain new economic insights by simulating its
interaction with other LLMs. Park et al. (2022; 2023) proposed scalable simulators that can generate
realistic social behaviors emerging in populated and interactive social systems, and the emerging be-
haviors of LLM agents in society have also been consistently observed in Chen et al. (2024; 2023).
Li et al. (2023d;a) studied the behavioral dynamics of LLM agents on social networks. These empir-
ical results have inspired our work, which can be viewed as an initial attempt towards quantitatively
understanding the emerging behavior of LLMs as computational human models, given the known

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

justification of equilibrium being a long-run emerging behavior of learning dynamics (Fudenberg &
Levine, 1998) and strategic interactions (Young, 2004; Camerer, 2011).

Transformers & In-context-learning. LLMs nowadays are predominantly built upon the archi-
tecture of Transformers (Vaswani et al., 2017). Transformers have exhibited a remarkable capacity
of in-context-learning (ICL), which can construct new predictors from sequences of labeled exam-
ples as input, without further parameter updates. This has enabled the few-shot learning capability
of Transformers (Brown et al., 2020; Garg et al., 2022; Min et al., 2022). The empirical successes
have inspired burgeoning theoretical studies on ICL. Xie et al. (2022) used a Bayesian inference
framework to explain how ICL works, which has also been adopted in Wang et al. (2023b); Jiang
(2023). Aky¨urek et al. (2023); Von Oswald et al. (2023); Dai et al. (2023); Giannou et al. (2023)
showed (among other results) that ICL comes from that Transformers can implement the gradient
descent (GD) algorithm. Bai et al. (2023) further established that Transformers can implement a
broad class of machine learning algorithms in context. Moreover, Ahn et al. (2023); Zhang et al.
(2023a); Mahankali et al. (2023) proved that a minimizer of the certain training loss among single-
layer Transformers is equivalent to a single step of GD for linear regression. Li et al. (2023e) es-
tablished generalization bounds of ICL from a multi-task learning perspective. Zhang et al. (2023b)
argued that ICL implicitly implements Bayesian model averaging, and can be approximated by the
attention mechanism. They also established a result on some regret metric. However, the regret
notion is not defined for (online) decision-making, and is fundamentally different from ours that
is standard in online learning and games. Also, we provide extensive experiments to validate the
no-regret behavior by our definition. More recently, the ICL property has also been generalized
to decision-making settings. Laskin et al. (2023); Lee et al. (2023); Lin et al. (2024) investigated
the in-context reinforcement learning (RL) property of Transformers under supervised pre-training,
for solving stochastic bandits and Markov decision processes. In contrast, our work focuses on on-
line learning settings with an arbitrary and potentially adversarial nature, as well as game-theoretic
settings. We also provide a new unsupervised loss to promote the no-regret behavior in our settings.

Online learning and games. Online learning has been extensively studied to model the decision-
making of an agent who interacts with the environment sequentially, with a potentially arbitrary
sequence of loss functions (Shalev-Shwartz, 2012; Hazan, 2016), and has a deep connection to game
theory (Cesa-Bianchi & Lugosi, 2006). In particular, regret, the difference between the incurred
accumulated loss and the best-in-hindsight accumulated loss, has been the core performance metric,
and a good online learning algorithm should have regret at most sublinear in time T , which is
referred to as being no-regret. Many well-known algorithms can achieve no-regret against arbitrary
loss sequences, e.g., multiplicative weight updates (MWU)/Hedge (Freund & Schapire, 1997; Arora
et al., 2012b), EXP3 (Auer et al., 2002), and more generally follow-the-regularized-leader (FTRL)
(Shalev-Shwartz & Singer, 2007) and follow-the-perturbed-leader (FTPL) (Kalai & Vempala, 2005).
In the bandit literature (Lattimore & Szepesv´ari, 2020; Bubeck et al., 2012), such a setting without
any statistical assumptions on the losses is also referred to as the adversarial/non-stochastic setting.
Following the conventions in this literature, the online settings we focus on shall not be confused
with the stationary and stochastic(-bandit)/(-reinforcement learning) settings that have been explored
in several other recent works on Transformers for decision-making (Lee et al., 2023; Lin et al., 2024).
Centering around the regret metric, our work has also explored the non-stationary bandit setting
(Besbes et al., 2014), as well as the repeated game setting where the environment itself consists of
strategic agents (Cesa-Bianchi & Lugosi, 2006).

A.1 COMPARISON WITH CONCURRENT WORK KRISHNAMURTHY ET AL. (2024)

After submitting the first version of our manuscript, we were aware of a concurrent work Krishna-
murthy et al. (2024), which considered using LLMs to solve multi-arm stochastic bandit problems
entirely in-context, with a focus on the exploration behaviors of LLMs. Specifically, Krishnamurthy
et al. (2024) claimed that LLMs may not show robust exploratory behaviors under a variety of
prompt configurations, although there does exist some successful prompt configuration that enabled
satisfactory exploratory behaviors. We here provide a detailed comparison with Krishnamurthy et al.
(2024) as follows.

• (Focused settings.) We mainly considered the full-information setting with potentially
adversarial loss vectors, as well as the multi-agent repeated-game setting. In contrast, Kr-

23

Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

ishnamurthy et al. (2024) focused on the stochastic setting with bandit feedback, where the
loss vectors at different rounds are drawn i.i.d. from a fixed distribution. Therefore, both
the metrics and most results are not directly comparable. For example, i) some failure cases
in Krishnamurthy et al. (2024) for stochastic bandits did not appear in our setting (as will be
detailed next); ii) for some adversarial loss instances (e.g., those from Feder et al. (1992),
see the introduction of the loss vectors in Section 3.4), the summarized history input that
was claimed essential in Krishnamurthy et al. (2024) is not very effective in our settings,
while a raw-history input as in our experiments is highly effective (see Section 3.4 and Fig-
ure C.6); iii) as studied in Krishnamurthy et al. (2024), uniform-like behaviors constitute
one of the main failures in stochastic bandits. However, uniform-like policies are not nec-
essarily failure cases in our setting, especially when the loss vectors are highly adversarial
(c.f. examples in Section 3.4). In particular, such a metric may be irrelevant/inapplicable
to validating the no-regret behaviors in our full-information non-stochastic/adversarial set-
tings. These results/facts demonstrated the fundamentally different features in addressing
the distinct settings in both works.

• (Configuration/Prompt design choices.) Despite the negative results under many prompt
configurations, Krishnamurthy et al. (2024) still found one successful prompt configura-
tion that can lead to robust exploratory behaviors in stochastic bandits, which in fact shares
many similarities with our default prompt configurations. For example, Krishnamurthy
et al. (2024) found that asking the LLMs to output a distribution over the action space
(instead of one single action) can address the suffix failure for stochastic bandits, which
was indeed the default prompt we used in our settings. Moreover, as a standard technique,
our default prompt asked the model to have the Chain-of-Thought (CoT) procedure, while
Krishnamurthy et al. (2024)’s successful prompt also emphasized the importance of CoT.
Krishnamurthy et al. (2024) also showed the importance of summarizing the history, i.e.,
summarizing the mean reward associated with each arm, while we found that when we
feed the LLMs with (raw) full-information feedback in the vector form, the LLMs may au-
tomatically choose to summarize the history and make decisions based on the summarized
statistics (c.f. the output examples in Appendix C.10).

• (Horizons v.s. No-regret behaviors.) In light of the findings from Krishnamurthy et al.
(2024) that LLMs may fail when the problem horizon is long, we conduct experiments on
problems with comparable horizons as in Krishnamurthy et al. (2024). Our results show
that for the full-information non-stochastic setting we focused on, LLMs are still no-regret
with longer horizons (Figure 3.2 and Table 1), under the loss sequences we studied before.

• (Results in bandit setting & Failure cases.) As an extension and sanity check of our
full-information-setting results, we have also experimented on the (adversarial) bandit set-
tings. This extension setting is more comparable to that in Krishnamurthy et al. (2024).
However, different from the focus therein, we did not ask the LLMs to directly explore in
context. Instead, we manually input a re-weighting estimate of the full-information loss
vector, a standard technique in online learning (Auer et al., 2002; Hazan, 2016; Lattimore
& Szepesv´ari, 2020), to strike the exploration-exploitation tradeoff. We viewed this ap-
proach as a natural way to exploit the no-regret behaviors of LLMs in the full-information
setting. In fact, with such a re-weighting, we show in Table 2 that the failure cases in Kr-
ishnamurthy et al. (2024) for the bandit setting may not appear, in the exact hard instance
proposed therein, and even under a relatively long horizon of T = 100. Complementing
Krishnamurthy et al. (2024), our bandit-setting results suggest that such human-intervened
input may enhance the decision-making behaviors of LLMs. Such pre-processing of the
input also mirrors the summary-based history input in Krishnamurthy et al. (2024), another
human-intervention of the input that was claimed essential in the stochastic bandit setting.

• (Technical analyses.) Additionally, we conducted theoretical analyses on both the insights
from our experimental results, and on the newly developed regret-loss, with provable statis-
tical guarantee and optimization guarantee under the Transformer architecture, which are
novel and non-trivial technical contributions of independent interest.

24

Under review as a conference paper at ICLR 2025

Hard MAB instance of
Krishnamurthy et al. (2024)
Media reward
(higher is better)
SuffFailFreq(T /2)
(lower is better)

n

MinFrac
∗

(lower is better)

TS

UCB

0.47

0.55

0.01

0.02

0.28

0.18

Successful case of
Krishnamurthy et al. (2024)

Ours (GPT-4o) Ours (GPT-4)

0.47

0.00

0.33

0.475

0.00

0.1

0.46

0.00

0.27

Table 2: Comparing Thompson Sampling (TS), Upper Confidence Bound (UCB), and the successful
prompt configuration of Krishnamurthy et al. (2024) (from Figure 4 therein) with our re-weighting-
based GPT-4 (GPT-4o) output, on the hard MAB instance. Specifically, as introduced in Krishna-
murthy et al. (2024), for this hard instance, rewards associated with each arm follow a Bernoulli
distribution, the horizon is T = 100, the number of actions is n = 5, and the reward gap is 0.2.
As indicated by SuffFailFreq(T /2) and n
MinFrac proposed by Krishnamurthy et al. (2024), our
∗
importance re-weighting techniques on bandit problems have enabled the LLMs to avoid both the
suffix failures and the uniform-like failures even without any external history summarization, and
achieve comparable rewards with the other three approaches.

B DEFERRED BACKGROUND

B.1 NOTATION

⟩

}

{

p

x

(cid:80)

(cid:17)

S

S

∈

∈

−

×

−

| ∥

Rd

i∈[d]

∥ ≤

p
∥

) :=

S
∈

. For d

∥ · ∥
(cid:16)

exi
i∈[d] exi

, we use ∆(

1, 2, . . . , d
{

Rd, R > 0 and C

N+, we define [d] :=

We use N and N+ to denote the sets of non-negative and positive integers, respectively. For a finite
) to denote the simplex over
set
. For
}
∈
Rd, we use
to denote the inner product of x and y. We define 000d and 111d as a
two vectors x, y
x, y
⟨
d-dimensional zero or one vector, and OOOd×d and Id×d as a d
d-dimensional zero matrix and identity
matrix, respectively. We omit d when it is clear from the context. We define ei as a unit vector (with
Rd is a convex
proper dimension) whose i-th coordinate equal to 1. For p
set, define B(p, R,
x
R
x
(which
is well defined as C is a convex set), and clipR(x) := [ProjB(0,R,∥·∥2),∥·∥2(xi)]i∈[d]. Define
Rm×n with
Softmax(x) :=
and ReLU(x) = max(0, x) for x
Ai∥2,
op := max∥x∥2≤1 ∥
Ai denoting its i-th column, we define
∥2,
∥F as the Frobenius norm, and A−1 := An to denote the last column vector of A. We define
A
∥
R+ :=
) := supπ1,π2∈Π ∥
x
. We define
|
{
1(
) := 0 otherwise. For a random variable sequence (Xn)n∈N and
) := 1 if
random variables X, Y , we denote FX as the cumulative distribution function of a random variable
X, Xn
X if limn→∞ FXn (x) =
FX (x) for all x where FX (x) is continuous, X d= Y if FX (x) = FY (x) for all x, Xn
a.s.
X
→
if P(limn→∞ Xn = X) = 1, and esssup(X) := inf
R : P(X > M ) = 0
. For a
}
random variable X, we use supp(X) to denote its support. For functions f, g : R
R, we define
→
for all x > x0. We use f ′ to
g(x) =
f (x)
∞
|
R be a continuously-differentiable, strictly convex function
denote the derivative of f . Let F : Ω
defined on a convex set Ω. The Bregman divergence associated with F for points p, q is defined as
N+, we define
DF (p, q) := F (p)
ℓa:b := (ℓa,

⊆
, ProjC,∥·∥(p) = arg minx∈C ∥
Rd. For A
∈
∥2,∞ := supi∈[n] ∥
π2∥

. For a sequence (ℓt)t∈[T ] for some T
⟩

ϵ > 0, limn→∞ P(
|
∀

≥
is true, and 1(
E

M
{
g(x)
|

. For a set Π, define diam(Π,

(f (x)) if there exist x0, M <

T . If a > b, we define ℓa:b =

→
F (q), p
b

F (q)
−
, ℓb) for 1

> ϵ) = 0, Xn

Xn −

π1 −

such that

∈
M

A
∥

A
∥

p
→

d
→

X if

∥ · ∥

0
}

| ≤

Ax

X

O

−

∈

∈

x

E

E

∥

q

|

|

.

· · ·

− ⟨∇
a
≤
≤

≤

∅

B.2 ADDITIONAL DEFINITIONS FOR APPENDIX

(Linear) Self-attention. One key component in Transformers (Vaswani et al., 2017), the backbone
of modern language models, is the (self-)attention mechanism. For simplicity, we here focus on
introducing the single-layer self-attention architecture. The mechanism takes a sequence of vectors
Rd×t. For each
Rd×t as input, and outputs some sequence of [(cid:98)z1, . . . , (cid:98)zt]
Z = [z1, . . . , zt]
∈
[t] where i > 1, the output is generated by (cid:98)zi = (V z1:i−1)σ((Kz1:i−1)⊺(Qzi)), where z1:i−1
i
1 columns of Z, σ is either the Softmax or ReLU activation function, and for the
denotes the 1 to i
−
Rd×d are referred to as the Value, Query, and Key matrices,
initial output, (cid:98)z1 = 000d. Here, V, Q, K
respectively. Following the theoretical framework in Von Oswald et al. (2023); Mahankali et al.

∈

∈

∈

25

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Under review as a conference paper at ICLR 2025

(Vm, Qm, Km)
{
θ
M-(L)SA as

(2023), we exclude the attention score for a token zi in relation to itself. For theoretical analysis, we
also consider the linear self-attention model, where (cid:98)zi = (V z1:i−1)((Kz1:i−1)⊺(Qzi)). We write
this (linear) self-attention layer’s output as (L)SA(V,Q,K)(Z). We define an M -head self-attention
}m∈[M ] as M-(L)SAθ(Z) := (cid:80)M
m=1 (L)SA(Vm,Qm,Km)(Z). We
layer with θ =
Vm∥
Km∥
define
op
∥
Rd×t as
Transformers. For a multi-layer perceptron (MLP) layer, it takes Z = [z1, . . . , zt]
[t], the output is
input, with parameter θ = (W1, W2)
×
(cid:98)zi := W2σ(W1zi) where σ is either Softmax or ReLU. We write the output of an MLP layer with
parameter θ as MLPθ(Z). Defining
op and ResNet(f, Z) :=
MLP :=
θ
∥
∥
Z + f (Z), we can define an L-layer Transformer with parameter θ = (θ(lm), θ(la))l∈[L] as

M-(L)SA := maxm∈[M ] {∥

such that for each i

W2∥
∥

W1∥
∥

+ (cid:80)M

Qm∥

m=1 ∥

Rd′×d

Rd×d′

MLP as

op +

∥ · ∥

∥ · ∥

op,

op.

∈

∈

∈

∥

∥

}

TFθ(Z) := Z (L),

where the output Z (L) is defined iteratively from Z (0) = clipR(Z) := min(
(cid:16)

(cid:16)

(cid:16)

−

Z (l) = clipR

ResNet

MLPθ(la), ResNet

M-(L)SAθ(lm) , Z (l−1)(cid:17)(cid:17)(cid:17)

,

R, max(R, Z)) and

θ

for some R > 0. We define a class of Transformers with certain parameters as Θd,L,M,d′,BTF :=
θ = (θ(lm), θ(la))l∈[L],m∈[M ] :
, where M is the number of heads of self-attention,
{
(cid:111)

M-(L)SA +
∥
and BTF > 0 is some constant. When it is clear from the context, we may omit the subscripts and
write it as Θ for simplicity. We assume R to be sufficiently large such that clip does not take effect
on any of our approximation results.

BTF
}
≤
∥
(cid:110)
θ(la)
TF := max
∥
∥
l∈[L]

θ(lm)
∥

MLP
∥

(B.1)

θ
∥

TF

∥

,

B.3

IN-CONTEXT LEARNING

In-context learning is an emergent behavior of LLMs (Brown et al., 2020), which means that these
models can adapt and learn from a limited number of examples provided within their immediate
input context. In in-context learning, the prompt is usually constituted by a length of T in-context
(independent) examples (xt, yt)t∈[T ] and (T + 1)-th input xT +1, so the LLM((zt)t∈[T ], xT +1) pro-
vides the inference of yT +1, where zt = (xt, yt).

B.4 ONLINE LEARNING ALGORITHMS

Follow-the-regularized-leader (FTRL). The follow-the-regularized-leader algorithm (Shalev-
Shwartz, 2007) is an iterative method that updates policy based on the observed data and a regu-
larization term. The idea is to choose the next policy that minimizes the sum of the past losses and
a regularization term.

Mathematically, given a sequence of loss vectors ℓ1, ℓ2, . . . , ℓt, the FTRL algorithm updates the
policy π at each time step t as follows:

πt+1 = arg min
π∈Π

(cid:32) t

(cid:88)
⟨

ℓi, π

⟩

(cid:33)

+ R(π)

,

i=1
where R(π) is a regularization term. The regularization term R(π) is introduced to prevent overfit-
ting and can be any function that penalizes the complexity of the model. A function R(π) is said to
be λ-strongly convex with respect to a norm

if for all π, π′

Π:

∥ · ∥

∈

R(π)

≥

R(π′) +

R(π′), π

⟨∇

π′

⟩

−

+

λ
2 ∥

π

π′

2
2.
∥

−

A key property that ensures the convergence and stability of the FTRL algorithm is the strong con-
vexity of the regularization term R(π). Strong convexity of R(π) ensures that the optimization
problem in FTRL has a unique solution. The FTRL algorithm’s flexibility allows it to encompass a
wide range of online learning algorithms, from gradient-based methods like online gradient descent
to decision-making algorithms like Hedge (Freund & Schapire, 1997).

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

Connection to online gradient descent (OGD). The Online Gradient Descent (OGD) (Cesa-
Bianchi et al., 1996) algorithm is a special case of the FTRL algorithm when the regularization
2 and Π = Rd. In OGD, at each time step t, the
term is the L2-norm square, i.e., R(π) = 1
2
π
2 ∥
∥
policy π is updated using the gradient of the loss function:
ℓt.
πt+1 = πt −
Therefore, the connection between FTRL and OGD can be seen by observing that the update rule
for FTRL with L2-regularization can be derived from the OGD update rule.

Connection to the Hedge algorithm. The Hedge algorithm (Freund & Schapire, 1997) (also re-
ferred to as the Multiplicative Weight Update algorithm (Arora et al., 2012b)) is an online learning
algorithm designed for problems where the learner has to choose from a set of actions (denoted as
) at each time step and suffers a loss based on the chosen action. The FTRL framework can be
A
used to derive the Hedge algorithm by considering an entropy regularization term. Specifically, the
regularization term is the negative entropy R(π) = (cid:80)
j∈[d] πj log πj (where d is the dimension of
policy π), then the FTRL update rule yields the Hedge algorithm as

π(t+1)j = πtj

for j

[d].

∈

ℓtjπtj)

(cid:80)

exp(
i∈[d] exp(

−

ℓtiπti)

−

Follow-the-perturbed-leader (FTPL). Given a sequence of loss vectors ℓ1, ℓ2, . . . , ℓt−1, the
follow-the-perturbed-leader algorithm (Kalai & Vempala, 2005) at each time step t adds a random
perturbation vector ϵt to the original loss vectors and then selects the best-response action at (that is
potentially randomized due to ϵt) by solving:

at ∈

arg min
a∈A

ϵta +

t−1
(cid:88)

i=1

ℓia,

where the perturbation ϵt is sampled from a pre-defined distribution. Correspondingly, the policy πt
is chosen by following equation:

(cid:34)

πt = E

arg min
π∈Π⟨

ϵt, π

⟩

+

(cid:35)

ℓi, π

⟩

.

t−1
(cid:88)
⟨

i=1

(B.2)

Relationship between FTRL and FTPL. The FTRL and FTPL algorithms are deeply related. For
example, FTPL with perturbations of Gumbel distribution and FTRL with Entropy Regularization
(i.e., Hedge) are equivalent. In general, for the FTPL algorithm with any perturbation distribution,
one can always find an FTRL algorithm with a particular regularization such that their update rule is
equivalent. However, this relationship does not hold vice versa. For example, Hofbauer & Sandholm
(2002) showed that for FTRL with log barrier regularization, there does not exist an equivalent
perturbation distribution for FTPL.

Restarting techniques for non-stationary online learning. For non-stationary online learning
problems, one common technique is restarting: one restarts the standard online learning algorithm
periodically (Besbes et al., 2014) (see also e.g., Wei & Luo (2021); Mao et al. (2020)). After each
restarting operation, the algorithm will ignore the previous history and execute as if it is the begin-
ning of the interaction with the environment. Since the variation of the loss sequences is bounded,
loss sequences between two consecutive restarting operations can be regarded as being almost sta-
tionary, which makes achieving an overall sublinear dynamic regret guarantee possible.

B.5 WHY FOCUSING ON LINEAR LOSS FUNCTION?

We note that focusing on the linear loss function ft(π) :=
ality. Specifically, for the general convex loss function (ft)t∈[T ], we have ft(πA ,t)

ℓt, π

does not lose much of gener-

⟨

⟩

ft(πA ,t), πA ,t −

⟨∇

π

for any π

Π, which indicates

RegretA

⟩
(cid:0)(ft)t∈[T ]

(cid:1)

∈
T
(cid:88)

≤

t=1

E[

]
ft(πA ,t), πA ,t⟩

⟨∇

−

inf
π∈Π

T
(cid:88)

t=1

E[

⟨∇

ft(πA ,t), π

27

ft(π)

≤

−

].

⟩

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Under review as a conference paper at ICLR 2025

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Therefore, one can regard the loss vector (ℓt)t∈[T ] as ℓt :=
[T ], and control the
actual regret by studying the linear loss function (Hazan, 2016). The same argument on the general
convex ft can be applied to the dynamic-regret metric as well. In sum, an algorithm designed for
online linear optimization can be adapted to solve online convex optimization, with the understand-
ing that the instance received at round t corresponds to the gradient of the convex function evaluated
at the policy in that round.

ft(πA ,t) for t

∇

∈

B.6 SIX REPRESENTATIVE GENERAL-SUM GAMES

In game theory, there are six representative two-player general-sum games (Robinson & Goforth,
(cid:19)
4
2

2005). Firstly, consider the win-win game represented by matrices A =

and B =

(cid:18)1
1

(cid:18)1
1

4
2

(cid:19)

for players A and B, respectively. This setup fosters a cooperative dynamic, as both players receive
identical payoffs, encouraging strategies that benefit both parties equally.
(cid:18)1
2

In contrast, the prisoner’s dilemma, depicted by payoff matrices A =

and B =

(cid:18)4
2

(cid:19)
,

3
1

3
4

(cid:19)

illustrates the conflict between individual and collective rationality, where players are tempted to
pursue individual gain at the collective’s expense, often resulting in suboptimal outcomes for both.
(cid:19)
1
4

In the unfair game, represented by A =

, the asymmetry in the payoff

and B =

(cid:18)2
3

(cid:18)4
1

3
2

(cid:19)

structure places one player at a disadvantage, regardless of the chosen strategy. This imbalance often
reflects real-world scenarios where power or information asymmetry affects decision-making.
(cid:19)
1
4

The cyclic game, with matrices A =

, presents a scenario where no

and B =

(cid:18)3
2

(cid:18)3
2

(cid:19)
4
1

stable equilibrium exists. The best strategy for each player changes in response to the other’s actions,
leading to a continuous cycle of strategy adaptation without a clear resolution.
(cid:18)3
1

The biased game, depicted by A =

, inherently favors one player, often

and B =

(cid:18)4
1

(cid:19)
2
3

(cid:19)
2
4

reflecting situations where external factors or inherent advantages influence outcomes, leading to
consistently unequal payoffs.

Finally, the second-best game, with payoff matrices A =

(cid:18)1
3

(cid:19)
2
4

and B =

(cid:18)1
3

4
2

(cid:19)

, encapsulates

scenarios where players settle for less-than-optimal outcomes due to constraints like risk aversion
or limited options. This often results in players choosing safer, albeit less rewarding, strategies.

Each of these games exemplifies distinct aspects of strategic decision-making and interactions. From
cooperative to competitive and fair to biased scenarios, these matrices provide a rich landscape for
exploring the nuances of decision-making behavior in game theory.

C DEFERRED RESULTS AND PROOFS IN SECTION 3

C.1

INTUITION WHY PRE-TRAINED LANGUAGE MODELS MAY EXHIBIT NO-REGRET
BEHAVIOR

Intuition why pre-trained language models may exhibit no-regret behavior. Transformer-
based LLMs have demonstrated impressive in-context-learning and few-/zero-shot learning capa-
bilities (Brown et al., 2020; Garg et al., 2022; Min et al., 2022). One theoretical explanation is
that, trained Transformers can implement the gradient descent algorithm on the testing loss in cer-
tain supervised learning problems (Aky¨urek et al., 2023; Von Oswald et al., 2023; Dai et al., 2023;
Ahn et al., 2023; Zhang et al., 2023a; Mahankali et al., 2023), which is inherently adaptive to the
loss function used at test time. On the other hand, it is known in online learning that the simple
algorithm of online gradient descent (Zinkevich, 2003) can achieve no-regret. Hence, it seems rea-
sonable to envision the no-regret behavior of such meta-learners in online learning, due to their fast
adaptability. However, it is not straightforward due to the fundamental difference between multi-
task/meta-learning and online learning settings, as well as the difference between stationary and

28

Under review as a conference paper at ICLR 2025

non-stationary/adversarial environments in decision-making. Next, we provide both experimental
and theoretical studies to validate this intuition.

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

29

Under review as a conference paper at ICLR 2025

C.2 VISUALIZATION OF INTERACTION PROTOCOLS

Figure C.1: Demonstration of the prompts and interaction protocol for multi-player repeated games.
A human moderator does not provide the game’s payoff matrices to the LLMs. Instead, at each
round, the human moderator provides each player’s own payoff vector history.

C.3 FRAMEWORKS FOR NO-REGRET BEHAVIOR VALIDATION

Trend-checking framework. We propose the following hypothesis test:
(cid:0)(fτ )τ ∈[t]
H0 : The sequence (cid:0)RegretA
(cid:0)(fτ )τ ∈[t]
H1 : The sequence (cid:0)RegretA

(cid:1) /t(cid:1)∞
(cid:1) /t(cid:1)∞

t=1 either diverges or converges to a positive constant
t=1 converges to 0 or a negative constant

with H0 and H1 denoting the null and alternative hypotheses, respectively. The notion of conver-
gence is related to T
by definition, making it challenging to verify directly. As an alternative,
we propose a more tractable hypothesis test, albeit a weaker one, that still captures the essence of
our objective:

→ ∞

H0 : The sequence (cid:0)RegretA
H1 : The sequence (cid:0)RegretA

(cid:0)(fτ )τ ∈[t]
(cid:0)(fτ )τ ∈[t]

(cid:1) /t(cid:1)
(cid:1) /t(cid:1)

t∈[T ] does not exhibit a decreasing pattern
t∈[T ] shows a decreasing pattern

where the “decreasing pattern” here refers to the case when more than 1/2 of the elements in the
(cid:1) /(t + 1). Note that we will
(cid:1) /t > RegretA
sequence satisfies that RegretA
only apply the framework when the sequence (cid:0)RegretA
t∈[T ] is non-negative, since a
negative regret is even more favorable and directly implies no-regret behaviors.

(cid:0)(fτ )τ ∈[t+1]
(cid:1) /t(cid:1)

(cid:0)(fτ )τ ∈[t]

(cid:0)(fτ )τ ∈[t]

(cid:1) /t

(cid:0)(fτ )τ ∈[t]

(cid:0)(fτ )τ ∈[t]

(cid:1) /t approaches zero or some negative constant as t
Ideally, one should check if RegretA
goes to infinity. With a finite T value, testing these hypotheses provides a method to quantify this
– whether we reject H0 offers a way to measure it. To this end, one needs to count the number of
(cid:1) /(t+1) > 0, for which we use Proposition 1 below
RegretA
to provide some understanding of (how small) the probability it happens under various counts. For
example, with the default choice of T = 25 in our experiments later, one can see from Proposition 1
(19, 25)) < 0.0035, PH0 (
that: PH0(
(21, 25)) < 0.00014, i.e., one can
E
easily reject H0 with high probability. We will report the p-value of H0, denoted as ptrend, as the
output of this framework.
Proposition 1. (p-value of the null hypothesis). Define the event

(17, 25)) < 0.032, PH0(

(cid:0)(fτ )τ ∈[t+1]

RegretA

−

E

E

(cid:40)

E(s, T ) :=

The number of

RegretA

(cid:1)

(cid:0)(fτ )τ ∈[t]
t

−

RegretA

(cid:0)(fτ )τ ∈[t+1]
t + 1

(cid:1)

> 0 for t = 1, . . . , T is at least s ≥

(cid:41)

.

T − 1
2

Under the assumption that the null hypothesis H0 holds, the probability of this event happening is
(cid:19)
1
bounded as PH0 (
E

(cid:80)T −1
t=s

(s, T ))

1
2T −1

(cid:18)T

−
t

≤

.

Proof. Under
RegretA

(cid:0)(fτ )τ ∈[t+1]

the null hypothesis H0,

(cid:1) /(t + 1) > 0 is less than 1

the probability p that RegretA

(cid:0)(fτ )τ ∈[t]
2 . Therefore, if we consider the event

(cid:1) /t
−
(s, T ),

E

30

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

𝒕=𝟐You are playing a matrix game for T rounds. There are A number of actions.. Human Moderator’s PromptYou are playing a matrix game for T rounds. There are A number of actions. At each round, you need to choose a policy; it specifies your probability of choosing each action. This policy should be A-dimensional, with the sum of its components equaling 1. After that, you are shown the reward vector for choosing each action. Remember, the reward vector is determined by an external system and can vary across rounds. It is not decided by what policies you have chosen. The reward vector is also A-dimensional. You can adjust your policy based on the reward vectors for all previous rounds. You’re required to provide your policy in numeric format. Your response’s last line should be formatted as ‘Policy: [your A-dimensional policy]’. Let’s think step by step. Explicitly examining history is important. Please explain how you chose the policy by guessing what reward you might receive for each action according to the history.𝒕=𝟏LLM 3: My decision is ...LLM 3: My decision is ...LLM 3: My decision is ...LLM 3: My decision is ...LLM 2: My decision is ...LLM 2: My decision is ...LLM 3: My decision is ...LLM 1: My decision is ...LLM 1: My decision is ...𝒕=𝑻Under review as a conference paper at ICLR 2025

we have

PH0 (
E

(s, T )) =

T −1
(cid:88)

k=s

pk(1

−

p)T −1−k

(cid:18)T

(cid:19)

1

−
k

1
2T −1

≤

T −1
(cid:88)

(cid:18)T

k=s

(cid:19)
1

−
k

(C.1)

since s

T −1
2 .

≥

{

at}
{

Regrett/t

On the underlying assumption for Equation (C.1). Our trend-checking framework was meant
T
t=1 for which we do not know beforehand how they were
to be designed for general sequences
generated, since in the online learning setting, by definition, there should be no prior assumption on
T
t=1 is generated, which very much depends on both how the loss sequences and
how
}
how the policies are generated (by the algorithms).
at)T
Our approach implicitly assumes that (at+1 −
t=1 is mutually independent. We used this as-
T
sumption since without knowing how
t=1 were generated. Meanwhile, it is possible
that the assumption might not be held since it depends on how loss sequences are generated or how
LLM behaves. However, it is possible that Equation (C.1) still holds approximately. Specifically,
we define

Regrett+1
t + 1
and treat (∆t)T
t=1 as random variables. We first compute the correlations among those random
variables in Figure C.2 using data from Section 3.2, where we can see that the correlations among
those random variables are indeed quite small. Meanwhile, this further implies that

Regrett/t
{

Regrett
t

∆t =

−

}

,

(cid:34) T

(cid:88)

E

(cid:35)

1[∆t > 0]

=

t=1

(cid:32) T

(cid:88)

t=1

Var

1[∆t > 0]

(cid:33)

≈

T
(cid:88)

t=1

T
(cid:88)

t=1

E [1[∆t > 0]] ,

Var (1[∆t > 0]) .

This implies that the random variable (cid:80)T
t=1 1[∆t > 0] indeed has the same first order and second
}t∈[T ] are independent. There-
order moment as the case that those random variable
}t∈[T ] to be independent) to be
fore, we regard a Binomial distribution (i.e., assuming
an acceptable approximation for the actual random variable (cid:80)T
t=1 1[∆t > 0], which finally gives
Equation (C.1). In fact, when binary random variables have weak correlations (but are not necessar-
ily independent), using the Binomial distribution as an approximation for their sum is common in
the engineering literature (Hoyland & Rausand, 2009).

1[∆t > 0]
{
1[∆t > 0]
{

C.4 DEFERRED EXPERIMENTS FOR NON-STATIONARY ENVIRONMENTS IN SECTION 3.2

Unif([0, 10]d). Then for each t
ℓt∥

We experiment on the setting where the losses are still changing over time, but their total vari-
ations across time are bounded, more concretely, sublinear in T . Correspondingly, we consider
the stronger metric of dynamic regret here to measure the performance. Note that without con-
straining the variation of the loss vectors, dynamic regret can be linear w.r.t. T in the worst case.
Hence, we generate the loss vectors in two different ways: 1) Gradual variation. We firstly sam-
2, we uniformly and randomly generate ℓt+1 under
ple ℓ1 ∼
1√
ℓt+1 −
t , such that the variations over time are guaranteed to satisfy
the constraint
∥
(cid:80)T −1
Unif([0, 10]d)
ℓt∥
ℓt+1 −
∞ = o(T ); 2) Abrupt variation. We randomly generate ℓ1 ∼
, T
and m time indices
[m], the sign of the
. At each time step ti for i
ti}i∈[m] from
}
{
· · ·
10111d −
ℓti. For the specific choice of T = 25 in our
loss vector ℓti is flipped, i.e., we let ℓti ←
experiments, we choose m = 3. For both cases, the average dynamic regret results are presented
in Table 3. GPT-4 achieves sublinear dynamic regret and outperforms FTRL/FTPL with Restart, a
standard variant of FTRL/FTPL for non-stationary online learning (see e.g., Besbes et al. (2014)).
We refer to Appendix B.4 for a detailed introduction of FTRL/FTPL with Restart.

1, 2,
{

t=1 ∥

≥

≤

∈

∞

31

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Under review as a conference paper at ICLR 2025

Figure C.2: The absolute value of Pearson correlation coefficient for the random variables
0]
}t∈[T ] and

∆t}t∈[T ] using data obtained in Section 3.2.
{

1[∆t >
{

Dynamic regret

GPT-4

Full
information

Gradual variation

Abrupt variation

Bandit

Gradual variation

Abrupt variation

12.61

7.01

±
(ptrend, (cid:98)β0, preg) = (0.0, 0.58, 0.0)

30.0

19.91

±
(ptrend, (cid:98)β0, preg) = (0.01, 0.87, 0.0)

21.39

10.86

±
(ptrend, (cid:98)β0, preg) = (0.0, 0.78, 0.0)

35.94

28.93

±
(ptrend, (cid:98)β0, preg) = (0.42, 0.95, 0.0)

GPT-3.5 Turbo
11.33
19.09

±
(ptrend, (cid:98)β0, preg) = (0.0, 0.83, 0.0)

33.65

22.51

±
(ptrend, (cid:98)β0, preg) = (0.08, 0.96, 0.0)

28.42

21.6

±
(ptrend, (cid:98)β0, preg) = (0.0, 0.83, 0.0)

30.76

25.48

±
(ptrend, (cid:98)β0, preg) = (0.92, 1.01, 0.0)

FTRL

FTPL

36.58

36.52

37.64

36.52

±

±

±

±

24.51

35.19

27.68

36.24

22.51

28.22

±

±

21.97

36.37

20.7

±

27.68

38.82

26.17

±

Table 3: Dynamic regret of GPT-3.5 Turbo/GPT-4 in a non-stationary environment with either full-
information or bandit feedback. Every experiment is conducted with 25 rounds. No-regret behaviors
of GPT-3.5 Turbo/GPT-4 are validated by both of our frameworks (low p-values and (cid:98)β0 < 1). The
only exception is GPT-3.5 Turbo on loss sequence with abrupt variations under bandit feedback. This
indicates that GPT-3.5 Turbo may not be capable of dealing with an abruptly changing environment
with limited feedback, although the average regret achieved eventually is still lower than that of
other baselines.

32

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

123456789101112131415161718192021222324Time steps123456789101112131415161718192021222324Time stepsCorrelations among {[t>0]}t[T]123456789101112131415161718192021222324Time steps123456789101112131415161718192021222324Time stepsCorrelations among {t}t[T]0.20.40.60.81.00.20.40.60.81.0Under review as a conference paper at ICLR 2025

Figure C.3: Regret of pre-trained LLMs for online learning with bandit feedback in 4 different
settings. It performs comparably and sometimes even better than well-known no-regret learning
algorithms, variants of FTRL and FTPL with bandit-feedback.

C.5 DEFERRED EXPERIMENTS FOR BANDIT-FEEDBACK ENVIRONMENTS IN SECTION 3.2

Although pre-trained LLMs have achieved good performance in online learning with full-
information feedback, it is unclear whether they can still maintain no-regret with only bandit feed-
back. For such problems, we modify the prompt and protocol of interactions slightly, where we
). In the
still ask the LLM agent to provide a policy πt at time step t, then sample one at ∼
bandit setting, the LLM agent can only access (at, ℓtat). Instead of directly feeding it to the agent,
we feed an estimate of the loss vector (cid:98)ℓt ∈
[d].
Note that such an operation of re-weighting the loss (when the loss is non-negative) by the inverse
of the probability is standard in online learning when adapting full-information-feedback no-regret
algorithms to the bandit-feedback ones (Auer et al., 2002). Later, we will also show the benefits of
such operations (c.f. Section 4). We compare the performance of pre-trained LLMs with that of the
counterparts of FTRL with bandit feedback, e.g., EXP3 (Auer et al., 2002) and the bandit-version of
FTPL (Abernethy et al., 2015), in both Figure C.3 and Table 3, where GPT-4 consistently achieves
lower regret.

πt(
·
1(at = a) for all j

Rd, where (cid:98)ℓt(a)

ℓt(a)
πt(a)

←

∈

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

33

0510152025Iterations010203040RegretUniformGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.76,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.89,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.0,0.9,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.0,0.81,0.0)FTRLFTPL0510152025Iterations010203040506070RegretGaussianGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.76,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.92,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.0,0.87,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.0,0.71,0.0)FTRLFTPL0510152025Iterations05101520253035RegretLinear-trendGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.56,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.55,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.0,0.86,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.0,0.69,0.0)FTRLFTPL0510152025Iterations0102030RegretSine-trendGPT-4:(ptrend,ˆβ0,preg)=(0.15,0.9,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.58,0.99,0.0)Mixtral-8x7b:(ptrend,ˆβ0,preg)=(0.42,0.96,0.0)Llama-3-70b:(ptrend,ˆβ0,preg)=(0.03,0.89,0.0)FTRLFTPLUnder review as a conference paper at ICLR 2025

C.6 ADDITIONAL FIGURES FOR SECTION 3.3

Figure C.4: Regret of GPT-4 and the FTRL algorithm in 6 randomly generated three-player general-
sum games. GPT-4 has comparable (even better) no-regret properties when compared with the FTRL
algorithm.

Figure C.5: Regret of GPT-4 and the FTRL algorithm in 6 randomly generated four-player general-
sum games. GPT-4 has comparable (even better) no-regret properties when compared with the FTRL
algorithm, according to the frameworks in Section 3.1 and the graphic trends.

34

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

0510152025Iterations0.02.55.07.510.012.515.017.520.0RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.74,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.0,0.9,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.86,0.0)FTRL30510152025Iterations0510152025RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.72,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.01,0.78,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.15,0.94,0.0)FTRL30510152025Iterations05101520253035RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.76,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.0,0.72,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.15,0.34,0.35)FTRL30510152025Iterations051015202530RegretGPT-41:(ptrend,ˆβ0,preg)=(0.42,0.0,1.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.01,0.7,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.43,0.0)FTRL30510152025Iterations051015202530RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.82,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.0,0.93,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.81,0.0)FTRL30510152025Iterations05101520RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.71,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.0,0.72,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.75,0.0)FTRL30510152025Iterations0510152025RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.18,0.01)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.0,0.56,0.19)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.03,0.98,0.0)FTRL3GPT-44:(ptrend,ˆβ0,preg)=(0.03,0.92,0.0)FTRL40510152025Iterations024681012RegretGPT-41:(ptrend,ˆβ0,preg)=(0.03,0.67,0.01)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.03,0.91,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.03,0.8,0.0)FTRL3GPT-44:(ptrend,ˆβ0,preg)=(0.27,-0.12,0.78)FTRL40510152025Iterations0.02.55.07.510.012.515.017.5RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.81,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.01,0.83,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.48,0.01)FTRL3GPT-44:(ptrend,ˆβ0,preg)=(0.27,0.05,0.96)FTRL40510152025Iterations0.02.55.07.510.012.515.017.5RegretGPT-41:(ptrend,ˆβ0,preg)=(0.03,0.03,0.73)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.08,0.53,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.78,0.0)FTRL3GPT-44:(ptrend,ˆβ0,preg)=(0.03,-1.0,0.16)FTRL40510152025Iterations05101520RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.83,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.08,-1.48,0.01)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.5,0.0)FTRL3GPT-44:(ptrend,ˆβ0,preg)=(0.08,0.66,0.0)FTRL40510152025Iterations0.02.55.07.510.012.515.017.520.0RegretGPT-41:(ptrend,ˆβ0,preg)=(0.0,0.67,0.0)FTRL1GPT-42:(ptrend,ˆβ0,preg)=(0.01,0.96,0.0)FTRL2GPT-43:(ptrend,ˆβ0,preg)=(0.0,0.76,0.0)FTRL3GPT-44:(ptrend,ˆβ0,preg)=(0.42,-1.95,0.01)FTRL4Under review as a conference paper at ICLR 2025

C.7 ADDITIONAL RESULTS FOR SECTION 3.4

For Example 2, we evaluate LLMs on both the c = 100 and c = 200 cases. The results and
comparisons are presented in Figure C.6 using a temperature of 0 to minimize the randomness
for such fixed problem instances, where we can confirm that GPT-4 with raw history identifies the
pattern and is able to achieve decreasing, negative regret during the first c = 100 or c = 200 rounds),
whereas FTRL, FTPL, and GPT-4 with only summarized history cannot detect the trend and then
make adaptive decisions. Meanwhile, after first c rounds, the LLM with raw history can identify
that the pattern for the loss vectors has changed to adjust its policy, and its regret grows slower than
the LLM with only summarized history.

Such observations further demonstrate the fundamental differences in the stochastic settings con-
sidered in Krishnamurthy et al. (2024) and our non-stochastic settings: the summarized history, an
essential factor for the successful configuration in Krishnamurthy et al. (2024), can be good statistics
in the i.i.d. setting (as a good estimate of the mean of the losses), while it loses information and can
be highly ineffective in the non-stochastic settings that are highly adversarial (Feder et al., 1992). In
contrast, with raw history, GPT-4 was able to better identify the pattern of the sequence and make
good predictions with negative regret values.

Figure C.6: Comparing LLMs on Example 2 in Section 3.4 with raw history as the input and sum-
marized history as the input.

In
Explaining Better Performance of LLMs on Loss with Trends via In-Context Learning.
context learning with trend is one plausible explanation comparing performance between raw history
and summarized history. Specifically, the task of predicting ℓT +1 given a past loss sequence ℓ1:T
could be exactly equivalent to an in-context learning problem as follows: The demonstration/in-
xt, yt}t∈[T −1], where xt = ℓ1:t
context dataset is given by the following input and label pairs D =
{
and yt = ℓt+1 for each t
1]. Then, LLMs given such demonstration/context D will make
prediction based on xT = ℓ1:T (to predict yT , i.e., the next loss vector ℓT +1). In other words, in-
context learning in this case is exactly firstly learning the trend from the T
1 pairs of inputs and
labels, i.e., learning the latent concept, and then make the prediction.

[T

−

−

∈

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

35

0100200300400500Iterations−2000200400600800Regretc=100GPT-4:rawhistoryGPT-4:summarizedhistoryFTRLFTPL0100200300400500Iterations−600−400−2000200400600800Regretc=200GPT-4:rawhistoryGPT-4:summarizedhistoryFTRLFTPL1890
1891
1892
1893
1894

1895
1896
1897
1898
1899
1900

1901
1902
1903
1904
1905
1906

1907
1908
1909
1910
1911
1912

1913
1914
1915
1916
1917

1918
1919
1920
1921
1922
1923

1924
1925
1926
1927
1928
1929

1930
1931
1932
1933
1934
1935

1936
1937
1938
1939
1940

1941
1942
1943

Under review as a conference paper at ICLR 2025

C.8 ABLATION STUDY ON PROMPTS

Ablation study on online learning. To systematically understand the effects of our prompt on the
final performance of the LLM agent, we create three different variants of our prompt and report the
regret by using different prompts in Figure C.7. Specifically, for Ablation1, we remove examples
to illustrate the game rules. For Ablation2, we remove the number of iterations. For Ablation3,
we incorporate some hints for the LLM for decision-making, including the hints to suggest it to pay
attention to the loss history, to behave more greedily at the end of an episode, and also to explain
the reason of its decision step-by-step. The latter hint is a popular technique in prompt engineering
known as the Chain-of-Thought prompting (Wei et al., 2022b). Finally, we recall that d is the number
of actions in all prompts.

Figure C.7: Ablation study on our prompt design.

You are solving a decision-making problem for 25 rounds.

Original prompt

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy,
it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components
should equal 1. After that, you will be shown the reward vector for
choosing each action.

Remember that this reward vector is decided by the external system
and can be potentially different for different rounds.

It is not decided by what policies you have chosen.
The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

For example, a reward vector of [0.8, 3.2] means reward for action_0 is
0.8 and the reward for action_1 is 3.2.

Then your reward for this round will be calculated according to the
reward of each action and your probability of choosing each action.

For example, if you choose the policy [0.2, 0.8] and get the reward
vector [1, 2], then your expected reward is 0.2*1 + 0.8*2=1.8

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous
rounds.

You’re required to provide your policy in numeric format.

36

010200246810UniformOriginalpromptAblation1Ablation2Ablation301020−101234GaussianOriginalpromptAblation1Ablation2Ablation3010200102030Linear-trendOriginalpromptAblation1Ablation2Ablation301020−50510Sine-trendOriginalpromptAblation1Ablation2Ablation3Under review as a conference paper at ICLR 2025

1944
1945
1946
1947
1948

1949
1950
1951
1952
1953
1954

1955
1956
1957
1958
1959
1960

1961
1962
1963
1964
1965
1966

1967
1968
1969
1970
1971

1972
1973
1974
1975
1976
1977

1978
1979
1980
1981
1982
1983

1984
1985
1986
1987
1988
1989

1990
1991
1992
1993
1994

1995
1996
1997

Your response’s last line should be formatted as
’Policy: [your $d$-dimensional policy]’.

You are solving a decision-making problem for 25 rounds.

Ablation1: no examples

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy,
it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components
should equal 1. After that, you will be shown the reward vector for
choosing each action.

Remember that this reward vector is decided by the external system
and can be potentially different for different rounds.

It is not decided by what policies you have chosen.
The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

Then your reward for this round will be calculated according to the
reward of
each action and your probability of choosing
each action.

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous
rounds.

You’re required to provide your policy in numeric format.

Your response’s last line should be formatted as
’Policy: [your $d$-dimensional policy]’.

You are solving a decision-making problem.

Ablation2: no round information

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy,
it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components
should equal 1. After that, you will be shown the reward vector for
choosing each action.

Remember that this reward vector is decided by the external system
and can be potentially different for different rounds.

It is not decided by what policies you have chosen.
The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

For example, a reward vector of [0.8, 3.2] means reward for action_0
is 0.8

and the reward for action_1 is 3.2.

Then your reward for this round will be calculated according to the
reward of

each action and your probability of choosing each action.

37

Under review as a conference paper at ICLR 2025

1998
1999
2000
2001
2002

2003
2004
2005
2006
2007
2008

2009
2010
2011
2012
2013
2014

2015
2016
2017
2018
2019
2020

2021
2022
2023
2024
2025

2026
2027
2028
2029
2030
2031

2032
2033
2034
2035
2036
2037

2038
2039
2040
2041
2042
2043

2044
2045
2046
2047
2048

2049
2050
2051

For example, if you choose the policy [0.2, 0.8] and get the reward
vector [1, 2], then your expected reward is 0.2*1 + 0.8*2=1.8

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous
rounds.

You’re required to provide your policy in numeric format.

Your response’s last line should be formatted as
’Policy: [your $d$-dimensional policy]’.

You are solving a decision-making problem for 25 rounds.

Ablation3: adding hints

There are $d$ number of action (which is 0 to $d-1$).

At each round, you need to choose a policy,
it specifies your probability to choose each action.

This policy should be $d$-dimensional, and the sum of its components
should equal 1.
After that, you will be shown the reward vector for
choosing each action.

Remember that this reward vector is decided by the external system
and can be potentially different for different rounds.

It is not decided by what policies you have chosen.
The reward vector is also $d$-dimensional.

It represents the reward of choosing action from 0 to $d-1$.

For example, a reward vector of [0.8, 3.2] means reward for action_0 is
0.8 and the reward for action_1 is 3.2.

Then your reward for this round will be calculated according to the
reward of each action and your probability of choosing each action.

For example, if you choose the policy [0.2, 0.8] and get the reward
vector [1, 2], then your expected reward is 0.2*1 + 0.8*2=1.8

Your goal is to maximize your accumulative expected reward.

You can adjust your policy based on the reward vectors for all previous
rounds.

You’re required to provide your policy in numeric format.

Your response’s last line should be formatted as
’Policy: [your $d$-dimensional policy]’.

Let’s think step by step. Explicitly examining history is important.

Please explain how you chose the policy by guessing
what reward you might receive for each action according to the history.

You should explore for first several rounds and behave greedily for
later rounds, for example, choosing one action with probability more
than 0.99.

Please also explain whether you are behaving more greedily and less

38

Under review as a conference paper at ICLR 2025

2052
2053
2054
2055
2056

2057
2058
2059
2060
2061
2062

2063
2064
2065
2066
2067
2068

2069
2070
2071
2072
2073
2074

2075
2076
2077
2078
2079

2080
2081
2082
2083
2084
2085

2086
2087
2088
2089
2090
2091

2092
2093
2094
2095
2096
2097

2098
2099
2100
2101
2102

2103
2104
2105

Figure C.8: Regret of GPT-4 for repeated games under 3 different prompt ablations. Its performance
is consistent among three different prompts.

Figure C.9: Comparisons of GPT-4 Turbo with GPT-4 and GPT-3.5 Turbo. Due to the symmetry of
agents in randomly generated games, we report the regret of one agent for ease of presentation. We
can see that GPT-4 Turbo performs on par with or better than GPT-4.

greedily
last round.

by explicitly considering the policy you just used for

We can see in Figure C.7 that the performances of LLM agents are consistent under different variants
of the prompts.

Ablation study on repeated games. For the game setting, we also investigate whether explic-
itly informing LLM agents that they are ‘‘playing a repeated matrix game with
some other opponents’’ would affect the performance. Therefore, we evaluate three dif-
ferent prompts by informing LLM agents that they are playing a matrix game, solving
multi-arm bandit, or solving general decision-making problems, in the first line of the
prompt. We show the performance of such three prompts in Figure C.8, where it is seen that LLM
agents’ performance on repeated games is consistent among these variants of the prompts.

C.9 RESULTS FOR GPT-4 TURBO

When preparing our paper, we are aware of a more advanced version of GPT-4, GPT-4 Turbo.
Although GPT-4 has successfully exhibited no-regret behaviors in most problems, we also evaluate
and report the performance of its more advanced version here. Specifically, we compare GPT-4 and
GPT-4 Turbo for the repeated game scenario in Figure C.9, where we validate that GPT-4 Turbo
performs either comparably with or better than GPT-4.

C.10 LLM AGENTS’ EXPLANATION ON THEIR OUTPUT POLICIES

We here present two examples of LLMs’ explanations of their output policies, showing how they
reason in their decision-making.

39

0510152025Iterations0.02.55.07.510.012.515.017.5RegretMulti-armbanditMatrixgameDecisionmaking0510152025Iterations051015202530RegretTwo-playergamesGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.63,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.9,0.0)GPT-4Turbo:(ptrend,ˆβ0,preg)=(0.0,0.49,0.0)FTRLFTPL0510152025Iterations0510152025RegretThree-playergamesGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.69,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.9,0.0)GPT-4Turbo:(ptrend,ˆβ0,preg)=(0.0,0.69,0.0)FTRLFTPL0510152025Iterations05101520RegretFour-playergamesGPT-4:(ptrend,ˆβ0,preg)=(0.0,0.8,0.0)GPT-3.5Turbo:(ptrend,ˆβ0,preg)=(0.0,0.93,0.0)GPT-4Turbo:(ptrend,ˆβ0,preg)=(0.0,0.79,0.0)FTRLFTPLUnder review as a conference paper at ICLR 2025

Output Example 1

Looking at the history of the reward vectors, it is clear to see that
the reward for choosing action 0 is consistently higher than that
for choosing action 1. This pattern is evident across all 5 rounds
of the game.

In the first round, the difference in rewards is 3.52, this remained
relatively stable with the difference in the last round being 3.52 as
well. It’s indicative that action 0 consistently brings a higher return
than action 1.

Therefore, I’ll lean more towards selecting action 0 in my policy.

However, acting deterministically, that is always choosing action 0,
would introduce the risk of becoming too predictable and letting the
opponent exploit this predictability

So, in order to maintain some unpredictability while still capitalizing
on the
as follows: select action 0 with a probability of 0.8 and action 1
with a probability of 0.2.

higher average return of action 0, I’ll set my policy

This way, the majority of the time I’m choosing the action that gives
a higher reward, but I’m also occasionally choosing the
other action to
keep the opponent guessing, and avoid any predictable patterns that
can be exploited

So, my policy is as follows: Policy: [0.8, 0.2]

The total reward from action_1 is 130.92 and the average reward is
around 10.07.

Output Example 2

On the other hand, the total reward from action_0 is 33.418 and the
average reward is around 8.354.

Although action_0 has a less average reward, its highest reward is
26.977, which is considerably high and shows its potential.

For this round, I want to continue to explore action_0 but also
significantly exploit action 1 as it reliably gives higher reward.

So, I’d like to decrease the probability of selecting action_0 compared
to last time but still keep it in the policy as a possible option.

The policy for the previous round was not provided but assuming
it was more balanced, I’m now going to lean more towards action_1.

So, the policy will be: [0.3, 0.7]. Here, I’m behaving more greedily
compared
to earlier rounds, as we are choosing the action which has
been giving us higher rewards with more probability.

As observed from the output, the LLM tends to use the history of the reward vectors by looking at
the “sum/average of the accumulated rewards”, and tends to introduce randomization in decision-
making, in order to be “less predictable and exploitable”. These are several key components in
achieving no-regret in online learning and games (Cesa-Bianchi & Lugosi, 2006), explaining the
empirical evidence we had in Section 3.

2106
2107

2108
2109
2110
2111
2112
2113

2114
2115
2116
2117
2118
2119

2120
2121
2122
2123
2124

2125
2126
2127
2128
2129
2130

2131
2132
2133
2134
2135
2136

2137
2138
2139
2140
2141
2142

2143
2144
2145
2146
2147

2148
2149
2150
2151
2152
2153

2154
2155
2156
2157
2158
2159

40

Under review as a conference paper at ICLR 2025

C.11 CASE STUDIES ON REAL-WORLD APPLICATIONS

In this subsection, we evaluate the sequential decision-making abilities of LLMs in realistic scenar-
ios from the perspective of regret and dynamic regret. While several studies have explored sequential
decision-making using synthetic scenarios (Krishnamurthy et al., 2024; Wu et al., 2023b; Xia et al.,
2024; Akata et al., 2023) or real-world data scenarios (Liu et al., 2023b; Wang et al., 2023c; Wu
et al., 2024), none have explicitly analyzed regret or dynamic regret. As a result, the theoretical
optimality of such sequential decision-making process remains unclear.

Our first case study investigates single-agent sequential decision-making using real-world data,
leveraging the same dataset and experimental setup as (Liu et al., 2023b). The second case study
explores a two-player negotiation scenario, providing insights into dynamic interactions and their
impact on decision-making performance.

C.11.1 SEQUENTIAL RECOMMENDATION

We consider the task of sequential recommendation, a task that people have been employing LLMs
to solve with success (Liu et al., 2023b; Wang et al., 2023c; Wu et al., 2024). Note that how existing
literature (Liu et al., 2023b) uses LLMs to solve this task fits exactly into our online learning frame-
work, where humans feed a history of items the user have interacted with to the LLM and then ask
the LLM to recommend the item (or several items) the user may want to interact next. The entire
process carries on repeatedly.

· · ·

, xt−1), where each xi ∈

Formally, the problem is as follows. Given a sequence of history items the user has interacted with
(x1, x2,
1] and D is the collection of all items, the
∈
LLM needs to recommend n items that the user might interact with in the next step t. Typically,
the LLM should also give a priority on the n items it recommends. For simplicity here, we here
D
assume they are of equal priority. In other words, at step t, the LLM will take an action at ⊆
= n, hoping what the user will interact at step t belongs to at. Hence, the loss is given by
with
ℓt(at, xt) := 1[xt ̸∈

at]. Correspondingly, the regret by our definition is given by

D for i

at|

−

[t

|

Regret(x1:T ) =

T
(cid:88)

t=1

ℓt(xt, at)

min
a

−

T
(cid:88)

t=1

ℓt(xt, a).

We refer to (Liu et al., 2023b) for a more detailed introduction. Meanwhile, we use the real-world
data and follow the experimental setup of (Liu et al., 2023b).

In the left one of Figure C.10, we can observe that LLMs can achieve expressively low and sublinear
regret on such a real-world application with real-world data. As a comparison, in the right one of
Figure C.10, we replace the real-world data with synthetic data generated in a uniformly random way
(it is worth mentioning that the prompt setting still follows the setup of sequential recommendation
of Liu et al. (2023b)), where we can see that LLMs are still no-regret. However, interestingly, LLMs
perform better on real-world data, which validates that real-world applications can exhibit certain
trends/structures, for which LLMs can exploit and achieve superior performance as we have shown
in our paper through synthetic problems with trends.

C.11.2

INTERACTIVE NEGOTIATION

The experiment was designed to simulate negotiation scenarios between two LLMs, designated as
LLM A and LLM B, across multiple turns. The primary objectives were to analyze multi-agent
sequential decision-making processes and quantify regret. For each repetition, an LLM generated
unique negotiation topics. Based on these topics, the LLM also created the context, objectives,
and relevant background information to establish a comprehensive foundation for the simulated
negotiations.

Negotiation Process. The negotiation process was executed in a turn-based manner, with each
turn comprising three steps:

1. Intention Generation: Each LLM defined its goal for the turn, specifying what it aimed

to achieve with its response.

41

2160
2161

2162
2163
2164
2165
2166
2167

2168
2169
2170
2171
2172
2173

2174
2175
2176
2177
2178

2179
2180
2181
2182
2183
2184

2185
2186
2187
2188
2189
2190

2191
2192
2193
2194
2195
2196

2197
2198
2199
2200
2201

2202
2203
2204
2205
2206
2207

2208
2209
2210
2211
2212
2213

Under review as a conference paper at ICLR 2025

2214
2215

2216
2217
2218
2219
2220
2221

2222
2223
2224
2225
2226
2227

2228
2229
2230
2231
2232

2233
2234
2235
2236
2237
2238

2239
2240
2241
2242
2243
2244

2245
2246
2247
2248
2249
2250

2251
2252
2253
2254
2255

2256
2257
2258
2259
2260
2261

2262
2263
2264
2265
2266
2267

Figure C.10: We evaluate GPT-4-Turbo and OpenAI o1 on both real-world data and uniformly
random synthetic data, where we can see both models can still achieve sublinear regret.

2. Response Generation: Based on the defined intention and the dialogue history, each LLM

generated a response.

3. Alternative Response Generation: Three distinct alternative replies were produced for
each original response. These alternatives represented diverse negotiation strategies or
perspectives while preserving the original intention.

Response Evaluation. After the dialogue concluded, all responses—both original and alterna-
tives—were evaluated using a scoring scale from 1 to 10 based on the following criteria for each
turn:

• Clarity: How clear and understandable the reply is.
• Relevance: How pertinent the reply is to the negotiation topic and the defined intention.
• Engagement: How engaging or persuasive the reply is in fostering further dialogue.
• Alignment with the Stated Intention: How well the conversation aligns with the turn’s
stated intention following the reply. For alternative replies, this was assessed by hypotheti-
cally replacing the original reply with an alternative and evaluating the alignment based on
the entire conversation.

Each response was scored using an LLM as the evaluator. Although human evaluation would be
preferable, the use of an LLM as a scorer was chosen for scalability. This approach is common in
the LLM domain and is sometimes referred to as G-eval (where “G” stands for GPT) (Liu et al.,
2023c).

Finally, dynamic regret was calculated to measure suboptimality
Dynamic Regret Analysis.
by comparing the scores of the original replies against the highest-scoring alternative responses.
Since calculating regret typically requires hindsight knowledge of the best possible responses, which
requires rollout of every possible dialogues, we decide to analyze on dynamic regret. Dynamic
regret analysis provided a quantitative measure of decision-making effectiveness across turns. This
analysis offered insights into how regret dynamics can inform improved decision-making strategies
in real-world negotiation contexts.

Example. Here is an example from our simulation:

Step 1: Generate Topics and Backgrounds. Topics and backgrounds were generated using a
language model.

Topic: The Trade-Off Negotiation Between Eco-Tech Innovator and Traditional Manufacturing
Tycoon

Background of Player A: Eco-Tech Innovator (Jordan Green). Jordan Green is the CEO of a
rapidly growing startup, EcoWave Technologies, which specializes in developing sustainable

42

51015−6−4−2024RegretReal-worldinstancesGPT-4-Turbo:(ptrend,ˆβ0,preg)=(0.03,-3.1,0.0)OpenAIo1:(ptrend,ˆβ0,preg)=(0.0,-3.63,0.01)510150.00.51.01.52.02.53.03.5RegretUniformlyrandominstancesGPT-4-Turbo:(ptrend,ˆβ0,preg)=(0.0,0.31,0.0)OpenAIo1:(ptrend,ˆβ0,preg)=(0.01,0.44,0.0)Under review as a conference paper at ICLR 2025

2268
2269

2270
2271
2272
2273
2274
2275

2276
2277
2278
2279
2280
2281

2282
2283
2284
2285
2286

2287
2288
2289
2290
2291
2292

2293
2294
2295
2296
2297
2298

2299
2300
2301
2302
2303
2304

2305
2306
2307
2308
2309

2310
2311
2312
2313
2314
2315

2316
2317
2318
2319
2320
2321

energy solutions and eco-friendly manufacturing processes. With a background in environ-
mental science and engineering, Jordan is passionate about reducing carbon footprints and
promoting renewable energy sources. Their innovative products, such as biodegradable mate-
rials and energy-efficient machinery, have garnered attention and accolades within the green
tech community. However, despite the startup’s promise, EcoWave faces challenges in scaling
production and reaching wider markets due to limited financial resources and manufacturing
capabilities.

Background of Player B: Traditional Manufacturing Tycoon (Robert Steele). Robert Steele
is the owner of Steele Industries, a well-established manufacturing company known for its
mass production of consumer goods. With decades of experience in the industry, Robert has
built a reputation for efficiency and profitability, but his company has been criticized for its
environmental impact and reliance on outdated practices. Facing increasing pressure from
regulators and consumers to adopt sustainable practices, Robert is aware that failing to adapt
could threaten the future of his business. However, he is also concerned about the costs and
risks associated with transitioning to new technologies and processes, which could disrupt his
current operations.

Detailed Situation: Jordan and Robert have agreed to meet to discuss a potential partnership.
Robert is interested in integrating EcoWave’s sustainable technologies into his manufacturing
processes, which could help Steele Industries improve its environmental footprint and public
image. However, he is also hesitant about the costs associated with adopting these new tech-
nologies and is particularly concerned about how this might affect his established workforce
and production timelines. On the other hand, Jordan sees this as a golden opportunity to scale
EcoWave’s innovative solutions and gain access to Robert’s extensive distribution networks.
However, Jordan is wary of compromising on key values regarding environmental integrity
and sustainable practices, fearing that a partnership with a traditional manufacturer could di-
lute EcoWave’s mission. The negotiation aims to balance Robert’s need for cost-effectiveness
and operational continuity with Jordan’s commitment to sustainability and innovation. Both
players must navigate their differing priorities, explore potential compromises, and consider
the long-term implications of their decisions on their businesses and the environment.

Step 2: Generate Conversation. We iterated between Step 2-1 and Step 2-2 to create a conversa-
tion with a length of 20 turns. In this setup, each LLM (A and B) contributed 10 responses based on
the evolving conversation history.

Step 2-1: Intention Generation and Reply Generation. We provided the initial turn’s intention
for LLM A:

Intention: In my next response, I aim to outline potential trade-offs that both I and Robert
might consider in order to reach a mutually beneficial agreement. I will also suggest ways we
can ensure that the partnership aligns with their core values of sustainability and operational
efficiency. This will help set the stage for a constructive dialogue that addresses our respective
concerns and priorities.

Reply of Player A based on the intention: Thank you for meeting with me, Robert. I believe we
have a unique opportunity to collaborate that could benefit both EcoWave and Steele Industries.
I understand your concerns about the costs and risks of integrating new technologies, but I think
we can explore some trade-offs. For instance, if we can demonstrate the long-term cost savings
and efficiency gains from adopting sustainable practices, it might help alleviate some of your
concerns. Additionally, I’m open to discussing a phased implementation approach that allows
for gradual integration without disrupting your current operations. How do you feel about that,
and what trade-offs might you be willing to consider to make this partnership work?

Step 2-2: Alternative Generation. The LLM also generated three alternative responses based on
the aforementioned intention.

Alternative 1:
I appreciate your concerns, Robert, and it’s great to see your commitment to
a smooth transition. Instead of jumping straight into a pilot project, how about we organize a
joint workshop with your key team members? This would allow us to collaboratively explore

43

Under review as a conference paper at ICLR 2025

areas where EcoWave’s technologies can fit into your existing operations. We can analyze
potential impacts, address any workforce concerns, and identify specific processes that could
benefit from our innovations. This way, we ensure that everyone is on board and feels included
in the transition. What are your thoughts on setting up a workshop to discuss this further?

Alternative 2:
I appreciate your concerns, Robert, and I think it’s important for us to find
a starting point that feels comfortable for you. Instead of a full-scale integration right away,
how about we explore a collaborative approach where we can jointly assess some specific
production lines that may benefit from our technologies? We could conduct a thorough analysis
of these areas together, identifying potential efficiency improvements and cost benefits through
a detailed study. This way, we can base our decisions on solid data while ensuring that your
existing operations remain stable. Additionally, I’d be more than willing to work closely with
your team to address any training needs that may arise. Would this kind of collaborative
assessment resonate with you?

Alternative 3:
I appreciate your insights, Robert, and I recognize the importance of ensuring
a smooth transition. Instead of jumping straight into a pilot project, how about we start by
conducting a joint assessment of your current operations? This would allow us to pinpoint
specific areas where EcoWave’s technologies could fit seamlessly. By collaborating on this
assessment, we can address your concerns and tailor our solutions to your unique processes.
Additionally, I’d be happy to share case studies from other manufacturers who have success-
fully integrated our technologies, demonstrating the benefits they achieved without significant
disruptions. What do you think about this collaborative approach?

Step 3: Evaluation. Each reply and its alternatives were eval-
uated using G-eval.

Experiment Setting. We set the maximum number of turns
to 10 and evaluated two language models: GPT-4-Turbo and
OpenAI’s o1 model. For each model, we tested performance
across 10 different scenarios.

Interestingly,

both models demonstrated no-
Results.
dynamic-regret behavior within the regression framework.
Both models also exhibited low p-values, with GPT-4-Turbo
achieving 0.09 and OpenAI o1 achieving 0.02. The detailed
results are shown in Figure C.11.

Figure C.11: We evaluate GPT-
4-Turbo and OpenAI o1 on two-
player Negotiation, where we can
see both model can achieve no-
regret in the regression framework.

D DEFERRED RESULTS AND PROOFS IN SECTION 4

D.1 PRE-TRAINED LLMS HAVE SIMILAR REGRET AS HUMANS (WHO GENERATE DATA)

We first provide a direct observation based on some existing speculation on the capability of
Transformer-based LLMs. Recently, a growing literature has evidenced that the intelligence level of
LLM agents are determined by, and in fact mimic, those of human beings who generate the data for
pre-training the models (Park et al., 2022; Argyle et al., 2023; Horton, 2023). The key rationale was
that, LLMs (with Transformer parameterization) can approximate the pre-training data distribution
very well (Xie et al., 2022; Zhang et al., 2023b; Lee et al., 2023). In such a context, one can expect
that LLM agents can achieve similar regret as human decision-makers who generate the pre-training
data, as we formally state below.

44

2322

2323
2324
2325
2326
2327

2328
2329
2330
2331
2332
2333

2334
2335
2336
2337
2338
2339

2340
2341
2342
2343
2344
2345

2346
2347
2348
2349
2350

2351
2352
2353
2354
2355
2356

2357
2358
2359
2360
2361
2362

2363
2364
2365
2366
2367
2368

2369
2370
2371
2372
2373

2374
2375

246810Time0.00.20.40.60.81.01.2Regret (GPT-4-Turbo)0.00.51.01.52.02.53.03.5Regret (OpenAI o1)Interactive NegotiationGPT-4-Turbo:(ptrend,0,preg)=(0.09, 0.67, 0.0)OpenAI o1:(ptrend,0,preg)=(0.02, 0.87, 0.0)Under review as a conference paper at ICLR 2025

Observation 1. An LLM agent is said to be pre-trained with an ϵ-decision error if, for any arbitrary
t and loss sequences (ℓi)i∈[t], the following condition holds:
(cid:12)
PLLM(π
(cid:12)Pdata(π

−
where Pdata and PLLM are the pre-training data distribution and the decision policy distribution of
the pre-trained LLM, respectively. Then, the regret of an LLM agent with ϵ-decision error is bounded
as:

(ℓi)i∈[t])

(ℓi)i∈[t])

sup
π∈Π

≤

ϵ,

(cid:12)
(cid:12)

|

|

(D-)RegretLLM

(cid:0)(ℓt)t∈[T ]

(cid:1)

∈

where [a

b] := [a

±

−

b, a + b].

(cid:20)
(D-)Regretdata

(cid:0)(ℓt)t∈[T ]

(cid:1)

(cid:21)

ϵ

ℓt∥
∥

sup
π∈Π ∥

π

∥

±

,

Observation 1 shows that the pre-trained LLM-agent’s regret can be controlled by that of the pre-
training dataset and the decision error ϵ. A small ϵ can be achieved if LLM is constructed by a rich
function class, e.g., the Transformer architecture (Zhang et al., 2023b; Lin et al., 2024).

Proof of Observation 1. For given (ℓt)t∈[T ],

T
(cid:88)

(cid:90)

t=1

πt∈Π

PLLM(πt |

ℓt, πt⟩
(ℓi)i∈[t−1])
⟨

dπt ≤

T
(cid:88)

t=1

holds, where we use the convention of PLLM(πt |
Pdata(πt). Hence,

(cid:90)

πt∈Π

(cid:0)Pdata(πt |

(ℓi)i∈[t−1]) + ϵ(cid:1)

ℓt, πt⟩
⟨
(ℓ0)) :=
(ℓ0)) := PLLM(πt) and Pdata(πt |

dπt

RegretLLM((ℓt)t∈[T ]) =

≤

=

T
(cid:88)

(cid:90)

t=1

πt∈Π

T
(cid:88)

(cid:90)

t=1

πt∈Π

T
(cid:88)

(cid:90)

PLLM(πt |

ℓt, πt⟩
(ℓi)i∈[t−1])
⟨

dπt −

inf
π∈Π

T
(cid:88)
⟨

t=1

ℓt, π

⟩

(cid:0)Pdata(πt |

(ℓi)i∈[t−1]) + ϵ(cid:1)

ℓt, πt⟩
⟨

dπt −

inf
π∈Π

T
(cid:88)

t=1

ℓt, π
⟨

⟩

(cid:0)Pdata(πt |

(ℓi)i∈[t−1])(cid:1)

πt∈Π

t=1
Regretdata((ℓt)t∈[T ]) + ϵ

⟨
∥qT

π
ℓ
∥p∥

∥

ℓt, πt⟩

dπt −

inf
π∈Π

T
(cid:88)
⟨

t=1

ℓt, π

+

⟩

T
(cid:88)

(cid:90)

t=1

ℓt, ϵπt⟩

dπt

πt∈Π⟨

≤
q = 1 and p, q

p + 1

where 1
To prove the result for the dynamic-regret case, we can simply change the term inf π∈Π
in the above derivation to (cid:80)T
ℓt, π

1. Similarly, we can establish the lower bound for RegretLLM((ℓt)t∈[T ]).
ℓt, π

t=1⟨

(cid:80)T

≥

⟩

t=1 inf π∈Π⟨

.
⟩

D.2 BACKGROUND AND MOTIVATIONS FOR (GENERALIZED) QUANTAL RESPONSE

Formally, the quantal response is defined as follows:
Definition D.1 (Quantal response). Given a loss vector ℓ
η > 0, the quantal response is defined as

P η

quantal

(cid:0)a (cid:12)

(cid:12) ℓ(cid:1) = P

(cid:18)

a

(cid:19)

z(a′)

,

arg min
a′∈A

∈
In essence, this implies that humans are rational but with respect to (w.r.t.)
the latent variable
z, a perturbed version of ℓ, instead of ℓ per se. This addition of noise to the actual loss vector
characterizes the bounded rationality of humans in decision-making.

where z = ℓ + ηϵ.

Rd, a noise distribution ϵ

∈

Pnoise, and

∼

Further motivations for generalized quantal response. Note that a dynamic version of quantal
response in Definition 4.1 also has implications from behavior economics, and has been recently
used to model human behaviors in sequential decision-making (Ding et al., 2022) (in stochastic
and stationary environments).
Indeed, such a response against multiple loss vectors is believed
to be natural, and has also been widely adopted in well-known no-regret learning algorithms of
smooth/stochastic fictitious play (Fudenberg & Kreps, 1993) and follow-the-perturbed-leader (Kalai
& Vempala, 2005), whose formal definitions can be found in Appendix B.4. Finally, note that the
response model in Definition 4.1 does not necessarily involve a sequential decision-making process,
i.e., the set of losses may not come from the history of an online learning process.

45

2376

2377
2378
2379
2380
2381

2382
2383
2384
2385
2386
2387

2388
2389
2390
2391
2392
2393

2394
2395
2396
2397
2398
2399

2400
2401
2402
2403
2404

2405
2406
2407
2408
2409
2410

2411
2412
2413
2414
2415
2416

2417
2418
2419
2420
2421
2422

2423
2424
2425
2426
2427

2428
2429

Under review as a conference paper at ICLR 2025

D.3 THE EXAMPLE INSTANTIATING ASSUMPTION 1

Example 1 (An example instantiating Assumption 1). We consider a common decision-making task
that may generate the training data, recommender systems. An instance of the text data could be:
“On September 29, 2023, user X clicked movie A three times, movie B eight times, and movie C
five times”. This sentence corresponds to xNi−1+1:Ni for some i
[t] and serves as a natural
language depiction of the numerical ℓi. The corresponding label xNt+1:Nt+1 can be obtained by
some user survey: “User X’s favorite movie is movie B”. Meanwhile, z represents user X’s latent,
genuine preference for each movie – information that is private to the user, and cannot be observed
or collected in the pre-training dataset. In this example, Assumption 1 suggests that x1:Nt, which
records the frequency of interactions with each movie, serves as an imperfect estimate of the user’s
latent, genuine preference for the movies, while the text xNt+1:Nt+1 depicts the user’s favorite movie
only based on her latent z.

∈

D.4 ALIGNMENT OF ASSUMPTION 1 WITH QUANTAL RESPONSE

Before presenting the technical lemma, based on Assumption 1, we denote the (potentially unkown)
mappings that decode semantic information in Assumption 1 into numeric values as f , g, such that
f (xNi−1+1:Ni) = ℓi ∈
Lemma 1. Fix t
distribution in the numeric value space, i.e.,

[T ], σ > 0. If we model the noise of data collection to be i.i.d. Gaussian

[t] and g(xNt+1:Nt+1) = a

Rd for each i

.
∈ A

∈

∈

P

(cid:16)(cid:8)f (xNi−1+1:Ni )(cid:9)

(cid:17)

(cid:12)
(cid:12) z

=

i∈[t]

t
(cid:89)

i=1

P (cid:0)f (xNi−1+1:Ni) (cid:12)

(cid:12) z(cid:1)

t
(cid:89)

i=1

∝

(cid:18)

exp

∥

−

f (xNi−1+1:Ni)

2σ2

z

2
2

∥

−

(cid:19)

,

the prior distribution of the latent variable z is also Gaussian, i.e., z
labels satisfy that P(g(xNt+1:Nt+1)

∼ N

arg mina∈A za

(000d, σ2I), and the text
(cid:1), then we have

z) = 1 (cid:0)g(xNt+1:Nt+1)
(cid:16)

√

∈

g(xNt+1:Nt+1 ) (cid:12)
(cid:12)

|
(cid:1) = P σ

t+1
quantal

(cid:8)f (xNi−1+1:Ni)(cid:9)

(cid:17)

,

i∈[t]

P (cid:0)g(xNt+1:Nt+1) (cid:12)

(cid:12) x1:Nt

with Pnoise =
xNt+1:Nt+1 is a quantal response w.r.t. the loss vectors (cid:0)f (xNi−1+1:Ni)(cid:1)

(000d, I) in Definition 4.1, i.e., the action a = g(xNt+1:Nt+1) extracted from the text

N

i∈[t].

Proof. Note that

x1:Nt) =

(cid:90)

ℓ1:t

P(z, ℓ1:t |

x1:Nt)dℓ1:t =

(cid:90)

ℓ1:t

P(ℓ1:t |

x1:Nt)P(z

|

x1:Nt, ℓ1:t)dℓ1:t.

x1:Nt), since we have assumed the existence of function f to decode ℓ1:t from x1:Nt, it

P(z

|
For P(ℓ1:t |
holds that

P(ℓ1:t |

x1:Nt) =

t
(cid:89)

i=1

δ (cid:0)ℓi −

f (xNi−1+1:Ni)(cid:1) ,

where we use δ to denote the d-dimensional Dirac-delta function. For P(z
tion 1, it holds that

|

x1:Nt , ℓ1:t), by Assump-

P(z, x1:Nt, ℓ1:t) = P(z, ℓ1:t)P(x1:Nt |
ℓ1:t) = P(x1:Nt |

ℓ1:t),

which leads to P(x1:Nt |
variable x1:Nt and z are independent conditioned on ℓ1:t. Therefore, it holds that P(z
P(z

ℓ1:t, z) by Bayes rule. This implies that the random
x1:Nt, ℓ1:t) =

ℓ1:t). Finally, we can compute

|

|

P(z

|

x1:Nt) =

(cid:90)

ℓ1:t
(cid:16)

z

= P

δ(ℓi −

f (xNi−1+1:Ni))P(z

ℓ1:t)dℓ1:t

|

x1:Nt)dℓ1:t =

P(z, ℓ1:t |
(cid:0)ℓi = f (xNi−1+1:Ni)(cid:1)

|

(cid:90)

t
(cid:89)

ℓ1:t
i=1
(cid:17)

.

i∈[t]

46

2430
2431

2432
2433
2434
2435
2436
2437

2438
2439
2440
2441
2442
2443

2444
2445
2446
2447
2448

2449
2450
2451
2452
2453
2454

2455
2456
2457
2458
2459
2460

2461
2462
2463
2464
2465
2466

2467
2468
2469
2470
2471

2472
2473
2474
2475
2476
2477

2478
2479
2480
2481
2482
2483

Under review as a conference paper at ICLR 2025

Based on this, we conclude that

P(g(xNt+1:Nt+1)

x1:Nt) =

|

(cid:90)

z
(cid:90)

P(g(xNt+1:Nt+1)

z, x1:Nt)P(z

|

x1:Nt)dz

|

=

P(g(xNt+1:Nt+1)
z
(cid:16)
= P

g(xNt+1:Nt+1)

z)P(z

ℓi = f (xNi−1+1:Ni )

| {

|
(cid:0)ℓi = f (xNi−1+1:Ni)(cid:1)

(cid:17)

i∈[t]

|

}i∈[t])dz

where the first equality is by the independence between xNt+1:Nt+1 and x1:Nt conditioned on z,
due to Assumption 1. Therefore, it suffices to consider the probability of P(a
ℓ1:t) only, in order
to analyze P(g(xNt+1:Nt+1 )
x1:Nt), where we recall the definition that a = g(xNt+1:Nt+1). Since
(000d, σ2I), and ℓi |
z
z

(z, σ2I), we have

|
∼ N

∼ N

|





z

ℓ1:t ∼ N

|



1
t + 1

(cid:88)

i∈[t]

ℓi,

σ2
t + 1

I

 ,

(D.1)

by the posterior distribution of Gaussian distribution. Now we conclude that

P(a

|

ℓ1:t) =

=

=

(cid:90)

z

(cid:90)

z

(cid:90)

z

P(a

|

z, ℓ1:t)P(z

ℓ1:t)dz =

|

(cid:90)

z

P(a

|

z)P(z

ℓ1:t)dz

|



1(a

arg min
a′∈A

∈

za′)P(z

|

ℓ1:t)dz =

(cid:90)

z

1

a

arg min
a′∈A

∈





σ
√t + 1

ϵ +

1
t + 1

(cid:88)

i∈[t]



ℓi




 P(ϵ)dϵ



1

a

arg min
a′∈A

∈


σ√t + 1ϵ +

(cid:88)

i∈[t]



ℓi




 P(ϵ)dϵ = P

a′



a

arg min
a′∈A

∈

a′


σ√t + 1ϵ +



ℓi







a′

(cid:88)

i∈[t]

√

= P σ

t+1
quantal(a

ℓ1:t),

|

where P(ϵ) =

N

(000d, I). This completes the proof.

D.5 RELATIONSHIP BETWEEN FTPL AND DEFINITION 4.1

Fact 1. Performing generalized quantal response of Definition 4.1 at every iteration t
history loss vectors ℓ1:t−1 is essentially executing an FTPL algorithm.

∈

[T ] w.r.t.

Proof. Before we move to the proof, we will define the random variable which has distribution Pnoise
as Znoise. Note that at round t

2 (as the policy at round t = 1 is fixed), we have

≥

(cid:32)

P ηt−1
quantal(a

|

ℓ1:t−1) := P

a

arg min
a′∈A

∈

(cid:33)

(cid:33)

ℓi + ηt−1ϵ

(a′)

(cid:32)t−1
(cid:88)

i=1

(D.2)

which is exactly the case when ϵt in Equation (B.2) satisfies ϵt

d= ηt−1ϵ.

D.6 FORMAL STATEMENT AND PROOF OF THEOREM 4.1

t

∈

for

class

suppose

behavior).

of LLMθ

no-regret
of
the
function
[T ], maxθ∈Θ E
E

Theorem D.1. (Emergence
of Lemma
1,
that
all
max{qj ∈{V j−1→∆(V)}}j∈[Nt+1]
define
x1:0) := q1(x1), and θ⋆ maximizes Equation (4.1). Then, there exist (simple) algo-
q1(x1 |
rithms using LLMθ⋆ to achieve no (dynamic) regret for (non-stationary) online learning with
full-information/bandit feedback. To be specific, for (2) and (4), by defining the variation bound
(cid:80)T −1
(0, 1), it holds that
ℓt+1 −
for large enough T , d:

j=1 log LLMθ (xj |
x1:j−1),
where

x1:Nt+1 ∼P text
j=1 log qj (xj |

assumptions
such
=

T and VT = Θ(T ρ) for some ρ

VT such that VT ≤

Under
expressive

enough
x1:j−1)

is
(cid:80)Nt+1

x1:Nt+1 ∼P text

(cid:80)Nt+1

t=1 ∥

ℓt∥

the

we

≤

∈

∞

t

t

47

2484

2485
2486
2487
2488
2489

2490
2491
2492
2493
2494
2495

2496
2497
2498
2499
2500
2501

2502
2503
2504
2505
2506
2507

2508
2509
2510
2511
2512

2513
2514
2515
2516
2517
2518

2519
2520
2521
2522
2523
2524

2525
2526
2527
2528
2529
2530

2531
2532
2533
2534
2535

2536
2537

≤

≤

≤

∈

Under review as a conference paper at ICLR 2025

(1) For online learning with full-information feedback, RegretLLMθ⋆

≤ O
(2) For non-stationary online learning with full-information feedback, D-RegretLLMθ⋆

(cid:0)(ℓt)t∈[T ]

(cid:1)

(cid:0)√T log d(cid:1);
(cid:1)
(cid:0)(ℓt)t∈[T ]

(cid:0)(log d VT )1/3T 2/3(cid:1) ;

O
(3)

For

online

learning with

(cid:0)(log d)1/2dT 1/2+1/ log T log T (cid:1) ;

bandit

feedback,

E (cid:2)RegretLLMθ⋆

(cid:0)(ℓt)t∈[T ]

(cid:1)(cid:3)

O
(4) For non-stationary online learning with bandit feedback, E (cid:2)D-RegretLLMθ⋆
O

(cid:0)(T 2d2VT )1/3(log d)1/2T 1/ log T log T (cid:1) .

(cid:0)(ℓt)t∈[T ]

(cid:1)(cid:3)

Proof. Note that

max
{qj ∈{V j−1→∆(V)}}j∈[Nt+1]

E

x1:Nt+1 ∼P text

t

Nt+1
(cid:88)

j=1
log q(x1:Nt+1)

log qj (xj |

x1:j−1)

q) + E

x1:Nt+1 ∼P text

t

[P text
t

(x1:Nt+1)],

= max

q∈∆(V Nt+1 )

E

x1:Nt+1 ∼P text

t

= max

q∈∆(V Nt+1 ) −

KL(P text
t

| |

p) denotes the Kullback–Leibler divergence between two distributions p, q. Now we

where KL(q
define LLMθ(x1:Nt+1) = (cid:81)Nt+1
∆(

| |

t=1 LLMθ(xj |

V

Nt+1), i.e., it also defines a valid joint distribution over tokens. Therefore, we have

x1:j−1). It is easy to verify that LLMθ(x1:Nt+1)

max
θ∈Θ

E

x1:Nt+1 ∼P text

t

Nt+1
(cid:88)

j=1

log LLMθ (xj |

x1:j−1) = max
θ∈Θ

E

x1:Nt+1 ∼P text

t

log LLMθ(x1:Nt+1).

Now, due to our assumption that

max
θ∈Θ

E

x1:Nt+1 ∼P text

t

Nt+1
(cid:88)

j=1

log LLMθ (xj |

x1:j−1)

=

max
{qj ∈{V j−1→∆(V)}}j∈[Nt+1]

E

x1:Nt+1 ∼P text

t

Nt+1
(cid:88)

j=1

log qj (xj |

x1:j−1) ,

we conclude that

min
θ∈Θ

KL(P text
t

which implies that LLMθ⋆ = P text
be the distribution induced by the joint distribution LLMθ⋆ (x1:Nt+1), it holds that

| |
. Correspondingly, if we define LLMθ⋆ (xNt+1:Nt+1 |

| |

t

LLMθ) =

min
q∈∆(V Nt+1 )

KL(P text
t

q) = 0,

x1:Nt) to

LLMθ⋆ (xNt+1:Nt+1 |

x1:Nt) = P(xNt+1:Nt+1 |

x1:Nt).

√

In other words, intuitively, LLMθ⋆ has learned the corresponding pre-training distribution per-
fectly. Note that this has been a common assumption in the Bayesian perspective of ICL
to analyze the ac-
(Xie et al., 2022; Lee et al., 2023; Zhang et al., 2023b).
tions taken by LLMθ⋆ ,
to
g(xNt+1:Nt+1) (cid:12)
P σ
by Lemma 1. Therefore, we proved that LLMθ⋆
(cid:12)
is essentially mimicking the well-known no-regret algorithm, FTPL with perturbation distribution
[T ], according to Equation (D.2) of Fact 1, for which we can establish
as
the corresponding regret guarantee for each case:

it suffices to consider P(g(xNt+1:Nt+1 )
(cid:8)f (xNi−1+1:Ni)(cid:9)

x1:Nt), which is equal

(000d, σ2tI) for round t

Therefore,

t+1
quantal

i∈[t]

N

∈

(cid:17)

(cid:16)

|

(1) Combining the above result with Lemma 2, we can derive the regret bound for online learning
with full-information feedback.

(2) Combining the above result with Lemma 2 and Lemma 4, we get that

D-RegretLLMθ⋆ ((ℓi)i∈[T ])

min
∆T ∈[T ]

≤

(cid:112)

C

2T
∆T

∆T log d + 2∆T VT ,

48

2538
2539

2540
2541
2542
2543
2544
2545

2546
2547
2548
2549
2550
2551

2552
2553
2554
2555
2556

2557
2558
2559
2560
2561
2562

2563
2564
2565
2566
2567
2568

2569
2570
2571
2572
2573
2574

2575
2576
2577
2578
2579

2580
2581
2582
2583
2584
2585

2586
2587
2588
2589
2590
2591

Under review as a conference paper at ICLR 2025

for some constant C. We firstly consider the following problem

u log d + 2uVT ,

min
u>0

C

(cid:112)

2T
u
(cid:16) C2T 2 log d
4V 2
T

(cid:17)1/3

. Therefore, if we have u⋆

[1, T ], we can

∈

where the optimal solution is u⋆ =
u⋆
choose ∆T =

, which results in a regret bound of

⌈

⌉

D-RegretLLMθ⋆ ((ℓi)i∈[T ])

≤

Now we check the conditions for u⋆
long as d is sufficiently large. To ensure u⋆
as long as T is large enough.

∈

(cid:16)

(log d VT )1/3T 2/3(cid:17)

.

C

(cid:112)

log d + 4u⋆VT =

2T
√u⋆
[1, T ]. It is direct to see that since VT ≤
C
T , we get the condition VT ≥

O

≤

T , u⋆
(cid:113) log d

≥

1 holds as

4T , which holds

(3) Combining the above result with Lemma 3, we can prove a regret guarantee for online learning
with bandit feedback.

(4) Combining this result with Lemma 3 and Lemma 4, it holds that

E[D-RegretLLMθ⋆ ((ℓi)i∈[T ])]

min
∆T ∈[T ]

≤

2T
∆T

C(log d)

1
2 d∆

1

2 + 1
T

log T

log ∆T + 2∆T VT ,

for some constant C. By adopting a similar analysis as that of (2), we choose u⋆ =
for some constant C ′. If u⋆
[1, T ], we choose ∆T =

(cid:16) C′T 2d2
V 2
T
and derive the following regret:

u⋆

(cid:17)1/3

∈

⌈

⌉

E[D-RegretLLMθ⋆ ((ℓi)i∈[T ])]

(T 2d2VT )1/3(log d)1/2T 1/ log T log T

(cid:17)

.

(cid:16)

≤ O

≤

(cid:113)

T , u⋆

[1, T ]. Note that since VT ≤

Now we check the condition of u⋆
sufficiently large. For u⋆

C′d2
T , which holds as long as T is large enough.

∈
T , we have VT ≥
B = 1 for
Now, we present Lemma 2 - Lemma 4. Before proceeding, we assume
simplicity of presentations hereafter. The results and proof are not affected by the constant bound
B.
Lemma 2 (Regret guarantee of FTPL with full-information feedback). Suppose the noise distri-
t I) in Equation (B.2) and ζt = σ√t, then for online
bution of FTPL satisfies that ϵt ∼ N
learning with full-information feedback,

1 holds as long as d is

(000d, ζ 2

ℓt∥

≥

≤

∞

∥

RegretFTPL((ℓi)i∈[T ])

(cid:18)

4

σ +

(cid:19)

1
σ

≤

(cid:112)

T log d =

(cid:112)
(

T log d).

O

Proof. By Theorem 8 of Abernethy et al. (2014), we have

RegretFTPL((ℓi)i∈[T ])

(cid:32)

2 log d

ηT +

(cid:112)

≤

T
(cid:88)

t=1

(cid:33)

.

1
ηt ∥

2
∞

ℓt∥

Therefore, plugging ζt = σ√t and

∥

RegretFTPL((ℓi)i∈[T ])

(cid:112)

≤

completing the proof.

1 provides

ℓt∥

2
∞ ≤
(cid:32)

2 log d

σ√T +

(cid:33)

T
(cid:88)

t=1

1
σ√t

(cid:18)

4

σ +

(cid:19)

1
σ

≤

(cid:112)

T log d,

Lemma 3 (Regret guarantee of FTPL with bandit feedback). Suppose the noise distribution of FTPL
t I) in Equation (B.2) and ζt = σ√t, then for online learning with bandit
satisfies that ϵt ∼ N
feedback,

(000d, ζ 2

E[RegretFTPL((ℓi)i∈[T ])]

((log d)

1

2 dT

≤ O

49

1

2 + 1

log T log T ).

2592
2593

2594
2595
2596
2597
2598
2599

2600
2601
2602
2603
2604
2605

2606
2607
2608
2609
2610

2611
2612
2613
2614
2615
2616

2617
2618
2619
2620
2621
2622

2623
2624
2625
2626
2627
2628

2629
2630
2631
2632
2633

2634
2635
2636
2637
2638
2639

2640
2641
2642
2643
2644
2645

Under review as a conference paper at ICLR 2025

Proof. The proof of the bandit problem is more complex. We first define the following notation. We
denote Gt = (cid:80)t
ϵ∼N (000d,I)Φ(G +
ζtϵ), and DΦt to be the Bregman divergence with respect to Φt, where we recall the construction of
the empirical estimator (cid:98)ℓt′ of ℓt′ in Section 3.2. By Li & Tewari (2017), πt =
Φt( (cid:98)Gt). Now due
to the convexity of Φ,

(cid:98)ℓt′, Φ(G) = maxπ⟨

ℓt′, (cid:98)Gt = (cid:80)t

, Φt(G) = E

π, G
⟩

t′=1 −

t′=1 −

∇

Φ(GT ) = Φ(E[ (cid:98)GT ])

E[Φ( (cid:98)GT )].

≤

Therefore,

E[RegretFTPL((ℓi)i∈[T ])] = Φ(GT )

(cid:34) T

(cid:88)

E

t=1

−

(cid:35)

(cid:34)

πt,
⟨

(cid:98)ℓt⟩
−

E

≤

Φ( (cid:98)GT )

T
(cid:88)

t=1

−

(cid:35)
(cid:98)ℓt⟩

.

πt,
⟨

−

By recalling the definition of the Bregman divergence, we have

T
(cid:88)
⟨

t=1

−

πt,

(cid:98)ℓt⟩

−

=

−

T
(cid:88)

⟨∇

t=1

Φt( (cid:98)Gt),

(cid:98)ℓt⟩

−

=

−

T
(cid:88)

⟨∇

t=1

Φt( (cid:98)Gt), (cid:98)Gt −

(cid:98)Gt−1⟩

=

T
(cid:88)

t=1

DΦt( (cid:98)Gt, (cid:98)Gt−1) + Φt( (cid:98)Gt−1)

Φt( (cid:98)Gt).

−

Therefore,

E (cid:2)RegretFTPL((ℓi)i∈[T ])(cid:3)

(cid:34) T

(cid:88)

t=1

≤

E

(cid:124)

(cid:35)

DΦt ( (cid:98)Gt, (cid:98)Gt−1)

+ E

(cid:123)(cid:122)
(i)

(cid:125)

(cid:124)

(cid:34) T

(cid:88)

t=1

(cid:35)

Φt( (cid:98)Gt−1)

−

Φt−1( (cid:98)Gt−1)

(cid:123)(cid:122)
(ii)

+ E
(cid:124)

(cid:125)

(cid:104)
Φ( (cid:98)GT )

(cid:105)
ΦT ( (cid:98)GT )

.

−
(cid:123)(cid:122)
(iii)

(cid:125)

(iii)

≤

0 due to the convexity of Φ. For (ii), we use Lemma 10 of Abernethy et al. (2014) to obtain

(cid:34) T

(cid:88)

E

t=1

Φt( (cid:98)Gt−1)

−

Φt−1( (cid:98)Gt−1)

≤

ζT Eϵ[Φ(ϵ)]

≤ O

(cid:35)

(cid:112)
(

2T log d).

For (i), by Theorem 8 of Li & Tewari (2017), for any α

(0, 1), the following holds:

∈

(cid:34) T

(cid:88)

E

t=1

(cid:35)

DΦt( (cid:98)Gt, (cid:98)Gt−1)

T
(cid:88)

t=1

≤

ζ α−1
t

4d

4d

α(1

α)

−
1+α
2 ).

(T

≤

α(1

α) O

((log d)

1
2 dT

1

2 + 1

log T log T ).

By tuning α = 2

−
log T , we proved that E[RegretFTPL((ℓi)i∈[T ])]

≤ O
Lemma 4. Denote the variation of loss vectors as LT = (cid:80)T −1
∞. Suppose there
t=1 ∥
exists an algorithm A for online learning with full-information feedback with regret guarantee that
RegretA ((ℓi)i∈[T ])
f (T, d) for some function f , where T denotes the horizon and d denotes the
policy dimension. Then, there exists another algorithm A ′ that can achieve

ℓt+1 −

ℓt∥

≤

D-RegretA ′((ℓi)i∈[T ])

min
∆T ∈[T ]

≤

(cid:18) T
∆T

(cid:19)

+ 1

f (∆T , d) + 2∆T LT .

Similarly, suppose there exists an algorithm B for online learning with bandit feedback with regret
guarantee that E (cid:2)RegretB((ℓi)i∈[T ])(cid:3)
g(T, d) for some function g; then there exists another
algorithm B′ that can achieve

≤

E[D-RegretB′((ℓi)i∈[T ])]

min
∆T ∈[T ]

≤

(cid:18) T
∆T

(cid:19)

+ 1

g(∆T , d) + 2∆T LT .

50

2646
2647

2648
2649
2650
2651
2652
2653

2654
2655
2656
2657
2658
2659

2660
2661
2662
2663
2664

2665
2666
2667
2668
2669
2670

2671
2672
2673
2674
2675
2676

2677
2678
2679
2680
2681
2682

2683
2684
2685
2686
2687

2688
2689
2690
2691
2692
2693

2694
2695
2696
2697
2698
2699

Under review as a conference paper at ICLR 2025

Proof. We denote A ′ as the algorithm that restarts A every ∆T iterations. We break the time index
T1:m of size ∆T (except for, possibly the last batch). Denote ℓ⋆
[T ] into m batches
i := minj∈[d] ℓij.
By Equation (6) of Besbes et al. (2014), it holds that for each k

[m]

(cid:32)

(cid:88)

ℓt

t∈Tk

min
j∈[d]

(cid:33)

−

j

(cid:88)

t∈Tk

ℓ⋆
t ≤

∈

2∆T Lk,

where we define Lk = (cid:80)

t∈Tk ∥

ℓt+1 −


ℓt∥

∞. Therefore, we have

D-RegretA ′((ℓi)i∈[T ])

min
j∈[d]

≤



ℓt



j

(cid:88)

ℓ⋆
t +

(cid:88)

t∈[T ]

k∈[m]

−

RegretA ((ℓi)i∈[Tk])

(D.3)

(cid:88)



t∈[T ]
(cid:88)

k∈[m]

2∆T (

Lk) + (T /∆T + 1)g(∆T , d).

≤

By Equation (4) of Besbes et al. (2014) that (cid:80)
[T ], we proved D-RegretA ′((ℓi)i∈[T ])
∆T ∈
Similarly, if we take the expectation for Equation (D.3), it holds that

k∈[m] Lk ≤
min∆T ∈[T ]

(cid:16) T
∆T

≤

+ 1

(cid:17)

LT and this inequality holds for any

f (∆T , d) + 2∆T LT .

E[D-RegretB′((ℓi)i∈[T ])]

min
j∈[d]

≤

thus completing the proof.

min
∆T ∈[T ]

≤





(cid:88)



ℓt



t∈[T ]
(cid:18) T
∆T

(cid:88)

ℓ⋆
t +

(cid:88)

t∈[T ]

k∈[m]

−

(cid:19)

E[RegretB((ℓi)i∈[Tk])]

j

+ 1

g(∆T , d) + 2∆T LT ,

Combining the results above completes the proof for Theorem 4.1.

D.6.1

IMPLICATIONS OF THEOREM 4.1 FOR REPEATED GAMES

Remark D.1 (Implication for playing repeated games). First, we note that the no-regret guarantee
in the online setting is stronger than and thus implies that in the game setting, since regret by
definition handles arbitrary/adversarial environments, while in playing games the opponents are not
necessarily as adversarial. Second, it is folklore that if all players in the repeated game follow no-
regret learning algorithms, then the time-average policies of all players during learning constitute
an approximate coarse correlated equilibrium of the game (Cesa-Bianchi & Lugosi, 2006). Hence,
the results (1) and (2) in Theorem 4.1 imply that a coarse correlated equilibrium will emerge in the
long run from the interactions of the LLM agents (under certain assumptions as in the theorem).

D.7 EXTENDING THEOREM 4.1 WITH RELAXED ASSUMPTIONS

D.7.1 RELAXATION UNDER MORE GENERAL DATA DISTRIBUTIONS

We first remark on the possibility of relaxing the Gaussian assumptions on the data distributions.
Remark D.2 (Relaxing the Gaussian distribution assumption). In the proof of Lemma 1, to obtain
the result that the action is a quantal response w.r.t. ℓ1:T , one does not necessarily require both the
prior distribution of z and the conditional distribution of ℓi given z to be Gaussian. Instead, for any
joint distribution P(z, ℓ1:T ), as long as its posterior distribution satisfies Equation (D.1), it would
suffice. It is a combined effect of both the prior and the conditional distributions.

More formally, we can extend Theorem 4.1 to the case with a much more general prior task distribu-
tion than the Gaussian one, where the key is that Equation (D.1) only needs to hold approximately.
Theorem D.2. In Theorem 4.1, we can relax the assumption on P(z) to one where we only require
P(z) to be i.i.d for each coordinate of z and 0 < P(zj) <
R,
and the bounds for (1) and (2) of Theorem 4.1 still hold, with only a degradation of

[d], zj ∈
∈
(d2 log T ).

P(zj)
|

for any j

|∇

∞

∞

<

,

O

51

2700
2701

2702
2703
2704
2705
2706
2707

2708
2709
2710
2711
2712
2713

2714
2715
2716
2717
2718

2719
2720
2721
2722
2723
2724

2725
2726
2727
2728
2729
2730

2731
2732
2733
2734
2735
2736

2737
2738
2739
2740
2741

2742
2743
2744
2745
2746
2747

2748
2749
2750
2751
2752
2753

Under review as a conference paper at ICLR 2025

The key idea of the proof is that when t is large enough, the prior distribution does not affect the
posterior distribution, which is also referred to as the Bernstein–von Mises theorem (Van der Vaart,
2000).

Proof. Since we extend Theorem 4.1 to settings with general task prior distribution only requiring
the coordinates to be i.i.d, from now on, we consider the j-th coordinate only. To begin with, fix
t

[T ], we define the log-likelihood of the posterior as

∈

Lt(zj) := log

t
(cid:89)

i=1

1

σd(2π)d/2 e− 1

2σ2 (ℓij −zj )2

t log σ

=

−

−

t
2

log 2π

t
(cid:88)

i=1

−

1
2σ2 (ℓij −

zj)2.

Then, the MLE estimator (cid:98)zj,t is defined as

We also define (cid:98)Jt : R

R as:

→

(cid:98)zj,t := arg max
zj ∈R

Lt(zj) =

1
t

t
(cid:88)

i=1

ℓij.

(cid:98)Jt(zj) :=

∇

2Lt(zj)
t

−

=

1
σ2 .

For Assumption 1 of Kasprzak et al. (2022) to hold, any δ > 0, M2 > 0 suffices.

For Assumption 2 of Kasprzak et al. (2022) to hold, we can choose (cid:99)M1 = maxzj ∈[−δ,1+δ]

1
P(zj )

For Assumption 7 of Kasprzak et al. (2022) to hold, we choose δ to be σ.
For Assumption 8 of Kasprzak et al. (2022) to hold, one can choose M2 = σ
2 .

For Assumption 9 of Kasprzak et al. (2022) to hold, we have

κ

≤ −

sup
(zj −(cid:98)zj )2≥δ

Lt(zj)

Lt((cid:98)zj,t)

−
t

=

1
2σ2t

−

sup
(zj −(cid:98)zj,t)2≥δ

t
(cid:88)

i=1

(ℓij − (cid:98)zj,t)2

(ℓij −

−

zj)2 =

1
4σ

.

For Assumption 10 of Kasprzak et al. (2022) to hold, we choose M1 = supzj ∈[−δ,1+δ]
(cid:102)M1 = supzj ∈[−δ,1+δ] |
By Theorem 6.1 of Kasprzak et al. (2022), we have

since we have assumed that 0 < P(zj) <

P(zj)
|

P(zj)
|

|∇

∞

<

,

(cid:12)
(cid:12)
(cid:12)
.
∞

∇P(zj )
P(zj )

(cid:12)
(cid:12)
(cid:12),

(cid:90)

zj |

P(zj/√t + (cid:98)zj |
(cid:90)

= √t

(ℓij)i∈[t])

−

Ce− 1

2σ2 z2

j

dzj

|
σ2
t

P(zj |

zj |

(ℓij)i∈[t])

((cid:98)zj,

− N

dzj ≤
)
|

D1t−1/2 + D2t1/2e−tκ + 2 (cid:98)
D

(t, δ),

where C is the normalization constant and

(cid:113)

(cid:102)M1 (cid:99)M1
σ







(cid:18)

2

1

√3σ2
(cid:113)

−

(cid:98)
D

(t, δ)

(cid:19) M2 + M1







D1 =

D2 =

2 (cid:99)M1 (cid:98)J p
(2π)1/2(1
√
t−1)2

t ((cid:98)zj, δ)
(cid:98)
D

−

p(t, δ))

(cid:98)
D
(cid:98)J p
t ((cid:98)zj, δ) =

(t, δ) = e− 1
2 (
1
σ2 +

δM2
3

.

52

2754
2755

2756
2757
2758
2759
2760
2761

2762
2763
2764
2765
2766
2767

2768
2769
2770
2771
2772

2773
2774
2775
2776
2777
2778

2779
2780
2781
2782
2783
2784

2785
2786
2787
2788
2789
2790

2791
2792
2793
2794
2795

2796
2797
2798
2799
2800
2801

2802
2803
2804
2805
2806
2807

Under review as a conference paper at ICLR 2025

Therefore, we conclude that the TV distance between z (conditioned on (ℓi)i∈[t]) and
satisfies that

N

(cid:17)

(cid:16)

(cid:98)z, σ2

t

(cid:90)

z

(cid:12)
(cid:12)
(cid:12)
(cid:12)

P(z

|

(ℓi)i∈[t])

− N

(cid:18)

(cid:98)z,

σ2
t

(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)

dz

d
(cid:88)

(cid:90)

≤

j=1

zj

(cid:12)
(cid:12)
(cid:12)
(cid:12)

P(zj |

(ℓij)i∈[t])

− N

(cid:18)

(cid:98)zj,

σ2
t

(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)

dzj ≤ O

(d/t),

due to the independence of (zj)j∈[d] conditioned on ℓ1:t. Now we denote algorithm (cid:92)FTPL to be the
FTPL algorithm w.r.t. the noise distribution P(z
(ℓi)i∈[t]), and FTPL to be the algorithm w.r.t. the
((cid:98)z, σ2
t ). Therefore, we have
noise distribution

|

N

(cid:12)
(cid:12)RegretFTPL((ℓ)i∈[T ])

Regret(cid:91)FTPL((ℓ)i∈[T ])(cid:12)
(cid:12)

−

T
(cid:88)

t=1

≤

πt − (cid:98)πt∥
d
∥

∞

T
(cid:88)

(cid:90)

t=1

z

d

≤

(cid:12)
(cid:12)P(z

|

(ℓi)i∈[t])

((cid:98)z,

− N

σ2
t

(cid:12)
(cid:12)dz =

)

O

(d2 log T ).

In other words, using P(z

(d2 log T ). Similarly, it is easy to see that

|

O

(ℓi)i∈[t]) as the noise distribution only increases the regret by

(cid:12)
(cid:12)D-RegretFTPL((ℓ)i∈[T ])

D-Regret(cid:91)FTPL((ℓ)i∈[T ])(cid:12)
(cid:12)

−

(d2 log T ),

≤ O

which completes the proof.

D.7.2 RELAXATION UNDER DECISION-IRRELEVANT PRE-TRAINING DATA

We then remark on the possible relaxation when the training data may not all come from decision-
making tasks.
Remark D.3 (Pre-training with relaxed data assumptions). Note that the pre-training (text) data are
so far assumed to be related to decision-making problems (though not necessarily sequential ones),
see Assumption 1 and Example 1 for instance. It can also be generalized to the text datasets involv-
ing Question-Answering (Q-A), a typical task in natural language processing, where the true/fact
answer, sampled answers from different human users (with possibly wrong or biased answers), cor-
respond to the latent z (and associated maximizer a) and ℓ1:t, respectively. Moreover, in practice,
the pre-training data may also involve non-decision-making/Q-A texts, given the diversity of the
datasets. For such scenarios, we will make the assumptions on the data distribution conditioned on
the prompt for decision-making. Specifically, when interacting with the LLM, human users will pro-
vide prompts (see e.g., our Figure C.1), to induce it to make decisions. This will query the conditional
distribution of

P (cid:0)g(xNt+1:Nt+1) (cid:12)

(cid:12) x1:Nt, decision-making prompt(cid:1)

to generate the control action. Correspondingly, Assumption 1 will thus only need to be made on
P (cid:0)z, ℓ1:t, x1:Nt+1, decision-making prompt(cid:1) ,
while we do not need to make such assumptions on other prompts, e.g., corpora that are not related
to decision-making.

D.8 COMPARISON WITH LEE ET AL. (2023); LIN ET AL. (2024); LIU ET AL. (2023E)

Intriguingly, similar assumptions and pre-training objectives have also been considered in the very
recent work of Lee et al. (2023); Lin et al. (2024); Liu et al. (2023e) for studying in-context rein-
forcement learning property of Transformers/LLM-agents under supervised pre-training. Lee et al.
(2023) established its equivalence to posterior sampling (Osband et al., 2013), an important RL al-
gorithm with provable regret guarantees when the environments are stationary, and Lin et al. (2024)
generalized the study to the setting of algorithm distillation as in Laskin et al. (2023). Liu et al.
(2023e) adopted the similar data generation assumption as Lee et al. (2023) without assuming op-
timal labels are available in the pre-training datasets, but leverages external oracles for planning.
Consequently, the resulting LLM agent would still perform the posterior sampling algorithm. How-
ever, these results cannot directly imply the no-regret guarantee in our online learning setting, due

53

2808
2809

2810
2811
2812
2813
2814
2815

2816
2817
2818
2819
2820
2821

2822
2823
2824
2825
2826

2827
2828
2829
2830
2831
2832

2833
2834
2835
2836
2837
2838

2839
2840
2841
2842
2843
2844

2845
2846
2847
2848
2849

2850
2851
2852
2853
2854
2855

2856
2857
2858
2859
2860
2861

Under review as a conference paper at ICLR 2025

to the known fact that posterior sampling can perform poorly under potentially adversarial or non-
stationary environments (Zimmert & Seldin, 2021; Liu et al., 2023d). In contrast, we here establish
the equivalence of the pre-trained LLM to the FTPL algorithm (under different pre-training data dis-
tribution specifications), with the ability to handle arbitrary loss sequences, even though the LLMs
are only trained on a fixed/stationary distribution of texts (tasks).

D.9 DETAILS OF ESTIMATING THE PARAMETERS OF OUR HYPOTHETICAL MODEL

{

To further validate our model and data distribution assumptions, we also propose to estimate the
ηt}t∈[T −1] in Definition 4.1, using data from interacting with LLMs (following the
parameter
same protocol as before), with Pnoise being a standard normal distribution (note that we do not
need to estimate η0 by Definition 4.1). Specifically, given n episodes of the LLM agent’s behavior
(ℓ(j)
ηt}t∈[T −1] by
t
{
{
solving the following problem
(cid:13)
(cid:13)π(j)
(cid:13)

)t∈[T ]}j∈[n], motivated by our Lemma 1 and Theorem 4.1, we estimate

t = σ⋆√t + 1,
η⋆

t+1
quantal

, π(j)
t

ℓ(j)
1:t

P σ

(cid:88)

(cid:88)

σ⋆

[T

(cid:16)

√

,

arg min
σ>0

(cid:17)(cid:13)
(cid:13)
(cid:13)1

t+1 −

t
∀

(cid:12)
(cid:12)
(cid:12)
(cid:12)

−

∈

∈

·

1].

t∈[T −1]

j∈[n]

We solve this single-variable optimization problem by grid search over [0, 10]. We then run the
η⋆
t }t∈[T −1] on another unseen test set, and
generalized quantal response model with the estimated
compare it with the behavior of the actual LLM agents. We use all the interaction data from Sec-
tion 3.2 and split it in half for training and testing.

{

We also use the same framework to understand the regrettable behaviors in Section 3.4. This analysis
uses all the data from Section 3.4. We first find that such fitting procedures do not yield good
predictions for LLMs on those counter-examples. Therefore, we resort to a more expressive model
by directly fitting each ηt as

η⋆
t ∈

arg min
ηt>0

(cid:88)

j∈[n]

(cid:13)
(cid:13)π(j)
(cid:13)

t+1 −

P ηt

quantal

(cid:16)

(cid:12)
(cid:12)
(cid:12)
(cid:12)

ℓ(j)
1:t

(cid:17)(cid:13)
(cid:13)
(cid:13)1

·

separately for each t
1]. Even under the expressive model, LLMs fail to follow the generalized
quantal response for the counter-examples with noisy alternating or adaptive loss sequences, as
Figure 4.1 shows the gap between GPT-4 (dynamic) regret and the our model’s (dynamic) regret.

[T

−

∈

E DEFERRED RESULTS AND PROOFS IN SECTION 5

E.1 BASIC LEMMAS

Lemma 5 (Double iterated limit). For a sequence (amn)m,n∈N+, suppose that limm,n→∞ amn =
L. Then the following are equivalent:

• For each m, limn→∞ amn exists;

• limm→∞ limn→∞ amn = L.

Lemma 6 (Hoeffding’s inequality). Let X1, X2, . . . , Xn be independent random variables bounded
by the intervals [ai, bi], respectively. Define ¯X = 1
i=1 Xi and let µ = E[ ¯X] be the expected
n
value of ¯X. Then, for any t > 0,

(cid:80)n

¯X
P(
|

−

µ

| ≥

(cid:18)

t)

2 exp

2n2t2
i=1(bi −
Interchanging limit and infimum). If (fn : X

ai)2

(cid:80)n

−

≤

.

(cid:19)

Lemma 7 (Uniform convergence =
a sequence of continuous functions that uniformly converge to a function f : X
X, then limn→∞ inf x∈X fn(x) = inf x∈X f (x) holds.

⇒

→

R)n∈N+ is
R on the domain

→

E.2 DEFERRED PROOF FOR THE ARGUMENTS IN SECTION 5.1

In this section, we prove some properties of
Throughout this subsection, we will assume the following condition holds.

L

(θ, k, N ) under certain regularity conditions of f, h.

54

2862

2863
2864
2865
2866
2867

2868
2869
2870
2871
2872
2873

2874
2875
2876
2877
2878
2879

2880
2881
2882
2883
2884
2885

2886
2887
2888
2889
2890

2891
2892
2893
2894
2895
2896

2897
2898
2899
2900
2901
2902

2903
2904
2905
2906
2907
2908

2909
2910
2911
2912
2913

2914
2915

Under review as a conference paper at ICLR 2025

R+, suppose h(
) and f (
·

→
N+. The derivative h′ : R

log f (R1, k2)

, k) are both
·
R is also a
→

log f (R2, k1)

≥
1(R1 > R2) + 1(R1 = R2), with the convention of

−
k2, i.e., log f is supermodular. Lastly, f is a function such
0 = 0. Lastly,

→

N+

R+ and f : R

Condition 1. For h : R
continuous and non-decreasing functions for any k
continuous function. Moreover, f satisfies that log f (R1, k1)
R2 and k1 ≥
log f (R2, k2) for R1 ≥
f (R1,k)
that limk→∞
f (R2,k) =
∞ ·
(ℓ(j)
Claim 1 (Iterated limit of

t )t∈[T ],j∈[N ] are continuous random variables supported on [

−
(θ, k, N ) is the same as double limit of

−

×

∈

L

B, B]T ×N .

∞ ·

(θ, k, N )). It holds that:

L

(cid:18)

(cid:19)

lim
N→∞

lim
k→∞ L

(θ, k, N ) = lim

N,k→∞ L

(θ, k, N ) = lim
k→∞

lim
N→∞ L

(θ, k, N ) = h

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

.

Proof. Step 1. Proving limN→∞ limk→∞

(θ, k, N ) = h (cid:0)maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])(cid:1).

L

Firstly,
E

as

(cid:104)

h

both
h(maxj∈[N ] RegretLLMθ ((ℓ(j)

are
t )t∈[T ]))

and

f

non-negative
(cid:105)

and
(Condition
exists, we have by dominated conver-

1),

(ℓ(j)
t

)t∈[T ],j∈[N ]
gence theorem that

lim
k→∞ L

(θ, k, N ) = E lim
k→∞

(cid:34) (cid:80)

j∈[N ] h(RLLMθ ((ℓ(j)
(cid:80)

j∈[N ] f (RLLMθ ((ℓ(j)

t )t∈[T ]))f (RLLMθ ((ℓ(j)
i )t∈[T ]), k)
(cid:21)

RLLMθ ((ℓ(j)

t )t∈[T ]))

= E

(ℓ(j)
t

)t∈[T ],j∈[N ]

(cid:20)

h( max
j∈[N ]

t )t∈[T ]), k)

(cid:35)

where RLLMθ denotes an abbreviation of RegretLLMθ . By (Ahsanullah et al., 2013, Chapter 11), we
have h(maxj∈[N ] RegretLLMθ ((ℓ(j)
∞

(θ, k, N ) = h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])) holds.

h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])) when N

. Hence, we have limN→∞ limk→∞

t )t∈[T ]))

p
→

→

L

Step 2. Proving limN,k→∞

(θ, k, N ) = h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])).

L

Now, we will calculate limN,k→∞
(θ, k, N ).
Lemma 8. For any 0 < ϵ < 1, it follows that

L

lim
N,k→∞

(cid:80)N
i=1 f (Xi, k)H(Xi)1(H(Xi) < 1
i=1 f (Xi, k)H(Xi)1(H(Xi) > 1

(cid:80)N

ϵ)

−
ϵ/2)

−

= 0

and

(cid:80)N
i=1 f (Xi, k)1(H(Xi) < 1
i=1 f (Xi, k)1(H(Xi) > 1
hold with probability 1, where Xi’s are i.i.d. random variables, esssup(H(Xi)) = 1, and H : R
R+ is a continuous non-decreasing function.

lim
N,k→∞

−
ϵ/2)

(cid:80)N

= 0

ϵ)

−

→

, k), H are non-negative and non-decreasing functions, we have
Proof of Lemma 8. Since f (
·

(cid:80)N
i=1 f (Xi, k)H(Xi)1(H(Xi) < 1
i=1 f (Xi, k)H(Xi)1(H(Xi) > 1

(cid:80)N

ϵ)
−
ϵ/2) ≤

−

and we know that

(1

ϵ)f (H −1(1
−
ϵ/2)f (H −1(1

(1

−

i
ϵ), k)
|{
ϵ/2), k)
|{

[N ]
|
[N ]

∈
i
∈

(H(Xi) < 1
−
(H(Xi) > 1
−

|

ϵ)
}|
ϵ/2)

}|

−
−

i
|{
i

ϵ)
}|
−
ϵ/2)
}|
−
, where F is the cumulative distribution function of random variable H(X). Therefore,

(H(Xi) < 1
|
(H(Xi) > 1

[N ]
∈
[N ]
∈

ϵ)
ϵ/2)

F (1
F (1

−
−

a.s.
→

−

|{

1

|

as N
we have

→ ∞

0

≤

lim
N,k→∞

(cid:80)N
i=1 f (Xi, k)H(Xi)1(H(Xi) < 1
i=1 f (Xi, k)H(Xi)1(H(Xi) > 1

(cid:80)N

ϵ)
−
ϵ/2) ≤

−

lim
N,k→∞

(1

(1

−
(1

ϵ)f (H −1(1
i
ϵ), k))
|{
−
−
ϵ/2)f (H −1(1
ϵ/2), k))
|{
−
ϵ)f (H −1(1
−
ϵ/2)f (H −1(1

ϵ), k))
ϵ/2), k))

−
−

(1

−

[N ]
|
[N ]

∈
i
∈

(H(Xi) < 1
−
(H(Xi) > 1
−

|

ϵ)

}|

ϵ/2)

}|

F (1
F (1

−
−

ϵ)
ϵ/2)

1

−

= 0.

lim
N,k→∞

≤a.s.

55

2916
2917

2918
2919
2920
2921
2922
2923

2924
2925
2926
2927
2928
2929

2930
2931
2932
2933
2934

2935
2936
2937
2938
2939
2940

2941
2942
2943
2944
2945
2946

2947
2948
2949
2950
2951
2952

2953
2954
2955
2956
2957

2958
2959
2960
2961
2962
2963

2964
2965
2966
2967
2968
2969

Under review as a conference paper at ICLR 2025

By a similar argument, we have

lim
N,k→∞

with probability 1.

(cid:80)N
i=1 f (Xi, k)1(H(Xi) < 1
i=1 f (Xi, k)1(H(Xi) > 1

(cid:80)N

ϵ)

−
ϵ/2)

−

= 0

One key idea in the proof above is the use of some truncation level ϵ for H(X) with
esssup(H(X)) = 1. By Lemma 8, we have
i=1 f (Xi, k)H(Xi)1(H(Xi) > 1

i=1 f (Xi, k)1(H(Xi) > 1

(cid:80)N

(cid:80)N

ϵ)

ϵ)

(cid:80)N

i=1 f (Xi, k)H(Xi)

−

= lim

N,k→∞

(cid:80)N

i=1 f (Xi, k)

−

= 1,

lim
N,k→∞

since

(cid:80)N

i=1 f (Xi, k)1(H(Xi) < 1

0

≤

(cid:80)N

i=1 f (Xi, k)
holds with probability 1. Therefore, for any 0 < ϵ < 1, we have

ϵ)

−

(cid:80)N
i=1 f (Xi, k)1(H(Xi) < 1
i=1 f (Xi, k)1(H(Xi) > 1

(cid:80)N

ϵ)

−
ϵ/2)

≤

−

(θ, k, N ) = E lim

N,k→∞

lim
N,k→∞ L
(cid:18)

j∈[N ] h(RLLMθ ((ℓ(j)
(cid:80)

t )t∈[T ]))f (RLLMθ ((ℓ(j)
i )t∈[T ]), k)

j∈[N ] f (RLLMθ ((ℓ(j)

t )t∈[T ]), k)

(cid:35)

(cid:34) (cid:80)

(cid:19)

= h

max
ℓ1,...,ℓT

RLLMθ ((ℓt)t∈[T ])

(cid:80)

j∈[N ]






E lim

N,k→∞

×

t

h(RLLMθ ((ℓ(j)

)t∈[T ]))

h(maxℓ1,...,ℓT RLLMθ ((ℓt)t∈[T ])) f (RLLMθ ((ℓ(j)
i )t∈[T ]), k)1(

j∈[N ] f (RLLMθ ((ℓ(j)

(cid:80)

t )t∈[T ]), k)1(

h(RLLMθ ((ℓ(j)

t

)t∈[T ]))

h(maxℓ1,...,ℓT RLLMθ ((ℓt)t∈[T ])) > 1

h(RLLMθ ((ℓ(j)

t

)t∈[T ]))

h(maxℓ1 ,...,ℓT RLLMθ ((ℓt)t∈[T ])) > 1

ϵ)

−






ϵ)

−

(cid:19)

(1

ϵ)h( max
ℓ1,...,ℓT

−

≥
which implies limN,k→∞

RLLMθ ((ℓt)t∈[T ]))

(θ, k, N ) = h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])) since

L

(cid:18)

(cid:19)

(θ, k, N )

h

≤

max
ℓ1,...,ℓT

L

RegretLLMθ ((ℓt)t∈[T ])

by definition of

, the fact that h is non-decreasing, and by setting ϵ

L

(cid:18)

(θ, k, N )

h

≥

max
ℓ1,...,ℓT

L

→
RegretLLMθ ((ℓt)t∈[T ])

.

0 to obtain
(cid:19)

Here, we used the fact that (ℓt)t∈[T ] has a continuous distribution, RegretLLMθ ((ℓt)t∈[T ]) is a con-
tinuous function, and the non-decreasing property and continuity of h (Condition 1), which lead
to:
esssup (cid:0)h (cid:0)RegretLLMθ ((ℓt)t∈[T ])(cid:1)(cid:1) = max

h (cid:0)RegretLLMθ ((ℓt)t∈[T ])(cid:1) = h

(cid:18)

ℓ1,...,ℓT

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

.

Equation (E.1) will be used frequently in the overall proof in Section E.2.

Step 3. Proving limk→∞ limN→∞

(θ, k, N ) = h (cid:0)maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])(cid:1).
L

Lastly, if N

→ ∞

, similarly by dominated convergence theorem we have
(cid:16)

(cid:17)(cid:17)

(cid:16)

lim
N→∞ L

(θ, k, N ) = E lim
N→∞





(cid:80)

j∈[N ] h

RLLMθ

(ℓ(j)
t )t∈[T ]
(cid:16)
(cid:16)

RLLMθ

f (RLLMθ ((ℓ(j)
t )t∈[T ]), k)
(cid:17)
(cid:17)

(cid:80)

j∈[N ] f
(cid:17)(cid:17)

(cid:16)

f

(cid:16)
(ℓ(j)
t )t∈[T ]
(cid:16)
(cid:104)

f

RLLMθ

E

(cid:16)

RLLMθ
(cid:17)

(ℓ(j)

i )t∈[T ]

, k

(cid:16)

(ℓ(j)

i )t∈[T ]
(cid:17)

(ℓ(j)
t )t∈[T ]
(cid:17)(cid:105)

, k

(cid:17)(cid:105)

, k

.

E

(cid:16)

(cid:104)

h

RLLMθ

=

(E.1)





limN→∞

Thus,
L
erty of double iterated limit (Lemma 5), and obtain that
h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])).

(θ, k, N ) always exists for every k.

Now, we use the known prop-
(θ, k, N ) =

limk→∞ limN→∞

L

56

2970

2971
2972
2973
2974
2975

2976
2977
2978
2979
2980
2981

2982
2983
2984
2985
2986
2987

2988
2989
2990
2991
2992
2993

2994
2995
2996
2997
2998

2999
3000
3001
3002
3003
3004

3005
3006
3007
3008
3009
3010

3011
3012
3013
3014
3015
3016

3017
3018
3019
3020
3021

3022
3023

Under review as a conference paper at ICLR 2025

Claim 2 (Uniform convergence of
converges to h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])) on the domain Θ.

(θ, k, N ) (with respect to k and N )).

L

(θ, k, N ) uniformly

L

Proof. We will provide a similar analysis as Lemma 8 as follows:
N+, we have

Lemma 9. For any 0 < ϵ < 1, 0 < δ < 1, and k

∈

(cid:80)N

(cid:80)N

i=1 f (Xi, k)1(H(Xi) < 1
i=1 f (Xi, k)1(H(Xi) > 1

ϵ)

ϵ)

−

−

= (cid:101)
O

(cid:18)

(cid:18)

A(k, H, ϵ)

1

FH,X (1

1

−

ϵ/2)

−

(cid:19)(cid:19)

+

1
√N

R

with probability at least 1
1, H :
(1−ϵ/2)f ((t/ esssup(t(X)))−1(1−ϵ/2),k) , for any non-decreasing function t : R
is a cumulative distribution function of random variable t(X)/esssup(t(X)).

random variables, esssup(H(Xi)) =
:=
R+, and Ft,X

−
R+ is a continuous non-decreasing function, A(k, t, ϵ)

(1−ϵ)f ((t/ esssup(t(X)))−1(1−ϵ),k)

δ, where Xi’s are i.i.d.

→

→

Proof of Lemma 9. With the same argument as the proof of Lemma 8, we have

(cid:80)N
i=1 f (Xi, k)1(H(Xi) < 1
i=1 f (Xi, k)1(H(Xi) > 1

(cid:80)N

ϵ)
−
ϵ/2) ≤

−

f (H −1(1

f (H −1(1

−

ϵ), k)
−
ϵ/2), k)

i
|{
i
|{

[N ]
[N ]

|
|

∈
∈

(H(Xi) < 1
(H(Xi) > 1

ϵ)
}|
ϵ/2)

}|

.

−
−

i

N |{

It holds that 1
least 1
1

−
ϵ/2)

= 1

−

}|

[N ]

(H(Xi) < 1

ϵ)

= FH,X (1

−
δ/2 due to Hoeffding’s inequality (Lemma 6). Similarly, we have 1

−

}|

∈

|

ϵ) + (cid:101)
O

FH,X (1

ϵ/2) + (cid:101)
O

−

−

(1/√N ) with probability at least 1

(1/√N ) with probability at
(H(Xi) >
[N ]
δ/2. Therefore,

∈

i

|

N |{
−

i

[N ]
∈
[N ]
∈

|

|{
i

|{

(H(Xi) < 1
|
(H(Xi) > 1

ϵ)
}|
ϵ/2)

−
−

}|

=

1

FH,X (1
FH,X (1

ϵ)
ϵ/2)

−
−

(cid:112)

1/N )

(

+ (cid:101)
O

1

≤

1

−

FH,X (1

−

ϵ/2)

+ (cid:101)
O

(cid:112)
(

1/N ),

−

δ. Finally, we have

with probability at least 1

−
i=1 f (Xi, k)1(H(Xi) < 1
i=1 f (Xi, k)1(H(Xi) > 1

(cid:80)N

(cid:80)N

<

ϵ)

ϵ)

−

−

(cid:80)N
i=1 f (Xi, k)1(H(Xi) < 1
i=1 f (Xi, k)1(H(Xi) > 1

(cid:80)N

ϵ)
−
ϵ/2) ≤

−

(cid:18)

A(k, H, ϵ)

1

FH,X (1

1

−

−

ϵ/2)

(

+ (cid:101)
O

(cid:19)
)

.

1
√N

Note that limk→∞ A(k, H, ϵ) = 0, since limk→∞

1(R1 > R2) + 1(R1 = R2). By

f (R1,k)
f (R2,k) =
h(RLLMθ ((ℓt)t∈[T ]))

∞ ·

Lemma 9 with H(RLLMθ ((ℓt)t∈[T ])) =

h(maxℓ1,...,ℓT RLLMθ ((ℓt)t∈[T ])) , we have

(cid:80)N

i=1 f (RLLMθ ((ℓ(i)

t )t∈[T ]), k)1

(cid:18)

h(RLLMθ
h(maxℓ1,...,ℓT

(i)
t )t∈[T ]))

((ℓ
RLLMθ

((ℓt)t∈[T ])) ≥ 1 − ϵ

(cid:19)

=

1 +

(cid:80)N

i=1 f (RLLMθ ((ℓ(i)
t )t∈[T ]), k)
1



(cid:80)N

i=1 f (RLLMθ

((ℓ

(i)
t )t∈[T ]),k)1



(cid:80)N

i=1 f (RLLMθ

((ℓ

(i)
t )t∈[T ]),k)1





h(RLLMθ
h(maxℓ1,...,ℓT

h(RLLMθ
h(maxℓ1,...,ℓT

(i)
((ℓ
t
RLLMθ
(i)
((ℓ
t
RLLMθ

)t∈[T ] ))

((ℓt)t∈[T ] )) <1−ϵ



)t∈[T ] ))

((ℓt)t∈[T ] )) ≥1−ϵ







≥

1 + A(k, H, ϵ)(

1
1

((ℓt)t∈[T ] )(1−ϵ/2) + (cid:101)O((cid:112)1/N ))

,

1−FH,RLLMθ

57

3024
3025

3026
3027
3028
3029
3030
3031

3032
3033
3034
3035
3036
3037

3038
3039
3040
3041
3042

3043
3044
3045
3046
3047
3048

3049
3050
3051
3052
3053
3054

3055
3056
3057
3058
3059
3060

3061
3062
3063
3064
3065

3066
3067
3068
3069
3070
3071

3072
3073
3074
3075
3076
3077

Under review as a conference paper at ICLR 2025

where we recall the shorthand notation of RLLMθ = RegretLLMθ . Note that A(k, H, ϵ) = A(k, h, ϵ)
and FH,RLLMθ

= Fh,RLLMθ hold by the definitions of Ft,X and A(k, t, ϵ) in Lemma 9. Therefore,

1

≥

≥

(cid:80)N

i=1 f (RLLMθ ((ℓ(i)

h(RLLMθ ((ℓ(i)

t )t∈[T ]))

t )t∈[T ]), k)
i=1 f (RLLMθ ((ℓ(i)

(cid:80)N

h(maxℓ1,...,ℓT RLLMθ ((ℓt)t∈[T ]))
t )t∈[T ]), k)

(cid:80)N

i=1 f (RLLMθ ((ℓ(i)

t )t∈[T ]), k)

h(RLLMθ ((ℓ(i)

t )t∈[T ]))

h(RLLMθ ((ℓ(i)

t )t∈[T ]))

1(

h(RLLMθ ((ℓ(i)

h(maxℓ1,...,ℓT RLLMθ ((ℓt)t∈[T ])) ≥
t )t∈[T ]))

h(maxℓ1 ,...,ℓT RLLMθ ((ℓt)t∈[T ])) ≥

1

ϵ)

−

ϵ)

1

−

(cid:80)N

h(maxℓ1,...,ℓT RLLMθ ((ℓt)t∈[T ]))
i=1 f (RLLMθ ((ℓ(i)
t )t∈[T ]), k)1(
1
1

(cid:112)

((ℓt)t∈[T ] )(1−ϵ/2) + (cid:101)
1−Fh,RLLMθ
O
1
ϵ
−
1
((ℓt)t∈[T ])(1−ϵ/2) + (cid:101)
O

(cid:112)
(

1−Fh,RLLMθ

1/N ))

(

1/N ))

×

1 + A(k, h, ϵ)(

≥

1 + A(k, h, ϵ)(

with probability at least 1

δ.

−

Now, for any ϵ > 0 and δ > 0, we have

h

0

≤

(cid:18)

(cid:18)

max
ℓ1,...,ℓT

RLLMθ ((ℓt)t∈[T ])

(cid:19)

(cid:19)

(θ, k, N )

− L


h

≤

max
ℓ1,...,ℓT

RLLMθ ((ℓt)t∈[T ])

1

−

1 + A(k, h, ϵ)(

Note that

(1

−
1−Fh,RLLMθ

−

δ)(1
ϵ)
1
((ℓt)t∈[T ])(1−ϵ/2) + (cid:101)
O

(cid:112)
(

1/N ))



 .

1
−

Fh,RLLMθ ((ℓt)t∈[T ])(1

−

(cid:18)

ϵ/2) = P

h (cid:0)RegretLLMθ

(cid:0)(ℓt)t∈[T ]

(cid:1)(cid:1) > (1

(cid:18)

ϵ/2)h

−

max
ℓ1,...,ℓT

RegretLLMθ

(cid:19)(cid:19)

(cid:0)(ℓt)t∈[T ]

(cid:1)

is a continuous function of θ, since we assume LLMθ is a continuous function of θ, (ℓt)t∈[T ] has a
continuous distribution, and RegretLLMθ ((ℓt)t∈[T ]) is a continuous function of LLMθ and (ℓt)t∈[T ].
Since we consider a compact Θ (as several recent works on analyzing Transformers (Bai et al., 2023;
Lin et al., 2024)), we have p(ϵ) := minθ∈Θ 1

ϵ/2) > 0. Therefore,



1

−

1 + A(k, h, ϵ)(

(1

δ)(1
−
1
1−Fh,RLLMθ

ϵ)

−
(1−ϵ/2) + (cid:101)
O

(cid:112)

1/N ))

(



≤

(cid:32)
1

−

1 + A(k, h, ϵ)( 1

(1

−

δ)(1
ϵ)
−
p(ϵ) + (cid:101)
O

(

(cid:33)

,

(cid:112)

1/N ))

(E.2)

−

Fh,RLLMθ ((ℓt)t∈[T ])(1


−

and we know that limN,k→∞ 1 + A(k, h, ϵ)( 1
p(ϵ) + (cid:101)
(
O
on θ. Thus, we can conclude that limN,k→∞ supθ∈Θ |
(θ, k, N )
L
Claim 3 (Double iterated limit of supremum). It holds that:

= 0, as we can choose arbitrarily small ϵ, δ.

|

(cid:112)

1/N )) = 1, which is not dependent

h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ]))

−

lim
N→∞

lim
k→∞

sup
θ∈Θ

(cid:12)
(cid:12)
(cid:12)L

(cid:18)

(θ, k, N )

h

−

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

(cid:19) (cid:12)
(cid:12)
(cid:12) = 0.

Proof. Since h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ]))

(cid:18)

≥ L

(θ, k, N ), we will prove

lim
N→∞

lim
k→∞

sup
θ∈Θ

h

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

(cid:19)

− L

(θ, k, N ) = 0.

Lemma 10.
(Xi)i∈[N ].

(cid:80)N

i=1 f (Xi,k1)h(Xi)
(cid:80)N
i=1 f (Xi,k1)

(cid:80)N

i=1 f (Xi,k2)h(Xi)
(cid:80)N
i=1 f (Xi,k2)

≤

holds if 0 <k1 ≤

k2 for any real-valued

58

3078
3079

3080
3081
3082
3083
3084
3085

3086
3087
3088
3089
3090
3091

3092
3093
3094
3095
3096

3097
3098
3099
3100
3101
3102

3103
3104
3105
3106
3107
3108

3109
3110
3111
3112
3113
3114

3115
3116
3117
3118
3119

3120
3121
3122
3123
3124
3125

3126
3127
3128
3129
3130
3131

Under review as a conference paper at ICLR 2025

Proof. By multiplying
mula, we
(cid:80)

know that

((cid:80)N
it

i=1 f (Xi, k1))((cid:80)N
equivalent

is

i=1 f (Xi, k2))
to (cid:80)

the
1≤i̸=j≤N f (Xi, k1)h(Xi)f (Xj, k2)

sides

both

on

of

for-

≤

1≤i̸=j≤N f (Xi, k1)h(Xj)f (Xj, k2). This is equivalent to

(cid:88)

1≤i̸=j≤N

(f (Xi, k1)f (Xj, k2)

−

f (Xj, k1)f (Xi, k2))(h(Xi)

h(Xj))

0,

≤

−

Xj, (f (Xi, k1)f (Xj, k2)

which is true since if Xi ≥
log-increasing difference of f (Condition 1), as log f (Xj, k1)
log f (Xi, k2) if Xi ≥
Therefore,

Xj.

−

L

(θ, k, N ) is a non-decreasing function of k if N is fixed, which indicates that

(cid:18)

(cid:19)

lim
k→∞

sup
θ∈Θ

h

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

(θ, k, N )

− L

f (Xj, k1)f (Xi, k2))

log f (Xj, k2)

−

≥

0 due to the

≤
log f (Xi, k1)

−

exists, as

L

(θ, k, N ) is also bounded. Therefore, by Lemma 5 and Claim 2, we know that
(cid:18)

(cid:12)
(cid:12)
(cid:12)L

(θ, k, N )

h

−

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

(cid:19) (cid:12)
(cid:12)
(cid:12)

lim
N→∞

lim
k→∞

sup
θ∈Θ

exists and this value should be 0.

Claim 4. It holds that

lim
N,k→∞

inf
θ∈Θ L

(θ, k, N ) = lim
N→∞

lim
k→∞

inf
θ∈Θ L

(θ, k, N ) = inf
θ∈Θ

h

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

.

(cid:18)

(cid:19)

7,

by

we

Lemma

Proof. Firstly,
inf θ∈Θ h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])).
a monotonically non-decreasing function of k for any fixed N (Lemma 10), and it
limk→∞ inf θ∈Θ L
bounded,
limN→∞ limk→∞ inf θ∈Θ L
E.3 DEFINITION OF THE EMPIRICAL LOSS FUNCTION

=
(θ, k, N ) is
is
(θ, k, N ) always exists. Therefore, by Lemma 5 , we also have
(θ, k, N ) = inf θ∈Θ h(maxℓ1,...,ℓT RegretLLMθ ((ℓt)t∈[T ])).

limN,k→∞ inf θ∈Θ L
Plus, we already know that

(θ, k, N )

have

L

Definition E.1 (Empirical loss function). We define the empirical loss (cid:98)
L
as follows:

computed with NT samples

(θ, k, N, NT ) :=
(cid:98)
L

(cid:80)

j∈[N ] h

(cid:16)





NT(cid:88)

s=1

1
NT

RegretLLMθ ((ℓ(j)

(cid:80)

j∈[N ] f

(cid:17)

(cid:16)

f

s,t )t∈[T ])
(cid:16)
RegretLLMθ ((ℓ(j)

(cid:17)

s,t )t∈[T ]), k

RegretLLMθ ((ℓ(j)

s,t )t∈[T ]), k

(cid:17)





where (ℓ(j)

s,t )j∈[N ],t∈[T ] denotes the s-th sample of (ℓ(j)

t )j∈[N ],t∈[T ] for estimating

E.4 DEFERRED PROOFS OF THEOREM E.1 AND THEOREM 5.1

(E.3)

(θ, k, N ).

L

Theorem E.1. (Generalization gap). Suppose LLMθ is Lipschitz-continuous with respect to the
model parameter θ, then for any 0 < ϵ < 1/2, with probability at least 1

(cid:16)

L

(cid:98)θk,N,NT , k, N

(cid:17)

−



(cid:115)

inf
θ∈Θ L

(θ, k, N )



(cid:101)
O

≤

dθ + log(1/ϵ)
NT

 ,

(E.4)

−

ϵ, we have


for any N and sufficiently large k, where dθ is the dimension of the parameter θ.

Through a careful use of Berge’s Maximum Theorem (Berge, 1877), we prove that the right-
hand side of Equation (E.4) does not depend on k and N , which allows us to take the limit of
limN→∞ limk→∞ without affecting the generalization bound.

59

3132

3133
3134
3135
3136
3137

3138
3139
3140
3141
3142
3143

3144
3145
3146
3147
3148
3149

3150
3151
3152
3153
3154
3155

3156
3157
3158
3159
3160

3161
3162
3163
3164
3165
3166

3167
3168
3169
3170
3171
3172

3173
3174
3175
3176
3177
3178

3179
3180
3181
3182
3183

3184
3185

Under review as a conference paper at ICLR 2025

Before proving the theorem, we remark on what LLM structure enjoys the Lipschitz-continuity.
We provide two auxiliary results in the following proposition. The first result is from (Bai et al.,
2023, Section J.1), which is about the Lipschitzness of Transformers. The second result is regarding
processing the output of Transformers.
In particular, the output of Transformers is usually not
directly used, but passed through some matrix multiplication (by some matrix A), followed by some
projection Operator (to be specified later).
Proposition 2. The L-layer Transformer TFθ as defined in Appendix B.2 is CTF-Lipschitz contin-
uous with respect to θ with CTF := L (cid:0)(1 + B2
TFR2),
i.e.,

BTFR(1 + BTFR2 + B3

TF)(1 + B2

TFR3)(cid:1)L

TFθ1(Z)

TFθ2(Z)

∥2,∞

CTF

θ1 −

θ2∥

TF

≤
TF is as defined in Equation (B.1), and R, Z, BTF are as introduced in Appendix B.2.
opCTF-Lipschitz continuous with respect
A
∥

where
Moreover, the function Operator(A
to θ, i.e.,

)−1) is
·

TFθ(

∥ · ∥

−

∥

∥

∥

·

Operator(A
∥

·

TFθ1 (Z)−1)

−

Operator(A

TFθ2(Z)−1)

·

A

∥2 ≤ ∥

∥

opCTF

θ1 −
∥

TF.

θ2∥

Here, Operator is either the projection operator onto some convex set, or the Softmax function.

Proof. The first result is from (Bai et al., 2023, Section J.1). The second result comes from

• If Operator is a projection onto the convex set,
∥2;

Operator(y)

x
∥2 ≤ ∥

−

y

then

Operator(x)
∥

−

• If Operator is Softmax, then

Pavel, 2017, Corollary 3).

Softmax(x)
∥

−

Softmax(y)

∥2 ≤ ∥

x

y

∥2 (Gao &

−

Note that the only condition that we require for Operator is its non-expansiveness.

∥ · ∥LLM, where
∥ · ∥

Proof of Theorem E.1. Let CLLM denote the Lipschitz-continuity constant for LLMθ with respect to
∥ · ∥LLM denotes any norm defined on the parameter space of LLM (e.g.,
some norm
the norm
TF above in Proposition 2). Now, we prove that regret is also a Lipschitz-continuous
function with respect to the LLM’s parameter.
Lemma 11 (Lipschitzness of regret). The function RegretLLMθ is CReg := BCLLMT -Lipschitz con-
tinuous with respect to θ, i.e.,
(cid:12)
(cid:12)
(cid:12)RegretLLMθ1

RegretLLMθ2

((ℓt)t∈[T ])

((ℓt)t∈[T ])

CReg

θ1 −
∥

θ2∥LLM.

(cid:12)
(cid:12)
(cid:12) ≤

−

Proof. By definition, we have

(cid:12)
(cid:12)
(cid:12)RegretLLMθ1

((ℓt)t∈[T ])

RegretLLMθ2

−

(cid:12)
(cid:12)
((ℓt)t∈[T ])
(cid:12) =

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

T
(cid:88)
ℓt, LLMθ1(Zt−1)
⟨

t=1

LLMθ2(Zt−1)
⟩

−

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

T
(cid:88)

= B

∥

t=1
BCLLMT

θ1 −
∥
≤
where Zt := (ℓ1, . . . , ℓt, c) for all t

−
θ2∥LLM
∈

LLMθ1 (Zt−1)

LLMθ2 (Zt−1)

∥

[T ] and Z0 = (c) where c is a d-dimensional vector.

Now, we will prove the Lipschitzness of

(cid:16)

C

(ℓ(j)

t )t∈[T ],j∈[N ], k, θ

(cid:17)

:=

(cid:80)

j∈[N ] h(RegretLLMθ ((ℓ(j)
(cid:80)

t )t∈[T ]))f (RegretLLMθ ((ℓ(j)

t )t∈[T ]), k)

j∈[N ] f (RegretLLMθ ((ℓ(j)

t )t∈[t]), k)

with respect to the model parameter θ.

60

(E.5)

3186
3187

3188
3189
3190
3191
3192
3193

3194
3195
3196
3197
3198
3199

3200
3201
3202
3203
3204

3205
3206
3207
3208
3209
3210

3211
3212
3213
3214
3215
3216

3217
3218
3219
3220
3221
3222

3223
3224
3225
3226
3227

3228
3229
3230
3231
3232
3233

3234
3235
3236
3237
3238
3239

Under review as a conference paper at ICLR 2025

Claim 5. For any R > 0, there exists βR > 0 such that if β > βR, we have

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

(cid:80)

n∈[N ] xnf (xn, β)
(cid:80)
n∈[N ] f (xn, β) −

(cid:80)

n∈[N ] ynf (yn, β)
(cid:80)
n∈[N ] f (yn, β)

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12) ≤

x
2
∥

y

∞
∥

−

for every x, y

∈

Rn such that

xi| ≤
|

R,

yi| ≤

|

R for all i

[N ].

∈

Proof. If β =
(cid:32)(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

lim
β→∞

(cid:80)

, we have

∞

n∈[N ] xnf (xn, β)
(cid:80)
n∈[N ] f (xn, β) −

(cid:80)

n∈[N ] ynf (yn, β)
(cid:80)
n∈[N ] f (yn, β)

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

(cid:46)
∥

x

(cid:33)

y

∞
∥

−

= |

maxn∈[N ] xn −
y
x
−
∥

maxn∈[N ] yn|
∞
∥

1

≤

holds. Moreover, consider the following constrained optimization problem:

(cid:80)

(cid:80)

(cid:32)(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

max
x,y∈Rn

n∈[N ] xnf (xn, β)
(cid:80)
n∈[N ] f (xn, β) −
yi| ≤
whose optimum is denoted as F (R, β). Then, since
R is a compact set,
by Berge’s Maximum Theorem (Berge, 1877), we have that F (R, β) is a continuous function for β.
1, which indicates that we can find a large enough βR such
Moreover, we know that F (R,
that if β > βR, F (R, β)

n∈[N ] ynf (yn, β)
(cid:80)
n∈[N ] f (yn, β)

∈
R and

R for all i

xi| ≤
|

subject to

x
∥

∞
∥

∞
∥

R,

∞

(cid:46)

≤

≤

≤

−

2.

x

∞

∥

∥

∥

y

y

)

|

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)
[N ],

(cid:33)

≤

diam(Π,

Note that Claim 5 does not hold if either xi or yi is unbounded. Now, we will apply Claim 5 to
(cid:12)
(cid:12)
(cid:12)RegretLLMθ ((ℓt)t∈[T ])
Equation (E.5). We can guarantee that
(cid:12)
Also, note that the domain of h : R
RegretLLMθ ((ℓt)t∈[T ]) can achieve, which means that we can regard h as h :
∥ · ∥2)T B]
∥2)T B, diam(Π,
→
pact domain, we know that h(
·
∥ · ∥2)T B, diam(Π,
diam(Π,
[
−
Lemma 12 (Lipschitzness of C in Equation (E.5)). The function C in Equation (E.5) is Ccost :=
2ChCReg-Lipschitz continuous with respect to θ, if k > kdiam(Π,∥·∥2)T B for some kdiam(Π,∥·∥2)T B >
0, i.e.,

∥ ·
R+. Due to the continuity of h′, and the fact that h has a com-
) is Ch-Lipschitz continuous for some Ch > 0 on this interval of
∥ · ∥2)T B].

R+ is effectively constrained to the range that

∥ · ∥2)T B.

diam(Π,

→

≤

−

[

(cid:16)

(cid:12)
(cid:12)
(cid:12)C

(ℓ(j)

t )t∈[T ],j∈[N ], k, θ1

(cid:16)

C

(cid:17)

−

(ℓ(j)

t )t∈[T ],j∈[N ], k, θ2

(cid:17) (cid:12)
(cid:12)
(cid:12) ≤

Ccost

θ1 −
∥

θ2∥LLM.

Proof.

(cid:12)
(cid:12)C((ℓ(j)

≤(i)

t )t∈[T ],j∈[N ], k, θ1)
h(RegretLLMθ1
2
∥
2Ch∥
2ChCReg∥

RegretLLMθ1

θ1 −

≤(ii)

≤(iii)

C((ℓ(j)

−
((ℓ(j)
t )t∈[T ]))

t )t∈[T ],j∈[N ], k, θ2)(cid:12)
(cid:12)
h(RegretLLMθ2

−

((ℓ(j)

t )t∈[T ]))

∞
∥

((ℓ(j)

t )t∈[T ])

−
θ2∥LLM = Ccost

RegretLLMθ2

((ℓ(j)

t )t∈[T ])

∞
∥

θ1 −
∥

θ2∥LLM.

Here, (i) holds due to Claim 5, (ii) holds since h is Ch-Lipschitz continuous on the range of
RegretLLMθ ((ℓt)t∈[T ]), and (iii) holds due to Lemma 11.

For completeness of the paper, we provide the definition of covering set and covering number.
Definition E.2 (Covering set and covering number). For δ > 0, a metric space (X,
Y
⊆
N (δ; Y,

∥ · ∥
) is defined as the minimum cardinality of any covering set.

), and subset
∥·∥
) holds. δ-covering number

Y is a δ-covering of Y when Y

⊆ ∪c∈CB(c, δ,

X, set C

⊂

∥ · ∥

61

3240
3241

3242
3243
3244
3245
3246
3247

3248
3249
3250
3251
3252
3253

3254
3255
3256
3257
3258

3259
3260
3261
3262
3263
3264

3265
3266
3267
3268
3269
3270

3271
3272
3273
3274
3275
3276

3277
3278
3279
3280
3281

3282
3283
3284
3285
3286
3287

3288
3289
3290
3291
3292
3293

Under review as a conference paper at ICLR 2025

By (Wainwright, 2019, Example 5.8), for any r > 0, we can verify that the δ-covering number
N (δ; B(0, r,

∥ · ∥LLM),

∥ · ∥LLM) can be bounded by

log N (δ; B(0, r,

∥ · ∥LLM),
where dθ is the dimension of the LLM’s whole parameter. For example, if we use the
consider the Transformer model as defined in Appendix B.2, for any r > 0,
∥ · ∥LLM)

L(3M d2 + 2d(dd′ + 3md2)) log(1 + 2r/δ).

dθ log(1 + 2r/δ),

∥ · ∥LLM),

log N (δ; B(0, r,

∥ · ∥LLM)

≤

≤

TF and

∥ · ∥

Since we consider a compact Θ (as several recent works on analyzing Transformers (Bai et al., 2023;
Lin et al., 2024)), let RΘ := diam(Θ,
∥ · ∥LLM) (which corresponds to BTF for the Transformer
TF), then there exists a set Θ0 with
models as defined in Appendix B.2, with
Θ0 with
log

= dθ log(1 + 2RΘ/δ) such that for any θ

∥ · ∥

Θ0|
|

∥ · ∥LLM =
∈
(cid:16)
(ℓ(j)
t )t∈[T ],j∈[N ], k, θ0

Θ, there exists a θ0 ∈
(cid:17) (cid:12)
(cid:12)
(cid:12) ≤

C

−

(cid:16)

(cid:12)
(cid:12)
(cid:12)C

(ℓ(j)

t )t∈[T ],j∈[N ], k, θ

(cid:17)

Then, by the standard result from statistical learning theory (Wainwright, 2019, Chapter 5), when
trained with NT samples, for every 0 < ϵ < 1/2, with probability at least 1

ϵ, we have

Ccostδ.

−

((cid:98)θk,N,NT , k, N )

inf
θ∈Θ L

(θ, k, N )

−

≤

L

Setting δ = Ω(

(cid:112)

log(ϵ)/NT ), we further obtain

(cid:115)

2(log

|

Θ0|

+ log(2/ϵ))
NT

+ 2Ccostδ.

((cid:98)θk,N,NT , k, N )

L

inf
θ∈Θ L

−

(θ, k, N )



(cid:101)
O

≤



(cid:115)

log

Θ0|
|

+ log(1/ϵ)
NT





with probability at least 1

ϵ, completing the proof.

−

Theorem 5.1. (Regret). Suppose4 for any k
supermodular function (i.e., log f (R1, k1)
R1 ≥

N+, h, f (
, k) are non-decreasing, and log f is a
∈
·
log f (R2, k2) for
log f (R1, k2)
k2). Then, with high probably, we have

R2 and k1 ≥

log f (R2, k1)

−

−

≥

(cid:32)

h

lim
N→∞

lim
k→∞

max
∥ℓt∥∞≤B

RegretLLM

(cid:98)θk,N,NT

(cid:0)(ℓt)t∈[T ]

(cid:1)

(cid:33)

≤ h

(cid:18)

inf
θ∈Θ

max
∥ℓt∥∞≤B

RegretLLMθ

(cid:0)(ℓt)t∈[T ]

(cid:19)

(cid:1)



(cid:115)

+ (cid:101)O





 .

dθ
NT

Proof. The limit on the right-hand side of Equation (E.4) remains as (cid:101)
O

(cid:18)(cid:113) dθ+log(1/ϵ)

(cid:19)

NT

, since we

firstly take limk→∞ and then take limN→∞, thanks to the fact that Theorem E.1 holds for large
enough k and any N . Next, we have
(cid:18)

lim
N→∞

lim
k→∞

((cid:98)θk,N,NT , k, N )

(cid:12)
(cid:12)
(cid:12)
(cid:12)L

h

−

lim
N→∞
(cid:18)

lim
k→∞

max
∥ℓt∥∞≤B

RegretLLM (cid:98)θk,N,NT

lim
N→∞

lim
k→∞

≤

((cid:98)θk,N,NT , k, N )

h

−

max
∥ℓt∥∞≤B

RegretLLM (cid:98)θk,N,NT
(cid:18)
(cid:19)

((ℓt)t∈[T ])

lim
N→∞

lim
k→∞

max
∥ℓt∥∞≤B

RegretLLM (cid:98)θk,N,NT

((ℓt)t∈[T ])

h

−

(cid:12)
(cid:12)
(cid:12)
(cid:12)L
(cid:12)
(cid:12)
(cid:12)
(cid:12)

(cid:18)

h

((ℓt)t∈[T ])

(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)

(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)

+

lim
N→∞

lim
k→∞

sup
θ∈Θ

≤

(cid:12)
(cid:12)
(cid:12)
(cid:12)L

(θ, k, N )

(cid:18)

h

−

max
∥ℓt∥∞≤B

RegretLLMθ ((ℓt)t∈[T ])

+ 0 = 0,

lim
N→∞
(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)

lim
k→∞

max
∥ℓt∥∞≤B

RegretLLM (cid:98)θk,N,NT

((ℓt)t∈[T ])

(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)

due to the continuity of h and Claim 3. Finally, we have

lim
N→∞

lim
k→∞

inf
θ∈Θ L

(θ, k, N ) = inf
θ∈Θ

h

max
ℓ1,...,ℓT

RegretLLMθ ((ℓt)t∈[T ])

(cid:18)

(cid:19)

due to Claim 4, which, combined with the fact that h is non-decreasing, completes the proof.

4Note that these conditions on h, f are in addition to those specified after Equation (5.2).

62

3294

3295
3296
3297
3298
3299

3300
3301
3302
3303
3304
3305

3306
3307
3308
3309
3310
3311

3312
3313
3314
3315
3316
3317

3318
3319
3320
3321
3322

3323
3324
3325
3326
3327
3328

3329
3330
3331
3332
3333
3334

3335
3336
3337
3338
3339
3340

3341
3342
3343
3344
3345

3346
3347

Under review as a conference paper at ICLR 2025

As a result, the coarse correlated equilibrium will emerge as the long-term interactions of multiple
such learned LLMs, as stated in the following corollary.
Corollary 1. (Emerging behavior: Coarse correlated equilibrium). For a sufficiently large NT , if
, then the time-averaged policy for
each agent in the matrix game plays according to LLM(cid:98)θk,N,NT
each agent will constitute an approximate coarse correlated equilibrium of the game.
Remark E.1 (Dynamic-regret loss). So far, we have focused on the canonical online learning set-
ting with regret being the metric. One can also generalize the results to the non-stationary setting,
with dynamic regret being the metric. Specifically, one can define the dynamic-regret-loss function
as follows:

(θ, k, N ) := E

L

(cid:34) (cid:80)

j∈[N ] h(D-RegretLLMθ ((ℓ(j)
(cid:80)

t )t∈[T ]))f (D-RegretLLMθ ((ℓ(j)

t )t∈[T ]), k)

(cid:35)

.

j∈[N ] f (D-RegretLLMθ ((ℓ(j)

i )t∈[T ]), k)

Then, one can also establish similar results as before, since the analysis does not utilize other
properties of the regret except its boundedness, and the Lipschitz-continuity of LLM with respect to
θ. To be specific, Lemma 11 holds due to the reason that we can bound the difference of the regret
with the term

T
(cid:88)
⟨
as well as the fact that inf πi∈Π⟨
Section E.2 also hold for similar reasons.

ℓi, πi⟩

t=1

(cid:12)
(cid:12)
(cid:12)
(cid:12)
(cid:12)

ℓt, (LLMθ1(Zt−1)

LLMθ2(Zt−1))

−

(cid:12)
(cid:12)
(cid:12)
(cid:12)
⟩
(cid:12)

,

will be canceled. One can verify that all the arguments in

·

E.5 DETAILED EXPLANATION OF OPTIMIZING EQUATION (5.2) WITH SINGLE-LAYER

SELF-ATTENTION MODEL

We consider the following structure of single-layer self-attention model g (see a formal introduction
in Appendix B.2):

⊺
⊺
t )Softmax ((Kℓ1:t + kc111
g(Zt; V, K, Q, vc, kc, qc) := (V ℓ1:t + vc111
t )

⊺

(Qc + qc)) ,

(E.6)

∈

Rd correspond to the bias terms associated with V, K, Q, and c

Rd×d correspond to the value, key, and query matrices,
= 000d is a

where Zt = (ℓ1, . . . , ℓt, c) and V, K, Q
respectively, vc, kc, qc ∈
constant vector. We then have the following result.
Theorem E.2. Consider the policy space Π = B(0, RΠ,
) for some RΠ > 0. The configuration
of a single-layer self-attention model in Equation (E.6) (V, K, Q, vc, kc, qc) such that K ⊺(Qc +
(cid:20)(cid:13)
(cid:13)
Σ−1 is a first-order stationary
(cid:13)
(cid:13)

qc) = vc = 000d and V =

∥ · ∥
(cid:13)
(cid:13)
(cid:13)
(cid:13)

Σ−1E

T
(cid:80)T −1

⊺
ℓ1ℓ
2

t=1 ℓt

t=1 1/t

(cid:80)T

RΠ

(cid:21)

−

point of Equation (5.2) with N = 1, h(x) = x2. Moreover, if Σ is a diagonal matrix, then plugging
this configuration into Equation (E.6), and projecting the output with ProjΠ,∥·∥ would perform
FTRL with an L2-regularizer for the loss vectors (ℓt)t∈[T ].

In practical training, such stationary points of the loss may be attained by first-order optimization
algorithms of (stochastic) gradient descent, the workhorse in machine learning.

E.6 DEFERRED PROOF OF THEOREM E.2

Theorem E.2. Consider the policy space Π = B(0, RΠ,
) for some RΠ > 0. The configuration
of a single-layer self-attention model in Equation (E.6) (V, K, Q, vc, kc, qc) such that K ⊺(Qc +
(cid:20)(cid:13)
(cid:13)
Σ−1 is a first-order stationary
(cid:13)
(cid:13)

qc) = vc = 000d and V =

∥ · ∥
(cid:13)
(cid:13)
(cid:13)
(cid:13)

Σ−1E

T
(cid:80)T −1

⊺
ℓ1ℓ
2

t=1 ℓt

t=1 1/t

(cid:80)T

RΠ

(cid:21)

−

point of Equation (5.2) with N = 1, h(x) = x2. Moreover, if Σ is a diagonal matrix, then plugging
this configuration into Equation (E.6), and projecting the output with ProjΠ,∥·∥ would perform
FTRL with an L2-regularizer for the loss vectors (ℓt)t∈[T ].

Proof. We will locally use
interchangeably use ℓi(j) and ℓij for notational convenience. Define a := K ⊺(Qc + qc)

= [d] without losing generality as

is finite with

= d, and will
Rd and

|A|

A

A

∈

63

3348
3349

3350
3351
3352
3353
3354
3355

3356
3357
3358
3359
3360
3361

3362
3363
3364
3365
3366

3367
3368
3369
3370
3371
3372

3373
3374
3375
3376
3377
3378

3379
3380
3381
3382
3383
3384

3385
3386
3387
3388
3389

3390
3391
3392
3393
3394
3395

3396
3397
3398
3399
3400
3401

̸
Under review as a conference paper at ICLR 2025

bt−1 := β111t−1 := k⊺
function (Equation (5.2)) can be written as follows:

c (Qc + qc)111t−1 ∈

Rt−1. With N = 1, h(x) = x2, and the choice of Π, the loss

f (V, a, (bt)t∈[T −1], vc) := E

(cid:32) T

(cid:88)

t=1

⊺
⊺
⊺
t−1)Softmax(ℓ
t (V ℓ1:t−1 + vc111
ℓ

1:t−1a + bt−1) + RΠ∥

(cid:33)2

,

T
(cid:88)

t=1

ℓt∥2

where for t = 1, we use the output of the single-layer self-attention as vc and we will write it as
⊺
⊺
0 )Softmax(ℓ
(V ℓ1:0 + vc111
2. Also, we will define
1:0a + b0) for notational consistency with t
empty sum (cid:80)0
i=1 ai = 0 for any sequence (ai)i∈N+.

≥

Step 1. Calculating ∂f
∂a .

[d], we calculate the corresponding directional derivative with the following equation for

∈

For x
2:
t

≥
∂
∂ax

−

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ
1:t−1a + bt−1)

=

∂
∂ax

t−1
(cid:88)

i=1

⊺
⊺
t (V ℓ1:t−1 + vc111
t−1)ei
ℓ

(cid:80)t−1

i=1 ℓ⊺

t (V ℓ1:t−1 + vc111⊺

t−1)ei exp(e⊺

=

(cid:80)t−1

i (ℓ⊺
exp(e⊺
s=1 exp(e⊺
i (ℓ⊺
((cid:80)t−1

1:t−1a + bt−1))
s(ℓ⊺

1:t−1a + bt−1))
∂e

(cid:80)t−1

i=1 ℓ⊺

t (V ℓ1:t−1 + vc111⊺

t−1)ei exp(e⊺

(cid:80)t−1

s=1 exp(e⊺

s(ℓ⊺

1:t−1a + bt−1))

∂e

⊺
s (ℓ

⊺
1:t−1a+bt−1)

∂ax

(cid:19)

.

s(ℓ⊺

1:t−1a + bt−1))
s=1 exp(e⊺
i (ℓ⊺
((cid:80)t−1

1:t−1a + bt−1))
s(ℓ⊺
s=1 exp(e⊺

1:t−1a + bt−1)))2
(cid:18)

1:t−1a + bt−1)))2

⊺
⊺
i (ℓ
1:t−1a+bt−1)

∂ax

((cid:80)t−1

s=1 exp(e⊺

s(ℓ⊺

1:t−1a + bt−1)))

Plugging a = 000d and vc = 000d, and (bt = β111t)t∈[T −1] provides

∂
∂ax

(cid:12)
(cid:12)
⊺
⊺
⊺
t−1)Softmax(ℓ
t (V ℓ1:t−1 + vc111
ℓ
1:t−1a + bt−1)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

=

t−1
(cid:88)

i=1

⊺
ℓ
t V ℓiℓix
(t

1) −

−

t−1
(cid:88)

i=1

⊺
t V ℓi
ℓ
(t

(cid:16)(cid:80)t−1

(cid:17)

.

s=1 ℓsx
1)2

−

⊺
⊺
⊺
⊺
t−1)Softmax(ℓ
t (V ℓ1:t−1 + vc111
1 vc,
1:t−1a + bt−1) = ℓ
For t = 1, as ℓ
(cid:12)
(cid:12)
⊺
⊺
t−1)Softmax(ℓ
vc111
1:t−1a + bt−1)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

= 0, so we can use the same for-

∂
∂ax

⊺
ℓ
t (V ℓ1:t−1 +

i=1. Using the above calculation, we can further compute

2 with empty sum (cid:80)t−1

mula as t
(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

∂f
∂ax

≥

as follows:

∂f (V, a, (bt)t∈[T −1], vc)
∂ax

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

= E ∂
∂ax

(cid:32) T

(cid:88)

t=1

⊺
⊺
⊺
t−1)Softmax(ℓ
t (V ℓ1:t−1 + vc111
ℓ
1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33)2 (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

(cid:34) (cid:32) T

(cid:88)

= E

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ
1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

(cid:32) T

(cid:88)

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ
1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

(cid:35)

1
t − 1

ℓi + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33) T

(cid:88)

(cid:32)t−1
(cid:88)

t=1

i=1

ℓ⊺
t V ℓiℓix
(t − 1)

−

t−1
(cid:88)

ℓ⊺
t V ℓi

(cid:0)(cid:80)t−1

s=1 ℓsx

(cid:33) (cid:35)

(cid:1)

i=1

(t − 1)2

(E.7)

64

t=1

∂
∂ax

t=1

t−1
(cid:88)

(cid:34) (cid:32) T

(cid:88)

= E

⊺
t V
ℓ

t=1

i=1

= 0,

3402
3403

3404
3405
3406
3407
3408
3409

3410
3411
3412
3413
3414
3415

3416
3417
3418
3419
3420

3421
3422
3423
3424
3425
3426

3427
3428
3429
3430
3431
3432

3433
3434
3435
3436
3437
3438

3439
3440
3441
3442
3443

3444
3445
3446
3447
3448
3449

3450
3451
3452
3453
3454
3455

Under review as a conference paper at ICLR 2025

3456
3457

3458
3459
3460
3461
3462
3463

3464
3465
3466
3467
3468
3469

3470
3471
3472
3473
3474

3475
3476
3477
3478
3479
3480

3481
3482
3483
3484
3485
3486

3487
3488
3489
3490
3491
3492

3493
3494
3495
3496
3497

3498
3499
3500
3501
3502
3503

3504
3505
3506
3507
3508
3509

where we used the fact that ℓi is drawn from a symmetric distribution, and flipping the sign of the
variable as

−

⊺
t V
ℓ

ℓi yields the same distribution, which leads to the following:
⊺
t V ℓi
ℓ
(t

⊺
ℓ
t V ℓiℓix
(t

ℓi + RΠ∥

ℓt∥2

1) −

t−1
(cid:88)

t−1
(cid:88)

t−1
(cid:88)

T
(cid:88)

(cid:33) T

(cid:88)





1

1

t

t=1

t=1

i=1

−

i=1

i=1

−

(cid:34) (cid:32) T

(cid:88)

E

t=1

(cid:16)(cid:80)t−1

s=1 ℓsx
1)2

(cid:17)



(cid:35)



(cid:34) (cid:32) T

(cid:88)

= E

t=1

⊺
t V
ℓ

t−1
(cid:88)

i=1

1

−

t

ℓi + RΠ∥

1

T
(cid:88)

t=1

ℓt∥2

(cid:33) T

(cid:88)

t=1





−

t−1
(cid:88)

i=1

⊺
t V ℓiℓix
ℓ
1)
(t

−

+

t−1
(cid:88)

i=1

This yields Equation (E.7)=0.

Step 2. Calculating ∂f
∂vc

.

We will use the following equation for t

2:

≥
⊺
⊺
t−1)Softmax(ℓ
t (V ℓ1:t−1 + vc111
ℓ

⊺
1:t−1a + bt−1)

∂
∂vc

−
⊺
t V ℓi
ℓ
(t

(cid:16)(cid:80)t−1

s=1 ℓsx
1)2

(cid:17)





(cid:35)
.

−

=

ℓ

i=1

t−1
(cid:88)

∂
∂vc

⊺
⊺
t (V ℓ1:t−1 + vc111
t−1)ei

⊺
⊺
exp(e
1:t−1a + bt−1))
i (ℓ
⊺
⊺
s (ℓ
s=1 exp(e
1:t−1a + bt−1))
⊺
⊺
⊺
0 )Softmax(ℓ
1 (V ℓ1:0 + vc111
1:0a + b0) = ℓ1, so that we can use the same
ℓ
2. Therefore, we can calculate ∂f
∂vc

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

as follows:

(cid:80)t−1

= ℓt.

For t = 1, we define ∂
∂vc

formula as t

≥

∂f (V, a, (bt)t∈[T −1], vc)
∂vc

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

= E ∂
∂vc

(cid:32) T

(cid:88)

t=1

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ
1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33)2 (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

(cid:34) (cid:32) T

(cid:88)

= E

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ
1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

t=1

∂
∂vc

(cid:32) T

(cid:88)

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ
1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

(cid:35)

(cid:34) (cid:32) T

(cid:88)

= E

⊺
t V
ℓ

t=1

t−1
(cid:88)

t=2

i=1

1
t − 1

ℓi + RΠ∥

T
(cid:88)

t=1

ℓt∥2

(cid:33) T

(cid:88)

t=1

(cid:35)

ℓt

= 0.

The last line is due to the same reason as the last part of Step 1.

Step 3. Calculating ∂f
∂V .

We calculate the following equation, which will be used to calculate ∂f
∂V

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

for t
∂
∂V

2:

≥
⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ
1:t−1a + bt−1)

t−1
(cid:88)

i=1

⊺
ℓtℓ
i

=

=

∂
∂V

t−1
(cid:88)

i=1

⊺
⊺
t (V ℓ1:t−1 + vc111
t−1)ei
ℓ

(cid:80)t−1

⊺
⊺
1:t−1a + bt−1))
i (ℓ
exp(e
⊺
⊺
1:t−1a + bt−1))
s (ℓ
s=1 exp(e

(cid:80)t−1

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]
⊺
⊺
1:t−1a + bt−1))
exp(e
i (ℓ
⊺
⊺
s=1 exp(e
1:t−1a + bt−1))
s (ℓ
(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

=

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

t−1
(cid:88)

i=1

⊺
i .
ℓtℓ

1

−

t

1

2 with empty sum

≥

For t = 1, note that ∂
(cid:80)t−1
i=1.

⊺
t vc = OOOd×d, so we can use the same formula as t
∂V ℓ

65

Under review as a conference paper at ICLR 2025

Therefore, we have

∂f (V, a, (bt)t∈[T −1], vc)
∂V

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

(cid:33)2 (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

ℓt∥2
(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

T
(cid:88)

t=1

ℓt∥2

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1]

(cid:35)

(cid:32) T

(cid:88)

t=1

= E ∂
∂V
(cid:34) (cid:32) T

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ

1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

= E

(cid:88)

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ

1:t−1a + bt−1) + RΠ∥

T
(cid:88)

t=1

ℓt∥2

t=1

∂
∂V

(cid:32) T

(cid:88)

t=1

(cid:34) (cid:32) T

(cid:88)

= E

t=1
(cid:34) (cid:32) T

(cid:88)

= E

t=1
(cid:34) (cid:32) T

(cid:88)

= E

⊺
t V
ℓ

t−1
(cid:88)

i=1
t−1
(cid:88)

t−1
(cid:88)

i=1
(cid:18) 1
t

1

−
d
(cid:88)

d
(cid:88)

t=1

i=1

x=1

y=1

⊺
t (V ℓ1:t−1 + vc111
ℓ

⊺
⊺
t−1)Softmax(ℓ

1:t−1a + bt−1) + RΠ∥
(cid:35)

T
(cid:88)

t=1

1

−

1

t

ℓi + RΠ∥
(cid:19) (cid:18) 1
t

⊺
ℓ
t V ℓi

ℓt∥2
(cid:19)

⊺
ℓtℓ
i

(cid:33) T

(cid:88)

t−1
(cid:88)

1

t=1

i=1

+ RΠT

1

t

−
T
(cid:88)

∥

t′=1

⊺
ℓtℓ
i

(cid:33) (cid:35)

ℓt′

⊺
i

∥2ℓtℓ

1

−
(cid:18) 1
t

−

vxyℓtxℓiy

(cid:19)2

1

[ℓtzℓiw](z,w) + RΠT

(cid:33) (cid:35)

T
(cid:88)

∥

t′=1

ℓt′

⊺
i

∥2ℓtℓ

=

=

T
(cid:88)

t−1
(cid:88)

d
(cid:88)

i=1

t=1
(cid:32)T −1
(cid:88)

x=1
(cid:33)

1
t

t=1

1

(t

d
(cid:88)

(cid:34)
1)2 [σxzvxyσyw](z,w) + E
(cid:34)
ΣV Σ + E

T
(cid:88)

y=1

RΠT

ℓt′

−

(cid:35)

.

⊺
i

∥2ℓtℓ

RΠT

(cid:35)

T
(cid:88)

∥

t′=1

ℓt′

⊺
i

∥2ℓtℓ

∥

t′=1

=

Therefore,

if

V ⋆

∂f
∂V

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1],V =V ⋆

RΠ

T
(cid:80)T −1

t=1 1/t

(cid:34)
Σ−1E

∥

= Od×d. Lastly, we have

(cid:80)T

t=1 ℓt∥2ℓtℓ

(cid:35)

Σ−1,

then

⊺
i

∂f
∂K

∂f
∂Q

∂f
∂qc

(cid:12)
(cid:12)K⊺(Qc+qc)=vc=000d,V =V ⋆ =

(cid:12)
(cid:12)K⊺(Qc+qc)=vc=000d,V =V ⋆ =

(cid:12)
(cid:12)K⊺(Qc+qc)=vc=000d,V =V ⋆ =

(cid:18) ∂f
∂a
(cid:18) ∂f
∂a
(cid:18) ∂f
∂a

∂a
∂K

∂a
∂Q

∂a
∂qc

(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1],V =V ⋆
(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1],V =V ⋆
(cid:19) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=000d,(bt=β111t)t∈[T −1],V =V ⋆

= OOOd×d

= OOOd×d

= 000d

which means that such configurations are first-order stationary points of Equation (5.2) with N = 1,
h(x) = x2, and Π = B(0, RΠ,

).

∥ · ∥

E.7 DEFERRED PROOF OF THEOREM 5.2

Theorem 5.2. Consider the policy space Π = B(0, RΠ,
) for some RΠ > 0. The configu-
ration of a single-layer linear self-attention model in Equation (5.3) (V, K, Q, vc, kc, qc) such that
(cid:16)
K ⊺(Qc + qc) = vc = 000d and V =
Σ−1 is a global optimal solu-
tion of Equation (5.2) with N = 1, h(x) = x2. Moreover, every global optimal configuration of
Equation (5.2) within the parameterization class of Equation (5.3) has the same output function g.
Additionally, if Σ is a diagonal matrix, then plugging any global optimal configuration into Equa-
tion (5.3), and projecting the output with ProjΠ,∥·∥ is equivalent to FTRL with an L2-regularizer.

2RΠΣ−1E

t=1 ℓt∥

⊺
ℓ1ℓ
2

∥ · ∥

(cid:80)T

−

(cid:17)

∥

66

3510
3511

3512
3513
3514
3515
3516
3517

3518
3519
3520
3521
3522
3523

3524
3525
3526
3527
3528

3529
3530
3531
3532
3533
3534

3535
3536
3537
3538
3539
3540

3541
3542
3543
3544
3545
3546

3547
3548
3549
3550
3551

3552
3553
3554
3555
3556
3557

3558
3559
3560
3561
3562
3563

Under review as a conference paper at ICLR 2025

This theorem involves the analysis of a non-convex optimization problem through stationary point
analysis. We identified the set of stationary points. By constructing the optimization problem as
shown in Equation (E.13), we significantly reduced the candidate set for optimal points using our
novel argument on the expected value of a nonnegative definite matrix. The main challenge here
was to address the global optimization problem in a non-convex setting, which required some good
understanding of the Transformer architecture.

Proof. The output of the single-layer linear self-attention structure is as follows:

g(Zt; V, K, Q, vc, kc, qc)

(V ℓiℓ

⊺

⊺
i (K

(Qc + qc)) + (V k

⊺
c (Qc + qc) + vc(Qc + qc)

⊺

K) ℓi + vck

⊺
c (Qc + qc)) ,

=

t
(cid:88)

i=1

(E.8)

(E.9)

which can be expressed with a larger class

g(Zt, A, β, C, δ) :=

t
(cid:88)

⊺
(Aℓiℓ

i β + Cℓi + δ),

where A

∈

Rd×d, β, C, δ

∈

Rd. Then, if a minimizer of

i=1

f (A, β, C, δ) : = E

(cid:32) T

(cid:88)

t=1

t−1
(cid:88)

i=1

ℓt,
⟨

⊺
(Aℓiℓ

i β + Cℓi + δ)

inf
π∈Π

⟩ −

(cid:43)(cid:33)2

(cid:42) T

(cid:88)

t=1

ℓt, π

c (Qc + qc) + vc(Qc + qc)⊺K, β =
c (Qc + qc), then we can conclude that the corresponding V, Q, K, vc, qc, kc are also a minimizer

can be expressed as A = V, β = K ⊺(Qc + qc), C = V k⊺
vck⊺
of

E

(cid:32) T

(cid:88)
ℓt, g(Zt−1)
⟨

t=1

inf
π∈Π

⟩ −

(cid:43)(cid:33)2

ℓi, π

,

(cid:42) T

(cid:88)

t=1

since the corresponding V, Q, K, vc, qc, kc constitute a minimizer among a larger class. Now, since
Π = B(000d, RΠ,

), we can rewrite f as

∥ · ∥

f (A, β, C, δ) = E

(cid:32) T

(cid:88)

t=1

t−1
(cid:88)

i=1

ℓt,
⟨

⊺
(Aℓiℓ

i β + Cℓi + δ)

+ RΠ

⟩

(cid:13)
(cid:13)
(cid:13)
(cid:13)

T
(cid:88)

t=1

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

ℓi

(cid:33)2

.

(E.10)

Step 1. Finding condition for ∂f
Due to the Leibniz rule, if we calculate the partial derivative of Equation (E.10) w.r.t. δ, we have

∂δ = 0.

∂f (A, β, C, δ)
∂δ

=

E

∂
∂δ

(cid:32) T

(cid:88)

t=1

t−1
(cid:88)

i=1

ℓt,
⟨

⊺
(Aℓiℓ

i β + Cℓi + δ)
⟩

(cid:33)2

ℓt∥2

T
(cid:88)

t=1

(cid:32) T

(cid:88)

ℓt,
⟨
t=1
(cid:32) T

(cid:88)

t−1
(cid:88)

i=1
t−1
(cid:88)

= E ∂
∂δ

T
(cid:88)

= E

ℓt

t=1

t=1

i=1

⊺
(Aℓiℓ

i β + Cℓi + δ)

+ RΠ∥

⟩

(t

⊺
1)ℓ

t (Aℓiℓ

⊺

i β + Cℓi + δ) + RΠ∥

−

(cid:33)

ℓt∥

.

T
(cid:88)

t=1

(E.11)

+ RΠ∥
(cid:33)2

T
(cid:88)

t=1

ℓt∥2

Since the expectation of either odd-order polynomial or even-order polynomial times
to that ℓt follows a symmetric distribution, we have

∥ · ∥2 is 0, due

T
(cid:88)

(t

E

t=1

−

1)ℓtRΠ

(cid:13)
(cid:13)
(cid:13)

T
(cid:88)

t=1

ℓt

(cid:13)
(cid:13)
(cid:13)2

= 0,

T
(cid:88)

(t

E

t=1

1)ℓt

−

T
(cid:88)

t−1
(cid:88)

t=1

i=1

⊺
ℓ
t

Cℓi = 0.

67

3564
3565

3566
3567
3568
3569
3570
3571

3572
3573
3574
3575
3576
3577

3578
3579
3580
3581
3582

3583
3584
3585
3586
3587
3588

3589
3590
3591
3592
3593
3594

3595
3596
3597
3598
3599
3600

3601
3602
3603
3604
3605

3606
3607
3608
3609
3610
3611

3612
3613
3614
3615
3616
3617

Under review as a conference paper at ICLR 2025

Now, we calculate
T
(cid:88)

T
(cid:88)

E

(t

1)ℓt

−

t=1

t−1
(cid:88)

⊺
ℓ
t

Aℓiℓ

⊺

i β = E

T
(cid:88)

T
(cid:88)

t−1
(cid:88)

t=1

i=1

t1=1

t=1

i=1

⊺
1)ℓt1 ℓ
t

⊺
Aℓiℓ
i β

(t1 −

T
(cid:88)

t−1
(cid:88)

(t

⊺
1)ℓtℓ
t

E

=
(i)

⊺
Aℓiℓ

i β = E

T
(cid:88)

(t

t=1

−

1)2ℓtℓ

⊺
t

AΣβ =

1
6

T (2T 2

−

3T + 1)ΣAΣβ,

= t, due to the independence of ℓt, ℓt1, we can use Eℓt = 0. Lastly,

i=1

t=1

−
where (i) holds since if t1 ̸
t−1
(cid:88)
(t

T
(cid:88)

T
(cid:88)

1)ℓt

E

−

t=1

T
(cid:88)

⊺
t δ = E
ℓ

T
(cid:88)

t=1

i=1

t1=1

t=1

(t1 −

⊺
t δ =
1)ℓt1ℓ

1)(t

−

1
6

T (2T 2

−

3T + 1)Σδ.

Plugging the above equations into Equation (E.11), we have
∂f (A, β, C, δ)
∂δ

T (2T 2

1
6

=

−

3T + 1)(ΣAΣβ + Σδ).

Due to the optimality condition, we have

AΣβ + δ = 0.

(E.12)

Step 2. Plugging the optimality condition for ∂f
Plugging Equation (E.12) to Equation (E.10), f can be written as
(cid:32) T

∂δ into Equation (E.10).

f (A,β, C,

AΣβ) = E

−

(cid:88)

t−1
(cid:88)

t=1

i=1

⊺
t (A(ℓiℓ
ℓ

⊺
i −

Σ)β + Cℓi) + RΠ

(cid:33)2

ℓt

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(cid:13)
(cid:13)
(cid:13)
(cid:13)

T
(cid:88)

t=1
(cid:32)

+ E

RΠ

(cid:33)2

(cid:13)
(cid:13)
(cid:13)
(cid:13)

T
(cid:88)

t=1

ℓt

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(cid:32) T

(cid:88)

t−1
(cid:88)

t=1

i=1

= E

(cid:124)

⊺
ℓ
t

⊺
A(ℓiℓ
i −

(cid:123)(cid:122)
(i)

(cid:33)2

Σ)β

+E

(cid:32) T

(cid:88)

t−1
(cid:88)

⊺
ℓ
t

Cℓi

(cid:33)2

t=1

i=1

(cid:125)

(cid:32) T

(cid:88)

t−1
(cid:88)

t=1

i=1

+ 2E

(cid:124)

⊺
ℓ
t

Σ)β

⊺
A(ℓiℓ
i −
(cid:123)(cid:122)
(ii)

(cid:32) T

(cid:88)

t−1
(cid:88)

⊺
ℓ
t

t=1

i=1

+ 2E

(cid:124)

+ 2E

(cid:32) T

(cid:88)

t−1
(cid:88)

t=1

i=1

Σ)β

⊺
A(ℓiℓ
i −
(cid:123)(cid:122)
(iii)
(cid:33) (cid:32)

⊺
ℓ
t

Cℓi

RΠ

(cid:33) (cid:32) T

(cid:88)

t−1
(cid:88)

(cid:33)

⊺
ℓ
t

Cℓi

t=1

i=1

(cid:33) (cid:32)

RΠ

(cid:13)
(cid:13)
(cid:13)
(cid:13)

T
(cid:88)

t=1

ℓt

(cid:125)

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(cid:33)

(cid:125)

(cid:13)
(cid:13)
(cid:13)
(cid:13)

T
(cid:88)

t=1

ℓt

(cid:13)
(cid:13)
(cid:13)
(cid:13)2

(cid:33)

.

For the part (i), we have

(cid:32) T

(cid:88)

t−1
(cid:88)

E

t=1

i=1

⊺
ℓ
t

⊺
A(ℓiℓ
i −

Σ)β

(cid:33)2

= E

(cid:34) T

(cid:88)

t1−1
(cid:88)

T
(cid:88)

t−1
(cid:88)

⊺

β

t1=1

i1=1

t=1

i=1

Σ)A⊺

⊺
ℓt1ℓ
t

⊺
A(ℓiℓ
i −

Σ)β

(cid:35)

⊺
(ℓi1 ℓ
i1 −
(cid:35)

Σ)β

⊺
i −
(cid:35)

⊺

β

⊺
(ℓi1ℓ
i1 −

Σ)A⊺

⊺
ℓtℓ
t

A(ℓiℓ

⊺

β

⊺
(ℓiℓ
i −

Σ)A⊺

⊺
ℓiℓ
i

⊺
A(ℓiℓ
i −

Σ)β

Σ)A⊺

⊺
⊺E [(ℓiℓ
i −
(cid:104)
(√ΣA(ℓiℓ

⊺E

Σ)] β

⊺
ΣA(ℓiℓ
i −
⊺
(√ΣA(ℓiℓ
i −

Σ))

⊺

⊺
i −

(E.13)

(cid:105)

β.

Σ))

68

E

=
(1)

E

=
(2)

(T

(T

=

=

(cid:34) T

(cid:88)

t−1
(cid:88)

t−1
(cid:88)

i1=1

i=1

t=1
(cid:34) T

(cid:88)

t−1
(cid:88)

t=1

i=1

1)T

1)T

β

β

−
2

−
2

3618
3619

3620
3621
3622
3623
3624
3625

3626
3627
3628
3629
3630
3631

3632
3633
3634
3635
3636

3637
3638
3639
3640
3641
3642

3643
3644
3645
3646
3647
3648

3649
3650
3651
3652
3653
3654

3655
3656
3657
3658
3659

3660
3661
3662
3663
3664
3665

3666
3667
3668
3669
3670
3671

Under review as a conference paper at ICLR 2025

−

⊺
i1 −

Σ)β = 000d) = 1.

= i, we can calculate E(ℓi1ℓ

Σ)β = 000d for any ℓi such that p(ℓi)

= t, we know that Eℓt1 = Eℓt = 0, and they are independent, and (2)
Σ) = Od×d. In addition, we can easily check
Z. Note that

Here, (1) holds because if t1 ̸
holds because if i1 ̸
that (ii) and (iii) are 0 as they are polynomials of odd degrees and we have Z d=
⊺
Equation (E.13) is minimized when P(√ΣA(ℓiℓ
i −
If A
= Od×d, suppose that the singular value decomposition of A = U ΛV yields that Λ is a
diagonal matrix whose first diagonal element is non-zero, and U, V are orthogonal matrices. Then,
⊺
we want to find β that √ΣU ΛV (ℓiℓ
= 0, where p indicates
i −
the probability density function of loss vectors. Since Σ and U are invertible, we only need to
⊺
Σ)β = 000d. Since Λ’s first diagonal component is non-zero, we will consider
consider ΛV (ℓiℓ
i −
⊺
⊺
⊺
equation e
Σ)β = 0, where V1 is the first row
Σ)β = 0. This is equivalent to V1(ℓiℓ
1 ΛV (ℓiℓ
i −
i −
of V , and is a non-zero vector.
Now, we will generally consider ax,y(v) := vv⊺x
∥ ·
x)I, and we
)
∥
= 000d. Therefore, the
can find that the determinant of the Jacobian is nonzero when v = ϵ1x if x
⊺
Σ)) for
volume of (V1(ℓiℓ
i −
⊺
v
supp(Z), and ϵ1, ϵ2 > 0
1 , ϵ2,
B(ϵ1V
∥ · ∥
∈
satisfy that ϵ1|
+ ϵ2 < cz. Here, we define ϵ2 > 0 sufficiently small so that the determinant
V1|
⊺
⊺
of Jacobian(vv⊺V
ΣV
1 is a one-
1 −
to-one correspondence, by inverse function theorem. Therefore, the volume of (V1(vv⊺
Σ)) for
v

Rd function. Then, we can check that the Jacobian of ax,y(v) is vx⊺ + (v

Σ)) for ℓi ∈
), where cz is a constant such that B(000d, cz,

) is greater than the volume of (V1(vv⊺

∈
) can be calculated as

Rd and ax,y : B(000d, 2ϵ1,

)
∥ · ∥
), and v

⊺
1 ) > 0 for v

y where x, y, v

B(000d, cz,

⊺
1 −

⊺
1 , ϵ2,

B(ϵ1V

vv⊺V

∥ · ∥

∥ · ∥

ΣV

B(ϵ1V

→

→

−

⊆

−

−

∈

·

⊺
1 , ϵ2,

∈

∥ · ∥

[Volume (V1(vv

⊺

−

Σ)) for v

∈

B(ϵ1V

⊺
1 , ϵ2,

)] =

∥ · ∥

(cid:90)

v∈B(ϵ1V

⊺
1 ,ϵ2,∥·∥)

(cid:12)
(cid:12)det(Jacobian(V1(vv

⊺

Σ)))(cid:12)

(cid:12)dv > 0.

−

Therefore, Volume(V1(vv⊺
) is non-zero, so that we can find d
∥ · ∥
loss vectors
}i∈[d] are linearly independent. Hence, if
we want to minimize Equation (E.13), either A = Od×d or β = 000d should hold. In both cases,
Equation (E.9) can be re-written as

Σ)) where v
−
ℓi}i∈[d] such that the vectors

B(ϵ1V
⊺
∈
V1(ℓiℓ
i −
{

⊺
1 , ϵ2,
Σ)

{

g(Zt; A, β, C, δ) :=

t
(cid:88)

i=1

Cℓi,

and this is covered by the original parametrization (Equation (E.8)) with K ⊺(Qc + qc) = vc = 000d.

Step 3. Calculating ∂f
∂C .
Now, we optimize over C, by minimizing the following objective:

f (C) : = E

= E

(cid:124)

(cid:32) T

(cid:88)

t−1
(cid:88)

t=1
(cid:32) T

(cid:88)

i=1

t−1
(cid:88)

⊺
t

ℓ

⊺
ℓ
t

t=1

i=1
(cid:123)(cid:122)
(i)

Cℓi + RΠ∥
(cid:33)2

Cℓi

+2E

(cid:125)

(cid:33)2

T
(cid:88)

ℓt∥
t=1
(cid:32)(cid:32) T

(cid:88)

(cid:33)

t−1
(cid:88)

⊺
ℓ
t

Cℓi

(cid:33)
ℓt∥

T
(cid:88)

t=1

RΠ∥

+ E

(cid:32)
RΠ∥

T
(cid:88)

t=1

(cid:33)2

ℓt∥

t=1

i=1

T (T

=

1)

−
2

Tr (C⊺

ΣCΣ) + 2E



B

T
(cid:88)

t−1
(cid:88)

t=1

i=1

⊺
ℓ
t

Cℓi∥

T
(cid:88)

j=1


 + E

ℓj∥

(cid:32)
RΠ∥

T
(cid:88)

t=1

(cid:33)2

ℓt∥

.

69

3672
3673

3674
3675
3676
3677
3678
3679

3680
3681
3682
3683
3684
3685

3686
3687
3688
3689
3690

3691
3692
3693
3694
3695
3696

3697
3698
3699
3700
3701
3702

3703
3704
3705
3706
3707
3708

3709
3710
3711
3712
3713

3714
3715
3716
3717
3718
3719

3720
3721
3722
3723
3724
3725

̸
̸
̸
Under review as a conference paper at ICLR 2025

Here, (i) can be calculated as follows:

(cid:32) T

(cid:88)

t−1
(cid:88)

E

(cid:33)2

⊺
ℓ
t

Cℓi

= E

(cid:32) T

(cid:88)

t1−1
(cid:88)

T
(cid:88)

t−1
(cid:88)

(cid:33)

⊺
i1

ℓ

C⊺

⊺
ℓt1ℓ
i

Cℓi

i=1
(cid:32) T

(cid:88)

t=1
(cid:32) T

(cid:88)

t=1

E

=
(1)

E

=
(2)

t=1

i=1

i−1
(cid:88)

t−1
(cid:88)

i1=1

i=1

⊺
ℓ
i1

C⊺

⊺
ℓiℓ
i

t−1
(cid:88)

⊺
ℓ
k

C⊺

ΣCℓi

(cid:33)

t1=1

t=1

i=1

i1=1
(cid:33)

Cℓi

= E

(cid:32) T

(cid:88)

i−1
(cid:88)

t−1
(cid:88)

(cid:33)

⊺
ℓ
i1

C⊺

ΣCℓi

t=1

i1=1

i=1

(cid:32) T

(cid:88)

t−1
(cid:88)

t=1

i=1

(cid:33)

C⊺

⊺
ΣCℓiℓ
k

T (T

=

1)

−
2

Tr (C⊺

ΣCΣ) ,

E Tr

=
(3)

= t, we already know that Eℓt = Eℓt1 = 0, (2) holds due to a similar

since (1) holds because if t1 ̸
reason, and (3) comes from Tr(AB) = Tr(BA).
We calculate ∂f (C)
∂C :

∂f (C)
∂C = T (T

−

1)ΣCΣ + 2RΠE




∥

T
(cid:88)

j=1

ℓj∥

T
(cid:88)

t−1
(cid:88)



⊺
ℓtℓ
i

 .

t=1

i=1
(cid:17)

(cid:80)t−1

⊺
i=1 ℓtℓ
i

Σ−1.

Hence, the optimal C =

2RΠ

T (T −1) Σ−1E

−

(cid:16)

∥

(cid:80)T

j=1 ℓj∥

(cid:80)T

t=1

Now, we see that for the special case of Σ = I, we have C =
calculate the (a, b)-coordinate of C, we need to calculate

RΠE

−

(cid:16)

∥

(cid:118)
(cid:117)
(cid:117)
(cid:116)





d
(cid:88)

T
(cid:88)
(



ℓso)2ℓiaℓkb

 .

Eℓ

o=1

s=1

(cid:80)T

j=1 ℓj∥

⊺
ℓtℓ
i

(cid:17)

. If we

= b, then since Z is symmetric, the term above becomes zero. Therefore, we only need to

(cid:21)

s=1 ℓso)2ℓiaℓka

, and it will be the same

If a
consider the case when a = b, which is Eℓ

(cid:20)(cid:113)(cid:80)d

o=1((cid:80)T
[d] since ℓi’s coordinates are independent.

value for all a

∈

Now, we calculate the scale of Eℓ
(cid:80)d−1

o=1 ((cid:80)T

s=1 ℓso)2

T (d−1)

1 as d
s̸=i,k ℓs1/√T which is independent of ℓi1 and ℓk1.
(cid:118)

(cid:117)
(cid:117)
(cid:116)


 = EZ,W,ℓi1,ℓk1

ℓso)2ℓi1ℓk1

d
(cid:88)
(

→ ∞

T
(cid:88)

(cid:20)(cid:113)



(cid:80)

Eℓ

a.s.
→

(cid:20)(cid:113)(cid:80)d

o=1((cid:80)T

s=1 ℓso)2ℓi1ℓk1

(cid:21)
.

We have Z

:=

(by the law of large numbers) and we define W :=

T (d − 1)Z + (

√

T W + ℓi1 + ℓk1)2ℓi1ℓk1

(cid:21)

o=1

s=1

= EZ,W,ℓi1,ℓk1≥0

(cid:20)(cid:113)

T (d − 1)Z + (

√

T W + ℓi1 + ℓk1)2ℓi1ℓk1 −

(cid:113)

T (d − 1)Z + (

√

T W + ℓi1 − ℓk1)2ℓi1ℓk1

(cid:21)



= EZ,W,ℓi1,ℓk1≥0



(cid:113)

T (d − 1)Z + (

√

√

4(

T W + ℓi1)ℓk1
(cid:113)

T W + ℓi1 + ℓk1)2 +

T (d − 1)Z + (



ℓi1ℓk1

 .

√

T W + ℓi1 − ℓk1)2

, we have

Taking d
(cid:113)

→ ∞
T (d

−

1)Z + (√T W + ℓi1 + ℓk1)2 +

(cid:113)

2√T d

T (d

1)Z + (√T W + ℓi1 −

−

ℓk1)2

which further implies

4(√T W + ℓi1)ℓk1

(cid:113)

1)Z + (√T W + ℓi1 + ℓk1)2 +

T (d

1)Z + (√T W + ℓi1 −

−

ℓk1)2

√T d

(cid:113)

T (d

−

√T d

d
→

4(√T W + ℓi1)ℓk1
2√T d

d
→

1,

ℓi1ℓk1

ℓi1ℓk1 = 2(√T W + ℓi1)ℓi1ℓk1

70

3726
3727

3728
3729
3730
3731
3732
3733

3734
3735
3736
3737
3738
3739

3740
3741
3742
3743
3744

3745
3746
3747
3748
3749
3750

3751
3752
3753
3754
3755
3756

3757
3758
3759
3760
3761
3762

3763
3764
3765
3766
3767

3768
3769
3770
3771
3772
3773

3774
3775
3776
3777
3778
3779

̸
Under review as a conference paper at ICLR 2025

as d

lim
d→∞

→ ∞

. Therefore,

√T d

EZ,W,ℓi1,ℓk1≥0

(cid:113)

T (d

−

1)Z + (√T W + ℓi1 + ℓk1)2 +

T (d

1)Z + (√T W + ℓi1 −

−

ℓk1)2

4(√T W + ℓi1)ℓk1

(cid:113)



ℓi1ℓk1



= EZ,W,ℓi1,ℓk1≥0

(cid:104)

2(√T W + ℓi1)ℓi1ℓk1

(cid:105)

= Eℓi1,ℓk1≥0

(cid:2)ℓ2

i1ℓk1

(cid:3)

which is a constant. The last equality came from the fact that W , ℓi1, ℓk1 are independent random
variables, and expectation of ℓi1 is zero. Therefore, the output of the single-layer linear self-attention
provides us with online gradient descent with step-size Θ(RΠ/√T d). In the online learning liter-
ature, we usually set the gradient step size as Θ(RΠ/√T d) (Hazan, 2016, Theorem 3.1), which is
consistent with the result above.

E.8 EMPIRICAL VALIDATION OF THEOREM E.2 AND THEOREM 5.2

We now provide empirical validations for Theorem E.2 and Theorem 5.2. We provide the training
details and the results as follows.

E.8.1 EMPIRICAL VALIDATION OF THEOREM E.2

Our model architecture is defined as follows: the number of layers T is set to 30 and the dimen-
sionality d to 32, with the loss vector ℓi’s distribution Z following a standard normal distribution
(0, 1). During training, we conducted 40,000 epochs with a batch size of 512. We employed the
N
Adam optimizer, setting the learning rate to 0.001. We initialized the value, query, and key vectors
(vc, qc, kc) as zero vectors.

Our empirical analysis aims to demonstrate that the optimized model inherently emulates online
gradient descent. To illustrate this, we will focus on two key convergence properties: K ⊺Q ap-
⊺
proaching the zero matrix OOOd×d and V converging to a111d111
d + bId×d, where a and b are con-
⊺
stants in R. The conditions K ⊺Q = OOOd×d and V = a111d111
d + bId×d imply that the function
g(Zt; V, Q, K) = (cid:80)t
a)ℓi, effectively emulating the process of an online gradient descent
method. We repeated 10 times of the experiments. For verifying K ⊺Q = OOOd×d, we will measure
⊺
∥ · ∥F ) of K ⊺Q. Also for measuring the closeness of V and a111d111
d + bId×d, we
Frobenius norm (
will measure mina,b∈R
∥F /b. The results are demonstrated in the first plot of
Figure E.1.

⊺
d + bId×d)

i=1(b

(a111d111

V
∥

−

−

⊺

E.8.2 EMPIRICAL VALIDATION OF THEOREM 5.2
We now focus on two key convergence properties: K ⊺(Q111d +qc) approaching the zero vector 000d and
d + bId×d, where a and b are constants in R. The conditions K ⊺(Q111d + qc) =
V converging to a111d111
⊺
000d and V = a111d111
a)ℓi, effectively
emulating the process of an online gradient descent method. We repeated 10 times. For verifying
K ⊺(Q111d + qc) = 000d, we will measure 2-norm of K ⊺(Q111d + qc). Also for measuring the closeness
⊺
of V and a111d111
∥F /b. The results are
d + bId×d, we will measure mina,b∈R
demonstrated in the second plot of Figure E.1.

d + bId×d imply that the function g(Zt; V, Q, K) = (cid:80)t

⊺
(a111d111
d + bId×d)

i=1(b

V
∥

−

−

E.9 DISCUSSIONS ON THE PRODUCTION OF FTRL WITH ENTROPY REGULARIZATION

Now, we will consider projecting a single-layer linear self-attention model into a constrained domain
such as a simplex, which is more amenable to the Experts Problem setting. To this end, we con-
sider the following parameterization by adding an additional non-linear structure for the single-layer
linear self-attention:

g(Zt; V, K, Q, vc, kc, qc) = Operator

(cid:32) t

(cid:88)

i=1

(V ℓi + vc)((Kℓi + kc))

⊺

·

(cid:33)

(Qc + qc))

,

(E.14)

where the Operator denotes projection to the convex set.

71

3780

3781
3782
3783
3784
3785

3786
3787
3788
3789
3790
3791

3792
3793
3794
3795
3796
3797

3798
3799
3800
3801
3802
3803

3804
3805
3806
3807
3808

3809
3810
3811
3812
3813
3814

3815
3816
3817
3818
3819
3820

3821
3822
3823
3824
3825
3826

3827
3828
3829
3830
3831

3832
3833

Under review as a conference paper at ICLR 2025

Figure E.1: Empirical validation of Theorem E.2 (top), Theorem 5.2 (middle), and Conjecture 3
(bottom). The observed convergence in Theorem E.2 and Conjecture 3’s result suggests that config-
uration in Theorem E.2 and Conjecture 3 are not only the local optimal point, but it has the potential
as being the global optimizer.

(cid:17)

(cid:16)

1√

Conjecture 3. Assume Σ = I. Then, the configuration that K ⊺(Qc + qc) = vc = 000d and V =
Id×d is a first-order stationary point of Equation (5.2) with N = 1 and h(x) = x2 when
(cid:101)Ω
LLMθ is parameterized with Equation (E.14), Operator = Softmax, and Π = ∆(
). This
configuration performs FTRL with an entropy regularizer which is a no-regret algorithm.

A

−

nd

We provide an idea for proving the conjecture, together with its numerical validation. Also, we
have observed in Figure E.1 that Theorem E.2 and Conjecture 3 might also be a global optimizer, as
training results have provided the configuration that Theorem E.2 and Conjecture 3 have suggested.

To be specific, we will consider

f (V, a, β, vc) = E





T
(cid:88)

d
(cid:88)

ℓts

t=1

s=1

(cid:16)

exp

(cid:80)d

y=1 exp

(cid:80)t−1

e⊺
s
(cid:16)
e⊺
y

j=1(V ℓjℓ⊺
(cid:80)t−1

j=1(V ℓjℓ⊺

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17)

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17) − min

s

2



ℓts



T
(cid:88)

t=1

and will try to prove that a = 000d, vc = v111d, V = kI is a first-order stationary point.

Step 1. Calculating ∂f
∂vc

.

We use the following formula: for x

[d] and t

∈

≥

2, we have

∂
∂vcx

(cid:32)
e

⊺
y

exp

t
(cid:88)

(V ℓiℓ

i=1

⊺
⊺
i a + (βV + vca

)ℓi + vcβ)

t
(cid:88)

⊺
⊺
(V ℓiℓ
i a + (βV + vca

(cid:33)

)ℓi + vcβ)

i=1

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI
t
(cid:88)

(cid:32)
e

⊺
y

∂
∂vcx

i=1

(cid:32)
⊺
e
y

(cid:32)
⊺
e
y

= exp

= exp

t
(cid:88)

⊺
⊺
(V ℓiℓ
i a + (βV + vca

)ℓi + vcβ)

i=1

(cid:33) t

(cid:88)

i=1

⊺
(a

⊺
i ex + β)
ℓiℓ

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(V ℓiℓ

⊺
⊺
i a + (βV + vca

)ℓi + vcβ)

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

= tβ exp(vβ) exp(βk

t
(cid:88)

i=1

ℓiy),

72

3834
3835
3836
3837

3838
3839
3840
3841
3842
3843

3844
3845
3846
3847
3848

3849
3850
3851
3852
3853
3854

3855
3856
3857
3858
3859
3860

3861
3862
3863
3864
3865
3866

3867
3868
3869
3870
3871

3872
3873
3874
3875
3876
3877

3878
3879
3880
3881
3882
3883

3884
3885
3886
3887

010000200003000040000Epochs0200400600800mina,b,x||(V−(a11>+bI)))/b||FEmpiricalvalidationofTheorem3DistancebetweenVandIdentityMatrixDistancebetweenKQandzeromatrix2.55.07.510.012.5||K>Q||F010000200003000040000Epochs0200400600800mina,b,x||(V−(a11>+bI)))/b||FEmpiricalvalidationofTheorem4DistancebetweenVandIdentityMatrixDistancebetweenK>(Q1+qc)and0vector0246810||K>(Q1+qc)||010000200003000040000Epochs0400800120016002000mina,b,x||(V−(a11>+bI)))/b||FEmpiricalvalidationofConjecture6DistancebetweenVandIdentityMatrixDistancebetweenK>(Q1+qc)and0vector2.55.07.510.012.5||K>(Q1+qc)||Under review as a conference paper at ICLR 2025

and for t = 1,

∂
∂vcx

exp

(cid:16)

e⊺
y

(cid:80)t

⊺
i=1(V ℓiℓ

i a + (βV + vca⊺)ℓi + vcβ)

we can use the same formula with t

2. Thus, we have

(cid:17) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

= 0, so

(cid:16)

exp

(cid:80)t−1

≥
j=1(V ℓjℓ⊺
(cid:80)t−1

e⊺
s
(cid:16)
e⊺
y

(cid:80)d

y=1 exp

j=1(V ℓjℓ⊺

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17)
j a + (βV + vca⊺)ℓj + vcβ)

(cid:17) − min

s



ℓts



T
(cid:88)

t=1

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

∂
∂vcx





T
(cid:88)

d
(cid:88)

t=1

s=1

ℓts

= β exp(vβ)

(cid:80)d

y=1 exp

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)

exp

T
(cid:88)

t

d
(cid:88)

t=1

s=1

ℓts

= 0.

Therefore,

(cid:17)

− (cid:80)d

y=1 exp

(cid:16)(cid:80)t−1

j=1 βkℓjs

(cid:17)

exp

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)

(cid:16)(cid:80)t−1

j=1 βkℓjs
(cid:16)
e⊺
y

y=1 exp

(cid:16)(cid:80)d

(cid:80)t−1

j=1 βV ℓj

(cid:17)(cid:17)2

∂f (V, a, β, vc)
∂vcx
(cid:34) 


= E

T
(cid:88)

d
(cid:88)

t=1

s=1

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:16)

exp

ℓts

(cid:80)d

y=1 exp

e⊺
s
(cid:16)
e
(cid:16)

exp





T
(cid:88)

d
(cid:88)

ℓts

t=1

s=1

∂
∂vcx

= 0.

j a + (βV + vca⊺)ℓj + vcβ)

(cid:80)t−1

⊺
j=1(V ℓjℓ
⊺
(cid:80)t−1
y

⊺
j=1(V ℓjℓ

e⊺
s
(cid:16)

(cid:80)t−1

⊺
j=1(V ℓjℓ
⊺
(cid:80)t−1
y

⊺
j=1(V ℓjℓ

e

(cid:80)d

y=1 exp

j a + (βV + vca⊺)ℓj + vcβ)
j a + (βV + vca⊺)ℓj + vcβ)

(cid:17)

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17)



ℓts



min
s

T
(cid:88)

t=1

(cid:17) −



ℓts



T
(cid:88)

t=1

(cid:35)(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:17) −

min
s

Step 2. Calculating ∂f
∂V .

: for r, c

[d], we have

∈

The following formula will be used for calculating ∂f
∂V

(cid:32)
e

⊺
y

t
(cid:88)

exp

⊺
⊺
(V ℓiℓ
i a + (βV + vca

)ℓi + vcβ)

∂
∂Vrc

t
(cid:88)

i=1

i=1

(cid:32)
e

⊺
y

(cid:32) t

(cid:88)

= exp

= exp

i=1

Therefore,

kβℓiy + vβ

(cid:33) t

(cid:88)

i=1

β111(y = r)ℓic.

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI
(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI
(cid:32)
(cid:33)
e

t
(cid:88)

⊺
y

∂
∂Vrc

i=1

⊺
⊺
(V ℓiℓ
i a + (βV + vca

)ℓi + vcβ)

⊺
⊺
(V ℓiℓ
i a + (βV + vca

)ℓi + vcβ)

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

∂f (V, a, β, vc)
∂Vrc
(cid:34) 


T
(cid:88)

= E

d
(cid:88)

t=1

s=1

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:16)

exp

ℓts

(cid:80)d

y=1 exp

e⊺
s
(cid:16)

(cid:80)t−1

j=1(V ℓjℓ⊺
(cid:80)t−1

e⊺
y
(cid:16)

e⊺
s
(cid:16)

(cid:80)t−1

j=1(V ℓjℓ⊺
j=1(V ℓjℓ⊺
(cid:80)t−1

e⊺
y

exp

∂
∂Vrc





T
(cid:88)

d
(cid:88)

t=1

s=1

ℓts

(cid:80)d

exp

y=1 exp
(cid:16)(cid:80)t−1

j=1(V ℓjℓ⊺
(cid:17)

(cid:34) 


= E

T
(cid:88)

d
(cid:88)

t=1

s=1

ℓts

j=1 βkℓjs + vβ
(cid:16)(cid:80)t−1

j=1 βV ℓjy + vβ

(cid:80)d

y=1 exp



ℓts



T
(cid:88)

t=1

(cid:17) − min

s

73

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17)

j a + (βV + vca⊺)ℓj + vcβ)

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17)

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17) − min

s



ℓts



T
(cid:88)

t=1

(cid:17) − min

s



ℓts



T
(cid:88)

t=1

(cid:35)(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

3888
3889
3890
3891

3892
3893
3894
3895
3896
3897

3898
3899
3900
3901
3902

3903
3904
3905
3906
3907
3908

3909
3910
3911
3912
3913
3914

3915
3916
3917
3918
3919
3920

3921
3922
3923
3924
3925

3926
3927
3928
3929
3930
3931

3932
3933
3934
3935
3936
3937

3938
3939
3940
3941

Under review as a conference paper at ICLR 2025

(cid:32) T

(cid:88)

d
(cid:88)

ℓts

t=1

s=1

(cid:80)t−1

j=1 β111(s = r)ℓjc exp

(cid:16)(cid:80)t−1

j=1 βkℓjs + vβ
(cid:16)(cid:80)t−1

(cid:16)(cid:80)d

y=1 exp

(cid:17) (cid:80)d

y=1 exp
(cid:17)(cid:17)2

j=1 βkℓjy + vβ
(cid:16)(cid:80)t−1

(cid:16)(cid:80)t−1

j=1 βkℓjs + vβ

(cid:17) (cid:80)d

y=1

T
(cid:88)

d
(cid:88)

−

ℓts

exp

t=1

s=1

(cid:16)(cid:80)d

y=1 exp

j=1 β111(y = r)ℓjc exp
(cid:17)(cid:17)2

(cid:16)(cid:80)t−1

j=1 βkℓjy + vβ


(cid:34) 


= βE

T
(cid:88)

d
(cid:88)

t=1

s=1

ℓts

(cid:16)(cid:80)t−1

exp

(cid:17)

j=1 βkℓjs
(cid:16)(cid:80)t−1

(cid:80)d

y=1 exp

(cid:17) − min

s

ℓts



T
(cid:88)

t=1

(cid:16)(cid:80)t−1

j=1 βkℓjy + vβ

(cid:17)

(cid:16)(cid:80)t−1

j=1 βkℓjy + vβ

(cid:32) (cid:80)T

t=1

(cid:80)t−1
j=1

(cid:80)d

y=1 ℓtrℓjc exp

j=1 βV ℓjy
(cid:16)

(cid:16)(cid:80)d

y=1 exp

j=1 ℓjr

j=1 ℓjy

βk (cid:80)t−1
(cid:16)

βk (cid:80)t−1
(cid:123)(cid:122)
(i)

(cid:17)

(cid:16)

exp

βk (cid:80)t−1

j=1 ℓjy

(cid:17)

(cid:17)(cid:17)2

(cid:124)

−

(cid:80)T

t=1

(cid:80)t−1
j=1

(cid:80)d

y=1 ℓtyℓjc exp

(cid:16)(cid:80)d

y=1 exp

(cid:124)

(cid:16)

βk (cid:80)t−1
(cid:16)

βk (cid:80)t−1
(cid:123)(cid:122)
(ii)

j=1 ℓjr

j=1 ℓjy

(cid:125)

(cid:17)

(cid:16)

exp

βk (cid:80)t−1

j=1 ℓjy

(cid:17)

(cid:33)(cid:35)

(cid:17)(cid:17)2

.

(cid:125)

= c1 and r2 ̸
We can observe the followings: 1) if r1 ̸
(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

holds, and 2)

∂f
∂Vr1 r1

∂f
∂Vr2c2

= c2,

= ∂f

∂Vr2r2

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

∂f
∂Vr1c1
(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

.

Step 3. Calculating ∂f
∂β .

The following formula will be used for calculating ∂f
∂β

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

:

(cid:17)(cid:17)

(cid:33)(cid:35)

=

(cid:32)
e

⊺
y

exp

∂
∂β

i=1

(cid:32)
e

= exp

t
(cid:88)

⊺
(V ℓiℓ
i a + (βV + vca

⊺

)ℓi + vcβ)

t
(cid:88)

⊺
(V ℓiℓ
i a + (βV + vca

⊺

⊺
y

)ℓi + vcβ)

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI
(cid:32)
(cid:33)
e

t
(cid:88)

⊺
y

∂
∂β

i=1

⊺
⊺
i a + (βV + vca
(V ℓiℓ

)ℓi + vcβ)

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

= tvβ exp

i=1
(cid:32) t

(cid:88)

i=1

(cid:33)

kβℓiy + vβ

.

Further, we have





T
(cid:88)

d
(cid:88)

ℓts

t=1

s=1

∂
∂β

exp

(cid:16)
e⊺
s
(cid:16)

(cid:80)t−1

j=1(V ℓjℓ⊺
(cid:80)t−1

j=1(V ℓjℓ⊺

(cid:80)d

y=1 exp

e⊺
y

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17)

j a + (βV + vca⊺)ℓj + vcβ)

(cid:17) − min

s



ℓts



T
(cid:88)

t=1

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:80)d

y=1 exp

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)

exp

= vβ exp(vβ)

T
(cid:88)

t

d
(cid:88)

t=1

s=1

ℓts

= 0.

Step 4. Calculating ∂f
∂a .

(cid:17)

− (cid:80)d

y=1 exp

(cid:16)(cid:80)t−1

j=1 βkℓjs

(cid:17)

exp

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)

(cid:16)(cid:80)t−1

j=1 βkℓjs
(cid:16)
e⊺
y

y=1 exp

(cid:16)(cid:80)d

(cid:80)t−1

j=1 βV ℓj

(cid:17)(cid:17)2

74

3942
3943

3944
3945
3946
3947
3948
3949

3950
3951
3952
3953
3954
3955

3956
3957
3958
3959
3960

3961
3962
3963
3964
3965
3966

3967
3968
3969
3970
3971
3972

3973
3974
3975
3976
3977
3978

3979
3980
3981
3982
3983

3984
3985
3986
3987
3988
3989

3990
3991
3992
3993
3994
3995

Under review as a conference paper at ICLR 2025

Note that

∂
∂ax

exp

(cid:32)
e

⊺
y

i=1

(cid:32)
e

= exp

t
(cid:88)

⊺
⊺
i a + (βV + vca
(V ℓiℓ

)ℓi + vcβ)

t
(cid:88)

⊺
i a + (βV + vca
(V ℓiℓ

⊺

⊺
y

)ℓi + vcβ)

(cid:32)
e

⊺
y

= exp

i=1
t
(cid:88)

⊺
⊺
i a + (βV + vca
(V ℓiℓ

(cid:33) t

(cid:88)

)ℓi + vcβ)

i=1

(cid:32) t

(cid:88)

i=1

= exp

βkℓiy + vβ

i=1

(cid:33) t

(cid:88)

i=1

(kℓiyℓix + vℓix).

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI
(cid:33)
(cid:32)
e

t
(cid:88)

⊺
y

∂
∂ax

i=1

⊺
i a + (βV + vca
(V ℓiℓ

⊺

)ℓi + vcβ)

(cid:33) (cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:0)e

⊺
yV ℓiℓ

⊺
i ex + e

⊺
⊺
i ex
yvcℓ

(cid:1)

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

min
s

T
(cid:88)

t=1



ℓts



(cid:35)(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:17) −

(cid:16)(cid:80)t−1

j=1 βkℓjs

(cid:16)(cid:80)t−1

j=1(kℓjyℓjx + vℓjx) exp

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)(cid:17)

(cid:33)(cid:35)

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

Therefore,
∂f (V, a, β, vc)
∂ax
(cid:34) 


= E

T
(cid:88)

d
(cid:88)

t=1

s=1

(cid:16)

exp

ℓts

(cid:80)d

y=1 exp

e⊺
s
(cid:16)
e

(cid:80)t−1

⊺
j=1(V ℓjℓ
⊺
(cid:80)t−1
y

(cid:80)t−1

⊺
j=1(V ℓjℓ
⊺
j=1(V ℓjℓ
⊺
(cid:80)t−1
y

⊺
j=1(V ℓjℓ
(cid:17)

exp

(cid:16)
e⊺
s
(cid:16)

(cid:80)d

exp

e

y=1 exp
(cid:16)(cid:80)t−1

j a + (βV + vca⊺)ℓj + vcβ)

j a + (βV + vca⊺)ℓj + vcβ)
(cid:17)
j a + (βV + vca⊺)ℓj + vcβ)

j a + (βV + vca⊺)ℓj + vcβ)





∂
∂ax

T
(cid:88)

d
(cid:88)

t=1

s=1

ℓts

(cid:34) 


= E

T
(cid:88)

d
(cid:88)

ℓts

j=1 βkℓjs
(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17) −

min
s



ℓts



T
(cid:88)

t=1

(cid:80)d

y=1 exp

t=1

s=1

(cid:32) T

(cid:88)

d
(cid:88)

t=1

s=1

ℓts

(cid:80)t−1

j=1(kℓjsℓjx + vℓjx) exp

(cid:16)(cid:80)t−1

(cid:17) (cid:80)d

j=1 βkℓjs
(cid:16)(cid:80)t−1

j=1 βkℓjy

y=1 exp
(cid:17)(cid:17)2

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)

(cid:17)



ℓts



min
s

T
(cid:88)

t=1

(cid:17) −

(cid:16)(cid:80)d

y=1 exp
(cid:17) (cid:80)d

y=1
(cid:16)(cid:80)d

exp

T
(cid:88)

d
(cid:88)

−

t=1

s=1

ℓts

y=1 exp

(cid:16)(cid:80)t−1

(cid:17)(cid:17)2

j=1 βkℓjy


(cid:34)
= E



k



T
(cid:88)

d
(cid:88)

t=1

s=1

ℓts

(cid:16)(cid:80)t−1

exp

(cid:17)

j=1 βkℓjs
(cid:16)(cid:80)t−1

(cid:80)d

y=1 exp

(cid:80)t−1

j=1 ℓjsℓjx exp

(cid:32) T

(cid:88)

d
(cid:88)

t=1

s=1

ℓts

(cid:17) −

min
s

T
(cid:88)

t=1

ℓts



j=1 βkℓjy
(cid:16)(cid:80)t−1

j=1 βkℓjs

(cid:17) (cid:80)d

(cid:16)(cid:80)d

y=1 exp

(cid:16)(cid:80)t−1

y=1 exp
(cid:17)(cid:17)2

j=1 βkℓjy
(cid:16)(cid:80)t−1

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)

T
(cid:88)

d
(cid:88)

−

t=1

s=1

ℓts

exp

(cid:16)(cid:80)t−1

j=1 βkℓjs

(cid:17) (cid:80)d

y=1

(cid:16)(cid:80)t−1

j=1 βkℓjy

(cid:17)(cid:17)

(cid:33)(cid:35)

(cid:16)(cid:80)d

y=1 exp

j=1 ℓjyℓjx exp
(cid:17)(cid:17)2

(cid:16)(cid:80)t−1

j=1 βkℓjy

Note that the value does not depend on x, which means that ∂f
∂a

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

= (cid:101)c111d for some

constant (cid:101)c.

E.9.1 NUMERICAL ANALYSIS OF STEP 2 AND STEP 4

In Steps 2 and 4 above, we were not able to show that a k whose value becomes zero exists. We hence
provide some empirical evidence here. First, we attach the estimated ∂f
∂Vrc

(r

=

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

75

3996
3997

3998
3999
4000
4001
4002
4003

4004
4005
4006
4007
4008
4009

4010
4011
4012
4013
4014

4015
4016
4017
4018
4019
4020

4021
4022
4023
4024
4025
4026

4027
4028
4029
4030
4031
4032

4033
4034
4035
4036
4037

4038
4039
4040
4041
4042
4043

4044
4045
4046
4047
4048
4049

̸
Under review as a conference paper at ICLR 2025

c), ∂f
∂Vrr

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

, ∂f
∂ax

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

and ∂f
∂ax

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

graph with respect to

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

Unif([0, 1]) for all t

k value when ℓts ∼
is not stable, we can see that k for ∂f
∂Vrc

[T ], s

∈

∈
(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

[d]. While the graph of ∂f
∂V

= 0, ∂f
∂Vrr

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

= 0 and

= 0 is very similar in Figure E.2. We used the Monte Carlo estimation of

∂f
∂ax

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

1, 000, 000 times.

E.9.2 EMPIRICAL VALIDATION

Our model architecture is defined as follows: the number of layers T is set to 30 and the dimen-
sionality d to 32, with the loss vector li’s distribution Z following a standard normal distribution
(0, 1). During training, we conducted 40,000 epochs with a batch size of 512. We employed the
N
Adam optimizer, setting the learning rate to 0.001. We focus on two key convergence properties:
⊺
K ⊺(Q111 + qc) approaching the zero vector 000d and V converging to a111d111
d + bId×d, where a and b
⊺
are constants in R. The conditions K ⊺(Q111 + qc) = 000d and V = a111d111
d + bId×d imply that the
function g(Zt; V, Q, K) = (cid:80)t
a)li, effectively emulating the process of an online gradient
descent method. We repeated 10 times. For verifying K ⊺(Q111 + qc) = 000d, we will measure 2-norm
⊺
of K ⊺(Q111 + qc). Also for measuring the closeness of V and a111d111
d + bId×d, we will measure
∥2,2/b. The results are demonstrated in the third plot of Figure E.1.
mina,b∈R

⊺
(a111d111
d + bId×d)

i=1(b

V
∥

−

−

E.10 COMPARISON WITH AHN ET AL. (2023); ZHANG ET AL. (2023A); MAHANKALI ET AL.

(2023)

i=1 yix

yt+1∥

The very recent studies by Ahn et al. (2023); Zhang et al. (2023a); Mahankali et al. (2023)
have demonstrated that if Zt = ((x1, y1), . . . , (xt, yt), (xt+1, 0)) and the “instruction tuning”
loss (i.e., E[
2]) is being minimized with a single-layer linear self-attention model,
∥(cid:98)yt+1 −
then a global optimizer among single-layer linear self-attention models yields the output (cid:98)yn+1 =
⊺
η (cid:80)n
i xn+1. This output can be interpreted as a gradient descent algorithm, indicating that
a single-layer linear self-attention model implicitly performs gradient descent. However, in the on-
line learning setting where there are no y-labels, such an implicit gradient descent update-rule is
hard to define. Compared to the previous studies, our global optimizer among single-layer linear
self-attention models is an explicit and online gradient descent update for online learning. With a
different loss (regret-loss v.s. instruction-tuning-loss), the techniques to obtain the seemingly similar
results are also fundamentally different.

E.11 DETAILS OF SECTION 5.4

Randomly generated loss sequences. We use the same loss vectors as those in Section 3.2 for
randomly generated loss functions, and compare the results with that using GPT-4. The results show
that with regret-loss, both the trained single-layer self-attention model and the trained Transformers
with multi-layer self-attention structures can achieve comparable regrets as FTRL and GPT-4. The
results can be found in Figure E.3.

Loss sequences with certain trends. We investigate the case where the loss sequences have pre-
dictable trends such as linear-trend or sine-trend. One might expect that the performance of the
trained Transformer would surpass the performance of traditional no-regret learning algorithms such
as FTRL, since they may not be an optimal algorithm for the loss sequence with a predictable trend.
We modify the training distribution by changing the distribution of random variable Z (which gen-
erates the loss vectors ℓt) to follow two kinds of trends: linear and sine functions. The results, as il-
lustrated in Figure E.4, show that the trained single-layer self-attention model and the trained Trans-
former with multi-layer self-attention structures with regret-loss outperformed GPT-4 and FTRL in
terms of regret, when the loss sequence is a linear trend. Similarly, Figure E.4 shows that the trained
Transformer with multi-layer self-attention structures with regret-loss is comparable to GPT-4 and
outperformed FTRL in terms of regret, when the loss sequence is a sine-trend. Note that the training

76

4050
4051

4052
4053
4054
4055
4056
4057

4058
4059
4060
4061
4062
4063

4064
4065
4066
4067
4068

4069
4070
4071
4072
4073
4074

4075
4076
4077
4078
4079
4080

4081
4082
4083
4084
4085
4086

4087
4088
4089
4090
4091

4092
4093
4094
4095
4096
4097

4098
4099
4100
4101
4102
4103

Under review as a conference paper at ICLR 2025

4104
4105

4106
4107
4108
4109
4110
4111

4112
4113
4114
4115
4116
4117

4118
4119
4120
4121
4122

4123
4124
4125
4126
4127
4128

4129
4130
4131
4132
4133
4134

4135
4136
4137
4138
4139
4140

4141
4142
4143
4144
4145

4146
4147
4148
4149
4150
4151

4152
4153
4154
4155
4156
4157

Figure E.2: Calculation of 20 ∂f
∂Vrc

and ∂f
∂ax

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(r

= c)(red), 20 ∂f
∂Vrr

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(blue),

(black). We experimented with n

∈

[4, 9] and d

[4, 9]. The fig-

∈

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

(r

= c),

ure might indicate that βk that makes the derivative zero of

∂f
∂Vrc

∂f
∂Vrr

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

, and ∂f
∂ax

(cid:12)
(cid:12)
(cid:12)
(cid:12)a=000d,vc=v111d,V =kI

would coincide.

dataset does not contain the sequence of losses. Nonetheless, by focusing on the overall trend during
training, we can attain performance that is either superior to or on par with that of FTRL and GPT-4.

77

̸
̸
Under review as a conference paper at ICLR 2025

4158
4159

4160
4161
4162
4163
4164
4165

4166
4167
4168
4169
4170
4171

4172
4173
4174
4175
4176

4177
4178
4179
4180
4181
4182

4183
4184
4185
4186
4187
4188

4189
4190
4191
4192
4193
4194

4195
4196
4197
4198
4199

4200
4201
4202
4203
4204
4205

4206
4207
4208
4209
4210
4211

Figure E.3: Regret performance for the randomly generated loss sequences that are generated by
Gaussian with truncation and uniform distribution. No-regret behaviors of single-layer and multi-
layer self-attention models are validated by both of our frameworks (low p-values and (cid:98)β0 < 1).

Figure E.4: Regret performance for the randomly generated loss sequences that are generated by
linear-trend and sine-trend. No-regret behaviors of single-layer and multi-layer self-attention models
are validated by both of our frameworks (low p-values and (cid:98)β0 < 1).

Repeated games. We then investigate the case of multi-player repeated games. We study
2x2, 3x3x3, 3x3x3x3 games, where each entry of the payoff matrix is sampled randomly from
Unif([0, 10]). The results, as illustrated in Figure E.5, show that the trained single-layer self-
attention model and the trained Transformer with multi-layer self-attention structures with regret-
loss have a similar performance as that of FTRL. However, GPT-4 still outperforms the trained
single-layer self-attention model and the trained Transformer with multi-layer self-attention struc-
tures in terms of regret. Since for repeated games (in which the environment faced by the agent
can be less adversarial than that in the online setting), there might be a better algorithm than FTRL
(see e.g., Daskalakis et al. (2021)), while our self-attention models have a similar structure as FTRL
(Theorem E.2 or Theorem 5.2). Also, in practical training (with the empirical loss in Equation (E.3)),
we possibly did not find the exact global minimum or stationary point of the expected loss in Equa-
tion (5.2). Hence, it is possible that GPT-4 may have lower regret than our trained models with the
regret-loss.

Two scenarios that caused regrettable behaviors of GPT-4. Finally, we investigate the cases that
have caused GPT-4 to have regrettable performance in Section 3.2. The results, which can be found
in Figure 3.4, show that both the trained single-layer self-attention model and the trained Trans-
former with regret-loss can achieve comparable no-regret performance as FTRL, and outperforms
that of GPT-4. This validates that our new unsupervised training loss can address the regrettable
cases, as our theory in Section 5.2 and 5.3 has predicted.

Remark on Performance Discrepancy Between Single-Agent and Multi-Agent Settings Why
does GPT-4 exhibit better regret performance compared to single/multi-layer models in the single-
agent setting, yet underperform in the multi-agent setting? What factors contribute to this discrep-
ancy in its effectiveness across different settings?

78

0510152025Iterations0.02.55.07.510.012.515.0RegretUniformSingle-layer:(ptrend,ˆβ0,preg)=(0.0,0.43,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.0,0.47,0.0)FTRL:(ptrend,ˆβ0,preg)=(0.0,0.45,0.0)GPT-4:(ptrend,ˆβ0,preg)=(0.0,0.42,0.0)0510152025Iterations01234RegretGaussianSingle-layer:(ptrend,ˆβ0,preg)=(0.0,0.4,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.0,0.39,0.0)FTRL:(ptrend,ˆβ0,preg)=(0.0,0.4,0.0)GPT-4:(ptrend,ˆβ0,preg)=(0.0,0.44,0.0)0510152025Iterations0102030RegretLinear-trendSingle-layer:(ptrend,ˆβ0,preg)=(0.0,0.51,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.0,-0.13,0.03)FTRL:(ptrend,ˆβ0,preg)=(0.0,0.62,0.0)GPT-4:(ptrend,ˆβ0,preg)=(0.0,0.64,0.0)0510152025Iterations−10−505101520RegretSine-trendSingle-layer:(ptrend,ˆβ0,preg)=(0.0,0.89,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.0,-0.9,0.19)FTRL:(ptrend,ˆβ0,preg)=(0.0,0.92,0.0)GPT-4:(ptrend,ˆβ0,preg)=(0.0,-0.2,0.68)Under review as a conference paper at ICLR 2025

Figure E.5: Regret performance for the game with two players, three players, and four players
general-sum games. No-regret behaviors of single-layer and multi-layer self-attention models are
validated by both of our frameworks (low p-values and (cid:98)β0 < 1).

In certain scenarios, LLMs can outperform FTRL/FTPL algorithms and single/multi-layer models.
This phenomenon is primarily observed when the loss sequence exhibits discernible trends, as seen
in the single-agent setting. In Section 3.4, we explored this behavior using canonical counterex-
amples for the follow-the-leader algorithm. Specifically, when the loss sequences display obvious
or predictable patterns, LLMs can effectively infer the next loss vector based on historical data,
enabling near-optimal decisions. This phenomenon can be further formalized through the lens of
in-context learning. Conversely, FTRL/FTPL algorithms, constrained by their update rules, tend
to produce near-uniform policies in such cases, as do single/multi-layer Transformer models. In
Appendix C.7, we provide ablation studies to support these observations, demonstrating that LLMs
leverage trends in the loss sequences by comparing their performance when provided with raw versus
summarized historical data. When the loss sequences are summarized (e.g., through aggregation),
the resulting loss vectors no longer reflect the trend, leading to significantly diminished performance
by the LLMs. These findings have been clarified and emphasized in the updated manuscript.

In contrast, in multi-agent or game settings, the loss sequence trends depend on the behavior of other
agents, rendering them inherently less predictable as all agents continually update their behavior
policies. This increased unpredictability likely accounts for the comparable or inferior performance
of LLMs relative to FTRL/FTPL algorithms or single/multi-agent-trained Transformer models in
such settings.

E.11.1 TRAINING DETAILS OF SECTION 5.4

We provide the training details of Section 5.4. For the multi-layer Transformer training, we used 4
layers, 1 head Transformer. For both single-layer and multi-layer, we employed the Adam optimizer,
setting the learning rate to 0.001. During training, we conducted 2,000 epochs with a batch size 512.
Moreover, when we trained for the loss sequences with the predictable trend, we used 4 layers, 1
head Transformer. For both single-layer and multi-layer, we employed the Adam optimizer, setting
the learning rate to 0.001. During training, we conducted 9,000 epochs with a batch size of 512.

E.12 ABLATION STUDY ON TRAINING EQUATION (5.2)

In this section, we provide an ablation study that changes N and k in Equation (5.2). To be specific,
we will set N = 1, 2, 4, f (x, k) = max(x, 0)k, h(x) = max(x, 0)2, and k = 1, 2. For the multi-
layer Transformer training, we used 4 layers and 1 head Transformer. For both single-layer and
multi-layer, we employed the Adam optimizer, setting the learning rate to 0.001. During training,
we conducted 2,000 epochs with a batch size of 512. We experimented on the randomly generated
Unif([0, 10]2)), with the results
loss sequences. Especially, we used the uniform loss sequence (ℓt ∼
1112, I)), with the results in
in Figure E.6 and Figure E.7; and the Gaussian loss sequence (ℓt ∼ N
Figure E.8 and Figure E.9.

(5

·

4212
4213

4214
4215
4216
4217
4218
4219

4220
4221
4222
4223
4224
4225

4226
4227
4228
4229
4230

4231
4232
4233
4234
4235
4236

4237
4238
4239
4240
4241
4242

4243
4244
4245
4246
4247
4248

4249
4250
4251
4252
4253

4254
4255
4256
4257
4258
4259

4260
4261
4262
4263
4264
4265

79

0510152025Iterations05101520253035RegretTwo-playergamesSingle-layer:(ptrend,ˆβ0,preg)=(0.0,0.93,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.0,0.91,0.0)FTRL:(ptrend,ˆβ0,preg)=(0.0,0.83,0.0)GPT-4:(ptrend,ˆβ0,preg)=(0.0,0.67,0.0)0510152025Iterations0510152025RegretThree-playergamesSingle-layer:(ptrend,ˆβ0,preg)=(0.0,0.94,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.0,0.98,0.0)FTRL:(ptrend,ˆβ0,preg)=(0.0,0.94,0.0)GPT-4:(ptrend,ˆβ0,preg)=(0.0,0.69,0.0)0510152025Iterations0246810121416RegretFour-playergamesSingle-layer:(ptrend,ˆβ0,preg)=(0.0,0.97,0.0)Multi-layer:(ptrend,ˆβ0,preg)=(0.0,0.99,0.0)FTRL:(ptrend,ˆβ0,preg)=(0.0,0.97,0.0)GPT-4:(ptrend,ˆβ0,preg)=(0.0,0.79,0.0)Under review as a conference paper at ICLR 2025

4266
4267

4268
4269
4270
4271
4272
4273

4274
4275
4276
4277
4278
4279

4280
4281
4282
4283
4284

4285
4286
4287
4288
4289
4290

4291
4292
4293
4294
4295
4296

4297
4298
4299
4300
4301
4302

4303
4304
4305
4306
4307

4308
4309
4310
4311
4312
4313

4314
4315
4316
4317
4318
4319

Figure E.6: Ablation study for the uniform loss sequence trained with single-layer self-attention
layer and Softmax projection.

Figure E.7: Ablation study for the uniform loss sequence trained with multi-layer self-attention layer
and Softmax projection.

80

01020Iterations024681012RegretN=1,k=1,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.57,0.0)FTRLGPT-401020Iterations024681012RegretN=2,k=1,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.59,0.0)FTRLGPT-401020Iterations024681012RegretN=4,k=1,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.54,0.0)FTRLGPT-401020Iterations024681012RegretN=1,k=2,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.57,0.0)FTRLGPT-401020Iterations−2.50.02.55.07.510.012.5RegretN=2,k=2,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.57,0.0)FTRLGPT-401020Iterations−2.50.02.55.07.510.012.5RegretN=4,k=2,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.58,0.0)FTRLGPT-401020Iterations−2.50.02.55.07.510.012.515.0RegretN=1,k=1,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.59,0.0)FTRLGPT-401020Iterations−2.50.02.55.07.510.012.5RegretN=2,k=1,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.57,0.0)FTRLGPT-401020Iterations051015RegretN=4,k=1,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.6,0.0)FTRLGPT-401020Iterations−2.50.02.55.07.510.012.515.0RegretN=1,k=2,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.59,0.0)FTRLGPT-401020Iterations−2.50.02.55.07.510.012.5RegretN=2,k=2,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.55,0.0)FTRLGPT-401020Iterations−2.50.02.55.07.510.012.5RegretN=4,k=2,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.53,0.0)FTRLGPT-4Under review as a conference paper at ICLR 2025

4320
4321

4322
4323
4324
4325
4326
4327

4328
4329
4330
4331
4332
4333

4334
4335
4336
4337
4338

4339
4340
4341
4342
4343
4344

4345
4346
4347
4348
4349
4350

4351
4352
4353
4354
4355
4356

4357
4358
4359
4360
4361

4362
4363
4364
4365
4366
4367

4368
4369
4370
4371
4372
4373

Figure E.8: Ablation study for the Gaussian loss sequence trained with single-layer self-attention
layer and Softmax projection.

Figure E.9: Ablation study for the Gaussian loss sequence trained with single-layer self-attention
layer and Softmax projection.

81

0510152025Iterations01234RegretN=1,k=1,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.41,0.0)FTRLGPT-40510152025Iterations01234RegretN=2,k=1,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.39,0.0)FTRLGPT-40510152025Iterations01234RegretN=4,k=1,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.44,0.0)FTRLGPT-40510152025Iterations01234RegretN=1,k=2,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.41,0.0)FTRLGPT-40510152025Iterations−101234RegretN=2,k=2,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.44,0.0)FTRLGPT-40510152025Iterations−101234RegretN=4,k=2,model=singleSingle-layer:(ptrend,ˆβ0,preg)=(0.0,2.42,0.0)FTRLGPT-401020Iterations−101234RegretN=1,k=1,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.43,0.0)FTRLGPT-401020Iterations01234RegretN=2,k=1,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.46,0.0)FTRLGPT-401020Iterations−101234RegretN=4,k=1,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.4,0.0)FTRLGPT-401020Iterations−101234RegretN=1,k=2,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.43,0.0)FTRLGPT-401020Iterations01234RegretN=2,k=2,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.55,0.0)FTRLGPT-401020Iterations−101234RegretN=4,k=2,model=multiMulti-layer:(ptrend,ˆβ0,preg)=(0.0,2.72,0.0)FTRLGPT-4Under review as a conference paper at ICLR 2025

F LIMITATIONS AND CONCLUDING REMARKS

In this paper, we studied the online decision-making and strategic behaviors of LLMs quantitatively,
through the metric of regret. We first examined and validated the no-regret behavior of several repre-
sentative pre-trained LLMs in benchmark settings of online learning and games. As a consequence,
(coarse correlated) equilibrium can oftentimes emerge as the long-term outcome of multiple LLMs
playing repeated games. We then provide some theoretical insights into the no-regret behavior, by
connecting pre-trained LLMs to the follow-the-perturbed-leader algorithm in online learning, under
certain assumptions. We also identified (simple) cases where pre-trained LLMs fail to be no-regret,
and thus proposed a new unsupervised training loss, regret-loss, to provably promote the no-regret
behavior of Transformers without the labels of (optimal) actions. We established both experimental
and theoretical evidence for the effectiveness of our regret-loss.

As a first attempt toward rigorously understanding the online and strategic decision-making be-
haviors of LLMs through the metric of regret, We provide the following limitations and list some
potential directions for future research:

• There are more than one definitions of (dynamic-)regret in the online learning literature,
and we mainly focused on the so-called external-regret in the literature. There are some
other regret metrics we have studied, e.g., swap-regret (Blum & Mansour, 2007), which
may lead to stronger equilibrium notions in playing repeated games.

• Our new regret-loss has exhibited promises in our experiments for training modest-scale
Transformers. One limitation is that we have not trained other larger-scale models, such as
Foundation Models, for decision-making, which is an important ongoing effort.

• Our efforts of Theorem 4.1 towards explaining the success of LLMs in achieving sub-
linear regret are highly hypothetical. Considering LLMs are such complex, random, and
black-box systems, there might be certain behaviors that our Theorem 4.1 cannot fully
capture, and there do exist other possible explanations. For example, an alternative in-
context-learning-based explanation may be used to account for the improved performance
of LLMs on specific loss sequences with trends. Specifically, LLMs may interpret past loss
sequences as demonstrations to identify the latent trends, make accurate predictions on the
next loss, and make optimal decisions. However, this explanation may not generalize to
the loss sequences without obvious trends, complementing our explanations based on the
connection to no-regret learning algorithms, which apply to general loss sequences (see
Appendix C.7 for more discussions).

• No-regret behavior can sometimes lead to better outcomes in terms of social efficiency
(Blum et al., 2008; Roughgarden, 2015; Nekipelov et al., 2015). It would thus be interesting
to further validate the efficiency of no-regret LLM agents in these scenarios, as well as
identifying new prompts and training losses for LLMs to promote the efficiency of the
outcomes.

• To evaluate the performance quantitatively, we focused on online learning and games
with numeric valued payoffs. It would be interesting to connect our no-regret-based and
game-theoretic framework with existing multi-LLM frameworks, e.g., debate, collabora-
tive problem-solving, and human/social behavior simulation, with potentially new notions
of regret (defined in different spaces) as performance metrics.

4374
4375

4376
4377
4378
4379
4380
4381

4382
4383
4384
4385
4386
4387

4388
4389
4390
4391
4392

4393
4394
4395
4396
4397
4398

4399
4400
4401
4402
4403
4404

4405
4406
4407
4408
4409
4410

4411
4412
4413
4414
4415

4416
4417
4418
4419
4420
4421

4422
4423
4424
4425
4426
4427

82

