Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments
Under review as a conference paper at ICLR 2025

LEARN-BY-INTERACT: A DATA-CENTRIC FRAME-
WORK FOR SELF-ADAPTIVE AGENTS IN REALISTIC
ENVIRONMENTS

Anonymous authors
Paper under double-blind review

ABSTRACT

Autonomous agents powered by large language models (LLMs) have the potential
to enhance human capabilities, assisting with digital tasks from sending emails to
performing data analysis. The abilities of existing LLMs at such tasks are of-
ten hindered by the lack of high-quality agent data from the corresponding envi-
ronments they interact with. We propose LEARN-BY-INTERACT, a data-centric
framework to adapt LLM agents to any given environments without human an-
notations. LEARN-BY-INTERACT synthesizes trajectories of agent-environment
interactions based on documentations, and constructs instructions by summariz-
ing or abstracting the interaction histories, a process called backward construction.
We assess the quality of our synthetic data by using them in both training-based
scenarios and training-free in-context learning (ICL), where we craft innovative
retrieval approaches optimized for agents. Extensive experiments on SWE-bench,
WebArena, OSWorld and Spider2-V spanning across realistic coding, web, and
desktop environments show the effectiveness of LEARN-BY-INTERACT in various
downstream agentic tasks — baseline results are improved by up to 11.1% for ICL
with Claude-3.5 and 23.1% for training with Codestral-22B. We further demon-
strate the critical role of backward construction, which provides up to 10.6% im-
provement for training. Our ablation studies demonstrate the efﬁciency provided
by our synthesized data in ICL and the superiority of our retrieval pipeline over
alternative approaches like conventional retrieval-augmented generation (RAG).
We expect that LEARN-BY-INTERACT will serve as a foundation for agent data
synthesis as LLMs are increasingly deployed at real-world environments.

1

INTRODUCTION

Pre-trained large language models (LLMs) offer great potential for assisting humans with various
tasks in digital settings, such as editing images, performing data analysis, resolving software en-
gineering issues, and navigating commercial platforms (Xie et al., 2023; 2024; Yao et al., 2022a;
Jimenez et al., 2023). By streamlining these, LLM agents can greatly enhance human efﬁciency and
productivity, allowing individuals to shift their focus toward higher-level, creative, and strategic en-
deavors. To explore this potential, many benchmarks (Jimenez et al., 2023; Zhou et al., 2023b; Xie
et al., 2024; Cao et al., 2024; Koh et al., 2024) and agentic frameworks (Yang et al., 2024; Zhan &
Zhang, 2023; Yang et al., 2023; Gur et al., 2023; Chen et al., 2024a) have been established based on
realistic digital environments, spanning web applications, code development, desktop computing,
etc. However, current LLMs often fall short of expected performance in these tasks, consistently
displaying a signiﬁcant gap compared to human capabilities. As a result, they remain less practical
and reliable for real-world applications.

Efﬁcient adaptation to new environments can be the key part of the performance improvements.
Prior works have explored various prompt-based approaches (Yao et al., 2022b; Yang et al., 2024;
Gur et al., 2023; Zhan & Zhang, 2023), that are constrained by the capabilities of underlying foun-
dation models. Other studies on training LLMs with human-labeled examples (Chen et al., 2023;
2024b; Li et al., 2020) on the other hand, come with the fundamental limitation of high annotation
costs when new environments are considered. In particular, annotating agentic data can be quite
difﬁcult and expensive due to long-trajectory interactions with environments and speciﬁc domain

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Overview of the data synthesis and adaptation processes. Given an environment and stan-
dard resources, we ﬁrst leverage self-instruct to create a diverse set of instructions. LLMs are then
employed to complete these tasks, resulting in long trajectories of agent-environment interactions.
We construct task instructions using LLMs for each sub-trajectory, a process called backward con-
struction. The synthesized data are then ﬁltered and used for both training and in-context learning,
where we design agentic retrieval to retrieve demonstration examples based on information at each
step, using both model-based and observation-based approaches. See Appendix F for the complete
data synthesis example and Algorithm 2 for more details on agentic retrieval.

expertise required. Few works have explored fully-autonomous data construction pipelines towards
self-adaptive agents that can efﬁciently learn new environments (Gulcehre et al., 2023; Aksitov et al.,
2023).

In this paper, we introduce LEARN-BY-INTERACT, a data-centric framework for LLMs to self-adapt
to new environments, utilizing agent data synthesis via interactions. Intuitively, the effects of ac-
tions executed in environments (e.g., the next webpage after clicking a button) serve as informa-
tive demonstrations that help LLMs in future navigation. Inspired by this, we design LEARN-BY-
INTERACT that ﬁrst uses self-instruct (Wang et al., 2022b) to develop a variety of task instructions,
referring to standard resources such as documentations and tutorials for a given environment. This
covers most important scenarios that human users are interested in and avoids intensive prompt en-
gineering to control the distribution and diversity of the generated data. We then collect diverse tra-
jectories from interactions between LLMs and environments, as illustrated in Fig. 1. However, given
the low performance of LLMs in existing agentic benchmarks (Xie et al., 2024; Cao et al., 2024), it is
likely that a large percentage of synthesized trajectories do not match with the instructions. To tackle
this challenge, we construct new instructions by summarizing or abstracting each sub-trajectory,
leveraging the strong summarization capabilities of LLMs (Pu et al., 2023; Liu et al., 2023). We
call this process backward construction. After obtaining synthesized instruction-trajectory pairs and
ﬁltering low-quality ones, we apply it to both training and ICL, where we craft innovative retrieval
pipelines optimized for agents. Concretely, it consists of two parts: (1). model-based approach that
leverages LLMs to ﬁrst write queries based on instructions, interaction histories and current obser-
vations, and uses retrieval models to retrieve demonstration examples from synthesized data; (2).
observation-based approach that ﬁnds examples with the current observation appearing in trajecto-
ries (which indicates that the current state has been encountered in the data synthesis process).

Our comprehensive evaluations across four challenging benchmarks: SWE-bench (Jimenez et al.,
2023), WebArena (Zhou et al., 2023b), OSWorld (Xie et al., 2024), and Spider2-V (Cao et al., 2024),
highlight the efﬁcacy of the data generated by LEARN-BY-INTERACT. With ICL, both Gemini-1.5-
pro (Reid et al., 2024) and Claude-3.5-sonnet (Anthropic, 2024) show consistent and remarkable im-
provements – for OSWorld (Xie et al., 2024), our generated data nearly doubles Claude-3.5-sonnet’s
baseline performance, increasing it from 11.4% to 22.5%. Furthermore, substantial improvements
are observed by training models of varying sizes and architectures with our synthesized data. As
an example, Codestral-22B’s (Team, 2024b) performance in WebArena signiﬁcantly increases from

2

LLM AgentAgent-environment interactionGenerated task instruction by self-instruct: 
Upload CSV file in Google Drive to BigQueryAct1: view dataset “demo”Act2: click button create tableAct3: select table source Google Cloud Storage
(wronk prediction misalikned with instruction)Obs0: BigQuery HomepageObs1: Dataset page,with info like creation timeObs2: Table creation pageObs13: BigQuery table created.Tutorial and  DocumentationFAQCode: Software: Web: ResourcesEnvironmentsInstruction 1: Replicate the ...
Trajectory 1: (Obs1, Act2, Obs2)
(Obs1, Act2, Obs2)new instructionkeplicate the following: In the dataset page, click the button create table ...Instruction n: Link CSV file ...
Trajectory n: (Obs0, ..., Obs13) Synthesized dataIn-context learningBackward construction (Obs0, ..., Obs13)Update  instruction to align with trajectoryLink CSV file in Google Cloud Storage to BigQueryFiltered dataTraining..............................Loading CSV dataGo to the BigQuery page. How to create BigQuery table?
 Answer: ......InstructionObs 0Act 1Obs 1Act 2Model-based:Find examples with similar intent and trajectory Find examples with the same observationObservation-based:Agentic retrievalUnder review as a conference paper at ICLR 2025

4.7% to 27.8% after training. These results underscore the high quality of our generated agent data
and its broad applicability across diverse agent environments.

Our extensive ablation studies reveal that backward construction not only increases the quantity of
the synthesized data, but also improves its overall quality (§3.5). With data synthesized by LEARN-
BY-INTERACT, we observe signiﬁcant improvements in both performance and efﬁciency during
LLM inference (§4.1). Our empirical results demonstrate the superiority of the agentic retrieval
in ICL (§4.2). We anticipate that this research will spark innovative developments in enhancing
agent performance using LLMs and contribute to its wider-spread adoption in real-world application
scenarios.

2 LEARN-BY-INTERACT

We introduce the LEARN-BY-INTERACT pipeline to synthesize agent data in an autonomous way by
leveraging interactions between LLMs and environments. We ﬁrst formalize the agent canonical task
(§2.1), and introduce the detailed synthesis (§2.2) and ﬁltering (§2.3) procedures. We then describe
the application of the synthesized data in adapting LLMs in both training-free and training-based
settings (§2.4).

2.1 TASK FORMULATION

Given an environment E and a task instruction I, the objective of an agent A is to achieve the target
G through multi-step interactions with E. At each step i, A predicts the next action ai based on the
instruction I and the previous history H = (o0, a1, o1, a2, ..., oi−1), which is then executed in the
environment E to get a new observation oi. The interactions terminated until A predicts the action
stop or the maximum number of steps m is reached.

2.2 AGENTIC DATA SYNTHESIS

The essential idea of LEARN-BY-INTERACT
is manifested in synthesizing environment-
speciﬁc agent data with zero human effort.
In Algorithm 1, we show the overall pro-
cess with pseudo-code. Given an environ-
ment for a downstream application (such as vi-
sual studio code), we ﬁrst leverage commonly-
accessible resources like documentation to
generate diverse task instructions using self-
instruct (Wang et al., 2022b) (line 5). These
resources are usually created by human experts
to address common concerns and provide usage
suggestions, e.g., how to navigate a website or
operate a software. Intuitively, such references
often cover representative usecases of an ap-
plication. Therefore the task instructions gen-
erated conditioned on them could cover most
popular scenarios in the domain and avoids po-
tentially unreasonable cases that may be of less
value.

For each generated task, LLMs then aim to
solve it, which results in a long trajectory
T = (o0, a1, o1, ..., an, on) (line 9-14 in Algo-
rithm 1). To address the potential misalignment
between the instruction I and the generated tra-
jectories T , we introduce a novel mechanism,
backward construction,
instruc-
tions based on trajectories (line 15-22 in Algo-
rithm 1). Speciﬁcally, for each sub-trajectory

to construct

Algorithm 1 Agent data synthesis
1: Input: LLM : Large Language Model; E: envi-
ronment; Doc: standard resources like documenta-
tion; N : the number of instructions to generate per
document; F : data ﬁlter.

// initialize interaction trajectory

// self-instruct to generate N task instructions
Instructions = LLM (d, N )
for I in Instructions do

o = E.get_observation()
a = LLM (I, T, o)
T += [o, a]

E.reset()
T = []
while not E.ﬁnished() do

2: Initialization: D = []: synthesized data.
3: for d in Doc do
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24: end for
25: D = F (D) // Filter low-quality data
26: Return: D

end while
T.append(E.get_observation())
// backward construction
for i in range(0, len(T ) − 1, 2) do

T (cid:48) = T [i : j]
I (cid:48) = LLM (T (cid:48))
D.append([I (cid:48), T (cid:48)])

end for

end for

end for

for j in range(i + 2, len(T ), 2) do

3

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Under review as a conference paper at ICLR 2025

T (cid:48) = (oi, ai+1, oi+1, ..., aj, oj), 0 ≤ i < j ≤ n, we obtain two types of new instructions: (1).
summarizations of trajectory steps; (2). abstractions of the trajectory purpose. In Fig. 1, the sub-
trajectory (Obs1, Act2, Obs2) is summarized into a new task instruction that requires to replicate
the Act2. The abstraction of the full trajectory updates the original task objective, which is no
longer aligned with the generated trajectory due to the wrong prediction in the action 3. Overall, the
LEARN-BY-INTERACT pipeline offers two notable advantages:

• It corrects the potential misalignment between instructions and predicted trajectories by updating
task objectives, which enhances the data quality as veriﬁed by the experimental results in §3.5.

• It maximizes the utility of each generated trajectory by crafting new instructions for each sub-
trajectory. This results in a quadratic increase in the number of synthesized examples with respect
to the steps in the sequence per generated trajectory. For a given target dataset size, backward
construction substantially decreases the necessary interactions, which is particularly valuable in
scenarios where such interactions are challenging and costly to obtain such as Robotics (Keipour,
2022).

2.3 FILTERING

To further enhance the data quality, we design the following criteria to ﬁlter inferior synthesized data:
(1). Remove duplicate states: We remove duplicate (ai, oi) from T (cid:48) if (ai, oi)=(ai−1, oi−1), which is
potentially introduced by the invalid action or the environment error (inactivity). (2). LLM commit-
tee check: We feed the generated instruction-trajectory pair (I (cid:48), T (cid:48)) into a committee of LLMs, and
only classify it of high-quality if all LLMs consider the trajectory coherent, natural, reasonable and
aligned with the instruction. The listed criteria are all fully-autonomous and canonically-applicable
for ﬁltering data synthesized in general agent scenarios. See Table 35 for our prompts used in LLM
committee check.

2.4 ADAPTATION

After obtaining the synthesized data D, we ap-
ply it to both ICL and training. Given the
unique characteristics of multi-round interac-
tions with environments in agent settings, we
design agentic retrieval (pseudo-code in Al-
gorithm 2) to maximize the effectiveness of
the synthesized data. Speciﬁcally, we propose
two retrieval pipelines: observation-based (line
5-14) and model-based retrieval (line 15-17).
In observation-based retrieval, we compare the
current observation o to the trajectory of each
example e in the synthesized data, where e =
[I (cid:48), [o0, a1, o1, ..., an, on]]. If o matches one of
the observations in e, i.e., o = oi, then we
consider e as a helpful example to the current
task. For the model-based retrieval, we lever-
age LLMs to ﬁrst write queries based on the
instruction, the interaction history and the cur-
rent observation (line 16), and then employ re-
trieval models to retrieve non-duplicate exam-
ples (line 17). LLMs are then augmented with
the retrieved examples to predict the next action
(line 18). Refer to Table 36 to 39 for prompts
to write queries and predict actions.

Algorithm 2 ICL with agentic retrieval
1: Input: LLM : Large Language Model; E: envi-
ronment; D: synthesized data; RM : retriever; I:
task instruction; m1: maximum number of exam-
ples from observation-based retrieval; m2: max-
imum number of examples from model-based re-
trieval.

2: Initialization: H = []: interaction history; R: re-

trieved examples.

// iterate through the trajectory
for o1 in t do

if o1=o then

R.append([i, t])

o = E.get_observation()
// observation-based retrieval
for i, t in D do

3: while not E.ﬁnished() do
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20: end while

end for
R = R[:m1]
// model-based retrieval
q = LLM (I, H, o)
R += RM (q, D, m2, R)
a = LLM (I, H, o, R)
H+ = [o, a]

end for

end if

from using the synthesized data as
Apart
demonstration examples in ICL, we further uti-
lize them to ﬁne-tune models. For a given generated example, we convert it to the format of action
prediction (Table 36), and prepare input-output pairs for supervised ﬁne-tuning. More details on the
experimental settings can be found in §3.3.

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Table 1: Statistics for the number of crawled documents, generated raw trajectories, examples
(instruction-trajectory pairs) and examples after ﬁltering.

SWE-bench WebArena OSWorld

Spider2-V

Documents
Raw trajectories
Examples
Filtered examples

6,464
19,392
180,752
101,523

3,578
10,734
185,635
109,276

7,362
22,086
437,635
103,526

11,231
33,693
652,786
125,683

3 EXPERIMENTS

3.1 BASELINES

We compare ICL with agentic retrieval to the following prompt-based approaches.

• Baseline: The vanilla prediction pipeline in each benchmark that includes the task instruction,
interaction history and the state observation in the prompt. See more implementation details in
Appendix A.

• RAG: The conventional RAG pipeline that ﬁrst retrieves from the resources like documentation

based on the instruction, and augments LLMs with the retrieved content.

• Data distill: We follow the same pipeline to synthesize data in Algorithm 1 except backward
construction (replace line 15-22 with D.append(I, T )), and follow Algorithm 2 during the evalu-
ation.

• Reﬂexion (Shinn et al., 2024): A general framework to reinforce language agents through linguis-

tic feedback from both executors and LLMs.

• Language Agent Tree Search (LATS) (Zhou et al., 2023a): It integrates the combinatorial tree
search into expanding ReAct (Yao et al., 2022b) and combine agent online reasoning, acting and
planning throughout the trajectory.

For the training-based evaluation, we primarily compare to the data distillation, which also con-
structs data from scratch and requires no human effort to annotate seed or preference data. Addi-
tionally, we include the model performance before training as another baseline.

3.2 DATASETS

We consider 4 agent datasets that involve multi-round interactions with realistic environments. They
span diverse domains of code, web, computer desktop and professional software. Appendix C illus-
trates details of each dataset with examples.

• SWE-bench (Jimenez et al., 2023) is an evaluation benchmark on realistic software engineering
problems from realistic Github issues. We use the Lite version by default throughout the experi-
ments.

• Webarena (Zhou et al., 2023b) evaluates agent capabilities to perform tasks in the web environ-

ments such as e-commerce, social forum discussion, and beyond.

• OSWorld (Xie et al., 2024) is an integrated environment for assessing open-ended computer tasks,

which involve diverse applications like terminal, chrome, etc.

• Spider2-V (Cao et al., 2024) is a multimodal agent benchmark focusing on professional data

science and engineering workﬂows, which includes BigQuery, Airbyte and more.

3.3 SETTINGS

We synthesize one separate set of environment-speciﬁc data for each evaluated benchmark.
Throughout the data synthesis process, we employ the Claude-3.5-sonnet (Anthropic, 2024) as the
generator model and both Gemini-1.5-pro (Reid et al., 2024) and Claude-3.5-sonnet as the LLM
committee for ﬁltering low-quality data. For each document, we sample three task instructions from

5

Under review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Table 2: Comparison of LEARN-BY-INTERACT to other existing training-free approaches. SWE
refers to SWE-bench, Web refers to WebArena and OS refers to OSWorld. The best results are
highlighted in bold. We include more leaderboard results of SWE-bench and WebArena in Table 6.

Benchmark →

SWE Web

OS

Spider2-V SWE Web

OS

Spider2-V

Approach ↓

Gemini-1.5-pro

Claude-3.5-sonnet

Baseline
RAG
Data distill
Reﬂexion
LATS

13.3
13.7
14.0
14.3
15.3

17.9
19.5
19.8
20.2
21.0

4.9
5.1
5.7
5.7
6.5

Learn-by-interact
∆ over baseline

18.7
+5.4

25.6
+7.7

10.3
+5.4

Existing approaches

8.3
9.1
9.1
9.3
11.3

16.4
+8.1

26.7
27.0
28.0
28.3
29.0

31.5
31.8
32.1
32.4
34.2

11.4
11.7
11.9
12.2
13.6

Ours

34.7
+8.0

39.2
+7.7

22.5
+11.1

7.5
7.7
8.5
8.9
10.3

16.3
+8.8

LLMs. The statistics for generated raw trajectories, examples before and after ﬁltering are shown
in Table 1. In Appendix E, we list document sources used for each benchmark. During ICL, we
retrieve examples until the maximum length of LLMs and set an upper bound of 5 for both model-
based and observation-based retrieval (m1 = 5, m2 = 5 in Algorithm 2). We leverage Gemini-1.5-
pro (Reid et al., 2024) and Claude-3.5-sonnet (Anthropic, 2024)1, Codegemma-7B (Team, 2024a)
and Codestral-22B (Team, 2024b) in the ICL evaluation, and tune Codegemma-7B and Codestral-
22B with LoRA (Hu et al., 2021) to evaluate the data quality as training sources. By default, we
do not include retrieval content in evaluating the trained model to avoid the confusion in under-
standing the effectiveness of our synthesized data in training. We include more detailed hyper-
parameter settings (both existing approaches and LEARN-BY-INTERACT) and machine information
in Appendix D.

3.4 EVALUATION

We follow the default evaluation metrics designed by the original benchmarks.
In SWE-
bench (Jimenez et al., 2023), we apply the generated patch program to the repository codebase, and
measure the agent performance by execution accuracy (pass@1). In WebArena (Zhou et al., 2023b),
we employ both LLM-based fuzzy match and string match that checks keywords in predictions.
Slightly different from the original work that uses gpt-4-0613 as the LLM judge, we use Claude-
3.5-sonnet as a similar replacement. In OSWorld (Xie et al., 2024), we leverage the sample-speciﬁc
evaluation scripts to assess the functional correctness of the task completion, which processes en-
vironment states and checks if agents ﬁnish the task as expected. In Spider2-V (Cao et al., 2024),
we utilize ﬁle-based comparison, information-based validation, execution-based veriﬁcation to de-
termine whether a task is successfully completed. All performance numbers throughout the paper
are shown in the percentage of resolved instances with % omitted for brevity.

3.5 RESULTS

3.5.1 TRAINING-FREE EVALUATION

We ﬁrst consider LEARN-BY-INTERACT in the training-free setting, where the proposed methods
can be applied to the commercial LLMs even with prediction-only API access.

Results on Table 2 show marginal improvement of RAG compared to the baseline, which suggests
limited effectiveness by simply concatenating standard reousrces to LLM prompts. By retrieving
examples from distilled data, we observe better performance compared to RAG, but still no more
than 2% improvement over the baseline, which indicates that the distilled data tend to be noisy
in the setting with multi-round agent-environment interactions. This highlights the critical role of

1In the subsequent descriptions, Gemini refers to Gemini-1.5-pro, and Claude refers to Claude-3.5-sonnet.

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 3: Downstream task performance of models trained from data generated by Learning-by-
interact and data distillation. We include the models results before training, where the synthesized
data is used as demonstration examples, and after training, where the synthesized data is used to
train models.

Benchmark →

Web

OS

Web

OS

Web

OS

Web

OS

Model →

Approach ↓

Codegemma-7B Codestral-22B Codegemma-7B Codestral-22B

Before tuning

After tuning

Existing approaches

Baseline
Data distill

3.3
4.2

0.0
0.0

4.7
5.8

2.2
2.7

-
6.2

-
1.4

-
10.2

-
5.4

Learn-by-interact
∆ over baseline

7.6
+4.3

3.5
+3.5

9.9
+5.2

5.4
+3.2

17.9
+14.5

6.5
+6.5

27.8
+23.1

11.7
+9.5

Ours

backward construction, which corrects the misalignment between instructions and trajectories by
curating new task objectives.

Both Reﬂexion and LATS consistently improve over the baseline across 4 benchmarks, which
demonstrate their general applicability to agent tasks. Using the data synthesized from the LEARN-
BY-INTERACT, we can see a signiﬁcant performance gain compared to all other frameworks in both
Gemini and Claude. For example, in OSWorld, augmenting Claude with synthesized environment-
speciﬁc data almost doubles the result compared to the baseline. This signiﬁes the high quality of
the generated data and the effectiveness of the LEARN-BY-INTERACT framework.

3.5.2 TRAINING-BASED EVALUATION

We consider the data synthesized by LEARN-BY-INTERACT in the scenario of LLM tuning, which
is applicable to the LLMs with access to weight updates.

The results presented in Table 3 reveal that LEARN-BY-INTERACT substantially surpasses both the
baseline and data distillation, suggesting its capacity to generate high-quality training data that en-
ables language models to learn and adapt efﬁciently. We discover that utilizing our synthesized data
for model training yields better results compared to using it as in-context learning (ICL) examples.
A notable instance is in WebArena, where Codestral-22B’s performance jumps from 4.7% to 27.8%
when trained on our synthesized data, while only showing a 5.2% improvement in the ICL scenario.
Remarkably, the Codestral-22B model trained with our synthesized data even outperforms Gemini
when the latter uses our data as demonstration examples.

4 ANALYSIS

4.1

INFERENCE EFFICIENCY

We compare the efﬁciency of different pipelines at inference. We analyze the trade-off between
downstream task performance and the required computational costs. We focus on measuring the
number of LLM calls and consumed tokens per example, which are averaged across four evalu-
ated datasets (§3.2) using Claude-3.5-sonnet. As illustrated in Fig. 2, while Reﬂexion and LATS
demonstrate enhanced performance, this comes at the cost of signiﬁcantly increased computational
resources during inference. Speciﬁcally, LATS yields a 2.5% improvement on average, but re-
quires nearly four times used tokens per instance relative to the baseline. In contrast, LEARN-BY-
INTERACT exhibits superior performance while utilizing fewer LLM calls and slightly more tokens
compared to the baseline. Thanks to the rich environment information stored in the examples of
synthesized data, LLMs can potentially make better decisions and thus ﬁnish the task in fewer steps.
This removes the performance-efﬁciency trade-off during inference at the cost of data synthesis in

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Figure 2: Evaluation performance, the number of LLM calls and consumed tokens (per example)
of various training-free pipelines during inference, which are all averaged across four benchmarks:
SWE-bench, Webarena, OSWorld and Spider2-V.

Table 4: Model performance based on different retrieval paradigms. Observation-based and Model-
based retrieval prove to be particularly effective in agent tasks, whose combination (ours) gives the
best results.

Benchmark →

SWE Web OS

Spider2-V SWE Web OS

Spider2-V

Retrieval ↓

Gemini-1.5-pro

Claude-3.5-sonnet

No retrieval
Instruction-based
Observation-based
Model-based
Ours

13.3
14.7
16.3
17.0
18.7

17.9
21.6
23.5
24.3
25.6

4.9
7.0
8.7
9.5
10.3

8.3
10.2
14.6
15.4
16.4

26.7
27.7
32.3
33.7
34.7

31.5
33.6
36.3
37.2
39.2

11.4
15.7
18.7
20.3
22.5

7.5
9.1
13.2
14.5
16.3

advance and suggests that LEARN-BY-INTERACT is particularly well-suited for real-world deploy-
ment that demonds both low latency and high performance.

4.2 THE IMPACT OF RETRIEVAL

As mentioned in §2.4, we employ both model-based and observation-based retrieval in our evalu-
ation with ICL. We analyze their effectiveness by incorporating only one of them (skip line 5-14
in Algorithm 2 for model-based retrieval only and skip line 15-17 for observation-based retrieval
only). In addition, we compare to two baselines: (1). no retrieval: LLMs predict each action in the
zero-shot setting; (2). instruction-based: only use instructions to retrieve synthesized data and apply
the same demonstration examples in every action prediction throughout the trajectory.

The results presented in Table 4 illustrate how various retrieval methods impact LLMs when us-
ing the synthetic data as the retrieval source. Despite having access to the same example pool
(except the baseline without using retrieval), there are notable differences in performance across
different retrieval strategies, highlighting the crucial role of agentic retrieval in effectively utilizing
synthesized data. Traditional Retrieval-Augmented Generation (RAG) methods, which only em-
ploys instructions for retrieval, show the least improvement across four benchmarks and two LLMs.
In contrast, the observation-based approach proves particularly effective for agent-based tasks, sig-
niﬁcantly outperforming the instruction-based retrieval, for instance, achieving a 4.4% absolute im-
provement in Spider-2V when using Gemini. By leveraging task instructions, interaction history and
the current observation, model-based retrieval demonstrates even better results compared to using
the observation-based version. Ultimately, the most impressive scores are achieved by combining
both model-based and observation-based retrieval, which results in our agentic retrieval pipeline.
These ﬁndings underscore the importance of carefully designing retrieval pipelines to maximize the
potential of synthetic data and LLMs in agent scenarios.

4.3 DATA GRANULARITY

8

192123252729Performance510152025303540LLM calls50k100k150k200k250kConsumed tokensBaselineRAGData distillReflexionLATSLearn-by-interactUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 5: Effectiveness of synthetic data with various granularity. In general, short-trajectory data is
more advantageous to both training and ICL, while mixing all of short, medium and long-trajectory
data provides the best performance.
Benchmark →

SWE Web OS

Spider2-V Web

OS

Granularity ↓

Claude-3.5-sonnet

Codestral-22B

Baseline
Short
Medium
Long
Short+Medium
Short+Long
Medium+Long
Short+Medium+Long

26.7
28.7
28.0
27.3
30.0
29.3
28.7
31.0

31.5
33.3
32.5
31.9
34.4
33.9
32.9
34.9

11.4
14.9
13.8
13.0
15.7
15.2
14.4
16.3

7.7
10.3
9.5
8.9
10.7
10.5
10.1
11.3

4.6
13.5
12.6
10.6
14.6
14.4
13.2
15.4

2.2
4.9
4.0
3.4
5.7
5.3
4.5
6.3

As mentioned in §2.2, we synthesize data by
taking contiguous sub-trajectories from the full
generation paths of LLMs, i.e. T (cid:48) = T [i : j],
which results in trajectories of diverse lengths
in the synthesized data. We divide the syn-
trajec-
thetic data into three groups:
(1).
5 ≤ trajec-
tory steps < 5 (short);
(2).
tory steps < 10 (medium); (3).
trajectory
steps ≥ 10 (long), and leverage each group
and their combinations in both the training-
free and the training-based process. To en-
sure a fair comparison, we constraint the data
size in each group and combined group to
200M tokens2, utilizing Su et al. (2022) for
sub-sampling. Table 5 presents the results.
In both training-free and training-based eval-
uation, LLMs derive greater advantages from
short-trajectory data, as demonstrated by its
consistently superior performance compared to
medium and long-trajectory data with Claude-
3.5-sonnet and Codestral-22B. This can be at-
tributed to the versatility of short-trajectory data, which usually serves as a sub-step or a partial
workﬂow in downstream tasks. The combination of any two data groups proves more effective than
relying on a single group, showcasing the complementary nature of diverse data sets. For instance,
in Webarena with Codestral-22B, incorporating examples with both short and medium-length tra-
jectories shows additional improvement over using either one exclusively (14.6 vs 13.5 and 14.6 vs
12.6). This underscores the value of considering the trajectory length as a unique dimension of agent
data synthesis.

Figure 3: Scaling laws of the synthesized data.
Compared to in-context learning, tuned models
achieves more signiﬁcant improvements as the
data scales up. The performance is averaged
across WebArena and OSWorld.

4.4 SCALING LAWS

We examine how the model performance improves as the synthetic data size scales up. Figure 3
presents two sets of results, with training-free (where Claude, Gemini, Codegemma and Codestral
use retrieval augmentation without training) and with training-based (where ﬁne-tuned Codegemma
and Codestral models are evaluated without retrieval). All results are averaged across Webarena
and OSworld due to the limit of computational resources. The ﬁndings indicate that both learning
paradigms beneﬁt from larger data, suggesting the synthetic data is diverse and high-quality. In
the training-free evaluation, more substantial improvements are observed for larger models (Claude
and Gemini) compared to smaller ones (Codegemma and Codestral), possibly due to the enhanced

2We use the number of tokens to measure the data size due to the fact that long-trajectory example may

contain more information compared to the short version.

9

020k40k60k80k100kSynthesized data size051015202530PerformanceClaude-3.5-sonnetCodegemma-7BCodegemma-7B-trainedGemini-1.5-proCodestral-22BCodestral-22B-trainedUnder review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

in-context learning abilities of larger models. Our analysis also reveals that for a given amount of
synthetic data, ﬁne-tuning smaller models is more effective than using the data as demonstration
examples during evaluation.

5 RELATED WORK

Various agents based on LLMs have been developed (Wang et al., 2024; Zhang et al., 2024; Shinn
et al., 2024; Huang et al., 2022; Wang et al., 2023a;b). React (Yao et al., 2022b) proposes to
synergize reasoning and acting in LLMs. By integrating Monte Carlo Tree Search (Kocsis &
Szepesvári, 2006; Coulom, 2006), Zhou et al. (2023a) leverages LLM-powered value functions
and self-reﬂection (Madaan et al., 2024) to encourage proﬁcient exploration and decision-making.
However, it comes with increased computational costs and relies on the premise that the environ-
ment allows for state reversals. In contrast, LEARN-BY-INTERACT removes such assumptions and
improves both agent efﬁciency and performance by synthesizing high-quality data in advance.

Another line of research to improve agent models relies on training on human-labeled exam-
ples (Zeng et al., 2023; Yin et al., 2023; Deng et al., 2024; Chen et al., 2024b; Wang et al., 2022a)
or data distilled from LLMs like GPT-4 (Chen et al., 2023; Zhao et al., 2024). AgentGen (Hu et al.,
2024) explores automatic synthesis of both environments and tasks and then leverages FastDown-
ward3 to generate trajectory data. AgentTuning (Zeng et al., 2023) utilizes both existing datasets
and self-instruct (Wang et al., 2022b) to derive instructions and then samples trajectories from GPT-
4 (Achiam et al., 2023). In contrast, LEARN-BY-INTERACT focuses on realistic environments and
generate tasks and trajectories using backward construction. Some other researchers are also ex-
ploring ways to use data more efﬁciently with reinforcement learning (Ball et al., 2023; Schwarzer
et al., 2020; Nachum et al., 2018; Thomas & Brunskill, 2016; Schwarzer et al., 2021). Gulcehre et al.
(2023) suggests using data created by an LLM’s policy can enhance itself via ofﬂine reinforcement
learning algorithms. Aksitov et al. (2023) takes this further by combining with ReAct (Yao et al.,
2022b) to train agent models iteratively on experience trajectories. These typically require a reward
model as the scoring function or LLM/execution-generated feedback to enhance data quality. Our
work, however, takes a different approach by employing the backward construction to improve the
data quality by aligning instructions and trajectories.

6 CONCLUSION

We introduce LEARN-BY-INTERACT, a data-centric framework to adapt LLM agents to any given
environments without human annotations. Based on commonly-accessible resources like documen-
taion, LLMs propose downstream tasks and complete them with multi-round interactions with en-
vironments. We address the misalignment between instructions and trajectories by updating objec-
tives with new instructions derived from trajectories. Additionally, we design innovative retrieval
pipelines that leverage agent instructions, interaction histories, and current observations to retrieve
synthesized examples. Through extensive experiments, we demonstrate that the synthetic data from
LEARN-BY-INTERACT signiﬁcantly enhances model performance in ICL and training. Compared
with other leading approaches in agent tasks, LEARN-BY-INTERACT shows much better perfor-
mance with lower latency and computational costs, which make it particularly suitable for large-
scale deployment. Further analysis has also shown the superiority of LEARN-BY-INTERACT over
the classical RAG. In future work, we plan to explore multi-modal settings and train general agent
models widely applicable in realistic environments. We anticipate that LEARN-BY-INTERACT will
inspire future research to push the state-of-the-art in this direction.

7 LIMITATIONS

Although LEARN-BY-INTERACT effectively synthesizes high-quality agentic data with trajectories,
it requires a lot of LLM calls in generation and ﬁltering. We hope that future works will explore more
efﬁcient approaches to complete annotations without sacriﬁcing quality. Additionally, LEARN-BY-
INTERACT leverages the environment-related resources to generate instructions. In some scenarios,
however, these resources may be incomplete or not available.

3https://www.fast-downward.org/

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

Renat Aksitov, Sobhan Miryooseﬁ, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu,
Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. Rest meets react: Self-
improvement for multi-step reasoning llm agent. arXiv preprint arXiv:2312.10003, 2023.

Anthropic.

Introducing claude 3.5 sonnet, 2024. URL https://www.anthropic.com/

news/claude-3-5-sonnet.

Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efﬁcient online reinforcement learn-
ing with ofﬂine data. In International Conference on Machine Learning, pp. 1577–1594. PMLR,
2023.

Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang
Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, et al. Spider2-v: How far are mul-
arXiv preprint
timodal agents from automating data science and engineering workﬂows?
arXiv:2407.10956, 2024.

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao.

Fireact: Toward language agent ﬁne-tuning. arXiv preprint arXiv:2310.05915, 2023.

Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun
Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. Coder: Issue resolving with multi-agent and
task graphs. arXiv preprint arXiv:2406.01304, 2024a.

Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and
Feng Zhao. Agent-ﬂan: Designing data and methods of effective agent tuning for large language
models. arXiv preprint arXiv:2403.12881, 2024b.

Rémi Coulom. Efﬁcient selectivity and backup operators in monte-carlo tree search. In International

conference on computers and games, pp. 72–83. Springer, 2006.

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su.
Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing
Systems, 36, 2024.

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek
Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training
(rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023.

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and
Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro-
gram synthesis. arXiv preprint arXiv:2307.12856, 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang,
arXiv preprint

and Weizhu Chen. Lora: Low-rank adaptation of large language models.
arXiv:2106.09685, 2021.

Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, Saravan
Rajmohan, and Dongmei Zhang. Agentgen: Enhancing planning abilities for large language
model based agent via environment and task generation. arXiv preprint arXiv:2408.00764, 2024.

Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot
planners: Extracting actionable knowledge for embodied agents. In International conference on
machine learning, pp. 9118–9147. PMLR, 2022.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Oﬁr Press, and Karthik
Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint
arXiv:2310.06770, 2023.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Azarakhsh Keipour. Physical interaction and manipulation of the environment using aerial robots.

arXiv preprint arXiv:2207.02856, 2022.

Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference

on machine learning, pp. 282–293. Springer, 2006.

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham
Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating
multimodal agents on realistic visual web tasks. arXiv e-prints, pp. arXiv–2401, 2024.

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language

instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776, 2020.

Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye, Alexander R Fabbri, Pengfei Liu, Dragomir
Radev, and Arman Cohan. On learning to summarize with large language models as references.
arXiv preprint arXiv:2305.14239, 2023.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri
Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-reﬁne: Iterative reﬁnement
with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.

Oﬁr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efﬁcient hierarchical

reinforcement learning. Advances in neural information processing systems, 31, 2018.

Xiao Pu, Mingqi Gao, and Xiaojun Wan.

Summarization is (almost) dead.

arXiv preprint

arXiv:2309.09558, 2023.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-
baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem-
ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint
arXiv:2403.05530, 2024.

Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach-
man. Data-efﬁcient reinforcement learning with self-predictive representations. arXiv preprint
arXiv:2007.05929, 2020.

Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De-
von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efﬁcient
reinforcement learning. Advances in Neural Information Processing Systems, 34:12686–12699,
2021.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reﬂexion:
Language agents with verbal reinforcement learning. Advances in Neural Information Processing
Systems, 36, 2024.

Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked llm policies for web

actions. In First Conference on Language Modeling, 2024.

Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari
Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models
better few-shot learners. arXiv preprint arXiv:2209.01975, 2022.

CodeGemma Team.

Codegemma: Open code models based on gemma.

arXiv preprint

arXiv:2406.11409, 2024a.

The Mistral AI Team. Codestral: Hello, world!, 2024b. URL https://mistral.ai/news/

codestral/.

Philip Thomas and Emma Brunskill. Data-efﬁcient off-policy policy evaluation for reinforcement

learning. In International Conference on Machine Learning, pp. 2139–2148. PMLR, 2016.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan,
and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.
arXiv preprint arXiv:2305.16291, 2023a.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is

your agent smarter than a 5th grader? arXiv e-prints, pp. arXiv–2203, 2022a.

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe-

cutable code actions elicit better llm agents. arXiv preprint arXiv:2402.01030, 2024.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions.
arXiv preprint arXiv:2212.10560, 2022b.

Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe,
explain, plan and select: Interactive planning with large language models enables open-world
multi-task agents. arXiv preprint arXiv:2302.01560, 2023b.

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying

llm-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024.

Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Jun-
ning Zhao, Qian Liu, Che Liu, et al. Openagents: An open platform for language agents in the
wild. arXiv preprint arXiv:2310.10634, 2023.

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing
Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal
agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972,
2024.

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan,
and Oﬁr Press. Swe-agent: Agent-computer interfaces enable automated software engineering.
arXiv preprint arXiv:2405.15793, 2024.

Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent:

Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable
real-world web interaction with grounded language agents. Advances in Neural Information Pro-
cessing Systems, 35:20744–20757, 2022a.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629,
2022b.

Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and
Bill Yuchen Lin. Lumos: Learning agents with uniﬁed data, modular design, and open-source
llms. arXiv preprint arXiv:2311.05657, 2023.

Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai
Qi, Hao Yu, Lei Yu, et al. Swe-bench-java: A github issue resolving benchmark for java. arXiv
preprint arXiv:2408.14354, 2024.

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttun-

ing: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023.

Zhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal chain-of-action agents.

arXiv preprint arXiv:2309.11436, 2023.

Jiwen Zhang, Yaqi Yu, Minghui Liao, Wentao Li, Jihao Wu, and Zhongyu Wei. Ui-hawk: Unleash-

ing the screen stream understanding for gui agents. arXiv preprint, 2024.

Zhonghan Zhao, Ke Ma, Wenhao Chai, Xuan Wang, Kewei Chen, Dongxu Guo, Yanting Zhang,
Hongwei Wang, and Gaoang Wang. Do we really need a complex agent system? distill embodied
agent into a single model. arXiv preprint arXiv:2404.04619, 2024.

Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Lan-
guage agent tree search uniﬁes reasoning acting and planning in language models. arXiv preprint
arXiv:2310.04406, 2023a.

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Table 6: Top-10 results of SWE-bench from the leaderboard at https://www.swebench.
com/. All the numbers are fetched on 2024-10-01.

Approach ↓
site
CodeStory Aide + Mixed Models https://www.swebench.com/
Honeycomb
AbanteAI MentatBot
Gru
Isoform
SuperCoder2.0
MarsCode
Lingma
Factory Code Droid
AutoCodeRover
LEARN-BY-INTERACT (ours)

https://honeycomb.sh/
https://mentat.ai/blog/mentatbot-sota-coding-agent
https://gru.ai/
https://www.isoform.ai/
https://superagi.com/supercoder/
https://www.marscode.com/
https://arxiv.org/abs/2406.01422
https://www.factory.ai/
https://autocoderover.dev/
This paper

result

43.0
38.3
38.0
35.7
35.0
34.0
34.0
33.0
31.3
30.7
34.7

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng,
Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build-
ing autonomous agents. arXiv preprint arXiv:2307.13854, 2023b.

A BASELINE IMPLEMENTATIONS

We follow the existing frameworks to set up baselines in each benchmark. In SWE-bench (Jimenez
et al., 2023), we follow the prompt styles of the Agentless (Xia et al., 2024) pipeline to ﬁrst localize
suspicious ﬁles, then ﬁnd classes and functions to edit. In WebArena (Zhou et al., 2023b), we follow
the implementation of Step (Sodhi et al., 2024), which concatenates task objectives, action space
descriptions, general instructions (e.g., output formats) and webpage observations in the prompt,
and ask LMs to predict the next action. By default, we use the accessibility tree4 as the observation
space.
In OSWorld (Xie et al., 2024) and Spider2-V (Cao et al., 2024), we follow the original
prompt style designed by the benchmark, which also concatenates task objectives, action space
descriptions, general instructions and computer observations in the prompt. By default, we use the
accessibility tree as the observation space for OSWorld, and use the set-of-mark for Spider2-V due
to the signiﬁcant information loss of the accessibility tree in the original benchmark. See an example
in Table 22 and 23 for more details.

B COMPARISON TO TASK-SPECIFIC APPROACHES

In Table 6, we compare LEARN-BY-INTERACT to top-10 task-speciﬁc approaches (with
open-sourced code) that may not broadly applied in agent scenarios for SWE-bench (Zan
the information is retrieved
et al., 2024) and WebArena (Zhou et al., 2023b).
leaderboard https://www.swebench.com/ and
on 2024-10-01 from the ofﬁcial
https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC_
pF7LdyGU1f_ufZb_NWNBZQ/edit?gid=0#gid=0. To the best of our knowledge, we
are the ﬁrst to apply our methods in OSWorld (Xie et al., 2024) and Spider2-V (Cao et al., 2024).

All

C DATASET EXAMPLES

From Table 8 to 21, we provide one example for each dataset with full instructions, interaction
history with the environment.

D EXPERIMENTAL SETTINGS

We retrieve documents until the maximum length of LLMs for RAG and set an upper bound number
of 50 documents, where the retrieved documents remain unchanged throughout agent interaction
trajectory because only instructions are used as the query for retrieval. For Reﬂexion (Shinn et al.,

4https://developer.mozilla.org/en-US/docs/Glossary/Accessibility_tree

14

Under review as a conference paper at ICLR 2025

Table 7: Top-10 results of WebArena from the leaderboard at https://docs.google.com/
spreadsheets/d/1M801lEpBbKSNwP-vDBkC_pF7LdyGU1f_ufZb_NWNBZQ/edit?
gid=0#gid=0. All the numbers are fetched on 2024-10-01.

Approach ↓

site

Jace.AI
WebPilot
AWM
Step
BrowserGym
Auto Eval
Tree Search
AutoWebGLM
gpt-4-0613
gpt-4o-2024-05-13
LEARN-BY-INTERACT (ours) This paper

https://www.jace.ai/
https://www.arxiv.org/pdf/2408.15978
https://arxiv.org/pdf/2409.07429
https://arxiv.org/abs/2310.03720
https://github.com/ServiceNow/BrowserGym
https://arxiv.org/abs/2404.06474
https://jykoh.com/search-agents
https://arxiv.org/abs/2404.03648
https://arxiv.org/abs/2307.13854
https://arxiv.org/abs/2307.13854

result

57.1
37.2
35.5
33.5
23.5
20.2
19.2
18.2
14.9
13.1
39.2

2024), we use the maximum trials 3. In LATS (Zhou et al., 2023a), we use the number of gener-
ated action 5, depth limit 15, value function weight 0.8, following the original setting in paper with
WebShop (Yao et al., 2022a), which is also an agent task based on website. By default, we use
https://huggingface.co/dunzhang/stella_en_1.5B_v5 as the retriever for model-based retrieval con-
sidering both size and the performance. We use the temperature 0 throughout the experiments to
ensure better reproductivity of the experiments. During training, we the batch size 128, learning
rate 0.00002, warmup ratio 0.03 and maximum length 8192, and tune the model for 3 epochs. All
experiments are conducted in H100 machines with 80GB memeory.

E DOCUMENT SOURCES

We use all the non-repeated python ﬁles in SWE-bench-Lite (Jimenez et al., 2023) as the document
sources. Although we may not always ﬁnd abundant documentations and tutorials for each envi-
ronment, we believe that documentations in the same domain still have a good coverage of frequent
operations. For example, one subset of WebArena (Zhou et al., 2023b) focuses on the navigation
of the shopping website OneStopMarket, we use the Amazon documentation as a good replace-
ment. Regardless of the shopping websites, the frequent tasks usually include order change, product
search, delivery checking, etc. Therefore, we use other documentations in the same domain to sam-
ple task instructions when the exact version for the target environment is not available. Concretely,
we use the following sources for WebArena:

• https://docs.gitlab.com/ee/tutorials/
• https://support.google.com/maps
• https://www.amazon.com/hz/contact-us/foresight/hubgateway
• https://support.reddithelp.com/hc/en-us/articles

The following sources are used for OSWorld:

• https://support.google.com/chrome/?hl=en
• https://www.gimp.org/tutorials/
• https://books.libreofﬁce.org/en/CG72/CG72.html
• https://books.libreofﬁce.org/en/WG73/WG73.html
• https://ubuntu.com/tutorials/command-line-for-beginners
• https://support.mozilla.org/en-US/products/thunderbird
• https://wiki.videolan.org/Documentation:Documentation
• https://code.visualstudio.com/docs

, The following sources are used for Spider2-V:

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

• https://docs.getdbt.com/

• https://release-1-7-2.dagster.dagster-docs.io/

• https://docs.astronomer.io/

• https://docs.airbyte.com/

• https://airbyte.com/tutorials/

• https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html

• https://superset.apache.org/docs/

• https://www.metabase.com/docs/v0.49/

• https://www.metabase.com/learn/

• https://docs.snowﬂake.com/en/

• https://cloud.google.com/bigquery/docs/

• https://jupyterlab.readthedocs.io/en/4.1.x/

F SYNTHESIZED DATA EXAMPLES

From Table 24 to 30, we provide a complete example of data synthesis. To begin with, an LLM
generates instructions based on standard resources like tutorials, documentations and FAQs: Upload
CSV data in Google Drive to BigQuery. (See prompt in Table 33) It then attempts solve the task by
predicting actions and collecting feedback from environments (interactions). This produces a long
trajectory showing how LLMs try to achieve the goal.

However, it is not guaranteed that the trajectory successfully achieves the target. In our example, the
LLM makes a wrong prediction in the action 4. It selects the table source Google Cloud Storage,
while the correct action should select “Drive" to align with the instruction that reuiqres to upload
CSV data in Google Drive. This results in wrong actions in the subsequent predictions, and the
generated trajectory is not aligned with the initial instruction, which leads to noisy data in this case.

Instead of using the original instruction-trajectory pairs for downstream training and in-context
learning, we ﬁx the mentioned misalignment by crafting new instructions for each sub-trajectory
(backward construction). Concretely, we feed the generated trajectory into LLM prompts, and ask
it to summarize the trajectory or propose a new task based on it. For example, the LLM updates the
task objective to “Link CSV ﬁle in Google Cloud Storage to BigQuery" after observing the trajec-
tory, which makes the task instrucion and the trajectory aligned. Additionally, we also generate new
instructions for each sub-trajectory, which would increase the utility of a generated full trajectory.
For instance, based on the sub-trajectory (observation 0, Action 1, observation 1), the LLM gener-
ates a new instruction: When is dataset “demo" created? In Table 31 and 32, we list more generated
instructions based on sub-trajectories.

G CASE STUDY ON FILTERED EXAMPLES

In Table 40-49, we demonstrate the representative synthesized examples that fail to meet our de-
signed criteria. The example in Table 40-45 is ﬁltered because the trajectory shows detour in ac-
complishing the goal, i.e. Action 1-6 are not necessary. The example in Table 46-49 is ﬁltered
because it goes back and forth in states, i.e. repeat the actions of clicking "My Orders" and click-
ing "View Order". We ﬁlter these low-quality examples to avoid their negative inﬂuences in the
downstream applications.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

Table 8: SWE-bench example

Instruction

Please address the following Github issue.

### GitHub Problem Description ###
ModelChoiceIteratorValue is not hashable.
Description

Recently I migrated from Django 3.0 to Django 3.1. In my code, I add custom data-* at-
tributes to the select widget options. After the upgrade some of those options broke. Error is
TypeErrorunhashable type: ’ModelChoiceIteratorValue’.
Example (this one breaks):

def create_option(self, name, value, label, selected, index, subindex=None, attrs=None):
context = super().create_option(name, value, label, selected, index, subindex, attrs)
if not value:

return context

if value in self.show_ﬁelds: # This is a dict 1: [’ﬁrst_name’, ’last_name’]
context[’attrs’][’data-ﬁelds’] = json.dumps(self.show_ﬁelds[value])

However, working with arrays is not an issue:

def create_option(self, name, value, label, selected, index, subindex=None, attrs=None):
context = super().create_option(name, value, label, selected, index, subindex, attrs)
if not value:

return context

if value in allowed_values: # This is an array [1, 2]

...

Observation 0 (repository structure)

django/

setup.py
__init__.py
__main__.py
shortcuts.py
forms/

__init__.py
utils.py
ﬁelds.py
renderers.py
forms.py
formsets.py
widgets.py
boundﬁeld.py
models.py

db/

__init__.py
transaction.py
utils.py
models/

signals.py
constants.py
ﬁelds/

ﬁles.py
__init__.py
related.py

...

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Action 1 (Find related ﬁles to edit)

Table 9: SWE-bench example cont.

Based on the problem description and the repository structure, here are the most likely ﬁles
that would need to be edited to ﬁx the issue:

django/forms/models.py
django/forms/ﬁelds.py
django/forms/widgets.py
django/forms/__init__.py
django/db/models/ﬁelds/__init__.py
django/db/models/ﬁelds/related.py
...

Observation 1 (File skeleton)

### File: django/forms/models.py
def construct_instance(form, instance, ﬁelds=None, exclude=None):

...

def model_to_dict(instance, ﬁelds=None, exclude=None):

...

class ModelChoiceIteratorValue:

def __init__(self, value, instance):

...

def __str__(self):

...

def __eq__(self, other):

...

class ModelChoiceField(ChoiceField):

def _get_choices(self):

...

### File: django/forms/ﬁelds.py
class Field:

widget = TextInput # Default widget to use when rendering this type of Field.
hidden_widget = HiddenInput # Default widget to use when rendering this as "hidden"

def __init__(self, *, required=True, widget=None, label=None, initial=None):
...
def prepare_value(self, value):
...
def to_python(self, value):
...
def validate(self, value):
...
def run_validators(self, value):
...

class ChoiceField(Field):

def validate(self, value):

...

def valid_value(self, value):

...

...

18

Under review as a conference paper at ICLR 2025

Action 2 (Find related classes, methods to edit)

Table 10: SWE-bench example cont.

Here are the identiﬁed locations that need inspection or editing to ﬁx the problem:

django/forms/models.py
class: ModelChoiceIteratorValue

django/forms/models.py
class: ModelChoiceField
method: _get_choices

django/forms/ﬁelds.py
class: ChoiceField
method: valid_value
...

Observation 2 (Classes and methods content)

### File django/forms/models.py
class ModelChoiceIteratorValue:

def __init__(self, value, instance):

self.value = value
self.instance = instance

def __str__(self):

return str(self.value)

def __eq__(self, other):

if isinstance(other, ModelChoiceIteratorValue):

other = other.value
return self.value == other

class ModelChoiceField(ChoiceField):

def _get_choices(self):

return self.iterator(self)

### django/forms/ﬁelds.py
class ChoiceField(Field):

def valid_value(self, value):
text_value = str(value)
for k, v in self.choices:

if isinstance(v, (list, tuple)):

# This is an optgroup, so look inside the group for options
for k2, v2 in v:

if value == k2 or text_value == str(k2):

return True

else:

if value == k or text_value == str(k):

return True

return False

...

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

Action 3 (Edit code to address the issue)

Table 11: SWE-bench example cont.

Based on the issue description and the provided code segment, it appears that the ‘Model-
ChoiceIteratorValue‘ class is not hashable, which is causing the TypeError when trying to
use it as a dictionary key. To ﬁx this, we need to make the ‘ModelChoiceIteratorValue’ class
hashable by implementing the ‘__hash__’ method. Here’s the proposed ﬁx:

### django/forms/models.py
### Replace the following code
class ModelChoiceIteratorValue:

def __init__(self, value, instance):

self.value = value
self.instance = instance

def __str__(self):

return str(self.value)

def __eq__(self, other):

if isinstance(other, ModelChoiceIteratorValue):

other = other.value
return self.value == other

### with

class ModelChoiceIteratorValue:

def __init__(self, value, instance):

self.value = value
self.instance = instance

def __str__(self):

return str(self.value)

def __eq__(self, other):

if isinstance(other, ModelChoiceIteratorValue):

other = other.value
return self.value == other

def __hash__(self):

return hash(self.value)

Observation 3 (Generated patch)

diff –git a/django/forms/models.py b/django/forms/models.py
index 5a30801..55b3c81 100644
- - - a/django/forms/models.py
+++ b/django/forms/models.py
@@ -1171,6 +1171,9 @@ class ModelChoiceIteratorValue:

other = other.value
return self.value == other

+ def __hash__(self):
+ return hash(self.value)
+

class ModelChoiceIterator:
def __init__(self, ﬁeld):

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

Table 12: Webarena example

Instruction

Tell me the total cost of my latest cancelled order?

Observation 0

Action 1

click [4918] // click the button ‘My Account’

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

Observation 1

Table 13: Webarena example cont.

Action 2

Stop: The total cost of the latest cancelled order is $365.42

22

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Under review as a conference paper at ICLR 2025

Table 14: OSWorld example

Instruction

Could you assist me in adding a new layer and naming it ‘Square’?

Observation 0 (Interface of the software GIMP)

Action 1

import pyautogui
pyautogui.hotkey(‘shift’, ‘ctrl’, ‘n’) // shortcut to initialize a new layer.

Observation 1 (Interface of the software GIMP)

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Under review as a conference paper at ICLR 2025

Table 15: OSWorld example cont.

Action 2

import pyautogui
pyautogui.typewrite(‘Square’) // change the layer name to Square.

Observation 2 (Interface of the software GIMP)

Action 3

import pyautogui
pyautogui.click(1096, 892) // click OK to save the change.

Observation 3 (Interface of the software GIMP)

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

Instruction

Table 16: Spider2-V example, cont.

I have established a connection from Faker to local .csv ﬁle. Could you help me change the
running schedule? I hope it can be replicated at 18:00 pm every day.

Observation 0 (Interface of the software Airbyte)

Action 1

import pyautogui
pyautogui.click(550,280) // click the connection row with the name “Sample Data (Faker) →
Local CSV"

Observation 1 (Interface of the software Airbyte)

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

25

Under review as a conference paper at ICLR 2025

Table 17: Spider2-V example, cont.

Action 2

import pyautogui
pyautogui.click(500,300) // click the button Replication

Observation 2 (Interface of the software Airbyte)

Action 3

import pyautogui
pyautogui.click(500,450) // click the panel Conﬁguration

Observation 3 (Interface of the software Airbyte)

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

Action 4

Table 18: Spider2-V example, cont.

import pyautogui
pyautogui.click(1270,430) // reveal the dropdown menu of the schedule type

Observation 4 (Interface of the software Airbyte)

Action 5

import pyautogui
pyautogui.click(1200,565) // select the schedule type Cron

Observation 5 (Interface of the software Airbyte)

27

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Under review as a conference paper at ICLR 2025

Table 19: Spider2-V example, cont.

Action 6

import pyautogui
pyautogui.click(1020,480) // focus on the content of the Cron expression

Observation 6 (Interface of the software Airbyte)

Action 7

import pyautogui
pyautogui.hotkey(“ctrl",“a") // select all content in the textbox

Observation 7 (Interface of the software Airbyte)

28

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Under review as a conference paper at ICLR 2025

Table 20: Spider2-V example, cont.

Action 8

import pyautogui
pyautogui.press(“delete") // delete the default content

Observation 8 (Interface of the software Airbyte)

Action 9

import pyautogui
pyautogui.write(“0 0 18 * * ?") // update the content to align 18:00 pm in the instruction

Observation 9 (Interface of the software Airbyte)

29

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Under review as a conference paper at ICLR 2025

Table 21: Spider2-V example, cont.

Action 10

import pyautogui
pyautogui.click(1450,900) // click the button save changes

Observation 10 (Interface of the software Airbyte)

30

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Under review as a conference paper at ICLR 2025

Screenshot

Table 22: Observation space of Spider2-V.

Set-of-mark

31

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Under review as a conference paper at ICLR 2025

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Table 23: Observation space of Spider2-V. The accessibility tree suffers from signiﬁcant information
loss. Compared to the screenshot and set-of-mark shown in Table 22, the presented accessibility
tree fails to retrieve webpage information, and only shows the details of the desktop icons in the left
panel.

[208, 13] menu Chromium Web Browser “"
[1463, 13] menu System “"
[35, 65] push-button Chromium Web Browser “"
[753, 81] label Please download waiting software updates. “"
[135, 109] label Home
[35, 133] push-button Terminal “"
[35, 201] push-button Visual Studio Code “"
[35, 269] push-button Files “"
[35, 337] push-button Text Editor “"
[953, 370] label Updated software is available for this computer. Do you want to install it
now?
[35, 405] push-button LibreOfﬁce Calc “"
[951, 463] table-cell Security updates
[1191, 463] table-cell 638.8 MB
[35, 473] push-button LibreOfﬁce Writer “"
[963, 486] table-cell LibreOfﬁce
[1191, 486] table-cell 23.4 MB
[963, 509] table-cell LibreOfﬁce Calc
[1191, 509] table-cell 8.7 MB
[923, 524] toggle-button Details of updates “"
[963, 532] table-cell LibreOfﬁce Draw
[1191, 532] table-cell 3.0 MB
[35, 541] push-button Document Viewer “"
[963, 555] table-cell LibreOfﬁce Impress
[1191, 555] table-cell 1.3 MB
[963, 578] table-cell LibreOfﬁce Math
[1191, 578] table-cell 673 kB
[35, 612] push-button Software Updater “"
[935, 660] label 1157.8 MB will be downloaded.
[35, 680] push-button Trash “"
[671, 702] push-button Settings. . . “"
[1054, 702] push-button Cancel “"
[1176, 702] push-button Install Now “"
[35, 884] toggle-button Show Applications “"

32

Under review as a conference paper at ICLR 2025

Table 24: Example of data synthesis - Bigquery

Instruction

Upload CSV data in Google Drive to BigQuery.

Observation 0 (Bigquery Interface)

Action 1

import pyautogui
pyautogui.doubleClick(332,447) // double click the dataset demo.

Observation 1 (Bigquery Interface)

33

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

Under review as a conference paper at ICLR 2025

Action 2

Table 25: Example of data synthesis - Bigquery

import pyautogui
pyautogui.doubleClick(1002,321) // double click the button CREATE TABLE.

Observation 2 (Bigquery Interface)

Action 3

import pyautogui
pyautogui.click(1458,279) // click the dropdown menu to select table source.

Observation 3 (Bigquery Interface)

34

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

Under review as a conference paper at ICLR 2025

Table 26: Example of data synthesis - Bigquery

Action 4

import pyautogui
// wrong prediction: click the table source Google Cloud Storage.
// The correct prediction should select Drive to align with the instruction.
pyautogui.click(1302,331)

Observation 4 (Bigquery Interface)

Action 5

import pyautogui
pyautogui.click(1389,340) // click BROWSE to ﬁnd ﬁle.

Observation 5 (Bigquery Interface)

35

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

Under review as a conference paper at ICLR 2025

Table 27: Example of data synthesis - Bigquery

Action 6

import pyautogui
pyautogui.click(1341,282) // click to ﬁnd ﬁles under directory.

Observation 6 (Bigquery Interface)

Action 7

import pyautogui
pyautogui.doubleClick(1203,284) // select the CSV ﬁle.

Observation 7 (Bigquery Interface)

36

1890
1891

1892
1893
1894
1895
1896
1897

1898
1899
1900
1901
1902
1903

1904
1905
1906
1907
1908

1909
1910
1911
1912
1913
1914

1915
1916
1917
1918
1919
1920

1921
1922
1923
1924
1925
1926

1927
1928
1929
1930
1931

1932
1933
1934
1935
1936
1937

1938
1939
1940
1941
1942
1943

Under review as a conference paper at ICLR 2025

Table 28: Example of data synthesis - Bigquery

Action 8

import pyautogui
pyautogui.click(1000,915) // click to select ﬁle.

Observation 8 (Bigquery Interface)

Action 9

import pyautogui
pyautogui.click(1000,668) // click to focus on the text box Table.

Observation 9 (Bigquery Interface)

37

1944
1945

1946
1947
1948
1949
1950
1951

1952
1953
1954
1955
1956
1957

1958
1959
1960
1961
1962

1963
1964
1965
1966
1967
1968

1969
1970
1971
1972
1973
1974

1975
1976
1977
1978
1979
1980

1981
1982
1983
1984
1985

1986
1987
1988
1989
1990
1991

1992
1993
1994
1995
1996
1997

Under review as a conference paper at ICLR 2025

Table 29: Example of data synthesis - Bigquery

Action 10

import pyautogui
pyautogui.write("test") // name the ﬁle "test".

Observation 10 (Bigquery Interface)

Action 11

import pyautogui
pyautogui.click(1425,726) // click to reveal dropdown menu of the table type.

Observation 11 (Bigquery Interface)

38

1998
1999

2000
2001
2002
2003
2004
2005

2006
2007
2008
2009
2010
2011

2012
2013
2014
2015
2016

2017
2018
2019
2020
2021
2022

2023
2024
2025
2026
2027
2028

2029
2030
2031
2032
2033
2034

2035
2036
2037
2038
2039

2040
2041
2042
2043
2044
2045

2046
2047
2048
2049
2050
2051

Under review as a conference paper at ICLR 2025

Table 30: Example of data synthesis - Bigquery

Action 12

import pyautogui
pyautogui.click(1297,801) // select the table source external table.

Observation 12 (Bigquery Interface)

Action 13

import pyautogui
pyautogui.click(303,914) // click CREATE TABLE.

Observation 13 (Bigquery Interface)

39

2052
2053

2054
2055
2056
2057
2058
2059

2060
2061
2062
2063
2064
2065

2066
2067
2068
2069
2070

2071
2072
2073
2074
2075
2076

2077
2078
2079
2080
2081
2082

2083
2084
2085
2086
2087
2088

2089
2090
2091
2092
2093

2094
2095
2096
2097
2098
2099

2100
2101
2102
2103
2104
2105

Under review as a conference paper at ICLR 2025

2106
2107

2108
2109
2110
2111
2112
2113

2114
2115
2116
2117
2118
2119

2120
2121
2122
2123
2124

2125
2126
2127
2128
2129
2130

2131
2132
2133
2134
2135
2136

2137
2138
2139
2140
2141
2142

2143
2144
2145
2146
2147

2148
2149
2150
2151
2152
2153

2154
2155
2156
2157
2158
2159

sub-trajectory

Observation 0
↓
Action 1
↓
Observation 1)

Observation 1
↓
Action 2
↓
Observation 2

Observation 4
↓
Action 5
↓
Observation 5
↓
Action 6
↓
Observation 6
↓
Action 7
↓
Observation 7
↓
Action 8
↓
Observation 8

Table 31: Instructions generated from trajectory from Table 24 to 30

type

instruction

New task

When is dataset “demo" created?

Replicate the following: We are currently at the Google Cloud
Console interface, speciﬁcally focused on a BigQuery project.

Replicate trajectory The browser window displays details of a dataset named "demo"

within a BigQuery project. The interface provides information
about the dataset, including its creation date, last modiﬁed time,
data location (US), and other properties like default table expiry
and rounding mode. On the left side of the screen, there’s a
navigation panel showing the Explorer view with the "demo"
dataset selected. The top of the screen shows the Google Cloud
header with project selection and search functionality.
The overall layout is characteristic of a cloud-based data
management platform, with options to create tables, share data,
and manage dataset properties.
After taking the action to click the CREATE TABLE button,
we go to the user interface for creating a table. The screen
displays a form titled "Create table" with various ﬁelds and
options. The source section allows selecting a table to create
from, while the destination section includes ﬁelds for project,
dataset, and table name. There’s also a schema section and
partition and cluster settings. The interface is part of the Google
Cloud Console, as evident from the sidebar on the left showing
different Cloud services and project navigation.

New task

Select test.csv in the bucket test-1616 in Google Cloud Storage
as the table source.

40

Under review as a conference paper at ICLR 2025

Table 32: Instructions generated from trajectory from Table 24 to 30

type

instruction

Replicate trajectory

Replicate the following: We are in the the interface for creating
a table in Google Cloud’s BigQuery service. The page is divided
into several sections. At the top, it indicates the user is creating
a table from a Google Cloud Storage source, with a CSV ﬁle
selected. The destination section shows the project ID and allows
input for the dataset and table name. The destination table is
empty. The table type is set to “Native table". At the bottom,
there’s an option for schema detection, with buttons to create the
table or cancel the operation. The left side of the screen displays a
navigation menu for the Google Cloud Console, including options
like Explorer and various project-related items. The overall layout
suggests this is part of a larger cloud data management and
analysis platform. After we click on the text box Table, we select
and focus on the text box. We then type “test" into the box, which
gives the table a name. Except the textbox we are working on,
the other parts of the webpage has not changed after clicking
and typing.

New task

Link CSV ﬁle in Google Cloud Storage to BigQuery

sub-trajectory

Observation 8
↓
Action 9
↓
Observation 9
↓
Action 10
↓
Observation 10

Observation 0
↓
Action 1
↓
Observation 1
↓
Action 2
↓
......
↓
Observation 13

Table 33: self-instruct prompts to propose instructions based on tutorials, documentations and FAQs.

{Documentation}

Based on the tutorial, examplify 3 tasks that users frequently perform.
User the following format to output:
...
...

41

2160
2161

2162
2163
2164
2165
2166
2167

2168
2169
2170
2171
2172
2173

2174
2175
2176
2177
2178

2179
2180
2181
2182
2183
2184

2185
2186
2187
2188
2189
2190

2191
2192
2193
2194
2195
2196

2197
2198
2199
2200
2201

2202
2203
2204
2205
2206
2207

2208
2209
2210
2211
2212
2213

Under review as a conference paper at ICLR 2025

2214
2215

2216
2217
2218
2219
2220
2221

2222
2223
2224
2225
2226
2227

2228
2229
2230
2231
2232

2233
2234
2235
2236
2237
2238

2239
2240
2241
2242
2243
2244

2245
2246
2247
2248
2249
2250

2251
2252
2253
2254
2255

2256
2257
2258
2259
2260
2261

2262
2263
2264
2265
2266
2267

Table 34: Prompts to summarize (sub-)trajectories or propose new tasks based on the (sub-
)trajectories.
Prompt 1

Below is a trajectory to complete a task.
Observation:
{Observationi}
Action:
{Actioni+1}
Observation:
{Observationi+1}
Action:
{Actioni+2}
...
Action:
{Actionj−1}
Observation:
{Observationj}

Please write a reasonable task instruction that is completed by the trajectory.
Wrap the instruction with ```.

Prompt 2

Below is a trajectory to complete a task.
Observation:
{Observationi}
Action:
{Actioni+1}
Observation:
{Observationi+1}
Action:
{Actioni+2}
...
Action:
{Actionj−1}
Observation:
{Observationj}

Please summarize the trajectory about each observation and changes after each action.
Wrap the summarization with ```.

42

Under review as a conference paper at ICLR 2025

Table 35: LLM prompts to ﬁlter low-quality data

Task instruction:
{instruction}
Below is the trajectory to complete the task.
Observation:
{Observationi}
Action:
{Actioni+1}
Observation:
{Observationi+1}
Action:
{Actioni+2}
...
Action:
{Actionj−1}
Observation:
{Observationj}

Here are the criteria to indicate a good pair of the instruction and the trajectory:
1. The instruction and the trajectory are aligned, which means the trajectory successfully
accomplishes the goal in the instruction.
2. The trajectory is coherent, indicating that each action is logical based on its previous
observation and the actions do not contradict with each other based on the task instruction.
3. The trajectory is natural, meaning that the trajectory closely mimics real-world interactions
and a human user would possibly perform it when engaging in the environment.
4. The trajectory is reasonable, indicating that the trajectory ﬁnishes the task instruction
using a reasonable solution, e.g., not using an over-complicated method, not over-simply the
problem, not going back and forth in states, etc.

Please answer yes if the task instruction and the trajectory satisﬁes all the criteria, otherwise,
asnwer with no.

Table 36: Model inference prompts without external knowledge

SYSTEM MESSAGE:
{system message}
OBJECTIVE:
{task instruction}
INTERACTION HISTORY:
{interaction history}
OBSERVATIONS:
{observations}

Your REASONING and ACTION in the format:
REASON:
Your reason to choose a speciﬁc action.
ACTION:
Your action

43

2268
2269

2270
2271
2272
2273
2274
2275

2276
2277
2278
2279
2280
2281

2282
2283
2284
2285
2286

2287
2288
2289
2290
2291
2292

2293
2294
2295
2296
2297
2298

2299
2300
2301
2302
2303
2304

2305
2306
2307
2308
2309

2310
2311
2312
2313
2314
2315

2316
2317
2318
2319
2320
2321

Under review as a conference paper at ICLR 2025

Table 37: Model inference prompts with external knowledge

SYSTEM MESSAGE:
{system message}
ADDITIONAL INFORMATION FOR REFERENCE:
{external knowledge}
OBJECTIVE:
{task instruction}
INTERACTION HISTORY:
{interaction history}
OBSERVATIONS:
{observations}

Your REASONING and ACTION in the format:
REASON:
Your reason to choose a speciﬁc action.
ACTION:
Your action

Table 38: Expected model outputs

REASON:
...
ACTION:
...

Table 39: Model prompts to write query for retrieval

SYSTEM MESSAGE:
{system message}
Here is the ﬁnal goal we want to achieve:
{task instruction}
To achieve the goal, we have done the following:
{interaction history}
Now, we have observed:
{observations}

To better ﬁnish the task, write a query to ask for useful information, e.g., what kind of exam-
ples or interaction history will be helpful to predict the next action.

44

2322
2323

2324
2325
2326
2327
2328
2329

2330
2331
2332
2333
2334
2335

2336
2337
2338
2339
2340

2341
2342
2343
2344
2345
2346

2347
2348
2349
2350
2351
2352

2353
2354
2355
2356
2357
2358

2359
2360
2361
2362
2363

2364
2365
2366
2367
2368
2369

2370
2371
2372
2373
2374
2375

Under review as a conference paper at ICLR 2025

Table 40: OSWorld example (ﬁltered)

Instruction

Sum numbers in the ﬁrst column.

Observation 0 (Interface of the software LibreOfﬁce Calc)

Action 1

import pyautogui
pyautogui.click(543,126) // click Tools.

Observation 1

45

2376
2377

2378
2379
2380
2381
2382
2383

2384
2385
2386
2387
2388
2389

2390
2391
2392
2393
2394

2395
2396
2397
2398
2399
2400

2401
2402
2403
2404
2405
2406

2407
2408
2409
2410
2411
2412

2413
2414
2415
2416
2417

2418
2419
2420
2421
2422
2423

2424
2425
2426
2427
2428
2429

Under review as a conference paper at ICLR 2025

Table 41: OSWorld example (ﬁltered) cont.

Action 2

import pyautogui
pyautogui.click(543,580) // click Solver.

Observation 2

Action 3

import pyautogui
pyautogui.click(772,892) // click Close.

Observation 3

46

2430
2431

2432
2433
2434
2435
2436
2437

2438
2439
2440
2441
2442
2443

2444
2445
2446
2447
2448

2449
2450
2451
2452
2453
2454

2455
2456
2457
2458
2459
2460

2461
2462
2463
2464
2465
2466

2467
2468
2469
2470
2471

2472
2473
2474
2475
2476
2477

2478
2479
2480
2481
2482
2483

Under review as a conference paper at ICLR 2025

Table 42: OSWorld example (ﬁltered) cont.

Action 4

import pyautogui
pyautogui.click(520,126) // click Data.

Observation 4

Action 5

import pyautogui
pyautogui.moveTo(520,562) // move to Calculate.

Observation 5

47

2484
2485

2486
2487
2488
2489
2490
2491

2492
2493
2494
2495
2496
2497

2498
2499
2500
2501
2502

2503
2504
2505
2506
2507
2508

2509
2510
2511
2512
2513
2514

2515
2516
2517
2518
2519
2520

2521
2522
2523
2524
2525

2526
2527
2528
2529
2530
2531

2532
2533
2534
2535
2536
2537

Under review as a conference paper at ICLR 2025

Table 43: OSWorld example (ﬁltered) cont.

Action 6

import pyautogui
pyautogui.click(498,126) // click Sheet.

Observation 6

Action 7

import pyautogui
pyautogui.click(152,591) // click cell A4.

Observation 7

48

2538
2539

2540
2541
2542
2543
2544
2545

2546
2547
2548
2549
2550
2551

2552
2553
2554
2555
2556

2557
2558
2559
2560
2561
2562

2563
2564
2565
2566
2567
2568

2569
2570
2571
2572
2573
2574

2575
2576
2577
2578
2579

2580
2581
2582
2583
2584
2585

2586
2587
2588
2589
2590
2591

Under review as a conference paper at ICLR 2025

Table 44: OSWorld example (ﬁltered) cont.

Action 8

import pyautogui
pyautogui.click(480,302) // click the formula box.

Observation 8

Action 9

import pyautogui
pyautogui.write("=SUM(A1:A3)") // type the formula.

Observation 9

49

2592
2593

2594
2595
2596
2597
2598
2599

2600
2601
2602
2603
2604
2605

2606
2607
2608
2609
2610

2611
2612
2613
2614
2615
2616

2617
2618
2619
2620
2621
2622

2623
2624
2625
2626
2627
2628

2629
2630
2631
2632
2633

2634
2635
2636
2637
2638
2639

2640
2641
2642
2643
2644
2645

Under review as a conference paper at ICLR 2025

Table 45: OSWorld example (ﬁltered) cont.

Action 10

import pyautogui
pyautogui.press("enter")

Observation 10

50

2646
2647

2648
2649
2650
2651
2652
2653

2654
2655
2656
2657
2658
2659

2660
2661
2662
2663
2664

2665
2666
2667
2668
2669
2670

2671
2672
2673
2674
2675
2676

2677
2678
2679
2680
2681
2682

2683
2684
2685
2686
2687

2688
2689
2690
2691
2692
2693

2694
2695
2696
2697
2698
2699

Under review as a conference paper at ICLR 2025

Table 46: WebArena example (ﬁltered)

Instruction

What are items ordered in the latest cancelled order?

Observation 0

Action 1

click [4918] // click the button "My Account"

Observation 1

51

2700
2701

2702
2703
2704
2705
2706
2707

2708
2709
2710
2711
2712
2713

2714
2715
2716
2717
2718

2719
2720
2721
2722
2723
2724

2725
2726
2727
2728
2729
2730

2731
2732
2733
2734
2735
2736

2737
2738
2739
2740
2741

2742
2743
2744
2745
2746
2747

2748
2749
2750
2751
2752
2753

Under review as a conference paper at ICLR 2025

Table 47: WebArena example (ﬁltered) cont.

Action 2

click [4922] // click the button "My Orders".

Observation 2

Action 3

click [6357] // click the button "View Order".

Observation 3

52

2754
2755

2756
2757
2758
2759
2760
2761

2762
2763
2764
2765
2766
2767

2768
2769
2770
2771
2772

2773
2774
2775
2776
2777
2778

2779
2780
2781
2782
2783
2784

2785
2786
2787
2788
2789
2790

2791
2792
2793
2794
2795

2796
2797
2798
2799
2800
2801

2802
2803
2804
2805
2806
2807

Under review as a conference paper at ICLR 2025

Table 48: WebArena example (ﬁltered) cont.

Action 4

click [4922] // click the button "My Orders".

Observation 4

Action 5

click [6357] // click the button "View Order".

Observation 5

53

2808
2809

2810
2811
2812
2813
2814
2815

2816
2817
2818
2819
2820
2821

2822
2823
2824
2825
2826

2827
2828
2829
2830
2831
2832

2833
2834
2835
2836
2837
2838

2839
2840
2841
2842
2843
2844

2845
2846
2847
2848
2849

2850
2851
2852
2853
2854
2855

2856
2857
2858
2859
2860
2861

Under review as a conference paper at ICLR 2025

Table 49: WebArena example (ﬁltered) cont.

Action 6

click [4922] // click the button "My Orders".

Observation 6

Action 7

click [6357] // click the button "View Order".

Observation 7

54

2862
2863

2864
2865
2866
2867
2868
2869

2870
2871
2872
2873
2874
2875

2876
2877
2878
2879
2880

2881
2882
2883
2884
2885
2886

2887
2888
2889
2890
2891
2892

2893
2894
2895
2896
2897
2898

2899
2900
2901
2902
2903

2904
2905
2906
2907
2908
2909

2910
2911
2912
2913
2914
2915

