AgentSquare: Automatic LLM Agent Search in Modular Design Space
Under review as a conference paper at ICLR 2025

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

AGENTSQUARE: AUTOMATIC LLM AGENT

SEARCH IN MODULAR DESIGN SPACE

Anonymous authors
Paper under double-blind review

ABSTRACT

Recent advancements in Large Language Models (LLMs) have led to a rapid
growth of agentic systems capable of handling a wide range of complex tasks.
However, current research largely relies on manual, task-specific design, limiting
their adaptability to novel tasks. In this paper, we introduce a new research prob-
lem: Modularized LLM Agent Search (MoLAS). We propose a modular design
space that abstracts existing LLM agent designs into four fundamental modules
with uniform IO interface: Planning, Reasoning, Tool Use, and Memory. Build-
ing on this design space, we present a novel LLM agent search framework called
AgentSquare, which introduces two core mechanisms, i.e., module evolution and
recombination, to efficiently search for optimized LLM agents. To further ac-
celerate the process, we design a performance predictor that uses in-context sur-
rogate models to skip unpromising agent designs. Extensive experiments across
six benchmarks, covering the diverse scenarios of web, embodied, tool use and
game applications, show that AgentSquare substantially outperforms hand-crafted
agents, achieving an average performance gain of 17.2% against best-known hu-
man designs. Moreover, AgentSquare can generate interpretable design insights,
enabling a deeper understanding of agentic architecture and its impact on task
performance. We believe that the modular design space and AgentSquare search
framework offer a platform for fully exploiting the potential of prior successful
designs and consolidate the collective efforts of research community. Code repo
is available at https://github.com/ICLR-10021/AgentSquare.

1

INTRODUCTION

The past few years have witnessed remarkable progress in the development of Large Language Mod-
els (LLMs) (Achiam et al., 2023; Touvron et al., 2023), giving rise to the proliferation of numer-
ous agentic systems (Weng, 2023; Shen et al., 2024). For example, “chain-of-thought” prompting
has unlocked the general-purpose reasoning capabilities of LLMs (Wei et al., 2022), and memory
mechanisms have been proven effective in simulating human behavioiur (Park et al., 2023). These
emerging LLM agents have demonstrated astonishing abilities to transform a wide range of tasks, in-
cluding solving mathematical problems (Romera-Paredes et al., 2024), navigating the web (Nakano
et al., 2021), providing financial advice (Ding et al., 2024) and informing medical decisions (Li
et al., 2024a). Therefore, the design of agentic systems plays a crucial role in harnessing the power
of LLMs for various downstream applications.

However, current research predominantly relies on manually designed agentic systems tailored for
specific tasks, which often depend heavily on expert insight and intensive human labor. Furthermore,
these task-specific agent designs frequently struggle to adapt to novel tasks. A few recent studies
have explored using LLMs to rewrite and optimize the prompts of existing agents (Fernando et al.,
2024; Yang et al., 2024). A more recent work introduces the idea to leverage LLMs to search the
entire agentic systems defined in code space (Hu et al., 2024), enabling the discovery of agents with
more flexible prompts, control flows, etc. However, these previous approaches are limited in their
ability to explicitly recombine the strengths of agentic modules discovered by different researches
and located in separate codebases. Another line of research focuses on optimizing the configuration

1

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: AgentSquare is a modular framework for designing and optimizing LLM agents. We
first propose a modular design space of LLM agents and extract 4 types of standardized modules
including planning, reasoning, tooluse, and memory. Based on this, we design a novel LLM agent
search framework to automatically discover good-performing agents.

of multi-agent systems (Chen et al., 2023; Yuan et al., 2024; Li et al., 2023; Zhuge et al., 2024;
Wang et al., 2023b). These efforts are orthogonal to the optimization of single-agent systems, as
they focus more on the role-playing and interaction patterns among multiple agents, rather than the
design of agentic modules.

This paper addresses a novel research problem — Modularized LLM Agent Search (MoLAS). The
goal is to automatically optimize LLM agent designs by leveraging the experience of published or
evaluated modules. Therefore, the core of our work is a modular design space for LLM agents,
comprising 4 categories of modules: Planning, Reasoning, Tool Use, and Memory. This design
space is abstracted from a thorough literature review of existing agentic systems (details provided in
Section 2). It is important to note that our goal is not to propose the most comprehensive, one-size-
fits-all LLM agent design space, but rather to demonstrate that our modular design space enables
researchers and intelligent search algorithms to fully exploit the potential of prior successful designs.
Thus, our key consideration is providing a standardized IO interface for these modules, facilitating
easy recombination of modules from different agentic systems and hence enabling efficient search
for novel agents. Our design space is also highly extensible, allowing new agentic systems to be
integrated as plug-in modules. Therefore, it provides a platform to consolidate the collective efforts
of the research community on LLM agents. The overview of this work is illustrated in Figure 1.

Building on this modular design space, we propose a novel LLM agent search framework called
AgentSquare. Specifically, AgentSquare optimizes LLM agents through the mechanisms of module
evolution and recombination. The module evolution mechanism leverages an evolutionary meta-
prompt to explore new modules through prompt-level optimization, which jointly models task de-
scriptions, existing modules, and the performance of evaluated modules. Besides, the module re-
combination mechanism performs module-level optimization by leveraging the reasoning power of
LLMs to strategically search for promising module combinations. To reduce the expensive eval-
uation costs of LLM agents, we further introduce a performance predictor that implements an in-
context surrogate model for newly proposed LLM agents, enabling us to skip unpromising candi-
dates and significantly accelerate the search process.

We conduct comprehensive evaluations on six widely adopted benchmarks, covering diverse use
cases in web, embodied, tool use and game scenarios. Our experiments show AgentSqaure can
discover novel LLM agents that outperform hand-crafted agents across all six benchmarks, scor-
ing an average performance gain of 17.2% compared to the best known human designs. Besides,
AgentSqaure also surpasses other search algorithms in terms of having a steeper optimization tra-
jectory. More importantly, case studies reveal that AgentSquare can provide human interpretable
design insights for newly discovered, good-performing agents.

The key contributions of this work are as follows:

• We propose a novel modular design space for LLM agents, enabling researchers to easily
build on previous successful designs and accumulate new discoveries as a community.

2

AgentSquareToolformerToolbenchDEPSCOTTOTGenerativeVOYAGERDilu…GameSimulationSelf-driving General-purposeReasoningToolUsePerformanceTOTVOYAGERGenrativeToolformerDiverse agentsPlanningReasoningMemoryTooluseNEWNEWStandardized agentsModule pool:::…EvaluationUnder review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Figure 2: Illustration of the modular agent design space and agentic workflow (left) and the stan-
dardized IO interface of four types of modules (right).

• We design the AgentSquare framework that efficiently searches for novel and good-
performing LLM agents via the novel mechanism of module evolution, module recom-
bination, and performance predictor.

• Experiments across six diverse tasks show that our method discovers novel LLM agents
that outperform all known human designs. Besides, AgentSqaure can generate human
interpretable design insights for these novel agents.

2 A MODULAR DESIGN SPACE OF LLM AGENTS

2.1 BACKGROUND

There are several recent studies explore the problem of prompting LLMs to design LLM agentic
systems. OPRO (Yang et al., 2024) and Promptbreeder (Fernando et al., 2024) can be viewed as
leveraging the reasoning power of LLMs to improve the prompt of LLM agents. More importantly,
ADAS introduces the idea of searching the entire agentic system defined in code space, and propose
a Meta Agent Search algorithm that discovers LLM agents outperforming state-of-the-art human
designs (Hu et al., 2024). Our main difference and contribution lie in introducing a modular design
space for LLM agents, which can provide a standard framework to support the convenient reuse of
existing successful agent components and fruitful innovative agent module discovery.

A modular design space for LLM agents facilitates the reuse of prior successful designs and sup-
ports the exploration of new architectures. At the core of such modularization is the standardization
of input-output interfaces, which ensures both extensibility and seamless integration with existing
designs. Many experts in the field have proposed building LLM agentic systems with key mod-
ular components from engineering (Weng, 2023) and cognitive perspectives (Sumers et al., 2023;
Shao et al., 2024). However, these proposals remain largely conceptual, lacking implementable so-
lutions to unify existing LLM agents. Besides, current LLM workflow program frameworks (e.g.,
LangChain and AutoGPT) only provide operation-level components, which cannot support module-
level search that best exploits the potential of prior successful designs.

To address these problems, we perform a comprehensive literature review of publications from
NeurIPS, ICML, and ICLR over the past three years. The review focuses on papers with the key-
words “LLM”, “Agent”, or “Large Language Model” in their titles while excluding works related to
multi-agent systems or agents that require additional training. Note that our aim is not to propose
the most comprehensive, one-for-all LLM agent design space, but to offer a standardized framework
that enables the recombination of existing agents and facilitates the discovery of new ones. As a
result, we sort out 16 popular LLM agents and abstract a modular design space with 1050 possible
combinations, which can be easily extended when new modules are discovered. Below, we describe
the agentic workflow and the function of four modules in our design space.

3

Standard IO InterfaceAbstractclass Memory():    Input : current situation         if current situation ! = ‘ ‘ :                add(current situation)         else:                Output : retrieve()    class Planning():    Input : task type, task description, feedback    Output : planclass Reasoning():    Input : task description, tool_instruction, feedback    Output : reasoning resultclass Tooluse():    Input : task description, tooluse_instruction,feedback   Output : tool responsePlanningReasoningTool UseMemoryselectfeedbackAddRetrieve…actfeedbackAgenticWorkflowoutputreturnModular Agent Design SpaceTool PoolMemoryDatabase: Module: Task Description: ResourcesTask DescriptionSubtask 1Subtask 2Subtask nEnvironmentUnder review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

2.2 WORKFLOW OVERVIEW

The proposed agent workflow operates through an iterative process with the interconnection of the
above four modules, as shown in Figure 2. Upon receiving a task d, the agent starts with the planning
module, decomposing it into n sub-tasks{s1, s2, . . . , sn}. Next, these sub-tasks are passed to the
reasoning module sequentially. Taking the sub-task si description as input, the reasoning module
explores to prompt LLMs to give the result. When reasoning encounters limitations in internal
knowledge of LLMs, the tool use module is activated to select an appropriate tool from the pre-
defined tool pool τ , supporting problem-solving. Besides, the reasoning process also accesses the
memory module which reads and writes necessary observations and experiences from a memory
database mem to help reasoning. The reasoning result of each sub-task will be transformed into
actions, guiding the agent to interact with the external environment. After all sub-tasks are finished
or the reasoning process gets stacked, the agent will activate the planning module to adjust the
plan with the received feedback. The agent conducts such a trial-and-error loop until the task d is
completed or the set maximum trial number is reached.

Planning. The planning module is responsible for decomposing the targeted task into smaller sub-
tasks. Given a task description d and optional feedback information f , the planning module P
strategically decomposes the targeted task into a sub-task sequence {s1, s2, . . . , sn} = P (d, f ).
Such decomposition is critical for handling very complex tasks with long-term characteristics, espe-
cially for agents in open-world environments such as MineCraft (Wang et al., 2024a;c).

Reasoning. LLMs have exhibited remarkable reasoning abilities under advanced prompting ap-
proaches such as CoT (Wei et al., 2022), ToT (Yao et al., 2024), and SoT (Shang et al., 2024),
shaping the foundation of the intelligence of LLM agents. The reasoning module R is invoked to
solve the sub-tasks sequentially after planning, which takes each sub-task si and optional feedback
information fi as input and outputs a solution ri = R(si, fi).

Tool use. The ability of using external tools (Shen et al., 2024; Schick et al., 2024) overcomes
the limitations of the LLM’s internal knowledge during the reasoning process. Formally, given
certain problem pij derived from the reasoning process of sub-task si and a pre-defined tool pool
τ , the tooluse module T selects the best-matched tool tij to address the problem, denoted as tij =
T (pij, τ ), where tij ∈ τ .

Memory. Memory plays a critical role by storing past thoughts, actions, and observations of
agents (Park et al., 2023; Shinn et al., 2024). During the reasoning process, these internal logs
are dynamically written to and retrieved from the memory database mem, controlled by the mem-
ory module M . The writing process can be expressed as mem = Mwrite(o, mem), where o denotes
the current observations. The retrieval process is m = Mretrieve(o, mem), where m denotes the
retrieved knowledge relevant to the current situation.

3 AGENTSQUARE FRAMEWORK

3.1 PROBLEM FORMULATION OF MOLAS

In the proposed modular design space, an LLM agent A can be instantiated with the combination
of a planning module P , a reasoning module R, a tooluse module T and a memory module M ,
denoted as A = (P, R, T, M ). Given the task description d and the set of all possible modules
with standardized IO interface {P, R, T, M}. We formulate an optimization problem for searching
LLM agent architectures within the modular design space. The objective is to identify the optimal
module combination in a solution space defined by a Cartesian product of four design dimensions
to maximize agent performance. Let the performance evaluation function of the task be Evald(·),
where the specific metric varies in different tasks as discussed in Appendix A.1. The optimization
problem of MoLAS is defined as follows:

arg max
P ∈P,R∈R,T ∈T,M ∈M

Evald(P, R, T, M ).

(1)

4

Under review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Figure 3: Overview of AgentSquare search framework. AgentSquare optimizes LLM agents through
the mechanisms of module evolution and recombination. We further introduce a performance pre-
dictor that implements an in- context surrogate model for efficient evaluation of novel agents.

3.2 AGENTSQUARE SEARCH ALGORITHM

Solving the optimization problem of MoLAS features three key challenges: (1) The search space,
defined as the Cartesian product of four orthogonal modules, is vast and hard to explore; (2) the
module sets encompass any code with standard IO interfaces, making the module selection an open-
ended problem; (3) the high costs of agent evaluation during the search process constrain the overall
search scale. To tackle these issues, we introduce AgentSquare, an automatic search framework to
optimize LLM agents within the modular design space. Facing the vast search space of MoLAS,
we propose module recombination operation utilizing LLMs to strategically reason to identify more
promising module combinations. Such operation broadens the coverage of child samples, overcom-
ing the limitations of prompt rewrite methods that explore only a restricted space. However, only
searching in the existing module combinations also narrows the search space, thus we propose mod-
ule evolution operation which employs an evolutionary meta-prompt to search new modules through
code-level optimization. This operation, combined with module recombination, enables the search
of any module combination in the open-ended solution space. Finally, to mitigate the high costs of
frequent evaluations of searched agents, we design a performance predictor as an in-context surro-
gate model for evaluating searched agents, significantly accelerating the search process and reducing
real-valued costs.

The overall framework of AgentSquare is illustrated in Figure 3 and the algorithm is presented in
Algorithm 1. Next, we detail the key components of the AgentSquare search process.

3.3

INITIALIZATION

Insights from existing AutoML studies indicate that a well-chosen initialization enhances warm-up
and improves search efficiency by avoiding unpromising populations (So et al., 2019; Yuan et al.,
2024). AgentSquare starts by initializing a global experience pool E = {(P, R, T, M, v)|P0 ∈
P, R0 ∈ R, T0 ∈ T, M0 ∈ M} to seed agents that are well-designed (as mentioned in Section 2)
along with their real-valued performance v. The module pools {P, R, T, M} are set to the standard-
ized modules extracted from these seed agents.

5

LLM𝑃′𝑅′𝑇′𝑀′𝑃0𝑅0𝑇0𝑀0𝑃0𝑅′𝑇0𝑀0𝑃′𝑅0𝑇0𝑀0𝑃0𝑅0𝑇′𝑀0𝑃0𝑅0𝑇0𝑀′LLM𝑃1𝑅′𝑇0𝑀0𝑃0𝑅3𝑇0𝑀0𝑃0𝑅′𝑇2𝑀0𝑃0𝑅′𝑇0𝑀4Initial AgentEvolutionRecombinationNewModuleNew AgentsNextEpisodePlanning𝑃0𝑅′𝑇0𝑀0𝑃0𝑅′𝑇2𝑀0𝑅1𝑅2𝑅3𝑅4𝑇1𝑇2𝑇3𝑇4𝑀1𝑀2𝑀3𝑀4𝑃1𝑃2𝑃3𝑃4Module Pool𝑃2𝑅2𝑇2𝑀2𝑃3𝑅3𝑇3𝑀3𝑃4𝑅4𝑇4𝑀4𝑃1𝑅1𝑇1𝑀1::::…New AgentsExperience Pool…WebshopTravelplannerAlfworldSciworldM3Tool…EvaluationEvaluationTask EvaluationReasoningTooluseMemoryAgent SearchFeedbackUnder review as a conference paper at ICLR 2025

3.4 MODULE RECOMBINATION

Given the vast solution space of MoLAS, relying solely on prompt rewriting leads to a lim-
ited exploration confined to the neighbor of the initial state. To expand the exploration space,
we propose leveraging LLMs as a self-adaptive proposer, which iteratively reason to identify
promising module combinations with accumulated experience beyond the original agent config-
uration. Denote the initial agent of the recombination phase as A0
r = (P0, R0, T0, M0), where
P0 ∈ P, R0 ∈ R, T0 ∈ T, M0 ∈ M. The module combination proposer LLM πθ incorporates
targeted task description d, existing module pools {P, R, T, M} and the performance experience of
searched module combinations E to propose promising new agents Ar:

Ar = πθ((P0, R0, T0, M0), d, N, P, R, T, M, E).

(2)

Based on the initial agent configuration A0
r } by
replacing certain modules of A0
r with alternatives from the module pool. For instance, a possible
∈ R is a different reasoning module selected from the
solution could be (P0, R
module pool. Then, the created N new agents are evaluated with a performance predictor πp (detail
in Seciton 3.6) and the best one goes to the next episode as initialization.

r, the LLM proposes N offspring {A1

, T0, M0), where R

r, ..., AN

r, A2

′

′

3.5 MODULE EVOLUTION

As mentioned above, the solution space for each module type is open-ended, allowing any code with
a standardized I/O interface. Consequently, searching only with module recombination narrows the
solution space and limits the upper bound of agent performance. To address this problem, we design
a module evolution operation with an evolutionary meta-prompt to search for new modules through
program-level optimization. This design is inspired by the iterative pipeline of FunSearch (Romera-
Paredes et al., 2024), which prompts LLMs to propose new solutions based on the target problem and
performance feedback from existing solutions. Building on this concept, we introduce a module-
programming LLM πξ to conduct agent search in our modular design space by jointly modeling task
descriptions, existing modules, and the performance of previously evaluated modules. Please note
we reuse parts of the open-source code from ADAS (Hu et al., 2024) to implement the optimization
procedure. Leveraging LLMs to search in the modular agent design space has several appealing
advantages. Compared with the unconstrained design space of LLM agents, searching functional
modules can produce a more focused and fruitful search space. Additionally, integrating existing
successful module designs with standard IO as in-context examples can better elicit the reflective
reasoning abilities of LLMs to identify previous key designs to help propose innovative ones. Denote
the initial agent in the module evolution stage as A0
0), the module programmer
LLM produces a population of child agents by evolving current modules of A0
e. Formally the module
evolution operation is denoted as follows:

e = (P

0, M

0, R

0, T

′

′

′

′

Ae = πξ((P

0, R

0, T

0, M

′

′

′

′

0), d, N, P, R, T, M, E).

(3)

The created new modules are appended to the standardized module pools {P, R, T, M} and each
module is used to individually mutate the initial agent, resulting in N child agents {A1
e }.
For example, (P ∗, R0, T0, M0) represents a solution where the planning module is mutated into a
new variant P ∗. These child agents are then real-tested and updated to the historical experience pool
E. The best-performing one is selected as the initial agent for the subsequent recombination phase.

e, ..., AN

e, A2

3.6 PERFORMANCE PREDICTOR

The last challenge in automatic agent search is the high API cost incurred during the evaluation of
each candidate agent. Many agent tasks require multiple steps and involve substantial input and
output tokens, leading to prohibitive evaluation costs. For instance, evaluating a simple CoT agent
based on GPT-4o in ALFWorld (Shridhar et al., 2021) requires around $60, making the agent search
economically unsustainable at scale. To tackle this issue, we propose incorporating an additional
LLM πp as a performance predictor to serve as an in-context surrogate model for novel agent eval-
uation, enabling the exclusion of unpromising candidates and significantly accelerating the search
process. Compared to real environment evaluation, such an in-context surrogate model requires
significantly fewer tokens, making it more cost-efficient and supporting larger-scale searches. Sim-
ilar approaches have been effectively applied in neural architecture search (NAS), where LLMs are

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Baseline Type

Hand-crafted Agents

Module Search

Prompt Search
Agent Search

Method
CoT
Cot-SC
Self-refine
ToT
Step Back
TP
HuggingGPT
Voyager
Generative Agents
DEPS
OPENAGI
Dilu
Random
Bayesian
OPRO
ADAS
AgentSquare

Tool

Web

Embodied

Game
Webshop ALFWorld SciWorld M3Tool TravelPlanner PDDL
0.542
0.495
0.514
0.476
0.486
0.518
0.584
0.412
0.553
0.591
0.616
0.463
0.660
0.650
0.589
0.568
0.669

0.405
0.426
0.567
0.437
0.279
0.404
0.481
0.425
0.477
0.459
0.510
0.433
0.620
0.634
0.380
0.543
0.695

0.448
0.461
0.442
0.453
0.434
0.387
0.354
0.247
0.402
0.278
0.322
0.475
0.438
0.502
0.309
0.475
0.524

0.697
0.656
0.654
0.741
0.220
0.576
0.680
0.776
0.663
0.740
0.718
0.682
0.704
0.749
0.569
0.754
0.781

0.487
0.413
0.000
0.380
0.000
0.430
0.510
0.523
0.480
0.540
0.533
0.360
0.563
0.537
0.523
0.373
0.583

0.485
0.512
0.461
0.501
0.468
0.398
0.519
0.366
0.499
0.481
0.506
0.451
0.533
0.549
0.505
0.521
0.607

Table 1: Performance comparison of searched agents from AgentSquare and (1) existing human-
designed agents (2) module search baselines (3) prompt search baselines based on GPT-4o on six
tasks across different domains.

leveraged to evaluate the performance of generated network architectures (Jawahar et al., 2023;
Chen et al., 2024a).

During the search process, newly created agents from module evolution are still tested in the real task
environment because these new modules never appear in the experience pool, and it is unsuitable to
use the performance predictor to provide predictions. During the module recombination operation,
the newly proposed agents are evaluated by the performance predictor, which leverages in-context
reasoning based on past agent combination performance to provide efficient performance prediction.
Here, given a newly searched agent A′, the performance predictor πp thoroughly considers task
descriptions d, module profiles and in-context performance examples of previously tested agents E
to score novel agents:

v′ = πp(A′, d, P, R, T, M, E),

(4)

where v′ is the predicted performance of the evaluated agent. Empirical results demonstrate that the
predicted performance of agents closely matches their actual performance, verifying the effective-
ness of the proposed performance predictor, which is detailed in Section 4.3.

4 EXPERIMENTS

4.1 EXPERIMENTAL SETUP

Task setup. We conduct experiments on six representative tasks covering four domains: embodied,
game, web and tool applications, which are widely adopted by existing LLM agent benchmarks (Ma
et al., 2024; Xi et al., 2024), more details are presented in Appendix A.1.

Baselines. We compare AgentSquare with four types of baselines including hand-crafted agents,
module-level search, prompt-level search and agent-search methods. More details are presented in
Appendix A.1.

AgentSquare setup. We implement AgentSquare and conduct experiments using both GPT-3.5-
turbo-0125 and GPT-4o (Achiam et al., 2023). To ensure a fair comparison, we use the same number
of few-shot examples across all methods. The initial agent is set as a random module combination,
and the search process terminates after 5 consecutive iterations without performance improvement.

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Figure 4: AgentSquare search trajectory on Alfworld and Webshop.

4.2 EXPERIMENTAL RESULTS

Main results. We conduct extensive experiments to compare our method against three types of
baselines on six tasks and present results based on GPT-4o in Table 1 and results on GPT-3.5 in Ta-
ble A.3. Additionally, we evaluate the agents’ API costs and provide a performance-cost comparison
in Figure A.7 to Figure A.12. From these results, we have the following observations:

• AgentSquare can effectively discover better agents compared with human-designed agents.
On the six representative agent tasks, the best agent searched by AgentSquare consistently out-
performs human-designed agents in terms of performance. Specifically, as shown in Table 1
and Table A.3, compared with the best human-designed agent, AgentSquare achieves an average
14.1% performance improvement on Webshop, 26.1% improvement on ALFWorld, 20.5% im-
provement on SciWorld, 30.6% improvement on M3Tool, 6.0% improvement on Travelplanner,
6.0% improvement on PDDL. Simultaneously, the best agent from AgentSquare is commonly
cost-efficient, which strikes the best performance-cost trade-off among all compared agents as
seen in Figure A.7 -Figure A.12. Since the search cost is a one-time expense and the searched
modules can be reused, it is not included in the above analysis, but separately listed in Table A.6.
• AgentSquare provides a more efficient searching approach for LLM agent optimization. To
further demonstrate the effectiveness of the search pipeline of AgentSquare, we also compare with
two types of agent searching methods including module-level and prompt-level search. Compared
with the best agent crafted from these searching methods, AgentSquare achieves an average 8.4%
performance improvement on Webshop, 8.1% improvement on ALFWorld, 11.0% improvement
on SciWorld, 12.8% improvement on M3Tool, 2.5% improvement on Travelplanner, 1.4% im-
provement on PDDL.

Search trajectory in AgentSquare. We present the search trajectory under 15 iterations using
AgentSquare based on GPT-4o and other searching methods on ALFWorld and Webhop tasks in
Figure 4. Results on other tasks are presented in Figure A.13 and A.14. AgentSquare demonstrates a
steady convergence trajectory, where more advanced agents are continually emerging during search.
In contrast, module-level searching methods including random and Bayesian search lack a clear and
insightful search direction. Prompt-level search methods such as OPRO are constrained by a limited
modification space, leading to minimal performance improvements. As a result, they all encounter
performance bottlenecks during the search process, resulting in sub-optimal agent architectures. Be-
sides, we find that simple module-level search methods such as random recombination greatly out-
performs prompt-level search, indicating the importance of searching in the modular design space.

4.3 ABLATION STUDY OF AGENTSQUARE

Effectiveness of module evolution and recombination. There are two key operations in the search-
ing framework of AgentSquare: module evolution which creates new modules and module recom-
bination which strategically recombines existing ones. To verify the effectiveness of each design,
we tested three variants of AgentSquare: the full model, a version without module evolution, and

8

123456789101112131415Iteration0.350.400.450.500.550.600.65Success ratePlanning: TDReasoning: CoTMemory:  adadda VoyagerPlanning: TDReasoning: SF-TOTMemory:  adadddadadaaaa Generative AgentsAgentSquare Search on Alfworld123456789101112131415161718Iteration0.480.500.520.540.560.580.600.62ScorePlanning: DEPSReasoning: CoT-SCMemory: adDiluPlanning: IOReasoning: HTSSMemory: DiluHTSSAgentSquare Search on WebshopBayesianRandomOPROAgentSquareBest of human designUnder review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Method
AgentSquare (full)
w/o module evolution
w/o module recombination

Webshop ALFWorld SciWorld M3Tool TravelPlanner PDDL
0.669
0.614
0.669

0.607
0.564
0.560

0.781
0.736
0.710

0.695
0.649
0.616

0.524
0.502
0.481

0.583
0.577
0.280

Table 2: Ablation study of AgentSquare on GPT-4o on six tasks across different domains.

Figure 5: Validation of the effectiveness of the performance predictor (correlation between the actual
and predicted performance) on each task.

a version without module recombination. The results based on GPT-4o and GPT-3.5 are presented
in Table 2 and Table A.5, respectively. It can be seen that dropping each design results in a no-
ticeable performance decline and the module recombination has a larger impact. Module recom-
bination significantly expands the search space, reducing the risk of falling into a local optima.
Meanwhile, module evolution facilitates the discovery of more advanced modules tailored to spe-
cific tasks. These two operations collaborate well ensuring the effectiveness of the search process in
AgentSquare.

Effectiveness of performance predictor. In this part, we verify the effectiveness of this design
empirically. Figure 5 illustrates the predicted performance of given agents versus their actual tested
performance based on both GPT-3.5 and GPT-4o on all six tasks. It can be found that the predicted
performance closely aligns with the actual performance, demonstrating the effectiveness of the per-
formance predictor. For instance, the evaluation cost of the predictor is only about 0.025% of the cost
of a full evaluation based on GPT-4o in ALFWorld, demonstrating its remarkable cost-efficiency.

4.4 DISCOVERED BEST AGENTS FROM AGENTSQUARE

In this section, we provide some illustrations of the searched best agents, especially some discovered
promising modules. Table A.4 summarizes the searched best agent from AgentSquare and the best
hand-crafted agents on all tasks. We can observe that AgentSquare can adaptively identify promising
agents with both previously existing and newly programmed modules tailored to the given task.
For instance, the discovered best agent for ALFWorld combines an existing well-designed memory
module from Generative Agents with newly created planning (named TD) and reasoning modules
(named SF-ToT). By comparison, the best hand-crafted agent Self-refine focuses only on reasoning
module design while overlooking other functional modules, leading to suboptimal performance.
Moreover, we illustrate two new modules and the human interpretable design insights discovered on
ALFWorld in Figure 6. More illustrations are listed in the Figure A.15 to Figure A.20.

5 RELATED WORK

5.1 LLM-BASED AUTONOMOUS AGENTS

LLM-based autonomous agents are an advanced AI system using a core LLM to manage external
functional modules and interact with environments. Recent studies have equipped LLM agents with
several LLM-centric functional modules including planning (Hao et al., 2023; Huang et al., 2023;
Zeng et al., 2024), reasoning (Wei et al., 2022; Yao et al., 2024; Shang et al., 2024), using tools (Shen
et al., 2024; Schick et al., 2024), and monitoring memory (Wang et al., 2024a; Park et al., 2023),

9

0.20.30.40.50.60.7Predicted performance0.20.30.40.50.60.7Actual tested performanceWebshop based on gpt-3.5Webshop based on gpt-4oAlfworld based on gpt-3.5Alfworld based on gpt-4o0.20.30.40.50.60.7Predicted performance0.20.30.40.50.60.7Actual tested performanceSciworld based on gpt-3.5Sciworld based on gpt-4oTravelplanner based on gpt-3.5Travelplanner based on gpt-4o0.20.30.40.50.60.7Predicted performance0.10.20.30.40.50.60.7Actual tested performanceM3tool based on gpt-3.5M3tool based on gpt-4oPddl based on gpt-3.5Pddl based on gpt-4oUnder review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Figure 6: New module discovered through AgentSquare search on ALFWorld.

greatly enhancing the capabilities of LLM agents. Along with the improvement of the single agent,
there’s another line of work trying to build more advanced multi-agent systems by strategically
organizing individual agents for both simulation (Li et al., 2023; Chen et al., 2023) and targeted task
solving (Qian et al., 2024; Xiao et al., 2023; Chen et al., 2024b; Li et al., 2024b). The emergence
of more and more sophisticated agent produces remarkable performance improvement, however,
their architectures and codebases differ greatly with each other. The lack of a unified design space
and consistent terminologies across individual works makes it hard to compare different agents,
understand their evolution routes, and guide new agent design directions.

5.2 AUTOMATIC DESIGN OF LLM-BASED AGENTS

LLM-based agent system, as the most advanced AI system, has not yet formed a unified design
space and an automatic design approach. Engineering-oriented open resources like LangChain1 and
BabyAGI2 has provided convenient ways to build an LLM-centric agentic system, however, they
still need human participation to organize different modules and can’t support the optimization of
the designed agent. Besides, there has been some conceptual framework trying to provide a unified
design principle of LLM agents, such as CoALA (Sumers et al., 2023). However, it’s still a vision
of how LLM agents should be in the future, without providing a practical design framework. More
importantly, there are several recent works that explore the problem of automating (at least part of)
the design of LLM agent systems defined on different search space. OPRO (Yang et al., 2024) and
Promptbreeder (Fernando et al., 2024) can be considered as using LLMs to optimize LLM agent
defined on prompt space. Besides, ADAS (Hu et al., 2024) propose to search the entire agentic
system defined on code space, enabling the search for LLM agents with more flexible prompts, tool
uses, control flows and more.

6 CONCLUSION

In this work, we introduce a novel modular design space for LLM agents, allowing researchers to
build upon successful prior designs and collectively accumulate new insights. Based on this, we pro-
pose a novel research problem, Modularized LLM Agent Search (MoLAS), which aims to automat-
ically optimize LLM agent designs by leveraging the knowledge gained from previously published
or evaluated modules. To address the challenge of vast search spaces, we present AgentSquare, an
automatic search framework to optimize LLM agents through module evolution and recombination.
We further introduce a performance predictor as an in-context surrogate model for evaluating novel
LLM agents to accelerate the search process. AgentSquare consistently discovers novel agents that
surpass all existing hand-crafted ones. Overall, our work offers a transition from studying individual
LLM agent designs to studying LLM agents within a modular design space, further consolidating
the collective efforts of the research community.

1https://github.com/langchain-ai/langchain
2https://github.com/yoheinakajima/babyagi

10

432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485UnderreviewasaconferencepaperatICLR202523456789API cost($)0.300.350.400.450.500.550.600.65AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareAlfworld based on GPT-3.520406080100API cost($)0.30.40.50.60.7AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareAlfworld based on GPT-4oFigure4:PerformanceversusAPIcostsvisualizationoneachtask.(All)123456789101112131415Iteration0.350.400.450.500.550.600.65Success rateReplace the memory module with Voyager after module recombinationPlanning moule = TD, Reasoning module = SF-TOTMemory module = Generative AgentsAgentSquare Search on AlfworldBayesianRandomQproAgentSquareBest of human design123456789101112131415161718Iteration0.480.500.520.540.560.580.600.62ScoreReplace the memory module with Dilu after module recombinationPlanning moule = IO, Reasoning module = HTSSMemory module = DiluAgentSquare Search on WebshopBayesianRandomQproAgentSquareBest of human designFigure5:MetaAgentsearchtrajectory(morehuman-designagents,specificmodulecombinationswhensurpassingbesthuman-designandthefinalevolvedagent,othersearchbaselines).Effectivenessofperformancepredictor.Inthispart,weverifytheeffectivenessofthisdesignempirically,takingWebshop,ALFWorldandM3Tooltasksasillustrativeexamples.Figure6il-lustratesthepredictedperformanceofgivenagentsversustheiractualtestedperformancebasedonbothGPT-3.5andGPT-4oonthethreetasks.Itcanbefoundthatthepredictedperformancecloselyalignswiththeactualperformance,demonstratingtheeffectivenessoftheintroducedperformancepredictor.Forinstance,theevaluationcostofthepredictorisonlyabout0.025%ofthecostofafullevaluationbasedonGPT-4oinALFWorld,demonstratingitsremarkablecost-efficiency.SF-TOTInsights:Developamodulethatnotonlygeneratesmultiplepathsandevaluatesthembutalsoincorporatesself-reflectionandself-improvementstrategies.classREASONING_SF_TOT():def__call__(self,...):prompt=f’’’Interactwithahouseholdtosolveatask.Yourinstructionsmustfollowtheexamples.Herearesomeexamples....’’’responses=llm_response(...)response=self.get_votes(...)response=self.refine(response,...)returnresponseTDInsights:Useatimeline-basedapproach,wheretasksaredecomposedintosub-taskswithexplicittemporaldependen-cies.classPLANNING_TD():def__call__(self,...):prompt=f’’’Youareaplannerwhodividesa{task_type}taskintoseveralsubtaskswithexplicittemporaldependencies.Considertheorderofactionsandtheirdependenciestoensurelogicalsequencing....’’’sub_plans=llm_response(...)returnsub_plans9Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

Angelica Chen, David Dohan, and David So. Evoprompting: language models for code-level neural

architecture search. Advances in Neural Information Processing Systems, 36, 2024a.

Lin Chen, Fengli Xu, Nian Li, Zhenyu Han, Meng Wang, Yong Li, and Pan Hui. Large language
model-driven meta-structure discovery in heterogeneous information network. In Proceedings of
the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 307–318,
2024b.

Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu,
Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and
exploring emergent behaviors. In The Twelfth International Conference on Learning Representa-
tions, 2023.

Han Ding, Yinheng Li, Junhao Wang, and Hang Chen. Large language model agent in financial

trading: A survey. arXiv preprint arXiv:2408.06361, 2024.

Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim
Rockt¨aschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-
first International Conference on Machine Learning, 2024.

Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al.
Openagi: When llm meets domain experts. Advances in Neural Information Processing Systems,
36, 2024.

Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning
with language model is planning with world model. In Proceedings of the 2023 Conference on
Empirical Methods in Natural Language Processing, pp. 8154–8173, 2023.

Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint

arXiv:2408.08435, 2024.

Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan
Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through
planning with language models. In Conference on Robot Learning, pp. 1769–1782. PMLR, 2023.

Ganesh Jawahar, Muhammad Abdul-Mageed, Laks VS Lakshmanan, and Dujian Ding. Llm perfor-
mance predictors are good initializers for architecture search. arXiv preprint arXiv:2310.16712,
2023.

Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang
Liu. Agent hospital: A simulacrum of hospital with evolvable medical agents. arXiv preprint
arXiv:2405.02957, 2024a.

Songwei Li, Jie Feng, Jiawei Chi, Xinyuan Hu, Xiaomeng Zhao, and Fengli Xu. Limp: Large
language model enhanced intent-aware mobility prediction. arXiv preprint arXiv:2408.12832,
2024b.

Yuan Li, Yixuan Zhang, and Lichao Sun. Metaagents: Simulating interactions of human behav-
iors for llm-based task-oriented coordination via collaborative generative agents. arXiv preprint
arXiv:2310.06500, 2023.

Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Ling-
peng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents.
arXiv preprint arXiv:2401.13178, 2024.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri
Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement
with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo-
pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted
question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and
Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings
of the 36th annual acm symposium on user interface software and technology, pp. 1–22, 2023.

Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen,
Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Pro-
ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 15174–15186, 2024.

Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog,
M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang,
Omar Fawzi, et al. Mathematical discoveries from program search with large language models.
Nature, 625(7995):468–475, 2024.

Timo Schick, Jane Dwivedi-Yu, Roberto Dess`ı, Roberta Raileanu, Maria Lomeli, Eric Hambro,
Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can
teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.

Yu Shang, Yu Li, Fengli Xu, and Yong Li. Defint: A default-interventionist framework for efficient

reasoning with hybrid large language models. arXiv preprint arXiv:2402.02563, 2024.

Chenyang Shao, Fengli Xu, Bingbing Fan, Jingtao Ding, Yuan Yuan, Meng Wang, and Yong Li.
Beyond imitation: Generating human mobility from context-aware reasoning with large language
models. arXiv preprint arXiv:2402.09836, 2024.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging-
gpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information
Processing Systems, 36, 2024.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion:
Language agents with verbal reinforcement learning. Advances in Neural Information Processing
Systems, 36, 2024.

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew
Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.
In
International Conference on Learning Representations, 2021.

David So, Quoc Le, and Chen Liang. The evolved transformer.

In International conference on

machine learning, pp. 5877–5886. PMLR, 2019.

Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive archi-

tectures for language agents. arXiv preprint arXiv:2309.02427, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee
Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and
efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan,
and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.
Transactions on Machine Learning Research, 2024a. ISSN 2835-8856.

Ruoyao Wang, Peter Jansen, Marc-Alexandre Cˆot´e, and Prithviraj Ammanabrolu. Scienceworld:
Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical
Methods in Natural Language Processing, pp. 11279–11298, 2022.

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Exe-
cutable code actions elicit better llm agents. In Forty-first International Conference on Machine
Learning, 2024b.

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha
Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language
models. In The Eleventh International Conference on Learning Representations, 2023a.

Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the
emergent cognitive synergy in large language models: A task-solving agent through multi-persona
self-collaboration. arXiv preprint arXiv:2307.05300, 2023b.

Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Shawn Ma, and Yitao Liang. De-
interactive planning with llms enables open-world multi-task

scribe, explain, plan and select:
agents. Advances in Neural Information Processing Systems, 36, 2024c.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny
Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in
neural information processing systems, 35:24824–24837, 2022.

Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao MA, Pinlong Cai, Min Dou, Botian Shi, Liang
He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language
models. In The Twelfth International Conference on Learning Representations, 2024.

Lilian Weng. Llm-powered autonomous agents.

lilianweng.github.io, Jun 2023. URL https:

//lilianweng.github.io/posts/2023-06-23-agent/.

Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural
architectures for neural architecture search. In Proceedings of the AAAI conference on artificial
intelligence, volume 35, pp. 10293–10301, 2021.

Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen
Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based
agents across diverse environments. arXiv preprint arXiv:2406.04151, 2024.

Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin
Fu, Tao Zhong, Jia Zeng, Mingli Song, et al. Chain-of-experts: When llms meet complex opera-
tions research problems. In The Twelfth International Conference on Learning Representations,
2023.

Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and
Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first
International Conference on Machine Learning, 2024.

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun
Chen. Large language models as optimizers. In The Twelfth International Conference on Learning
Representations, 2024.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable
real-world web interaction with grounded language agents. Advances in Neural Information Pro-
cessing Systems, 35:20744–20757, 2022.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik
Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad-
vances in Neural Information Processing Systems, 36, 2024.

Junchi Yu, Ran He, and Zhitao Ying. THOUGHT PROPAGATION: AN ANALOGICAL AP-
In The

PROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS.
Twelfth International Conference on Learning Representations, 2024.

Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Dongsheng Li, and Deqing Yang. Evoa-
gent: Towards automatic multi-agent generation via evolutionary algorithms. arXiv preprint
arXiv:2406.14228, 2024.

Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, and Yong
Li. Perceive, reflect, and plan: Designing llm agent for goal-directed city navigation without
instructions. arXiv preprint arXiv:2408.04168, 2024.

13

Under review as a conference paper at ICLR 2025

Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le,
and Denny Zhou. Take a step back: Evoking reasoning via abstraction in large language models.
In The Twelfth International Conference on Learning Representations, 2024.

Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural
architecture search. In International conference on machine learning, pp. 7603–7613, 2019.

Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and J¨urgen
Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International
Conference on Machine Learning, 2024.

14

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Under review as a conference paper at ICLR 2025

A APPENDIX

A.1 EXPERIMENTAL SETUP

Task setup. We evaluate AgentSquare and compared methods on six representative tasks covering
four key domains which are widely adopted by existing LLM agent benchmarks (Ma et al., 2024;
Xi et al., 2024):

• Embodied: ALFWorld (Shridhar et al., 2021) with text-based household tasks where agents nav-
igate and interact with objects using text commands, ScienceWorld (Wang et al., 2022) with inter-
active science tasks requiring agents to navigate rooms and perform experiments, testing scientific
commonsense;

• Game: PDDL (Ma et al., 2024) including many strategic games where agents use PDDL expres-

sions to complete tasks;

• Web: WebShop (Yao et al., 2022) focusing on online shopping tasks where agents browse and

purchase products based on user instructions;

• Tool: TravelPlanner (Xie et al., 2024) with many travel planning tasks where agents use tools
and data to create detailed plans, (6)M3ToolEval (Wang et al., 2024b) including complex tasks
requiring multi-turn interactions with multiple tools.

The specific performance evaluation metric varies in different tasks, following the evaluation set-
tings in their original work. Specifically, the evaluation metric is “success rate” for ALFWorld and
M3ToolEval, “task score (defined as the average reward obtained across episodes)” for Webshop,
“progress rate” for SciWorld and PDDL, and “micro pass rate” for TravelPlanner.

Baselines. We compare AgentSquare with four types of baselines:

• Hand-crafted agents. We compare with 12 hand-crafted agents including CoT (Wei et al., 2022),
CoT-SC (Wang et al., 2023a), Self-refine (Madaan et al., 2024), ToT (Yao et al., 2024), Step
back (Zheng et al., 2024), Thought propagation (Yu et al., 2024), HuggingGPT (Shen et al., 2024),
Voyager (Wang et al., 2024a), Generative Agents (Park et al., 2023), DEPS (Wang et al., 2024c),
OPENAGI (Ge et al., 2024)and Dilu (Wen et al., 2024).

• Module search methods. We compare with two module-level agent optimization methods in-
cluding the random combination of existing modules and Bayesian (Zhou et al., 2019) module
combination optimization inspired by Bayesian optimization in NAS (White et al., 2021).

• Prompt search methods. We select OPRO (Yang et al., 2024) as a representative prompt-level
optimization approach, which leverages LLMs as optimizers by generating and refining instruc-
tions through iterative prompts.

• Agent search methods. We select ADAS (Hu et al., 2024) which optimizes the entire agentic

system in code space as the agent search baseline.

AgentSquare setup. We implement AgentSquare and conduct experiments using both GPT-3.5-
turbo-0125 and GPT-4o (Achiam et al., 2023). To ensure a fair comparison, we use the same number
of few-shot examples across all methods. The initial agent is set as a random module combination,
and the search process terminates after 5 consecutive iterations without performance improvement.

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Algorithm 1: Algorithm of AgentSquare
Input: Initial agent A0, targeted task descriptions d, maximum evolution episode K,

population size N per evolution phase, standardized module pools {P, R, T, M},
experience pool E
Output: The evolved agent A∗
t ← 1 // Current search episode
A0
while t ≤ K do
e, A2
e } ← πξ(A0
r ← arg max{Evald(A0

e, d, N, P, R, T, M, E) // Module evolution
e), Evald(A1
e )} // Select the

e ← A0 // Initialization of the module evolution phase

e), ..., Evald(AN

e, ..., AN

{A1
A0

best-performing generated agent
r } ← πθ(A0
r, A2
e ← arg max{Evald(A0

r, d, N, P, R, T, M, E) // Module recombination
r), Evald(A1

r )} // Select the

r), ..., Evald(AN

r, ..., AN

{A1
A0

best-performing generated agent

t ← t + 1

end
A∗ ← A0
e
return A∗

Method Type

Hand-crafted Agents

Module Search

Prompt Search
Agent Search

Method
CoT
CoT-SC
Self-refine
ToT
Step Back
TP
HuggingGPT
Voyager
Generative Agents
DEPS
OPENAGI
Dilu
Random
Bayesian
OPRO
ADAS
AgentSquare

Tool

Web

Embodied

Game
Webshop ALFWorld SciWorld M3Tool TravelPlanner PDDL
0.151
0.178
0.109
0.147
0.137
0.122
0.212
0.149
0.123
0.186
0.178
0.054
0.216
0.210
0.179
0.186
0.219

0.369
0.381
0.388
0.381
0.375
0.299
0.502
0.369
0.388
0.474
0.448
0.291
0.569
0.611
0.376
0.274
0.651

0.172
0.181
0.098
0.189
0.128
0.139
0.012
0.008
0.144
0.017
0.008
0.131
0.235
0.217
0.193
0.193
0.285

0.504
0.527
0.439
0.510
0.478
0.429
0.518
0.427
0.539
0.555
0.507
0.418
0.562
0.581
0.507
0.519
0.617

0.142
0.105
0.222
0.143
0.027
0.168
0.270
0.301
0.153
0.308
0.257
0.000
0.367
0.269
0.032
0.217
0.432

0.080
0.167
0.000
0.163
0.120
0.063
0.470
0.480
0.060
0.500
0.430
0.137
0.473
0.497
0.513
0.410
0.520

Table A.3: Performance comparison of searched agents from AgentSquare and (1) existing human-
designed agents (2) module search baselines (3) prompt search baselines based on GPT-3.5 on six
tasks across different domains.

Task
Webshop
ALFWorld
SciWorld
M3Tool

Planning Reasoning Tooluse
HTSS
SF-ToT
CoT

/
/
/

IO
TD
Voyager
/

CoT-SC ToolBF

TravelPlanner DEPS

PDDL

IR

CoT
CASRC

TH
/

Memory
Dilu
Generative Agents
Hier
/
/
Generative Agents

Best Hand-crafted Agents
HuggingGPT
Self-refine
Voyager
Toolbench
DEPS
OPENAGI

Table A.4: Comparison between the searched best agent from AgentSquare and the best human-
designed agent on all tasks.

16

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Method
AgentSquare(full)
w/o module evolution
w/o module recombination

Webshop ALFWorld SciWorld M3Tool TravelPlanner PDDL
0.219
0.202
0.173

0.617
0.595
0.578

0.432
0.288
0.310

0.285
0.236
0.258

0.651
0.623
0.546

0.520
0.483
0.267

Table A.5: Ablation study of AgentSquare on GPT-3.5 on six tasks across different domains.

Avg cost (GPT-3.5)
Iterations (GPT-3.5)
Avg cost (GPT-4o)
Iterations (GPT-4o)

Webshop ALFWorld SciWorld M3Tool TravelPlanner PDDL
$2.70
17
$26.94
12

$3.16
23
$10.51
15

$2.43
18
$26.03
14

$1.92
8
$42.14
9

$4.25
21
$13.96
18

$1.84
8
$29.75
9

Table A.6: Average API cost per search iteration and the total number of iterations until termination
for AgentSquare using GPT-3.5 and GPT-4o across six tasks.

Figure A.7: Performance versus API costs visualization on ALFWorld task.

Figure A.8: Performance versus API costs visualization on Webshop.

Figure A.9: Performance versus API costs visualization on Sciworld.

17

23456789API cost($)0.300.350.400.450.500.550.600.65AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareAlfworld based on GPT-3.520406080100API cost($)0.30.40.50.60.7AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareAlfworld based on GPT-4o246810API cost($)0.4250.4500.4750.5000.5250.5500.5750.6000.625AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareWebshop based on GPT-3.520406080100120API cost($)0.400.450.500.550.60AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareWebshop based on GPT-4o1.01.52.02.53.0API cost($)0.00.10.20.30.4AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareSciworld based on GPT-3.51015202530API cost($)0.20.30.40.50.60.70.8AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareSciworld based on GPT-4oUnder review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure A.10: Performance versus API costs visualization on M3tool.

Figure A.11: Performance versus API costs visualization on Travelplanner.

Figure A.12: Performance versus API costs visualization on PDDL.

18

0.51.01.52.02.53.03.54.0API cost($)0.000.050.100.150.200.25AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareM3Tool based on GPT-3.51015202530API cost($)0.250.300.350.400.450.50AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareM3Tool based on GPT-4o0.20.40.60.81.0API cost($)0.00.10.20.30.40.5AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareTravelplanner based on GPT-3.5810121416182022API cost($)0.00.10.20.30.40.50.6AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquareTravelplanner based on GPT-4o1.52.02.53.03.54.0API cost($)0.0500.0750.1000.1250.1500.1750.2000.225AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquarePddl based on GPT-3.51015202530API cost($)0.400.450.500.550.600.65AccuracyCoTCoT-SCSelf-refineToTStep BackTPHuggingGPTVoyagerGenerative AgentsDEPSOPENAGIDiluAgentSquarePddl based on GPT-4oUnder review as a conference paper at ICLR 2025

Figure A.13: AgentSquare search trajectory on M3tool and PDDL (more hand-crafted agents, spe-
cific module combinations when surpassing best hand-crafted and the final evolved agent, other
search baselines).

Figure A.14: AgentSquare search trajectory on Sciworld and Travelplanner (more hand-crafted
agents, specific module combinations when surpassing best hand-crafted and the final evolved agent,
other search baselines).

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

1234567891011121314Iteration0.160.180.200.220.240.260.28Success rateReplace the tooluse module with toolbench after module recombination Reasoning module = COT-SCTooluse module = ToolBFAgentSquare Search on M3toolBayesianRandomOPROAgentSquareBest of human design123456789101112Iteration0.540.560.580.600.620.640.66Progress rateReplace the memory module  after module evolutionPlanning moule = IR, Reasoning module = CASRCMemory module = Generative AgentsAgentSquare Search on PDDLBayesianRandomOPROAgentSquareBest of human design123456789Iteration0.00.10.20.30.4Progress rateReplace the planning module  after module evolutionPlanning module = Voyager, Reasoning module = CoTMemory module = HierAgentSquare Search on SciworldBayesianRandomOPROAgentSquareBest of human design123456789Iteration0.480.500.520.540.560.58Pass rateReplace the planning module with DEPS after module recombinationPlanning moule = DEPS, Reasoning module = CoTTooluse module = THAgentSquare Search on TravelplannerBayesianRandomOPROAgentSquareBest of human designUnder review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Figure A.15: New module discovered through AgentSquare search on Webshop.

20

113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187UnderreviewasaconferencepaperatICLR2025HTSSInsights:Observingthecurrentperformanceofthereasoningmodules,itseemsthattech-niqueslikeChain-of-Thought(CoT)andTree-of-Thoughts(ToT)haveofferedimprove-mentsbybreakingdownthetaskintosmallerstepsandevaluatingmultiplereasoningpaths.TheSelf-Consistency(SC)approachalsoshowspromisebygeneratingmultipleanswersandvotingonthem,whiletheSelf-Refinemoduleusesiterativeimprovementbasedonfeedback.OverallIdea:Tofurtherenhancetheperformance,wecanintegrateacombinationofthesetechniquesintoasinglemodule.Specifically,wecancombinetheTree-of-Thoughts(ToT)approachwithSelf-Consistency(SC)andSelf-Refine.Thiscombinedapproachwouldin-volvegeneratingmultiplereasoningpaths,evaluatingthemtochoosethebestpath,andtheniterativelyrefiningthechosenpathbasedonfeedback.Implementation:1.GenerateMultipleReasoningPaths:UsetheTree-of-Thoughts(ToT)approachtogeneratemultiplereasoningpaths.2.EvaluateandChoosetheBestPath:UseSelf-Consistency(SC)toevaluatethesepathsbyvotingandselectingthemostcommonoraccuratepath.3.RefinetheChosenPath:UseSelf-Refinetoiterativelyimprovethecho-senpathbasedonfeedback.4.PromptStructure:Craftpromptsthatencouragestep-by-stepproblem-solving,refertosimilarsolvedexamples,andrefineoutputbasedonfeedback.classREASONING_HYBRID_TOT_SC_SELFREFINE():def__init__(self,profile_type_prompt,memory,tooluse,llms_type):self.feedback=’’self.profile_type_prompt=profile_type_promptself.memory=memoryself.llm_type=llms_type[0]self.tooluse=toolusedef__call__(self,task_description:str,tool_instruction:str=’’,feedback:str=’’):task_name=re.findall(r’Instruction:\s+(.*?)\s+\[Search\]’,task_description)ifself.memoryisnotNone:self.task_name_cache=task_name[1]self.memory_cache=self.memory(task_description)iftask_description.count(’Reasoning’)==2:self.memory_cache=self.memory_cache.split(’Observation’)[0]eliftask_description.count(’Reasoning’)==4:self.memory_cache=’Observation’.join(self.memory_cache.split(’Observation’)[0:3])else:self.memory_cache=self.memory_cacheelse:self.memory_cache=’’ifself.tooluseisnotNone:tooluse=self.tooluse(task_description,tool_instruction)else:tooluse=’’split_text=task_description.rsplit(’WebShop’,1)examples=split_text[0]task_description=’WebShop’+split_text[1]prompt=’’’{tooluse}Solvethetaskstepbystep.Youroutputmustfollowtheexamplesprocess.Don’trefineyoursearch.Youhavetochooseonefromalistofitems.{memory}{examples}{task_description}’’’prompt=prompt.format(task_description=task_description,examples=examples,memory=self.memory_cache,tooluse=tooluse)reasoning_results=llm_response(prompt=prompt,model=self.llm_type,temperature=0.1,stop_strs=[’\n’],n=3)#EvaluateandchoosethebestpathfromcollectionsimportCounterstring_counts=Counter(reasoning_results)best_path=string_counts.most_common(1)[0][0]#Refinethechosenpathbasedonfeedbackrefined_result=self.refine(best_path)#reasoning_result=self.refine(reasoning_result)returnrefined_result22Under review as a conference paper at ICLR 2025

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Figure A.16: New module discovered through AgentSquare search on Sciworld.

21

124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295UnderreviewasaconferencepaperatICLR2025HierInsights:Theproposedmemorymodule’shierarchicalstructureofferssignificantadvan-tagesintaskmanagementforintelligentagents.Bybreakingdowneachtaskintosmallersub-tasksstoredseparately,thesystemenablesfocusedinformationretrieval,allowingtheagenttoaccessonlytherelevantdatainsteadofsiftingthroughanentiretasktrajectory.OverallIdea:Myproposedmemorymodulewillfocusoncreatingahierarchicalmemorystructure,whereeachtaskisbrokendownintosmallersub-tasksandeachsub-taskisstoredseparately.Thisapproachallowstheagenttoretrievefocusedinformationonspecificsub-tasksratherthananentiretasktrajectory.Additionally,thismemorymodulewillincludeafeedbackmechanismtoimprovememoryrelevanceandaccuracyovertime.Implementation:Theimplementationinvolvesmodifyingthememorymoduletostoreandretrievesub-tasktrajectoriesandintroduceafeedbackloopforcontinuousimprovement.classMEMORY_HIER():def__init__(self,llms_type)->None:self.llm_type=llms_type[0]self.embedding=OpenAIEmbeddings()db_path=os.path.join(’./db’,’memory/’)ifos.path.exists(db_path):shutil.rmtree(db_path)self.scenario_memory=Chroma(embedding_function=self.embedding,persist_directory=db_path)def__call__(self,current_situation:str=’’):if’Thecorrecttrajectoryis’incurrent_situation:self.addMemory(current_situation)else:returnself.retrieveMemory(current_situation)defretrieveMemory(self,query_scenario):sub_task_name=query_scenarioifself.scenario_memory._collection.count()==0:print("Thememoryvectordatabaseisempty.Cannotperformsearch.")return’’similarity_results=self.scenario_memory.similarity_search_with_score(sub_task_name,k=3)fewshot_results=[]foridxinrange(0,len(similarity_results)):fewshot_results.append(similarity_results[idx][0].metadata[’sub_task_trajectory’])return"\nHerearesimilarsub-tasksandthecorrecthandlingtrajectoriesinthesecases:"+’,’.join(fewshot_results)defaddMemory(self,current_situation):sub_task_trajectory=current_situationsub_task_name=re.search(r’Sub-task:(.*?)\nThecorrecttrajectoryis’,current_situation)ifsub_task_nameisnotNone:sub_task_descrip=sub_task_name.group(1)doc=Document(page_content=sub_task_descrip,metadata={"sub_task_name":sub_task_descrip,’sub_task_trajectory’:sub_task_trajectory})id=self.scenario_memory.add_documents([doc])deffeedbackMechanism(self,current_situation,was_successful):ifwas_successful:self.addMemory(current_situation)else:sub_task_name=re.search(r’Sub-task:(.*?)\nTheincorrecttrajectoryis’,current_situation)ifsub_task_nameisnotNone:sub_task_descrip=sub_task_name.group(1)doc_id=self.scenario_memory.search_documents(sub_task_descrip)[0].idself.scenario_memory.delete_document(doc_id)24Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

Figure A.17: New module discovered through AgentSquare search on M3tool.

22

113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187UnderreviewasaconferencepaperatICLR2025ToolBFInsights:Thepreviouslydiscoveredarchitecturesindicatethatleveragingmultipleinterac-tionsormultipleattemptstoidentifythemostsuitabletoolcanenhanceperformance(asinToolformer).Additionally,usingavectorsimilarityapproachtoretrievethemostrelevanttools(asinToolbench)seemspromising.OverallIdea:Iproposecombiningthevectorsimilarityapproachwithmultipleattemptstomaximizethechancesofselectingtheoptimaltool.Specifically,IwillaugmenttheTool-benchapproachbymakingmultiplecallstotheLLMtogenerateseveralpotentialsolutionsandthenselectingthebestonethroughavotingmechanism.Implementation:TheimplementationwillinvolveconvertinginstructionsandAPIdocu-mentationintovectorrepresentations,retrievingthemostrelevantAPIs,generatingmultipleresponsesusingtheLLM,andfinallyselectingthebestresponseusingavotingmechanism.classTOOLUSE_TOOLBENCHFORMER():def__init__(self,llms_type):self.llm_type=llms_type[0]self.scenario_memory={}forname,toolsintooluse_IO_pool.items():db_path=os.path.join(’./db’,f’api_pool{name}/’)self.embedding=OpenAIEmbeddings()self.scenario_memory[name]=Chroma(embedding_function=self.embedding,persist_directory=db_path)api_pattern=re.compile(r"\[(\d+)\]([ˆ:]+):(.+?)(?=\[\d+\]|\Z)",re.DOTALL)api_matches=api_pattern.findall(tools)documents=[]formatchinapi_matches:api_id,api_name,api_description=matchfirst_sentence=api_description.split(’.’)[0].strip()+’.’full_description=f"[{api_id}]{api_name}:{api_description.strip()}"doc=Document(page_content=first_sentence,metadata={"name":api_name.strip(),"description":full_description})documents.append(doc)self.scenario_memory[name].add_documents(documents)def__call__(self,task_description,tool_instruction,feedback_of_previous_tools):similarity_results=self.scenario_memory[task_description].similarity_search_with_score(tool_instruction,k=4)tool_pool=[]foridxinrange(0,len(similarity_results)):tool_pool.append(similarity_results[idx][0].metadata[’description’])prompt=f’’’Youhaveaccesstothefollowingtools:{tool_pool}Youneedtoselecttheappropriatetoolfromthelistofavailabletoolsaccordingtothetaskdescriptiontocompletethetask:{tool_instruction}Youmustusethetoolsbyoutputingthetoolnamefollowedbyitsarguments,delimitedbycommas.Youcanoptionallyexpressyourthoughtsusingnaturallanguagebeforeyouraction.Forexample,’Thought:Iwanttousetool_nametodosomething.Action:<youractiontocalltool_name>EndAction’.Youcanonlyinvokeonetoolatatime.Youmustbeginyourtoolinvocationwith’Action:’andenditwith’EndAction’.Yourtoolinvocationformatmustfollowtheinvocationformatinthetooldescription.{feedback_of_previous_tools}’’’strings=llm_response(prompt=prompt,model=self.llm_type,temperature=0.1,n=3)string=self.get_votes(tool_pool,tool_instruction,feedback_of_previous_tools,strings)returnstring22Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Figure A.18: New module discovered through AgentSquare search on Travelplanner.

23

118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241UnderreviewasaconferencepaperatICLR2025THInsights:Fromthecurrentlyexploredarchitectures,the’Toolformer’approachseemstohavethehighestperformanceat0.56,whichsuggeststhatgeneratingmultiplecandidateresponsesandthenvotingonthebestoneiseffective.Anotherobservationisthatahierar-chicalsearchapproachlikein’Anytool’mighthelpinbettercategorizingandselectingtoolsbasedonthetask.OverallIdea:I’llcombinethehierarchicalsearchstrategywiththecandidateresponsegen-erationandvotingmethod.Thiswillinvolvefirstcategorizingthetoolbasedonthetaskdescriptionandthengeneratingmultiplecandidateresponsestoselectthebestone.Thisshouldleveragethestrengthsofbothmethods.Implementation:Iwillimplementaclasswherethetoolisfirstselectedusingahierarchi-calsearchstrategy,andthenmultipleresponsesaregeneratedfortheselectedtool,followedbyavotingmechanismtoidentifythebestresponse.classTOOLUSE_TH():def__init__(self,llms_type):self.llm_type=llms_type[0]self.tool_description=functions_infoself.tool_pool=travelplanner_toolpool()category_prompt=category_prompt()string=llm_response(prompt=category_prompt,model=self.llm_type,temperature=0.1)dict_strings=re.findall(r"\{[ˆ{}]*\}",string)self.dicts=[ast.literal_eval(ds)fordsindict_strings]def__call__(self,task_description,tool_instruction,feedback_of_previous_tools):prompt=f’’’{self.dicts}Youneedtoselecttheappropriatetoolcategoryfromthelistofavailabletoolsaccordingtothetaskdescriptiontocompletethetask:{task_description}{tool_instruction}Youcanonlyinvokeonecategoryatatime.{feedback_of_previous_tools}Outputcategorynamedirectly.Youroutputshouldbeofthefollowingformat:Categoryname:’’’category_name=llm_response(prompt=prompt,model=self.llm_type,temperature=0.1).split(’:’)[-1].strip()matching_dict=Nonefordinself.dicts:ifd.get(’categoryname’)==category_name:matching_dict=dbreakifmatching_dictand’toollist’inmatching_dictandmatching_dict[’toollist’]:matched_tools={tool:self.tool_description[tool]fortoolinmatching_dict[’toollist’]iftoolinself.tool_description}else:matched_tools=random.choice(list(self.tool_description.keys()))prompt=f’’’{matched_tools}Theuser’squeryis:{task_description}Thetool-useinstructionforcurrenttaskis:{tool_instruction}Youcanonlyinvokeonetoolatatime.{feedback_of_previous_tools}Youanswershouldfollowtheformat:tool_type[tool_arg],suchasFlightSearch[NewYork,London,2022-10-01]’’’strings=llm_response(prompt=prompt,model=self.llm_type,temperature=0.1,n=3)string=get_votes(matched_tools,tool_instruction,feedback_of_previous_tools,strings)returnstring23Under review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Figure A.19: New module discovered through AgentSquare search on Pddl.

24

113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187UnderreviewasaconferencepaperatICLR2025CASRCInsights:Thecurrentapproacheshaveexploreddirectreasoning,step-by-step(Chain-of-Thought),andself-refinementtechniques.Notably,the’Chain-of-Thought’and’Self-Refine’methodshaveshownpotentialbydecomposingthetaskanditerativelyimprovingthesolu-tion.However,despitetheseefforts,theperformancestillhoversaround50-55%,indicatingroomforimprovement.OverallIdea:Tofurtherenhancetheperformance,Iproposecombiningelementsfromthehigh-performingmethods(Chain-of-ThoughtandSelf-Refine)withanewfocuson.classREASONING_CONTEXT_AWARE_SELF_REFINE_COT():def__init__(self,profile_type_prompt,memory,tooluse,llms_type):self.feedback=’’self.profile_type_prompt=profile_type_promptself.memory=memoryself.llm_type=llms_type[0]self.tooluse=tooluseself.context=’’def__call__(self,task_description:str,tool_instruction:str=’’,feedback:str=’’):ifself.memoryisnotNone:memory=self.memory(task_description)else:memory=’’ifself.tooluseisnotNone:tooluse=self.tooluse(task_description,tool_instruction)else:tooluse=’’#Setuptheinitialpromptprompt=f’’’Solvethetaskstepbystep.Interactwithahouseholdtosolveatask.Yourinstructionsshouldfollowtheexamples.{memory}{tooluse}Hereisoneexample.Task:"id":-1,"task":"pddl","goal":"Thegoalistosatisfythefollowingconditions:shot1containscocktail6.shot2containsingredient1.shot3containsingredient2.","subgoals":["shot1containscocktail6.","shot2containsingredient1.","shot3containsingredient2."],"difficulty":"hard","additional_info":"subtask":"barman"Inthistask,thegoalistohave3shotscontainingspecificingredientsorcocktails.Thistaskfallsunderthe"barman"category,whichinvolvesmixingandservingdrinks.Assumingabartenderrobotwithagripperhand,let’sbreakdownthestepstoachievethis:1.Thefirstsubgoalistohaveshot1containcocktail6.Therobotwillneedtolocatecocktail6,graspitusingitsgripper,andpouritintoshot1.2.Thesecondsubgoalistohaveshot2containingredient1.Therobotwilldothesameasinstep1,butthistimelocatingingredient1andpouringitintoshot2.3.Thethirdsubgoalistohaveshot3containingredient2.Therobotwillagainrepeattheprocess,locatingingredient2andpouringitintoshot3.Afterthesesteps,therobotwillhaveachievedallthesubgoals,thuscompletingthemaintask.ATTENTION:Youshouldansweravalidactiondirectly!Now,hereisthetaskyouneedtosolve:{task_description}’’’#print(’prompt:’,prompt)#input()#prompt=prompt.format(task_description=task_description,memory=memory,tooluse=tooluse)reasoning_result=llm_response(prompt=prompt,model=self.llm_type,temperature=0.1,stop_strs=[’\n’]).replace(’>’,’’).strip()reasoning_result=self.refine(task_description,reasoning_result)returnreasoning_result22Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

Figure A.20: New module discovered through AgentSquare search on Pddl.

25

113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187UnderreviewasaconferencepaperatICLR2025IRInsights:TomaximizetheperformanceoftheagentonALFworldtasks,weshouldcon-siderincorporatingfeedbackloopsanditerativerefinementintheplanningprocess.Fromthediscoveredarchitectures,itseemsthatthemosteffectivemodules(DEPSandopenagi)providedetailedsub-goalsandmakeuseofiterativeimprovementsbasedonfeedback.OverallIdea:Ournextplanningmodulewillfocusoniterativeplanningwithfeedbackin-corporation.Aftergeneratinganinitialsetofsub-tasks,themodulewillprompttheLLMtorefinetheplanbyexplicitlycheckingdependenciesandcompletenessofthesub-tasks.Implementation:Wewillcreateaplanningmodulethatgeneratesaninitialsetofsub-tasksandthenrefinesitbasedonfeedback.Thisrefinementwillensurethatthesub-tasksareco-herent,minimal,andcomplete,ensuringbetterperformanceinsequentialdecision-makingtasks.classPLANNING_ITERATIVE_REFINEMENT():def__init__(self,llms_type):self.plan=[]self.llm_type=llms_type[0]def__call__(self,task_type,task_description,feedback):few_shot=’’’Goal:Thegoalistosatisfythefollowingconditions:b1isonb2.,b2isonb3.\nObservation:B1isonthetable.B2isonthetable.B3isonthetable.Robotarmisempty.Theb1isclear.Theb2isclear.Theb3isclear.sub-task1:{{’description’:’Ineedtostackb2onb3first’,’reasoninginstruction’:’b2isonb3’,’tooluseinstruction’:None}}sub-task2:{{’description’:’ThenIneedtostackb1onb2’,’reasoninginstruction’:’b1isonb2’,’tooluseinstruction’:None}}’’’prompt=’’’Youareaplannerwhodividesa{task_type}taskintoseveralsubtasks.First,generateaninitialsetofsubtaskstoachievethefinalgoal.Aftergenerating,refinethesubtasksbyensuringtheycoverallnecessarysteps,areinthecorrectorder,andhavenoredundancies.Youroutputformatshouldfollowtheexamplebelow.Thefollowingaresomeexamples:Task:{example}’’’iffeedback==’’:prompt=prompt+’’’Task:{task_description}’’’prompt=prompt.format(example=few_shot,task_description=task_description,task_type=task_type)else:prompt=prompt+’’’end--------------------Reflexion:{feedback}Task:{task_description}’’’prompt=prompt.format(example=few_shot,task_description=task_description,task_type=task_type,feedback=feedback)#Initialresponseinitial_response=llm_response(prompt=prompt,model=self.llm_type,temperature=0.1)initial_dict_strings=re.findall(r"\{[ˆ{}]*\}",initial_response)initial_dicts=[ast.literal_eval(ds)fordsininitial_dict_strings]#Refinementphaserefinement_prompt=’’’Youareanexpertplannertaskedwithrefiningthefollowingsubtasks.Ensureallnecessarystepsarecovered,theyareinthecorrectorder,andtherearenoredundancies.Youroutputformatshouldfollowtheexamplebelow.Thefollowingaresomeexamples:Task:{example}end--------------------Subtasks:{subtasks}’’’.format(subtasks=initial_dicts,example=few_shot)refined_response=llm_response(prompt=refinement_prompt,model=self.llm_type,temperature=0.1)refined_dict_strings=re.findall(r"\{[ˆ{}]*\}",refined_response)refined_dicts=[ast.literal_eval(ds)fordsinrefined_dict_strings]self.plan=refined_dictsreturnself.plan