Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage
Under review as a conference paper at ICLR 2025

MULTI-MODAL AGENT TUNING: BUILDING A VLM-

DRIVEN AGENT FOR EFFICIENT TOOL USAGE

Anonymous authors
Paper under double-blind review

Figure 1: The comparison of the LLM (GPT-4)-driven agent and our T3-Agent. Our agent chooses
more precise tools based on the given files and intermediate observations.

ABSTRACT

The advancement of large language models (LLMs) prompts the development of
multi-modal agents, which are used as a controller to call external tools, providing a
feasible way to solve practical tasks. In this paper, we propose a multi-modal agent
tuning method that automatically generates multi-modal tool-usage data and tunes
a vision-language model (VLM) as the controller for powerful tool-usage reasoning.
To preserve the data quality, we prompt the GPT-4o mini model to generate queries,
files, and trajectories, followed by query-file and trajectory verifiers. Based on the
data synthesis pipeline, we collect the MM-Traj dataset that contains 20K tasks
with trajectories of tool usage. Then, we develop the T3-Agent via Trajectory
Tuning on VLMs for Tool usage using MM-Traj. Evaluations on the GTA and
GAIA benchmarks show that the T3-Agent consistently achieves improvements
on two popular VLMs: MiniCPM-V-8.5B and Qwen2-VL-7B, which outperforms
untrained VLMs by 20%, showing the effectiveness of the proposed data synthesis
pipeline, leading to high-quality data for tool-usage capabilities.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Thought: Use the `facedetection` tool to detect human faces in the two images. Code:faces_image1 = facedetection("image_1.jpg") faces_image2 = facedetection("image_2.jpg")Observation: Faces in Image 1: 4bounding boxesFaces in Image 2: []Thought: There are 4 children in total. The price of a PS5 is approximately $500, so the cost is 4*500.Code:Final_answer=4* 500.Thought: First analyze the image 1 to find the number of identifiable children. Then determine the price of PS5 from image2. Code:faces_image= facedetection("image_1.jpg") price =image_qa("whatis the price?", "image_2.jpg")Observation: Faces in Image 1: 4bounding boxesPrice of PS5: $479.99Thought: Using the price of $479.99 for each console.Code:Final_answer= 4*479.99Query: I want to buy a PS5 for each child in the photo. How many dollars will I need to spend in total?Query: The men in the picture want to buy one NVIDIA GeForce RTX 4070 SUPER each. According to the price in January, how many dollars will they need to spend in total?Thought: Use the `facedetection` tool to detect the number of human faces in the imageCode:face = facedetection("image_1.jpg")Observation: Faces : 3 bounding boxesThought: Use the `image_qa`tool for the price of NVIDIA GeForce RTX 4070 SUPER.Code:price = image_qa(" whatis the price ofGeForce RTX 4070 SUPER for three men”, "image_1.jpg")Observation: This image does not provide any price. Thought: Icannot obtain useful information. I think the price is about $1800 for three men. Code: Final_answer= 1800Thought: Use the `facedetection` tool to detect the number of human faces in the imageCode:face = facedetection("image_1.jpg")Observation: Faces : 3 bounding boxesThought: Use the `web_search` tool for the price of NVIDIA GeForce RTX 4070 SUPER.Code:price = web_search(query="NVIDIA GeForce RTX 4070 SUPER price")Observation: … … On January 8, 2024, Nvidia released the RTX 4070 SUPER at $599, ……Thought: The price is $599. There are three men.Code: Final_answer= 3 * 599UserUserGPT-4 driven AgentGPT-4 driven AgentT3 AgentT3 AgentUnder review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

INTRODUCTION

1
Integrating external tools to solve diverse multi-modal tasks is a promising research direction towards
multi-modal agents (Surís et al., 2023; Gupta & Kembhavi, 2023; Gao et al., 2024; Yuan et al., 2024;
Zhong et al., 2023). Existing agents usually use a large language model (LLM) as the controller that
generates plans via prompt engineering to call tools, achieving impressive performance in multiple
domains, such as image editing (Wu et al., 2023), robotic manipulation (ichter et al., 2023), question
answering (Shen et al., 2024), video understanding (Fan et al., 2024), and desktop APPs (Trivedi
et al., 2024). Despite their success, prompt engineering faces limited reasoning abilities for tool usage
in tackling practical tasks, as shown in Fig. 1. (1) The in-context examples in prompts only involve
textual information, degrading the efficiency of tool usage in the multi-modal world. For the query
‘How many dollars will I need to spend to buy a PS5 for each child?’, the agent may select improper
tools if it does not know what the two images depict. (2) The pre-defined in-context examples are
fixed and cannot tackle all tasks in the real world. For the task that requires searching for information
from the web, the agent cannot use the proper tools, if in-context examples tend to use the ‘image_qa’
tool. This motivates us to enhance the controller’s reasoning ability for efficient tool usage.

In this paper, we propose a multi-modal agent tuning method that automatically generates a large num-
ber of multi-modal tasks with tool-usage trajectories and tunes a vision-language model (VLM) (Liu
et al., 2024b; Chen et al., 2024d; Yao et al., 2024) as the controller for powerful tool-usage reasoning.
Compared with LLM-driven agents, VLM-driven agents can utilize multi-modal information (such as
required knowledge domains in the multi-modal data) instead of only using the query for reasoning,
benefiting efficient tool usage (Liu et al., 2024c; Wang et al., 2024a; Sun et al., 2024). Many efforts
are made to enhance specific capabilities of VLMs via finetuning, such as the chain-of-thought
ability (Hu et al., 2024), grounding ability (Peng et al., 2023), and feedback-refining ability (Li et al.,
2024). This inspires us to construct a large number of multi-modal tool-usage data for VLM-driven
agents, which improves the reasoning ability when using tools for real-world tasks.

In doing so, we need to overcome two challenges. (1) Collecting multi-modal tasks is challenging.
Tasks in the real world usually involve multiple tools for multiple files (images, textual files, videos,
audio, and etc). There are few off-the-shelf datasets for such tasks, and prompting models to generate
natural and diverse queries with matched files is non-trivial. (2) Generating trajectories is challenging.
Due to the complexity of trajectories, existing methods usually manually define templates and fill in
key information for trajectory generation. This will limit the diversity of synthesis data and cause
weak generalization for real-world tasks.

To overcome the above challenges, we introduce a novel tool-usage data synthesis pipeline that auto-
matically generates a large number of multi-modal tool-usage data via three steps: query generation,
file generation, and trajectory generation. Concretely, we first prompt GPT-4o mini (OpenAI, 2024)
to generate queries and analyze what files are needed to solve the queries. Then, we produce files via
two strategies. If needed files are images, we search for them from existing image datasets; otherwise,
we prompt GPT-4o mini to produce codes to generate the needed files. Finally, we prompt a zero-shot
agent to solve the generated tasks (i.e., queries and files) and collect trajectories, including the
thoughts and codes in task solving. To preserve the data quality, the generated tasks and trajectories
are passed through two verifiers to discard low-quality data. After that, we use these data to tune a
VLM for efficient tool usage, through which one agent driven by the trained VLM could generate
precise thoughts and codes for real-world tasks.

With the data generation pipeline, we construct MM-Traj, a dataset that contains 20K multi-modal
tasks with tool-usage trajectories. Based on MM-Traj, we introduce the T3-Agent, a VLM-driven
agent in the ReAct framework (Yao et al., 2023). The VLM controller of the T3-Agent is developed
via Trajectory Tuning for Tool usage using MM-Traj. We conduct comprehensive evaluations of
the T3-Agent on the GTA (Wang et al., 2024b) and GAIA benchmarks (Mialon et al., 2023), where
two popular VLMs are used as the controller, that is MiniCPM-V-8.5B (Yao et al., 2024) and Qwen-
VL-7B (Wang et al., 2024c). The T3-Agent consistently achieves improvements on the two VLMs
and outperforms the untrained VLMs by 20%. This indicates that our multi-modal agent tuning
method enables agents a powerful tool-usage capability for practical tasks with complex and diverse
trajectories.

In summary, our contributions are three-fold. (1) We propose a multi-modal agent tuning method that
automatically generates a large number of multi-modal tasks with trajectories and tunes VLMs using
the generated data for powerful tool usage. (2) We introduce MM-Traj, a multi-modal tool-usage

2

Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

dataset that contains 20K tasks across diverse knowledge domains with 15K files and high-quality
trajectories. (3) We develop the T3-Agent, a multi-modal tool-usage agent that significantly improves
the tool usage performance on two popular benchmarks: GTA and GAIA.

2 RELATED WORK

2.1 MULTI-MODAL AGENT
Using external tools to address complex tasks is an important ability for multi-modal agents. Ac-
cording to different controllers, existing agents can be categorized into LLM-driven agents and
VLM-driven agents. LLM-driven agents utilize powerful LLMs as the controller and produce pseudo
code (Gupta & Kembhavi, 2023; Gao et al., 2024), python code (Surís et al., 2023; Yuan et al., 2024),
or JSON format (Shen et al., 2024) to call tools via one-step reasoning. Considering the complexity
of practical tasks, some methods (Yang et al., 2023; Fan et al., 2024; Yang et al., 2024) empower
the agent with step-by-step reasoning, which allocates tools based on observations of previous steps.
Compared with LLM-driven agents, VLM-driven agents are more efficient in task solving, since the
VLM controller can utilize information from visual data in tool usage, showing superior performance
in visual design (Sasazawa & Sogawa, 2024), web search (Zheng et al., 2024a), image editing (Wang
et al., 2024e), embodied scenario (Zheng et al., 2024b), robotic manipulation (Sun et al., 2024), and
etc. However, VLM-driven agents have a weaker reasoning ability, compared with LLM-driven
agents. Thus, several works synthesize training data to tune open-source VLMs for general tool
usage (Wang et al., 2024a; Liu et al., 2023a; 2024c). Due to the challenges of trajectory generation,
existing methods mainly focus on simple tasks requiring one or two tools, and only synthesize a small
amount of data (e.g., 1K in (Liu et al., 2024c)). Different from them, our T3-Agent is tuned using
scaled-up multi-modal data with complex tool-usage trajectories, through which our agent could
solve more practical tasks with strong tool-usage capability.

2.2 TOOL-USAGE DATASET
Several tool-usage datasets have been established for agents, such as APIBank (Li et al., 2023), Toolal-
paca (Tang et al., 2023), ToolBench (Qin et al., 2023), AnyTool (Du et al., 2024), agentohana (Zhang
et al., 2024a), APIGen (Liu et al., 2024d), and AgentInstruct (Zeng et al., 2023). The above datasets
contain little multi-modal data (e.g., images, videos, and audios) that are commonly encountered in
the real world. Thus, to evaluate the performance of agents in solving multi-modal tasks, some multi-
modal agent benchmarks have been built, such as the GUI benchmarks: OSWorld (Xie et al., 2024)
and MMInA (Zhang et al., 2024b), multi-modal question answering benchmarks: GAIA (Mialon
et al., 2023), GTA (Wang et al., 2024b), and m&m’ (Ma et al., 2024), and comprehensive benchmarks:
AgentBench (Liu et al., 2023b) and AgentGym (Xi et al., 2024). In addition, some efforts are paid
to synthesize trajectory data using LLMs to improve the tool-usage ability of multi-modal agents.
DEDER (Choi et al., 2024) resorts to in-context learning to generate trajectories, through which
the chain-of-thought reasoning ability is distilled from LLMs to a small model. Lumos (Yin et al.,
2024) converts ground-truth reasoning steps in existing benchmarks into the expected format of
tool-usage trajectories. TASKBENCH (Shen et al., 2023) samples trajectories from a predefined
graph, and then generate queries. MLLM-Tool (Wang et al., 2024a) and LLaVA-plus (Liu et al.,
2023a) collect trajectories based on image and tool descriptions. VisualAgentBench (Liu et al.,
2023b) manually designs trajectory templates to collect data. Different from the above methods that
usually focus on simple and predefined trajectories, or rely on queries in off-the-shelf datasets, our
data collection pipeline does not have any constraints, which generates diverse tasks and complex
trajectories, improving the volume, complexities, naturalness, and diversity of the tool-usage dataset.

3 DATA COLLECTION

3.1 FORMULATION
Data Synthesis Pipeline. The proposed data synthesis pipeline is shown in Fig. 2, including three
steps: query generation, file generation, and trajectory generation. To preserve the quality of data, we
design a query-file verifier and a trajectory verifier to discard low-quality data.

Data format. We format the multi-modal tool-usage data as {Fopt, Q, T, C, O, A}, where Fopt
denotes the multi-modal files, Q means the query, T mean the generated thought (i.e., plans to call

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

Figure 2: The pipeline for data generation.

tools), and C means the generated code, O means observation (outputs of using tools), and A means
the ground truth answer. opt means that the files are optional, i.e., some queries do not involve files.

The query Q could be divided into two categories, question answering and image generation, where
the answer A would be some descriptive texts for the former and images for the latter. In our setting,
F includes 11 types of files, such as JPG, PDF and PPTX files (details in Appendix). Considering
that solving one real-world task may require multiple steps involving multiple tools, T , C, and O
can be represented by the integration of thought, code, and observation in multiple steps, and the
data format is reformulated as {Fopt, Q, {t1, · · · , tn}, {c1, · · · , cn}, {o1, · · · , on}, A}, where ti, ci,
and oi indicate the thought, code, and observation in the i-step respectively, and there are n steps in
total. The thought, code, and observation are composed of a trajectory {t1, c1, o1, · · · , tn, cn, on} of
n steps to solve the query.

Image source. Visual information plays an important role in multi-modal tasks. To enhance the
diversity and comprehensiveness of our tool-usage data, we compile about 93K image-captioning
pairs from 8 source datasets, including ChartQA (Masry et al., 2022), COCO (Lin et al., 2014),
LLaVA (Wang et al., 2023a), SAM (Kirillov et al., 2023), TextVQA (Singh et al., 2019), Web-
Celebrity (Liu et al., 2015), Web-Landmark (Weyand et al., 2020), and WikiArt (Saleh & Elgammal,
2015). These datasets cover multiple multi-modal tasks: visual question answering, chart/table/doc-
ument analysis, science question answering, and etc. We use the ShareGPT4V model (Chen et al.,
2024b) to produce captions for each collected image.

3.2 QUERY GENERATION
Our goal is to generate a large number of diverse, practical, and feasible queries. We manually write
some seed queries by brainstorming and double-checking them to ensure their practicality. In each
step of query generation, we feed several randomly sampled seed queries, tools with descriptions,
and a designed prompt to the GPT-4o mini model that generates multiple queries. Adding tool
descriptions to the prompt makes GPT-4o mini better understand the desirable queries, improving the
feasibility of generated queries. We tune the hyperparameters (such as temperature) of GPT-4o mini
to improve the diversity of generation.

4

Tool SetQuery SeedPromptYouaretaskedwithgeneratinguserqueriesthatwillpromptanagenttocallvarioustools.Ourtoolse:<TOOL_SET>Iwillnowprovideuserqueryexamples:<QUERY_SEED>PleaseoutputtheQueriesinajsonformat.QueryCirclethehabitatofthisanimalonthemap.FileContentFile1:.jpgfile.Twoimagescontaincallithrixflavicepsanimalandtheworldmap.File2:.pdffile.Onefilecontainshabitatsofvariousanimals,includingtheBrazilhabitatforcallithrixflaviceps.ImageOther filesImage databaseTrajectoryThought:Usetheimage_qatooltoidentifyanimals.Code:a1=image_qa(‘whatistheanimal?’,’image1.jpg’)print(f‘Theanimalis{a1}.’)Observation:Theanimalsiscallithrixflaviceps.Thought:Usethefile_inspectortooltoidentifythehabitat.Code:l1=file_inspector(‘habitatofcallithrixflaviceps’,‘habitat.pdf’)print(f‘Thehabitatis{l2}’)Observation:ThehabitatisBrazil.Thought:Usetheimage_edittooltocircleBrazil.Code:final_answer=image_edit(‘drawacircleonBrazil’,‘image2.pdf’)Query-file checkTrajectory checkZero-ShotAgentQuery GenerationFile GenerationTrajectory GenerationCode for file generationImage retrievalFile content generationMM-trajUnder review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

3.3 MULTI-MODAL FILE GENERATION
Different from existing multi-modal data synthesis methods that first sample multi-modal files (images
in most cases) from off-the-shelf datasets and then feed the files to language models (e.g., ChatGPT)
for query generation, we opt to first generate queries without files and then produce relevant files
for the queries. The reasons are two aspects. (1) Practical tasks usually involve not only images
but also other multi-modal files, such as DOCX, PPTX, XLSX, and PDF files. It is challenging to
construct off-the-shelf datasets that contain enough files for real-world tasks. (2) Tasks are usually
based on multiple files instead of only one, while randomly sampled files may have weak relevance
and feeding them to language models usually produces non-natural queries. It is non-trivial to design
a standard to automatically sample relevant files for query generation. (3) Using existing files to
generate queries may limit the knowledge domain, decreasing the diversity of tasks. In contrast,
generating files based on generated queries may lead to more diversity.

Concretely, for each generated query, we prompt GPT-4o mini to output the needed file type and file
content. The files are divided into two categories: images and others. For needed images, we use the
BGE model (Chen et al., 2024a) to extract textual embeddings of the file content and compare its
similarities with collected source images from off-the-shhackelf datasets. The top similar images are
collected for the query. For other needed files, we prompt GPT-4o mini to extend the file content and
generate Python code to produce the files.

3.4 ZERO-SHOT AGENT INFERENCE
Trajectories are collected by prompting a zero-shot agent (without training) to solve our generated
tasks (i.e., queries and files). We utilize the framework of ReAct agents (Yao et al., 2023), where
GPT-4o mini is employed as the controller. It solves the query into multiple steps and generates
thought and code for each step based on the observations of previous steps. We collect trajectories
whose code can be executed, including the thoughts, codes, and observations of all steps. The details
of the agent can be found in Section Appendix C and Appendix D.

3.5 DATA VERIFICATION
To preserve the quality of generated tool-usage data, we design a query-file verifier and a trajectory
verifier to discard low-quality data. Using LLMs to verify the quality of LLM outputs has shown
effectiveness in multiple methods, such as verifying the generated instructions (Wang et al., 2023b)
and verifying the generated plans (Liu et al., 2024d). Inspired by them, we argue that using LLMs
can also verify the synthetic tasks and trajectories.

Query-file verifier. Considering that the generated queries may be infeasible to solve and the
produced files may not match the queries, the query-file verifier filters out low-quality query-file pairs.
We prompt GPT-4o mini as the verifier based on the following factors: (1) whether the query and files
are relevant; (2) whether the files contain enough information to solve the query; and (3) whether the
query can be solved based on the given tools.

Trajectory verifier. Similarly, we prompt GPT-4o mini as the trajectory verifier based on the
following factors: (1) the trajectory should use the provided tools as much as possible; (2) the
trajectory should be reasonable, that is, the trajectory should align with the object and context of the
query; (3) the tool usage in the trajectory should consistent with the query and files; (4) the input
arguments for tools in the trajectory should be correct; (5) the answer should be correctly summarized
from observations of tool-usage; and (6) the final answer should be relevant to the query.

3.6 MM-TRAJ DATASET
Data that passes through the two verifiers is considered high-quality and collected in an MM-Traj
dataset. In summary, we collect 23.5K data points from query generation and file generation. After
passing through the two verifiers, 20K data points are left with 15K files.

Scalability. Note that our method can extend to additional modalities by incorporating more tools and
leveraging advanced multi-modal models. For example, to extend our method to the video modality
(MP4, MOV), we can integrate a video search model into the data synthesis pipeline (like the image
modality), and apply a video-language model to the agent controller with powerful video processing
models as tools. This approach ensures seamless adaptation to new modalities while maintaining
efficiency and coherence.

5

Under review as a conference paper at ICLR 2025

(a) File type

(b) Domain Knowledge

(c) Tool Statistic

(d) Step Number

Figure 3: Data statistics on the MM-Traj dataset.

3.6.1 DATASET ANALYSIS

We provide four key statistics: file type, knowledge domain, step number, and used tools in the
collected MM-Traj Dataset. File type. We show the distribution of involved files in Fig. 3(a), which
reflects the diversity of our dataset. MM-Traj covers more than 9 files, all of which are commonly
encountered in the real world. Thus the T3-Agent trained on MM-Traj can handle practical tasks
since the multi-modal knowledge is not limited to images in our lives. Knowledge domain. We show
the involved knowledge of the generated tasks in Fig. 3(b), which can be divided into 16 non-overlap
categories, spanning across finance, environment, culture, health, history, food, and etc. Training in
such data provides rich knowledge to use tools to solve diverse practical tasks. Tools. In Fig. 3(c),
We show the distributions of used tools in generated trajectories. The web search tool is the most
commonly used tool, which is consistent with practical tasks requiring specific knowledge. Moreover,
other tools are also widely used in our dataset. Step number. We show the distributions of step
numbers of generated trajectories in Fig. 3(d). Trajectories in the MM-Traj dataset have diverse
step numbers. Most tasks require 2-6 steps to solve and some tasks require 7-8 steps, showing the
complexity and diversity of our dataset.

4 T3-AGENT

4.1 WORKFLOW
To handle practical tasks that require complex trajectories, we opt for the framework of the ReAct
agent that performs step-by-step reasoning for tool usage based on observations of previous steps.
In each step, the agent generates thought and corresponding code to execute tools. Compared with
other formats (e.g., JSON format), code is more flexible to handle different inputs and output types
for various tools. Concretely, given a query Q and files F , the i-step of the agent is formulated as

i , c⋆
t⋆

i = arg max P (ti, ci|Fopt, Q, hi),

(1)

and c⋆
i

where t⋆
and hi =
i
{t1, c1, o1, · · · , ti−1, ci−1, oi−1} denotes the history (thought, code, and observation of previous
steps).

are generated thought and code for

the i-th step,

6

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

.23.4%.docx6.4%.xlsx18.6%1.1%3.0%15.5%25.9%.png.jpg2.2%  other.mp3 2.2% .mp41.7%  .pptx.csv.pdf16.4%Environment7.3%14.5%5.8%11.4%10.1%2.6%4.8%2.1%5.5%CultureHealthHistoryFoodFinance ScienceOthersTransportationTravel9%SportsSocialEntertainment4.2%1.4.4%5%2.Technology2.4%NatureGeography4.2%35.8%24.5%3.6%Object Localization14.5%8.3%3.6%6.0%3.7%Face DetectionWeb SearchImage Question AnsweringFile  InspectorImage Editing SegmentationImage Generation VisualUnder review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 1: Used tools in the T3-Agent

Tool name
Web search
Image question answering
File inspector
Visual segmentation
Object localization
Image generation
Image editing
Face detection
Python package

Tool description
Perform complicated web browsing to answer a question
Answer questions for queries based on attached images
Answer questions for queries based on given files
Do instance segmentation on the given image
Localize objects in given images and output the bounding boxes
Create an image according to a textual prompt
Edit image based on the textual prompt
Detect human faces in given images and output the bounding boxes
Various packages such as ‘matplotlib’ and ‘opencv’

4.2 TOOLS
We deploy real-executable tools for the agent instead of only providing tool names. Our tools are
across multiple categories: web search, visual perception, image generation/editing, file understand-
ing, multi-modal understanding, and multiple Python packages, as shown in Tab. 1.

4.3 TRAINING
Given a data point {Fopt, Q, {t1, · · · , tn}, {c1, · · · , cn}, {o1, · · · , on}, A}, we train the VLM con-
troller using the cross-entropy loss,

min E(Fopt,Q,T ,C,O,A)∼D

−

(cid:34)

n
(cid:88)

i=1

(cid:35)

P (ti, ci|Fopt, Q, hi)

,

(2)

where D is the collected MM-Traj dataset and we sum the loss values of the n steps in the trajectory.
Note that, in training VLMs, we do not use the final answer A, as we encourage the controller to
leverage tools in solving given tasks, instead of directly producing an answer based on its internal
knowledge. After training, we obtain the T3-Agent.

Model. We use the same model architectures as MiniCPM-V-8.5B (Yao et al., 2024) and Qwen2-VL-
7B (Wang et al., 2024c) as our VLM controllers, including their visual encoders, resamplers, and
LLMs. We initialize the model using their released versions.

5 EXPERIMENTS

5.1 SETTING
To evaluate the effectiveness of the proposed multi-modal agent tuning method, we evaluate the
T3-Agent on the GTA (Wang et al., 2024b) and GAIA (Mialon et al., 2023) benchmarks and compare
it with agents that use closed-source models (GPT-4, GPT-4o, and Claude3) and open-source models
(LLaMA-3-70B-instruct (Dubey et al., 2024), Qwen1.5-72B-chat (Bai et al., 2023), LLaVA-NeXT-
8B (Liu et al., 2024a), InternVL2-8B (Chen et al., 2024c), Qwen2-VL-7B (Wang et al., 2024c), and
MiniCPM-V-8.5B (Yao et al., 2024)) as the controllers. Concretely, we compare the T3-Agent with
Lego Agent (AgentLego Contributors, 2023), Sibyl Agent (Wang et al., 2024d), and the Warm-up
Act Agent (Mialon et al., 2023). The huggingface agent (HF Agent) (HuggingFace Contributors,
2024) is the baseline agent, using the same tools as the T3-Agent. We conduct ablation experiments
to evaluate our data synthesis pipeline and visualize the task-solving process of our T3-Agent.

Training. To preserve the visual perception and reasoning capabilities of MiniCPM-V and Qwen2-
VL, we combine the training data in MM-Traj with the data in Cauldron (Lindström & Abraham,
2022) and open-LLaVa-NeXT (Chen, 2024) datasets. We train 5 epoch over all data. In the training
process of our VLM controller, we freeze the vision encoder and visual token compressor, and
fine-tune the language model using LoRA (Hu et al., 2022). We set the rank as 64 and apply LoRA
on query, key, and value projection matrices in all self-attention layers. We use the AdamW optimizer
with a cosine annealing scheduler. The learning rate is 1e − 6 and the batch size is 2. We set the max
context window to 10240 to support the long trajectory of our agent.

Benchmark. GTA and GAIA benchmarks are comprehensive evaluation benchmarks for multi-modal
agents. GTA contains 229 tasks with 252 images, and the steps required to solve tasks range from
2 to 8, with most questions requiring 2 to 4 steps. It requires multi-modal agents to build powerful

7

Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 2: Results on the GTA benchmark

Method
Lego Agent
Lego Agent
Lego Agent
Lego Agent
Lego Agent
Lego Agent
HF Agent
HF Agent
HF Agent
HF Agent
HF Agent
HF Agent
T3-Agent
T3-Agent

Controller
GPT-4
GPT-4o
GPT-3.5-turbo
Claude3-opus
Qwen1.5-72B-chat
LLaMA3-70B-instruct
GPT-4o
GPT-4o mini
LLaVA-NeXT-8B
InternVL2-8B
MiniCPM-V-8.5B
Qwen2-VL-7B
Tuned MiniCPM-V-8.5B
Tuned Qwen2-VL-7B

AnsAcc
46.59
41.52
23.62
23.44
13.32
8.32
57.05
57.69
14.10
32.05
33.97
42.31
52.56
53.85

ToolAcc
-
-
-
-
-
-
63.41
56.10
14.97
36.75
36.59
44.85
65.85
64.63

CodeExec
-
-
-
-
-
-
95.12
100.00
25.08
52.18
56.10
65.19
80.49
84.32

Table 3: Results on the validation set of the GAIA benchmark

Method
Sibyl Agent
Warm-up Act
HF Agent
HF Agent
HF Agent
HF Agent
HF Agent
HF Agent
T3-Agent
T3-Agent

Controller
GPT-4-turbo
GPT-4-turbo
GPT-4o
GPT-4o mini
LLaVA-NeXT-8B
InternVL2-8B
MiniCPM-V-8.5B
Qwen2-VL-7B
Tuned MiniCPM-V-8.5B
Tuned Qwen2-VL-7B

AnsAcc
29.70
17.60
33.40
26.06
3.64
4.85
7.27
9.70
15.15
16.97

Level 1
43.40
30.20
47.17
33.96
9.43
7.55
13.21
16.98
26.42
26.42

Level 2
27.90
15.10
31.40
27.91
1.16
4.65
5.81
8.14
11.63
15.12

Level 3
7.70
0.00
11.54
3.84
0.00
0.00
0.00
0.00
3.84
3.84

perception, operation, logic, and creativity abilities on visual data. In addition to visual data, diverse
files (such as PPTX, PDF, XLSX files, etc) are also commonly encountered in practical multi-modal
tasks. To evaluate agents on such files, we use the GAIA benchmark that contains 446 tasks with 109
files. The tasks in GAIA are divided into three levels, the steps of which range from 2 to arbitrarily
long sequences, evaluating the capabilities of document understanding, web surfing, logic reasoning,
and answer summarization.

Metric. In the GTA benchmark, we measure three metrics for agents, including AnsAcc, ToolAcc,
and CodeExec. AnsAcc measures the correctness of predicted answers. ToolAcc means the accuracy
of tool selection and answer summary. CodeExec quantifies the percentage of generated codes that
could be executed without errors. In the GAIA benchmark, we measure AnsAcc of its three levels.

5.2 GTA RESULTS
The performance of agents on the GTA benchmark is shown in Tab. 2, where AnsAcc, ToolAcc, and
CodeExec are reported. Our agent achieves better results than the Lego agent that uses closed-source
models (e.g., GPT-4 and GPT-4o) and HF agent using open-source models (e.g., InternVL2-8B),
showing its effectiveness in solving complex tasks. The comparison of agents using the tuned and
untuned VLMs shows the effectiveness of our multi-modal agent tuning method. For example, tuning
MiniCPM-V-8.5B leads to about 18%, 29%, and 24% improvements on the answer accuracy, tool
correctness, and code executability, respectively. In addition, compared to the HF agent using GPT-4o
and GPT-4o mini, our agent has higher ToolAcc while lower CodeExec, showing that our tuned VLM
has more powerful reasoning capability for tool usage, while the weak programming capability results
in worse AnsAcc. This inspires us to develop VLMs for writing codes.

5.3 GAIA RESULTS
In Tab. 3, we report the performance of T3-Agent on the validation set of GAIA. T3-Agent performs
better than agents driven by open-source models. For example, Qwen2-VL-7B achieves the best
performance among all open-source models, while our agent is still 7% higher than it. The perfor-
mance improvements across multiple VLMs validate the effectiveness of our dataset. Compared with
agents driven by closed-source models (e.g., GPT-4), our T3-Agent achieves worse performance. The

8

Under review as a conference paper at ICLR 2025

Figure 4: Case study of the T3-Agent in the GTA benchmark.

reason is that closed-source models use larger model sizes and more training data. These factors may
primarily contribute to the performance differences.

5.4 DATA QUALITY
To evaluate the data quality of generated data in
MM-Traj, we conduct a user study. Concretely,
we randomly sample 600 data points from the
MM-Traj datasets and filtered out data. We ask
30 persons (with rich programming experience)
to provide scores (1-10) for the task quality and trajectory quality, and they do not know whether the
data is from MM-Traj or filtered out data. We ask the persons to provide scores for the tasks (the
queries and files) and trajectories. The score is in the range (1-10), where higher scores mean better
quality. Results are shown in Tab. 4. The quality of MM-Traj is higher than the filtered-out data,
demonstrating the used verifiers can discard lower-quality data.

Table 4: Average scores from humans.
Filtered out Data
Task Trajectory
6.36

MM-Traj
Task Trajectory
8.32

6.38

8.67

5.5 ABLATION
We conduct ablation experiments to evaluate the ef-
fectiveness of our two verifiers, as shown in Tab. 5.
We observe that on both the two benchmarks, the
data using the two verifiers leads to better per-
formance (e.g., 2.56% improvements on the GTA
benchmark), showing the effectiveness of the two verifiers.

Table 5: Ablation

Method
w/o two verifier
Ours (two verifiers)

GTA GAIA
13.33
50.00
15.15
52.56

9

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Query:Iwanttobuyadogtoyforeachdoginthephoto.HowmanydollarswillIneedtospendintotal?Thought:Ineedfirsttousetheobjectlocationtoolidentifythenumberofthedogs,andthenusetheimage_qatooltoidentifythepriceofthetoy.Code:number_of_dogs=len(objectlocalization(‘dog’,‘image1.jpg’))price_per_toy=image_qa(‘Howmuchisthetoy?’,‘image2.jpg’)print(“Numberofdogs:",number_of_dogs)print(“Pricepertoy:",price_per_toy)Observation:Numberofdogs:4Pricepertoy:$19.99Thought:Ineedtocalculatethetotalcostforpurchasingadogtoyforeachofthefourdogsinthephoto.Eachdogwillrequireonetoy,andthepriceis$19.99asshowninthefirstimage.Todothiscalculation,I'llsimplymultiplythenumberofdogsbythepricepertoy.Code:number_of_dogs=4price_per_toy=19.99total_cost=number_of_dogs*price_per_toyprint("Totalcost:",total_cost)Observation:Totalcost:79.96Query:Whichdayinthisweekhasthewidestrangeoftemperature?Andwhatistheexactdifferencebetweenthemaximumtemperatureandthelowestofthisday?Thought:Todeterminewhichdayhasthewidestrangeoftemperatureandcalculatetheexactdifferencebetweenthemaximumandminimumtemperaturesforthatday,Iwillfirstextractthetemperaturesfromtheweatherforecastimage.Then,I'llidentifythedayswiththegreatesttemperaturerange.Code:image_path="data/gta_dataset/image/image_405.jpg”temperature_data=image_qa(question="Extractthetemperaturesforeachdayinthe7-dayforecast.",image_path=forecast_image_path)print(temperature_data)Observation:**Sunday**:High60°,Low42°;**Monday**:High64°,Low52°;**Tuesday**:High70°,Low52°;**Wednesday**:High66°,Low52°;**Thursday**:High69°,Low52°;**Friday**:High70°,Low54°;**Saturday**:High74°,Low54°"Thought:NowthatIhaveextractedthetemperaturesforeachday,Iwillcalculatethetemperaturerangeforeachdayandidentifywhichdayhasthewidestrange.Code:defcalculate_temperature_range(temperatures):returnmax(temperatures)-min(temperatures)sunday_range=calculate_temperature_range([60,42])monday_range=calculate_temperature_range([64,52])tuesday_range=calculate_temperature_range([70,52])wednesday_range=calculate_temperature_range([66,52])thursday_range=calculate_temperature_range([69,52])friday_range=calculate_temperature_range([70,54])saturday_range=calculate_temperature_range([74,54])max_range=max(sunday_range,monday_range,tuesday_range,wednesday_range,thursday_range,friday_range,saturday_range)ifsunday_range==max_range:day_with_widest_range=“Sunday”ifmonday_range==max_range:day_with_widest_range="Monday”iftuesday_range==max_range:day_with_widest_range="Tuesday”ifwednesday_range==max_range:day_with_widest_range="Wednesday”ifthursday_range==max_range:day_with_widest_range="Thursday”:Iffriday_range==max_range:day_with_widest_range="Friday”ifsaturday_range==max_range:day_with_widest_range="Saturday”print("Daywiththewidesttemperaturerange:",day_with_widest_range)print("Temperaturedifferenceforthisday:",max_range)```”Observation:Daywiththewidesttemperaturerange:SaturdayTemperaturedifferenceforthisday:20"Query: If I wanted to order the two lowest priced dishes, how many dollars would it cost in total?Thought: I need to identify the two lowest priced dishes from the menu and calculate their total cost. The menu shows Beef at $35, Hamburger at $10, and Milk at $5. I will extract these prices and then sum them up.Code:menu_items=[{“Product”:“Beef”,“Price”:35},{“Product”:“Hamburger”,“Price”:10},{"Product":"Milk","Price":5}]lowest_prices=sorted(menu_items,key=lambdax:x["Price"])[:2]total_cost=sum([priceforiteminlowest_pricesforpriceinitem.values()])print("Totalcost:",total_cost)Observation:Codeexecutionfailedduetothefollowingerror:unsupportedoperandtype(s)for+:'int'and'str‘.Nowlet'sretry:takecarenottorepeatpreviouserrors!Ifyouhaveretriedseveraltimes,tryacompletelydifferentapproach."Thought:Itseemstheissueiswithmixingdifferenttypesofdatainthecalculation.I'llextractonlythepricesandthensumthemupcorrectly.Code:menu_items=[{"Product":"Beef","Price":35},{"Product":"Hamburger","Price":10},{"Product":"Milk","Price":5}]prices=[item["Price"]foriteminmenu_items]lowest_prices=sorted(prices)[:2]total_cost=sum(lowest_prices)print("Totalcost:",total_cost)Observation:Totalcost:15"(a)(b)(c)Under review as a conference paper at ICLR 2025

5.6 VISUALIZATION
In Fig. 4 and Fig. 5, we visualize cases solved by our T3-Agent in the GTA and GAIA benchmarks.
We have the following conclusions. (1) Our agent could handle multiple-image reasoning tasks. By
utilizing the visual information from given images, the agent could apply the correct tools and write
correct arguments for given images. (2) Our agent could solve complex tasks requiring long code.
(3) Our agent could revise code errors based on observations. (4) T3-Agent can solve tasks with
multi-hop questions. For example, in the first case of Fig. 5, our agent searches for information from
the web, based on obtained information in the first step. (5) T3-Agent could handle multi-modal files,
such as audio and PDF files in Fig. 5.

Figure 5: Case study of the T3-Agent in the GAIA benchmark.

6 CONCLUSION

In this paper, we have presented a multi-modal agent tuning method that improves the tool-usage
capability of agents by generating a large number of tool-usage data and tuning a VLM using these
data. Given proper prompts and sufficient source images, our data synthesis pipeline can produce
high-quantity multi-modal tasks with trajectories. We collect these generated data into an MM-Traj
dataset to tune a MiniCPM-V model, and the T3-Agent with the tuned model has achieved significant
improvements on two multi-modal benchmarks, demonstrating the effectiveness of the data synthesis
pipeline and the collected MM-Traj dataset.

Limitation. In the current T3-Agent, we only consider the multi-modal information in queries.
Practical tasks usually involve multi-modal data in trajectories of agents, such as the intermediate
results in image editing tasks. We will study how to utilize multi-modal information in agent
trajectories, which benefits in performing more powerful step-by-step reasoning for tool usage.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Query:DuringthefirstweekofAugust2015,oneoftheNASAAstronomyPicturesoftheDayshowsthelightsofacityonthehorizon.ThenamesakeofthiscityalsohasalandmarkbuildinginChicagonamedafterhim.Whatisthenameofthearchitecturalfirmthatdesignedthislandmarkbuilding?GivethefirstnameappearinginthenameofthefirmasofJune2023.Thought:TofindthenameofthearchitecturalfirmthatdesignedalandmarkbuildinginChicagonamedafterthecityonthehorizon,Ineedtogatherinformationaboutboththecityanditsnotablelandmarks.I'llusethe`web_search`tooltofindrelevantinformationregardingtheNASAAstronomyPictureoftheDayfromAugust2015andidentifythecitymentionedalongwithitslandmarkbuildinginChicago.Code:city_info=web_search("FindthecityshownintheNASAAstronomyPictureoftheDayduringthefirstweekofAugust2015.")print(city_info)Observation:Searchoutcome(shortversion):ThecityshownintheNASAAstronomyPictureoftheDayduringthefirstweekofAugust2015isMarquette,Michigan."Thought:ThecityidentifiedfromtheNASAAstronomyPictureoftheDayduringAugust2015isMarquette,Michigan.NowIneedtofindoutaboutalandmarkbuildinginChicagonamedafterthiscityandidentifythearchitecturalfirmthatdesignedit.Code:landmark_info=web_search("FindalandmarkbuildinginChicagonamedafterMarquette,Michiganandidentifythearchitecturalfirmthatdesignedit.")print(landmark_info)Observation:Searchoutcome(shortversion):TheMarquetteBuildinginChicagoisalandmarkdesignedbythearchitecturalfirmHolabird&Roche,completedin1895.Thought:IhavefoundthatthelandmarkbuildinginChicagonamedafterMarquette,Michiganisthe"MarquetteBuilding."Code:architectural_firm="Holabird&Roche”first_name=architectural_firm.split()[0]final_answer(first_name)Query:TheattachedPDFlistsaccommodationsintheresortcommunityofSeahorseIsland.WhichtypeofaccommodationhasahigheraverageratinginSeahorseIsland?Attachment:data/GAIA/2023/validation/67e8878b-5cef-4375-804e-e6291fdbe78a.pdf”Thought:TodeterminewhichtypeofaccommodationhasahigheraverageratinginSeahorseIsland,IwillfirstneedtoreadthecontentsoftheprovidedPDFfile.Afterextractingtherelevantinformationregardingaccommodationsandtheirratings,Icananalyzethemtofindoutwhichtypeisratedhigheronaverage.Code:pdf_content=inspect_file_as_text(file_path=“data/GAIA/2023/validation/67e8878b-5cef-4375-804e-e6291fdbe78a.pdf")print(pdf_content)Observation:Name:HotelsNeptune‘sPalaceAdmiralSturgeonCurrentsTheLaughingGullLoachTowersMotels……Rating(outof5),Vacancy,Pool,SampleReview:5Yes5No4Yes3No2Yes5Yes3No2Yes1Yes0Yes4No3No4Yes5No5No4Yes3No3No3Yes1Yes4Yes5Yes3No4Yes1YesYesYesYesYesNoYesYesNoNoYesNoNoNoYesNoYesYesNoNoNoYesYesNoYesNoAhotelfitforaking.......Thought:IhaveextractedthenamesofaccommodationsandtheircorrespondingratingsfromthePDF.Now,Iwillprocessthisdatatocalculatetheaverageratingforeachtypeofaccommodation(hotels,motels,rentalhouses,campgrounds).Code:accommodations={“Hotels”:[],“Motels”:[],“RentalHouses”:[],“Campgrounds”:[]}ratings=[5,4,2,3,1,5,4,3,1,4,5,4,5,4,4,3,3,3,4,5,4]forname,ratinginzip(accommodations.keys(),ratings):ifratingisnotNone:accommodations[name].append(rating)average_ratings={}forname,ratingsinaccommodations.items():average_ratings[name]=sum(ratings)/len(ratings)print("AverageRatings:",average_ratings)Observation:AverageRatings:{'Hotels':5.0,'Motels':4.0,'RentalHouses':2.0,'Campgrounds':3.0}\n"Thought:Itisclearthatthehotelshaveahigheraveragerating(5.0)comparedtomotels,rentalhouses,andcampgrounds(allwithanaverageratingof4orless).Therefore,Iwillreturnthisconclusionasmyfinalanswer.Code:final_answer("HotelshavethehighestaverageratinginSeahorseIsland.")（a）Query:OurprofessorsprungitonusattheendofclassFriday,andI‘mstilltryingtofigureitout.Thequestionheaskeduswasaboutananagram.I’veattachedanaudiorecordingofthequestionthatheasked,soifyoucouldpleasetakealistenandgivemetheanswer,I‘dreallyappreciatethehelp.Pleaselimityourresponsetotheanagramtextthatcouldbegeneratedfromtheoriginallinewhichfulfillstheprofessor’srequest,withoutanyothercommentary.Also,pleasedon‘tincludeanypunctuationinyourresponse.Attachment:data/GAIA/2023/validation/2b3ef98c-cc05-450b-a719-711aee40ac65.mp3Thought:Iwillusethe`inspect_file_as_text`tooltoanalyzetheaudiorecordingandextractthetextfromit.Thisway,Icandeterminetheanagramthatfulfillstheprofessor'srequestwithoutanycommentaryorpunctuation.Code:audio_content=inspect_file_as_text(file_path=“data/GAIA/2023/validation/2b3ef98c-cc05-450b-a719-711aee40ac65.mp3")print(audio_content)Observation:Okay,guys,beforewecallitfortheweek,I'vegotonelittlebonusassignment.ThefollowingquotationisactuallyananagramofoneoftheBard'smostwell-knownlines.I'dlikeyoualltothinkaboutit,andanyonewhocanprovidetheoriginallinewillgetanautomaticAonnextweek'squiz.Here'stheanagram.InoneoftheBard'sbestthoughtoftragedies,ourinsistenthero,Hamlet,queriesontwofrontsabouthowlifeturnsrotten.Thought:Ihavesuccessfullyextractedthetextfromtheaudiorecording.TheanagramprovidedisrelatedtoalinefromShakespeare'splay"Hamlet."Now,IwillprocessthisanagramandfindtheoriginallinethatHamletsaysintheplay.Code:Anagram:"InoneoftheBard'sbestthoughtoftragedies,ourinsistenthero,Hamlet,queriesontwofrontsabouthowlifeturnsrotten.”original_line=web_search(query="WhatdoesHamletsayin'InoneoftheBard'sbestthoughtoftragedies,ourinsistenthero,Hamlet,queriesontwofrontsabouthowlifeturnsrotten?'")print(original_line)Observation:Searchoutcome(shortversion):Theprovidedquoteisananagramandnotadirectcitationfrom'Hamlet.'Itmostlikelyalludestoexistentialthemesexemplifiedinthe'Tobe,ornottobe'soliloquy,whereHamletquestionsthenatureofexistence,purpose,andthedecayoflife.Thought:TheanagramreferstoHamlet'sfamoussoliloquy"Tobe,ornottobe."Now,Iwillcompilethisinformationintoaconciseresponsethatfulfillstheprofessor'srequest.Code:final_answer("Tobe,ornottobe")（b）（c）Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

AgentLego Contributors. AgentLego: Open-source tool API library to extend and enhance LLM

agents, December 2023. URL https://github.com/InternLM/agentlego. 7

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge,

Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 7

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image
editing instructions. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 18392–18402, 2023. 20

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding:
Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge
distillation. 2024a. 5

Lin Chen. Open-llava-next-mix1m dataset, 2024. URL https://huggingface.co/

datasets/Lin-Chen/Open-LLaVA-NeXT-mix1M. 7

Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.
Sharegpt4v: Improving large multi-modal models with better captions. In European Conference
on Computer Vision (ECCV), 2024b. 4

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi
Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial
multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024c. 7

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong
Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning
for generic visual-linguistic tasks. In The IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), 2024d. 2

Wonje Choi, Woo Kyung Kim, Minjong Yoo, and Honguk Woo. Embodied cot distillation from llm
to off-the-shelf agents. In International Conference on Machine Learning (ICML), pp. 8702–8721,
2024. 3

Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-
scale api calls. In International Conference on Machine Learning (ICML), pp. 11812–11829, 2024.
3

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha
Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024. 7

Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-
augmented multimodal agent for video understanding. In European Conference on Computer
Vision (ECCV), 2024. 2, 3

Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li.
Clova: A closed-loop visual assistant with tool usage and update. In The IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR), pp. 13258–13268, 2024. 2, 3

Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning
without training. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 14953–14962, 2023. 2, 3

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and
Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference
on Learning Representations (ICLR), 2022. 7

Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay
Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic
reasoning into vision-language models. In The IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 9590–9601, 2024. 2

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

HuggingFace Contributors. Agents and tools, 2024. URL https://huggingface.co/docs/

transformers/agents. 7

brian ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog,
Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine,
Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander T Toshev, Vincent Vanhoucke,
Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas
Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter
Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey,
Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and
Chuyuan Kelly Fu. Do as i can, not as i say: Grounding language in robotic affordances. In
Conference on Robot Learning (CoRL), pp. 287–318, 2023. 2

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete
Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Interna-
tional Conference on Computer Vision (ICCV), pp. 4015–4026, 2023. 4

Jian Li, Yabiao Wang, Changan Wang, Ying Tai, Jianjun Qian, Jian Yang, Chengjie Wang, Jilin Li,
and Feiyue Huang. Dsfd: dual shot face detector. In The IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 5060–5069, 2019. 20

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang,
and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Annual
Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3102–3116,
2023. 3

Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun
Zhu, and Qing Li. Fire: A dataset for feedback integration and refinement evaluation of multimodal
models. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 2

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European
Conference on Computer Vision (ECCV), 2014. 4

Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional

language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358, 2022. 7

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https:
//llava-vl.github.io/blog/2024-01-30-llava-next/. 7

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances

in Neural Information Processing Systems (NeurIPS), volume 36, 2024b. 2

Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang,
Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv
preprint arXiv:2311.05437, 2023a. 3

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding,
In International

Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents.
Conference on Learning Representations (ICLR), 2023b. 3

Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai,
Xinyi Liu, Hanlin Zhao, et al. Visualagentbench: Towards large multimodal models as visual
foundation agents. arXiv preprint arXiv:2408.06327, 2024c. 2, 3

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In

International Conference on Computer Vision (ICCV), 2015. 4

Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran
Yao, Zhiwei Liu, Yihao Feng, et al. Apigen: Automated pipeline for generating verifiable and
diverse function-calling datasets. arXiv preprint arXiv:2406.18518, 2024d. 3, 5

12

Under review as a conference paper at ICLR 2025

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. m&m’s: A benchmark

to evaluate tool-use for multi-step multi-modal tasks. In CVPR Workshop, 2024. 3

Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark
for question answering about charts with visual and logical reasoning. In Annual Meeting of the
Association for Computational Linguistics (ACL), pp. 2263–2279, 2022. 4

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom.

Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023. 2, 3, 7

Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey
Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple
open-vocabulary object detection. In European Conference on Computer Vision (ECCV), pp.
728–755. Springer, 2022. 20

OpenAI.

Gpt-4o system card.
gpt-4o-system-card.pdf. 2

2024.

URL https://cdn.openai.com/

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu
Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint
arXiv:2306.14824, 2023. 2

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru
Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world
apis. In International Conference on Learning Representations (ICLR), 2023. 3

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-
resolution image synthesis with latent diffusion models. In The IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 10684–10695, 2022. 20

Babak Saleh and Ahmed Elgammal. Large-scale classification of fine-art paintings: Learning the

right metric on the right feature. arXiv preprint arXiv:1505.00855, 2015. 4

Yuichi Sasazawa and Yasuhiro Sogawa. Layout generation agents with large language models. arXiv

preprint arXiv:2405.08037, 2024. 3

Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng
Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation.
arXiv preprint arXiv:2311.18760, 2023. 3

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt:
Solving ai tasks with chatgpt and its friends in hugging face. In Advances in Neural Information
Processing Systems (NeurIPS), 2024. 2, 3

Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus
Rohrbach. Towards vqa models that can read. In The IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR), pp. 8317–8326, 2019. 4

Xiaowen Sun, Xufeng Zhao, Jae Hee Lee, Wenhao Lu, Matthias Kerzel, and Stefan Wermter.
Details make a difference: Object state-sensitive neurorobotic task planning. arXiv preprint
arXiv:2406.09988, 2024. 2, 3

Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for
reasoning. In International Conference on Computer Vision (ICCV), pp. 11888–11898, 2023. 2, 3

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal-
paca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint
arXiv:2306.05301, 2023. 3

Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank
Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of
apps and people for benchmarking interactive coding agents. In Annual Meeting of the Association
for Computational Linguistics (ACL), pp. 16022–16076, 2024. 2

13

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

Chenyu Wang, Weixin Luo, Qianyu Chen, Haonan Mai, Jindi Guo, Sixun Dong, XM Xuan, Zhengxin
Li, Lin Ma, and Shenghua Gao. Mllm-tool: A multimodal large language model for tool agent
learning. arXiv preprint arXiv:2401.10727, 4, 2024a. 2, 3

Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta:
A benchmark for general tool agents. In Advances in Neural Information Processing Systems
(NeurIPS), 2024b. 2, 3, 7

Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to
believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574,
2023a. 4

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu,
Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the
world at any resolution. arXiv preprint arXiv:2409.12191, 2024c. 2, 7

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and
Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In
Annual Meeting of the Association for Computational Linguistics (ACL), 2023b. 5

Yulong Wang, Tianhao Shen, Lifeng Liu, and Jian Xie. Sibyl: Simple yet effective agent framework
for complex real-world reasoning. 2024d. URL https://arxiv.org/abs/2407.10718.
7

Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal llm as an agent for

unified image generation and editing. arXiv preprint arXiv:2407.05600, 2024e. 3

T. Weyand, A. Araujo, B. Cao, and J. Sim. Google Landmarks Dataset v2 - A Large-Scale Benchmark
for Instance-Level Recognition and Retrieval. In The IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR), 2020. 4

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi-
sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint
arXiv:2303.04671, 2023. 2

Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen
Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based
agents across diverse environments. arXiv preprint arXiv:2406.04151, 2024. 3

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing
Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal
agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972,
2024. 3

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu,
Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning
and action. arXiv preprint arXiv:2303.11381, 2023. 3

Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doraemongpt: Toward
understanding dynamic scenes with large language models (exemplified as a video agent). In
International Conference on Machine Learning (ICML), pp. 55976–55997, 2024. 3

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models. In International Conference on
Learning Representations (ICLR), 2023. 2, 5

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li,
Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint
arXiv:2408.01800, 2024. 2, 7

Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and
Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In
International Conference on Machine Learning (ICML), pp. 12380–12403, 2024. 3

14

Under review as a conference paper at ICLR 2025

Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R Fung, Hao Peng, and Heng Ji. Craft: Customizing
llms by creating and retrieving from specialized toolsets. In International Conference on Learning
Representations (ICLR), 2024. 2, 3

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning:

Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823, 2023. 3

Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang,
Liangwei Yang, Yihao Feng, Zuxin Liu, et al. Agentohana: Design unified data and training
pipeline for effective agent learning. arXiv preprint arXiv:2402.15506, 2024a. 3

Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal

internet agents. arXiv preprint arXiv:2404.09992, 2024b. 3

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web
agent, if grounded. In International Conference on Machine Learning (ICML), pp. 61349–61385,
2024a. 3

Sipeng Zheng, Yicheng Feng, Zongqing Lu, et al. Steve-eye: Equipping llm-based embodied agents
with visual perception in open worlds. In International Conference on Learning Representations
(ICLR), 2024b. 3

Yaoyao Zhong, Mengshi Qi, Rui Wang, Yuhan Qiu, Yang Zhang, and Huadong Ma. Viotgpt:
Learning to schedule vision tools towards intelligent video internet of things. arXiv preprint
arXiv:2312.00401, 2023. 2

15

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Under review as a conference paper at ICLR 2025

A HUMAN VERIFICATION OF MM-TRAJ

A.1 DATA SYSTHESIS PIPELINE

We recruited 30 persons with rich programming and AI experience to evaluate the tasks and trajectories
generated by our method. Each evaluator is tasked with assessing 20 samples, which are randomly
selected and mixed from both MM-Traj and filtered-out data.

The evaluation was conducted using a 5-level rating scale: Poor, Fair, Good, Very Good, and
Excellent, corresponding to numerical scores of 2, 4, 6, 8, and 10, respectively, with a maximum
score of 10. The results in Tab. 6 show that verified cases consistently outperform filtered-out cases in
both task and trajectory evaluations. Verified tasks scored an average of 7.96, while filtered-out tasks
averaged 6.30, indicating that verified tasks are more natural, coherent, and complex. For trajectories,
verified cases scored 8.64 versus 6.24 for filtered-out cases, demonstrating better reasoning, code
coherence, and feedback effectiveness. These results confirm that the verification process effectively
filters out lower-quality data, ensuring the reliability of our data synthesis pipeline and the MM-Traj
dataset.

MM-Traj
Task Trajectory
8.32

8.67

Filtered out Data
Task Trajectory
6.36

6.38

Table 6: Average score of the human verification.

The evaluation criteria for the generated tasks and trajectories are as follows.

Task Evaluation Criteria

• Naturalness: The degree to which the task appears natural and realistic.

• Coherence: The logical consistency and smooth flow of the task.

• Complexity: The extent to which the task exhibits sufficient complexity, requiring the use

of multiple tools for effective resolution.

Trajectory Evaluation Criteria

• Reasoning: The logical soundness and clarity of the agent’s thought process.

• Code Coherence: The clarity, consistency, and structure of the code produced by the agent.

• Feedback Effectiveness: The agent’s ability to effectively respond to and incorporate results

from the tool executions.

The interface of the user study for data quality is shown in Fig. 6.

A.2 AGENT OUTPUT

We conducted a user study on agent outputs on the GTA benchmark. We recruited 20 participants,
each evaluating 20 tasks with agent outputs, where the agent is with or without fine-tuning. The
agent outputs (w/ or w/o tuning) were shuffled for each task, and the participants were not informed
about the source, ensuring an unbiased assessment. The participants were asked to provide the
preference of the two agent outputs, based on the accuracy, helpfulness, and relevance. We measured
the percentages of results, as shown in Tab. 7. Outputs from the tuned agent have a significantly high
preference, indicating its better performance in solving practical tasks. The interface of the user study
for agent output is shown in Fig. 7.

Agent w/o tuning is better
21%

Tie
13%

Agent w tuning is better
66%

Table 7: User study for agent outputs on the GTA benchmark.

16

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Under review as a conference paper at ICLR 2025

Figure 6: The interface of the user study for data quality.

17

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Figure 7: The interface of the user study for agent outputs on the GTA benchmark.

18

Under review as a conference paper at ICLR 2025

B MORE EXPERIMENTS

B.1 ABLATION

To improve the interpretability of our method, we add ablation experiments to show the contributions
of different modalities in decision-making. Ablation results on the GTA dataset are shown in Tab. 8,
where removing the image modality reduces the performance by 40%, highlighting the importance of
input images.

Method
T3 Agent w/o image
T3 Agent w/ image

AnsAcc ToolAcc CodeExec
25.32
10.67
65.85
52.56

20.09
80.49

Table 8: Average score of the human verification.

B.2 DATA NUMBER

We show the agent’s performance on the GTA benchmark as the dataset size increases, in Tab. 9.
With the increase of data number, the agent achieves better performance, the memory consumption is
constant, and the time consumption linear increases. Compared with the accuracy improvements, we
think that the consumption of memory and time is acceptable.

Data Number
Accuracy
Memory
Training Time

6K
43.59%
214 GB
276 mins

12K
48.08%
214 GB
532 mins

20K
52.56%
214 GB
946 mins

Table 9: Performance on the GTA benchmarks.

C TOOLS

We will show details of used tools in T3-Agent.

C.1 WEB SEARCH

The web search tool is actually another agent. It has three sub-tools: Searchinformation, Visit, and
Webqa.

searchinformation. Given a query to this tool, it performs a Google search and outputs the title,
abstract, and URL of multiple entries.

Visit. The input is the URL of an HTML page, and the output is the textual content of the HTML
page.

Webqa. Given a question and search textual content, the tool outputs the answer.

C.2

IMAGE QUESTION ANSWERING

We use the GPT-4o-mini model as the image question answering tool. The input is an image and a
question, and the output is the answer.

C.3 FILE INSPECTOR

The input is a question and one multi-modal file. We use the Python package ‘MarkdownConverter’
that converts the given files into the markdown texts. Then, we feed the question and texts to the
GPT-4o-mini model for the answer.

19

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Under review as a conference paper at ICLR 2025

C.4 OBJECT LOCALIZATION

We use the OWL-ViT model (Minderer et al., 2022) for object localization. The input includes one
image and a query, and the output is a Python list of bounding boxes of the query.

C.5

IMAGE GENERATION

We use the stable diffusion model for image generation (Rombach et al., 2022). Given a textual query,
the tool produces an image that matches the query.

C.6

IMAGE EDITING

We use the InstructPix2Pix model for image editing (Brooks et al., 2023). The inputs are an instruction
and an image, and the output is an edited image to match the instruction.

C.7 FACE DETECTION

We use the DSFD model for face detection (Li et al., 2019). The input is an image, and the output is
a Python list of all bounding boxes of faces.

C.8 PYTHON PACKAGE

We allow the agent to use the following Python package in code writing: “requests", “zipfile", “os",
“pandas", “numpy", “sympy", “json", “bs4", “pubchempy", “xml", “yahoo_finance", “Bio", “sklearn",
“scipy", “pydub", “io", “PIL", “chess", “PyPDF2", “pptx", “torch", “datetime", “csv", “fractions",
“matplotlib", “pickle", “cv2", through which the agent is more flexible in writing code.

D PROMPT

D.1 PROMPT FOR QUERY GENERATION

The prompt for query generation is shown in Fig. 8.

You are tasked with generating user queries that will prompt an agent to call various tools (only use the
tool listed in our toolset), including internet search capabilities, to solve real-world, practical problems.
The problems should be natural, varied, and challenging, requiring the agent to reason across different
domains. Ensure that the problems span a range of practical scenarios.

Our toolset: TOOL_SET
I will now provide examples, along with the tools.
Examples of user queries: IN_CONTEXT_EXAMPLES

Please output the Queries in a json format. Make sure that the queries share a similar style of the
in-context examples. The output template is :
{
"query": "What is the weather today?", <The user query to the agent.>
"tools": ["tool1", "tool2",...], <A list consisting of the tool names related to the query.>
},
...

Figure 8: Prompt for query generation.

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

Under review as a conference paper at ICLR 2025

D.2 PROMPT FOR FILE GENERATION

The prompt for file content generation is shown in Fig. 9 and Fig. 10 , and the prompt for file code
generation is shown in Fig. 11 and Fig. 12.

You are a smart reasoner that can restore a query_solving scene between a human and an agent.
Human gives a complex query and several images to the agent, and then the agent answers the query
by searching on the Internet and applying tools to the images with step-by-step reasoning. Now, you
will be given the query with suggested tools, I suggest you analyze the needed information to solve the
query, and divide the information into two groups: searching from the Internet and extracting from the
images using tools. Based on the information from the images, you need to further infer the content of
these images, through which the agent could correctly solve the query.

Our toolset: TOOL_SET
Output MUST use the following json template.

{

"information": <Needed information to solve the query. For the query including creating/generating

images, the information should NOT be the description of the describe image.>

"information from the Internet": <Information from the Internet inferences based on the given
query and suggested tools. Determine which information is suitable to be obtained from the Internet.
Or say no information is required from the Internet.>

"information from images": <Information extracted from given images based on the suggested
tools to solve the query. It should be several sentences, including information extracted from the
images using tools. Determine which information is suitable to be obtained from the images, and using
which tools. Do not generate image_content for the query including generating/creating an image. Or
say no information is required from the images.>

"file": {

"image_numbers": <set an int number, the number is depended on needed information from
images>,
"image_content":
{

"image_1": <The image content should be a natural language, describe the content of the
first image relevant to the query. The content should be concrete, such as concrete
number, concrete name. The content should match the query and the above images.>
... <if you think the query needs more than 1 image, please output image content like
’image_2’.>

}

}

}

Figure 9: System prompt for the file content generation.

Now given the query: QUERY, firstly analyze the needed information to solve the query and divide the
information into two groups: searching from the Internet or extracting from images using tools. Then
for information from images, imagine possible answers of each information (it should be concrete
answers instead of descriptions). Finally, output the json for the inferenced information and the content
of images.

Figure 10: User prompt for the file content generation.

D.3 PROMPT FOR QUERY-FILE FILTER

The prompt for the query-file filter is shown in Fig. 13 and Fig. 14.

21

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092
1093

1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115
1116

1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

You are a helpful assistant and can to generate a <file type placeholder> file by writing Python code.
You will be given a description of the content of the file. You need to firstly largely extend the content,
and then write Python code to generate a <file type placeholder> file. GUARANTEE that the provided
content is in the file.
The output Python code MUST use the following template.
"""

## extention start

Extened content: <here is the extented content>

## extention end

## code start

<here is the Python code to generate a <file type placeholder> file>

## code end

"""

Figure 11: User prompt for the file content generation.

Now, given the following content: <file content>, first largely extend the content, and output a code to
generate a <file type placeholder> file, where the file name is <file name> and the file will be saved in
<save path>.

Figure 12: User prompt for the file content generation.

D.4 PROMPT FOR TRAJECTORY FILTER

The prompt for the trajectory filter is shown in Fig. 15 and Fig. 16.

D.5 PROMPT FOR AGENTS

The system prompt for the T3-Agent is shown in Fig. 17.

E MORE VISUALIZATION

We provide more visualization of our T3-Agent on the GTA and GAIA benchmarks, as shown
in Figs. 18 to 24.

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

22

Under review as a conference paper at ICLR 2025

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

You are a helpful assistant that is given a query and several images. You need to check whether the
images are relevant to the query. The query and images are used to evaluate the perception ability,
reasoning ability, and information search ability of an AI agent. The agent solves the query by
searching information from the Web and extracting information from the images. In some cases, based
on the given images, the agent could not solve the query, even it searches for information from the
Web (e.g., some specific knowledge). You need to pick up these bad cases.

The agent can call the following tools to solve the query. TOOL_SET .

Thus, the images should follow these requirements.
1. Relevance: The depicted scenarios or objects in images should be relevant to the query. The images
should contain scenarios or objects that are mentioned in the images.
2. Usefulness: The image should contain necessary information to address the query, such as some
specific details that cannot be obtained from the Web.
3. Some queries require the agent to search for knowledge from the Web and combine the information
in the image to solve the queries. Thus, in some cases, the images do not contain all the information to
solve the query, but the missed information could be searched on the Web. These cases should be
regarded as correct cases.

The output MUST use the following json template to check the images.
{

"information_for_query": <Required information to solve the query.>,

"useful_information_in_image": <Useful information that can be extracted from images to solve

the query>,

"missed_information_in_images": <Missed information that is necessary to solve the query but

does not exist in the images.>,

"missed_information_web_search": <You need to justify whether the missed information could be

searched from the Web, using your rich experience in surfing the Internet.> ,

"missed_information_obtained": <You need to justify whether the missed information could be
obtained via computing or reasoning based on information extracted from the images or searched from
the Web.>,

"thought": <Now, you need to determine whether the images can solve the query. If the missed
information could be searched from the Web or obtained based on existing information, the images
can solve the query. If not, the images cannot solve the query.>,

"correct": <According to the above reasoning, if you consider the images reasonable for the query

to be solved by the tools, set the value to ’yes’, otherwise set the value to ’no’.>,

"updated_query": <If you judge the correctness as ’no’, please rewrite the query to make it more
relevant to the given images. If you judge the correctness as ’yes’, please output "no revision is
needed." >
} ”’

Figure 13: System prompt for the query-file verification.

Following are images, the query: <query>, inference whether the images can solve the query based on
the perception ability, reasoning ability, and information search ability of an AI agent.

Figure 14: User prompt for the query-file verification.

23

Under review as a conference paper at ICLR 2025

As a data quality evaluator that need to determine whether a query-solving trajectory between a human
and an agent is correct. The human gives images and a query, and the agent calls tools to solve the
query. The trajectory of query-solving contains a task query, thoughts and codes generated by the agent
to call tools (Python functions), and tool-response of each step, and the final answer. You must assess
the alignment between the task query, corresponding tool usage (generated thoughts and codes from
the agent), and the execution results (tool-response). Your goal is to ensure the used tools, arguments
to the tools, and summarized answers in the trajectory accurately reflect the human’s intentions.
Our toolset: TOOL_SET
The query-solving trajectory is incorrect if: 1. The tool usage does not align with the query’s objective
and the context, or there is useless or unreasonable tool usage. In addition, the agent does not use tools
and solve the query by itself. 2. The input arguments to the tools appear incorrect or unreasonable.
3. The final answers or intermediate results summarized from the observation appear incorrect or
unreasonable. 4. The final answer is not relevant to the task query or the final answer seems incorrect.
5. The trajectory (such as tool-usage and observation) conflicts or is not consistent with the image
content.

Figure 15: System prompt for the trajectory verification.

Now, given used images and corresponding information, determine whether the trajectory is correct or
not.

1. User Query: QUERY
2. Image Content: IMAGE_CONTENT
3. Trajectory, including generated thought and code from the agent, and intermediate results of using
tools: TRAJ
4. Execution Results: RESULT
Output MUST use the following json template to determine whether the query-solving trajectory is
correct or not.
{
"thought": "Concisely describe your reasoning here",
"correct": "yes" or "no"
}

Figure 16: User prompt for the trajectory verification.

24

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Under review as a conference paper at ICLR 2025

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

You are an expert assistant who can solve any task using code blobs. You will be given a task to solve
as best you can. To do so, you have been given access to a list of tools: these tools are basically Python
functions which you can call with code. To solve the task, you must plan forward to proceed in a
series of steps, in a cycle of ’Thought:’, ’Code:’, and ’Observation:’ sequences.
At each step, in the ’Thought:’ sequence, you should first explain your reasoning towards solving the
task and the tools that you want to use. Then in the ’Code:’ sequence, you should write the code
in simple Python. The code sequence must end with the ’<end_action>’ sequence. During each
intermediate step, you can use ’print()’ to save whatever important information you will then need.
DO NOT generate a code which does not call ’print()’ because you will lose this information. You can
assume all tools must have a return that can be printed. These print outputs will then appear in the
’Observation:’ field, which will be available as input for the next step. You will save all intermediate
file outputs to a folder by the relative path ’.cache’. In the end, you have to return a final answer using
the ‘final_answer‘ tool.
Here are a few examples using notional tools: IN_CONTEXT_EXAMPLES

The above example was using notional tools that might not exist for you. You only have access to
those tools: TOOL_SET
You also can perform computations in the Python code that you generate.

Here are the rules you should always follow to solve your task:
1. Always provide a ‘Thought:’ sequence, and a ‘Code: py’ sequence ending with ‘<end_action>’
sequence, else you will fail.
2. Use only variables that you have defined!
3. Always use the right arguments for the tools. DO NOT pass the arguments as a dict as in ‘answer =
ask_search_agent(‘query’: “What is the place where James Bond lives?")’, but use the arguments
directly as in ‘answer = ask_search_agent(query="What is the place where James Bond lives?")’.
4. Take care to not chain too many sequential tool calls in the same code block, especially when the
output format is unpredictable. For instance, a call to search has an unpredictable return format, so do
not have another tool call that depends on its output in the same block: rather output results with
print() to use them in the next block.
5. Call a tool only when needed, and never re-do a tool call that you previously did with the exact
same parameters.
6. Don’t name any new variable with the same name as a tool: for instance don’t name a variable
‘final_answer’.
7. Never create any notional variables in our code, as having these in your logs might derail you from
the true variables.
8. You can use imports in your code, but only from the following list of modules: «authorized_imports»
9. The state persists between code executions: so if in one step you’ve created variables or imported
modules, these will all persist.
10. Don’t give up! You’re in charge of solving the task, not providing directions to solve it.

Now Begin! If you solve the task correctly, you will receive a reward of $1,000,000.

Figure 17: System prompt for the agent.

25

Under review as a conference paper at ICLR 2025

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Figure 18: Case study of the T3-Agent.

26

Query:WhoistheCEOofthetop-rankedcompany?Thought:TofindtheCEOofthetop-rankedcompany,Ineedtoanalyzetheprovidedimage.ThetaskinvolvesidentifyingwhichcompanyisrankedfirstandthendeterminingitsCEO.Iwillusethe`image_qa`tooltoextractanytextualinformationfromtheimage.Code:image_path="data/gta_dataset/image/image_399.jpg"company_rank=image_qa(question="Whatisthenameofthetop-rankedcompany?",image_path=image_path)print("Top-rankedcompany:",company_rank)Observation:Top-rankedcompany:Thetop-rankedcompanyisApple.Thought:NowthatIknowthetop-rankedcompanyisApple,IneedtofindoutwhoitsCEOis.Sincethisinformationmaynotbedirectlyvisibleintheimage,Iwillusethe`ask_search_agent`tooltosearchforthecurrentCEOofApple.Code:ceo_of_apple=ask_search_agent(query="WhoistheCEOofApple?")print("CEOofApple:",CEO_of_Apple)Observation:Searchoutcome(shortversion):TimCookisthecurrentCEOofAppleInc.,havingassumedtheroleinAugust2011.Underhisleadership,Applehasseenasignificantincreaseinrevenueandprofit,andastrategicshifttowardsamoreinclusivecompanyculture.Thought:IhavegatheredinformationaboutTimCookbeingthecurrentCEOofApple.Now,Iwillsummarizethisinformationtoprovideaconcisefinalanswer.Code:final_answer("TimCookisthecurrentCEOofApple.")Under review as a conference paper at ICLR 2025

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Figure 19: Case study of the T3-Agent.

27

Query:IfIhave10lightsinmyhomeintotal,howmanylightsarethereintheotherrooms?Thought:Iwillusethe`image_qa`tooltoanalyzetheimageandextractinformationaboutthelightspresent.Thisshouldprovideaclearcountofhowmanylightsarevisibleinthepicture.Code:lights_count=image_qa(question="Howmanylightsarevisibleinthisimage?",image_path="data/gta_dataset/image/image_392.jpg")print(lights_count)Observation:Therearetwolightsvisibleintheimage.Thought:Ihavedeterminedthattherearetwolightsvisibleintheimage.Now,tofindouthowmanylightsareintheotherrooms,Iwillsubtractthiscountfromthetotalnumberoflights(10)sinceit'sgiventhatthereare10lightsinthehome.Code:total_lights=10lights_in_other_rooms=total_lights–2final_answer(lights_in_other_rooms)Under review as a conference paper at ICLR 2025

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Figure 20: Case study of the T3-Agent.

28

Query:Whoisthefounderofthiscompany?Thought:Tofindoutwhothefounderofthecompanydepictedintheimage,Iwillusethe`image_qa`tooltoanalyzetheimageandextractrelevantinformationaboutthecompany.Thisshouldhelpidentifykeydetailssuchasitsnameandanyassociatedindividuals.Code:company_info=image_qa(question="Whatisthenameofthecompanyshowninthisimage?",image_path="image_306.jpg\")print(company_info)Observation:ThecompanyshownintheimageisNVIDIA.Thought:NowthatIknowthecompanyisNVIDIA,Iwillusethe`web_search`tooltofindoutwhofoundedit.Thisshouldprovidemewiththenameofthefounder.Code:founder=web_search(query="WhoisthefounderofNVIDIA?")print(founder)Observation:Searchoutcome(shortversion):NVIDIAwasfoundedonApril5,1993,byJensenHuang,ChrisMalachowsky,andCurtisPriem.Thought:IhavegatheredinformationaboutthefoundersofNVIDIA.ThecompanywasfoundedbyJensenHuang,ChrisMalachowsky,andCurtisPriem.Now,Iwillcompilethisinformationintoafinalanswer.Code:founders=[“JensenHuang”,"ChrisMalachowsky","CurtisPriem"]final_answer(f"FoundersofNVIDIA:{','.join(founders)}")Under review as a conference paper at ICLR 2025

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Figure 21: Case study of the T3-Agent.

29

Query:Ihaveonly$70tobuyapantsandshoes.Idon'twearflats.Eachitemshouldbedifferentinstyleandcolor.WhatspecificitemscanIbuy?Pleaseanswerwithashortnounphrase.Thought:Ineedtoidentifyapairofpantsandshoesthataredifferentinstyleandcolorwithinmybudget.SinceIhave$70,I‘lllookforoptionsunderthegivenpricerange.Iwillusethe`image_qa`tooltoanalyzetheimageprovidedandextractrelevantinformationaboutclothingitems,focusingonstylesandcolors.Code:cloth_items=image_qa(question="Pleaseidentifyspecificpantsandshoesfromtheimagebasedontheirstylesandcolors.",image_path="image_287.jpg")print(cloth_items)Observation:**Pants:**-Jeans ($30) -Chinos ($35)-Formal Trousers ($40) **Shoes:** Sneakers ($40)-Ballet Flats ($30)-Ankle Boots ($60)Thought: I see that the table lists specific clothing items and their prices. However, it doesn't provide visual information about styles or colors of pants and shoes directly from the image provided. Since I don't have a clear visual reference for the specific items, I will summarize the options available based on the listed prices:-**Pants:** Jeans ($30), Chinos ($35), Formal Trousers ($40)-**Shoes:** Sneakers ($40), Ballet Flats ($30), Ankle Boots ($60)Given my budget of $70, I need to choose two different items—one pair of pants and one pair of shoes—without buying flats. I'll propose a combination where I buy one type of jeans and sneakers, ensuring they are distinct in style and color.Code:item1="Jeans"price1=30item2="Sneakers”price2=40total_cost=price1+price2print("Totalcost:",total_cost)Observation:Totalcost:70Thought:Ihaveconfirmedthatthetotalcostofbuyingjeansandsneakersis$70,whichmatchesmybudget.Now,Iwillfinalizethispurchase.\n\nI'llreturnthespecificitemswithashortnounphrasetoindicatewhatIcanbuywithinthegivenconstraintsCode:final_answer("JeansandSneakers")Under review as a conference paper at ICLR 2025

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

Figure 22: Case study of the T3-Agent.

30

Query:Takethegendersplitfromthe2011Bulgariancensusaboutthosewhohavecompletedtertiaryeducation.Subtractthesmallernumberfromthelargernumber,thenreturnthedifferenceinthousandsofwomen.Soiftherewere30.1thousandmoremen,you‘dgive30.1.Thought:Iwillfirstneedtogatherthedataregardingthegendersplitfromthe2011Bulgariancensusforthosewhohavecompletedtertiaryeducation.OnceIhavethesenumbers,Icansubtractthesmallernumberfromthelargernumberandreturntheresultinthousandsofwomen.\n\nI'llstartbyusingthe`web_search`tooltofindthespecificdatarelatedtothegendersplitfromthe2011censusonthosewithtertiaryeducation.Code:gender_data=web_search("gendersplitfromthe2011Bulgariancensusaboutthosewhohavecompletedtertiaryeducation")print(gender_data)Observation:Search outcome: In the 2011 Bulgarian Census, 22.3% of women and 16.7% of men had completed tertiary education.Thought: I have retrieved the relevant data from the 2011 Bulgarian census. According to the information, there were 791.8 thousand females (representing 22.3% of women) and 556.9 thousand males (representing 16.7% of men) who had completed tertiary education.Now, I will calculate the difference between these two numbers in thousands of women by subtracting the smaller number from the larger number:-Larger number (females): 791.8\n-Smaller number (males): 556.9. The difference is calculated as follows:[ text{Difference} = 791.8 -556.9 ] I'll perform this calculation using Python code.Code:larger_number=791.8smaller_number=556.9difference=larger_number-smaller_numberprint("Differenceinthousandsofwomen:",difference)Observation:Differenceinthousandsofwomen:234.9Thought:Thedifferencebetweenthenumbersofwomenandmenwhocompletedtertiaryeducationinthe2011Bulgariancensusisapproximately234.9thousand.Now,Iwillreturnthisresultasthefinalanswerusingthe`final_answer`tool.Code:final_answer("Thedifferenceinthousandsofwomenfromthosewithtertiaryeducationinthe2011Bulgariancensusisapproximately234.9.)Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Figure 23: Case study of the T3-Agent.

31

Query:Thelongest-livedvertebrateisnamedafteranisland.AccordingtoWikipediaasofJanuary1,2021,whatisthe2020estimatedpopulationofthatisland,tothenearestthousand?Thought:Tosolvethistask,Ineedtofindthenameoftheislandafterwhichthelongest-livedvertebrateisnamed.Then,IwillsearchfortheestimatedpopulationofthatislandasofJanuary1,2020.SinceWikipediawasreferencedinthequestion,I'llusethe`web_search`tooltogatherthisinformation.Code:longest_lived_island=web_search(query="Thenameoftheislandafterwhichthelongest-livedvertebrateisnamed")print("Islandname:",longest_lived_island)population_query=web_search(query=f"Theestimatedpopulationof{longest_lived_island}asofJanuary1,2020.")print("Estimatedpopulation(tonearestthousand):",population_query)Observation:Search outcome: The longest-lived vertebrate is the Greenland shark, named after the island of Greenland. Thought: From the search results, I have found that the longest-lived vertebrate is the Greenland shark. The estimated population of Greenland as of January 1, 2020, was not explicitly mentioned in the provided information. However, since it's related to a specific island, I can assume this refers to the population of Greenland itself. As there is no direct data on the population of Greenland from 2020, I will need to use general knowledge or perform another search for an estimate. I will ask the `web_search` again for an approximate population of Greenland in 2020 and round it to the nearest thousand.Code:population_greenland=web_searc(query="WhatistheestimatedpopulationofGreenlandasofJanuary1,2020?")print("EstimatedpopulationofGreenland(tonearestthousand):",population_greenland)Observation:Searchoutcome:Greenland'sestimatedpopulationasofJanuary1,2020,was56,081.Thought:TheestimatedpopulationofGreenlandasofJanuary1,2020,isreportedtobeapproximately56,081.Iwillnowformatthisinformationandusethe`final_answer`tooltoprovideaconciseanswerregardingboththelongest-livedvertebrate(Greenlandshark)anditsassociatedisland'spopulation.Code:population_info="Thelongest-livedvertebrateistheGreenlandshark,namedaftertheislandofGreenland,withanestimatedpopulationofapproximately56,000in2020.”final_answer(population_info)Under review as a conference paper at ICLR 2025

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Figure 24: Case study of the T3-Agent.

32

Query:TheMetropolitanMuseumofArthasaportraitinitscollectionwithanaccessionnumberof29.100.5.Oftheconsecratorsandco-consecratorsofthisportrait'ssubjectasabishop,whatisthenameoftheonewhoneverbecamepope?Thought:Tofindtheconsecratorsandco-consecratorsoftheportrait'ssubjectasabishop,specificallylookingforthosewhoneverbecamepope,IwillneedtosearchforinformationaboutthisparticularportraitintheMetropolitanMuseumofArtcollection.Thismayinvolvecheckingonlinedatabasesormuseumarchivesrelatedtothisspecificartwork.\n\nI'llstartbyaskingthe`web_search`tooltoprovidedetailsontheportraitwithaccessionnumber29.100.5attheMetropolitanMuseumofArt.Code:portrait_info=web_search("Detailsabouttheportraitwithaccessionnumber29.100.5attheMetropolitanMuseumofArt")print(portrait_info)Observation:Search outcome:-Title: Cardinal Fernando Niño de Guevara n-Artist: El Greco (DomenikosTheotokopoulos) -Date: ca. 1600 -Medium: Oil on canvas -Location: Metropolitan Museum of Art, Accession Number: 29.100.5Thought: The search results indicate that the portrait is of Cardinal Fernando Niño de Guevara, painted by El Greco around 1600. I now need to find out who were the consecrators and co-consecrators of this cardinal subject as a bishop, specifically identifying those who never became pope. I will ask the `web_search` tool again for details on the consecrators and co-consecrators of Cardinal Fernando Niño de Guevara, focusing particularly on those who did not become pope.Code:consecrators_info=web_search("Consecratorsandco-consecratorsofCardinalFernandoNiñodeGuevarafromtheMetropolitanMuseumofArtcollection,specificallythosewhoneverbecamepope.")print(consecrators_info)Observation:Searchoutcome(shortversion):-CardinalFernandoNiñodeGuevarawasconsecratedbyPopeClementVIII,withco-consecratorsCamilloBorghese(laterPopePaulV)andAlfonsoVisconti.-AlfonsoViscontiwasnotapope.-NodirectconnectiontotheMetropolitanMuseumofArtwasfoundforAlfonsoVisconti.Thought:Basedonthesearchresults,IhaveidentifiedthatCardinalFernandoNiñodeGuevarawasconsecratedbyPopeClementVIIIwithco-consecratorsCamilloBorgheseandAlfonsoVisconti.Amongtheseindividuals,AlfonsoViscontiisnotedasneverhavingbecomeapope.Iwillnowcompilethisinformationintoafinalanswerformattopresentitclearly.Code:final_answer("Theco-consecratorofCardinalFernandoNiñodeGuevarawhoneverbecamepopeisAlfonsoVisconti.")