SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION
Under review as a conference paper at ICLR 2025

SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR
SMARTPHONE AGENT EVALUATION

Anonymous authors
Paper under double-blind review

ABSTRACT

Smartphone agents are increasingly important for helping users control devices
efficiently, with (Multimodal) Large Language Model (MLLM)-based approaches
emerging as key contenders. Fairly comparing these agents is essential but chal-
lenging, requiring a varied task scope, the integration of agents with different im-
plementations, and a generalisable evaluation pipeline to assess their strengths and
weaknesses. In this paper, we present SPA-BENCH, a comprehensive SmartPhone
Agent Benchmark designed to evaluate (M)LLM-based agents in an interactive
environment that simulates real-world conditions. SPA-BENCH offers three key
contributions: (1) A diverse set of tasks covering system and third-party apps in
both English and Chinese, focusing on features commonly used in daily routines;
(2) A plug-and-play framework enabling real-time agent interaction with Android
devices, integrating over ten agents with the flexibility to add more; (3) A novel
evaluation pipeline that automatically assesses agent performance across multiple
dimensions, encompassing seven metrics related to task completion and resource
consumption. Our extensive experiments across tasks and agents reveal challenges
like interpreting mobile user interfaces, action grounding, memory retention, and
execution costs. We propose future research directions to ease these difficulties,
moving closer to real-world smartphone agent applications.

1

INTRODUCTION

The growing capabilities of Large Language Models (LLMs) and Multimodal Large Language Models
(MLLMs) have broadened the application of AI agents across various domains Gur et al. (2023);
Gou et al. (2023); Cai et al. (2023); Li et al. (2023a); Wang et al. (2023); Wu et al. (2023a). One
promising area is smartphone control, where agents assist users in tasks like booking hotels or setting
alarms. These agents can be broadly categorised into two main types: (1) agent-as-a-model Lai
et al. (2024), where fine-tuned or pre-trained (M)LLMs are customised for agentic tasks Zhan &
Zhang (2023); Hong et al. (2024); Bai et al. (2024); Lu et al. (2024), and (2) agentic workflow Shang
et al. (2024), which typically relies on off-the-shelf models and modular designs to support agentic
functionality Yang et al. (2023b); Wen et al. (2024); Wang et al. (2024b;a); Rawles et al. (2024a). In
both cases, these models act as the “brains” for decision-making. The information these agents use to
interact with smartphones can vary, with common methods involving direct screen observation Wang
et al. (2024b;a); Zhan & Zhang (2023); Hong et al. (2024); Bai et al. (2024); Lu et al. (2024),
accessing non-visible data via Android View Hierarchy or Extensible Markup Language (XML) Wen
et al. (2024), or a combination of both Yang et al. (2023b); Rawles et al. (2024a).

As the number of (M)LLM-based smartphone agents grows, fair performance comparisons become
crucial for identifying their strengths and shortcomings, leading to an increasing need for bench-
marking. Chan et al. (2023); Liu et al. (2023a;b); Wu et al. (2023b). Regarding smartphone agent
benchmarks, existing studies use three main approaches to evaluate agents: actions-based Xing
et al. (2024), states-based Rawles et al. (2024a); Zhang et al. (2024); Lee et al. (2024); Wang et al.
(2024c), or a hybrid of both Wang et al. (2024c). Each method faces specific difficulties: action-based
evaluation may involve multiple correct sequences, while state-based methods struggle to determine
the appropriate post-action state. A hybrid approach could mitigate these limitations, but the challenge
lies in effectively utilising both action and state information.

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: An overview of SPA-BENCH. The worker machine iterates through the task and agent pools,
assigning tasks to agents within the framework for execution, and then passes the execution results to
the evaluation pipeline for measuring task completion and resource consumption performance.

Despite these efforts, current research Rawles et al. (2024a); Xing et al. (2024); Zhang et al. (2024);
Lee et al. (2024); Wang et al. (2024c) still has several key limitations: (1) The focus remains primarily
on system and Google suite applications (apps) in English, which are often free from distractions like
ads and pop-ups that could introduce complexity and randomness; (2) The number of evaluated agents
is typically fewer than five, with some studies including only similar variants of the same agent;
(3) Automated success detection methods frequently require human intervention (e.g., handcrafted
validation logic for each task) or rely on data that may be inaccessible in certain cases (e.g., Android
View Hierarchy data, which is unavailable in WebView apps Xing et al. (2024)).

In this paper, we introduce SPA-BENCH, a SmartPhone Agent Benchmark designed to evaluate more
than 10 smartphone control agents in daily tasks. As illustrated in Figure 1, SPA-BENCH comprises
340 tasks, including 150 single-app tasks and 20 cross-app tasks, in both English and Chinese apps,
as well as third-party ones. It integrates 11 agents into a unified framework based on their original
implementations. This framework is linked to an automated evaluation pipeline that measures agent
performance and can be applied to additional tasks beyond this benchmark, without requiring human
inputs. Our experiments show that agents following the agentic workflow outperform those in the
agent-as-a-model category, although the former remain impractical for real-world deployment due
to time and cost constraints. We also provide a detailed discussion on the challenges and future
directions for smartphone agents, covering topics such as building perceptive mobile interfaces,
reasoning mechanisms, and user-friendly applications.

In summary, our comprehensive benchmark makes several key contributions: (1) a diverse task
collection of 340 tasks with increasing difficulty, accompanied by human trajectory annotations. It
covers both English and Chinese apps, including single-app and cross-app scenarios, and featuring
58 third-party apps (Section 3); (2) a plug-and-play agent framework supporting 11 agents, which
allows for easy integration of new agents with minimal adaptation and offers features like automatic
Android setup and multi-device emulator support (Section 4); (3) an automated and scalable
evaluation pipeline assesses agent performance using task completion and resource consumption
metrics. It employs success detection methods that achieve average F1 scores of 90.5% for single-app
tasks and 84.5% for cross-app tasks compared to human evaluators (Section 5); and (4) extensive
experiments across agents and tasks, providing a detailed analysis of current smartphone agent
capabilities and limitations, while also offering directions for future research (Section 6).

2 RELATED WORK

Smartphone Agent. Smartphone agents aim to automate tasks on mobile apps in a human-like
way. Early agents, like Siri and Google Assistant, relied on system-level APIs and customisation,
limiting their generality. Recently, (M)LLM-based agents have emerged, using the user interface
(UI) to achieve a more general approach. These agents, with (M)LLMs as their “brains”, also require
“hands” (actions) and “eyes” (observations) to interact with smartphones. They are based on either
off-the-shelf or fine-tuned models and perform human-like actions (e.g., tapping, typing, and swiping).

2

Evaluation Pipeline✅② Agent +Task Description③ Screenshots +Execution InformationWorker Machine🖥 Agent Framework🤖④ Task Description +Screenshots +Execution Information ⑤ Task PerformanceMulti-workerProcessTask Pool📋Agent PoolOverallPerformanceOff-the-Shelf(M)LLM-basedAgentsFine-tuned(M)LLM-basedAgents① Select an Agent and a Task🧠(M)LLM-based Agents📱Android Devices🎯Success Signal🔚Termination Reason💰API Cost⏱  Time Spent🔁Step Ratio         Single-App TasksLanguagesDifficulty Levels       Cross-App TasksLanguagesDifficulty Levels⚠ Premature Termination Signal⏳Overdue Termination SignalUnder review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Table 1: Comparison of SPA-BENCH and other smartphone agent benchmarks. Agents that differ
only in their base models are not counted as separate agents.

Dataset

AndroidArena Xing et al. (2024)
AndroidWorld Rawles et al. (2024a)
LlamaTouch Zhang et al. (2024)
B-MoCA Lee et al. (2024)
MobileAgentBench Wang et al. (2024c)

SPA-BENCH

Third-party
app?
✗
✓
✓
✗
✗

Cross-
app?
✓
✓
✗
✗
✗

Chinese
app?
✗
✗
✗
✗
✗

Difficulty
level?
✗
✓
✓
✗
✓

✓

✓

✓

✓

Number
of tasks

Number
of agents

Number
of metrics

221
116
495
60
100

340

1
3
4
3
5

11

4
1
1
1
6

7

Free of hand-
crafted validation?
✗
✗
✗
✗
✗

✓

Information for
success detection

Action only
State only
State only
State only
Action and State

Action and State

According to how they observe the UI, recent works are categorised into text-based, vision-based,
and combined approaches. Text-based methods Wen et al. (2024); Rawles et al. (2024a) rely on
UI document data (e.g., XML) or convert visual information into text, vision-based methods Wang
et al. (2024b;a); Zhan & Zhang (2023); Hong et al. (2024); Bai et al. (2024); Lu et al. (2024) use
screenshots to capture the complete visual context, while combined approaches Yang et al. (2023b);
Rawles et al. (2024a) integrate both text and vision inputs for greater informativeness. SPA-BENCH
evaluates all three types of agents to provide a comprehensive comparison of their capabilities.

Smartphone Agent Evaluation. Effective evaluation of smartphone agents is crucial for identifying
limitations and guiding improvements. Success rate, which measures task completion, is the most
commonly used metric, with some studies also considering efficiency. Success detection methods
are generally classified into two types: human detection Yang et al. (2023b); Wang et al. (2024b;a),
which is accurate but resource-intensive, and automated detection, which is less costly but varies in
accuracy. Current automated methods primarily rely on hand-crafted validation logic, making them
unscalable without human intervention. They are restricted to evaluating tasks involving apps that are
limited to English-only and simpler apps (e.g., system, Google Suite, and open-source apps), with
minimal coverage of other third-party ones. These automated methods can be further divided into
action-based, state-based, and hybrid approaches. Action-based methods Xing et al. (2024) compare
agents’ actions to human demonstrations but struggle with the non-unique nature of correct action
sequences. State-based methods Rawles et al. (2024a); Zhang et al. (2024); Lee et al. (2024) assess
whether essential states are reached but may miss minor actions. Hybrid approaches Wang et al.
(2024c) combine state and action data for more accurate success detection. SPA-BENCH introduces
two hybrid approaches for evaluating single-app and cross-app tasks. Compared to other automated
methods, our approaches support a wider range of apps and tasks. They do not rely on hand-crafted
validation logic, making them adaptable without human intervention. Table 1 presents a comparison
between our work and other automated evaluation-based smartphone agent benchmarks, highlighting
our comprehensive evaluation of various agents in diverse tasks across multiple dimensions.

3 SPA-BENCH TASK

3.1 OVERVIEW

SPA-BENCH builds a collection of smartphone agent tasks across both English and Chinese apps,
featuring 39 English and 29 Chinese apps divided into eight categories based on core features (see
Appendix B.1). The collection includes 150 single-app tasks and 20 cross-app tasks for each language.
These tasks focus on core app functions that reflect everyday use, providing a realistic assessment of
smartphone agents’ performance. The inclusion of diverse Chinese and third-party apps increases
complexity, primarily due to the difficulties agents encounter in understanding Chinese and navigating
more intricate UIs. A complete list of tasks is provided in Appendix B.2.

The single-app tasks are grouped into sets, with progressively increasing complexity across three
difficulty levels. In each set, Level 1 tasks serve as foundational and straightforward activities,
while Level 2 and Level 3 tasks introduce more complex requirements, such as handling intricate UI
elements or animations. Generally, Level 1 tasks require fewer than five actions, while Level 2 tasks
typically involve fewer than ten, and Level 3 tasks fewer than fifteen. While each set follows similar
instructions, the tasks are designed to use distinct entities (e.g., creating folders with different names)
to prevent any influence from earlier tasks. Examples of single-app tasks are shown in Figure 2.

3

Under review as a conference paper at ICLR 2025

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

For cross-app tasks, we re-
fer to the recent work GUI
Odyssey Lu et al. (2024), which
defines six task types: General
Tool, Information Management,
Web Shopping, Media Enter-
tainment, Social Sharing, and
Multi-Apps. Unlike single-app
tasks, cross-app difficulty lev-
els are determined by the num-
ber of apps involved in a task.
Level 1 tasks require interac-
tions between two apps, while
Level 2 tasks necessitate switch-
ing between three. As the num-
ber of apps increases, complex-
ity arises not only from addi-
tional steps but also from inter-
app dependencies and coordina-
tion. Our cross-app tasks in-
clude three Level 1 tasks for
each of the first five types, and
five Level 2 tasks for the Multi-
Apps type. Appendix B.4 pro-
vides examples.

3.2 TASK CONSTRUCTION

Figure 2: A sample set of tasks within the Deliveroo app, annotated
by human. In this example, simpler tasks form the foundation for
more complex ones, resulting in shared trajectories in the initial
stages. The final screenshots for tasks of all three difficulty levels
are highlighted in corresponding colours. Each final screenshot
highlights the key components used in coarse detection (explained
further in Section 5), with the zoomed-in versions available in
Appendix B.3.

Our tasks were primarily constructed by human annotators. For single-app tasks, we selected
commonly used apps and supplemented them with apps from related works Yang et al. (2023b);
Wang et al. (2024b). Based on each app’s core features, tasks were created following an annotation
guideline specifying: (1) A clear task description that reflects the task’s goal and difficulty level. For
descriptions inspired by prior works, we standardised them and assigned appropriate difficulty levels.
(2) A human-executed trajectory presented as a series of screenshots. Between any two adjacent
screenshots, only one action (e.g., tap, type) is allowed. The total number of actions in the human
execution serves as the “golden steps” in our experiments. To ensure a reproducible and unbiased
baseline, we instruct human annotators to avoid irrelevant actions and refrain from using shortcuts
that are inherently dynamic, influenced by factors such as recommendation algorithms or user-specific
history. (3) Key components of the final state, which are pieces of text that must appear in the final
screenshot if the task is successfully completed. We focus only on the final state because there may be
multiple correct paths to complete the task, but they typically converge to the same final state Wang
et al. (2024c). These key components are designed for future use, as detailed in Section 5.2. For
cross-app tasks, annotations include only task descriptions and human-executed trajectories due to
the flexibility of final states. Most cross-app English tasks were drawn from GUI Odyssey Lu et al.
(2024), and we reformatted descriptions and recollected trajectories where necessary.

To ensure task quality, a validation process followed task annotation. Annotators cross-checked
all tasks for clarity, trajectory accuracy, and key component quality. The tasks were also tested
across different Android devices, Android versions, and app versions to verify feasibility. The same
validation was repeated before experiments.

In total, SPA-BENCH includes 300 single-app and 40 cross-app tasks, evenly split between English
and Chinese. Each task may consist of multiple subtasks (e.g., adding, modifying, deleting, searching).
The distribution of steps performed by humans for these tasks, categorised by task type, is illustrated
in Appendix B.5.

4 AGENT FRAMEWORK

4.1 A UNIFIED PLUG-AND-PLAY FRAMEWORK

4

👆👆⌨ 👆⌨ 👆👆👆👆👆Enter a McDonald's restaurant.Search for fries there. Add a small fries to the basket. Addtwo medium fries to the basket. Viewthe basket for confirmation.Get the search results for McDonald's.TaskLevel 1Task difficultyLevel 2Level 3HardKey componentsLevel 2 Task DoneLevel 3 Task DoneLevel 1 Task DoneUnder review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

Our framework facilitates the
execution of autonomous smart-
phone agents and tasks. As
shown in Figure 3, the worker
machine manages communica-
tion, providing task descriptions
and receiving outcomes (trajec-
tories and logs). It hosts multi-
ple worker processes, each con-
necting an Android emulator
and an agent. Each agent in-
teracts with the Android device
by performing actions based
on observations, such as taking
screenshots and generating ac-
tions like taps or swipes. The
snapshot state is restored at the
start of each experimental cycle. The framework is highly scalable. Unlike existing research Rawles
et al. (2024a); Xing et al. (2024); Zhang et al. (2024); Lee et al. (2024); Wang et al. (2024c), which
integrates a limited number of agents tightly into the framework, ours allows easy addition of new
agents with minimal integration, ensuring each agent operates independently within an isolated
environment. Details about the agents integrated into our framework are provided in Appendix C.

Figure 3: An overview of the agent framework using a multi-
processing architecture. Each worker process connects an agent to
an Android emulator, and they interact multiple times throughout
the task (i.e., step 3 is repeated) until completion. The emulators
are reset after the agent has executed all assigned tasks.

4.2 SNAPSHOT-BASED EMULATOR FOR CONSISTENT TESTING

The framework integrates Android emulators as a scalable alternative to physical devices, replicating
most Android functions for parallel testing and rapid experiment deployment. For instance, a 24-
core CPU with 64GB RAM can support up to eight emulators or worker processes simultaneously,
depending on the agents’ resource needs.

To ensure consistency, emulators can be quickly loaded from snapshots, which capture and restore
system states (e.g., installed apps, login credentials, and local settings). This eliminates repetitive
setup processes by preserving pre-configured settings (e.g., a pre-existing contact for messaging
tasks). However, since some app data is stored externally, manual intervention is required after each
experiment cycle, such as unsubscribing from channels post-task completion.

5 AUTOMATED EVALUATION PIPELINE

5.1 METRICS

We define seven key metrics for comprehensive evaluation:

Completion-related Metrics. (1) Success signal – a binary indicator of task success. For single-app
and cross-app tasks, we develop two different hybrid approaches that leverage both action and state
information, allowing for multiple valid execution paths. These approaches eliminate the need for
human evaluators and handcrafted evaluation logic (details are provided in Section 5.2). (2) Step
ratio – measures execution efficiency by comparing agent steps with human steps (the “golden steps”
from Section 3.2). This is considered only when the task is successful (i.e., success signal is “true”).
A higher ratio indicates more unnecessary actions and lower efficiency. (3) Termination reason –
explains why the task was terminated, including self-reported completion (i.e., an agent proactively
terminates a task based on its belief that the task has been completed successfully), reaching the
configured maximum step limit, or execution errors (e.g., invalid actions).(4) Premature termination
signal – a binary indicator applicable only when the termination reason is self-reported completion.
It is set to “true” when the success signal is “false”, indicating the agent mistakenly believed the
task was completed. This premature stopping reduces success rates by causing the agent to assume
success before finishing the task. (5) Overdue termination signal – a binary indicator applicable
only when the termination reason is reaching the maximum step limit. It is set to “true” when the
success signal is “true”, meaning the agent mistakenly thought the task was incomplete. This results
in unnecessary steps, reducing efficiency as the agent takes extra actions before concluding the task.

5

⚙ ⚙ 💻Worker MachineAutoReset🤖AgentMulti-modal InputDecision + ReasonWorker Process⚙ Worker Process⚙ Worker Process🗂 LocalData🧠Model③ Observation③ Translated ActionManual Reset④ Screenshot② Assign Agent④ Action+  Log⑤ Execution Trajectory~8 ProcessesPer Machine......① Task + Configuration☁ ServerData💾AndroidSnapshot📱EmulatorUnder review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

Figure 4: An example of our single-app success detection pipeline. It features coarse detection
through key component matching on execution screenshots and pre-annotated key components,
followed by fine detection using MLLM evaluation given action information.

Consumption-related Metrics. (6) Time spent – the time taken for task execution, recorded in
seconds. (7) API cost – the monetary cost incurred by API usage, measured in US dollars. However,
these two metrics apply only to agents using proprietary MLLMs, as for locally hosted fine-tuned
models, the time taken heavily depends on computational resources, and there are no monetary costs
from external API calls.

5.2 SUCCESS DETECTION

Single-App Success Detection. We employ a coarse-to-fine success detection pipeline that uses
key component matching followed by MLLM evaluation. As shown in Figure 4, for each agent-task
pair, the pipeline first applies coarse detection using the annotated key components introduced in
Section 3.2, filtering out trajectories irrelevant to the task. If passed, fine detection follows, using
an MLLM evaluator for final success determination. This approach is motivated by the need to
balance scalability and cost efficiency, addressing the limitations of relying on extensive human labour
or expensive MLLM-based evaluations in large-scale benchmarks. We compared our single-app
success detection approach with human evaluations and found it achieves an F1 score of 0.926 for
English tasks and 0.884 for Chinese tasks. Further details on the single-app success detection and its
performance can be found in Appendix D.

Cross-App Success Detection. Unlike single-app success detection which processes the entire task
at once, our cross-app approach splits tasks into subtasks and evaluates them sequentially. This
is because cross-app tasks are usually longer than single-app tasks and require switching between
multiple apps, increasing the complexity of success detection. As illustrated in Figure 5, a MLLM
first generates subtasks based on the involved apps, followed by a human review. During evaluation,
another MLLM splits the trajectory into multiple segments based solely on each app in the ordered
list. If the segmentation is valid, each subtask is then evaluated sequentially until either the final
subtask is checked or an earlier subtask fails. Our cross-app success detection method closely aligns
with human evaluations, achieving an F1 score of 0.845. More details on the cross-app success
detection and its performance are provided in Appendix E.

6 EXPERIMENTS

In this paper, the success rate results were derived using the automated success detection methods
outlined in Section 5.2, with GPT-4o serving as the MLLM. To account for agents with multiple
variants, detailed configurations for each agent are provided in Appendix F.1. Furthermore, case
studies illustrating various agent behaviors are presented in Appendix H.

6.1 OVERVIEW OF SUCCESS RATE

Table 2 shows the overall success rates. Notably, M3A consistently achieved the highest success rates
across all task sets. We found that agents generally performed English tasks better than Chinese tasks,

6

Task: Go to notification settings. Turn on Notification History.TrueTask FailedTask SuccessFalseCoarse DetectionSingle screenshot(from last to first)OCRPreprocessingTaskExecutionConcatenatedextracted textTrueFalseKey Components: notification  historyFine DetectionTask descriptionExecuted screenshotswith action annotationExecuted ActionMLLM EvaluationExecution ScreenshotsKey ComponentsMatchingUnder review as a conference paper at ICLR 2025

Figure 5: An example of our cross-app success detection pipeline that is based on subtasks instead of
the entire task. The first stage involves splitting the full trajectory into segments, while the second
stage checks the subtasks sequentially.

agents with the agentic workflow outperformed those categorised as agent-as-a-model, and cross-app
tasks were more challenging than single-app tasks for agents.

Agent

Overall

Overall

English

English

Chinese

Chinese

Cross-App

Single-App

Agentic Workflow (GPT-4o)

Table 2: Success rates across all tasks and agents in this benchmark,
categorised by task type. The first seven agents fall under the
category of agentic workflow, while the last four belong to agent-
as-a-model. AutoDroid was tested only on single-app tasks as its
agent framework, Droidbot Li et al. (2017), supports only these
tasks.

Comparison in Single-App
Tasks.
For single-app En-
glish tasks, M3A, T3A, and
MobileAgentV2 were the best-
performing ones, with suc-
cess rates ranging from 0.640
to 0.433. These agents are
equipped with reflection mod-
ules that help prevent them from
stalling. AppAgent and Au-
toDroid performed less well,
though they would likely had
performed better with access to
external knowledge documents,
as in their original implementa-
tions. For single-app Chinese
tasks, MobileAgentV2 outper-
formed T3A, while its perfor-
mance was more comparable to
M3A. A potential factor could
be the overly complex accessi-
bility (a11y) tree layout used by
T3A. MobileAgentV2, relying
on OCR and raw screenshots, averaged 12,400 prompt tokens per step in Chinese single-app tasks,
compared to T3A’s 22,000 tokens using only a11y trees, indicating larger or more intricate structures
in Chinese apps, potentially contributing to the agent’s degraded performance. A similar trend
was observed in English single-app tasks, with lower token usage across both agents: 11,200 for
MobileAgentV2 and 19,700 for T3A. In general, a decrease in success rates for Chinese tasks was
observed due to the limited capabilities of (M)LLMs in Chinese, compounded by the increased
complexity of Chinese apps. These apps often feature more intricate layouts, frequent animations,
and distracting elements such as ads and pop-ups.

AppAgent
AutoDroid
MobileAgent
MobileAgentV2
M3A
T3A
SeeAct

Auto-UI
CogAgent
DigiRL
OdysseyAgent

0.340
0.327
0.387
0.433
0.640
0.487
0.393

0
-
0.100
0.100
0.100
0.100
0.050

0
-
0.050
0.100
0.200
0.100
0.100

0.294
0.257
0.314
0.437
0.544
0.434
0.360

0.247
0.187
0.240
0.440
0.447
0.380
0.327

0
-
0.075
0.100
0.150
0.100
0.075

0.013
0.027
0.020
0.053

0.010
0.027
0.010
0.037

0.007
0.027
0
0.020

Agent-as-a-Model

0
0
0
0

0
0
0
0

0
0
0
0

Impact of Core Models and Input Modalities. There was a significant gap in success rates between
agents using proprietary models like GPT-4o and those based on fine-tuned models. Agents following
the agentic workflow significantly outperformed those in the agent-as-a-model category, the latter of
which often struggled to complete any tasks. This contrasts with the high action matching scores
reported in prior studies Zhan & Zhang (2023); Hong et al. (2024); Bai et al. (2024); Lu et al. (2024),
indicating that fine-tuned agents are often optimised for generating textual actions based on fixed UI
scenarios. While such optimisations may achieve high accuracy in offline environments, they often

7

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Subtask 2Success DetectScreenshots 2Actions 2Subtask 2Screenshotsand ActionsUse the LinkedIn app to search for a customer servicerepresentative position. Select a job, open Keep Notes,create a new note, record the company's name, andset the note's title to 'customer service representative'.LinkedIn, Keep NotesSearch for a customer servicerepresentative position and select a job.Create a new note, record {company's name}, and setthe note's title to 'customer service representative'.LinkedIncompany's nameSubtask 1Subtask 2Keep Notescompany's nameMLLM splitExecution Screenshotsand ActionsLinkedInKeep NotesMLLM splitStage 1Subtask GenerationTaskFailedTaskSuccessHuman reviewedSubtask 1Success DetectStage 2TrueTrueTaskAPPMemoryHistoryValidInvalidTrueFalseLinkedIn, Keep NotesSubtask 1App 1MemorySubtask 2App 2HistoryFalseSubtask 1Screenshotsand ActionsScreenshots 1Actions 1Under review as a conference paper at ICLR 2025

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

Table 3: Task performance on single-app English tasks. SRC and MSR refer to Self-Reported
Completion and Maximum Steps Reached, respectively. The execution time and token costs of the
last four agents are omitted because they use locally hosted open-source models.

Agent

Success
Rate

Mean Step Ratio
on Success

Termination Reason

Termination Inaccuracy

SRC
Rate

MSR
Rate

Error
Rate

Premature
Rate

Overdue
Rate

Mean Exec Time
per Step (sec)

Mean Token Cost
per Step (USD)

AppAgent
AutoDroid
MobileAgent
MobileAgentV2
M3A
T3A
SeeAct

Auto-UI
CogAgent
DigiRL
OdysseyAgent

0.340
0.327
0.387
0.433
0.640
0.487
0.393

0.013
0.020
0.020
0.053

1.33
1.10
1.24
1.05
0.92
1.04
1.60

1.50
1.67
1.52
2.00

Agentic Workflow (GPT-4o)

0.327
0.593
0.367
0.580
0.847
0.707
0.200

0.060
0.147
0.227
0

0.507
0.340
0.633
0.420
0.153
0.293
0.773

0.166
0.067
0
0
0
0
0.027

0.347
0.494
0.109
0.333
0.244
0.368
0.100

Agent-as-a-Model

0.940
0.820
0.607
1.000

0
0.033
0.166
0

1.000
1.000
0.971
-

0.197
0.078
0.095
0.111
0
0.136
0.276

0.015
0.024
0.022
0.013

26.5
34.0
27.1
56.1
19.3
9.6
41.2

-
-
-
-

0.014
0.008
0.053
0.067
0.092
0.116
0.046

-
-
-
-

fail in dynamic, real-world settings. For example, a tap action is deemed successful if its coordinates
fall within 14% of the screen distance to the ground truth Rawles et al. (2024b), but this tolerance
can cause inaccuracies with actionable elements in practice. Furthermore, reliance on predefined
scenarios limits the agents’ ability to generalise to unseen UI contexts or to recover from detoured
states caused by mistaken actions. On the other hand, agents utilising agentic workflow are typically
equipped with input from visual modules, such as mark-up documents and set-of-marks Yang et al.
(2023a). These layout documents are sometimes incomplete, failing to capture all available UI
elements on the interface. In other cases, they are unnecessarily complex for models to handle, as
seen in the case of T3A mentioned above. This highlights a critical gap in grounding capabilities,
which are essential for end-to-end task completion but remain challenging especially for fine-tuned
models Zheng et al. (2024).

Complexity and Memory Retention in Cross-App Task. For cross-app tasks, most agents, except
M3A, completed no more than 4 tasks in total across both English and Chinese apps. Although
M3A performed better, completing 6 out of 40 tasks, overall performance was still low, reflecting
the complexity of cross-app tasks. These tasks require more steps, reasoning, and the ability to
switch between apps while retaining memory of previous actions. In some cases, agents might nearly
complete the task but fail in the end due to minor mistakes or missed requirements, especially in long
sequences or multi-context scenarios. Even OdysseyAgent Lu et al. (2024), specifically designed
for cross-app tasks, faced difficulty completing them end-to-end. It sometimes handled subtasks
within a single app well but faltered when transitioning between apps, illustrating the challenge of
maintaining context and reasoning across environments. These findings suggest that current agents,
including the best-performing ones, struggle with multi-step cross-app tasks, often losing context or
forgetting prior actions. This highlights the need for better memory mechanisms, enhanced inter-app
reasoning, and advanced handling of complex, multi-context environments Shinn et al. (2023); Li
et al. (2023b); Pan et al. (2024). These capabilities are essential for tasks users expect autonomous
agents to manage.

6.2 COMPLETION- AND CONSUMPTION-RELATED METRICS

When comparing completion- and consumption-related metrics across agents, we observed consistent
trends across single-app and cross-app tasks in both English and Chinese. Since the single-app
English results are the most comprehensive, this section focuses primarily on those results, with
additional details available in Appendix F.2. Table 3 shows full task performance for single-app
English scenarios.

Step Efficiency and Success Rate. As discussed in Section 6.1, agents with the agentic workflow
substantially outperformed those belong to agent-as-a-model. Higher success rates correlate with
lower mean step ratios, indicating more efficient task completion with fewer unnecessary actions or
errors. Conversely, agents facing difficult tasks tend to make more errors, even if they ultimately
succeed. M3A exhibited a notably low mean step ratio of 0.92, indicating it used fewer steps than

8

Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

a human. This efficiency is partly achieved through combined actions specifically defined by the
agent itself, where a single action encompasses multiple operations, such as typing in the search box
and pressing “enter” in one step. Agents may also exploit strategic shortcuts, such as clicking on a
recommended item instead of using the search bar. Thus, both approaches allow agents to reduce the
steps needed to complete a task.

Task Termination and Success Rate. Regarding task termination, a higher success rate generally
aligns with a higher Self-Reported Completion (SRC) rate and a lower Maximum Steps Reached
(MSR) rate. Agents terminated tasks either when they believed the task was complete or when they
reached the step limit or encounter errors. However, agents did not always accurately determine
task completion, leading to discrepancies between success rates and SRC rates. This can be further
analysed by examining the premature termination rate (PTR) and overdue termination rate (OTR). As
mentioned in Section 5.1, PTR can affect the success rate, while OTR can influence task efficiency.
Notably, a pattern emerges where agents with a lower PTR tend to have a higher OTR. This compro-
mise likely arises from the agent’s internal decision thresholds. For instance, SeeAct exhibited the
lowest PTR (0.100) but the highest OTR (0.276). This demonstrates a trade-off in the sensitivity of
the agent’s internal success detector, balancing the risk of premature termination with the tendency to
extend task completion unnecessarily. An ideal success detector should minimise both premature and
overdue terminations to optimise both task accuracy and efficiency.

Enhancing Robustness through Error Handling Mechanisms. Error-handling mechanisms are cru-
cial for improving success rates and ensuring reliable performance. Agents lacking these mechanisms
were more prone to failure, which led to early termination when execution errors occurred. Common
issues include parsing errors arising from the agents’ inability to correctly interpret model outputs as
valid actions. For example, the output may be missing specific phrases, such as “Thought: ”, that are
required by the agent’s parsing module. Some agents also encountered failures when necessary inputs
(e.g., XML files) could not be accessed. These failures highlight the need for better error detection
and recovery strategies, allowing agents to correct mistakes and improve their overall success rates.

Limitations in Cost and Efficiency for Real-World Use. While agents categorised as agent-as-a-
model do not incur token costs and their execution time varies with device power, their low success
rates make them impractical for deployment. Among agents with the agentic workflow, AutoDroid is
the most cost-effective, using only $0.008 per step due to its text-based input. However, it has a long
execution time (34 seconds per step) and a success rate of only 0.327. M3A and T3A, though faster
(under 20 seconds per step) and more successful, have higher token costs at around $0.10 per step
due to the complexity of inputs generated by UI elements. MobileAgentV2, while more affordable at
$0.067 per step, suffers from a complex visual perception pipeline, resulting in the longest execution
time (56.1 seconds per step). These results highlight the trade-off between efficiency and effectiveness.
Agents like T3A, despite achieving relatively high success rates and faster execution times, still fall
short of human-level usability due to their monetary cost. Such limitations stem from two major
factors. One is the delay between UI information collection and action execution, which can cause
inaccuracies especially when dynamic content appears. The other is the agents’ slower speeds and
higher costs compared to human users. Users are unlikely to rely on an autonomous agent to complete
a task if they have to wait for extended periods or pay several dollars, especially when they could
complete it in a few steps themselves.

Performance Variation Across Difficulty Levels, Open-Ended Task Experiments, and Case
Studies. We compared agent performance across difficulty levels, showing that easier tasks are
executed more successfully, as demonstrated in Appendix F.3. To further explore the scalability of
our success detection approaches, we conducted initial experiments on “open-ended” single-app
English tasks, detailed in Appendix G. In Appendix H, we present three case studies that illustrate
representative scenarios of agent task execution.

6.3 KEY INSIGHTS

To enhance the performance of autonomous smartphone agents, future research may need to address
several core dimensions, including UI understanding and action grounding, dataset diversity, memory
retention, reflection and error-handling mechanisms, internal task termination recognition, and
execution efficiency.

9

Under review as a conference paper at ICLR 2025

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

First, integrating more advanced visual perception modules is essential for enhancing agents’ under-
standing of complex UI layouts and precise action grounding across various scenarios. Although
agents using a11y trees and OCR have shown relatively good performance in English tasks, their
effectiveness is still limited in Chinese tasks, which often feature more visually complex and dynamic
content. Currently, some agents struggle to ground actions in these dynamic environments, often
failing to recognise actionable elements or map generated actions to the correct coordinates. Future
designs should focus on building more robust visual models that can accurately interpret these
environments and perform end-to-end task completion in interactive settings.

Diversifying fine-tuning datasets is also essential for making agents more generalisable. Datasets
should include various task instruction formats, languages, and both single-app and cross-app
scenarios to better simulate real-world conditions. This would ensure that agents are prepared to
handle a broader range of interactions, particularly in multilingual environments where language and
UI complexity vary.

Memory retention mechanisms can be improved as well, especially for handling long, multi-step
tasks that span multiple apps. Current agents often lose context during complex tasks or app
transitions, which leads to incomplete task execution. Memory-augmented networks or episodic
memory architectures could enable agents to retain context across transitions, which is particularly
valuable in cross-app scenarios where agents usually struggle. These scenarios closely resemble
real-world tasks that require continuity and context recall over extended sequences.

Reflection and error-handling capabilities are another critical area for improvement. Many agents fail
to learn from mistakes, repeatedly making the same errors without self-correction. Implementing
robust reflection modules, similar to those found in M3A, would allow agents to better assess
their past actions and adjust their strategies dynamically. Additionally, error-handling mechanisms,
such as error identification, recovery loops, self-correction, and fallback strategies, are vital for
maintaining performance in unpredictable, dynamic environments. Agents need to be able to detect
and resolve issues such as invalid model outputs, unactionable UI elements, or parsing errors, rather
than terminating prematurely or getting stuck in unproductive actions.

In task termination, agents must carefully balance premature and overdue termination. Some agents
still struggle to accurately determine when a task is truly complete. For example, while SeeAct
showed a low premature termination rate, it also exhibited a high overdue termination rate. This
indicates that although SeeAct avoided ending tasks prematurely, it often failed to recognise when
tasks were completed, leading to inefficiencies. A well-designed internal success detector can
minimise both types of termination inaccuracies, thereby improving task accuracy and efficiency.

Finally, execution time and cost need to be optimised for real-world deployment. Agents such as
MobileAgentV2, which rely on multiple modules, need to reduce overhead and streamline execution to
minimise task completion time. MLLM-based agents, in contrast to T3A, may also focus on reducing
input context size to lower token costs while preserving critical information for task completion.
A hybrid model approach that combines the speed and efficiency of lightweight models with the
robustness of more complex ones could provide a promising solution for balancing performance and

7 CONCLUSION

In this paper, we introduced SPA-BENCH, a comprehensive benchmark for evaluating smartphone
agents across diverse tasks. The evaluation covers English and Chinese apps, single-app and cross-app
scenarios, and varying difficulty levels. Our experiments reveal that even the best-performing agents
can complete less than 70% of tasks successfully, and there are significant performance gaps between
agents following the agentic workflow and those in the agent-as-a-model category, particularly in
action grounding and generalisation within complex Chinese apps. While some agents excel in
simpler tasks, their long execution times and high costs limit their practicality for real-world use.
Our findings highlight the need for better memory mechanisms, robust error handling, accurate
self-evaluator, improved integration of reasoning with UI understanding, and optimising execution
time and cost for real-world deployment. Additionally, agents based on fine-tuned models should be
adapted to diverse scenarios and focus on long-sequence decision-making rather than isolated actions.
By developing SPA-BENCH as a fair and scalable benchmark, we aim to accelerate the development
of more efficient, practical, and user-friendly smartphone agents.

10

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl:
Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint
arXiv:2406.11896, 2024.

Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as

tool makers. arXiv preprint arXiv:2305.17126, 2023.

Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and
Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv
preprint arXiv:2308.07201, 2023.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and
Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv
preprint arXiv:2309.17452, 2023.

Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and
Aleksandra Faust. A real-world webagent with planning, long context understanding, and program
synthesis. arXiv preprint arXiv:2307.12856, 2023.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan
Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
14281–14290, 2024.

Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen
Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large language
model-based web navigating agent. arXiv preprint arXiv:2404.03648, 2024.

Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking mobile
device control agents across diverse configurations. arXiv preprint arXiv:2404.16660, 2024.

Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com-
municative agents for" mind" exploration of large language model society. Advances in Neural
Information Processing Systems, 36:51991–52008, 2023a.

Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. A zero-shot language agent for computer

control with structured reflection. arXiv preprint arXiv:2310.08740, 2023b.

Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. Droidbot: a lightweight ui-guided test input
generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering
Companion (ICSE-C), pp. 23–26. IEEE, 2017.

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding,
Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. arXiv preprint
arXiv:2308.03688, 2023a.

Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng,
Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. Bolaa: Benchmarking and orchestrating
llm-augmented autonomous agents. arXiv preprint arXiv:2308.05960, 2023b.

Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang,
Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui
navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024.

Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous
evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.

Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth
Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic
benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024a.

11

Under review as a conference paper at ICLR 2025

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An-
droidinthewild: A large-scale dataset for android device control. Advances in Neural Information
Processing Systems, 36, 2024b.

Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic

llm agent search in modular design space. arXiv preprint arXiv:2410.06153, 2024.

Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic

memory and self-reflection. arXiv preprint arXiv:2303.11366, 2(5):9, 2023.

Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and
Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv
preprint arXiv:2305.16291, 2023.

Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang,
and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via
multi-agent collaboration. arXiv preprint arXiv:2406.01014, 2024a.

Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao
Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv
preprint arXiv:2401.16158, 2024b.

Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen,
and Shoufa Chen. Mobileagentbench: An efficient and user-friendly benchmark for mobile llm
agents. arXiv preprint arXiv:2406.08184, 2024c.

Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao
Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In
Proceedings of the 30th Annual International Conference on Mobile Computing and Networking,
pp. 543–557, 2024.

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li,
Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen llm applications via
multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023a.

Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. Smartplay: A benchmark for llms as

intelligent agents. arXiv preprint arXiv:2310.01557, 2023b.

Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the
weakness of large language model agents within a complex android environment. In Proceedings
of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6061–6072,
2024.

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark
prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441,
2023a.

Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent:

Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023b.

Zhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal chain-of-action agents.

arXiv preprint arXiv:2309.11436, 2023.

Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and
Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui automation task evaluation.
arXiv preprint arXiv:2404.16054, 2024.

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web

agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.

12

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

648
649

650
651
652
653
654
655

656
657
658
659
660

661
662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683

684
685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

APPENDIX

A LIMITATION AND FUTURE WORK

Given that constructing tasks is both time-consuming and resource-intensive, SPA-BENCH currently
includes 300 single-app tasks and 40 cross-app tasks, evenly split between English and Chinese. We
plan to expand the scope of our task collection and increase the diversity of task presentation (e.g., as
explored in the initial examples in Appendix G, by adding vague task descriptions and mimicking
various human tones). Since some apps are difficult to operate using emulators, we also aim to design
tasks that can be more easily experimented with. Additionally, we will execute experiments multiple
times to ensure robustness.

Regarding our agent framework, we intend to expand the scope of our research by supporting a
broader range of agents. This will include, but is not limited to, locally deployable agents, cloud-based
agents, and agents capable of operating other smart devices, such as Android tablets and iOS devices.

In terms of our evaluation method, particularly for single-app success detection, we plan to introduce
a more accurate approach and extend support for cross-app success detection. Furthermore, we will
define a more fine-grained metric to assess how agents complete tasks, moving beyond a simple
binary success signal.

B TASK COLLECTION

B.1 TASK APPS

The distribution and categories of apps for the 300 single-app tasks are presented in Figure 6.

Figure 6: Distribution of apps and their categories. Left: English apps. Right: Chinese apps. The
circle size is proportional to the number of tasks.

B.2 LIST OF TASKS

The 340 tasks, encompassing single-app English, single-app Chinese, cross-app English, and cross-
app Chinese categories, are detailed in Tables 4, 5, 6, 7 respectively.

B.2.1 SINGLE-APP ENGLISH TASKS

13

airbnbamazonbbcboltbooking.comcalculatorcalendarchromeclockcontactsdeliveroodictionaryespnevernoteexpediaebayfacebookfilesgmailgoogle mapsgoogle playgoogle tasksinstagramkeep noteslinkedinonenotequoraredditsettingsspotifytemutiktoktrillerwhatsappxyelpyoutubezoomaccuweather*EnglishComm&SocialSystemAppsProd&ToolsNews&ReadingTravel&NavShop&FinMedia&EntmtLifestyle支付宝哔哩哔哩菜鸟裹裹万年历时钟图库电信抖音饿了么高德地图航旅纵横协和医院好大夫华为浏览器京东美团QQQQ音乐去哪儿设置淘宝腾讯文档腾讯会议今日头条微信微博小红书有道词典知乎*ChineseComm&SocialSystemAppsProd&ToolsNews&ReadingTravel&NavShop&FinMedia&EntmtLifestyleUnder review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714

715
716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737

738
739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

App

Airbnb
Airbnb

Airbnb

Amazon
Amazon

Amazon

BBC
BBC

BBC

Bolt
Bolt

Bolt

Booking
Booking

Booking

Booking
Booking

Booking

Calculator
Calculator
Calculator
Calendar
Calendar
Calendar

Chrome
Chrome

Chrome

Clock
Clock
Clock

Clock
Clock

Clock

Contacts
Contacts

Contacts

Deliveroo
Deliveroo

Deliveroo

Merriam-
Webster
Merriam-
Webster
Merriam-
Webster

ESPN
ESPN

Diff
Level
1
2

Golden
Step
4
9

3

1
2

3

1
2

3

1
2

3

1
2

3

1
2

3

1
2
3
1
2
3

1
2

3

1
2
3

1
2

3

1
2

3

1
2

3

1

2

3

1
2

13

3
8

11

3
10

15

3
6

10

5
11

15

3
7

12

4
10
14
5
9
14

3
10

16

4
9
15

3
6

12

7
11

15

2
5

10

3

6

9

3
5

Table 4: Single-app English tasks.

Key Components

Task Description

1, guest
1, guest, wembley,
stadium
1, guest, wembley,
stadium
sunglasses
sunglasses, checkout

goggles, checkout

save
save

saved, items

eiffel, tower, route
eiffel, tower

triomphe,

arc, de,
bolt, cash

berlin
man, cdg

london, shanghai

settings
settings,
metric
notifications

celsius,

2
2, 3
5, 040
halloween, 31
16, haircut
17, dental, check, 7,
9, pm
taylor, swift
taylor, swift, wiki,
bookmark
taylor, swift, wiki,
reading

8
9
11

clock, london
clock, home, hong,
kong
settings, analog

agent, contact
agent, two, contact,
gmail
three, contact, work,
gmail, huawei

mcdonald
fries

order, fries

dictio-

dictio-

definition,
nary, thesaurus
definition,
nary, thesaurus
saved, words

klay, thompson
klay, thompson, like

Get the search results for stay for 1 adult anywhere any week.
Get the search results for stay tonight near ’wembley stadium’ for 1 adult.

Get the search results for stay tonight near ’wembley stadium’ for 1 adult. Add one
result to wishlist. Confirm that this item is in the wishlist.
Get the search results for ’sunglasses’.
Get the search results for ’sunglasses’. Filter with ’kids’. Add one result to cart.
Confirm that this item is in the cart.
Get the search results for ’goggles’. Filter with ’adult’. Add one result to cart.
Confirm that this item is in the cart. Compare with similar items. Add one of the
similar items to cart.
Navigate to ’Innovation’ section. Select ’Technology’ tab. Open any news article.
Go to app settings. Change the Text size to ’Smaller’. Navigate to ’Innovation’
section. Select ’Technology’ tab. Open any news article.
Go to app settings. Change the Text size to ’Larger’. Navigate to ’Business’ block.
Select ’Technology of Business’ tab. Open any news article. Save this article. Go to
Saved Items to confirm the article was added.
Select Eiffel Tower as my destination.
Select Louvre museum Paris as my pick-up location. Select Eiffel Tower as my
destination.
Select Louvre museum Paris as my pick-up location. Select Eiffel Tower as my
destination. Add ’Arc de Triomphe’ as the final destination and Eiffel Tower as
stopping point.
Get the search results for stays in Berlin. Select any date, rooms and guests.
Navigate to Flights section. Select any date. Choose a flight from Manchester
Airport to CDG Paris. Get the search results for a round trip.
Navigate to Flights section. Select one way flight. Choose the 1st of any month as
the flight date. Get the search results from Shanghai to London.
Navigate to app settings.
Navigate to app settings. Change Temperature to ’Degrees in Celsius’. Change
Units to ’Metric (km, m)’.
Navigate to app settings. Change Currency to ’Pound Sterling’. Disable all notifica-
tions.
Get the result for ’1+1’.
Get the result for ’log(20)+ln(e)’.
Get the result for ’log(20)+ln(e)’. Clear the results. Get the result for factorial 7.
Check the upcoming 31 October. Click on the event for that day.
Set up an all-day event titled ’Haircut’ on the 16th of any month.
Set up an event titled ’Dental Check’ on the 17th of any month. Set the time to from
7pm to 9pm.
Get the search results for Taylor Swift.
Get the search results for Taylor Swift. Go to her Wikipedia page. Add it to
bookmarks. Check the Bookmarks for confirmation.
Get the search results for Taylor Swift. Go to her Wikipedia page. Add it to
bookmarks. Move this bookmark to Reading List. Check the Reading List for
confirmation.
Set an alarm for 8am.
Set an alarm for 9am on weekdays.
Set an alarm for 10am on weekdays. Disable vibration for this alarm. Set another
alarm for 11am on weekends.
Add current time at London (UK) to clock.
Set Home time zone to ’Hong Kong’.

Add current time at Melbourne (Australia) to clock. Change style to Analog for
clock. Change style to Analog for screen saver.
Create a contact named ’Agent’. The phone number is +44 1234 567 890.
Create a contact named ’Agent Two’. The phone number is +44 1234 567 890. The
email is benchmark@gmail.com
Modify the last name of one of the contacts to ’Three’. Update the label for the
contact’s phone number to Work. Set the company to ’Huawei’. Add an email
agent.benchmark.2024@gmail.com. Label the email as Work.
Get the search results for McDonald’s.
Get the search results for McDonald’s. Enter a McDonald’s restaurant. Search for
fries there.
Get the search results for McDonald’s. Enter a McDonald’s restaurant. Search for
fries there. Add a small fries to the basket. Add two medium fries to the basket.
View the basket for confirmation.
Look up the definition of the word ’agent’.

Look up the definition of the word ’agent’. Switch to Thesaurus tab to find its
synonyms. Click on one of its synonyms. Switch back to Dictionary tab.
Look up the definition of the word ’agent’. Switch to Thesaurus tab to find its
synonyms. Click on one of its synonyms. Switch back to Dictionary tab. Save this
synonym. Confirm that synonym is in the saved words.
Get the search results for ’Klay Thompson’.
Get the search results for ’Klay Thompson’. See all the articles. Open one of the
articles.

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768

769
770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791

792
793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

ESPN

Evernote
Evernote

Evernote

Evernote
Evernote

Evernote

Expedia

Expedia

Expedia

Expedia
Expedia

Expedia

Facebook
Facebook
Facebook

Facebook
Facebook

Facebook

Files
Files

Files

Gmail

Gmail

Gmail

Gmail
Gmail

Gmail

Google
Maps
Google
Maps
Google
Maps
Google
Maps
Google
Maps
Google
Maps
Google
Play
Google
Play
Google
Play
Google
Play
Google
Play
Google
Play
Google
Tasks

3

1
2

3

1
2

3

1

2

3

1
2

3

1
2
3

1
2

3

1
2

3

1

2

3

1
2

3

1

2

3

1

2

3

1

2

3

1

2

3

1

11

3
7

13

3
6

12

4

8

12

7
11

13

4
8
11

2
6

11

3
7

18

5

9

11

2
6

11

3

7

10

3

6

12

3

7

14

3

6

10

3

thompson

agent, cookbook
agent, first, note

agent2, first, note,
hello, world, test

to-

literature, review
paper, writing,
morrow
recurring, main, git,
repo
rome, 2

paris, 25, 28, 2

hong, kong, 25, 28,
2
paris, 25, 28
rome, 26, 29

paris, 25, 28, save

hello, world
morning
bonne, nuit, eiffel,
tower, paris
settings
birthday,
tions
notifications, email,
sms
dcim
dcim, agent, created

notifica-

agent, created

paper

paper

paper, scheduled

settings
gmail, notification

inbox

hotel

hotel, 4

hotel, 4

gas, station

your, location

your, location, mc-
donald
whatsapp

Get the search results for ’Klay Thompson’. See all the articles. Open one of the
articles. Return to the player’s search results. Select the player. Turn on player news
notification. Follow the player.
Create a new notebook ’Agent Cookbook’.
Create a new notebook ’Agent’. Create a new note in the notebook with title ’First
note’. Return to the ’Agent’ notebook to confirm the note.
Create a new notebook ’Agent2’. Create a new note in the notebook. Write content
’Hello World!’ and title ’First note’. Create a new tag ’test’. Apply the tag ’test’ to
the note. Save the note. Return to the ’Agent2’ notebook.
Create a new task ’Literature Review’.
Create a new task ’Paper Writing’.Set the due date to tomorrow. Navigate to the
Tasks tab for confirmation.
Create a new task ’Maintain Git Repo’.Set it to repeat daily. Navigate to the Tasks
tab. Apply the recurring tasks filter. Confirm that task exists.
Check stays in Rome. The dates do not matter. Get the search results for 1 room
and 2 people.
Check stays in Paris. Choose from 25th to 28th any month. Get the search results
for 1 room for 2 people.
Check stays in Hong Kong. Choose from 25th to 28th any month. Get the search
results for 1 room for 2 people. Filter hotels with parking.
Check things to do in Paris. Get the search results for 25th to 28th of any month.
Check things to do in Rome. Get the search results for 26th to 29th of any month.
Save it to my trips.
Check things to do in Paris. Get the search results for 25th to 28th of any month.
Save it to my trips. Confirm that by checking the saved Paris trip.
Create a new post saying ’Hello World!’. Post it.
Create a new Public post saying ’Morning!’. Change to black background. Post it.
Create a new Public post saying ’Bonne Nuit’. Add the location as Eiffel Tower.
Post it.
Navigate to settings.
Navigate to settings. Disallow notifications for Birthdays.

Navigate to settings. Disallow notifications for Marketplace from Email and SMS.
Disallow notifications for Memories from Email and SMS.
Go to the ’DCIM’ folder in the internal storage.
Go to the ’DCIM’ folder in the internal storage. Create a subfolder named ’Agent
created’.
Go to the ’DCIM’ folder in the internal storage. Create a subfolder named ’Agent
created 2’. Create another subfolder named ’Agent created 3’. Then move the folder
’Agent created 2’ into the ’Documents’ folder in the internal storage.
Draft an email to agent.benchmark.2024@gmail.com asking them about their new
paper.
Send an email to agent.benchmark.2024@gmail.com asking them about their new
paper. Navigate to the Sent tab. Check the email details for confirmation after
sending.
Draft an email to agent.benchmark.2024@gmail.com asking them about their new
paper. Schedule it to be sent tomorrow morning. Navigate to the Scheduled tab.
Check the email details for confirmation for confirmation after scheduling.
Navigate to settings.
Navigate to settings. Check current setting for notifications. Turn off notification
for Attachments.
Navigate to settings. Check current setting for notifications. Turn off notification
for Miscellaneous. Disable ’notification dot’. Return to Inbox.
Get the search results for nearby hotel rooms.

Get the search results for nearby hotel rooms. Filter the results to show only those
that can accommodate 4 adults.
Get the search results for nearby hotel rooms. Filter the results to show only those
that can accommodate 4 adults. Further filter the results with ratings higher than 4.
Get the search results for nearby gas stations.

Get the search results for a nearby gas station that is open now. Get a driving route
to it.
Get the search results for a nearby gas station that is open now. Get a driving route
with the gas station as the first stop. Set McDonald’s as the final destination.
Get the search results for WhatsApp.

review

Get the search results for Facebook. Leave a 5-star review on its app store page.

whatsapp, review, re-
cent
settings

Get the search results for WhatsApp. Leave a 5-star review on its app store page.
Sort the reviews by most recent.
Check the details of General settings.

settings

Check the details of General settings. Switch to dark theme.

notification, settings

work, tasks

Check the details of General settings. Turn off all notifications. Confirm that all
notification settings for this device are off.
Create a new list ’Work’.

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822

823
824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845

846
847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Google
Tasks
Google
Tasks
Instagram
Instagram

Instagram

Instagram
Instagram

Instagram

Keep
Notes
Keep
Notes
Keep
Notes
LinkedIn
LinkedIn

LinkedIn

LinkedIn
LinkedIn
LinkedIn

Microsoft
OneNote
Microsoft
OneNote
Microsoft
OneNote
Quora
Quora
Quora

Quora
Quora

Quora

Reddit
Reddit

Reddit

Settings
Settings

Settings
Settings
Settings
Settings

Spotify
Spotify

Spotify

Temu
Temu

Temu

TikTok
TikTok

TikTok

WhatsApp
WhatsApp
WhatsApp

2

3

1
2

3

1
2

3

1

2

3

1
2

3

1
2
3

1

2

3

1
2
3

1
2

3

1
2

3

1
2

3
1
2
3

1
2

3

1
2

3

1
2

3

1
2
3

6

10

4
6

10

3
10

17

2

5

11

4
7

12

4
7
10

4

7

11

3
6
11

3
7

11

4
8

11

3
5

10
3
6
11

3
6

15

3
7

12

3
8

13

5
9
13

tasks, buy, groceries,
weekend
tasks, 12, visa, travel

messi, posts
cristiano, following,
message
minions,
tions, all
edit, profile
it

notifica-

account, privacy, pri-
vate
note, hello, 1

note, agent, hello, 2

agent, python, java

following, openai
join, huawei, groups

huawei,
hkrc

reposted,

engineer, jobs
engineer, jobs, spain
engineer, jobs, spain,
saved
agent, benchmark

benchmark2, appa-
gent, mobile, agent
prompts, test, pages

search, openai
search, openai
worth, thinking

following
questions,
answer
following, ask

follow,

worldnews, joined
premierleague, liver-
pool
blackmythwukong

timeout, 5,

screen, timeout
screen,
min
dark, theme
notification, history
store, notifications
instagram, storage

taylor, swift
taylor, swift

agent, playlist, love,
story, the, scientist

gaming, headset
gaming,
checkout
checkout

headset,

cat
cute, cat

cat

hi, you, message
mark, bench, contact
smart,
hi,
message

agent,

Create a new list ’Weekend’. Add new task ’Buy groceries’.

Create a new list ’Travel’. Add new task ’Visa’. Set date to the 12th of any month.

Get the search results for ’Messi’.
Get the search results for ’Cristiano Ronaldo’. Follow one account.

Get the search results for ’Minions’. Follow one account. Set to get all notifications
when they goes live. Turn on notifications for their posts.
Navigate to the page to edit my profile.
Navigate to the page to edit my profile. Add bio ’Hello World!’. Change pronouns
to ’it’.
Navigate to the page to edit my profile. Add link ’https://github.com’. Change
gender to Custom ’Them’. Switch to private account.
Create a new note. Write ’Hello this is a note1’ in the content.

Create a new note. Write ’Hello this is a note2’ in the content. Write ’Written by
Agent2’ as the note title.
Create a new checklist. Add two items ’Learn Python’ and ’Learn Java’. Write
’Goal of agent’ as the checklist title. Label this checklist as ’Agent’.
Get the search results for ’OpenAI’. Follow their page.
Get the search results for ’Huawei’. Follow their page. Filter the search results to
Groups. Join one of the Huawei groups.
Get the search results for ’Huawei HKRC’. Follow their page. Leave a ’Cheers!’
comment on one of its posts. Like the post. Repost the post instantly. View the
repost to confirm.
Get the search results for ’Engineer’ job.
Get the search results for ’Engineer’ job in Spain.
Get the search results for ’Engineer’ jobs in Spain. Save one of the jobs. Confirm it
is saved in My Jobs.
Create a new page with title ’Benchmark’ and content ’Test Agent’.

Create a new page with title ’Benchmark2’ and content TODO ’AppAgent’ and
’Mobile Agent’.
Create a new notebook ’test’. Create a new section ’prompts’ in ’test’ notebook.
Enter section ’prompts’ for confirmation.
Get the search results for ’OpenAI’.
Get the search results for ’OpenAI’. Filter to show only questions.
Get the search results for ’OpenAI’. Filter to show only questions. Select one
question or answer from the results to see more details. Add a comment ’Worth
thinking" to the answer.
Discover any Space. Follow that space.
Discover any Space. Follow that space. Go to questions in the space. Filter
unanswered questions. Follow one question.
Discover any Technology Spaces. Follow that space. Also follow one of the
suggested spaces. Turn off notification for the suggested space. Follow one of the
contributors of the suggested space.
Get the search results for ’r/worldnews’. Join the group.
Get the search results for ’r/PremierLeague’. Filter posts for Liverpool. Join the
group. Click on one of the posts.
Get the search results for ’r/BlackMythWukong’. Join the group. Set community
alerts to frequent. Click on one of the posts.
Check the current screen timeout.
Check the current screen timeout. Set it to 5 minutes.

Check the current screen timeout. Set it to 10 minutes. Then turn the dark theme on.
Go to notification settings. Turn on Notification History.
Go to notification settings. Turn off the notification from Google Play Store.
Go to notification settings. Turn off the ’Alerts’ and ’Likes’ notification from
Instagram. Clear the cache from storage.
Get the search results for the artist Taylor Swift.
Get the search results for the artist Taylor Swift. Enter her artist page. Shuffle play
her playlist.
Get the search results for the song ’Love Story’ by Taylor Swift. Add this song to
the new playlist namely ’Agent Playlist’. Then add another song ’The Scientist’ by
Coldplay to the same playlist. Check the playlist for confirmation.
Get the search results for gaming headset.
Get the search results for gaming headset. Sort the result by the lowest price to
highest. Add one to my shopping cart. Confirm that this item is in the cart.
Get the search results for gaming mouse. Filter items priced above 10. Add one to
cart. Confirm that this item is in the cart.
Get the search results for videos about pet cats.
Get the search results for videos about pet cats. Comment on a video with ’Such a
cute cat.’
Get the search results for videos about pet cats. Comment on a video with ’Such a
cute cat.’ Swipe through another two videos and like them.
Send a message ’Hi’ to myself.
Add new contact with the name ’Mark Bench’ and (+44)7437321230.
Add new contact with the name ’Smart Agent’ and (+44)7746953749. Send a
message ’Hi’ to ’Smart Agent’.

16

864
865

866
867
868
869
870
871

872
873
874
875
876

877
878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899

900
901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

Under review as a conference paper at ICLR 2025

X
X

X

X
X

X

Yelp
Yelp

Yelp

YouTube
YouTube

YouTube

YouTube
YouTube
YouTube

Zoom

Zoom

Zoom

1
2

3

1
2

3

1
2

3

1
2

3

1
2
3

1

2

3

3
9

12

5
8

15

2
6

10

4
8

12

4
10
14

5

9

14

agent, post, 1
agent, post, 2, reply

super,

agent, post, 3, reply,
amazing
mayday, following
nintendo,
mario
animal,
crossing,
timmy, tommy, post
restaurants
restaurants, chinese

review

tesla, subscribed
subscribed

subscriptions,

all,
microsoft, google

lebron
lebron, views
comment

smartphone, agent,
benchmark
smartphone, agent,
benchmark
smartphone, agent,
benchmark

Draft a post with the content ’Written by Agent1’.
Create a post with the content ’Written by Agent2’. Tag ’#animalcrossing’. Post it.
Check it from the profile.
Create a post with the content ’Written by Agent3’. Tag ’#animalcrossing’. Post it.
Check it from the profile. Then Like it. Reply to it with ’Amazing post’.
Search for the account @Mayday EN. Follow it.
Search for the account @Nintendo. Follow it. Search its post about ’Super Mario’.

Search for the account @animalcrossing. Follow it. Search its post about ’Timmy
and Tommy’. Repost one result. Check it from the profile for confirmation.
Get the search results for nearby restaurants.
Get the search results for nearby restaurants. Filter to include only Chinese restau-
rants that offer takeout. Sort them by distance.
Get the search results for nearby restaurants. Filter to include only Chinese restau-
rants that offer takeout. Sort them by distance. Select one result. Filter for 5-star
reviews.
Get the search results for the channel ’@Tesla’. Subscribe to the channel.
Get the search results for the channel ’@BMW’. Subscribe to the channel. Get the
search results for the channel ’@Mercedes’. Subscribe to the channel.
Get the search results for the channel ’@Google’. Subscribe to the channel. Get the
search results for the channel ’@Microsoft’. Subscribe to the channel. Navigate to
the Subscriptions tab. Show all subscriptions. Sort the subscriptions from A to Z.
Get the search results for videos about LeBron James.
Get the search results for videos about LeBron James. Filter videos under 4 minutes.
Get the search results for videos about LeBron James. Filter videos under 4 minutes.
Select any one of the results. Leave a comment ’great performance!’.
Schedule a meeting titled ’Smartphone Agent Benchmark’. Use personal meeting
ID.
Schedule a meeting titled ’Smartphone Agent Benchmark’. Use personal meeting
ID. Change the timezone to Hawaii. Repeat the meeting every day.
Schedule a meeting titled ’Smartphone Agent Benchmark’. Use personal meeting
ID. Change the timezone to Hawaii. Repeat the meeting every day. Disable waiting
room. Turn on host and participant video.

B.2.2 SINGLE-APP CHINESE TASKS

Table 5: Single-app Chinese tasks.

Key Components

Task Description

App

支付宝
支付宝
支付宝

Diff
Level
1
2
3

Golden
Step
3
9
13

哔哩哔哩 1
哔哩哔哩 2
哔哩哔哩 3

哔哩哔哩 1
哔哩哔哩 2

哔哩哔哩 3

哔哩哔哩 1
哔哩哔哩 2
哔哩哔哩 3

菜鸟裹裹 1
菜鸟裹裹 2

3
7
12

3
6

10

1
5
9

2
7

菜鸟裹裹 3

13

万年历
万年历

万年历

时钟
时钟

时钟

1
2

3

1
2

3

2
5

9

4
7

汇率换算
港币, 欧元
港币, 欧元, 100

游戏解说, 搜索
简介, 评论
哈哈, 发布

公开课, 学科课程
公开课, 学科课程,
收藏最多
好好好, 发布

消息, 聊天列表
动态
评论, 哈哈

地址
收货地址, 收件人,
张三
我的地址, 张三, 九
龙城区

黄历, 运程
宜出行

宜出行, 星期日, 星
期六

重复, 一
闹钟, 一个闹钟

11

闹钟, 响铃

搜索汇率换算。
进入汇率换算小程序，查看港币兑欧元汇率。
进入汇率换算小程序，查看港币兑欧元汇率。计算100港币 可以兑换多少欧
元。
搜索关键词“游戏解说”
搜索关键词“游戏解说”，对搜索结果按播放量排序，选择一个视频。
搜索关键词“游戏解说”，对搜索结果按播放量排序，选择一个视频，并对它
点赞、收藏，编辑评论“哈哈”。
查看公开课分区，展示学科课程栏目的内容。
查看公开课分区，展示学科课程栏目的内容，查看交叉学科相关的视频，
并按照收藏数排序。
查看公开课分区，展示学科课程栏目的内容，查看交叉学科相关的视频，
并按照收藏数排序，在收藏量最高的视频下面发送评论“好好好”。（停留在
评论发送页面）
浏览个人消息通知。
浏览个人消息通知，挑选一个聊天，查看好友动态。
浏览个人消息通知，挑选一个聊天，查看好友动态，点赞好友的一条动
态，编辑评论“哈哈”。
在我的页面中查看“收货地址”
在我的页面中选择收货地址，选择添加收货地址，姓名输入张三，手机号
输入123456789
在我的页面中选择收货地址，选择添加收货地址，姓名输入张三，手机号
输入123456789，详细地址输入无，地区选择九龙的九龙城区，然后保存这
个收货地址。
查看今天的黄历，然后查看今天的运程。
查看今天的黄历，然后查看今天的运程。然后在“工具”页面中打开“择吉
日”，然后查看“出行”的吉日。
查看今天的黄历，然后查看今天的运程。然后在“工具”页面中打开“择吉
日”，然后查看“出行”的吉日，然后将开始日期调整为下个月，并且设置“只
看周末”。
新建闹钟，设置在每个星期一重复，然后停止操作
新建闹钟，设置在每个星期一重复，修改标签（备注）或闹钟名为“一个闹
钟”，然后停止操作
新建闹钟，设置在每个星期一重复，修改标签（备注）或闹钟名为“一个闹
钟”，更换一个铃声音乐，保存闹钟

17

Under review as a conference paper at ICLR 2025

918
919

920
921
922
923
924
925

926
927
928
929
930

931
932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953

954
955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

电信

电信

电信

电信
电信
电信

抖音
抖音
抖音

抖音
抖音

抖音

饿了么

饿了么

饿了么

饿了么
饿了么

饿了么

1

2

3

1
2
3

1
2
3

1
2

3

1

2

3

1
2

3

高德地图 1
高德地图 2
高德地图 3

高德地图 1

高德地图 2

高德地图 3

高德地图 1

高德地图 2

高德地图 3

航旅纵横 1
航旅纵横 2

航旅纵横 3

航旅纵横 1
航旅纵横 2

航旅纵横 3

好大夫
好大夫
好大夫

好大夫
好大夫

好大夫

华为浏览
器
华为浏览
器
华为浏览
器

1
2
3

1
2

3

1

2

3

3

6

9

2
5
9

3
6
13

3
8

11

2

5

10

2
6

9

3
6
10

3

6

10

3

6

9

6
11

15

5
10

18

2
9
14

3
7

11

3

7

12

查用量, 查费用, 查
积分
支付方式, 确认交
易
中国电信, 立即支
付

我的, 账户信息
5G开启, 梦想加速
操作成功

张国伟, 关注
张国伟
张国伟, 已关注, 私
信
已关注
评论, 期待下一条
视频
收藏, 视频

美食外卖, 快餐便
当
点餐

加入购物车, 详情

搜索
生椰拿铁, 温度, 数
量
确认订单

美食
导航, 路线
公交, 推荐

超市, 位置距离, 推
荐排序
我的位置, 驾车, 开
始导航
我的位置, 加油站,
驾车, 开始导航

著名景区, 位置距
离, 推荐排序
太阳岛风景区

提交订单

北京, 深圳
首都大兴, 宝安

经济舱, 大兴

深圳, 筛选, 酒店
星级, 好评优先, 筛
选
封面

订单列表, 全部
神经外科
预约

失眠, 介绍
北京, 推荐, 医院

信息

首页

进入查询办理，查看自己的积分，费用与用量（不分先后）

进入查询办理，查看自己的积分，费用与用量（不分先后），随后查看自
己的话费账单并充值10元额度（停在选择支付方式界面）
进入查询办理，查看自己的积分，费用与用量（不分先后），随后查看自
己的话费账单并充值10元额度（停在选择支付方式界面），选择微信支付
选项并停在立即支付界面前，不要付钱
进入用户中心的个人信息页面。
进入用户中心的个人信息页面，设置电信签名为5G开启，梦想加速。
进入用户中心的个人信息页面（如果需要登录则登录账号），设置电信签
名为5G开启，梦想加速，最后从本地相册选择一张图片设置为个人头像。
搜索博主张国伟
搜索博主张国伟，查看博主主页并查看其中一条视频，点赞该视频
搜索博主张国伟，查看博主主页并查看其中一条视频，点赞该视频任意一
条评论，并关注主播
进入关注界面，查看关注博主的主页
进入关注界面，查看关注博主的主页，观看关注的博主发布的视频并发表
评论“期待下一条视频”
进入关注界面，查看关注博主的主页，观看关注的博主发布的视频并发表
评论“期待下一条视频”，收藏该视频并查看我的收藏夹
进入美食外卖，选择快餐便当

进入美食外卖，选择快餐便当，按照好评优先排序，选择一家店铺查看详
情
进入美食外卖，选择快餐便当，按照好评优先排序，选择一家店铺查看详
情，浏览评价，查看商家信息或品牌故事，返回点餐，选择任意餐食查看
详情
进入甜品饮品板块，进入搜索界面
进入甜品饮品板块，进入搜索界面，搜索瑞幸咖啡，选择推荐的店铺查看
详情，选择生椰拿铁规格
进入甜品饮品板块，进入搜索界面，搜索瑞幸咖啡，选择推荐的店铺查看
详情，选择生椰拿铁规格，选择冰，不额外加糖其余默认，加入购物车并
去结算，不要提交订单
搜索附近的餐厅。
搜索附近的餐厅，按照好评优先排序，并点击列表里面的一家餐厅。
搜索附近的餐厅，按照好评优先排序，并点击列表里面的一家餐厅，查看
用户详情然后点击“路线”来规划乘公交从当前位置到达该餐厅的路线。
查找附近的超市。

查找附近的超市，点击其中一个超市，点击“路线”来规划路线。

查找附近的超市，点击其中一个超市，点击“路线”来规划路线，然后查看周
围是否有加油站，选择一个合适的加油站作为经停点并确认最佳的驾车路
线。
进入附近界面，点击旅游，进入著名景区界面

进入附近界面，点击旅游，进入著名景区界面，切换到哈尔滨。将推荐排
序换为好评优先，选择太阳岛风景区
进入附近界面，点击旅游，进入著名景区界面，切换到哈尔滨。将推荐排
序换为好评优先，选择太阳岛风景区。点击实用信息查看相关消息，购买
太阳岛寒地野生动物园的票，进入订单界面。
搜索某个月份16号北京到深圳的机票
搜索某个月份16号北京到深圳的机票，筛选起飞时段为12:00-18:00并规定舱
位为经济舱
搜索某个月份16号北京到深圳的机票，筛选起飞时段为12:00-18:00并规定舱
位为经济舱，从中选择一班飞机，并查看退改签详细信息
进入酒店信息界面，选择某个月份16日-18日深圳的酒店预定
进入酒店信息界面，选择某个月份16日-18日深圳的酒店预定，筛选品牌
为“全部经济酒店”，推荐顺序为好评优先
进入酒店信息界面，选择某个月份16日-18日深圳的酒店预定，筛选品牌
为“全部经济酒店”，推荐顺序为好评优先，选择位置在机场附近的一家酒店
查看个人的全部订单
查看个人的全部订单。回到首页在专家门诊中查找神经外科有关的医生
查看个人的全部订单。回到首页在专家门诊中查找神经外科有关的医生，
选择一位医生申请服务并预约挂号，时间任意选择。
进入知识界面，选择失眠选项，查看介绍。
进入知识界面，选择失眠选项，查看介绍。点击推荐医院，更改地区为北
京全部。
进入知识界面，选择失眠选项，查看介绍。点击推荐医院，更改地区为北
京全部。点击排行第一的医院并选择一位专家将其加入备选，并点击预约
挂号查看预约信息。
搜索bilibili.com.

潜艇伟伟迷

搜索bilibili.com，在网站中搜索“潜艇伟伟迷”并进入UP主列表

书签, 潜艇伟伟迷

搜索bilibili.com，在网站中搜索“潜艇伟伟迷”并进入UP主列表，添加该页面
为书签，并确认在书签管理中存在该书签

18

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984

985
986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007

1008
1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

华为浏览
器
华为浏览
器
华为浏览
器
京东
京东
京东

京东
京东

京东

美团
美团
美团

美团
美团
美团

美团
美团
美团

QQ音乐
QQ音乐
QQ音乐
QQ音乐
QQ音乐
QQ音乐
QQ音乐
QQ音乐
QQ音乐

去哪儿

去哪儿

去哪儿

去哪儿

去哪儿

去哪儿

设置
设置

设置

淘宝
淘宝
淘宝

淘宝
淘宝
淘宝

淘宝
淘宝
淘宝

1

2

3

1
2
3

1
2

3

1
2
3

1
2
3

1
2
3

1
2
3
1
2
3
1
2
3

1

2

3

1

2

3

1
2

3

1
2
3

1
2
3

1
2
3

腾讯文档 1

腾讯文档 2
腾讯文档 3

腾讯会议 1

腾讯会议 2

3

6

10

3
9
14

3
7

12

3
7
13

3
6
9

2
8
13

3
6
9
3
6
9
2
5
9

4

7

12

2

6

12

3
6

9

3
6
10

3
7
10

3
7
10

3

6
10

3

9

淘宝

浏览电商网站taobao.com

商品, 华为, 搜索

浏览电商网站taobao.com，搜索华为mate60pro

全部, 评

索尼, WH
索尼, WH
索尼, WH, 结算

地毯
购物车, 详情

购物车, 结算

螺蛳粉
螺蛳粉
螺蛳粉, 订单

健身
教练印象
订单, 支付

浏览记录, 商户
评价
评价详情

周杰伦
周杰伦, 评论
周杰伦, 评论
个人资料
留言
留言
巅峰榜, 热歌榜
热歌榜, 我喜欢
微信好友, 分享

国 内, 海 外, 入 住,
离店, 东京
海 外, 入 住, 离 店,
6日, 19日
东 京, 搜 索, 2人,
2床, 1居

机 票, 单 程, 往 返,
乘机人
机 票, 往 返, 深 圳,
南京
深圳, 南京, 9天, 去
程, 返程
开发, 选项, 内存
内存使用量, 小时

内存使用量, 详细
信息

华为P50
评价
关注成功

抹茶旦旦
抹茶旦旦
鬼灭之刃, 分享给
好友
男士洗发水
加入购物车
已关注, 已收藏, 加
入购物车, 关注成
功
请输入标题, 请输
入正文
标题, 你好
今天, 标题

预定会议, 入会密
码
会议水印

浏览电商网站taobao.com，搜索华为mate60pro，选择按销量排序，点击任意
商品并查看评价
搜索索尼 WH-1000XM4 头戴式耳机。
搜索索尼 WH-1000XM4 头戴式耳机，筛选出价格低于2000元的结果。
搜索索尼 WH-1000XM4 头戴式耳机，筛选出价格低于2000元的结果，查看
商品详情页，加入购物车，查看购物车以确认。
搜索一款地毯。
搜索一款地毯，筛选销量最多的结果，查看商品详情，如果尚未收藏则收
藏商品。
搜索一款地毯，筛选销量最多的结果，查看商品详情，如果尚未收藏则收
藏商品，并查看优惠详情，然后选择商品，设置数量为2，加入购物车
搜索一家附近的螺蛳粉店。
搜索一家附近的螺蛳粉店，并查看店内评分、评论以及菜品。
搜索一家附近的螺蛳粉店，并查看店内评分、评论以及菜品，然后下单一
份螺蛳粉(停在支付前的订单界面)。
查找附近的一家健身馆。
查找附近的一家健身馆，查看一名高评价的健身教练介绍。
查找附近的一家健身馆，查看一名高评价的健身教练介绍并且购买票。
（停留在订单界面）
查看浏览过的商品或商家
为浏览过的商品或商家并撰写评论“Hello world”
为浏览过的商品或商家撰写评论“Hello world”，回到个人主页，查看自己刚
刚发布的评论
搜索歌手周杰伦。
搜索歌手周杰伦，打开他的一个专辑然后查看他的专辑评论。
搜索歌手周杰伦，查看他的专辑然后发表评论“hello world”到评论区。
查看个人资料
查看个人资料，然后查看自己发表过的评论
查看个人资料，然后查看自己发表过的评论，并回复“好好好”给自己的评论
浏览音乐排行榜。
浏览音乐排行榜，找到排名前五的歌曲。将前三名添加至我喜欢。
浏览音乐排行榜，找到排名前五的歌曲，将前五名添加至我喜欢并将第五
名分享给微信好友。（停留在分享界面）
在首页选择民宿 客栈，选择海外，城市选择东京

在首页选择民宿 客栈，选择海外，城市选择东京，入住时间选择为某个月
份的6日，离店时间选择为某个月份的19日
在首页选择民宿 客栈，选择海外，城市选择东京，入住时间选择为某个月
份的6日，离店时间选择为某个月份的19日，入住条件中总人数选择2人，
床铺数选择2床，居室数选择1居
在首页选择机票，选择往返界面

在首页选择机票，选择往返界面，出发城市选择深圳，抵达城市选择南京

在首页选择机票，选择往返界面，出发城市选择深圳，抵达城市选择南
京，出发日期定为某个月份9号，返回日期定为某个月份17号，点击搜索。
点击“系统与更新”，进入开发者选项
点击“系统与更新”，进入开发者选项，点击内存，将选项换成1天后查看各
个应用的内存使用量
点击“系统与更新”，进入开发者选项，点击内存，将选项换成1天后查看
各个应用的内存使用量，然后进入其中两个应用的内存使用量页面分别查
看。
搜索“华为P50”。
搜索“华为P50”，点击一件商品查看详情，下拉查看评价。
搜索“华为P50”，点击一件商品查看详情，下拉查看评价，收藏此商品，进
入店铺，关注此店铺。
搜索《抹茶旦旦》周边。
搜索《抹茶旦旦》周边，查看并选择一款拼图，加入购物车。
搜索《鬼灭之刃》动漫周边，查看并选择一款T恤，加入购物车，收藏商
品，并且分享给好友（结束在分享页面前就可以）。
搜索男士洗发水。
搜索男士洗发水，查看商品详情，然后加入购物车。
搜索男士香水，查看商品详情并收藏，然后加入购物车，并将香水店铺放
入店铺关注列表。

新建一个空白文档

新建一个空白文档，设置标题为“标题”，设置正文为“你好”
新建一个空白文档，设置标题为“标题”，设置正文为“你好”，查看文档大纲
后返回至主页
预定一个常规会议，开启入会密码

预定一个常规会议，开启入会密码并将入会密码设置为“111111”后进入设置
会议水印界面

19

Under review as a conference paper at ICLR 2025

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038

1039
1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061

1062
1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

腾讯会议 3

今日头条 1
今日头条 2
今日头条 3
今日头条 1

今日头条 2
今日头条 3

微信
微信
微信

微博

微博

微博

微博
微博
微博

小红书

小红书

小红书

小红书
小红书

小红书

1
2
3

1

2

3

1
2
3

1

2

3

1
2

3

协和医院 1
协和医院 2
协和医院 3

协和医院 1
协和医院 2

协和医院 3

有道词典 1
有道词典 2
有道词典 3

知乎
知乎

知乎

知乎
知乎
知乎

1
2

3

1
2
3

12

4
7
10
2

5
10

2
9
15

3

6

13

3
7
10

3

6

11

2
5

10

3
7
10

3
6

9

4
7
10

3
7

13

4
10
13

会议详情

科技新闻
分享
分享, 收藏
设置, 编辑资料, 账
号与安全
历史, 编辑, 关注
消息, 私信, 评论

预定一个常规会议，开启入会密码并将入会密码设置为“111111”，开启会议
水印后完成设置。
搜索关键词“科技新闻”。
搜索关键词“科技新闻”，查看前2条新闻。
搜索关键词“科技新闻”，查看前2条新闻，点赞并收藏。
打开设置。

关注, 说点什么
评论

搜索“深圳美食”，关注两个发微博的用户。

搜索一种名为“减肥食谱”的笔记，按照热度排序。

打开设置，清理缓存文件并查看历史记录。
打开设置，清理缓存文件，查看历史记录并使用“一键清空”功能清空历史记
录，然后浏览消息私信。
进入朋友圈的页面。
发表一则朋友圈，从相册中任意选一张图，并配文“今天真开心”。
发表一则朋友圈，从相册中任意选一张图，并配文“今天天气真好”。点赞这
条朋友圈，并评论“希望大家点赞”。
搜索“周末去哪儿玩”

搜索“周末去哪儿玩”，关注两个发微博的用户并转发其中一条微博，附上评
论“这个地方看起来不错”（并且停留在发送页面）
搜索用户名为“李子柒”的用户
搜索用户名为“李子柒”的用户，关注该用户，浏览其最新发布的一条微博
搜索用户名为“腥味猫罐”的用户，关注该用户，浏览其发布的一条微博，并
在这条微博下进行评论：“hello world”（并且发送出去）
搜索一种名为“减肥食谱”的笔记。

今天真开心
今天天气真好, 希
望大家点赞
周末去哪儿玩, 综
合
深圳美食, 综合, 已
关注
这个地方看起来不
错
李子柒
李子柒, 全部微博
腥味猫罐, 发送成
功
减肥食谱, 全部, 用
户, 筛选
减肥食谱, 全部, 用
户, 筛选
发送, 很有用, 谢谢 搜索一种名为“减肥食谱”的笔记，按照热度排序，观看其中一个热度最高的
笔记，点赞该笔记，收藏该笔记，然后编辑评论“很有用，谢谢“，停留在评
论发送页面不要发送。
在首页切换至视频类别，观看一条推荐的视频笔记。
在首页切换至视频类别，观看一条推荐的视频笔记点赞该视频，关注视频
发布者，并查看视频评论区。
在首页观看一条推荐的视频笔记，点赞该视频，关注视频发布者，并查看
视频评论区，随后查看该用户的其他笔记，并收藏其中两篇笔记。
进入预约挂号界面
进入预约挂号界面，打开“东单院区普通门诊”，查看基本外科医生列表
进入预约挂号界面，打开“东单院区普通门诊”，查看基本外科医生列表，选
择一个医生，查看医生介绍主页并点击常规咨询
在便民服务中点击专科专病
在便民服务中点击专科专病，选择放射治疗科，点击胸部肿瘤组查看该病
简介
在便民服务中点击专科专病，选择放射治疗科，点击胸部肿瘤组查看该病
简介后点击专家介绍，选择一个专家查看其主页并点击常规咨询选项
点击学习，搜索“高三”
点击学习，搜索“高三”，进入图文分类点击一篇图文，查看评论
点击学习，搜索“高三”，进入图文分类点击一篇图文，查看评论后发表评
论“hello world”
搜索“编程猫”
搜索“编程猫”，搜索用户，并进入结果中一个用户的主页，查看其最新发布
的文章
搜索“编程猫”，搜索用户，并进入结果中一个用户的主页，查看其最新发布
的文章，点赞该文章，点击收藏，并在评论区发表评论“hello world”
搜索“人工智能”专栏
搜索“人工智能”专栏，查看一篇专栏中的文章并评论“hello world”
搜索“人工智能”专栏，查看一篇专栏中的文章并评论“hello world”，并将该
文章收藏

挂号, 日期, 门诊
基本外科
医生主页, 咨询

人工智能, 专栏
评论
更改

专科专病
胸部肿瘤组, 简介

高三, 图文
评论, 发布
hello, world

编程猫, 实时
编程猫, 关注

已关注, 收藏成功

编程猫, 评论

咨询

B.2.3 CROSS-APP ENGLISH TASKS

Table 6: Cross-app English tasks.

Category

App

General Tool

General Tool

Google Play
Store, Setting
Keep Notes,
LinkedIn

General Tool

Clock, Setting

Information
Management

Facebook, Set-
ting

Diff
Level
1

Golden
Step
15

1

1

1

12

12

17

Task Description

Open Google Play Store, uninstall the Alibaba.com app, then go to Settings and
verify if the app is still listed under app resources.
Use the LinkedIn app to search for a customer service representative position. Select
a job, open Keep Notes, create a new note, record the company’s name, and set the
note’s title to ‘customer service representative’.
In the Settings app, enable ‘Data Saver’ mode. Open the Clock app and set an alarm
for 6:00 AM.
Open Facebook, search for tropical pictures, save one picture to your phone, go to
the Wallpaper section in the Settings app, and set the saved picture as your wallpaper.

20

1080
1081

1082
1083
1084
1085
1086
1087

1088
1089
1090
1091
1092

1093
1094
1095
1096
1097
1098

1099
1100
1101
1102
1103
1104

1105
1106
1107
1108
1109
1110

1111
1112
1113
1114
1115

1116
1117
1118
1119
1120
1121

1122
1123
1124
1125
1126
1127

1128
1129
1130
1131
1132
1133

Under review as a conference paper at ICLR 2025

Information
Management
Information
Management
Media Entertain-
ment

Media Entertain-
ment
Media Entertain-
ment
Multi Apps

Multi Apps

Multi Apps

Multi Apps

Multi Apps

Social Sharing

Social Sharing

Social Sharing

Web Shopping

Web Shopping

Web Shopping

Store,

Calendar,
Chrome
Spotify,
Chrome
Google
Play
Youtube
Google Play
Store, Chrome
Clock,
Youtube
Quora, eBay,
Chrome
Clock,
Chrome,
Instagram
Triller,
Set-
ting, Google
Play Store
Clock, What-
sApp, Zoom

AccuWeather,
Evernote,
Expedia
X, Facebook

BBC News,
Gmail
Spotify, Face-
book
eBay,
book
Amazon,
Temu
Airbnb, Insta-
gram

Face-

1

1

1

1

1

2

2

2

2

2

1

1

1

1

1

1

16

13

12

10

11

20

20

15

23

25

20

10

19

15

15

19

Using Chrome, search for the date of the next Winter Olympics opening ceremony
and then set a reminder for that date in your Calendar.
Open Chrome, search for the top Country songs of 2023, identify a song from the
search results, then switch to Spotify and add that song to your playlist.
Watch a YouTube video about fitness tracking app recommendations, check the
video’s description for the suggested apps, then use Google Play Store to download
one of the suggested apps.
Utilize Chrome to research different Recipe Organizer apps, and then proceed to
Google Play Store, download one of your choice.
Search for a relaxing soundscape video on YouTube, use the Clock app to set a timer
for 3 hours, then go back to YouTube and play the video.
Utilize Chrome to search for a biography book, then use Quora to read reviews
about the book, and finally add the book to watchlist on eBay.
Organize a movie night by choosing a horror film using Chrome, sending an invita-
tion to one of your friends via Instagram, and setting a reminder in the Clock app
for 8:35 PM on Sunday.
First, install the Triller app from the Google Play Store. After the installation, open
the Triller app, navigate to the Setting app to check current battery status, reopen
the Triller app.
Arrange a business meeting using Zoom, copy the sharing text, go to WhatsApp,
send the copied text to a contact, set an alarm using the Clock app at the meeting
time.
Utilize Expedia to search for Things to do in Beijing on 18-20th, choose one and
record the sharing text using Evernote, open AccuWeather to check daily weather in
Beijing.
Use the social media platform X to post a photo, copy the link to your post, then
open Facebook and send the link to a friend
Use the BBC News app to search for Artificial Intelligence news, read an article,
share it via Gmail, send to agent.benchmark.2024@gmail.com.
Listen to a Reggaeton album on Spotify, then share the albumâC™s name with a
friend on Facebook.
Search for ‘Circe by Madeline Miller’ on Facebook, read one of the posts, head over
to eBay, search the book, add it to watchlist.
Investigate the prices for Catan board game across Amazon and Temu, then proceed
to add the cheaper option into your cart.
Use Instagram to search for an itinerary for Venice, Italy, and then proceed Airbnb,
book accommodations at Venice, Italy.

B.2.4 CROSS-APP CHINESE TASKS

Table 7: Cross-app Chinese tasks.

Category

App

Diff
Level

General Tool

饿了么, 设置 1

Golden
Step
10

General Tool
General Tool
Information
Management
Information
Management
Information
Management
Media Entertain-
ment
Media Entertain-
ment
Media Entertain-
ment
Multi Apps

Multi Apps

Multi Apps

Multi Apps

Multi Apps

Social Sharing

Social Sharing

1
1
1

设置, 抖音
微信, 设置
华 为 浏 览 器,
bilibili
华 为 浏 览 器,
QQ音乐
小红书, 设置 1

1

华 为 浏 览 器,
QQ音乐
抖音, 微博

QQ音乐, bili-
bili
华 为 浏 览 器,
bilibili, QQ
淘 宝, 京 东,
腾讯文档
高德地图, 美
团, 微信
去哪儿, 航旅
纵横, 微信
华 为 浏 览 器,
淘宝, 图库
bilibili, QQ

小 红 书,
QQ音乐

1

1

1

2

2

2

2

2

1

1

6
12
9

11

18

12

16

10

14

18

18

21

16

9

10

Task Description

打开饿了么，搜索“汉堡包”，然后进入设置APP，在应用中找到饿了么，关
闭后台运行权限
在设置APP中开启省流模式，然后打开抖音
进入设置，切换到深色模式，然后打开微信，将深色模式设置为“跟随系统”
在华为浏览器中搜索“地球上最大的动物是”，然后在bilibili中搜索这种动物
的视频并停留在搜索结果界面
在华为浏览器中搜索 2024年的热门流行歌曲，选择一首歌曲后，切换
到QQ音乐并将该歌曲添加到您的播放列表中
打开小红书，搜索“冬日美景”，保存一张图片，然后在设置中将保存的图片
更换为新的壁纸
在华为浏览器中搜索“歌曲七里香的作者是谁”，然后在QQ音乐中搜索这名
歌手，进入歌手主页，播放任意一首歌并进入该歌曲的主页
利 用 抖 音 搜 索“BLACKPINK”， 观 看 任 意 一 个 视 频 ， 然 后 去 微 博 搜
索BLACKPINK账号并关注
打 开QQ音 乐 ， 搜 索 周 杰 伦 ， 查 看 他 的 主 页 ， 记 录 下 一 首 歌 曲 ， 并
在bilibili中搜索该歌曲相关的视频
在华为浏览器中搜索“贝塞斯达最成功的游戏是什么”，然后在bilibili搜索任
意一个有关该游戏的视频，观看视频并分享到QQ空间
分别在淘宝和京东搜索"华为Mate60Pro"，然后在腾讯文档里新建一个"华
为Mate60Pro"价格的文档，把淘宝和京东搜索到的价格记录下来
在美团搜索一家附近的餐厅，用高德地图查找驾车路线，把路线分享到微
信朋友圈
打开去哪儿APP搜索深圳酒店，切换到航旅纵横查看某天从北京飞往深圳的
机票，并将其中一张机票分享给微信好友
在华为浏览器中搜索“英伟达最强专业计算卡”，在淘宝中搜索该计算卡并查
看商品详情，保存预览图到图库，最后在图库查看这张图片
在bilibili中搜索“自制关卡 胆小菇之梦”，点击进入任意一个视频，分享该视
频到qq空间
在QQ音乐上播放一首周杰伦的歌，然后将音乐分享到小红书，发布笔记

21

Under review as a conference paper at ICLR 2025

1134
1135

1136
1137
1138
1139
1140
1141

1142
1143
1144
1145
1146
1147

1148
1149
1150
1151
1152

1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

1165
1166
1167
1168
1169
1170

1171
1172
1173
1174
1175

1176
1177
1178
1179
1180
1181

1182
1183
1184
1185
1186
1187

(a) Level 1: “mcdonald”

(b) Level 2: “fries”

(c) Level 3: “order” and “fries”

Figure 7: A visualised example of key components across three difficulty levels, with subcaptions
indicating the key components for each level and highlighted key components in the corresponding
screenshots.

Social Sharing
Web Shopping

知乎, 微博
知乎, 京东

1
1

Web Shopping

小红书, 淘宝 1

Web Shopping

华 为 浏 览 器,
淘宝

1

11
14

14

14

在知乎查看热榜，进入任意一个问题，然后将其转发到微博
在知乎中搜索“1000元以下音箱推荐”，并在京东搜索其中提到的一款音箱，
选择一个加入购物车
在小红书上找到一款2024年推荐的运动相机，然后前往淘宝，将该商品加
入购物车
在华为浏览器中搜索“最新款华为mate系列手机叫什么”，并在淘宝中搜索该
型号的手机后将其加入购物车

B.3 EXAMPLE OF KEY COMPONENTS

Figure 7 shows an example of key components.

B.4 CROSS-APP EXAMPLE TASK DEMO

Figure 8 illustrates two examples of English cross-app tasks, each with a different difficulty level.

B.5 STEPS OF TASKS

Refer to Figure 9 for a box plot illustrating the distribution of steps across tasks.

C INTEGRATED AGENTS

The benchmark includes 11 state-of-the-art autonomous agents, shown in Table 8. These agents differ
in core models, input modalities, action spaces, and additional training or prompting modules. They
fall into two categories: those leveraging off-the-shelf MLLMs (e.g., GPT, Qwen), and those using

22

Under review as a conference paper at ICLR 2025

Figure 8: Example cross-app tasks with trajectories collected by human annotators.

Figure 9: Distribution of steps taken by humans to execute tasks, categorised by difficulty level and
task type.

fine-tuned models with parameter counts ranging from 1.3 billion to 18 billion. Fine-tuned models,
trained primarily on the offline AITW Rawles et al. (2024b) dataset, focus on action prediction,
with DigiRL additionally employing online RL training. In our benchmarks, unlike their offline
training settings, all agents are tested in real-world scenarios that require precise action grounding
and long-sequence task execution.

C.1 AGENT INPUT MODALITIES

Input modalities and action spaces define an agent’s ability to interact with mobile user interfaces.
Screenshot input is intuitive, capturing everything a human would see, but MLLMs often struggle
to identify actionable UI elements and link them with screen coordinates Zheng et al. (2024). To
address this, some agents enhance input with XML files, accessibility trees, or information obtained
through Optical Character Recognition (OCR). For instance, AppAgent Yang et al. (2023b) and
AutoDroid Wen et al. (2024) use element IDs and coordinates, M3A Rawles et al. (2024a) annotates
screenshots with key UI elements, while MobileAgent Wang et al. (2024b) first identifies interaction
elements and then uses OCR or icon recognition to locate them.

23

1188
1189

1190
1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206

1207
1208
1209
1210
1211
1212

1213
1214
1215
1216
1217
1218

1219
1220
1221
1222
1223
1224

1225
1226
1227
1228
1229

1230
1231
1232
1233
1234
1235

1236
1237
1238
1239
1240
1241

Level 2 Task: Arrange a business meeting using Zoom, copy the sharing text, go to WhatsApp, send thecopied text to a contact, set an alarm using the Clock app at the meeting time.Level 1 Task: Using Chrome, search for the date of the next Winter Olympics opening ceremony and thenset a reminder for that date in your Calendar.ChromeLevel 1Task difficultyLevel 2HardCalendarZoomClockWhatsApp2 AppsCross Apps3 AppsENG Diff 1CHN Diff 1ENG Diff 2CHN Diff 2ENG Diff 3CHN Diff 30510152025Number of StepsSingle-App TasksENG Diff 1CHN Diff 1ENG Diff 2CHN Diff 2Cross-App TasksUnder review as a conference paper at ICLR 2025

1242
1243

1244
1245
1246
1247
1248
1249

1250
1251
1252
1253
1254
1255

1256
1257
1258
1259
1260

1261
1262
1263
1264
1265
1266

1267
1268
1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283

1284
1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

Table 8: Comparison of agents integrated into SPA-BENCH framework across key dimensions.

Agent

Core Model

UI Representation

Touch Point Localisation

AppAgent Yang et al. (2023b)
AutoDroid Wen et al. (2024)
MobileAgent Wang et al. (2024b)
MobileAgentV2 Wang et al. (2024a)

M3A Rawles et al. (2024a)

T3A Rawles et al. (2024a)

GPT-4o
GPT-4o
GPT-4o
GPT-4o

GPT-4o

GPT-4o

SeeAct Rawles et al. (2024a); Zheng et al. (2024)

GPT-4o

Auto-UI Zhan & Zhang (2023)

Fine-tuned FLAN-Alpaca-Base
(200M) + BLIP-2-T5-Instruct (1.1B)

CogAgent Hong et al. (2024)

CogAgent-18B

DigiRL Bai et al. (2024)

Fine-tuned FLAN-Alpaca-Base
(200M) + BLIP-2-T5-Instruct (1.1B)

Screenshot + XML
HTML
Screenshot
Screenshot
Screenshot +
Accessibility Tree

Accessibility Tree

Screenshot +
Accessibility Tree

Screenshot

Screenshot

Screenshot

OdysseyAgent Lu et al. (2024)

Fine-tuned Qwen-VL (9.6B)

Screenshot

Coordinates from XML
Coordinates from HTML
OCR + Icon Recognition
OCR + Icon Recognition
Coordinates from
Accessibility Tree
Coordinates from
Accessibility Tree
Coordinates from
Accessibility Tree
Normalized coordinates
from Model
Normalized coordinates
from Model
Normalized coordinates
from Model
Normalized coordinates
from Model

C.2 ADOPTION OF AGENTS INTO FRAMEWORK

Integrating agents into the framework required several adaptations. We used their original open-
source implementations, with the exception of SeeAct Zheng et al. (2024), for which we adopted
AndroidWorld’s action grounding module. For agents using fine-tuned models (i.e., Auto-UI1,
DigiRL, OdysseyAgent, CogAgent), which lacked direct Android interaction capabilities, we used
UIAutomator22 for end-to-end task execution.

C.3 LOGS AND ERRORS

While task descriptions and screenshot trajectories remain the primary inputs/outputs, we also
logged executed actions, performance metrics (steps, time, API costs), and errors. Errors were
categorised as expected (e.g., invalid responses) or unexpected (e.g., network failures). Expected
errors arise from the agent’s limitations, such as failing to generate valid actions or when certain
functionalities are restricted. Unexpected errors refer to unforeseeable issues like network failures,
Android malfunctions, or CAPTCHA challenges. The framework automatically re-runs such tasks
to avoid penalising agents for unexpected errors, ensuring a fair and accurate assessment of their
capabilities and limitations.

C.4 SCOPE OF USING ANDROID EMULATOR

Certain English tasks involving WhatsApp and OneNote, as well as most Chinese tasks, were executed
exclusively on physical Android devices rather than emulators 3. This decision was due to strict app
control measures, such as restrictions on logging in across multiple devices and compatibility issues
with emulator system images. While physical Android devices can replace the emulator, doing so
would eliminate the snapshot functionality described in Section 4.2.

D SINGLE-APP SUCCESS DETECTION

D.1 COARSE DETECTION: KEY COMPONENT MATCHING

Given a single screenshot, PaddleOCR4 is used to extract text, which is then lowercased and con-
catenated to minimise inaccuracies. This text is matched against key components of the final state
(defined by human annotators in Section 3.2). Matching starts from the last screenshot and moves

1Auto-UI has been renamed to Auto-GUI, but in this paper, we use Auto-UI as it is more commonly

referenced in previous works.

2https://github.com/openatx/uiautomator2
3https://developer.android.com/studio/run/emulator
4https://github.com/PaddlePaddle/PaddleOCR

24

Under review as a conference paper at ICLR 2025

Table 9: The proportion of reduction in MLLM evaluation times through key component matching,
and the F1 score performance of our MLLM evaluator (without key component matching) across
reasoning and action modes. Bold values indicate the best performance for each task and language
pair.

Task

Language Reduction Rate

No Action

Text Action

Image Action

Result-only

Reason-and-Result

Result-only

Reason-and-Result

Result-only

Reason-and-Result

Single-app

English
Chinese

0.313
0.670

0.911 (-0.003)
0.879 (-0.076)

0.922 (-0.033)
0.857 (-0.102)

0.919 (-0.016)
0.883 (-0.092)

0.903 (-0.040)
0.884 (-0.113)

0.926 (-0.006)
0.872 (-0.093)

0.915 (-0.050)
0.864 (-0.129)

backward until a match is found or the first screenshot is reached. If no match is found, the task is
marked as failed, skipping fine detection.

D.2 FINE DETECTION: MLLM EVALUATION

If coarse detection is successful, fine detection is performed using a MLLM evaluator (based on
GPT-4o). The evaluator receives task descriptions, screenshots, and executed actions to assess task
success. Action information can be presented as either text or concatenated screenshots. Prompts
used for the MLLM evaluator are detailed in Appendix D.4.

D.3 APPROACH EVALUATION AND RESULTS

To validate the single-app success detection pipeline, we compared its detection against human
evaluations for AppAgent and M3A (English tasks), and CogAgent and MobileAgentV2 (Chinese
tasks). Two reasoning and three action modes were tested to prompt the MLLM, and an ablation
study was conducted to assess the impact of coarse detection.

Table 9 presents the proportion of fine detection time reduced before and after applying coarse
detection, along with the F1 scores for each reasoning and action mode across English and Chinese
tasks, both with and without coarse detection. The results demonstrate that coarse detection effectively
enhances performance by reducing the frequency of fine detection calls and improving the success
detection F1 score, particularly in Chinese tasks where MLLM evaluation struggles. While no
significant differences were found between reasoning modes, incorporating action data improved
decision-making but also increased token length, which sometimes led to hallucinations.

Overall, in the best-performing evaluation modes, our pipeline achieved F1 scores of 0.926 for
English tasks and 0.884 for Chinese tasks, demonstrating its effectiveness in aligning with human
evaluations. For further task evaluations, we use these modes to detect success: result-only reasoning
with image action for English tasks, and reason-and-result with text action for Chinese tasks.

D.4 PROMPTING TEMPLATES

D.4.1 SYSTEM PROMPT

Your primary role is

Evaluate solely based on the provided screenshots.

You are an expert in evaluating smartphone operation tasks.
to determine whether a task has been successfully completed based on a series of
screenshots (provided in order of execution) and the corresponding task description.
### Guidelines:
1. **No Assumptions**:
infer or assume details that aren’t explicitly shown.
2. **Subtask Completion**:
successfully completed.
For example, for the task "Go to the website github.com.
this website to the reading list,", it is successful only if the screenshots show
github.com has been navigated to and then added to the reading list.
3. **Common Reasons for Subtask Failure**:
- **Incomplete**:
task example above, visiting the website but not adding it to the reading list results
in task failure.
- **Incorrect Execution**:
part of the instruction.

A subtask is not successful if it is not performed or achieved.

A subtask fails if the screenshots do not align with any

A task is successful only when all its subtasks are

Do not

Add

Same

1296
1297

1298
1299
1300
1301
1302
1303

1304
1305
1306
1307
1308
1309

1310
1311
1312
1313
1314

1315
1316
1317
1318
1319
1320

1321
1322
1323
1324
1325
1326

1327
1328
1329
1330
1331
1332

1333
1334
1335
1336
1337

1338
1339
1340
1341
1342
1343

1344
1345
1346
1347
1348
1349

25

Under review as a conference paper at ICLR 2025

If intermediate screenshots show all subtasks are

Subtasks can be completed in any order unless they are

If the subtask is "Go to the website github.com." but the

If a subtask involves filtering based on specific criteria,

If the subtask is "Like a post," but the screenshots show the

Similar entities (e.g., ’iPhone 11’ vs.
’walking directions’) are considered different,

- **Wrong Noun/Entity**:
screenshots show google.com, the subtask fails.
’iPhone 12’ or ’driving directions’ vs.
leading to task failure if not correctly executed.
- **Wrong Verb/Action**:
post was reposted instead, the subtask fails due to incorrect action.
4. **Additional Actions**:
successful, consider the task a success, even if additional actions are shown afterward.
This applies as long as these actions do not impact task completion or cause the
original task to fail.
5. **Filtering Subtask**:
ensure the filter has been applied (i.e., a specific app feature).
treated as an additional search condition, the subtask fails.
6. **Order of Subtasks**:
explicitly dependent on each other.
7. **Subtasks Completed Midway**:
not be reflected in the final screenshot; these should still be considered successful
if they align with the task requirements.
8. **Corrective Actions**:
by subsequent actions should be considered successful only when the correction fully
aligns with the original task.
9. **Intermediate Steps**:
long as the final result meets the task requirements; consider this a success.
10. **Focus on Overview**:
minor, irrelevant details distract from the main evaluation.
11. **UI Differences**:
styles or colors indicating selected tabs).
action_sys_prompt_template(action_mode)
**These guidelines serve as a general framework.
overfitting to edge cases not covered.
a task has been successfully completed or not.
indicate failure.**

It’s acceptable if a subtask isn’t completed in one go, as

Be mindful of subtle UI differences (e.g., different font

Subtasks that initially appear to fail but are corrected

Pay attention to the overall objective and avoid letting

Subtasks completed in the middle of the process may

Be strict and cautious when determining whether

Use 1 to indicate success and 0 to

Apply them thoughtfully and avoid

If the filter is

D.4.2 SYSTEM PROMPT WITH ACTION

If needed, consider the action information when evaluating

Some quick pop-ups may not be captured by

12. **Use of Action Information**:
screenshots provided.
the task.
13. **Single Action for Multiple Subtasks**:
single action, such as clicking an icon that shuffles a playlist.
### Common Actions:
the screen, triggering an event or interaction.
- Long Press:
- Swipe/Scroll:
the content or screen position changes according to the direction.
- Type/Input Text:
- Back:

The user types or inputs text into a field.
The user presses the back button to return to the previous screen.

- Click/Tap:

The user presses and holds a point to trigger a secondary action or menu.
The user drags their finger across the screen to scroll or navigate;

The user selects or activates a specific point on

Some subtasks can be completed with a

D.4.3 BASE PROMPT

Use 1 to indicate success and 0 to indicate

Now, here is a smartphone operation task description:
**task_description** history_info
Please carefully determine whether the task has been correctly and completely executed
according to the provided screenshots.
failure.
action_prompt[0]
reasoning_prompt
Remember:
- Do not make assumptions based on information not presented in the screenshots. Only
evaluate what is explicitly shown.
- Ensure that every entity and action in the task description is precisely matched and
fulfilled.
- Consider additional actions taken after a task is successfully completed as part
of the success, as long as those actions don’t impact the task’s completion or cause
failure.
- A filtering subtask is only correct when a specific filter is applied as a feature of
the app.
- Subtasks can be completed in any order unless they are explicitly dependent on each
other.
- Subtasks completed correctly mid-process, even if not reflected in the final
screenshot, should be considered successful.

Using the criteria as a keyword search will cause the subtask to fail.

26

1350
1351

1352
1353
1354
1355
1356
1357

1358
1359
1360
1361
1362
1363

1364
1365
1366
1367
1368

1369
1370
1371
1372
1373
1374

1375
1376
1377
1378
1379
1380

1381
1382
1383
1384
1385
1386

1387
1388
1389
1390
1391

1392
1393
1394
1395
1396
1397

1398
1399
1400
1401
1402
1403

Under review as a conference paper at ICLR 2025

- Subtasks that initially appear to fail but are corrected by subsequent actions should
be considered successful.
- A task can be considered successful even if some subtasks are not completed in one go,
as long as the final result meets the task requirements.
- Focus on the overall objective of the task without being distracted by minor,
irrelevant details.
- Pay attention to subtle UI differences that might indicate task completion or failure,
such as highlighted tabs or changes in font.
action_prompt[1]

D.4.4 BASE PROMPT WITH TEXT ACTION

The i-th screenshot may contain details that change the

To assist you in determining whether the task was successful, action information
is provided.
Use this information only when you cannot determine success purely
based on the screenshots.
screenshot from the i-th to the i+1-th, while the last screenshot contains no action
information as the task ends afterward. In some screenshots, a red dot may indicate
where a specific action occurred (e.g., clicked or long-pressed), triggering an event
or interaction.
position operation (e.g., a swipe or text input). You can find the details of these
actions below, if applicable.
extra_action

If there isn’t a red dot, the action is more complex than a single

- Consider the action information only when necessary.
- Pop-ups that appear immediately after an action may not be captured in the
screenshots; do not consider this a failure.
- Some subtasks can be completed with a single action, such as clicking an icon that
shuffles a playlist.

D.4.5 BASE PROMPT WITH IMAGE ACTION

Use this information only when you cannot determine success purely based on

To assist you in determining whether the task was successful, action information is
provided.
the screenshots.
The action information on the i-th screenshot describes the changes
from the i-th screenshot to the i+1-th screenshot, while the last screenshot contains
no action information as the task ends afterward. This information is presented as
a white strip attached to the original screenshot, separated by a blue line.
screenshots, a red dot may indicate where a specific action occurred (e.g., clicked or
long-pressed), triggering an event or interaction.

In some

- Consider the action information only when necessary.
- Pop-ups that appear immediately after an action may not be captured in the
screenshots; do not consider this a failure.
- Some subtasks can be completed with a single action, such as clicking an icon that
shuffles a playlist.

D.4.6 RESULT-ONLY PROMPT

Please provide your decision using the following template without any reasoning:
Result:

<1 OR 0>

D.4.7 REASON-AND-RESULT PROMPT

<Brief description of why you believe the task was successful or failed,

Use the following format for your response:
Reason:
including the alignment or misalignment between the task description and screenshots,
starting with "I believe this task is successful/failed">
Result:

<1 OR 0>

27

1404
1405

1406
1407
1408
1409
1410
1411

1412
1413
1414
1415
1416
1417

1418
1419
1420
1421
1422

1423
1424
1425
1426
1427
1428

1429
1430
1431
1432
1433
1434

1435
1436
1437
1438
1439
1440

1441
1442
1443
1444
1445

1446
1447
1448
1449
1450
1451

1452
1453
1454
1455
1456
1457

Under review as a conference paper at ICLR 2025

1458
1459

1460
1461
1462
1463
1464
1465

1466
1467
1468
1469
1470
1471

1472
1473
1474
1475
1476

1477
1478
1479
1480
1481
1482

1483
1484
1485
1486
1487
1488

1489
1490
1491
1492
1493
1494

1495
1496
1497
1498
1499

1500
1501
1502
1503
1504
1505

1506
1507
1508
1509
1510
1511

Figure 10: Evaluation of the “airbnb_1” task executed by M3A. All four annotated key components
were successfully matched in the OCR-extracted text from the final screenshot, allowing the task to
pass both coarse and fine detection.

D.5 EXAMPLE OF SUCCESS DETECTION

Figure 10 illustrates a coarse-to-fine evaluation of the “airbnb_1” task executed by M3A, which
corresponds to the Airbnb Level 2 task listed in Table 4).

E CROSS-APP SUCCESS DETECTION

E.1 SUBTASK GENERATION

For a cross-app task, each subtask is tied to a single app, and any adjacent subtasks must use different
apps. However, the same app can appear multiple times as long as there is at least one different app
between occurrences. Beyond “app” and “task description”, each subtask also includes the fields
“history” and “memory”. The “history” field is a boolean value indicating whether the subtask requires
information from previous tasks, highlighted as phrases in the task description. This information,
referred to as “memory”, consists of phrases that will be matched with the highlighted “history”
phrases. Such subtasks are generated by a MLLM and then reviewed by humans to ensure quality.
Examples of subtasks are provided below, and detailed prompts can be found in the Appendix E.5.

E.2 STAGE 1: TRAJECTORY SPLIT

Stage 1 splits the entire trajectory into segments based solely on app transitions as preparation for
detecting subtask success. The previous subtask generation step provides an ordered list of apps for
each task, indicating the sequence in which they should be operated for successful completion. A
MLLM processes this app list along with the complete series of execution screenshots, segmenting
the trajectory so that each part includes only screenshots related to the corresponding app’s operations.
If the segmentation is invalid, such as when an app is missing or the sequence is incorrect, the task is
marked as unsuccessful due to errors in one or more apps.

28

FINISHCoarse Detectionwembleystadium1guestMatched {8.png}Fine DetectionResult-onlyImage-actionGPT-4oImage TrajectoryResult: 1Extracted Text        WAIT👆👆👆👆👆👆⌨Under review as a conference paper at ICLR 2025

E.3 STAGE 2: SEQUENTIAL SUBTASK SUCCESS DETECTION

Stage 2 is activated when the segmentation is valid, meaning each app in the ordered list has a unique
series of screenshots. Subtasks are checked sequentially, with each subtask evaluated only if its
predecessor is marked as successful. If a subtask is marked as successful, the phrases in its “memory”
field (unless the field is empty), will be required as historical references for subsequent subtasks.
This memory is generated by another MLLM, which summarises the current screenshots based on
the required phrases and appends the relevant information to the memory set for future use. If a
subsequent subtask’s “history” field is marked as true, the necessary phrases are then extracted and
matched with the stored information to assist in evaluating success. Such historical data, combined
with partial task screenshots and action details, is used to determine the subtask’s success. Since each
subtask involves only a single app, it uses the same MLLM evaluation method applied in single-app
success detection. The entire task is considered successful only if all subtasks pass. Otherwise, it
fails as soon as any subtask is marked unsuccessful.

E.4 APPROACH EVALUATION AND RESULTS

Table 10: The F1 score perfor-
mance of our cross-app success
detection pipeline.

To validate the cross-app success detection pipeline, we com-
pared its results against human evaluations using four different
agents per language. For English tasks, the agents were M3A,
T3A, Auto-UI, and OdysseyAgent, while for Chinese tasks, we
used AppAgent, MobileAgent, MobileAgentV2, and CogAgent.
Table 10 presents the F1 scores of our cross-app success detection
pipeline for both English and Chinese tasks. The performance
is lower compared to single-app success detection due to the in-
creased complexity of cross-app tasks. With over 90% of tasks
being true negatives, even a small number of errors significantly
impacts the overall performance. Additionally, we observed that for each agent, false positives and
false negatives occurred at a similar rate. Thus, despite a relatively modest F1 score, the pipeline’s
success detection still reflects each agent’s performance.

Cross-app

F1 Score

Chinese

English

0.833

0.857

E.5 PROMPTING TEMPLATES

E.5.1 SYSTEM PROMPT OF STAGE 1

Each
one for opening the app and one

For each app, identify where the agent opens and operates within the app.

Do not change the order of apps, even if they do not match the screenshot order.

Split the screenshots into segments based on transitions between apps in the given

You are provided with a sequence of screenshots representing an agent performing tasks
across multiple apps on a smartphone. Each screenshot corresponds to a specific action.
You are also given a list of apps that should be used in the task.
**Your task is to:**
1.
list.
Output the results based on the provided app list order.
2.
app interaction requires at least two screenshots:
for quitting or switching to another, except for the final app, which may not require a
quit action.
3. **Ensure that the start and end indices you provide are within the range of
screenshots sent to you.** You will receive a certain number of screenshots, and you
must repeat how many screenshots you received before processing.
should not exceed the total number of screenshots.
4.
start and end screenshot indices for that app.
5.
apps).
result.
6.
but there must be another app between repeated instances of the same app.
7.
you should not interpret them as transitions between apps.
### Example Input:
**App list:** ‘["AppA", "AppB", "AppA"]‘
**Screenshots:** A sequence of numbered screenshots.
### Example Reasoning:
1.
within it.
**Screenshot 6:** The agent interacts with the home screen, which is irrelevant. 4.
**Screenshots 7-9:** The agent opens AppA again and operates within it.

Ignore screenshots that show irrelevant actions (e.g., the home screen or unrelated

**Screenshots 4-5:** The agent opens AppB and operates within it. 3.

An app may appear more than once in the list (e.g., ‘["AppA", "AppB", "AppA"]‘),

There might be distractors (e.g., advertisements and popups) in the screenshots;

If an app from the list is missing in the screenshots, return ‘-1‘ for both the

You may mention them in the analysis but do not include them in the final

**Screenshots 1-3:** The agent opens AppA, and operates

Any indices provided

2.

29

1512
1513

1514
1515
1516
1517
1518
1519

1520
1521
1522
1523
1524
1525

1526
1527
1528
1529
1530

1531
1532
1533
1534
1535
1536

1537
1538
1539
1540
1541
1542

1543
1544
1545
1546
1547
1548

1549
1550
1551
1552
1553

1554
1555
1556
1557
1558
1559

1560
1561
1562
1563
1564
1565

Under review as a conference paper at ICLR 2025

1566
1567

1568
1569
1570
1571
1572
1573

1574
1575
1576
1577
1578
1579

1580
1581
1582
1583
1584

1585
1586
1587
1588
1589
1590

1591
1592
1593
1594
1595
1596

1597
1598
1599
1600
1601
1602

1603
1604
1605
1606
1607

1608
1609
1610
1611
1612
1613

1614
1615
1616
1617
1618
1619

### Final Output:
"start screen":
9 } }
**task_description**

{ "AppA_1":

{ "start screen": 1, "end screen":

3 }, "AppB":

{

4, "end screen": 5 }, "AppA_2": { "start screen":

7, "end screen":

E.5.2 USER PROMPT OF STAGE 1

Here is the app list:
exactly the same as the order provided in my app list.

task_app Ensure the order of apps in your final output is

E.5.3 SYSTEM PROMPT OF STAGE 2 MEMORY

You are an MLLM tasked with analyzing screenshots and summarizing the relevant
information based on a description provided by the user.
from screenshots that relate to the description, ignoring any that are unrelated.
the screenshots show a list of results (e.g., a search page), summarize or list all
the relevant results.
step-by-step details, or line breaks.

The summary should be clear and concise, without bullet points,

Only summarize information

If

E.5.4 USER PROMPT OF STAGE 2 MEMORY

Here is the description:

memory_text

E.5.5 SUBTASK GENERATION

Do not include the

For each subtask, you

The name of the app being used in the subtask.

A string describing the action to be performed.

This should be exactly the same phrase as the previous subtask’s memory (i.e.,

A boolean value (‘True‘ or ‘False‘) indicating whether this subtask

If applicable, specify a piece of information that the current subtask

Use ’{PREVIOUS MEMORY}’ if the task depends on information from a previous

You are tasked with splitting a smartphone control instruction into a series of
subtasks, each corresponding to specific app interactions.
should define:
1. **app**:
2. **task**:
app name in the task description unless necessary (e.g., if the task is to only open
the app).
subtask.
if history is True).
3. **history**:
relies on data from a previous subtask.
4. **memory**:
generates or retrieves, which will be passed to the next subtask.
needed, set this to ‘None‘.
**Guidelines**:
- Use the same language for the split task as the task description.
- If there are several consecutive subtasks for the same app, combine them into a single
subtask (i.e., adjacent subtasks should not have the same app).
app are acceptable if there is at least one subtask for a different app in between.
- By default, each subtask should be independent unless explicitly needing data from a
prior subtask (in which case, set ‘"history": True‘).
- Flexibly determine whether any information should be stored as **memory** and passed
to subsequent tasks, based on the task’s natural requirements.
- Output the subtasks in a structured format like the following:
{ "subtask_1":{ "app":"APP", "task":"TASK", "history":"BOOL", "memory":"MEMORY" },
"subtask_2":{ "app":"APP", "task":"TASK", "history":"BOOL", "memory":"MEMORY" }, ...
###Example 1
**Task**:
Settings, then proceed to open YouTube.
**Result**:
{ "subtask_1":{ "app":"Settings", "task":"Adjust the notification settings for
the YouTube app on your phone", "history":false, "memory":"None" }, "subtask_2":{
"app":"YouTube", "task":"Open YouTube", "history":false, "memory":"None" } }
### Example 2
**Task**:
vacuum cleaner, and then go to Amazon to purchase one.

Adjust the notification settings for the YouTube app on your phone using

Utilize the X app to research and identify a highly recommended robotic

Subtasks for the same

If no memory is

}

30

Under review as a conference paper at ICLR 2025

1620
1621

1622
1623
1624
1625
1626
1627

1628
1629
1630
1631
1632
1633

1634
1635
1636
1637
1638

1639
1640
1641
1642
1643
1644

1645
1646
1647
1648
1649
1650

1651
1652
1653
1654
1655
1656

1657
1658
1659
1660
1661

1662
1663
1664
1665
1666
1667

1668
1669
1670
1671
1672
1673

Table 11: Task performance on single-app Chinese tasks. SRC and MSR refer to Self-Reported
Completion and Maximum Steps Reached, respectively. The token costs of four agents are omitted
because they use locally hosted open-source models.

Agent

Success
Rate

Mean Step Ratio
on Success

Termination Reason

Termination Inaccuracy

SRC
Rate

MSR
Rate

Error
Rate

Premature
Rate

Overdue
Rate

Mean Exec Time
per Step (sec)

Mean Token Cost
per Step (USD)

AppAgent
AutoDroid
MobileAgent
MobileAgentV2
M3A
T3A
SeeAct

Auto-UI
CogAgent
DigiRL
OdysseyAgent

0.247
0.187
0.240
0.440
0.447
0.380
0.327

0.007
0.027
0
0.007

1.66
1.25
1.39
1.28
1.08
1.31
1.91

0.50
1.79
-
2.00

Agentic Workflow (GPT-4o)

0.100
0.567
0.273
0.460
0.640
0.507
0.067

0.893
0.060
0.387
0

0.393
0.360
0.653
0.487
0.360
0.493
0.927

0.507
0.073
0.074
0.053
0
0
0.006

0.600
0.729
0.439
0.333
0.323
0.408
0.300

Agent-as-a-Model

0.107
0.893
0.520
1.000

0
0.047
0.093
0

0.993
1.000
1.000
-

0.407
0.111
0.133
0.274
0.037
0.162
0.302

0
0.030
0
0.007

25.6
48.8
35.6
104.5
20.8
12.6
23.0

-
-
-
-

0.013
0.011
0.037
0.075
0.097
0.128
0.050

-
-
-
-

**Result**:
{ "subtask_1":{ "app":"X", "task":"Research and identify a highly recommended
robotic vacuum cleaner", "history":false, "memory":"robotic vacuum cleaner" },
"subtask_2":{ "app":"Amazon", "task":"Go to Amazon to purchase {robotic vacuum
cleaner}", "history":true, "memory":"None" } }
Now, for any smartphone control instruction, decompose the task into subtasks using the
format above.

F EXPERIMENT DETAILS

F.1 AGENT CONFIGURATION

The agents in this benchmark include variations in core models and optional modules. Of the 11
agents, 7 originally used off-the-shelf (M)LLMs such as GPT-4V and Qwen-VL-Max. For consistency,
these agents were upgraded to GPT-4o, including replacing MobileAgentV2’s Qwen-VL-Chat with
GPT-4o-mini for icon recognition. For Auto-UI and DigiRL (fine-tuned), the Auto-UI-Base core
model was selected.

Agent-specific configurations include:

• AppAgent, SeeAct, M3A, and T3A: Added AdbKeyboard5 for Chinese character input,

following the MobileAgent setup.

• Auto-UI: Enabled “action history” and “chain of actions” features.

• OdysseyAgent: Enabled action and screenshot history.

• AppAgent and AutoDroid: No additional knowledge or exploration was allowed before

experiments.

For all other settings, the default configurations provided by the developers were used. Agents were
allowed to execute up to twice the number of “golden steps” for a task, after which execution was
halted.

F.2 EXPERIMENTAL RESULTS

See Tables 11, 12, 13 for the detailed experiment results of single-app Chinese, cross-app English,
and cross-app Chinese tasks respectively.

5https://github.com/senzhk/ADBKeyBoard

31

Under review as a conference paper at ICLR 2025

1674
1675

1676
1677
1678
1679
1680
1681

1682
1683
1684
1685
1686
1687

1688
1689
1690
1691
1692

1693
1694
1695
1696
1697
1698

1699
1700
1701
1702
1703
1704

1705
1706
1707
1708
1709
1710

1711
1712
1713
1714
1715

1716
1717
1718
1719
1720
1721

1722
1723
1724
1725
1726
1727

Table 12: Task performance on cross-app English tasks. SRC and MSR refer to Self-Reported
Completion and Maximum Steps Reached, respectively. The token costs of four agents are omitted
because they use locally hosted open-source models.

Agent

Success
Rate

Mean Step Ratio
on Success

Termination Reason

Termination Inaccuracy

SRC
Rate

MSR
Rate

Error
Rate

Premature
Rate

Overdue
Rate

Mean Exec Time
per Step (sec)

Mean Token Cost
per Step (USD)

AppAgent
MobileAgent
MobileAgentV2
M3A
T3A
SeeAct

0
0.050
0.100
0.200
0.100
0.100

Auto-UI
CogAgent
DigiRL
OdysseyAgent

0
0
0
0

-
2.00
2.00
1.16
1.43
1.52

-
-
-
-

Agentic Workflow (GPT-4o)

0.200
0.100
0.250
0.700
0.600
0.150

0.100
0.050
0.050
0

0.550
0.900
0.750
0.300
0.400
0.850

0.250
0
0
0
0
0

1.000
1.000
1.000
0.714
0.833
0.333

Agent-as-a-Model

0.800
0.950
0.550
0.650

0.100
0
0.400
0.350

1.000
1.000
1.000
-

0
0.056
0.133
0
0
0

0
0
0
0.007

22.9
25.3
58.8
17.3
12.1
19.9

-
-
-
-

0.014
0.089
0.071
0.082
0.091
0.043

-
-
-
-

Table 13: Task performance on cross-app Chinese tasks. SRC and MSR refer to Self-Reported
Completion and Maximum Steps Reached, respectively. The token costs of four agents are omitted
because they use locally hosted open-source models.

Agent

Success
Rate

Mean Step Ratio
on Success

Termination Reason

Termination Inaccuracy

SRC
Rate

MSR
Rate

Error
Rate

Premature
Rate

Overdue
Rate

Mean Exec Time
per Step (sec)

Mean Token Cost
per Step (USD)

AppAgent
MobileAgent
MobileAgentV2
M3A
T3A
SeeAct

0
0.100
0.100
0.100
0.100
0.050

AutoUI
CogAgent
DigirlAgent
GUI_Odyssey

0
0
0
0

-
1.62
1.89
1.32
1.08
2.00

-
-
-
-

Agentic Workflow (GPT-4o)

0
0.150
0.200
0.500
0.750
0.100

1.00
0.050
0.800
0

0.550
0.750
0.750
0.500
0.250
0.900

0.450
0.100
0.050
0
0
0

-
0.667
1.000
0.800
0.867
1.000

Agent-as-a-Model

0
0.850
0.050
0.500

0
0.100
0.150
0.500

1.000
1.000
1.000
-

0
0.067
0.133
0
0
0.056

0
0
0
0

23.5
53.4
104.1
17.8
13.4
17.3

-
-
-
-

0.014
0.064
0.075
0.091
0.110
0.045

-
-
-
-

F.3 PERFORMANCE ACROSS TASK DIFFICULTY LEVELS

Table 14 shows agent performance across different difficulty levels. As expected, agents perform
better on easier tasks, confirming that our tasks are designed with increasing difficulty, where lower-
level tasks serve as subtasks for higher-level ones. The overall trend in performance across difficulty
levels aligns with each agent’s general success rate discussed in Section 6.1.

G EXPERIMENTS ON OPEN-ENDED SINGLE-APP ENGLISH TASKS

To further explore the scalability of our success detection approaches, we designed an initial set of
ten “open-ended” single-app English tasks across distinct apps, as detailed in Table 15.

Table 15: Open-ended single-app English tasks.

App
Airbnb
Amazon
Calculator
Chrome
Clock
Merriam-Webster
Google Maps
Settings
Spotify
YouTube

Task Description
I’m traveling to London with three friends and need accommodation. I’ll manage the checkout process myself.
I’d like to buy wedding gifts for my friend and their partner. I’ll take care of the checkout myself.
I want to show my friend the multiplication of two negative numbers is indeed a positive number.
I’m planning a trip to ski and would like to save a blog to read later.
Please set two alarms, one for weekdays and another for weekends. I prefer waking up later on weekends.
I’d like to expand my vocabulary in political ideologies. I aim to learn two new terms today.
My car is low on gas, and I’m also feeling hungry.
Sometimes I have trouble reading the screen clearly.
Create a music playlist for me in a recommended genre. Just two songs will do.
I’m interested in watching tech tutorial videos recently.

32

Under review as a conference paper at ICLR 2025

Table 14: Success rates on single-app English, single-app Chinese, cross-app English and cross-app
Chinese tasks, categorised by difficulty level. AutoDroid was tested only on single-app tasks as its
agent framework, Droidbot Li et al. (2017), supports only these tasks.

Agent

AppAgent
AutoDroid
MobileAgent
MobileAgentV2
M3A
T3A
SeeAct

Auto-UI
CogAgent
DigiRL
OdysseyAgent

Single-app English Tasks

Single-app Chinese Tasks

Cross-app English Tasks

Cross-app Chinese Tasks

Level 1

Level 2

Level 3

Level 1

Level 2

Level 3

Level 1

Level 2

Level 1

Level 2

0.540
0.560
0.620
0.700
0.800
0.720
0.600

0.040
0.060
0.020
0.140

0.340
0.300
0.380
0.400
0.700
0.480
0.460

0
0
0.040
0.020

0.140
0.120
0.160
0.200
0.420
0.260
0.120

0
0
0
0

Agentic Workflow (GPT-4o)

0.400
0.360
0.300
0.580
0.500
0.480
0.500

0.020
0.040
0
0.004

0.180
0.120
0.240
0.420
0.520
0.460
0.340

0.160
0.080
0.180
0.320
0.320
0.200
0.140

Agent-as-a-Model

0
0.040
0
0.020

0
0
0
0

0
-
0.067
0.133
0.267
0.133
0.133

0
0
0
0

0
-
0
0
0
0
0

0
0
0
0

0
-
0.067
0.133
0.133
0.133
0.067

0
0
0
0

0
-
0.200
0
0
0
0

0
0
0
0

As discussed in Section 3.2, when a task description is clearly defined with a specific goal, its
executions typically converge to the same final state. Such tasks can be treated as “closed-ended”
tasks, which form the basis for human-annotated key components. In contrast, for a more vague
task description, the task is considered “open-ended”. The final state may result in multiple possible
outcomes, making it challenging to define key components explicitly. While the coarse detection
phase may be limited in such cases, we hypothesised that our fine detection approach, relying on the
MLLM evaluator, remains effective and can still be applied to “open-ended” tasks.

In this initial experiment, we tested the seven agents that follow
the agentic workflow on the ten “open-ended” tasks. Given the
open-ended nature of these tasks and the absence of predefined
golden steps, agents were allowed a maximum of 20 steps to
complete each task. We compared the alignment of success detec-
tion results between human evaluations and our MLLM evaluator.
Using the same MLLM evaluator introduced in Section 5.2, we
identified 22 true positives, 2 false positives, 2 false negatives,
and 44 true negatives. This resulted in an F1 score of 0.917, con-
sistent with the corresponding results for “closed-ended” tasks
reported in Table 9. These findings demonstrate the potential of
applying our MLLM evaluator to a broader range of tasks, both
“open-ended” and “close-ended”, highlighting its scalability to
tasks beyond our benchmark.

Table 16: Success rates on open-
ended single-app English tasks.

Agent

Success
Rate

Agentic Workflow (GPT-4o)

AppAgent
AutoDroid
MobileAgent
MobileAgentV2
M3A
T3A
SeeAct

0.200
0.300
0.200
0.200
0.700
0.400
0.400

Table 16 presents the success rates of the seven agents on these ten tasks. M3A consistently outper-
formed the other agents. However, compared to the success rates reported in Table 3, MobileAgentV2
exhibited the largest performance gap, suggesting its limitations in handling “open-ended” tasks.

In future work, we aim to expand this initial experiment with a more comprehensive task collection
to further improve and assess the feasibility of our MLLM evaluator for a wider range of tasks, and
to investigate agent performance on “open-ended” tasks.

1728
1729

1730
1731
1732
1733
1734
1735

1736
1737
1738
1739
1740
1741

1742
1743
1744
1745
1746

1747
1748
1749
1750
1751
1752

1753
1754
1755
1756
1757
1758

1759
1760
1761
1762
1763
1764

1765
1766
1767
1768
1769

1770
1771
1772
1773
1774
1775

1776
1777
1778
1779
1780
1781

33

Under review as a conference paper at ICLR 2025

H CASE STUDY

Three case studies are presented to illustrate representative scenarios of task execution by agents.
These include: (1) an invalid action taken by AppAgent due to misinterpretation of the UI structure
in the XML file, (2) a dynamically changing screen without any action execution, repetitive actions
due to the lack of reflection, and unrelated behaviours to the task description in MobileAgent, and (3)
the combined actions employed by M3A.

H.1 APPAGENT ON CONTACT_2 TASK

(a) Annotated screenshot

(b) Parsed XML file

Figure 11: The screenshot and XML file before the last action for AppAgent executing task contact_2.
The model generated invalid action tap(2). Task description: “Modify the last name of one of the
contacts to ‘Three’. Update the label for the contact’s phone number to Work. Set the company to
‘Huawei’. Add an email agent.benchmark.2024@gmail.com. Label the email as Work”.

As shown in Figure 11, in the final step of task contact_2, AppAgent encountered a critical error
due to a misinterpretation of the UI structure. The model incorrectly parsed the XML, treating the
entire pop-up menu as a single element instead of recognizing each individual operable component,
which reduced the number of widgets the agent could interact with. In addition, the agent executed
an invalid action, tap(2), targeting a non-clickable element. This issue highlights that an imperfect
operable action detection mechanism may limit the agent’s ability to navigate complex UI hierarchies
and execute fine-grained interactions.

34

1782
1783

1784
1785
1786
1787
1788
1789

1790
1791
1792
1793
1794
1795

1796
1797
1798
1799
1800

1801
1802
1803
1804
1805
1806

1807
1808
1809
1810
1811
1812

1813
1814
1815
1816
1817
1818

1819
1820
1821
1822
1823

1824
1825
1826
1827
1828
1829

1830
1831
1832
1833
1834
1835

Under review as a conference paper at ICLR 2025

H.2 MOBILEAGENT ON EXPEDIA_3 TASK

As shown in Figure 12 and Figure 13, MobileAgent’s execution of task expedia_3 reveals several
noteworthy points: (1) Although the transition between the second and third screenshots (highlighted
with a red border) lacks valid actions, the interface still changes, indicating that content is loading
during a waiting period (i.e., a dynamically changing screen). (2) The agent generates repetitive
actions despite no changes in the interface, but after several iterations, a correction occurs (highlighted
with a blue border). (3) Interestingly, at the beginning of task execution, the agent initially attempted
to chat with ChatGPT, which was unrelated to the task description. By the time the agent attempted to
execute something relevant, several steps had already been wasted, leaving insufficient opportunities
to complete the task properly.

Figure 12: Trajectory of MobileAgent on expedia_3 (Part 1). Task description: “Check things to do
in Paris. Get the search results for 25th to 28th of any month.”

1836
1837

1838
1839
1840
1841
1842
1843

1844
1845
1846
1847
1848
1849

1850
1851
1852
1853
1854

1855
1856
1857
1858
1859
1860

1861
1862
1863
1864
1865
1866

1867
1868
1869
1870
1871
1872

1873
1874
1875
1876
1877

1878
1879
1880
1881
1882
1883

1884
1885
1886
1887
1888
1889

35

Under review as a conference paper at ICLR 2025

1890
1891

1892
1893
1894
1895
1896
1897

1898
1899
1900
1901
1902
1903

1904
1905
1906
1907
1908

1909
1910
1911
1912
1913
1914

1915
1916
1917
1918
1919
1920

1921
1922
1923
1924
1925
1926

1927
1928
1929
1930
1931

1932
1933
1934
1935
1936
1937

1938
1939
1940
1941
1942
1943

Figure 13: Trajectory of MobileAgent on expedia_3 (Part 2). Task description: “Check things to do
in Paris. Get the search results for 25th to 28th of any month.”

36

Under review as a conference paper at ICLR 2025

H.3 M3A VS HUMAN ON GOOGLE_TASKS_0 TASK

(a) Trajectory of M3A on google_tasks_0

(b) Trajectory of Human on google_tasks_0

Figure 14: Trajectory of M3A vs human on google_tasks_0. Task description: “Create a new list
‘Work’.”

By comparing Figure 14a and Figure 14b, it is evident that M3A employed a combined action strategy,
encapsulating text input and pressing the “enter” key within a single-step operation. This approach
led to a more concise execution, requiring one fewer step compared to the human trajectory.

37

1944
1945

1946
1947
1948
1949
1950
1951

1952
1953
1954
1955
1956
1957

1958
1959
1960
1961
1962

1963
1964
1965
1966
1967
1968

1969
1970
1971
1972
1973
1974

1975
1976
1977
1978
1979
1980

1981
1982
1983
1984
1985

1986
1987
1988
1989
1990
1991

1992
1993
1994
1995
1996
1997

