# Agent-E: From Autonomous Web Navigation to Foundational Design Principles in
Agentic Systems

Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan,
Aditya Vempaty, Ravi Kokku

({tea, deepak, prasenjit, ashish, aditya<EMAIL_ADDRESS>
Emergence AI
July, 2024)

###### Abstract

AI Agents are changing the way work gets done, both in consumer and enterprise
domains. However, the design patterns and architectures to build highly
capable agents or multi-agent systems are still developing, and the
understanding of the implication of various design choices and algorithms is
still evolving. In this paper, we present our work on building a novel web
agent, Agent-E 111Our code is available at
https://github.com/EmergenceAI/Agent-E. Agent-E introduces numerous
architectural improvements over prior state-of-the-art web agents such as
hierarchical architecture, flexible DOM distillation and denoising method, and
the concept of change observation to guide the agent towards more accurate
performance. We first present the results of an evaluation of Agent-E on
WebVoyager benchmark dataset and show that Agent-E beats other SOTA text and
multi-modal web agents on this benchmark in most categories by 10-30%. We then
synthesize our learnings from the development of Agent-E into general design
principles for developing agentic systems. These include the use of domain-
specific primitive skills, the importance of distillation and de-noising of
environmental observations, the advantages of a hierarchical architecture, and
the role of agentic self-improvement to enhance agent efficiency and efficacy
as the agent gathers experience.

##  1 Introduction

Recent advancements in large language models have led to significant interest
in the development of autonomous agents that can execute complex tasks on the
web [Zheng et al., 2024, He et al., 2024, Lutz et al., 2024] and on device
[Bai et al., 2024, Wen et al., 2024]. Automation of complex and repetitive
tasks presents an invaluable opportunity to increase individual and
organizational efficiency. In addition, robust automation systems will unlock
new use cases and experiences by enabling collaboration between users and AI
agents working together to accomplish sophisticated tasks.

Figure 1 shows a simplified anatomy of a web agent. A typical web-agent, at
its most basic, has two key capabilities: the capability to sense the web page
and the capability to act on the web page.

  * •

Sensing: Sensing the state of the web-page, for a web agent, typically
involves encoding the Document Object Model (DOM) of the page [Nakano et al.,
2022, Lutz et al., 2024], using the accessibility tree of the page [He et al.,
2024] and/or using screenshots of the page [He et al., 2024].

  * •

Acting: The action space can be comprised of simple actions such as navigating
to URLs, clicking on elements, and entering text in a field, or composite
actions comprising of several simple actions. An example of composite action
is used by Lutz et al. [Lutz et al., 2024], which they call ‘Input’. ‘Input’
selects a text box, deletes any existing content, inputs text and presses
submit button.

Based on the user’s task and the current state of the page, the agent can
create a plan i.e., it determines the subsequent next action or series of
actions required to complete the task.

Figure 1: Simplified anatomy of web agents

Developing a robust web agent that can autonomously perform tasks on the
browser presents multiple challenges. Firstly, HyperText Markup Language
(HTML) Document Object Models (DOMs) can be noisy and expansive, so that they
often exceed an LLM’s context window. This renders them unusable without
intelligent simplification or de-noising steps. Further, even with
simplification, information may exceed the context window limit of modern LLMs
after a few interactions. This is an important challenge that restricts their
use for complex tasks [He et al., 2024]. Secondly, websites are primarily
designed for human visual consumption, wherein information is organized to
prevent information overload for a user; thus, websites utilize established
human-computer interaction patterns that may not be optimal for agent
interaction. Complex widgets, such as date selectors, are easy for humans to
operate but pose difficulties for agents [Lutz et al., 2024]. Thirdly, while
humans can naturally execute complex web tasks (e.g., finding the cheapest
flight between two destinations), agents require detailed multi-step planning
to execute such tasks. Further, the plan may need multiple revisions if the
initial approach proves unfruitful.

Despite these challenges, recent work has demonstrated promising outcomes with
web agents executing diverse tasks on the internet. Yet, state-of-the-art web
agents leave much to be desired in terms of practical usefulness [Wornow et
al., 2024]. Most previous studies simply report success rates. Task success
rates show that web agents are more error-prone compared to a human performing
the same task and are not yet ready for effective mainstream use [Lutz et al.,
2024, He et al., 2024, Zhou et al., 2023].

Task success rates, while important, may not tell the whole story when it
comes to evaluating web-agents. For example Kapoor et al. [Kapoor et al.,
2024] note that since that the underlying LLMs are stochastic systems, simple
retry mechanisms often improve success rates. Thus, simply measuring task
success rates is misleading. There are other important factors that influence
the utility and user experience of using agentic systems, such as:

  * •

Task completion times: The duration required for the system to complete a
task, which impacts overall efficiency and productivity. Task completion times
are especially relevant for online tasks or tasks with a human in the loop.
The task completion times can be influenced by extrinsic factors (network
speed, computing environment influencing loading times of web pages) or
intrinsic factors (token generation speed of the LLMs, the path taken by the
agent to perform the task, errors made during execution etc). Despite the
variability, task completion times is an important measure since it can be
critical in many real world applications.

  * •

Cost: The total expenditure involved in executing the task. The cost is a
variable measure and depends on the underlying model and the per token
input/output cost of different model providers. This is also a ‘point in time’
measure, since the pricing models set by LLM providers change frequently.
While not fully encompassing, a good proxy for cost is number of LLM calls
involved.

  * •

Error-awareness: The capability of the system to recognize and report its own
errors or failures, ensuring reliability and ability to provide a seamless
user experience by highlighting and recommending fallback options. Ability of
system to detect that it has not completed the task appropriately will also
open up avenues to collect human demonstration (e.g. I could not complete this
task, can you demonstrate how you would do it?) and a path towards continuous
improvement.

In this paper, we introduce Agent-E, a state-of-the-art web agent capable of
performing complex web-based tasks. Central to Agent-E are two LLM-powered
components: the planner agent and the browser navigation agent. The planner
agent is responsible for task planning and task management. It breaks down the
user task into a sequence of sub tasks and delegates them one at a time to the
browser navigation agent. The browser navigation agent is tasked with
executing the individual sub tasks by sensing the page using different DOM
distillation capabilities available to it, finding the next actions to execute
and reporting its task success or failure back to the planner. This loop is
executed iteratively until the task is completed. This tiered architecture
ensures that the planner agent is insulated from the overwhelming and noisy
details of the website and DOM, and the browser navigation agent is freed from
the complexities of the overall task planning and orchestration.

Agent-E can run in two modes: autonomous mode and human-in-the-loop mode. In
the autonomous mode, Agent-E performs the task end to end as best as it can,
and in the human-in-the-loop mode, it falls back to the user when it
encounters steps that it cannot accomplish (e.g. logging to a web page, solve
a captcha triggered by the website) or to ask for clarifications when the task
itself is ambiguous. We believe human-in-the-loop workflows are critical for
adoption of agentic systems. However, since there are no web automation
benchmarks for human-in-the-loop agents [Kapoor et al., 2024], in this paper
we will focus on Agent-E as an autonomous web agent.

We evaluate Agent-E in autonomous mode on the WebVoyager benchmark [He et al.,
2024], where we achieve a new state-of-the-art result of 73.2%, 20% higher
than previous state-of-the-art for text-only [Lutz et al., 2024] and 16%
higher than previous state-of-the-art multi-modal web agent [He et al., 2024].
For specific sites like Wolfram Alpha, it achieves upto 30% improvement.

###  1.1 Contributions

  * •

We introduce a novel hierarchical architecture for web agents that enables the
execution of more complex tasks through a clear separation of roles and
responsibilities between a planner agent and a browser navigation agent. We
illustrate the utility of the hierarchical architecture for capabilities such
as task verification, backtracking, and error recovery, with examples where
these capabilities are invoked.

  * •

We propose a flexible DOM distillation approach whereby, given a task, the
browser navigation agent can choose the most suitable DOM representation from
three different implementations. We demonstrate its effectiveness with
examples.

  * •

We highlight the benefits of ‘Change observation’, a paradigm conceptually
similar to Reflexion [Shinn et al., 2024]. After the execution of each action,
the outcome (change in state) is monitored and used to provide verbal feedback
to the browser navigation agent. This guides the agent towards better
awareness of the current state and more accurate performance.

  * •

We report detailed end-to-end evaluations of Agent-E on the WebVoyager
benchmark and show that it achieves a new state-of-the-art results with a
73.2% success rate. This marks a 20% improvement over the previous text-only
web agent and a 16% increase over the former state-of-the-art multi-modal web
agent.

  * •

Beyond task success rates, we are the first to report additional metrics such
as error awareness, task completion times, and the number of LLM calls,
offering a baseline for comprehensive evaluation of future web agents.

  * •

Agent-E is a result of numerous experimentation and exploration into the space
of AI Agents. We synthesize our learning from the development of Agent-E into
design principles for development of agentic systems that we believe
generalize beyond Agent-E and web automation, and provide useful insights to
practitioners.

##  2 Agent-E: System Description

Figure 2 shows the high level architecture of Agent-E. It comprises of two
LLM-powered agents: Planner Agent and Browser Navigation Agent, and two
executor components: Planner Skills executor and Browser Navigation Skills
executor. Each LLM-powered agent has skills associated with it, which are
python functions that are described to the LLM for function calling. We do not
make any distinction between skills associated with sensing (e.g. _Get DOM_)
and skills associated with acting on the page (e.g. _click on element_). The
executor components execute the function suggested by the LLM and relays the
response back to the LLM.

Agent-E is built using Autogen, the open-source programming framework for
building multi-agent collaborative systems [Wu et al., 2023b] and
Playwright222https://playwright.dev/ for browser control. Its architecture
leverages the interplay between skills and agents as shown in Figure 2.

Figure 3 shows a conceptual flow diagram of Agent-E. Given a new user task,
the planner decomposes the task into a sequence of steps that need to be
executed. The next step is then delegated to the browser navigation agent for
execution. Browser navigation agent is implemented as a nested chat that is
freshly instantiated by the Planner for each run (i.e., it does not contain
previous steps in its chat history). Browser navigation agent has a set of
predefined ‘primitive’ or foundational skills for observing denoised browser
state and controlling the browser instance using Playwright. Figure 4 shows
the skills available to the browser navigation agent. The browser navigation
agent uses the skills available to it to perform the sub task and return a
summary of actions it took to perform the task and/or answer the planner if
the task was a question. Figure 5 shows an example communication between
planner and the browser navigation agent and Figure 6 shows the browser
navigation agent using the primitive skills available to it to perform a
related sub task.

Figure 2: A high level architecture of Agent-E Figure 3: Conceptual flow
diagram of Agent-E. Individual blocks represent functions; In Agent-E, a
single LLM call is used to perform multiple functions. Figure 4: Skills
registered to the Browser Navigation Agent for sensing and acting on the web
page. Figure 5: An example of Agent-E execution highlighting communication
between the planner and browser navigation agent for the user task Find the
price of Canva Teams subscription and minimum number of users required for it

###  2.1 Skills Design for Browser Navigation Agent

There are two key differences between Agent-E and previous web agent
implementations such as Wilbur [Lutz et al., 2024] and WebVoyager [He et al.,
2024] in terms of skills .

  * •

Sensing Skills: Agent-E supports multiple DOM synthesis techniques that allows
the browser navigation agent to choose the approach best suited for the task
(see Figure 4). If the task is to summarise information on a page, it can
simply use Get DOM with text_only content type. If the task is to identify and
execute a search on a page, it can use the content type input_fields. If the
task is to list all the interactive elements on a page, it can use all_fields.
This optimizes the information available to the agent and prevents the
problems associated with noisy DOM. Another key difference is our DOM de-
noising techniques for _all_fields_ and _input_fields_ attempts to preserve
the parent child relationship of elements wherever possible and relevant. This
is unlike some of the other previous implementations which uses a flat DOM
encoding (e.g. [Lutz et al., 2024]). Further, to make identifying and
interacting with HTML elements easier, Agent-E injects a custom identifier
attribute (mmid) to each element as part of sensing, similar to [Zhou et al.,
2023] and [He et al., 2024].

  * •

Action Skills: All the action skills are designed to not only execute an
action but also report on any change in state as an outcome of the action, a
concept we call ‘Change observation’. This is conceptually similar to
Reflexion paradigm [Shinn et al., 2024] which uses verbal reinforcement to
help agents learn from prior failings. However, a key difference is that
change observation is not directly associated or limited to a prior failure.
The observation returned can be any type of outcome of the action. For
example, a click action may return a response Clicked the element with mmid
25. As a consequence, a popup has appeared with following elements. Such
verbal responses nudges the agent towards the correct next step.

Change Observation capability is implemented by observing changes in selected
attributes of elements (e.g aria-expanded) and Mutation Observer Web API
333https://developer.mozilla.org/en-US/docs/Web/API/MutationObserver that
allow observing changes in DOM immediately following an action execution. This
is especially useful for extremely dynamic pages such as a flight booking
website and nudges the LLM to take appropriate next step given the task. (see
Figure 6 for an example)

Figure 6: An example of Agent-E nested chat execution loop for the sub task
“Navigate to the plans and pricing section” which is part of the larger task
introduced earlier “Find the price of Canva Teams subscription and minimum
number of users required for it”

##  3 Evaluation

###  3.1 Introduction to WebVoyager

WebVoyager [He et al., 2024] is a recent web agent benchmark that consists of
web navigation tasks across 15 real websites as shown Figure 7. Each website
has about 40-46 tasks resulting in a benchmark dataset of 643 tasks (see Table
1 for example tasks). These tasks could be completed through DOM manipulation
(textual) as well as augmenting with image understanding (multi-modal). We
chose WebVoyager since it covers a diverse range of tasks across different
live, dynamic and representative websites. Other alternative benchmarks either
focused solely on single task domain [Yao et al., 2022], used custom created
websites that are significantly simpler than real-world versions in terms of
DOM complexity [Zhou et al., 2023], or used cached versions of real-websites
only supporting a fixed route for the agent and not allowing free form
exploration [Deng et al., 2024]. One of the issues with WebVoyager though is
that it uses static dates for some of the tasks (e.g. Search a hotel with free
WiFi and air conditioning in Bali from Jan 1 to Jan 4, 2024.). The static
dates makes some of these tasks unachievable since the dates have passed. We
manually went through the tasks to identify ones with fixed dates in the tasks
that make the task unachievable, and added 8 months to the dates (i.e., Search
a hotel with free WiFi and air conditioning in Bali from Aug 1 to Aug 4,
2024.).

Figure 7: WebVoyager Websites Website |  Task
---|---
Allrecipes |  Find a recipe for Baked Salmon that takes less than 30 minutes to prepare and has at least 4-star rating based on user reviews.
Booking |  Find the cheapest available hotel room for a three-night stay from 1st Jan in Jakarta. The room is for 2 adults, just answer the cheapest hotel room and the price.
Github |  Search for an open-source project related to ‘climate change data visualization’ on Github and the report the project with the most stars.
Google Flights |  Search a one-way flight from Dublin to Athens Greece for 1 Adult that leaves on December 30 and analyze the price graph for the next 2 months.
Table 1: Example tasks from WebVoyager benchmark.

###  3.2 Evaluation Setup

The entire benchmark was divided among 5 human evaluators who ran 125-130
tasks each. For each task, the evaluators were instructed to classify the task
as pass or fail along with a textual reason in case of failures. They were
instructed to mark the task as pass if it was completed successfully in full
(in case the task has multiple parts). The evaluators ran the task from India
during IST office hours (this may have implications when you interpret or
compare measures such as task completion times). For the evaluation, Agent-E
was used in autonomous mode (i.e., _ask user for input_ skill was disabled)
and used GPT-4-Turbo as the LLM for both planner and browser navigation agent.

###  3.3 Measures

We report four important measures that are relevant for comprehensive
evaluation of web agent and understanding their practical implementation
readiness.

  * •

Task success rates: The percentage of tasks that Agent-E performed
successfully across websites.

  * •

Self-aware vs Oblivious failure rates: Detecting when the task was not
completed successfully is of utmost importance, since it can be used for
enabling a fallback workflow, to notify the user of failure or use as an
avenue to gather human demonstration for the same task. Self-aware failures
are failures where agent is aware of its own failure in completing the task
and responds with a final message explicitly stating so, e.g. I’m unable to
provide a description of the first picture due to limitations in accessing or
analyzing visual content. or ‘Due to repeated rate limit errors on GitHub
while attempting to refine the search…’. The failures could be due to
technical reasons or agent deeming the task unachievable since it could not
complete the task after repeated attempts. On the other hand, oblivious
failures are cases were the agent wrongly answers the question or performs the
wrong action (e.g. adds the wrong product to cart or provides a wrong
information). For mainstream utility, oblivious failures should be as minimal
as possible. For the current evaluation, failures were categorized to self-
aware and oblivious failures by manual annotation. However, it would be
trivial to employ an LLM critique to automatically do the same task, similar
to [Wornow et al., 2024].

  * •

Task completion times: The average time required to complete the task, across
websites for failed and successful tasks.

  * •

Total number of LLM calls: The average number of LLM calls (both planner and
browser navigation agent) that was required to perform the task. This includes
both successful and failure cases.

###  3.4 Result: Quantitative

Publication | Task success rates on websites
---|---
| Allrecipe | Amazon | Apple | Arxiv | Github | Booking | ESPN | Coursera
He et al. 2024 (text) | 57.8 | 43.1 | 36.4 | 50.4 | 63.4 | 2.3 | 28.6 | 24.6
He et al. 2024 (multi) | 51.1 | 52.9 | 62.8 | 52.0 | 59.3 | 32.6 | 47.0 | 57.9
Lutz et al. 2024 (text) | 60 | 43.9 | 60.5 | 51.2 | 22.0 | 38.6 | 59.1 | 51.1
Agent-E (text) | 71.1 | 70.7 | 74.4 | 62.8 | 82.9 | 27.3 | 77.3 | 85.7
Failure modes | Agent-E Error Analysis on Websites
Overall failures % | 28.9 | 29.3 | 25.6 | 37.2 | 17.1 | 72.7 | 22.7 | 14.3
Self-aware failures % | 17.8 | 14.6 | 9.3 | 18.6 | 12.2 | 4.5 | 13.6 | 4.8
Oblivious failures % | 11.1 | 14.6 | 16.3 | 18.6 | 4.9 | 68.2 | 9.1 | 9.5
TCT | Agent-E Avg. Task Completion Times (seconds)
TCT (Success) | 116 | 286 | 122 | 137 | 104 | 183 | 187 | 119
TCT (Failed) | 196 | 246 | 200 | 176 | 384 | 317 | 387 | 177
LLM Calls | Agent-E Avg. Number of LLM calls
Total | 22 | 23.1 | 21.5 | 25.5 | 21.5 | 36.4 | 24.0 | 25.5
Planner | 6.5 | 6.4 | 5.9 | 6.9 | 5.4 | 6.6 | 6.3 | 6.3
Browser Nav Agent | 15.5 | 16.7 | 15.6 | 18.6 | 16.1 | 29.8 | 17.7 | 19.2
Table 2: Evaluation of Agent-E on WebVoyager. Publication | Task success rates on websites
---|---
| Dictionary | BBC | Flights | Maps | Search | Hug.Face | Wolfram | Overall
He et al. 2024 (text) | 66.7 | 45.2 | 7.1 | 62.6 | 75.2 | 31.0 | 60.2 | 44.3
He et al. 2024 (multi) | 71.3 | 60.3 | 51.6 | 64.3 | 77.5 | 55.8 | 60.9 | 57.1
Lutz et al. 2024 (text) | 86.0 | 81.0 | 0.0 | 39.0 | 67.4 | 53.5 | 65.2 | 52.6
Agent-E (text) | 81.4 | 73.8 | 35.7 | 87.8 | 90.7 | 81.0 | 95.7 | 73.1
Failure modes | Agent-E Error Analysis on Websites
Overall failures % | 18.6 | 26.2 | 64.3 | 12.2 | 9.3 | 19.0 | 4.3 | 26.9
Self-aware failures % | 16.2 | 9.6 | 57.1 | 12.0 | 4.6 | 14.3 | 2.1 | 14.1
Oblivious failures % | 2.4 | 16.6 | 7.1 | 0 | 4.6 | 4.7 | 2.1 | 12.8
TCT | Agent-E Avg. Task Completion Times (seconds)
TCT (Success) | 98 | 105 | 244 | 127 | 106 | 140 | 68 | 150
TCT (Failed) | 136 | 110 | 234 | 177 | 135 | 167 | 94 | 220
LLM Calls | Agent-E Avg. Number of LLM calls per Task
Total | 22.0 | 21.3 | 53.8 | 22.9 | 19.4 | 22.8 | 14.5 | 25.0
Planner | 6.6 | 6.0 | 11.4 | 5.8 | 5.6 | 6.2 | 4.4 | 6.4
Browser Nav Agent | 15.4 | 15.3 | 42.2 | 17.0 | 13.7 | 16.6 | 10.15 | 18.6
Table 3: Evaluation of Agent-E on WebVoyager (Contd.)

In this section, we will present quantitative results of how well Agent-E
performs in WebVoyager benchmark. Tables 2 and 3 shows the summary of
evaluation of Agent-E on WebVoyager.

#### Comparison with prior state-of-the-art agents

Agent-E overall successfully completed 73% of the tasks and outperforms prior
state-of-the-art text-only web agent WILBUR [Lutz et al., 2024] by 21% and
state-of-the-art multi-modal web agent [He et al., 2024] by 16%. The
effectiveness of Agent-E varied across websites from 27.3% (Booking) to 95.7%
(WolframAlpha). Comparing by websites, Agent-E outperformed both Wilbur [Lutz
et al., 2024] and WebVoyager multi-modal agent [He et al., 2024] in 11 out of
15 websites. Booking.com and Google Flights continue to be websites, where all
web agents, including Agent-E does not do well.

It is important to note that [Lutz et al., 2024] uses task and website-
specific prompting and [He et al., 2024] uses vision for observing the page.
In contrast, Agent-E is a generic text-only web agent which does not employ
any task or website-specific instructions. This suggests that there is likely
room for further improvement in Agent-E using some of these strategies.

#### Agent-E failure modes

Overall, Agent-E was self-aware of the failures for more than 52% of the
failed tasks. Typically, Self-aware failures occur when the reason for failure
are technical in nature (e.g., navigation issues, inability to extract certain
information from DOM elements such as Iframes, canvas or images, inability to
operate a button, anti-scraping policies employed by websites, inability to
find the answer despite multiple attempts etc.). The Oblivious failures are
scenarios where Agent-E gives a response that was wrong. These are typically
scenarios where agent overlooks certain task requirements and provides an
answer that only partially meets the requirements. It could also stem from DOM
observation issues (e.g., not being aware that the date got reset due to
incorrect format in Google Flights) or website capability understanding issues
(e.g., not using advanced search capability when needed, or assuming search
functionalities are perfect and every search result will completely satisfy
the search requirements). Similar error modes were also observed by [He et
al., 2024] who classify them as agent hallucinations.

#### Task Completion Times

Agent-E on an average took 150 seconds to successfully complete a task and 220
seconds for completion when the task was a failure. The longer duration for
failed tasks is interesting and also expected, since given a difficult task,
Agent-E may try multiple approaches to complete the task before giving up on
it. The difference in task completion times across websites (e.g., 68 seconds
to successfully complete a task in WolframAlpha vs. 286 seconds in Amazon)
also reflects the differences in task and website complexity. Amazon.com is a
feature-rich and content-heavy website (for example, the Amazon and
Booking.com homepage contains 3189 and 2,952 DOM elements respectively, in
contrast to 1520 in Google Flights and 590 for WolframAlpha homepages). When
the website is content-heavy and has numerous DOM elements, the DOM de-noising
step takes more time, and the LLM token count for browser navigation agent may
increase as well.

#### LLM Calls

On an average Agent-E took 25 LLM calls to execute a task (6.4 calls by the
planner and almost 3 times as much by the browser navigation agent). The
average number of LLM calls per website, as expected, is consistent with task
completion times. Unfortunately, no prior studies have reported an end-to-end
number of LLM calls required for task completion. While [Lutz et al., 2024]
provides the number of LLM calls required by one (out of three) of the LLM-
based components in their system, this information is insufficient for a
conclusive comparison.

###  3.5 Result: Qualitative

In this section, we will present qualitative results of concrete examples
showing how different design choices made in Agent-E help perform complex web
tasks.

#### Hierarchical Planner helps error detection and recovery

The hierarchical architecture allows easily detecting and recovering from
errors. The planner agent is prompted to perform verification (by asking
questions or asking confirmation) as part of the plan whenever necessary.
Figure 8 shows an example instance where the planner agent asks the browser
navigation agent for more information (i.e., list the search results), and
from the response (i.e., there are no specific search results) identifies that
it may have made an error by making the search query too focused. In the
example, the planner creates a new plan of action for performing the task.
Another common pattern in the evaluation was the planner’s ability to detect
errors and easily backtrack to a previous page to continue execution. Given
that the planner has the URL of the page at each step available to it, it
allows the planner to effortlessly backtrack to a previous page by adding it
as a step in the plan (e.g., navigate to the search result page using the
<url>).

Figure 8: An example instance of Agent-E detecting and recovering from errors.
The conversation is truncated with ‘…’ to enhance readability in the image.

#### Support for multiple DOM observation methods

HTML DOM can be extremely large (e.g., YouTube Homepage with all the DOM
elements and their attributes is about 800,000 tokens). Thus, it is important
to denoise and encode the DOM such that only task relevant information is
presented to the LLM. However, information relevant for a given task is very
dependent on the task at the hand. Some tasks may only need a complete textual
representation (e.g., summarise the current page), some tasks may only need
input fields and buttons (e.g., search on google). On the other hand, more
exploratory tasks, may need a complete representation of the page (e.g., what
elements are on this page). Most previous web agent work have used a single
DOM representation, e.g. [Zhou et al., 2023] used accessibility tree, multi-
modal web agent in [He et al., 2024] used screenshots and [Lutz et al., 2024]
used direct encoding and denoising of the HTML DOM. However, in our view,
there is no single DOM observation method that suits all the tasks. Thus,
Agent-E supports three different DOM representation methods text_only,
input_fields, all_fields. This allows Agent-E to flexibly select the DOM
representation that it feels is best suited for the task. Also, it allows for
backup, that if one does not work as expected, Agent-E can fall back to a
different representation. There were numerous examples in our benchmark where
this multiple DOM representation was useful. Figure 9 illustrates an example
where Agent-E adaptively uses all_fields DOM representation for interaction
and text_only for summarization.

Figure 9: Providing multiple options for DOM observation allows to flexibly
select one fit for task. The conversation is truncated with ‘…’ to enhance
readability in the image.

#### Change observation helps grounding

Change observation is a concept we introduced in which each action execution
also observes for changes in the state and returns a linguistic feedback to
the LLM. Some typical scenarios where this is useful is where the browser
navigation agent tries to click on a navigation item (e.g., click on the
soccer link on ESPN.com), however instead of navigating to the relevant
section, it may instead open a popup menu that requires further selection.
Another common example is when browser navigation agent attempts to set the
source airport in a flight booking websites and a list of possible airports
open up as a drop down. In both these cases, the interaction is not yet
complete, but the browser navigation agent may assume it is (i.e., navigation
to soccer section would require clicking on a new link from the popup menu,
setting source airport would require selecting one of the option from the drop
down). In both these cases, the click skills will return a feedback to the LLM
that as a consequence of the click, a menu has appeared where you may need to
make further selection. See Figure 6 for an example. Conceptually, the purpose
of the change observation is to provide linguistic feedback to the LLM whether
the action has been completed without any errors and did it lead to any
tangible change in the environment, in order to inform subsequent actions. We
also envision efficiency improvements if the change observation can return a
list of elements, so that LLM can make subsequent selection without again
using Get DOM skill to observe the state of the DOM. Change observation is
adjacent to the concept of Reflexion [Shinn et al., 2024]. However, there are
nuanced differences between the two. Reflexion paradigm provides feedback of a
prior failure. However, change observation is not directly associated or
limited to a prior failure. The observation returned can be any outcome of the
action. Reflexion uses an LLM to analyse the scalar ‘success/failure’ signal
from an action and current trajectory to produce the verbal feedback to the
agent. Reflexion did not provide any tangible improvement when directly
applied to web automation in the Webshop[Yao et al., 2022] benchmark. On the
other hand, change observation is implemented using classical code-based
approaches to observe the consequence of an action and linguistic feedback of
actions is implemented using heuristic-based approach to help the system to be
better aware of the current state of the environment, and nudge the system
towards the correct next action.

##  4 Discussion

In this section, we will synthesize our learnings from the development and
evaluation of Agent-E into a series of agent design principles. We believe
these principles can be generalized beyond the domain of web automation.

###  4.1 Agent Design Principles

  1. 1.

Well crafted set of primitive skills can enable powerful use-cases: A well
crafted ensemble of foundational skills can serve as a building block to
support more complex functionalities. LLMs can effectively combine these
skills to unlock a broad range of usecases. The analysis of the intended use
case is crucial to arrive at the requisite primitive skills. In the case of
Agent-E these primitive skills were click, enter text, get DOM, Open URL and
Press Keys. These were only a subset of what a user could potentially perform
on a page (e.g. we did not support drag, double click, right click, tab
management, etc. We considered the primitive skills we enabled in Agent-E to
be enough for vast majority of general web automation tasks. Nonetheless, if
we were to specialise Agent-E to work on certain websites where right-click to
select a functionality is a prominent interaction pattern, we would need to
introduce that as a primitive skill.

  2. 2.

Consider hierarchical architecture for complex tasks: Hierarchical
architecture can be useful in agents with multiple LLM-based components. It
allows execution of more complex tasks through a clear separation of roles and
responsibilities. Hierarchical architecture excels in scenarios where tasks
can be decomposed into sub-tasks that need to be handled at different levels
of granularity. Additionally, it aids in the identification of tasks that can
be executed in parallel, potentially leading to performance enhancements. It
also supports the development and improvement of various components in
isolation.
It is important to keep in mind that hierarchical architecture may not be
suitable for all tasks. In case of Agent-E, if all we had to support were,
e.g., navigate to specific URLs or perform simple web search, a hierarchical
architecture may be over-complicated. In such cases, a much simpler
architecture may suffice.

  3. 3.

Perform payload denoising when relevant: Mitigating noise in the payload is
critical in creating reliable, cost and time efficient agents. Noise could be
in the form of unnecessary or irrelevant information, which could lead to
incorrect or sub-optimal performance. It is also important to keep in mind
that what is considered noise may be dependent on the task.
Simple techniques such as filtering irrelevant data, transforming the data to
an easily consumable format, and focusing on key information can contribute to
more accurate decision-making by the agent. This is especially important in
environments with a high degree of uncertainty or very large input payload. In
the case of Agent-E, we performed denoising on the HTML DOM. Further, we
provided multiple DOM observation capabilities that the agent could adaptively
use given the task requirements.

  4. 4.

Provide linguistic feedback of actions: Our exploration into the space of
agents capable of performing actions that has tangible consequences suggests
that linguistic feedback of actions could be useful to help executor agents to
be better aware of the environment and any consequence of the actions (e.g., a
dialog box appeared as a consequence of the click action). Change observation
helps refine the agent’s subsequent actions by providing a clear narrative of
cause and effect, and also improved awareness of the environment. This could
apply in a variety of usecases such as desktop automation or automations in
the physical space (e.g robot control).

  5. 5.

Support human in the loop architecture when necessary: Supporting human in the
loop architecture is critical for agentic systems. Given that agentic systems
are so new, users may not trust the system completely. Further, there will
arise scenarios where the agent may need to have varying degree of user
involvement for a multitude of reasons: e.g., to get more clarification on the
task, when there is ambiguity how to proceed, to approve critical actions,
perform some mission critical actions that the agent cannot or should not
perform, audit the final output and re-invoke if needed, or take over
responsibility when agent identifies that it cannot perform the task. Human in
the loop architecture could also be leveraged for self-improvement where the
agent identifies opportune times to gather human demonstration (e.g. Sorry, I
could not do this task. Can you show me how you would do it?. Thus,
incorporating mechanisms for human oversight can play a pivotal role in
building that trust, and seamless handover.
Human in the loop could mean different levels of human involvement. From
simple notifications of task progress, human involvement at key decision
points, all the way to a handover of an incomplete task to a human.

  6. 6.

Analyse, reflect and aggregate past experiences routinely for self-
improvement: For Agentic systems to be adopted widely, they need to
(gradually) achieve close to human-level performance. In the context of web
automation, Agent-E could perform 73.1% of the tasks with an average task
completion time of 150-220 second and 25 LLM calls. While very promising, this
is not practical for production use-cases in terms of both effectiveness and
efficiency.
A simple agentic solution to improving efficiency is to cache LLM calls. These
mechanisms are supported out-of-the-box by agent development frameworks such
as Autogen [Wu et al., 2023b]. However, a naive caching may not work well in
dynamic contexts (e.g., a web site will continuously change in terms of
content, advertisements etc.). A better approach would be to establish offline
workflows that routinely analyse, reflects on and aggregates past tasks and
human demonstrations to convert them to more classical automation workflows.
These classical approach could be re-triggered upon a new task if it matches a
workflow that it has encountered in the past, and use the exploratory agentic
approach only as a fallback. This would mean the task could be completed
faster and cheaper.

  7. 7.

Introduce internal and external guardrails into the Agentic System: To ensure
safe and effective operation of an agentic system, it’s important to introduce
various guardrails into its operation. These guardrails can be in the form of
rules, guidelines or constitution that the system must adhere to. Some of the
guardrails may be external to system, e.g., a task appropriateness guardrail
may filter out inappropriate tasks even before agentic system is invoked. In
addition, there could be guardrails intrinsic to the system such as guardrails
against prompt injection attacks, and operational guardrails that limits how
the agent performs different tasks.
In the context of Agent-E, the operational guardrails include domain
boundaries within which Agent-E should operate (e.g., intranet of an
organisation), task specific boundaries (e.g., Agent-E should only respond to
a medical question from authoritative websites) or user-configured boundaries
(e.g., for shopping, use only Amazon)

  8. 8.

Choose between generic agent vs. task specific agent: Generic agentic systems
by definition can perform a wide range of tasks. However, in many practical
implementations, a more focused set of capability may be desirable. For
example, Agent-E is a generic web agent that can perform a wide range of tasks
on the internet, but not necessarily optimized for any specific task. It would
be possible to optimize Agent-E for specific type of tasks (e.g., form
filling) or specific websites (e.g., Atlassian Confluence pages) to achieve
significantly higher performance. Depending on the use case, an optimized
agent may suit better for certain workflows than a generic version.

##  5 Related Work

LLM-based Planning and Reasoning Over the last few years, large language
models (LLMs) have demonstrated extraordinary abilities in text generation,
code generation, and the generation of natural language multi-step reasoning
traces. Spurred by these, there has been much work in the use of LLMs to solve
multi-step reasoning and planning problems. The many variants of ‘chain-of-
thought’ techniques [Wei et al., 2023, Chu et al., 2023] encourage the LLM to
produce a series of tokens with causal decoding that drive towards the
solution of problems in math, common-sense reasoning and other similar tasks
[Chowdhery et al., 2022, Fu et al., 2023, Li et al., 2023, Mitra et al.,
2024]. With tool-usage for sensing and acting, LLMs have also been used to
drive planning in software environments and embodied agents e.g., [Baker et
al., 2022, Wang et al., 2023a, Wang et al., 2023b, Irpan et al., 2022,
Bousmalis et al., 2023, Wu et al., 2023a, Bhateja et al., 2023]. Finally,
there has been related work investigating the limits of LLMs when it comes to
planning and validation. For examples of negative results, see [Valmeekam et
al., 2023b, Momennejad et al., 2023, Valmeekam et al., 2023a, Huang et al.,
2023, Kambhampati et al., 2024] among others.

Specialized Agents for Repetitive Tasks Beyond the examples above, and as
described in Section 1, there has been much recent interest in building
specialized agents for the web [Zheng et al., 2024, He et al., 2024, Lutz et
al., 2024] and on device [Bai et al., 2024, Wen et al., 2024]. Also related is
recent work on building agentic workflows to replace robotic process
automation [Wornow et al., 2024]. Further afield, the work on building agents
and training language models for API usage is also related, given that many
software tasks and workflows involve the use of APIs; examples include
[Hosseini et al., 2021, Patil et al., 2023, Qin et al., 2024] and many more.

Hierarchical Planning The notion of hierarchical AI planning has been around
for five decades or more. Instead of planning directly in the space of low-
level primitive actions, planning in a space of ‘high-level actions’
constrains the size of the plan length (and hence the size of the plan-space),
which can result in more effective and efficient search. Examples from prior
work include [Tate, 1977, Nau et al., 1991, Marthi et al., 2007] and many
more; see [Russell and Norvig, 2009] for more details. Also related is the use
of temporal abstractions in planning and in reinforcement learning, for
example the use of options in [Sutton et al., 1999, Bacon et al., 2017]. In
recent years, multiple papers have proposed the use of hierarchical planning
for solving tasks in complex environments or with embodied agents; examples
include [Wang et al., 2022, Irpan et al., 2022] and several others.

##  6 Conclusion

In this paper, we introduced Agent-E, a novel web agent designed to perform
complex web-based tasks that makes use of numerous architectural improvements
over prior state-of-the-art web agents such as hierarchical architecture,
flexible DOM distillation and denoising method and concept of change
observation to guide the agent towards more accurate performance.

Agent-E was evaluated on the WebVoyager benchmark, achieving a state-of-the-
art success rate of 73.2%, a significant improvement over previous text-only
and multi-modal web agents. Beyond task success rates, we also reported on
additional metrics such as error awareness, task completion times, and the
number of LLM calls, providing a more comprehensive evaluation of Agent-E’s
performance.

We presented our learnings in the form of eight general design principles for
developing agentic systems that can be applied beyond the realm of web
automation.

## References

  * [Bacon et al., 2017] Bacon, P.-L., Harb, J., and Precup, D. (2017).  The option-critic architecture.  AAAI Conference on Artificial Intelligence.
  * [Bai et al., 2024] Bai, H., Zhou, Y., Cemri, M., Pan, J., Suhr, A., Levine, S., and Kumar, A. (2024).  Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning.  arXiv preprint arXiv:2406.11896.
  * [Baker et al., 2022] Baker, B., Akkaya, I., Zhokhov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. (2022).  Video pretraining (vpt): Learning to act by watching unlabeled online videos.
  * [Bhateja et al., 2023] Bhateja, C., Guo, D., Ghosh, D., Singh, A., Tomar, M., Vuong, Q., Chebotar, Y., Levine, S., and Kumar, A. (2023).  Robotic offline rl from internet videos via value-function pre-training.
  * [Bousmalis et al., 2023] Bousmalis, K., Vezzani, G., Rao, D., Devin, C., Lee, A. X., Bauza, M., Davchev, T., Zhou, Y., Gupta, A., Raju, A., Laurens, A., Fantacci, C., Dalibard, V., Zambelli, M., Martins, M., Pevceviciute, R., Blokzijl, M., Denil, M., Batchelor, N., Lampe, T., Parisotto, E., Żołna, K., Reed, S., Colmenarejo, S. G., Scholz, J., Abdolmaleki, A., Groth, O., Regli, J.-B., Sushkov, O., Rothörl, T., Chen, J. E., Aytar, Y., Barker, D., Ortiz, J., Riedmiller, M., Springenberg, J. T., Hadsell, R., Nori, F., and Heess, N. (2023).  Robocat: A self-improving foundation agent for robotic manipulation.
  * [Chowdhery et al., 2022] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. (2022).  Palm: Scaling language modeling with pathways.
  * [Chu et al., 2023] Chu, Z., Chen, J., Chen, Q., Yu, W., He, T., Wang, H., Peng, W., Liu, M., Qin, B., and Liu, T. (2023).  A survey of chain of thought reasoning: Advances, frontiers and future.
  * [Deng et al., 2024] Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. (2024).  Mind2web: Towards a generalist agent for the web.  Advances in Neural Information Processing Systems, 36.
  * [Fu et al., 2023] Fu, Y., Peng, H., Ou, L., Sabharwal, A., and Khot, T. (2023).  Specializing smaller language models towards multi-step reasoning.
  * [He et al., 2024] He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. (2024).  Webvoyager: Building an end-to-end web agent with large multimodal models.  arXiv preprint arXiv:2401.13919.
  * [Hosseini et al., 2021] Hosseini, S., Awadallah, A. H., and Su, Y. (2021).  Compositional generalization for natural language interfaces to web apis.  arXiv preprint arXiv:2112.05209.
  * [Huang et al., 2023] Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. (2023).  Large language models cannot self-correct reasoning yet.
  * [Irpan et al., 2022] Irpan, A., Herzog, A., Toshev, A. T., Zeng, A., Brohan, A., Ichter, B. A., David, B., Parada, C., Finn, C., Tan, C., Reyes, D., Kalashnikov, D., Jang, E. V., Xia, F., Rettinghouse, J. L., Hsu, J. C., Quiambao, J. L., Ibarz, J., Rao, K., Hausman, K., Gopalakrishnan, K., Lee, K.-H., Jeffrey, K. A., Luu, L., Yan, M., Ahn, M. S., Sievers, N., Joshi, N. J., Brown, N., Cortes, O. E. E., Xu, P., Sampedro, P. P., Sermanet, P., Ruano, R. J., Julian, R. C., Jesmonth, S. A., Levine, S., Xu, S., Xiao, T., Vanhoucke, V. O., Lu, Y., Chebotar, Y., and Kuang, Y. (2022).  Do as i can, not as i say: Grounding language in robotic affordances.  arXiv preprint arXiv:2204.01691.
  * [Kambhampati et al., 2024] Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., and Murthy, A. (2024).  Llms can’t plan, but can help planning in llm-modulo frameworks.  arXiv preprint arXiv:2402.01817.
  * [Kapoor et al., 2024] Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. (2024).  Ai agents that matter.  arXiv preprint arXiv:2407.01502.
  * [Li et al., 2023] Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. (2023).  Textbooks are all you need ii: phi-1.5 technical report.
  * [Lutz et al., 2024] Lutz, M., Bohra, A., Saroyan, M., Harutyunyan, A., and Campagna, G. (2024).  Wilbur: Adaptive in-context learning for robust and accurate web agents.  arXiv preprint arXiv:2404.05902.
  * [Marthi et al., 2007] Marthi, B., Russell, S., and Wolfe, J. (2007).  Angelic semantics for high-level actions.  International Conference on Automated Planning and Scheduling.
  * [Mitra et al., 2024] Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. (2024).  Orca-math: Unlocking the potential of slms in grade school math.
  * [Momennejad et al., 2023] Momennejad, I., Hasanbeig, H., Vieira, F., Sharma, H., Ness, R. O., Jojic, N., Palangi, H., and Larson, J. (2023).  Evaluating cognitive maps and planning in large language models with cogeval.
  * [Nakano et al., 2022] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. (2022).  Webgpt: Browser-assisted question-answering with human feedback.  arXiv preprint arXiv:2112.09332.
  * [Nau et al., 1991] Nau, D., Cao, Y., Lotem, A., and Muñoz-Avila, H. (1991).  Shop: Simple hierarchical ordered planner.  International Joint Conference on Artificial Intelligence.
  * [Patil et al., 2023] Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. (2023).  Gorilla: Large language model connected with massive apis.  arXiv preprint arXiv:2305.15334.
  * [Qin et al., 2024] Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Hong, L., Tian, R., Xie, R., Zhou, J., Gerstein, M., Li, D., Liu, Z., and Sun, M. (2024).  Toolllm: Facilitating large language models to master 16000+ real-world apis.  International Conference on Learning Representations.
  * [Russell and Norvig, 2009] Russell, S. J. and Norvig, P. (2009).  Artificial Intelligence: a modern approach.  Pearson, 3 edition.
  * [Shinn et al., 2024] Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. (2024).  Reflexion: Language agents with verbal reinforcement learning.  Advances in Neural Information Processing Systems, 36.
  * [Sutton et al., 1999] Sutton, R., Precup, D., and Singh, S. (1999).  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  Artificial Intelligence Journal.
  * [Tate, 1977] Tate, A. (1977).  Generating project networks.  International Joint Conference on Artificial Intelligence.
  * [Valmeekam et al., 2023a] Valmeekam, K., Marquez, M., and Kambhampati, S. (2023a).  Can large language models really improve by self-critiquing their own plans?
  * [Valmeekam et al., 2023b] Valmeekam, K., Sreedharan, S., Marquez, M., Olmo, A., and Kambhampati, S. (2023b).  On the planning abilities of large language models (a critical investigation with a proposed benchmark).
  * [Wang et al., 2023a] Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. (2023a).  Voyager: An open-ended embodied agent with large language models.
  * [Wang et al., 2022] Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., and Liang, Y. (2022).  Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.  Advances in Neural Information Processing Systems, 37.
  * [Wang et al., 2023b] Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., and Liang, Y. (2023b).  Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.
  * [Wei et al., 2023] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2023).  Chain-of-thought prompting elicits reasoning in large language models.
  * [Wen et al., 2024] Wen, H., Li, Y., Liu, G., Zhao, S., Yu, T., Li, T. J.-J., Jiang, S., Liu, Y., Zhang, Y., and Liu, Y. (2024).  Autodroid: Llm-powered task automation in android.  In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543–557.
  * [Wornow et al., 2024] Wornow, M., Narayan, A., Opsahl-Ong, K., McIntyre, Q., Shah, N. H., and Re, C. (2024).  Automating the enterprise with foundation models.  arXiv preprint arXiv:2405.03710.
  * [Wu et al., 2023a] Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. (2023a).  Unleashing large-scale video generative pre-training for visual robot manipulation.
  * [Wu et al., 2023b] Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. (2023b).  Autogen: Enabling next-gen llm applications via multi-agent conversation framework.  arXiv preprint arXiv:2308.08155.
  * [Yao et al., 2022] Yao, S., Chen, H., Yang, J., and Narasimhan, K. (2022).  Webshop: Towards scalable real-world web interaction with grounded language agents.  Advances in Neural Information Processing Systems, 35:20744–20757.
  * [Zheng et al., 2024] Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. (2024).  Gpt-4v (ision) is a generalist web agent, if grounded.  arXiv preprint arXiv:2401.01614.
  * [Zhou et al., 2023] Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. (2023).  Webarena: A realistic web environment for building autonomous agents.  arXiv preprint arXiv:2307.13854.


# Semantic Segmentation for Autonomous Driving: Model Evaluation, Dataset
Generation, Perspective Comparison, and Real-Time Capability

SENAY CAKIR , MARCEL GAUß , KAI HÄPPELER , YASSINE OUNAJJAR ,
FABIAN HEINLE , AND REINER MARCHTHALER
1 Esslingen University of Applied Sciences, Flandernstraße 101, 73732
Esslingen am Neckar, GERMANY

###### Abstract

Environmental perception is an important aspect within the field of autonomous
vehicles that provides crucial information about the driving domain, including
but not limited to identifying clear driving areas and surrounding obstacles.
Semantic segmentation is a widely used perception method for self-driving cars
that associates each pixel of an image with a predefined class. In this
context, several segmentation models are evaluated regarding accuracy and
efficiency. Experimental results on the generated dataset confirm that the
segmentation model FasterSeg is fast enough to be used in realtime on low-
power computational (embedded) devices in self-driving cars. A simple method
is also introduced to generate synthetic training data for the model.
Moreover, the accuracy of the first-person perspective and the bird’s eye view
perspective are compared. For a $320\times 256$ input in the first-person
perspective, FasterSeg achieves $65.44\,\%$ mean Intersection over Union
(mIoU), and for a $320\times 256$ input from the bird’s eye view perspective,
FasterSeg achieves $64.08\,\%$ mIoU. Both perspectives achieve a frame rate of
$247.11$ Frames per Second (FPS) on the NVIDIA Jetson AGX Xavier. Lastly, the
frame rate and the accuracy with respect to the arithmetic 16-bit Floating
Point (FP16) and 32-bit Floating Point (FP32) of both perspectives are
measured and compared on the target hardware.

###### Index Terms:

Autonomous driving, deep learning, image segmentation, real-time semantic
segmentation, supervised learning, synthetic data generation

##  I Introduction

Autonomous vehicles have a variety of different sensor systems onboard to
detect obstacles, lanes, free parking spaces, etc. [1], [2]. A frequently
applied technique in this field is image segmentation, which uses camera
images to classify each pixel. The predicted images can be used to plan the
vehicle’s behavior and avoid collisions[3], [4]. This work was conducted in
the context of the Carolo-Cup, a student competition providing student teams
with a platform for the design and implementation of autonomous Radio
Controlled (RC) vehicles. They must accomplish various driving tasks such as
parking or overtaking in an imitated environment containing obstacles,
intersections, parking spaces, and more. Furthermore, RC vehicles use embedded
hardware to run the sensing, planning, control algorithms, etc. The algorithms
must therefore run in realtime so that the vehicle can drive smoothly and
reliably. [5]
To classify the pixels of the images delivered by the camera built on top of
the vehicle, several image segmentation models were evaluated. A dataset
representing the imitated environment is required to train the segmentation
neural network. In this context, synthetic images generated with a simulation
are combined with real images of the Carolo-Cup environment to compose the
training dataset. Supervised learning is used in this work because each image
of the dataset has its corresponding ground truth. The motivation of this work
is to generate a dataset that mainly contains synthetic data to avoid high
labeling effort. Thus, the routes can be generated in a simulation and must
not be replicated. Moreover, a state-of-the-art image segmentation model is
applied in realtime on a comparatively slow embedded hardware. Additionally,
the potential of the bird’s eye view perspective is examined. Both the overall
accuracy mean Intersection over Union (mIoU) and the accuracy Intersection
over Union (IoU) of each class are then investigated more closely. This paper
attempts to answer four main questions:

  * •

Which image segmentation model is fast and accurate enough for the Carolo-Cup?

  * •

How to easily generate labeled synthetic data?

  * •

Is the bird’s eye view perspective a better alternative compared to the first-
person perspective?

  * •

What impact does the 16-bit Floating Point (FP16) and the 32-bit Floating
Point (FP32) arithmetic have on the model accuracy and the real-time
capability?

This paper is organized as follows. First, different segmentation models are
evaluated to find a suitable option for this work. Secondly, a method for
generating labeled synthetic data is described. Lastly, two different
experiments are conducted, using the selected segmentation model trained with
the generated data. The first experiment examines the accuracy of two models
trained with data from two different perspectives: the first-person and the
bird’s eye view perspective. The second experiment explores the real-time
capability and accuracy regarding the arithmetic FP16 and FP32 of both models.
The intended contributions of this study are the following:

  * •

The development of a simple, yet effective method to generate synthetic data
representing an imitated environment for autonomous vehicles.

  * •

Exploring the possibility of executing semantic segmentation on low-power
embedded devices using images from the bird’s eye view perspective.

##  II Related work

The following section describes related work which is relevant. First, the
fundamentals of synthetic data generation are introduced. In the second
paragraph, an overview of real data sources is provided. Then, various image
segmentation models are evaluated and compared in terms of accuracy and frame
rate. Finally, the chosen image segmentation model is further described.

###  II-A ROAD GENERATION AND SIMULATION

Gazebo is chosen as the simulation environment to replicate realistic driving
scenes. The synthetic routes used in Gazebo can be generated as images using a
road generator provided by [6]. These images can be directly rendered in the
simulation environment. To create various routes, the road generator is
extended with the objects listed in Fig. 2. Furthermore, the generator is
customized to create equivalent annotated routes [7].

###  II-B SOURCES OF REAL DATA

In addition to the generated synthetic data, images from real routes in
imitated environments are used. These real images are provided by various
research teams such as Spatzenhirn (University of Ulm) [8], ISF Löwen
(Technical University of Braunschweig) [9], KITcar (Karlsruhe Institute of
Technology) [10], and it:movES (Esslingen University of Applied Sciences)
[11]. Different environments offer a relatively high diversity of real images
which can be very useful for training and testing an image segmentation model.
The images are recorded using different cameras.

###  II-C EVALUATION OF STATE-OF-THE-ART IMAGE SEGMENTATION MODELS

Image segmentation is an important part of visual perception systems for
autonomous vehicles. It can be described as separating an image into any
segments. Image segmentation can be divided into semantic segmentation and
instance segmentation. Semantic segmentation refers to the process of
assigning a label to each pixel of a picture. Instance segmentation extends
the semantic segmentation scope further by detecting each instance of the
object within the image and delineating it with a bounding box or segmentation
mask. [27]
To interpret the images of the vehicle’s environment, different instance and
semantic segmentation models were evaluated. The goal of the evaluation is to
find a model that achieves a high inference frame rate measured in Frames per
Second (FPS) while concurrently maintaining high detection accuracy.
Table I lists the frame rate as well as the accuracy of some state-of-the-art
instance segmentation models on different GPUs. The models were rated using
the MS COCO benchmark dataset [12], and the Average Precision (AP) was used as
an accuracy metric. Table II lists some state-of-the-art semantic segmentation
models rated with the Cityscapes benchmark dataset [13]. The $mIoU$ is used to
measure the models’ accuracies. The $mIoU$ is defined as follows:

$mIoU=\frac{1}{N}\sum_{n=1}^{N}IoU=\frac{1}{N}\sum_{n=1}^{N}\frac{TP}{(TP+FP+FN)}$
(1)

where $TP$, $FP$, and $FN$ are the numbers of true positive, false positive,
and false negative pixels determined over the whole test set. $N$ is the
number of the defined classes. [28]
The evaluation demonstrates that semantic segmentation models achieve higher
frame rates and accuracy values using higher image resolution. Therefore
semantic segmentation is chosen due to it’s likelihood to run successfully in
realtime on the used hardware. As Table II indicates, models such as BiSeNetV2
[21] or FasterSeg [22] achieve a comparatively high frame rate, and the
accuracy is only slightly smaller compared to other models like U-HarDnet-70
[18] or SwiftNetRN-18 [20]. To reach real-time capability and good accuracy on
the used embedded hardware, FasterSeg is chosen as a segmentation model for
this work. FasterSeg attains the highest frame rate of the compared models
with 163.9 FPS and a relatively good $mIoU$ with $71.5\,\%$.

TABLE I: Evaluation of the instance segmentation models. Model |  Resolution |  AP [$\%$] |  FPS |  GPU
---|---|---|---|---
BlendMask-512 |  550 x 550 |  36.8 |  25.0 |  GeForce GTX
[14] |  |  |  |  1080 Ti
YOLACT-550++ |  550 x 550 |  34.1 |  33.5 |  GeForce GTX
[15] |  |  |  |  Titan Xp
CenterMask |  580 x 600 |  38.3 |  35.0 |  GeForce GTX
[16] |  |  |  |  Titan XP
YOLACT-550 [17] |  550 x 550 |  29.8 |  33.5 |  GeForce GTX
|  |  |  |  Titan Xp
TABLE II: Evaluation of the semantic segmentation models. Model |  Resolution |  mIoU |  FPS |  GPU
---|---|---|---|---
|  |  [$\%$] |  |
U-HarDnet-70 [18] |  1024 x 2048 |  75.9 |  53.0 |  GeForce GTX
[19] |  |  |  |  1080 Ti
SwiftNetRN-18 [20] |  1024 x 2048 |  75.5 |  39.9 |  GeForce GTX
|  |  |  |  1080 Ti
BiSeNetV2 [21] |  1024 x 2048 |  72.6 |  156.0 |  GeForce GTX
|  |  |  |  1080 Ti
FasterSeg [22] |  1024 x 2048 |  71.5 |  163.9 |  GeForce GTX
|  |  |  |  1080 Ti
BiSeNet [23] |  1024 x 2048 |  68.4 |  105.8 |  Titan Xp

###  II-D FASTERSEG

FasterSeg is an automatically designed neural network for semantic
segmentation that was discovered by a novel Neural Architecture Search (NAS)
framework. This network has an efficient search space with multi-resolution
branching. With the intention of reaching an optimal trade-off between low
latency and high accuracy, this network integrates a fast inference speed,
competitive accuracy and a fine grained latency regularization. Knowledge
Distillation (KL) is a technique used by FasterSeg to transfer knowledge from
a large complex network (teacher $T$) to a much smaller network (student $S$).
Fig. 1 shows that the searched teacher is a sophisticated network based on the
same search space and supernet weights $W$ as the student. FasterSeg uses two
sets of architectures in one supernet to simultaneously search for a
complicated teacher $(\alpha_{T},\beta_{T})$ and a lightweight student
$(\alpha_{S},\beta_{S},\gamma_{S})$. FasterSeg training can be broken down
into four stages:

  * •

Search the architecture

  * •

Pre-train the supernet

  * •

Pre-train the teacher network

  * •

Pre-train the student network

In all experiments conducted the supernet is pre-trained for 20 epochs without
changing the architecture parameters. Then the architecture search is done for
30 epochs. The epoch values are the same as used for the search experiments
run by the FasterSeg developing team. [22]

Figure 1: During search (left), the co-searching framework optimizes two
architectures, and during training from scratch (right), it distills from a
complicated teacher to a light student using KL [22]. Figure 2: Definition of
the existing objects. Each color represents the corresponding class, which
should be recognized. Figure 3: System overview of the synthetic data
generation.

##  III Proposed approach

The definitions of the labels with various scenarios of the imitated
environment are shown in Fig. 2. In this section, the generation of the
labeled data is described. Additionally, a method to transform the images into
a bird’s eye view perspective is presented.

###  III-A SYNTHETIC DATA GENERATION

The process of synthetic data generation is shown in its entirety in Fig. 3.
This process is divided into three fields: the road generator, the simulation,
and the image processing. Furthermore, the automation level of each task
within the fields is visualized with a corresponding color. In the following,
the tasks of each main field of the figure are described.

####  III-A1 Road Generator

Synthetic data generation starts with the creation of a route layout. High
diversity and different constellations are essential for accurate predictions.
Therefore, various configurations of parking zones, intersections, center line
types, missing lines, objects, and curves with different radii and angles must
be created. A raw and an annotated route are automatically generated using the
designed layout. The road generator also creates x- and y- coordinates, as
well as the yaw angle. This represents the spatial orientation for the
trajectory of the simulated vehicle. The coordinates run along the center of
the right lane. Special driving maneuvers, such as overtaking, parking, or
crossing the intersection from different directions, must be added manually.

####  III-A2 Simulation

As described in Section II-A, Gazebo is selected as the simulation environment
to generate synthetic training data for this work. The simulator produces
realistic first-person perspective image sequences that replicate real driving
scenarios. To achieve this, two different virtual vehicles are rendered in the
simulator, each one driving on a different route created by the road
generator. The first vehicle is driving on the raw route, while the second one
is assigned the colored route. Due to the fact Gazebo is based on Robotic
Operating System (ROS), the architecture and therefore the behavior of both
vehicles are similar. The virtual vehicles were built based on the RC vehicle
used at Esslingen University. After rendering both vehicles and routes in the
simulation environment, the trajectory generated by the road generator is
published using ROS. The publishing of the trajectory is executed for both
vehicles at the same time. The camera topics are then recorded to produce two
synchronized sequences of raw and colored images. These images are finally
sent to the next stage for processing. Fig. 4 shows a flow chart of the
trajectory publishing process.

Figure 4: Flow chart of the publishing process of the trajectory in Gazebo.

####  III-A3 Image Processing

To process the annotations, FasterSeg requires an 8-bit grayscale image where
each pixel contains the class ID. Therefore, each pixel of the colored image
is replaced by the corresponding class ID using a lookup table. The result is
an 8-bit grayscale image with IDs representing each class. Additionally, a
Region of Interest (ROI) is set to exclude undetectable objects near the
horizon. FasterSeg also requires the image height and width to be divisible by
64. The generated images are thus checked and eventually downscaled.

###  III-B REAL DATA GENERATION

To cover all driving scenarios and achieve optimal predictions, synthetic data
must be extended with real images. It is necessary to consider special
features that are not included in the simulation like natural lightning,
blurred scenes, and surroundings beyond the route, as illustrated in Fig. 5.
Hence, real images containing these features are added to the dataset. The
real images include different driving maneuvers as well as objects with
various orientations and visibility. In addition, depending on the data
source, various image resolutions, grayscale and colored images, and various
RC vehicles are used (II-B).

###  III-C BIRD’S EYE VIEW TRANSFORMATION

There are multiple ways to transform a first-person perspective image into a
bird’s eye view perspective. The warp perspective mapping method [26] is used
for this work since no intrinsic nor extrinsic parameters of the cameras are
available. This mapping method is suitable for several different camera models
and does not require additional calibration. The mapping process consists of
selecting four points $X_{ego}$ on the ground plane from the first-person
perspective and their corresponding points $X_{bird}$ from the bird’s eye view
perspective as illustrated in Fig. 6. $X_{ego}$ of the input image will be
viewed as $X_{bird}$. The mapping from $X_{ego}$ to $X_{bird}$ can be
expressed as:

$X_{bird}=H\cdot X_{ego}$ (2)

The transformation matrix $H$ can be calculated using the equation above. The
matrix $H$ is then used to map the input images from the first-person to the
output of the bird’s eye view perspective using a pixel-by-pixel process [25].

Figure 5: In the left image [10], surroundings beyond the track like legs,
feets and shoes are depicted in the background. The right image [11] shows a
reflection on the track caused by natural lightning. Figure 6: Bird’s eye view
transformation process using four mapping points illustrated in red.

##  IV Experiments

In this section, the conducted experiments to test the performance of the
proposed model on the generated dataset as well as the respective results are
described. In this context, the accuracy of the first-person and bird’s eye
view perspectives are compared. Also, the frame rate and the accuracy of both
perspectives are measured and compared using different model arithmetic on the
NVIDIA Jetson AGX Xavier board.

###  IV-A COMPARISON OF FIRST-PERSON AND BIRD’S EYE VIEW PERSPECTIVE REGARDING
ACCURACY

This experiment compares the accuracy of the semantic segmentation model
regarding the first-person and the bird’s eye view perspectives. For this
purpose, two FasterSeg models are trained using a dataset from the first-
person and the bird’s eye view perspectives. In the following, the dataset and
the hyperparameters, which were used to train the models, are described.
Finally, the results of this experiment are presented.

####  IV-A1 DATASET

Table III describes the dataset used for the training of the FasterSeg models.
The dataset consists of synthetic and real images, received from the it:movES
team. The images are divided into three sets: a training set (Train)
containing $75\,\%$ of the images, a validation set (Val) consisting of
$25\,\%$ of the images, and a test set (Test) composed of 20 real images used
to measure the accuracy of the models. To train the bird’s eye view model, all
the images are transformed into the bird’s eye view perspective using the
method described in section III-C. Both FasterSeg models are trained using the
same resolution ($320\times 256$) to objectively compare both perspectives. It
is important to consider that this dataset contains only 11 objects instead of
the initial 14 illustrated in Fig. 2.

TABLE III: The dataset generated for the perspective comparison experiment. Data source |  Resolution |  Resolution |  Train |  Val |  Test
---|---|---|---|---|---
|  first-person perspective |  bird’s eye view perspective |  |  |
it:movES (real) |  320 x 256 |  320 x 256 |  55 |  18 |  20
it:movES |  320 x 256 |  320 x 256 |  576 |  192 |  0
(synthetic) |  |  |  |  |
Sum |  |  |  631 |  210 |  20

####  IV-A2 SETTINGS

The training process of FasterSeg is divided into four substeps [22], [24].
Note that only the teacher network is used in this experiment. The configured
hyperparameters, such as the number of epochs and the number of iterations per
epoch, are listed in Table IV. The training runs on an NVIDIA Tesla V100S-PCI
GPU.

TABLE IV: The hyperparameters adjusted for the perspective comparison experiment. Substep |  Epochs |  Iterations per epoch |  Batch size |  Initial learning rate
---|---|---|---|---
Pretrain the supernet |  20 |  400 |  3 |  $2\cdot 10^{-2}$
Search the architecture |  30 |  400 |  2 |  $1\cdot 10^{-2}$
Train the teacher network |  249 |  1000 |  12 |  $1\cdot 10^{-2}$

####  IV-A3 RESULTS

Table V lists the predictions’ accuracies of the trained models. Strikingly,
the bird’s eye view perspective achieves an mIoU that is almost as good as the
first-person perspective. Considering the IoU of each class, the double solid
center line and the stop line reach a much higher IoU in the first-person
perspective than the bird’s eye view perspective. On the other hand, free
parking space is predicted better in the bird’s eye view perspective.
Fig. 7 shows several test predictions and their corresponding ground truth
images. Although some of the images have light reflections on the track, this
has no apparent impact on the predictions’ quality. Note that the predictions
are less accurate when the vehicle changes lanes. The left and the right lanes
are often confused during such maneuvers. Furthermore, both perspectives
achieve an inference frame rate of 247.11 FPS on the NVIDIA Jetson AGX Xavier.

Figure 7: Visual predictions of the trained FasterSeg models using the test dataset. First and second rows show the predictions and the corresponding ground truth images in the first-person perspective. Third and fourth rows illustrate the predictions and the corresponding ground truth images in the bird’s eye view perspective. Each color corresponds to a predefined class. TABLE V: The measured accuracies of the first-person and bird’s eye view models tested with uniform resolution. Class |  IoU [$\%$] |  IoU [$\%$]
---|---|---
|  first-person |  bird’s eye view
|  perspective |  perspective
unlabeled |  92.74 |  96.40
left lane |  74.94 |  78.32
right lane |  83.97 |  79.43
dashed center line |  51.57 |  53.84
double solid center line |  89.01 |  53.32
starting line |  65.71 |  60.51
stop line |  50.48 |  27.37
crosswalk |  76.26 |  88.89
free parking space |  7.49 |  45.79
free parking area |  62.27 |  56.93
mIoU |  65.44 |  64.08

###  IV-B REAL-TIME CAPABILITY AND ACCURACY IN RELATION TO THE ARITHMETIC

This experiment can be divided into two parts. The first part deals with the
analysis of the real-time capability of FasterSeg. The second part examines
the accuracy of FasterSeg concerning the arithmetic. For this purpose, two
models are trained using a dataset different from the experiment above. The
dataset consists of various resolution images from the first-person and the
bird’s eye view perspectives. The inference is conducted on the NVIDIA Jetson
AGX Xavier using the FP16 and the FP32 arithmetic. The TensorRT framework is
used to perform the inference. In the following, the used dataset and
hyperparameters are described. Finally, the results of the experiment are
presented.

####  IV-B1 DATASET

The dataset used to train the models is listed in Table VI. It consists of
synthetic and real images from different data sources with different
resolutions. The dataset is also divided into three sets as described in
IV-A1. The test set used to compute the accuracy consists of 208 real images.
Note that the dataset is significantly larger than the dataset used in the
previous experiment. The dataset is also transformed into the bird’s eye view
perspective to train the second FasterSeg model.

TABLE VI: The dataset generated for the real-time capability and the arithmetic accuracy comparison experiment. Data source |  Resolution |  Resolution |  Train |  Val |  Test
---|---|---|---|---|---
|  first-person perspective |  bird’s eye view perspective |  |  |
it:movES (real) |  1280 x 960 |  320 x 256 |  48 |  16 |  30
KITcar |  1280 x 640 |  320 x 320 |  6 |  2 |  31
ISF Löwen |  768 x 384 |  320 x 320 |  14 |  4 |  69
Spatzenhirn |  2048 x 1536 |  256 x 256 |  17 |  6 |  78
it:movES |  1280 x 960 |  320 x 256 |  18221 |  6072 |  0
(synthetic) |  |  |  |  |
Sum |  |  |  18306 |  6100 |  208

####  IV-B2 SETTINGS

The adjusted hyperparameters for the FasterSeg models are listed in Table VII.
Only the teacher network is used for this experiment. The training runs on an
NVIDIA Tesla V100S-PCI GPU.

TABLE VII: The hyperparameters adjusted for the real-time capability and the arithmetic accuracy comparison experiment. Substep |  Epochs |  Iterations per epoch |  Batch size |  Initial learning rate
---|---|---|---|---
Pretrain the supernet |  20 |  3051 |  3 |  $2\cdot 10^{-2}$
Search the architecture |  30 |  4576 |  2 |  $1\cdot 10^{-2}$
Train the teacher network |  420 |  1526 |  12 |  $1\cdot 10^{-2}$

####  IV-B3 RESULTS

The first part of the experiment consists of measuring the inference frame
rate of the bird’s eye view and the first-person FasterSeg models using the
FP16 and the FP32 arithmetic on an NVIDIA Jetson AGX Xavier board. Table VIII
lists the measurements’ results. The results demonstrate the real-time
capability of the FasterSeg model on an NVIDIA Jetson AGX Xavier board. The
bird’s eye view model reaches very high frame rate values, which concludes it
is highly dependent on the resolution. Furthermore, the frame rate using FP16
is approximately twice as high as the frame rate with FP32.
The second part investigates the impact of the used arithmetic on the achieved
accuracy. Therefore, the bird’s eye view and the first-person FasterSeg models
are compared using the FP16 and the FP32 arithmetic. Table IX illustrates the
achieved accuracies (mIoU) of the trained models. Note that the dataset used
to train the models consists of images with various resolutions from different
sources leading to relatively low accuracies. The results demonstrate that the
arithmetic has no significant influence on the accuracy.

TABLE VIII: The frame rate of the bird’s eye view and the first-person FasterSeg models using different arithmetic measured on NVIDIA Jetson AGX Xavier. |  Data source |  Resolution |  FPS FP16 |  FPS FP32
---|---|---|---|---
bird’s eye view |  Spatzenhirn |  256 x 256 |  319.85 |  190.91
it:movES |  320 x 256 |  274.11 |  153.16
KITcar |  320 x 320 |  231.13 |  118.91
ISF Löwen |  320 x 320 |  230.81 |  118.68
first- person |  ISF Löwen |  768 x 384 |  113.93 |  55.22
KITcar |  1280 x 640 |  48.66 |  20.87
it:movES |  1280 x 960 |  32.79 |  15.19
Spatzenhirn |  2048 x 1536 |  14.27 |  6.39
TABLE IX: The models accuracies measured on NVIDIA Jetson AGX Xavier. |  Data source |  Resolution |  mIoU FP16 |  mIoU FP32
---|---|---|---|---
|  |  |  [$\%$] |  [$\%$]
bird’s eye view |  Spatzenhirn |  256 x 256 |  28.0206 |  28.0234
it:movES |  320 x 256 |  25.7181 |  25.7040
KITcar |  320 x 320 |  13.1934 |  12.1047
ISF Löwen |  320 x 320 |  11.8073 |  11.8127
first- person |  ISF Löwen |  768 x 384 |  44.2960 |  44.2849
KITcar |  1280 x 640 |  41.1073 |  41.1028
it:movES |  1280 x 960 |  37.3627 |  37.3606
Spatzenhirn |  2048 x 1536 |  55.7952 |  55.7936

##  V Conclusion

In this paper, the semantic segmentation model FasterSeg was investigated
regarding the accuracy and the real-time capability in the Carolo-Cup
environment on NVIDIA Jetson AGX Xavier embedded hardware. Synthetic images,
which were generated using a semi-automated process, as well as real images
were used to train the FasterSeg model. The experimental evaluation
demonstrated that FasterSeg model reaches an accuracy of over 64 % and a frame
rate of 247.11 FPS in a Carolo-Cup environment.
A process chain was designed to facilitate the generation of labeled synthetic
data. This procedure starts with the semi-automated generation of route
layouts and trajectories. Images of a modeled Carolo-Cup environment are then
recorded, using a Gazebo simulation. Finally, the colored images of the
simulation are used to create a suitable annotated dataset for the training.
The dataset was also extended by a small amount of manually labeled real
images to improve the predictions’ accuracy.
In addition, the dataset in the first-person perspective was transformed into
the bird’s eye view perspective to train a separate FasterSeg model. A
comparison regarding the accuracy of both perspectives was performed. The
comparison showed that both perspectives achieve similar mIoUs. Note that some
classes are better predicted in one perspective or the other. Depending on the
use case and class relevance, both perspectives provide acceptable
predictions.
Furthermore, the real-time capability experiment shows that an FP16 arithmetic
achieves approximately twice the frame rate compared to an FP32 arithmetic.
However, the arithmetic of the inference had no significant impact on the
accuracy of the models.

## References

  * [1] T. Nguyen and M. Yoo, ”Fusing LIDAR sensor and RGB camera for object detection in autonomous vehicle with fuzzy logic approach,” 2021 International Conference on Information Networking (ICOIN), 2021, pp. 788-791, doi: 10.1109/ICOIN50884.2021.9334015.
  * [2] K. Min, S. Han, D. Lee, D. Choi, K. Sung and J. Choi, ”SAE Level 3 Autonomous Driving Technology of the ETRI,” 2019 International Conference on Information and Communication Technology Convergence (ICTC), 2019, pp. 464-466, doi: 10.1109/ICTC46691.2019.8939765.
  * [3] L. Bartolomei, L. Teixeira and M. Chli, ”Perception-aware Path Planning for UAVs using Semantic Segmentation,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 5808-5815, doi: 10.1109/IROS45743.2020.9341347.
  * [4] M. Hua, Y. Nan and S. Lian, ”Small Obstacle Avoidance Based on RGB-D Semantic Segmentation,” 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019, pp. 886-894, doi: 10.1109/ICCVW.2019.00117.
  * [5] Technische Universität Braunschweig, ”Carolo-Master-Cup@Home Regulations 2021,” 20 Jan 2021. [Online]. Available: https://www.tu-braunschweig.de/fileadmin/Redaktionsgruppen/Institute_Fakultaet_5/Carolo-Cup/Basic-Cup_Regulations_210120.pdf. [Accessed February 2021].
  * [6] H. Kirchner, Randomized Road Generation and 3D Visualization, TU Munich. [Online]. Available: https://github.com/tum-phoenix/drive_sim_road_generation. [Accessed May 2020].
  * [7] S. Cakir, M. Gauß, K. Häppeler, and Y. Ounajjar, Road generation to use in a simulation for semantic segmentation, Esslingen University of Applied Sciences. [Online]. Available: https://github.com/Sinop97/drive_sim_road_generation. [Accessed July 2021].
  * [8] Team Spatzenhirn, University of Ulm. [Online]. Available: https://www.uni-ulm.de/en/in/spatzenhirn/. [Accessed July 2021].
  * [9] Team ISF Löwen, Institute of Software Engineering and Automotive Informatics, TU Braunschweig. [Online]. Available: http://www.isf-loewen.de/. [Accessed July 2021].
  * [10] Team KITcar, Karlsruhe Institute of Technology. [Online]. Available: https://kitcar-team.de/. [Accessed July 2021].
  * [11] Team it:movES, Esslingen University of Applied Sciences. [Online]. Available: https://www.hs-esslingen.de/informatik-und-informationstechnik/forschung-labore/projekte/interne-projekte/. [Accessed July 2021].
  * [12] Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., … & Zitnick, C. L. (2014, September). Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740-755). Springer, Cham
  * [13] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes Dataset for Semantic Urban Scene Understanding,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3213-3223, doi: 10.1109/CVPR.2016.350.
  * [14] H. Chen, K. Sun, Z. Tian, C. Shen, Y. Huang and Y. Yan, ”BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8570-8578, doi: 10.1109/CVPR42600.2020.00860.
  * [15] D. Bolya, C. Zhou, F. Xiao and Y. J. Lee, ”YOLACT++ Better Real-Time Instance Segmentation,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 2, pp. 1108-1121, 1 Feb. 2022, doi: 10.1109/TPAMI.2020.3014297
  * [16] Y. Lee and J. Park, ”CenterMask: Real-Time Anchor-Free Instance Segmentation,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 13903-13912, doi: 10.1109/CVPR42600.2020.01392.
  * [17] D. Bolya, C. Zhou, F. Xiao and Y. J. Lee, ”YOLACT: Real-Time Instance Segmentation,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 9156-9165, doi: 10.1109/ICCV.2019.00925.
  * [18] P. Chao, C. -Y. Kao, Y. Ruan, C. -H. Huang and Y. -L. Lin, ”HarDNet: A Low Memory Traffic Network,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3551-3560, doi: 10.1109/ICCV.2019.00365.
  * [19] Papers with code, “HarDNet: A Low Memory Traffic Network,” Accessed: Jul 01, 2021. [Online]. Available: https://paperswithcode.com/paper/hardnet-a-low-memory-traffic-network
  * [20] M. Oršic, I. Krešo, P. Bevandić, and S. Šegvić, “In Defense of Pre-trained ImageNet Architectures for Real-time Semantic Segmentation of Road-driving Images,” Apr. 12, 2019, arXiv:1903.08469v2. [Online]. Available: https://arxiv.org/pdf/1903.08469v2.pdf
  * [21] C. Yu, C. Gao, J. Wang, G. Yu, C. Shen and N.Sang, “BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation,” Apr. 5, 2020, arXiv:2004.02147v1. [Online]. Available: https://arxiv.org/pdf/2004.02147v1.pdf
  * [22] W.Chen, X. Gong, X. Liu, Q. Zhang, Y. Li and Z. Wang, “FASTERSEG: SEARCHING FOR FASTER REAL-TIME SEMANTIC SEGMENTATION,” Jan. 16, 2020, arXiv:1912.10917. [Online]. Available: https://arxiv.org/pdf/1912.10917.pdf
  * [23] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N.Sang. (2018). Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 325-341).
  * [24] S. Cakir, M. Gauß, K. Häppeler, and Y. Ounajjar, How to train FasterSeg for custom objects, Esslingen University of Applied Sciences. [Online]. Available: https://github.com/Gaussianer/FasterSeg. [Accessed July 2021].
  * [25] DUONG, Tin Trung, PHAM, Cuong Cao, TRAN, Tai Huu-Phuong, et al. Near real-time ego-lane detection in highway and urban streets. In : 2016 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia). IEEE, 2016. p. 1-4.
  * [26] KIM, ZuWhan. Robust lane detection and tracking in challenging scenarios. IEEE Transactions on intelligent transportation systems, 2008, vol. 9, no 1, p. 16-26.
  * [27] S. Minaee, Y. Y. Boykov, F. Porikli, A. J. Plaza, N. Kehtarnavaz and D. Terzopoulos, ”Image Segmentation Using Deep Learning: A Survey,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2021.3059968
  * [28] Everingham, M., Eslami, S.M.A., Van Gool, L. et al. The Pascal Visual Object Classes Challenge: A Retrospective. Int J Comput Vis 111, 98?136 (2015). https://doi.org/10.1007/s11263-014-0733-5