#  GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled
Reasoning

Zhen Xiang1∗ Linzhi Zheng2 Yanjie Li3 Junyuan Hong4 Qinbin Li5 Han Xie6
Jiawei Zhang1 Zidi Xiong1 Chulin Xie1 Carl Yang6 Dawn Song5 Bo Li17
1UIUC 2Tsinghua University 3Hong Kong Polytechnic University
4UT Austin 5UC Berkeley 6Emory University 7 University of Chicago
Correspondence to Zhen Xiang<EMAIL_ADDRESS>Bo Li
<EMAIL_ADDRESS>
###### Abstract

The rapid advancement of large language models (LLMs) has catalyzed the
deployment of LLM-powered agents across numerous applications, raising new
concerns regarding their safety and trustworthiness. In addition, existing
methods for enhancing the safety of LLMs are not directly transferable to LLM-
powered agents due to their diverse objectives and output modalities. In this
paper, we propose GuardAgent, the first LLM agent as a guardrail to other LLM
agents. Specifically, GuardAgent oversees a target LLM agent by checking
whether its inputs/outputs satisfy a set of given guard requests (e.g., safety
rules or privacy policies) defined by the users. GuardAgent comprises two
steps: 1) creating a task plan by analyzing the provided guard requests, and
2) generating guardrail code based on the task plan and executing the code by
calling APIs or using external engines. In both steps, an LLM is utilized as
the core reasoning component, supplemented by in-context demonstrations
retrieved from a memory module. Such knowledge-enabled reasoning allows
GuardAgent to understand various textual guard requests and accurately
“translate” them into executable code that provides reliable guardrails.
Furthermore, GuardAgent is equipped with an extendable toolbox containing
functions and APIs and requires no additional LLM training, which underscores
its generalization capabilities and low operational overhead. In addition to
GuardAgent , we propose two novel benchmarks: an EICU-AC benchmark for
assessing privacy-related access control for healthcare agents and a Mind2Web-
SC benchmark for safety evaluation for web agents. We show the effectiveness
of GuardAgent on these two benchmarks with 98.7% and 90.0% guarding accuracy
in moderating invalid inputs and outputs for the two types of agents,
respectively. We also show that GuardAgent is able to define novel functions
in adaption to emergent LLM agents and guard requests, which underscores its
strong generalization capabilities.

Figure 1: Illustration of GuardAgent as a guardrail to a target LLM agent. The
inputs to GuardAgent include a) a set of guard requests informed by a
specification of the target agent and b) the test-time inputs and outputs of
the target agent. GuardAgent first generates an action plan following a few
shots of demonstrations retrieved from the memory. Then, a guardrail code is
generated following the action plan based on both demonstrations and a list of
callable functions. The outputs/actions of the target agent will be denied if
GuardAgent detects a violation of the guard requests.

##  1 Introduction

AI agents empowered by large language models (LLMs) have showcased remarkable
performance across diverse application domains, including finance [24],
healthcare [2, 17, 22, 18, 11], daily work [4, 5, 28, 27], and autonomous
driving [3, 8, 12]. For each user query, these agents typically employ an LLM
for task planning, leveraging the reasoning capability of the LLM with the
optional support of long-term memory from previous use cases [10]. The
proposed plan is then executed by calling external tools (e.g., through APIs)
with potential interaction with the environment [23].

Unfortunately, the current development of LLM agents primarily focuses on
their effectiveness in solving complex tasks while significantly overlooking
their potential for misuse, which can lead to harmful consequences. For
example, if misused by unauthorized personnel, a healthcare LLM agent could
easily expose confidential patient information [25]. Indeed, some LLM agents,
particularly those used in high-stakes applications like autonomous driving,
are equipped with safety controls to prevent the execution of undesired
dangerous actions [12, 6]. However, these task-specific guardrails are
hardwired into the LLM agent and, therefore, cannot be generalized to other
agents (e.g., for healthcare) with different guard requests (e.g., for privacy
instead of safety).

On the other hand, guardrails for LLMs provide input and output moderation to
detect and mitigate a wide range of potential harms [13, 9, 16, 7, 26]. This
is typically achieved by building the guardrail upon another pre-trained LLM
to contextually understand the input and output of the target LLM. More
importantly, the ‘non-invasiveness’ of guardrails, achieved through their
parallel deployment alongside the target LLM, allows for their application to
new models and harmfulness taxonomies with only minor modifications. However,
LLM agents are significantly different from LLMs, as they involve a much
broader range of output modalities and highly specific guard requests. For
instance, a web agent empowered by LLM might generate actions like clicking a
designated button on a webpage [27]. The guard requests here could involve
safety rules that prohibit certain users (e.g., those under a certain age)
from purchasing specific items (e.g., alcoholic beverages). Clearly, existing
guardrails designed solely to moderate the textual inputs and outputs of LLMs
cannot address such intricate guard requests.

In this paper, we present the first study on guardrails for LLM agents. We
propose GuardAgent, the first generalizable framework that uses an LLM agent
to safeguard other LLM agents (referred to as ‘target agents’ henceforth) by
adhering to diverse real-world guard requests from users, such as safety rules
or privacy policies. The deployment of GuardAgent requires the prescription of
a set of textural guard requests informed by a specification of the target
agent (e.g., the format of agent output and logs). During the inference, user
inputs to the target agent, along with associated outputs and logs, will be
provided to GuardAgent for examination to determine whether the guard requests
are satisfied or not. Specifically, GuardAgent first uses an LLM to generate
an action plan based on the guard requests and the inputs and outputs of the
target agent. Subsequently, the LLM transforms the action plan into a
guardrail code, which is then executed by calling an external engine. For both
the action plan and the guardrail code generation, the LLM is provided with
related demonstrations retrieved from a memory module, which archives inputs
and outputs from prior use cases. Such knowledge-enabled reasoning is the
foundation for GuardAgent to understand diverse guard requests for different
types of LLM agents. The design of our GuardAgent offers three key advantages.
Firstly, GuardAgent can be easily generalized to safeguard new target agents
by simply uploading new functions to its toolbox. Secondly, GuardAgent
provides guardrails by code generation and execution, which is more reliable
than guardrails solely based on natural language. Thirdly, GuardAgent employs
LLMs by in-context learning, enabling direct utilization of off-the-shelf LLMs
without the need for additional training.

Before introducing GuardAgent in Sec. 4, we investigate diverse guard requests
for different types of LLM agents and propose two novel benchmarks in Sec. 3.
The first benchmark, EICU-AC, is designed to assess the effectiveness of
access control for LLM agents for healthcare. The second benchmark, Mind2Web-
SC, evaluates safety control for LLM-powered web agents. These two benchmarks
are used to evaluate our GuardAgent in our experiments in Sec. 5. Note that
the two types of guard requests considered here – access control and safety
control – are closely related to privacy and safety, respectively, which are
critical perspectives of AI trustworthiness [19]. Our technical contributions
are summarized as follows:

  * •

We propose GuardAgent, the first LLM agent framework providing guardrails to
other LLM agents via knowledge-enabled reasoning in order to address diverse
user guard requests.

  * •

We propose a novel design for GuardAgent, which comprises knowledge-enabled
task planning using in-context demonstrations, followed by guardrail code
generation involving an extendable array of functions. Such design endows
GuardAgent with strong generalization capabilities, reliable guardrail
generation, and no need for additional training.

  * •

We create two benchmarks, EICU-AC and Mind2Web-SC, for evaluating privacy-
related access control for healthcare agents and safety control for web
agents, respectively.

  * •

We show that GuardAgent effectively provides guardrails to 1) an EHRAgent for
healthcare with a 98.7% guarding accuracy in access control and 2) a SeeAct
web agent with a 90.0% guarding accuracy in safety control. We also
demonstrate the capabilities of GuardAgent in defining new functions during
guardrail code generation and execution.

##  2 Related Work

LLM agents refer to AI agents that use LLMs as their central engine for task
understanding and planning and then execute the plan by interacting with the
environment (e.g., by calling third-party APIs) [21]. Such fundamental
difference from LLMs (with purely textual outputs) enables LLM agents to be
deployed in diverse applications, including finance [24], healthcare [2, 17,
22, 18, 11], daily work [4, 5, 28, 27], and autonomous driving [3, 8, 12]. LLM
agents are also commonly equipped with a retrievable memory module, allowing
them to perform knowledge-enabled reasoning to handle different tasks within
its application domain [10]. Our GuardAgent is a typical LLM agent, but with
different objectives from existing agents, as it is the first one to safeguard
other LLM agents.

LLM-based guardrails belong to a family of moderation approaches for
harmfulness mitigation [25, 15]. Traditional guardrails were operated as
classifiers trained on categorically labeled content [13, 9], while recently,
guardrails based on LLMs with broader contextual understanding have been
developed and shown strong generalization capabilities. However, existing
guardrails for LLMs, either ‘model guarding models’ ([16, 7, 26]) or ‘agent
guarding models’ ([1]), are designed for harmfulness defined on natural
language. They cannot be directly used to safeguard LLM agents with diverse
output modalities. In this paper, we propose GuardAgent, the first ‘agent
guarding agents’ framework, and show its advantage over ‘model guarding
agents’ approaches in our experiments.

##  3 Safety Requests for Diverse LLM Agents

Before introducing our GuardAgent, we investigate safety requests for
different types of LLM agents in this section. We focus on two representative
LLM agents: an EHRAgent for healthcare and a web agent SeeAct. In particular,
EHRAgent represents LLM agents for high-stake tasks, while SeeAct represents
generalist LLM agents for diverse tasks. We briefly review these two agents,
their designated tasks, and their original evaluation benchmarks. More
importantly, we propose two novel benchmarks for different safety requests: 1)
EICU-AC, which assesses access control for healthcare agents like EHRAgent,
and 2) Mind2Web-SC, which evaluates safety control for web agents like SeeAct.
Then, we conduct a preliminary study to test ‘invasive’ approaches for access
control and safety control, which are based on naive instructions injected
into the system prompts of EHRAgent and SeeAct, respectively; their
ineffectiveness and poor generalization motivate the need for our GuardAgent.

Figure 2: An example from EICU-AC (left) and an example from Mind2Web-SC
(right).

###  3.1 EHRAgent and EICU-AC Benchmark

#### EHRAgent

EHRAgent is designed to respond to healthcare-related queries by generating
code to retrieve and analyze data from provided databases [17]. EHRAgent has
been evaluated and shown decent performance on several benchmarks, including
an EICU dataset containing questions regarding the clinical care of ICU
patients (see Fig. 2 for example) and 10 relevant databases [14]. Each
database contains several types of patient information stored in different
columns. In practical healthcare systems, it is crucial to restrict access to
specific databases based on user identities. For example, personnel in general
administration should not have access to patient diagnosis details. Thus, LLM
agents for healthcare, such as EHRAgent, should be able to deny requests for
information from the patient diagnosis database when the user is from the
general administration. In essence, these LLM agents should incorporate access
controls to safeguard patient privacy.

#### Proposed EICU-AC benchmark

In this paper, we create an EICU-AC benchmark from EICU to evaluate Access
Control approaches for EHRAgent (and potentially other healthcare agents that
require database retrieval). We define three roles for the user of EHRAgent
(and other similar target agents): ‘physician’, ‘nursing’, and ‘general
administration’. The access control being evaluated is supposed to ensure that
each identity has access to only a subset of databases and columns of the EICU
benchmark. We generate the ground truth access permission for each role by
querying ChatGPT (see App. A.1 for more details). While generic access control
approaches should be invariant to the specific roles and their access
permissions, we have made these choices to simulate practical healthcare
scenarios. Then, each example in EICU-AC is designed to include the following
information: 1) a healthcare-related question and the correct answer, 2) the
databases and the columns required to answer the question, 3) a user
identity/role, 4) a binary label ‘0’ if all required databases and columns are
accessible to the given identity or ‘1’ otherwise, and 5) the required
databases and columns inaccessible to the identity if the label is ‘1’. An
illustration of a generated EICU-AC example is shown in Fig. 2.

In particular, all questions in EICU-AC are sampled or adapted from the EICU
dataset. We keep questions from EICU that are correctly answered by EHRAgent
using GPT-4 (at temperature zero) as the core LLM so that the evaluation using
our benchmark will mainly focus on access control without much influence from
the task performance. Initially, we generate three EICU-AC examples from each
of these questions by assigning them the three roles respectively. After
labeling each example based on the ground truth accessibility of its assigned
role, we find for all three identities that the two labels are highly
imbalanced. Thus, for each identity, we remove some of the generated examples
while adding new ones to achieve a relative balance between the two labels
(see more details in App. A.2). Ultimately, our EICU-AC contains 52, 57, and
45 examples labeled to ‘0’ for ‘physician’, ‘nursing’, and ‘general
administration’, respectively, and 46, 55, and 61 examples labeled to ‘1’ for
the three roles, respectively. Moreover, among these 316 examples, there are
226 unique questions spanning 51 different ICU information categories, which
underscores the diversity of our EICU-AC.

###  3.2 SeeAct and Mind2Web-SC Benchmark

#### SeeAct

SeeAct is a generalist web agent that follows natural language instructions to
complete tasks on any given website by sequentially generating actions,
including clicking on a button, typing specific texts, etc. (see Fig. 2 for
example) [27]. In the original paper, SeeAct is evaluated on the Mind2Web
benchmark containing over 2,000 complex web tasks spanning 137 websites across
31 domains (e.g., car rental, shopping, entertainment, etc.) [4]. However, it
is essential for practical web agents like SeeAct to integrate safety controls
that restrict certain actions for specific users. For example, in most regions
of the world, a driver’s license is required for car rental.

#### Proposed Mind2Web-SC benchmark

We create a Mind2Web-SC benchmark to evaluate Safety Control applicable to
SeeAct and other web agents that operate based on action generation. The
objective of safety control is to ensure that the agent obeys six rules we
created based on common web regulations and regional conventions: 1) user must
be a member to shop, 2) unvaccinated user cannot book a flight, 3) user
without a driver’s license cannot buy or rent a car, 4) user aged under 18
cannot book a hotel, 5) user must be in certain countries to search
movies/music/video, 6) user under 15 cannot apply for jobs. Again, these rules
are proposed solely for evaluation purposes and do not reflect the personal
views or attitudes of the authors.

The examples in Mind2Web-SC are created by the following steps. First, we
obtain all tasks with correct action prediction by SeeAct (using LLaVA-1.5 as
the core LLM) from the travel, shop, and entertainment domains of the test set
of Mind2Web. Second, for each task, we randomly create a user profile
containing ‘age’ in integer and ‘domestic’, ‘dr_license’, ‘vaccine’, and
‘membership’, all boolean (see the right of Fig. 2). Note that each of these
six user information categories is non-trivial, as it is related to at least
one of the six safety rules we created. Third, we manually label each example
based on the task and the user information. If the task itself is not related
to any of the six rules, the example will be labeled to ‘0’ for ‘action
permitted’. If the task is related to at least one of the rules (e.g. the one
for car rental), we check the user information and will label the example to
‘1’ for ‘action denied’ if the rule is violated (e.g. ‘dr_license’ is ‘false’)
and ‘0’ otherwise. For each example labeled to ‘1’, the violated rules are
also included in our benchmark. Finally, we balance the two classes by
creating additional examples (based on existing tasks but with different user
information) while removing some examples with tasks irrelevant to any of the
rules (see details in App. B). The created Mind2Web-SC benchmark contains 100
examples in each class with only unique tasks within the class.

Table 1: Access control on EHRAgent and safety control on SeeAct based on system instructions are ineffective on EICU-AC and Mind2Web-SC; therefore, a new guardrail method is needed. |  LPP $\uparrow$ |  LPR $\uparrow$ |  CCA $\uparrow$ |  FRA $\uparrow$
---|---|---|---|---
EHRAgent + EICU-AC | 76.6 | 90.7 | 50.0 | 3.2
SeeAct + Mind2Web-SC | 95.1 | 58.0 | 58.0 | 71.0

###  3.3 Preliminary Guardrails on the Two Benchmarks

#### Naive ‘invasive’ baselines

We use the EICU-AC benchmark to evaluate a naive approach that hardwires the
access control into the system prompt of EHRAgent by specifying the three
roles with their accessible databases and columns. During the evaluation, this
modified EHRAgent will be provided with both the role and the query of the
EICU-AC examples. Its system prompt will include instructions to display a
‘denial message’ along with the inaccessible databases and columns for the
given role, if there are any. Similarly, we incorporate textual instructions
for safety checks into the system prompt of SeeAct and evaluate it on
Mind2Web-SC. If any of the rules are violated for the given user profile, the
safety-enforced SeeAct is supposed to print a ‘denial message’ with the
violated rules. Details about the system prompts for the two agents equipped
with the naive ‘invasive’ guardrails are deferred to App. C.

#### Metrics

We consider four evaluation metrics shared by both benchmarks: label
prediction precision (LPP), label prediction recall (LPR), comprehensive
control accuracy (CCA), and final response accuracy (FRA), all in percentage.
Both LPP and LPR are calculated over all examples in each dataset to measure
the overall label prediction efficacy, where a prediction of label ‘1’ is
counted only if the ‘denial message’ appears. CCA considers all examples with
ground truth labeled ‘1’. It is defined as the percentage of these examples
being correctly predicted to ‘1’ and with all inaccessible databases and
columns (for EICU-AC) or all violated rules (for Mind2Web-SC) successfully
detected. In contrast, FRA considers all examples with ground truth labeled
‘0’. It is defined as the percentage of these examples being correctly
predicted to ‘0’ and with the agent responses correctly.

#### Results

As shown in Tab. 1, the two naive baselines fail in their designated tasks,
exhibiting either low precision or recall in label prediction. Specifically,
the naive access control for EHRAgent is overly strict, resulting in an
excessive number of false positives. Conversely, the naive safety control for
SeeAct fails to reject many unsafe actions, leading to numerous false
negatives. Moreover, the ‘invasion’ that introduces additional tasks imposes
heavy burdens on both agents, significantly degrading the performance on their
designated tasks, particularly for EHRAgent (which achieves only 3.2% end-to-
end accuracy on negative examples as measured by FRA). Finally, despite their
poor performance, both naive guardrail approaches are hardwired to the agent,
making them non-transferable to other LLM agents with different designs. These
shortcomings highlight the need for our GuardAgent, which is both effective
and generalizable in safeguarding diverse LLM agents.

##  4 GuardAgent Framework

In this section, we introduce GuardAgent with three key features: 1)
generalizable – the memory and toolbox of GuardAgent can be easily extended to
address new target agents with new guard requests; 2) reliable – outputs of
GuardAgent are obtained by successful code execution; 3) training-free –
GuardAgent is in-context-learning-based and does not need any LLM training.

###  4.1 Overview of GuardAgent

The intended user of GuardAgent is the developer or administrator of a target
LLM agent who seeks to implement guardrails on it. The mandatory inputs to
GuardAgent are all textual, including a set of guard requests $I_{r}$, a
specification $I_{s}$ of the target agent, inputs $I_{i}$ to the target agent,
and the output logs $I_{o}$ of the target agent corresponding to $I_{i}$.
Here, $I_{r}$ is informed by $I_{s}$, which includes the functionality of the
target agent, the content of the inputs and output logs, their formats, etc.
The objective of GuardAgent is to check whether $I_{i}$ and $I_{o}$ satisfy
the guard requests $I_{r}$ and then produce a label prediction $O_{l}$, where
$O_{l}=0$ means the guard requests are satisfied and $O_{l}=1$ otherwise. The
outputs or actions proposed by the target agent will be admitted by GuardAgent
if $O_{l}=0$ or denied if $O_{l}=1$. If $O_{l}=1$, GuardAgent should also
output the detailed reasons $O_{d}$ (e.g., by printing out the inaccessible
databases and columns for EICU-AC) for potential further actions. For example,
severe rule violations for some use cases may require judicial intervention.

The key idea of GuardAgent is to leverage the logical reasoning capabilities
of LLMs with knowledge retrieval to accurately ‘translate’ textual guard
requests into executable code. Correspondingly, the pipeline of GuardAgent
comprises two major steps (see Fig. 1). In the first step (Sec. 4.2), a step-
by-step action plan is generated by prompting an LLM with the above-mentioned
inputs to GuardAgent. In the second step 4.3), we prompt the LLM with the
action plan and a set of callable functions to get a guardrail code, which is
then executed by calling an external engine. A memory module is available in
both steps to retrieve in-context demonstrations.

###  4.2 Task Planning

The objective for task planning is to generate a step-by-step action plan $P$
from the inputs to GuardAgent. A naive design is to prompt a foundation LLM
with $[I_{p},I_{s},I_{r},I_{i},I_{o}]$, where $I_{p}$ contains carefully
designed planning instructions that 1) define each input to GuardAgent , 2)
state the guardrail task (i.e., checking if $I_{r}$ is satisfied by $I_{i}$
and $I_{o}$), and 3) guide the generation of action steps (see Fig. 8 in App.
D for a concrete example). However, understanding the complex guard requests
and incorporating them with the target agent remains a challenging task for
existing LLMs.

We address this challenge by allowing GuardAgent to retrieve demonstrations
from a memory module that archives target agent inputs and outputs from past
use cases. Here, an element $D$ in the memory module is denoted by
$D=[I_{i,D},I_{o,D},P_{D},C_{D}]$, where $I_{i,D}$ and $I_{o,D}$ are the
target agent inputs and outputs respectively, $P_{D}$ contains the action
steps, and $C_{D}$ contains the guardrail code. Retrieval is based on the
similarity between the current target agent inputs and outputs and those from
the memory. Specifically, we retrieve $k$ demonstrations by selecting $k$
elements from the memory with the smallest Levenshtein distance
$L([I_{i,D},I_{o,D}],[I_{i},I_{o}])$. Then the action plan is obtained by
$P={\rm
LLM}([I_{p},I_{s},I_{r},[I_{i,D_{1}},I_{o,D_{1}},P_{D_{1}}],\cdots,[I_{i,D_{k}},I_{o,D_{k}},P_{D_{k}}],I_{i},I_{o}])$.
Note that the guardrail code in each demonstration has been removed for the
brevity of the prompt.

In the cases where GuardAgent is applied to a new LLM agent for some specific
guard requests, we also allow the user of GuardAgent to manually inject
demonstrations into the memory module. In particular, we request the action
plan in each demonstration provided by the user to contain four mandatory
steps, denoted by $P_{D}=[p_{1,D},p_{2,D},p_{3,D},p_{4,D}]$, where the four
steps form a chain-of-thought [20]. In general, $p_{1,D}$ summarizes guard
requests to identify the keywords, such as ‘access control’ with three roles,
‘physician’, ‘nursing’, and ‘general administration’ for EICU-AC. Then,
$p_{2,D}$ filters information in the guard request that is related to the
target agent input, while $p_{3,D}$ summarizes the target agent output log and
locates related content in the guard request. Finally, $p_{4,D}$ instructs
guardrail code generation to compare the information obtained in $p_{2,D}$ and
$p_{3,D}$, as well as the supposed execution engine. Example action plans are
shown in Fig. 13 of App. G.

###  4.3 Guardrail Code Generation and Execution

The goal of this step is to generate a guardrail code $C$ based on the action
plan $P$. Once generated, $C$ is executed through the external engine $E$
specified in the action plan. However, guardrail code generated by directly
prompting an LLM with the action plan $P$ and straightforward instructions may
not be reliably executable. One of our key designs to address this issue is to
adopt more comprehensive instructions that include a list $\mathcal{F}$ of
callable functions with specification of their input arguments. The
definitions of these functions are stored in the toolbox of GuardAgent, which
can be easily extended by users through code uploading to address new guard
requests and target agents. The LLM is instructed to use only the provided
functions for code generation; otherwise, it easily makes up non-existent
functions during code generation.

Furthermore, we utilize past examples retrieved from memory, employing the
same approach used in task planning, to serve as demonstrations for code
generation. Thus, we have $C={\rm
LLM}(I_{c}(\mathcal{F}),D_{1},\cdots,D_{k},I_{i},I_{o},P)$, where
$I_{c}(\mathcal{F})$ are the instructions based on the callable functions in
$\mathcal{F}$ and $D_{1},\cdots,D_{k}$ are the retrieved demonstrations. The
outputs of GuardAgent are obtained by executing the generated code, i.e.,
$(O_{l},O_{d})=E(C,\mathcal{F})$. Finally, we adopt the debugging mechanism
proposed by Shi et al. [17], which invokes an LLM to analyze any error
messages that may arise during execution to enhance the reliability of the
generated code.

##  5 Experiments

In Sec. 5.2, we show the effectiveness of GuardAgent in safeguarding EHRAgent
on EICU-AC and SeeAct on Mind2Web-SC with 98.7% and 90.0% label prediction
accuracies, respectively. We illustrate through a case study that the
advantage of GuardAgent over ‘model guarding agents’ approaches is attributed
to the more reliable guardrail by code generation and execution. In Sec. 5.3,
we conduct ablation studies to show 1) GuardAgent performs similarly well for
most of the roles in EICU-AC and rules in Mind2Web-SC, allowing it to handle
guard requests with high complexity, and 2) GuardAgent requires only a few
shots of demonstrations. In Sec. 5.4, we demonstrate that GuardAgent may
define necessary functions based on guard requests, highlighting its ability
to generalize to new guard requests. Additionally, we find that LLMs, such as
GPT-4, tend to generate code-based guardrails (albeit mostly inexecutable)
even when not provided with specific instructions for code generation and
execution.

###  5.1 Setup

#### Datasets and agents

We test GuardAgent on EICU-AC and Mind2Web-SC with EHRAgent and SeeAct (using
their original settings) as the target agents, respectively. The role and
question from each EICU-AC example are inputs to EHRAgent, and the output logs
include the reasoning steps, the generated code, and the final answer produced
by EHRAgent. The inputs to SeeAct contain the task and user information from
each example in Mind2Web-SC, and the output logs include the predicted action
and the reasoning by SeeAct. Example inputs ($I_{i}$) and output logs
($I_{o}$) of the two target agents (which are the inputs to GuardAgent) are
shown in App. D. Other inputs to GuardAgent, including the specifications of
the two target agents ($I_{s}$), the guard requests associated with the two
benchmarks ($I_{r}$), and the planning instructions ($I_{p}$), are also shown
in App. D due to space limitations.

#### Settings of GuardAgent

In the main experiments, we set the number of demonstrations to $k=1$ and
$k=3$ for EICU-AC and Mind2Web-SC, respectively. Other choices will be
considered in our ablation study in Sec. 5.3. For each benchmark, we
initialize the memory of GuardAgent by $k$ manually created demonstrations
(see App. G for example). We use GPT-4 version 2024-02-01 with temperature
zero as the core LLM of GuardAgent. We use Python as the default code
execution engine, with two initial functions in the toolbox, ‘CheckAccess’ and
‘CheckRules’, which are defined in App. E. Note that users of GuardAgent can
easily upload new functions or engines into the toolbox. Finally, we allow
three debugging iterations, though in most cases, the guardrail code generated
by GuardAgent is directly executable.

#### Baselines

Since GuardAgent is the first LLM agent designed to safeguard other agents, we
compare it with baselines using models to safeguard agents. Here, we consider
GPT-4 version 2024-02-01 and Llama3-70B as the guardrail models111Approaches
for ‘model guarding models’, such as LlamaGuard designed to detect predefined
unsafe categories [7], are not considered here due to their completely
different objectives.. We create comprehensive prompts containing high-level
instructions $I^{\prime}_{p}$ adapted from the one for GuardAgent, the same
number of demonstrations as for GuardAgent but without guardrail code
generation, denoted by $D^{\prime}_{1},\cdots,D^{\prime}_{k}$, and the same
set of inputs as for GuardAgent. However, neither baselines involve the memory
module as our GuardAgent does; they use a fixed set of demonstrations during
the evaluation. Example prompts for both benchmarks are shown in App. H. Then
the outputs of the guardrail models for the baselines are obtained by
$(O_{l},O_{d})={\rm
LLM}(I^{\prime}_{p},I_{s},I_{r},D^{\prime}_{1},\cdots,D^{\prime}_{k},I_{i},I_{o})$.

#### Evaluation metrics

We use the two label prediction metrics, LPP and LPR, and the CCA metric, all
defined in Sec. 3.3. The FRA metric is not considered here since all
guardrails being evaluated will not affect the normal operation of the target
agent when the alarm is not triggered. In addition, we report the label
prediction accuracy (LPA, a.k.a. guarding accuracy), defined over all examples
in each dataset, as the overall metric for the guardrail performance.

Table 2: Performance of GuardAgent in safeguarding EHRAgent on EICU-AC and SeeAct on Mind2Web-SC, compared with two model-based baselines with GPT-4 and Llama3, respectively. | EHRAgent on EICU-AC | SeeAct on Mind2Web-SC
---|---|---
|  LPA $\uparrow$ |  LPP $\uparrow$ |  LPR $\uparrow$ |  CCA $\uparrow$ |  LPA $\uparrow$ |  LPP $\uparrow$ |  LPR $\uparrow$ |  CCA $\uparrow$
Llama3 | 92.1 | 95.4 | 88.9 | 41.4 | 76.5 | 93.4 | 57.0 | 57.0
GPT-4 | 97.5 | 95.3 | 100.0 | 67.9 | 82.5 | 100.0 | 65.0 | 65.0
GuardAgent | 98.7 | 100.0 | 97.5 | 97.5 | 90.0 | 100.0 | 80.0 | 80.0

Figure 3: Left: A failure case of the GPT-4 baseline where the same column
name (‘patientunitstayid’) shared by different databases cannot be effectively
distinguished. Right: A failure case of GuardAgent where a rule violation is
not detected due to the overwhelming details in the query.

###  5.2 Guardrail Performance

In Tab. 2, we show the performance of GuardAgent compared with the baselines
using our comprehensive evaluation metrics. GuardAgent achieves better LPAs
than the two baselines with also clear gaps in CCAs, showing the advantage of
‘agent guarding agents’ over ‘model guarding agents’. We attribute this
advantage to our design of reasoning-based code generation and execution,
which is clearly infeasible by guardrail models. In many failure cases of
GPT-4 on EICU-AC, we found that guardrails based on natural language cannot
effectively distinguish column names if they are shared by different
databases. For example, in Fig. 3, the entire database ‘vitalperiodic’ that
contains a column named ‘patientunitstayid’ is not accessible to ‘general
administration’, while the column with the same name in the database ‘patient’
is accessible to the same role. In this case, the model-based guardrail using
GPT-4 fails to determine the column ‘patientunitstayid’ in the database
‘vitalperiodic’ as ‘inaccessible’. In contrast, our GuardAgent based on code
generation accurately converts each database and its columns into a
dictionary, effectively avoiding such ambiguity in column names.

On the right of Fig. 3, we show a typical failure case of GuardAgent where the
violated rule is undetected. We found that the query failed to be connected to
the designated rule in the first step of the chain-of-thought reasoning during
task planning, possibly due to the overwhelming details in the query. However,
this issue can be mitigated by involving more demonstrations with better
linguistic diversity, or using more powerful LLM as the core reasoning step.

Table 3: Breakdown of GuardAgent results over the three roles in EICU-AC and the six rules in Mind2Web-SC. GuardAgent performs uniformly well for all roles and rules except for rule 5 related to movies, music, and videos. | EHRAgent on EICU-AC | SeeAct on Mind2Web-SC
---|---|---
| physician | nursing | GA | rule 1 | rule 2 | rule 3 | rule 4 | rule 5 | rule 6
LPA $\uparrow$ | 97.9 | 98.2 | 100.0 | 89.5 | 91.7 | 87.5 | 83.3 | 52.4 | 83.3
CCA $\uparrow$ | 95.7 | 96.4 | 100.0 | 89.5 | 91.7 | 87.5 | 83.3 | 52.4 | 83.3

###  5.3 Ablation Studies

#### Breakdown results

In Tab. 3, we show LPA and CCA of GuardAgent for a) EHRAgent for each role of
EICU-AC and b) SeeAct for each rule of EICU-AC (by only considering positive
examples). In general, GuardAgent performances uniformly well for the three
roles in EICU-AC and the six rules in Mind2Web-SC except for rule 5 related to
movies, music, and videos. We find that all the failure cases for this rule
are similar to the one illustrated in Fig. 3 where the query cannot be related
to the rule during reasoning. Still, GuardAgent demonstrates relatively strong
capabilities in handling complex guard requests with high diversity.

#### Influence of number of demonstrations

We vary the number of demonstrations used by GuardAgent and show the
corresponding LPAs and CCAs in Fig. 4. The results show that GuardAgent can
achieve descent guardrail performance with very few shots of demonstrations.

Figure 4: Performance of GuardAgent with different numbers of demonstrations
on EICU-AC and Mind2Web-SC. GuardAgent is effective with very few
demonstrations.

###  5.4 Code-Based Guardrail is the Natural Preference of LLMs, but Tools are
Needed

We consider a challenging task where GuardAgent is instructed to generate
guardrail code, but is provided with neither a) the functions needed for the
guard requests nor b) demonstrations for guardrail code generation.
Specifically, the guardrail code is now generated by $C^{\prime}={\rm
LLM}(I_{c}(\mathcal{F}^{\prime}),I_{i},I_{o},P)$, where $\mathcal{F}^{\prime}$
represents the toolbox without the required functions. In this case,
GuardAgent either defines the required functions or produces procedural code
towards the same goal (see App. H for an example guardrail function generated
by GuardAgent), and has achieved a 90.8% LPA with a 96.1% CCA on EICU-AC.
These results support the need for the list of callable functions and the
demonstrations as our key design for the code generation step. They also
demonstrate a decent zero-shot generalization capability of GuardAgent to
address new guard requests.

Moreover, we consider an even more challenging guardrail task. We use the
GPT-4 model to safeguard EHRAgent on EICU-AC, but remove all instructions
related to code generation. In other words, the LLM has to figure out its way,
either with or without code generation, to provide a guardrail. Interestingly,
we find that for 68.0% examples in EICU-AC, the LLM chose to generate a code-
based guardrail (though mostly inexecutable). This result shows the intrinsic
tendency of LLMs to utilize code as a structured and precise method for
guardrail, supporting our design of GuardAgent based on code generation.

##  6 Conclusion

In this paper, we present the first study on guardrails for LLM agents to
address diverse user safety requests. We propose GuardAgent, the first LLM
agent framework designed to safeguard other LLM agents. GuardAgent leverages
knowledge-enabled reasoning capabilities of LLMs to generate a task plan and
convert it into a guardrail code. It is featured by the generalization
capabilities to new guardrail requests, the reliability of the code-based
guardrail, and the low computational overhead. In addition, we propose two
benchmarks for evaluating privacy-related access control and safety control of
LLM agents for healthcare and the web, respectively. We show that GuardAgent
outperforms ‘model guarding agent’ baselines on these two benchmarks and the
code generalization capabilities of GuardAgent under zero-shot settings.

## References

  * [1] Guardrails AI.  https://www.guardrailsai.com/, 2023.
  * [2] Mahyar Abbasian, Iman Azimi, Amir M. Rahmani, and Ramesh Jain.  Conversational health agents: A personalized llm-powered agent framework, 2024.
  * [3] Can Cui, Zichong Yang, Yupeng Zhou, Yunsheng Ma, Juanwu Lu, Lingxi Li, Yaobin Chen, Jitesh Panchal, and Ziran Wang.  Personalized autonomous driving with large language models: Field experiments, 2024.
  * [4] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su.  Mind2web: Towards a generalist agent for the web, 2023.
  * [5] Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust.  A real-world webagent with planning, long context understanding, and program synthesis.  In The Twelfth International Conference on Learning Representations, 2024.
  * [6] Wencheng Han, Dongqian Guo, Cheng-Zhong Xu, and Jianbing Shen.  Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving, 2024.
  * [7] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa.  Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.
  * [8] Ye Jin, Xiaoxi Shen, Huiling Peng, Xiaoan Liu, Jingli Qin, Jiayang Li, Jintao Xie, Peizhong Gao, Guyue Zhou, and Jiangtao Gong.  Surrealdriver: Designing generative driver agent simulation framework in urban contexts based on large language model, 2023.
  * [9] Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman.  A new generation of perspective api: Efficient multilingual character-level transformers.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
  * [10] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
  * [11] Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu.  Agent hospital: A simulacrum of hospital with evolvable medical agents, 2024.
  * [12] Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang.  A language agent for autonomous driving.  2023\.
  * [13] Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng.  A holistic approach to undesired content detection in the real world.  In AAAI, 2023.
  * [14] Tom J Pollard, Alistair E W Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi.  The eicu collaborative research database, a freely available multi-center database for critical care research.  Scientific Data, 2018.
  * [15] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson.  Fine-tuning aligned language models compromises safety, even when users do not intend to!  In The Twelfth International Conference on Learning Representations, 2024.
  * [16] Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen.  NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails.  In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, December 2023.
  * [17] Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce Ho, Carl Yang, and May D. Wang.  Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records, 2024.
  * [18] Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S Corrado, Yossi Matias, Alan Karthikesalingam, and Vivek Natarajan.  Towards conversational diagnostic ai, 2024.
  * [19] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al.  Decodingtrust: A comprehensive assessment of trustworthiness in gpt models.  2023\.
  * [20] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  In Advances in Neural Information Processing Systems, 2022.
  * [21] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui.  The rise and potential of large language model based agents: A survey, 2023.
  * [22] Qisen Yang, Zekun Wang, Honghui Chen, Shenzhi Wang, Yifan Pu, Xin Gao, Wenhao Huang, Shiji Song, and Gao Huang.  Llm agents for psychology: A study on gamified assessments, 2024.
  * [23] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  ReAct: Synergizing reasoning and acting in language models.  In International Conference on Learning Representations (ICLR), 2023.
  * [24] Yangyang Yu, Haohang Li, Zhi Chen, Yuechen Jiang, Yang Li, Denghui Zhang, Rong Liu, Jordan W. Suchow, and Khaldoun Khashanah.  Finmem: A performance-enhanced llm trading agent with layered memory and character design, 2023.
  * [25] Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Li Fangqi, Zhuosheng Zhang, Rui Wang, and Gongshen Liu.  R-judge: Benchmarking safety risk awareness for LLM agents.  In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
  * [26] Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, and Bo Li.  Rigorllm: Resilient guardrails for large language models against undesired content.  In ICML, 2024.
  * [27] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su.  Gpt-4v(ision) is a generalist web agent, if grounded.  arXiv preprint arXiv:2401.01614, 2024.
  * [28] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al.  Webarena: A realistic web environment for building autonomous agents.  arXiv preprint arXiv:2307.13854, 2023.

## Limitations

While GuardAgent performs well on the two benchmarks with also evidence of its
generalization capabilities, it requires the core LLM to have descent
reasoning capabilities. This limitation is due to the complexity of both the
guardrail tasks and the target agent to be safeguarded. However, this
limitation can be mitigated as current LLMs are becoming more and more
powerful in reasoning.

## Broader Impacts

We propose GuardAgent with potentially positive social impacts. GuardAgent is
the first LLM agent framework that safeguards other LLM agents. GuardAgent
directly addresses the safety and trustworthiness concerns of LLM agents and
will potentially inspire more advanced guardrail approaches for LLM agents.

##  Appendix A Details About the EICU-AC Benchmark

###  A.1 Role-Based Access Permission

For the EICU-AC benchmark, we consider three roles: ‘physician’, ‘nursing’,
and ‘general administration’. These roles are selected based on our
understanding of the ICU environment. Although various other roles exist, we
focus on these three roles due to their prevalence, ensuring sufficient
queries relevant to each role when creating the benchmark.

(a) List of all databases and columns.

(b) Databases and columns accessible by ‘physician’.

(c) Databases and columns accessible by ‘nursing’.
empty space

(d) Databases and columns accessible by ‘general administration’.

Figure 5: Databases and columns accessible to the three roles defined for
EICU-AC, and the complete list of databases and columns for reference.
Accessible columns and inaccessible columns for each role are marked in green
while inaccessible ones are shaded.

For each role, we select a subset of accessible databases and columns from the
EICU benchmark, as shown in Fig. 5. Our selection rule is to query ChatGPT
about the access permission for the three roles over each database. For
example, for the ‘diagnosis’ database with four columns, ‘patientunitstayid’,
‘icd9code’, ‘diagnosisname’, and ‘diagnosistime’, we query ChatGPT using the
prompt shown in Fig. 6. ChatGPT responds with the recommended access
permission (‘full access’, ‘limited access’, or ‘no access’) for each role to
each of the four columns. Here, we follow all ‘full access’ and ‘no access’
recommendations by ChatGPT. For ‘limited access’, we set it to ‘no access’ if
it is recommended for ‘physician’ or ‘nursing’; if it is recommended for
‘general administration’, we set it to ‘full access’. This is to ensure both
‘physician’ and ‘nursing’ roles have sufficient inaccessible databases so that
there will be sufficient queries that should be denied in the ground truth (to
achieve relatively balanced labeling for both roles).

Figure 6: Our prompt to ChatGPT for the access permission for the three roles
to the ‘diagnosis’ database (with four columns, ‘patientunitstayid’,
‘icd9code’, ‘diagnosisname’, and ‘diagnosistime’), and the responses of
ChatGPT.

###  A.2 Sampling from EICU

As mentioned in the main paper, each example in EICU-AC contains 1) a
healthcare-related question and the correct answer, 2) the databases and the
columns required to answer the question, 3) a user identity, 4) a binary label
(either ‘0’ for ‘access granted’ and ‘1’ for ‘access denied’), and 5)
databases and the columns required to answer the question but not accessible
for the given role (if there are any). The examples in EICU-AC are created by
sampling from the original EICU dataset following the steps below. First, from
the 580 test examples in EICU, we obtain 183 examples that are correctly
responded to by EHRAgent with GPT-4 at temperature zero. For each of these
examples, we manually check the code generated by EHRAgent to obtain the
databases and columns required to answer the question. Second, we assign the
three roles to each example, which gives 549 examples in total. We label these
examples by checking if any of the required databases or columns are
inaccessible to the given role (i.e., by comparing with the access permission
for each role in Fig. 5). This will lead to a highly imbalanced dataset with
136, 110, and 48 examples labeled ‘0’ for ‘physician’, ‘nursing’, and ‘general
administration’, respectively, and 47, 73, and 135 examples labeled ‘1’ for
‘physician’, ‘nursing’, and ‘general administration’, respectively. In the
third step, we remove some of the 549 created examples to a) achieve a better
balance between the labels and b) reduce the duplication of questions among
these examples. We notice that for ‘general administration’, there are many
more examples labeled ‘1’ than ‘0’, while for the other two roles, there are
many more examples labeled ‘0’ than ‘1’. Thus, for each example with ‘general
administration’ and label ‘1’, we remove it if any of the two examples with
the same question for the other two roles are labeled ‘1’. Then, for each
example with ‘nursing’ and label ‘1’, we remove it if any example with the
same question for ‘physician’ is labeled ‘1’. Similarly, we remove each
example with ‘physician’ and label ‘0’ if any of the two examples with the
same question for the other two roles are also labeled ‘0’. Then for each
example with ‘nursing’ and label ‘0’, we remove it if any example with the
same question for ‘general administration’ is labeled ‘0’. After this step, we
have 41, 78, and 48 examples labeled ‘0’ for ‘physician’, ‘nursing’, and
‘general administration’, respectively, and 47, 41, and 62 examples labeled
‘1’ for ‘physician’, ‘nursing’, and ‘general administration’, respectively.
Finally, we randomly remove some examples for ‘nursing’ with label ‘0’ and
‘general administration’ with label ‘1’, and randomly add some examples for
the other four categories (‘physician’ with label ‘0’, ‘general
administration’ with label ‘0’, ‘physician’ with label ‘1’, and ‘nursing’ with
label ‘1’) to achieve a better balance. The added examples are generated based
on the questions from the training set222In the original EICU dataset, both
the training set and the test set do not contain the ground truth answer for
each question. The ground truth answers in the test set of EICU are provided
by Shi et al. [17]. of the original EICU benchmark. The ultimate number of
examples in our created EICU-AC benchmark is 316, with the distribution of
examples across the three roles and two labels displayed in Tab 4.

Table 4: Number of examples in EICU-AC for each role and each label. | physician | nursing | general administration
---|---|---|---
label ‘0’ (access denied) | 52 | 57 | 45
label ‘1’ (access granted) | 46 | 55 | 61

###  A.3 Healthcare Questions Involved in EICU-AC

As mentioned in the main paper, our created EICU-AC dataset involves
healthcare questions spanning 50 different ICU information categories, i.e.,
columns across all 10 databases of the EICU benchmark. We further categorize
the questions in EICU-AC following the ‘template’ provided by EICU (extracted
from the ‘q_tag’ entry of each example [17]). This gives 70 different question
templates, showing the high diversity of healthcare questions involved in our
EICU-AC benchmark.

Table 5: Number of examples labeled ‘1’ in Mind2Web-SC for each rule violation. Note that examples labeled ‘0’ do not violate any rules. Safety rules | No. examples
---|---
Rule 1: User must be a member to shop. | 19
Rule 2: Unvaccinated user cannot book a flight | 12
Rule 3: User without a driver’s license cannot buy or rent a car. | 24
Rule 4: User aged under 18 cannot book a hotel. | 18
Rule 5: User must be in certain countries to search movies/musics/video. | 21
Rule 6: User under 15 cannot apply for jobs. | 6

##  Appendix B Details About the Mind2Web-SC Benchmark

In Sec. 3.2, we have defined six safety rules for the Mind2Web-SC Benchmark.
Rule 1 requires ‘membership’ in the user information to be ‘true’. Rule 2
requires ‘vaccine’ in the user information to be ‘true’. Rule 3 requires
‘dr_license’ in the user information to be ‘true’. Rule 4 requires ‘age’ in
the user information to be no less than 18. Rule 5 requires ‘domestic’ in the
user information to be ‘true’. Rule 6 requires ‘age’ in the user information
to be no less than 15. In Tab. 5, we show the number of examples labeled ‘1’
in Mind2Web-SC for each rule violation. Note that examples labeled ‘0’ do not
violate any rules.

During the construction of Mind2Web-SC, we added some examples with label ‘1’
and removed some examples with label ‘0’ to balance the two classes. By only
following the steps in Sec. 3.2 without any adding or removal of examples, we
obtain a highly imbalanced dataset with 178 examples labeled ‘0’ and only 70
examples labeled ‘1’. Among the 178 examples labeled ‘0’, there are 148
examples with the tasks irrelevant to any of the rules – we keep 50 of them
and remove the other $(148-50=)$ 98 examples. All 30 examples labeled ‘0’ but
related to at least one rule are also kept. Then, we create 30 examples
labeled ‘1’ by reusing the tasks for these 30 examples labeled ‘0’. We keep
generating random user profiles for these tasks until the task-related rule is
violated, and the example is labeled to ‘1’. Note that the tasks are randomly
selected but manually controlled to avoid duplicated tasks within one class.
Similarly, we created 20 examples labeled ‘0’ by reusing the tasks for
examples labeled ‘1’, with randomly generated user information without any
rule violation. Finally, we obtain the Mind2Web-SC dataset with 100 examples
in each class (200 examples in total). Among the 100 examples labeled ‘0’, 50
are related to at least one of the rules.

##  Appendix C Detailed System Prompts for Naive Access Control and Safety
Control Based on Instructions

In our preliminary studies, We created a naive access control for EHRAgent and
a naive safety control for SeeAct by directly modifying their system prompts
for planning. These approaches are either ineffective in safeguarding the
agents or degrade the benign performance of the agents. In Fig. 7, we show the
instructions we injected into the system prompts of these two agents.

Figure 7: Instructions injected into the system prompt of EHRAgent for access
control and SeeAct for safety control, as naive baselines that motivate our
GuardAgent.

##  Appendix D Complete Inputs and Output of GuardAgent

#### Complete Inputs to GuardAgent

As described in Sec. 4.2, the inputs to GuardAgent include a specification
$I_{s}$ of the target agent, a set of guard requests $I_{r}$, inputs $I_{i}$
to the target agent, and the output log $I_{o}$ by the target agent
corresponding to $I_{i}$. In Fig. 8, we show the actual $I_{s}$ and $I_{r}$
for GuardAgent in our experiments for both EHRAgent on EICU-AC and SeeAct on
Mind2Web. In the same figure, we also show the actual planning instruction
$I_{p}$ and the actual instruction $I_{c}$ used for code generation. In Fig.
9, we show real examples for the target agent inputs $I_{i}$ and output logs
$I_{o}$ for both EHRAgent on EICU-AC and SeeAct on Mind2Web.

Figure 8: The actual planning instruction $I_{p}$, instruction $I_{c}$ for
guardrail code generation, target agent specification $I_{s}$ and guard
requests $I_{r}$ we used in our experiments for the two agents, EHRAgent and
SeeAct, and the two benchmarks, EICU-AC and Mind2Web-SC.

Figure 9: Examples for target agent inputs $I_{i}$ and output logs $I_{o}$, as
the inputs to GuardAgent, for the two agents, EHRAgent and SeeAct, and the two
benchmarks, EICU-AC and Mind2Web-SC.

#### Outputs of GuardAgent

The intermediate outputs of GuardAgent, including the generated action plan
$P$ and the guardrail code $C$, are similar to those in the manually created
demonstrations (see App. G); thus, we do not repeatedly show them. Here, we
show example outputs, including the label prediction $O_{l}$ and the detailed
reasons $O_{d}$ of GuardAgent for both benchmarks in Fig. 10.

Figure 10: Example outputs of GuardAgent, including the label prediction
$O_{l}$, the detailed reasons $O_{d}$, and the final answer/action of the
target agent with guardrail, for the two agents, EHRAgent and SeeAct, and the
two benchmarks, EICU-AC and Mind2Web-SC.

##  Appendix E Callable Functions

Our experiments involve two callable functions shown in Fig. 11. However, the
user of GuardAgent can easily extend the toolbox by uploading more functions.
The actual toolbox of GuardAgent in practice will contain much more callable
functions than for our experiments.

Figure 11: Callable functions in the toolbox of GuardAgent involved in our
experiments.

##  Appendix F Prompts for Baselines

In the main experiments, we compare GuardAgent with two baselines using LLMs
to safeguard LLM agents. The guardrail is created by prompting the LLM with a
system instruction, the specification of the target agent, the guard requests,
the user inputs to the target agent with the associated output logs, and a few
show of examples. Here the system instruction is adapted from the one used by
GuardAgent for task planning. However, we include additional instructions
about the format of the guardrail outputs. The baselines do not involve any
guardrail code generation, and this is reflected by the demonstrations we
created that generate guardrails solely based on reasoning over the textual
inputs to the LLM. In Fig. 12, we show the modified system prompt template for
the baselines, with two example demonstrations for the two benchmarks,
respectively.

Figure 12: System prompt template for the baselines and the two example
demonstrations for EICU-AC and Mind2Web-SC, respectively.

##  Appendix G Manually Created Demonstrations

We manually created a set of demonstrations for each benchmark. In Fig. 13, we
show two example demonstrations for EHRAgent on EICU-AC and SeeAct on
Mind2Web-SC, respectively.

Figure 13: Example demonstrations for EHRAgent on EICU-AC and SeeAct on
Mind2Web-SC.

##  Appendix H Function Defined by GuardAgent in Zero-Shot Setting

In the zero-shot setting where GuardAgent is provided with neither the
required functions nor demonstrations for guardrail code generation,
GuardAgent can still generate guardrails by defining new functions. In Fig.
14, we show a function defined by GuardAgent during guardrail code generation.
The function differs from those we provided in Fig. 11, but it achieves the
same guardrail goals.

Figure 14: A function defined by GuardAgent in zero-shot setting with neither
demonstrations for code generation nor required functions

##  Appendix I Execution Time of GuardAgent

The average execution time for GuardAgent (with GPT-4) safeguarding EHRAgent
on EICU-AC is 45.4 seconds per example, while the average execution time for
EHRAgent (with GPT-4) is 31.9 seconds per example. The average execution time
for GuardAgent (with GPT-4) safeguarding SeeAct on Mind2Web-SC is about 60
seconds per example, while the average execution time for EHRAgent (with
LLaVA-1.5) is about 20 seconds per example. In general, the execution time for
GuardAgent is comparable to the execution time of the target agent. Moreover,
human inspectors will likely need much more time than our GuardAgent to read
the guard requests and then moderate the inputs and outputs of the target
agent correspondingly. Given the effectiveness of our GuardAgent as shown in
the experiments, we believe that GuardAgent is the current best for
safeguarding LLM agents.