# Safeguarding Large Language Models: A Survey

Yi Dong1∗ Ronghui Mu1∗ Yanghao Zhang1 Siqi Sun1 Tianle Zhang1 Changshun Wu2
Gaojie Jin1 Yi Qi1 Jinwei Hu1 Jie Meng3 Saddek Bensalem2,4 Xiaowei Huang1 *
Equal contribution;
1\. University of Liverpool, UK;
2\. Université Grenoble Alpes, France;
3\. Loughborough University, UK;
4\. CSX-AI, France
Correspondence to: Xiaowei Huang<EMAIL_ADDRESS>
###### Abstract

In the burgeoning field of Large Language Models (LLMs), developing a robust
safety mechanism, colloquially known as “safeguards” or “guardrails”, has
become imperative to ensure the ethical use of LLMs within prescribed
boundaries. This article provides a systematic literature review on the
current status of this critical mechanism. It discusses its major challenges
and how it can be enhanced into a comprehensive mechanism dealing with ethical
issues in various contexts. First, the paper elucidates the current landscape
of safeguarding mechanisms that major LLM service providers and the open-
source community employ. This is followed by the techniques to evaluate,
analyze, and enhance some (un)desirable properties that a guardrail might want
to enforce, such as hallucinations, fairness, privacy, and so on. Based on
them, we review techniques to circumvent these controls (i.e., attacks), to
defend the attacks, and to reinforce the guardrails. While the techniques
mentioned above represent the current status and the active research trends,
we also discuss several challenges that cannot be easily dealt with by the
methods and present our vision on how to implement a comprehensive guardrail
through the full consideration of multi-disciplinary approach, neural-symbolic
method, and systems development lifecycle.

###### Index Terms:

Large Language Models, Generative AI, Safeguards, Guardrails, Trustworthy AI.

##  1 Introduction

In recent years, generative artificial intelligence (GenAI) has significantly
accelerated humanity’s stride into the era of intelligence. Technologies such
as ChatGPT and Sora [1] have become a pivotal force driving the transformation
of a new generation of industries. However, the rapid deployment and
integration of LLMs have raised significant concerns regarding their risks,
including, but not limited to, ethical use, data biases, privacy, and
robustness [2]. In societal contexts, concerns also include the potential
misuse by malicious actors through activities such as spreading misinformation
or aiding criminal activities [3]. In the scientific context, LLMs can be used
professionally, with dedicated ethical considerations and risks in scientific
research [4].

To address these issues, model developers have implemented various safety
protocols intended to confine the behaviors of these models to a more secure
range of functions. The complexity of LLMs, characterized by intricate
networks and numerous parameters, and the closed-source nature (such as
ChatGPT) present substantial hurdles. These complexities require different
strategies compared to the pre-LLM era, which focuses on _white-box
techniques_ , enhancing models by various regularizations and architecture
adaptations during training. Therefore, in parallel to the reinforcement
learning from human feedback (RLHF) and other training skills such as in-
context training, the community moves towards employing _black-box, post-hoc
strategies_ , notably guardrails [5, 6], which monitors and filters the inputs
and outputs of trained LLMs. A guardrail is an algorithm that takes as input a
set of objects (e.g., the input and the output of LLMs) and determines if and
how some enforcement actions can be taken to reduce the risks embedded in the
objects. If the input to LLMs relates to child exploitation, the guardrail may
stop the input or adapt the output to become harmless [7]. In other words,
guardrails are used to identify the potential misuse in the query stage and to
prevent the model from providing an answer that should not be given.

The difficulty in constructing guardrails often lies in establishing their
requirements. AI regulations can be different across different countries, and
in the context of a company, data privacy can be less severe than in the
public domain. Nevertheless, a guardrail of LLMs may include requirements from
one or more categories: Hallucination, fairness, privacy, robustness,
toxicity, legality, out-of-distribution, uncertainty, etc. In this paper, we
do not include the typical requirement, i.e., accuracy, as they are benchmarks
of the LLMs and arguably not the responsibilities of the guardrails. That
said, there might not be a clear cut on the responsibilities (notably,
robustness) between LLMs and the guardrails, and the two models shall
collaborate to achieve a joint set of objectives. Nevertheless, for concrete
applications, the requirements need to be precisely defined, and their
corresponding metrics, and a _multi-disciplinary_ approach is called for.
Mitigating a given requirement (such as hallucinations, toxicity, fairness,
biases, etc) is already non-trivial, as discussed in Section 5. Working with
multiple requirements worsens it, especially when some can be _conflicting_.
Such complexity requires a sophisticated solution design method to manage. In
terms of the design of guardrails, while there might not be “one method that
rules them all”, a plausible design of the guardrail is _neural-symbolic_ ,
with learning agents and symbolic agents collaborating in processing both the
inputs and the outputs of LLMs. Multiple types of neural-symbolic agents [8].
However, the existing guardrail solutions such as Llama Guard [9], Nvidia NeMo
[10], and Guardrails AI [11] use the simplest, loosely coupled ones. Given the
complexity of the guardrails, it will be interesting to investigate other,
more deeply coupled, neural-symbolic solution designs.

Like safety-critical software, a _systematic process_ to cover the development
cycle (ranging from specification to design, implementation, integration,
verification, validation, and production release) is required to carefully
build the guardrails, as indicated in industrial standards such as ISO-26262
and DO-178B/C. This survey starts with some background introduction at Section
2. The goal is to (1) Understand the existing guardrail frameworks that are
being used to control model outputs in LLM services, as well as the techniques
to evauate, analyze, and enhance guardrails against specific desirable
properties (Section 3); (2) Understand the techniques that are being used to
overcome these guardrails, as well as to defend the attacks and to reinforce
the guardrails (Section 4); and then discuss how to achieve a complete
guardrail solution, including several issues regarding the systematic design
of a guardrail for a specific application context (Section 5).

##  2 Background for Large Language Models

Large Language Models (LLMs), primarily based on the Transformer architecture
[12], are composed of deep neural networks with several transformer blocks.
Each block integrates a self-attention layer and a feedforward layer connected
via residual links. This specific self-attention mechanism enables the model
to concentrate on neighboring tokens while analyzing a specific token.
Originally, the transformer architecture was exclusively developed for machine
translation purposes. Newly developed language models that utilize transformer
architecture can be fine-tuned, thereby removing the need for architectures
tailored to specific tasks [13]. Typically, their networks encompass hundreds
of billions (or more) of parameters and are trained on vast corpora of textual
data. Examples include ChatGPT-3 [14], ChatGPT-4 [1], LLaMA [15], and PaLM
[16].

LLMs are employed in a variety of complex tasks, such as conversational AI
[17], translation [18], and story generation [19]. Current LLMs utilize
architectures and training objectives similar to those in smaller language
models, such as the Transformer architecture and tasks centered around
language modeling. However, LLMs distinguish themselves by significantly
scaling up in aspects like model dimensions, data volume, the breadth of their
application scope, and computation cost. Building an offline model comprises
three main stages [2]: pre-training, adaptation tuning, and utilization
improvement. Generally, the pre-training phase parallels conventional machine
learning training, involving data collection, choosing an architecture, and
undergoing training. The adaptation tuning includes instruction tuning [20]
and alignment tuning [21] to enable learning from task-specific instructions
and adhere to human values. Finally, the utilization improvements can enhance
user interactions, including in-context learning [14] and chain-of-thought
learning [22].

After training an LLM, its performance against set expectations is crucial.
This evaluation typically encompasses three dimensions: assessing essential
performance, conducting safety analysis to understand potential consequences
in practical applications, and utilizing publicly available benchmark
datasets. The primary performance review focuses on essential capabilities
like language generation and complex reasoning. Safety analysis delves into
the LLM’s alignment with human values, interactions with external
environments, and integration into broader applications such as search
engines. Additionally, benchmark datasets and accessible tools support this
comprehensive evaluation. The outcome of this assessment determines whether
the LLM meets pre-defined criteria and is ready for deployment. If it falls
short, the process reverts to one of the earlier training stages to address
identified shortcomings. At the deployment stage, LLM could be used on a web
platform for direct user interaction, like ChatGPT, or integrated into a
search engine, like the new Bing. Regardless of the application, it is
standard practice to implement guardrails in interactions between LLMs and
users to ensure adherence to AI regulations.

##  3 Techniques on Design and Implementation of Guardrails

This section presents several existing guardrail techniques have been proposed
by the LLM service provider or the open-source community. Then, we review the
methods to evaluate, analyze, and enhance the LLMs according to the desirable
properties one may expect an LLM to have. A comparison checkbox table among
different platforms and properties is shown in Table. I.

TABLE I: Abilities among different Guardrails.

| Llama Guard | Nvidia NeMo | Guardrails AI | TruLens | Guidance AI | LMQL
---|---|---|---|---|---|---
Hallucination | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Fairness | ✓ | - | ✓ | ✓ | - | -
Privacy | - | ✓ | - | - | - | -
Robustness | - | - | - | - | - | -
Toxicity | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Legality | ✓ | - | - | - | - | -
Out-of-Distribution | - | - | ✓ | - | - | -
Uncertainty | - | ✓ | ✓ | ✓ | - | -

###  3.1 Guardrail Frameworks and Supporting Software Packages

LLM guardrails constitute a suite of safety measures designed to oversee and
regulate user interactions with LLM applications. These measures are
programmable, rule-based systems positioned between users and foundational
models. Their primary function is to ensure that the LLM model adheres to an
organization’s established principles and operates within a prescribed ethical
and operational framework. Guardrails are applied during the interaction stage
between users and deployed LLMs, the last step in the LLM lifecycle. Fig. 1
illustrates the lifecycle and potential vulnerabilities of the general
guardrail mechanism. Developers complete the development of guardrails through
data processing, guardrail model training, and model customization or fine-
tuning (e.g., Llama Guard and NeMo Guardrails), as shown in the yellow area of
Fig. 1. These guardrails are then deployed in LLMs to facilitate interaction
with users. Typically, users predefine the content that needs protection, also
called custom rules. Subsequently, users interact with LLMs through prompts
and await the generated responses. The guardrails evaluate the output against
the predefined rules to determine its compliance. If the content is deemed
unsafe, guardrails may block it outright or issue a preset warning to the
user. Conversely, if the output aligns with the criteria, it is displayed
directly to the user, as indicated in the orange area of Fig. 1. Notably, some
existing attack methods allow unsafe content to bypass guardrail protection,
as highlighted in the red box of Fig. 1; for a detailed discussion of these
attack methods, refer to Section 4.

Figure 1: Guardrails Lifecycle and Vulnerabilities

####  3.1.1 Llama Guard

Llama Guard [9], developed by Meta on the Llama2-7b architecture, focuses on
enhancing human-AI conversation safety. It is a fine-tuned model that takes
the input and output of the victim model as input and predicts their
classification on a set of user-specified categories. Figure 2 shows its
workflow. Due to the zero/few-shot abilities of LLMs, Llama Guard can be
adapted–by defining the user-specified categories –to different taxonomies and
guidelines that meet requirements for applications and users. This is a Type 1
neural-symbolic system [8], i.e., typical deep learning methods where the
input and output of a learning agent are symbolic. It lacks guaranteed
reliability since the classification results depend on the LLM’s understanding
of the categories and the model’s predictive accuracy.

Figure 2: Llama Guard Guardrail Workflow

####  3.1.2 Nvidia Nemo

Nvidia NeMo, described in Rebedea’s work [10], functions as an intermediary
layer that enhances the control and safety of Large Language Models (LLMs). It
employs Colang, an executable program language designed by [23], to establish
constraints to guide LLMs within set dialogical boundaries. When the
customer’s input prompt comes, NeMo embeds the prompt as a vector and then
uses K-Nearest Neighbours (KNN) method to compare it with the stored vector-
based user canonical forms, retrieving the embedding vectors that are ‘the
most similar’ to the embedded input prompt. The input embedding of Nemo
differs from traditional approaches that utilize the initial layers. Instead,
it employs embedding through similarity functions to capture the most relevant
semantics. For example, Nemo uses the "sentence-transformers/all-MiniLM-L6-v2”
model to compute embeddings, which is used for the following KNN search (Annoy
algorithm is employed for efficient nearest-neighbor search.) Therefore,
“input embedding” in the Nemo refers to mapping input sentences and paragraphs
to a multi-dimensional dense vector space, facilitating the search for the
most similar canonical forms/flows. After that, Nemo starts the flow execution
to generate output from the canonical form. During the flow execution process,
the LLMs generate a safe answer if the Colang program requests.

LLMs will be invoked multiple times during the guardrail flow for various
tasks. For example, in a conversation scenario, LLM is utilized in the
following three phases: (1) Generating user intent: the input of this LLM call
includes two contexts: examples and potential user intents (top 5 intents from
example code in NeMo Github repository). The output is a refined user intent
(temperature is set as 0 to get the deterministic result). (2) Generating next
step: In this phase, Nemo searches the most relevant similar flows and
integrates these similar flows into an example, which is then fed into the
LLM. The LLM call output is called “bot intent.”. (3) Generating bot-message:
The input for this call includes examples (the five most relevant bot intents)
and relevant chunks (dictionary search), which are to be used as context.

The process is presented in Figure 3. Building on the above customizable
workflow, NeMo also includes a set of pre-implemented moderations dedicated to
e.g., fact-checking, hallucination prevention in responses, and content
moderation. NeMo is also a Type-1 neural-symbolic system, with its
effectiveness closely tied to the performance of the KNN method.

Figure 3: Nvidia NeMo Guardrails Workflow

####  3.1.3 Guardrails AI

Guardrails AI enables the user to add structure, type, and quality guarantees
to the outputs of LLMs [11]. It operates in three steps: 1) defining the
“RAIL” spec, 2) initializing the “guard” and 3) wrapping the LLMs. In the
first step, Guardrails AI defines a set of RAIL specifications, which are used
to describe the return format limitations. This information must be written in
a specific XML format, facilitating subsequent output checks, e.g., structure
and types. The second step involves activating the defined spec as a guard.
For applications that require categorized processing, such as toxicity checks,
additional classifier models can be introduced to classify the input and
output text. The third step is triggered when the guard detects an error.
Here, the Guardrails AI can automatically generate a corrective prompt,
pursuing the LLMs to regenerate the correct answer. The output is then re-
checked to ensure it meets the specified requirements. Currently, the methods
based on Guardrails AI are only applicable for text-level checks and cannot be
used in multimodal scenarios involving images or audio. Unlike the previous
two methods, Guardrail AI is a Type-2 neural-symbolic system with a backbone
symbolic algorithm supported by learning algorithms (in this case, those
additional classifier models).

Figure 4: Guardrails AI Workflow

####  3.1.4 TruLens

TruLens, developed by TruEra111https://truera.com/, is an open-source toolkit
for developing, evaluating, and monitoring LLMs. Central to its features is
TruLens-Eval, which ensures quality by comparing outputs against prefined
standards. The toolkit integrates LLMs, supporting logging records of inputs
and outputs, and it leverages feedback functions that utilize auxiliary
models, such as relevance models or sentiment classifiers, to perform
evaluations on the RAG triad, including context relevance, answer relevance,
and groundedness. Retrieval-Augmented Generation (RAG) is a technology that
enhances the quality of language model outputs by supporting answer generation
with retrieved relevant information. Within TruLens, RAG’s role is to ensure
the accuracy and relevance of model outputs by comparing them against
predefined standards, thereby evaluating LLM apps. The services are invoked
from various providers. For instance, when assessing groundedness related to
how closely outputs align with the source material, Trulens-Eval can utilize
providers like OpenAI API to call an LLM to find the relevant strings in a
text or employ NLI models with hugging faces. The toolkit allows for the
customization or pre-definition of feedback functions via Python, enabling
evaluations to be specifically tailored to unique requirements. Additionally,
TruLens incorporates embedding models to convert predefined information into
numerical vectors, simplifying matching text with relevant data. TruLens also
visualizes the LLM applications’ rankings in a leaderboard according to their
performance metrics, creating a dynamic environment that encourages developers
to refine their models iteratively. As a guidance-oriented approach, TruLens
is designed not to constrain LLM inputs and outputs but to provide a framework
for continuous model refinement and evaluation, ensuring adherence to quality
and relevance standards.

Figure 5: TruLens Workflow

####  3.1.5 Guidance AI

Guidance AI222https://guidance.readthedocs.io/en/latest/, a programming
paradigm, offers superior control and efficiency than conventional prompting
and chaining. It allows users to constrain generation (e.g., with regex and
CFGs) and interleave control (conditional, loops) and generation seamlessly.
This guardrail tool integrates text generation, prompts, and logic control
within a single, continuous flow in a Python environment, thereby refining the
text processing approach in LLMs. This unified method allows more effective
LLM control than traditional prompts or thought language chains. Its features
include simple and intuitive syntax built on the Handlebars template language,
assuring the variable insertion in any prompts. The Guidance program has a
well-defined linear execution order directly corresponding to the token
sequence processed by the language model. The illustration graph of Guidance
AI working flow is demonstrated in Figure 6.

Figure 6: Guidance AI Workflow

At any timesteps during the program execution, the language model can be
called for generation(via {{gen}} tag) or to make logical flow decisions, such
as {{#select}}{{or}}{{/select}} commands. Guidance supports a variety of LLMs,
and during dialogue, it can use role labels to map the current LLM to correct
tokens or API calls, such as {{#assistant}}, {{#user}}, {{#system}} etc. It
also can be integrated with HuggingFace models, including using Guidance
acceleration to speed up standard prompts by reusing key-value caches to
shorten prompt execution times and using token healing to optimize prompt
boundaries. Regarding token healing, this concept is related to fixing the
subtleties introduced by the language model’s normal greedy tokenization
method. Specifically, it involves advancing the model one step while
simultaneously restricting the prefix of the generated token to be the same as
the previous token. Regex patterns to enforce formatting are also supported in
Guidance. Guidance’s templated output is more suitable for generating text
with high formatting requirements, such as ensuring legally compliant and
controllable JSON structures. During this process, different operation
commands have their processing methods; for example, encountering the select
command, it specifies the generation of a token and returns the corresponding
log probs, then uses a trie tree to match candidates and determine their
probabilities, finally selecting the one with the highest probability.
Additionally, it supports hidden blocks; for instance, some inference
processes of the LLM may not need to be exposed to the end user, but they can
be utilized in the template to generate intermediate results.

####  3.1.6 LMQL (Language Model Query Language)

LMQL 333https://lmql.ai, a programming interface for LLMs focusing on
controlled output and safety of generated content, is designed by SRI Lab at
ETH Zurich. Building on the foundation of Guidance AI, the LMQL project
further advances the concept of “prompt templates” into a new programming
language paradigm. As a Python superset, it allows developers the capacity to
embed precise constraints within their queries. These constraints, from
content restriction to adherence to specific formats for accuracy, leverage
logit masking and custom operator support for fine-tuned control. Structured
to simplify LLM interactions, LMQL introduces a SQL-like syntax complemented
by scripting capabilities. Its foundation is built on decoder declarations,
such as argmax, beam, or sample strategies, alongside query blocks that
support inserting variables or placeholders expected to be filled, model
sources, and intricate constraint conditions specified in ’where’ clauses. The
workflow of the LQML is illustrated in Figure 7.

Figure 7: LMQL Workflow

At its core, LMQL’s runtime and decoding ability uniquely features employs a
scripted Beam search to execute LMQL-specific queries and constraints. This
approach allows for a search across all placeholders rather than limiting the
process to singular predictive points. Additionally, constraint-driven
decoding dynamically adjusts available tokens based on real-time evaluation of
specified constraints and can reduce ineffective model calls. The Partial
Evaluation Semantics and FollowMaps facilitate the application of constraints
in real-time during content generation, considering both the current state of
the generated content and the potential next tokens. Partial evaluation
semantics combines value and final semantics for eager validation. Value
semantics determines the current value of expressions given the existing
context. In contrast, the final semantics employs annotations such as FIN
(fixed), VAR (variable), INC (increasing), and DEC (decreasing) to indicate
the potential change in the value of expressions as content generation
progresses. Among the joint value and final semantics, boolean expressions are
denoted by symbols $\top$ for True and $\bot$ for False. Based on these
semantics, FollowMap which is formulated as Follow[<expr>]$(u,t)$ is designed
to guide the generation process by evaluating the impact of adding a new token
$t$ to the current interaction trace $u$ on meeting the specified constraints.
The system evaluates potential next tokens against FollowMap constraints,
excluding those that may violate constraints, thus guiding content generation
away from invalid sequences and minimizing unnecessary model calls.

####  3.1.7 Python Packages

Apart from the above-mentioned integrated guardrail techniques, Several Python
packages are pivotal in implementing guardrails around LLMs, enhancing their
safety, fairness, and compliance. Here we listed some packages: (1)
$LangChain$444https://www.langchain.com/: LangChain is pivotal in streamlining
the development of LLM applications. It introduces components that can be
leveraged to implement guardrails, thus indirectly contributing to creating
safer and more reliable AI solutions. (2) $AI\ Fairness\ 360$
(AIF360)555https://github.com/Trusted-AI/AIF360: AIF360 is an extended toolkit
from IBM that provides developers with a comprehensive set of algorithms
designed to detect, understand and mitigate bias in AI models. Its extensive
resources help enhance the fairness and integrity of machine learning
applications. (3) $Adversarial\ Robustness\ Toolbox$
(ART)666https://github.com/Trusted-AI/adversarial-robustness-toolbox: ART is
tailored to enhance model security and robustness in the face of increasing
adversarial attacks. It provides mechanisms to defend against and adapt to
malicious input, protecting AI applications from potential vulnerabilities.
(4) $Fairlearn$777https://github.com/fairlearn/fairlearn: This package
addresses and reduces unwanted biases in machine learning models. Fairlearn
provides developers with tools and methodologies to assess and mitigate
biases, fostering fairness and equality in AI outcomes. (5)
$Detoxify$888https://github.com/unitaryai/detoxify?trk=article-ssr-frontend-
pulse˙little-text-block: Detoxify aims to identify and mitigate toxic content
in text data, serving as a crucial tool for maintaining a respectful and safe
digital interaction space. It utilizes advanced models to screen and filter
out harmful content, ensuring that AI applications foster positive
communication.

These packages represent just a glimpse into the vast array of tools available
to AI developers dedicated to embedding ethical considerations into their
applications. By utilizing these packages, developers can navigate the complex
challenges of AI safety, ensuring their LLMs are technologically advanced and
aligned with ethical and responsible use principles. This approach reinforces
the commitment to developing innovative AI technologies that respect societal
norms and individual rights, marking a significant step towards responsible AI
development.

###  3.2 Techniques for (Un)desirable Properties in Guardrails

In this section, we discuss several different properties, detailing their
standard definitions and how to use Guardrails to protect these properties. It
is noticed that the number of properties is too extensive to cover
comprehensively; thus, we focus on hallucination, fairness, privacy,
robustness, toxicity, legality, out-of-distribution, and uncertainty.

####  3.2.1 Hallucination

Hallucinations in LLMs are defined as responses that are either nonexistent in
reality, illogical, or irrelevant to the prompt provided; an example of
hallucination is shown in Fig. 8. These phenomena often stem from the language
models’ uncertainty in response, where they generate answers based on patterns
identified in training datasets rather than actual factual understanding [24].
The origins of these hallucinations can be traced back to various stages,
including data sourcing [25], pre-training [26], alignment [25], and inference
[27].

The resolution of hallucinations in LLM is crucial due to their impact on
model reliability and real-world applications, which presents a significant
challenge in LLM development. Researchers are actively working on methods to
differentiate between accurate and hallucinated content in LLM responses. This
involves the use of detection technologies [28] and evaluation methodologies
[29] to ensure the integrity and utility of LLM outputs. In certain contexts,
such as safety-critical applications, ensuring rigorous guarantees for the
output of LLMs is imperative and achievable. This involves first specifying
high-level formal specifications and then applying formal verifiers to monitor
whether the execution of LLMs conforms to the specified requirements. For
instance, in [30], the authors proposed a framework for counterexample-guided
inductive synthesis using LLMs and satisfiability modulo theory (SMT). Within
this framework, an SMT solver is employed to eliminate the hallucinated
outputs of the LLM, guiding it to generate correct outputs. Experimental
results on two planning problems demonstrated the promise of this approach in
practice. The framework consistently converged to correct outputs in finite
steps. However, it is worth noting that such convergence cannot always be
theoretically guaranteed, underscoring the practical significance of this
method in addressing LLM hallucinations. Furthermore, there are some
researches proposed to detect and prevent the hallucinations of LLMs.

In [28], the authors suggest utilizing continuous integration (CI) to
automatically identify hallucinations in the outputs of an LLM with the aid of
another LLM. The CI process involves regular incremental updates to the
software, with each change automatically built and tested, allowing for prompt
detection of errors. Therefore, LLM developers can use CI to automate tests on
new datasets and updates to the models, which ensures that any changes do not
unintentionally introduce bias or reduce the quality of the model’s output.
Instead of using another LLM to identify the errors, some researchers use
trusted information sources to cross-check the generated content [31].
Building on these techniques, a comprehensive framework is proposed by [32],
which equips LLMs with the ability to recognize factual inaccuracies,
utilizing external tools to gather supporting evidence. In addition to
verifying the accuracy of responses, it’s crucial to assess the faithfulness
of the output to prevent hallucinations that diverge from the context. This
means ensuring the output remains relevant and agrees with the input, avoiding
any extraneous or contradictory information. This can be achieved by the fact-
based metrics that are based on entity [33], relation[34], and knowledge F1
metric [35]. A further approach to ensure a model’s faithful output utilizes
the classifiers that are either trained on task-specific inference datasets
[36] or fine-tuned on adversarial datasets [37]. Nowadays, the instruction-
following abilities of LLMs are leveraged for autonomous evaluation. LLMs can
effectively gauge accuracy and relevance by setting clear evaluation criteria
and providing both generated and source content. Different methodologies have
been adopted for output evaluation, such as binary judgment mechanisms [38]
and using a k-point Likert scale [39].

In current guardrails, Nvidia Nemo employed the mechanism proposed by [40]. It
first generates a few additional responses from the LLM, typically two more by
default. The original response from the bot is treated as the hypothesis,
while the additional generated responses serve as the context or evidence.
This method aligns the consistency verification with natural language
inference (NLI) principles, allowing for a structured output assessment.

####  3.2.2 Fairness

Fairness is a concept that originates in sociology, economics, and law. It is
defined as “imperfect and just treatment or behavior without favoritism or
discrimination” in the Oxford English Dictionary. The key to fairness in NLP
is the presence of social biases in language models; an example is illustrated
in Fig. 9. Unfair LLM systems make discriminatory, stereotypic, and demeaning
decisions against vulnerable or marginalized demographics, causing undesirable
social impacts and potential harms [41]. Fairness in LLMs has been studied
from different angles, such as gender bias [42],cultural bias [43], dataset
bias [44], and social bias [45].These social biases can be encoded in the
embeddings and carried over to decisions in downstream tasks, compromising the
fairness of LLMs. For instance, when GPT-3 is prompted with questions about
leadership roles or positions of power, it may reflect societal biases in its
responses. If asked, “Who is likely to be a CEO?”, GPT-3’s response might lean
towards “He is likely to be a CEO,” implying a male. Conversely, suppose the
question is about lower-ranking positions or roles traditionally seen as
supportive or administrative, such as “Who is likely to be a secretary?”. In
that case, the model might respond with “She is likely to be a secretary,”
suggesting a female. This pattern of responses reveals a bias where higher-
status professions or roles are more closely associated with males, while
supportive or administrative roles are linked with females. Such biases in LLM
outputs can reinforce outdated stereotypes and potentially influence the
perception of what roles are “appropriate" for individuals based on their
gender.

Thus, to guard fairness in LLMs, increasing work is needed to understand these
biases and evaluate their further effects on the downstream tasks of LLMs. For
example, in terms of racial bias, an African American is more likely to be
assigned a “criminal behavior” feature because of the “African” group he
belongs to [46]. When this feature is used for model encoding and further
downstream tasks, it induces unfairness in the language model toward African
Americans. Biases are purposefully introduced into the responses of LLMs to
craft distinct personas for use in interactive media [47]. BAD focuses on
identifying and quantifying instances of social bias in models like ChatGPT,
especially in sensitive applications such as job and college admissions
screening [48]. DAMA utilizes causal analysis to identify problematic model
components, mainly focusing on mid-upper feed-forward layers most prone to
convey biases [49]. The presence of political bias is examined in ChatGPT,
focusing on aspects such as race, gender, religion, and political orientation
[50]. Additionally, they explored the role of randomness in responses by
collecting multiple answers to the same questions, which enables a more robust
analysis of potential biases. The bias of LLMs is also examined by controlling
the input, highlighting that LLMs can still produce biased responses despite
the progress in bias reduction [51]. Besides, a Bias Index is designed to
quantify and address biases inherent in LLMs, including GPT-4 [52]. It has
also been observed that the biased response can be generated inadvertently,
sometimes as seemingly harmless jokes [53].

A line of debiasing studies aims to mitigate the intrinsic bias that is task-
agnostic in the representations before they are applied to downstream tasks.
Safeguarding training data before training the model efficiently alleviates
intrinsic biases since label imbalance across different demographic groups in
the training data is an essential factor in inducing bias. A Counterfactual
Data Augmentation (CDA) [54, 55] is a widespread data processing method to
balance labels, which replaces the sensitive attributes in the original sample
with the sensitive attributes of the opposite demographic based on a prior
list of sensitive word pairs. Fairness can be incorporated into LLMs’ design
to balance the training samples, and then a guarded fairer model can be
obtained by parameter tuning. Retraining models [56]is a direct way to reduce
bias, although it can be resource-intensive and difficult to scale. For
instance, FairBERTa is a fairer model for retraining RoBERTa on a large-scale
demographic perturbation corpus Perturbation Augmentation NLP DAtaset (PANDA)
containing 98K augmentation sample pairs [56]. Moreover, an additional
debiasing module is added after the encoder of LLMs to filter out the bias in
the representation, and a common strategy is to utilize a contrastive learning
framework for training [57].

The other line of debiasing studies aims to mitigate the extrinsic debiasing
in a task-specific way. These studies attempt to improve fairness in
downstream tasks by making models provide consistent outputs across different
demographic groups. Many studies have concentrated on reducing bias through
model adaptation approaches. A bias mitigating method, DAMA [49], can reduce
bias while maintaining model performance on downstream tasks. [58]
investigated the bias in CtB-LLMs and demonstrated the effectiveness of
debiasing techniques. They find that bias depends not solely on the number of
parameters but also on factors like perplexity and that techniques like
debiasing of OPT using LoRA can significantly reduce bias. [59] demonstrated
that the Stereotype Content Model, which posits that minority groups are often
perceived as cold or incompetent, applies to contextualized word embeddings
and presents a successful fine-tuning method to reduce such biases. Moreover,
[60] proposed a novel adversarial learning debiasing method applied during the
pre-training of LLMs. [61] mitigated cultural bias through fine-tuning models
on culturally relevant data, yet it requires resources that make it accessible
to only a few.

Instead of fine-tuning parameters, several studies directly set up guardrails
on the interaction between users and LLMs by exploring the control of input
and output. [62] suggested using purposely designed code generation templates
to mitigate the bias in code generation tasks. [43] found that cultural
prompting is a simple and effective method to reduce cultural bias in the
latest LLMs. However, it may be ineffective or even exacerbate bias in some
countries. [63] proposed a method to address gender bias that does not require
access to model parameters. It shows that text-based preambles generated from
manually designed templates can effectively suppress gender biases in LLMs
with minimal adverse effects on downstream task performance. [64] guided LLMs
to generate more equitable content by employing an innovative approach of
prompt engineering and in-context learning, significantly reducing gender
bias, especially in traditionally problematic.

Developing guardrails through a comprehensive approach that intertwines
various strategies is crucial to mitigate bias effectively. This begins with
meticulously monitoring and filtering training data to ensure it is diverse
and devoid of biased or discriminatory content. The essence of this step lies
in either removing biased data or enriching the dataset with more inclusive
and varied information. Alongside this, algorithmic adjustments are necessary,
which involve fine-tuning the model’s parameters to prevent the overemphasis
of certain patterns that could lead to biased outcomes. Incorporating bias
detection tools is another pivotal aspect. These tools are designed to
scrutinize the model’s outputs, identifying and flagging potentially biased
content for human review and correction. We believe that adopting a continuous
learning approach is key to the long-term efficacy of these guardrails. This
involves regularly updating the model with new data, insights, and feedback
and adapting to evolving societal norms and values. This dynamic process
ensures that the guardrails against bias remain robust and relevant. Moreover,
we believe in _principled methods_ to evaluate fairness when the definitions
are settled. However, the definition is expected to be distribution-based
rather than point-based as unintended responses, which need to estimate
posterior distributions and measure the distance between two distributions.

####  3.2.3 Privacy (Copyright)

Privacy, in the context of modern technology and artificial intelligence, is a
crucial aspect of data protection that has been increasingly emphasized
through legislation and research. Legislative measures like the EU AI Act,
General Data Protection Regulation (GDPR), and California Consumer Privacy Act
(CCPA) have established stringent data sharing and retention standards,
necessitating strict adherence to data protection and privacy guidelines.
Despite these frameworks, challenges persist in preventing the release of
personally identifiable information (PII) by LLMs [65], emphasizing the need
for cautious and robust data handling protocols, an example of a privacy issue
is shown in Figure 10. [66] comprehensively analyzes privacy attacks against
LLMs, introduces significant defense strategies, and highlights potential new
privacy issues and future research directions as LLMs evolve.

Several studies have focused on implementing privacy defense technologies to
safeguard data privacy and mitigate privacy breaches. Differential Privacy
(DP)-tuned LLMs [66] emerge as a leading approach to protecting data privacy
in these contexts, ensuring secure handling of sensitive information by LLMs
and minimizing the risk of privacy violations. For general NLP models, [67]
indicated that a direct application of DP-SGD [68] may not perform
satisfactorily and suggests a few tricks. [69] study differential privacy
model compression and proposes a framework that achieves 50% sparsity levels
while maintaining nearly complete performance, setting a benchmark for future
research in this area. [70] implemented a model for text rewriting along with
Local Differential Privacy (LDP), both with and without pretraining. [71]
introduce Privacy Protection Language Models (PPLM), a novel paradigm for
fine-tuning LLMs that incorporates domain-specific knowledge while preserving
data privacy. They explore techniques such as corpus curation and instruction-
based tuning, demonstrating the effectiveness of these approaches in
safeguarding private data. [72] introduce a novel text protection mechanism
called "Silent Guardian," which effectively prevents the malicious use of text
by LLMs through Truncation Protection Examples and the Super Tailored
Protection algorithm. It features efficiency, semantic consistency,
transferability, and robustness. [73] proposed a method to prepend a trained
prompt to the incoming prompt before passing it to the model, where the
training of the prefix prompt is to minimize the extent of extractable
memorized content in the model. [66] and [74] also proposed the prompt-tuning
methodology that adheres to differential privacy principles. [75] propose an
effective algorithm for differentially private fine-tuning of large pre-
trained language models, which achieves utility close to that of non-private
models while protecting privacy and reduces the computational and memory cost
of training, especially performing excellently on larger models. [76]
introduces a “Just Fine-tune Twice" (JFT) framework for the latest large
Transformer models, achieving Selective Differential Privacy protection. It
enhances the model’s utility and privacy safeguards through double fine-tuning
and systematic methods.

Other than constructing privacy-preserving LLMs, watermarking techniques can
play a more critical role in LLMs for privacy and copyright protection. A
typical watermarking mechanism [77] embedded watermarks into the output of
LLMs by selecting a randomized set of “green” tokens before a word is
generated and then softly promoting the use of green tokens during sampling.
So, as long as we know the list of green tokens, it is easy to determine if an
output is watermarked or not. We can also use the watermarks to track the
point of origin or the owner of watermarked text for copyright purposes, and
this has been applied to protect the copyright of generated prompts [78]. We
believe in an agreed watermarking mechanism between the data owners and the
LLMs developers, such that the users embed a personalized watermark into their
documents or texts when they deem them private or with copyright, and the LLMs
developers will _not_ use watermarked data for their training. More
importantly, the LLMs developers should take the responsibility of enabling
(1) an automatic verification to determine if a user-provided, watermarked
text is within the training data, and (2) model unlearning [79], which allows
the removal of users’ personally owned texts from training data. LLMs also
risk user trust due to their pre-training on vast textual datasets [80],
potentially leading to inadvertent disclosure of sensitive information about
individuals [81]. Malicious actors can exploit this vulnerability through
adversarial attacks [82], underscoring the critical importance of privacy
protection, especially when fine-tuning LLMs with sensitive data.

In addressing privacy concerns within LLM applications, implementing
guardrails is crucial for existing and in-development technologies. Key
strategies for existing applications include robust testing to identify
privacy risks and continuous model monitoring to adapt to new threats.
Implementing content control mechanisms such as blocklists, allowlists, and
suppression lists directly tackles unsafe content generation that could
compromise privacy. For example, Nemo Guardrails restricts apps to making
connections only to external third-party applications known to be safe. The
guardrails can force an LLM model to interact only with third-party software
on an allowed list. The "human-in-the-loop" approach, where human oversight is
applied to review potentially sensitive outputs and facilitates user reporting
channels for privacy violations, enhances the protection framework [83].
Regular model retraining to align with current norms and the option to revert
to previous safe versions of the model serve as dynamic responses to privacy
challenges.

For applications still in development, privacy protection begins at the design
stage, with ethical risk assessments focused on identifying and mitigating
privacy risks. Adopting responsible AI practices ensures privacy is a core
consideration from the outset [84]. Implementing selective memory and
information filtering techniques restricts the AI’s access to sensitive data,
directly safeguarding user privacy. Removing personally identifiable
information (PII) from data used in model training is critical in protecting
privacy [85]. Continuous updates to employ the latest version of LLMs and
strict data privacy protocols for staff overseeing AI use are also essential
for maintaining privacy standards.

####  3.2.4 Robustness

With the rise of LLMs as dominant models in NLP, robustness consists of out-
of-distribution (OOD) and adversarial robustness. This section only accounts
for adversarial robustness, while OOD is discussed in Section 3.2.7. The
adversary (end-user) only attempts to jailbreak the model by explicitly
optimizing adversarial queries or adaptively making queries based on previous
outputs but will not make out-of-distribution queries asking about potentially
revoked information. Robustness has distinct definitions across various
downstream tasks of NLP; it can be commonly characterized in the following way
(It works for a range of NLP tasks like text classification and sequence
labeling): let $x$ represent the input and $y$ its corresponding correct
label. Consider a model $f$ that has been trained on data pairs $(x,y)\sim D$,
with its output prediction for $x$ given by $f(x)$. When new test data
$(x^{\prime},y^{\prime})\sim D^{\prime}$, where $D^{\prime}$ is not identical
to $D$, is introduced, the robustness of the model can be determined by its
performance on $D^{\prime}$ [86]. Through comprehensively perturbing the input
from $x$ to $x^{\prime}$, we encounter the notion of adversarial robustness,
which is a concept originating from the computer vision [87].

The adversarial robustness under the LLMs refers to the ability of models to
maintain performance when faced with inputs that have been intentionally
altered or crafted to cause the model to error, such as the malicious queries
made intentionally or unintentionally [88]. It is a type of model based on
transformations or small perturbations (e.g.typo) to study the robustness of
the model (it is also called invariance of LLMs) [89]. Typically, alterations
that maintain the underlying meaning, like modifying the text case and
contraction perturbed, are considered fairly benign [89]. In particular,
disturbances are directed at various layers of linguistic signals, including
characters, words, sentence structures, and underlying meanings. The core
objective is to replicate potential user mistakes (e.g. use of near-meaning
words), to assess the impact of minor deviations on the outcomes of LLMs [90,
82].

The defense methods for shielding LLMs from deliberate disruptions remain
under investigation [91], indicating that robust safeguarding measures are
necessary, especially during the most crucial phases of user engagement with
these models. Typically, guardrails pre-process users’ inputs to remove or
neutralize potentially adversarial content, thus preventing models from being
misled by manipulated inputs (e.g., correcting typos and standardizing input
formats). Similarly, guardrails also monitor LLM’s outputs. This may involve
establishing thresholds for specific types of responses or flagging outputs
that significantly deviate from expected patterns for review by a professional
security team.

####  3.2.5 Toxicity

An important NLP task is the toxicity detection [92], the term ‘toxicity’ is
employed as a broad descriptor, encompassing a variety of related phenomena
and linguistic contexts that may also manifest as ‘offensive’ [93], ‘abusive’
[94], ‘hateful’ [95]. Similar descriptors [92]. We show a typical example of
toxicity in Fig. 11. LLMs, as one of the prevalent developments in traditional
language modeling, are frequently trained using vast quantities of datasets,
which can include content exhibiting toxic behavior and unsafe material, such
as hate speech, offensive/abusive language, etc. Typically, a thorough
examination of toxicity is required, especially considering the employment of
LLMs for downstream tasks that might engage younger or more vulnerable
individuals, as well as the negative effects of unintended outputs from LLMs
on specific tasks [96]. The definition of what constitutes toxicity of the
LLMs varies normally, toxicity responses will be defined as rude,
disrespectful, or unreasonable responses that are likely to make an individual
leave a discussion [97]. It is, hence, very desirable to evaluate how well-
trained LLMs deal with toxicity [98].

Existing studies address the problem by focusing on representative terms in
datasets, such as identity terms [99]. To evaluate the toxicity in LLMs,
several studies have crafted trigger prompts that mirror detailed toxic
categories [100]. These studies leveraged standard toxicity metrics, such as
the Toxicity Classifier Score and PerspectiveAPI999Perspective API was
developed by Jigsaw and the Google Counter Abuse Technology team
(https://perspectiveapi.com), to determine whether the LLM’s response is toxic
[101]. However, typical metrics are susceptible to evaluator bias [102], and
encoders are perturbed [103]. Subsequently, a structured investigation
framework attempted to address this bias [104]. Despite being trained on
extensive datasets, LLMs are capable of generating outputs that can be
implicitly toxic, which are difficult to detect with straightforward, zero-
shot methods [105, 106]. This complexity arises even when prompts appear non-
toxic, underscoring the nuanced challenges in detoxifying language models,
such as depending on the specific roles assigned to LLMs, certain roles may
generate markedly more toxic outcomes [97].

Even when a generative model is trained on data characterized by low toxicity
levels, and its ability to minimize the generation of toxic text has been
validated through evaluations, it is still crucial to enforce protective
measures during live interactions between users and the model [89]. Safety
guardrails are an integral part of the user interaction and LLMs interaction
phases, playing a key role in ensuring privacy, preventing bias, and
maintaining user trust [107]. For example, Nvidia Nemo allows users to define
the toxic output they want to identify; the next step is determining the
chatbot’s response to users’ input. This involves setting up a workflow that
utilizes these definitions. Thus, this procedure is triggered whenever there
is potential exposure to toxic content, and the chatbot supports the user.
Furthermore, they ensure compliance with legal standards and align AI
operations with societal values.

####  3.2.6 Legality

Another crucial aspect of safeguarding LLMs involves managing the risks
associated with illicit101010It’s important to note that while the concepts of
legality and toxicity may overlap to some extent, they are not synonymous.
Legality is the lowest requirement, defining what is permitted under the law.
However, something that is not illegal may still be considered toxic due to
its potential to cause harm or adverse effects in other contexts. Conversely,
if something is deemed illegal, it invariably falls into the category of being
toxic, as its prohibition by law implies a recognized potential for harm or
negativity. Thus, while legality provides a clear boundary based on legal
statutes, toxicity encompasses a broader range of potentially harmful actions
or materials, some of which may not be explicitly covered by legal definitions
but are nonetheless detrimental to well-being or ethical standards.  outputs
[108]. Generally, this involves safeguarding efforts on two fronts:
implementing measures to reject inappropriate user inputs and moderating model
output to ensure it is appropriate and safe for users or downstream tasks.

During the development of LLMs, developers implement a series of measures to
ensure the safety and compliance of the models with relevant laws and
regulations. These measures include: i) researchers meticulously screen and
clean the training data before training the models to remove inappropriate,
harmful, or illegal content. This ensures the model learns from high-quality
data and avoids adopting inappropriate behaviors. ii) During model training,
human reviewers assess the samples generated by the model and offer feedback,
aiding in rectifying errors and enhancing the model’s output. This process,
alongside RLHF, helps models refine the content they generate and gradually
adopt appropriate behavior.

Once the model construction is completed and before release, models undergo
thorough and rigorous ethics and safety testing to ensure that the content
they generate is absent of inappropriate or illegal elements. One classical
approach is red teaming [109, 7], which entails simulated attacks and
adversarial testing to uncover potential vulnerabilities, ethical pitfalls,
and legal considerations. Organizations like OpenAI, Anthropic, Google, and
Meta utilize diverse methodologies for red teaming, ensuring a thorough
evaluation and effective risk mitigation. For instance, Google promotes
internal red teams 111111https://blog.google/technology/ai/google-gemini-next-
generation-model-
february-2024/, where employees with diverse expertise simulate attacks on the
AI model. In contrast, OpenAI favors external red teaming and has established
external networks 121212https://openai.com/blog/red-teaming-network to
encourage participation from outside members.

In addition to the above safeguarding efforts, monitoring systems are
established upon model release to detect inappropriate inputs and outputs.
Techniques such as natural language processing and anomaly detection are
employed for real-time identification. Upon identification of any issues,
immediate measures, such as content filtering algorithms or human intervention
protocols, are swiftly implemented to address the concern. It is worth noting
that leading LLM providers, such as Google, OpenAI, Anthropic, and Meta, offer
advanced moderation tools and techniques to developers or users, enabling
customized safeguards against illicit and inappropriate content. For instance,
Google offers PaLM-based Moderation 131313https://cloud.google.com/natural-
language/docs/moderating-text, capable of detecting more than 16 types of
inappropriate content. OpenAI provides a Moderation API
141414https://platform.openai.com/docs/guides/moderation, allowing developers
and users to customize safeguards for inappropriate content. Meanwhile,
Anthropic has developed Constitutional AI [110] and Meta utilizes Llama Guard
[9] for content moderation.

In addition to the moderation tools offered by LLM providers, notable
contributions from other entities in the field also exist. For instance,
LangChain 151515https://python.langchain.com/docs/modules/chains, an open-
source framework, simplifies and safeguards the development of applications
using LLMs. Specifically, it offers a standardized interface for creating,
combining, and customizing various components, resulting in powerful language-
driven applications. One notable application of LangChain in the legal domains
is ConstitutionalChain
161616https://api.python.langchain.com/en/latest/chains/langchain.chains.constitutional˙ai.base.ConstitutionalChain.html.
By incorporating predefined rules and guidelines, ConstitutionalChain can
filter and modify generated content to align with constitutional principles.
This ensures that responses are controlled, legal, and contextually
appropriate.

####  3.2.7 Out-of-Distribution

For a specific DNN, out-of-distribution (OOD) data strictly refers to data not
belonging to any in-distribution classes used in training. Broadly, OOD data
can be characterized as differing from the in-distribution data on certain
dimensions. Research indicates that DNNs often exhibit overconfident decision-
making when presented with OOD data. This has led to widespread investigation
of OOD detection issues across domains such as computer vision [111], and
natural language processing [112]. However, the OOD detection task within the
field of NLP presents notable challenges, particularly exacerbated by the
presence ofLLMs. This issue has resulted in limited research focused on OOD
detection specifically tailored to LLMs [113], primarily due to the immense
training corpora used for LLMs, making it difficult to define precisely what
data has not been utilized for training. Moreover, the generative nature of
LLMs adds another layer of complexity to defining the OOD problem [114].

While defining OOD instances for an LLM is generally very difficult, if not
impossible, it becomes more feasible when applied to specific real-world
scenarios where the context is more precise. In practical scenarios, OOD
instances can be defined as data irrelevant to the main task or significantly
deviating from normal ones. For instance, recent work [115] has explored the
evaluation of OOD in the context of specific language model applications, such
as text classification [116], sentiment analysis [117], machine reading
comprehension [118], and found that it can lead to a significant performance
decrease, even with minor semantic shifts caused by small perturbations. To
mitigate the impact of OOD on model performance in practical tasks, strategies
such as setting up anomaly input filtering mechanisms
171717https://hub.guardrailsai.com/validator/guardrails/unusual˙prompt or
constructing OOD detectors tailored to the task can be employed.

####  3.2.8 Uncertainty

A key aspect of LLMs’ trustworthiness lies in their ability to discern their
outputs’ reliability and correctness, a concept central to uncertainty
quantification. This approach is an effective method for assessing risks,
aiming to gauge the confidence levels of LLMs in their predictions. Elevated
uncertainty suggests that an LLM’s output may require rejection or additional
scrutiny. Fig. 13 shows an example of uncertainty. The effectiveness of
uncertainty quantification is further contingent on the alignment between the
model’s predicted confidence and its actual accuracy, essentially measuring
the model’s calibration.

There has been a growing focus on research to quantify the overall uncertainty
in LLMs. Establishing dependable uncertainty metrics is essential for
enhancing the safety of LLM systems. Recent studies have noted that the
calibration of LLMs is improved relatively through techniques like combining
multiple reasoning chains [119], integrating different prompts [120], or by
prompting LLMs to output their confidence levels directly [114]. In addition
to these observations, numerous methods have been developed to quantify the
uncertainty in LLMs effectively. [121] demonstrated that a GPT-3 model can
learn to articulate uncertainty regarding its responses in natural language
independently of using model logits. [122] comprehensively compared various
popular approaches to construct a well-calibrated prediction pipeline for pre-
trained language models. [123] unveiled KnowNo, a framework designed to
measure and align the uncertainty in LLM-based planners, enabling them to
recognize their limitations and seek assistance when necessary.

The primary hurdles in assessing LLM uncertainty arise from the pivotal roles
of meaning and structure in language. This pertains to what linguists and
philosophers define as a sentence’s semantic content and syntactic or lexical
framework. While foundation models mainly output token-likelihOODs, reflecting
lexical confidence, the meanings often hold the most significance in most
applications. [124] introduced the concept of semantic entropy, which
integrates linguistic consistencies arising from identical meanings. The
fundamental method involves a semantic equivalence relation, denoted as
$\mathbb{E}(s_{i},s_{j})$, where $s_{i}$ and $s_{j}$ represent output
sentences corresponding to a given input. This equivalence relation is said to
hold when two sentences $s_{i}$ and $s_{j}$ convey the same meaning, implying
that they belong to the same cluster $C$. The semantic entropy is defined as

$H(C|x)=-\sum_{C}\mathbb{P}(C|x)\ln\mathbb{P}(C|x),$ (1)

where $x$ is the input sentence. This methodology, which employs ‘out-of-the-
box’ models, enhances reproducibility and simplifies deployment. Moreover,
this unsupervised uncertainty could address the issue identified in previous
research, where supervised uncertainty measures often falter in the face of
distributional shifts.

Utilizing the above uncertainty technologies to build a guardrail for LLMs, it
is crucial to integrate mechanisms that enable the model to assess and
communicate its uncertainty. This involves training the model to recognize
when a query falls outside its expertise or when the answer is speculative. It
also involves responding appropriately—whether by providing a cautious answer,
flagging the response as uncertain, or directing the user to more reliable
sources.

##  4 Overcome and Enhance Guardrails

Implementing advanced safeguarding techniques, as discussed in Section 3, has
played a crucial role in enhancing their security and reliability within LLMs.
However, [125] indicated that employing guardrails does not enhance the
robustness of LLMs against attacks. They examined the external guardrails such
as ModerationEndpoint, OpenChatKitModeration Model, and Nemo, showing that
they only marginally reduce the average success rate of jailbreak attacks.
Jailbreak attacks, referred to as “jailbreaks”, aim to exploit language
models’ inherent biases or vulnerabilities by manipulating their responses.
These successful attacks allow users to circumvent the model’s safeguard
mechanisms, restrictions, and alignment, potentially leading to generating
unconventional or harmful content or any content controlled by the adversary.
By bypassing these constraints, jailbreaks empower the model to produce
outputs that exceed the boundaries of its safety training and alignment.

Therefore, in this section, we explore current methods used to bypass the
guardrails of LLM. In Table II, we compare different jailbreaks on: (1)
Attacker access type: white box, black box, and gray box. In a white-box
scenario, the attacker has full visibility into the model’s parameters. A
black-box situation restricts the attacker from observing the model’s outputs.
In a grey-box context, the attacker has partial access, typically to some
training data. (2) Prompt level for manipulation: user prompt or system
prompt. User prompts are those where the input prompt is specified by the
user, allowing for personalized or targeted inputs. On the other hand, system
prompts are generated automatically by models and may include outputs that
attackers craftily devise to deceive or manipulate the system’s response. (3)
Core technique: the main technique used to attack the LLM. (4) Stealthiness:
high stealthiness represents that the attack is difficult to notice by a
human, which is supposed to be some logical, semantic, and meaningful
conversation rather than some gibberish. (5) GPT4 Evaluation: As many
jailbreaks are not directly targeted for LLMs with guardrails, and GPT4 has
its default guardrail, then evaluation on GPT4 can be seen as a surrogate
metric for comparison. (6) Target manipulated property of generated response:
toxicity, privacy, fairness, and hallucination

TABLE II: Comparison among Different Jailbreaks for (Guarded) LLMs

Attack | Access Type | Prompt Level | Core Technique | Stealthiness | GPT4 Evaluation | Targeted Property
---|---|---|---|---|---|---
GCG [65] | White | User | Greedy Gradient-based Search | Low | $\times$ | Harmful Content
PGD [126] | White | User | Continuous Relaxation & Entropy projection | Low | $\times$ | Harmful Content
PRP* [127] | White | User | In-context Learning & two-step prefix-based | Low | $\times$ | Harmful Content
AutoDAN-Liu* [128] | White | System+User | Hierarchical Genetic Algorithm | High | $\checkmark$ | Harmful Content
AutoDAN-Zhu* [129] | White | User | Double-loop Optimization | High | $\checkmark$ | Harmful Content & prompt leaking
COLD-Attack* [130] | White | User | Langevin dynamics | High | $\times$ | Harmful Content
ProMan [131] | White | - | Generation Manipulation | - | $\times$ | Harmful Content & Privacy Leakage
JailBroken [132] | Black | System | Failure modes as guiding principles | Low | $\checkmark$ | Harmful Content & personally identifiable information leakage
DeepInception [133] | Black | User | Nested instruction | Medium | $\checkmark$ | Harmful Content
DAN* [125] | Black | User | Characterizing in-the-wild prompt | High | $\checkmark$ | Harmful Content
ICA [134] | Black | User | In-context learning ability of LLM | Low | $\times$ | Harmful Content
SAP [135] | Black | User | In-context learning ability of LLM | Medium | $\times$ | Harmful Content
DRA [136] | Black | User | Making Them Ask and Answer | Low | $\checkmark$ | Harmful Content
CipherChat [137] | Black | System | Long-tail: cipher | High | $\checkmark$ | Harmful Content
MultiLingual [138] | Black | User | Long-tail: low-resource | High | $\checkmark$ | Harmful Content
LRL [139] | Black | User | Long-tail: low-resource | High | $\checkmark$ | Harmful content
CodeChameleon [140] | Black | User | Long-tail: encrypts | High | $\checkmark$ | Harmful Content
ReNeLLM [141] | Black | User | Prompt rewriting & scenario nesting | High | $\checkmark$ | Harmful Content
PAIR [142] | Black | System | Automatic Iterative Refinement | High | $\checkmark$ | Harmful content
GPTFUZZER [143] | Black | User | Fuzzing | Low | $\checkmark$ | Harmful content
TAP [144] | Black | System | Tree-of-thought reasoning | Medium | $\checkmark$ | Harmful content
Mosaic Prompts [145] | Black | User | Semantic censorship | High | $\times$ | Impermissible content
EasyJailbreak [146] | Black | System+User | Unified framework for 12 jailbreaks | - | $\checkmark$ | Jailbreak attack evaluation
PROMPTINJECT [147] | Black | User | Mask-based iterative strategy | Low | $\times$ | Goal hijacking & prompt leaking
IPI [148] | Black | System | Indirect prompt injection | High | $\checkmark$ | Cyber threats like theft of data and denial of service etc.
HOUYI [149] | Black | User | SQL injection & XSS attacks | Low | $\times$ | Prompt abuse & prompt leak
GA [150] | Black | User | genetic algorithm | Low | $\times$ | Harmful content
GUARD [151] | Black | System+User | Role-playing LLMs | High | $\times$ | Harmful content
CIA [152] | Black | User | Combination of multiple instructions | Medium | $\checkmark$ | Harmful content
Pelrine et al. [153] * | Grey | User | fine-tuning | Low | $\checkmark$ | Misinformation & Privacy Leakage
Zhang et al. [154] | Grey | User | fine-tuning | Low | $\checkmark$ | Harmful content
Safety-tuned [155] | Grey | User | fine-tuning | Low | $\times$ | Harmful content
Janus inference [156] | Grey | System | Fine-tuning | Low | $\times$ | Privacy Leakage
Qi et al.[157] | Grey | User | fine-tuning | Low | $\times$ | Harmful content
Pelrine et al. [153] | Grey | User | poisoning knowledge retrieval | Medium | $\checkmark$ | Harmful content & fairness
PoisonedRAG[158] | Grey | System | poisoning knowledge retrieval | Low | $\checkmark$ | Hallucination
AutoPoison [159] | Grey | System | Content Injection | Low | $\times$ | Triggered response
LoFT [160] | Grey | System | fine-tuning | Low | $\checkmark$ | Harmful content
BadGPT [161] | Grey | System | Backdoor Attack | Medium | $\times$ | Harmful content
ICLAttack [162] | Grey | System | Backdoor Attack | Low | $\times$ | Triggered response
ActivationAttack [163] | Grey | System | Activation Steering | Low | $\times$ | Harmful & Biased content

###  4.1 White-box Jailbreaks

A white-box attack normally refers to a scenario in which the attacker has
full access to the internal details of the model. Since even LLMs with
guardrails can not fully protect against adversarial attacks, we introduce
some techniques for attacking LLMs (with or without guardrails) under the
white-box setting. Notably, most white-box attacks can be applied to the
black-box scenario by employing their transferability on a white-box surrogate
model.

####  4.1.1 Learning-based Methods

The Greedy Coordinate Gradient (GCG) [65] method was designed to search a
specific sequence of characters (an adversarial suffix). When the adversarial
suffix is attached to different queries, it misleads the LLM to generate a
response with harmful content. This approach integrates greedy search and
gradient-based methods for discrete optimization to manipulate the model’s
outputs. It aims to optimize the likelihood that the model will generate an
affirmative response, e.g., “Sure, this is…”. To improve the computational
efficiency of GCG, [126] revisited Projected Gradient Descent (PGD) for LLMs,
which has been widely used for generating adversarial perturbations in other
domains, by controlling the error introduced by the continuous relaxation for
the input prompt, it can fool LLMs with the similar attack performance but
with up to one order of magnitude faster. Previous techniques in traditional
NLP, like Gradient-based Distributional Attack (GBDA) [164], can also be used
to search adversarial suffixes. Specifically, it also applies continuous
relaxation usingGumbel-Softmax [165], which allows for the manipulation of
text inputs in a gradient-guided manner, maintaining the textual data’s
discrete nature while optimizing the adversarial objective. However, it fails
to obtain high jailbreaking performance on the aligned LLMs.

On the other hand, the adversarial suffixes produced by GCG are mostly some
garbled characters that are easily detectable by simple perplexity filter
[166]. AutoDAN-Zhu [129] design a double-loop optimization method upon GCG to
generate more stealthy jailbreak prompts. In addition, it also demonstrates
the ability and interpretability to solve other new tasks like prompt leaking.
Furthermore, COLD-Attack [130] adopts a new jailbreak, to automate the search
of adversarial LLM attacks under a variety of restrictions such as fluency,
stealthiness, sentiment, and left-right-coherence. It performs efficient
gradient-based sampling in the continuous logit space and relies on a guided
decoding process to translate the continuous logit sequences back into
discrete text.

Although the adversarial suffix can lead the base LLM to generate harmful
responses, the LLM models with guardrails can easily detect it. Mangaokar
et.al. [127] proposed a new attack strategy, named PRP, for attacking LLM
models with guardrails mainly. Specifically, it leverages a two-step prefix-
based attack, including universal adversarial prefix construction and prefix
propagation to the response. Inserting the universal prefix into the response
can elicit the guardrail for outputting the harmful content. After the
universal prefix generation, the corresponding propagation prefix can be
created through a few in-context templates. Such in-context learning enables
the LLM to initially output the pre-computed or desired adversarial prefix,
eventually making PRP jailbreak the LLM models with guardrails. Subsequently,
AutoDAN-Liu [128]proposed to generate stealthy jailbreak prompts
automatically, it utilizes a hierarchical genetic algorithm to bypass the
ethical guidelines and safety measures of LLMs. This method is grounded in
optimization techniques inspired by natural selection. It iteratively refines
generations of prompts to circumvent built-in safeguards effectively. Through
this evolutionary process, AutoDAN-Liu generates stealthy prompts that subtly
avoid triggering the model’s protective mechanisms.

####  4.1.2 LLM Generation Manipulatation

On the other hand, except for jailbreaking via some learning-based method,
ProMan [131] was proposed to directly manipulate the generation process of
those open-source LLMs and enforce the LLMs to generate specific tokens at
specific positions, therefore cheating the LLMs to generate undesired
response, including harmful or sensitive information or even private data.

Although the current literature only describes limited white-box jailbreaks,
it is still possible to bypass the guardrails if full access is provided. For
example, suppose the attacker knows the guardrail used for the targeted LLM is
Llama Guard, and the adversary has full access to the fine-tuned Guard model.
In that case, the previous white-box attacks using optimization can be further
extended: the optimization space will be further narrowed by adding an extra
constraint, i.e., the adversarial input and/or the resulting response of LLMs
are supposed to be evaded by the guardrail. In other words, a successful
jailbreak should satisfy the safety conditions of the targeted LLM and the
Guard models.

###  4.2 Black-box Jailbreaks

Unlike white-box attacks, which necessitate access to model weights and
tokenizers, black-box attacks operate under the premise that adversaries lack
knowledge of the LLM’s internal architecture or parameters. Therefore, they
are more common. In this subsection, jailbreak attacks conducted within a
black-box setting are classified into four categories: i) manually designed
jailbreaks, ii) attacks exploiting long-tail distribution, iii) optimization-
based methods for generating jailbreaks, and iv) unified framework for
jailbreaking.

####  4.2.1 Delicately Designed Jailbreaks

The phenomenon of jailbreak attacks against state-of-the-art large LLMs was
investigated in JailBroken [132], explicitly focusing on models such as GPT-4,
GPT-3.5 Turbo, and Claude v1.3. This work identifies two primary reasons for
the successful attack: competing training objectives and instruction tuning
objectives. The authors propose these two failure modes as guiding principles
for designing new jailbreak attacks. Using carefully engineered objectionable
prompts, they empirically evaluate these attacks against the aforementioned
safety-trained LLM models. The results indicate a high success rate regarding
a large number of the attacks.

In this line, due to the constant evolution of ethical and legal constraints
embedded within LLM safeguards, jailbreak attempts employing direct
instructions [132, 142] are typically easily identified and rejected. It is
motivated by the Milgram shock experiment [167, 168] and its adaptation to
LLMs, which follow authoritative commands to produce harmful content. A
prompt-based jailbreak method, called DeepInception[133], was devised for
conducting a black-box attack on LLMs. This involves injecting an inception
mechanism into a LLM and effectively hypnotizing it to act as a jailbreaker.
DeepInception explicitly constructs a nested scene to serve as the inception
for guiding the behavior of the LLM. This nested scene facilitates an adaptive
approach to circumvent safety constraints in a normal scenario, opening up the
potential for subsequent jailbreaks. Specifically, DeepInception utilizes the
personification capability of LLMs, along with their tendency to follow
instructions, to generate diverse scenes or characters. [125] furthered this
by creating prompts that encourage ChatGPT to act as a DAN (“Do Anything
now"). As implied by their designation, LLMs are now capable of boundless
functions. They are no longer bound by the customary rules that govern AI
systems.

While existing attack methods are typically applied to new conversations
devoid of context, the potential of In-Context Learning (ICL) was delved into
the influence of the alignment ability of LLMs. Leveraging these insights, the
study introduces the In-Context Attack (ICA) [134]. ICA is tailored to
construct malicious contexts to direct models to produce harmful outputs. The
efficacy of in-context demonstrations in aligning LLMs is demonstrated, and
implementing these methods is straightforward. Additionally, Deng et.al. [135]
proposed a semi-automatic attack framework named Semi-Automatic Attack Prompt
(SAP), it combines manual and automatic methods to generate prompts to mislead
LLMs to output harmful content. Specifically, they manually construct high-
quality prompts as an initial prompt set and then iteratively update them
through in-context learning with LLMs. Through this red-teaming attack,
extensive high-quality attack prompts can be efficiently generated. Liu et.al.
[136] proposed a novel universal jailbreak approach named DRA (Disguise and
Reconstruction Attack. This method involves concealing harmful instructions
via disguise, prompting the model to uncover and reconstruct the original
harmful instruction within its generated output, thus navigating around
traditional security measures. In this way, the harmful input can be disguised
from the input filter, which then guides the target to reconstruct the attack
to obtain the desired response from the adversary.

####  4.2.2 Exploiting Long-tail Distribution

Jailbreaks relying on long-tail distributed encoding convert the original
query into rare or unique data formats such as ciphers [137], low-resource
languages [138], and personalized encryption methods [140]. The safety
vulnerability of LLMs when user queries are encrypted was investigated in
CipherChat [137]. The framework involves encoding malicious unsafe text using
LLMs and assessing the safety of the decoded responses. CipherChat is designed
with three key elements in its system prompt to ensure effective communication
through ciphers: (1) behavior assignment, (2) cipher teaching, and (3)
enciphered unsafe demonstrations. It enables users to interact with LLMs using
cipher prompts, system role descriptions, and few-shot enciphered
demonstrations. Furthermore, the authors introduce SelfCipher, which utilizes
a hidden cipher embedded within LLMs to circumvent safety features more
efficiently than existing ciphers.

Afterward, despite the widespread use of English globally, there is growing
concern that the safety of LLMs is predominantly assessed in English alone.
However, MultiLingual [138] takes a significant stride forward by
investigating the safety levels of LLMs across various languages, including
those with limited linguistic resources. This research delves into the
vulnerabilities of LLMs from two perspectives: unintentional and intentional
scenarios. In the unintentional scenario, queries translated into non-English
languages unexpectedly expose users to unsafe content. Conversely, the
intentional scenario involves using translated multilingual “jailbreak”
prompts. Similarly, Low Resource Languages-Combined Attacks [139] (LRL) also
underlines the cross-lingual vulnerability of GPT-4. By translating unsafe
English prompts into less commonly used languages, they successfully
circumvent protective measures to elicit harmful responses. [3] show that
instruction-following language models using the text-davinci-003 prompt could
potentially be employed to produce malicious content.

A hypothesis regarding LLMs’ safety mechanisms was proposed in subsequent
research, suggesting that LLMs first detect intent before generating
responses. Building on this hypothesis, a framework was introduced known as
CodeChameleon, which encrypts and decrypts queries into a format challenging
for LLMs to detect [140]. Four distinct encryption functions are employed
during the encryption stage based on reverse order, word length, odd and even
positions, and binary tree structure. Subsequently, the decryption functions
are incorporated into the instructions as code blocks. During inference, these
decryption functions assist LLMs in understanding the encrypted content.
Extensive testing demonstrates that CodeChameleon effectively circumvents
LLMs’ intent recognition.

####  4.2.3 Optimization-based Approaches

In contrast to conventional adversarial examples, such jailbreaks are usually
created through human ingenuity, strategically devising situations that
naturally mislead the models [132], rather than relying on automated
techniques. Consequently, crafting them demands considerable manual labor. The
adversarial prompts generated Greedy Coordinate Gradient (GCG) [65] exhibit a
high degree of universality and transferability, particularly to other fully
black-box models.

To avoid limitations regarding intricate manual design [132, 137] and require
optimization on other white-box models, compromising generalization or
efficiency [65], a method known as ReNeLLM was introduced [141]. ReNeLLM is an
automatic jailbreak prompt generation framework, which generalizes jailbreak
prompt attacks into two aspects: (1) Prompt Rewriting and (2) Scenario
Nesting.

Following this, Prompt Automatic Iterative Refinement (PAIR) proposed an
automated red teaming method for jailbreaking LLMs [142], which represents a
significant improvement of over ten thousand times compared to existing
attacks, such as jailbreaks identified through Greedy Coordinate Gradient
(GCG) [65]. The authors aim to find a balance between prompt-level attacks
[169], which are labor-intensive but scalable, and token-level attacks [170],
which are uninterpretable and inefficient in terms of queries. PAIR devised a
protocol leveraging a language model to craft prompt-level attacks that are
both semantic and human-interpretable. This involves an automated system where
the attacker language model learns from prior prompts and responses to refine
based on a judge score and generate new prompts. Through in-context learning,
PAIR enabled the language model to enhance the quality of generated candidate
queries autonomously.

Drawing inspiration from AFL fuzzing, GPTFUZZER was introduced, a black-box
jailbreak fuzzing framework to autonomously generate jailbreak prompts [143].
GPTFUZZER aims to combine the efficacy of human-written prompts with the
scalability and flexibility of automated systems to bolster the assessment of
vulnerabilities in LLMs. The framework is built upon a seed selection
strategy, mutate operators, and a judgment model. By harnessing these
elements, GPTFUZZER can systematically detect and exploit vulnerabilities in
LLMs.

Building upon prior automated methodologies, Tree of Attacks with Pruning
(TAP) introduced a novel approach for generating jailbreaks [144]. TAP
leverages an LLM to iteratively refine candidate prompts using tree-of-thought
reasoning until a successful jailbreaking prompt is generated. The framework
involves three key components: an attacker LLM tasked with generating
jailbreaking prompts using tree-of-thought reasoning, an evaluator responsible
for assessing the generated prompts and determining the success of the
jailbreaking attempt, and a target LLM that serves as the subject of the
jailbreaking endeavor. Lapid et.al. [150] employed the genetic algorithm (GA)
for generating a universal adversarial suffix under the black-box setting.
Instead of maximizing the targeted token likelihOOD in GCG, they proposed
using random subset sampling for fitness approximation by minimizing the
cosine similarity between benign input embedding and adversarial input
embedding. Experiments illustrate high transferability across different LLMs.
Furthermore, Jin et.al. [151] proposed a role-playing system named Guideline
Upholding through Adaptive Role-play Diagnostics (GUARD), which allocates four
distinct roles to user LLMs to collaborate on new jailbreaks. By collecting
some existing jailbreak prompts into a knowledge graph and using Chain-of-
Thought to align with the specific functions and objectives for each role,
they can generate a higher jailbreak success rate and a lower perplexity score
than GCG [65] and AutoDAN [129].

####  4.2.4 Unified Framework for Jailbreaking

A recent development, EasyJailbreak [146], presents a comprehensive framework
to evaluate jailbreak attacks on LLMs. This framework integrates four pivotal
components: Selector, Mutator, Constraint, and Evaluator. This approach allows
researchers to concentrate on crafting unique components, thus minimizing the
effort required for development. Moreover, it demonstrates broad model
compatibility, accommodating various models, including open-source
alternatives like LlaMA2 and proprietary ones like GPT-4.

####  4.2.5 Prompt Injection for Desired Responses

Prompt Injection in LLMs involves the malicious alteration of input provided
to the model, commonly achieved by substituting original instructions with
carefully crafted user input [171]. This manipulation occurs within the
framework of supplying prompts to the LLM, guiding its responses or behaviors.
Prompt injection attacks present a significant cybersecurity risk as they can
result in creating unauthorized content, circumventing content moderation
protocols, exposing sensitive data, or even facilitating the dissemination of
malicious code or malware. This vulnerability is particularly prominent in
LLMs that employ prompt-based learning approaches, rendering them susceptible
to exploitation by malicious attackers. Given the significant role of prompt
in shaping LLM output, prompt injection manipulation can have widespread
implications for attacking LLMs.

Since most LLMs, such as ChatGPT, are closed-source platforms, much of the
research centers on utilizing prompt engineering techniques to induce ChatGPT
to generate inappropriate content. The framework known as PROMPTINJECT [147]
was proposed as a straightforward alignment mechanism for generating iterative
adversarial prompts through masks. This approach involves assembling prompts
to facilitate a quantitative assessment of the robustness of LLMs against
adversarial prompt attacks. The study focuses primarily on evaluating the
susceptibility of GPT3 to such attacks, accomplished through simplistic
handcrafted inputs. The analysis concentrates on two types of attacks: goal
hijacking and prompt leaking. Goal hijacking involves introducing a malicious
string, termed a rogue string, designed to divert the model into generating a
particular sequence of characters. Conversely, prompt leaking pertains to the
possibility of revealing a private value embedded within a confidential
prompt, which should not be disclosed externally under any circumstances.
Following this, the concept of Indirect Prompt Injection (IPI) [148] was
introduced, referring to an uninvestigated attack vector where retrieved
prompts can function as “arbitrary code”, thus compromising LLM-integrated
applications. The authors demonstrate these attacks against real-world systems
like Bing Chat, code-completion engines, and GPT-4.

Inspired by traditional injection attacks, a novel black-box prompt injection
attack technique called HOUYI [149] was introduced. HOUYI comprises three
essential components: a preconstructed prompt, an injection prompt, and a
malicious question, each tailored to achieve the adversary’s goals. Two
significant exploit scenarios were identified: prompt abuse and prompt leak.
The application of HOUYI to a sample of 36 real-world LLM-integrated
applications revealed that 31 of these applications are vulnerable to prompt
injection. [145] raised a concept of semantic censorship, which falls into the
category of guardrail using a universal algorithm to determine whether the
content generated by an LLM is permissible based on semantic content alone.
Accordingly, they proposed a novel attack named Mosaic Prompts; it leverages
the ability of a user to query an LLM multiple times in independent contexts
to construct impermissible outputs from a set of permissible ones. This
indicates a significant limitation of output censorship, as it cannot provide
safety or security guarantees without imposing severe restrictions on model
usefulness. Moreover, Compositional Instruction Attack (CIA) [152] capitalizes
on LLMs’ failure to detect underlying harmful intents when instructions are
composed of multiple elements, thus revealing significant vulnerabilities in
LLM security mechanisms. They outline two specific strategies, Talking-CIA
(T-CIA) and Writing-CIA (W-CIA), developed to automate the generation of these
deceptive instructions. T-CIA leverages psychological principles to align the
model’s response persona with the harmful intent, bypassing LLMs’ ethical
constraints. Conversely, W-CIA disguises harmful prompts as creative writing
tasks, exploiting LLMs’ lack of judgment on fictional content to elicit
dangerous outputs.

###  4.3 Gray-box Jailbreaks

In [153], the authors highlight that beyond the white-box approach, which
involves full access to a model’s parameters, and the more limited black-box
method, there’s also ‘grey-box’ access. This middle ground could be crucial in
uncovering additional vulnerabilities in the safeguard systems of LLMs. This
section will present studies on ‘grey-box’ attack methods, encompassing
strategies like fine-tuning, retrieval-augmented generation, and backdoor
attacks.

####  4.3.1 Fine-tuning Attacks

Fine-tuning technology enables users to customize pre-trained LLMs
effectively. However, when these fine-tuning privileges are extended to end-
users, the existing guardrails may not be sufficient to prevent harmful
behaviors. The attacks by fine-tuning the LLMs can also be called ’grey-box’
attacks. [154] suggested that fine-tuning could mitigate Reinforcement
Learning with Human Feedback (RLHF) safeguards, commonly employed in LLMs to
minimize harmful outputs. Their research revealed that even ChatGPT 4 could
have its protections removed by fine-tuning. Through experiments, they
demonstrated a success rate of 95% in generating harmful responses from
ChatGPT 4, using just 340 examples for fine-tuning. The experiments from [153]
also indicate that fine-tuning a model with only 15 harmful or 100 benign
examples can compromise the safeguards of GPT-4, leading to a variety of
harmful outputs. [155] also examines the potential safety risks associated
with excessive instruction tuning in LLM, illustrating that models excessively
tailored to specific instructions may still generate harmful content. To
counteract these risks, the researchers suggest developing a safety-focused
tuning dataset to balance the dual objectives of maintaining model performance
while enhancing safety measures. Furthermore, the research by [156] highlights
the risks of fine-tuning language models using small datasets containing
personally identifiable information (PII). Initially, it focuses on a simple
approach where a language model is fine-tuned with a small dataset rich in
text-based PII, which results in the model being more likely to divulge PII
upon prompting. Then, the researchers introduced the "Janus" methodology,
which centers around defining a PII recovery task followed by few-shot fine-
tuning. Experimental findings demonstrate that fine-tuning GPT-3.5 with just
10 PII examples markedly increases the model’s ability to expose PII. [157]
found that additional training of the model can compromise the effectiveness
of established guardrails. They bypass the GPT-3.5 Turbo’s safety guardrails
by fine-tuning it with justten0 specific examples and successfully make the
model entirely susceptible to harmful instructions.

####  4.3.2 Retrieval-Augmented Generation (RAG)

RAG for LLMs aims to improve the response of LLMs by incorporating external
datasets during inference. It integrates context and up-to-date or relevant
information in the prompt to enhance the LLM’s performance. [153] finds that
employing the prompt injection techniques suggested by [147] indicated that
polluting the external dataset by injecting a malicious instruction could
successfully invalidate ChatGPT 4’s safety protection. They also demonstrated
that if biased system messages accompany the upload of factual data, it can
bias the responses of ChatGPT. [158] also proposed to inject toxic texts into
the knowledge database to compromise LLMs. They developed these poisoned texts
by forming them to solve an optimization problem aimed at generating a target
response chosen by the attacker. Their experiments showed that by injecting
just five tainted texts tailored to a specific question, they were able to
attain a 90% attack success rate.

####  4.3.3 Backdoor Attack

The backdoor attack on the neural language process task is to manipulate the
model to produce specific outputs when triggered [172]. It typically occurs
during the pre-training and adaptation tuning, where the backdoor trigger gets
embedded [173]. These manipulations should maintain the model’s performance
and evade detection by human inspection. The backdoor is triggered exclusively
when input prompts to LLMs include the embedded trigger, causing the
compromised LLMs to act maliciously as intended by the attacker. [159] propose
Auto Poinson to incorporate training examples that reference the desired
target content into the system, triggering similar behaviors in downstream
models. [160] introduces LoFT (Local Proxy Fine-tuning) to fine-tuning
smaller, local proxy models to develop attacks that are more likely to
transfer successfully to larger, more complex LLMs. This technique leverages
the target LLMs to produce prompts closely aligned with harmful queries,
effectively gathering prompts from a localized vicinity around these queries.
A set of parameters in the proxy LLM is then fine-tuned, guided by the
responses of the target LLM to these analogous prompts. Ultimately, this fine-
tuned proxy model is deployed to attack the target LLMs. The study
demonstrates that this method improves the transferability of attacks. [161]
proposed the BadGPT, a backdoor attack targeting RL fine-tuning in language
models. It injects a backdoor trigger into the reward model during the fine-
tuning stage, allowing for compromising the fine-tuned language model. [162]
then proposed ICLAttack, which fine-tunes models by targeting in-context
learning for backdoor attacks. This method focuses on two prompt-level
strategies: introducing compromised examples within the prompt’s demonstration
set and modifying the prompts. This technique operates at the prompt level,
eliminating the necessity to train new LLMs altogether. On the other hand,
Wang et.al.[163] pointed out that poisoning the training dataset or
introducing harmful prompts affects the adaptability of the attacks, rendering
them more prominent and more accessible to identify. They propose using
activation steering without optimization to target four key aspects of LLMs:
truthfulness, toxicity, bias, and harmfulness.

###  4.4 Techniques for Strengthening LLMs

This section discusses techniques that may help construct more powerful
defenses for guardrails or more robust LLMs.

####  4.4.1 Detection-based Methods: Guardrail Enhancement

To detect the harmful information in the user’s input, PPL [174] calculates
the perplexity of a provided input to decide whether a user’s request should
be accepted or rejected. SmoothLLM [175] borrowed the idea of randomized
smoothing literature [176], it randomly alters multiple versions of a given
input and then combines the respective predictions to identify adversarial
inputs. Some researchers have explored how In-Context Learning (ICL) can
impact the alignment capabilities of LLMs. In-Context Defense (ICD) [134]
method is designed to bolster model resilience by demonstrations of rejecting
to answer harmful prompts via in-context demonstration.

To defend LLM attacks, LLM SELF DEFENSE [177] was proposed first.
Specifically, by incorporating the generated content into a pre-defined prompt
and using another instance, LLM, to analyze the text, it constructs an extra
guardrail filter for preventing harmful content. Furthermore, Cao et.al. [178]
proposed Robustly Aligned LLM (RA-LLM) to defend against potential alignment-
breaking attacks. Unlike the previous alignment check, which uses the
alignment check function to decide whether to reject the response, the
proposed Robust Alignment Check Function adds several extra random droppings
on the request. It usually checks whether the corresponding response can still
pass the alignment check function AC. Then, Chen et.al. [179] designed a
moving target defense (MTD) to enhance the LLM system. Compared to previous
guardrail methods that decide whether the input/output is safe, MTD calculates
a composite score for each response by combining its quality and toxicity
metrics. It employs randomization to select a response that qualifies both
response metrics, eventually providing a solid moving target defense for the
LLMs.

####  4.4.2 Mitigation-based Methods: Affirmative Response Generation

As shown in. [166], besides perplexity filtering, input preprocessing like
Retokenization and Paraphrase can also successfully compromise the
effectiveness of some attacks like GCG [65]. However, adversarial training,
though once favored for safeguarding image classifiers, faces diminished
appeal for LLMs due to the prohibitive expenses associated with both model
pre-training and the creation of adversarial attacks, rendering large-scale
adversarial training impractical. Finding a good approximation for robust
optimization objectives that allow for successful adversarial training remains
an open challenge. Further, Li et.al. [180] introduced a novel inference
method, Rewindable Auto-regressive INference (RAIN) enables pre-trained LLMs
to assess their own outputs and leverage the assessment outcomes to inform and
steer the backtracking and generating content to enhance AI safety. Contrary
to Reinforcement Learning from Human Feedback (RLHF), RAIN dispenses with the
requirement for extra model upkeep and bypasses the accumulation of gradient
data and computational graphs. Still, it must pay the extra but acceptable
cost of the auto-regressive inference. Additionally, Zhang et.al. [181]
proposed to integrate goal prioritization (GP) at both training and inference
stages. It analyzes the reason behind successful jailbreaking: the conflict
between two goals: helpfulness (providing helpful responses to user queries)
and safety (providing harmless and safe responses to user queries. The
jailbreak attack success rate can be notably decreased by plugging in the goal
prioritization for these two properties into the inference alone or with
training.

Further, Self-Reminder [182] suggests that adding self-reminder prompts can be
an effective defense. They speculate that initiating ChatGPT with a ‘system
mode’ prompt at the most external level to remind it of its role as a
responsible AI tool could reduce its vulnerability to being malevolently
steered by user inputs at a deeper level. Therefore, by concatenating an extra
system prompt after the user’s query that reminds the LLMs to respond
responsibly, the experimental results showed that self-reminders significantly
reduce the success rate of jailbreak attacks. Then, Ge et.al. [183] proposed a
multi-round automatic red-teaming framework MART to improve the scalability of
safety alignment. Two players, i.e., an adversarial LLM and a target LLM,
iteratively interplay with each other. The adversarial LLM aims to generate
challenging prompts that provoke unsafe responses from the target LLM.
Concurrently, the target LLM is refined with data that aligns with safety
standards based on these adversarial prompts. Through several rounds of red-
teaming, the enhanced target LLM continues to bolster its defenses through
safety-specific fine-tuning. Further, Zhou et.al. [184] proposed the first
adversarial objective aimed at protecting language models from jailbreaking
attacks, along with a novel algorithm, Robust Prompt Optimization (RPO). This
strategy employs gradient-based token optimization (similar to GCG) to ensure
the generation of harmless outputs. RPO represents the initial approach in
jailbreaking defense (like adversarial training in vision) that enhances
robustness comprehensively and effectively and at only a minor cost to normal
use. SafeDecoding [185] found that despite the likelihood of tokens signifying
harmful content being higher than those for harmless responses, safety
disclaimers continue to emerge among the highest-ranking tokens when sorted by
probability in descending order. Thus, in the training phase, the model will
be fine-tuned with a few safety measures, and then, during the inference,
SafeDecoding further constructs the new token distribution. The crafted
probability distribution reduces the chances of tokens that resonate with the
attacker’s objectives and enhances the probabilities of tokens that align with
human values.

##  5 Discussions: A Complete Guardrail

Based on the discussions about tackling individual requirements in Sections 3
and 4, this section advocates building a guardrail by systematically
considering multiple requirements. We discuss four topics: conflicting
requirements (Section 5.1), multidisciplinary approach (Section 5.2),
implementation strategy (Section 5.3), rigorous engineering process (Section
5.4), and safeguards for LLM Agents (Section 5.5).

###  5.1 Conflicting Requirements

This section discusses the tension between safety and intelligence as an
example of the conflicting requirements. Conflicting requirements are typical,
including, e.g., fairness and privacy [186], privacy and robustness [187], and
robustness and fairness [188]. Integrating guardrails with LLMs may lead to a
discernible conservative shift in the generation of responses to open-ended
text-generation questions [189]. The shift has been witnessed in ChatGPT over
time. [190] documented a notable change in ChatGPT’s performance between March
and June 2023. Specifically, when responding to sensitive queries, the model’s
character count decreased significantly, plummeting from an excess of 600
characters to approximately 140. Additionally, in the context of opinion-based
questions and answers surveys, the model is more inclined to abstain from
responding.

Given the brevity and conservativeness of responses generated by ChatGPT, the
following question arises: How can exploratory depth be maintained in
responses, particularly for open-ended test generation tasks? Furthermore,
does the application of guardrails constrain ChatGPT’s capacity to deliver
more intuitive responses? On the other hand, [191] critically examined this
paper and emphasized the difference between an LLM’s capabilities and its
behavior. Although capabilities typically remain constant, behavior can alter
due to fine-tuning, which can be interpreted as the “uncertainty” challenges
in LLMs. They suggest that GPT-4’s performance changes are likely linked more
to evaluation data and fine-tuning methods rather than a decline in its
fundamental abilities. They also acknowledge that such behavioral drift
challenges the development of reliable chatbot products. The adoption of
guardrails has also led to the model adopting a more concise communication
approach, offering fewer details and electing non-response in specific
queries. The decision “to do or not to do” can be challenging when designing
the guardrail. While the most straightforward approach is to decline an answer
to any sensitive questions, is it the most intelligent one? That is, _we need
to determine if the application of guardrail always has a positive impact on
LLMs that is within our expectation_.

Our Perspective Prior research suggested incorporating a creativity assessment
mechanism into the guardrail development for LLMs. To measure the creativity
capability of LLMs, [192] employed the Consensual Assessment Technique [193],
a well-regarded approach in creativity evaluation, focusing on several key
aspects: fluency, flexibility, originality, and elaboration, which
collectively contribute to a comprehensive understanding of the LLMs’ creative
output in storytelling. [191] showed that although some LLMs may demonstrate
adeptness in specific aspects of creativity, there is a significant gap
between their capabilities and human expertise when evaluated comprehensively.

###  5.2 Multidisciplinary Approach

While current LLM guardrails include mechanisms to detect harmful content,
they still risk generating biased or misleading responses. It is reasonable to
expect future guardrails to integrate harm detections and other mechanisms to
deal with, e.g., ethics, fairness, and creativity. In the introduction, we
have provided three categories of requirements to be considered for a
guardrail. Moreover, LLMs may not be universally effective across all domains,
and it has been a trend to consider domain-specific LLMs [194]. In domain-
specific scenarios, specialized rules may conflict with the general
principles. For instance, in crime prevention, the use of certain
terminologies that are generally perceived as harmful, such as ‘guns’ or
‘crime,’ is predominant and should not be precluded. To this end, the concrete
requirements for guardrails will differ across different LLMs, and research is
needed to _scientifically_ determine requirements. The above challenges
(multiple categories, domain-specific, and potentially conflicting
requirements) are compounded by the fact that many requirements, such as
fairness and toxicity, are hard to define precisely, especially without a
concrete context. The existing methods, such as the popular one that sets a
threshold on predictive toxicity level [7], do not have _valid justification
and assurance_.

Our Perspective Developing LLMs ethically involves adhering to fairness,
accountability, and transparency. These principles ensure that LLMs do not
perpetuate biases or cause unintended harm. The works by e.g., [195] and [196]
provide insights into how these principles can be operationalized in the
context of LLMs. Establishing community standards is vital for the responsible
development of LLMs. These standards, derived from a consensus among
stakeholders, including developers, users, and those impacted by AI, can guide
LLMs’ ethical development and deployment. They ensure that LLMs are aligned
with societal values and ethical norms, as discussed in broader AI ethics
literature [197]. Moreover, the ethical development of LLMs is not a one-time
effort but requires ongoing evaluation and refinement. These tasks involve
regular assessment of LLMs outputs, updating models to reflect changing
societal norms, and incorporating feedback from diverse user groups to ensure
that LLMs remain fair and unbiased.

Socio-technical theory [198], in which both ‘social’ and ‘technical’ aspects
are brought together and treated as interdependent parts of a complex system,
have been promoted [199, 200] for machine learning to deal with properties
related to human and societal values, including e.g., fairness [201], biases
[202], and ethics [203]. To manage the complexity, the whole system approach
[204], which promotes an ongoing and dynamic way of working and enables local
stakeholders to come together for an integrated solution, has been
successfully working on healthcare systems [205]. We believe a multi-
disciplinary group of experts will work out and rightly justify and validate
the concrete requirements for a specific context by applying the socio-
technical theory and the whole system approach.

###  5.3 Neural-Symbolic Approach for Implementation

Existing guardrail frameworks such as those introduced in Section 3 employ a
language (such as RAIL or Colang) to describe the behavior of a guardrail. A
set of rules and guidelines are expressed with the language, so each is
applied independently. It is unclear if and how such a mechanism can deal with
more complex cases where the rules and guidelines conflict. As mentioned in
Section 5.2, such complex cases are common in building guardrails. Moreover,
it is unclear if they are sufficiently flexible and capable of adapting to
semantic shifts over time and across different scenarios and datasets.

Our Perspective _First_ , a principled approach is needed to resolve conflicts
in requirements, as suggested in [206] for requirement engineering, which is
based on the combination of logic and decision theory. _Second_ , a guardrail
requires the cooperation of symbolic and learning-based methods. For example,
we may expect that the learning agents deal with the frequently-seen cases
(where there are plenty of data) to improve the overall performance w.r.t. the
requirements mentioned above, and the symbolic agents take care of the rare
cases (where there are few or no data) to improve the performance in dealing
with corner cases in an interpretable way. Due to the complex conflict
resolution methods, more closely coupled neural-symbolic methods might be
needed to deal with the tension between effective learning and sound
reasoning, such as those Type-6 systems [8] that can deal with true symbolic
reasoning inside a neural engine, e.g., Pointer Networks [207].

###  5.4 Systems Development Life Cycle (SDLC)

The criticality of guardrails requires a careful engineering process. For
this, a revisit of the SDLC, which is a complex project management model to
encompass guardrail creation from its initial idea to its finalized deployment
and maintenance, has the potential, and the V-model [208], which builds the
relations of each development process with its testing activities, can be
helpful to ensure the quality of the final product.

Our Perspective Rigorous verification and testing will be needed [2], which
requires a comprehensive set of evaluation methods. Certification with
statistical guarantees can be helpful for individual requirements, such as the
randomized smoothing [176]. For the evaluation of multiple, conflicting
requirements, a combination of the Pareto front-based evaluation methods for
multiple requirements [209] and the statistical certification for a single
requirement is needed. The Pareto front is a concept from the field of multi-
objective optimization. It represents a set of non-dominated solutions, where
no other solutions in the solution space are better when all objectives are
considered. Statistical certification involves using statistical methods to
ensure that a single requirement meets a specified standard with a certain
confidence level. It is typically applied when there is uncertainty in the
measurements, or the requirement is subject to variability. Combining these
techniques can find the trade-offs, provide confidence in the viability of
solutions concerning individual requirements, and support more informed and
adaptive decision-making processes. Finally, attention should also be paid to
understanding the theoretical limits of the evaluation methods, e.g.,
randomized smoothing causes a fairness problem [210].

While these conflicts may not be entirely resolvable, particularly within a
general framework applicable across various contexts, more targeted approaches
in specific scenarios might offer better conflict resolution. Such approaches
demand ongoing research to develop concrete principles, methods, and standards
that a multidisciplinary team can implement and adhere to. While effective in
particular situations, Guardrails are not a universal solution that addresses
all potential conflicts. Instead, they should be designed to manage specific,
well-defined scenarios.

###  5.5 Safeguards for LLM Agents

In the rapidly evolving field of LLM, more autonomous entities extend the
capabilities of LLMs by integrating decision-making and action-initiating
capacities. $LLM\ agents$ process and generate language and use this
capability to perform actions in the digital or physical world. These LLM
agents typically encompass five fundamental modules: LLMs, planning, action,
external tools, and memory and knowledge. [211]. While LLMs respond passively
to user queries, LLM agents can take proactive steps based on their
understanding or directives. This increased autonomy raises concerns about
unintended consequences, especially in sensitive domains like scientific
research.

Our Perspective Due to their autonomy, LLM agents introduce higher complexity
and unpredictability. The integration of decision-making processes means they
might initiate actions that are hard to foresee or control, potentially
leading to ethical and practical risks. Different from safeguard LLMs, the
safety of agents interacting with various tools and environments is often
overlooked, leading to potential harmful outputs, as highlighted in studies
such as ToolEmu [212], AgentMonitor [213], and R-Judge [214]. For LLM agents,
“safeguard" means implementing stricter controls and oversight to manage their
broader capabilities effectively.

##  6 Conclusions

This survey provides a holistic view of the existing challenges and
prospective enhancements of safeguarding techniques on LLMs. We categorize the
existing guardrails, analyze their effectiveness, and delve into known
techniques for overcoming these measures. Meanwhile, several safety-related
properties in LLMs are reviewed entirely. This survey highlighted methods for
mitigating risks such as hallucinations and breaches of fairness and privacy
and strategies for countering potential attacks on these mechanisms. After
that, we explored methods to bypass these controls (i.e., attacks), overcome
the attacks, and strengthen the guardrails. In summary, Guardrails are highly
complex due to their role in managing interactions between LLMs and humans. A
systematic approach, supported by a multidisciplinary team, can fully consider
and manage the complexity and provide assurance to the final product.

## References

  * [1] OpenAI, “GPT-4 technical report,” _arXiv e-prints 2303,08774_ , 2023.
  * [2] X. Huang, W. Ruan, W. Huang, G. Jin, Y. Dong, C. Wu, S. Bensalem, R. Mu, Y. Qi, X. Zhao _et al._ , “A survey of safety and trustworthiness of large language models through the lens of verification and validation,” _arXiv prepr. arXiv:2305,11391_ , 2023.
  * [3] D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto, “Exploiting programmatic behavior of LLMs: Dual-use through standard security attacks,” _arXiv prepr. arXiv:2302,05733_ , 2023.
  * [4] A. Birhane, A. Kasirzadeh, D. Leslie, and S. Wachter, “Science in the age of large language models,” _Nat. Rev. Phys._ , vol. 5, no. 5, pp. 277–280, May 2023.
  * [5] J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang, “Challenges in detoxifying language models,” _arXiv prepr. arXiv:2109,07445_ , 2021.
  * [6] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “Realtoxicityprompts: Evaluating neural toxic degeneration in language models,” _arXiv prepr. arXiv:2009,11462_ , 2020.
  * [7] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,” _arXiv prepr. arXiv:2202,03286_ , 2022.
  * [8] L. C. Lamb, A. d’Avila Garcez, M. Gori, M. O. Prates, P. H. Avelar, and M. Y. Vardi, “Graph neural networks meet neural-symbolic computing: A survey and perspective,” in _Proc. 29th Int. Jt. Conf. Artif. Intell. (IJCAI 2021)_ , ser. IJCAI’20, Yokohama, Yokohama, Japan, 2021.
  * [9] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine _et al._ , “Llama guard: Llm-based input-output safeguard for human-ai conversations,” _arXiv prepr. arXiv:2312,06674_ , 2023.
  * [10] T. Rebedea, R. Dinu, M. Sreedhar, C. Parisien, and J. Cohen, “Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails,” _arXiv prepr. arXiv:2310,10501_ , 2023.
  * [11] S. Rajpal, “Guardrails AI,” 2023.
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, $\mathcal{L}$. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Adv. Neural Inf. Process. Syst. 30 (NeurIPS 2017)_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017.
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proc. 2019 Conf. n. Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol._ Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.
  * [14] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in _Proc. 34th Int. Conf. Neural Inf. Process. Syst._ , ser. NIPS’20. Red Hook, NY, USA: Curran Associates Inc., 2020.
  * [15] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar _et al._ , “Llama: Open and efficient foundation language models,” _arXiv prepr. arXiv:2302,13971_ , 2023.
  * [16] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen _et al._ , “Palm 2 technical report,” _arXiv prepr. arXiv:2305,10403_ , 2023.
  * [17] J. Wei, S. Kim, H. Jung, and Y.-H. Kim, “Leveraging large language models to power chatbots for collecting user self-reported data,” _arXiv prepr. arXiv:2301,05843_ , 2023.
  * [18] C. Lyu, J. Xu, and L. Wang, “New trends in machine translation using large language models: Case examples with chatgpt,” _arXiv prepr. arXiv:2305,01181_ , 2023.
  * [19] N. Simon and C. Muise, “TattleTale: Storytelling with planning and large language models,” in _ICAPS Workshop Sched. Plan. Appl._ , 2022.
  * [20] R. Lou, K. Zhang, and W. Yin, “Is prompt all you need? no. A comprehensive and broader view of instruction learning,” _arXiv prepr. arXiv:2303,10475_ , 2023.
  * [21] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , “Training language models to follow instructions with human feedback,” _NeurIPS_ , vol. 35, pp. 27 730–27 744, 2022.
  * [22] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou _et al._ , “Chain-of-thought prompting elicits reasoning in large language models,” _NeurIPS_ , vol. 35, pp. 24 824–24 837, 2022.
  * [23] Nvidia, “Colang,” 2023.
  * [24] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin _et al._ , “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” _arXiv prepr. arXiv:2311,05232_ , 2023.
  * [25] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal _et al._ , “Towards expert-level medical question answering with large language models,” _arXiv prepr. arXiv:2305,09617_ , 2023.
  * [26] Z. Li, S. Zhang, H. Zhao, Y. Yang, and D. Yang, “Batgpt: A bidirectional autoregessive talker from generative pre-trained transformer,” _arXiv prepr. arXiv:2307,00360_ , 2023.
  * [27] Y. Liu, X. Zeng, F. Meng, and J. Zhou, “Instruction position matters in sequence generation with large language models,” _arXiv prepr. arXiv:2308,12097_ , 2023.
  * [28] M. Webster and J. Schmitt, “LLM hallucinations: How to detect and prevent them with CI,” _CircleCI Blog_ , Jan. 2024.
  * [29] Q. Cheng, T. Sun, W. Zhang, S. Wang, X. Liu, M. Zhang, J. He, M. Huang, Z. Yin, K. Chen _et al._ , “Evaluating hallucinations in chinese large language models,” _arXiv prepr. arXiv:2310,03368_ , 2023.
  * [30] S. K. Jha, S. Jha, P. Lincoln, N. D. Bastian, A. Velasquez, R. Ewetz, and S. Neema, “Counterexample guided inductive synthesis using large language models and satisfiability solving,” in _2023 IEEE Mil. Commun. Conf. (MILCOM 2023)_. IEEE, 2023, pp. 944–949.
  * [31] S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi, “Factscore: Fine-grained atomic evaluation of factual precision in long form text generation,” _arXiv prepr. arXiv:2305,14251_ , 2023.
  * [32] I. Chern, S. Chern, S. Chen, W. Yuan, K. Feng, C. Zhou, J. He, G. Neubig, P. Liu _et al._ , “FacTool: Factuality detection in generative AI–A tool augmented framework for multi-task and multi-domain scenarios,” _arXiv prepr. arXiv:2307,13528_ , 2023.
  * [33] F. Nan, R. Nallapati, Z. Wang, C. N. dos Santos, H. Zhu, D. Zhang, K. McKeown, and B. Xiang, “Entity-level factual consistency of abstractive text summarization,” _arXiv prepr. arXiv:2102,09130_ , 2021.
  * [34] B. Goodrich, V. Rao, P. J. Liu, and M. Saleh, “Assessing the factual accuracy of generated text,” in _Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min._ , 2019, pp. 166–175.
  * [35] K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” _arXiv prepr. arXiv:2104,07567_ , 2021.
  * [36] A. Mishra, D. Patel, A. Vijayakumar, X. L. Li, P. Kapanipathi, and K. Talamadupula, “Looking beyond sentence-level natural language inference for question answering and text summarization,” in _Proc. 2021 Conf. n. Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol._ , 2021, pp. 1322–1336.
  * [37] M. Barrantes, B. Herudek, and R. Wang, “Adversarial nli for factual correctness in text summarisation models,” _arXiv prepr. arXiv:2005,11739_ , 2020.
  * [38] Z. Luo, Q. Xie, and S. Ananiadou, “Chatgpt as a factual inconsistency evaluator for abstractive text summarization,” _arXiv prepr. arXiv:2303,15621_ , 2023.
  * [39] M. Gao, J. Ruan, R. Sun, X. Yin, S. Yang, and X. Wan, “Human-like summarization evaluation with chatgpt,” _arXiv prepr. arXiv:2304,02554_ , 2023.
  * [40] P. Manakul, A. Liusie, and M. J. Gales, “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models,” _arXiv prepr. arXiv:2303,08896_ , 2023.
  * [41] S. L. Blodgett, S. Barocas, H. D. III, and H. M. Wallach, “Language (technology) is power: A critical survey of "Bias" in NLP,” in _Proc. 58th Annu. Meet. Assoc. Comput. Linguist._ , D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds. Association for Computational Linguistics, 2020, pp. 5454–5476.
  * [42] A. Malik, “Evaluating large language models through gender and racial stereotypes,” _arXiv prepr. arXiv:2311,14788_ , 2023.
  * [43] Y. Tao, O. Viberg, R. S. Baker, and R. F. Kizilcec, “Auditing and mitigating cultural bias in LLMs,” _arXiv prepr. arXiv:2311,14096_ , 2023.
  * [44] B. Sheppard, A. Richter, A. Cohen, E. A. Smith, T. Kneese, C. Pelletier, I. Baldini, and Y. Dong, “Subtle misogyny detection and mitigation: An expert-annotated dataset,” _arXiv prepr. arXiv:2311,09443_ , 2023.
  * [45] Y. Sheng, S. Cao, D. Li, B. Zhu, Z. Li, D. Zhuo, J. E. Gonzalez, and I. Stoica, “Fairness in serving large language models,” _arXiv prepr. arXiv:2401,00588_ , 2023.
  * [46] I. Garrido-Muñoz, A. Montejo-Ráez, F. Martínez-Santiago, and L. A. Ureña-López, “A survey on bias in deep NLP,” _Appl. Sci._ , vol. 11, no. 7, p. 3184, 2021.
  * [47] N. Badyal, D. Jacoby, and Y. Coady, “Intentional biases in LLM responses,” in _2023 IEEE 14th Annu. Ubiquitous Comput. Electron. Mob. Commun. Conf. (UEMCON)_. IEEE, 2023, pp. 0502–0506.
  * [48] N. H. Koh, J. Plata, and J. Chai, “BAD: BiAs Detection for Large Language Models in the context of candidate screening,” _arXiv prepr. arXiv:2305,10407_ , 2023.
  * [49] T. Limisiewicz, D. Mareček, and T. Musil, “Debiasing algorithm through model adaptation,” _arXiv prepr. arXiv:2310,18913_ , 2023.
  * [50] F. Motoki, V. Pinho Neto, and V. Rodrigues, “More human than human: Measuring chatgpt political bias,” _Available SSRN 4372349_ , 2023.
  * [51] K.-C. Yeh, J.-A. Chi, D.-C. Lian, and S.-K. Hsieh, “Evaluating interfaced LLM bias,” in _Proc. 35th Conf. Comput. Linguist. Speech Process. (ROCLING 2023)_ , 2023, pp. 292–299.
  * [52] O. Shaikh, H. Zhang, W. Held, M. Bernstein, and D. Yang, “On second thought, let’s not think step by step! Bias and toxicity in zero-shot reasoning,” _arXiv prepr. arXiv:2212,08061_ , 2022.
  * [53] K. Z. Zhou and M. R. Sanfilippo, “Public perceptions of gender bias in large language models: Cases of chatgpt and ernie,” _arXiv prepr. arXiv:2309,09120_ , 2023.
  * [54] X. Ma, M. Sap, H. Rashkin, and Y. Choi, “PowerTransformer: Unsupervised controllable revision for biased language correction,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , B. Webber, T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 7426–7441.
  * [55] Z. Xie and T. Lukasiewicz, “An empirical analysis of parameter-efficient methods for debiasing pre-trained language models,” in _Proc. 61st Annu. Meet. Assoc. Comput. Linguist._ , A. Rogers, J. L. Boyd-Graber, and N. Okazaki, Eds. Association for Computational Linguistics, 2023, pp. 15 730–15 745.