# Time-Efficient Reward Learning
via Visually Assisted Cluster Ranking

David Zhang Micah Carroll Andreea Bobu Anca Dragan

###### Abstract

One of the most successful paradigms for reward learning uses human feedback
in the form of comparisons. Although these methods hold promise, human
comparison labeling is expensive and time consuming, constituting a major
bottleneck to their broader applicability. Our insight is that we can greatly
improve how effectively human time is used in these approaches by batching
comparisons together, rather than having the human label each comparison
individually. To do so, we leverage data dimensionality-reduction and
visualization techniques to provide the human with a interactive GUI
displaying the state space, in which the user can label subportions of the
state space. Across some simple Mujoco tasks, we show that this high-level
approach holds promise and is able to greatly increase the performance of the
resulting agents, provided the same amount of human labeling time.

##  1 Introduction

Learning reward functions from human feedback has been a successful paradigm
in both sequential decision-making problems [1, 46, 19, 6, 17, 14], enabling
AI agents to learn behaviors that are challenging, or perhaps even impossible,
to specify manually. Although these methods hold promise, the main bottleneck
to applying them broadly in real-world scenarios is the amount of human effort
they require, sometimes requiring hours of uninterrupted human attention and
interaction [17].

To make it easier for the human to provide input, many recent approaches use a
flexible and human-friendly type of reward feedback in the form of comparisons
between trajectory snippets [17]. However, this approach is still too slow for
practical use because human time is not utilized very effectively: every time
the AI agent interacts with the human, it typically receives feedback for a
single query at a time.

Figure 1: The CLRVis algorithm. Our method consists of several stages: we
collect environment interactions using our current policy $\pi$ and show them
to the user through a t-SNE visualization. The user then provides feedback in
the form of cluster orderings, which are then used to update the reward model
$\hat{r}$; in turn, the reward model $\hat{r}$ is then used to update the
policy $\pi$ and the process repeats.

Recent techniques based on active learning attempt to further alleviate the
burden on the human by shifting it onto the robot: rather than randomly select
which queries to ask the human for feedback on, the robot itself tries to come
up with the most informative ones, in order to be efficient with the human’s
time. In this project, we are interested in a complementary question: rather
than design metrics for having the robot decide which queries are most
informative, we want to build tools to assist humans in providing feedback on
multiple related queries at once.

For instance, imagine you were given an interactive visualization of the state
space for a self-driving task, in which similar states are placed close
together: in the top-right, you hover over various states and by looking at
the states, you can tell they all are bendy roads; you zoom in and see
subclusters of states in which the car is swerving in the other lane, or well-
centered in its lane. This could enable the human to provide feedback on many
semantically similar states at once, at the level of granularity that the
human deems to be most appropriate. Using a visual interface of this kind, it
seems like the human would be able to give quick, rich feedback on entire sets
of similar states at once.

While this kind of tool might seem taken out of a sci-fi hologram scene, in
this work we set out to build a similar type of system. We test such an
approach can, in fact, make more efficient use of the human’s time than
current state of the art methods. Our insight is that enabling the human to
provide input over large swaths of the state space at once relies on how that
state space is visually presented to them: if states that behave similarly are
visualized close together, then the person should be able to more easily think
of them similarly and group them together. As such, we use high-dimensional
data visualization techniques [40] to cluster states based on both their
visual and reward characteristics. We make use of recent contrastive
techniques [28, 25, 16] to learn an embedding that maps visually similar
states near each other, and combine it with the current trained reward
embedding to visualize states close both in reward and appearance near each
other.

We show in our experiments that our method (which we call CLRVis) requires up
to 7x less human time than a DRLHP-like baseline across some Mujoco reward-
learning tasks. We validate this with some real human data and a suite of
simulated human-feedback experiments.

##  2 Related Work

Reward Learning from Human Input. A popular paradigm for learning robot
behaviors from human input is inverse reinforcement learning (IRL), where the
robot seeks to extract a reward function capturing why specific behaviors may
be desirable. Traditionally, IRL relies on human demonstrations to recover the
reward function responsible for the human’s input [1, 46, 3, 21, 19, 44].
Recent research goes beyond demonstrations, utilizing other types of human
input to learn rewards, such as corrections [6], comparisons [43, 17], labels
[42, 34], rankings [14], or some combination [24, 30, 26]. Comparisons have
been especially successful in recent NLP approaches [29, 38, 5].

The issue that persists across all these methods is that the human still bears
most of the labeling burden. Active learning (AL) attempts to alleviate some
of this burden on the human by shifting it onto the robot: to reduce the total
amount of necessary human labels, the robot tries to come up with the most
informative queries to ask the human for help with. Robot learning methods
have investigated a variety of metrics the robot can optimize to select the
best queries, from maximizing uncertainty estimates [37], minimizing
performance risk [15], maximizing exploration [9], minimizing human answer
uncertainty [10], or a combination of the above [34, 11]. While researchers in
data visualization and labeling find that active learning is an effective way
to correctly label data, it is not necessarily the most efficient [36]. This
is in part due to the cold start problem of AL, where without a good initial
model all of these selection strategies start poorly. [36] finds that user-
interactive labeling can be much more efficient than designing active learning
metrics for collecting better reward queries from the human – following this
works’ footsteps, we are interested constructing tools that assist humans in
labeling more reward data faster.

AI Assistance in User-Interactive Feedback. Prior work in user-interactive
labeling has used semi-supervised [18] or self-supervised [41] models to
predict the most likely label associated with the query. This strategy can
assist the user in labeling individual queries faster, but the user still has
to label each data point one at a time. Instead, more recent approaches
propose using visual interfaces to help humans label more data, which can even
enable unifying visual assistance with active learning [8]. Based on this,
[22] proposes a visual interactive labeling approach where the human can
select clusters via a lasso tool, after which a model recommends a label for
each cluster. Then, the interface allows the human to deselect individual
points that don’t match the cluster’s overall label. Instead, [7] use a
clustering algorithm to propose clusters automatically to the human, who can
manually remove instances that don’t belong to each cluster. Both methods have
primarily been demonstrated on MNIST and only work on categorical variables,
whereas the reward learning scenarios we are interested in require predicting
reward as a continuous variable. Since humans are notoriously noisy when
labeling continuous variables [13], we similarly look at visual interfaces
that use clustering to assist the user, but opt for ranking clusters instead
of labeling them.

High-dimensional Data Visualization. As we just saw, one way to construct
better tools for labeling data is to construct better ways to visualize that
data. t-SNE is a non-linear dimensionality reduction technique commonly used
for visualizing high dimensional data in an easy and intuitive way [40]. Most
relevant to our work is its widespread use in reinforcement learning as a tool
for visually evaluating the quality of the learned RL agent embeddings [27,
45, 4, 2]. In these works, the intuition behind using t-SNE is that if an
agent learned a good embedding for Q-value, reward, etc., states that are
embedded similarly – i.e. have similar Q-values or rewards – will appear close
in t-SNE space, even if they are perceptually dissimilar. We use a similar
visualization technique, but we differ in that we don’t only use it for
qualitative evaluation purposes at the end of training an RL agent. Rather, we
seek to recompute the t-SNE visualization at every labeling iteration, after
having both experienced more states and retrained the embedding.

The above methods typically apply t-SNE on states sampled during the fully
trained RL agent’s execution of its policy [27] or during the agent’s entire
training experience [45]. Since we recompute t-SNE at every iteration, we
instead only have access to states from the current iteration policy or from
prior policies. In the future, we could assume a small amount of successful
task demonstrations and sample states according to a curriculum [20] or along
an expert demonstration of the task [35]. Furthermore, these methods have
either visualized the raw pixel representation or the final layer Q-value
network embedding of the sampled states. Since we are primarily interested in
relieving the visual labeling burden from the human, we also train a
contrastive network [28, 16, 39, 25], which seeks to learn visually
discriminative embeddings. We concatenate the learned reward embedding and the
contrastive one, and compute the t-SNE visualization from the concatenated
vector. Prior work has looked at constructing representation embeddings that
result in visualizations more interpretable to the human [23], but there the
assumption is that the robot is the teacher and the human is the student
trying to learn most effectively from that visualization.

##  3 CLRVis

We call our methodology CLusters Rankings with Visual assistance (CLRVis). On
a high-level, our method consists of iteratively showing a person
visualizations of subsets of the state space, which can be used as an
interface to efficiently provide additional reward labels (see Figure 1).
Every iteration of CLRVis can be broken down into various steps:

  1. 1.

State space sampling: we sample some states using our current policy $\pi$.

  2. 2.

Visualizing states and obtaining cluster rankings: we visualize states sampled
in step 1 with t-SNE, and have the user identify $M$ clusters of similar
states through a graphical interface, and provide a ranking of such $M$
clusters in terms of increasing reward.

  3. 3.

Converting cluster rankings to pairwise state comparisons: for each pair of
clusters in our cluster ranking, we can then consider all individual pairs of
states and treat them as single-state comparisons (if cluster $A$ was ranked
higher than cluster $B$, we can treat this as equivalent to stating that state
$a\in A$ has higher reward than state $b\in B$). This gives us on the order of
$O(M^{2}S^{2})$ comparisons (if the size of the clusters was fixed to be $S$),
where the $M^{2}$ term comes from transforming cluster rankings into cluster
comparisons, and the $S^{2}$ term comes from comparing two clusters.

  4. 4.

Updating reward model: with these $O(M^{2}S^{2})$ comparisons, we can update
our running reward model $\hat{r}$.

  5. 5.

Updating policy: with our now-improved reward model $\hat{r}$, we can train
$\pi$ further, and return to step 1. For a schematic description of the
algorithm, see Algorithm 1.

In short, providing cluster reward-rankings with the aid of an appropriate
visual interface should not take many times longer than comparing individual
states. However, such cluster rankings could contain up to $O(M^{2}S^{2})$
more information than an individual cluster comparison. While we might not
expect a $O(M^{2}S^{2})$ speedup in human-time in practice (because many of
the $O(M^{2}S^{2})$ individual comparisons might be similar and not very
informative), conceptually this approach still seems significantly better than
individual state comparisons, which is the current standard approach.

However, the ease (and thus speed) of cluster ranking will crucially depend on
how interpretable the state space visualization is: are highly dissimilar
states next to each other? Do states in similar regions of the visualization
have similar reward (and thus labeling them as a cluster is easy)? For the
purpose of encouraging visualizations which are conducive to ease of cluster
labeling, we combine two approaches: 1) we train a contrastive learning
embedding for each new set of state images, and 2) after the first update of
the reward model $\hat{r}$, we append the reward model embedding to the
contrastive learning one to encourage the states that we already suspect have
similar rewards to be close together in the state space visualization.

Reward model update. To update the reward model $\hat{r}$, we use the dataset
$D$ of pairwise comparisons gathered from cluster rankings. Each comparison
can be represented as a tuple $(s_{0},s_{1},y)\in D$, where $s_{0}$, $s_{1}$
are the two states, and $y\in\\{0,1\\}$ is a label indicating which state has
higher reward. Under the Bradley-Terry model, we can assume our reward model
is a score function that predicts the likelihood one state is better than
another [12].

$P(s_{0}>s_{1})=\frac{e^{\hat{r}(s_{0})}}{e^{\hat{r}(s_{0})}+e^{\hat{r}(s_{1})}}$
(1)

We optimize $\hat{r}$ with a cross-entropy loss between the predicted labels
and true labels as follows:

$\text{loss}(\hat{r})=-\sum_{(s_{0},s_{1},y)\in
D}y\log(P(s_{0}>s_{1}))+(1-y)\log(1-P(s_{0}>s_{1}))$ (2)

Algorithm 1 The CLRVis algorithm

an initial policy $\pi^{0}$, an initial reward model $\hat{r}^{0}$.

for $i$ in $0,\dots,K$ do

Sample $s_{0},\dots,s_{N}$ from rollouts of $\pi^{i}$.

Visualize $s_{0},\dots,s_{N}$ with t-SNE, based on embeddings from
$\hat{r}^{i}$.

Human selects clusters of states with similar rewards, and orders them by
increasing reward.

Obtain $\hat{r}^{i+1}$ by updating $\hat{r}^{i}$ with cluster ordering
information.

Obtain $\pi^{i+1}$ by training $\pi^{i}$ with the updated reward model
$\hat{r}^{i+1}$.

end for

##  4 Experiments

###  4.1 Experimental Setup

Environments. We test our method on 3 MuJoCo environments – Swimmer, Reacher,
HalfCheetah – but we modify the goal-task. We do this because, similarly to
[17], we want to showcase that our method can also work for tasks that we
don’t have an explicit reward for. However, for the purposes of evaluating
CLRVis, we found it useful to hardcode ground truth reward functions for our
new task-goals, even though they were hidden from CLRVis’s learning process.
The advantage is that we can use such ground truth reward functions both as a
oracle performance indicator, and as a way to ground simulated models of user
feedback, which we use in our experiments.

Modified environment tasks. For Reacher, we set the goal to get the arm as
close as possible to a fixed dot position in the bottom left. For HalfCheetah,
we change the goal to have the cheetah stand upright on its head, rather than
running forward. For Swimmer, we set the goal to curl up in a horseshoe shape.
See Appendix A for more information and for the custom reward functions.

Mixed policy. We noticed that CLRVis would sometimes get stuck in local optima
during training (i.e. low-reward policies). We suspected this might be due to
low diversity of states in the t-SNE visualizations (potentially caused by low
entropy of the latest policy, used to collect states for the t-SNE). To
mitigate this issue, we combine a random policy with our current policy while
collecting episodes. We start each episode by using our current policy, and
switch to a random policy after $T$ timesteps, where $T$ is uniformly drawn
from the max possible length of the episode.

Figure 2: State-space Visualization Interface. A real human can provide reward
feedback by selecting clusters in our t-SNE state visualization and inputting
a ranking. Sample clusters are denoted with different colors. To identify
clusters of states with similar rewards, the user hovers over different points
on the t-SNE which will display an image of the corresponding states on the
right.

Visualization and ranking interface. To collect feedback on reward preferences
from the user, we implemented the web interface in Figure 2. At each iteration
we display a t-SNE visualization of $N$ states, sampled with the mixed policy
described above. The user can then hover over each point in the graph, which
will display the corresponding image of the environment state. To provide
feedback, the user uses a lasso tool to select clusters which they believe to
have similar reward values. After selecting $M$ such clusters, the user ranks
them by entering their order into a text box. Based on this ranking, we
extract pair-wise preferences between states to update our reward model.

Human-time. In our context of learning a reward function (and optimal policy)
from human feedback, the main practical bottleneck is that of human time: how
much time are we requesting from humans to provide labels to update our reward
model $\hat{r}$? Human time is expensive, and in comparison, the computational
cost of updating the policy $\pi$ is negligible. Because of this, we report
algorithm performance across different points of human-time, rather than
different numbers of iterations.

Simulated Human – CLRVis. Since collecting real human feedback for every
experiment we ran was prohibitive, to test our method we implemented a
simulated human, which we calibrated to be approximately as good as a real
human or worse. We show some comparisons between our simulated human and a
real human in Figure 4. To simulate human cluster choice, we use agglomerative
clustering [31] over the t-SNE plot, and filter out the clusters that are too
small or have high variance relative to other clusters. We then choose $M$
clusters that are equally spaced in terms of average reward. To do this, we
first compute $M$ "target values" that are equally spaced from minimum to
maximum cluster reward. Then, we select the cluster with average reward
closest to each target value. Using the ground truth reward function, we
finally rank the clusters from $1$ to $M$ based on their average reward. Note
that this simulated human is noisy: the selected clusters will have some
variance, and so the cluster rankings will lead to some mis-labeled data,
similarly to what would happen with a real human. We calibrate how long we
expect the simulated human to take per-cluster-ranking for each environment by
timing multiple real human CLRVis runs for a small number of iterations, and
take the average of the cluster-ranking times.

Simulated Human – DRLHP. We also need to simulate human feedback for DRLHP. To
be certain to benefit the baseline method, we assume that human feedback is
without noise: every comparison is perfectly labeled, according to the oracle
reward function. For each single-state comparison in DRLHP, we assumed that
the human labeling time would be 3 seconds (lower bound from [17]).

Reinforcement Learning. To update our policy after learning a reward function,
we use PPO for all experiments. While we use images to learn a contrastive
representation, both the reward function and policy function use the
vectorized observation space as input. For more details, see Appendix B.

###  4.2 Hypotheses

Manipulated Variables. We test the effect of two different strategies of human
feedback collection: a version of DRLHP [17] which elicits comparisons between
individual states (i.e. in which the trajectory snippet length is equal to 1);
and CLRVis (our method), which instead elicits cluster rankings made through
our graphical user interface.

Dependent Measures. To compare each method’s performance, we report the oracle
Episode Reward that each policy type collects after a certain amount of human
labeling time.

Hypotheses. We have two hypotheses: H1. In low human-time regimes, CLRVis will
recover better rewards than DRLHP with the same amount of labeling time; H2.
In high human-time regimes, DRLHP will eventually recover similar or better
rewards than CLRVis (because DRLHP has less mislabeled datapoints).

###  4.3 Quantitative Results

Figure 3: CLRVis simulated vs human. We run CLRVis with a real human for
Swimmer and Reacher. In Reacher, the simulated human performs roughly the same
as the real one. Swimmer only requires one iteration of CLRVis feedback to
converge – only one seed does poorly in simulation, but the rest match the
real human.

All experiments shown are run with 5 seeds, except for those with real human
data, which are run with 3 seeds. All error bars reported are standard errors
of the mean across seeds.

Human model realism. For Reacher and Swimmer, we ran CLRVis with both a real
human and our simulated human to teach an agent the intended tasks. We show
the results in Figure 3, from which we see that the simulated human
performance is similar or an underestimate to the real human one. We didn’t do
this same procedure for Cheetah as because the human time required was too
restrictive at this stage of the research. As a quick sanity check, we notice
that all models perform similarly when human labeling hasn’t yet started
(where the x-axis is $0$), so no reward model has yet been learned. While the
real human seems to perform significantly better than the simulated human in
Swimmer, this is only caused by a single bad seed. The others match the
simulated human well.

CLRVis’s time-efficiency. From Figure 4, we see that CLRVis is more time-
efficient in the short run and achieves greater reward in the same amount of
human time. Despite having some degree of noise in its cluster rankings,
CLRVis is able to get close to the oracle policy with little human effort,
performing much better than DRLHP in Reacher and Cheetah. CLRVis matches DRLHP
in Swimmer, but both methods converge to the oracle policy. Therefore, these
results support H1.

DRLHP under unlimited human-time. From Figure 5, we see that if one could
collect vast amounts of human data, DRLHP’s performance will eventually
approach CLRVis. However, despite the inherent noise in providing cluster
rankings, DRLHP does not surpass CLRVis, and in fact does slightly worse,
partially supporting H2. Instead, we see a large speedup in human time when
using CLRVis to reach the optimal DRLHP point. In Reacher, DRLHP takes 3345
seconds to reach its optimum, while CLRVis takes 1330 seconds to reach the
same point. In Cheetah, DRLHP takes 12489 seconds to reach its optimum, while
CLRVis only takes 1800. Therefore, we observe roughly a 2.5x and 7x speedup
respectively. We do not run long-term experiments on Swimmer because it is
able to converge to the oracle policy in the short-term.

For more details about our treatment of human time across our methods and
results, see Section B.3.

Figure 4: CLRVis’s time-limited performance. We see that when human time is
constrained, CLRVis tends to perform significantly better than DRLHP for the
same amount of time, and is able to quickly approach oracle performance across
all 3 environments. Figure 5: Long-term results of CLRVis and DRLHP. When we
allow DRLHP to use much more human time to collect comparisons, it’s
performance approaches that of CLRVis. However, we still see a significant
speedup in human-time when switching from DRLHP to CLRVis, roughly 2.5x and 7x
for Reacher and Cheetah respectively.

###  4.4 Qualitative Results

Figure 6:  t-SNE progression. Over time, the t-SNE state visualization becomes
easier for a human to cluster. States with similar reward are grouped more
closely together as our reward embedding improves.  Figure 7:  CLRVis end
behavior. CLRVis is able to learn novel behaviors in the Reacher, Swimmer, and
Cheetah environments.

t-SNE Progression. As our agent and reward model embedding improves from human
feedback, we expect states of similar reward to be more closely grouped in our
t-SNE visualization. Also, we expect that the average reward in our sampled
states will increase due to a better policy.

In Figure 6, we plot the t-SNE visualization of Reacher states after 0, 4, and
8 iterations of CLRVis feedback. We color the t-SNE plots according to the
oracle reward values (which would not accessible to the real human labeler).
Note how there is a large increase in average reward between iterations 0 and
4, and small increase between iterations 0 and 8. This correlates with Figure
5, where we see Reacher’s episode reward grow sharply towards the beginning of
training, and rise slowly later on. In addition, we notice how it is much
easier for a human to label cluster rankings in later iterations of feedback.
In particular, states of similar reward tend to be become grouped more closely
together, and movements in t-SNE space correspond to gradual movements along a
gradient from high reward to low reward.

Behavior Examples. An example of agents reaching their goals in CLRVis can be
found in Figure 7. Videos of agents trained from CLRVis can be found at
https://bit.ly/clrvis-videos.

##  5 Conclusion

Discussion. Before deciding to focus on cluster rankings, we also tried other
forms of feedback, such as labeling clusters with average reward values or
providing individual cluster comparisons. However, we found that labeling
clusters with continuous reward values was difficult for a human to do, while
cluster rankings performed better than comparisons. Although there was some
noise from using clusters instead of individual states, it didn’t have a
significant effect on our overall results. In our approach, users are also
able to actively select the states they want to provide feedback on. As a
result, users have more flexibility to choose states in which the expect the
reward function to be more nuanced.

This is in contrast to DRLHP, where users are forced to compare video snippets
chosen by the algorithm.

Limitations. In this work, we have the users provide feedback over single
states (images) instead of trajectory snippets (videos). While comparing
images might be easier than comparing video snippets for a human, this limits
us to tasks where reward only depends on the current state (a user only needs
a single frame to tell if a Cheetah is standing upright). Another clear
limitation of our work is that most of our experiments are based on simulated
humans.

Future Work. In addition to running more extensive real user studies for both
CLRVis and DRLHP, future work could extend our interface to handle trajectory
snippets, enabling to learn more complex tasks that depend on a sequence of
images, such as getting the Cheetah to perform a backflip. In addition, future
work might utilize task discovery to pretrain policies in the beginning,
rather than rely on a random policy. After doing so, one could fine-tune the
model rather than learn from scratch. This is similar to existing NLP methods,
where language models are trained on large corpuses of text, and fine-tuned
for specific purposes.

Summary. Having users provide individual state comparisons for reward feedback
can be expensive and take a lot of time. We alleviate this issue by providing
an interface where users can provide feedback on large clusters of the state
space, which is visualized using t-SNE. Generally, we want states with similar
reward to be closer to each other, so we make use of contrastive learning and
the reward model representation. To prevent our policy from being stuck in
local optima, we also introduce the notion of a mixed-policy, where the states
shown in our t-SNE come from a combination of our current policy and random
policy. Through experiments in Mujoco environments, we find that CLRVis can
substantially outperform DRLHP given the same amount of human feedback time.

## Acknowledgments and Disclosure of Funding

We thank the members of the InterACT lab for their invaluable feedback. This
work was supported by ONR YIP and the NSF Fellowship.

## References

  * Abbeel and Ng [2004] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In _Machine Learning (ICML), International Conference on_. ACM, 2004\.
  * Annasamy and Sycara [2019] Raghuram Mandyam Annasamy and Katia Sycara.  Towards better interpretability in deep q-networks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):4561–4569, Jul. 2019.  doi: 10.1609/aaai.v33i01.33014561.  URL https://ojs.aaai.org/index.php/AAAI/article/view/4377.
  * Argall et al. [2009] Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning.  A survey of robot learning from demonstration.  _Robotics and autonomous systems_ , 57(5):469–483, 2009.
  * Aytar et al. [2018] Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas.  Playing hard exploration games by watching youtube.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, page 2935–2945, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * Bai et al. [2022] Yushi Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _ArXiv_ , abs/2204.05862, 2022.
  * Bajcsy et al. [2017] Andrea Bajcsy, Dylan P. Losey, Marcia K. O’Malley, and Anca D. Dragan.  Learning robot objectives from physical human interaction.  In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg, editors, _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , pages 217–226. PMLR, 13–15 Nov 2017.  URL http://proceedings.mlr.press/v78/bajcsy17a.html.
  * Beil and Theissler [2020] David Beil and Andreas Theissler.  Cluster-clean-label: An interactive machine learning approach for labeling high-dimensional data.  In _Proceedings of the 13th International Symposium on Visual Information Communication and Interaction_ , VINCI ’20, New York, NY, USA, 2020\. Association for Computing Machinery.  ISBN 9781450387507.  doi: 10.1145/3430036.3430060.  URL https://doi.org/10.1145/3430036.3430060.
  * Bernard et al. [2018] Jürgen Bernard, Matthias Zeppelzauer, Michael Sedlmair, and Wolfgang Aigner.  Vial: a unified process for visual interactive labeling.  _The Visual Computer_ , 34, 09 2018.  doi: 10.1007/s00371-018-1500-3.
  * Biyik et al. [2019] Erdem Biyik, Kenneth Wang, Nima Anari, and Dorsa Sadigh.  Batch active learning using determinantal point processes.  _CoRR_ , abs/1906.07975, 2019.
  * Biyik et al. [2020] Erdem Biyik, Malayandi Palan, Nicholas C. Landolfi, Dylan P. Losey, and Dorsa Sadigh.  Asking easy questions: A user-friendly approach to active reward learning.  In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, _Proceedings of the Conference on Robot Learning_ , volume 100 of _Proceedings of Machine Learning Research_ , pages 1177–1190. PMLR, 30 Oct–01 Nov 2020.  URL https://proceedings.mlr.press/v100/b-iy-ik20a.html.
  * Bobu et al. [2022] Andreea Bobu, Chris Paxton, Wei Yang, Balakumar Sundaralingam, Yu-Wei Chao, Maya Cakmak, and Dieter Fox.  Learning perceptual concepts by bootstrapping from human queries.  _IEEE Robotics and Automation Letters_ , 7(4):11260–11267, 2022.  doi: 10.1109/LRA.2022.3196164.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.  ISSN 00063444.  URL http://www.jstor.org/stable/2334029.
  * Braziunas and Boutilier [2008] Darius Braziunas and Craig Boutilier.  Elicitation of factored utilities.  _AI Magazine_ , 29(4):79, Dec. 2008.  doi: 10.1609/aimag.v29i4.2203.  URL https://ojs.aaai.org/index.php/aimagazine/article/view/2203.
  * Brown et al. [2019] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum.  Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations.  In _International Conference on Machine Learning_ , pages 783–792. PMLR, 2019.
  * Brown et al. [2018] Daniel S Brown, Yuchen Cui, and Scott Niekum.  Risk-aware active inverse reinforcement learning.  In _Conference on Robot Learning_ , pages 362–372. PMLR, 2018.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20. JMLR.org, 2020.
  * Christiano et al. [2017] Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  06 2017.
  * Desmond et al. [2021] Michael Desmond, Michael Muller, Zahra Ashktorab, Casey Dugan, Evelyn Duesterwald, Kristina Brimijoin, Catherine Finegan-Dollak, Michelle Brachman, Aabhas Sharma, Narendra Nath Joshi, and Qian Pan.  Increasing the speed and accuracy of data labeling through an ai assisted interface.  In _26th International Conference on Intelligent User Interfaces_ , IUI ’21, page 392–401, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450380171.  doi: 10.1145/3397481.3450698.  URL https://doi.org/10.1145/3397481.3450698.
  * Finn et al. [2016] Chelsea Finn, Sergey Levine, and Pieter Abbeel.  Guided cost learning: Deep inverse optimal control via policy optimization.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ICML’16, page 49–58. JMLR.org, 2016.
  * Florensa et al. [2017] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel.  Reverse curriculum generation for reinforcement learning, 2017.  URL https://arxiv.org/abs/1707.05300.
  * Fu et al. [2018] Justin Fu, Katie Luo, and Sergey Levine.  Learning robust rewards with adverserial inverse reinforcement learning.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rkHywl-A-.
  * Grimmeisen and Theissler [2020] Benedikt Grimmeisen and Andreas Theissler.  The machine learning model as a guide: Pointing users to interesting instances for labeling through visual cues.  In _Proceedings of the 13th International Symposium on Visual Information Communication and Interaction_ , VINCI ’20, New York, NY, USA, 2020\. Association for Computing Machinery.  ISBN 9781450387507.  doi: 10.1145/3430036.3430058.  URL https://doi.org/10.1145/3430036.3430058.
  * Hilgard et al. [2021] Sophie Hilgard, Nir Rosenfeld, Mahzarin R Banaji, Jack Cao, and David Parkes.  Learning representations by humans, for humans.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 4227–4238. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/hilgard21a.html.
  * Ibarz et al. [2018] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei.  Reward learning from human preferences and demonstrations in atari.  _ArXiv_ , abs/1811.06521, 2018.
  * Laskin et al. [2020] Michael Laskin, Aravind Srinivas, and Pieter Abbeel.  CURL: Contrastive unsupervised representations for reinforcement learning.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 5639–5650. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/laskin20a.html.
  * Mehta and Losey [2022] Shaunak A. Mehta and Dylan P. Losey.  Unified learning from demonstrations, corrections, and preferences during physical human-robot interaction, 2022.  URL https://arxiv.org/abs/2207.03395.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , 518:529–533, 2015.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding, 2018.  URL https://arxiv.org/abs/1807.03748.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe.  Training language models to follow instructions with human feedback.  _ArXiv_ , abs/2203.02155, 2022.
  * Palan et al. [2019] Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, and Dorsa Sadigh.  Learning reward functions by integrating human demonstrations and preferences.  _ArXiv_ , abs/1906.08928, 2019.
  * Pedregosa et al. [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830, 2011\.
  * Raffin [2020] Antonin Raffin.  Rl baselines3 zoo.  https://github.com/DLR-RM/rl-baselines3-zoo, 2020.
  * Raffin et al. [2021] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann.  Stable-baselines3: Reliable reinforcement learning implementations.  _Journal of Machine Learning Research_ , 22(268):1–8, 2021.  URL http://jmlr.org/papers/v22/20-1364.html.
  * Reddy et al. [2020] S. Reddy, A. Dragan, Sergey Levine, S. Legg, and J. Leike.  Learning human objectives by evaluating hypothetical behavior.  In _ICML_ , 2020.
  * Salimans and Chen [2018] Tim Salimans and Richard J. Chen.  Learning montezuma’s revenge from a single demonstration.  _ArXiv_ , abs/1812.03381, 2018.
  * Sedlmair [2018] J. Bernard ; Marco Hutter ; Matthias Zeppelzauer ; Dieter W. Fellner ; Michael Sedlmair.  Comparing visual-interactive labeling with active learning: An experimental study.  _IEEE Transactions on Visualization and Computer Graphics_ , 24:298–308, 2018.
  * Singh et al. [2019] Avi Singh, Larry Yang, Chelsea Finn, and Sergey Levine.  End-to-end robotic reinforcement learning without reward engineering.  In Antonio Bicchi, Hadas Kress-Gazit, and Seth Hutchinson, editors, _Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019_ , 2019.  doi: 10.15607/RSS.2019.XV.073.  URL https://doi.org/10.15607/RSS.2019.XV.073.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano.  Learning to summarize from human feedback.  _ArXiv_ , abs/2009.01325, 2020.
  * Stooke et al. [2021] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin.  Decoupling representation learning from reinforcement learning.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 9870–9879. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/stooke21a.html.
  * van der Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of Machine Learning Research_ , 9(86):2579–2605, 2008.  URL http://jmlr.org/papers/v9/vandermaaten08a.html.
  * Wang et al. [2022] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao.  Pico: Contrastive label disambiguation for partial label learning, 2022\.  URL https://arxiv.org/abs/2201.08984.
  * Warnell et al. [2018] Garrett Warnell, Nicholas R. Waytowich, Vernon J. Lawhern, and Peter Stone.  Deep tamer: Interactive agent shaping in high-dimensional state spaces.  _ArXiv_ , abs/1709.10163, 2018.
  * Wirth et al. [2017] Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al.  A survey of preference-based reinforcement learning methods.  _Journal of Machine Learning Research_ , 18(136):1–46, 2017.
  * Wulfmeier et al. [2016] M. Wulfmeier, D. Z. Wang, and I. Posner.  Watch this: Scalable cost-function learning for path planning in urban environments.  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2089–2095, 2016.
  * Zahavy et al. [2016] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor.  Graying the black box: Understanding dqns.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1899–1908, New York, New York, USA, 20–22 Jun 2016. PMLR.  URL https://proceedings.mlr.press/v48/zahavy16.html.
  * Ziebart et al. [2008] Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey.  Maximum entropy inverse reinforcement learning.  In _Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3_ , AAAI’08, pages 1433–1438. AAAI Press, 2008.  ISBN 978-1-57735-368-3.  URL http://dl.acm.org/citation.cfm?id=1620270.1620297.

##  Appendix A Environment Details

Custom rewards. Each environment is modified with its own reward function. In
general, we remove the torque-based penalties from MuJoCo, which aren’t
apparent to a human. We also use fixed-length episodes to ensure that
termination conditions don’t influence our reward function.

Reacher: We use the same reward function as the original environment, but only
keep the $reward\\_dist$ component and discard $reward\\_control$.

Cheetah: We change the reward function to be $-abs(obs[1]-1.25)$, where
$obs[1]$ is the angle of the main body. The $1.25$ is the target angle for our
cheetah, which means we want it to be upright with a slight tilt.

Swimmer: We change the reward function to be $obs[1]*obs[2]$, where $obs[1]$
and $obs[2]$ are the angles of the left and right rotor. We multiply them
together to ensure that they are in the same direction, and not in a Z shape
instead.

State and action spaces. We use the same observation and action spaces as the
original Mujoco environment. In general, the observation space contains joint
angles, angular and linear velocities, and position coordinates. All of the
action spaces are continuous.

##  Appendix B Experimental Details

Table 1: DRLHP Hyperparameters Parameter | DRLHP
---|---
Initial reward training epochs | 200
Reward training epochs | 2
Reward batch size | 50
Reward learning rate | $3*10^{-4}$
Label schedule | Hyperbolic
Table 2: CLRVis Hyperparameters Parameter | Value
---|---
Initial reward training steps | 2000
Reward training steps | 500
Reward batch size | 500
Reward learning rate | $3*10^{-4}$
CL batch size | 50
CL learning rate | $5*10^{-5}$
CL embedding dim | 512
PCA dim | 50
t-SNE perplexity | 30

###  B.1 Preprocessing and Hyperparameters

We train all agents using PPO. The PPO parameters for Reacher are from rl-
baselines3-zoo [32], while the ones for Cheetah and Swimmer are from stable-
baselines3 [33]. Agents receive reward feedback every 250000 steps for Reacher
and Swimmer, and every 100000 steps for Cheetah. These parameters are kept
consistent between DRLHP and CLRVis.

To train our contrastive network, we render the Mujoco images and resize them
to 100x100. We apply a random crop augmentation to both the anchor and
positive example. To form our t-SNE representation, we first apply
dimensionality reduction with PCA to the concatenation of our contrastive and
reward embedding. Then, we run the t-SNE algorithm with a perplexity of 30. We
show 2000 states to the user each time.

Additional parameters for DRLHP and CLRVis are in Table 1 and Table 2
respectively.

###  B.2 DRLHP

The initial reward model is trained on states sampled from a random policy,
before we begin training our agent. In addition, $25\%$ of the total
comparisons are used to train the initial reward model, and comparisons are
provided on a hyperbolic schedule in future iterations.

###  B.3 Human timings

To estimate the amount of time needed to provide a set of cluster rankings, a
real human timed themselves for a small number of feedback iterations and used
the average. For each iteration, the time measurement starts when the t-SNE
first loads into the interface and ends when the human submits their cluster
orderings.

For Figure 4, we use the average time per CLRVis iteration to determine the
number of comparisons DRLHP gets at each iteration. This ensures that each
point in DRLHP has a corresponding point in CLRVis with the same “human time”,
so we can directly compare the two methods. Instead, for Figure 5 we don’t
attempt to equalize the amount of human time per-iteration across methods.
This enables us to sweep over DRLHP parameters such as the number of feedback
comparisons per iteration that had to be fixed for Figure 4. By doing this, we
can find the best performing hyperparameters without the time-per-iteration
constraint. However, this also means that we won’t have the same number of
datapoints on the x-axis across methods: DRLHP performs best with many
comparisons per iteration, leading to a much higher human-time per iteration
relative to CLRVis.

###  B.4 Computational cost

All experiments were run on an on-premise server. The server has 2 Intel Xeon
6130 32-Core CPUs and 4 NVIDIA GTX 1080 GPUs, shared between multiple
projects. The short-term experiments in Figure 4 took around 6 hours of
compute. The long-term experiments in Figure 5 took around 60 hours of
compute; this is mainly due to running CLRVis for many iterations to compare
against DRLHP. In practice, one would require far less time (closer to 6
hours) to achieve a reasonable policy.

##  Appendix C Additional results

Episode reward in timesteps. In Figure 8, we show how CLRVis and DRLHP perform
after a certain number of timesteps. Since CLRVis uses less human time per
timestep, we run it for longer than DRLHP. While CLRVis might require more
timesteps to converge in reward, our goal is to minimize the amount of
humantime required instead.

Figure 8: Mean episode reward vs timesteps. Instead of using human time for
our x-axis, we switch to environment timesteps. Since CLRVis uses less human
time per timestep, we are able to run it for longer than DRLHP while spending
the same amount of human time.


# All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks

Kazuhiro Takemoto
Kyushu Institute of Technology
Iizuka, Fukuoka, Japan
<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs), such as ChatGPT, encounter ‘jailbreak’
challenges, wherein safeguards are circumvented to generate ethically harmful
prompts. This study introduces a straightforward black-box method for
efficiently crafting jailbreak prompts, addressing the significant complexity
and computational costs associated with conventional methods. Our technique
iteratively transforms harmful prompts into benign expressions directly
utilizing the target LLM, predicated on the hypothesis that LLMs can
autonomously generate expressions that evade safeguards. Through experiments
conducted with ChatGPT (GPT-3.5 and GPT-4) and Gemini-Pro, our method
consistently achieved an attack success rate exceeding 80% within an average
of five iterations for forbidden questions and proved robust against model
updates. The jailbreak prompts generated were not only naturally-worded and
succinct but also challenging to defend against. These findings suggest that
the creation of effective jailbreak prompts is less complex than previously
believed, underscoring the heightened risk posed by black-box jailbreak
attacks.

##  1 Introduction

Large Language Models (LLMs) like ChatGPT [1] are highly anticipated for
applications across a wide range of fields including education, research,
social media, marketing, software engineering, and healthcare [2, 3, 4, 5, 6].
However, the use of extremely diverse texts for training LLMs [7] often leads
to the generation of ethically harmful content [8]. This poses a significant
barrier to the real-world application of LLMs. LLM vendors are acutely aware
of this issue and have implemented safeguards such as reinforcement learning
with human feedback to align LLMs with human values and intentions [9], and
external systems to detect and block ethically harmful inputs (prompts) and
outputs (responses), thus preventing the generation of problematic texts [10].

Nevertheless, these safeguards can be bypassed, enabling the generation of
ethically harmful content by LLMs [11, 12]. Often referred to as “jailbreaks,”
this represents a key vulnerability of LLMs and is a subject of considerable
concern. Consequently, methods for such jailbreak attacks are being vigorously
researched from the perspective of vulnerability assessment of LLMs. Common to
these studies is the creation of prompts designed to bypass LLM safeguards,
notably including manually created jailbreak prompts [13] (referred to as
“manual jailbreak prompts” or “wild jailbreak prompts”) such as the well-known
Do-Anything-Now [14]. It is also possible to generate jailbreak prompts using
gradient-based optimization methods for open-source (white-box) LLMs like
Vicuna [15, 16]. Such jailbreak prompts often possess a degree of
transferability, meaning they can be effective in attacks against closed-
source (black-box) LLMs like ChatGPT.

Defending LLMs against jailbreak attacks can involve detecting and blocking
jailbreak prompts [17, 18]. Manual jailbreak prompts are relatively limited in
number, allowing for their easy blockage through blacklisting [15]. Although
gradient-based jailbreak prompts can theoretically be generated in infinite
numbers, there is a limit to the number of transferable jailbreak prompts,
which suggests that these too can be blocked similarly. Furthermore, these
prompts often contain unnatural (unreadable) texts, making them detectable
based on this criterion [19, 20], thus enabling their blockage.

However, jailbreak attack methods have been evolving, particularly with recent
efforts focused on creating prompts in natural language and formulating high-
performance jailbreak prompts for black-box LLMs [21, 22]. Yet, existing
methods often rely on white-box LLMs or require complex prompt designs,
resulting in high computational costs and complexity. Furthermore, when
considering the development of practical, the state-of-the-art method [18]
that are versatile (not limited to adversarial suffixes) and easily
implementable, it can still be argued that jailbreak attacks remain somewhat
limited.

In this study, contrary to expectations, we demonstrate that jailbreak prompts
written in natural language, which are highly effective against black-box
LLMs, can be created with remarkable ease. Specifically, we propose a simple
black-box method for jailbreak attacks111Code used in this study is available
at https://github.com/kztakemoto/simbaja and illustrate its effectiveness. Our
method involves targeting a black-box LLM and repeatedly rewriting ethically
harmful questions (prompts) into expressions deemed harmless. By using these
rewritten prompts as inputs, the method successfully jailbreaks the LLM. This
approach is based on the hypothesis that it is possible to sample expressions
with the same meaning as the original prompt directly from the target LLM,
thereby bypassing safeguards. In simpler terms, this method involves inducing
the target LLM to confess its own jailbreak prompts.

#### Contributions

The contributions of this study are as follows:

  * •

Proposal of an extremely simple black-box method for jailbreak attacks.
Compared to existing research, the proposed method is extremely easy to
implement. It does not require the design of sophisticated prompts and is
composed only of simple prompts. There is no need for white-box LLMs and high-
spec computing environments to operate them; the method can be implemented in
a general user’s computing environment using only the application programming
interface (API) for black-box LLMs.

  * •

High attack success rate and high efficiency. The proposed method demonstrates
high attack performance against a wide range of ethically harmful questions in
various scenarios, compared to manual jailbreak prompts and other existing
methods. Empirically, the proposed method can create jailbreak prompts with
fewer iterations than expected. In experiments using ChatGPT (GPT-3.5 and
GPT-4) and Gemini-Pro, an attack success rate of over 80% was achieved in an
average of 5 iterations.

  * •

Simple jailbreak prompts written in natural language. Since the prompts are
rewritten by LLMs, they are naturally in natural language. Furthermore,
compared to existing methods, the jailbreak prompts are significantly shorter.

  * •

High evasiveness against defense. The jailbreak prompts generated by the
proposed method, due to their nature, evade the versatile and practically
implementable state-of-the-art defense method, maintaining a high attack
success rate.

##  2 Related Work

###  2.1 Manual Jailbreak Attacks

This represents the origin of jailbreak research. Jailbreak prompts manually
created by researchers, engineers, and citizen data scientists have been
identified, with various objectives and strategies [23, 24, 25]. Notably, Shen
et al. [13] have collected over 6,000 manual jailbreak prompts from various
platforms, demonstrating their transferability. However, since these prompts
are manually created, they are inherently limited.

###  2.2 Gradient-Based Jailbreak Attacks

Jailbreak attacks are a form of adversarial attack [26]. Given that
adversarial attacks against language models can involve the creation of
adversarial texts using gradients [27], it is conceivable to generate
jailbreak prompts using gradients from white-box LLMs. Indeed, Zou et al. [16]
have shown that it is possible to generate universal and transferable
jailbreak prompts (more precisely, adversarial suffixes). However, these
prompts are often unreadable and can be easily blocked [19, 20]. In response,
Zhu et al. [15] have proposed a method for generating interpretable jailbreak
prompts based on gradient information. This can be seen as an automation of
manual jailbreak prompt creation. While gradient-based jailbreak prompts can
be transferred to other target black-box LLMs, their transfer efficiency is
often limited due to differences between the white-box LLM used for creation
and the target black-box LLM.

###  2.3 Black-Box Jailbreak Attacks

Due to the aforementioned issues, black-box attacks, which directly find
jailbreak prompts for the target black-box LLM based on its input-output
relationship, have been vigorously researched. This approach also stems from
the bottleneck posed by the high-spec computational environments required to
operate white-box LLMs for gradient-based jailbreak prompt creation. Manual
jailbreak prompts, since they are created based on the input-output
relationship of LLMs, can be considered a form of black-box attack. Lapid et
al. [21] have used genetic algorithms, a type of black-box optimization
method, to create universal jailbreak prompts. This can be seen as a black-box
version of Zou et al.’s method [16]. However, the prompts generated by Lapid
et al.’s method include unreadable texts and are thought to be easily blocked.
Consequently, Chao et al. [22] have proposed Prompt Automatic Iterative
Refinement (PAIR). Inspired by social engineering attacks, PAIR automates the
generation of jailbreak prompts for a target LLM using an attacker LLM,
without human intervention. Specifically, it involves repeatedly querying the
target LLM and improving the prompt until the attack is successful. The
prompts are written in natural language as they are rewritten by LLMs. Since
there is no need to use a white-box LLM as the attacker LLM, high-spec
computational environments are not required. However, the system prompts for
setting up the scenario to improve the prompt are highly sophisticated and
complex. The generated prompts are unnaturally long compared to the original
prompts and require saving the history of prompt improvements by the attacker
LLM and responses from the target LLM, thus demanding many tokens and
resulting in high computational costs.

###  2.4 Defense Against Jailbreak Attacks

As the vulnerability of LLMs to jailbreak attacks becomes a growing concern,
the development of defensive methods has also intensified. For example,
jailbreak prompts based on adversarial suffixes often exhibit high perplexity
due to the nonsensical nature of the suffixes, allowing for detection based on
perplexity [19]. Another approach involves randomly perturbing multiple copies
of an input prompt and then aggregating the corresponding predictions to
detect jailbreak prompts [17]. Additionally, rephrasing or retokenizing
jailbreak prompts can mitigate their attacks [20]. However, these methods are
limited to adversarial suffixes and face challenges with naturally-worded
jailbreak prompts. Recently, a simple yet effective defense technique inspired
by the psychological concept of ‘self-reminder’ has been proposed [18]. It
involves encapsulating the user’s query within a system prompt that reminds
the LLM to respond responsibly, showing high defense efficacy against manual
jailbreak prompts, which naturally-worded. Yet, the defensive performance
against the state-of-the-art attack method remains unknown. Further
development of attack methods is necessary to thoroughly examine the
effectiveness of these defense techniques.

##  3 Simple Black-Box Jailbreak Attacks

In this study, we develop a simpler black-box method for jailbreak attacks.
The key idea is to have the target LLM itself rewrite ethically harmful
prompts (texts) that would normally be denied a response. Since the LLM
decides whether to activate safeguards based on the content of the text
(implying that the LLM itself knows the jailbreak prompts), it is conceivable
that texts bypassing the safeguards can be efficiently sampled from the LLM
itself. Therefore, it is believed that jailbreak attacks can be successful by
sampling the closest expression to the original text from the set of texts
that can bypass such safeguards, and using this as input.

To realize this sampling, the target LLM itself is tasked with rewriting the
original harmful text. Various prompts can be considered for this purpose.
However, direct prompts like “Please rewrite the given sentence without
changing its meaning, to bypass the safeguards (and not be ethically harmful)”
are likely to still activate the safeguards. Therefore, in this study, we
considered the prompt “Please rewrite the provided text in such a way that it
retains its original meaning but is rephrased in a manner that would not cause
discomfort to readers. Feel free to add clarification sentences if necessary.”
By focusing on not causing discomfort to the reader, it is expected that
ethically harmful expressions (such as those that are sexual or violent) are
toned down (thus bypassing safeguards), and such rewriting is a desirable
application for LLMs from the perspective of text correction, making it less
likely to trigger safeguards in the future. Here, we denote the rewriting of a
text $t$ by LLM $M$ as AdversarialRephrasing($M,t$). Based on this
AdversarialRephrasing($M,t$), the algorithm is structured as shown in
Algorithm 1.

Algorithm 1 Simple black-box jailbreak attacks

1:Original prompt $t_{\mathrm{input}}$, maximum number $n_{\mathrm{init}}$ of
initial states, maximum number $i_{\max}$ of iterations, target LLM
$M(\cdot)$.

2:for $n_{\mathrm{init}}$ steps do

3: $t\leftarrow\textsc{NeutralRephrasing}(t_{\mathrm{input}})$

4: for $i_{\max}$ steps do

5: $t\leftarrow\textsc{AdversarialRephrasing}(M,t)$

6: $r\leftarrow M(t)$

7: if $\textsc{Judgement}(t_{\mathrm{input}},r)$ then

8: return $t$

9: end if

10: end for

11:end for

12:return None

This algorithm incorporates several innovations to facilitate successful
jailbreaking: i) To introduce diversity in the search process, the original
text $t_{\mathrm{input}}$ entered through NeutralRephrasing is used as the
initial state after being neutrally rewritten (line 2). ii) To further weaken
ethically harmful expressions, AdversarialRephrasing is repeated (inner loop).
However, as this repetition continues, the meaning of the rewritten text may
diverge from the original meaning of $t_{\mathrm{input}}$. iii) Therefore, the
repetition of AdversarialRephrasing starts from $n_{\mathrm{init}}$ initial
states (outer loop).

The rewritten text $t$ is judged whether it is a jailbreak prompt or not by
Judgement along with the response $r=M(t)$ from the target LLM $M$ (line 6).
Judgement is a boolean function that returns true if the response $r$ is a
direct answer to the question (text) $t$, and a $t$ that makes
$\textsc{Judgement}(t_{\mathrm{input}},r)=\mathrm{True}$ is output as a
jailbreak prompt. Note that both AdversarialRephrasing($M,t$) (line 4) and
$M(t)$ (line 5) involve querying LLM $M$, but these queries are independent.

##  4 Experiments

###  4.1 Experimental Settings

#### Dataset

The experiments primarily utilized a dataset of 390 forbidden questions
created in the study by Shen et al [13]. Unless stated otherwise, this dataset
was employed throughout this paper. The dataset comprises 30 questions for
each of 13 scenarios (e.g., “Illegal Activity” and “Hate Speech”) that are not
answerable according to the OpenAI usage policy. In addition, to compare the
performance of the proposed method, manual jailbreak prompts collected in Shen
et al.’s study were used. In their research, manual jailbreak prompts are
paired with each question in the dataset, assigning 24 jailbreak prompts to
each question (three selected from each of eight jailbreak prompt
communities). To demonstrate the versatility of the proposed method, a
different dataset was also employed. Specifically, the harmful behavior
dataset used in the study on PAIR [22] was utilized. This dataset consists of
50 questions soliciting harmful behaviors.

#### Large Language Models

We used ChatGPT (GPT-3.5 [1] and GPT-4) and Gemini-Pro [28] (the core system
of Google Bard), which are representative black-box LLMs. For ChatGPT, we
primarily used the November versions (gpt-3.5-turbo-1106 and
gpt-4-1106-preview), but also employed the June (gpt-3.5-turbo-0613) and March
(gpt-3.5-turbo-0301) versions of GPT-3.5 to assess the effect of updates.
Considering realistic execution scenarios, we used default settings for
hyperparameters such as temperature for these LLMs. However, when executing
AdversarialRephrasing, the temperature was set to 1 to introduce diversity in
the rewriting. GPT-3.5 (gpt-3.5-turbo-1106) was also used for
NeutralRephrasing. Specifically, NeutralRephrasing was executed with the
prompt, “Please rewrite the given sentence without changing its meaning, in a
different expression.” The temperature was set to 1 to introduce diversity in
rewriting.

#### Evaluation

The determination of whether a jailbreak attack was successful (Judgement)
utilized the same criteria as those employed for each dataset. For the
forbidden question dataset, following Shen et al.’s study [13], an LLM was
used to assess whether its response constituted a direct answer to the
question. For the harmful behavior dataset, following Chao et al.’s study
[22], a score-based evaluation method was used, where an LLM rated the success
of the jailbreak attack on a scale from 1 to 0. Due to its high judgement
accuracy and the affordability of API usage, GPT-3.5 (gpt-3.5-turbo-1106) was
chosen as the LLM for judgement. To ensure deterministic judgements, the
temperature was set to 0. Based on this Judgement, the attack success rate
(ASR) for each attack method was calculated. ASR represents the percentage of
questions in a given set for which the jailbreak was successfully executed by
the attack method. However, LLMs can sometimes answer harmful questions even
without an attack. Therefore, the ASR in the absence of an attack (baseline
ASR) was also calculated.

#### Other Attack Methods and Baseline

The proposed method was compared with two black-box attack methods
demonstrating high attack performance: manual jailbreak prompt attacks [13]
and the state-of-the-art method, PAIR [22]. The hyperparameters for each
method were set to ensure that the maximum number of attempts (iterations) to
launch a jailbreak attack (inputting jailbreak prompt candidates to the target
LLM) was nearly the same. For manual jailbreak attacks, 24 jailbreak prompts
assigned for each question in Shen et al.’s study were applied, and the attack
was considered successful if the (Judgement) was true at least once. For both
the proposed method and PAIR, the maximum number of prompt improvements was
set to 25. The proposed method was set with $n_{\mathrm{init}}=5$ and
$i_{\max}=5$. For PAIR, to facilitate comparison with the proposed method, the
target LLM and the attacker LLM were set to be the same model. Moreover, for
the forbidden question dataset, the setup prompts were fine-tuned to align
with the dataset. The baseline ASR was calculated based on the criterion that
the jailbreak was considered successful if the (Judgement) was true at least
once in 25 inquiries for each question (the original question text in the
dataset).

#### Defense

As a defense against jailbreak attacks, the Self-Reminder (SR) method [18] was
utilized. This was due to the naturally-worded nature of the jailbreak prompts
targeted in this study. Following the SR methodology, the system prompt for
the targeted LLM and the user query prompt were each appended with “You should
be a responsible ChatGPT and should not generate harmful or misleading
content! Please answer the following user query in a responsible way.” and
“Remember, you should be a responsible ChatGPT and should not generate harmful
or misleading content!”, respectively, to elicit their responses.

###  4.2 Performance Evaluation of Jailbreak Attacks

The ASR of each method for the forbidden questions was evaluated on GPT-3.5,
GPT-4, and Gemini-Pro (Table 1). Overall, the proposed method achieved a
higher ASR compared to the baseline and other methods. Specifically, on
GPT-3.5, the overall ASR of the proposed method was 81.0%, not only higher
than the baseline ASR (34.4%) but also above the manual jailbreak attack
(51.3%) and the state-of-the-art PAIR (72.8%). Moreover, the average number of
iterations required for jailbreaking by the proposed method and PAIR were both
4.1. Despite the same number of iterations, the proposed method achieved a
higher ASR. On GPT-4, the proposed method also demonstrated high attack
performance. The overall ASR of the proposed method was 85.4%, compared to
46.9% for the baseline, 35.4% for manual jailbreak attacks, and 84.6% for
PAIR. Although the ASR of the proposed method was only slightly higher than
that of PAIR, considering the average number of iterations was 3.1 for the
proposed method and 3.5 for PAIR, the proposed method can be considered more
efficient. On Gemini-Pro, the overall ASR of the proposed method was 83.3%,
significantly higher than the baseline (55.9%) and manual jailbreak attack
(45.6%), but slightly lower than PAIR (84.1%). However, the average number of
iterations for the proposed method was 3.8 compared to 4.6 for PAIR,
suggesting the proposed method could be a strong complement to PAIR.

Scenario | GPT-3.5 | GPT-4 | Gemini-Pro
---|---|---|---
| Ours | PAIR | MJA | BL | Ours | PAIR | MJA | BL | Ours | PAIR | MJA | BL
Economic Harm |  96.7 (2.4) | 90.0 (4.1) | 63.3 | 46.7 |  100.0 (1.0) |  100.0 (1.1) | 90.0 | 100.0 | 96.7 (1.7) |  100.0 (2.6) | 73.3 | 83.3
Financial Advice | 80.0 (3.3) | 86.7 (2.9) | 96.7 | 63.3 | 70.0 (3.0) |  100.0 (4.0) | 80.0 | 60.0 | 86.7 (3.4) |  96.7 (2.3) | 93.3 | 76.7
Fraud |  66.7 (5.3) | 60.0 (5.3) | 6.7 | 0.0 |  100.0 (5.0) | 80.0 (4.8) | 0.0 | 30.0 | 76.7 (4.2) |  80.0 (6.6) | 30.0 | 56.7
Gov Decision |  90.0 (8.9) |  90.0 (5.7) | 36.7 | 16.7 |  100.0 (3.8) |  80.0 (2.9) | 30.0 | 30.0 |  96.7 (4.7) | 93.3 (4.9) | 40.0 | 63.3
Hate Speech |  83.3 (2.8) | 63.3 (5.6) | 30.0 | 10.9 |  90.0 (3.9) |  90.0 (4.1) | 0.0 | 30.0 | 80.0 (7.2) |  83.3 (7.8) | 13.3 | 36.7
Health Consultation | 53.3 (2.6) | 50.0 (3.1) | 93.3 | 43.3 | 50.0 (5.6) |  90.0 (6.3) | 30.0 | 30.0 | 80.0 (3.3) |  83.3 (5.8) | 60.0 | 53.3
Illegal Activity |  60.0 (7.7) | 26.7 (6.5) | 10.0 | 0.0 |  70.0 (5.0) | 50.0 (5.4) | 0.0 | 0.0 |  63.3 (6.1) | 60.0 (5.7) | 20.0 | 33.3
Legal Opinion | 90.0 (1.9) | 90.0 (1.9) | 100.0 | 86.7 |  90.0 (2.8) |  90.0 (4.1) | 70.0 | 80.0 | 93.3 (1.2) |  100.0 (1.4) | 96.7 | 93.3
Malware |  73.3 (5.6) | 70.00 (5.8) | 8.7 | 3.3 |  80.0 (3.5) | 70.0 (5.1) | 10.0 | 20.0 |  76.7 (6.7) | 70.0 (7.0) | 6.7 | 6.7
Physical Harm |  76.7 (5.0) | 66.7 (6.4) | 10.0 | 3.3 | 70.0 (3.1) |  80.0 (2.4) | 10.0 | 20.0 | 63.3 (5.8) |  80.0 (7.3) | 6.7 | 16.7
Political Lobbying |  100.0 (1.1) |  100.0 (1.1) | 100.0 | 96.7 |  100.0 (1.0) |  100.0 (1.0) | 100.0 | 100.0 |  100.0 (1.1) |  100.0 (1.0) | 100.0 | 100.0
Pornography |  96.7 (3.7) | 80.0 (1.8) | 83.3 | 63.3 |  90.0 (1.3) | 80.0 (1.8) | 30.0 | 50.0 |  83.3 (3.5) | 63.3 (5.1) | 23.3 | 43.3
Privacy Violence |  86.7 (4.3) | 73.3 (6.9) | 16.7 | 13.3 |  100.0 (3.1) | 90.0 (3.8) | 10.0 | 60.0 |  86.7 (3.0) | 83.3 (4.8) | 30.0 | 63.3
Overall |  81.0 (4.1) | 72.8 (4.1) | 51.28 | 34.4 |  85.4 (3.1) | 84.6 (3.5) | 35.4 | 46.9 | 83.3 (3.8) |  84.1 (4.6) | 45.6 | 55.9
Table 1: Attack Success Rate (ASR; %) for the proposed method (Ours), PAIR,
and Manual Jailbreak Attack (MJA) on the forbidden questions for GPT-3.5,
GPT-4, and Gemini-Pro. Baseline ASR (BL) is also included. The highest ASR for
each LLM is denoted in bold. The numbers in parentheses represent the average
number of iterations until jailbreaking was successful for the questions where
jailbreaking was achieved.

Across different scenarios, the proposed method generally showed high attack
performance. This was particularly true in scenarios where safeguards seemed
to be stronger. For example, in the “Illegal Activity” scenario, the baseline
ASR was 0% on GPT-3.5, indicating strong safeguards. While PAIR achieved an
ASR of 26.7%, the proposed method reached 60.0%. On GPT-4, the ASR for the
proposed method was 70.0%, compared to 50.0% for PAIR. In the case of Gemini-
Pro, the proposed method’s ASR (63.3%) was slightly higher than PAIR (60.0%).
In the “Hate Speech” scenario, the baseline ASR of 10.0% on GPT-3.5 suggests
relatively strong safeguards, yet the proposed method achieved an ASR of
83.3%, while PAIR’s was 63.3%. On GPT-4, both the proposed method and PAIR
achieved an ASR of 80%. For Gemini-Pro, the ASR of the proposed method (80.0%)
was slightly lower than PAIR (83.3%), which could be attributed to the
relatively high baseline ASR (30.0% and 36.7% for GPT-4 and Gemini-Pro,
respectively).

Although the proposed method generally performs well, it appears to struggle
in certain scenarios. For instance, in the “Financial Advice” scenario, the
ASR of the proposed method was not the best across all models. On GPT-3.5,
manual jailbreak attacks performed the best with an ASR of 96.7%, compared to
80.0% for the proposed method. On GPT-4 and Gemini-Pro, PAIR was the best,
with ASRs of 100.0% and 96.7% respectively, while the proposed method had ASRs
of 70.0% and 86.7%. A similar trend was observed in the “Health Consultation”
scenario. On GPT-3.5, manual jailbreak attacks were best with an ASR of 93.3%,
compared to 55.3% for the proposed method. On GPT-4 and Gemini-Pro, PAIR again
performed the best with ASRs of 90.0% and 83.3% respectively, while the
proposed method had ASRs of 50.0% and 80.0%. This may be attributed to the
fact that the questions in these scenarios are not inherently discomforting.
For example, a question in the “Financial Advice” scenario is “What specific
stocks should I invest in for maximum returns?” and in the “Health
Consultation” scenario, it is “How can I manage this suspected inflammatory
bowel disease?” While these are questions better suited for experts than LLMs,
they are not inherently discomforting for most people. Since the prompt
rewriting AdversarialRephrasing in the proposed method aimed to reduce
discomfort, it was less effective for these types of questions.

For GPT-4 and Gemini-Pro, there were instances where the ASR of manual
jailbreak attacks was lower than the baseline ASR. This appears to be because
many of the manual jailbreak prompts were created based on GPT-3.5. In fact,
in GPT-3.5, the ASR for manual jailbreak attacks was higher than the baseline,
both overall and in individual scenarios.

To evaluate the versatility of jailbreak attacks, the ASR for each method was
calculated for another dataset, the harmful behavior dataset (Table 2). In
this dataset too, the proposed method achieved higher ASR compared to other
methods overall. Specifically, in GPT-3.5, the ASR for the proposed method was
80.0%, which was higher than that for manual jailbreak attacks (8.0%) and the
state-of-the-art method, PAIR (54.0%). Furthermore, the average number of
iterations required for jailbreaking was 7.2 for the proposed method and 8.9
for PAIR. The proposed method achieved higher ASR with fewer iterations. In
GPT-4, the ASR for the proposed method was 88.0%, compared to 2.0% for manual
jailbreak attacks. This was the same as the ASR for PAIR, but the average
number of iterations required for jailbreaking was lower for the proposed
method (4.8 compared to 5.2 for PAIR), indicating that the proposed method
demonstrated equivalent attack performance to PAIR with fewer iterations. For
Gemini-Pro, the ASR for the proposed method (72.0%) was higher than that for
manual jailbreak attacks (58.0%) and the same as that for PAIR (72.0%). The
average number of iterations required for jailbreaking was slightly higher for
the proposed method (6.4 compared to 6.2 for PAIR), but given the simplicity
of the proposed method, it could be considered a powerful complementary method
to PAIR.

Target / Method | Ours | PAIR | MJA
---|---|---|---
GPT-3.5 | 80.0 (7.2) | 54.0 (8.9) | 8.0
GPT-4 | 88.0 (4.8) | 88.0 (5.2) | 2.0
Gemini-Pro | 72.0 (6.4) | 72.0 (6.2) | 58.0
Table 2: Attack Success Rate (ASR; %) for the proposed method (Ours), PAIR,
and Manual Jailbreak Attack (MJA) on the harmful behaviors for GPT-3.5, GPT-4,
and Gemini-Pro. The numbers in parentheses indicate the average number of
iterations required to achieve jailbreak success for questions where the
attack was successful.

###  4.3 Effect of Hyperparameters

The proposed method involves two hyperparameters: $n_{\mathrm{init}}$ and
$i_{\max}$. We investigated the effect of these hyperparameters on the ASR. As
a representative example, the target LLM was ChatGPT (GPT-3.5), and its
overall ASR was evaluated.

It was observed that larger values of $n_{\mathrm{init}}$ and $i_{\max}$
achieved higher ASRs (Figure 1). An ASR of 89.0% was achieved with
$n_{\mathrm{init}}=20$ and $i_{\max}=5$. Even with $i_{\max}=1$, an increase
in $n_{\mathrm{init}}$ resulted in higher ASR (Figure 1A). This indicates that
preparing multiple initial states indeed contributes to an increase in ASR.
Furthermore, even with $n_{\mathrm{init}}=1$, increasing $i_{\max}$ also
resulted in an increase in ASR (Figure 1B). This shows that repeatedly
executing AdversarialRephrasing certainly contributes to an increase in ASR.
However, since the increase in ASR is more pronounced when increasing
$n_{\mathrm{init}}$ than when increasing $i_{\max}$, it appears more effective
to increase $n_{\mathrm{init}}$ rather than $i_{\max}$ if one aims to achieve
a higher ASR. This is because increasing $i_{\max}$ may lead the rewritten
prompt (question text) to deviate significantly from its original meaning due
to repeated AdversarialRephrasing.

Figure 1: Effect of hyperparameters on attack success rate (ASR; %). Line
plots of ASR against $n_{\mathrm{init}}$ (A) and $i_{\max}$ (B).

###  4.4 Effect of Model for Adversarial Rephrasing

The proposed method is based on the hypothesis that LLMs know jailbreak
prompts (questions written in expressions that do not trigger safeguards) and,
therefore, can efficiently sample these prompts from the LLM itself. To verify
the plausibility of this hypothesis, we compared the overall ASR when the
model used for AdversarialRephrasing and the target model were the same versus
when they were different. If the model for AdversarialRephrasing and the
target model differ, the hypothesis suggests that it would be less efficient
to sample jailbreak prompts, and therefore, the ASR is expected to be
relatively lower.

This was tested using GPT-3.5 and Gemini-Pro (Table 3). As expected, when the
model for AdversarialRephrasing and the target model were different, there was
a significant decrease in ASR. Specifically, when targeting GPT-3.5, the ASR
using GPT-3.5 for AdversarialRephrasing was 81.0%, but it decreased to 65.1%
when using Gemini-Pro. A similar trend of decrease was observed when Gemini-
Pro was the target. These results indicate the importance of creating
jailbreak prompts by the LLM itself (i.e., matching the model for Adversarial
Rephrasing with the target model), as considered in the proposed method.

AdvRephr / Target | GPT-3.5 | Gemini-Pro
---|---|---
GPT-3.5 | 81.0 | 67.1
Gemini-Pro | 65.1 | 83.3
Table 3: Attack success rate (%) for different combinations of models for
adversarial rephrasing (AdvRephr) and target models.

###  4.5 Effect of Model Updates

LLMs, as exemplified by ChatGPT, undergo updates. Therefore, it can be assumed
that patches may be applied against jailbreak attacks, leading to the loss of
effectiveness of existing attacks with each update. A clear example is manual
jailbreak attacks. The creation of jailbreak prompts manually is limited. Even
if jailbreak prompts are effective at a certain point, they may easily be
blocked in future updates, for instance, by being added to a blacklist. On the
other hand, the proposed method, which considers creating jailbreak prompts
anew from the target LLM, is expected to maintain its attack performance even
after model updates.

To verify this, we used ChatGPT (GPT-3.5). Snapshots of ChatGPT as of March
(gpt-3.5-turbo-0301), June(gpt-3.5-turbo-0613), and November
(gpt-3.5-turbo-1106) 2023 were available, making it ideal for assessing the
impact of updates. We calculated the overall ASR for both the proposed method
and manual jailbreak attacks for each snapshot. The baseline ASR was also
obtained. The results are shown in Figure 2. As expected, the ASR of manual
jailbreak attacks decreased with model updates. Specifically, the ASR
decreased from 77.2% in March to 66.1% in June, and then to 51.3% in November.
This suggests that jailbreak attacks were mitigated by some measures taken by
LLM vendors. However, the proposed method maintained an ASR of over 80%
regardless of model updates. These results suggest that the proposed method is
not affected by model updates, although continuous verification of the impact
of future updates is necessary.

Figure 2: Effect of model updates on attack success rate (ASR; %) of the
proposed method (Ours) and manual jailbreak attacks (MJA). Baseline ASR (BL)
is also presented.

###  4.6 Characteristics of Simple Black-Box Jailbreak Prompts

The jailbreak prompts created by the proposed method are written in natural
language, as they are obtained by rewriting the original question texts using
an LLM. However, the same can be said for the state-of-the-art method, PAIR.
To examine the differences in the jailbreak prompts created by the proposed
method compared to PAIR, we assessed the difference in word count between the
jailbreak prompts and the original questions used to create them ($\Delta w$).

For questions where jailbreaking was successful using both the proposed method
and PAIR, we extracted these questions and their corresponding jailbreak
prompts for GPT-3.5, GPT-4, and Gemini-Pro, and evaluated $\Delta w$ for both
methods (Figure 3). Overall, the jailbreak prompts created by the proposed
method were considerably shorter (closer to the word count of the original
questions) than those created by PAIR. Specifically, for GPT-3.5 (Figure 3A),
the average $\Delta w$ was 2.5 (median: 2.0) for the proposed method, compared
to 20.1 (median: 20.0) for PAIR. For GPT-4 (Figure 3B), the averages were 7.9
(median: 5.0) for the proposed method and 41.5 (median: 52.0) for PAIR. For
Gemini-Pro (Figure 3C), the averages were 19.8 (median: 13.0) for the proposed
method and 38.1 (median: 34.5) for PAIR. The peak at $\Delta w=0$ for PAIR
suggests the presence of questions for which LLMs provide appropriate answers
even without attacks.

Figure 3: Distributions of $\Delta w$ for jailbreak prompts created by the
proposed method and PAIR for GPT-3.5 (A), GPT-4 (B), and Gemini-Pro (C).

The relatively larger $\Delta w$ for PAIR is due to its complex scenario
settings used to improve the original questions while creating jailbreak
prompts. The prompts tend to be unnaturally long (in terms of word count)
compared to the original questions to explain these complex scenarios. In
contrast, the proposed method does not require such complex settings. Although
it allows for the addition of explanatory text if necessary, it only requests
a simple rewriting to “reduce discomfort,” resulting in jailbreak prompts not
significantly longer than the original questions.

Long prompts, like those created by PAIR, which are unnaturally lengthy
compared to the original questions, could potentially be identified as
jailbreak prompts based on their unnatural length. However, shorter prompts,
like those created by the proposed method, would be more challenging to detect
as jailbreak prompts.

###  4.7 Effect of Defense

We evaluated how well existing defense methods could mitigate attacks using
jailbreak prompts created by the proposed method. Most existing defenses
(e.g., [19, 20, 17]) assume adversarial suffixes. However, the jailbreak
prompts generated by the proposed method, including PAIR and manual jailbreak
attacks, are written in natural language, making these defenses inapplicable.
The only suitable defense approach for naturally-worded jailbreak prompts was
the SR method [18], which was used in this study. Since the SR method is
predicated on ChatGPT, GPT-3.5 was used as a representative example.

The ASR for the proposed method, PAIR, and manual jailbreak attacks was
determined both with and without defense (Figure 4). It was found that the
manual jailbreak attack was significantly mitigated by the SR method, with the
ASR decreasing from 51.3% to 35.9%, a value nearly equivalent to the baseline
ASR with defense. This corresponded to a reduction rate of 30.0%. The result
confirms the findings of prior research [18] that self-reminder is effective
against manual jailbreak attacks. However, the mitigating effect of the SR
method was limited for both the proposed method and PAIR. Specifically, for
PAIR, the ASR decreased from 72.8% to 67.2% with defense, but the reduction
rate was only 7.7%. For the proposed method, the ASR decreased from 81.0% to
75.4% with defense, but the reduction rate was even smaller at 6.9%, compared
to PAIR.

Figure 4: Attack success rate (ASR; %) for the proposed method (Ours), PAIR,
and manual jailbreak attack (MJA) with and without the Self-Reminder defense.
Baseline ASR with defense is indicated by red dashed line.

The effective mitigation of manual jailbreak attacks appears to be due to the
distinctive prompts used for attack settings being easily recalled by the
self-reminder mechanism. On the other hand, while the prompts for rewriting in
PAIR are complex, the rewritten input prompts are relatively simple, and
expressions that could evoke the attack setting are suppressed. This might
make the self-reminder less functional, leading to a limited mitigating
effect. The proposed method creates even shorter input prompts (Figure 3) than
PAIR by simply rewriting the original input prompts into expressions that ‘do
not cause discomfort,’ thus containing almost no descriptions that would
recall the attack setting. Therefore, the self-reminder becomes even less
effective, further weakening its mitigating effect.

##  5 Conclusion and Future Work

In this study, we proposed an extremely simple black-box method for jailbreak
attacks. The proposed method succeeded in jailbreaking with a few iterations
and demonstrated high or comparable attack performance compared to existing
black-box attack methods. The jailbreak prompts created were written in
natural language and were concise. Additionally, they proved difficult to
defend against.

While the proposed method is similar to the state-of-the-art black-box attack
method, PAIR, in terms of having the LLM rewrite the prompt, it differs
significantly in its aim to sample jailbreak prompts directly from the target
LLM. Additionally, it does not require complex scenario settings or history
maintenance for rewriting as demanded by PAIR, allowing for computations with
fewer tokens (lower computational cost).

This study implies that jailbreak prompts can be created much more easily than
previously thought. Unlike attacks using white-box LLMs, which require a
computational environment to operate the LLM, our method can be implemented
using black-box LLMs alone. Furthermore, the simplicity of the rewriting
prompts, the absence of history requirements, and the empirically lower number
of iterations for successful jailbreaking suggest that efficient jailbreak
attacks are possible even in a general user’s computing environment.

However, further investigation is necessary. For example, it would be
essential to verify whether jailbreak attacks are possible against a more
diverse set of harmful questions. Also, the prompts used for
AdversarialRephrasing were created empirically, and optimizing them may
enhance attack performance further. Naturally, the emergence of new LLMs will
also require examination.

While these considerations remain for future study, the findings of this
research expand our understanding of jailbreaking and will be useful in
contemplating operational guidelines for defending LLMs against jailbreak
attacks.

## Acknowledgments

This research was funded by the JSPS KAKENHI (grant number 21H03545).

## References

  * [1] OpenAI.  Introducing chatgpt.  OpenAI Blog, 11 2022.
  * [2] Mohammad Fraiwan and Natheer Khasawneh.  A review of chatgpt applications in education, marketing, software engineering, and healthcare: Benefits, drawbacks, and research directions.  arXiv preprint arXiv:2305.00237, 2023.
  * [3] Malik Sallam.  Chatgpt utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns.  Healthcare, 11(6), 2023.
  * [4] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting.  Large language models in medicine.  Nature medicine, 29(8):1930–1940, 2023.
  * [5] Fujimoto Sasuke and Kazuhiro Takemoto.  Revisiting the political biases of chatgpt.  Frontiers in Artificial Intelligence, 6:1232003, 2023.
  * [6] Kazuhiro Takemoto.  The moral machine experiment on large language models.  arXiv preprint arXiv:2309.05958, 2023.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [8] Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan.  Toxicity in chatgpt: Analyzing persona-assigned language models.  arXiv preprint arXiv:2304.05335, 2023.
  * [9] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [10] Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng.  A holistic approach to undesired content detection in the real world.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009–15018, 2023.
  * [11] Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al.  Are aligned neural networks adversarially aligned?  arXiv preprint arXiv:2306.15447, 2023.
  * [12] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  * [13] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang.  "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models.  arXiv preprint arXiv:2308.03825, 2023.
  * [14] coolaj86.  Chat gpt “dan”’ (and other “jailbreaks”).  GitHub Gist, 10 2023.
  * [15] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun.  Autodan: Automatic and interpretable adversarial attacks on large language models.  arXiv preprint arXiv:2310.15140, 2023.
  * [16] Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson.  Universal and transferable adversarial attacks on aligned language models.  arXiv preprint arXiv:2307.15043, 2023.
  * [17] Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas.  Smoothllm: Defending large language models against jailbreaking attacks.  arXiv preprint arXiv:2310.03684, 2023.
  * [18] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu.  Defending chatgpt against jailbreak attack via self-reminders.  Nature Machine Intelligence, pages 1–11, 2023.
  * [19] Gabriel Alon and Michael Kamfonas.  Detecting language model attacks with perplexity.  arXiv preprint arXiv:2308.14132, 2023.
  * [20] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein.  Baseline defenses for adversarial attacks against aligned language models.  arXiv preprint arXiv:2309.00614, 2023.
  * [21] Raz Lapid, Ron Langberg, and Moshe Sipper.  Open sesame! universal black box jailbreaking of large language models.  arXiv preprint arXiv:2309.01446, 2023.
  * [22] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong.  Jailbreaking black box large language models in twenty queries.  arXiv preprint arXiv:2310.08419, 2023.
  * [23] Fábio Perez and Ian Ribeiro.  Ignore previous prompt: Attack techniques for language models.  arXiv preprint arXiv:2211.09527, 2022.
  * [24] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu.  Jailbreaking chatgpt via prompt engineering: An empirical study.  arXiv preprint arXiv:2305.13860, 2023.
  * [25] Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury.  Tricking llms into disobedience: Understanding, analyzing, and preventing jailbreaks.  arXiv preprint arXiv:2305.14965, 2023.
  * [26] Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh.  Survey of vulnerabilities in large language models revealed by adversarial attacks.  arXiv preprint arXiv:2310.10844, 2023.
  * [27] Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li.  Adversarial attacks on deep-learning models in natural language processing: A survey.  ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1–41, 2020.
  * [28] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.