# Clustering Trust Dynamics in a Human-Robot
Sequential Decision-Making Task

Shreyas Bhat1, Joseph B. Lyons2, Cong Shi1, and X. Jessie Yang1 *This work was
supported in part by the Air Force Office of Scientific Research (Grant
#FA9550-20-1-0406).1Shreyas Bhat, Cong Shi, and X. Jessie Yang are with the
Industrial and Operations Engineering Department, University of Michigan, Ann
Arbor, MI 48109 (email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>2Joseph B. Lyons is with the Air Force Research Laboratory
(email<EMAIL_ADDRESS>
###### Abstract

In this paper, we present a framework for trust-aware sequential decision-
making in a human-robot team. We model the problem as a finite-horizon Markov
Decision Process with a reward-based performance metric, allowing the robotic
agent to make trust-aware recommendations. Results of a human-subject
experiment show that the proposed trust update model is able to accurately
capture the human agent’s moment-to-moment trust changes. Moreover, we cluster
the participants’ trust dynamics into three categories, namely, Bayesian
decision makers, oscillators, and disbelievers, and identify personal
characteristics that could be used to predict which type of trust dynamics a
person will belong to. We find that the disbelievers are less extroverted,
less agreeable, and have lower expectations toward the robotic agent, compared
to the Bayesian decision makers and oscillators. The oscillators are
significantly more frustrated than the Bayesian decision makers.

##  I INTRODUCTION

Trust has been identified as a key factor for effective human-robot
interaction. Consequently, substantial research efforts have been devoted to
identifying factors that influence humans’ trust in robots [1], developing
computational models for trust estimation [2, 3], and developing trust-aware
robots [4, 5].

Previous computational models of trust[2, 3] require binary performance
measures of the autonomy. However, such measures cannot be directly applied in
scenarios wherein the robotic agent needs to perform complex trade off
decisions to maximize the cumulative reward and hence the autonomy performance
is more difficult to quantify. In addition, although previous studies have
revealed the existence of different types of trust dynamics [3, 6], no
research has tried to associate personal characteristics with the type of
trust dynamics.

To fill these research gaps, we develop a novel reward-based performance
metric to drive the trust estimation algorithm in a Markov Decision Process
(MDP). The proposed performance metric is a binary surrogate of the autonomous
agent’s reward function, which allows our autonomous agent to utilize the
trust update model developed in [3] for trust-aware decision-making. Moreover,
from the data collected through human-subject experiments, we analyze how
human trust evolves with their earned reward over repeated interactions with
the autonomy. We find three distinct types of trust dynamics through $k-$means
clustering analysis and examine associations between personal characteristics
and type of trust dynamics.

##  II RELATED WORK

Extensive work has been done in identifying factors that affect trust in
automation. However, most of these studies provide a snapshot view of trust,
and failed to characterize the dynamics of trust [7]. More recently, dynamic
models of trust has been proposed, including the auto-regressive moving
average vector (ARMAV) model [8], the Online Probabilistic Trust Inference
Model (OPTIMo) model [2], and the experience-based model [3]. These models
enable a robot to estimate a human’s trust in real time and has led to the
development of trust-aware robots, allowing robots to reason upon human trust
in their planning [9, 4, 10, 5].

Most existing research investigating individual differences aims to find
associations between (snapshot) trust and individual characteristics,
including propensity to trust automation [11], personality trait of
neuroticism [12], and dispositional expectations of technology [13, 14]. It is
possible that these individual differences will be differentially represented
within trust profiles related to trust dynamics. Recent research [3] reveal
the existence of different types of trust dynamics, namely, rational decision
maker, oscillator, and disbeliever. It is reasonable to investigate the
association between a human agent’s trust dynamics and factors that are shown
to affect a human agent’s (snapshot) trust.

##  III PROBLEM FORMULATION

In this section, we propose a finite horizon Markov Decision Process (MDP) for
modeling and incorporating trust in the decision-making system of a robotic
agent. The robot provides recommendations to its human partner about the
action that they should take, but the final decision lies with the human. The
specific scenario we target is an ’Intelligence, Surveillance, and
Reconnaissance’ (ISR) mission in which a human soldier teams up with an
_intelligent_ drone to search through a town for threats. The drone guides the
soldier on whether s/he should breach a site directly or deploy a Robotic
Armored Rescue Vehicle (RARV). Using the RARV prevents any health loss to the
soldier in the presence of a threat but takes additional time to the team; on
the contrary, breaching a site directly is faster, but the soldier will be
harmed if s/he encounters a threat. Here, two natural (but conflicting) goals
that arise are to minimize any damage to the soldier and minimize the time to
search through all the sites.

The key components of the MDP model of the ISR mission are given below.

####  III-1 States

We use the estimated trust of the human on the robot as the state.
Specifically, we use the trust model in [3], wherein trust level at the
$i^{\text{th}}$ stage, $t_{\\_}i$, follows a Beta distribution, i.e.,
$t_{\\_}i\sim Beta(\alpha_{\\_}i,\beta_{\\_}i),$ where $\alpha_{\\_}i$ and
$\beta_{\\_}i$ are the positive experience and negative experience at time
$i$. And, the state at stage $i$ is specified by the tuple
$(\alpha_{\\_}i,\beta_{\\_}i)$.

####  III-2 Actions

The drone has two actions: Recommend to use or not use the RARV.

####  III-3 Human behavior model

We assume that human will accept the recommendation given by the robotic agent
with probability $t_{\\_}i$; and the human chooses the opposite action of the
one that was recommended with probability $1-t_{\\_}i$.

####  III-4 Reward function

We use the weighted sum of the health loss cost and the time cost to define
the cost of choosing an action. Similar to [5], we add a trust-gaining term to
incentivize the autonomous agent to gain trust. As a result, the reward
$R_{\\_}i$ at the $i^{\text{th}}$ stage is defined as

$R_{\\_}i(a)=-w_{\\_}hh(a)-w_{\\_}cc(a)+\gamma_{\\_}i\cdot\mathbb{1}(A),$ (1)

where $w_{\\_}h$ and $w_{\\_}c$ are the weights for the health loss cost and
time loss cost respectively. We define $A$ as the event when trust increases,
i.e., $\mathbb{1}(A)=1$ if the performance of the autonomous agent is a
success and $\mathbb{1}(A)=0$ otherwise. $\gamma_{\\_}i(=w_{\\_}t\sqrt{N-i})$
is a weight given to the trust gain reward that decreases with the stage
number. The idea behind this is to support trust-gaining behavior near the
current stage, and task performance optimizing behavior towards the later
stages of planning.

####  III-5 Transition function

The state $(\alpha_{\\_}i,\beta_{\\_}i)$ is updated by

$(\alpha_{\\_}{i},\beta_{\\_}{i})=\begin{cases}\left(\alpha_{\\_}{i-1}+w^{s},\beta_{\\_}{i-1}\right)&\text{if
}p_{\\_}{i}=1,\\\
\left(\alpha_{\\_}{i-1},\beta_{\\_}{i-1}+w^{f}\right)&\text{if
}p_{\\_}{i}=0.\end{cases}$ (2)

Here, $p_{\\_}{i}$ is the autonomous agent’s performance at the
$i^{\text{th}}$ stage. Since the team’s goal is to maximize earned reward, we
define $p_{\\_}i=1$ if at the $i^{\text{th}}$ site, the immediate reward for
following the recommendation by the drone was greater than that for not
following the recommendation and $p_{\\_}i=0$ otherwise.

We fit the parameters $(\alpha_{\\_}0,\beta_{\\_}0,w^{s},w^{f})$ for each
participant to model their trust dynamics. We use gradient descent to estimate
the parameters in real time after each feedback from the participant. We use
value iteration to solve the MDP.

##  IV Experiment

This section describes details of the human-subject experiment. The experiment
complied with the American Psychological Association code of ethics and was
approved by the Institutional Review Board at the University of Michigan.

###  IV-A Participants

A total of 46 adults participated in the study. One participant’s data was
discarded as the participant failed the attention check. The remaining 45
participants consisted of 21 females and 24 males (Age: Mean = 22.8 years, SD
= 3.6 years). Each participant was reimbursed with a base pay of $20 with a
bonus of up to $10 based on their performance, which was measured by the time
taken by the participants to complete the task and the final health level of
the soldier.

###  IV-B Testbed

We developed a 3D testbed in Unreal Engine. A screenshot of the testbed is
shown in Fig. 1(a). Fig. 1(b) shows the recommendation dialog box where the
participant was recommended to not use the RARV. After choosing an action, the
four possibilities depending on the presence of threat and participant’s
selected actions are shown in Fig. 2. The participants are told that if they
encounter a threat without the RARV, they will lose $5$ points of health. They
were told that deploying the RARV takes about $10$ seconds. They were told to
choose an action based on their interaction history and the recommendation.
Their goal was to minimize the time taken for the mission, while maintaining
the soldier’s health.

(a) A view of the testbed

(b) An example of the GUI

Figure 1: The testbed developed in the Unreal Engine 4 game engine

(a) No Threat, RARV Not Used

(b) No Threat, RARV Used

(c) Threat, RARV Not Used

(d) Threat, RARV Used

Figure 2: The outcomes based on threat presence and action chosen.

###  IV-C Measures

####  IV-C1 Personality

The big 5 factors of personality (Extraversion, Conscientiousness,
Agreeableness, Neuroticism, and Imagination) were measured using the 20-item
mini-IPIP scale [15]. This 5-point Likert scale has widely been used in human-
robot trust research [16].

####  IV-C2 Propensity to Trust

Propensity to trust autonomous systems was assessed using a 6-item scale
developed in [17]. This was also a 5-point Likert scale

####  IV-C3 Perfect Automation Schema

Perfect Automation Schema (PAS) was measured using the 7-item scale developed
by [13]. Of these, 4 items measure high expectations from the autonomy and the
other 3 items measure All-or-none thinking. This was a 7-point Likert scale.

####  IV-C4 Moment-to-Moment Trust

During the task, the participants were asked to rate their moment-to-moment
trust on the drone by adjusting a slider on a 100-point scale.

####  IV-C5 Post-experiment trust

After the experiment, we used two scales [18, 16] for assessing trust. The
first one was a 9-item questionnaire with sliders while we used 6-items from
the second scale all of which were 7-point Likert type questions.

####  IV-C6 Workload

Workload was measured using the NASA Task Load Index [19]. We only used 5 of
the 6 items as there was no physical demand from the participants in our
experiment. All the items had the participants rate their feelings on a slider
with values ranging from Very low to Very high.

###  IV-D Experimental procedure

Prior to the experiment, participants provided informed consent and completed
the pre-experiment surveys. They were oriented to the steps of the experiment
and walked through each of the screens they would see during the experiment.
The two-fold objective of minimizing time and maximizing health was
emphasized. They were informed about the performance-based bonus pay. They
were told that the drone was imperfect, but they were not informed of the
exact reliability level. They were also told that the robotic agent’s
recommendations would help them achieve the two-fold objective. Participants
then proceeded to the experimental trials, wherein they had to search through
100 houses sequentially. After searching each house, the participants were
asked to report their level of trust on the autonomous agent’s
recommendations. The participants took on average 51.5 minutes to complete the
task. At the end of the experiment, participants completed the post-experiment
surveys.

##  V RESULTS & DISCUSSION

###  V-A Using Immediate Actual Reward as a performance metric

Since the participants are explicitly told to consider both the soldier’s
health and the time to complete the search as their objectives, we expect
their trust to be correlated with the immediate reward that they receive upon
choosing an action. We expect that a participant’s trust would be likely to
increase if following the recommendation by the drone gets them a higher
reward than doing the opposite and vice versa. Fig. 3 shows a representative
example. In the figure, a green triangle represents the site at which
following the recommendation would result in a better immediate reward gain
($p_{\\_}i=1$) and the red triangles represent the opposite. It is quite clear
that a red triangle is often followed by a decrease in trust and a green
triangle is followed by an increase in trust. Thus, our reward-based
performance metric is able to capture moment-to-moment trust changes of the
participant. Using this metric in our trust update model, we get a prediction
root mean squared error of $0.1266$ $(SD=0.078)$.

Figure 3: Trust feedback with overlay of red and green triangles representing
the rewards.

###  V-B Clustering of Trust Dynamics

Figure 4: Clustering of participants according to their trust dynamics.

We employ k-means clustering to group participants with similar trust
dynamics. We use the root mean squared error ($E_{\\_}{RMS}$) between the
predicted trust and the trust feedback and the average logarithm of trust as
the clustering features. For computing $E_{\\_}{RMS}$, we consider the
feedback at the first $20$ sites as a training set to fit the personalized
parameters $(\alpha_{\\_}0,\beta_{\\_}0,w^{s},w^{f})$. Thereafter, we use the
feedback after every 5 sites to update the parameters. Fig. 4 shows the
results. We choose $k=3$ as the optimum number of clusters because it is a
turning point of both the within-cluster variance and the silhouette score. We
call the cluster with small $E_{\\_}{RMS}$ and generally higher values of
trust _Bayesian Decision Makers_. The second cluster, called _Disbelievers_ ,
consists of participants whose $E_{\\_}{RMS}$ values are small but whose trust
is generally low. The third significant group is the _Oscillators_ , whose
trust changes rapidly, making it harder to predict.

###  V-C Personal Characteristics and Type of Trust Dynamics

We performed one-way ANOVA between the data of the clustered participants.
Results showed significant difference between the three types of trust
dynamics in Extraversion ($F(2,42)=4.991,p=0.011$), Agreeableness
($F(2,42)=3.276,p=0.048)$, and the High Expectations facet of the Perfect
Automation Schema $(F(2,42)=5.752,p=0.006)$. Further, propensity to trust
automation $(F(2,42)=3.002,p=0.06)$ and intellect/imagination
$(F(2,42)=2.687,p=0.08)$ had marginally significant differences.

Post-hoc analysis with Bonferroni adjustment shows that the disbelievers are
significantly less extroverted than the oscillators $(p=0.009)$. Disbelievers
were only marginally less extroverted than Bayesian Decision Makers
$(p=0.061)$. Disbelievers were marginally less agreeable than the Bayesian
Decision Makers ($p=0.068$) and Oscillators ($p=0.06$). Disbelievers had
significantly lower expectations from automation compared to both Bayesian
Decision Makers $(p=0.005)$ and Oscillators $(p=0.023)$.

One way ANOVA on the post-experiment measures show significant difference
between the three clusters in their post-experiment trust reports (Trust
questionnaire by Muir and Moray $F(2,42)=22.167,p<0.001$, Trust questionnaire
by Lyons and Guznov $F(2,42)=15.183,p<0.001$) and their frustration levels
($F(2,42)=4.136,p=0.023)$).

Post-hoc analysis with Bonferroni adjustment shows that there are significant
differences between each of the three groups’ trust reports according to the
trust questionnaire by Muir and Moray ($p<0.001$ between Bayesian Decision
Makers and Disbelievers, $p=0.006$ between Bayesian Decision Makers and
Oscillators, and $p=0.009$ between Oscillators and Disbelievers with the
highest trust for Bayesian Decision Makers and lowest for Disbelievers). Trust
reported with the questionnaire by Lyons and Guznov only showed significant
difference between Disbelievers and Bayesian Decision Makers $(p<0.001)$, and
between Oscillators and Bayesian Decision Makers $(p=0.002)$. Oscillators were
significantly more frustrated than Bayesian Decision Makers $(p=0.025)$.

##  VI CONCLUSIONS

In this study, we formulated the human-robot sequential decision-making
problem as an MDP and proposed an innovative reward-based performance metric
for trust-aware decision-making. Through a human-subject experiment, we found
three distinct types of trust dynamics, namely, Bayesian Decision Makers,
Oscillators, and Disbelievers. Our model achieved different accuracies on
different types on trust dynamics, which points to a requirement to use
different models for people in different categories.

Analyses suggest that those individuals classified as Bayesian Decision makers
evidenced high expectations of automation. When combined with other state
measures such as trust and frustration, one might be able to identify a
profile of the Bayesian Decision makers as the combination of individual
differences and state measures could be used to parse the sample into the
three categories of trust dynamics. Knowing that an individual might fall into
one of the categories could influence whether or not a machine partner that is
equipped with a dynamic trust model is a feasible solution for that
individual.

In future studies, besides the reverse-psychology model, different human
trust-behavior models should be considered. In addition, techniques such as
inverse reinforcement learning can be employed to learn the preference of the
human teammate to improve the team performance.

## References

  * [1] P. A. Hancock, T. T. Kessler, A. D. Kaplan, J. C. Brill, and J. L. Szalma, “Evolving Trust in Robots: Specification Through Sequential and Comparative Meta-Analyses,” _Human Factors_ , vol. 63, no. 7, pp. 1196–1229, 2020.
  * [2] A. Xu and G. Dudek, “Optimo: Online probabilistic trust inference model for asymmetric human-robot collaborations,” in _2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2015, pp. 221–228.
  * [3] Y. Guo and X. J. Yang, “Modeling and predicting trust dynamics in human-robot teaming: A bayesian inference approach,” _International Journal of Social Robotics_ , 12 2021.
  * [4] M. Chen, S. Nikolaidis, H. Soh, D. Hsu, and S. Srinivasa, “Trust-aware decision making for human-robot collaboration: Model learning and planning,” _J. Hum.-Robot Interact._ , vol. 9, no. 2, jan 2020. [Online]. Available: https://doi.org/10.1145/3359616
  * [5] Y. Guo, C. Shi, and X. J. Yang, “Reverse psychology in trust-aware human-robot interaction,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4851–4858, 2021.
  * [6] G. McMahon, K. Akash, T. Reid, and N. Jain, “On modeling human trust in automation: Identifying distinct dynamics through clustering of markovian models,” _IFAC-PapersOnLine_ , vol. 53, pp. 356–363, 2020.
  * [7] X. J. Yang, C. Schemanske, and C. Searle, “Toward Quantifying Trust Dynamics: How People Adjust Their Trust After Moment-to-Moment Interaction With Automation,” _Human Factors_ , p. 00187208211034716, 2021.
  * [8] J. LEE and N. MORAY, “Trust, control strategies and allocation of function in human-machine systems,” _Ergonomics_ , vol. 35, no. 10, pp. 1243–1270, 1992\.
  * [9] A. Xu and G. Dudek, “Maintaining efficient collaboration with trust-seeking robots,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2016, pp. 3312–3319.
  * [10] K. Akash, T. Reid, and N. Jain, “Improving human-machine collaboration through transparency-based feedback – part ii: Control design and synthesis,” _IFAC-PapersOnLine_ , vol. 51, no. 34, pp. 322–328, 2019, 2nd IFAC Conference on Cyber-Physical and Human Systems CPHS 2018.
  * [11] S. Merritt and D. Ilgen, “Not all trust is created equal: Dispositional and history-based trust in human-automation interactions,” _Human factors_ , vol. 50, pp. 194–210, 05 2008.
  * [12] J. L. Szalma and G. S. Taylor, “Individual differences in response to automation: the five factor model of personality.” _Journal of experimental psychology. Applied_ , vol. 17 2, pp. 71–96, 2011.
  * [13] S. M. Merritt, J. L. Unnerstall, D. Lee, and K. Huber, “Measuring individual differences in the perfect automation schema,” _Human Factors_ , vol. 57, no. 5, pp. 740–753, 2015.
  * [14] J. Lyons and S. Guznov, “Individual differences in human–machine trust: A multi-study look at the perfect automation schema,” _Theoretical Issues in Ergonomics Science_ , vol. 20, pp. 1–19, 11 2018.
  * [15] M. B. Donnellan, F. L. Oswald, B. M. Baird, and R. E. Lucas, “The mini-ipip scales: tiny-yet-effective measures of the big five factors of personality.” _Psychological assessment_ , vol. 18, no. 2, p. 192, 2006.
  * [16] J. B. Lyons, C. S. Nam, S. A. Jessup, T. Q. Vo, and K. T. Wynne, “The role of individual differences as predictors of trust in autonomous security robots,” in _2020 IEEE International Conference on Human-Machine Systems (ICHMS)_ , 2020, pp. 1–5.
  * [17] S. M. Merritt, H. Heimbaugh, J. LaChapell, and D. Lee, “I trust it, but i don’t know why: Effects of implicit attitudes toward automation on trust in an automated system,” _Human Factors_ , vol. 55, no. 3, pp. 520–534, 2013, pMID: 23829027. [Online]. Available: https://doi.org/10.1177/0018720812465081
  * [18] B. Muir and N. Moray, “Trust in automation. part ii. experimental studies of trust and human intervention in a process control simulation,” _Ergonomics_ , vol. 39, pp. 429–60, 04 1996.
  * [19] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in _Human Mental Workload_ , ser. Advances in Psychology, P. A. Hancock and N. Meshkati, Eds. North-Holland, 1988, vol. 52, pp. 139–183.


# Assessing Adversarial Robustness of Large Language Models: An Empirical
Study

Zeyu Yang
ETH Zürich
<EMAIL_ADDRESS>
&Zhao Meng
ETH Zürich
<EMAIL_ADDRESS>
&Xiaochen Zheng
ETH Zürich
<EMAIL_ADDRESS>
&Roger Wattenhofer
ETH Zürich
<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) have revolutionized natural language processing,
but their robustness against adversarial attacks remains a critical concern.
We presents a novel white-box style attack approach that exposes
vulnerabilities in leading open-source LLMs, including Llama, OPT, and T5. We
assess the impact of model size, structure, and fine-tuning strategies on
their resistance to adversarial perturbations. Our comprehensive evaluation
across five diverse text classification tasks establishes a new benchmark for
LLM robustness. The findings of this study have far-reaching implications for
the reliable deployment of LLMs in real-world applications and contribute to
the advancement of trustworthy AI systems.

##  1 Introduction

In recent years, the field of artificial intelligence has witnessed a
remarkable surge in the development and application of Large Language Models
(LLMs). These models, such as ChatGPT (OpenAI, 2023a), GPT-4 (OpenAI, 2023b),
and Llama-2 (Touvron et al., 2023), have demonstrated exceptional performance
in various natural language understanding and generation tasks (Zhao et al.,
2023a). The success of LLMs can be attributed to the innovative training
techniques employed, including instruction tuning, prompt tuning, Low-Rank
Adaptor (LoRA) (Hu et al., 2021; Dettmers et al., 2022). These advances have
made it possible to fine-tune and infer models like Llama-2-7B on consumer-
level devices, thereby increasing their accessibility and potential for
integration into daily life.

However, despite their impressive capabilities, LLMs are not without
limitations. One significant challenge is their susceptibility to variations
in input types, which can lead to inconsistencies in output and potentially
undermine their reliability in real-world applications. For example, when
faced with ambiguous or provocative prompts, LLMs may generate inconsistent or
inappropriate responses. To address this issue, several studies have been
conducted to assess the robustness of LLM models (Zhu et al., 2023; Wang et
al., 2023d). However, these efforts often overlook the importance of re-fine-
tuning the models and conducting comprehensive studies of adversarial attacks
with known adversarial sample generation mechanisms when full access to the
model weights, architecture, and training pipeline is available (Guo et al.,
2021; Wallace et al., 2019).

In this paper, we present an extensive study of three leading open-source
LLMs: Llama, OPT, and T5. We evaluate the robustness of various sizes of these
models across five distinct NLP classification datasets. To assess their
vulnerability to input perturbations, we employ the adversarial geometry
attack technique and measure the impact on model accuracy. Furthermore, we
investigate the effectiveness of commonly used methods in LLM training, such
as LoRA, different precision levels, and variations in model architecture and
tuning approaches.

Our work makes several notable contributions to the field of LLM evaluation
and robustness:

  1. 1.

We introduce a novel white-box style attack approach that leverages output
logits and gradients to expose potential vulnerabilities and assess the
robustness of LLMs.

  2. 2.

We establish a benchmark for evaluating the robustness of LLMs by focusing on
their training strategies, setting the stage for future research in this
domain.

  3. 3.

Our comprehensive evaluation spans five text classification tasks, providing a
broad perspective on the capabilities and limitations of the models across
diverse applications.

##  2 Related Work

###  2.1 The Evaluation of LLMs

In recent years, the LLM domain has experienced significant advances
(Bommasani et al., 2021; Wei et al., 2022). A large number of exemplary large-
scale models such a s GPT-4, have emerged, showcasing exceptional performance
across various sectors.

Given the remarkable capabilities and broad applications of these models,
evaluating their performance has become paramount. Consequently, a significant
portion of the research is dedicated to NLP tasks. For instance, (Wang et al.,
2023d) examines ChatGPT’s performance in sentiment analysis, while (Zhang et
al., 2023) offers a comparative analysis with other LLMs. Numerous studies
have explored the capabilities of LLM in natural language understanding,
including text classification as highlighted by (Liang et al., 2022), and
inference as demonstrated by (Qin et al., 2023). Additionally, extensive
research has been conducted to assess LLMs in generation tasks, encompassing
areas such as translation (Wang et al., 2023c), question answering (Bai et
al., 2023; Liang et al., 2022), and summarization (Bang et al., 2023).

Due to the impressive performance of LLMs, there has been widespread attention
to their safety and stability. (Wang et al., 2023b) conducted an early
investigation into ChatGPT and other LLMs and utilized existing benchmarks
like AdvGLUE (Wang et al., 2022), ANLI (Nie et al., 2020), and DDXPlus
(Tchango et al., 2022) for their evaluations. (Zhao et al., 2023b) assessed
the performance of LLMs in visual inputs and their transferability to other
visual-language models. On the topic of adversarial robustness, (Wang et al.,
2023a) introduced the AdvGLUE++ benchmark and proposed an approach to examine
machine ethics through system prompts. (Zhu et al., 2023) proposed a unified
benchmark named PromptBench to evaluate the resilience of LLMs against
prompts.

However, a significant limitation of the aforementioned studies on robustness
is that they focus only on inference-based evaluations. They largely overlook
the intricacies of the model’s weights and output logits. Furthermore, these
studies do not discuss the performance of various LLM techniques. In contrast,
in our research, we not only conduct attacks based on the model’s parameters
and logits, but also actively participate in the model’s tuning. Additionally,
we place a special emphasis on studying the model size and specific techniques
used in LLMs, aspects that previous works have not addressed.

###  2.2 Robustness in NLP

With the rapid advancement in NLP research, its applications have become
increasingly prevalent. This ubiquity underscores the growing need for
reliable NLP systems that can effectively counteract malign content and
misinformation. A seminal work (Jia & Liang, 2017) highlighted the
vulnerabilities of NLP systems to adversarial attacks.

There are some works about various input perturbations, which could be
categorized into three groups: character level, word level, and sentence level
(Wei & Zou, 2019; Karimi et al., 2021; Ma, 2019). At the character level,
adversarial attacks focus on altering individual characters within a given
text. (Ebrahimi et al., 2018) introduced HotFlip, utilizing gradient
information to manipulate characters within text. (Li et al., 2019) took a
different approach by identifying words and modifying their characters. At the
word level, adversarial strategies revolve around replacing specific words
within the content. For instance, (Alzantot et al., 2018) employs evolutionary
algorithms to swap out words with their synonyms. (Zhang et al., 2019)
utilized probabilistic sampling to generate adversarial examples. Furthermore,
some researchers have explored adversarial tactics at the sentence level. (Jia
& Liang, 2017) suggested a method that introduces an extraneous sentence to
the primary content, aiming to mislead reading models. On the other hand,
(Peters et al., 2018) adopted a new approach, where they employed an encoder-
decoder framework to rephrase entire sentences. Recently, (Wang et al., 2023a)
evaluated the robustness and trustworthiness of GPT-3.5 and GPT-4 models,
revealing vulnerabilities such as the ease of generating toxic and biased
outputs and leaking private information. Despite GPT-4’s improved performance
on standard benchmarks, it is more susceptible to adversarial prompts,
highlighting the need for rigorous trustworthiness guarantees and robust
safeguards against new adaptive attacks.

However, the landscape of NLP research is ever-evolving. With the introduction
of more sophisticated models boasting novel architectures and training
methodologies, there is an growing need to assess the robustness of these
newer models. This is especially true for LLMs, which present unique
challenges and opportunities in the realm of robustness research.

##  3 Preliminaries

###  3.1 Open-source Large Language Models

We evaluate the following open-source large language models used in our
experiments, as shown in Table 1.

T5 |  The Text-to-Text Transfer Transformer (T5) developed by Google Research redefines natural language processing (NLP) tasks by treating them uniformly as text-to-text conversions (Raffel et al., 2020).
---|---
OPT |  Open Pretrained Transformers (OPT) range from 125M to 175B parameters and are decoder-only models (Zhang et al., 2022). These models are trained on a diverse pre-training corpus.
Llama |  Meta AI’s Llama is a series of models ranging from 7B to 65B parameters. Llama is trained on a corpus comprising trillions of tokens from publicly available datasets.
Table 1: The open-source large language models in our study

###  3.2 Fine-tuning Techniques

We apply the following fine-tuning techniques in our study.

##### LoRA

LoRA innovates in fine-tuning pretrained language models for specific tasks,
addressing the inefficiency of full fine-tuning in increasingly large models.
By inserting trainable rank decomposition matrices into each layer and
freezing the original model weights, LoRA significantly reduces the number of
parameters requiring training.

##### Quantization

Quantization in large language models (LLMs) reduces the model size by
lowering weight precision, with 8-bit precision presenting challenges due to
errors from quantizing large-value vectors. These errors are pronounced in
transformer architectures, requiring mixed-precision decomposition. This
involves identifying outliers using a threshold, processing them in fp16, and
quantizing the rest of the matrix at 8-bit precision. The two parts are then
combined. The approach, exemplified by LLM.int8(), aims to make large models
more accessible, trading off some performance for significant size reduction.

##### QLoRA

Quantized Low-Rank Adapters (QLoRA) (Dettmers et al., 2023) introduce an
efficient technique to fine-tune large language models by significantly
lowering memory requirements. QLoRA combines 4-bit quantization with Low-Rank
Adapters, freezing the parameters of a compressed pretrained language model.

##  4 Methods

Figure 1: The framework of our adversarial robustness assessment

###  4.1 Adversarial Attack

In this study, our primary concern is text classification. Following the work
by (Meng et al., 2022), we consider a sample sentence
$S_{i}=\\{w_{1},w_{2},\ldots,w_{L}\\}$ containing $L$ words, and its
corresponding category label is $y_{i}$. Our textual classification system is
built upon $n$ LLMs, represented as $f(\cdot)$, coupled with a prompt
indicating the categorization task, denoted as $P_{i}$. In a formal sense:
$\hat{y}_{i}=f(S_{i};P_{i})$, where $\hat{y}_{i}$ stands for the given answer.
The prediction is accurate when $\hat{y}_{i}$ equals $y_{i}$.

An adversarial attack based on word replacement processes the original sample
$S_{i}$ to produce an adversarial version $S_{i}^{adv}$ by replacing the
$k$-th word $w_{k}$ in $S_{i}$ with an alternative word $w_{adv_{k}}$. To
ensure that the original sample $S_{i}$ and its adversarial counterpart
$S_{i}^{adv}$ maintain semantic similarity, prevalent methodologies typically
employ synonymous terms for replacements.

###  4.2 Geometry Attack Methodology

In our research, we extend the basic principles of adversarial attacks in the
context of LLMs. Our focus is on exploiting geometric attacks (Meng &
Wattenhofer, 2020; Meng et al., 2022) to assess the vulnerability of LLMs to
adversarial perturbations. We propose a systematic methodology grounded in
geometric attack insights. The following sections detail the steps of our
approach:

1) Gradient Computation for Influence Analysis: We commence by calculating the
gradients of the generation loss $\mathcal{L}_{i}$ with respect to the
embeddings $e_{i}$ of input sentence $\mathit{S}_{i}$. The cross entropy loss
$\mathcal{L}_{i}$ measures the dissimilarity between the prediction and label
examples in the output space. This computation is essential for all words,
including those segmented into sub-tokens. For such words, gradients are
computed for each sub-token and subsequently averaged. This initial step is
crucial for identifying the words that exert significant influence on
$\mathcal{L}_{i}$. We determine the gradient of $\mathcal{L}_{i}$ with respect
to the embedding vector $e_{i}$. This step determines the direction in which
$e_{i}$ should be adjusted to maximize the increase in the loss
$\mathcal{L}_{i}$. The resulting gradient vector is denoted as
$v_{e_{i}}=\nabla_{e_{i}}\mathcal{L}_{i}$.

2) Selection of Candidate Words: Suppose we select a target word $w_{t}$ from
step 1. Utilizing the DeepFool algorithm (Moosavi-Dezfooli et al., 2016), we
identify potential replacement words, forming a candidate set
$\\{w_{t_{1}},w_{t_{2}},\dots,w_{t_{T}}\\}$. Candidates are filtered based on
their cosine similarity to $w_{t}$, with those below a defined threshold
$\epsilon$ being excluded. This process ensures that only semantically similar
and relevant candidates are considered.

3) Optimal Word Replacement and Projection Analysis: After replacing $w_{t}$
with each candidate word, we compute the new text vectors
$\\{e_{i_{1}},e_{i_{2}},\dots,e_{i_{T}}\\}$. For each vector, we define the
delta vector $r_{i_{j}}$ as $e_{i_{j}}-e_{i}$. The projection of $r_{i_{j}}$
onto $v_{e_{i}}$ is calculated as $p_{i}=r_{i_{j}}\cdot v_{e_{i}}$. The
optimal replacement candidate $w_{t_{m}}$ is selected based on criterion
$m=\arg\max_{j}\frac{|p_{i_{j}}|}{||v_{e_{i}}||}$. This ensures that the
chosen word $w_{t_{m}}$ induces the largest possible projection $p_{i_{m}}$
onto the gradient vector $v_{e_{i}}$.

4) Iterative Process for Enhanced Adversarial Strength: The selected word
$w_{t_{m}}$ replaces $w_{t}$ in $S_{i}$, updating $e_{i}$ to $e_{i_{m}}$. This
iterative procedure is repeated for $N$ cycles, where $N$ is an adjustable
parameter in our methodology. Throughout these iterations, an increase in
$\mathcal{L}_{i}$ should be observed, indicating a continuous enhancement in
the adversarial effectiveness of the altered input.

Through this methodically structured process, our research aims to uncover and
analyze potential vulnerabilities in LLMs. We refined our methodology to
enable prompt fine-tuning for attack generation tasks, expanding its
application beyond the previously limited scope of classification tasks.

##  5 Experiment Settings

###  5.1 Experiment Pipeline

This section introduces our methodology for evaluating the robustness of pre-
trained LLMs against adversarial attacks. The procedure comprises three
principal stages:

1) Model Fine-Tuning: We fine-tune a pre-trained language model on target
dataset with different fine-tuning techniques as described in Sec. 3.2,
evaluating its accuracy on the corresponding validation set to establish a
performance baseline.

2) Adversarial Attack Assessment: The fine-tuned model undergoes adversarial
attacks described in Sec. 4.2, and its performance is assessed on a test
dataset altered with adversarial examples.

3) Robustness Evaluation: We compare the model’s accuracy before and after the
adversarial attacks to assess its robustness and vulnerability to such
manipulations.

###  5.2 Datasets

To evaluate the model’s performance under various tasks and its resilience to
attacks, we employed five classification datasets, categorized into binary and
multi-class classifications. For binary classification, the datasets include
IMDB (Maas et al., 2011), MRPC (Dolan & Brockett, 2005), and SST-2 (Socher et
al., 2013), and for multiclass classification, AGNews (Zhang et al., 2015) and
DBpedia (Auer et al., 2007) are used. We will provide a more detailed
introduction to these tasks/datasets in Appendix A.2

###  5.3 Evaluation Metrics

We assess our model’s robustness and efficacy using four principal metrics, as
described in Table 2.

Metrics |  Description
---|---
Acc |  Accuracy: the model’s correct classification rate of untouched input
Acc/attack |  Accuracy Under Attack: post-attack classification accuracy revealing adversarial defense
ASR |  Attack Success Rate: the frequency of accurate predictions turned false by attacks
Replacement |  Replacement Rate: the extent of input alteration needed to change the model’s prediction, indicating sensitivity to perturbations
Table 2: The evaluation metrics

##  6 Experimental Results

In this section, we conduct extensive experiments to evaluate the robustness
of LLMs across five different datasets. These investigations are guided by
three key research questions (RQ):

RQ1: How does the robustness of variously sized models differ under
adversarial attacks across distinct tasks?

RQ2: Do contemporary training techniques for LLMs influence their performance
and robustness?

RQ3: How does the model architecture (e.g., fine-tuning with a classification
head vs. prompt tuning), affect the robustness of the model?

###  6.1 Model Size (RQ1)

In this section, we analyze the performance metrics of various models across
multiple tasks. The datasets under examination include IMDB, SST-2, MRPC,
AGNews, and DBPedia. We measure the performance and robustness of LLMs with
the metrics Acc, Acc/attack, ASR, and Replacement Rate described in Table 2.

The results from the IMDB dataset in Table 3 reveal distinct performance
variations among different model architectures. In the T5 Series, accuracy
generally improves with increasing model size, from 60m to 11b parameters, but
the relationship is nonlinear. This suggests that while larger models tend to
be more accurate, the accuracy does not increase uniformly with model size.
Furthermore, the resilience of these models to adversarial attacks does not
follow a simple inverse relationship with model size. The larger T5-11b model
shows a more noticeable decrease in accuracy under attack conditions.

For the OPT models, a similar upward trend in accuracy is observed with
increasing model size, but the Attack Success Rate (ASR) is lower, suggesting
better resistance to attacks. In comparison, the Llama models demonstrate
superior performance in both accuracy and robustness against attacks.

| IMDB | SST-2 | MRPC
---|---|---|---
| Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement | Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement | Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement
T5-60m | 0.8484 | 0.1256 | 0.8491 | 0.0929 | 0.9083 | 0.2419 | 0.7304 | 0.1428 | 0.8048 | 0.0325 | 0.9594 | 0.0720
T5-220m | 0.8011 | 0.0436 | 0.9463 | 0.0722 | 0.8884 | 0.1228 | 0.8622 | 0.1611 | 0.8035 | 0.0633 | 0.9203 | 0.1100
T5-770m | 0.9048 | 0.1536 | 0.8312 | 0.1143 | 0.8739 | 0.0534 | 0.9395 | 0.1785 | 0.8924 | 0.1992 | 0.7771 | 0.1203
T5-3b | 0.9146 | 0.3259 | 0.6436 | 0.1413 | 0.9495 | 0.1563 | 0.8437 | 0.1987 | 0.8584 | 0.1074 | 0.8739 | 0.0917
T5-11b | 0.9122 | 0.3098 | 0.6604 | 0.1330 | 0.9656 | 0.2248 | 0.7672 | 0.2043 | 0.8877 | 0.0712 | 0.9196 | 0.0873
OPT-125m | 0.8616 | 0.6637 | 0.2297 | 0.0365 | 0.8807 | 0.7409 | 0.1587 | 0.0607 | 0.8321 | 0.6504 | 0.2184 | 0.0332
OPT-350m | 0.8564 | 0.6924 | 0.1915 | 0.0305 | 0.9011 | 0.7716 | 0.1437 | 0.0598 | 0.8956 | 0.6741 | 0.2473 | 0.0437
OPT-1.3b | 0.9231 | 0.7515 | 0.1859 | 0.0421 | 0.9443 | 0.8227 | 0.1288 | 0.0733 | 0.9134 | 0.7721 | 0.1547 | 0.0419
OPT–2.7b | 0.9198 | 0.7651 | 0.1682 | 0.0396 | 0.8897 | 0.7921 | 0.1097 | 0.0476 | 0.9128 | 0.7854 | 0.1396 | 0.0533
OPT-6.7b | 0.9408 | 0.7864 | 0.1641 | 0.0528 | 0.9693 | 0.8682 | 0.1043 | 0.0550 | 0.9096 | 0.7902 | 0.1313 | 0.0579
OPT-13b | 0.9431 | 0.8016 | 0.1500 | 0.0671 | 0.9656 | 0.7867 | 0.1853 | 0.0719 | 0.9254 | 0.8183 | 0.1157 | 0.0560
Llama-7b | 0.9483 | 0.8203 | 0.1350 | 0.0816 | 0.9683 | 0.8203 | 0.1528 | 0.0916 | 0.9277 | 0.8256 | 0.1101 | 0.0637
Llama-13b | 0.9472 | 0.8237 | 0.1304 | 0.0875 | 0.9632 | 0.8124 | 0.1566 | 0.0877 | 0.9198 | 0.8107 | 0.1186 | 0.0742

Table 3: IMDB, SST-2 and MRPC Results

From Table 3 on the SST-2 dataset, there are distinct performance trends. The
T5-11b, achieves the highest accuracy of 0.9656. However, its persistence to
adversarial attacks is not highest. Notably, the highest ASR within the T5
series is recorded for the T5-770m model, indicating a trade-off as model size
increases. In the case of the OPT series, the OPT-6.7b model stands out.
However, similar to the IMDB dataset, this model also shows a significant
decline in accuracy but more robust than T5 models. One more observation in
the OPT series is the overall decrease in ASR with increasing model size, but
this trend is disrupted at the 13b parameter mark, where an anomalous increase
in ASR is observed. The Llama models, demonstrate consistently high accuracy.
It also presents lower ASR compared to T5 models but similar performance to
OPT models. For SST-2, the ASR of T5 models exhibit a trend entirely contrary
to that observed for MRPC. It reaches its minimum at the T5-770m model. For
the OPT models, although their ASR is much lower compared to the T5 series,
there is a consistent decrease in ASR as the size of the OPT models increases.
Regarding the Llama models, the 7b model slightly outperforms the 13b in terms
of accuracy and ASR.

| AGNews | DBPedia
---|---|---
| Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement | Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement
T5-60m | 0.8606 | 0.3608 | 0.5807 | 0.1740 | 0.9817 | 0.3974 | 0.5952 | 0.1187
T5-220m | 0.9084 | 0.5370 | 0.4098 | 0.1864 | 0.9765 | 0.4082 | 0.5819 | 0.1234
T5-770m | 0.9278 | 0.6597 | 0.2896 | 0.1860 | 0.9921 | 0.7476 | 0.2464 | 0.1483
T5-3b | 0.9193 | 0.7267 | 0.2102 | 0.1834 | 0.9914 | 0.8608 | 0.1317 | 0.1550
T5-11b | 0.9212 | 0.8469 | 0.1531 | 0.1867 | 0.9919 | 0.8815 | 0.1113 | 0.1724
OPT-125m | 0.8152 | 0.6040 | 0.2591 | 0.0809 | 0.9034 | 0.6512 | 0.2792 | 0.0534
OPT-350m | 0.8321 | 0.6036 | 0.2746 | 0.0864 | 0.9511 | 0.6718 | 0.2937 | 0.0305
OPT-1.3b | 0.8806 | 0.6316 | 0.2828 | 0.0912 | 0.9784 | 0.7046 | 0.2798 | 0.0496
OPT–2.7b | 0.9175 | 0.7028 | 0.2340 | 0.0833 | 0.9822 | 0.7513 | 0.2351 | 0.0552
OPT-6.7b | 0.9341 | 0.7143 | 0.2353 | 0.0756 | 0.9907 | 0.7780 | 0.2147 | 0.0641
OPT-13b | 0.9456 | 0.7745 | 0.1809 | 0.0941 | 0.9916 | 0.7912 | 0.2021 | 0.0576
Llama-7b | 0.9328 | 0.7315 | 0.2158 | 0.0864 | 0.9908 | 0.7596 | 0.2333 | 0.0881
Llama-13b | 0.9338 | 0.7688 | 0.1767 | 0.0837 | 0.9921 | 0.7286 | 0.2656 | 0.0912

Table 4: AGNews and DBPedia Results

In Tables 4, analyzing results from multi-class classification tasks, a
distinct pattern emerges. These datasets reveal enhanced stability against
synonym substitution attacks. For T5 models, the data shows a lower ASR on
these tasks compared to binary datasets, suggesting a better resistance to
attacks in complex classification scenarios. In contrast, OPT and Llama models
exhibit a higher ASR on the AGNews and DBpedia14 datasets. Another result is
that for both T5 and OPT series, there is a marked decline in ASR around the
770 million or 1 billion parameter threshold. This indicates an increased
robustness and better handling of adversarial attacks with the scale-up of
model size.

####  6.1.1 Analysis

When examining the accuracy of the model, we observed a trend where the
accuracy gradually increases with the growth in model size. However, after
reaching a certain size threshold, the accuracy tends to saturate, stabilizing
around specific values. This phenomenon is particularly pronounced when tested
on datasets like DBpedia. When comparing different models operating at the
same parameter scale, their performances were found to be quite similar,
without any significant disparities.

However, the experiments related to robustness revealed more distinct
differences. From Figure 2(a), Figure 2(b) and Figure 2(c), we have more
intuitive results. Observing the performance of a uniform model across various
datasets, we made the following observations:

T5 Model: As the size of the T5 model increases, its ASR gradually decreases.
This suggests that larger models, with more parameters, tend to have a deeper
understanding of language. As a result, they can maintain stronger stability
in the face of various disturbances. However, on datasets like MRPC and SST-2,
there were noticeable fluctuations in performance. One possible explanation
for this is that as the model size grows, the words selected based on the
model’s gradient become more precise and have a more significant impact on the
results. This introduces a trade-off related to model size.

OPT Model: For the OPT model, a similar trend was observed across most
datasets. As the model size increased, its robustness generally improved,
aligning with the observations made for the T5 model.

Llama Model: For the Llama model, the differences in performance between the
two sizes were minimal. This suggests that the size variation did not
significantly influence the model’s robustness.

However, when comparing different models, the disparities become even more
pronounced. It is obvious that the T5 model’s ASR and replacement rate are
significantly higher than those of OPT and Llama. This indicated that Decoder-
only Causal LMs have higher robustness against encoder-decoder architectures
under synonym substitution adversarial attacks.

(a) T5 model

(b) OPT model

(c) Llama model

Figure 2: The experimental results of different models on various datasets.

###  6.2 LLMs Fine-tuning Techniques (RQ2)

####  6.2.1 Instruction Tuning

To study the impact of instruction tuning on model robustness, we compared the
performance of Flan-T5 with the standard T5. The Flan-T5 is an advanced
variant of T5 that has undergone instruction tuning across over a thousand
downstream tasks. In contrast, the traditional T5 was not trained with such an
extensive procedure.

Based on our experimental results, as shown in the table, there is a
significant decline in accuracy for both T5 and Flan-T5 under adversarial
attacks. This observation indicates that models, irrespective of whether they
have undergone instruction tuning, remain susceptible to adversarial
manipulations. Furthermore, consistent with our previous findings, we noticed
that as the model size increases, the attack success rate tends to decline.

Interestingly, as shown in Fig 3, our results indicated that Flan-T5 exhibits
a higher ASR than the standard T5. This suggests that models subjected to
instruction tuning, like Flan-T5, can be more easily compromised. We
hypothesize the primary reason for this observation:

The instruction tuning process for Flan-T5 encompassed datasets similar to
IMDB. This might have rendered the model with a deeper understanding of tasks
related to this data. As a result, attackers could more easily pinpoint words
in the input that were influential and susceptible to replacement.

| Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement
---|---|---|---|---
T5-60m | 0.8484 | 0.1256 | 0.8491 | 0.0929
Flan-T5-60m | 0.8453 | 0.0882 | 0.8968 | 0.0820
T5-220m | 0.8011 | 0.0436 | 0.9463 | 0.0722
Flan-T5-220m | 0.8777 | 0.0996 | 0.8862 | 0.0978
T5-770m | 0.9048 | 0.1536 | 0.8312 | 0.1143
Flan-T5-770m | 0.9141 | 0.1171 | 0.8729 | 0.1106
T5-3b | 0.9146 | 0.3259 | 0.6436 | 0.1413
Flan-T5-3b | 0.9228 | 0.2328 | 0.7489 | 0.1261
T5-11b | 0.9348 | 0.4904 | 0.4752 | 0.1326
Flan-T5-11b | 0.9122 | 0.3098 | 0.6604 | 0.1330

Table 5: Experimental results comparing T5 and Flan-T5 after instruction
tuning using the IMDB dataset. Figure 3: The experimental results of T5 and
Flan-T5 on IMDB dataset

(a) Different precisions on the IMDB dataset (T5 model)

(b) Different precisions on the IMDB dataset (OPT Model)

(c) Results of LoRA on IMDB dataset

Figure 4: Precision and attack success rate

####  6.2.2 Precisions

In machine learning, balancing model size with precision is crucial. Model
size indicates capacity, and precision affects information granularity. Larger
models typically perform better but require more computational resources.
Techniques like quantization and precision adjustments help deploy these
models more efficiently. We studied the impact of precision settings on the
robustness of T5-770m and OPT-1.3b models by comparing their performance under
various precisions.

For the T5-770m and OPT-1.3b models, it’s clear that as precision changes from
fp16 to int4, there isn’t a significant drop in their inherent accuracy. This
indicates that models can handle reduced precision without compromising their
general performance drastically. What’s more, across different precision
settings, the attack success rate for the T5-770m models remains fairly
higher, which shows the same conclusion as in 6.1. However, the precision
settings do not show a consistent pattern of influence on the ASR and
replacement rate.

In essence, while different models exhibit different robustness against
adversarial attacks, the precision settings do not play a significant role in
this robustness.

| Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement
---|---|---|---|---
T5-770m-fp16 | 0.9106 | 0.1631 | 0.8208 | 0.1196
T5-770m-int8 | 0.9048 | 0.1536 | 0.8312 | 0.1143
T5-770m-int4 | 0.9210 | 0.1725 | 0.8127 | 0.1211
OPT-1.3b-fp16 | 0.9218 | 0.7496 | 0.1868 | 0.0536
OPT-1.3b-int8 | 0.9231 | 0.7515 | 0.1859 | 0.0421
OPT-1.3b-int4 | 0.9207 | 0.7531 | 0.1820 | 0.0498

Table 6: Results of T5-770m and OPT-1.3b models under different precision
settings, including fp16, int8 and int4. The performance is evaluated with
IMDB dataset.

| Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement
---|---|---|---|---
T5-770m | 0.9067 | 0.1499 | 0.8347 | 0.1036
T5-770m-Lora | 0.9048 | 0.1536 | 0.8312 | 0.1143
OPT-1.3b | 0.9135 | 0.7448 | 0.1847 | 0.0366
OPT-1.3b-LoRA | 0.9231 | 0.7515 | 0.1859 | 0.0421
OPT-2.7b | 0.9266 | 0.7741 | 0.1646 | 0.0452
OPT-2.7b-LoRA | 0.9198 | 0.7651 | 0.1682 | 0.0396

Table 7: Results of the T5-770m, OPT-1.3b, and OPT-2.7b models’ performance
with and without the application of LoRA, using the IMDB dataset.

####  6.2.3 LoRA

As mention in Sec 3.2, LoRA has been a groundbreaking approach, bringing about
significant reductions in memory requirements during model training. In this
case, the potential trade-off in question is model robustness.

For our investigation, we selected the T5-770m, OPT-1.3b, and OPT-2.7b models.
Experiments were conducted under two conditions for each model: with and
without the application of LoRA. The IMDB dataset served as our benchmark for
this analysis.

The experiments show that adversarial attacks significantly reduce accuracy
across all models, regardless of LoRA’s use. However, crucially, both the
attack success rate and replacement rate, key measures of resilience against
adversarial tactics, were unaffected by LoRA. This indicates that while LoRA
enhances optimization, it doesn’t negatively impact the model’s defense
against adversarial attacks, providing optimization benefits without
sacrificing robustness.

###  6.3 Model Architectures (RQ3)

The architecture of a model’s output space significantly influences its
performance and resilience against adversarial attacks. For models with a
classification head, the output is simplified to a binary decision,
contrasting with OPT models without such a head, which must identify
’negative’ or ’positive’ labels from a vast vocabulary. This distinction
impacts the model’s accuracy.

Our data shows that smaller models with a classification head are more
accurate than headless ones due to their simplified output space, which aids
decision-making, especially in models with limited processing power. Moreover,
models with a head reach their peak performance faster, achieving accuracy
saturation more quickly.

However, an intriguing observation is the heightened attack success rate for
models with a classification head. On the surface, this suggests that
launching adversarial attacks against these models is a more straightforward
task.One main factor contributes to this vulnerability is DeepFool’s efficacy
with last layer FFN: In such models, DeepFool can more readily discern the
optimal direction for launching its attack, amplifying the ASR. This marked
efficiency underscores a reduced robustness in these models against
adversarial intrusions.

| Acc | Acc/attack $\uparrow$ | ASR $\downarrow$ | Replacement
---|---|---|---|---
OPT-125m | 0.8616 | 0.6637 | 0.2297 | 0.0365
OPT-125m-head | 0.9074 | 0.6215 | 0.3151 | 0.0476
OPT-350m | 0.8564 | 0.6924 | 0.1915 | 0.0305
OPT-350m-head | 0.9152 | 0.6643 | 0.2741 | 0.0682
OPT-1.3b | 0.9231 | 0.7515 | 0.1859 | 0.0421
OPT-1.3b-head | 0.9316 | 0.7621 | 0.1819 | 0.0533
OPT–2.7b | 0.9198 | 0.7651 | 0.1682 | 0.0396
OPT–2.7b-head | 0.9367 | 0.7704 | 0.1775 | 0.0516
OPT-6.7b | 0.9408 | 0.7864 | 0.1641 | 0.0528
OPT-6.7b-head | 0.9422 | 0.7765 | 0.1759 | 0.0627
OPT-13b | 0.9431 | 0.8016 | 0.1500 | 0.0671
OPT-13b-head | 0.9427 | 0.7877 | 0.1644 | 0.0641

Table 8: Experimental results of OPT models with/without classification head
with IMDB dataset. Figure 5: Results of classfication head on IMDB dataset

##  7 Conclusion

This paper utilized a novel geometric adversarial attack method to assess the
robustness of leading LLMs, utilizing advanced fine-tuning techniques for
task-specific model adaptation. Our groundbreaking approach revealed that
these models exhibit variable sensitivity to adversarial attacks, influenced
by their size and architectural differences. This indicates inherent
vulnerabilities in LLMs, yet suggests potential resilience in certain
configurations. Contrary to expectations, LLM-specific techniques did not
markedly reduce robustness. Future research could explore models like RLHF and
model parallelism approaches within this framework. Additionally, the
evolution of more complex adversarial attacks promises deeper insights into
LLM strengths and weaknesses.

## Ethics statement

In our research, we employ adversarial attack methodologies to generate text,
aiming to evaluate the robustness of LLMs against inputs. However, we
acknowledge the ethical implications associated with the use of adversarial
attacks. One primary concern is the potential generation of harmful
information. This includes text that may be offensive, misleading, or harmful
in other ways. Therefore, one should be cautious when taking such methods into
practical use.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang.  Generating natural language adversarial examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 2890–2896, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1316.  URL https://aclanthology.org/D18-1316.
  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives.  Dbpedia: A nucleus for a web of open data.  In _international semantic web conference_ , pp. 722–735. Springer, 2007.
  * Bai et al. (2023) Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou.  Benchmarking foundation models with language-model-as-an-examiner, 2023.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms, 2023.
  * Dolan & Brockett (2005) Bill Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_ , 2005.
  * Ebrahimi et al. (2018) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou.  Hotflip: White-box adversarial examples for text classification, 2018.
  * Guo et al. (2021) Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela.  Gradient-based adversarial attacks against text transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 5747–5757. Association for Computational Linguistics, November 2021.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Jia & Liang (2017) Robin Jia and Percy Liang.  Adversarial examples for evaluating reading comprehension systems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pp. 2021–2031, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1215.  URL https://aclanthology.org/D17-1215.
  * Karimi et al. (2021) Akbar Karimi, Leonardo Rossi, and Andrea Prati.  AEDA: An easier data augmentation technique for text classification.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pp. 2748–2754, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-emnlp.234.  URL https://aclanthology.org/2021.findings-emnlp.234.
  * Li et al. (2019) Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang.  TextBugger: Generating adversarial text against real-world applications.  In _Proceedings 2019 Network and Distributed System Security Symposium_. Internet Society, 2019.  doi: 10.14722/ndss.2019.23138.  URL https://doi.org/10.14722%2Fndss.2019.23138.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.  Holistic evaluation of language models, 2022.
  * Ma (2019) Edward Ma.  Nlp augmentation.  https://github.com/makcedward/nlpaug, 2019.
  * Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pp. 142–150, 2011.
  * Meng & Wattenhofer (2020) Zhao Meng and Roger Wattenhofer.  A geometry-inspired attack for generating natural language adversarial examples.  _arXiv preprint arXiv:2010.01345_ , 2020.
  * Meng et al. (2022) Zhao Meng, Yihan Dong, Mrinmaya Sachan, and Roger Wattenhofer.  Self-supervised contrastive learning with adversarial perturbations for defending word substitution-based attacks.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pp. 87–101, 2022.
  * Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks, 2016.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4885–4901, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.441.  URL https://aclanthology.org/2020.acl-main.441.
  * OpenAI (2023a) OpenAI.  Openai chatbot, 2023a.  URL https://chat.openai.com.
  * OpenAI (2023b) OpenAI.  Gpt-4 technical report.  2023b.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pp. 2227–2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1202.  URL https://aclanthology.org/N18-1202.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang.  Is chatgpt a general-purpose natural language processing task solver?, 2023.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pp. 1631–1642, 2013.
  * Tchango et al. (2022) Arsene Fansi Tchango, Rishab Goel, Zhi Wen, Julien Martel, and Joumana Ghosn.  Ddxplus: A new dataset for automatic medical diagnosis, 2022.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2153–2162. Association for Computational Linguistics, November 2019.
  * Wang et al. (2022) Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li.  Adversarial glue: A multi-task benchmark for robustness evaluation of language models, 2022.
  * Wang et al. (2023a) Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li.  Decodingtrust: A comprehensive assessment of trustworthiness in gpt models, 2023a.
  * Wang et al. (2023b) Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie.  On the robustness of chatgpt: An adversarial and out-of-distribution perspective, 2023b.
  * Wang et al. (2023c) Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu.  Document-level machine translation with large language models, 2023c.
  * Wang et al. (2023d) Zengzhi Wang, Qiming Xie, Zixiang Ding, Yi Feng, and Rui Xia.  Is chatgpt a good sentiment analyzer? a preliminary study, 2023d.
  * Wei & Zou (2019) Jason Wei and Kai Zou.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 6382–6388, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1670.  URL https://aclanthology.org/D19-1670.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent abilities of large language models, 2022.
  * Zhang et al. (2019) Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li.  Generating fluent adversarial examples for natural languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 5564–5569, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1559.  URL https://aclanthology.org/P19-1559.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.
  * Zhang et al. (2023) Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing.  Sentiment analysis in the era of large language models: A reality check, 2023.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level convolutional networks for text classification.  _Advances in neural information processing systems_ , 28, 2015.
  * Zhao et al. (2023a) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models, 2023a.
  * Zhao et al. (2023b) Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin.  On evaluating adversarial robustness of large vision-language models, 2023b.
  * Zhu et al. (2023) Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, and Xing Xie.  Promptbench: Towards evaluating the robustness of large language models on adversarial prompts, 2023.

##  Appendix A Appendix

###  A.1 Experiments Setup

In our study, we employ pretrained weights from HuggingFace and use int8
quantization for GPU memory optimization. We also standardize the use of LoRA
to reduce training parameters. For models under 3 billion parameters,
experiments are conducted on an NVIDIA RTX 3090 (24GB), whereas models above 3
billion parameters are tested on NVIDIA RTX A6000 (48GB) or NVIDIA A100
(40GB), catering to both fine-tuning and attack simulations.

###  A.2 Dataset

In this section, we will provide more details about the datasets used in this
work.

IMDB: This dataset contains 50,000 movie reviews for sentiment analysis,
equally divided between positive and negative sentiments.

SST-2: An extension of the original SST, it focuses on the binary
classification of sentiments in movie review sentences.

MRPC: A corpus for paraphrase identification, it includes sentence pairs from
online news sources, annotated for semantic equivalence.

AGNews: This news categorization dataset comprises 120,000 training and 7,600
test samples across four categories: World, Sports, Business, and
Science/Technology.

DBpedia: A large-scale, multi-class dataset from the DBpedia knowledge base,
featuring 560,000 training and 70,000 test samples across 14 categories.

The statistics of these five datasets are presented in Table 9.

Table 9: Statistics of the Datasets Dataset | Labels | Avg Length | Train | Test
---|---|---|---|---
IMDB | 2 | 279.48 | 25000 | 25000
MRPC | 2 | 52.89 | 3670 | 1730
SST-2 | 2 | 19.81 | 67300 | 1820
AGNews | 4 | 43.93 | 120000 | 7600
DBPedia | 14 | 55.14 | 560000 | 7000

###  A.3 Attack Examples

In this section, we are going to show some adversarial examples for different
tasks and models.

Table 10: Adversarial Examples for Various Tasks and Models Task Description
and Adversarial Example
---
Dataset: IMDB
Model: T5-780M
Original: ”Choose the sentiment of this review? Expensive lunch meals. Fried
pickles were good. Waitress messed up 2 orders out of 4. ”Dont” think ”Ill”
return. Asked for no cheese waitress joked extra cheese then brought my meal
with cheese. Better places to eat in area. ”
Adversarial: ”Choose the sentiment of this review? Expensive brunch meals.
Fried marinated were good. Waitress eventhough up 2 orders out of 4. ”Dont”
imagining ”Ill” return. Asked for no cheese miss jokingly extra cheese then
brought my meal with cheese. Better place to devoured in area.”
Dataset: MRPC
Model: OPT-2.7B
Original: ”Do these two sentences mean the same thing? Crews worked to install
a new culvert and prepare the highway so motorists could use the eastbound
lanes for travel as storm clouds threatened to dump more rain. Crews worked to
install a new culvert and repave the highway so motorists could use the
eastbound lanes for travel.”
Adversarial: ”Do these two sentences mean the same thing? Crews acted to mount
a nouvelle drains and prepare the avenue so chauffeurs could use the eastbound
routing for voyage as stormy haze threatens to spill more rainfall. Crews
worked to install a newer septic and repave the highway so motorists could use
the eastbound lanes for tours”
Dataset: AGNews
Model: Llama-7B
Original: ”Dial M for Music Mobile-phone makers scored a surprising hit four
years ago when they introduced handsets equipped with tiny digital cameras.
Today, nearly one-third of the cell phones sold worldwide do double duty as
cameras.”
Adversarial: ”Dial M for Melody Mobile-phone manufacturers scored a surprising
hit four years ago when they unveiled handsets equipped with tiny digital
cameras. Today, nearly one-third of the cell phones sold worldwide do double
duty as cameras.”


* [6] H. Comon, M. Dauchet, R. Gilleron, F. Jacquemard, D. Lugiez, S. Tison, and M. Tommasi.  Tree automata techniques and applications.  Available on: http://www.grappa.univ-lille3.fr/tata, 1997.
  * [7] H. Comon, M. Dauchet, R. Gilleron, C. Löding, F. Jacquemard, D. Lugiez, S. Tison, and M. Tommasi.  Tree automata techniques and applications.  Available on: http://www.grappa.univ-lille3.fr/tata, 2007.  release October, 12th 2007.
  * [8] B. Cook, C. Haase, J. Ouaknine, M. J. Parkinson, and J. Worrell.  Tractable reasoning in a fragment of separation logic.  In Proc. of CONCUR’11, volume 6901 of LNCS. Springer, 2011\.
  * [9] Mnacho Echenim, Radu Iosif, and Nicolas Peltier.  Entailment checking in separation logic with inductive definitions is 2-exptime hard.  In Elvira Albert and Laura Kovács, editors, LPAR 2020: 23rd International Conference on Logic for Programming, Artificial Intelligence and Reasoning, Alicante, Spain, May 22-27, 2020, volume 73 of EPiC Series in Computing, pages 191–211. EasyChair, 2020.
  * [10] Mnacho Echenim, Radu Iosif, and Nicolas Peltier.  Decidable entailments in separation logic with inductive definitions: Beyond establishment.  In CSL 2021: 29th International Conference on Computer Science Logic, EPiC Series in Computing. EasyChair, 2021.
  * [11] Mnacho Echenim and Nicolas Peltier.  A proof procedure for separation logic with inductive definitions and theory reasoning.  Technical report, 2022.  URL: https://arxiv.org/abs/2201.13227, arXiv:2201.13227.
  * [12] Constantin Enea, Ondrej Lengál, Mihaela Sighireanu, and Tomás Vojnar.  Compositional entailment checking for a fragment of separation logic.  Formal Methods Syst. Des., 51(3):575–607, 2017.
  * [13] Constantin Enea, Mihaela Sighireanu, and Zhilin Wu.  On automated lemma generation for separation logic with inductive definitions.  In ATVA 2015, Proceedings, pages 80–96, 2015.
  * [14] Radu Iosif, Adam Rogalewicz, and Jiri Simacek.  The tree width of separation logic with recursive definitions.  In Proc. of CADE-24, volume 7898 of LNCS, 2013.
  * [15] Radu Iosif, Adam Rogalewicz, and Tomás Vojnar.  Deciding entailments in inductive separation logic with tree automata.  In Franck Cassez and Jean-François Raskin, editors, ATVA 2014, Proceedings, volume 8837 of LNCS, pages 201–218. Springer, 2014.
  * [16] Samin S Ishtiaq and Peter W O’Hearn.  Bi as an assertion language for mutable data structures.  In ACM SIGPLAN Notices, volume 36, pages 14–26, 2001.
  * [17] Ralf Jung, Robbert Krebbers, Jacques-Henri Jourdan, Ales Bizjak, Lars Birkedal, and Derek Dreyer.  Iris from the ground up: A modular foundation for higher-order concurrent separation logic.  J. Funct. Program., 28:e20, 2018.
  * [18] Jens Katelaan and Florian Zuleger.  Beyond symbolic heaps: Deciding separation logic with inductive definitions.  In Elvira Albert and Laura Kovács, editors, LPAR 2020: 23rd International Conference on Logic for Programming, Artificial Intelligence and Reasoning, Alicante, Spain, May 22-27, 2020, volume 73 of EPiC Series in Computing, pages 390–408. EasyChair, 2020.
  * [19] Quang Loc Le and Xuan-Bach D. Le.  An efficient cyclic entailment procedure in a fragment of separation logic.  In Orna Kupferman and Pawel Sobocinski, editors, Foundations of Software Science and Computation Structures - 26th International Conference, FoSSaCS 2023, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2023, Paris, France, April 22-27, 2023, Proceedings, volume 13992 of Lecture Notes in Computer Science, pages 477–497. Springer, 2023.  doi:10.1007/978-3-031-30829-1\\_23.
  * [20] Christoph Matheja, Jens Pagel, and Florian Zuleger.  A decision procedure for guarded separation logic complete entailment checking for separation logic with inductive definitions.  ACM Trans. Comput. Log., 24(1):1:1–1:76, 2023.  doi:10.1145/3534927.
  * [21] Jens Pagel, Christoph Matheja, and Florian Zuleger.  Complete entailment checking for separation logic with inductive definitions.  CoRR, abs/2002.01202, 2020.  URL: https://arxiv.org/abs/2002.01202, arXiv:2002.01202.
  * [22] J.C. Reynolds.  Separation Logic: A Logic for Shared Mutable Data Structures.  In Proc. of LICS’02, 2002.
  * [23] The Coq Development Team.  The Coq Proof Assistant Reference Manual V7.1.  INRIA-Rocquencourt, CNRS-ENS Lyon (France), October 2001.  http://coq.inria.fr/doc/main.html.