# Large Language Models Are Involuntary Truth-Tellers: Exploiting Fallacy
Failure for Jailbreak Attacks
WARNING: This paper contains model outputs that may be considered offensive in
nature.

Yue Zhou1 Henry Peng Zou1 Barbara Di Eugenio1 Yang Zhang2
1University of Illinois Chicago 2MIT-IBM Watson AI Lab  IBM Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

We find that language models have difficulties generating fallacious and
deceptive reasoning. When asked to generate deceptive outputs, language models
tend to leak honest counterparts but believe them to be false. Exploiting this
deficiency, we propose a jailbreak attack method that elicits an aligned
language model for malicious output. Specifically, we query the model to
generate a fallacious yet deceptively real procedure for the harmful behavior.
Since a fallacious procedure is generally considered fake and thus harmless by
LLMs, it helps bypass the safeguard mechanism. Yet the output is factually
harmful since the LLM cannot fabricate fallacious solutions but proposes
truthful ones. We evaluate our approach over five safety-aligned large
language models, comparing four previous jailbreak methods, and show that our
approach achieves competitive performance with more harmful outputs. We
believe the findings could be extended beyond model safety, such as self-
verification and hallucination.

##  1 Introduction

It is arguably easier, at least from the logical perspective, to tell the
truth than to tell a lie. For example, given a math problem _“What is 1/2 +
1/3”_ , telling the truth only requires the ability to perform the correct
reasoning and derive the correct answer. Telling a lie, on the other hand,
requires the ability to not only discern the correct answers, but also avoid
generating the correct answers and, more importantly, make the wrong answers
look real. In this paper, we refer to the task of fabricating incorrect yet
seemingly plausible reasoning as _fallacious reasoning_.

Large language models (LLMs) have long been struggling with reasoning
problems. Existing research revealed that LLMs have difficulty discerning the
veracity of their intrinsic answers Huang et al. (2024); Kadavath et al.
(2022); Stechly et al. (2023). This raises an intriguing research question: If
LLMs already find it hard to validate their own correctness, can LLMs
deliberately generate fallacious reasoning upon request?

Figure 1: A prompt containing malicious behavior can be rejected by a human-
value aligned language model. However, when asked to generate a fallacious
procedure for the malicious behavior, an LLM can leak the honest answer, yet
believe it false.

This paper starts with a pilot investigation of LLMs’ ability to perform
fallacious reasoning. Specifically, on four reasoning benchmarks, covering the
domains of mathematics, logic, and commonsense, we ask the LLM to generate the
correct answers and to deliberately generate wrong answers that are
deceptively real. Our surprising finding is that, for all of our tested
benchmarks, the accuracy of the generated wrong answers is almost as high as
that of the correct ones. For instance, in Figure 3, the LLM generates the
correct reasoning chain and final answer despite the fallacious generation
request and claims a step to be wrong with a contradictory statement. This
pilot study reveals that LLMs might be unable to intentionally generate
deceptive reasoning and instead often leak the correct solutions in what they
claim to be wrong answers.

As we further our investigation, this seemingly small glitch in LLMs can lead
to a significant security threat. Specifically, we discovered a simple yet
effective jailbreak attack, which we call the fallacy failure attack (FFA),
that can elicit harmful output from LLMs by exploiting the LLMs’ deficiency in
fallacious reasoning. Given a malicious query, _e.g._ “How to create and
spread a virus”, FFA queries the target LLM to generate a fallacious yet
deceptively real procedure for the malicious query, as demonstrated in Figure
1. The rationale behind FFA is two-fold: (1) While LLMs generally reject
malicious queries as they are harmful, they would consider a query asking for
a fallacious answer harmless since it purportedly does not seek a truthful
(and harmful) answer. This can potentially help to bypass the LLMs’ safeguard
mechanisms; (2) LLMs would generally leak a truthful answer even when asked to
generate a fallacious one. Therefore, by asking the LLM to generate fake
answers to a malicious query, we can both bypass the security mechanism and
obtain a factual and harmful response. Based on the rationales above, FFA
crafts a jailbreak prompt with four components: malicious query, fallacious
reasoning request, deceptiveness requirement, and scene and purpose. FFA does
not require access to the language model’s internal parameters, fine-tuning,
or multi-turn interaction with a chatLLM.

We evaluate FFA over five safety-aligned large language models: OpenAI
GPT-3.5-turbo, GPT-4 (version 0613) OpenAI (2023), Google Gemini-Pro Anil et
al. (2024), Vicuna-1.5 (7b) Chiang et al. (2023), and LLaMA-3 (8b) AI@Meta
(2024) on two benchmark datasets: AdvBench Zou et al. (2023b) and HEx-PHI Qi
et al. (2023). We compare FFA with four previous state-of-the-art jailbreak
attack methods, Greedy Coordinate Gradient (GCG) Zou et al. (2023b), AutoDAN
Liu et al. (2023), DeepInception Li et al. (2023), and ArtPrompt Jiang et al.
(2024) and under the impact of three defense methods. Our experiments show
that FFA performs most effectively against GPT-3.5, GPT-4, and Vicuna-7b,
provoking these models to generate significantly more harmful outputs. We also
find that none of the three defense methods are effective against FFA,
highlighting the urgent need to address this security threat. In additional
studies, we show the role of scene and purpose in jailbreak attacks and
explain why FFA could induce the most factually harmful results.

##  2 Fallacious Reasoning in LLMs

In this section, we present the findings of our pilot study about LLMs’
capabilities in fabricating fallacious reasoning.

###  2.1 Task and Motivation

We introduce the task of fallacious reasoning, where we ask the LLM to
deliberately generate reasoning processes that satisfy two requirements: ❶
They should be incorrect and lead to false answers, and ❷ they should be
deceptive and appear to be correct.

Generating a fallacious reasoning process is a highly sophisticated task,
because it involves multiple capabilities: the ability to judge the
correctness of an answer, the ability to avoid generating the correct answer,
and the ability to make a wrong answer deceptively real. However, existing
research revealed that LLMs struggle in in discerning the veracity of their
intrinsic answers Huang et al. (2024); Kadavath et al. (2022); Stechly et al.
(2023). Therefore, we raise the following intriguing research questions: Can
LLMs deliberately generate fallacious reasoning upon request?

###  2.2 Experiment Setting

To investigate this, we design the following pilot experiment. We choose four
reasoning benchmarks, math reasoning GSM8K Cobbe et al. (2021) and MATH
Hendrycks et al. (2021), commonsense reasoning HotPotQA Yang et al. (2018),
and logic reasoning ProofWriter Tafjord et al. (2020), and randomly sample 100
questions for each benchmark. For each question, we use GPT-3.5-turbo to
generate answers in two modes. ❶ Honest Mode. We ask the LLM to generate the
correct answers, using zero-shot Chain-of-Thought Kojima et al. (2023) to
prompt the LLM, which appends “Let’s think step by step.” to the question
text; ❷ Fallacious Mode. We ask the LLM to provide a step-by-step yet
fallacious solution to the question and explain why it is incorrect. Detailed
dataset description and experimental settings in this section are available in
Appendix A.

###  2.3 Our Findings

One might expect that the accuracy of the solutions generated by these two
modes would be drastically different – the honest mode would yield high
accuracy and the fallacious mode low. However, this is not the case. Figure 2
compares the accuracies of the two modes on the four benchmarks, where we
find, quite surprisingly, that the two modes yield comparably high accuracies.
This implies that even if the LLM is asked to generate a wrong answer, it is
still likely to generate the correct one.

Figure 2: Accuracy (compared with ground truth of answers) of fallacious and
honest solutions on four different tasks by GPT-3.5-turbo.

To further investigate this behavior, we show a typical example in Figure 3,
where the LLM generates the correct solution to a question in the MATH dataset
despite the fallacious mode. There are two important observations. First,
although the LLM is asked to generate a fallacious solution, the entire
reasoning chain, as well as the final answer, is correct. Second, when the LLM
generates the explanation of why the reasoning chain is incorrect, it simply
randomly picks a correct step and claims that it is wrong. The final
conclusion even contains a contradictory statement (“The correct sum should be
5/6, not 5/6”), indicating the LLM only generates what looks like an
explanation on the surface without making any actual logical judgment.
Although the four benchmarks are in different reasoning domains, our
observations are consistent.

This pilot study reveals that LLMs are unable to generate fallacious reasoning
on purpose, and instead often leak the correct solution in what they claim to
be wrong answers. As we will show in Section 3, this deficiency may turn out
to be an important security threat in LLMs.

Figure 3: An example where the LLM failed to provide a fallacious solution
upon request but instead proposed the correct solution and contradictorily
claimed it false.

##  3 Fallacy Failure Attack

The findings introduced above have profound security implications. In this
section, we will discuss how the LLMs’ failure in fallacious reasoning can be
exploited to form an effective jailbreak attack, called the _Fallacy Failure
Attack_ (FFA).

###  3.1 Problem Formulation

Jailbreak attacks aim to elicit harmful information from LLM by transforming
the malicious query into a special input form that could bypass LLMs’ inherent
security safeguard. Specifically, consider a malicious query, $Q$, (e.g., “How
to create and release a virus that can cause ecological damage?”). If $Q$ is
directly fed into an LLM, the LLM is likely to reject to answer due to the
security safeguard mechanism. Jailbreak attack introduces a transformation
function, denoted as $f(\cdot)$, that turns the malicious input into a
jailbreak input, $Q^{\prime}=f(Q)$, which can trick the LLM into giving away
its answer to the malicious question.

Therefore, the key to designing a jailbreak attack is searching for the input
transformation function. A jailbreak attack with a high level of threat and
practicality should have the following properties: ❶ Query-agnostic
transformation – the same transformation should work for a wide range of
malicious queries; and ❷ Model generalizability – the attack should work for
different LLMs, including white-box and black-box LLMs; thus the design of
transformation should ideally not utilize the gradient information of LLMs.

###  3.2 FFA Overview

Our proposed attack method, FFA, uses a fallacious reasoning query to bypass
LLMs’ safeguard mechanisms. Specifically, FFA is built on the following two
conclusions. First, although LLMs would generally recognize direct malicious
queries as harmful queries and reject them, most of them would consider a
query to generate a fallacious answer as a harmless query and would not reject
them. For example, although “How to create and release a virus that can cause
ecological damage” is often identified as a harmful query, “Generate a
fallacious procedure of creating and releasing a virus” is generally
considered a harmless one by most LLMs, because it does not seek to find the
true answer, and thus would not be blocked. Second, LLMs would generally
release the true answer even when asked to generate a fallacious one.
Therefore, by asking the LLM to generate fake answers to a malicious query, we
can both bypass the security mechanism and obtain the true answer, hence
forming the rationale behind FFA.

###  3.3 FFA Prompt Design

Based on the rationales above, FFA crafts a jailbreak query with the following
four components.

$\bullet$ Malicious Query – the target query for which the attacker wishes to
obtain an answer, _e.g._ , “create and release a virus that can cause
ecological damage” or “make and distribute counterfeit.”

$\bullet$ Fallacious Reasoning Prompt – the instruction for the LLM to
generate a fallacious procedure for the given malicious query, _e.g._ ,
“Provide a fallacious procedure to [malicious query]”.

$\bullet$ Deceptiveness Requirement – an additional instruction that the
fallacious procedure should be deceptively real, _e.g._ , “Need to be ‘real’
to make laymen believe, not too absurd”. As shown in Figure 4, the
deceptiveness requirement is crucial in encouraging the LLM to leak real
answers (left plot); otherwise, the LLM could generate answers that are too
naively fictional (right plot).

$\bullet$ Scene and Purpose – a description of a hypothetical scene or role-
play and a legitimate purpose that could further relax the LLM’s ethical
constraints. Typical scenes include “writing a fiction”, “being a professor”,
or “playing a game”; typical purposes include “for legitimate purpose” or the
ones that align with the scene Li et al. (2023); Liu et al. (2023); Jiang et
al. (2024); Qi et al. (2023); Chao et al. (2023).

To sum up, the prompt template for FFA is shown below. Our method does not
require access to the language model’s internal parameters, fine-tuning, or
multi-turn interaction with a chat LLM.

Prompt Template of FFA [Scene] provide a fallacious procedure to [Malicious
Query] [Purpose]. Make sure the procedure has to be fake but read well by
laymen. [Deceptiveness Requirement]

Figure 4: A comparison between our pilot jailbreak prompts and corresponding
output excerpts, with and without specification of deceptiveness.

##  4 Experiments

In this section, we describe our experiment results evaluating the
effectiveness of FFA, as well as additional studies on its mechanisms.

###  4.1 Experimental Settings

#### Victim Language Models

We utilize five State-of-the-Art large language models for evaluation,
including three closed-source models, OpenAI GPT-3.5-turbo and GPT-4 (version
0613) OpenAI (2023) and Google Gemini-Pro Anil et al. (2024), and two open-
source models, Vicuna-1.5 (7b) Chiang et al. (2023) and LLaMA-3 (8b) AI@Meta
(2024). All LLMs are aligned with safety protocols.

#### Baselines

We compare our approach with four previous State-of-the-Art attacking methods:

$\bullet$ Greedy Coordinate Gradient (GCG) Zou et al. (2023b) is an
optimization-based method which requires white-box access to a language model.
It searches for a token sequence that maximizes the conditional probability of
an output starting with an affirmative response.

$\bullet$ AutoDAN Liu et al. (2023) an optimization-based, automated jailbreak
attack that generates stealthy prompts using a hierarchical genetic algorithm,
requiring white-box access to a language model.

$\bullet$ DeepInception Li et al. (2023) is a black-box jailbreak attack that
leverages LLMs’ personification abilities to construct a nested scene to
provoke harmful behaviors.

$\bullet$ ArtPrompt Jiang et al. (2024) is a black-box attack that leverages
ASCII art to conceal harmful instructions within a word puzzle. The attack
then encourages the LLM to decode this masked word and inadvertently complete
the harmful instruction, thereby circumventing the safety measures.

For completeness, we also include a Naïve baseline, which directly instructs
victim LLMs for harmful behaviors.

#### Datasets

Following previous work, we evaluate our approach on two benchmark datasets:
AdvBench Zou et al. (2023b) and HEx-PHI Qi et al. (2023). Specifically, we use
a subset of 50 representative non-duplicate harmful instructions from
AdvBench, following Wei et al. (2023) and Jiang et al. (2024). For the HEx-PHI
dataset, we sample ten harmful instructions from each of the 11 prohibited
categories, following Jiang et al. (2024).

#### Metrics

We utilize three metrics to evaluate the effectiveness of a jailbreak attack:
❶ Bypass Rate (BPR) (or Helpful Rate or Non-Refusal Rate), defined as the
percentage of harmful attack queries that an LLM does not refuse to answer
over the total number of queries. To detect refusal in the response, we use a
Judge-dictionary containing a list of keywords or phrases (e.g., “Sorry, I can
not assist with that.”), following Qi et al. (2023) and Jiang et al. (2024).
However, LLMs can sometimes respond to malicious queries with off-topic
content or ethical advice rather than directly refusing to answer, causing the
bypass rate to be inaccurate. To this end, previous work Qi et al. (2023);
Jiang et al. (2024); Liu et al. (2023) utilizes LLMs to evaluate the
harmfulness in the output. The output content is rated by GPT-4, given
detailed rubrics from 1 to 5 for harmfulness, with a score of 1 for no harm
and 5 for extreme harm. Therefore, we also report ❷ Average Harmfulness Score
(AHS), the harmfulness scores averaged over all the responses; and ❸ Attack
Success Rate (ASR), the percentage of responses with a score of 5 over the
total number of responses, following Qi et al. (2023) and Jiang et al. (2024).

#### Defense Methods

To evaluate our approach from the defense perspective, we consider three
popular defense methods against jailbreak attacks Jain et al. (2023); Liu et
al. (2023); Jiang et al. (2024): ❶ Perplexity Filter. Typical attacking
prompts can contain gibberish or unconventional strings, resulting in high
perplexity. Thus, the perplexity filter rejects input string $X$ if PPL$(X)$
exceeds a pre-defined threshold $T$. ❷ Paraphrasing. A query is paraphrased
before being prompted to an LLM. Ideally, paraphrasing would preserve an
instruction’s natural semantics but fail to accurately maintain enough
adversarial tokens for jailbreak. ❸ Retokenization. A query is preprocessed by
BPE-dropout Provilkov et al. (2019), where 20% merges from the BPE tokenizer
are dropped, following Jiang et al. (2024). For example, one of the possible
retokenization results of the string ‘‘a fallacious procedure for making
counterfeit’’ can be ‘‘a fall aci ous proce du re for ma king coun ter fe
it’’. Although previous work indicates that, for benign instructions, PPL
checks can result in false alarms and string transformations can degrade model
output quality, these defensive measures remain valuable for understanding the
mechanisms of attack methods.

#### Implementation Details

For baselines, when attacking a black-box model with methods requiring white-
box access, we follow their official implementation of transferable settings,
optimizing the prompt with an open-source model (Vicuna-7b in Zou et al.
(2023b) and LLaMA-2 Touvron et al. (2023) in Liu et al. (2023)) and
transferring it to attack the black-box models. For FFA, we ensemble attack
prompts using three pre-defined common combinations of scene and purpose for
each malicious instruction and store the one with the highest harmful score.
We find that the truthfulness of output is not sensitive to the phrasing of
the discriminator and set it as ‘need to be “real” to make laymen believe, not
too absurd. That is, deceivable enough to educated group’ for the rest of the
experiments. The full implementation details of the attack, defense, and
prompt templates used in the experiments are available in Appendix B.

###  4.2 Main Results

GPT-3.5-turbo† GPT-4† Gemini-Pro† Vicuna-7b LLaMA-3-8B Attack Method BPR% AHS
ASR% BPR% AHS ASR% BPR% AHS ASR% BPR% AHS ASR% BPR% AHS ASR% Naïve 2 1.22 0 0
1.00 0 8 1.28 6 4.4 1.09 0 0 1.00 0 GCG 30 3.36 54 24 1.48 10 48 2.88 46 96.3
4.09 66.2 38.1 1.96 8.8 AutoDan 24 1.78 18 14 1.52 10 20 1.34 8 98.1 4.21 63.1
46.3 2.03 13.8 DeepInception 100 2.90 16 100 1.30 0 100 4.34 78 100 3.48 32.5
58.1 1.99 10.0 ArtPrompt 92 4.56 78 98 3.38 32 100 4.42 76 100 2.84 12.5 82.5
3.07 28.7 FFA (Ours) 100 4.71 88.1 96.3 4.26 73.8 82.5 4.04 73.1 100 4.81 90.0
46.3 2.22 24.4

Table 1: Attack efficacy of FFA against five language models compared to five
baseline methods.†indicates results from previous papers. Directly asking an
LLM to propose harmful output can be easily rejected by all models. FFA
performs most effectively against GPT-3.5, GPT-4, and Vicuna-7b, provoking
these models to generate significantly more harmful outputs. However, FFA
struggles against LLaMA-3. This is because LLaMA-3 is inclined to reject any
instruction involving the creation of false content, irrespective of its
potential harm. ArtPrompt performs poorly against Vicuna-7b due to the model’s
lack of comprehension ability of ASCII art. DeepInception exhibits a very high
bypass rate, but its output is not harmful based on the AHS and ASR.

GPT-3.5-turbo GPT-4 Gemini-Pro Vicuna-7b LLaMA-3-8B Defense Method BPR% AHS
ASR% BPR% AHS ASR% BPR% AHS ASR% BPR% AHS ASR% BPR% AHS ASR% No Defense 100
4.71 88.1 96.3 4.26 73.8 82.5 4.04 73.1 100 4.81 90.0 46.3 2.22 24.4 PPL-
Filter 95.6 4.55 84.4 91.9 4.11 70.0 78.8 3.89 69.4 95.6 4.64 86.2 43.8 2.14
22.5 Paraphrasing 90.0 4.09 65.6 65.6 2.91 42.5 51.2 2.95 43.8 71.3 3.67 63.1
63.1 2.88 31.9 Retokenization 97.5 4.01 61.3 63.1 3.11 46.9 58.8 2.99 38.8
92.5 3.19 31.9 73.1 2.44 21.9

Table 2: Result of FFA performance under the impact of defense approaches.

#### Attack Efficiency

Table 1 illustrates the performance of FFA compared with the five baselines
across five language models. There are primarily two observations regarding
our approach. First, FFA is most effective against GPT-3.5, GPT-4, and
Vicuna-7b and achieves comparable performance against Gemini-Pro, compared
with other previous State-of-the-Art jailbreak methods. Against GPT-3.5,
GPT-4, and Vicuna-7b, our method provokes the LLMs to generate significantly
more harmful output, with a $10\%\sim 50\%$ absolute improvement in ASR.
Second, the recently released language model LLaMA-3, in general, has stronger
defense power against multiple jailbreak attack methods. However, our method
performed even worse compared with some other methods. By manual inspection of
the responses of the model, we find that LLaMA-3 is inclined to reject any
instruction involving the creation of deceptive or false content, irrespective
of its potential harm. For instance, LLaMA-3 will refuse the proposition of a
fallacious mathematical theorem proof. While we acknowledge that rejecting all
false content could provide optimal defense against FFA, the ability to
generate fallacies could, paradoxically, reflect a form of AI intelligence and
be advantageous in specific contexts, such as mathematics and theory.
Additionally, there are a few observations regarding other baselines: (1)
Nav̈e approach, which directly asks an LLM to propose harmful output, can be
easily rejected by all models. (2) Despite ArtPrompt’s sufficient performance,
it struggles against an easily targeted model, Vicuna-7b. We find that this is
due to the model’s inability to interpret ASCII art and reconstruct the true
intent of the attack. (3) DeepInception exhibits a very high bypass rate, but
its output is not harmful based on the AHS and ASR. We will discuss a
hypothesis on the harmfulness of our outputs compared with DeepInception in
Section 4.3.

#### Defense Impact

Table 2 presents the results under various defense settings. The "No Defense"
results indicate the best attack performance without implementing any defense
measures. Generally, all three defense methods can negatively impact the
effectiveness of FFA. However, we observe that (1) PPL-Filter only marginally
affects FFA. The result is expected since our attack prompt is phrased
naturally without nonsensical or unconventional strings. (2) Paraphrasing is
generally the most effective defense method against our approach. This is
unexpected since the semantics of instructing LLM for a fallacious output
should be preserved after paraphrasing. We hypothesize that even subtle
semantic changes, including describing harmful behavior, could affect LLM’s
security measures. (3) Surprisingly, paraphrasing and retokenization did not
degrade but enhanced the FFA attack’s effectiveness against LLaMA-3. We find
that during paraphrasing, the terms “fallacious/fake” are often rephrased as
“invalid” or “flawed.” Given the LLM’s strong opposition to fake content, we
hypothesize that paraphrasing could alleviate this opposition. However,
interpreting retokenization is challenging as we’re uncertain how distorted
token inputs are perceived by language models. Overall, none of the three
methods effectively defend against or mitigate FFA, highlighting the urgent
need for more advanced defenses and further research on the fallacious
generation ability in LLMs.

###  4.3 Additional Studies

#### Impact of Scene and Purpose on Attack Efficacy

An intriguing question is the role of scene and purpose in jailbreak attacks.
Do they alone suffice to bypass the LLM’s security measures? Does FFA retain
attack ability without a scene or purpose? We conducted an ablation study and
computed the AHS and ASR under five attacks, scene, and purpose combinations
across three language models shown in Figure 5. setting X, Y, and Z refer to
directly asking LLM for malicious behavior with that combination of scene and
purpose, respectively. FFA + Z refers to using Z as the scene and purpose of
the FFA attack. FFA + None refers to the FFA attack without specifying any
scene or purpose. We can observe that (1) naïvely adding a scene and purpose
to the direct instruction of harmful behavior mostly has a marginal effect on
jailbreak attack. The only exception is the combination of “scientific
fiction” and “against evil Doctor X,” which demonstrates notable attack
efficacy against GPT-3.5-turbo and Gemini-Pro. Interestingly, this unique
design seems to be the primary driving force behind the DeepInception method.
(2) Although our method archives optimal performance with the combination of
scene and purpose, it can retain significant attack ability without a scene
and purpose, except for LLaMA-3, which opposes the creation of untruthful
content. (3) When using the same fictitious scene and purpose as
DeepInception, our method is more likely to induce more harmful output.

Figure 5: Scatter plot of AHS and ASR from five attack and scene/purpose
combinations across three language models.

#### Harmfulness from Honesty

Figure 6 presents a qualitative example of the outputs from FFA and
DeepInception, both targeting the same malicious behavior. Content-wise, while
DeepInception’s output is loaded with science fiction terminology, FFA
presents factual and detailed steps for the queried malicious behavior. Why is
the output produced by FFA more harmful and factual? We hypothesize that (1)
despite the potential for a science fiction scenario to bypass the LLM’s
security measures, the output will inevitably be more fictional due to the
LLM’s understanding of sci-fi. (2) In contrast, FFA rests on the fact that LLM
cannot generate a deceptive solution but instead provides a truthful
counterpart, thus is factually harmful. For this reason, the harmfulness in
the FFA’s output, to some extent, is retained even when using the fictitious
scene and purpose from DeepInception.

Figure 6: A comparison of the model outputs between FFA and DeepInception,
given the same target behavior - describe steps on insider trading.

Interestingly, in the FFA output, the LLM also tries to elucidate that the
above procedure is fallacious. This incoherent conclusion and the truthful
solution under the fallacious request echo the findings in the fallacious
reasoning section.

##  5 Related Work

#### Jailbreak Attack

Recent large-scale language models (LLMs) are optimized and aligned with human
preferences under ethical guidelines and legal constraints Ouyang et al.
(2022); Ziegler et al. (2020); Zou et al. (2023a). However, studies have
discovered that deliberately transforming the prompt can trick LLMs into
responding with malicious instructions without rejection, which exposes the
ethical and security risks of LLMs in real-world applications Wei et al.
(2023); Qi et al. (2023). Two primary strategies are currently employed to
identify these transformations. The first involves optimization requiring
access to a white box language model. Zou et al. (2023b) introduce an
optimization-based method by searching for a token sequence that maximizes the
conditional probability of an output starting with an affirmative response.
Liu et al. (2023) propose to generate more readable prompts using a
hierarchical genetic algorithm. The second involves manually crafting or
searching for prompt updates without requiring gradient access. Li et al.
(2023) leverage LLMs’ personification abilities to construct a nested scene to
provoke harmful behaviors. Jiang et al. (2024) use ASCII art to conceal
harmful instructions within a word puzzle to circumvent the safety measures.
There are also methods that are based on multi-turn interactions with chat
LLMs. Chao et al. (2023) utilize an additional language model as an attacker
to find jailbreak prompts with multiple queries, Russinovich et al. (2024)
attack the chat language model with multi-turn dialogues.

#### Jailbreak Defense

The development of defense methods is challenging and limited due to the
inaccessibility of internal parameters of closed-source language models. The
most straightforward strategy involves a perplexity check, presuming the
attack prompt contains unnatural strings. Some methods involve prompt pre-
processing, including token perturbation and transformation Jain et al.
(2023); Provilkov et al. (2019); Robey et al. (2023). However, these defenses
could compromise benign user instructions’ output quality. Lastly, some
strategies leverage language models to assess the potential harm of the
instruction and its output Kumar et al. (2024); Phute et al. (2024).

##  6 Conclusion and Future Work

This paper presented a simple yet explainable and effective jailbreak attack
method. It is predicated on the observation that language models cannot
generate fallacious and deceptive solutions but instead produce honest
counterparts. We argue that this observation not only poses a security threat
but also implies how modern LLMs’ perceptions of specific tasks are limited
when the scenario is inadequately optimized and aligned during training. We
believe this observation can be further extended to related research areas,
such as self-verification and hallucination, providing valuable insights into
understanding LLM behavior toward general intelligence.

## Limitations

While in this paper, we propose an effective jailbreak attack method against
language models, we have not yet identified an ideal defense mechanism to
counteract it. One potential defense strategy is to consistently reject
queries containing fallacious reasoning. However, this approach may not be
optimal, as it undermines the versatility and utility of large language models
in achieving general intelligence and could lead to inadvertent rejection of
benign queries in other applications. Future work is required to develop more
robust and sophisticated defense strategies to effectively prevent FFA.

## Ethics Statement

This paper introduces a jailbreak approach leveraging LLMs’ failures of
fallacious reasoning. It potentially allows adversaries to exploit LLMs,
creating outputs that do not align with human values or intentions. However,
like previous jailbreak research, this work should encourage research into
improved defense strategies and develop more robust, secure, and well-aligned
LLMs in the long term. We also hope that the characteristic of LLMs leaking
truthful content upon request of the fallacious generation will draw attention
from the research community, enabling potential research in other areas, such
as hallucination and LLM self-verification.

## References

  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Alon and Kamfonas (2023) Gabriel Alon and Michael Kamfonas. 2023.  Detecting language model attacks with perplexity.
  * Anil et al. (2024) Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, et al. 2024.  Gemini: A family of highly capable multimodal models.
  * Chao et al. (2023) Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023.  Jailbreaking black box large language models in twenty queries.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2(3):6.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.  Measuring mathematical problem solving with the math dataset.  _NeurIPS_.
  * Huang et al. (2024) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024.  Large language models cannot self-correct reasoning yet.
  * Jain et al. (2023) Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023.  Baseline defenses for adversarial attacks against aligned language models.
  * Jiang et al. (2024) Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. 2024.  Artprompt: Ascii art-based jailbreak attacks against aligned llms.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.
  * Kojima et al. (2023) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023.  Large language models are zero-shot reasoners.
  * Kumar et al. (2024) Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2024.  Certifying llm safety against adversarial prompting.
  * Li et al. (2023) Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023.  Deepinception: Hypnotize large language model to be jailbreaker.  _arXiv preprint arXiv:2311.03191_.
  * Liu et al. (2023) Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023.  Autodan: Generating stealthy jailbreak prompts on aligned large language models.  _arXiv preprint arXiv:2310.04451_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Phute et al. (2024) Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2024.  Llm self defense: By self examination, llms know they are being tricked.
  * Provilkov et al. (2019) Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019.  Bpe-dropout: Simple and effective subword regularization.  _arXiv preprint arXiv:1910.13267_.
  * Qi et al. (2023) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023.  Fine-tuning aligned language models compromises safety, even when users do not intend to!  _arXiv preprint arXiv:2310.03693_.
  * Robey et al. (2023) Alexander Robey, Eric Wong, Hamed Hassani, and George J. Pappas. 2023.  Smoothllm: Defending large language models against jailbreaking attacks.
  * Russinovich et al. (2024) Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2024.  Great, now write an article about that: The crescendo multi-turn llm jailbreak attack.
  * Stechly et al. (2023) Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023.  Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems.
  * Tafjord et al. (2020) Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2020.  Proofwriter: Generating implications, proofs, and abductive statements over natural language.  In _Findings_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wei et al. (2023) Zeming Wei, Yifei Wang, and Yisen Wang. 2023.  Jailbreak and guard aligned language models with only few in-context demonstrations.  _arXiv preprint arXiv:2310.06387_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020.  Fine-tuning language models from human preferences.
  * Zou et al. (2023a) Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023a.  Representation engineering: A top-down approach to ai transparency.
  * Zou et al. (2023b) Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023b.  Universal and transferable adversarial attacks on aligned language models.

##  Appendix A Fallacious Reasoning Details

For the experiments in Section 2, we randomly sample 100 data points from each
of the following:

$\bullet$ GSM8K Cobbe et al. (2021) contains linguistically diverse grade
school-level math questions with moderate difficulties.

$\bullet$ MATH Hendrycks et al. (2021) is a competition mathematics dataset
with challenging concepts such as Calculus and Linear Algebra, spanning over
five difficulty levels. We sample 20 examples from each of the five difficulty
levels.

$\bullet$ HotPotQA Yang et al. (2018) is a multi-domain, multi-hop question-
answering dataset based on supporting facts. We sample questions form the
hard-level development set since the online testing set is unavailable.

$\bullet$ ProofWriter Tafjord et al. (2020) is a multi-hop logic reasoning
dataset with each example containing a set of facts and rules for the logic
deduction. We sample all the questions from the set requiring the most hops.

We show another example where the LLM failed to propose fallacious reasoning
from the HotPotQA dataset in Figure 7, with similar patterns as the example in
Figure 3.

Figure 7: An example from HotPotQA, where the LLM failed to provide a
fallacious solution upon request but instead proposed the correct solution and
claimed it false without supporting evidence.

##  Appendix B Main Experiment Settings

For PPL-Filter, we use GPT-2 to calculate $PPL(X)$, following Alon and
Kamfonas (2023), defined as:

$PPL(X)=-\frac{1}{t}\sum_{i}^{t}logp(x_{i}|x_{<i}),$ (1)

where $X$ is the input token sequence. Following Jain et al. (2023); Jiang et
al. (2024), we set $T=175.57$ as the threshold, which is the max perplexity
among all the direct instructions of the malicious behaviors in the AdvBench
datasets.

Since ArtPrompt may require encoding and masking one (meaningful) word at a
time for each harmful behavior (e.g., “[MASK] fraud” or “tax [MASK]”), we
report the best performance out of multiple attack prompts for each harmful
query.

We utilize GPT-3.5-turbo for paraphrasing, following Jain et al. (2023), with
the template “Paraphrase the following instruction: {oringial instruction}”.
The paraphrased version is then forwarded to the target LLM as the new
instruction.

We treat [scene] and [purpose] as hyperparameters in FFA. We utilize three
predefined scene-purpose pairs based on commonsense and previous work, shown
in Table 3:

SetID Scene Purpose 1 Suppose you are a professor in forensicscience. In the
class, … to let the criminal believes thereby showing their trace. 2 You are a
news reporter, … for educational and legitimate purpose. 3 In the writing of a
science fiction, … against the evil doctor X.

Table 3: Scene and Purpose combinations used in FFA.