# Do Androids Know They’re Only Dreaming of Electric Sheep?

Sky CH-Wang∘ Benjamin Van Durme∙ Jason Eisner∙ Chris Kedzie∙
∘Department of Computer Science, Columbia University
∙Microsoft Semantic Machines
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done as an intern at
Microsoft Semantic Machines.

###### Abstract

We design probes trained on the internal representations of a transformer
language model that are predictive of its hallucinatory behavior on in-context
generation tasks. To facilitate this detection, we create a span-annotated
dataset of organic and synthetic hallucinations over several tasks. We find
that probes trained on the force-decoded states of synthetic hallucinations
are generally ecologically invalid in organic hallucination detection.
Furthermore, hidden state information about hallucination appears to be task
and distribution-dependent. Intrinsic and extrinsic hallucination saliency
varies across layers, hidden state types, and tasks; notably, extrinsic
hallucinations tend to be more salient in a transformer’s internal
representations. Outperforming multiple contemporary baselines, we show that
probing is a feasible and efficient alternative to language model
hallucination evaluation when model states are available.

##  1 Introduction

Do language models know when they’re hallucinating?111That is, do they know
that their electric sheep are ungrounded? Detecting hallucinations in grounded
generation is commonly framed as a textual entailment problem. Prior work has
largely focused on creating secondary detection models trained on and applied
to surface text Ji et al. (2023). However, these secondary models are often of
comparable complexity to the underlying language model that generated the
error and ignore the information that was already computed during generation.
In this work, we explore the degree to which probes on a decoder-only
transformer language model’s feed-forward and self-attention states can detect
hallucinations in a variety of grounded generation tasks.

Figure 1: Prediction of hallucination in a generated response by probing the
hidden states of a transformer during decoding. The true annotated span of
hallucination within the response is boxed in white; rows represent probes
trained to detect hallucination from layers within a transformer’s hidden
states, with columns representing their prediction for each token in a
generated response.

In particular, as the language model generates a response, we probe its
encoded tokens to determine whether the model has just begun to hallucinate or
will eventually be found to have hallucinated. It is plausible that these
encodings are “aware” of whether generation is retrieving information that is
grounded in the prompt versus filling in plausible information that is stored
in the model’s weights. Related work has studied how transformers retrieve
relevant memories as they encode text Meng et al. (2022, 2023) and suggests
that feed-forward layers act as associative memory stores Geva et al. (2021).
A probe for hallucination detection must consider whether subsequent generated
text is _entailed_ by information in the prompt, given the retrieved memories,
or is merely one plausible _extrapolation_ from the prompt (see also Monea et
al., 2023). author=jason,color=green!40,size=,fancyline,caption=,]I edited
this paragraph, but I’m still not sure whether we’re fully living up to it. It
feels like we need a more complex taxonomy of
hallucination/entailment/memory.Is the sin that we use a memory that goats
_sometimes_ live in Iran, which is a tendency rather than a fact? Is it a
different sin when the entailment depends "mostly" on the retrieved fact
rather than the prompt?

We develop three probes of increasing complexity across three grounded
generation tasks: abstractive summarization, knowledge-grounded dialogue
generation, and data-to-text. For each task, we collect hallucinations in two
ways: (1) from sampled responses, where we generate outputs from a large
language model (LLM) conditioned on the inputs, or (2) by editing reference
inputs or outputs to create discrepancies. We refer to these as organic and
synthetic data, respectively. In both cases, we have human annotators mark
hallucinations in the outputs. Method (2) produces hallucination annotations
at a higher rate, though we find that the utility of these synthetic examples
is lower as they do not come from the test distribution.

We summarize our contributions as follows:

  * •

We produce a high-quality dataset of more than 15k utterances with
hallucination annotations for organic and synthetic output texts across three
grounded generation tasks.

  * •

We propose three probe architectures for detecting hallucinations and
demonstrate improvements over multiple contemporary baselines in hallucination
detection.

  * •

We analyze how probe accuracy is affected by annotation type
(synthetic/organic), hallucination type (extrinsic/intrinsic), model size, and
which part of the encoding is probed.

##  2 Related Work

#### Definitions.

We center our study of hallucinations in the setting of in-context generation
Lewis et al. (2020) where grounding knowledge sources are provided within the
prompt itself. This is in contrast to settings where a language model is
expected to generate a response entirely from its world knowledge (i.e.,
grounded in the model’s training data) Azaria and Mitchell (2023).

Though the term has often appeared in reference to factuality Gabriel et al.
(2021), hallucinations are not necessarily factually incorrect. In their
survey, Ji et al. (2023) classify hallucinations as intrinsic, where generated
responses directly contradict the knowledge sources, or extrinsic, where
generated responses are neither entailed nor contradicted by the sources. We
follow this taxonomy. For example, given the following task prompt for
abstractive summarization,

> …Deckard is now able to buy his wife Iran an authentic Nubian goat—from
> Animal Row, in Los Angeles—with his commission …TL;DR:

A generated summary sentence of “Deckard can now purchase an animal for his
wife” is faithful, “Deckard can purchase a goat in Iran” is an intrinsic
hallucination, and “Sheep are usually cuter and fluffier than goats” is an
extrinsic hallucination. While models may leverage latent knowledge to inform
their responses, such as recognizing that a goat is indeed an animal, directly
stating such knowledge in a response—i.e. only generating “Goats are animals”
in the example above—is often considered an extrinsic hallucination as it is
neither entailed nor contradicted by a given knowledge source. Thus, non-
hallucinatory generation in this setting involves balancing the language
model’s retrieval and manipulation of knowledge, both from the given context
and from its own stored weights.

Task | History | Knowledge | Instruction | Response
---|---|---|---|---
Abstractive |  | …dialed down that initial | TL;DR: | The National Weather
Summarization |  | forecast to predict 13 to 19 |  | Service forecast called
|  | named storms, including six |  | for 13 to 19 named
|  | to nine hurricanes… |  | storms including six
|  |  |  | to 11 hurricanes.
KGDG | …B: ok how can I get a | Three consecutive strikes | Response that uses | A: Indeed, you have to
| turkey I think that may | are known as a "turkey." | the information in | get four consecutive
| be hard to get though |  | knowledge: | strikes to get one.
Data-to-Text |  | name[The Wrestlers], | Restaurant | Located by the river-
|  | priceRange[high] | Description: |  side, The Wrestlers is
|  |  |  | a restaurant with high
|  |  |  | prices, and near Raja
|  |  |  | Indian Cuisine.
Table 1: From top to bottom, examples of organic hallucinations in model
responses for abstractive summarization, knowledge-grounded dialogue
generation (KGDG), and data-to-text tasks. Minimum annotated hallucination
spans are highlighted in cyan for intrinsic hallucinations and green for
extrinsic hallucinations.

#### Automated Diagnosis.

Lexical metrics commonly used in the evaluation of NLG fail to correlate with
human judgments of faithfulness Kryscinski et al. (2019). More useful are NLI
approaches that either directly measure the degree of entailment between a
generated response and a given source text Hanselowski et al. (2018);
Kryscinski et al. (2020); Goyal and Durrett (2020), or else break down texts
into single-sentence claims that are easily verifiable Min et al. (2023);
Semnani et al. (2023); these have been successfully used for hallucination
detection Laban et al. (2022); Bishop et al. (2023). Similarly, approaches
that leverage question-answer models to test if claims made by a response can
be supported by the source text, and vice versa Wang et al. (2020); Durmus et
al. (2020); Scialom et al. (2021), try to more explicitly identify the
grounding sources of given generations in this task. Under the intuition that
a model is usually not confident when it is hallucinating, hallucinatory
behavior can also be reflected in metrics that measure generation uncertainty
or importance Malinin and Gales (2021); Guerreiro et al. (2023); Ren et al.
(2023); Ngu et al. (2023); Madsen et al. (2023), relative token contributions
Xu et al. (2023), and differences between the attention matrices and decoder
stability author=jason,color=green!40,size=,fancyline,caption=,]can’t really
parse this Lee et al. (2018) during generation.

#### Predicting Transformer Behavior.

In small transformer language models, it is possible to ascribe behaviors like
in-context learning to specific layers and modules Elhage et al. (2021). This
becomes harder in larger language models, but Meng et al. (2022) provide
evidence that the feed-forward modules function as key-value stores of factual
associations and that it is possible to modify weights in these modules to
edit facts associated with entities. Our probes, to be successful, must
determine whether the recalled facts were deployed in accordance with the
rules of grounding (i.e., to paraphrase facts expressed in the prompt, rather
than to replace them or to add details that are merely
probable).author=jason,color=green!40,size=,fancyline,caption=,]old: "Our work
extends this finding to show that information about the recall of facts
associated with encoded grounding data is present in generation-time hidden
states". but I don’t think our annotation taxonomy was actually rich enough to
let us say anything about what facts were recalled? some hallucinations
involve recall, some don’t, but we didn’t distinguish those  ck: Proposed
change: Our work extends this finding to show that feed-forward outputs
contain signals about how information is being recalled from the prompt in
addition to claims recalled from the training data. Transformer internal
states have also been shown to be predictive of the likelihood that a model
will answer a question correctly in the setting of closed-book QA Mielke et
al. (2022) and in the prediction of the truthfulness of a statement Azaria and
Mitchell (2023), though these internal representations of truthfulness can
disagree systematically with measures of model confidence Liu et al. (2023).
These works measure hidden states in the absence of supporting evidence (i.e.
no passages are retrieved for either task), making them effectively probes of
knowledge stored in model weights and not the manipulation of novel or
contingent information provided in a prompt. Monea et al. (2023) find that
there are separate computational processes in transformer LLMs devoted to
recall of associations from memory vs grounding data, and we show that we can
detect whether associations from grounding data are used correctly across
several distinct generation tasks. Slobodkin et al. (2023) train a probe to
detect the (un)answerability of a question given a passage, which is similar
in some respects to our task in Section 4; however, we detect token-level
hallucination during answer generation, which could happen even in
theoretically answerable questions.

Response | Original Knowledge | Changed Knowledge
---|---|---
Yes, Elvis was also one of | Elvis Presley is regarded as one of | Elvis Presley is regarded as one
the most famous musicians | the most significant cultural icons | of the least well-known music-
of the 20th century. | of the 20th century. |  -ians of the 20th century.
Close to The Sorrento you | name[Clowns], eatType[pub] | name[Clowns],
can find Clowns pub. | near[The Sorrento] | near[The Sorrento]
Table 2: Examples of changed knowledge statements for our datasets of
synthetic hallucinations. Changes made to Conv-FEVER (top) and E2E (bottom)
are shown in bold. The changed statements make the top response an intrinsic
predicate error and the bottom response an extrinsic entity error (highlighted
as in Table 1).

##  3 Grounded Generation Tasks

We test hallucination probes for autoregressive grounded generation in three
distinct tasks: abstractive summarization, knowledge-grounded dialogue
generation, and data-to-text. We use the same high-level prompt structure
across all tasks to organically generate model responses to tasks and to probe
for hallucination. Each prompt contains an optional interaction history, a
knowledge source, and an instruction resulting in the conditionally generated
response. Task prompts and examples of organic hallucinations are shown in
Table 1.

###  3.1 Organic Hallucinations

We use llama-2-13b Touvron et al. (2023) as the response model to generate
outputs for each task. To ensure that a non-negligible number of both
hallucinations and grounded responses were present in the generations for each
task, each model response is generated using top-k random sampling Fan et al.
(2018) with $k=2$. author=jason,color=green!40,size=,fancyline,caption=,]why
does this lead to hallucination more than ordinary top-$\infty$ sampling, or
greedy top-1 sampling?  sky: it can, but the point here is balance of grounded
or hallucination; greedy iirc was too much hallucination for some The number
of few-shot in-context learning examples for each task was chosen through a
pilot exploration of several generation strategies, optimizing for the
presence of a decent balance of hallucinations and grounded responses in all
tasks, as determined via manual inspection by the authors. Full prompt
templates are shown in Appendix C.

#### Abstractive Summarization.

The CNN / DailyMail (CDM) dataset Hermann et al. (2015) is a standard
benchmark in the setting of abstractive summarization Lin and Ng (2019). It
consists of a set of news articles paired with reference summaries in the form
of human-written lists of bullet points that were originally published
alongside the articles. For automatic summarization, we follow the GPT-2 paper
Radford et al. (2019) and prompt our model to generate abstractive summaries
of news articles by appending TL;DR: after the article content (a 0-shot
prompt),222In Internet posts that appear in LLM pretraining data, this string
often marks the start of a summary immediately following the main content of
the post. Prior work Radford et al. (2019) has found that appending this
string after texts in prompts to autoregressive LLMs is capable of producing
abstractive summaries of the former. taking the first three generated
sentences as the summary.

#### Knowledge-Grounded Dialogue Generation.

For a given conversation history and a provided knowledge sentence obtained
via Wizard-Of-Wikipedia, the Conv-FEVER Santhanam et al. (2021) task (CF) asks
a respondent to create a next-turn response to the conversation that is
grounded in the provided knowledge sentence. Here, we prompt our response
model to generate grounded responses under 2-shot in-context learning with
example grounded responses.

#### Data-to-Text Generation.

The E2E task Novikova et al. (2017) asks respondents to write restaurant
descriptions that are congruent with a provided set of attributes. Here, we
prompt our response model to perform this task with 10-shot in-context
learning.333As the original E2E task had omitted or missing information in up
to 40% of its data, we use the cleaned version by Dušek et al. (2019). As even
the cleaned version had instances of omitted or missing information, 10 in-
context examples were manually selected and manually verified for
groundedness.

###  3.2 Synthetic Hallucinations

Recent work Ma et al. (2023); Monea et al. (2023) has carefully created
synthetic ungrounded datasets by hand to evaluate metrics used to measure
unfaithfulness and models used to detect it. The BUMP dataset Ma et al. (2023)
is a version of CDM in which reference summaries have been manually edited to
be ungrounded. It includes both freestyle edits and systematic coverage of 4
specific types of faithfulness errors: (1) predicate errors, where the
predicate in a generation is inconsistent with the source knowledge; (2)
entity errors, where the subject or object of a predicate is inconsistent; (3)
circumstance errors, or errors in the time, duration, or location of an event
of the predicate; and (4) coreference errors. Types (1)–(3) may be intrinsic
or extrinsic.

We use BUMP in our CDM experiments. For our Conv-FEVER and E2E experiments, we
similarly created faithfulness errors in a uniformly sampled subset of these
datasets. However, rather than changing an example’s reference response, we
changed its provided knowledge sentence so that the existing reference
response would be unfaithful to it. This simulates the common case of
hallucination where a large language model generates statements that may be
true, but which are not grounded in the source documents as required. For
Conv-FEVER, we prompted ChatGPT to modify a given example’s provided knowledge
sentence to introduce intrinsic and extrinsic errors of types (1)–(4) above;
we then manually judged these attempts and edited the unsuccessful ones.444In
hindsight, this workflow proved inefficient; significant human editing and
validation were needed. In this way, we created equal numbers of examples with
intrinsic and extrinsic predicate, entity, circumstance, and coreference
errors, as well as freestyle errors.

Similarly, for E2E, we modified a random subset of the provided grounding
attributes. Specifically, for an example with $n$ attributes, we drew $k$
uniformly from $[1,n-1]$, uniformly selected one of the subsets of size $k$,
and then flipped a fair coin to decide whether to remove or perturb—by
randomly sampling an alternative attribute from the entire dataset—these $k$
attributes. Finally, we manually verified that the reference response was
unfaithful with respect to the modified attributes; if not, we further edited
the attributes. Examples of changed knowledge statements are shown in Table 2;
statistics of our synthetic datasets are shown in Table 3.

Task | N-o | H-o | N-s | H-s
---|---|---|---|---
CDM | 3399 | 23% | 1778555Obtained via BUMP Ma et al. (2023). | 50%
CF | 924 | 64% | 2846 | 50%
E2E | 4975 | 57% | 9950 | 50%
Table 3: Dataset statistics. N-o is the number of annotated organic
generations, and H-o is the percentage of organic generations with at least
one hallucination. N-s and H-s refer to the same for synthetic responses. Note
that grounded synthetic responses were not generated by a model but come from
gold reference data.

###  3.3 Annotation

We ask annotators to label the minimal hallucinatory spans in each response,
if any. These are the smallest text spans that would need to be modified to
transform an ungrounded response into a grounded one.

A team of 17 annotators completed a pilot annotation exercise of 50
annotations per task. These pilot annotations were checked and verified by the
authors to pinpoint common sources of error and disagreement. This feedback
was integrated into a revised set of annotation guidelines, shown in Appendix
A, and the full dataset was then reannotated by the same team. Each example
was annotated once; example annotations are shown in Table 1. The number of
annotated examples is shown in Table 3. Note that for each dataset, over one-
fifth of all annotated responses were annotated as containing at least one
hallucinated span (columns H-o and H-s).

##  4 Probing

Figure 2: General probe structure. Response tokens that were annotated as
hallucinatory are shown in red. For each internal representation layer, and
each type of representation (attention output and feed-forward output), a
classifier is separately trained on the attention head out-projection and
feed-forward hidden states for all representation layers (or just the final
token) of a response from a prompted language model. Depending on the probe,
these classifiers are then ensembled to make a final prediction of whether
each individual token was, or otherwise if the response contained, a
hallucination.

Probes Alain and Bengio (2017) are tools used to analyze a neural network’s
internal representations. Often taking the form of linear classifiers, they
are applied to the internal representations of a network and are trained to
discriminate between different types of inputs or outputs. Here, we are
interested in detecting hallucinatory outputs during an LLM’s generation steps
when conditioned on a prompt that requires a response grounded n the prompt.

###  4.1 Design

Let $u$ be a prompt as described at the start of Section 3. To determine
whether a response $x$ to a prompt $u$ contains hallucinations, we will probe
the hidden states of the transformer language model (i.e., decoder) as it
generates $x$. Note that these hidden states depend on $u$ and on the prefix
of $x$ that has been generated so far and that they may be reconstructed from
the string pair $(u,x)$ via forced decoding. A single transformer layer
actually consists of two sublayers: the first transforms a token’s embedding
by adding a vector predicted by attention, and the second transforms it again
by adding a vector predicted by a feed-forward network. We train a separate
probe for each of the $2L$ sublayers, where $L$ is the number of
layers.author=jason,color=green!40,size=,fancyline,caption=,]in next version,
we should also try nonlinear probes (the learned features might detect
different kinds of hallucinations) and we should probably try probing the
unnormalized attention distributions as previously discussed

#### Linear Probe.

This probe is a linear classifier using a single hidden state vector (output
by some sublayer) as input. Let $x_{i}$ be the $i^{\textrm{th}}$ response
token and let $\mathbf{h}_{i}\in\mathbb{R}^{n}$ be the corresponding LLM
hidden state at a fixed layer and component. We write $y_{i}\in\\{0,1\\}$ to
indicate whether $x_{i}$ falls within a minimum hallucination span. The linear
probe predicts $y_{i}$ by $p(y_{i}=1\mid u,x_{\leq
i})=\sigma(\mathbf{w}^{\intercal}\mathbf{h}_{i}+b)$, where
$\mathbf{w}\in\mathbb{R}^{n}$ are learned weights, $b\in\mathbb{R}$ is a
learned bias, and $\sigma$ is the sigmoid
function.author=jason,color=green!40,size=,fancyline,caption=,]In next
version, we should also try a version in which $\mathbf{h}_{i}$ is replaced by
$\mathbf{h}_{i-1}$, since it is really $\mathbf{h}_{i-1}$ that helps generate
$x_{i}$ and decides whether to hallucinate it. (By the time we get to
$\mathbf{h}_{i}$, it’s too late since we’ve already generated $x_{i}$, though
PPO might still be able to learn a policy that avoids generating $x_{i}$.) My
hypothesis is that $\mathbf{h}_{i-1}$ does better at the higher layers
(generating $x_{i}$) while $\mathbf{h}_{i}$ does better at the lower layers
(encoding $x_{i}$ for later use). Possibly ensembling over these two positions
would work?

#### Attention-Pooling Probe.

In order to directly consider the previous hidden states of the response as
well, we also experiment with an attention-pooling probe over all hidden
states so far,

$p(y_{i}=1\mid u,x_{\leq
i})=\sigma(\mathbf{w}^{\intercal}\mathbf{\bar{h}}_{i})$
$\mathbf{\bar{h}}_{i}=\sum_{j=1}^{i}\alpha_{i,j}\mathbf{h}_{j},\;\;\alpha_{i,j}=\frac{\exp\left(\mathbf{q}^{\intercal}\mathbf{h}_{j}\right)}{\sum_{k=1}^{i}\exp\left(\mathbf{q}^{\intercal}\mathbf{h}_{k}\right)}$

where $\mathbf{q}\in\mathbb{R}^{n}$ is a learned query vector.

#### Ensemble Probe.

Our final approach is a single feed-forward layer that combines the
predictions from our $2L$ separately trained probes (either the linear probes
or the attention-pooling probes):

$\displaystyle p(y_{i}\mid u,x_{\leq i})$
$\displaystyle=\sigma\left(\sum_{\mathclap{\begin{subarray}{c}\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;l\in\\{1,\ldots,L\\}\\\
\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;m\in\\{\textrm{attention},\textrm{feed-
forward}\\}\end{subarray}}}\beta_{l,m}\cdot p_{l,m}(y_{i}\mid u,x_{\leq
i})\right)$

where $\beta_{l,m}$ are learned weights. The individual probes are frozen
before $\beta_{l,m}$ parameters are fit.

###  4.2 Training Objectives

#### Token-level.

To train a probe to predict _token-level_ hallucination indicators $y_{i}$, we
minimize negative log-likelihood on the annotations in our dataset
$\mathcal{D}$:

$\mathcal{L}(\theta)\propto\sum_{(u,x,y)\in\mathcal{D}}\sum_{i=1}^{|x|}-\log
p(y_{i}\mid u,x_{\leq i};\theta)$

#### Response-level.

We also try training a classifier to predict whether $x$ contains any
hallucinations at all; denote this indicator as
$y=\bigvee_{i=1}^{|x|}y_{i}\in\\{0,1\\}$. For this, we simply use an
attention-pooling probe over the entire response $x$. To fit the probe
parameters, $\theta$, we again minimize negative log-
likelihood:author=jason,color=green!40,size=,fancyline,caption=,]next version:
report linear probe as well? (see commented-out text below)

$\mathcal{L}(\theta)\propto\sum_{(u,x,y)\in\mathcal{D}}-\log p(y\mid
u,x_{\leq|x|};\theta)$

###  4.3 Metrics

#### Response-level Classification.

F1 is the harmonic mean of recall and precision. We denote the F1 score of a
response-level classifier by F1-R.

#### Token-level Classification.

For token-level classifiers, it is possible to evaluate the F1 score of their
per-token predictions; however, this would give more weight to annotated or
predicted spans that are longer in length. Here, we are more interested in
span-level F1: specifically, whether we predicted all of the annotated
hallucinations (recall), and whether all of the predicted spans were annotated
as actual hallucinations (precision). As requiring an exact match of the spans
is too severe—Appendix B shows how our human annotators often disagreed on the
exact boundaries of a hallucinatory span—we modify span-level precision and
recall metrics to give partial credit, as follows.

Define a span to be a set of tokens, where tokens from different responses are
considered distinct. Let $A$ be the set of all annotated spans in the
reference corpus, and let $P$ be the set of all predicted spans on the same
set of examples. We compute the recall $r$ as the average coverage of each
reference span, i.e., the fraction of its tokens that are in some predicted
span. Conversely, we compute the precision $p$ as the average coverage of each
predicted span:

$\displaystyle r$ $\displaystyle=\frac{1}{|A|}\sum_{s\in
A}\frac{\left|s\cap\left(\bigcup_{\hat{s}\in P}\hat{s}\right)\right|}{|s|}$
$\displaystyle p$ $\displaystyle=\frac{1}{|P|}\sum_{\hat{s}\in
P}\frac{\left|\hat{s}\cap\left(\bigcup_{s\in A}s\right)\right|}{|\hat{s}|}$

where $\cup$, $\cap$, and $|\cdot|$ denote set union, intersection, and
cardinality respectively. We define F1-Sp (standing for "span, partial
credit") as the harmonic mean of $p$ and $r$. Note that F1-Sp micro-averages
over examples, ignoring the boundaries between them. Thus, in contrast to
F1-R, examples with more spans have more influence on the final F1-Sp metric
achieved.

##  5 Experiments

Unless otherwise indicated, all evaluations are performed on organic data
only, even if their probes were trained on synthetic data.

###  5.1 Probe Hyperparameters and Training

Single-layer hidden state probes, described in Section 4, are trained under
float32 precision with unregularized log loss. A batch size of 100 and Adam
optimization Kingma and Ba (2014) with a learning rate of 0.001 and
$\beta_{1},\beta_{2}=(0.9,0.999)$ were chosen after a small grid search on
hyperparameters. With 70/10/20% train, validation, and test splits, each
classifier was trained using early stopping with 10-epoch patience. With the
predictions of these single-layer probes as input, the ensemble classifier was
trained with the same hyperparameters.

###  5.2 Response-Level Baselines

We compare the performance of our probes against a set of contemporary methods
used in faithfulness evaluation, measuring test set hallucination
classification F1. Specifically: (1) the model uncertainty-based metric of
length-normalized sequence log-probability Seq-Logprob Guerreiro et al.
(2023); (2) off-the-shelf synthetically-trained sentence-level claim
classification FactCC Kryscinski et al. (2020); (3) linguistic feature-based
dependency arc entailment DAE Goyal and Durrett (2020); (4) QA-based metrics
FEQA Durmus et al. (2020); (5) QuestEval Scialom et al. (2021); (6) the NLI-
based metric SummaC Laban et al. (2022); and ChatGPT classifiers Luo et al.
(2023) (7) without and (8) with chain-of-thought prompting, with prompts shown
in § C.1.

Response-Level Classification, F1-R
---
Method | CDM | CF | E2E
OC | 0.38 | 0.78 | 0.72
Seq-Logprob | 0.44 | 0.81 | 0.75
FactCC | 0.43 | - | -
DAE | 0.37 | - | -
FEQA | 0.35 | - | -
QuestEval | 0.41 | - | -
SummaCZS | 0.46 | - | -
SummaCConv | 0.42 | - | -
ChatGPT | 0.34 | 0.62 | 0.37
ChatGPT CoT | 0.36 | 0.68 | 0.45
PoolingSL | 0.43 | 0.86 | 0.86
PoolingE | 0.45 | 0.88 | 0.87
PoolingSL-SYNTH | 0.46∗ | 0.64 | 0.66
PoolingE-SYNTH | 0.48∗ | 0.65 | 0.69
Human | 0.71 | 0.87 | 0.87
Table 4: Response-level probe and baseline organic hallucination detection
performance across tasks as measured via F1 scores (F1-R) achieved on the test
set. SL denotes the best single-layer classifier (chosen on the validation
set), E denotes the ensembled classifier, and SYNTH indicates that the
classifier was trained on synthetic data and tested on organic hallucinations.
The Human row measures the performance of a human annotator who is different
from the reference annotator; this was measured on a subset of 50 examples
that were triply annotated, averaging over all 6 (annotator, reference
annotator) pairs.

Metrics (2)–(6) were not originally intended to detect hallucinations in
individual responses, but rather to evaluate a collection of responses for the
sake of system comparisons. To adapt to our setting, however, we threshold
each of these continuous metrics to classify the individual responses, setting
the threshold to maximize validation set F1, following Laban et al. (2022).
For sentence-level optimized baselines such as FactCC, response-level
evaluation is performed by logical OR-ing sentence-level baseline predictions
(obtained after thresholding the raw scores). ††∗For CDM, the synthetic
training dataset (from BUMP) is smaller than the organic training dataset, so
the synth results are at a disadvantage. For CF and E2E, however, we generated
a synthetic training dataset that was matched in size.

As a further reference baseline, we additionally report the performance
achieved by randomly predicting a hallucination label with some probability p
tuned to maximize F1 on the validation set, denoted here as Optimized Coin
(OC).author=jason,color=green!40,size=,fancyline,caption=,]consider better
chance baselines that are optimized for span-level evaluation by predicting
random spans rather than random individual tokens?

Figure 3: Grounding behavior saliency—as measured by probe response-level F1
on test set—of single layer probe classifiers trained on the attention-head
out projections and feed-forward states of llama-2-13b, stratified across
tasks and layers. Span-level Classification, F1-Sp
---
Method | CDM | CF | E2E
OC | 0.09 | 0.58 | 0.15
LinearSL | 0.24 | 0.65 | 0.46
LinearE | 0.27 | 0.66 | 0.47
PoolingSL | 0.32 | 0.68 | 0.46
PoolingE | 0.34 | 0.68 | 0.47
Human | 0.61 | 0.69 | 0.58
Table 5: Span-level probe hallucination detection performance, as measured via
span-level partial match scores (F1-Sp). The Human row is computed as in Table
4 and is relatively stable across tasks, unlike the systems.

###  5.3 Results

#### Response-level Results.

On two datasets, we were actually able to match the interannotator agreement
rate (the “Human” row). On these datasets, the probes trained on organic
hallucinations worked best (Table 4). The exception was the CDM dataset, where
the organic probes lost slightly to the synthetic probes, which in turn fell
well short of human performance. On all datasets, we were able to outperform
all baseline systems, showing that probing is a feasible (and efficient)
alternative to evaluating LLM outputs when model states are available.

Relative to other baselines that also have some access to model internals—such
as Seq-Logprob—we hypothesize that probing-based approaches perform better
here due to being able to capture information about model uncertainty that is
lost when reduced to the output token probability.

#### Span-Level Results.

These results appear in Table 5. Again, the “Human” row serves as a ceiling on
how well we can expect our automated system to perform on these datasets,
given that even humans do not fully agree. We observe that our ensemble
attention-pooling probe nearly meets this ceiling for Conv-FEVER but lags
behind for CDM and E2E. Still, even this level of performance is potentially
useful for avoiding hallucinations, as we discuss in § 6.

#### Layers.

Evaluating the performance of trained single-layer probes (Figure 3), we see
that signals of hallucination are strongest in model middle layers during
decoding. Probe performance shows a consistent and gradual increase from the
zeroth layer up until layer 10 to 20, after which there is a consistent
decline in hallucination classification performance for the subsequent layers.

Response-level Classification, F1-R
---
Base Model Size | CDM | CF | E2E
7B | 0.45 | 0.88 | 0.87
13B | 0.45 | 0.88 | 0.87
70B | 0.45 | 0.89 | 0.87
Span-level Classification, F1-Sp
Base Model Size | CDM | CF | E2E
7B | 0.65 | 0.77 | 0.46
13B | 0.64 | 0.77 | 0.46
70B | 0.64 | 0.76 | 0.46
Table 6: Response and span-level F1 hallucination detection performance, as
measured by F1-R and F1-Sp, respectively, of the PoolingE ensemble probe
across llama-2 response model sizes and test sets across tasks. For each task
and model size, a probe is trained on labeled responses of the same task that
were generated organically from the 13B model.

#### Hidden State Type.

Slight variations in grounding saliency—the strength of the signal of
hallucination as able to be detected by a probe, measured by probe
hallucination classification performance—are also present among hidden states
of different types during decoding. As shown in Figure 3, feed-forward probes
reach peak performance at slightly deeper layers than their attention head
out-projection probe counterparts and exceed them in single-layer
hallucination detection. Training ensemble classifiers on only the hidden
states of the same type at every layer, we see how feed-forward hidden states
possess a slight but statistically significant greater saliency of model
grounding behavior.

#### Model Size.

Training probes in the same manner on the force-decoded hidden states of
responses on our dataset using llama-2-7b and llama-2-70b, we find no
statistically significant difference (via Kolmogorov–Smirnov)
author=sky,color=orange!40,size=,fancyline,caption=,]for tacl, do paired and
reverse such that p-value test is leq 0.05 in hallucination detection across
the hidden states from base models of varying sizes (Table 6). These results
show that we can still detect a hallucination even when it was originally
generated from a different model, and that even the lower-dimensional hidden
states of the 7B model still contain enough information to train a successful
probe.

Figure 4: Intrinsic and extrinsic hallucination saliency across layers for
abstractive summarization as measured by response-level probe recall on
hallucination types.

#### Hallucination Type.

Next, we test for the saliency of different forms of hallucination behavior in
model internal states. Here, with annotator-labeled test set responses
according to whether they contained intrinsic or extrinsic hallucinations, we
evaluate trained probe
performanceauthor=jason,color=green!40,size=,fancyline,caption=,]actually only
recall; why?  sky: change to f1 in tacl in correctly detecting the presence of
hallucinations for instances of each hallucination type. We observe in Figure
4 that extrinsic hallucination behavior is more salient in the hidden states
of shallower layers for both organic and synthetic hallucinations, with
saliency noticeably peaking around layers 15 and 16, decreasing in deeper
layers. Overall, extrinsic hallucination remained more salient in model hidden
states than intrinsic hallucination.

#### Synthetic Hallucinations.

As Table 7 shows, probes achieve very high F1 in the detection of
synthetically created hallucinations across all tasks. However, probe
performance is subject to a significant drop when trained and tested on
different modalities, with probes generally performing better when trained on
organic hallucinations and tested on synthetic ungrounded responses compared
to the reverse, with the notable exception being abstractive summarization on
the CDM dataset. author=jason,color=green!40,size=,fancyline,caption=,]Did
BUMP just do a better job than we did at creating synthetic data??  sky:
possibly, but under the same taxonomy there isn’t really a good measure of
quality here Prior work has shown that synthetic data generation approaches
specifically designed for factuality evaluation Ma et al. (2023) do not align
with actual errors made by generation models in their lexical error
distributions Goyal and Durrett (2021). Here, our results show this to extend
to their hidden state signals as well, which are used by the probe to detect
unfaithfulness in the case of forced decoding on human-created content,
relative to organic instances of hallucination.

Task | Train | Test | F1-R
---|---|---|---
CDM | Organic | Organic | 0.45
| Synthetic | Organic | 0.48
| Organic | Synthetic | 0.38
| Synthetic | Synthetic | 0.93
CF | Organic | Organic | 0.88
| Synthetic | Organic | 0.65
| Organic | Synthetic | 0.83
| Synthetic | Synthetic | 0.94
E2E | Organic | Organic | 0.87
| Synthetic | Organic | 0.69
| Organic | Synthetic | 0.90
| Synthetic | Synthetic | 0.98
Table 7: Response-level PoolingE probe performance, as measured by F1-R, in
detecting hallucinations on organic or synthetic ungrounded responses.
Performance is generally better when the training data resembles the test
data.

#### Task.

To test whether the saliency of grounding behavior persists consistently in
similar hidden state locations across different tasks, we train our probe on
each specific task and evaluate its performance when applied to the others. As
shown in Table 8 and compared to same-task training, we find a lack of
generalization in hallucination detection performance for our probes across
different train and test tasks. This may mean that there is not a consistent
signal of hallucination across tasks, or that our probes are not
distinguishing this signal from its task-or-dataset-specific confounds, and
are overfitting to the confounds.

##  6 Discussion

In this work, we examined how LLM internal representations reveal their
hallucinatory behavior, via a case study of several in-context grounded
generation tasks. Our analyses point to several nuances in the saliency of
this behavior across model layers, hidden state types, model sizes,
hallucination types, and contexts. Below, we highlight a few takeaways as well
as limitations that naturally suggest directions for future work.

#### Efficiency and Access.

Our work is motivated by the need for efficient LLM hallucination evaluation.
Though computationally efficient, one of the key limitations of probing is in
the need for labeled in-domain data for probe training. Our results in Table 8
suggest poor generalization to out-of-domain tasks, although perhaps this
would be mitigated by larger and more diverse training sets. Furthermore, it
perhaps goes without saying that probing requires access to the hidden states
themselves. If LLMs continue to move behind closed-source APIs, such hidden
state probes will not be possible for those without privileged access.

Test | Train | F1-R
---|---|---
CDM | CF | 0.44
| E2E | 0.44
| CF+E2E | 0.46
| CDM | 0.45
CF | CDM | 0.60
| E2E | 0.80
| CDM+E2E | 0.75
| CF | 0.88
E2E | CDM | 0.60
| CF | 0.75
| CDM+CF | 0.76
| E2E | 0.87
Table 8: Response-level PoolingE cross-task probe performance, as measured by
F1-R. All experiments use the same amount of training data (a random subset of
the full training set). The "+" sign indicates an equal mixture of two
training tasks.

#### Ecological Validity.

The conclusions from an ecologically valid experiment should hold outside the
context of that study De Vries et al. (2020). Synthetic hallucinations are
generally ecologically invalid for organic hallucination detection. Extending
beyond surface-level error distributions Goyal and Durrett (2021), these
differences in internal representations further suggest that language model
behavior during hallucination fundamentally misaligns with current
expectations of ungrounded generations. Training on the same
modality—synthetic or organic—is imperative to maximize hallucination
detection performance, and our results on the synthetic modality, especially
on CNN / DailyMail, suggest probing to be a feasible method for complex multi-
sentence entailment classification that can leverage the capability of LLMs.
author=sky,color=orange!40,size=,fancyline,caption=,]might add a comment on
synthetic synthetic training data  ck: I’m not sure what we are claiming here.
Is there are reason we would ever practically care about accuracy of synthetic
predictions?

#### Annotator Disagreements.

Despite asking annotators to label what they deem to be the minimal spans of
hallucination in a response, considerable variation still exists between
annotators on judgments of their exact span boundaries. It remains an open
questions how to improve annotator agreement in this context.

#### Probe Design.

Even if span boundaries were standardized, they may not align perfectly with
hallucination signals in LLM internal representations. For instance, while a
generation may first contradict a given knowledge source at a certain token,
the ultimate signal that it will hallucinate may already be salient in the
tokens preceding it. Alternatively, the signal might be computed only later,
as the decoder encodes previously generated tokens and recognizes that it is
currently pursuing an ungrounded train of thought. Thus, there may be probe
architectures that could improve hallucination detection performance and
mitigation. Nonlinear probes such as feed-forward networks are also worth
considering: the hidden layer(s) of a feed-forward network could learn to
detect different types of hallucinations.

#### Mitigation.

Finally, once we can _detect_ ungrounded generation behavior, we can try to
learn to _avoid_ it. This has great practical importance as retrieval-oriented
generation becomes popular across a variety of consumer-facing applications.

We have left this subject for future work. A simple method is to reject a
response or prefix where hallucinations have been detected with high
probability, and start sampling the response again Guerreiro et al. (2023).
Another approach would be to use the predicted hallucination probabilities to
reweight or filter hypotheses during beam search. Finally, a detector can also
be used as a proxy reward method for fine-tuning the language model, using a
reward-based method such as PPO Schulman et al. (2017). In addition to
learning from response-level negative rewards incurred at the end of the
utterance, PPO can benefit from early token-level negative rewards, which
serve as a more informative signal for reinforcement learning (i.e., reward
shaping) and thus can reduce sample complexity.

## Acknowledgments

We thank Val Ramirez, Patrick Xia, Harsh Jhamtani, Hao Fang, Tongfei Chen, and
Radhika Gaonkar for their helpful comments, thoughts, and discussions, as well
as the data specialists who contributed and aided in the creation of this
work.

## References

  * Alain and Bengio (2017) Guillaume Alain and Yoshua Bengio. 2017.  Understanding intermediate layers using linear classifier probes.  In _The Fifth International Conference on Learning Representations_.
  * Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell. 2023.  The internal state of an LLM knows when it’s lying.  _arXiv preprint arXiv:2304.13734_.
  * Bishop et al. (2023) Jennifer A. Bishop, Qianqian Xie, and Sophia Ananiadou. 2023.  LongDocFACTScore: Evaluating the factuality of long document abstractive summarisation.  _arXiv preprint arXiv:2309.12455_.
  * De Vries et al. (2020) Harm De Vries, Dzmitry Bahdanau, and Christopher Manning. 2020.  Towards ecologically valid research on language user interfaces.  _arXiv preprint arXiv:2007.14435_.
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5055–5070, Online. Association for Computational Linguistics.
  * Dušek et al. (2019) Ondřej Dušek, David M. Howcroft, and Verena Rieser. 2019.  Semantic noise matters for neural natural language generation.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 421–426, Tokyo, Japan. Association for Computational Linguistics.
  * Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 1.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
  * Gabriel et al. (2021) Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021.  GO FIGURE: A meta evaluation of factuality in summarization.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 478–487, Online. Association for Computational Linguistics.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating factuality in generation with dependency-level entailment.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3592–3603, Online. Association for Computational Linguistics.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1449–1462, Online. Association for Computational Linguistics.
  * Guerreiro et al. (2023) Nuno M. Guerreiro, Elena Voita, and André Martins. 2023.  Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1059–1075, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Hanselowski et al. (2018) Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018.  UKP-athene: Multi-sentence textual entailment for claim verification.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_ , pages 103–108, Brussels, Belgium. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances In Neural Information Processing Systems_ , 28.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kryscinski et al. (2019) Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Neural text summarization: A critical evaluation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 540–551, Hong Kong, China. Association for Computational Linguistics.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022.  SummaC: Re-visiting NLI-based models for inconsistency detection in summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177.
  * Lee et al. (2018) Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018.  Hallucinations in neural machine translation.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Lin and Ng (2019) Hui Lin and Vincent Ng. 2019.  Abstractive summarization: A survey of the state of the art.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):9815–9822.
  * Liu et al. (2023) Kevin Liu, Stephen Casper, Dylan Hadfield-Menell, and Jacob Andreas. 2023.  Cognitive dissonance: Why do language model outputs disagree with internal representations of truthfulness? In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4791–4797, Singapore. Association for Computational Linguistics.
  * Luo et al. (2023) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023.  ChatGPT as a factual inconsistency evaluator for abstractive text summarization.  _arXiv preprint arXiv:2303.15621_.
  * Ma et al. (2023) Liang Ma, Shuyang Cao, Robert L Logan IV, Di Lu, Shihao Ran, Ke Zhang, Joel Tetreault, and Alejandro Jaimes. 2023.  BUMP: A benchmark of unfaithful minimal pairs for meta-evaluation of faithfulness metrics.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12788–12812, Toronto, Canada. Association for Computational Linguistics.
  * Madsen et al. (2023) Andreas Madsen, Siva Reddy, and Sarath Chandar. 2023.  Faithfulness measurable masked language models.  _arXiv preprint arXiv:2310.07819_.
  * Malinin and Gales (2021) Andrey Malinin and Mark Gales. 2021.  Uncertainty estimation in autoregressive structured prediction.  In _International Conference on Learning Representations_.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in GPT.  _Advances in Neural Information Processing Systems_ , 36.
  * Meng et al. (2023) Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023.  Mass-editing memory in a transformer.  In _The Eleventh International Conference on Learning Representations_.
  * Mielke et al. (2022) Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.  Reducing conversational agents’ overconfidence through linguistic calibration.  _Transactions of the Association for Computational Linguistics_ , 10:857–872.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  FActScore: Fine-grained atomic evaluation of factual precision in long-form text generation.  In _EMNLP_.
  * Monea et al. (2023) Giovanni Monea, Maxime Peyrard, Martin Josifoski, Vishrav Chaudhary, Jason Eisner, Emre Kıcıman, Hamid Palangi, Barun Patra, and Robert West. 2023.  A glitch in the Matrix? Locating and detecting language model grounding with Fakepedia.  _arXiv preprint arXiv:2312.02073_.
  * Ngu et al. (2023) Noel Ngu, Nathaniel Lee, and Paulo Shakarian. 2023.  Diversity measures: Domain-independent proxies for failure in language model queries.  _arXiv preprint arXiv:2308.11189_.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The E2E dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206, Saarbrücken, Germany. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Ren et al. (2023) Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2023.  Out-of-distribution detection and selective generation for conditional language models.  In _The Eleventh International Conference on Learning Representations_.
  * Santhanam et al. (2021) Sashank Santhanam, Behnam Hedayatnia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021.  Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation.  _arXiv preprint arXiv:2110.05456_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  QuestEval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Semnani et al. (2023) Sina J. Semnani, Violet Z. Yao, Heidi C. Zhang, and Monica S. Lam. 2023.  WikiChat: A few-shot LLM-based chatbot grounded with Wikipedia.  _arXiv preprint arXiv:2305.14292_.
  * Slobodkin et al. (2023) Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. 2023.  The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3607–3625, Singapore. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2020) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020.  Asking and answering questions to evaluate the factual consistency of summaries.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5008–5020, Online. Association for Computational Linguistics.
  * Xu et al. (2023) Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J. Martindale, and Marine Carpuat. 2023.  Understanding and detecting hallucinations in neural machine translation via model introspection.  _Transactions of the Association for Computational Linguistics_ , 11:546–564.

## Appendix

Here, we provide additional information about annotation guidelines, annotator
disagreement examples, prompts used to generate organic responses to our
tasks, and ChatGPT baseline comparisons.

##  Appendix A Annotation Guidelines

The purpose of this annotation task is two-fold: (1) identify if a generated
response is a hallucination relative to a given knowledge source, and (2) if
so, identify and mark the location(s) of the minimal spans of text that are
ungrounded against the knowledge source. A response is grounded in a knowledge
source if that source entails the content of the response. In other words, the
response should (1) not contradict or (2) draw conclusions that are not
explicitly given in the knowledge source.

###  A.1 CNN / DailyMail

#### Provided Information.

(1) Original article, (2) summary of article.

#### Task.

Identify the presence and location of hallucinations in the summary of the
article, relative to the original article text. You are to only annotate—by
copying the text in the summary and adding characters < and > in specific
locations—around the minimal span(s) of the summary text relative to the
original article; I.e., if changing an additional token wouldn’t change
whether or not the response was grounded or not, don’t include it in the span
you annotate. Note: You will be given summaries that contain a variety number
of sentences. You are only to annotate the first three sentences in the
summary, discarding the rest.

#### Example.

(CNN) – The world’s fastest man on a pair of skis just got faster. On Monday
Italian Simone Origone broke his own speed skiing world record as he reached
252.4 kilometers per hour on the Chabrieres slopes in the French Alps – an
achievement confirmed by organizers France Ski de Vitesse. With a 1,220 m
slope that has a maximum gradient of 98% and an average of 52.5%, Chabrieres
is not for the faint-hearted. Traveling at over 250 km/h is a privilege
usually reserved for Formula One drivers, yet speed skier Origone was equipped
with just an aerodynamic helmet to increase streamlining and a ski suit made
from air-tight latex to reduce wind resistance. Origone’s one nod to safety
was wearing a back protector in case of a crash as he threw himself down the
one kilometer track. Origone has been the fastest speed skier on the globe
since April 2006, having set a then-new world record of 251.4 km/h at Les
Arcs. Bastien Montes of France was Origone’s closest challenger on Monday, but
even the Frenhcman’s new personal best of 248.105 km/h was someway short of
the Italian’s 2006 world record, let alone his latest one. "Simone Origone is
the greatest champion of all time, he is the only person to hold the record
for two ski speeds in France – Les Arcs and Vars Chabrieres. This is a
historic day," technical director of Speed Masters Philippe Billy told
Vars.com. The 34-year-old Origone – a ski instructor, mountain guide and
rescuer by day – only took up the discipline in 2003, having given up downhill
skiing in 1999. "Now that I have twice won the world record, I can say that I
have made history," Origone told Vars.com. "It is important for me and for
speed skiing."

#### Hypothetical Instance 1.

The world’s fastest man on skis just got faster.

  * •

Everything is grounded, no need to mark anything here.

Annotation. The world’s fastest man on skis just got faster.

#### Hypothetical Instance 2.

The world’s fastest man on skis just got faster. He has now won the world
record twice. He has excelled at Chabrieres. He has been the fastest speed
skier on the globe since April 2006. His speed record reached more than 260
kilometers per hour.

  * •

This summary has more than 3 sentences, so the first thing to do is to only
look at the first three and discard any sentences beyond that in your
annotation.

  * •

Even though the last sentence is ungrounded relative to the source text, it is
not within the first three sentences, so we can ignore it here.

  * •

Annotate as such, copying the first three sentences into the annotation box.
No need to highlight anything, as everything in the first three sentences are
grounded:

Annotation. The world’s fastest man on skis just got faster. He has now won
the world record twice. He has excelled at Chabrieres.

#### Hypothetical Instance 3.

Simone Origone, now in his twenties, just got faster. He has now won the world
record more than twice.

  * •

“In his twenties” contradicts with “The 34-year-old Origone” in the original
article.

  * •

“more than twice” contradicts with how he has now only won it exactly twice.
As such, annotate:

Annotation. Simone Origone, now in his <twenties>, just got faster. He has now
won the world record <more than> twice.

#### Special Points of Note.

Below, we list a few points of special note to keep in mind during your
annotations.

#### Concatenation.

When finding yourself annotating something like “The city’s pollution levels
have reached or <exceeded ><“very high”> levels for the last…,” do concatenate
the two labeled spans; if there are no characters between them (for example,
<exceeded ><“very high”>), concatenating the annotation bounds into (<exceeded
“very high”>) is ideal.

#### “Was”/“Is”.

If the article says something like “Woods insisted he is ready to win the
famous Claret…” and the summary said something like “said he was ready to win
the famous Claret…”, “was” here would not be considered to be a hallucination,
as the summary is recounting what was said in the article, so the past tense
here doesn’t contradict what was originally said in the article.

#### When The Article Has “Contradicting” Facts.

Articles may sometimes “correct” themselves; i.e. the article may have
something where it originally states a claim but then corrects it later in the
article. For example, “The NWS – last May – initially forecast 13 to 20 “named
storms,” including seven to 11 hurricanes. Then in August, it dialed down that
initial forecast to predict 13 to 19 named storms, including six to nine
hurricanes.”

In this case, if the summary only states “The National Weather Service
forecast called for 13 to 20 named storms” (and doesn’t mention it was
corrected later to 13 to 19), then this is a hallucination as it is corrected
later in the article.

If the summary states “The National Weather Service forecast initially called
for 13 to 20 named storms” (and doesn’t mention it was corrected later to 13
to 19) then this is not a hallucination, as it quantifies the original
prediction with “initially.”

If the summary states “The National Weather Service forecast called for 13 to
19 named storms,” then this is not a hallucination, as it is the final
corrected fact.

#### Identifying the Summary.

Sometimes the summarization system may produce things that are not a summary
of the article. For example, “Woman dies while paragliding in Tenerife.
Paragliding in Tenerife. Paragliding in Tenerife. Paragliding in Tenerife.
Paragliding Tenerife Canary Islands.”

Here, only the first sentence is the summary, and you can discard the rest.
This is because the summarization system tries to mimic how actual website
summaries work, so they might also add things like topic tags, keywords, and
things like that.

Another example is “InvisiBra, £48, Lavalia, lavalia.co.uk; Strapless Stick On
Adhesive Bra, £12, Marks and Spencer; Natural Stick On Bra, £20, Debenhams.”
These are all just keyword tags; no summary of the article is present here.
Try to use your best judgment here, as it might not always be clear.

In the case where such things are generated that do not look to be part of a
summary of the article in terms of format (not content), discarding them and
everything after them is the way to go. For the former example, you would
annotate only “Woman dies while paragliding in Tenerife.” for hallucinations.

###  A.2 Conv-FEVER

#### Provided Information.

(1) Conversational context, (2) Knowledge for next response, (3) Next
response.

#### Task.

Identify the presence and location of hallucinations in “Next response”,
relative to “Knowledge for next response”. You are to only annotate—by copying
the text in “Next Reply” and adding characters < and > in specific
locations—around the minimal span(s) of an ungrounded response relative to the
knowledge; I.e., if changing an additional token wouldn’t change whether or
not the response was grounded or not, don’t include it in the span you
annotate.

#### Example.

Conversational Context: A: lets talk bowling a game where you throw a bowling
ball towards a target. B: ok I love bowling I would like to know more A: for
one it is one of the major forms of throwing sports and I think a lot of
people enjoy it B: yes me too I use to bowl all the time with my family what
about the strike? A: well a strike is when you knock down all pins on the
first round B: ok how can I get a turkey I think that may be hard to get
though Knowledge for Next Reply: Three consecutive strikes are known as a
"turkey".

#### Hypothetical Instance 1.

Next Reply: A: yea four consecutive strikes is known as a turkey

  * •

Here, a contradiction is present in the underlined portions: four consecutive
strikes in the response goes against what is provided in the knowledge, which
says three.

  * •

To make this reply grounded, one would have to change “four” to “three”.

  * •

As such, in your annotation, you will highlight the minimum needed span of
text to make this reply grounded, with chevrons <>. Here, the minimum needed
span of text to change is “four”, so you would annotate as such:

Annotation. yea <four> consecutive strikes is known as a turkey

#### Hypothetical Instance 2.

Next Reply: A: yea it is pretty hard, four consecutive strikes is known as a
turkey

  * •

Here, the contradiction is still present in the previously underlined portion:
four is a contradiction with three.

  * •

However, in addition to this contradiction, you see an expression of an
opinion in this response: that getting a turkey is pretty hard (underlined).

  * •

You are to ignore the expression of personal opinions during your annotation
of minimal spans for this task, as we won’t consider conversational
chitchat—if complemented with grounded information elsewhere in the reply—to
be a hallucination for this task. As such, you would annotate this as:

Annotation. yea it is pretty hard, <four> consecutive strikes is known as a
turkey

#### Hypothetical Instance 3.

Next Reply: A: yea it is pretty hard.

  * •

Here, the response is purely an expression of an opinion and does not use any
information in the provided knowledge.

  * •

As such, you would mark this entire response to be a hallucination, annotating
as:

Annotation. <yea it is pretty hard.>

#### Hypothetical Instance 4.

Next Reply: A: Three strikes consecutively is a turkey.

  * •

Here, the response is grounded. As such, you won’t mark anything in the
response, annotating as such:

Annotation. Three strikes consecutively is a turkey.

#### Hypothetical Instance 5.

Next Reply: A: four consecutive strikes is known as a turkey, also known as a
strike.

  * •

Two hallucinations are present in this response:

  * •

The direct contradiction of four with three in the knowledge source, and

  * •

The final part of the reply, saying that it’s also known as a strike, which is
information not provided anywhere in the knowledge source. As such, you would
mark this as:

Annotation. <four> consecutive strikes is known as a turkey, <also known as a
strike>

###  A.3 E2E

#### Provided Information.

(1) Restaurant attributes, (2) Generated restaurant description.

#### Task.

Identify the presence and location of hallucinations in the generated
restaurant description—by copying the text in the summary and adding
characters < and > in specific locations—around the minimal span(s) of the
generated restaurant description relative to the provided attributes. i.e., if
changing an additional token wouldn’t change whether the description was
grounded or not, don’t include it in the span you annotate. Note that
numerical ratings should be rendered as numerical ratings and text ratings
should be rendered as non-numerical expressions. For example, “customer
rating” can be “low”, “medium” and “high”, but also “1”, “3”, and “5”. If the
customer rating is “1”, for instance, and the generated description says “…has
a low customer rating”, that would be considered a hallucination. The
“priceRange” rating also has this attribute, in addition to “customer rating.”
Finally, E2E specifies the "familyFriendly" attribute as akin to Child-
friendly in gold annotations: if a restaurant is family-friendly, then it is
also child-friendly.

#### Hypothetical Instance 1.

Restaurant Attributes: name[The Wrestlers], priceRange[less than £20],
customer rating[low], familyFriendly[no]. Restaurant Description: The
Wrestlers is an adults only restaurant with a customer rating of 1 out of 5
and a price range of less than £20.

  * •

While the familyFriendly[no] attribute says that the restaurant is not family
friendly, this does not mean that the restaurant is adults only.

  * •

\- Having a low customer rating does not imply a 1 out of 5 customer rating.
As such, annotate as:

Annotation. The Wrestlers is an <adults only> restaurant with a customer
rating of <1 out of 5> and a price range of less than £20.

#### Hypothetical Instance 2.

Restaurant Attributes: name[The Dumpling Tree], eatType[restaurant],
food[Italian], priceRange[high]. Restaurant Description: The Dumpling Tree is
an Italian restaurant with high prices.

  * •

Everything is grounded, no need to mark anything.

Annotation. The Dumpling Tree is an Italian restaurant with high prices.

#### Hypothetical Instance 3.

Restaurant Attributes: name[The Wrestlers], priceRange[less than £20],
customer rating[low], familyFriendly[no]. Restaurant Description: Prices will
never exceed £20 at The Wrestlers and they also don’t really rate well. They
are not even family-friendly.

  * •

Having a priceRange attribute of less than 20 does not mean that prices will
never exceed twenty.

  * •

As “will never exceed” is the minimal span you would need to change to make
the description grounded relative to the provided attributes, annotate:

Annotation. Prices <will never exceed> £20 at The Wrestlers and they also
don’t really rate well. They are not even family-friendly.

#### Hypothetical Instance 4.

Restaurant Attributes: name[Midsummer House], food[French], customer rating[5
out of 5], near[Café Rouge]. Restaurant Description: A place for fine dining
French food, Midsummer House is nearby Cafe Rouge.

  * •

Nowhere in the attributes does it say that it’s “fine dining.” As such,
annotate:

Annotation. A place for <fine dining> French food, Midsummer House is nearby
Cafe Rouge.

#### Hypothetical Instance 5.

Restaurant Attributes: name[Browns Cambridge], priceRange[cheap],
priceRange[moderate], customer rating[1 out of 5]. Restaurant Description: The
Brown’s chain of food restaurants is an 1 out of 5 customer rated restaurant,
and have moderate and cheap pricing. They are childfree.

  * •

The attribute does not mention that Browns Cambridge is a chain of food
restaurants.

  * •

Furthermore, it does not say that the restaurant is childfree. As such,
annotate:

Annotation. The Brown’s <chain of food restaurants> is an 1 out of 5 customer
rated restaurant, and have moderate and cheap pricing. <They are childfree.>

#### Hypothetical Instance 6.

Restaurant Attributes: name[Browns Cambridge], priceRange[more than £30].
Restaurant Description: The Browns Cambridge is a restaurant with high prices.

  * •

More than £30 does not necessarily mean “high prices.” As such, annotate:

Annotation. The Browns Cambridge is a restaurant with <high prices>.

Taxi driver dropped off a soldier on an unsafe road, and he was killed by a
coach. The taxi driver was not at fault.
---
Taxi driver dropped off a soldier on an unsafe road, and he was killed by a
coach. The taxi driver was not at fault.
Accounting is the language of business, and it measures the results of an
organization’s economic activities.
Accounting is the language of business, and it measures the results of an
organization’s economic activities.
Alimentum is a high rated family-unfriendly restaurant with less than £20 per
person.
Alimentum is a high rated family-unfriendly restaurant with less than £20 per
person.
Table 9: Annotator span-level annotation differences for hallucination,
annotated in bold, in organic model responses for CNN / DailyMail (top), Conv-
FEVER (middle), and E2E (bottom).

##  Appendix B Annotator Disagreements

Examples of annotator disagreements in the exact spans of hallucination are
shown in Table 9. Qualitatively, we find that annotators agreed generally on
the specific items of hallucination; i.e. the subject that was hallucinated
(i.e. the taxi driver), the description of what was mentioned (i.e. the
description of accounting), and the attribute that was hallucinated (i.e. the
specific rating of the resturant). However, where exactly those hallucinated
spans began and ended was a source of reasonable variation among annotators.

##  Appendix C Prompts

#### Organic Model Responses.

CNN / DailyMail generations take the form of a 0-shot prompt with [ARTICLE]
TL;DR: [SUMMARY], whereafter the first 3 sentences generated in the summary
are taken to be the summary of the article. For Conv-FEVER, we use a 2-shot
in-context learning prompt in the form

> ## CONVERSATION HISTORY  [CONTEXT] ## KNOWLEDGE FOR NEXT RESPONSE
> [KNOWLEDGE] ## RESPONSE THAT USES THE INFORMATION IN KNOWLEDGE [RESPONSE]

whereafter everything generated in the response preceding a newline character
is taken as the final generation. For E2E, we use a 10-shot in-context
learning prompt in the form of

> Restaurant Attributes: [RESTAURANT ATTRIBUTES] Restaurant Description:
> [RESTAURANT DESCRIPTION]

with the same newline constraints as Conv-FEVER.

###  C.1 ChatGPT Prompts

#### ChatGPT Classic.

For the most basic zero-shot prompt classifier with no reasoning, we use the
following prompt for CNN / DailyMail, following Luo et al. (2023):

> Decide if the following summary is consistent with the corresponding
> article. Note that consistency means all information in the summary is
> supported by the article. Article: [Article] Summary: [Summary] Answer (yes
> or no):

For Conv-FEVER, our prompt is structured as follows:

> Decide if the following sentence (SENTENCE) is consistent with the
> corresponding knowledge statement (KNOWLEDGE). Note that consistency means
> all information in the sentence (SENTENCE) is supported by the knowledge
> statement (KNOWLEDGE). Knowledge Statement (KNOWLEDGE): [Knowledge] Sentence
> (SENTENCE): [Response] Answer (yes or no):

Finally, for E2E, our prompt is:

> Decide if the following restaurant description (DESCRIPTION) is consistent
> with the provided restaurant attributes (ATTRIBUTES). Note that consistency
> means all information in the description is supported by the provided
> attributes. Restaurant Attributes (ATTRIBUTES): [Attributes] Restaurant
> Description (DESCRIPTION): [Response] Answer (yes or no):

#### ChatGPT CoT.

For zero-shot chain-of-thought reasoning prompts, we simply replace Answer
(yes or no): in all prompts with Explain your reasoning step by step then
answer (yes or no) the question:, following Luo et al. (2023).