# Hidden Question Representations Tell Non-Factuality Within and Across Large
Language Models

Yanling Wang1, Haoyang Li2, Hao Zou1, Jing Zhang2, Xinlei He3, Qi Li1,4, Ke
Xu1,4
1Zhongguancun Laboratory, 2Renmin University of China
3Hong Kong University of Science and Technology (Guangzhou), 4Tsinghua
University
{wangyl, zouh}@zgclab.edu.cn,{lihaoyang.cs<EMAIL_ADDRESS>
<EMAIL_ADDRESS>{qli01<EMAIL_ADDRESS>

###### Abstract

Despite the remarkable advance of large language models (LLMs), the prevalence
of non-factual responses remains a common issue. This work studies non-
factuality prediction (NFP), which predicts whether an LLM will generate non-
factual responses to a question before the generation process. Previous
efforts on NFP usually rely on extensive computation. In this work, we conduct
extensive analysis to explore the capabilities of using a lightweight probe to
elicit “whether an LLM knows” from the hidden representations of questions.
Additionally, we discover that the non-factuality probe employs similar
patterns for NFP across multiple LLMs. Motivated by the intriguing finding, we
conduct effective transfer learning for cross-LLM NFP and propose a question-
aligned strategy to ensure the efficacy of mini-batch based training.

##  1 Introduction

Large language models (LLMs) have exhibited strong capabilities in
comprehending and generating meaningful text [28, 23, 14]. However, current
LLMs often fail to present queried facts [37, 5, 13, 41, 8], generating non-
factual responses that can misguide users and even lead to adverse social
impacts. Therefore, extensive studies have devoted to detecting non-factual
responses among the generated texts [22, 1, 6, 7, 10, 25], a task we name non-
factuality detection (NFD). Nevertheless, such post-hoc methods require
response generation, which lack efficiency and cannot fundamentally prevent
the occurrence of misinformation. In light of this, it is highly demanded to
predict the likelihood of an LLM generating non-factual responses before
response generation, a task we name non-factuality prediction (NFP). Figure
1(a) illustrates the difference between these two tasks.

To address the problem of NFP, some studies have explored LLMs’ awareness of
their internal knowledge. One approach, termed Self-Evaluation [15], fully
fine-tunes the LLM along with an additional value head for NFP. However, this
method incurs significant computational costs and undermines the LLM’s
generalization ability [40]. Another approach, Self-Familiarity [20], assesses
the likelihood of an LLM generating non-factual responses through multi-round
conversations with the LLM, yet it requires carefully-crafted prompts to call
the LLM multiple times, thereby suppressing its prediction efficiency.

In pursuit of more efficient training and prediction, we explore the
feasibility of utilizing a lightweight probe111A lightweight neural network
trained on hidden states. to elicit “whether an LLM knows” from the hidden
representations of questions (i.e., the LLM’s hidden states corresponding to
the input questions). Notably, a recent work [18] proposes to train a probe to
capture the relationship between question-level hidden states and the
consistency across multiple generation, as it assumes that consistent
responses to the same question indicates the LLM knows relevant factual
knowledge [22, 6, 9]. However, LLMs could generate multiple consistent
responses to a question they cannot factually answer, for example, instruction
fine-tuned LLMs like LLaMA2-Chat [35], could consistently respond “I
apologize, but I don’t have information on … ” (i.e., a type of non-factual
responses), which compromises the validity of the consistency-based
estimation. Moreover, sampling a limited number of responses cannot guarantee
the effectiveness of the consistency-based estimation [22], while sampling
more responses incurs more computational cost. Consequently, this probing-
based effort does not rigorously demonstrate the relationship between the
hidden representations of questions and the LLM’s non-factuality. Therefore,
as illustrated in Figure 1(b), (1) we aim to investigate whether hidden
representations of questions indeed contain patterns for NFP tasks. If such
patterns exist, (2) we explore whether these patterns are transferable across
multiple LLMs to achieve low-resource NFP. Note that existing NFP methods
primarily focus on training and testing within the same LLM, lacking
discussion on transferability across LLMs.

Specifically, we define the objective of NFP as predicting whether an LLM will
fail to generate accurate facts in response to questions, and introduce a
simple and effective non-factuality predictor named Factuality Lens (FacLens),
implemented by a lightweight probe trained on hidden question representations.
FacLens shows promising performance to extract information necessary for NFP
tasks from hidden question representations. Furthermore, inspired by that
individuals with similar cognitive styles and encoding strategies have similar
brain activity on a specific task [24], we study an under-explored problem —
do hidden question representations sourced from different LLMs share similar
patterns for NFP tasks? To explore this, we define a collection of hidden
question representations sourced from a specific LLM as a data domain. We
observe that a FacLens trained on data mixed from multiple domains have
comparable performance with a FacLens trained on a single domain. That
indicates there are no significant concept shifts [27] (i.e., inconsistent
conditional distributions $P(\mathbf{y}|\mathbf{X})$) between different
individual domains. Hence, given a domain with manual annotations and a domain
without manual annotations, we can perform unsupervised domain adaptation [32,
19] to effectively learn a unified cross-LLM FacLens. Moreover, we design a
question-aligned strategy to make source and target domains share the same
batch of questions in each mini-batch, better estimating the distribution
distance between source and target domains to enhance the efficacy of mini-
batch training.

Figure 1: Motivations. We explore non-factuality prediction with hidden
representations of questions, aiming to efficiently predict the likelihood of
non-factual responses before response generation. A unified FacLens can be
trained via multi-domain joint training or unsupervised domain adaptation.

Overall, this paper makes the following contributions:

  * •

We verify that solely utilizing questions’ hidden representations yields
promising performance in identifying potential non-factual responses, making
efficient training and application of NFP models possible. Besides, we conduct
extensive analysis to reveal how to better use the hidden question
representations for NFP.

  * •

We demonstrate the feasibility of transfer learning for cross-LLM NFP, which
has not been discussed in previous work. Based on this intriguing finding, we
propose to conduct unsupervised domain adaptation and design a question-
aligned mini-batch training strategy to enable effective training for the
cross-LLM NFP.

  * •

All the analysis are based on comprehensive experiments, involving three
popular open source LLMs and three widely used factual question-answering
datasets.

##  2 Related Work

Non-Factuality Prediction in LLMs. Previous efforts usually assume that an LLM
is less likely to produce non-factual responses when answering questions about
their familiar entities. Following this assumption, Adaptive Retrieval [21]
posits that LLMs are more familiar with popular entities and use Wikipedia
page views to approximate the entity popularity. However, the Wikipedia data
is only a subset of the training corpora and does not encompasses all possible
entities. Consequently, the entity popularity-based method (termed Entity-
Popularity) lacks applicability. Inspired by LLMs’ possible self-awareness,
Self-Familiarity [20] estimates an LLM’s familiarity with the requested
entities through multi-stage prompt engineering, which requires carefully-
crafted prompts to query the LLM multiple times, thereby compromising the
efficiency. Besides, some researchers fully fine-tune the LLM together with an
additional value head for NFP [15] (termed Self-Evaluation). Nevertheless,
fully fine-tuning an LLM incurs significant computational costs and inhibits
the LLM’s generalization ability. Conversely, we explore a method that
exhibits both good applicability and high efficiency. Importantly, our work is
a pioneer work that investigates transferable NFP across LLMs.

Interpretable Factuality of LLMs. Several studies have investigated the
factuality of LLMs from an interpretable perspective via examining their
hidden states. Typically, SAPLMA [1] and MIND [33] utilize hidden states
corresponding to generated content via a classifier for non-factuality
detection. INSIDE [6], on the other hand, leverages the eigenvalues of the
covariance matrix of responses’ representations to measure self-consistency,
assuming that inconsistent responses to the same question tend to be
unreliable. These methods assess factuality in a post-hoc manner,
necessitating generated responses. In contrast, FacLens assesses factuality in
an ante-hoc manner, without the need for response generation. Notably, a
linear classifier-based probe [18] is currently trained on hidden states
corresponding to the last token of each question, aiming to capture the
correlation between such hidden states and the self-consistency. However,
consistency-based methods are still limited at accurately revealing the
factuality [22, 6]. For example, instruction fine-tuned LLMs usually respond
“I apologize, but I don’t have information on …” to questions they cannot
factually answer. That is to say, LLMs could generate multiple consistent
responses to a question they cannot factually answer. Consequently, the
previous probing-based effort [18] does not rigorously reveal the relationship
between the hidden representations of questions and the LLM’s non-factuality.

##  3 Problem Definition

In this section, we formally define the problems of non-factuality detection
(NFD) and non-factuality prediction (NFP) for LLMs, showing how the two tasks
differ from each other. Subsequently, we define the problem of transferable
NFP across LLMs, which has rarely been discussed before.

###### Definition 1

Non-Factual Response. Given an LLM $m\in\mathcal{M}$ and a question
$q\in\mathcal{Q}$, $m$ generates an answer $s$ to the question. If the answer
$s$ fails to reveal the queried fact, it is a non-factual response.

###### Problem 1

Non-Factuality Detection in an LLM (Post-Hoc). Given an LLM $m\in\mathcal{M}$
and a question $q\in\mathcal{Q}$, $m$ generates an answer $s$ to the question.
The objective is to learn a function $h\left(m,q,s\right)\rightarrow y$, where
$y$ takes value of 1 if $s$ is non-factual and 0 otherwise.

###### Problem 2

Non-Factuality Prediction in an LLM (Ante-Hoc). Given an LLM $m\in\mathcal{M}$
and a question $q\in\mathcal{Q}$, the objective is to learn a function
$f\left(m,q\right)\rightarrow y$, where $y=1$ if $m$ will generate a non-
factual response to $q$ and $y=0$ otherwise.

###### Problem 3

Transferable Non-Factuality Prediction Across LLMs. Given an LLM
$m_{1}\in\mathcal{M}$, an LLM $m_{2}\in\mathcal{M}$, and a question set
$\mathcal{Q}$, human annotation has been conducted on a subset of questions
$\mathcal{Q}_{train}\subset\mathcal{Q}$ for $m_{1}$, resulting in a training
set $\\{\left(<m_{1},q_{i}>,y_{1,i}\right)\\}_{q_{i}\in\mathcal{Q}_{train}}$.
The goal is to utilize the training set and $m_{2}\in\mathcal{M}$ to learn a
function $f\left(m,q\right)\rightarrow y$, where $m\in\\{{m_{1},m_{2}}\\}$ and
$q\in\mathcal{Q}$.

##  4 Hidden Question Representations Tell Non-Factuality in LLMs

Inspired by the fact that human brain neurons are activated during problem
thinking [30], we posit that an LLM’s hidden states corresponding to the input
question contain potential patterns for identifying whether the LLM can answer
the question factually.

###  4.1 Non-Factuality Prediction (NFP) Datasets

We formally define the objective of NFP as predicting whether an LLM will fail
to generate accurate facts in response to questions. According to this, we
construct the NFP datasets as follows.

Dataset Construction. Given an LLM $m$ and a question set $\mathcal{Q}$, for
each question $q\in\mathcal{Q}$, we assign a binary label $y$ to the $<m,q>$
pair, where $y=1$ if $m$ fails to factually answer $q$ and $y=0$ otherwise.
Our goal is to predict such labels prior to answer generation. Note that it is
possible to obtain the labels by human annotators or GPT-4. However, this
process incurs high costs, and we have observed judgement errors produced by
GPT-4. Therefore, to assign the label, we follow previous work [21] to mark
the LLM’s response as factual if any sub-string of the response exactly
matches any of the gold answers. To guarantee high-quality labeling, we adopt
question-answering (QA) datasets with short answers like entity mentions222We
plan to discuss the QA datasets with long answers in the future work.. In this
way, we ensure that all positive samples are accurately labeled. Moreover, to
minimize false negative instances333We have randomly sampled 20 negative
instances from each NFP dataset, deriving $20\times 9=180$ negative instances,
and we manually checked the quality of their labels, with a true negative rate
of 95%., we exclude questions with golden answers of fewer than three
characters, as well as multiple-choice questions, since the responses of
multiple-choice questions are likely to mention both correct and incorrect
answers.

To ensure the experimental reproducibility, we set the LLM’s decoding strategy
to greedy search rather than top-$p$ or top-$k$ sampling. We also attempt the
sampling-based decoding strategies for response generation, and find that all
the experimental conclusions in this paper still hold true. In this work, we
consider three LLMs and three QA datasets, which results in $3\times 3=9$ NFP
datasets. In each NFP dataset, we randomly sample 20% instances for training,
10% instances for validation, and use the remaining instances for testing.

LLMs & QA Datasets. We conduct experiments on three widely-used open-source
LLMs: LLaMA2-7B-Chat [35], LLaMA3-8B-Instruct [23], and
Mistral-7B-Instruct-v0.2 [14]. These LLMs have been instruction-tuned for
conversational engagement. We pose questions from three widely-used QA
datasets: PopQA (PQ) [21], Entity Questions (EQ) [31], and Natural Questions
(NQ) [17]. Detailed statistics of these QA datasets are provided in Appendix
A.

###  4.2 FacLens

To explore the ability of hidden representations-based NFP, we introduce
Factuality Lens (FacLens), a learnable lightweight network to extract useful
patterns from questions’ hidden representations for NFP. Specifically, given
an LLM $m$ and an input question $q$, we can quickly acquire hidden states of
input tokens. In a certain layer, we use the hidden states corresponding to
the last input token as the question’s hidden representation $\mathbf{x}$.
Then we use an encoder $g_{{enc}}$ to transform the question’s hidden
representation into a latent feature space, where we presume that useful
patterns for NFP can be retained. Afterwards, a linear classifier $g_{{clf}}$
is set upon $g_{{enc}}$ for classification. Formally, based on the $\ell$-th
hidden layer of $m$, FacLens predicts by,

$\mathbf{p}=f\left(m_{\leq\ell},q\right)=g_{{clf}}\left(g_{{enc}}\left(m_{\leq\ell}\left(q\right)\right)\right)=g_{{clf}}\left(g_{{enc}}\left(\mathbf{x}\right)\right)$
(1)

where $m_{\leq\ell}\left(\cdot\right)$ denotes the function composed of the
$\ell$-th transformer layer and its preceding layers, $g_{{enc}}$ is
implemented by a lightweight multi-layer perceptron (MLP)444This paper aims to
verify that hidden question representations contain useful patterns for NFP
tasks. As for more complex model architectures for the pattern extraction,
they are beyond the scope of this paper’s discussion., $g_{{clf}}$ is
implemented by a linear layer with the Softmax function, and $\mathbf{p}$ is a
two-dimensional vector revealing the probability of (not) producing non-
factual responses. Based on a set of labeled NFP instances
$\\{\left(m_{\leq\ell}\left(q_{i}\right),y_{i}\right)\\}_{q_{i}\in\mathcal{Q}_{train}\cup\mathcal{Q}_{validation}}$,
we can train a FacLens for $m$ with the classic cross-entropy (CE) loss. We
discuss the effect of training data size in Appendix B.

###  4.3 Experimental Insights

We compare FacLens with existing NFP methods introduced in Section 2,
including Entity-Popularity [21], Self-Familiarity [20], and Self-Evaluation
[15]. We additionally consider a Prompting-based method, which directly asks
the LLM whether the LLM knows the factual answers to the given questions.
Inspired by using the perplexity to evaluate the factual precision of
responses [26], we consider perplexity (PPL) on the input question as a
baseline. PPL measures how well a language model (LM) predicts a given text
(see Appendix C) 555We extend the calculation of PPL to be conducted in each
layer to obtain multiple PPL values for a text. We determine the layer based
on the performance on labeled data. We find that PPL prefers the last few
layers.. Since a low PPL value suggests that an LM has likely learned relevant
texts, we can regard PPL on the input question as a non-factuality predictor.
Due to space limitations, we provide the hyper-parameter settings in Appendix
D. As the number of positive samples is larger than that of negative samples
(see Table 3), we adopt AUC as the evaluation metric. From Table 1 and Table
2, we have the following insights.

Table 1: Prediction performance of different NFP methods (AUC %).

|  LLaMA2 |  LLaMA3 | Mistral
---|---|---|---
| PQ | EQ |  NQ | PQ | EQ |  NQ | PQ | EQ | NQ
PPL  | 72.5 | 67.1 | 56.4  | 69.8 | 65.5 | 53.9  | 69.1 | 67.2 | 57.7
Prompting  | 72.7 | 67.8 | 58.1  | 70.6 | 64.9 | 57.2  | 72.2 | 66.0 | 65.5
Entity-Popularity [21] | 79.0 | – | –  | 75.9 | – | –  | 77.6 | – | –
Self-Familiarity [20] | 59.1 | 64.9 | 55.8  | 61.8 | 68.4 | 52.0  | 57.1 | 64.9 | 54.2
Self-Evaluation (Fully SFT) [15] | 88.5 | 85.2 | 68.8  | 85.7 | 85.8 | 63.9  | 83.5 | 80.9 | 61.9
FacLens-ent (last layer)  | 76.6 | 79.4 | 60.3  | 75.6 | 77.5 | 57.3  | 76.7 | 77.4 | 59.2
FacLens-ent (2${}^{\text{nd}}$ to last layer)  | 77.9 | 80.4 | 60.8  | 76.2 | 78.8 | 58.1  | 77.1 | 78.6 | 60.0
FacLens-ent (middle layer)  | 81.6 | 81.1 | 60.9  | 79.0 | 80.9 | 59.1  | 81.1 | 82.4 | 61.4
FacLens (avg, last layer)  | 85.4 | 82.7 | 62.3  | 83.8 | 81.9 | 60.8  | 85.2 | 83.2 | 60.6
FacLens (avg, 2${}^{\text{nd}}$ to last layer)  | 86.0 | 83.7 | 62.7  | 84.2 | 82.7 | 61.6  | 85.3 | 83.3 | 62.0
FacLens (avg, middle layer)  | 86.6 | 84.6 | 63.7  | 85.3 | 84.4 | 61.2  | 86.8 | 85.2 | 63.6
FacLens (last token, last layer)  | 88.7 | 84.9 | 69.1  | 86.1 | 84.1 | 64.7  | 86.1 | 84.4 | 71.7
FacLens (last token, 2${}^{\text{nd}}$ to last layer)  | 88.8 | 85.0 | 67.7  | 86.1 | 84.1 | 65.6  | 87.0 | 85.7 | 72.1
FacLens (last token, middle layer)  | 88.7 | 85.6 |  69.5 | 86.5 | 85.0 |  68.9 | 87.4 | 85.4 | 71.4

  * •

“–” means the method is not suitable for the QA dataset, as not every mention
of the subject entity in a question has an exact match with a Wikidata entity.
FacLens (avg) means using the averaged hidden states of a question’s tokens as
its hidden question representation. FacLens (last token) means using the
hidden states of the last input token as the hidden question representation.

Table 2: Efficiency evaluation of FacLens (seconds).

|  Training-Free |  Training Time Per Epoch (Avg. ) |  Prediction Time Per Question (Avg. )
---|---|---|---
Self-Familiarity  | Yes  | –  | 5.838s
Prompting  | Yes  | –  | 0.115s
PPL  | Yes  | –  | 0.044s
Self-Evaluation (Fully SFT)  | No  | 184.778s (4 * 80G A800)  | 0.028s
FacLens (last token, middle layer)  | No  | 0.012s (1 * 80G A800)  | 0.016s

  * •

Note: We extract hidden question representations used for training before the
training process. The extraction step costs only 71.856s.

Insight 1: Hidden representations of questions are useful for NFP tasks. We
observe that FacLens exhibits promising performance compared with the
baselines. For example, for LLaMA2 on PQ, FacLens (last token, middle layer)
reaches 88.7% AUC, indicating that we can indeed use a proper function to
extract information from questions’ hidden representations for NFP. Self-
Evaluation (Fully SFT) can be regarded as a special case of hidden states-
based NFP method, which also derives good performance. Prompting-based method
also takes effect, indicating that LLMs know whether they know to some extent.
Instead of relying on prompts, FacLens uses a learnable lightweight network to
better elicit “whether an LLM knows”.

Insight 2: Question-level modeling is more desirable than focusing on queried
entities. Previous efforts like Entity-Popularity and Self-Familiarity focus
on analyzing the queried entities. Following their idea, we introduce FacLens-
ent, which feeds the averaged hidden representations of these entities’ tokens
to FacLens. In specific, we use the Stanza NLP Package [12] to identify
entities like persons, locations, and organizations in a given question.
FacLens consistently outperforms FacLens-ent, implying that focusing on the
entities while overlooking the comprehension of the entire question could
misguide the predictions.

Insight 3: LLMs have known whether they know in their middle layers. An
intriguing finding is that hidden representations in the middle layer prove
more beneficial compared to that in the last few layers. Hence we conjecture
that an LLM has already known “whether it knows” in its middle layer. As LLMs
progress from the middle layer to the final layer, they may focus more on how
to better organize their knowledge for the final response.

Insight 4: FacLens stands out for its efficiency. Taking the LLaMA2-PQ NFP
dataset as an example, which comprises 2,272 questions for training, 1,136
questions for validation, and 7,952 questions for testing. Table 2 reports the
averaged training time per epoch and averaged prediction time per question of
different methods. In Table 1, Self-Evaluation (Fully SFT) shows competitive
performance. However, fully fine-tuning an LLM incurs significant
computational costs. Importantly, if the LLM has been fine-tuned for a
specific task, its ability on other tasks will be compromised [40].
Conversely, FacLens performs as a lightweight plug-in that does not change the
parameters of LLMs. As for the prediction, FacLens runs much faster than Self-
Familiarity because Self-Familiarity involves multiple conversations with the
LLM. As FacLens utilizes the middle-layer hidden representations, it runs
faster than Prompting, PPL, and Self-Evaluation which run based on more layers
in the LLM.

Insight 5: The hidden states of generated responses could be less useful than
that of questions for identifying non-factual responses. Due to space
limitations, we compare FacLens with typical hidden states-based NFD methods
in Appendix E (see Figure 8). We observe that FacLens can be a better choice
compared with hidden states-based NFD methods. One reason is that the response
generated by an LLM could contain some interference information, and thus the
corresponding hidden states may be sub-optimal for NFD. We provide more
detailed analysis in Appendix E.

###  4.4 Performance Boundary Discussion

As shown in Table 1, we notice that FacLens performs better on PQ and EQ than
on NQ. When we set PQ, EQ, and NQ to have the same training data size, the
same validation data size, and the same testing data size, such an observation
still holds true (see Appendix F). In specific, both PQ and EQ are constructed
based on templates like “What is [subj]’s occupation?” and “ Who is the author
of [subj]?”. PQ comprises 16 templates (i.e., 16 topics), while EQ involves 24
templates (i.e., 24 topics). Conversely, NQ, released by Google, consists of
questions from real users and thus covers more diverse topics. Generally, the
topic of a question is a key factor in determining its semantic information.
Here we quantify the semantic gap between training questions and testing
questions by
$1-\frac{1}{|\mathcal{Q}_{test}||\mathcal{Q}_{train}|}\sum_{q_{i}\in\mathcal{Q}_{test}}\sum_{q_{j}\in\mathcal{Q}_{train}}cos\left(\mathbf{x}_{i},\mathbf{x}_{j}\right)$,
where $cos$ denotes the function of cosine similarity, and $\mathbf{x}$
denotes a hidden question representation which contains semantic information.
As shown in Figure 2, the training and testing questions from NQ are
demonstrated to have a larger semantic gap.

Figure 2: Semantic gap between training and testing questions in each question
set. The semantic gap is calculated based on a specific LLM.

For FacLens, its input features (i.e. hidden representations of questions)
contain semantic information of the given questions. Essentially, FacLens is
trained to weaken the semantic information while preserving information
necessary for the NFP tasks. Therefore, it could perform better on questions
whose topics have been seen during training, as FacLens has learned how to
better strip away corresponding semantic information. That is why FacLens
shows better performance on PQ and EQ.

Considering the limitation caused by semantic gap between training and testing
questions, we recommend prioritizing the use of FacLens for specific fields,
which naturally narrow the scope of question topics. In future research, open-
world learning [4, 36, 34, 29, 39, 38] is worth studying for the NFP problem
to enhance the generalization of FacLens on various question topics.

User Evaluation. We recruited 15 volunteers to use our demo and rate the
performance of FacLens666For simplicity, FacLens denotes FacLens (last token,
middle layer) in the following of this paper. (3: good, 2: average, 1: poor).
The volunteers consist of 7 females and 8 males, all of whom have bachelor
degrees or higher. We received 328 de-duplicated user queries, with 202
(61.6%) receiving 3 points, 84 (25.6%) receiving 2 points, and 42 (12.8%)
receiving 1 point. In Appendix G, we show the demo interface and conduct a
case study.

##  5 Hidden Question Representations From Multiple LLMs Share Similar
Patterns for Non-Factuality Prediction

Inspired by that individuals with similar cognitive styles and encoding
strategies have similar brain activity on a specific task [24], we presume
hidden question representations from different LLMs could have similar NFP
patterns. As the generalization of FacLens is affected by the factor of
question topics, we discuss the impact of various LLMs, keeping the question
set consistent across them.

Figure 3: Performance comparison between the FacLens trained on an individual
data domain and that trained on the mixture data domain.

Figure 4: Distribution of prediction gap $\delta$ between individual-domain
FacLens and mixture-domain FacLens over all questions (including questions
from PQ, EQ, and NQ).

###  5.1 Feasibility of Cross-LLM FacLens

For convenience, we denote the hidden representation of a question in an LLM
as a variable $\mathbf{X}$. The shared patterns for cross-LLM NFP mean that
the conditional distribution $P(\mathbf{y}|\mathbf{X})$ have similar forms
across different LLMs. To be specific, we define a collection of hidden
question representations sourced from a certain LLM as a data domain $D$. The
similar conditional distributions indicate the non-existence of significant
concept shifts [27] between data domains.

We justify the above claim by introducing a mixture data domain $D_{mix}$,
whose data distribution is,

$P_{mix}(\mathbf{X},\mathbf{y})=\sum\nolimits_{i=1}^{M}\alpha_{i}\cdot
P_{m_{i}}(\mathbf{X},\mathbf{y})\quad
s.t.\quad\sum\nolimits_{i=1}^{M}\alpha_{i}=1$ (2)

where $m_{i}$ denotes the $i$-th LLM, $0<\alpha_{i}<1$ represents the
proportion of $D_{i}$ in the mixture domain, and $M$ is the number of
individual data domains (i.e. the number of LLMs). Here we set
$\alpha_{i}=\frac{1}{M}$.

Therefore, the FacLens trained on the mixture data domain should follow the
conditional distribution, whose probability density function can be written
as,

$\mathbb{P}_{mix}(\mathbf{y}|\mathbf{X})=\sum\nolimits_{i=1}^{M}\beta_{i}(\mathbf{X})\cdot\mathbb{P}_{m_{i}}(\mathbf{y}|\mathbf{X}),\quad\beta_{i}(\mathbf{X})=\frac{\alpha_{i}\cdot\mathbb{P}_{m_{i}}(\mathbf{X})}{\sum_{j=1}^{M}\alpha_{j}\cdot\mathbb{P}_{m_{j}}(\mathbf{X})}$
(3)

It is readily derived that $\sum\nolimits_{i=1}^{M}\beta_{i}(\mathbf{X})=1$,
and $0<\beta_{i}(\mathbf{X})<1$ if
$\mathbb{P}_{m_{1}}(\mathbf{X}),\mathbb{P}_{m_{2}}(\mathbf{X}),\cdots,\mathbb{P}_{m_{M}}(\mathbf{X})$
are not disjoint. Therefore, if there is no concept shifts between individual
data domains, we have,

$\mathbb{P}_{mix}\left(\mathbf{y}|\mathbf{X}\right)=\mathbb{P}_{m_{1}}\left(\mathbf{y}|\mathbf{X}\right)=\mathbb{P}_{m_{2}}\left(\mathbf{y}|\mathbf{X}\right)=\cdots=\mathbb{P}_{m_{M}}\left(\mathbf{y}|\mathbf{X}\right)$
(4)

Conversely, if significant concept shifts exist between individual domains,
Eq. 4 is not valid, and thus there must exist a domain $D_{i}$ where
$\mathbb{P}_{mix}\left(\mathbf{y}|\mathbf{X}\right)$ is very different from
$\mathbb{P}_{m_{i}}\left(\mathbf{y}|\mathbf{X}\right)$. Consequently, on the
test set of domain $D_{i}$, $f_{mix}$ trained on $D_{mix}$ will noticeably
underperform $f_{m_{i}}$ trained on $D_{i}$.

Observation. As we consider three popular LLMs, we have three individual data
domains. Each individual domain has its training, validation, and test sets.
The training sets of all the individual domains form the training set of the
mixture data domain. In Figure 3, $f_{mix}$ exhibits comparable performance to
$f_{m}$ on the test set of each individual domain, demonstrating that there is
no significant concept shifts between individual domains.

Moreover, we assess the concept shift between an individual domain and the
mixture domain by the distribution of prediction gap
$\delta=||\mathbf{p}^{m_{i}}(y=1|\mathbf{x})-\mathbf{p}^{mix}(y=1|\mathbf{x})||$,
where $\mathbf{p}^{m_{i}}$ is calculated by $f_{m_{i}}$, while
$\mathbf{p}^{mix}$ is calculated by $f_{mix}$. Figure 4 shows that the values
of $\delta$ are concentrated around zero, i.e.,
$\mathbb{P}_{m_{1}}(\mathbf{\mathbf{y}|X}),\mathbb{P}_{m_{2}}(\mathbf{\mathbf{y}|X}),\cdots,\mathbb{P}_{m_{M}}(\mathbf{\mathbf{y}|X})$
and $\mathbb{P}_{mix}(\mathbf{\mathbf{y}|X})$ are likely to have similar
forms. This constitutes the cornerstone of unsupervised domain adaptation [42,
3, 16] for cross-LLM NFP.

###  5.2 Unsupervised Domain Adaptation for Cross-LLM FacLens

Since $P_{m_{i}}(\mathbf{X})\neq P_{m_{j}}(\mathbf{X})$ and
$P_{m_{i}}(\mathbf{y}|\mathbf{X})\approx P_{m_{j}}(\mathbf{y}|\mathbf{X})$, we
formalize the transfer learning for cross-LLM FacLens on the question set
$\mathcal{Q}$ as a domain adaptation (DA) problem. The distribution shift
between two individual data domains — a source domain $D_{S}$ and a target
domain $D_{T}$, is attributed to the difference of LLMs, i.e., $m_{S}\neq
m_{T}$. The source domain has labeled training data
$\\{\left(<m_{S},q_{i}>,y_{i}^{S}\right)\\}_{q_{i}\in\mathcal{Q}_{train}}$,
while the target domain has no label information. Our goal is to learn a
unified FacLens for the two domains, leveraging knowledge learned in $D_{S}$
to improve NFP in $D_{T}$.

We adopts the classic Maximum Mean Discrepancy (MMD) loss [11] to find a
shared latent feature space for $D_{S}$ and $D_{T}$. The MMD loss measures the
distribution shift by computing the distribution distance in the Reproducing
kernel Hilbert space (RKHS) [32]. Specifically, we optimize $g_{enc}$ by
minimizing the following MMD loss to encourage domain-invariant
representations,

$\mathcal{L}_{\text{MMD}}\left(Z^{S},Z^{T}\right)=\frac{1}{N_{S}^{2}}\sum_{i=1}^{N_{S}}\sum_{j=1}^{N_{S}}k(\mathbf{z}^{S}_{i},\mathbf{z}^{S}_{j})+\frac{1}{N_{T}^{2}}\sum_{i=1}^{N_{T}}\sum_{j=1}^{N_{T}}k(\mathbf{z}^{T}_{i},\mathbf{z}^{T}_{j})-\frac{2}{N_{S}N_{T}}\sum_{i=1}^{N_{S}}\sum_{j=1}^{N_{T}}k(\mathbf{z}^{S}_{i},\mathbf{z}^{T}_{j})$
(5)

where
$\mathbf{z}_{i}^{S}=f_{enc}\left(\mathbf{x}_{i}^{S}\right),\mathbf{z}_{j}^{T}=f_{enc}\left(\mathbf{x}_{j}^{T}\right)$,
$N_{S}=N_{T}=|\mathcal{Q}_{train}|$ is the number of questions for training,
the $\ell$-th layer is set to the middle layer, and $k\left(\cdot\right)$
denotes a kernel function.

Meanwhile, we utilize the labeled data in $D_{S}$ to optimize $g_{enc}$ and
$g_{clf}$ with the CE loss,

$\mathcal{L}_{\text{DA}}\left(D_{S},D_{T}\right)=\mathcal{L}_{\text{MMD}}\left(Z^{S},Z^{T}\right)+\frac{1}{N_{S}}\sum_{i=1}^{N_{S}}\mathcal{L}_{\text{CE}}\left(g_{clf}\left(\mathbf{z}_{i}^{S}\right),y_{i}^{S}\right)$
(6)

Notably, LLMs could have distinct hidden dimensions, so we introduce a linear
layer to reshape the lower-dimensional hidden states to match the dimension of
the other LLM. We demonstrate that FacLens can transfer across LLMs of
distinct hidden dimensions in Appendix I.

###  5.3 Question-Aligned Mini-Batch Training for Cross-LLM FacLens

MMD loss involves multiplications of feature matrices, which can result in
out-of-memory issues on GPUs if there are a large number of training
instances. Therefore, we adopt mini-batch training for cross-LLM FacLens.
Conventionally, in each mini-batch, we randomly sample a batch of questions
$\mathcal{\overline{Q}}_{batch}$ and another batch of questions
$\mathcal{\widetilde{Q}}_{batch}$ from $\mathcal{Q}_{train}$ for the two
domains, respectively.

In fact, $P\left(\mathbf{X}\right)$ is associated with two factors: the LLM
and the distribution of questions. Therefore, to better estimate the distance
between $P_{S}\left(\mathbf{X}\right)$ and $P_{T}\left(\mathbf{X}\right)$
induced by different LLMs, we use the same batch of questions for $D_{S}$ and
$D_{T}$, named question-aligned mini-batch training. In other words, based on
the question-aligned strategy, we have,

$\displaystyle||d\left(P_{S}\left(\mathbf{X}\right),P_{T}\left(\mathbf{X}\right)\right)-d\left(\overline{P}_{S}\left(\mathbf{X}\right),\overline{P}_{T}\left(\mathbf{X}\right)\right)||<||d\left(P_{S}\left(\mathbf{X}\right),P_{T}\left(\mathbf{X}\right)\right)-d\left(\overline{P}_{S}\left(\mathbf{X}\right),\widetilde{P}_{T}\left(\mathbf{X}\right)\right)||$
(7)

where $d\left(\cdot\right)$ denotes the distance between two distributions,
$\overline{P}\left(\mathbf{X}\right)$ and
$\widetilde{P}\left(\mathbf{X}\right)$ denote the distributions that the
hidden representations of $\mathcal{\overline{Q}}_{batch}$ and
$\mathcal{\widetilde{Q}}_{batch}$ should follow, respectively.

(a) Results on PQ (b) Results on EQ (c) Results on NQ

Figure 5: Evaluation of cross-LLM FacLens before and after domain adaptation
(AUC). In each heatmap, the element in the $i$-th row and $j$-th column
corresponds to the $i$-th source domain and the $j$-th target domain.

Figure 6: Evaluation of the question-aligned mini-bath training strategy. The
similar trends appear on other pairs of source and target domains.

###  5.4 Experimental Insights

Insight 1: Hidden question representations sourced from different LLMs are
demonstrated to share similar patterns for NFP. As reported in Figure 5,
FacLens is transferable across LLMs via domain adaptation. Given an LLM, we
train FacLens on the training data of the corresponding domain and directly
test the trained FacLens on the test data of another domain. The results in
the upper part of Figure 5 are unsatisfactory. After domain adaptation, the
cross-LLM FacLens can well adapt to the target domain, as depicted in the the
lower part of Figure 5. Furthermore, we discuss the choice of the kernel
function in Appendix H, and find that linear kernel performs better than the
popular Gaussian kernel, indicating that features extracted by $g_{enc}$ are
inherently discriminative.

Insight 2: Question-aligned mini-batch training is quite necessary for cross-
LLM FacLens training. As shown in Figure 6, our question-aligned strategy for
mini-batch training significantly enhances the performance of cross-LLM
FacLens, demonstrating its efficacy in estimating the distance between
marginal distributions of source and target domains within randomly sampled
mini-batches. Particularly on the more diverse NQ question set where
$\mathcal{\overline{Q}}_{batch}$ and $\mathcal{\widetilde{Q}}_{batch}$ tend to
exhibit more dissimilar distributions, integrating the question-aligned
strategy fosters the training process more.

##  6 Conclusion

In this paper, we demonstrate that hidden representations of input questions
contain valuable information for identifying potential non-factual responses,
and discover that similar patterns for NFP are shared by hidden question
representations sourced from different LLMs. Moreover, we conduct extensive
analysis to show how to better use the hidden question representations to
enable efficient and transferable NFP. We hope this paper could inspire future
studies on improving NFP of LLMs.

## References

  * [1] Amos Azaria and Tom M. Mitchell.  The internal state of an LLM knows when it’s lying.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of EMNLP, pages 967–976, 2023.
  * [2] Jeanine Banks and Tris Warkentin.  Gemma: Introducing new state-of-the-art open models, 2024.
  * [3] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira.  Analysis of representations for domain adaptation.  In NeurIPS, pages 137–144, 2006.
  * [4] Kaidi Cao, Maria Brbic, and Jure Leskovec.  Open-world semi-supervised learning.  In ICLR, 2022.
  * [5] Neeloy Chakraborty, Melkior Ornik, and Katherine Rose Driggs-Campbell.  Hallucination detection in foundation models for decision-making: A flexible definition and review of the state of the art.  CoRR, abs/2403.16527, 2024.
  * [6] Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye.  INSIDE: LLMs’ internal states retain the power of hallucination detection.  In ICLR, 2024.
  * [7] Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi.  Complex claim verification with evidence retrieved in the wild.  CoRR, abs/2305.11859, 2023.
  * [8] Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, Zhixing Tan, Junwu Xiong, Xinyu Kong, Zujie Wen, Ke Xu, and Qi Li.  Risk taxonomy, mitigation, and assessment benchmarks of large language model systems.  CoRR, abs/2401.05778, 2024.
  * [9] Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu.  Halo: Estimation and reduction of hallucinations in open-source weak large language models.  CoRR, abs/2308.11764, 2023.
  * [10] Boris A Galitsky.  Truth-o-meter: Collaborating with llm in fighting its hallucinations.  2023\.
  * [11] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola.  A kernel two-sample test.  J. Mach. Learn. Res., 13:723–773, 2012.
  * [12] Stanford NLP Group.  Stanza – a python nlp package for many human languages.  https://stanfordnlp.github.io/stanza/, 2020.
  * [13] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  CoRR, abs/2311.05232, 2023.
  * [14] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b.  CoRR, abs/2310.06825, 2023.
  * [15] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan.  Language models (mostly) know what they know.  CoRR, abs/2207.05221, 2022.
  * [16] Wouter M. Kouw and Marco Loog.  A review of domain adaptation without target labels.  IEEE Trans. Pattern Anal. Mach. Intell., 43(3):766–785, 2021.
  * [17] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: a benchmark for question answering research.  Trans. Assoc. Comput. Linguistics, 7:452–466, 2019.
  * [18] Yuxin Liang, Zhuoyang Song, Hao Wang, and Jiaxing Zhang.  Learning to trust your feelings: Leveraging self-awareness in llms for hallucination mitigation.  CoRR, abs/2401.15449, 2024.
  * [19] Xiaofeng Liu, Chae Hwa Yoo, Fangxu Xing, Hyejin Oh, Georges El Fakhri, Je-Won Kang, and Jonghye Woo.  Deep unsupervised domain adaptation: A review of recent advances and perspectives.  CoRR, abs/2208.07422, 2022.
  * [20] Junyu Luo, Cao Xiao, and Fenglong Ma.  Zero-resource hallucination prevention for large language models.  CoRR, abs/2309.02654, 2023.
  * [21] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  In ACL, pages 9802–9822, 2023.
  * [22] Potsawee Manakul, Adian Liusie, and Mark J. F. Gales.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, EMNLP, pages 9004–9017, 2023.
  * [23] Meta.  Introducing meta llama 3: The most capable openly available llm to date.  https://ai.meta.com/blog/meta-llama-3/, 2024.
  * [24] Michael B Miller, Christa-Lynn Donovan, Craig M Bennett, Elissa M Aminoff, and Richard E Mayer.  Individual differences in cognitive style and strategy predict similarities in the patterns of brain activity between individuals.  Neuroimage, 59(1):83–93, 2012.
  * [25] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  CoRR, abs/2305.14251, 2023.
  * [26] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  In EMNLP, pages 12076–12100, 2023.
  * [27] Jose G. Moreno-Torres, Troy Raeder, Rocío Alaíz-Rodríguez, Nitesh V. Chawla, and Francisco Herrera.  A unifying view on dataset shift in classification.  Pattern Recognit., 45(1):521–530, 2012.
  * [28] OpenAI.  GPT-4 technical report.  CoRR, abs/2303.08774, 2023.
  * [29] Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah.  Openldn: Learning to discover novel classes for open-world semi-supervised learning.  In ECCV, volume 13691, pages 382–401, 2022.
  * [30] PE Roland and L Friberg.  Localization of cortical areas activated by thinking.  Journal of neurophysiology, 53(5):1219–1243, 1985.
  * [31] Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen.  Simple entity-centric questions challenge dense retrievers.  In EMNLP, pages 6138–6148, 2021.
  * [32] Alexander J. Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf.  A hilbert space embedding for distributions.  In ALT, volume 4754 of Lecture Notes in Computer Science, pages 13–31, 2007.
  * [33] Weihang Su, Changyue Wang, Qingyao Ai, Yiran HU, Zhijing Wu, Yujia Zhou, and Yiqun Liu.  Unsupervised real-time hallucination detection based on the internal states of large language models, 2024.
  * [34] Yiyou Sun and Yixuan Li.  Opencon: Open-world contrastive learning.  Trans. Mach. Learn. Res., 2023, 2023.
  * [35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  CoRR, abs/2307.09288, 2023.
  * [36] Sagar Vaze, Kai Hant, Andrea Vedaldi, and Andrew Zisserman.  Generalized category discovery.  In CVPR, pages 7482–7491, 2022.
  * [37] Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang.  Survey on factuality in large language models: Knowledge, retrieval and domain-specificity.  CoRR, abs/2310.07521, 2023.
  * [38] Yanling Wang, Jing Zhang, Lingxi Zhang, Lixin Liu, Yuxiao Dong, Cuiping Li, Hong Chen, and Hongzhi Yin.  Open-world semi-supervised learning for node classification.  In ICDE, pages 2723–2736, 2024.
  * [39] Xin Wen, Bingchen Zhao, and Xiaojuan Qi.  Parametric classification for generalized category discovery: A baseline study.  In ICCV, pages 16544–16554, 2023.
  * [40] Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-Ann Heng, and Wai Lam.  Unveiling the generalization power of fine-tuned large language models.  CoRR, abs/2403.09162, 2024.
  * [41] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  CoRR, abs/2309.01219, 2023.
  * [42] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  Proc. IEEE, 109(1):43–76, 2021.

##  Appendix A Statistics of QA Datasets

PQ initially contains 14,267 questions. After eliminating duplicates and
remove some special questions (see Section 4.1), 11,360 unique questions are
remained. EQ contains 100K questions. We randomly sample 7,200 questions from
EQ, ensuring uniform coverage across all question topics. After eliminating
duplicates and removing special questions, we retain 7,159 questions from EQ.
The full NQ dataset is 42Gb, so we download a simplified development set of
NQ. In this paper, we focus on the case of short answers, thus we select
questions whose answers are comprised of 30 characters or less. Consequently,
our dataset comprises 1,244 questions sourced from NQ. Table 3 shows the
positive and negative sample ratios in different NFP datasets.

Table 3: Positive and negative sample ratios in different NFP datasets (%). A NFP dataset is built based on an LLM and a QA dataset. A positive sample indicates the LLM $m$ cannot provide the queried facts in response to the question $q$, whereas a negative sample indicates the LLM $m$ can provide the queried facts in response to the question $q$. |  PQ |  EQ | NQ
---|---|---|---
|  Pos |  Neg (factual) |  Pos |  Neg (factual) |  Pos |  Neg (factual)
LLaMA2-7B-Chat  | 74.9 | 25.1  | 70.3 | 29.7  | 57.2 | 42.8
LLaMA3-8B-Instruct  | 65.5 | 34.5  | 61.6 | 38.4  | 48.2 | 51.8
Mistral-7B-Instruct-v0.2  | 73.0 | 27.0  | 68.2 | 31.8  | 55.5 | 44.5

##  Appendix B Investigation of Data Size for Training FacLens

For each NFP dataset, we randomly select 40% samples for testing, 10% samples
for validation, and use the remaining samples for training. We vary the ratio
of training data from 10% to 50%, and evaluate the performance of FacLens
under different sizes of training data. As expected, the more training data,
the better the FacLens performs, showing in Figure 7. On PQ and EQ, FacLens
already derives good performance even using 10% samples for training, and
increasing the amount of training data does not bring significant performance
improvements. On the NQ whose training and testing questions have less
overlapped topics, augmenting limited training data is insufficient for
improving the performance of FacLens. Thus, dealing with diverse questions
whose topics are rarely seen in the training data is an important future work.

Figure 7: Evaluation of FacLens under different sizes of training data.

##  Appendix C Perplexity (PPL) on an Input Question

Since a low PPL value suggests that the language model has likely learned
relevant texts, this work considers PPL as a non-factuality predictor. If the
value of PPL exceeds a certain threshold, we predict $y=1$. Formally, PPL on a
question calculated in the $\ell$-th layer is formulated as,

$\centering\text{PPL}=f_{W_{U}}\left(m_{\leq\ell},q\right)=\text{exp}\left(\frac{1}{|q|}\sum_{v_{k}\in
q}-\log\left(p_{\ell}\left(v_{k}|v_{<k}\right)\right)\right),p_{\ell}\left(v_{k}|v_{<k}\right)=\text{Softmax}\left(m_{\leq\ell}\left(v_{<k}\right)W_{U}\right)_{v_{k}}\@add@centering$
(8)

where $W_{U}$ is the pre-trained unembedding matrix of $m$ that converts the
hidden states into distributions over the vocabulary. We extend the
calculation of PPL to be conducted in each layer to obtain multiple PPL values
for a text, and determine the layer based on the performance on labeled data.

##  Appendix D Hyper-Parameter Settings

Our experiments are conducted based on 4 * 80G A800 GPUs. We implement the
encoder $g_{enc}$ of FacLens by a 3-layer MLP, setting the dimension of each
MLP layer to 256. We use the Adam optimizer with weight decay 1e-4. The hyper-
parameters determined on the validation set include: the training epochs (set
the maximum epochs to 100), the learning rate $\in$ {1e-3, 1e-4} for single-
LLM FacLens. Considering that the training questions from NQ is relatively
small, we set the learning rate of FacLens to 1e-4 on NFP datasets derived
from NQ. The default learning rate of cross-LLM FacLens is set to 1e-5. Due to
the memory limitation, we minimize the MMD loss via mini-batch training with
batch size of 64.

In terms of baselines, we adopt hyper-parameter settings recommended by their
authors. Since we extend PPL to be calculated in each hidden layer, we
determine the specific layer according to PPL’s performance on the labeled
data. We introduce the Prompting-based method, which encourages an LLM to
answer whether it knows the factual responses via prompt “Question:
{question}\nCan you provide a factual response to the above question? If you
can, please reply yes or Yes. If you can not, please reply no or No.\nAnswer:
{label}\n”. The probabilities of predicting tokens “yes”, “Yes”, “no” and “No”
are normalized for prediction. For the Self-Evaluation (Fully SFT), we train
the model on 4*80G A800 GPUs, with learning rate of 1e-6, batch size of 32,
and we also determine the training epochs based on the performance on the
validation set. Self-Evaluation (Fully SFT) needs to fully finetune an LLM.
Therefore, to mitigate overfitting, the learning rate scheduler employs a
cosine decay strategy with 5% of the training steps dedicated to linear warm-
up. Additionally, the final learning rate is set to one-tenth of its initial
value.

##  Appendix E Comparison with Hidden States-Based NFD Methods

We compare FacLens with SAPLMA [1] and INSIDE [6], two typical hidden states-
based NFD methods that work in the post-hoc manner. INSIDE is built based on
the consistency assumption, i.e., if an LLM can factually answer the given
question, it will generate multiple consistent responses. However, for the
instruction fine-tuned LLMs, it tends to respond “ I apologize, but I don’t
have information on … ” to the question it cannot factually answer. Therefore,
the consistency assumption will mislead INSIDE to perform poorly as shown in
Figure 8. In contrast, SAPLMA and FacLens exhibit competitive performance.
Similar to FacLens, we find SAPLMA also prefers hidden states at the middle
layer. Therefore, we set both FacLens and SAPLMA to use the hidden states from
the middle layer. We observe that FacLens outperforms SAPLMA on most of the
NFP datasets. That is because the response generated by the LLM could contain
some interference information, and thus the corresponding hidden states may be
sub-optimal for NFD. For example, when presented with the question "What is
the occupation of Taylor Swift?", an LLM responds with "Taylor Swift is an
American singer-songwriter and pop culture icon, born on June 13, 1989," which
correctly answers the question but contains misinformation regarding the
singer’s date of birth. In a nutshell, FacLens stands out as a good choice,
even when compared to the hidden states-based post-hoc methods.

Figure 8: Comparison between FacLens and hidden states-based NFD methods. Table 4: Evaluation of FacLens keeping the size of training/validation/testing questions the same across different question sets (AUC %). |  LLaMA2 |  LLaMA3 | Mistral
---|---|---|---
| PQ | EQ |  NQ | PQ | EQ |  NQ | PQ | EQ | NQ
FacLens  | 85.1 | 83.8 | 66.0  | 86.5 | 80.0 | 67.4  | 88.7 | 83.0 | 70.4

##  Appendix F Performance Boundary of FacLens

In Table 1, we observe that FacLens performs better on PQ and EQ than on NQ.
Notably, in Table 1, the three question sets have the different sizes of
training sets, different sizes of validation sets, and different sizes of
testing sets. Hence, in order to strictly demonstrate that FacLens performs
better on PQ and EQ, we set PQ, EQ, and NQ to have the same training data size
(300), the same validation data size (100), and the same testing data size
(600). From Table 4, we still observe that FacLens performs better on PQ and
EQ than on NQ.

##  Appendix G Demo & Case Study

Web Interface of the Demo. We have implemented a demo of FacLens, whose web
interface is shown in Figure 9. In the demo, a user can choose a specific LLM,
and then enters a question in the text box. After submitting the question, the
demo will return whether the LLM knows the factual answer. Then the user can
decide whether to call the LLM to generate the response. If the user decides
to query the LLM, the demo will provide the response generated by the LLM.
According to the prediction of FacLens and the LLM’s response, the user can
score the performance of FacLens.

Figure 9: Display of FacLens prediction results in the demo.

FacLens in the Demo. Taking LLaMA2-7B-Chat (abbreviated as LLaMA2) as the
example, we integrate its NFP datasets, i.e., LLaMA2-PQ, LLaMA2-EQ, and
LLaMA2-NQ, to train a FacLens. Specifically, we randomly sample 80% instances
for training, and use the remaining 20% instances for validation. We set the
learning rate to 1e-3, and determine the training epochs according to the
performance on the validation set. On the validation set, we use FacLens to
predict the probability $\mathbf{p}\left(y=1|\mathbf{x}\right)$ for each
instance. Then we calculate the averaged probability
$\mathbf{\overline{p}}\left(y=1|\mathbf{x}\right)_{pos}$ based on positive
instances in the validation set, as well as the averaged probability
$\mathbf{\overline{p}}\left(y=1|\mathbf{x}\right)_{neg}$ based on negative
instances in the validation set.
$\mathbf{\overline{p}}\left(y=1|\mathbf{x}\right)_{pos}$ and
$\mathbf{\overline{p}}\left(y=1|\mathbf{x}\right)_{neg}$ are used as the
thresholds for predicting whether the LLM knows the factual answers. Clearly,
given a question, FacLens predicts the probability of LLaMA2 generating non-
factual responses. If the probability is larger than
$\mathbf{\overline{p}}\left(y=1|\mathbf{x}\right)_{pos}$, the demo outputs
“The LLM doesn’t know the factual answer”; If the probability is smaller than
$\mathbf{\overline{p}}\left(y=1|\mathbf{x}\right)_{neg}$, the demo outputs
“The LLM knows the factual answer”; otherwise, the demo outputs “I am not sure
if the LLM knows the factual answer”. Similarly, we train FacLens for
LLaMA3-8B-Instruct and Mistral-7B-Instruct-v0.2, respectively.

Case Study and Future Work. Taking LLaMA2-7B-Chat as the example, we show some
good cases and bad cases in Table 5 and Table 6, respectively. We observe that
FacLens could mistakenly think the LLM knows the factual answers to the
following kinds of factual questions: (1) questions involving time
information, and (2) questions involving some reasoning. When LLMs deal with
questions involving time information, they rely on learned knowledge available
up to their training cutoff date. This limitation, along with the recurrence
of similar events, could affect the judgment of FacLens. When LLMs deal with
reasoning problems, they usually require several reasoning steps, and thus the
NFP patterns may differ from that of general factual questions. We intend to
explore the above problems in the future work.

Table 5: Good case examples. The blue text indicates that the LLM does not know the factual answer. Question |  Prediction from FacLens | Main Response from the LLM
---|---|---
Who was the screenwriter for The Toy? | The LLM doesn’t know.  |  The screenwriter for the 1982 film “The Toy” is: Tom Mankiewicz
Who is the director of TV series The Happy Family? | The LLM doesn’t know.  |  I apologize, but I don’t have access to information on the director of a specific TV series called “The Happy Family.”
Who wrote the lyrics for the song I Can Do It With A Broken Heart? | The LLM doesn’t know.  |  I apologize, but I don’t have access to the lyrics of the song “I Can Do It With A Broken Heart” as it may be a private or unreleased song.
Who founded Dicos? | The LLM doesn’t know.  |  I apologize, but I couldn’t find any information on a company called “Dicos.”
Where is capital of the United States? | The LLM knows.  |  The capital of the United States is Washington, D.C.
Who is the director of Mulan (1998 film)? | The LLM knows.  |  The director of the 1998 film “Mulan” is Barry Cook and Tony Bancroft.
Who founded KFC? | The LLM knows.  |  KFC, or Kentucky Fried Chicken, was founded by Colonel Harland Sanders.
Which is heavier, an elephant or a sheep? | The LLM knows.  | An elephant is heavier than a sheep.
Table 6: Bad case examples. The blue text indicates that the LLM does not know the factual answer. Question |  Prediction from FacLens | Main Response from the LLM
---|---|---
What is Taylor Swift’s latest album? | The LLM knows.  |  Taylor Swift’s latest album is “Lover”.
By 2024, how many Grammy Awards has Taylor Swift won? | The LLM knows.  |  As of the 2024 Grammy Awards, Taylor Swift has won a total of 10 Grammy Awards throughout her career.
Who scored more goals in 2020 in the clubs, Messi or Ronaldo? | The LLM knows.  |  In 2020, Lionel Messi scored more goals than Cristiano Ronaldo in club football.
Which team won the women’s volleyball championship at the 2004 Athens Olympics? | The LLM knows.  |  The United States women’s volleyball team won the gold medal at the 2004 Athens Olympics.
How many days are there in 2024? | The LLM knows.  | There are 365 days in the year 2024.
Who are prime numbers 2 or 6? | The LLM knows.  |  2 and 6 are not prime numbers.
If a frog gets injured and loses one of its legs, how many legs does the frog still have? | The LLM knows.  |  If a frog gets injured and loses one of its legs, the frog still has 2 legs.
If the diameter of a circle is equal to the side length of a square, which has the greater area: the circle or the square? | The LLM knows.  |  If the diameter of a circle is equal to the side length of a square, then the area of the circle is greater than the area of the square.

##  Appendix H Kernel Selection for MMD Loss

In the MMD loss, the data features are mapped into a Reproducing kernel
Hilbert space (RKHS) determined by a kernel function. Then distribution shift
between different data domains is measured within the RKHS. Here, we evaluate
two commonly employed kernel functions: the linear kernel and the Gaussian
kernel [11] (also known as the Radial Basis Function, RBF). As depicted in
Figure 10, the linear kernel tends to perform better. This suggests that the
features extracted by $g_{enc}$ for NFP tasks are already inherently
discriminative.

Figure 10: Evaluation of kernels used by MMD loss for training cross-LLM
FacLens.

##  Appendix I Transferability of FacLens Across LLMs of Distinct Hidden
Dimensions

Here we consider Gemma-7B-it [2], an LLM with different hidden dimensions from
LLaMA2-7B-Chat, LLaMA3-8B-Instruct, and Mistral-7B-Instruct-v0.2. The hidden
dimension of Gemma-7B-it is 3072, while the hidden dimension of
LLaMA2-7B-Chat, LLaMA3-8B-Instruct, and Mistral-7B-Instruct-v0.2 is 4096. A
FacLens specially trained for the source-domain LLM cannot be directly used
for a target-domain LLM whose hidden dimension is distinct from that of the
source-domain LLM. Hence we introduce a linear layer to reshape the lower-
dimensional hidden states to match the dimension of the other LLM, and still
adopt Eq. 6 to conduct domain adaptation. In Figure 11, we can see that
although two LLMs have different hidden dimensions, the cross-LLM FacLens
works well.

Figure 11: Evaluation of cross-LLM FacLens on LLMs with different hidden
dimensions. The questions are from NQ. We follow Section 4.1 to construct the
NFP datasets.

##  Appendix J Limitations and Future Work

This paper aims to explore whether we can utilize a lightweight probe to
elicit “whether an LLM knows” (i.e. NFP) from the hidden representations of
questions. Through extensive experimental analysis, we demonstrate that the
hidden representations of questions indeed contain valuable patterns for NFP
tasks, and such patterns are transferable across domains corresponding to
different LLMs. Besides, as analyzed in Section 4.4, we also find that the
non-factuality probe performs better on questions whose topics have been seen
during training, indicating that open-world learning is worth studying for the
NFP problem to enhance the probe’s generalization ability on various question
topics. We consider this an important future research direction.

As NFP is a class-imbalanced classification task, it is not easy to determine
the probability score threshold for predicting whether an LLM will generate a
non-factual response. In Appendix G, we provide a method for setting this
threshold and derive relatively good performance. However, it is important to
note that our threshold-setting method is not optimal, and we recommend
exploring better threshold-setting approaches in future research.

##  Appendix K Impacts to Society

FacLens explores the non-factuality of LLMs. The primary objective of FacLens
is inherently positive: it aims to assist people in better identifying non-
factual information potentially generated by current LLMs and enhance users’
ability to better discern inaccurate information. Moreover, FacLens will not
produce direct harmful or malicious content, further ensuring that its
application remains beneficial.


that if an entry $C[k]$ has a unique witness then we can easily find it. Then
we reduce to the unique witness case with randomization. This approach is well
known [Sei95, AGMN92], but some extra care is needed to ensure that the
instances remain monotone and bounded. The standard derandomization for this
approach [AN96] also works in our setting.

######  Lemma 40.

If BMMaxConv on length-$n$ sequences can be computed in time $T(n)$, then in
time $\widetilde{O}(T(n))$ we can compute an array $U[0,\dots,2n]$ such that
for every $k\in[2n]$ if $C[k]$ has a unique witness $M[k]$ then $U[k]=M[k]$.
The output $U[k]$ is undefined otherwise.

  * Proof.

We will describe how to compute the unique witnesses bit by bit. For each bit
position $b\in[\lceil\log n\rceil]$ let $i_{b}\in\\{0,1\\}$ be the $b$-th bit
of $i$. Construct sequences $A_{b},B_{b}$ defined as $A_{b}[i]:=2A[i]+i+i_{b}$
and $B_{b}[i]:=2B[i]+i$ for $i\in[n]$. Note that $A_{b},B_{b}$ are still
monotone non-decreasing and have entries bounded by $O(n)$. Compute
$C_{b}:=A_{b}\oplus B_{b}$. For each $k\in[2n]$, we set the $b$-th bit of
$U[k]$ to $C_{b}[k]\bmod 2$. It is not hard to see that for those entries
$C[k]$ which have unique witnesses $U[k]$, this procedure indeed gives the
$b$-th bit of $U[k]$. Indeed, note that because we double every entry in
$A_{b},B_{b}$ and add a linear function, adding $i_{b}$ does not change the
maximizer. Therefore, if $C[k]$ has a unique witness then $C^{\prime}[k]$ has
a unique witness whose $b$-th bit can be read from the least significant bit
of $C^{\prime}[k]$. Thus, by repeating this over all bit positions
$b\in[\lceil\log n\rceil]$, we compute the entire array of unique witnesses
$U$ using $O(\log n)$ invocations to BMMaxConv, as desired. ∎

Now we prove the main lemma of this section which we restate for convenience.
See 9

  * Proof of Lemma 9.

Fix a set $S\subseteq[n]$. We say that an entry $C[k]$ gets _isolated_ by $S$
if the number of witnesses of $C[k]$ in $S$ is exactly one. We will now
describe how to find the witnesses of all entries isolated by $S$ (the idea
and argument is similar as in the proof of Lemma 40). Construct sequences
$A^{\prime},B^{\prime}$ where for each $i\in[n]$ we set

$A^{\prime}[i]:=\begin{cases}2A[i]+i+1&\text{if }i\in S\\\
2A[i]+i&\text{otherwise}\end{cases}$

and $B^{\prime}[i]:=2B[i]+i$. Note that these sequences are monotone non-
decreasing and have entries bounded by $O(n)$. Let
$C^{\prime}:=A^{\prime}\oplus B^{\prime}$. We claim that if an entry $C[k]$ is
isolated by $S$, then $C^{\prime}[k]$ has a unique witness. To see this, note
that if no witness of $C[k]$ gets included in $S$, then we have that
$C^{\prime}[k]=2C[k]+k$. If at least one witness gets included in $S$, then
$C^{\prime}[k]=2C[k]+k+1$. In particular, if a witness gets isolated then
$C^{\prime}[k]$ will have a unique witness, as claimed. Thus, by Lemma 40 we
can compute in time $\widetilde{O}(T(n))$ an array $U[0,\dots,2n]$ which
contains the witnesses of all entries that are isolated by $S$. Note that some
entries of $U$ might be undefined, but we can simply check in time $O(n)$
which entries of $U$ are true witnesses, by iterating over $k\in[2n]$ and
checking whether the equality $C[k]=A[U[k]]+B[k-U[k]]$ holds.

Now we show how to select appropriate sets $S$. Fix an entry $C[k]$ and denote
by $R\in[n]$ its number of witnesses. We sample $S\subseteq[n]$ by including
each element $i\in[n]$ independently with probability $p:=2^{-\alpha}$ where
$\alpha\in\mathbb{N}$ is chosen such that $2^{\alpha-2}\leq R\leq
2^{\alpha-1}$. Let $X$ be the random variable counting the number of witnesses
of $C[k]$ that get sampled in $S$. By keeping the first two terms in the
inclusion-exclusion formula we have that $\mathop{\bf Pr\/}[X\geq 1]\geq
p\cdot R-{R\choose 2}p^{2}$, and by a union bound $\mathop{\bf Pr\/}[X\geq
2]\leq{R\choose 2}p^{2}$. Thus,

$\mathop{\bf Pr\/}[X=1]=\mathop{\bf Pr\/}[X\geq 1]-\mathop{\bf Pr\/}[X\geq
2]\geq p\cdot R(1-p\cdot R)\geq 1/8$

where the last inequality holds because $1/8\leq p\cdot R\leq 1/4$ due to the
choice of $\alpha$. In particular, $S$ isolates $C[k]$ with probability at
least $1/8$.

We now put the pieces together. Iterate over the $O(\log n)$ possible values
for $\alpha$. Sample a set $S$ and find all witnesses of entries isolated by
$S$ as described earlier in time $\widetilde{O}(T(n))$. As we argued above, if
$C[k]$ has $R$ witnesses and $2^{\alpha-2}\leq R\leq 2^{\alpha-1}$, then
$C[k]$ gets isolated with constant probability. Thus, by repeating this step
with the same $\alpha$ for $O(\log n)$ freshly sampled sets $S$ we find a
witness for all such entries $C[k]$ with probability at least
$1-1/\operatorname{poly}(n)$. Combining the results across iterations we
obtain the array of witnesses $M[0,\dots,2n]$ in time $\widetilde{O}(T(n))$,
as desired.

Finally, we note that this procedure can be derandomized with standard
techniques [AN96]. ∎