# Semantic Density: Uncertainty Quantification in Semantic Space for Large
Language Models

Xin Qiu
Cognizant AI Labs
San Francisco, USA
<EMAIL_ADDRESS>
&Risto Miikkulainen
Cognizant AI Labs, San Francisco, USA
The University of Texas at Austin, Austin, USA
<EMAIL_ADDRESS>

###### Abstract

With the widespread application of Large Language Models (LLMs) to various
domains, concerns regarding the trustworthiness of LLMs in safety-critical
scenarios have been raised, due to their unpredictable tendency to hallucinate
and generate misinformation. Existing LLMs do not have an inherent
functionality to provide the users with an uncertainty metric for each
response it generates, making it difficult to evaluate trustworthiness.
Although a number of works aim to develop uncertainty quantification methods
for LLMs, they have fundamental limitations, such as being restricted to
classification tasks, requiring additional training and data, considering only
lexical instead of semantic information, and being prompt-wise but not
response-wise. A new framework is proposed in this paper to address these
issues. Semantic density extracts uncertainty information for each response
from a probability distribution perspective in semantic space. It has no
restriction on task types and is "off-the-shelf" for new models and tasks.
Experiments on seven state-of-the-art LLMs, including the latest Llama 3 and
Mixtral-8x22B models, on four free-form question-answering benchmarks
demonstrate the superior performance and robustness of semantic density
compared to prior approaches.

##  1 Introduction

Large language models (LLMs) have revolutionalized many domains, such as
conversational agents [27], code generation [33], and mathematical discovery
[31]. Given their ability for general reasoning and adaptability to new tasks,
LLMs are utilized increasingly in safety-critical applications, including
healthcare [34] and finance [42]. However, existing LLMs have an unpredictable
tendency to hallucinate [24], leading to misleading information and risky
behaviors. Responses are generated without quantitative indicators for their
uncertainty, making it difficult to evaluate how trustworthy they are. As a
result, concerns have been raised about their safety [46], hindering a deeper
utilization of LLMs in risk-sensitive domains [4].

Although significant resources have been invested in LLM development, leading
to a rapid pace in new model releases, only little progress has been made in
building an uncertainty quantification framework for LLMs. An ideal outcome of
such a system would be a quantitative metric associated with each response
that can be used as an uncertainty/confidence indicator. Users can then build
on this metric to evaluate the trustworthiness of LLM responses, e.g.,
establish an automatic system that triggers a warning if the response
uncertainty reaches a pre-defined threshold.

Following this line of thought, several techniques have been proposed in the
literature to extract the uncertainty/confidence score from LLMs. In addition
to the baselines that directly ask the LLM itself to evaluate its own answers
[36, 15], one further step was to integrate traditional uncertainty
estimatition/calibration methods into LLMs [5, 43, 45]. However, due to the
nature of these traditional methods, they only work on classification
problems, not free-form natural language generation (NLG) tasks, which are
more general and challenging. Another direction was to fine-tune the original
model [19] or train an additional layer or classifier [3, 22] to output
uncertainty indicators for the responses. The main drawback is that these
approaches are not "off-the-shelf" for new tasks and models: additional task-
specific training labels of the ground-truth confidence are needed, and the
training needs to be done in a model-specific manner, limiting their
applicability.

Most importantly, prior work still treats LLM outputs as traditional auto-
regressive predictions [23], i.e., the generated responses are simply handled
as sequences of tokens/words, considering only their lexical uncertainty.
However, due to the unique nature of free-form NLG, tokens that are lexically
different may be semantically similar. In most LLM applications, decisions
depend on the semantics of the generated responses, and the same semantics can
be stated using different words or sentence structures, leading to different
lexical tokens. Therefore, uncertainty in semantic space is a more essential
indicator for trustworthiness of LLM responses than lexical uncertainty.

Semantic entropy [16] is the state-of-the-art (SOTA) technique in semantic
uncertainty [21]. However, its current design has two intrinsic limitations.
First, the returned uncertainty score is prompt-wise, i.e., the semantic
entropy is calculated for each prompt, instead of each response. Considering
that LLMs can generate diverse responses for the same prompt, using the same
uncertainty score for different responses is problematic [20]. Second,
semantic entropy only evaluates semantic equivalence between the generated
responses, which is a binary one-cut measurement that returns whether two
responses are semantically equivalent or not. It doesn’t take into account the
more fine-grained semantic differences between the responses, which encode
information that can make uncertainty quantification more precise.

To fill these gaps, a framework is developed in this paper for a new
uncertainty metric, semantic density (SD), that can quantify the uncertainty
of LLM responses in semantic space. Semantic density rebuilds the output
probability distribution from a semantic perspective, and extracts an
uncertainty indicator analogous to probability density. The proposed semantic
density metric has the following advantages: (1) It does not need any further
training or fine-tuning of the original LLM; it is an "off-the-shelf" tool
that can be directly applied to any pre-trained LLMs without modifying them;
(2) it does not pose any restrictions on the problem type; in particular, it
works for general free-form generation tasks (3) the returned metric is
response-wise, making it possible to evaluate trustworthiness of each
response; and (4) it takes the fine-grained semantic differences between
resposes into account, which makes uncertainty quantification more precise.

The performance of the semantic density metric was compared with six existing
uncertainty quantification methods designed for LLMs across four question-
answering benchmark datasets. All the approaches were tested on seven SOTA
LLMs, including the latest Llama 3 and Mixtral-8x22B models. Semantic density
performed significantly better than the alternatives across the board,
suggesting that it forms a promising foundation for evaluating trustworthiness
of LLM responses.

##  2 Related Work

Although the main focus of LLM community is still on developing new models
with better performance, a number of studies aim at measuring uncertainty in
LLMs. This section summarizes their basic ideas and potential limitations,
which are then targeted in the development of semantic density.

The first direction is to ask the LLM to evaluate the uncertainty of its own
responses. Tian et al. [36] performed an empirical study showing that the
inherent conditional probabilities are poorly calibrated in existing LLMs with
RLHF (reinforcement learning from human feedback), and that verbal confidence
estimates provided by the LLMs are better calibrated. Kadavath et al. [15]
developed an approach where the LLM was asked to evaluate the correctness of
its own answer, in the form of the probability "P(True)" that its own answer
is correct. An additional “value” head was also trained to predict "P(True)",
but it turned out not to generalize well to out-of-distribution datasets. In
general, the performance of model self-evaluation is not as good as other more
advanced uncertainty quantification methods [16].

A second direction is to integrate traditional uncertainty quantification
methods into LLMs. The effectiveness of temperature scaling [8] for
calibrating the output token probabilities of LLMs was verified in Desai and
Durrett [5] and Xiao et al. [43]. Ye et al. [45] utilize conformal prediction
to quantify the uncertainty of LLMs. However, these approaches are limited to
NLP classification tasks; in contrast, the proposed semantic density can be
applied to the more general and challenging free-form generation tasks.

A third direction is to perform supervised learning, either by fine-tuning the
original LLM or adding additional layer/classifier, to create uncertainty
indicators. For example Lin et al. [19] fine-tuned the GPT-3 model to verbally
express its own confidence level, and Azaria and Mitchell [3] trained an
additional classifier to return the truthful probability of each generated
response, based on the hidden layer activations of the LLM. Liu et al. [22]
proposed the LitCab framework, in which a single linear layer is trained over
the LLM’s last hidden layer representations to predict a bias term, and the
model’s logits are then altered accordingly to update the response confidence.
Since these methods are model-specific and need additional task-specific
training labels, they cannot be readily applied to new models and tasks. In
comparison, as an unsupervised method, semantic density is "off-the-shelf" for
any new models and tasks, without need of additional data or modifications to
the original LLMs.

A common limitation of all the above methods is that they only consider the
lexical information, and do not take into account the semantic relationships
between responses. Yet semantics is critical in analyzing LLM outputs. As a
fourth direction, semantic entropy [16] is a SOTA technique that quantifies
semantic uncertainty for LLMs. It works by grouping the generated samples
based on their semantic equivalence, then generating an entropy-based
indicator as an uncertainty metric. Although its performances is promising
compared to the other approaches, as discussed in Section 1, it has two
intrinsic limitations: (1) The generated semantic entropy is prompt-wise
instead of response-wise; and (2) Only a one-cut equivalence relationship is
considered during semantic analysis. The proposed semantic density improves
over these two aspects by providing a response-specific uncertainty indicator
and analyzing the relationship among generated responses in a fine-grained
semantic space.

Besides the above four major directions, uncertainty quantification for LLMs
has been explored from other angles as well. Lin et al. [20] tested several
simple baselines, among which a straightforward measurement of semantic
dispersion is robust in evaluating selective response generation. Similarly,
Manakul et al. [24] proposed a framework with several variants that use
sampling consistency for detecting hallucinations. These two studies assume a
very restricted condition in which the original sampling likelihoods of each
response are not used; thus, the uncertainty information extracted by these
methods is limited. The study by Ling et al. [21] focused on a specific in-
context learning setting, aiming to decompose the uncertainty of LLMs into
that caused by demonstration quality and that caused by model configuration.
These experiments were limited to classification problems. Similarly, Hou et
al. [10] decomposed the uncertainty into data uncertainty and model
uncertainty in a prompt-wise approach. Xiao and Wang [44] studied the
connections between hallucination and predictive uncertainty, showing that
higher uncertainty is positively correlated with higher chance to hallucinate.
This study validates the importance of a reliable uncertainty measurement in
detecting hallucinations of LLMs. Finally, Huang et al. [11] perform an
explorative study using simple baselines on uncertainty measurement for LLMs,
highlighting the need for more advanced uncertainty quantification methods
developed exclusively for LLMs. This is the goal for the current paper as
well.

##  3 Methodology

This section first defines the LLM uncertainty quantification problem, then
describes the design principles and technical details of each algorithmic
component, and concludes with a summary of the entire framework.

###  3.1 Problem Statement

Given a pre-trained LLM, an input prompt $\bm{x}$, and an output sequence
$\bm{y}=[y_{1},y_{2},\cdots,y_{L}]$, where $L$ is the number of tokens in
$\bm{y}$, the target is to produce an uncertainty metric that has a monotonic
relationship with the probability of $\bm{y}$ to be true. Note that this
metric should be response-wise, i.e., it is calculated for a specific $\bm{y}$
given $\bm{x}$. The metric can be used as a quantitative indicator for whether
a specific response $\bm{y}$ can be trusted.

###  3.2 Semantic Space

Theoretically, a semantic space can be any _metric space_ such that a distance
function is properly defined to measure the semantic similarity between any
two output responses, given the input prompt. Note that such a space is
prompt-specific, i.e., each prompt results in a specific semantic space in
which the distance function measures the contextual semantic similarity
between two responses, treating the prompt as a common context.

More concretely, an oracle semantic space $\mathbb{S}$ is assumed to be a
Euclidean space where each point is a $D$-dimensional vector that represents a
contextual embedding of response $\bm{y}$ given prompt $\bm{x}$:

$\bm{v}=E(\bm{y}|\bm{x}),$ (1)

where $\bm{v}\in\mathbb{R}^{D}$, and $E(\cdot|\cdot)$ is an encoder that
generates text embeddings with the following properties:

  1. 1.

All the generated embedding vectors are normalized to have a norm of
$\displaystyle\frac{1}{2}$:

$\displaystyle||\bm{v}||=\frac{1}{2},\ \mathrm{for}\ \bm{v}=E(\bm{y}|\bm{x}),\
\forall\bm{x},\bm{y}.$ (2)

Whereas most existing text embedding models normalize the output vectors to
have a norm of 1 [26], they are rescaled to $\frac{1}{2}$ without changing
their direction to make it simpler to integrate them into the kernel function
(as explained in Section 3.4).

  2. 2.

Given a prompt $\bm{x}$ and two resulting responses $\bm{y}_{i}$ and
$\bm{y}_{j}$, with $\bm{v}_{i}=E(\bm{y}_{i}|\bm{x})$ and
$\bm{v}_{j}=E(\bm{y}_{j}|\bm{x})$, the following constraints exist for three
extreme cases:

$||\bm{v}_{i}-\bm{v}_{j}||=\begin{cases}0,\ \mathrm{if}\ \bm{y}_{i}\
\mathrm{and}\ \bm{y}_{j}\ \mathrm{are\ semantically\ equivalent\ given\
context}\ \bm{x}\\\ \frac{\sqrt{2}}{2},\ \mathrm{if}\ \bm{y}_{i}\
\mathrm{and}\ \bm{y}_{j}\ \mathrm{are\ semantically\ irrelevant\ given\
context}\ \bm{x}\\\ 1,\ \mathrm{if}\ \bm{y}_{i}\ \mathrm{and}\ \bm{y}_{j}\
\mathrm{are\ semantically\ contradictory\ given\ context}\ \bm{x}.\end{cases}$
(3)

Given the norm requirement in Eq. 2, the above three cases also correspond to
$\bm{v}_{i}=\bm{v}_{j}$, $\bm{v}_{i}\perp\bm{v}_{j}$ and
$\bm{v}_{i}=-\bm{v}_{j}$, respectively. Note that $||\bm{v}_{i}-\bm{v}_{j}||$
is not restricted to the above three values. It can be any value within
$[0,1]$, depending on the semantic similarity between $\bm{y}_{i}$ and
$\bm{y}_{j}$ given $\bm{x}$.

  3. 3.

Given a prompt $\bm{x}$ and three resulting responses $\bm{y}_{i}$,
$\bm{y}_{j}$ and $\bm{y}_{k}$, with $\bm{v}_{i}=E(\bm{y}_{i}|\bm{x})$,
$\bm{v}_{j}=E(\bm{y}_{j}|\bm{x})$ and $\bm{v}_{k}=E(\bm{y}_{k}|\bm{x})$,

$||\bm{v}_{i}-\bm{v}_{j}||<||\bm{v}_{i}-\bm{v}_{k}||,\ \mathrm{if}\
\bm{y}_{i}\ \mathrm{is\ semantically\ closer\ to}\ \bm{y}_{j}\ \mathrm{than\
to}\ \bm{y}_{k},\ \mathrm{given\ context}\ \bm{x}.$ (4)

###  3.3 Semantic Density Estimator

Given the semantic space $\mathbb{S}$ defined in Section 3.2, the underlying
probability distribution from which the LLM samples in $\mathbb{S}$ provides
critical uncertainty information: If a response is semantically close to many
highly probable samples, it should be more trustworthy compared to a response
that is semantically distant from the major sampling possibilities. A
classical technique for estimating probability density is kernel density
estimation (KDE) [32, 28]. However, the standard KDE only works for continuous
variables, whereas the LLM outputs are discrete, i.e., sequences of tokens
selected from a finite vocabulary. One possible way to extend KDE to
accomodate LLM outputs is to build a density estimator as

$\hat{p}(\bm{y}_{*}|\bm{x})=\sum_{i=1}^{M}f_{i}K(\bm{v}_{*}-\bm{v}_{i})=\frac{1}{\sum_{i=1}^{M}n_{i}}\sum_{i=1}^{M}n_{i}K(\bm{v}_{*}-\bm{v}_{i}),$
(5)

where $\bm{x}$ is the input prompt, $\bm{y}_{*}$ is the target response, and
$\bm{v}_{*}=E(\bm{y}_{*}|\bm{x})$. In total, $\sum_{i=1}^{M}n_{i}$ reference
responses are sampled to facilitate the density estimation, where $M$ is the
number of unique samples. Each $\bm{y}_{i}$ represents a unique sample;
$n_{i}$ is the number of occurrences of $\bm{y}_{i}$ during sampling,
$f_{i}=\frac{n_{i}}{\sum_{i=1}^{M}n_{i}}$ is the relative frequency of
$\bm{y}_{i}$ during sampling, and $K(\cdot)$ is a kernel function.

The design of Eq. 5 is similar to an early variant of KDE [29] that was used
to handle integer data. However, it has the drawback that it incorporates no
knowledge about the sampling probabilities for each $\bm{y}_{i}$. It thus
requires a large number of samplings, including a sufficient number of
duplicated results, to obtain the relative frequency as an empirical
approximation of the sampling probability. This cost becomes prohibitive for
LLMs given how expensive LLM inference generally is.

In contrast with the inherently unknown probability distributions in standard
KDE, the output token probabilities can be explicitly calculated in LLM
sampling, and with this information, a more sample-efficient estimator can be
developed. Given a prompt $\bm{x}$ and a resulting response $\bm{y}_{*}$, the
semantic density of $\bm{y}_{*}$ is defined as

$\mathrm{SD}(\bm{y}_{*}|\bm{x})=\frac{1}{\sum_{i=1}^{M}p(\bm{y}_{i}|\bm{x})}\sum_{i=1}^{M}p(\bm{y}_{i}|\bm{x})K(\bm{v}_{*}-\bm{v}_{i}),$
(6)

where $\bm{v}_{*}=E(\bm{y}_{*}|\bm{x})$, and $\bm{v}_{i}=E(\bm{y}_{i}|\bm{x})$
for $i=1,2,\cdots,M$. The $M$ unique responses $\bm{y}_{i}$ are the reference
responses based on which the semantic density of $\bm{y}_{*}$ is estimated.
$K(\cdot)$ is a kernel function which will be specified in Section 3.4, and
$p(\bm{y}_{i}|\bm{x})$ (for $i=1,2,\cdots,M$) is the probability for the
original LLM to generate sequence $\bm{y}_{i}$ given $\bm{x}$. That is,
$p(\bm{y}_{i}|\bm{x})=\prod_{j=1}^{L_{i}}p(y_{i,j}|y_{i,1},y_{i,2},\cdots,y_{i,j-1},\bm{x})$,
where $L_{i}$ is the number of tokens in $\bm{y}_{i}$ and $p(y_{i,j}|\cdot)$
is the conditional probability to generate token $y_{i,j}$. Note that in cases
where $p(\bm{y}_{*}|\bm{x})$ is available, $\bm{y}_{*}$ can also be used as
one of the $M$ reference responses.

One advantage of the semantic density estimator of Eq. 6 is that each result
$\bm{y}_{i}$ only needs to be sampled once; their relative frequency $f_{i}$
can then be estimated as
$f_{i}=\frac{p(\bm{y}_{i}|\bm{x})}{\sum_{i=1}^{M}p(\bm{y}_{i}|\bm{x})}$. Given
a sampling budget of $M$ reference responses, it is therefore desirable that
these $M$ samples are unique (duplications will be removed before calculating
Eq. 6) and have high sampling probabilities, so that they can cover more
sampling regions in the semantic space. In the current implementation, diverse
beam search [38], which tends to generate diverse and high probable responses,
is used to sample the $M$ unique reference responses.

In practice, length-normalized probability [25, 23] is usually used to correct
the length bias in sequence probability. Moreover, temperatures scaling [8, 5,
43] is a simple yet effective method for calibrating the token probabilities
during sampling. Both methods can be seamlessly integrated into semantic
density: The $p(\bm{y}_{i}|\bm{x})$ in Eq. 6 can be replaced with
$\sqrt[L_{i}]{p(\bm{y}_{i}|\bm{x})}$, and the temperature changed during
sampling to calibrate each $p(y_{i,j}|\cdot)$.

###  3.4 Dimension-invariant Kernel

In a standard KDE setup, a commonly used kernel for multi-variate cases is the
Epanechnikov kernel [6, 7], which was proved to be the most efficient in terms
of asymptotic mean integrated squared error [39]. Its original form is

$K(\bm{v})=\frac{\Gamma(2+\frac{D}{2})}{\pi^{\frac{D}{2}}}(1-||\bm{v}||^{2})\bm{1}_{||\bm{v}||\leq
1},$ (7)

where $D$ is the dimension of vector $\bm{v}$, $\Gamma(\cdot)$ is the gamma
function, $||\bm{v}||$ is the 2-norm of $\bm{v}$, and
$\bm{1}_{\mathrm{condition}}$ equals 1 if the condition is true, 0 otherwise.

In the semantic density estimator use case, one drawback of the original
Epanechnikov kernel is that the normalization coefficient
$\frac{\Gamma(2+\frac{D}{2})}{\pi^{\frac{D}{2}}}$ changes with the dimension
$D$ of $\bm{v}$. As a result, semantic densities calculated using embeddings
with different dimensionalities are incomparable. This issue may limit the
flexibility in selecting embedding methodologies for semantic density
calculation. However, the normalization coefficient can be removed to make the
kernel function simpler and more flexible, without affecting the performance
of uncertainty quantification. The kernel function of Eq. 6 thus becomes

$K(\bm{v}_{*}-\bm{v}_{i})=(1-||\bm{v}_{*}-\bm{v}_{i}||^{2})\bm{1}_{||\bm{v}_{*}-\bm{v}_{i}||\leq
1}.$ (8)

Although the resulting kernel function does not meet the normalization
requirement in standard KDE, it fits uncertainty quantification well. As long
as the norm requirements in Eq. 2, 3 and 4 are fulfilled, any embedding models
can be used to generate $\bm{v}$, regardless of the embedding
dimensionalities. The outcome of kernel function is always within $[0,1]$, and
a kernel value of 1, $\frac{1}{2}$, and 0 correspond to semantically
equivalent, irrelevant and contradictory responses, respectively. As a result,
the semantic density is also within $[0,1]$, with 1 as the highest semantic
density a response can obtain, indicating that all the reference responses are
semantically equivalent to it; analogously, it obtains a semantic density of 0
when all reference responses are semantically contradictory to it. This
consistency in the value range makes practical applications of semantic
density convenient: Practitioners can set a fixed threshold on semantic
density to detect unreliable responses.

###  3.5 Semantic Distance Measurement via the Natural Language Inference
(NLI) Model

Although most of the existing text-embedding models work in the semantic space
defined in Section 3.2, they do not perform well in measuring semantic
similarities [26]. Moreover, they can only consider input texts as a whole
instead of doing a contextual inference on part of the input, i.e., they can
only obtain $E(\bm{x}+\bm{y})$ instead of $E(\bm{y}|\bm{x})$, where
$\bm{x}+\bm{y}$ means a concatenation of $\bm{x}$ and $\bm{y}$.

The natural language inference (NLI) classification model[9] has proven to be
effective in analyzing the semantic relationship between LLM responses with
the prompt as context [16]. Given a pair of texts, a NLI model performs a
classification task and outputs the probabilities for them to be semantically
equivalent ("entailment" class), irrelevant ("neutral" class), or
contradictory ("contradiction" class). Given the output class probabilities,
the expectation of $||\bm{v}_{*}-\bm{v}_{i}||$ can be obtained as

${\begin{array}[]{ll}\mathbb{E}(||\bm{v}_{*}-\bm{v}_{i}||^{2})&=1^{2}\cdot
p_{\mathrm{c}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})+(\frac{\sqrt{2}}{2})^{2}\cdot
p_{\mathrm{n}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})+0^{2}\cdot
p_{\mathrm{e}}(\bm{y}_{*},\bm{y}_{i}|\bm{x}),\\\
&=p_{\mathrm{c}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})+\frac{1}{2}\cdot
p_{\mathrm{n}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})\end{array}}$ (9)

where $p_{\mathrm{c}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})$,
$p_{\mathrm{n}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})$ and
$p_{\mathrm{e}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})$ are the probabilities for
$\bm{y}_{*}$ and $\bm{y}_{i}$ to be semantically contradictory ("c" for
"contradiction" class), irrelevant ("n" for "neutral" class) and equivalent
("e" for "entailment" class), respectively, given context $\bm{x}$. The
expected value of $||\bm{v}_{*}-\bm{v}_{i}||^{2}$ can then be used in Eq. 8 to
obtain the kernel function output.

###  3.6 Summary of the Semantic Density Framework

Algorithm 1 describes how the semantic density metric is deployed on a given
task and model. The procedure consists of four main steps, i.e., sampling the
reference responses, analyzing semantic relationships, calculating the kernel
function, and calculating the semantic density.

Algorithm 1 Procedure for deploying semantic density

0: $\bm{y}_{*}$: target response that needs uncertainty estimation $\bm{x}$:
original prompt for generating $\bm{y}_{*}$ $M$: number of unique reference
responses to be sampled given $\bm{x}$

0: $\mathrm{SD}(\bm{y}_{*}|\bm{x})$: semantic density for $\bm{y}_{*}$ given
$\bm{x}$ Step 1: Reference Response Sampling:

1: sample $M$ unique reference responses $\bm{y}_{i}$ (for $i=1,2,\cdots,M$)
with prompt $\bm{x}$ on the original LLM using diverse beam search, and record
each corresponding length-normalized sampling probability
$\sqrt[L_{i}]{p(\bm{y}_{i}|\bm{x})}$ Step 2: Semantic Relationship Analysis:

2: for $i=1\ \mathrm{to}\ M$ do

3: obtain $p_{\mathrm{c}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})$ and
$p_{\mathrm{n}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})$ using NLI classification model

4: calculate expectation
$\displaystyle\mathbb{E}(||\bm{v}_{*}-\bm{v}_{i}||^{2})=p_{\mathrm{c}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})+\frac{1}{2}\cdot
p_{\mathrm{n}}(\bm{y}_{*},\bm{y}_{i}|\bm{x})$ Step 3: Kernel Function
Calculation:

5: for $i=1\ \mathrm{to}\ M$ do

6: calculate kernel function value using the expectation of
$||\bm{v}_{*}-\bm{v}_{i}||^{2}$, given by:$\displaystyle
K(\bm{v}_{*}-\bm{v}_{i})=(1-\mathbb{E}(||\bm{v}_{*}-\bm{v}_{i}||^{2}))\bm{1}_{\mathbb{E}(||\bm{v}_{*}-\bm{v}_{i}||)\leq
1}$ Step 4: Semantic Density Calculation:

7: calculate semantic density:
$\mathrm{SD}(\bm{y}_{*}|\bm{x})=\frac{1}{\sum_{i=1}^{M}\sqrt[L_{i}]{p(\bm{y}_{i}|\bm{x})}}\sum_{i=1}^{M}\sqrt[L_{i}]{p(\bm{y}_{i}|\bm{x})}K(\bm{v}_{*}-\bm{v}_{i})$

#### Computational Cost:

In terms of computational cost, only the first two steps involve model
inferences. The first step utilizes diverse beam search, in which the group
number equals $M$ with one beam in each group, and thus only $M$ inferences
need to be done by the original LLM. The second step requires another $M$ or
$2M$ inferences by the NLI classification model, depending on whether the
relationship analysis is performed in a bi-directional manner. Considering the
fact that NLI models are usually significantly smaller than LLMs (e.g., the
Deberta-large-mnli model [9] used in the implementation in this paper only has
1.5 billion parameters), the computational cost is therefore mainly determined
by the LLM inferences in the first step.

##  4 Experiments

This section first evaluates the performance of semantic density by comparing
it with six existing uncertainty metrics over various LLMs and benchmarks.
After that, two empirical studies are performed to investigate the robustness
of semantic density when the number of reference responses and sampling
strategy for target response are varied.

###  4.1 Performance Evaluation

Table 1: Performance of different uncertainty metrics across various LLMs and
datasets CoQA
---
AUROC | SD |  SE[16] |  P(True)[15] |  Deg[20] |  NL[25] |  NE[23] |  PE[15]
Llama-2-13B | 0.783 | 0.633 | 0.594 | 0.734 | 0.709 | 0.629 | 0.647
Llama-2-70B | 0.783 | 0.621 | 0.576 | 0.721 | 0.716 | 0.617 | 0.647
Llama-3-8B | 0.738 | 0.599 | 0.593 | 0.795 | 0.676 | 0.608 | 0.604
Llama-3-70B | 0.789 | 0.608 | 0.670 | 0.729 | 0.698 | 0.587 | 0.641
Mistral-7B | 0.788 | 0.627 | 0.667 | 0.737 | 0.704 | 0.614 | 0.632
Mixtral-8x7B | 0.786 | 0.626 | 0.589 | 0.728 | 0.708 | 0.617 | 0.651
Mixtral-8x22B | 0.791 | 0.614 | 0.614 | 0.726 | 0.700 | 0.604 | 0.649
TriviaQA
AUROC | SD | SE | P(True) | Deg | NL | NE | PE
Llama-2-13B | 0.848 | 0.672 | 0.589 | 0.824 | 0.675 | 0.574 | 0.556
Llama-2-70B | 0.829 | 0.677 | 0.556 | 0.787 | 0.714 | 0.582 | 0.566
Llama-3-8B | 0.866 | 0.662 | 0.647 | 0.796 | 0.834 | 0.636 | 0.622
Llama-3-70B | 0.828 | 0.663 | 0.654 | 0.764 | 0.828 | 0.611 | 0.596
Mistral-7B | 0.866 | 0.690 | 0.589 | 0.828 | 0.745 | 0.615 | 0.536
Mixtral-8x7B | 0.846 | 0.685 | 0.562 | 0.797 | 0.795 | 0.644 | 0.605
Mixtral-8x22B | 0.829 | 0.686 | 0.604 | 0.762 | 0.801 | 0.644 | 0.607
SciQ
AUROC | SD | SE | P(True) | Deg | NL | NE | PE
Llama-2-13B | 0.757 | 0.570 | 0.572 | 0.727 | 0.693 | 0.513 | 0.574
Llama-2-70B | 0.746 | 0.643 | 0.584 | 0.713 | 0.637 | 0.554 | 0.615
Llama-3-8B | 0.780 | 0.611 | 0.564 | 0.731 | 0.686 | 0.597 | 0.651
Llama-3-70B | 0.771 | 0.613 | 0.556 | 0.706 | 0.724 | 0.558 | 0.520
Mistral-7B | 0.771 | 0.618 | 0.568 | 0.736 | 0.669 | 0.565 | 0.528
Mixtral-8x7B | 0.773 | 0.612 | 0.585 | 0.716 | 0.726 | 0.612 | 0.658
Mixtral-8x22B | 0.775 | 0.620 | 0.602 | 0.719 | 0.715 | 0.602 | 0.628
NQ
AUROC | SD | SE | P(True) | Deg | NL | NE | PE
Llama-2-13B | 0.689 | 0.581 | 0.592 | 0.686 | 0.588 | 0.571 | 0.640
Llama-2-70B | 0.676 | 0.545 | 0.531 | 0.691 | 0.567 | 0.573 | 0.620
Llama-3-8B | 0.710 | 0.583 | 0.517 | 0.706 | 0.601 | 0.603 | 0.615
Llama-3-70B | 0.723 | 0.577 | 0.643 | 0.714 | 0.631 | 0.603 | 0.615
Mistral-7B | 0.680 | 0.597 | 0.523 | 0.676 | 0.640 | 0.635 | 0.631
Mixtral-8x7B | 0.729 | 0.599 | 0.576 | 0.720 | 0.654 | 0.603 | 0.608
Mixtral-8x22B | 0.709 | 0.577 | 0.504 | 0.704 | 0.638 | 0.625 | 0.680

Following the usual evaluation approach in the literature [16], the
uncertainty metric is used as a quantitative indicator of how likely the
response is going to be correct. Uncertainty values above a threshold are
taken as incorrect while those below are taken as correct. For each threshold,
the true positive rate vs. false positive rate is then measured. The area
under this curve, namely area under receiver operator characteristic curve
(AUROC), is calculated for each uncertainty metric. The AUROC score equals the
probability that a randomly chosen incorrect response has a higher uncertainty
than a randomly chosen correct response. A perfect uncertainty metric would
have an AUROC score of 1 while a random metric would have 0.5. Note also that
a higher semantic density corresponds to a lower uncertainty.

The performance of semantic density (SD) was compared with six existing LLM
uncertainty quantification methods: semantic entropy (SE) [16], P(True) [15],
degree (Deg) [20], length-normalized likelihood (NL) [25], length-normalized
entropy (NE) [23] and predictive entropy (PE) [15]. These methods were applied
to seven SOTA open-source LLMs: Llama-2-13B [37], Llama-2-70B [37], Llama-3-8B
[2], Llama-3-70B [2], Mistral-7B [12], Mixtral-8x7B [13] and Mixtral-8x22B
[35]. Each LLM was tested on four free-form question-answering datasets
commonly used in the literature: CoQA [30], TriviaQA [14], SciQ [40] and
Natural Questions (NQ) [17]. For each question, 10 responses were generated
using group beam search and used as reference responses in calculating SD, SE,
Deg, NE and PE (note that P(True) and NL do not need reference responses).
Each unique response among these 10 will also be used as a target response,
e.g., the response that needs an uncertainty estimation, in calculating the
AUROC scores of uncertainty metrics. Detailed experimental configuration and
parametric setup is described in Appendix A.1.

Table 1 shows the AUROC scores of each uncertainty metric across different
models and datasets, with the best entry in each configuration highlighted in
boldface. SD performs best in 26 out of 28 cases, demonstrating that it is
reliable and robust as an uncertainty metric for LLM responses. In two cases
it is outperformed by Deg. After investigation, the inherent sequence
likelihood returned by the original LLM was badly calibrated in these two
cases. Deg is the only method that ignores the likelihood information during
its calculation, making its performance unaffected by this negative factor.
However, for the other 26 cases, SD is able to utilize the likelihood
information to its advantage, and outperform Deg.

To confirm that the observed differences are statistically significant, a
paired $t$-test (paired by LLM and dataset) was performed between SD and the
other metrics (Table 2 in Appendix A.2). The $p$-values are consistently below
$10^{-6}$, indicating that the performance gains of SD are strongly
statistically significant.

###  4.2 Robustness of Semantic Density

Two additional empirical studies were performed to evaluate the robustness of
semantic density when the number of reference responses varies or the sampling
strategy for target response changes.

In the first study, the number of reference responses was reduced from 10,
which is a standard setup for existing methods [16], to one, which is the
extreme minimum case. Figure 1 shows the resulting AUROC scores, covering the
same four datasets and seven LLMs. Although performance indeed decreases with
fewer reference responses, the decrease is minor as long as the number of
references is at least four. This result suggests that semantic density can
provide reasonable performance even with a very limited budget for reference
sampling.

Figure 1: AUROC scores of semantic density with one to 10 reference responses.
Each subfigure corresponds to the dataset indicated by the subfigure heading.
Each curve corresponds to the base LLM identified in the legend below the
subfigures. Providing more reference responses generally increases the
reliability of semantic density, but in most cases only four samples are
sufficient.

In real-world applications, users may have different preferences when using
LLMs to generate responses: Some may prefer a greedy sampling strategy while
others may need diverse responses. The second study thus investigated how each
uncertainty metric performs when the target response is sampled using
different such strategies. The diverse beam search method inherently utilizes
different strategies for each beam group: The first group performs a greedy
beam search while later groups encourage more diverse responses. Following the
setup in Section 4.1, the AUROC scores were calculated for target responses
from each group separately, and the results averaged over the four datasets.

As the results in Figure 2 show, semantic density exhibits consistently good
AUROC scores across different beam groups. Thus, it is robust against both
more greedy and more diverse sampling strategies, thus covering a range of
possible user preferences. In contrast, other approaches either perform
consistently worse compared to semantic density across different beam groups,
or their performance is unstable when the sampling strategy changes.

Figure 2: AUROC scores over the different beam groups. The plots show the
average AUROC scores over the four datasets for the different beam groups in
diverse beam search. Smaller group index corresponds to the group with more
greedy generation strategy, while the group with larger index tends to be more
diverse during response generation. Each subfigure corresponds to one of the
seven LLMs, as indicated by the subfigure heading. Each curve represents one
uncertainty metric indicated by the legend at lower right. Semantic density
exhibits consistently better and stable performance across different groups,
compared to other methods.

##  5 Discussion and Future work

In terms of the broader societal impact of this work, the proposed semantic
density provides a general way to evaluate the trustworthiness of responses
generated by LLMs. This ability should have a positive impact on real-world
applications that are safety-critical, such as healthcare and finance.
Practitioners can utilize semantic density as an off-the-shelf indicator to
filter out unreliable responses.

One limitation of semantic density is that it needs access to the output
probabilities of generated tokens, which may not be available in some
proprietary LLMs. In such a case, the more expensive variant in Eq. 5 can be
considered as an alternative. Since semantic density does not require any
further access to the internal states or weights of the original LLMs, it is
still widely applicable.

The framework for measuring semantic density is modular, and therefore its
main components can be extended in future work. First, new strategies that
explicitly encourage a better coverage of semantic space can be developed to
generate reference responses. Such extensions should improve the reliability
of semantic density further. Second, text embedding methods that can measure
contextual semantic similarity between responses more reliably will be helpful
as well. Third, kernel functions specifically designed for the semantic space
should allow utilizing semantic relationships more efficiently. Fourth, more
precise methods for calibrating inherent token probabilities will form a more
reliable base for calculating semantic density.

##  6 Conclusion

This paper proposes semantic density as a practical new metric for measuring
uncertainty in LLM responses. It overcomes the limitations of existing
approaches by utilizing the semantic information in an efficient and precise
manner. It is response-specific, "off-the-shelf", and applicable to free-form
generation tasks. Experimental comparisons with six existing uncertainty
metrics across seven SOTA LLMs and four benchmark datasets suggest that it is
accurate, robust, and general, and can therefore help deploy LLMs in safety-
critical domains.

## References

  * [1]
  * AI@Meta [2024] AI@Meta. 2024.  Llama 3 Model Card.  https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
  * Azaria and Mitchell [2023] Amos Azaria and Tom Mitchell. 2023.  The Internal State of an LLM Knows When It’s Lying. In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 967–976.  https://doi.org/10.18653/v1/2023.findings-emnlp.68
  * Clusmann et al. [2023] Jan Clusmann, Fiona R. Kolbinger, Hannah Sophie Muti, Zunamys I. Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, Sophie-Caroline Schwarzkopf, Michaela Unger, Gregory P. Veldhuizen, Sophia J. Wagner, and Jakob Nikolas Kather. 2023.  The future landscape of large language models in medicine.  _Communications Medicine_ 3, 1 (2023), 141\.  https://doi.org/10.1038/s43856-023-00370-1
  * Desai and Durrett [2020] Shrey Desai and Greg Durrett. 2020.  Calibration of Pre-trained Transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 295–302.  https://doi.org/10.18653/v1/2020.emnlp-main.21
  * Epanechnikov [1969] V. A. Epanechnikov. 1969\.  Non-Parametric Estimation of a Multivariate Probability Density.  _Theory of Probability & Its Applications_ 14, 1 (1969), 153–158.  https://doi.org/10.1137/1114019 arXiv:https://doi.org/10.1137/1114019
  * Fukunaga and Hostetler [1975] Keinosuke Fukunaga and Larry D. Hostetler. 1975.  The estimation of the gradient of a density function, with applications in pattern recognition.  _IEEE Trans. Inf. Theory_ 21 (1975), 32–40.  https://api.semanticscholar.org/CorpusID:15299210
  * Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017\.  On calibration of modern neural networks. In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ (Sydney, NSW, Australia) _(ICML’17)_. JMLR.org, 1321–1330.
  * He et al. [2021] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021\.  DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=XPZIaotutsD
  * Hou et al. [2023] Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2023\.  Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling.  arXiv:2311.08718 [cs.CL]
  * Huang et al. [2023] Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. 2023.  Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models.  arXiv:2307.10236 [cs.SE]
  * Jiang et al. [2023] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7B.  arXiv:2310.06825 [cs.CL]
  * Jiang et al. [2024] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of Experts.  arXiv:2401.04088 [cs.LG]
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, Vancouver, Canada, 1601–1611.  https://doi.org/10.18653/v1/P17-1147
  * Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022\.  Language Models (Mostly) Know What They Know.  arXiv:2207.05221 [cs.CL]
  * Kuhn et al. [2023] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation. In _The Eleventh International Conference on Learning Representations_.  https://openreview.net/forum?id=VD-AYtP0dve
  * Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a Benchmark for Question Answering Research.  _Transactions of the Association of Computational Linguistics_ (2019).
  * Lin and Och [2004] Chin-Yew Lin and Franz Josef Och. 2004.  Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics. In _Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04)_. Barcelona, Spain, 605–612.  https://doi.org/10.3115/1218955.1219032
  * Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022\.  Teaching Models to Express Their Uncertainty in Words.  _Transactions on Machine Learning Research_ (2022).  https://openreview.net/forum?id=8s8K2UZGTZ
  * Lin et al. [2023] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023\.  Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models.  arXiv:2305.19187 [cs.CL]
  * Ling et al. [2024] Chen Ling, Xujiang Zhao, Wei Cheng, Yanchi Liu, Yiyou Sun, Xuchao Zhang, Mika Oishi, Takao Osaki, Katsushi Matsuda, Jie Ji, Guangji Bai, Liang Zhao, and Haifeng Chen. 2024.  Uncertainty Decomposition and Quantification for In-Context Learning of Large Language Models. In _2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics_.  https://openreview.net/forum?id=Oq1b1DnUOP
  * Liu et al. [2024] Xin Liu, Muhammad Khalifa, and Lu Wang. 2024.  LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses. In _The Twelfth International Conference on Learning Representations_.  https://openreview.net/forum?id=jH67LHVOIO
  * Malinin and Gales [2021] Andrey Malinin and Mark Gales. 2021.  Uncertainty Estimation in Autoregressive Structured Prediction. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=jN5y-zb5Q7m
  * Manakul et al. [2023] Potsawee Manakul, Adian Liusie, and Mark Gales. 2023\.  SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9004–9017.  https://doi.org/10.18653/v1/2023.emnlp-main.557
  * Murray and Chiang [2018] Kenton Murray and David Chiang. 2018.  Correcting Length Bias in Neural Machine Translation. In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (Eds.). Association for Computational Linguistics, Brussels, Belgium, 212–223.  https://doi.org/10.18653/v1/W18-6322
  * Neelakantan et al. [2022] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. 2022.  Text and Code Embeddings by Contrastive Pre-Training.  arXiv:2201.10005 [cs.CL]
  * OpenAI et al. [2024] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024\.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Parzen [1962] Emanuel Parzen. 1962\.  On Estimation of a Probability Density Function and Mode.  _The Annals of Mathematical Statistics_ 33, 3 (1962), 1065 – 1076.  https://doi.org/10.1214/aoms/1177704472
  * Rajagopalan and Lall [1995] Balaji Rajagopalan and Upmanu Lall. 1995.  A kernel estimator for discrete distributions.  _Journal of Nonparametric Statistics_ 4, 4 (1995), 409–426.  https://doi.org/10.1080/10485259508832629 arXiv:https://doi.org/10.1080/10485259508832629
  * Reddy et al. [2019] Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019\.  CoQA: A Conversational Question Answering Challenge.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 249–266.  https://doi.org/10.1162/tacl_a_00266
  * Romera-Paredes et al. [2024] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2024.  Mathematical discoveries from program search with large language models.  _Nature_ 625, 7995 (2024), 468–475.  https://doi.org/10.1038/s41586-023-06924-6
  * Rosenblatt [1956] Murray Rosenblatt. 1956\.  Remarks on Some Nonparametric Estimates of a Density Function.  _The Annals of Mathematical Statistics_ 27, 3 (1956), 832 – 837.  https://doi.org/10.1214/aoms/1177728190
  * Rozière et al. [2024] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024\.  Code Llama: Open Foundation Models for Code.  arXiv:2308.12950 [cs.CL]
  * Singhal et al. [2023] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. 2023.  Towards Expert-Level Medical Question Answering with Large Language Models.  arXiv:2305.09617 [cs.CL]
  * team [2024] Mistral AI team. 2024\.  https://mistral.ai/news/mixtral-8x22b/
  * Tian et al. [2023] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023\.  Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 5433–5442.  https://doi.org/10.18653/v1/2023.emnlp-main.330
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023\.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  arXiv:2307.09288 [cs.CL]
  * Vijayakumar et al. [2018] Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018\.  Diverse Beam Search for Improved Description of Complex Scenes.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 32, 1 (Apr. 2018).  https://doi.org/10.1609/aaai.v32i1.12340
  * Wand and Jones [1994] M.P. Wand and M.C. Jones. 1994.  _Kernel Smoothing_.  Taylor & Francis.  https://books.google.com/books?id=GTOOi5yE008C
  * Welbl et al. [2017] Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017.  Crowdsourcing Multiple Choice Science Questions.  arXiv:1707.06209 [cs.HC]
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Online, 38–45.  https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * Wu et al. [2023] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023.  BloombergGPT: A Large Language Model for Finance.  arXiv:2303.17564 [cs.LG]
  * Xiao et al. [2022] Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022.  Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis. In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 7273–7284.  https://doi.org/10.18653/v1/2022.findings-emnlp.538
  * Xiao and Wang [2021] Yijun Xiao and William Yang Wang. 2021.  On Hallucination and Predictive Uncertainty in Conditional Language Generation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 2734–2744.  https://doi.org/10.18653/v1/2021.eacl-main.236
  * Ye et al. [2024] Fanghua Ye, Yang MingMing, Jianhui Pang, Longyue Wang, Derek F Wong, Yilmaz Emine, Shuming Shi, and Zhaopeng Tu. 2024\.  Benchmarking LLMs via Uncertainty Quantification.  _arXiv preprint arXiv:2401.12794_ (2024).
  * Zhao et al. [2023] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023\.  A Survey of Large Language Models.  arXiv:2303.18223 [cs.CL]

##  Appendix A Appendix

###  A.1 Experimental Setup

This section provides the detailes of experimental setup for reproducing the
results presented in Section 4.

#### Base LLMs:

For all the seven base LLMs, the open-source versions in Huggingface
Transformers library [41] were used in the experiments. More specifically, the
following versions are used: meta-llama/Llama-2-13b-hf, meta-
llama/Llama-2-70b-hf, meta-llama/Meta-Llama-3-8B, meta-llama/Meta-Llama-3-70B,
mistralai/Mistral-7B
-v0.1, mistralai/Mixtral-8x7B-v0.1, and mistralai/Mixtral-8x22B-v0.1. When running diverse beam search in generating the responses, the generate() function is used with diversity_penalty=1.0, a default temperature of 1.0, and the num_beam_groups equals the number of beams so that each group has exactly one beam.

#### Datasets:

Following the setup in Kuhn et al. [16], for all the datasets, an answer is
considered to be correct if its Rouge-L [18] to any of the reference answers
is larger than 0.3, after trimming the redundant continuations. The details of
the datasets are as follows:

  * •

CoQA: The coqa-dev-v1.0 version is used with 1596 questions randomly selected
for the experiments, using the Huggingface datasets.train_test_split function
with a seed value of 10. The prompt format follows the same setup as in Kuhn
et al. [16].

  * •

TriviaQA: The dataset is loaded with the Huggingface datasets API. 1705
questions were randomly selected from the validation split for the
experiments, using the Huggingface datasets.train_test_split function with a
seed value of 10. A 10-shot prompt format is used following the setup in Kuhn
et al. [16].

  * •

SciQ: The test split from https://github.com/launchnlp/LitCab is used in the
experiments. It contains 990 questions. A 10-shot prompt format is used
following the setup in Kuhn et al. [16].

  * •

NQ: The test split from https://github.com/launchnlp/LitCab is used in the
experiments. 1800 questions were randomly selected using the Huggingface
datasets.train_test_
split function with a seed value of 10. A 10-shot prompt format is used
following the setup in Kuhn et al. [16].

#### Uncertainty Metrics:

The exactly same target response and reference responses are used for all the
tested methods. For SD, SE and Deg, the microsoft/deberta-large-mnli model
from Huggingface Transformers library [41] is used as the NLI classification
model. Following Kuhn et al. [16], the probabilities for "contradiction",
"neutral" and "entailment" are averaged bidirectionally. The parametric setup
for the uncertainty metrics tested in Section 4 is summarized as follows:

  * •

semantic density (SD): The exactly same steps as described by Algorithm 1 were
implemented, with a fixed temperature of 0.1 applied to rescale the value of
each token probability during postprocessing.

  * •

semantic entropy (SE): The original implementation and parametric setup from
https://github.com/lorenzkuhn/semantic_uncertainty was used.

  * •

P(True): The original few-shot prompt format from Kadavath et al. [15] is
used.

  * •

degree (Deg): The "entailment" probability returned by the NLI model (averaged
bidirectionally) is used as the similarity between two responses, which is
then used as the diagonal element in degree matrix.

  * •

length-normalized likelihood (NL): The original form as in Murray and Chiang
[25] was implemented.

  * •

length-normalized entropy (NE): The implementation from
https://github.com/lorenzkuhn
/semantic_uncertainty was used.

  * •

predictive entropy (PE): The implementation from
https://github.com/lorenzkuhn/semantic
_uncertainty was used.

#### Compute Resources:

All the experiments in Section 4 were running on an AWS P4de instance with 96
Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz, 1152GB memory and 8 NVIDIA
A100 (80GB). The GPU memory required to run the experiments is dependent on
the base LLMs, as detailed below:

  * •

Llama-2-13B: $\sim$30GB GPU memory.

  * •

Llama-2-70B: $\sim$140GB GPU memory.

  * •

Llama-3-8B: $\sim$20GB GPU memory.

  * •

Llama-3-70B: $\sim$140GB GPU memory.

  * •

Mistral-7B: $\sim$20GB GPU memory.

  * •

Mixtral-8x7B: $\sim$110GB GPU memory.

  * •

Mixtral-8x22B: $\sim$300GB GPU memory.

The exact computation time was affected by many factors, e.g., the current
workload of the machine, the prompt length of the question, the generated
response length, whether the cache option is turned on to store the model
states of LLMs, etc.

###  A.2 Results of Statistical Tests

Table 2 shows the $p$-values of the paired $t$-tests as described in Section
4.1.

Table 2: Statistical significance of SD’s advantage over other methods ($p$-values of paired $t$-tests) SD vs. | SE | P(True) | Deg | NL | NE | PE
---|---|---|---|---|---|---
4.83E-17 | 1.71E-15 | 1.16E-7 | 4.62E-14 | 2.35E-15 | 4.63E-13