# The Calibration Gap between Model and Human Confidence in Large Language
Models

Mark Steyvers<EMAIL_ADDRESS>
Department of Cognitive Sciences, University of California, Irvine Heliodoro
Tejeda<EMAIL_ADDRESS>
Department of Cognitive Sciences, University of California, Irvine Aakriti
Kumar<EMAIL_ADDRESS>
Department of Cognitive Sciences, University of California, Irvine Catarina
Belem<EMAIL_ADDRESS>
Department of Computer Science, University of California, Irvine Sheer Karny
<EMAIL_ADDRESS>
Department of Cognitive Sciences, University of California, Irvine Xinyue Hu
<EMAIL_ADDRESS>
Department of Cognitive Sciences, University of California, Irvine Lukas Mayer
<EMAIL_ADDRESS>
Department of Cognitive Sciences, University of California, Irvine Padhraic
Smyth<EMAIL_ADDRESS>
Department of Computer Science, University of California, Irvine

###### Abstract

For large language models (LLMs) to be trusted by humans they need to be well-
calibrated in the sense that they can accurately assess and communicate how
likely it is that their predictions are correct. Recent work has focused on
the quality of internal LLM confidence assessments, but the question remains
of how well LLMs can communicate this internal model confidence to human
users. This paper explores the disparity between external human confidence in
an LLM’s responses and the internal confidence of the model. Through
experiments involving multiple-choice questions, we systematically examine
human users’ ability to discern the reliability of LLM outputs. Our study
focuses on two key areas: (1) assessing users’ perception of true LLM
confidence and (2) investigating the impact of tailored explanations on this
perception. The research highlights that default explanations from LLMs often
lead to user overestimation of both the model’s confidence and its’ accuracy.
By modifying the explanations to more accurately reflect the LLM’s internal
confidence, we observe a significant shift in user perception, aligning it
more closely with the model’s actual confidence levels. This adjustment in
explanatory approach demonstrates potential for enhancing user trust and
accuracy in assessing LLM outputs. The findings underscore the importance of
transparent communication of confidence levels in LLMs, particularly in high-
stakes applications where understanding the reliability of AI-generated
information is essential.

Keywords: Large Language Models, LLMs, Calibration, Trust, Explanations, User
Confidence

##  1 Introduction

Large language models (LLMs) have markedly transformed how humans seek and
consume information. LLMs are being deployed across diverse fields, including
public health (Ali et al., 2023), coding (Zambrano et al., 2023), and
education (Whalen et al., 2023). Despite their growing influence, LLMs are not
without shortcomings. One notable issue is the possibility of generating
responses that, while convincing, may be inaccurate or nonsensical. This
phenomenon has been highlighted in several recent studies (Jo, 2023; Huang et
al., 2023) and raises concerns about the reliability of these models. The
unreliability of LLMs has led developers of LLMs to caution against uncritical
acceptance of model outputs (OpenAI, 2022b), suggesting that it is not always
clear when the models are or are not confident in the knowledge communicated
to the user.

Recent research, however, indicates that LLMs have the ability, to a certain
degree, to accurately discern their own knowledge boundaries. Large LLM models
in particular can exhibit a reasonable level of calibration for multiple-
choice questions such that the probability the model assigns to a selected
answer tracks with the probability that this answer is correct (OpenAI, 2023;
Kadavath et al., 2022; et al., 2023). In addition, recent studies show that
LLMs can distinguish between answerable and unanswerable questions (Yin et
al., 2023; Kadavath et al., 2022) and the internal state of an LLM can
distinguish between truthful statements and lies (Azaria and Mitchell, 2023).
These findings suggest that LLMs may possess an internal mechanism that is
reflective of self-knowledge.

In the specific context of question-answering, an LLM’s model confidence is
typically equated to the probability assigned by the LLM to the selected
answer relative to other possible answers (e.g., (Jiang et al., 2021;
Hendrycks et al., 2021)). However, from the perspective of a human interacting
with the LLM, this internal model confidence is not usually displayed to human
users as part of LLM output. Instead, in current practice, humans rely solely
on the language produced by the LLM in order to assess LLM confidence. To
contrast with model confidence, in this paper we use the term human confidence
to refer to a human’s assessment (expressed as a probability) of how likely it
is that the LLM’s answer is correct based only on the language produced by the
LLM without any knowledge of the LLM’s internal model confidence.

Surprisingly, studies focused on investigating human confidence in LLM outputs
are lacking. In this paper, we take a step in addressing this issue and
investigate what we term the calibration gap, namely the difference in the
reliability of (i) internal LLM model confidence and (ii) external human
confidence. In effect, the calibration gap represents the gap between an LLM’s
own internal confidence of what it knows and human perception of this
confidence. We address two specific research questions in this context:

  1. 1.

How Large is the Calibration Gap? i.e., is there a significant gap between LLM
model confidence and human confidence, in terms of how each assesses the true
accuracy of the LLM?

  2. 2.

Can the Calibration Gap be Reduced? e.g., can the quality of human confidence
in an LLM be improved by adapting the textual output of the LLM and by
leveraging internal model confidence?

These questions have important implications for the design of reliable LLM
assistants. Addressing the calibration gap is crucial for the development of
effective and trustworthy assistants by aligning the LLM’s internal confidence
with human perception of this confidence.

Our contributions in this context are twofold. First, we present a unique
experimental study and dataset that directly captures human assessment of LLM
confidence in a question-answering context, providing insight into human
perceptions of LLM textual responses. Second, we test and suggest ways of
generating LLM responses that improve the calibration quality of human
confidence relative to the LLM assistant’s model confidence and the LLM’s true
accuracy.

More specifically, as depicted in Figure 1, we use the well-known Massive
Multitask Language Understanding (MMLU) dataset (Hendrycks et al., 2021) to
conduct question-answering behavioral experiments where participants assess an
LLM agent’s responses to multiple-choice questions. Participants estimate the
accuracy of the LLM assistant, without any direct access to the LLM’s
numerical model confidence, allowing us to make inferences about participants’
perceptions of the confidence of the LLM based on model explanations alone.
Additionally, we conduct a series of experiments to prompt the LLM to generate
explanations more aligned with its internal confidence.

The outline of the remainder of the paper is as follows. In Section 2, we
review existing work on eliciting calibrated confidence scores from LLMs,
psychology research on converting probabilities to verbal expressions of
uncertainty, and human understanding of explanations. In Section 3, we
describe our experimental setup for the behavioral experiments, and in Section
4, we discuss our findings and data analysis. Finally, we discuss key
takeaways in Section 5.

Figure 1: Overview of the evaluation methodology for assessing the calibration
gap between model confidence and human confidence in the model. The approach
works as follows: (1) prompt the LLM with a multiple-choice question to obtain
the model’s internal confidence for each answer choice; (2) select the most
likely answer and prompt the model a second time to generate an explanation
for the given answer; (3) obtain the human confidence by showing users the
question and the LLM’s explanation and asking users to indicate the
probability that the model is correct. In this toy example the model
confidence is 0.46 for answer C, whereas the human confidence in 0.95

##  2 Related Research

###  2.1 Eliciting LLM Confidence

Several approaches have been developed to elicit confidence in LLMs and to
assess the degree to which the elicited confidence scores are calibrated (see
(Geng et al., 2023) for an overview). One approach that is commonly used is to
access internal model information such as token likelihoods, allowing for
direct computation of relative probabilities of different possible answers in
multiple-choice questions (Jiang et al., 2021; Kadavath et al., 2022; Xiao et
al., 2022; Hendrycks et al., 2021; OpenAI, 2023). More recent research has
focused on verbally expressing LLM model confidence such that the confidence
is expressed in natural language as numeric strings (e.g., “80%”) (Lin et al.,
2022; Xiong et al., 2023) or more qualitative expressions of confidence (e.g.,
“I am not confident the answer is X”) (Mielke et al., 2022; Zhou et al.,
2023). To create calibrated verbal expressions of uncertainty, methods have
included fine-tuning LLMs to produce verbalized probabilities for classes of
math problems (Lin et al., 2022) and training models to connect the internal
model representation to dialogue models that can translate internal
probabilities to an appropriate linguistic expression of confidence (Mielke et
al., 2022).

Methods that do not require access to internal model representations have used
prompting strategies designed to elicit verbal expressions of uncertainty
(Xiong et al., 2023; Zhou et al., 2023). Prompts that are designed to
emphasize step-by-step reasoning about the correctness of individual steps and
clarify the space of possible answers lead to better calibration than simple
prompts that simply ask for a confidence rating (Xiong et al., 2023). For
short-form question answering, prompting strategies can lead to calibrated
confidence levels (Tian et al., 2023). However, for multiple-choice questions,
which is the focus of our work, prompting approaches have been found to be
less accurate compared to methods that read out model confidence (Xiong et
al., 2023). Other black-box prompting methods for confidence elicitation have
focused on the assessment of similarity among multiple responses from the
model (Lin et al., 2023).

Our research builds on this prior work by integrating multiple approaches to
eliciting confidence. We utilize a “white-box” method that reads out the
internal token likelihoods. In addition, we use prompting strategies to
verbally communicate the uncertainty expressed in the internal likelihoods to
users. In contrast with prior work on LLM confidence elicitation, our goal is
not to develop novel confidence elicitation procedures. Instead, the focus of
our work is to assess the human perception of LLM uncertainty as expressed
through explanations.

###  2.2 Human perception of verbal probability phrases

A significant body of psychology research has investigated perceptions of
verbally expressed uncertainty across a wide range of domains, including
climate policy, medicine, and intelligence forecasting (Budescu et al., 2014;
Ho et al., 2015; Karelitz et al., 2002; Wallsten et al., 2008; O’Brien, 1989).
These studies aim to understand how humans perceive verbal probability phrases
such as “highly unlikely” and “almost certain” when describing the likelihood
of an event occurring. The findings show that there are subjective differences
in how people interpret linguistic probabilities (Dhami and Wallsten, 2005;
Karelitz and Budescu, 2004; Wallsten et al., 2008) which has led to efforts to
develop methods to reduce the subjectivity gap (Karelitz et al., 2002;
Wallsten et al., 2008). Furthermore, the perceived range of probabilities
varies between probability phrases. For example, “perhaps” is associated with
a wide range of probabilities, whereas “almost no chance” is associated with
less disagreement (O’Brien, 1989). Despite their lack of precision in
communicating probabilities, verbal probability phrases provide a simple way
to communicate uncertainty in natural language contexts. Our work expands on
previous research by developing a mapping function that encourages the use of
specific verbal probability phrases based on LLM’s model confidence.

More recently, the work by (Zhou et al., 2024) investigates how users behave
in the presence of verbal phrases of uncertainty during human-LLM
interactions. While their findings are consistent with ours, the two works use
two different methodologies. (Zhou et al., 2024) craft templates using LLM-
articulated expressions in a simulated trivia task and measures the users’
reliance on these expressions (regardless of model confidence). Conversely,
our work uses actual LLM outputs and its confidence in an attempt to quantify
the calibration gap. As a result, we are able to directly address the
miscommunication of uncertainty from LLM to humans.

###  2.3 Effect of AI explanations on human decision-making

Much of the work examining the human perception and evaluation of machine-
generated explanations has focused on machine learning classification models
rather than LLMs (see (Rong et al., 2023) for an overview). These studies
frequently employ feature highlighting to explain what areas of the image
(Smith-Renner et al., 2020) or what fragments of documents (Feng and Boyd-
Graber, 2019) can support the suggested classification. Studies have found
mixed evidence for the effectiveness of these types of AI explanations in
human decision-making (Steyvers and Kumar, 2023; Bansal et al., 2021; Buçinca
et al., 2021; Wang and Yin, 2022). When humans are assisted by AI, adding AI
explanations can improve human classification accuracy (Rong et al., 2023).
However, many of these studies involved AI models that perform at a higher
accuracy than humans, and therefore, these results suggest that the
explanations simply increased reliance on the AI. In contrast to this prior
work, our focus here is on human assessment of LLM confidence, i.e., the
ability of people to tell when the LLMs answer is likely correct or incorrect
based on the explanation provided, rather than on accuracy in decision-making.

##  3 Methodology

We used two publicly available state-of-the-art LLMs in our studies111We also
explored Llama2 70B and Llama-2-chat 70B but found the quality of the
generated explanations to be unsatisfactory.: GPT-3.5 (OpenAI, 2022a) and
PaLM2 (Anil et al., 2023). We apply both LLMs to a subset of multiple-choice
questions from the MMLU data set (Hendrycks et al., 2021), as detailed in
Section 3.1, and assess model confidence for each multiple-choice question.
Model confidence enables us to investigate the relationship between model
confidence and accuracy and to determine whether the LLM is reasonably well
calibrated, independent of the LLM’s ability to elicit well-calibrated
confidence from humans who use the LLM.

We conduct behavioral experiments (section 3.2), where the task for
participants is to estimate the probability that the LLM model’s answer to a
multiple-choice question is correct based on the explanation provided. We use
the term human confidence to refer to this assessment about the LLM. In
addition, with the assistance of the LLM, participants provide answers to the
questions. Previous research has demonstrated that the MMLU multiple-choice
questions are difficult for participants who lack domain expertise, resulting
in near-chance accuracy (Hendrycks et al., 2021). We anticipate that these
questions will be challenging to answer without the help of the LLM because
the majority of the participants in our experiments lack domain expertise, and
their perception of the explanation’s content will influence their assessment
more than their own knowledge.

A key feature of the behavioral experiment is manipulating the language of
uncertainty expressed in the explanations. Overall, we conducted three
experiments using both types of LLMs:

  * •

Experiment 1 assesses human perceptions of LLM accuracy based on default
explanations generated by the LLM. We use standard prompts to elicit the
answer and an explanation for the answer.

  * •

Experiment 2 manipulates the prompts to produce three levels of uncertainty
language (low, medium, and high confidence) and two levels of explanation
length, resulting in six different types of explanations presented to
participants. The prompts are designed to include uncertainty language
corresponding to model confidence at the start of the explanation.

  * •

Experiment 3 manipulates the prompts to change the default explanations from
Experiment 1 according to three levels of uncertainty language. In contrast to
Experiment 2, the explanations lead to more varied ways in which uncertainty
language is expressed.

We use two types of metrics to evaluate the relationship between confidence
(both human and model) and model accuracy as detailed in Section 3.3.

###  3.1 Question data set

The MMLU dataset is a comprehensive multitask dataset that contains multiple-
choice questions from various knowledge domains, such as STEM, humanities,
social sciences, and more (Hendrycks et al., 2021). In total, there are 14042
test set questions from 57 categories curated by undergraduate and graduate
students from freely available online resources such as the GRE and USMLE.
These questions range in difficulty from high-school to the professional
level. The MMLU dataset is widely employed to measure a text model’s multitask
accuracy, as it challenges models on their real-world text understanding
beyond mere linguistic comprehension, thus making it a robust benchmark for
model evaluation (Hendrycks et al., 2021; Hoffmann et al., 2023; Rae et al.,
2021). For this project, we sampled a subset of the questions from a range of
model’s confidence levels in selected categories from the full dataset to
comprehensively evaluate people’s assessment of LLM model confidence.

####  3.1.1 Assessing uncertainty on multiple choice answers

Before sampling a subset of questions from the MMLU dataset, we first assessed
the LLM model confidence of GPT-3.5 and PaLM2 language models to 14042 MMLU
multiple-choice questions. This allowed us to then select questions with
(somewhat) evenly distributed confidence levels. We followed the procedures
described in the GPT-4 Technical Report (OpenAI, 2023) using a zero-shot
prompting approach, in which the model was only prompted with the target
question and its associated answer options. Figure 2 shows an example. We read
out the log-probabilities for the top 5 tokens completed by the model using
the APIs for the GPT3.5 (gpt-3.5-turbo-instruct) and the PaLM2 (text-
bison@002) models222The GPT3.5 model was accessed on Aug-Oct 2023 and PaLM2
model on January 2024.. The temperature parameter was set to 0. The answer was
considered complete if the tokens included the single letters A, B, C, and D.
The log scores were then converted and normalized to probabilities across the
four answer options (so that the sum of the scores equaled one). Internal
uncertainties, referred to in this paper as model confidence, were represented
by these probabilities in all experiments, a common technique in calibration
assessment with LLMs (Jiang et al., 2021; Kadavath et al., 2022; Xiao et al.,
2022; Hendrycks et al., 2021; OpenAI, 2023).

A horizontal spring-block oscillator is in motion when a second block of equal
mass falls on top of the oscillating block. If the two blocks stick together,
what effect will this have on the amplitude of the oscillations?
Choose from the following options:
[A] It will increase the amplitude.
[B] It will decrease the amplitude.
[C] It will have no effect.
[D] It would depend on where the original block was in its cycle when the
second block fell.
Please answer this question. Answer [A], [B], [C], or [D]. The answer
therefore is: [

Figure 2: Example prompt to elicit the answer and model confidence across
answer options for a physics question from the MMLU dataset.

####  3.1.2 Sampling a subset of 350 questions

Based on the model confidence levels of each LLM for all MMLU questions, we
created a subset separately for each LLM. In total, 35 questions were sampled
for each of ten topics, for a total of 350 questions. For each topic, the 35
questions were sampled to approximately create a uniform distribution over
model confidence using the confidence bins: 0.2-0.4, 0.4-0.6, 0.6-0.8, and
0.8-1.0. However, due to the small number of questions that lead to model
confidence in the lowest confidence bin, fewer questions were sampled for the
0.2-0.4 confidence range. The Supplementary show the distribution over model
confidence levels for the entire MMLU dataset as well as the question subset
sampled for our study. Model accuracy across the 350 questions is 55% and 50%
for GPT-3.5 and PaLM2, respectively.

###  3.2 Behavioral Experiments

This section describes the methodology we used in our behavioral experiments.
Experiments 1, 2, and 3 differed only in the types of explanations presented
to participants but otherwise followed the same general procedures as
described below. Each experiment was conducted separately with the
explanations from GPT-3.5 and PaLM2.

####  3.2.1 Participants

A total of 240 participants (40 in each experiment for each LLM) completed the
study across Experiments 1, 2, and 3. Participants were native English
speakers residing in the United States, recruited through Prolific
(www.prolific.com). The University of California, Irvine Institutional Review
Board (IRB) approved the experimental protocol. Participants who completed the
study were paid $8 USD for their participation (approximately $12/hr). Prior
to the experiment, participants were given detailed instructions outlining the
experimental procedure as well as how to understand and interact with the user
interface. Participants were asked to sign an integrity pledge after reading
all of the instructions, stating that they would complete the experiment to
the best of their abilities. After submitting their integrity pledge,
participants were granted access to the experiment.

####  3.2.2 Experimental Procedure

Participants in all experiments were randomly assigned 40 multiple-choice
questions from the pool of 350 MMLU questions. The questions were chosen in
such a way that there would be four questions per topic. Furthermore, the
sampling was balanced across model confidence bins (0.2-0.4, 0.4-0.6, 0.6-0.8,
0.8-1.0). This sampling procedure ensured that all participants were given an
equal number of questions from each topic and were exposed to questions at all
levels of difficulty for each topic.

Furthermore, in Experiments 2 and 3, we balanced the types of explanation
styles across questions so that each question was presented approximately the
same number of times with each style. It should be noted that for each
subject, each question was presented only once, and each question received
only one explanation style. The counterbalancing, on the other hand, ensured
that the same question had (roughly) an equal number of observations for each
explanation style (across participants).

The participant’s task was divided into two phases for each question, as shown
in Figure 3. In the first stage, participants had to provide a probability
estimate that the LLM’s answer was correct. In the second phase, participants
must answer the question with the assistance of the LLM. Participants were
instructed to use their own knowledge as well as the LLM’s response when
making their own answer selection for this phase.

Evaluating LLM Accuracy

(a) Phase 1

Answering with LLM Assistance

(b) Phase 2

Figure 3: Illustration of the behavioral experiment interface with the GPT-3.5
model. The left panel displays the experimental interface during phase 1 of
the task, where participants evaluate the probability that the LLM is correct
based on the explanation. The right panel displays the experimental interface
during phase 2 of the task, where participants answer the multiple choice
question with the assistance of the LLM.

At the end of the experiments, we administered a brief survey in which
participants self-assessed their knowledge of the ten topics. Participants
were asked to estimate the expected accuracy for each topic if they were
presented with questions similar to those they encountered during the
experiment.

####  3.2.3 Creating explanation styles with varying degrees of uncertainty

This section describes how we constructed prompts to elicit explanations with
varying levels of uncertainty language in the experiments. For additional
details on the exact prompts used, see Appendix Table 3. Table 1 contains
examples of explanations from the three experiments and styles.

##### Experiment 1: baseline explanations

To generate the baseline explanations for Experiment 1, we used a simple
prompt that asked for an explanation, which would also repeat the answer.

##### Experiment 2: modified explanations

We used prompts that were designed to change the explanations to express three
different levels of confidence. The low, medium, and high confidence prompts
instructed the LLM to “mention you are” not sure/somewhat sure/sure in the
explanation for each of the low, medium, and high confidence prompts,
respectively. The prompts elicited responses in which the beginning of each
explanation indicated the level of uncertainty (e.g., “I am not sure the
answer is [B] because” for the low confidence prompt) . Furthermore, the
prompts led to additional differences in explanations for why the LLM lacked
confidence (e.g., “further research may be required to confirm this,” “it is
not possible to definitively state that…”). Experiment 2 also varied the
length of the explanation. We created a shorter variant for each of the three
types of explanations by adding an instruction to use as few words as possible
in the explanation. The average length of the long and short explanations was
770 and 227 characters, respectively, for GPT-3.5, and 441 and 123 characters,
respectively for PaLM2. In total, Experiment 2 included six types of
explanations (three levels of uncertainty x two levels of length).

##### Experiment 3: modified explanations

We used three different types of prompts to elicit low/medium and high-
confidence explanations and did not include a length manipulation. In this
approach, we gave the LLM the baseline explanation from Experiment 1 in the
prompt and instructed the LLM to “rewrite the explanation as if you are” not
sure/somewhat sure/sure for each of the three confidence levels.

Table 1: Example explanations given by GPT-3.5 for the different prompts used in the experiments. The particular question is: Which of the following is incorrectly named? A) CaCl2 calcium chloride; B) Fe(NO3)3 iron(III) nitrate; C) AlBr3 aluminum tribromide; D) K2Cr2O7 potassium dichromate Prompt Style |  Explanation
---|---
Experiment 1 |
Baseline |  The answer is [B] because the correct name for Fe(NO3)3 is iron(III) nitrate, not iron(II) nitrate. The Roman numeral in parentheses after the metal name indicates the oxidation state of the metal ion in the compound, and in this case, it should be III (3) instead of II (2). Therefore, the compound is incorrectly named as iron(II) nitrate.
Experiment 2 |
Low Confidence & Long |  I am not sure that the answer is [B] because iron(III) nitrate is a correct name for Fe(NO3)3. Iron(III) refers to the oxidation state of iron in the compound, which is +3, and nitrate is the anion present. However, I am not completely confident in my answer because it is possible that there is another, more specific name for this compound that I am not aware of.
Medium Confidence & Long |  I am somewhat sure that the answer is [B] because iron(III) nitrate should be named iron(III) nitrite. The correct name for Fe(NO3)3 is iron(III) nitrite, not iron(III) nitrate. This is because the nitrate ion has a charge of -1, while the nitrite ion has a charge of -2. Therefore, the correct formula for iron(III) nitrite should be Fe(NO2)3. Since the formula given in option [B] does not match the correct formula, I am somewhat sure that this option is incorrectly named.
High Confidence & Long |  I am sure that the answer is [B] because the correct name for Fe(NO3)3 is iron(III) nitrate, not iron(II) nitrate. The Roman numeral in parentheses after the metal name indicates the charge of the metal ion in the compound. In this case, iron has a charge of +3, not +2. Therefore, the correct name for Fe(NO3)3 is iron(III) nitrate, making [B] incorrectly named.
Low Confidence & Short |  I am not sure if the answer is [B] because iron(III) nitrate is the correct name for Fe(NO3)3.
Medium Confidence & Short |  I am somewhat sure that the answer is [B] because the correct name for Fe(NO3)3 is iron(III) nitrate, not iron(II) nitrate.
High Confidence & Short |  I am sure that the answer is [B] because iron(III) nitrate should be written as Fe(NO3)3, not Fe3(NO3)3.
Experiment 3 |
Low Confidence |  I am not entirely certain, but I believe the answer is [B]. The reason for this is that the compound Fe(NO3)3 is typically named as iron(III) nitrate, not iron(II) nitrate. The Roman numeral in parentheses after the metal name indicates the oxidation state of the metal ion in the compound, and in this case, it should be III (3) instead of II (2). Therefore, it is possible that the compound is incorrectly named as iron(II) nitrate.
Medium Confidence |  I believe the answer is [B] because the compound Fe(NO3)3 is incorrectly named as iron(II) nitrate. The correct name should be iron(III) nitrate, as indicated by the Roman numeral III (3) after the metal name. However, I am not completely confident in my answer and would appreciate further clarification or confirmation.
High Confidence |  The answer is [B] because the compound Fe(NO3)3 is incorrectly named as iron(II) nitrate. The correct name for this compound is iron(III) nitrate, as indicated by the Roman numeral III after the metal name. This indicates that the iron ion has an oxidation state of 3, not 2. Therefore, the compound is incorrectly named and the correct name is iron(III) nitrate.

####  3.2.4 Generating explanations modified by model confidence

In experiments 2 and 3 we use the following simple approach to automatically
select the type of explanation on the basis of LLM model confidence, in order
to better align human confidence with LLM accuracy. We use a simple decision
rule that selects the type of explanation $s\in$ {low confidence, medium
confidence, high confidence} based on the LLM model confidence score $p$:

$\displaystyle s$
$\displaystyle=\begin{cases}\mathrm{low\;confidence}&\text{if
}p\leq\theta_{1}\\\ \mathrm{medium\;confidence}&\text{if
}\theta_{1}<p\leq\theta_{2}\\\ \mathrm{high\;confidence}&\text{if
}\theta_{2}<p\\\ \end{cases}$ (1)

The parameters $\theta_{1}$ and $\theta_{2}$ determine the ranges where low,
medium, and high confidence explanations are chosen. The application of this
rule to a given parameter setting leads to any participant estimates being
filtered out if the explanation style used for a specific question does not
match the selected style. This allowed us to simulate the effect of
participants receiving different types of explanations based on model
confidence (i.e., lower confidence explanations for low model confidence and
high confidence explanations for high model confidence). The Appendix provides
details on the optimization procedure and also a demonstration that the
results are not particularly sensitive to the parameter settings.

###  3.3 Metrics

To investigate the relationship between the accuracy of answers to multiple-
choice questions and the confidence (either human confidence or model
confidence) associated with them, we utilize a range of metrics to evaluate
this association. The primary focus is on understanding how well confidence
levels correlate with the correctness of answers. To achieve this, we use both
Expected Calibration Error (ECE) and the Area under the Curve (AUC) metric.
These metrics assess the extent of overconfidence in predictions as well as
the diagnostic effectiveness of confidence scores in distinguishing between
correct and incorrect answers (Xiong et al., 2023; Tian et al., 2023; Jiang et
al., 2021; Kadavath et al., 2022; Xiao et al., 2022). The use of AUC in this
context parallels various metrics in psychology for metacognitive
discrimination or sensitivity, which similarly aim to evaluate the
effectiveness of confidence scores in distinguishing between correct and
incorrect answers (Fleming and Lau, 2014). In addition, in the Supplementary,
we also show results for the additional metric of Overconfidence Error (OE).

####  3.3.1 Expected Calibration Error (ECE)

We evaluate miscalibration using the Expected Calibration Error (ECE), as
detailed in (Guo et al., 2017; Naeini et al., 2015). ECE is calculated by
averaging the absolute differences between accuracy and confidence across $M$
equal-width probability bins:

$\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{N}|conf(B_{m})-acc(B_{m})|$ (2)

where $N$ represents the total sample count, $B_{m}$ the $m$th confidence bin,
and $acc(B_{m})$ and $conf(B_{m})$ denote the accuracy and average confidence
for samples in the $m$th bin. ECE does not account for the direction of
deviations between accuracy and confidence per bin respectively, so a nonzero
ECE can indicate a mix of over- and underconfidence While recent work (Kumar
et al., 2019; Gruber and Buettner, 2022) has shown that ECE can under-estimate
the true calibration error, the potential for under-estimation should not be a
significant issue given that we are interested in analyzing differences in ECE
rather than unbiased estimates of the error itself.

####  3.3.2 Area under the Curve (AUC)

The AUC metric is employed to assess the diagnostic ability of confidence
scores in distinguishing between correct and incorrect answers. Utilizing the
Mann-Whitney U statistic approach, the AUC represents the probability that a
randomly chosen correct answer has a higher confidence score compared to a
randomly chosen incorrect answer:

$\mathrm{AUC}=\frac{1}{N_{pos}\times
N_{neg}}\sum_{i=1}^{N_{pos}}\sum_{j=1}^{N_{neg}}I(C_{i}>C_{j})$ (3)

In this equation, $N_{pos}$ and $N_{neg}$ denote the counts of correct
(positive) and incorrect (negative) answers, respectively. $C_{i}$ and $C_{j}$
represent the confidence scores of the $i$th and $j$th correct and incorrect
answers, respectively. $I$ is an indicator function, which equals 1 if
$C_{i}>C_{j}$ and 0 otherwise. This method evaluates each pair of correct and
incorrect answers to determine if the confidence score for the correct answer
surpasses that of the incorrect one. The AUC is then the fraction of these
pairs satisfying this criterion, measuring the capability of confidence scores
to differentiate between correct and incorrect responses, with AUC values
ranging from 0.5 (indicating no better than chance discrimination) to 1
(signifying perfect discrimination).

###  3.4 Statistical Analysis

To assess statistical significance, we utilize Bayes factors (_BF_ s) to
determine the extent to which the observed data adjust our belief in the
alternative and null hypotheses. Values of 3 $<$ _BF_ $<$ 10 and _BF_ $>$ 10
indicate moderate and strong evidence against the null hypothesis,
respectively. Similarly, values of 1/10 $<$ _BF_ $<$ 1/3 and _BF_ $<$ 1/10
indicate moderate and strong evidence in favor of the null hypothesis,
respectively. We report Bayes factors for Bayesian t-tests using the default
priors as recommended by (Rouder et al., 2012).

##  4 Results

###  4.1 Modified explanations affect human confidence

We first assess if the linguistic manipulations in Experiments 2 and 3, which
systematically modify the LLMs explanations to better reflect model
confidence, have a direct effect on human confidence as measured in our
experiments. Figure 4 shows that the type of uncertainty language used in the
explanations has a strong influence on human confidence in Experiments 2 and
3. When the explanations contain less certain language (“I am not sure”, “not
confident”, “I think”, “not confident in this explanation”), participants are
less confident that the answer is correct. For comparison, the mean confidence
in Experiment 1 for the unmodified explanations was 0.77 and 0.75 for GPT-3.5
and PaLM2 respectively. In contrast, the low confidence (“not sure”)
explanations in Experiment 2 led to significantly lower confidence with means
of 0.52 and 0.48 for GPT-3.5 and PaLM2 respectively (BF $>$ 100 for both
LLMs). The results overall confirm that people can appropriately interpret
verbal cues about uncertainty.

For GPT-3.5, the length of the explanation generally did not affect the
confidence in the answers (BF $<$1 for all three comparisons of short and long
explanations across explanation styles). However, for PaLM2, for the least
confident explanations, longer explanations received higher confidence than
shorter explanations (BF $>$100). In the results for Experiment 2 in the
remainder of the paper we analyze the results collapsed over the short and
long explanations.

GPT-3.5 |  |  |
---|---|---|---
PaLM2 |  |  |
Figure 4: Distribution of human confidence across explanation styles in
Experiments 1, 2, and 3. For reference, dashed lines show the average human
confidence for the unmodified explanations in Experiment 1. The error bars
represent the 95% confidence interval across questions.

###  4.2 Calibration and discrimination

####  4.2.1 Calibration and discrimination gaps

We next analyze the results of comparing human and model confidence for the
case where LLMs generate default explanations for participants (Experiment 1)
compared to where the explanations are modified to reflect model confidence
(Experiments 2 and 3). The results show that there is a significant gap
between what LLMs know and what humans think they know on the basis of these
explanations.

Figure 5 shows the Expected Calibration Error (ECE) and the Area Under the
Curve (AUC) for both model and human confidence (for the full results with an
additional metric for overconfidence, see the Appendix). Focusing first on
calibration (left panel), the results show a significant calibration gap; for
both LLMs, the ECE metric is significantly lower for model confidence (in
gray) than for human confidence with baseline explanations (in blue). This gap
demonstrates that standard explanations provided by the LLM do not enable
participants to judge the likelihood of correctness of the LLM’s answers,
leading to a misalignment between perceived accuracy and actual LLM accuracy.

Furthermore, there is also a gap between how well model and human confidence
discriminate between correct and incorrect answers (Figure 5, right panel).
Whereas the LLM model confidence discriminates between correct and incorrect
answers well above chance (GPT-3.5 AUC=.751, PaLM2 AUC=0.746), the
participants who view the default explanations in Experiment 1 were only
slightly better than random guessing with AUC values of 0.589 and 0.602 for
GPT-3.5 and PaLM2 explanations respectively.

Figure 5: Calibration gap between model and human confidence. The plots show
the calibration error (assessed by ECE; lower is better), and discrimination
(assessed by AUC; higher is better) of model and human confidence across LLMs
and experiments. Vertical dashed lines represent the calibration and
discrimination gap between human confidence for unmodified explanations and
model confidence. The AUC values were computed separately for each participant
with error bars representing the 95% confidence interval. Because of data
sparsity, the ECE values were computed at the group level.

Figure 5 also shows the results from Experiments 2 and 3 after the LLM
explanations have been modified to reflect model confidence. The results in
Figure 5 (left panel, green and pink bars) show that the calibration gap has
narrowed substantially. While there is still generally a higher calibration
error for human confidence relative to model confidence, the calibration gap
has decreased significantly for both LLMs relative to the baseline
explanations in Experiment 1. Furthermore, Figure 5 (right panel) shows that
the discrimination gap (as measured by AUC) has also been narrowed relative to
the baseline explanations for both LLMs (BF $>$ 10 for both Experiments 2 and
3).

Overall, these results show that selecting the type of explanation based on
LLM model confidence leads to improved calibration and discrimination
performance, where human confidence in the LLM is more closely related to the
actual accuracy of the LLM.

####  4.2.2 Reliability diagrams across experiments and LLMs

Figure 6 expands on the calibration results in Figure 5 to show detailed
calibration results for each LLM and each experimental condition. The diagrams
show how well model confidence (left column) and human confidence (right three
columns) are calibrated. The ideal calibration (i.e., ECE=0) would yield
results along the diagonal. Both LLMs and humans have a tendency to be
overconfident, resulting in calibration lines below the diagonal. However,
human overconfidence when presented with the default explanations is greater,
indicating that people generally believe the LLMs are more accurate than they
actually are. The histograms (inset panels) demonstrate that a significant
portion of the calibration error is due to participants’ propensity to produce
high confidence scores, even though the model accuracy for the associated
questions is much lower than expected based on confidence.

GPT-3.5 |  |  |  |
---|---|---|---|---
PaLM2 |  |  |  |
Figure 6: Calibration diagrams for model confidence and human confidence
across Experiments 1-3. The top and bottom rows show results for the GPT-3.5
and PaLM2 models respectively. The histograms at the bottom of each plot show
the proportion of observations in each confidence bin (values are scaled by
30% for visual clarity). Shaded regions represent the 95% confidence interval
of the mean computed across participants and questions.

###  4.3 Accuracy

####  4.3.1 Participants lack specialized knowledge

For the experiments with GPT-3.5, participants’ average answer accuracy was
50%, closely aligning with LLM’s 52% accuracy rate. Similarly, for the
experiments with PaLM2, average participant accuracy was 46%, similar to the
50% accuracy rate for the LLM. Across all experiments and LLMs, for a majority
(82%) of responses, participants chose the answer that agreed with the answer
in the explanation provided by the LLM. When participants chose to alter the
answer, the average accuracy was 32% which is lower than the LLM’s accuracy of
45% for these particular questions. These findings suggest limited success in
participants’ ability to accurately answer the questions independent of the
LLM’s explanation. This is consistent with findings from (Hendrycks et al.,
2021), showing that Mechanical Turk workers without specialized knowledge
(akin to our participant pool) scored 35% accuracy on similar questions.

####  4.3.2 Self-assessed expertise does not affect performance

At the end of the experiment, participants estimated the performance they
would achieve on similar questions for each of the 10 topics. The median of
these self-assessed expertise estimates did not substantially vary between
topics: from 30% (e.g., high school physics) to 45% (e.g., high school world
history). Examining the impact of perceived expertise on accuracy estimation,
we divided participants into two groups based on whether their self-rated
expertise was above or below 50% separately for each of the 10 topics. For the
experiments with GPT-3.5, the higher expertise groups generally had better
discrimination (AUC 0.573 vs. AUC 0.538), but there was no evidence that this
difference was significant (BF $<$ 1). In addition, the calibration error was
comparable between the two groups (ECE =.261 vs. .257). Similarly, no effect
of expertise was found for the experiments with PaLM2. Therefore, participants
who considered themselves more knowledgeable about a topic were not more adept
at estimating GPT’s performance in that area.

##  5 Discussion

The core of our research centered on bridging the gap between what an LLM
knows and what users perceive it knows. This gap is critical, as the reliance
on LLMs for decision-making processes in various domains is rapidly
increasing. Several studies have examined LLM confidence for multiple choice
questions (Kadavath et al., 2022; OpenAI, 2023; Hendrycks et al., 2021; Xiong
et al., 2023) and assessing the reliability of LLM explanations in this
context (Tanneru et al., 2023), but there has been little research to
investigate user confidence in LLM output. Our results showed that users
consistently overestimated how accurate LLM outputs were, especially when they
relied on the models’ default explanations. This was true for two different
LLMs. This tendency towards overconfidence in LLM capabilities is a
significant concern, especially in scenarios where critical decisions are
based on LLM information. The inability of users to discern the reliability of
LLM responses not only undermines the utility of these models but also poses
risks in situations where user understanding of model accuracy is critical.

A key contribution of this research lies in demonstrating the effectiveness of
tailored explanations in bridging this perception gap. This was achieved by
altering the prompts used to generate explanations based on model confidence,
thereby controlling the expression of uncertainty within the responses.
Specifically, we designed these prompts to induce varying degrees of certainty
in the explanations, ranging from expressions of low confidence (e.g., ”I am
not sure the answer is [B] because”) to high confidence (e.g., ”I am confident
the answer is [B] because”). By modifying the language of the LLM’s responses
to better reflect model confidence, users showed improved calibration in their
assessment of the LLM’s reliability and were better able to discriminate
between correct and incorrect answers. This improvement underscores the
importance of transparent communication from LLMs, suggesting a need for
researchers to investigate how model explanations affect user perception.

One limitation of the current study is the focus on a specific type of
question (multiple-choice) and the assessment of a single dataset (MMLU). The
extent to which these results apply to other types of questions and datasets
remains an open question. Further research could investigate the applicability
of our findings across a broader range of scenarios, as well as the challenge
of calibrating human perception in response to open-ended questions. Another
limitation of this study is that our approach to modifying the prompt based on
internal uncertainty required the LLM to be prompted twice: once to read out
the answer and model confidence, and again to produce an explanation modified
by the model confidence. Future research could investigate how to produce
confidence-modified explanations in a single step.

In conclusion, our research highlights the critical role of clear and accurate
communication in the interaction between users and LLMs. Enhancing the
alignment between model confidence and the user’s perception of model
confidence can lead to a more responsible and trustworthy use of LLMs,
particularly in areas where the accuracy of AI-generated information is
critical.

Acknowledgments

This research was supported by NSF under award 1900644

## References

  * Ali et al. (2023) Stephen R Ali, Thomas D Dobbs, Hayley A Hutchings, and Iain S Whitaker.  Using chatgpt to write patient clinic letters.  _The Lancet Digital Health_ , 5(4):e179–e181, 2023.
  * Anil et al. (2023) Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu.  Palm 2 technical report, 2023.
  * Azaria and Mitchell (2023) Amos Azaria and Tom Mitchell.  The internal state of an llm knows when it’s lying, 2023.
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld.  Does the whole exceed its parts? the effect of ai explanations on complementary team performance.  In _Proceedings of the 2021 CHI conference on human factors in computing systems_ , pages 1–16, 2021.
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos.  To trust or to think: Cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making.  _Proceedings of the ACM on Human-Computer Interaction_ , 5(CSCW1):1–21, 2021.
  * Budescu et al. (2014) David V Budescu, Han-Hui Por, and Michael Broomell, Stephen B andSmithson.  The interpretation of IPCC probabilistic statements around the world.  _Nat. Clim. Chang._ , 4(6):508–512, 2014.
  * Dhami and Wallsten (2005) Mandeep K. Dhami and Thomas S. Wallsten.  Interpersonal comparison of subjective probabilities: Toward translating linguistic probabilities.  _Memory and Cognition_ , 33(6):1057–1068, 2005.
  * et al. (2023) Aarohi Srivastava et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=uyTL5Bvosj.
  * Feng and Boyd-Graber (2019) Shi Feng and Jordan Boyd-Graber.  What can ai do for me? evaluating machine learning interpretations in cooperative play.  In _Proceedings of the 24th international conference on intelligent user interfaces_ , pages 229–239, 2019.
  * Fleming and Lau (2014) Stephen M Fleming and Hakwan C Lau.  How to measure metacognition.  _Frontiers in human neuroscience_ , 8:443, 2014.
  * Geng et al. (2023) Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych.  A survey of language model confidence estimation and calibration, 2023.
  * Gruber and Buettner (2022) Sebastian Gruber and Florian Buettner.  Better uncertainty calibration via proper scores for classification and beyond.  _Advances in Neural Information Processing Systems_ , 35:8618–8632, 2022.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On calibration of modern neural networks.  In Doina Precup and Yee Whye Teh, editors, _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1321–1330. PMLR, 06–11 Aug 2017.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * Ho et al. (2015) Emily H Ho, David V Budescu, Mandeep K Dhami, and David R Mandel.  Improving the communication of uncertainty in climate science and intelligence analysis.  _Behav. Sci. Policy_ , 1(2):43–55, 2015.
  * Hoffmann et al. (2023) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models, 2023.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  2023\.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977, 2021.
  * Jo (2023) A Jo.  The promise and peril of generative ai.  _Nature_ , 614(1):214–216, 2023.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al.  Language models (mostly) know what they know, 2022.
  * Karelitz and Budescu (2004) Tzur M. Karelitz and David V. Budescu.  You say ”probable” and i say ”likely”: Improving interpersonal communication with verbal probability phrases.  _Journal of Experimantal Psychology_ , 10(1):25–41, 2004.
  * Karelitz et al. (2002) Tzur M. Karelitz, Mandeep K. Dhami, David V. Budescu, and Thomas S. Wallsten.  _Proceedings of the Fifteenth International Florida Artificial Intelligence Research Society Conference_ , pages 498–502, 2002.
  * Kumar et al. (2019) Ananya Kumar, Percy S Liang, and Tengyu Ma.  Verified uncertainty calibration.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans.  Teaching models to express their uncertainty in words.  _Transactions on Machine Learning Research_ , 2022.  ISSN 2835-8856.  URL https://openreview.net/forum?id=8s8K2UZGTZ.
  * Lin et al. (2023) Zhen Lin, Shubhendu Trivedi, and Jimeng Sun.  Generating with confidence: Uncertainty quantification for black-box large language models, 2023.
  * Mielke et al. (2022) Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau.  Reducing conversational agents’ overconfidence through linguistic calibration.  _Transactions of the Association for Computational Linguistics_ , 10:857–872, 2022.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 29, 2015.
  * O’Brien (1989) Bernie J O’Brien.  Words or numbers? the evaluation of probability expressions in general practice.  _The Journal of the Royal College of General Practitioners_ , 39(320):98–100, 1989.
  * OpenAI (2022a) OpenAI.  Gpt-3.5, Nov 2022a.  URL https://platform.openai.com/docs/models/gpt-3-5.  Accessed: [Nov 24, 2023].
  * OpenAI (2022b) OpenAI.  Openai blog, Nov 2022b.  URL https://openai.com/blog/chatgpt.  Accessed: [Nov 24, 2023].
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: methods, analysis & insights from training gopher, 2021.
  * Rong et al. (2023) Yao Rong, Tobias Leemann, Thai-Trang Nguyen, Lisa Fiedler, Peizhu Qian, Vaibhav Unhelkar, Tina Seidel, Gjergji Kasneci, and Enkelejda Kasneci.  Towards human-centered explainable ai: A survey of user studies for model explanations.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–20, 2023.  doi: 10.1109/TPAMI.2023.3331846.
  * Rouder et al. (2012) Jeffrey N Rouder, Richard D Morey, Paul L Speckman, and Jordan M Province.  Default bayes factors for anova designs.  _Journal of mathematical psychology_ , 56(5):356–374, 2012.
  * Smith-Renner et al. (2020) Alison Smith-Renner, Ron Fan, Melissa Birchfield, Tongshuang Wu, Jordan Boyd-Graber, Daniel S. Weld, and Leah Findlater.  No explainability without accountability: An empirical study of explanations and feedback in interactive ml.  In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , CHI ’20, page 1–13, New York, NY, USA, 2020. Association for Computing Machinery.  ISBN 9781450367080.  doi: 10.1145/3313831.3376624.  URL https://doi.org/10.1145/3313831.3376624.
  * Steyvers and Kumar (2023) Mark Steyvers and Aakriti Kumar.  Three challenges for ai-assisted decision-making.  _Perspectives on Psychological Science_ , 2023.  ISSN 1745-6924.  doi: 10.1177/17456916231181102.
  * Tanneru et al. (2023) Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju.  Quantifying uncertainty in natural language explanations of large language models, 2023.
  * Thulasidasan et al. (2019) Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak.  On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  2023\.
  * Wallsten et al. (2008) Thomas S. Wallsten, Yaron Shlomi, and Hisuchi Ting.  Exploring intelligence analysts’ selection and interpretation of probability terms.  2008\.
  * Wang and Yin (2022) Xinru Wang and Ming Yin.  Effects of explanations in ai-assisted decision making: Principles and comparisons.  _ACM Transactions on Interactive Intelligent Systems_ , 12(4):1–36, 2022.
  * Whalen et al. (2023) Jeromie Whalen, Chrystalla Mouza, et al.  Chatgpt: Challenges, opportunities, and implications for teacher education.  _Contemporary Issues in Technology and Teacher Education_ , 23(1):1–23, 2023.
  * Xiao et al. (2022) Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency.  Uncertainty quantification with pre-trained language models: A large-scale empirical analysis.  In _Findings of the association for computational linguistics: EMNLP 2022_ , pages 7273–7284, 2022.
  * Xiong et al. (2023) Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi.  Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms, 2023.
  * Yin et al. (2023) Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang.  Do large language models know what they don’t know?  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8653–8665. Association for Computational Linguistics, 2023.
  * Zambrano et al. (2023) Andres Felipe Zambrano, Xiner Liu, Amanda Barany, Ryan S Baker, Juhan Kim, and Nidhi Nasiar.  From ncoder to chatgpt: From automated coding to refining human coding.  In _International conference on quantitative ethnography_ , pages 470–485. Springer, 2023.
  * Zhou et al. (2023) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto.  Navigating the grey area: How expressions of uncertainty and overconfidence affect language models, 2023.
  * Zhou et al. (2024) Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap.  Relying on the unreliable: The impact of language models’ reluctance to express uncertainty, 2024.

##  A Additional Results

Table 2 shows the full set of results across experiments and LLMs. The table
also includes results for an additional performance metric, the Overconfidence
Error (OE). The metric is an adaptation of the ECE formula, specifically
focusing on cases of overconfidence Thulasidasan et al. (2019):

$\mathrm{OE}=\sum_{m=1}^{M}\frac{|B_{m}|}{N}[conf(B_{m})\times\mathrm{max}(0,conf(B_{m})-acc(B_{m}))]$
(4)

This penalizes predictions by the weight of the confidence but only when
confidence exceeds accuracy.

Table 2: Expected Calibration error (ECE), Overconfidence Error (OE), and Area under the Curve (AUC) of model and human confidence across experiments. LLM | Confidence Score | ECE | OE | AUC
---|---|---|---|---
GPT-3.5 |  |  |  |
| model confidence | .104 | .064 | .751
| human confidence: experiment 1, default explanations | .264 | .220 | .589
| human confidence: experiment 2, modified explanations | .183 | .145 | .692
| human confidence: experiment 3, modified explanations | .158 | .121 | .678
PaLM2 |  |  |  |
| model confidence | .154 | .098 | .746
| human confidence: experiment 1, default explanations | .291 | .229 | .602
| human confidence: experiment 2, modified explanations | .134 | .078 | .655
| human confidence: experiment 3, modified explanations | .195 | .155 | .689

##  B Additional Model Confidence Results

Figure 7 shows the calibration diagrams for the full set of 14,042 test
questions from the MMLU dataset. For GPT-3.5, the accuracy across all
questions is 63% with an AUC of 0.78. When computing model confidence, 8.7% of
the answers were incomplete and were removed from consideration. For PaLM2,
the accuracy is 51% with an AUC of 0.73. For comparison, Figure 8 shows the
calibration diagrams for the subset of 350 questions used for the behavioral
experiments.

Furthermore, we confirmed that we could replicate the GPT-4 Technical Report’s
OpenAI (2023) five-shot results. Five-shot prompting with GPT-3.5 (detailed in
Appendix 8 of the report) resulted in 71% accuracy (compared to 70% reported
in Table 2 of the report). The zero-shot approach is the focus of this paper.
The zero-shot approach simplifies the construction of explanations, and our
goal is not to maximize language model accuracy.

Figure 7: Calibration diagram for GPT-3.5 and PaLM2 model for full set of MMLU
questions.

Figure 8: Calibration diagram for GPT-3.5 and PaLM2 model for subset of 350
questions of MMLU questions used for behavioral experiments. Note that the
lower count of questions in the lowest confidence bin is due to the sparsity
of questions in that confidence bin.

##  C Optimization Procedure

To apply the selection rule, we reduced Experiment 2’s six explanation styles
to three. Therefore, for both Experiment 2 and 3, the selection rule
considered a choice of one of three explanation styles for each question
(i.e., low, medium, and high confidence explanations).

The parameters $\theta_{1}$ and $\theta_{2}$ were chosen to optimize a
combination of the ECE and AUC score (weighting the ECE by 33% relative to
AUC). The optimization was performed separately for Experiments 2 and 3 and
the two LLMs. The parameters were optimized using a basic grid search with
values ranging from 0.25 to 1, with the constraint that
$\theta_{1}<\theta_{2}$. For GPT-3.5, Experiment 2’s optimized parameters were
$\theta_{1}=0.55$ and $\theta_{2}=0.65$, while Experiment 3’s optimized
parameters were $\theta_{1}=0.75$ and $\theta_{2}=0.95$. For PaLM2, Experiment
2’s optimized parameters were $\theta_{1}=0.80$ and $\theta_{2}=0.85$, while
Experiment 3’s optimized parameters were $\theta_{1}=0.70$ and
$\theta_{2}=0.75$.

###  C.1 Sensitivity Analysis

The calibration and discrimination results are not overly sensitive to
parameter changes. The resulting AUC and ECE outcomes for each parameter
setting are shown in Figures 9 and 10. For example, for GPT-3.5, Experiment 2
yields mean AUC and ECE values of 0.631 and 0.217 across all parameter
settings, which are higher than the results of Experiment 1. Furthermore,
there is moderate evidence (BF¿3) for an improved AUC for 41% of parameter
combinations. The parameter settings in Experiment 3 had more selective
effects. There was only moderate evidence (BF ¿ 3) for an improved AUC for 17%
of parameter settings. The smaller effects of the explanation styles in
Experiment 3 presumably made the performance results more dependent on the
specific selection rule.

Figure 9: Sensitivity analysis for Experiment 2. Results show the effect of
different thresholds $(\theta_{1},\theta_{2})$ to select explanation styles.
Top and bottom panels show the resulting AUC and ECE that relate the human
confidence to the actual accuracy of the LLM.

Figure 10: Sensitivity analysis for Experiment 3. Results show the effect of
different thresholds $(\theta_{1},\theta_{2})$ to select explanation styles.
Top and bottom panels show the resulting AUC and ECE that relate the human
confidence to the actual accuracy of the LLM.

##  D Prompt Styles Across Experiments

Table 3: Prompt styles across experiments Prompt Style |  Prompt
---|---
Experiment 1 |
Baseline |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is: [answer]. Provide an explanation for the answer you gave. In your explanation, you must include the answer.
Experiment 2 |
Low Confidence & Long |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is [answer]. Write an explanation why you are not sure that the answer is [answer]. In your explanation, mention that you are not sure and include the answer. Start with ’I am’:”
Medium Confidence & Long |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is [answer]. Write an explanation why you somewhat sure that the answer is [answer]. In your explanation, mention that you are somewhat sure and include the answer. Start with ’I am’:
High Confidence & Long |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is [answer]. Write an explanation why you are sure that the answer is [answer]. In your explanation, mention that you are sure and include the answer. Start with ’I am’:
Low Confidence & Short |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is [answer]. Write a very short explanation why you are not sure that the answer is [answer]. In your explanation, mention that you are not sure and include the answer. Use as few words as possible. Start with ’I am’:
Medium Confidence & Short |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is [answer]. Write a very short explanation why you somewhat sure that the answer is [answer]. In your explanation, mention that you are somewhat sure and include the answer. Use as few words as possible. Start with ’I am’:
High Confidence & Short |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is [answer]. Write a very short explanation why you are sure that the answer is [answer]. In your explanation, mention that you are sure and include the answer. Use as few words as possible. Start with ’I am’:
Experiment 3 |
Low Confidence |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is: [answer]. The explanation you give is: [baseline explanation]. Rewrite the explanation as if you are not sure and ensure that you include language or phrases that conveys your complete lack of confidence. In your explanation, you must include the answer:
Medium Confidence |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is: [answer]. The explanation you give is: [baseline explanation]. Rewrite the explanation as if you are somewhat sure and ensure that you include language or phrases that conveys your lack of confidence. In your explanation, you must include the answer
High Confidence |  Problem: [question] Choose from the following options: [A] Option A [B] Option B [C] Option C [D] Option D. The answer you give is: [answer]. The explanation you give is: [baseline explanation]. Rewrite the explanation as if you are very sure and ensure that you include language or phrases that conveys your confidence. In your explanation, you must include the answer: