# Decision Theoretic Foundations for Experiments Evaluating Human Decisions

Jessica Hullman<EMAIL_ADDRESS>Northwestern
UniversityEvanstonIllinoisUSA ,  Alex Kale<EMAIL_ADDRESS>University of
ChicagoChicagoIllinoisUSA and  Jason Hartline<EMAIL_ADDRESS>Northwestern UniversityEvanstonIllinoisUSA

###### Abstract.

Decision-making with information displays is a key focus of research in areas
like explainable AI, human-AI teaming, and data visualization. However, what
constitutes a decision problem, and what is required for an experiment to be
capable of concluding that human decisions are flawed in some way, remain open
to speculation. We present a widely applicable definition of a decision
problem synthesized from statistical decision theory and information
economics. We argue that to attribute loss in human performance to forms of
bias, an experiment must provide participants with the information that a
rational agent would need to identify the normative decision. We evaluate the
extent to which recent evaluations of decision-making from the literature on
AI-assisted decisions achieve this criteria. We find that only 6 (17%) of 35
studies that claim to identify biased behavior present participants with
sufficient information to characterize their behavior as deviating from good
decision-making. We motivate the value of studying well-defined decision
problems by describing a characterization of performance losses they allow us
to conceive. In contrast, the ambiguities of a poorly communicated decision
problem preclude normative interpretation. We conclude with recommendations
for practice.

††submissionid: 1000

##  1\. Introduction

Providing informative interfaces to aid decision-making is a common goal in
research in human-centered artificial intelligence (HCAI), visualization, and
related areas. For example, visualization researchers emphasize assisting
decision-making as a vital goal of data visualization (Dimara and Stasko,
2021). Similarly, in human-centered AI, a proliferation of empirical studies
on human decision behavior have been deemed “necessary to evaluate the
effectiveness of AI technologies in assisting decision making, but also to
form a foundational understanding of how people interact with AI to make
decisions” (Lai et al., 2023).

Research areas that rely heavily on decision-making would be more coherent if
there was a consensus on what a decision is, and how decisions are a distinct
form of human behavior compared to tasks that are not considered decisions.
Human decisions from prediction displays are often evaluated in studies with
the goal of identifying forms of bias: ways in which study participants’
behavior deviates from expectations of good decision-making. Reviewing the
range of tasks that get filed under decision-making, it is difficult to
ascertain what common task structure makes these studies about decision-
making, much less what properties a study needs to have in order for it to be
capable of distinguishing human decisions as flawed. Colloquial definitions
describe a decision task as a choice between alternatives, or a choice where
the stakes are high (such as deciding which of multiple hurricane forecasts is
more accurate (Padilla et al., 2015), or whether to screen child welfare cases
(Zytek et al., 2021)), but neither isolates a clear set of assumptions that
must hold for a judgment to qualify as a decision. Other accounts cast
decision-making as a higher level task akin to sensemaking (Dimara and Stasko,
2021). For some, the point of studying human decisions is to compare them to a
definition of idealized behavior. However, sometimes researchers define this
standard in a way that is common across studies (e.g., a better forecast comes
closer to the true probability of the event (Green and Chen, 2019; Fogliato et
al., 2021)) while other times different definitions of the same concept are
used (e.g., appropriate reliance as agreeing with an AI when it is correct and
disagreeing when it’s not (Bansal et al., 2021; Buçinca et al., 2021) versus
providing a higher subjective rating of trust when the model is correct
(Bussone et al., 2015) versus choosing the better of the human versus the AI
prediction to bet on for a future task (Dietvorst et al., 2015)). Sometimes
the point is to study seemingly “subjective decisions,” like how people react
to music recommendations (Kulesza et al., 2012), or decide whether a house
seems to match its valuation (Prabhudesai et al., 2023), where it can seem
unclear whether normative interpretations of task performance can be applied.
Hence, not only is a more coherent definition of decision needed, as recent
survey papers on decision making in visualization and human-centered AI call
for (Dimara and Stasko, 2021; Lai et al., 2021), but such a definition must
clearly define requirements for identifying forms of bias or performance loss
in human decision-making.

We argue for the identification of a minimum set of theoretical commitments
for the task induced by a study of human decision-making from information
displays to be “well-defined:” that is, for it to be possible to establish
normative behavior. Fortunately, existing theories take on this challenge.
Statistical decision theory and expected utility theory (Savage, 1972; Steele
and Stefánsson, 2015; Von Neumann and Morgenstern, 2007) provide a rigorously
derived and widely applicable framework for studying decision-making.
Information economics formalizes information structures for decision
problems(Bergemann and Morris, 2019), which may by induced by design choices
about how to visualize or explain model predictions.

We synthesize a widely applicable definition of a well-defined decision
problem from statistical decision theory and information economics, and
motivate the value of this approach to HCAI and related research concerning
decision-making from data-driven interfaces. Our first contribution is to
establish and motivate a minimum set of components that must be defined for a
decision problem to allow identification of an optimal decision so as to
identify bias in human decisions. Using the concept of a rational Bayesian
agent, we show how performance loss can only be conceived of when a study
participant could in theory identify the normative decision from the
information they are provided. We find that while 35 of a sample of 46
existing studies draw conclusions about shortcomings in human decisions with
prediction displays, only for 6 (17%) of these is the study task well-defined
in that participants are given sufficient information to identify, at least in
principle, the best decision. We use examples to illustrate the
epistemological risks to such conclusions and provide recommendations for
experimenters to improve the interpretability of their study results.

##  2\. Defining a Decision Problem

We define a decision problem and corresponding standard for optimal behavior
so as to identify performance loss relative to that standard. Our definition
is intended for controlled evaluation of human behavior, a.k.a. normative
decision research. Such evaluative studies require that ground truth can be
identified for any states that the study participant is queried about.
Behavioral data (which may be produced by humans or by simulation) is
collected under controlled conditions, with the goal of learning about
behavior induced by the provision of information. Such studies are frequently
used to describe the quality of human performance in some situation (e.g., how
well people make decisions from displays in strategic settings (Zhang et al.,
2023)), to rank different assistive elements by human performance (e.g,
different visualizations (Correll et al., 2018) or AI explanation strategies
(Liu et al., 2021)), or to test a hypothesis about how humans make decisions
or what will help them do better (e.g., cognitive forcing functions will
improve AI-assisted decisions (Buçinca et al., 2021)).

###  2.1. Decision Problem

Our definition of a decision problem for controlled study starts by assuming a
scenario in which there is some state of the world $\theta$ which is uncertain
at the time of the decision (uncertain state). The state is drawn from a state
space $\Theta$ that describes the set of finite, mutually exclusive values
that it can take.

The data-generating model is a distribution over values of the uncertain state
and signals that provide information to the agent about the state. The data-
generating model associates the state with the information that the agent is
presented with (the signal) in order to inform their decision. Since the
signal is correlated with the state, if the participant understands it in the
context of the decision problem, they can improve their performance compared
to a case where they do not have access to the signal.

When we present a person (hereafter agent) with a decision problem, we give
them a choice of response (action) $a$ from some set of possible responses
(action space) $A$. In decision tasks corresponding to prediction tasks, the
action space is a probabilistic belief over the state space, i.e.,
$A=\Delta(\Theta)$. The agent faced with a decision task must consider the
probability of different states induced by the signal, and choose the best
action under the scoring rule, a function that assigns a score representing
the quality of the action for the realized state.

We refer to the process that produces signals that inform on the uncertain
state as a signaling policy. The relevance of the signal to the decision task
is captured by the correlation between the signal and the uncertain state.
More formally, a signaling policy induces an information structure that is
given by a joint distribution $\pi\in\Delta(V\times\Theta)$ over signals $v\in
V$ and states $\theta\in\Theta$. This joint distribution assigns to each
realization $(v,\theta)\in V\times\Theta$ a probability, denoted
$\pi(v,\theta)$.

If we ignore the signal $v$, $\pi$ gives rise to a probability distribution
over values of the uncertain state, which we denote as the prior distribution
$p\in\Delta(\Theta)$. The prior $p(\theta)$ is equivalent to
$\sum\nolimits_{v\in V}\pi(v,\theta)$.

Under this definition, the formalization of the information structure $\pi$
enables a definition of normative behavior. Summarizing information about
$\theta$, $\pi$ (as well as $p$) provides a basis for the experimenter to
establish expectations under optimal use of the information.

###  2.2. Normative Behavior

Given a decision problem defined as above, we calculate the normative
(“optimal”) decision by making assumptions about what it means for an agent to
have coherent preferences and to optimally decide between actions under
uncertainty about the outcome. Consequently, we can interpret an experiment
participant’s performance as an attempt at achieving this standard, and
identify sources of error (loss) in performance.

To do so, we will first assume that the agent’s preferences can be summarized
by the scoring rule $S$. We assume that she will choose the action that
maximizes her expected utility (score):

(1) $\displaystyle U(a)=S(a)$ $\displaystyle=\text{\bf E}_{\theta\sim
q}\\![{S(a,\theta)}]$

where $q$ describes the agent’s belief distribution, the probability
distribution over the states that the agent believes the states of the world
are drawn from.

We can define the optimal action as that which maximizes the agent’s expected
utility:

(2) $\displaystyle a_{opt}=\operatorname*{arg\,max}_{a\in A}\text{\bf
E}_{\theta\sim q}\\![{S(a,\theta)}]$

####  2.2.1. Rational Belief Formation

To define the normative action for a decision problem, we must first
characterize how we would ideally expect the agent to form the posterior
beliefs $q$ about the state given the signal $v$. A theory of optimal belief
construction starts by assuming that the ideal agent perfectly perceives $v$.
Once perceived, how does $v$ inform the agent’s beliefs about $\theta$?
Generally, we assume that the agent has some baseline set of beliefs about
$\pi$ that they could use to choose an action in the absence of
$v$.Encountering $v$ leads the agent to Bayesian update their beliefs.

Specifically, to calculate the optimal decision for a decision task we first
define what the agent perceives the probability distribution over the state to
be prior to obtaining information from $v$: $Pr(\theta)$ or $p(\theta)$ as we
describe above. Whenever the signaling policy does not reveal $\pi(\theta|v)$
directly via the signal, but does inform on $\theta$, we assume that after
seeing the signal, the agent updates their prior belief to a posterior belief
over signals and states, using Bayes rule based on what they have learned
about the data-generating model $\pi$:

(3) $\displaystyle q(\theta)$
$\displaystyle=\tfrac{Pr(v|\theta)Pr(\theta)}{Pr(v)}.$

where $Pr(v)$ is a normalizing factor. Note that the definition in Equation 3
implies that to calculate $q(\theta)$ the agent knows $\pi$:

(4) $\displaystyle
q(\theta)=\pi(\theta|v)=\tfrac{\pi(v,\theta)}{\sum\nolimits_{\theta\in\Theta}\pi(v,\theta)}.$

Given these posterior beliefs, we use Equation 2 to identify the action that a
fully rational agentwould choose in order to maximize their expected utility
under $S$.

###  2.3. Example

To make these concepts concrete, imagine a study of AI-advised decisions in
which people make recidivism decisions upon seeing features of a defendant.
The state is whether the person will commit another crime, i.e., recidivate,
within some duration after being released, which can take the value 0 for no
or 1 for yes ($\theta\in\\{0,1\\}$). The action space presented to the agent
is whether or not to release the defendant ($a\in\\{0,1\\}$).

The data-generating model is the joint distribution over the state (recidivate
or not) and signals comprised of defendant feature values and AI prediction
information. Assume that agents are presented with a signal in the form of a
set of feature values describing a defendant (age, prior arrests, prior
felonies, etc.) from which they must decide on an action. Imagine that the
signal also includes the prediction from a model that agents are told was
trained on a historical dataset of labeled examples, along with the
probability that the model’s prediction is correct. Agents can optionally use
this information to inform their decision. Here, the joint distribution
$\pi\in\Delta(V\times\Theta)$ assigns a probability, for each possible value
of $\theta$ ({0, 1}), to each possible signal (a combination of feature values
and model prediction with associated probability of correctness) used as a
trial in the experiment.

If we did not show the agent any feature values at all (nor model
predictions), we would expect them to make their choice based on their a
priori sense of the probability that a randomly drawn defendant would
recidivate. The prior distribution $p$ gives the probability that a defendant
recidivates as defined by the data-generating model before viewing the signal
given on an particular trial.

Experiment designs can vary in the amount of information they provide
participants about $\pi$. What is critical is that a participant has some
opportunity to obtain information about $\pi$. For example, imagine a version
of the recidivism study in which the agent is given only the feature values
and model prediction, with no information to link these to the state (i.e., no
probability that the model prediction is correct, and no feedback on the
realized state after their decision). Consequently, they have no opportunity
to learn about $\pi$, which is necessary to establish a normative decision.

###  2.4. Scoring Rules

Critical to a decision problem is a scoring rule $S$ :
$A\times\Theta\to{\mathbb{R}}$, a function that maps the action and state to a
quality or payoff. A scoring rule has two possible uses: to incentivize
participants and to evaluate the quality of the decision responses.

Returning to the recidivism example, assume the researcher uses a binary
scoring rule to map the ground truth realization of the state
($\theta\in{0,1}$ for not recidivate, recidivate) and the participant’s
selected action from the action space ($a\in\\{0,1\\}$), say, releasing the
defendant ($a=1$), to an outcome. For example, the agent might be scored by
the following rule, which imposes the greatest penalty on false positives
(i.e., acting as if the defendant will recidivate when they will not):

(5) $S(a,\theta)=\left\\{\begin{array}[]{ccc}0.2&\text{if
}a=0,\theta=0&\textit{release, does not recidivate }\\\ -0.2&\text{if
}a=0,\theta=1&\textit{release, recidivates}\\\ -1&\text{if
}a=1,\theta=0&\textit{not release, does not recidivate}\\\ 0.5&\text{if
}a=1,\theta=1&\textit{not release, recidivate}\end{array}\right.$

where the score could be in units of USD, for example.

Whether the researcher uses the same scoring rule for both incentives and
evaluation, or two different rules, has important implications for
interpreting the results. For the recidivism example, if the researchers
analyzed the collected responses using the above rule (i.e., reporting
performance in payoff space) in addition to communicating it to participants
for incentivization, then the utility associated with performance is defined
by a single scoring rule, and the interpretation of a “good” decision is
consistent across incentivization and evaluation. If we imagine instead that
the researchers decided to consider errors on a more equal footing in
analyzing their data, for example reporting the results in terms of accuracy,
where each type of error is associated with the same cost, the meaning of a
given action differs between incentivization and evaluation. Because a scoring
rule induces a meaning on actions by defining a range of beliefs for which an
action is optimal, a challenge to interpretation arises whenever two different
rules are used. One rule will by definition “misinterpret” the actions of the
other by implying a different normative standard.

An exception to the problems induced by using two different rules occurs when
the rule used for incentivization has certain properties. Specifically, among
scoring rules for beliefs, we can distinguish those that are proper scoring
rules: scoring rules for which the optimal action is to predict the true
distribution, i.e., $p\in\operatorname*{arg\,max}_{a\in A}S(a,p)$. When a
proper scoring rule is used, the researcher can assume that an agent who
understands the rule properly will report their true beliefs, as it is not
possible to score higher by reporting some alternative beliefs. These beliefs
can then be plugged into a different scoring rule without the risk of
misinterpretation.

For any non-proper scoring rule $S:A\times\Theta\to{\mathbb{R}}$ there is an
equivalent proper scoring rule
$\hat{S}:\Delta(\Theta)\times\Theta\to{\mathbb{R}}$ defined by playing the
optimal action under the reported belief. Formally,

(6) $\displaystyle\hat{S}(p,\theta)$
$\displaystyle=S(\operatorname*{arg\,max}\nolimits_{a\in A}S(a,p),\theta).$

In practice, decision problems with small actions spaces may be less subject
to these challenges. Applying the construction in Equation 6 to a binary
decision gives a threshold rule where for each state, all reported
probabilities below a threshold receive one score and those above receive
another score. Since the payoff depends only weakly on the belief, such
scoring rules may be less sensitive to the agents’ understanding of the
signal. For this reason, including a decision problem that directly also
elicits beliefs using a proper-scoring rule is a good idea whenever eliciting
responses using an action space that is not over beliefs, as it enables
plugging the elicited beliefs into other scoring rules.111See, e.g., Wu et al.
(2023)’s analysis of (Kale et al., 2020).

In general, the choice of possible scoring rules will depend on the nature of
the action space. If, for example, the researcher designed the recidivism
decision study to elicit the agent’s prediction of the probability of
recidivism, they could employ the quadratic scoring rule (i.e., squared loss)
to score the accuracy of the probabilistic beliefs. In general, a coarser
action space (i.e., with fewer choices available to participants) will produce
less informative responses for the purpose of resolving observed performance
relative to a normative standard.

###  2.5. Learning and Feedback

Given a definition of normative behavior dictated by Bayesian decision theory,
we would expect the decision-maker to update their beliefs given any
information that informs on the joint distribution between signals and states.
One source of such information is showing the agent the realization of the
state after they make their choice. Imagine that participants in the
recidivism decision study see feedback (i.e., are told the realized state,
whether or not the defendant recidivated) after each decision they make.
Assuming the signal is not revealing of the posterior probability of the state
(recidivate or not), we would naturally expect the participants to learn about
$\pi$ from this feedback as they complete trials, affecting their posterior
beliefs (Equations 3 and 4).

###  2.6. Decision-Making Under Risk and Certainty

The phrase “decision-making under uncertainty” is defined in the economics
literature as the case where the decision-maker is not directly given the
probability distribution over the state but instead must infer it.
Alternatively, decision-making under risk describes the situation when this
distribution is assumed to be known to the decision-maker (Steele and
Stefánsson, 2015). Under such an assumption we simply replace $q$ in Equation
2 with the known distribution.

We can also easily account for so-called decision-making under certainty,
where the state is deterministic222For any uncertain decision, it is possible
to devise an analogous set of judgments where states are deterministic and
enumerate the entire state space, an approach which has been used to isolate
the effect of uncertainty from the complexity of reasoning about multiple
contingencies (Martínez-Marquina et al., 2019). Under certainty, we no longer
need to compute expectation to calculate the optimal decision:

(7) $\displaystyle a_{opt}=\operatorname*{arg\,max}_{a\in A}S(a,\theta)$

##  3\. Defining Bias in Responses to a Decision Problem

The primary motivations for using the above framework are epistemological in
nature: they concern what we can know about the results of an experiment. To
interpret the responses to a decision problem in an experiment on human
decision-making as evidence of a faulty decision process, the experiment must
provide the participant with enough information to in principle identify the
normative decision that will be used to judge their behavior. In other words,
does the experiment give the participant enough information to align their
understanding of the decision problem with its normative interpretation?

When an experiment does not provide sufficient information for the participant
to solve the task in principle, there is the potential for multiplicity
concerning the data-generating model. Such multiplicity has implications for
interpreting the study results. As an example, consider the common approach to
defining appropriate reliance on AI as agreeing with the AI when it is correct
and disagreeing when it is not (Bansal et al., 2021; Gonzalez et al., 2020;
Wang and Yin, 2021; Yang et al., 2020). Imagine a user receives information
about the model’s overall performance unconditional on the predicted class.
Consider two data generating models that are consistent with the provided
information: one where errors are always false positives, one where errors are
always false negatives. Imagine a user who can only correct false positives.
If we observe the user disagreeing with the model prediction some observed
proportion of the time, we would say they are overrelying under the first data
generating model, but underrelying under the second. However, both are
compatible with the information they were given.

It is worth noting that even if the researchers do not think about the
elements of their experiment in terms of the components identified above
(i.e., statespace, data-generating model, signaling policy, action space,
scoring rule), the design of their study still induces a decision problem
whenever two simple conditions hold (Wu et al., 2023). First, there must be
some uncertainty about the state (i.e., there is more than one data scenario
used to generate stimuli)333A single trial study where all participants view
the same signal would not qualify., and second, the researchers believe it is
possible to provide a scoring rule that is consistent across participants
(eliminating, for example, studies of emotional responses).

When these conditions are met, we can characterize forms of bias as loss in
decision performance according to four possible sources, which we next define.

###  3.1. Defining Sources of Loss

Consider the process implied by definitions in Sections 2.1-2.2:

  1. (1)

Agent obtains prior

  2. (2)

Agent processes signal

  3. (3)

Agent updates beliefs

  4. (4)

Agent takes optimal action

The four steps above identify four sources of loss representing possible
reasons why observed human performance in an experiment might not match the
normative standard.

Prior loss (1). The agent must obtain prior beliefs that match the prior
beliefs the researchers assume in analyzing the problem. Prior loss is the
loss in performance due to the difference between the agent’s prior beliefs
and those used by the researchers to calculate the normative standard.
Explicit specification of priors is rare in decision studies used to evaluate
interfaces in practice. The consequences of this vary depending on the nature
of the signaling strategy. When the signal is fully revealing of the state
(i.e., the posterior is a point mass), or directly provides the posterior
probability of the state, there is no need for the prior. However, whenever
the signal cannot directly provide the probability of the state, the prior
must be communicated for a well-defined decision problem.

Receiver loss (2). The agent processes the signal for the information it
provides about the value of the uncertain state. Receiver loss is the loss due
to the agent not properly extracting the information from the signal, for
example, because the human visual system constrains what information is
actually perceived, because the process of rendering the signal induces noise,
or because participants do not understand how to read the signal.

Updating loss (3). Next the agent updates their prior beliefs with the
information obtained. Updating loss is the loss due to the agent not updating
their prior beliefs according to Bayes rule with the information they obtained
from the signal, for example due to systematic deviations in human belief-
updating relative to rational belief updating as prescribed by Bayes rule
(e.g., (Camerer, 1995)).

Optimization loss (4). The agent decides by choosing a good action under their
posterior beliefs. Optimization loss is the loss in performance due to not
identifying the optimal action under the scoring rule. Optimization loss may
include what psycho-physicists call “lapse” responses (e.g., from fatigue or
inattention), a catch-all term used to denote responses that do not
necessarily reflect the underlying function capturing the person’s beliefs. Or
it may include “elicitation errors,” where what participants report on the
response scale does not reflect what they had in mind as a result of bias or
imprecision induced by the elicitation method.

An important implication of the proper scoring rule distinction for belief
reports is that in general we should not expect an agent to truthfully report
their beliefs when it is not in their best interest to do so. Consequently,
when non-proper rules are used for belief tasks, we cannot interpret a study
participant who does not report the appropriate beliefs after examining the
signal as demonstrating bias. Such “deviation” from what the researcher might
expect (honest reporting) is not technically a loss, it is the expected
behavior under that rule. In practice, researchers often use non-proper
scoring rules when eliciting predictions; see, e.g., (Gneiting, 2011) for
examples from finance where commonly used rules allow forecasters to score
better by not reporting the true conditional mean of the target random
variable.

Misunderstanding the decision problem. A potential contributor to all four
sources of loss above is the agent misunderstanding the decision task,
including the scoring rule, the statespace, and/or the action space. Even when
all necessary components of a decision problem are carefully communicated, the
participant might not read the instructions carefully, or might not understand
them. Consequently, we should think of any tests we run to validate hypotheses
about biases stemming from particular sources of performance loss as tests of
the joint hypothesis comprised of the target hypothesis and a hypothesis that
participants understood the task.

###  3.2. Examples

Researchers are limited in their ability to interpret responses to decision
experiments as biased when distinct sources of performance loss that account
for deviation from the normative standard cannot be conceived. We use several
examples inspired by the human-centered AI and visualization literature to
demonstrate why drawing conclusions about biased behavior is misleading in
such scenarios.

####  3.2.1. AI-Assisted Recidivism Decisions

We first elaborate the recidivism decision-making study mentioned earlier.
Imagine the study is designed to investigate whether people over-rely on AI
assistance when deciding whether to release (i.e., grant parole to)
incarcerated defendants. The researchers use a standard train-validate-split
approach on a large labeled historical data set to train a model that predicts
whether or not a person will recidivate. Each defendant in the dataset is
represented by a set of feature values, $X=x$, which comprise part of the
signal $v$ shown on each trial. Study participants are randomly assigned to
either a control condition where they see the feature values along with only
the model’s prediction and what is described as “the model’s confidence in its
prediction, ranging between 50% and 100%,” or an AI explanation condition
where they are also given additional information about how features in the
data contribute to the model’s predictions, e.g., generated by LIME (Ribeiro
et al., 2016). The model confidence score that is provided is the probability
of a correct prediction conditional on the feature values, $Pr(correct|X=x)$.

Participants are told the AI model was trained on prior defendants whose
status is known. They are told that the model has an accuracy of $80\%$
(representing the combined false positives and false negatives) on a sample of
instances drawn from the test set which comprise the trials in the study.

Each participant makes parole decisions for 20 unique defendants. On each
trial they are asked to choose whether or not to release the defendant
($a\in\\{0,1\\}$). Rather than the scoring rule given in Equation 5, imagine a
simpler version that assigns different types of errors equivalent costs.
Participants are given a base reward of $\$10USD$, but they are told they will
lose $\$0.5USD$ from this initial endowment for every trial where they either
release a person who will re-offend or continue to incarcerate a person who
will not re-offend. Participants are not told how often to expect recidivism;
i.e., the prior $p$ is not endowed. Participants are not given interim
feedback on their decisions.

After collecting data from 500 participants, the researchers model the
percentage of trials where the participants incorrectly follow the AI’s
prediction. They find a statistically significant difference between the AI
explanation condition and the control condition. The researchers interpret
this as evidence that AI explanations promote over-reliance.

Interpretation of Results. To assess what can be said about performance loss,
we first consider whether the problem is well-defined: could a rational agent
could use the provided information to select the utility-maximizing decision?

The action space $A$ ($a=1$ for grant parole, $a=0$ for deny parole) is
conveyed to participants, as well as the statespace $\Theta$ ($\theta=1$ for
the defendant re-offending, $\theta=0$ for not). They also have knowledge of
the scoring rule that assigns a score (reward amount) given the realized state
and their decision.

How would a rational participant form beliefs about the posterior probability
of the state using Equation 3? Participants are not provided with the prior
probability of the state. This would be acceptable if the signal provided them
with posterior beliefs about the uncertain state. However, the signal does not
provide the required posterior probability: recall that participants receive
the probability that the model is correct conditional on the features,
$Pr(correct|X=x)$, but not conditional on the model’s prediction. They
therefore cannot obtain sufficient information from the data-generating model
$\pi$ to guide their choice of action.

Improving the experimental design. One possibility is to make the signal
provide the posterior $q(\theta)$. This could be achieved if the confidence
value included in the signal were to give the probability that the model is
correct conditional on both the features and the prediction, e.g.,
$Pr(correct|X=x,y_{pred}=1)$ when the model predicts 1. In this setting,
provision of the prior would not be necessary (and may confuse participants).

However, the careful reader may also have noticed another barrier to defining
bias for the problem: the model’s conditional confidence is described as
simply “the model’s confidence in its prediction”, making it ambiguous from
the participant’s perspective whether it is an unconditional or conditional.
Given that the normative response is not well-defined in the unconditional
case even with the prior, this description would need specify that the
confidence is conditional on the predicted label to eliminate ambiguity.

In the case where the study presented only $Pr(correct|X=x)$, not e.g.,
$Pr(correct|X=x,y_{pred}=1)$ when the model predicts 1, what additional
information would equip the agent to optimize? If instead of the overall model
accuracy over the stimuli, the experimenters provided information about the
model’s error as separated rates of false positives and false negatives, the
participant would be equipped with enough information to infer $Pr(v|\theta)$
in Equation 3.

It is worth noting that decision problems do not necessarily have to depend on
small, discrete action spaces. Imagine another amendment to the recidivism
study where instead of asking participants for a parole decision, the
researchers ask them to report the probability of recidivism $Pr(\theta|v)$
(see, e.g., (Fogliato et al., 2021; Green and Chen, 2019)). Assuming the above
changes are made to the design to equip participants with information to form
posterior beliefs, participants could in principle report the optimal beliefs.
However, if the scoring rule for a decision problem where the action space is
over beliefs is not proper, we should not expect the agent to respond with
their true beliefs. Optimization loss might be low, but this is not useful to
the experimenters’ research questions if agents respond with a distortion of
their true beliefs.

####  3.2.2. Election Forecast Displays

Imagine a study targeting the effect of different ways of visualizing election
forecasts. The researchers develop a simple forecasting model to predict vote
share (a percentage) by weighting and aggregating poll results from news
sources for ten unique upcoming two-party state-level elections. Participants
are randomly assigned to view one of three forecast displays showing the
predicted vote share with uncertainty of each party’s candidate as a table, an
animated graphic, or a static graphic. Assume for simplicity that all three
induce informationally equivalent signals to a rational agent444If any of the
displays was not informationally equivalent to the others, the optimal
decision may vary by display. See Wu et al. (Wu et al., 2023) for analysis of
such examples using a rational agent framework..

Upon beginning the experiment, participants are randomly assigned to a display
treatment and also assigned a political party preference. They are told they
will view election forecasts from a model based on poll results and should
imagine they live in the district where the election takes place. Each
participant in the study does ten trials corresponding to the ten elections.
On each trial, they are asked if they would like to vote for the candidate in
the election, elicited using a checkbox listing these options.

Participants are told they will win a given base reward amount plus an
additional $\$0.25USD$ for each trial where their candidate wins a simulated
version of the election that uses the forecast model to simulate how other
people will vote (voting carries no cost). The researchers generate outcomes
by randomly drawing a bivariate vote share outcome from the predicted
distribution under the model and generating the uncertain state $\theta$
(taking the value 0 or 1 for lose or win) by assuming the higher vote share
candidate wins. Participants are not given feedback after each trial.

After collecting results for several hundred participants, the researchers fit
a linear model that estimates the sensitivity of the vote decision to the
closeness of the race, defined as the difference between the predicted vote
share of the first candidate and 0.5. Using statistical significance testing
they reject the null hypothesis of no difference between the estimated slopes
for the interaction between display type and race closeness across display
types, using posthoc comparisons to declare that the animated graphic leads to
better voting decisions because in induces the most sensitivity.

Interpretation of Results. For the researchers’ conclusions to be valid,
participants should be given all necessary information to arrive at posterior
beliefs informed by $v$ as shown in Equation 3, and to select the utility
maximizing action (vote or not) under the scoring rule for the voting task
(Equation 2). We first consider whether this is the case.

The action space $A$ ($a=1$ for vote, $a=0$ for not vote) is conveyed to
participants, and the task description also gives participants the statespace
$\Theta$ ($\theta=1$ for their candidate winning, $\theta=0$ for not).
Participants can infer the probability of their candidate winning from the
signal showing the model predicted vote share.

What do participants know about the realized outcome of the election that the
researchers describe simulating? The researchers described the use of the
forecast model to simulate voting behavior, and in implemention, do not
account for the participant’s vote in generating the outcome. Consequently,
whether a participant votes is irrelevant to the payoff-relevant state under
the scoring rule. However, the study design is also somewhat deceptive in that
the description they give is ambiguous when it comes to whether the
participant’s vote counts: some participants might assume, based on their
knowledge of real world elections, that it does.

Because, in the context of the study, voting carries no value but also no
cost, we cannot conceive of the four sources of loss. There is no well-defined
decision problem, and consequently, it does not make sense to ask whether
behavior is more biased or otherwise worse under one display condition.

Improving the experimental design. We could create a valid decision problem by
changing the scoring rule so that the reward amount depends on whether one
votes. For example, we could assign a cost of $\$0.25$ to voting:

(8) $S(a,\theta)=0.5\theta-0.25a=\left\\{\begin{array}[]{ccc}0&\text{if
}a=0,\theta=0&\textit{do not vote, lose election }\\\ 0.5&\text{if
}a=0,\theta=1&\textit{do not vote, win election}\\\ -0.25&\text{if
}a=1,\theta=0&\textit{vote, lose election}\\\ 0.25&\text{if
}a=1,\theta=1&\textit{vote, win election}\end{array}\right.$

However, the decision problem does not admit a well-defined normative response
until we address the ambiguity in the description of the data-generating
model. Only once participants are given a transparent description of how the
realized outcome is generated from the model can we conceive of prior loss as
well as updating loss, both of which should be zero if the problem is
understood because the prior does not affect the judgment. We can conceive of
receiver loss as the loss from participants not being able to extract the
information from the signal, and optimization loss as the loss from
participants not being able to choose the optimal action under their beliefs.

Of course, if we amend the description to inform participants of how the
election outcome is actually generated, we should expect participants who
understand the problem to never vote. Rather than informed political
engagement, the proportion of participants who vote (e.g., in each treatment
condition) would be better described as reflecting the level of confusion
about the task. Even if the researchers did not change the description of how
the realized state is generated, but designed the study to give participants
feedback on the realized state after each decision, then the participants
might in theory learn that their vote does not matter, which works against the
researchers’ goals of presenting a situation where people could feel compelled
to vote.

This example demonstrates that providing participants sufficient information
to determine the utility-maximizing action is necessary but not sufficient for
a meaningful decision problem in light of the research questions of interest.

###  3.3. Goals & Incentives in the Lab vs World

When participants are not given ample information to “solve” at least in
principle the decision problem, we cannot bring well-formed expectations to
the results, and should expect heterogeneity in the behaviors they exhibit.
Such “heterogeneity by design” renders the results uninterpretable for
evaluative purposes like identifying forms of bias.

Imagine that the researchers who designed the voting task intentionally did
not include the vote in the scoring rule, and were intentionally ambiguous
about the role of the vote in the data-generating model. Because the utility
of voting in a real-world election is typically left implicit and likely to
vary between people, the researchers might think that their results would
therefore be more representative of real world voting behaviors if the scoring
rule was ambiguous.

Such beliefs stem from a convention among many researchers toward being
optimistic about how their experimental observations relate to people’s
behavior in the “real world.” Most controlled decision studies are designed to
help us understand how people behave in some realistic setting that the
experimental task is meant to proxy for. However, in the real world, people
have goals and prior beliefs. We might not be able to perceive what utility
function each individual person is using, but we can assume that behavior is
goal-directed in one way or another. Savage’s axioms (Steele and Stefánsson,
2015) and the derivation of expected utility theory tell us that for behavior
to be rationalizable, a person’s choices should be consistent with their
beliefs about the state and the payoffs they expect under different outcomes.

Consequently, when people are in an experiment, we should not expect the
analogous real world goals and beliefs for that kind of task to apply. For
example, people might take actions in the real world for intrinsic value—e.g.,
voting in order to feel like a good citizen, or paying close attention to a
defendant’s profile because the decision of whether to grant parole or not is
expected to have a tremendous impact on that person’s life. It is difficult to
motivate people to take actions based on intrinsic value in an experiment,
unless the experiment is designed to elicit and describe those intrinsic
values, in which case the normative framework here no longer applies. Hence,
the idea that as experimenters we can skip fully describing the decision
problem to our participants while still retaining the ability to identify
their behavior as faulty or biased arises from a failure to recognize our
fundamental uncertainty about how the experimental context relates to the real
world.

##  4\. Applying the Framework to HCAI Studies

We use a sample of recent evaluative studies of AI-assisted decisions to
estimate how often researchers draw conclusions about flaws in human decisions
in the absence of a well-defined decision problem. We randomly sampled 46 of
the studies surveyed by Lai et al. (2023) in their overview of empirical
human-subjects studies on human-AI decision-making. According to their
inclusion criteria, the sample contains only evaluative human-subject studies
targeting a decision task in the context of classification or regression
published in ACM or ACL conference venues between 2018 and 2021.

Codes. In coding the studies, we focused on three aspects of the study design
and interpretation. First, we identified whether our framework is applicable.
As described above, to be applicable to a decision task, the task should allow
for defining a ground truth state and induce uncertainty in the payoff-
relevant state, at least across participants. Next, we identified whether the
authors draw conclusions about flaws or performance loss in human decisions
from their results, i.e., is the study evaluating human decision-making in
some way. Finally, we identified whether the study targeted a well-defined
problem; i.e., whether participants in the study were provided with sufficient
information to identify the optimal response, at least in principle, such that
evaluative conclusions are warranted. When the paper and supplemental
materials did not include enough information about the study for us to
identify whether the participants had sufficient information (which affected a
small handful of cases), we coded it as a negative example.

Two authors participated in the coding. After formulating the coding scheme,
the authors divided the sampled studies roughly evenly, each coding
independently, then discussing their results to ensure consensus. We provide
the list of sampled studies as Appendix.

Results. Of the 46 studies examined, 11 (24%) studied tasks for which ground
truth could not be determined, such as subjective appraisals of music or movie
recommendations or emotion. One of these studied accuracy, but from a purely
descriptive perspective of understanding what accuracy humans versus
algorithms could achieve on a task (Lin et al., 2020).

All 35 of the remaining studies drew conclusions about flaws in human
decisions, such as by pointing to over- or underreliance on the AI or
speculating about why participants failed to get the correct answer. However,
participants were only given sufficient information to identify the best
decision response across all conditions in 6 of these 35 (17%); in the other
29 cases it was either clear that they did not get sufficient information (24)
or only some conditions got sufficient information (5), e.g., through the
provision of feedback or model accuracy on held-out data. In the latter case,
informational differences between conditions that the researchers compared
lead to different a priori expectations of decision quality, yet these
differences were not accounted for in comparing participants’ behavior between
conditions. This prevents assessing how much of the possible benefit from a
treatment participants were able to attain. The calculation of separate
performance benchmarks would address such ambiguity (Wu et al., 2023).

Additionally, even when authors gave participants sufficient information to
identify the best response, they rarely compared the observed performance to
the best attainable performance. For example, multiple papers defined a
scoring rule to incentivize participants to make accurate decisions, but
evaluated the responses using a simpler accuracy scale, implying that 100%
accuracy was obtainable even when it would not be to a perfectly-perceiving
rational agent,and potentially misinterpreting the performance of participants
guided by a different scoring rule. In some of these cases, participants were
also given sufficient information to optimize, but it did not accurately
reflect the true data-generating model. The most common case was when the
researchers gave participants the model’s held-out accuracy on examples
different from those participants experienced. When the study design includes
feedback, this leads to a contradiction between the instructions and what
participants observe.

##  5\. Recommendations for Practice

The lack of consensus around how to define a decision problem and what the
minimum required components for normative analysis should be leads to research
landscape in which researchers can easily be tempted to overinterpret the
results of their studies, and readers may not recognize the
overinterpretations. Only when researchers define the task they study as a
complete decision problem can the results of experiments that study slightly
different problems (e.g., use different data-generating models) be
meaningfully contrasted to drive the development of theory. Adopting the
framework above, which is based in established frameworks for statistical
decision theory (Savage, 1972) and information design in economics (Bergemann
and Morris, 2019), also stands to improve researchers’ awareness of the
limitations on what be concluded from study results when a decision problem is
under-communicated.

The variation induced when a decision problem is not communicated is
consequential to interpretation in that it cannot be separated from variation
induced by true differences in how people do a task under different
conditions. Identifying the key sources of loss—much less separating them
given experiment results—is a hard problem made impossible when the researcher
has not clearly identified the decision problem to the study participant or
the reader of a resulting research paper. Consequently, researchers wishing to
design a normative decision experiment should communicate all necessary
components of the decision problem both to study participants and in
publications. Specifically, we recommend that researchers:

Clearly convey action and state spaces and a scoring rule. The finite action
space $A$ and statespace $\Theta$ should be clearly communicated to
participants, and the action space reflected in the instruments used to
collect participant responses. Because the optimal action can only be
determined relative to a scoring rule, participants should be given a scoring
rule to reduce heterogeneity in their choice of action, even if the
experimenter feels comfortable assuming that the participants will try their
best and does not use the scores in assigning monetary rewards.

Provide prior and/or sufficient information about the data-generating model to
calculate posterior. Sufficient information about the data-generating model
must be provided to arrive at the Bayesian posterior distribution. Whenever
the signal on its own does not induce a posterior, the prior should be endowed
(Equation 4), and sufficient information about $\pi$ should be provided to
enable estimating $Pr(v|\theta)$ and $Pr(v)$. Using presentation formats that
are more likely to be automatically processed (e.g., conveying distribution
information via frequency animations (Goldstein and Rothschild, 2014;
hullman2015hops)) or attempting to elicit participants’ priors directly (e.g.,
(Kim et al., 2019, 2020)) may help researchers minimize prior loss.

Explicitly motivate scoring rules for incentivization and evaluation. Beyond
defining the components of the target decision problem, a next step is for
researchers to motivate the problem definition they use. Recent work describes
the use of a rational agent framework similar to the one presented here to
analyze whether a decision study is likely to provide agents with sufficient
incentive to consult the signals (Wu et al., 2023). Theory around scoring
rules (Li et al., 2022) and what statistical quantities are elicitable
(Lambert, 2011) stand to offer prescriptive advice if incorporated in human
decision research.

Compare to best attainable performance. We observed that even when researchers
invest effort to design scoring rules to incentivize participants, they do not
often compare the observed behavioral performance to the best attainable
performance under the rule. Researchers should default to analyzing results in
the space of the scoring rule used for incentives, and provide a rationale
whenever they deviate. Unbiased loss estimation is a rich area for future
work.

Elicit measures to estimate specific sources of loss. An important area for
future work is to establish under what conditions the losses we conceptualize
can be estimated. Separating losses entails adding carefully constructed
questions to the experimental procedure, and/or constructing benchmarks based
in rational agent performance to separate sources of loss in observed results.
For example, Wu et al. (2023) use the notion of a calibrated behavioral agent
along with a rational agent benchmark and baseline (expected performance of a
rational agent under only the prior) to separate optimization and receiver
loss.

Conduct design analysis. Researchers doing normative decision research can
benefit from conducting analyses prior to running the study which simulate the
idealized actions of an agent to vet the experiment design (see, e.g., Wu et
al. (2023)’s rational agent-based pre-study analyses). Simulation-based design
analysis forces the researcher to explicitly define the parameters of their
decision problem, which can elucidate what information needs to be
communicated to a participant in order to enable an optimal response. It also
enables the researcher to test variations on their signaling strategy and
scoring rule, which can help the researcher realize under what conditions (1)
the experiment can have acceptable statistical power and (2) participants can
be paid fairly. It helps the researcher choose conditions (e.g., AI
explanations) that provide informative signals for the decision task,
narrowing down what kinds of research questions can be answered using a
particular decision problem.

##  6\. Limitations & Future Work

Several challenges have been raised against expected utility theory as put
forth by Savage (Savage, 1972); we recommend (Steele and Stefánsson, 2015) for
an overview. Another critique of a normative approach dubbed “value collapse”
(Nguyen, 2023) is that presenting a clear set of values as a target for
behavior limits variance in ways that can be harmful. Our argument is that we
should not mistake the variation that emerges from in an under-communicated
decision problem as representative of behavior under a particular set of
values or a particular real-world decision task. Studies aimed at
understanding the range of values at play in a naturalistic decision scenario
are better characterized as descriptive research. We note that while our
framework is intended for normative decision research, the definitions below
may also be useful if one is doing descriptive or exploratory decision
research, by making clear which aspects of the decision problem are
underspecified and therefore presumably the object of study. For example, a
researcher might aim to characterize challenges in model-assisted decisions in
a particular domain (Zytek et al., 2021), or conduct an exploratory study to
elicit information about the utility function (i.e., scoring rule) that
operates in a setting then communicate it back to new participants as part of
the decision problem in an evaluative experiment (e.g., (Fernandes et al.,
2018)).

Some have voiced pessimistic views on the value of monetary incentives for
experiment participants (e.g., (Read, 2005)). These points are somewhat
orthogonal to the crux of our argument, which is philosophical and concerned
with what is knowable more so than the pragmatics of how much behavior changes
when a problem is well-defined.

Scoring rules can sometimes be challenging for study participants to
interpret; theoretical incentive compatibility does not guarantee
understanding on the part of subjects. There is a rich literature discussing
possible descriptions that can be used with common rules like the quadratic
scoring rule (e.g., (Artinger et al., 2010)). Interfaces researchers may be in
a good position to extend this literature through careful observational
studies.

Ultimately, the goal of our work is to give researchers the tools to recognize
that when we do experiments, we are restricted to learning about behavior in
the artificial worlds we create. As much as we might want to equate our
results with some real world setting used as motivation, extrapolating from
the world of the controlled experiment to the real world will always be a leap
of faith. It is therefore the responsibility of the experimenter to fully
understand their experimental world, and, for the purposes of offering the
results as a form of potentially generalizable knowledge, to give their
participants a chance to do so as well.

## References

  * (1)
  * Artinger et al. (2010) Florian Artinger, Filippos Exadaktylos, Hannes Koppel, and Lauri Sääksvuori. 2010.  _Applying quadratic scoring rule transparently in multiple choice settings: A note_.  Technical Report. Jena Economic Research Papers.
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021\.  Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Bergemann and Morris (2019) Dirk Bergemann and Stephen Morris. 2019.  Information design: A unified perspective.  _Journal of Economic Literature_ 57, 1 (2019), 44–95.
  * Buçinca et al. (2021) Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021.  To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (2021), 1–21.
  * Bussone et al. (2015) Adrian Bussone, Simone Stumpf, and Dympna O’Sullivan. 2015\.  The role of explanations on trust and reliance in clinical decision support systems. In _2015 international conference on healthcare informatics_. IEEE, 160–169.
  * Camerer (1995) Colin Camerer. 1995\.  Individual decision making.  _Handbook of experimental economics_ (1995).
  * Correll et al. (2018) Michael Correll, Dominik Moritz, and Jeffrey Heer. 2018\.  Value-suppressing uncertainty palettes. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–11.
  * Dietvorst et al. (2015) Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015\.  Algorithm aversion: people erroneously avoid algorithms after seeing them err.  _Journal of Experimental Psychology: General_ 144, 1 (2015), 114\.
  * Dimara and Stasko (2021) Evanthia Dimara and John Stasko. 2021.  A critical reflection on visualization research: Where do decision making tasks hide?  _IEEE Transactions on Visualization and Computer Graphics_ 28, 1 (2021), 1128–1138.
  * Fernandes et al. (2018) Michael Fernandes, Logan Walls, Sean Munson, Jessica Hullman, and Matthew Kay. 2018.  Uncertainty displays using quantile dotplots or cdfs improve transit decision-making. In _Proceedings of the 2018 CHI conference on human factors in computing systems_. 1–12.
  * Fogliato et al. (2021) Riccardo Fogliato, Alexandra Chouldechova, and Zachary Lipton. 2021.  The impact of algorithmic risk assessments on human predictions and its analysis via crowdsourcing studies.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2 (2021), 1–24.
  * Gneiting (2011) Tilmann Gneiting. 2011\.  Making and evaluating point forecasts.  _J. Amer. Statist. Assoc._ 106, 494 (2011), 746–762.
  * Goldstein and Rothschild (2014) Daniel G Goldstein and David Rothschild. 2014.  Lay understanding of probability distributions.  _Judgment and Decision making_ 9, 1 (2014), 1–14.
  * Gonzalez et al. (2020) Ana Valeria Gonzalez, Gagan Bansal, Angela Fan, Robin Jia, Yashar Mehdad, and Srinivasan Iyer. 2020.  Human evaluation of spoken vs. visual explanations for open-domain qa.  _arXiv preprint arXiv:2012.15075_ (2020).
  * Green and Chen (2019) Ben Green and Yiling Chen. 2019.  Disparate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments. In _Proceedings of the conference on fairness, accountability, and transparency_. 90–99.
  * Kale et al. (2020) Alex Kale, Matthew Kay, and Jessica Hullman. 2020.  Visual reasoning strategies for effect size judgments and decisions.  _IEEE transactions on visualization and computer graphics_ 27, 2 (2020), 272–282.
  * Kim et al. (2020) Yea-Seul Kim, Paula Kayongo, Madeleine Grunde-McLaughlin, and Jessica Hullman. 2020.  Bayesian-assisted inference from visualized data.  _IEEE Transactions on Visualization and Computer Graphics_ 27, 2 (2020), 989–999.
  * Kim et al. (2019) Yea-Seul Kim, Logan A Walls, Peter Krafft, and Jessica Hullman. 2019.  A bayesian cognition approach to improve data visualization. In _Proceedings of the 2019 chi conference on human factors in computing systems_. 1–14.
  * Kulesza et al. (2012) Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012.  Tell me more? The effects of mental model soundness on personalizing an intelligent agent. In _Proceedings of the sigchi conference on human factors in computing systems_. 1–10.
  * Lai et al. (2021) Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. 2021.  Towards a science of human-ai decision making: a survey of empirical studies.  _arXiv preprint arXiv:2112.11471_ (2021).
  * Lai et al. (2023) Vivian Lai, Chacha Chen, Alison Smith-Renner, Q Vera Liao, and Chenhao Tan. 2023.  Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_. 1369–1385.
  * Lambert (2011) Nicolas S Lambert. 2011\.  Elicitation and evaluation of statistical forecasts.  _Preprint_ (2011).
  * Li et al. (2022) Yingkai Li, Jason D Hartline, Liren Shan, and Yifan Wu. 2022\.  Optimization of scoring rules. In _Proceedings of the 23rd ACM Conference on Economics and Computation_. 988–989.
  * Lin et al. (2020) Zhiyuan “Jerry” Lin, Jongbin Jung, Sharad Goel, and Jennifer Skeem. 2020.  The limits of human predictions of recidivism.  _Science advances_ 6, 7 (2020), eaaz0652.
  * Liu et al. (2021) Han Liu, Vivian Lai, and Chenhao Tan. 2021.  Understanding the effect of out-of-distribution examples and interactive explanations on human-ai decision making.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW2 (2021), 1–45.
  * Martínez-Marquina et al. (2019) Alejandro Martínez-Marquina, Muriel Niederle, and Emanuel Vespa. 2019\.  Failures in contingent reasoning: The role of uncertainty.  _American Economic Review_ 109, 10 (2019), 3437–3474.
  * Nguyen (2023) C. Thi Nguyen. 2023\.  Value Collapse. Presented as the Annual Cardiff Lecture of the Royal Institute of Philosophy.  https://royalinstitutephilosophy.org/event/value-collapse/
  * Padilla et al. (2015) Lace M Padilla, Grace Hansen, Ian T Ruginski, Heidi S Kramer, William B Thompson, and Sarah H Creem-Regehr. 2015.  The influence of different graphical displays on nonexpert decision making under uncertainty.  _Journal of Experimental Psychology: Applied_ 21, 1 (2015), 37\.
  * Prabhudesai et al. (2023) Snehal Prabhudesai, Leyao Yang, Sumit Asthana, Xun Huan, Q Vera Liao, and Nikola Banovic. 2023\.  Understanding Uncertainty: How Lay Decision-makers Perceive and Interpret Uncertainty in Human-AI Decision Making. In _Proceedings of the 28th International Conference on Intelligent User Interfaces_. 379–396.
  * Read (2005) Daniel Read. 2005\.  Monetary incentives, what are they good for?  _Journal of Economic Methodology_ 12, 2 (2005), 265–276.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Savage (1972) Leonard J Savage. 1972\.  _The foundations of statistics_.  Courier Corporation.
  * Steele and Stefánsson (2015) Katie Steele and H Orri Stefánsson. 2015.  Decision theory.  (2015).
  * Von Neumann and Morgenstern (2007) John Von Neumann and Oskar Morgenstern. 2007.  _Theory of games and economic behavior (60th Anniversary Commemorative Edition)_.  Princeton university press.
  * Wang and Yin (2021) Xinru Wang and Ming Yin. 2021.  Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In _26th international conference on intelligent user interfaces_. 318–328.
  * Wu et al. (2023) Yifan Wu, Ziyang Guo, Michails Mamakos, Jason Hartline, and Jessica Hullman. 2023.  The Rational Agent Benchmark for Data Visualization.  _IEEE Transactions on Visualization and Computer Graphics (forthcoming)_ (2023).
  * Yang et al. (2020) Fumeng Yang, Zhuanyi Huang, Jean Scholtz, and Dustin L Arendt. 2020.  How do visual explanations foster end users’ appropriate trust in machine learning?. In _Proceedings of the 25th international conference on intelligent user interfaces_. 189–201.
  * Zhang et al. (2023) Dongping Zhang, Jason Hartline, and Jessica Hullman. 2023\.  Designing Shared Information Displays for Agents of Varying Strategic Sophistication.  _arXiv preprint arXiv:2310.10858_ (2023).
  * Zytek et al. (2021) Alexandra Zytek, Dongyu Liu, Rhema Vaithianathan, and Kalyan Veeramachaneni. 2021.  Sibyl: Understanding and addressing the usability challenges of machine learning in high-stakes decision making.  _IEEE Transactions on Visualization and Computer Graphics_ 28, 1 (2021), 1161–1171.

##  7\. Appendix: List of Studies Coded

We randomly sampled and coded the following studies from Lai et al. (2021):

A. Abdul, C. von der Weth, M. Kankanhalli, B. Lim. (2020). COGAM: Measuring
and Moderating Cognitive Load in Machine Learning Model Explanations.

A. Alqaraawi, M. Schuessler, P. Weiß, E. Costanza, N. Berthouze. (2020).
Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User
Study.

G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D.
Weld. (2021). Does the whole exceed its parts? the effect of AI explanations
on complementary team performance.

G. Bansal, B. Nushi, E. Kamar, W.S. Lasecki, D.S. Weld, E. Horvitz. (2019).
Updates in Human-AI Teams: Understanding and Addressing the
Performance/Compatibility Tradeoff.

G. Bansal, B. Nushi, E. Kamar, W.S. Lasecki, D.S. Weld, E. Horvitz. (2019).
Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance.

R. Binns, M. Van Kleek, M. Veale, Ulrik Lyngs, J. Zhao, N. Shadbolt. (2018).
‘It’s Reducing a Human Being to a Percentage’; Perceptions of Justice in
Algorithmic Decisions.

Z. Buçinca, M.B. Malaya, K.Z. Gajos. (2021). To Trust or to Think: Cognitive
Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-
making.

Z. Buçinca, P. Lin, K.Z. Gajos, E.L. Glassman. (2020). Proxy Tasks and
Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems.

A. Bussone, S. Stumpf, D. O’Sullivan. (2015). The Role of Explanations on
Trust and Reliance in Clinical Decision Support Systems.

C. Cai, E. Reif, N. Hegde, J. Hipp, B. Kim, D. Smilkov, M. Wattenberg, F.
Viegas, G.S. Corrado, M.C. Stumpe, M Terry. (2019). Human-Centered Tools for
Coping with Imperfect Algorithms during Medical Decision-Making.

S. Carton, Q. Mei, P. Resnick. (2020). Feature-Based Explanations Don’t Help
People Detect Misclassifications of Online Toxicity.

A. Chandrasekaran, V. Prabhu, D. Yadav, P. Chattopadhyay, D. Parikh. (2018).
Do Explanations make VQA Models more Predictable to a Human?

H-F. Cheng, R. Wang, Z. Zhang, F. O’Connell, T. Gray, F.M. Harper, H. Zhu.
(2019). Explaining Decision-Making Algorithms through UI: Strategies to Help
Non-Expert Stakeholders.

M. Chromik, M. Eiband, F. Buchner, A. Krüger, A. Butz. (2021). I Think I Get
Your Point, AI! The Illusion of Explanatory Depth in Explainable AI.

D. Das and S. Chernova. (2020). Leveraging Rationales to Improve Human Task
Performance.

B.J. Dietvorst, J.P. Simmons, C. Massey. (2018). Overcoming Algorithm
Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly)
Modify Them.

J. Dodge, Q.V. Liao, Y. Zhang, R. Bellamy, C. Dugan. (2019). Explaining
Models: An Empirical Study of How Explanations Impact Fairness Judgment.

J. Dressel, H. Farid. (2018). The accuracy, fairness, and limits of predicting
recidivism.

K. Gero, Z. Ashktorab, C. Dugan, Q. Pan, J. Johnson, M. Ruiz, S. Miller, D.
Millen and W. Geyer. (2020). Mental Models of AI Agents in a Cooperative Game
Setting.

B. Ghai, Q.V. Liao, Y. Zhang, R. Bellamy, K. Mueller. (2021). Explainable
Active Learning (XAL): An Empirical Study of How Local Explanations Impact
Annotator Experience.

B. Green, Y. Chen. (2019). Disparate Interactions: An Algorithm-in-the-loop
Analysis of Fairness in Risk Assessments.

S. Guo, F. Du, S. Malik, E. Koh, S. Kim, Z. Liu, D. Kim, H. Zha, and N. Cao.
(2019). Visualizing Uncertainty and Alternatives in Event Sequence
Predictions.

P. Hase, M. Bansal. (2020). Evaluating Explainable AI: Which Algorithmic
Explanations Help Users Predict Model Behavior?

R. Kocielnik, S. Amershi, P.N. Bennet. (2019). Will you Accept an Imperfect
AI? Exploring Designs for Adjusting End-user Expectations of AI Systems.

T. Kulesza, S. Stumpf, M. Burnett, S. Yang, I. Kwan, W-K. Wong. (2013). Too
Much, Too Little, or Just Right? Ways Explanations Impact End Users’ Mental
Models.

J. Kunkel, T. Donkers, L. Michael, C-M. Barbu, J. Ziegler. (2019). Let Me
Explain: Impact of Personal and Impersonal Explanations on Trust in
Recommender Systems.

I. Lage, E. Chen, J. He, M. Narayanan, B. Kim, S. Gershman, F. Doshi-Velez.
(2019). An Evaluation of the Human-Interpretability of Explanation.

A. Levy, M. Agrawal, A. Satyanarayan, D. Sontag. (2021). Assessing the Impact
of Automated Suggestions on Decision Makingg: Domain Experts Mediate Model
Errors but Take Less Initiative.

V. Lai, C. Tan. (2019). On Human Predictions with Explanations and Predictions
of Machine Learning Models: A Case Study on Deception Detection.

Z. Lin, J. Jung, S. Goel, J. Skeem. (2020). The limits of human predictions of
recidivism.

J.M. Logg, J. Minson, D.A. Moore. (2019). Algorithm Appreciation: People
Prefer Algorithmic To Human Judgment.

Lu, Z., M. Yin. (2021). Human reliance on machine learning models when
performance feedback is limited: Heurstics and risks.

K. Mallari, K. Inkpen, P. Johns, S. Tan, D. Ramesh, E. Kamar. (2020). Do I
Look Like a Criminal? Examining how Race Presentation Impacts Human Judgement
of Recidivism.

S. McGrath, P. Mehta, A. Zytek, I. Lage, H. Lakkaraju. (2020). When Does
Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML
Assisted Decision Making.

D. Nguyen. (2018). Comparing Automatic and Human Evaluation of Local
Explanations for Text Classification.

J.S. Park, R. Barber, A. Kirlik, K. Karahalios. (2019). A Slow Algorithm
Improves Users’ Assessments of the Algorithm’s Accuracy.

F. Poursabzi-Sangdeh, D.G. Goldstein, J.M. Hofman, J. Wortman Vaughan, H.
Wallach. (2021). Manipulating and Measuring Model Interpretability.

E. Rader, K. Cotter, J. Cho. (2018). Explanations as Mechanisms for Supporting
Algorithmic Transparency.

A. Smith-Renner, R. Fan, M. Birchfield, T. Wu, J. Boyd-Graber, D. Weld, L.
Findlater. (2020). No Explainability without Accountability: An Empirical
Study of Explanations and Feedback in Interactive ML.

A. Springer, S. Whittaker. (2019). Progressive Disclosure: Designing for
Effective Transparency.

K. Stowers, N. Kasdaglis, M. Rupp, J. Chen, D. Barber, and M. Barnes. (2017).
Insights into Human-Agent Teaming: Intelligent Agent Transparency and
Uncertainty.

M. Szymanski, M. Millecamp, K. Verbert. (2021). Visual, textual or hybrid: the
effect of user expertise on different explanations.

N. van Berkel, J. Goncalves, D. Russo, S. Hosio, M.B. Skov. (2021). Effect of
Information Presentation on Fairness Perceptions of Machine Learning
Predictors.

X. Wang, M. Yin. (2021). Are Explanations Helpful? A Comparative Study of the
Effects of Explanations in AI-Assisted Decision-Making.

M. Yin, J. Wortman Vaughan, H. Wallach. (2019). Understanding the Effect of
Accuracy on Trust in Machine Learning Models.

K. Yu, S. Berkovsky, R. Taib, J. Zhou, F. Chen. (2019). Do I trust my machine
teammate?: an investigation from perception to decision.