# Disciplining deliberation: a sociotechnical perspective on machine learning
trade-offs

Sina Fazelpour  Northeastern UniversityBoston, MassachusettsUnited States
<EMAIL_ADDRESS>
(2020)

###### Abstract.

This paper focuses on two highly publicized formal trade-offs in the field of
responsible artificial intelligence (AI)—between predictive accuracy and
fairness and between predictive accuracy and interpretability. These formal
trade-offs are often taken by researchers, practitioners, and policy-makers to
directly imply corresponding tensions between underlying values. Thus
interpreted, the trade-offs have formed a core focus of normative engagement
in AI governance, accompanied by a particular division of labor along
disciplinary lines. This paper argues against this prevalent interpretation by
drawing attention to three sets of considerations that are critical for
bridging the gap between these formal trade-offs and their practical impacts
on relevant values. I show how neglecting these considerations can distort our
normative deliberations, and result in costly and misaligned interventions and
justifications. Taken together, these considerations form a sociotechnical
framework that could guide those involved in AI governance to assess how, in
many cases, we can and should have higher aspirations than the prevalent
interpretation of the trade-offs would suggest. I end by drawing out the
normative opportunities and challenges that emerge out of these
considerations, and highlighting the imperative of interdisciplinary
collaboration in fostering responsible AI.

††copyright: acmcopyright††journalyear: 2020††doi:
10.1145/1122445.1122456††conference: FAccT ’21: ACM Conference on Fairness,
Accountability, and Transparency; March 2021; Virtual††booktitle: FAccT ’21:
ACM Conference on Fairness, Accountability, and Transparency, March 2021,
Virtual††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††copyright: none

##  1\. Introduction

Our aims and values are diverse and many. So, unsurprisingly, in many cases we
can’t have them all, as interventions that realize some will sacrifice others.
In contending with this reality, we thus regularly face difficult questions
about value trade-offs. Formal analyses of decision scenarios can offer
valuable assistance in our deliberations about these questions. These analyses
can provide decision-makers with a precise perspective for identifying when
their values (suitably operationalized) come into conflict, and assessing
different ways of navigating those tensions. When used appropriately,
incorporating these formal perspectives into practical reasoning promises to
bring discipline and rigor to our deliberations.

This paper focuses on two highly publicized formal trade-offs in the field of
responsible artificial intelligence (AI)—between _predictive accuracy_ and
_fairness_ and between _predictive accuracy_ and _interpretability_. Ideally,
the adoption of AI tools in social decision-making should enhance decision
quality in a way that promotes societal values such as reliability, fairness,
transparency, trust, and safety. As the two trade-offs purport to show,
however, it may be inherently impossible for AI models to simultaneously
promote formal operationalizations of all these relevant values in the general
case. Ensuring fairness (in some statistical sense) might necessitate
relinquishing the opportunity of deploying more predictively accurate models
(Kleinberg, 2018; Corbett-Davies et al., 2017). Similarly, the most accurate
models might be “blackboxes” that lack interpretability (in some sense), whose
deployment can threaten values that interpretability is said to support, such
as trust, safety, or procedural fairness (Murdoch et al., 2019; Molnar, 2022).

As policymakers seek to better understand value tensions that can emerge in
responsible AI governance,111See for example the National Institute of
Standard and Technology’s influential AI risk management framework, and the
roadmap for its development, including “Guidance on the tradeoffs and
relationships that may exist among trustworthiness characteristics”:
https://www.nist.gov/itl/ai-risk-management-framework/roadmap-nist-artificial-
intelligence-risk-management-framework-ai. these formal trade-offs have become
a core locus of normative and policy engagement (Fleisher, 2022; London, 2019;
Kearns and Roth, 2019; Babic et al., 2021; Rudin, 2019; Tabassi, 2023;
Johnson, 2021; Loi and Christen, 2021; Fazelpour and Danks, 2021). But,
questions remain as to how we should interpret the value implications of these
trade-offs. As I explain in Section 2, at a high level of abstraction, both
trade-offs can be seen as potential upshots of learning AI models under
constraints that encode desiderata other than maximizing predictive accuracy.
While relevant to our practical reasoning, there is a gap between these model
properties and the impacts of adopting those models on the relevant values.
How should we understand the relation between these formal trade-offs, on the
one hand, and the interrelation between underlying societal values, on the
other?

A prevalent attitude towards this question is to reason as if there is a
direct correspondence here. In normative literature, this attitude often
involves a two-fold process: first, assuming that, given the formal trade-
offs, there also exists a tension between corresponding values; second,
offering justifications or mechanisms for some way of resolving the value
tension, and assuming that it has direct implications for choosing models that
(supposedly) prioritize the relevant values. Babic et al. (2021), for example,
suggest that while in some healthcare scenarios (e.g., diagnosis) accuracy may
take precedence, in other cases (e.g., allocating scarce resources)
transparency—seen as a means to procedural fairness—should take priority.
Taking this to have direct implications for model selection, they note that
“in such contexts, even if interpretable AI/ML is less accurate, we may prefer
to trade off some accuracy, the price we pay for procedural fairness” (Babic
et al., 2021, p. 286). A similar approach pervades other discussions of the
trade-offs (London, 2019; Loi and Christen, 2021; Rahwan, 2018). In the
reverse direction, many technical works seek to provide a set of suitable
models, each exhibiting a different way of striking the trade-off (Kearns and
Roth, 2019; Molnar, 2022), from which stakeholders can choose based on their
values and priorities. In this way, this prevalent attitude also involves a
convenient division of labor, across disciplinary lines.

In this paper, I argue against this attitude. As emphasized by an emerging
body of work, understanding the social impacts of AI requires adopting a
sociotechnical perspective that encompasses not only the properties of AI
models in isolation, but also those of the technological, psychological, and
social processes that critically shape the design, development, and deployment
of those models (Selbst et al., 2019; Suresh and Guttag, 2019). In Section 3,
I will show how adopting this sociotechnical perspective complicates the
translation of formal model trade-offs into interrelations between values.
Specifically, I integrate and thematize different strands of emerging research
into three sets of considerations that are critical for interpreting the
practical significance of the model-level trade-offs: considerations of
validity and instrumental relevance (Section 3.1), compositionality (Section
3.2), and dynamics (Section 3.3).

Taken together, these considerations form a sociotechnical framework that
could be adopted by those involved in the evaluation, design, and governance
of AI-based decision systems. I draw out these broader epistemic, ethical, and
policy implications in Section 4. To be sure, attending to these issues does
not obviate the need for navigating various value tensions. In fact, as I also
discuss in Section 4, doing so brings into focus new trade-offs and tensions.
Nor does it mean that existing normative discussions about these two formal
trade-offs are not highly valuable. What these considerations demonstrate is
that the relation between formal model-level properties (e.g., model-level
predictive loss) and the corresponding values (e.g., decisional accuracy) is
not a straightforward one, and involves many assumptions. Importantly, closely
examining these assumptions reveals that sometimes sacrificing one formal
property for another at the model level can improve both values at the
practical level. That is, in many cases, we can and should have higher
aspirations than the direct correspondence interpretation would suggest.
Importantly, however, taking advantage of these possibilities requires that we
broaden the scope of normative engagement with AI technologies and better
appreciate the critical importance of interdisciplinary collaboration in
fostering responsible AI.

##  2\. Trade-offs in AI-based decision-making

This section provides the intuition behind the accuracy-fairness and accuracy-
interpretability trade-offs and motivates the need for scrutinizing their
interpretation in research and policy discussions. It is worth noting that
discussions of both trade-offs predate recent AI and ML applications. The
accuracy-fairness trade-off, for example, has been discussed in economics
(Young, 1994) and education (Willingham and Cole, 2013). And the accuracy-
interpretability trade-off has been discussed by statisticians (Plate, 1999).

###  2.1. Prediction-based decision-making

Let us ground the discussion using a typical binary classification task, where
we hope to learn the relation between a set of input features $X$ and a target
label $Y$, with supports $\mathcal{X}$ and $\mathcal{Y}=\\{0,1\\}$,
respectively. With this relation in hand, given a feature vector
$x\in\mathcal{X}$, we can then infer the corresponding label $y\in\\{0,1\\}$,
and use this prediction to inform or drive a relevant decision. In the context
of informing content moderation decisions, for example, $X$ and $Y$ might
correspond to information about social media posts including their metadata
and a label indicating whether they contain offensive language, respectively.
Supervised learning algorithms help us do this by providing a mapping from
datasets of past observations to predictive models (or hypotheses) that can be
used to make inferences about new cases. That is, given a dataset of labeled
examples, such algorithms output a predictive model
$h:\mathcal{X}\rightarrow\\{0,1\\}$, from a set of possible predictive models
$\mathcal{H}$. Typically, these algorithms are designed to find a model
$h^{*}\in\mathcal{H}$ that optimizes predictive performance according to some
evaluation function, such as minimizing a loss function like empirical risk
$\mathcal{\hat{L}}$:

$h^{*}=\operatorname{argmin}_{h\in\mathcal{H}}\mathcal{\hat{L}}(h)$

In practice, there is often a multiplicity of models that perform comparably
with respect to this optimization objective (Coston et al., 2021; Marx et al.,
2020; D’Amour, 2021). Let us refer to these as _the set of accurate models_ ,
$\mathcal{H}_{A}$, which, at their worst, only exhibit $\epsilon$ more loss
than our best predictive case $\hat{\mathcal{L}}(h^{*})$222This formulation is
helpful because it leaves it open how many solutions there exists to the
optimization problem, which can be more than one even when $\epsilon=0$. As
Marx et al. (2020) note, the choice of small $\epsilon$ can be perfectly
justifiable, as small differences in training error, of the sort captured by
the empirical risk function $\hat{\mathcal{L}}(h)$, does not necessarily
correlate with significant performance differences in deployment (See also
D’Amour et al., 2020a).:

$\mathcal{H}_{A}:=\\{h\in\mathcal{\mathcal{H}}:\hat{\mathcal{L}}(h)\leq\hat{\mathcal{L}}(h^{*})+\epsilon\\}$

With this notations in hand, we can get an intuition about the trade-offs by
considering the act of _enforcing_ fairness or interpretability as an
intervention that imposes a non-trivial constraint on the set of available
predictive models, and in doing so potentially restricts our access to only a
subset of eligible models in $\mathcal{H}$ that may not intersect with
$\mathcal{H}_{A}$ (see Figure 1(a)).

Of course, there can also be models that do satisfy these additional formal
constraints, without thereby suffering in predictive accuracy. This is
because, despite exhibiting similar predictive performance, the solutions in
the set of accurate models, $\mathcal{H}_{A}$, can have significantly
different characteristics when it comes to fairness (Coston et al., 2021; De-
Arteaga et al., 2022) and interpretability (Semenova et al., 2019; D’Amour et
al., 2020a), formally construed. Accordingly, it would be a mistake to assume
a priori that there will in fact be a cost to enforcing fairness- or
interpretability-motivated constraints (See also Rodolfa et al., 2021; Rudin,
2019). An exciting avenue of current research is build on existing
transparency documentation tools, such as model cards (Mitchell et al., 2019),
to better inform regulators and auditors about the potential normative
significance of the models within the set of accurate models (See Black et
al., 2022; Coston et al., 2021). That said, let us examine in more detail how
demands for fairness and interpretability have been formulated as enforcing
constraints on learning.

###  2.2. Fairness as a constraint

(a)

(b)

(c)

Figure 1. Figure (a) represents a characterization of the two trade-offs in
terms of a possible relation in the hypothesis space $\mathcal{H}$ between the
set of accurate models $\mathcal{H}_{A}$ and the set of models that satisfy a
fairness constraint $\mathcal{H}_{F}$. The figure is inspired by Figure 1 in
Dziugaite et al. (2020). (b) This figure, adapted from Kearns and Roth (2019),
offers a toy illustration of the accuracy-fairness Pareto frontier, while
Figure (c), adapted from Gunning et al. (2019) (itself adapted from DARPA XAI
presentation) offers a widely shared illustration of the accuracy-
interpretability trade-off.

The concern about accuracy-fairness trade-off most naturally arises when we
focus on the properties of predictive models (Kearns and Roth, 2019; Corbett-
Davies et al., 2017). Even given this relatively restrictive lens, there is
considerable debate about the appropriate formalization of fairness in terms
of model properties (Chouldechova, 2017; Barocas et al., 2019; Corbett-Davies
et al., 2017). We can largely sidestep these disagreements, however, because
in practice many proposed fairness measures can be formulated as non-trivial
parity constraints on the joint distribution $p(X,Y,\hat{Y},A)$, where $A$
denotes membership in a protected group and $\hat{Y}=h(X)$ denotes model
predictions (Fazelpour and Lipton, 2020; Barocas et al., 2019). For example,
according to some, fairness might demand that the chosen predictive model
exhibit equal sensitivity across different demographic groups (i.e.,
$p(\hat{Y}=1\mid Y=1,A=a)=p(\hat{Y}=1\mid Y=1,A=a^{\prime})$) (Hardt et al.,
2016).

Achieving “fairness”, from this perspective, requires enforcing the relevant
parity constraint(s) (Corbett-Davies et al., 2017). This might be done, for
example, by adding fairness considerations as constraints in optimization
(e.g., Zafar et al., 2017), or as penalty terms incorporated in the
optimization objective (e.g., Kamishima et al., 2011). Roughly speaking, many
of these algorithmic manipulations have the effect of restricting the choice
of predictive models to a _subset_ of possible predictive models that satisfy
the relevant parity constraint(s) $\mathcal{H}_{F}\subseteq\mathcal{H}$. This
act of constraining model choice might come with a predictive cost, however,
insofar as the best predictive models in this restricted space of “fair”
models $\mathcal{H}_{F}$, at best, only equal the predictive performance of
the models from the broader hypothesis class $\mathcal{H}$. That is,

$\text{Since }\mathcal{H}_{F}\subseteq\mathcal{H}\text{, then
}\min_{h\in\mathcal{H}_{F}}\mathcal{\hat{L}}(h)\geq\min_{h\in\mathcal{H}}\mathcal{\hat{L}}(h)$

Put in terms of the set of accurate models, there might be a predictive “cost”
to enforcing fairness-related parity constraints in that $\mathcal{H}_{F}$ may
not include any of the accurate models:
$\mathcal{H}_{A}\cap\mathcal{H}_{F}=\emptyset$. In such cases, enforcing those
constraints will result in drops in accuracy.

###  2.3. Interpretability as a constraint

While widely discussed in research and policy literature, the trade-off
between accuracy and model interpretability (transparency, explainability, or
other cognates) is more circuitous with respect to its technical underpinnings
as well as the potential value tensions it supposedly signifies. The trade-off
is less straightforward in terms of value tensions because “interpretability”
is typically not sought as an end in itself. Rather, its value is often tied
to a varied set of goals, such as increased understanding, enhancing user
trust and decision support quality, improving deployment performance and
safety, fostering autonomy and procedural fairness, and ensuring public
accountability (Lipton, 2018; Murdoch et al., 2019; Krishnan, 2020; Jobin et
al., 2019; Creel, 2020; Vredenburgh, 2021; Fleisher, 2022).

Even when we assume model interpretability (in some sense) is the requisite
means for achieving (some of) those aims, there are substantial disagreements
about what precise properties would render a model “interpretable” in the
relevant sense (Lipton, 2018; Krishnan, 2020). Once more, though, we can
sidestep these more specific disagreements by broadly characterizing
“interpretability” as a non-trivial constraint, whose enforcement will
restrict one’s search (or learning) to a _subset_ of predictive models
$\mathcal{H}_{I}\subseteq\mathcal{H}$ judged to satisfy that constraint
(Dziugaite et al., 2020). This broad characterization captures various
attempts to interpret “interpretability” in terms of some well-defined
technical notion (low model complexity, variable decomposability, …), without
committing to one operationalization, which might be contested. It also
captures some of the so-called post-hoc explainability methods that seek to
render blackbox models interpretable, insofar as those methods may depend on
restrictions to $\mathcal{H}$, whether by being model-specific by design or by
depending on certain properties of target models for their proper functioning
in practice (Murdoch et al., 2019). Given this characterization, the accuracy-
interpretability trade-off can again be seen as an upshot of enforcing a non-
trivial constraint on learning—namely, “interpretability”, whatever it might
imply. Thus viewed, enforcing interpretability can result in a loss in
predictive accuracy, when $\mathcal{H}_{A}\cap\mathcal{H}_{I}=\emptyset$. In
such cases, one could quantify (in context) the “price” of interpretability in
terms of the incurred loss of predictive accuracy (Bertsimas et al., 2019).

To capture this framing, in what follows we adopt a general understanding of
the accuracy-interpretability trade-off that nonetheless tends to be operative
in technical, philosophical and policy discussions: cases where we might need
to sacrifice predictive accuracy to make gains in understanding of model
behavior (broadly construed), in order to promote one of the aims for which
interpretability, explainability, or transparency is taken to be a means (see,
e.g., Miller, 2019; Fleisher, 2022).

###  2.4. Direct correspondence and a convenient division of labor

How should we understand the relation between the formal properties (and
trade-offs) of AI models and the (potentially varied) impacts of deploying
those models on corresponding values? As mentioned above, a prevalent
interpretation of the trade-offs is to proceed as if there is a direct
correspondence between these two. Importantly, this prevalent interpretation
suggests a straightforward division of labor between technical and normative
efforts.

On the technical side, for example, Kearns and Roth (2019) see the role of
formal tools as providing guidance for normative deliberation by mapping the
accuracy-“fairness”333Or more accurately, fairness-motivated statistical
parity Pareto frontier—consisting of the set of models that cannot be improved
in terms of one of these formal measures without incurring a loss on the other
(see Figure 1(b)). Within this framing, normative deliberation has its place
in choosing a specific model, depending on the contextually relevant values
and priorities. Similarly, other works aim to provide guidance about how
decision-makers, once they have decided that some sacrifice in predictive
accuracy is justified, might go about choosing an appropriate “interpretable”
model (e.g., Molnar, 2022) (see also Figure 1(c)). Conversely, many
philosophical and policy discussions simply take it for granted that the
formal trade-offs imply the existence of tensions between underlying _values_.
And, they see their task as providing means of resolving those tensions,
perhaps by offering justifications for when, why, or to what extent some value
should be prioritized (London, 2019; Loi and Christen, 2021; Babic et al.,
2019), or by proposing participatory mechanisms for making such determinations
(Rahwan, 2018; Lee et al., 2019).

We now turn to the different considerations that complicate this picture,
discussing the broader implications of this change of perspective for this
understanding of the division of labor in Section 4.

##  3\. A sociotechnical perspective on interpreting the trade-offs

Useful as it might be for motivating the discussion of the formal trade-offs,
the characterization of AI-based decision-making in the previous section
involves significant, and potentially problematic simplifications.
Specifically, researchers have argued against framing the value implications
of AI models solely in terms of model properties, in isolation from their
context of development and use (Selbst et al., 2019; Fazelpour and Lipton,
2020; Herington, 2020; Raji et al., 2022; Suresh and Guttag, 2019). Instead,
there have been calls for adopting a sociotechnical perspective that
encompasses not only the properties of AI models, but also other technical,
psychological, organizational, and social factors that shape the life-cycle of
AI-based decision systems, and ultimately their societal impacts. In this
section, I approach the trade-offs from this perspective, discussing three
sets of considerations that are required for bridging the gap between the
formal, model-level trade-offs described above and their practical value
impacts. As we will see, in each case, attending to these considerations
reveals that we can have things better than the formal trade-offs might have
us expect.

###  3.1. Validity and relevance

Trade-offs between two formal constructs (e.g., some measure of accuracy and
fairness) have the purported normative implications, only if those constructs
aptly track our values. But this assumption can fail, when invalid
operationalization, measurement, and estimation procedures and practices
result in disconnects between formal constructs and the epistemic, ethical, or
legal aims and values that those constructs are supposed to capture.

Consider first the notion of “predictive accuracy”, which is implicated in
both trade-offs. In many prediction-based decision settings, institutions care
about inferring outcomes that are ambiguous, latent, and contested, such as
being a patient with “severe healthcare needs” (Obermeyer et al., 2019), a
child “at risk” (Saxena et al., 2020), or a tweet containing “hate speech”
(Waseem, 2016). Let us refer to this outcome that we practically care about as
$Y_{c}$. Rendering $Y_{c}$ suitable for machine learning application often
requires selecting a simplified and unambiguously measurable proxy outcome,
$Y$. This is not a trivial task, and often involves various value judgments.
In practice, $Y$ might not be a valid proxy for $Y_{c}$ for a variety of
reasons, ranging from lack of construct validity in operationalization to
challenges of bias and validity in measurement, estimation, and aggregation
(Fazelpour and Danks, 2021; De-Arteaga et al., 2018; Kleinberg et al., 2018a;
Jacobs and Wallach, 2021).

Importantly, for our purposes, the accuracy-fairness trade-off can arise when
the extent of the disconnect between $Y$ and $Y_{c}$ is not evenly distributed
across the population of interest (De-Arteaga et al., 2022).444Intuitively,
this can be thought of as potentially resulting in a statistical dependence
between $Y$ and some protected attribute $A$ not because one exists in
reality, but as a result of differential validity in our categorization and
data collection practices. That is, when $Y$ exhibits differential validity
across groups, optimizing for predictive accuracy can undermine some fairness-
motivated parity constraints (and vice versa). In a salient example of such
differential validity in healthcare, the disconnect between a patient’s
“healthcare need” and its operationalization in terms of “healthcare
expenditure” particularly harmed Black patients, for whom expenditure was
particularly an unreliable proxy due to a variety of factors including
justice-related ones (Obermeyer et al., 2019). Similar issues arise when our
measurement, estimation, and aggregation techniques exhibit differential
validity across protected groups (Coston et al., 2023; Jacobs and Wallach,
2021).555For example, if our data collection and aggregation practices make it
more likely that tweets from certain marginalized groups are more likely to be
labeled as “toxic speech” (Davani et al., 2022).

In such cases, it would be a serious error to take the accuracy-fairness
trade-off as an inescapable fact and to focus our deliberative efforts simply
on selecting among the set of Pareto optimal solutions. Instead, we need to
address the underlying issue of differential validity by adopting better
sociotechnical practices in problem formulation, operationalization, and data
collection (Hellman, 2020; Obermeyer et al., 2019; Kleinberg et al., 2018b).
Taking the trade-off at face value not only obscures the underlying epistemic-
ethical problems; doing so also prevents us from recognizing that we can have
things better than the formal trade-offs lead us to think.

Similar validity concerns arise for formal notions of“interpretability”.
Currently, there exists a considerable disconnect between the underlying
values for which “interpretability” is sought (e.g., trust, improved decision
quality, safety, recourse, …) and the technical operationalizations of the
term (e.g., in terms of model complexity, variable decomposability, model
type, …) (Krishnan, 2020; Lipton, 2018). Indeed, in some cases, formal
operationalizations appear to undermine the very values they are meant to
support, such as when individuals lose their trust in algorithmic predictions
upon seeing that they originated from a simple decision tree, whose lack of
complexity was meant to invite user trust (Lu et al., 2019). When a measure of
“interpretability” lacks validity in this sense, it is unclear why we would
want it, especially if attaining it comes at a cost to predictive accuracy.
What is more, even if interpretability (in some sense) is _a_ means to those
underlying values, it may not be the only, or even the most effective, means
in context. In healthcare settings, for example, London (2019) proposes a
number of alternative, potentially less costly pathways for achieving some of
the aims for which interpretability is sought (see also Krishnan, 2020).

This is, of course, not to say that interpretability—in the broader sense
defined in Section 2.3 and suitably specified—is not critical for supporting
key epistemic, social, and ethical values in many settings, as will be
discussed below (See also Murdoch et al., 2019; Vredenburgh, 2021; Creel,
2020). But the _practical_ implications of a potential accuracy-
interpretability trade-off is dubious, if the formalization of
interpretability lacks validity or when there exist other sociotechnical
interventions that can better promote the values that animate concerns about
interpretability without imposing a cost on predictive accuracy. Closely
inspecting the validity and instrumental relevance of the formal constructs
can uncover normatively significant disconnects and alternatives. In many
cases, addressing these underlying issues can enable us to robustly promote
both sets of relevant values in ways that remain outside the purview of the
narrow formal characterization of the trade-offs.

###  3.2. Compositionality

In many cases of social concern, algorithmic tools are not stand-alone
decision-makers, but function as part of broader decision-making systems.
Examples include healthcare (Kompa et al., 2021), child welfare services (De-
Arteaga et al., 2020), loan approvals (Paravisini and Schoar, 2013), and fact-
checking (Guo et al., 2022), where algorithms assist (rather than replace)
human experts. In such cases, the focus should shift from the AI model’s
isolated performance to how its integration can improve the overall decision
quality, especially compared to the status quo of unaided human experts (See
Green and Chen, 2019). It thus become crucial to examine how properties of AI
models (accuracy, fairness, interpretability, or their respective trade-offs)
contribute to the broader decision-making systems those AI models are
integrated in.

Works in sociology, organizational science, and philosophy demonstrate that
the epistemic norms and properties of groups cannot necessarily be inferred
from those of their individual members (Mayo-Wilson et al., 2011; O’Connor and
Weatherall, 2019). The concept of complementarity in teaming and collective
intelligence illustrates this divergence (Page, 2019; Steel et al., 2018). It
suggests that, especially in complex tasks, a group’s performance hinges on
how its members’ cognitive tools (information, background knowledge, decision
heuristics, …) interlink. For instance, one member’s unique and different
perspective can offer a breakthrough on a problem that hinders another.
Conversely, this synergy is lost, if members’ strengths and weaknesses are too
similar (Bang and Frith, 2017). Importantly, selecting individuals with
appropriate complementary cognitive capabilities can mean not selecting the
“best” performing individuals (according to some measure) (Hong and Page,
2004).

(a)

(b)

Figure 2. The importance of complementarity when predictive models function as
decision support. (a) represents individual performance of two algorithms
$a_{1}$ and $a_{2}$ and a human expert $h$ on two equally encountered feature
regions, $\mathcal{X}_{1}$ and $\mathcal{X}_{2}$, measured by a loss function
$\mathcal{L}(.)$. Overall, $a_{1}$ has the best predictive performance,
followed by $a_{2}$. Human $h$ performs worst on the task.Notice, that the
mistakes are not similarly distributed across feature regions. (b) represents
team performance. $h+a_{2}$ is the best team, due to the complementarity in
their performance. Notice that the best predictive model $a_{1}$ is not the
best decision support. It is assumed that a system integrator (possibly $h$
herself) allocates tasks according to each team member’s ability on a given
feature region.

The logic of complementarity also applies to cases where AI models function as
decision aids. Consider our classification task from 2.1, where the feature
space is divided into two equally probable regions $\mathcal{X}_{1}$ and
$\mathcal{X}_{2}$ (e.g., corresponding to different types of patients in a
healthcare setting) (see Figure 2).666The example is based on Donahue et al.
(2022) Evaluating predictive performance as before, we might have:

  1. i

Algorithm $a_{1}$ with equal performance in both regions
$\hat{\mathcal{L}}(a_{1}\mid\mathcal{X}_{1})=\hat{\mathcal{L}}(a_{1}\mid\mathcal{X}_{2})=0.5$,
and so $\hat{\mathcal{L}}(a_{1})=0.5$ overall;

  2. ii

A less accurate algorithm $a_{2}$ with
$\hat{\mathcal{L}}(a_{2}\mid\mathcal{X}_{1})=0.45$ and
$\hat{\mathcal{L}}(a_{2}\mid\mathcal{X}_{2})=0.65$, and so
$\hat{\mathcal{L}}(a_{2})=0.55$ overall;

  3. iii

A human $h$, who is the least accurate overall, with
$\hat{\mathcal{L}}(h\mid\mathcal{X}_{1})=0.75$ and
$\hat{\mathcal{L}}(h\mid\mathcal{X}_{2})=0.45$, and so
$\hat{\mathcal{L}}(h)=0.6$.

Suppose knowing these, a decision-maker (possibly $h$ herself) can allocate
tasks to individuals accordingly. Which of $a_{1}$ or $a_{2}$ should partner
$h$? If we simply focused on overall performance, it might be tempting to
select $a_{1}$ due to its superior accuracy.777In fact, this is currently
standard practice where predictive models are primarily optimized for
individual-level properties. But, the concept of complementarity suggests a
different approach. Precisely on account of suitable differences in their
predictive capabilities, the human-AI team $h+a_{2}$ turns out to be the best
overall team with $\hat{\mathcal{L}}(h+a_{2})=0.45$, outperforming the
$h+a_{1}$ team as well as $a_{1}$ (Figure 2(b)). Crucially, then, the most
accurate model is not necessarily part of the most accurate human-AI team.

As this example shows, for groups to benefit from complementarity, there needs
to be relevant variability in individual performance, stemming from underlying
differences in background knowledge, representation, available information, or
heuristics. Crucial too are appropriate task allocation (who should do what
cases) and effective information integration (whose prediction should weigh
how much). Recent works on human-AI collaboration have explored these factors:
by characterizing the extent of variability required for complementarity
(Donahue et al., 2022) and mechanisms that could give rise to such differences
between humans and AI models (Rastogi et al., 2022); by designing optimization
objectives to suitably complement human experts (Mozannar and Sontag, 2020);
by developing optimal task allocation schemes (Madras et al., 2018); and by
facilitating effective information integration (Bansal et al., 2019a).888Some
of this work can also be seen as ways of incorporating other group-level
norms. For example, works on tools that ask for second opinions (e.g., Kompa
et al., 2021; Raghu et al., 2019) can be seen as computational implementations
of contestability considerations—proposed by many philosophers as a group-
level epistemic norm (e.g., Anderson, 2006; Landemore, 2020).

These works underscore that properly evaluating AI model properties and trade-
offs depends on understanding their contribution to the qualities of the
broader decision-making system. Many of the above techniques can result in
reduced model accuracy, but improve accuracy at the system-level. Notably, in
some cases, model accuracy is traded for improved understanding. This might be
done to help experts construct suitable mental models of the AI behavior
(Bansal et al., 2019a) or be better placed to combine AI outputs with their
own knowledge (Caruana et al., 2015). More generally, interpretability, as
defined in Sec 2.3, can also alleviate experts’ distrust of AI models,
promoting better informational uptake and integration (Zerilli et al.,
2022).999Some techniques for providing explanations (e.g., local saliency
explanations) have been shown to have detrimental impact on the quality of AI-
informed decisions by increasing experts reliance on (and reducing their
vigilance about) AI output, even in cases of incorrect AI predictions (Bansal
et al., 2021). But, caution is needed when interpreting such findings. We must
distinguish between the values of interpretability and the effectiveness of
particular interpretability-seeking techniques in achieving those values; the
fact that some techniques can result in misleading explanations does not
undermine the value of explanations (Lombrozo, 2011) or improved understanding
(Grimm, 2012) in general. Moreover, even when sometimes unjustified, increased
reliance overall on AI can still be a net epistemic good, depending on the
status quo of unaided human decision making. Thus, a perceived compromise at
the model-level can in fact translate into an advantage at the broader system-
level.

The study of “fairness under composition” reveals a similar insight (Dwork and
Ilvento, 2018; Wang et al., 2021). As Chouldechova and Roth (2020) note “often
the composition of multiple fair components will not satisfy any fairness
constraint … [while] the individual components of a fair system may appear to
be unfair in isolation” (p. 86). Here too, moving beyond a model-centric view
is essential to recognize potential (individual vs. system) divergences and
understand how principled model-level losses in a given dimension can result
in system-level gains along that same dimension.

Of course, as will be discussed in Section 4, this system-level perspective
doesn’t mean that “we can have all the good things at once” without facing
trade-offs. Rather, the suggestion is that we _can_ have higher aspirations
compared to what a simple interpretation of the two trade-offs would suggest.
When predictive models compose just one part of broader decision-making
systems, it is a _category mistake_ to focus on their properties apart from
those of the other parts (e.g., user characteristics and capabilities,
organizational norms). In such cases, the simple interpretation of the two
trade-offs is not only misguided, it can also distract us from other promising
(and challenging) epistemic and ethical considerations.

###  3.3. Deployment dynamics

To responsibly integrate AI models into social decision pipelines, it’s
essential to understand more than just model properties within a static
testing environment. It also requires a grasp of the mechanisms governing the
dynamic interactions of AI outputs with their organizational and social
embeddings. Understanding these dynamics is key for properly contextualizing
model-level trade-offs. What appears as a sacrifice of one desideratum for
another from a static perspective could be necessary for ensuring sustained
gains in the former in the longer term.

Consider first how our thinking about fairness and accuracy can be informed by
better understanding social dynamics.101010For a similar discussion on
accuracy-fairness see De-Arteaga et al. (2022) and Fazelpour et al. (2021)
Findings from workforce diversity research offer a salient example. These
works show that the benefits of increased team diversity—often understood in
this literature to be maximal at demographic parity—is not linear, and
depends, among other things, on how diverse a team _initially_ is (Phillips,
2017; Bear and Woolley, 2011; Post and Byron, 2015). That is, while moving
towards demographic parity can have negative effects on team performance in
highly homogeneous teams, it can improve performance in groups that are
already (to some extent) diverse (Steel et al., 2018). Accordingly, making
short-term utility losses for the sake of parity on fairness-related grounds
can result in robust, long-term utility gains (understood as improved team
performance on a variety of tasks). In other cases, making short-term gains in
fairness at the cost of accuracy can result in compounding injustices in the
longer term (Fazelpour et al., 2021). Take, for example, fairness-related
disparities in lending that arise because of conditions affecting the
repayment ability of members of certain disadvantaged groups. In such cases,
achieving fairness-related parities requires reducing predictive accuracy,
resulting in granting loans to individuals who may not be able to repay them.
Yet, the resulting disproportionate defaults can widen existing discrepancies
(e.g., due to its impacts on other opportunities via damaging credit scores)
(Liu et al., 2018; D’Amour et al., 2020b).

Examining these dynamics is also crucial for improved reasoning about longer
term predictive accuracy and its relationship with interpretability. Issues
around updating AI models offer an apt example here. Updating can improve
predictive accuracy by leveraging increased data, especially from post-
deployment observations; it is also crucial for addressing dynamic
distribution shifts (e.g., due to environmental changes or strategic behavior
of decision subjects) that can undermine predictive capabilities (Babic et
al., 2019; Zrnic et al., 2021). 111111Babic et al. (2019) provide a thoughtful
discussion of policy and regulatory challenges around these updates.

But the accuracy of the updated AI model with respect to a static dataset is
not the only consideration. We also need to consider the dynamic interactions
between the updated model and the rest of the decision-making system (Bansal
et al., 2019b; Srivastava et al., 2020; Bansal et al., 2019a). For example,
while enhancing a model’s overall accuracy, updates can degrade accuracy in
cases where users had come to justifiably rely on the model (Bansal et al.,
2019b). These unexpected shifts can hurt the overall human-AI performance, and
undermine trust, in ways that negatively affect experts’ long-term adoption of
AI tools (Dietvorst et al., 2015). Updating that ensures sustained
improvements can thus require potential sacrifices of model accuracy for the
sake of backward compatibility (Bansal et al., 2019a, b; Srivastava et al.,
2020).121212For example, because some interventions to make updated models
backward compatible restrict the space of learning according to some
consistency criteria (with past prediction). These sacrifices can be due to
interventions that make the models more interpretable (in the sense discussed
in Section 2.1), and can result in performance gains at the system level over
time.

More generally, AI models are often embedded in complex epistemic practices
(Creel, 2020; Fleisher, 2022). Appreciating the dynamic and evolving nature of
these practices should inform our understanding of the relation between
accuracy and interpretability. As noted by Rudin (2019), for example,
interpretable models that exhibit lower accuracy on a static dataset (compared
to a “blackbox” model) may afford better opportunities for understanding and
iteratively refining the knowledge discovery process, thus improving accuracy
in the longer run (See also Murdoch et al., 2019).

Overall, then, static properties of predictive models do not tell the full
story about what we stand to gain or lose by adopting the models in complex,
changing environments. What might appear to be an inescapable trade-off from a
static perspective may not be one, once we adopt a broader and longer term
perspective. Similarly, justifications developed in the abstract about how
such trade-offs should be navigated may provide hardly any epistemic or
ethical reassurance in complex social systems.

##  4\. Discussion

Having discussed each of the considerations separately, let us bring out some
of the shared lessons.

###  4.1. Expanding normative engagement, new opportunities, new challenges

The previous section showed how the direct correspondence interpretation of
the formal trade-offs can result in misplaced justifications, and
interventions that miss out on better epistemic and ethical possibilities.
Importantly, achieving these better possibilities requires that we expand the
scope of epistemic and ethical normative engagements to include issues that
arise throughout the operationalizations, measurement, and modeling practices
that influence the validity of model properties (Section 3.1) as well as the
cognitive, organizational, and social factors that shape the situated impacts
of AI models in setting (Sections 3.2 and 3.3). And we must do so in an
integrated and coherent manner.

For example, in his insightful discussion of the accuracy-interpretability
trade-off, London (2019) highlights how, in medical practice, decisional
accuracy and reliability take precedence over interpretability and mechanistic
understanding. Should we thus dismiss the search for interpretable models,
when they come at a cost to model accuracy? Not insofar as London’s discussion
does not attend to considerations of compositionality and dynamics that (as
discussed in Sections 3.2 and 3.3) are critical in many medical settings where
AI models are used as decision supports. Importantly, what these
considerations show is that even when, as London argues, decisional accuracy
matters more than mechanistic understanding, the best way to improve AI-
enabled decision-making in dynamic medical environments may not be choosing
the most accurate model. Indeed, as argued above, this may require selecting
an interpretable, even if less predictively accurate model.

As discussed above, the sociotechnical perspective can bring into view new
design and governance opportunities for improving the overall quality of AI-
based decision-making (e.g., focusing on team complementarity, instead of
individual performance). This does not mean that we never face those value
trade-offs. Indeed, this perspective also reveals difficult questions about
other value tensions. For example, enhanced participation and contestation can
improve group decision-making in a way that can be pertinent to considerations
ranging from operationalization and measurement (Martin Jr et al., 2020; Lee
et al., 2019) to human-AI teaming (Kompa et al., 2021; Raghu et al., 2019) in
the AI lifecycle. But, they can also result in delays and even undermine group
cohesion and decision quality (Fazelpour and De-Arteaga, 2021; Dobbe et al.,
2020). Similarly, when thinking about issues of compositionality, we are
likely to face the so-called diversity-stability trade-off familiar from other
complex systems, where too much variability—a requirement for
complementarity—can destabilize the system and degrade performance (Page,
2019; Eliassi-Rad et al., 2020). The crucial point is that these practically
relevant debates can remain outside the purview of normative deliberation
about AI governance, if attention is instead focused on a formal trade-off
that might not have the purported value significance.

###  4.2. The meaning of “AI talent” and the need for interdisciplinarity

As mentioned in Section 2.1, the direct correspondence interpretation is often
accompanied by an implicit division of labor between technical and normative
efforts. In particular, it suggests that once researchers, stakeholders, or
policy-makers decide which of their values should be prioritized in a given
context, the rest is technical work. Notably, these judgments about the
relevance and priority of values are often made without close engagement with
the various choices involved throughout the design, development, and
deployment processes of AI. For example, arguments about the priority of
accuracy over interpretability are offered by considering the demands of
different medical applications, as opposed to how AI models are designed for
and embedded into those applications.

This conception of the disciplinary division of labor may also underpin recent
governmental orders and directives that aim to promote responsible AI. For
example, in its recent efforts to promote responsible AI, the United States
government has launched an AI Talent Surge.131313See for example, the recent
directives by the Biden-Harris Administration:
https://www.whitehouse.gov/briefing-room/statements-releases/2024/01/29/fact-
sheet-biden-harris-administration-announces-key-ai-actions-following-
president-bidens-landmark-executive-order/ Importantly, however, the notion of
the “AI talent” that is meant to help “federal agencies to responsibly
leverage AI” is understood exclusively in technical terms in terms of data
science and tech talent.141414See https://ai.gov/apply/ But, if the
considerations above are on the right track, this is serious misconception.

As discussed above, the prevalent interpretation of the formal trade-offs can
mislead our normative deliberation and result in costly disconnects between
implementations and justifications that are meant to ground them. A
sociotechnical perspective not only offers a safeguard against misinterpreting
the trade-offs (and model properties more generally); it can also offer
improved epistemic and ethical opportunities. But taking advantage of these
opportunities requires serious interdisciplinary collaborations. Advancing our
understanding of any of the considerations above (e.g., operationalizing
social phenomena, human-AI complementarity and collaboration) requires drawing
on significant domain expertise as well as the knowledge and methodologies
from a variety of disciplines, including those in the humanities and social
sciences (For related points see Fazelpour and De-Arteaga, 2021; Stinson and
Vlaad, 2024; Rudin, 2019).

Of course, in taking an interdisciplinary approach to responsible AI
seriously, we need to address many challenging questions. For example, at an
interpersonal level, diverse and interdisciplinary teams often face
communicative problems (O’Rourke et al., 2013; Page, 2019). What type of
upskilling is needed to address these challenges in the context of responsible
AI design and governance? At an organizational level, successful
interdisciplinary collaboration appears to require factors such as sustained
leadership support, egalitarian power relations between groups, and more
(Phillips, 2017). What interventions and incentives can effectively realize
these factors in the responsible AI ecosystem? At a structural level, what are
the appropriate organizational structures for effective interdisciplinarity
(e.g., should we have teams of mostly interdisciplinary individuals or
disciplinary clusters with interdisciplinary bridges; what type of publication
and conferences venues are most suitable)? (Leydesdorff et al., 2008) The
first step towards making advances on these questions is to appreciate that
responsible AI governance cannot simply be achieved by the vision of
disciplinary division of labor underpinning the prevalent interpretation. If
“AI talent” is key to responsible innovation and design, then that talent
needs to be understood in an interdisciplinary way.

##  5\. Conclusion

With the increasing social use of AI, we need to carefully examine the
epistemic and ethical questions posed by these technologies. Taken together,
the three sets of considerations discussed above can provide a lens that can
assist those involved in evaluating, designing, and investigating these
systems. While the focus here is on the two formal trade-offs in AI-based
decision-making, the considerations may be applicable to formal trade-offs in
other normatively significant and policy-relevant scenarios. This is because
turning a policy scenario into one amenable to formal treatment often requires
abstractions and assumptions about the broader sociotechnical context that
often tend to be neglected in interpreting. The discussion can thus
potentially offer something to researchers interested in those other
scenarios. Hopefully, the considerations above allow us to leverage these
formal trade-offs in our practical deliberations, without allowing a
misleading interpretation of them punish our normative aspirations in a
negative sense.

## References

  * (1)
  * Anderson (2006) Elizabeth Anderson. 2006.  The epistemology of democracy.  _Episteme_ 3, 1-2 (2006), 8–22.
  * Babic et al. (2019) Boris Babic, Sara Gerke, Theodoros Evgeniou, and I Glenn Cohen. 2019.  Algorithms on regulatory lockdown in medicine.  _Science_ 366, 6470 (2019), 1202–1204.
  * Babic et al. (2021) Boris Babic, Sara Gerke, Theodoros Evgeniou, and I Glenn Cohen. 2021.  Beware explanations from AI in health care.  _Science_ 373, 6552 (2021), 284–286.
  * Bang and Frith (2017) Dan Bang and Chris D Frith. 2017.  Making better decisions in groups.  _Royal Society open science_ 4, 8 (2017), 170193.
  * Bansal et al. (2019a) Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019a.  Beyond accuracy: The role of mental models in human-AI team performance. In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , Vol. 7. 2–11.
  * Bansal et al. (2019b) Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. 2019b.  Updates in human-AI teams: Understanding and addressing the performance/compatibility tradeoff. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 2429–2437.
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021.  Does the whole exceed its parts? the effect of AI explanations on complementary team performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Barocas et al. (2019) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019.  _Fairness and Machine Learning_.  fairmlbook.org.  http://www.fairmlbook.org.
  * Bear and Woolley (2011) Julia B Bear and Anita Williams Woolley. 2011.  The role of gender in team collaboration and performance.  _Interdisciplinary science reviews_ 36, 2 (2011), 146–153.
  * Bertsimas et al. (2019) Dimitris Bertsimas, Arthur Delarue, Patrick Jaillet, and Sebastien Martin. 2019.  The price of interpretability.  _arXiv preprint arXiv:1907.03419_ (2019).
  * Black et al. (2022) Emily Black, Manish Raghavan, and Solon Barocas. 2022.  Model Multiplicity: Opportunities, Concerns, and Solutions. In _2022 ACM Conference on Fairness, Accountability, and Transparency_ (Seoul, Republic of Korea) _(FAccT ’22)_. Association for Computing Machinery, New York, NY, USA, 850–863.
  * Caruana et al. (2015) Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015.  Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_. 1721–1730.
  * Chouldechova (2017) Alexandra Chouldechova. 2017.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ 5, 2 (2017), 153–163.
  * Chouldechova and Roth (2020) Alexandra Chouldechova and Aaron Roth. 2020.  A snapshot of the frontiers of fairness in machine learning.  _Commun. ACM_ 63, 5 (2020), 82–89.
  * Corbett-Davies et al. (2017) Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017.  Algorithmic decision making and the cost of fairness. In _Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining_. 797–806.
  * Coston et al. (2023) Amanda Coston, Anna Kawakami, Haiyi Zhu, Ken Holstein, and Hoda Heidari. 2023.  A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms. In _2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)_. IEEE, 690–704.
  * Coston et al. (2021) Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. 2021.  Characterizing fairness over the set of good models under selective labels. In _International Conference on Machine Learning_. PMLR, 2144–2155.
  * Creel (2020) Kathleen A Creel. 2020.  Transparency in complex computational systems.  _Philosophy of Science_ 87, 4 (2020), 568–589.
  * D’Amour (2021) Alexander D’Amour. 2021.  Revisiting Rashomon: A Comment on” The Two Cultures”.  _Observational Studies_ 7, 1 (2021), 59–63.
  * D’Amour et al. (2020a) Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. 2020a.  Underspecification presents challenges for credibility in modern machine learning.  _arXiv preprint arXiv:2011.03395_ (2020).
  * D’Amour et al. (2020b) Alexander D’Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D Sculley, and Yoni Halpern. 2020b.  Fairness is not static: deeper understanding of long term fairness via simulation studies. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 525–534.
  * Davani et al. (2022) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 92–110.
  * De-Arteaga et al. (2018) Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. 2018.  Learning under selective labels in the presence of expert consistency.  _arXiv preprint arXiv:1807.00905_ (2018).
  * De-Arteaga et al. (2022) Maria De-Arteaga, Stefan Feuerriegel, and Maytal Saar-Tsechansky. 2022.  Algorithmic fairness in business analytics: Directions for research and practice.  _Production and Operations Management_ 31, 10 (2022), 3749–3770.
  * De-Arteaga et al. (2020) Maria De-Arteaga, Riccardo Fogliato, and Alexandra Chouldechova. 2020.  A case for humans-in-the-loop: Decisions in the presence of erroneous algorithmic scores. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Dietvorst et al. (2015) Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015.  Algorithm aversion: people erroneously avoid algorithms after seeing them err.  _Journal of Experimental Psychology: General_ 144, 1 (2015), 114.
  * Dobbe et al. (2020) Roel IJ Dobbe, Thomas Krendl Gilbert, and Yonatan Mintz. 2020.  Hard Choices in Artificial Intelligence: Addressing Normative Uncertainty through Sociotechnical Commitments. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 242–242.
  * Donahue et al. (2022) Kate Donahue, Alexandra Chouldechova, and Krishnaram Kenthapadi. 2022.  Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness.  _arXiv preprint arXiv:2202.08821_ (2022).
  * Dwork and Ilvento (2018) Cynthia Dwork and Christina Ilvento. 2018.  Fairness under composition.  _arXiv preprint arXiv:1806.06122_ (2018).
  * Dziugaite et al. (2020) Gintare Karolina Dziugaite, Shai Ben-David, and Daniel M Roy. 2020.  Enforcing Interpretability and its Statistical Impacts: Trade-offs between Accuracy and Interpretability.  _arXiv preprint arXiv:2010.13764_ (2020).
  * Eliassi-Rad et al. (2020) Tina Eliassi-Rad, Henry Farrell, David Garcia, Stephan Lewandowsky, Patricia Palacios, Don Ross, Didier Sornette, Karim Thébault, and Karoline Wiesner. 2020.  What science can do for democracy: a complexity science approach.  _Humanities and Social Sciences Communications_ 7, 1 (2020), 1–4.
  * Fazelpour and Danks (2021) Sina Fazelpour and David Danks. 2021.  Algorithmic bias: Senses, sources, solutions.  _Philosophy Compass_ 16, 8 (2021), e12760.
  * Fazelpour and De-Arteaga (2021) Sina Fazelpour and Maria De-Arteaga. 2021.  Diversity in sociotechnical machine learning systems.  _arXiv preprint arXiv:2107.09163_ (2021).
  * Fazelpour and Lipton (2020) Sina Fazelpour and Zachary C Lipton. 2020.  Algorithmic fairness from a non-ideal perspective. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 57–63.
  * Fazelpour et al. (2021) Sina Fazelpour, Zachary C Lipton, and David Danks. 2021.  Algorithmic Fairness and the Situated Dynamics of Justice.  _Canadian Journal of Philosophy_ (2021), 1–17.
  * Fleisher (2022) Will Fleisher. 2022.  Understanding, idealization, and explainable AI.  _Episteme_ 19, 4 (2022), 534–560.
  * Green and Chen (2019) Ben Green and Yiling Chen. 2019.  Disparate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments. In _Proceedings of the conference on fairness, accountability, and transparency_. 90–99.
  * Grimm (2012) Stephen Grimm. 2012.  The value of understanding.  _Philosophy Compass_ 7, 2 (2012), 103–117.
  * Gunning et al. (2019) David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. 2019.  XAI—Explainable artificial intelligence.  _Science Robotics_ 4, 37 (2019), eaay7120.
  * Guo et al. (2022) Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022.  A Survey on Automated Fact-Checking.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 178–206.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nati Srebro. 2016.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ 29 (2016).
  * Hellman (2020) Deborah Hellman. 2020.  Measuring algorithmic fairness.  _Virginia Law Review_ 106, 4 (2020), 811–866.
  * Herington (2020) Jonathan Herington. 2020.  Measuring fairness in an unfair World. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 286–292.
  * Hong and Page (2004) Lu Hong and Scott E Page. 2004.  Groups of diverse problem solvers can outperform groups of high-ability problem solvers.  _Proceedings of the National Academy of Sciences_ 101, 46 (2004), 16385–16389.
  * Jacobs and Wallach (2021) Abigail Z Jacobs and Hanna Wallach. 2021.  Measurement and fairness. In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_. 375–385.
  * Jobin et al. (2019) Anna Jobin, Marcello Ienca, and Effy Vayena. 2019.  The global landscape of AI ethics guidelines.  _Nature Machine Intelligence_ 1, 9 (2019), 389–399.
  * Johnson (2021) Gabbrielle M Johnson. 2021.  Algorithmic bias: on the implicit biases of social technology.  _Synthese_ 198, 10 (2021), 9941–9961.
  * Kamishima et al. (2011) Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. 2011.  Fairness-aware learning through regularization approach. In _2011 IEEE 11th International Conference on Data Mining Workshops_. IEEE, 643–650.
  * Kearns and Roth (2019) Michael Kearns and Aaron Roth. 2019.  _The ethical algorithm: The science of socially aware algorithm design_.  Oxford University Press.
  * Kleinberg (2018) Jon Kleinberg. 2018.  Inherent trade-offs in algorithmic fairness. In _Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems_. 40–40.
  * Kleinberg et al. (2018a) Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018a.  Human decisions and machine predictions.  _The quarterly journal of economics_ 133, 1 (2018), 237–293.
  * Kleinberg et al. (2018b) Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Cass R Sunstein. 2018b.  Discrimination in the Age of Algorithms.  _Journal of Legal Analysis_ 10 (2018), 113–174.
  * Kompa et al. (2021) Benjamin Kompa, Jasper Snoek, and Andrew L Beam. 2021.  Second opinion needed: communicating uncertainty in medical machine learning.  _NPJ Digital Medicine_ 4, 1 (2021), 1–6.
  * Krishnan (2020) Maya Krishnan. 2020.  Against interpretability: a critical examination of the interpretability problem in machine learning.  _Philosophy & Technology_ 33, 3 (2020), 487–502.
  * Landemore (2020) Hélène Landemore. 2020.  _Open democracy: Reinventing popular rule for the twenty-first century_.  Princeton University Press.
  * Lee et al. (2019) Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, et al. 2019\.  WeBuildAI: Participatory framework for algorithmic governance.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–35.
  * Leydesdorff et al. (2008) Loet Leydesdorff, Robert L Goldstone, Thomas Schank, et al. 2008\.  Betweenness centrality and the interdisciplinarity of cognitive science.  _Cognitive Science: A Multidisciplinary Journal_ (2008).
  * Lipton (2018) Zachary C Lipton. 2018.  The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ 16, 3 (2018), 31–57.
  * Liu et al. (2018) Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018.  Delayed impact of fair machine learning. In _International Conference on Machine Learning_. PMLR, 3150–3158.
  * Loi and Christen (2021) Michele Loi and Markus Christen. 2021.  Choosing how to discriminate: navigating ethical trade-offs in fair algorithmic design for the insurance sector.  _Philosophy & Technology_ 34 (2021), 967–992.
  * Lombrozo (2011) Tania Lombrozo. 2011.  The instrumental value of explanations.  _Philosophy Compass_ 6, 8 (2011), 539–551.
  * London (2019) Alex John London. 2019.  Artificial intelligence and black-box medical decisions: accuracy versus explainability.  _Hastings Center Report_ 49, 1 (2019), 15–21.
  * Lu et al. (2019) Joy Lu, Dokyun Lee, Tae Wan Kim, and David Danks. 2019.  Good explanation for algorithmic transparency.  _Available at SSRN 3503603_ (2019).
  * Madras et al. (2018) David Madras, Toni Pitassi, and Richard Zemel. 2018.  Predict responsibly: improving fairness and accuracy by learning to defer.  _Advances in Neural Information Processing Systems_ 31 (2018).
  * Martin Jr et al. (2020) Donald Martin Jr, Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S Isaac. 2020.  Participatory problem formulation for fairer machine learning through community based system dynamics.  _arXiv preprint arXiv:2005.07572_ (2020).
  * Marx et al. (2020) Charles Marx, Flavio Calmon, and Berk Ustun. 2020.  Predictive multiplicity in classification. In _International Conference on Machine Learning_. PMLR, 6765–6774.
  * Mayo-Wilson et al. (2011) Conor Mayo-Wilson, Kevin JS Zollman, and David Danks. 2011.  The independence thesis: When individual and social epistemology diverge.  _Philosophy of Science_ 78, 4 (2011), 653–677.
  * Miller (2019) Tim Miller. 2019.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial intelligence_ 267 (2019), 1–38.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting. In _Proceedings of the conference on fairness, accountability, and transparency_. 220–229.
  * Molnar (2022) Christoph Molnar. 2022.  _Interpretable Machine Learning_ (2 ed.).  christophm.github.io/interpretable-ml-book/
  * Mozannar and Sontag (2020) Hussein Mozannar and David Sontag. 2020.  Consistent estimators for learning to defer to an expert. In _International Conference on Machine Learning_. PMLR, 7076–7087.
  * Murdoch et al. (2019) W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. 2019.  Definitions, methods, and applications in interpretable machine learning.  _Proceedings of the National Academy of Sciences_ 116, 44 (2019), 22071–22080.
  * Obermeyer et al. (2019) Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019.  Dissecting racial bias in an algorithm used to manage the health of populations.  _Science_ 366, 6464 (2019), 447–453.
  * O’Connor and Weatherall (2019) Cailin O’Connor and James Owen Weatherall. 2019.  _The misinformation age_.  Yale University Press.
  * O’Rourke et al. (2013) Michael O’Rourke, Stephen Crowley, Sanford D Eigenbrode, and JD Wulfhorst. 2013.  _Enhancing communication & collaboration in interdisciplinary research_.  sage publications.
  * Page (2019) Scott Page. 2019.  _The diversity bonus_.  Princeton University Press.
  * Paravisini and Schoar (2013) Daniel Paravisini and Antoinette Schoar. 2013.  _The incentive effect of scores: Randomized evidence from credit committees_.  Technical Report. National Bureau of Economic Research.
  * Phillips (2017) Katherine W Phillips. 2017.  Commentary. What is the real value of diversity in organizations? Questioning our assumptions.  In _The diversity bonus_. Princeton University Press, 223–246.
  * Plate (1999) Tony A Plate. 1999.  Accuracy versus interpretability in flexible modeling: Implementing a tradeoff using gaussian process models.  _Behaviormetrika_ 26, 1 (1999), 29–50.
  * Post and Byron (2015) Corinne Post and Kris Byron. 2015.  Women on boards and firm financial performance: A meta-analysis.  _Academy of management Journal_ 58, 5 (2015), 1546–1571.
  * Raghu et al. (2019) Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. 2019.  Direct uncertainty prediction for medical second opinions. In _International Conference on Machine Learning_. PMLR, 5281–5290.
  * Rahwan (2018) Iyad Rahwan. 2018.  Society-in-the-loop: programming the algorithmic social contract.  _Ethics and information technology_ 20, 1 (2018), 5–14.
  * Raji et al. (2022) Inioluwa Deborah Raji, I Elizabeth Kumar, Aaron Horowitz, and Andrew Selbst. 2022.  The fallacy of AI functionality. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_. 959–972.
  * Rastogi et al. (2022) Charvi Rastogi, Liu Leqi, Kenneth Holstein, and Hoda Heidari. 2022.  A unifying framework for combining complementary strengths of humans and ML toward better predictive decision-making.  _arXiv preprint arXiv:2204.10806_ (2022).
  * Rodolfa et al. (2021) Kit T Rodolfa, Hemank Lamba, and Rayid Ghani. 2021.  Empirical observation of negligible fairness–accuracy trade-offs in machine learning for public policy.  _Nature Machine Intelligence_ 3, 10 (2021), 896–904.
  * Rudin (2019) Cynthia Rudin. 2019.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ 1, 5 (2019), 206–215.
  * Saxena et al. (2020) Devansh Saxena, Karla Badillo-Urquiola, Pamela J Wisniewski, and Shion Guha. 2020.  A human-centered review of algorithms used within the us child welfare system. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Selbst et al. (2019) Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019.  Fairness and abstraction in sociotechnical systems. In _Proceedings of the conference on fairness, accountability, and transparency_. 59–68.
  * Semenova et al. (2019) Lesia Semenova, Cynthia Rudin, and Ronald Parr. 2019.  A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning.  _arXiv preprint arXiv:1908.01755_ (2019).
  * Srivastava et al. (2020) Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. 2020.  An empirical analysis of backward compatibility in machine learning systems. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3272–3280.
  * Steel et al. (2018) Daniel Steel, Sina Fazelpour, Kinley Gillette, Bianca Crewe, and Michael Burgess. 2018.  Multiple diversity concepts and their ethical-epistemic implications.  _European journal for philosophy of science_ 8 (2018), 761–780.
  * Stinson and Vlaad (2024) Catherine Stinson and Sofie Vlaad. 2024.  A feeling for the algorithm: Diversity, expertise, and artificial intelligence.  _Big Data & Society_ 11, 1 (2024), 20539517231224247.
  * Suresh and Guttag (2019) Harini Suresh and John V Guttag. 2019.  A framework for understanding unintended consequences of machine learning.  _arXiv preprint arXiv:1901.10002_ 2, 8 (2019).
  * Tabassi (2023) Elham Tabassi. 2023.  Artificial Intelligence Risk Management Framework (AI RMF 1.0).  (2023).
  * Vredenburgh (2021) Kate Vredenburgh. 2021.  The right to explanation.  _Journal of Political Philosophy_ (2021).
  * Wang et al. (2021) Xuezhi Wang, Nithum Thain, Anu Sinha, Flavien Prost, Ed H Chi, Jilin Chen, and Alex Beutel. 2021.  Practical Compositional Fairness: Understanding Fairness in Multi-Component Recommender Systems. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. 436–444.
  * Waseem (2016) Zeerak Waseem. 2016.  Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In _Proceedings of the first workshop on NLP and computational social science_. 138–142.
  * Willingham and Cole (2013) Warren W Willingham and Nancy S Cole. 2013.  _Gender and fair assessment_.  Routledge.
  * Young (1994) H Peyton Young. 1994.  _Equity_.  Princeton University Press.
  * Zafar et al. (2017) Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. 2017.  Fairness constraints: Mechanisms for fair classification. In _Artificial Intelligence and Statistics_. PMLR, 962–970.
  * Zerilli et al. (2022) John Zerilli, Umang Bhatt, and Adrian Weller. 2022.  How transparency modulates trust in artificial intelligence.  _Patterns_ (2022), 100455.
  * Zrnic et al. (2021) Tijana Zrnic, Eric Mazumdar, Shankar Sastry, and Michael Jordan. 2021.  Who Leads and Who Follows in Strategic Classification?  _Advances in Neural Information Processing Systems_ 34 (2021).