# Fair Enough?
A map of the current limitations of the requirements to have “fair”
algorithms.

Alessandro Castelnovo111The views and opinions expressed are those of the
authors and do not necessarily reflect the views of Intesa Sanpaolo, its
affiliates or its employees<EMAIL_ADDRESS>Nicole
Inverardi22footnotemark: 2<EMAIL_ADDRESS>Gabriele Nanino
<EMAIL_ADDRESS>Ilaria Giuseppina Penco33footnotemark: 3
<EMAIL_ADDRESS>Daniele Regoli44footnotemark: 4
<EMAIL_ADDRESS>Data Science & Artificial Intelligence,
Intesa Sanpaolo S.p.A., Italy  Dept. of Informatics, Systems and
Communication, Univ. of Milano Bicocca, Italy

###### Abstract

In the recent years, the raise in the usage and efficiency of Artificial
Intelligence and, more in general, of Automated Decision-Making systems has
brought with it an increasing and welcome awareness of the risks associated
with such systems. One of such risks is that of perpetuating or even
amplifying bias and unjust disparities present in the data from which many of
these systems learn to adjust and optimise their decisions. This awareness has
on one side encouraged several scientific communities to come up with more and
more appropriate ways and methods to assess, quantify, and possibly mitigate
such biases and disparities. On the other hand, it has prompted more and more
layers of society, including policy makers, to call for “fair” algorithms. We
believe that while a lot of excellent and multidisciplinary research is
currently being conducted, what is still fundamentally missing is the
awareness that having “fair” algorithms is _per se_ a nearly meaningless
requirement, that needs to be complemented with a lot of additional _societal
choices_ to become actionable. Namely, there is a hiatus between what the
society is demanding from Automated Decision-Making systems, and what this
demand actually _means_ in real-world scenarios. In this work, we outline the
key features of such a hiatus, and pinpoint a list of fundamental ambiguities
and attention points that we _as a society_ must address in order to give a
concrete meaning to the increasing demand of fairness in Automated Decision-
Making systems.

###### keywords:

Fairness; Bias; Artificial Intelligence; Machine learning

## Introduction

Artificial Intelligence (AI) is undoubtedly experiencing an epoch of
unparalleled accomplishments. In particular, Machine Learning (ML) approaches
to AI —i.e., methods based on mathematical and statistical tools that enable
automated systems to learn patterns from data— are fostering the development
of numerous fruitful applications in an incredibly broad range of fields,
including computer vision, predictive analytics, and natural language
processing.

In the recent years, increasing attention is being devoted to the risks that
AI and, more broadly, Automated Decision-Making (ADM), inevitably brings with
it. One of such risks is that ADM systems, and ML approaches in particular,
are prone to learning from data a wide range of historical and social biases,
and to repeating and amplifying them at a scale that is challenging to monitor
and regulate. With the term fair-AI, or fair-ML, the literature usually
denotes the relatively recent area of research committed to study how to
assess and correct biases involved in algorithmic decision-making processes.
The 2023 Stanford AI Index report (Maslej et al., 2023), on the basis of data
about paper submissions to top-level conferences, says “Interest in AI ethics
continues to skyrocket”.

Along with this raising attention from the academic world, concerns for the
increasing use and diffusion of AI and ADM systems in many industrial sectors
emerged from ever broader layers of society. The issue of ethics in AI started
gaining popularity, usually under umbrella terms such as “Trustworthy AI” or
“Responsible AI”. In AI-related contexts (both academics and non-academics),
several _suitcase words_ began to be broadly utilised, such as “bias”,
“fairness”, “interpretability” and so on (Lipton and Steinhardt, 2018).

More concretely, the AI Index report (Maslej et al., 2023), based on
AIAAIC555AI, Algorithmic, and Automation Incidents and Controversies,
www.aiaaic.org/home. data, highlights that the number of reported incidents
related to the ethical misuse of AI is rapidly rising.

Against this background, in the spring of 2021 the European Commission
published the “Proposal for a Regulation of the European Parliament and of the
Council laying down harmonised rules on Artificial Intelligence” (The European
Commission, 2021b), the very first worldwide legal framework for AI, which
will likely be released in its final version in the coming months.666On
December 9, 2023, the European Parliament and Council reached a provisional
agreement on the Artificial Intelligence Act (AI Act). The agreed text will
now have to be formally adopted by both Parliament and Council to become EU
law. See https://www.europarl.europa.eu/news/en/press-
room/20231206IPR15699/artificial-intelligence-act-deal-on-comprehensive-rules-
for-trustworthy-ai. Since then, several other countries started discussing and
working on regulatory frameworks for AI systems: the Blueprint for AI in the
U.S., the pro-innovation approach in the U.K., to name a few. We refer once
more to the 2023 Stanford AI Index report (Maslej et al., 2023), that records
37 bills related to AI passed into law throughout the world in 2022. Among
these, it is worth mentioning the Spanish Comprehensive Law for Equal
Treatment and Non-Discrimination, that contains specific provisions for AI
algorithms involved in public administrations’ decision-making to take into
account bias-minimisation criteria. In November 2023, the U.K. government
hosted the first AI Safety Summit,777www.aisafetysummit.gov.uk. gathering
together international governments, leading AI companies, civil society groups
and experts to consider the risks of AI. In that context, the establishment of
a new AI Safety Institute was announced (gov.uk, 2023).

As a result, public and private actors who do employ AI and ADM in their
processes are being driven to attempt and satisfy this requirement for ethics
and justice. Practitioners and researches have access to several “fairness
toolkits” (Bellamy et al., 2018; Weerts et al., 2023; Wexler et al., 2020,
etc.), with the aim of assessing and mitigating unjust disparities in
algorithmic outcomes. Unfortunately, the level of maturity and consistency of
such toolkits appears to be still very low, and their effectiveness in
addressing the overall challenge of “fairness” is debated (Lee and Singh,
2021).

We think it is useful to briefly recall that the rising attention and the
lively debate that we are nowadays experiencing around the topic of fairness
in ADM bears interesting similarities with the “fairness movement” in academic
research that followed the approval of the US Civil Rights Act in 1964, as
beautifully discussed by Hutchinson and Mitchell (2019). In particular, the
decade after the approval of the Act saw a boom of research into how to
mathematically measure unfair discrimination in standardised tests within the
educational and employment communities, often with a focus on race. By the mid
1970s, researchers came up with multiple statistical measures of fairness and
their mutual incompatibility, in striking resemblance to the current
quantitative definitions and the related debate (see Section 3.3).
Unfortunately, by the late 1970s, that movement had mostly disappeared,
probably due in part to a general lack of consensus and clarity over standards
for determining what constitutes unfair discrimination and, more importantly,
to the ethical and legal principles that underpin fairness (Cole and Zieky,
2001). An issue that even the current “reboot” of quantitative research on
fairness hasn’t adequately addressed, yet.

### Contribution

This work aligns with a relatively recent stream of research that takes a
critical focus on the overall topic of fairness in Automated Decision-Making
(Ruggieri et al., 2023; Saxena et al., 2023; Buyl and De Bie, 2022; Selbst et
al., 2019; Cooper et al., 2021; Hoffmann, 2019). The critic is not on the
topic _per se_ —whose importance is not disputed— but rather on some usually
overlooked subtleties and assumptions, that often leads to over-reliance and
misplaced trust, that can in turn effectively lead to a deterioration of trust
in ADM in the long term.

Among the generic risks of “blindly” embracing simplified recipes, we can cite
the so-called _Automation Bias_ , namely the propensity to place unmotivated
trust on automated decisions, or —worse— the possibility of cherrypicking
certain simple approaches promoting the false perception that an ADM system
respects ethical values. Aivodji et al. (2019) has evocatively named the
latter _Fairwashing_.888More precisely, Aivodji et al. (2019) coined the term
‘fairwashing’ regarding black-box interpretability techniques used to promote
false perception of compliance to ethical values, but the concept has a
straightforward extension to fair-AI techniques.

Even if most of the ambiguities and attention points that we survey in what
follows have already been introduced and discussed separately by other
authors, on the one hand we try to give an overall perspective, grounding such
ambiguities on few foundational sources; on the other hand we approach the
topic as a call for action, placing the focus on the fact that most of the
ambiguities are a matter of decisions that are not technical in nature, but
rather _societal_ , and lie at the intersection of very diverse disciplines.

We primarily adopt the viewpoint of practitioners and developers of ADM
systems, who must deal with the practical difficulties of identifying
potential instances of unfair discrimination in real-world situations and
possibly mitigating them when appropriate.

##  1 The sources of ambiguities

Even if it is to some extent debatable, we can reasonably agree that the
overall perimeter of _unjust_ discrimination is group specific. By this, we
mean that we regard discrimination as _unjust_ with respect to some specific
dimensions, or attributes, that we “as a society” deem to be somehow
protected. Therefore, making decisions involving those attributes is
considered morally not justifiable. Typical examples of such attributes are
gender, ethnicity, religion, etc.

The two foundational points we want to raise are the following:

######  Source S1 (Sensitive attributes choice).

It is not that discrimination _per se_ is unjust —this would be trivial and
would somehow throw away the entire (data-driven) decision-making paradigm—
only discrimination with respect to _some attributes_ that we have either to
list and agree upon or to define with some reasonable criterion.

######  Source S2 (What we mean by unfair discrimination).

We need to clarify what we mean by making decisions _involving_ such
attributes, and in what cases such decision-making represent an unjust
discrimination.

These two seemingly superficial, or at least very abstract points are at the
root of all the ambiguities and subtleties of fairness in decision-making. The
first point underlies all the problems about what attributes we should monitor
and what should we do with their interactions. The second point underlies the
problems of defining/clarifying the mechanisms that we deem are not morally
acceptable through which a protected attribute can impact the decision.

Both points are ultimately a matter of collective and agreed upon choice.
Unfortunately, no mathematical model, sophisticated as it may be, can tell us
what are the characteristics that should be protected from discrimination or,
e.g., whether employing the level of income for loan decisions is morally
legitimate even if income is on average lower for female applicants.

We devote Section 2 to schematise and discuss the most important ambiguities
originating from Source S1, while we do the same in Section 3 for Source S2.
Section 4 is devoted to analyse the importance of causality when dealing with
discrimination, and about its limitations.

In Table 1, we summarise the key points raised throughout the paper.

Table 1: Schematic summary of current ambiguities and attention points of
fairness requirements for ADM.

Source |  | Type |  | Issue
---|---|---|---|---
S1: Sensitive attributes choice |  | Ambiguity |  |  A1: Protected groups
| Ambiguity |  |  A2: Sensitive data collection
| Ambiguity |  |  A3: Group aggregation
| Ambiguity |  |  A4: Intersectional bias
S2: What we mean by unfair discrimination |  | Ambiguity |  |  A5: Direct and indirect discrimination
| Ambiguity |  |  A6: Objective justification
| Attention Point |  |  AP1 Over-reliance on observational metrics
| Ambiguity |  |  A7: The threshold problem
| Attention Point |  |  AP2 Assumptions of causal structure
| Attention Point |  |  AP3: Use of human attributes as causes

##  2 Sensitive attributes choice

###  2.1 When a given attribute should be _protected_?

High level non-discrimination principles can be found in several legislative
frameworks. At the European level, the starting point of analysis on this
topic is the EU Charter of Fundamental Rights. In particular, art 21,1 reads:
Any discrimination based on any ground such as sex, race, colour, ethnic or
social origin, genetic features, language, religion or belief, political or
any other opinion, membership of a national minority, property, birth,
disability, age or sexual orientation shall be prohibited. It is worth noting
that the EU Charter is addressed to institution and bodies of the European
Union and to the Member States, and not to the private sector, at least not
directly. To give concrete application to these principles, various directives
have been adopted over time. They detail provisions for specific protected
groups and/or specific domains, such as work, environment, or access to goods
and services. See the EU non-discrimination website and Wachter et al. (2021)
for more details, and Table 2 for a schematic overview.

Moreover, there are domains where the EU Charter is explicitly referred to as
a source of high level principles to be observed, but without any concrete
details about their implementation. An example is the Consumer Credit
Directive, which claims: This Directive respects fundamental rights and
observes the principles recognised in particular by the Charter of Fundamental
Rights of the European Union. In particular, this Directive seeks to ensure
full respect for the rules on protection of personal data, the right to
property, non-discrimination, protection of family and professional life, and
consumer protection pursuant to the Charter of Fundamental Rights of the
European Union. The European Commission has in place a proposal of revision of
the Directive on consumer credits (The European Commission, 2021a) that
contains the following Article 6, explicitly on non-discrimination: Member
States shall ensure that the conditions to be fulfilled for being granted a
credit do not discriminate against consumers legally resident in the Union on
ground of their nationality or place of residence or on any ground as referred
to in Article 21 of the Charter of Fundamental Rights of the European Union,
when those consumers request, conclude or hold a credit agreement or
crowdfunding credit services within the Union. which seems to suggest that all
attributes referred to in the EU Charter of Fundamental Rights are to be
considered sensitive for credit access purposes.

Something not dissimilar can be found in the US legislation. We refer to
Barocas et al. (2019, chap 6) and Barocas and Selbst (2016) for more details.

Table 2: Schematic summary of protected categories explicitly covered by EU
Directives on non-discrimination. See also the EU non-discrimination website.

Directive |  | year |  | Domain of application |  | Protected categories
---|---|---|---|---|---|---
Race Equality Directive (Directive 2000/43/EC) |  | 2000 |  |  employment, social protection, healthcare, education, access to and supply of goods and services which are available to the public |  | race and ethnic origin
Employment Directive (Directive 2000/78/EC) |  | 2000 |  | working environment |  |  religion or belief, disability, age, sexual orientation
Gender Access Directive (Directive 2004/113/EC) |  | 2004 |  |  access to and supply of goods and services |  | gender
Gender Equality Directive (Directive 2006/54/EC) |  | 2006 |  | employment |  | gender

In general, we can say that there are high level principles requiring non-
discrimination with respect to several dimensions and some more specific
legislation regulating single domains and/or single sensitive attributes (such
as gender or “race”), but besides that, there is ambiguity regarding what
individual characteristics should be considered as protected or sensitive.

Namely, we can summarise by formulating the following:

######  Ambiguity A1 (Protected groups).

Given a specific phenomenon, what are the groups of people that we should
consider as protected, and with respect to which we therefore have to take
care of assessing and avoiding any unjust discrimination?

###  2.2 Sensitive data collection

Suppose that your team is in charge of developing a Decision System to rank
applicants to a job posting in order to prioritise interviews. The data you
collect come entirely from submitted CVs and the online application form,
where applicants are asked questions on standard personal data (name, gender,
address, date and place of birth), previous work experience, education, and
skills. While developing the system, you take particular care of avoiding
unjustified dependencies of the outputs on information such as gender and
nationality of the applicants. You think you have done all that was possible
to prevent any form of unjust discrimination, but when you present your work
to your boss, she points out that you actually have no control whatsoever
about discrimination with respect to a lot of other sensitive information
—such as political and religious opinion— for the very trivial fact that you
don’t have such information to begin with. We know very well that Fairness
Through Unawareness, i.e. being blind to sensitive information, is in general
not enough to prevent unjust discrimination, or at least some forms of unjust
discrimination (see, e.g. Barocas et al., 2019; Castelnovo et al., 2022;
Kusner et al., 2017). This simple example raises the following:

######  Ambiguity A2 (Sensitive data collection).

Should developers of ADM systems keep track of all the sensitive attributes
that they wouldn’t otherwise record, for the sole purpose of assessing unjust
discrimination with respect to those attributes?

According to Buyl and De Bie (2022), sensitive data collection may not only be
morally unacceptable, but also extremely unlikely to occur because people are
typically reluctant to share personal information. This could actually pose a
significant barrier to the development of ADM systems.

Notice that the issue of sensitive data collection is also explicitly
considered by the European Commission’s proposal for an AI Act (The European
Commission, 2021b), where it is foreseen that developers of AI systems may
process special categories of personal data “To the extent that it is strictly
necessary for the purposes of ensuring bias monitoring, detection and
correction” (The European Commission, 2021b, art 10, 5). It is clear that this
is only a chance given to the provider, and that the use of personal data must
not be extended to other uses.

###  2.3 The problem of aggregation

Even once we agree on what are the dimensions that contain groups of people to
be protected against unjust discrimination, we still have to face the problem
of how to aggregate individuals along those dimensions. The simplest example
in this respect is that of age. Suppose that we are in a situation in which we
agree that there should be no discrimination based on the age of the
applicants. We can take the example of credit lending: in the U.S., the Equal
Credit Opportunity Act forbids to discriminate “on the basis of race, color,
religion, national origin, sex or marital status, or age” ( Equal Credit
Opportunity Act, §1691(a)(1)). What do we actually mean by age? Should we
consider separate protected groups for each year of age (with the assumption
that we have access to that information)? Or is it enough to aggregate
individuals into a handful of broader classes, e.g. $<30$, $[30,60]$, $>60$?
If we opt for the broader classes, how do we select the thresholds? Should we
take quantiles of the age distribution of our data, or should we use some
common-sense knowledge of how people are actually segmented in the society?

######  Ambiguity A3 (Group aggregation).

The specific identification of most attributes that are commonly considered
protected depends on alternative ways of aggregating individuals: what
strategy should developers follow to choose the proper aggregation when
assessing unjust discrimination?

This same problem arises for more or less all the potentially protected
attributes, think e.g. of profession, place of birth, political opinions,
disability, ethnicity. Notice that the way in which we aggregate individuals
into groups may have a significant impact on the assessment of bias and
discrimination along that dimension. It may happen, e.g., that individuals
below 30 years of age are not discriminated against people over-60, but if you
focused on the narrow range of 20-25, individuals in this group may still
undergo big disparities compared to people in the range 60-65.

It is important to note that, at least for some of the potentially protected
characteristics, there are reservations and discussions on the prospect of
placing people into rigid, exclusive categories (Lu et al., 2022). For
instance, multiracial individuals come from various racial groupings. Indeed,
at least on a biological/genetic level, race and ethnicity are now seen as
extremely fluid and nuanced ideas rather than simple categorical attributes.
Gender and sexual orientation are subject to very comparable criticism.
Despite appearing somewhat abstract, the question of whether it is possible to
describe entire groups of individuals based on (roughly stable) human
characteristics is a serious one, which we will revisit in Section 4.1.

###  2.4 How shall we deal with several sensitive attributes _at the same
time_?

Most of the literature on fairness in ML deals with a single sensitive
dimension, usually assumed to be binary. Even if this is hardly the case in
real-world scenarios, it is true that, when facing with multiple sensitive
variables, one could simply repeat the assessment for each of the variables,
separately.

However, this may hide forms of _intersectional bias_ (Buolamwini and Gebru,
2018; Roy et al., 2023). Namely, the fact that the decision system may have an
equal impact on men and women, and an equal impact on black individuals and
white individuals, but still present significant disparities between black
women and white men.

This issue arises whenever we assess discrimination with respect to groups:
achieving some form of parity at the group level may hide disparities _within_
the group, e.g. at the intersection of multiple sensitive characteristics.
Kearns et al. (2018) call this phenomenon _fairness gerrymandering_. It is
true that this drawback might be easily resolved, in principle, by conducting
a fairness evaluation on all potential sensitive group intersections.
Unfortunately, this is not a very practical solution, since the number of
subgroups to take into account increases exponentially as more dimensions are
added, and similarly, the number of data samples in each subgroup rapidly
decreases, making it extremely difficult to draw any reliable statistical
conclusion (Roy et al., 2023; Kong, 2022).

Roy et al. (2023) further underline that the multi-dimensional aspect of
discrimination exacerbates the already mentioned issues of aggregation and
categorisation of individuals into groups, and is of course very much
connected to the choice of protected groups.

######  Ambiguity A4 (Intersectional bias).

Is it fair enough to evaluate unjust discrimination with respect to sensitive
attributes separately? If not, which combinations of sensitive characteristics
should we give priority to (given that we cannot realistically hope to assess
all possible combinations)?

##  3 What we mean by unfair discrimination

Agreeing on what we mean in concrete terms when we state that a particular
group of individuals is discriminated against is one of the trickiest issues
to deal with. Quantitative studies to assess the presence of unjust
discrimination has naturally turned mainly to legal literature and anti-
discrimination legislation frameworks to come up with concrete mechanisms
underlying “unfair” practices in decision-making.

The legal debate on the topic, as well as the legislative provisions, is
highly diversified, and in various respects still very much open. Moreover,
the concept of discrimination has deep roots in both political and moral
philosophy, with arguments that go beyond the focus on how to fairly compare
people belonging to different social groups, and rather discuss on what
grounds individuals should be compared at all, with debates going as far as
discussing merit and desert as true drivers of individual successes.

In light of this, it should come as little surprise that the need for “fair”
algorithms has a variety of nuances and complexities, many originating from
very fundamental and hotly contested conceptions, which we can barely hope to
be met at all.

In the following section, we summarise the basic legal notions that have
guided most of the literature on quantitative assessment of unjust
discrimination of protected groups in ADM.

###  3.1 Direct and indirect discrimination

The key distinction when analysing the concept of discrimination in legal
situations is arguably that between _direct discrimination_ and _indirect
discrimination_ (Altman, 2020). Roughly speaking, an _indirect discrimination_
happens when the decision depends on a sensitive characteristic _through_
other variables, therefore not explicitly. While an analysis based on causal
reasoning allows to be more specific and detailed —we refer to Section 4 for
such a discussion— an example can provide a straightforward intuition. Suppose
you have to develop a decision system to support credit lending decisions
based on available applicants’ characteristics. Among these characteristics
there are, e.g., the level of income, the requested amount, the record of past
loans, and the gender attribute. The system will show a _direct
discrimination_ with respect to gender if it is aware of the gender attribute,
and uses it (together with the other variables) to estimate the optimal
outcome. For instance, a Machine Learning model that is given access to
historical data on loan applications and their (non-)repayment, may learn a
statistical association between gender and loan repayments, thus effectively
assigning a bonus/malus weight for the _sole gender attribute_ of the
applicant. Conversely, if the system is not aware of the gender attribute, or
—more precisely— if it does not assign any contribution to the gender
attribute itself, the system is not showing a direct discrimination. However,
it may nonetheless undergo _indirect discrimination_ with respect to gender
since it outputs decisions on the basis of the level of income, which is
associated to gender. In this last case, the system shows an indirect
discrimination with respect to gender, mediated by the level of income.

The direct vs indirect distinction for discrimination is definitely not the
only one proposed in the literature or present in legislative frameworks. This
approach is mainly found in the European Union and U.K. laws —see, e.g.,
Directive 2006/54/EC and Directive 2000/43/EC, and also Barnard and Hepple
(2000)— while U.S. anti-discrimination laws rely on a similar distinction
between _disparate treatment_ and _disparate impact_ , the former representing
forms of explicit discrimination while the latter encompassing a broader set
of indirect mechanisms —see, e.g., Barocas and Selbst (2016) and Barocas et
al. (2019, chapter 6).

Another well known distinction is the one made within the framework of the
theory of _equality of opportunity_ , i.e. the idea supporting the need to
‘level the playing field’ for all individuals: different degrees of successes
and failures of individuals are fairly distributed only when they “play” on a
field without “slopes” that may advantage some with respect to others. Experts
identify a wide range of possibilities, from a _formal equality of
opportunity_ , proposing the avoidance of explicit discrimination of protected
groups, to forms of more and more _substantive equality of opportunity_ ,
supporting the avoidance of more subtle and indirect mechanisms of
discrimination (and sometimes equated to the _equality of outcome_) —see
Barnard and Hepple (2000) for a legal perspective and Elford (2023) for a
philosophical one. Unfortunately, as it is the case for most of the concepts
in discrimination theory, the level of consensus about what equality of
opportunity actually means in real-world scenarios is incredibly low. Quoting
Bevir (2010): There is widespread agreement that equality of opportunity is a
good thing, even a constituent of a just society, but very little consensus on
what it requires. Defenders of equality of opportunity suppose that it
requires people to be able to compete on equal terms, on a “level playing
field,” but they disagree over what it means to do so. They believe that
equality of opportunity is compatible with, and indeed justifies, inequalities
of outcome of some sort, but there is considerable disagreement over precisely
what degree and kind of inequalities it justifies and how it does so.

Against this background, while it is widely recognised that direct
discrimination is hardly acceptable, the debate on indirect discrimination is
much more nuanced, as we shall also discuss later on (Section 3.2 and, for a
causality-based view, Section 4). Moreover, two factors, among others,
contribute to make the distinction at times quite loose and difficult to
maintain:

  1. 1.

a system may not explicitly use sensitive characteristics, but still produce
an indirect discrimination through variables that are _per se_ not sensitive,
but very much associated to sensitive information;

  2. 2.

oftentimes, one way to avoid indirect discrimination is to provide favourable
conditions to the otherwise unfavoured group. Unfortunately, using dual
standards on the basis of sensitive characteristics is precisely what direct
discrimination means.

Point 1 is the well known problem of _proxy variables_ , that may be exploited
—possibly with an explicit discriminatory intent— in order to obtain a
discriminatory outcome while maintaining a formal absence of direct
discrimination. Ingold and Spoer (2016) describe the situation of the Amazon
same-day delivery service in 6 metropolitan areas in the U.S.: they document a
strong correlation between neighbourhoods not reached by the same-day delivery
service and the presence of a significant majority of black residents. The
most striking example is that of Boston, where the very central neighbourhood
of Roxbury (with a 59 percent presence of black residents) was not granted the
same-day delivery service, while all the areas surrounding Roxbury are
eligible for the service.999As explicitly declared in Ingold and Spoer (2016),
Amazon later extended its same-day delivery service to the entire metropolitan
area of Boston, as well as to that of New York and Chicago. Amazon declared
that demographics of neighbourhoods played no role in their choices. In cases
like this, it is indeed difficult to disentangle direct and indirect forms of
discrimination, and agree on what we should and should not tolerate as “fair”
practices.

Point 2 is an example of the so-called _affirmative_ (or _positive_) _action_
, i.e. providing more favourable conditions to groups of individuals
traditionally discriminated against. The other side of the coin is known as
_reverse discrimination_ , i.e. maintaining dual standards not justified by
task-related skills and characteristics (Lipton et al., 2018). Kim (2022)
refers to Ricci v. DeStefano, where a case of affirmative action by the City
of New Haven’s fire department in favour of black firefighters has been judged
to be unfairly discriminating the supposed favoured group (white). The U.S.
Supreme Court decision on Ricci v. DeStefano reports that there was not enough
evidence that the performed disparate treatment by the City has avoided a
disparate impact:101010Incidentally, in partial support to Section 4 on
causality, notice that this formulation has a genuine counterfactual nature.
We conclude that race-based action like the City’s in this case is
impermissible under Title VII unless the employer can demonstrate a strong
basis in evidence that, had it not taken the action, it would have been liable
under the disparate-impact statute.

As previously pointed out, the landscape is complex and extremely diverse, and
a thorough discussion is well beyond the scope and ambition of this work.
However, we want to make clear that there is no consensus in the scientific
communities on what mechanisms actually constitute an unjust discrimination,
particularly regarding indirect forms and their potential tension with the
direct ones.

In light of this, we state the following:

######  Ambiguity A5 (Direct and indirect discrimination).

When evaluating unjust discrimination, should developers of ADM systems take
into account all the potential direct and indirect ways by which sensitive
characteristics may have affected the outcome? Is it acceptable to engage in
direct discrimination with the purpose of preventing indirect discrimination?

###  3.2 What characteristics represent a source of objective justification
for discrimination?

Ambiguity A5 is largely intertwined with the discussion on how to deal with
indirect discrimination, once we agree that also indirect discrimination may
be unacceptable in some cases.

The focus of the debate is sometimes presented as the identification of
‘business necessities’. With this term, one loosely refers to the possible
presence of characteristics that are fundamentally “relevant” to the target
estimation, therefore their use to perform ADM is justifiable even when such
characteristics bear some dependence on sensitive dimensions. Quoting Barocas
and Selbst (2016) This defense [of the presence of a ‘business necessity’] is,
in a very real sense, the crux of disparate impact analysis. The problem is
how to decide when a variable is eligible to represent a business necessity.

Incidentally, notice that also direct discrimination may be at times bypassed
by business necessities: Romei and Ruggieri (2014) suggests the example of the
necessity to take explicitly into account gender when advertising for an actor
to play, e.g., a male role.

Legislative frameworks make explicit reference to this possibility. For
instance the Directive 2006/54/EC on non-discrimination of men and women in
matters of employment and occupation, has provisions of objective
justification already in the very definition of indirect discrimination (art
2,1(b)) ‘indirect discrimination’: where an apparently neutral provision,
criterion or practice would put persons of one sex at a particular
disadvantage compared with persons of the other sex, unless that provision,
criterion or practice is objectively justified by a legitimate aim, and the
means of achieving that aim are appropriate and necessary The key ingredients
to have an objective justification for indirect discrimination are here
identified to be a “legitimate aim”, and “appropriate and necessary means” to
achieve it.

Similar arguments can be found in U.S. laws. The US Civil Rights Act Title VII
on non-discrimination in the workplace, among the criteria in support of the
presence of unjust discrimination, has the following: a complaining party
demonstrates that a respondent uses a particular employment practice that
causes a disparate impact on the basis of race, color, religion, sex, or
national origin and the respondent fails to demonstrate that the challenged
practice is job related for the position in question and consistent with
business necessity Here “job-relatedness” and “business necessity” are cited
as legitimate reasons to have indirect discrimination.

Barocas and Selbst (2016) discuss in details the evolution and oscillation of
the interpretation of such terminology by the U.S. Supreme Court, somehow
emphasising the ambiguity of it. The point is that all these concepts are
inherently fuzzy. One may interpret a characteristic to be job related if it
correlates with job performance, and similarly job performance may be
considered a legitimate aim.

Albach and Wright (2021) present the results of extensive surveys (2157 Amazon
Mechanical Turk workers) on different domains to understand what are the
_perceived_ drivers of fairness in Machine Learning applications.
Interestingly, they find that ‘relevance’ and ‘increases accuracy’ are the 2
dimensions that explain most of the perceived fairness of a given variable.
More precisely, in a first study, the attribute ‘increases accuracy’ was not
among the exposed ones, and ‘relevance’ was by far the most important
dimension explaining perceived fairness. Acknowledging the possible ambiguity
of the term ‘relevance’, the authors designed further studies, either
replacing ‘relevance’ with ‘increases accuracy’, or adding ‘increases
accuracy’ to the possible drivers. They conclude that most of the times the
survey participants perceive ‘relevance’ as a placeholder for the
characteristic of increasing performance, and that this single dimension
contributes the most to the perceived fairness of a given variable.

These results seem to support the loose interpretation of business necessities
as any variable able to increase the predictive performance. But notice that
this interpretation strongly undermines the very concept of indirect
discrimination: if all the variables that correlates with the outcome can be
used legitimately irrespective of their dependence on sensitive
characteristics, than we hardly have indirect discrimination.

Against this background, practitioners that need to assess and possibly
mitigate any unjust discrimination when developing an ADM system, face the
following:

######  Ambiguity A6 (Objective justification).

What qualities should an attribute have, if any, in order to be eligible for
use in automatic judgements, even if it serves as a basis for indirect
discrimination?

#### Long-term impact

A very subtle issue is what happens on the long-term given some notion of
unjust discrimination. It is not at all clear that “fair” ADM systems are
going to push society towards an actual equality of opportunity. Suppose we
acknowledge that income is allocated on average unevenly between men and
women, and that this is due to historical bias that, as a society, we aim to
eradicate. Should “fair” ADM systems avoid gender disparities due to income in
their outcomes? This would mean that ADM systems should favour women over men,
at the same level of income. In the long run, is this aligned with the overall
goal of eradicating gender bias? Needless to say, the answer to this question
is: we don’t have a clue. Indeed, a “broad view” of equality of opportunity
(Barocas et al., 2019) is not focused on ADM at all, but is much more top-down
and global, discussing how society overall should be designed, how its
institutions should function, what are the incentives that, in the long run,
would tend to actually level the playing field for every individual, without
the need of any “local” intervention on single ADM systems. As an example of
this broader spirit, we like to cite the article 3 of the Italian
Constitution, which reads: It is the duty of the Republic to remove those
obstacles of an economic or social nature which constrain the freedom and
equality of citizens, thereby impeding the full development of the human
person and the effective participation of all workers in the political,
economic and social organisation of the country.

###  3.3 The risk of over-reliance on fairness metrics

Sometimes, the various forms of unjust discrimination that we have encountered
so far are associated with specific _observational metrics_. With this term,
we mean quantities that can be computed by having access to samples of the
joint distribution $P(X,A,Y,S)$, where variable $A$ represents the sensitive
attribute, $X$ denotes the set of other (non-sensitive) characteristics, $Y$
is the target variable, and $S$ is the model outcome.

While the discussion of fairness metrics is out of scope in this study —there
is plenty of works in the literature on the topic, see, e.g., Castelnovo et
al. (2022); Mitchell et al. (2021) and references therein— we here provide
some examples to highlight how they enter in our discussion. In general,
observational metrics are expressed as statistical relationships among (some
of) the variables introduced above (indeed, they are sometimes called
_statistical metrics_). In particular, we can distinguish the following
classes of observational metrics (Barocas et al., 2019):

  * 1.

independence: $S\perp\\!\\!\\!\perp A$;

  * 2.

conditional independence: $S\perp\\!\\!\\!\perp A\mid X^{\prime}$, with
$X^{\prime}\subseteq X$;

  * 3.

separation: $S\perp\\!\\!\\!\perp A\mid Y$;

  * 4.

sufficiency: $Y\perp\\!\\!\\!\perp A\mid S$.

Independence is usually associated with disparate impact: the fact that the
model outcome bears no dependence on the sensitive attribute, directly
translates into a substantial parity in the outcomes. In the simple case of
binary classifications, where $S\in\\{0,1\\}$,111111More precisely, most
binary classifiers outputs a real-valued _score_ , which is an estimate of the
probability of the input belonging to the positive class. In those cases, the
binary outcome is a consequence of thresholding the score. independence is
indeed equivalent to what is known as _Demographic Parity_ (or _Statistical
Parity_), namely the equality of acceptance rate among all the demographic
groups along the sensitive dimension.

The introduction of business needs is associated sometimes with conditional
independence, and sometimes with separation. The idea behind these
associations is the following: if we agree that some variables (that we
collectively label with $X^{\prime}\subseteq X$) are acceptable grounds to
make decisions irrespective of their dependence on sensitive attributes (i.e.,
they represent business necessities), then the natural quantity to inspect is
the independence of the outcome on sensitive attributes _stratified_ by those
variables. Namely, if we agree that the use of the level of income for credit
decisions is justified by business needs, then we may require to have equality
of acceptance rate between, say, male and female applicants _within the same
level of income_. In other words, we are requiring that the only gender
disparity we are willing to tolerate is the one justified by the level of
income. In a seminal paper on Conditional Demographic Parity, Wachter et al.
(2021) suggest the use of such type of metric in relation to the EU non-
discrimination legal framework.

Similarly, in the case of separation metrics, we allow only disparities
justified by the target variable: e.g., we tolerate gender disparities as long
as they are justified by actual disparities in repayment rates. This is in
line with the discussion made in Section 3.2 on business necessities: if
business needs can be justified on grounds of evidence of predictive
(therefore, often, economic) performance, then metrics of the separation class
are indeed more appropriate.

Notice, incidentally, that the set of variables $X^{\prime}$ used as
conditions determines how much the corresponding independence statement is
focusing on smaller and smaller groups. Fair-AI literature discusses the
distinction between _individual_ and _group_ metrics, depending on the
granularity of the observations compared: while group metrics are computed as
averages over groups that are then compared to one another, individual metrics
consider comparison among individuals. At the level of observational metrics,
this distinction is somehow overlapping to that between direct and indirect
discrimination: namely, the more granular we push the metrics the more we
concentrate on direct effects, and vice versa (Castelnovo et al., 2022). We
shall see in Section 4 that these relationships can indeed be decoupled with
the lens of causal reasoning. To the best of our knowledge, this subtlety is
absent in most legal debates.

While these arguments are valid, they unfortunately conceal certain possible
weaknesses that we will cover in the reminder of the section.

#### The label problem

Many of the statistical metrics (namely, those of the separation and
sufficiency classes) rely on the comparison between model outcomes ($S$) and
target labels ($Y$). Unfortunately, relying on labels is delicate, for (at
least) two types of drawbacks. One the one hand, we have a problem of actual
_access_ to labels: e.g., when granting loans, there is no information about
applicants that don’t receive loans in the first place (namely, access to $Y$
is possible only for applicants with positive outcome $S$). Moreover, even
when we do have access to labels, oftentimes there may be a significant delay,
as it is the case for repayment of loans, or for job performance in recruiting
decision-making: in such scenarios we can monitor labels (and thus fairness
metrics based on those labels) only considerably _ex-post_.121212Of course,
this _ex-post_ effect is there also when monitoring algorithmic performance
with respect to such targets. On the other hand, there may be an issue of
_label bias_ (Obermeyer et al., 2019): this happens when seemingly effective
proxies for ground truth are chosen as target, but there are mechanisms (often
involved and ignored by the developer and/or the decision-maker) by which
those proxies embed forms of bias. Incidentally, notice that the issue of a
sound choice of the target variables in fair-AI goes even beyond the choice of
a statistical metric potentially grounded on biased labels: optimising an ADM
system by learning from biased labels is of course problematic under multiple
perspectives (Baumann et al., 2023).

#### The incompatibility problem

It is well-known that statistical metrics are not, in general, mutually
compatible (Kleinberg et al., 2016; Barocas et al., 2019). E.g., one cannot
hope to have an ADM system presenting no Demographic Disparities and, at the
same time, having equal error rates for different (sensitive) groups of
individuals. This is true for all the classes of metrics listed above, a part
for degenerate cases. This evidence implies that, when using such metrics, one
must make choices, that are ultimately associated with the perspective one has
on unjust discrimination. Some works have proposed guidelines —usually in the
form of decision trees or diagrams— to help finding the most appropriate
statistical metric given domain-specific constraints (see, e.g., Makhlouf et
al., 2021b, a; Haeri et al., 2022; Smith et al., 2023). However, as the
authors of such works clearly acknowledge, the process of following the
proposed decision diagrams is itself complicated, involving necessarily multi-
disciplinary competencies, and in any case they warn not to take these
diagrams too categorically or as a set of well-established prescriptions. To
make things even more blurry, if it is true that having perfect parity with
respect to different metric classes is mathematically impossible, allowing a
limited level of disparity may be attainable with multiple metrics at the same
time (Bell et al., 2023), suggesting that focusing too much on a single metric
maybe counter-productive after all.

#### The threshold problem

Once practitioners have computed the values of observational metrics, they
face the problem of “deciding” whether the found numbers are enough to raise a
warning or not. A 10% difference in acceptance rate between male and female is
fair enough? What about using the ratio of acceptance rates instead of the
difference? Borrowing from Ruggieri et al. (2023) “The apparently innocuous
choice between the algebraic operators (1) or (2) [difference vs ratio],
however, has an enormous impact on how decisions are affected.” The threshold
problem is as trivial as it sounds: statistical metrics are real-valued
measures of (some notion of) disparities, they are not a binary trigger. Some
attempts to face this issue can be found in the U.S. context. One is the well-
known _four-fifths rule_ (Equal Employment Opportunity Commission, 2015),
stating that an acceptance rate (in recruiting setting) of the ‘unfavoured’
group less than 80% of the acceptance rate of the ‘favoured’ group constitutes
an evidence of disparate impact.131313Incidentally, notice that this is a rule
on the _ratio_ of acceptance rates. To be clear, the Equal Employment
Opportunity Commission (2015) adds several _caveats_ to that rule, that are
often overlooked, e.g. that a greater ratio may still constitute an adverse
impact “when significant in both statistical and practical terms”, while
smaller ratios may not be enough evidence for adverse impact, e.g. when they
are based on small samples. Other approaches rely on statistical significance
of proportions: we can mention the _Castaneda rule_ (Romei and Ruggieri,
2014), named after the Castaneda v. Partida law case, that prescribes a
significance test on $P(A=1\mid S=1)$. More precisely, if the proportion of
protected individuals selected ($P(A=1\mid S=1)$) departs more than “two or
three” (_sic._) standard deviations from what we expect in the independence
scenario ($P(A=1)\times P(S=1)$), then a significant disparity is considered
at play.141414In Castaneda v. Partida a binomial distribution is assumed for
the proportion $P(A=1\mid S=1)$. Roughly speaking, considering a normal
approximation to the binomial proportion, then 2 (3) standard deviations
correspond to a 95% (99.7%) confidence interval. In other words, a disparity
is assumed whenever the observed proportion would be observed less than 5%
(0.03%) of the times if no real disparity was present.

In general, developers of ADM systems are therefore challenged by the
following:

######  Ambiguity A7 (The threshold problem).

Given an observational metric, on what grounds should we decide how small the
measured disparity should be in order to be considered fair enough?

#### Lack of causality

Another drawback of purely observational metrics is that they are blind with
respect to the underlying generation mechanisms: it is well known that
statistical correlations among variables may have very different natures. They
may be due to a direct causation of one variable on another; to causation
mediated by other variables; or may be spurious, i.e. due to some common
factor causing both of them; or may even be spuriously created when
conditioning on a common effect (Peters et al., 2017; Neal, 2020). More on the
topic of causality in fair-AI in Section 4.

A vivid critic of the “metric” approach to fairness can be found in Selbst et
al. (2019), where the authors face the fair-AI problem from the perspective of
Science and Technology Studies. In particular, they talk about a _Formalism
Trap_ : fairness and discrimination are complex concepts, that contain many
subtle dimensions, among which a _procedural_ , a _contextual_ , and a
_contestable_ dimension, that cannot be grasped by relatively simple
statistical quantities. Of course, this does not imply that statistical
measures are altogether wrong or useless, but only that they must be utilised
as tools, to be set in a wider context.

Indeed, survey studies about fairness _perception_ in ML show a very diverse
and nuanced landscape, where perspectives on fairness are associated with
socio-demographic factors of the participants (Grgić-Hlača et al., 2022), and
consensus on fairness notions is very low (Starke et al., 2022; Harrison et
al., 2020).

Overall, considering together all these aspects about observational metrics,
we can state the following:

######  Attention Point AP1 (Over-reliance on observational metrics).

Purely observational fairness metrics should be taken with a grain of salt.
They can and should be used as a means for a deeper reasoning on the
mechanisms underlying a phenomenon, rather than a final word on the presence
or lack of unjust discrimination.

##  4 The importance of causality

Legal provisions against discrimination frequently have a counterfactual bent,
either overtly or covertly.

Regarding direct discrimination, the Directive 2006/54/EC has the following
definition: where one person is treated less favourably on grounds of sex than
another is, has been or would be treated in a comparable situation while, in
Carson v. Bethlehem Steel Corp., we read: The central question in any
employment-discrimination case is whether the employer would have taken the
same action had the employee been of a different race (age, sex, religion,
national origin, etc.) and everything else had remained the same.

Regarding indirect discrimination, the 2015 U.S. Supreme Court decision for
Texas Dept. of Housing & Community Affairs v. Inclusive Communities Project,
Inc. established a “robust causality” requirement for the plaintiff to
demonstrate the presence of disparate impact.

In general, it is reasonable to claim that, to inspect whether or not an
unjust discrimination is present, we need to understand what are the causal
relationship among relevant variables and sensitive ones. E.g., how do we
disentangle direct and indirect discrimination? How do we know that a variable
that we can argue to be necessary for our business is the one responsible for
the observed disparate impact?

$D$$G$$Y$? Figure 1: 1973 Berkeley graduate admission. $G$ represents gender,
$D$ the chosen Department and $Y$ the admission outcome. The dashed line
represents the possible direct effect of gender on the admission choice.

A well-known and extensively studied example of the importance of causal
reasoning to assess fairness in decision-making is that of the graduate
admissions at Berkeley in 1973 as discussed by Bickel et al. (1975).
Historical data aggregated over the six largest departments showed a
significant face-value disparity of admission rate between male and female
applicants, with male being accepted with a rate 14 percentage points higher
(44% vs 30%). This seemingly huge discrepancy in favour of men applicants
turned out to be _reversed_ in 4 out of 6 departments, when inspecting
admission rates data for each of the 6 department _separately_. This is an
example of the so-called _Simpson’s paradox_ , the well known fact in
probability and statistics by which aggregate trends may disappear or revert
at a more granular level, due to relative frequencies of individuals in
subpopulations. In the Berkeley admission case, men applicants were much more
numerous in departments where the admission rate was higher (both for men and
women), while female applicants concentrated more on departments with lower
acceptance rate.

The bottom line151515First drawn by Bickel et al. (1975) and then deeply
discussed by Judea Pearl; see e.g. the popular audience book by Pearl and
Mackenzie (2018). is that, when conditioned on department choice, the observed
gender disparity disappears, revealing the absence of a direct discrimination
by admission committees. The observed disparity can be seen as an indirect
effect due to the fact that female applicants tended to choose more
competitive departments. In the language of causal theory, department choice
is a _mediator_ between gender and admission (Figure 1).

We refer to Barocas et al. (2019) and Plecko and Bareinboim (2022) for an in-
depth analysis of this exemplar case, we here simply rephrase what is
beautifully reported in Barocas et al. (2019), namely that even when the
causal structure is revealed and we know the mechanism by which the disparity
among genders appears on the observational data, the question about
discrimination is still there, we have merely “scratched the surface”. Namely,
one could (and should?) ask and investigate the reasons why women tended to
apply to more competitive departments in the first place. In fact, what the
causal structure has revealed is that there is no _direct_ discrimination, and
that the entire observed disparity is a consequence of the choice of the
department by the applicants. But the causal structure cannot say whether this
channel of disparity is a “fair” one or not. Namely, we are back at the
discussions of Sections 3.1-3.2 and to Ambiguities A5-A6.

It is interesting to underline that, even when employing observational
metrics, the process of choosing which of the many available metrics are the
most appropriate for a particular setting is a form of causal reasoning in
disguise. Indeed, in the Attention Point AP1 we suggest to use observational
metrics precisely as a means to understand the mechanisms underlying a
specific observational distribution. As hinted in Section 3.3, metrics based
on conditional disparities are appropriate when we assume the presence of
legitimate _mediators_ : conditioning on a mediator is precisely the way
through which we can isolate other _unwanted_ sources of disparities,
analogously to the 1973 Berkeley admission case.

In some sense, the causal framework allows to capture more clearly what we
should compare in order to asses a specific discrimination mechanism.
Analogously to the well-known Fairness Through Awareness approach (Dwork et
al., 2012), that suggests to compare observed individuals on the basis of a
to-be-defined similarity metric, the causal framework suggests to compare a
observed individual to a counterfactual self for whom sensitive conditions are
changed in a “controlled” way. Depending on the “causal channels” that we
block or let free, we can pinpoint different discrimination mechanisms.

In the Berkeley graduate admission example we may flip gender _while keeping
the department fixed_ , thus measuring the direct effect only. Or we could
flip gender and _let the department free to vary_ : a male transported in the
counterfactual world where he is a woman would have himself chosen higher
demanding departments, thus this feature would change as well. In this case we
are targeting both direct and indirect effects, the latter channeled through
the department choice.

$Z$$A$$W$$Y$ Figure 2: Standard Fairness Model (Plecko and Bareinboim, 2022).
The bidirected dashed edge represents the possible presence of hidden
confounding.

Building on seminal works on causality in fair-AI, such as those by Kusner et
al. (2017) and Chiappa (2019), Plecko and Bareinboim (2022) perform a
remarkable analysis of fair-AI through the lens of causal theory. They suggest
a basic causal graph (Figure 2), that they call Standard Fairness Model,
containing 4 categories of nodes: the sensitive attribute $A$, the target
variable $Y$, mediator variables $W$ and other non-mediator variables $Z$.
Variables $Z$ can have impact on $Y$, and they can be confounded with $A$,
meaning that there may be unobserved common causes to $Z$ and $A$. The Boston
same-day delivery example discussed in Section 3.1 can be seen as an example
of confounding, where ethnicity of residents ($A$) has some ancestor cause in
common with the neighbourhood they end up living in ($Z$). Given this basic
structure, they identify 3 building blocks of the unconditional dependence of
$Y$ on $A$:

  1. 1.

_Direct Effects_ : represented by the direct arrow from $A$ to $Y$, measurable
by the variation in $Y$ resulting from flipping the attribute $A$ while
keeping $Z$ and $W$ fixed;

  2. 2.

_Indirect Effects_ : represented by arrows going through $W$, measurable by
the variation in $Y$ resulting from the flipping $A$ and letting $W$ vary, but
keeping $Z$ and $A$ fixed;

  3. 3.

_Spurious Effects_ : represented by the confounding mechanism through $Z$,
measurable by the variation in $Y$ for different _observed_ values of $A$,
without considering the causal effects of $A$.

For the mathematically inclined, these three types of effect read:

$\displaystyle\text{DE}_{a_{0},a_{1};X^{\prime}}=$ $\displaystyle
P(Y_{a_{1},W_{a_{0}}}\mid X^{\prime})-P(Y_{a_{0}}\mid X^{\prime}),$ (1a)
$\displaystyle\text{IE}_{a_{0},a_{1};X^{\prime}}=$ $\displaystyle
P(Y_{a_{0},W_{a_{1}}}\mid X^{\prime})-P(Y_{a_{0}}\mid X^{\prime}),$ (1b)
$\displaystyle\text{SE}_{a_{0},a_{1};X^{\prime}}=$ $\displaystyle
P(Y_{a_{0}}\mid a_{1},X^{\prime})-P(Y_{a_{0}}\mid a_{0},X^{\prime});$ (1c)

where, e.g., the expression $Y_{a_{1},W_{a_{0}}}$ denotes the counterfactual
distribution of $Y$ when $A$ is set to $a_{1}$ but $W$ is kept at the value it
would have taken when $A$ is set to $a_{0}$. We have here denoted with
$X^{\prime}$ a generic set of variables, it could be, e.g., $A$ itself, or
$Z$, etc. The conditioning on $X^{\prime}$ deserves some additional
explanation. Recall that the conditioning event represents the subset of
individuals that happen to meet the given characteristics: conditioning on
$A=a_{0}$ means to restrict the attention to individuals that we observe
having value $a_{0}$ for $A$. As mentioned in Section 3.3, the more we narrow
down with the conditioning event, the more the effect we are capturing is
individual in nature. If we stick to pure observational distributions, this
concept remains somewhat ambiguous, since the type of the effect and the
conditioning are entangled to one another. With the tools of causal analysis
things become much more transparent: on one side we have the type of mechanism
we are targeting, on an independent level we have the resolution with which we
are looking at those mechanisms. The two extremes are:

  1. 1.

the individual level, or _unit_ level, where we have a measure of the effect
for each single individual;

  2. 2.

the unconditional level, or _population_ level, where we have a single measure
representing the overall average effect.

Plecko and Bareinboim (2022) call the individual level metrics _more powerful_
, since the larger the granularity, the more we can average out effects that
are nevertheless present at a finer-grained level. Indeed, whenever we measure
a null effect by equations (1), we can only say that there is no
discrimination at the level $X^{\prime}$, but we cannot draw any conclusions
on finer levels. On the contrary, if we get a non-zero metric we can be sure
that some discrimination is at play. Unfortunately, the finer-grained we aim,
the higher the chance that we cannot compute the effect on the basis of
observational data alone, and we need to add further modelling assumptions —an
issue known as _identifiability_ of causal effects (for further details, see
Plecko and Bareinboim, 2022). In particular, individual effects are
intrinsically not computable via observable data alone, since an individual
cannot be _observed_ into distinct states. In contrast, if we average at a
coarser-grained level, we can compare similar individuals (i.e., sharing the
same values of the covariates $X^{\prime}$ that identify the desired level)
_observed_ into different treatment states.

Notice that spurious effects lie at a different level with respect to direct
and indirect ones: while the latter are variations in outcome _in response_ to
intervention on $A$ and/or its descendants, spurious effects are not causally
_after_ $A$, but _before_ it. This is why Eq. (1c) is the only one in which
the change in $A$ is _observed_ and not due to _intervention_. Moreoveor, to
isolate the effect, the causal consequences of a change in $A$ are suppressed
by measuring the counterfactual distribution of $Y_{a_{0}}$. Drawing again
from the Boston same-day delivery example, if we intervened on ethnicity we
would have no consequences on having/not-having access to the same-day
delivery service. To measure the dependence we have to compare an observed
black individual (say $A=a_{0}$) to an observed white individual ($A=a_{1}$)
in the counterfactual setting where she is black ($Y_{a_{0}}$).

We have here focused on the simple causal structure presented in Figure 2, but
of course things can be much more involved than that. In fact, Plecko and
Bareinboim (2022) extend their analysis by “unpacking” the confounder and
mediator variables ($Z$ and $W$, respectively), therefore providing an
exhaustive framework including cases when $Z$ and $W$ are multiple, and also
accounting for the possibility that any of those variables represent a
business necessity.

With this concise overview of causality-based fair-AI, we hope to have made
two points clear: _i)_ that, on the one hand, the causal framework is a tool
in and of itself for analysing the phenomenon under study in greater detail,
and _ii)_ that, on the other hand, going deeper often reveals richer and more
complex structures, as we have seen with the introduction of spurious effects
or the decoupling of individual-vs-population level and the mechanisms type
level.

Overall, we encourage a discussion on the variables at play and their
particular role before even thinking about any fairness metric to be computed
and monitored.

###  4.1 Limits of causality-based approaches

Unfortunately, causality-aware approaches come with plenty of drawbacks.

One of the problems, that we briefly mentioned above, is that some of the more
individual-level discrimination effects are not computable with the sole
observational distribution and the (assumed) causal structure
(_identifiability_ problem): the functional relationships among the variables
are needed, the so-called _structural equations_. Relying on structural
equations introduces an additional level of assumptions and estimations beyond
those of the graph structure (see, e.g., Peters et al., 2017; Neal, 2020).

In general, causal theory requires rather strong assumptions, some of which
are not verifiable. Blind computation of causal effects when assumptions do
not hold can lead to false results, and therefore to risk of harmful actions
in response to such results. Fawkes et al. (2022) argue that the usual
assumptions that underlie causal analysis are not really realistic when
dealing with fairness, and call for more care when assuming the underlying
class of causal models.

######  Attention Point AP2 (Assumptions of causal structure).

Causal tools require strong, often unverifiable, assumptions. Downstream
consequences of wrong assumptions can lead to wrong or even harmful actions.
Therefore, particular care must be taken when relying on such tools.

While Attention Point AP2 calls for care in the assumptions underlying causal
_relationships_ , at an even more fundamental level, some scholars have raised
concerns regarding causal graph _nodes_. In particular, about the very
possibility of using human attributes as nodes in a causal graph. The point is
that, when we make causal statements about human attributes, oftentimes the
direction of causality is far from being obvious. This ambiguity is beyond the
“standard” difficulty of causality assumptions that we have mentioned above:
it is rooted in the very meaning of human attributes. Hu and Kohler-Hausmann
(2020) make the example of _religious belief_ : in particular, they mention
the fact that ‘being catholic’ can be seen both as a cause and as a
consequence of a lot of different characteristics, such as ‘believe in the
resurrection of Christ’, or ‘believe in saints’. Barocas et al. (2019) also
make a similar example: we can reasonably assume that one’s religious
affiliation may causally influence her/his education, thus seeing education as
a consequence of religious belief. But we can just as well agree on the
opposite: the level of education has impacts on one’s spiritual attitudes and
believes. The point is that education and religious believes are too
intertwined to be cast in a simple cause-effect relationship. Hu and Kohler-
Hausmann (2020) talk about _constitutive_ relationship as opposed to _causal_
relationship. A constitutive relationship indicates the proprieties in virtue
of which an object —in our case a human attribute— is what it is. In this
view, the propriety of ‘believing in the resurrection of Christ’ is neither a
cause nor causes that of ‘being catholic’. It rather constitutes it along with
many other proprieties bundled together (e.g., the duty of going to mass, the
infallibility of the Pope). By that it is meant that without this specific
belief (and other rooted practices), the meaning of the social category
‘catholic’ would be totally different. If this ontology of human attributes is
correct, what insight can we gain for causal approaches to fairness? A first
suggestion is that the difficulties in drawing causal arrows come from the
attempt to provide causal interpretation of a non-causal relation. Following
Hu and Kohler-Hausmann (2020), we understand that constitutive proprieties
play an important role in cases of discrimination based on sensitive
attributes. Bearing some physical traits can rarely by itself justify
discriminatory outcome. In most cases unfair outcome will be motivated by
inter-subjective believes and practices that are constitutive proprieties of
sensitive attribute as social kind. While rather informative, these remarks
poses a serious question to causal approaches in fairness and may encourage
toward more encompassing methods of modelling inferences.

Among the experts in causal theory, one school of thought adheres to the well-
known motto “no causation without manipulation” (Holland, 1986): to draw
causal inference on something, we should be able to manipulate it. Following
this perspective, a notable proposal is that of shifting from modelling an
attribute to modelling the _exposure to attribute_ (Greiner and Rubin, 2011).
In practical terms, this implies exposing someone to something that is
manipulable and that represents a given attribute, such as the _declared_
name, age, and other features on a CV. We can easily think of manipulating
(i.e., faking) information declared on a CV, without the need of manipulating
the actual characteristics of the individual. Unfortunately, even if useful
and effective in some cases (Bertrand and Mullainathan, 2004), this comes with
drawbacks as well. In order to make this strategy work, it appears that we
need to model the causal implications of a change in a declared attribute,
thus understanding all the different perceptions that an attribute could
possibly generate (Barocas et al., 2019). Ultimately, there is no way of
coming up with such a model without an understanding of the social perception
and meaning of the actual attribute and category, that is the thing that
raised the doubts to begin with. In other words, modelling perception to
sensitive attributes is as problematic as modelling the attributes themselves.
This aligns with the alternative view of causality as the identification of a
_generative process_ , rather than relying on direct manipulation only
(Goldthorpe, 2001). Moreover, it appears that intervening on exposure to
attributes triggers different downward changes. Considering the example of
declared origin on a CV, if we tweak the declared origin we are acting on the
social perception of that attribute, but we miss all the (causal) connections
that would be present if we tweaked the true origin (whatever this may mean),
e.g. education, and so forth. These difficulties may unveil a forced
commitment to some constructivist theories of human attributes (Mallon, 2016).

In general, the problem of causality in social studies is very much debated,
several alternative definitions and mechanisms have been proposed and
analysed, but there is no clear consensus in the scientific community
(Hirschman and Reed, 2014; Goldthorpe, 2001), which calls for care in handling
such concepts. The inherent conclusion is that causal modelling and related
discrimination issues vitally depend on our ability to understand the
relationships between human attributes, users’ perception of them, and the
representational power of the objects we choose as nodes in the graph.

We can summarise these arguments by stating the following:

######  Attention Point AP3 (Use of human attributes as causes).

If it is true that causal mechanisms underpinning phenomena are paramount to
evaluate the presence of possible unjust discrimination, a particular care
must be taken on the meaning we assign to nodes representing human attributes
and social categories.

#### Remark

By raising these Attention Points we are by no means supporting the idea that
causal theory shouldn’t be used any longer because there are too many inherent
dangers and drawbacks. Quite the opposite, we promote the use of causal tools,
and recognise the value in such approaches, but we also call for an awareness
of their weak points and limitations.

##  5 Conclusions

In this work, we support the view that the understanding of the landscape of
unjust discrimination in ADM, despite the impressive work done in the last
decade or so by the fair-AI community, is not yet mature enough to be “put
into practice”. In particular, there are gaps at the intersection between the
mathematical and statistical tools developed by statisticians and AI
researchers, the legal non-discrimination provisions, and the ethical and
social notions of fairness. We believe these gaps are deep, and not so easy to
bridge, in part precisely because they lie at the boundaries between quite
different worlds.

The goal of this work is to create a map of the barriers that now make it
difficult both for researchers in the field of artificial intelligence and for
practitioners and developers of ADM systems to meet the societal requirement
of having “fair” algorithms. We believe that the requirement is still too
vague and that, in order to be put in place, we have to resolve a set of
ambiguities, most of which are societal in nature rather than technical. Given
their strong multidisciplinary content, these ambiguities will be best
addressed and discussed when researchers and practitioners from the various
fields involved (e.g., statisticians, AI experts, ethicists, and legal
experts) collaborate toward the goal, potentially creating a shared vocabulary
and set of working notions. In fact, this is the spirit that also guided this
work.

As a final remark, notice that there are of course other problematic aspects
of unjust discrimination in ADM systems, that are somehow out of the scope of
this work and would have required a much broader analysis. We can cite, e.g.,
how to face the challenge of bias in models with unstructured data such as
images and text, especially in generative AI models such as the modern Large
Language Models; or more technological problems, such as those raised by the
modularity of AI systems, that are usually composed by several steps and
components, making it complicated to clarify how fairness issues may propagate
through the process.

## Author contributions

D.R. conceived the main ideas presented and outlined a first draft of the
manuscript. D.R. and A.C. contributed to the overall structure of the
manuscript. G.N. contributed, in particular, in shaping and writing the
section on limitations of causal theory. All authors contributed to the
discussion of the ideas presented and to the overall review of the manuscript.

## References

  * ITC (1947) , 1947.  Italian Constitution.  URL: https://www.senato.it/istituzione/la-costituzione.
  * Aivodji et al. (2019) Aivodji, U., Arai, H., Fortineau, O., Gambs, S., Hara, S., Tapp, A., 2019.  Fairwashing: the risk of rationalization, in: Chaudhuri, K., Salakhutdinov, R. (Eds.), Proceedings of the 36th International Conference on Machine Learning, PMLR. pp. 161–170.  URL: https://proceedings.mlr.press/v97/aivodji19a.html.
  * Albach and Wright (2021) Albach, M., Wright, J.R., 2021\.  The role of accuracy in algorithmic process fairness across multiple domains, in: Proceedings of the 22nd ACM Conference on Economics and Computation, Association for Computing Machinery, New York, NY, USA. p. 29–49.  doi:10.1145/3465456.3467620.
  * Altman (2020) Altman, A., 2020.  Discrimination, in: Zalta, E.N. (Ed.), The Stanford Encyclopedia of Philosophy. Winter 2020 ed.. Metaphysics Research Lab, Stanford University.
  * Barnard and Hepple (2000) Barnard, C., Hepple, B., 2000\.  Substantive equality.  The Cambridge Law Journal 59, 562–585.  doi:10.1017/S0008197300000246.
  * Barocas et al. (2019) Barocas, S., Hardt, M., Narayanan, A., 2019.  Fairness and Machine Learning: Limitations and Opportunities.  fairmlbook.org.  http://www.fairmlbook.org.
  * Barocas and Selbst (2016) Barocas, S., Selbst, A.D., 2016\.  Big data’s disparate impact.  California law review , 671–732doi:10.15779/Z38BG31.
  * Baumann et al. (2023) Baumann, J., Castelnovo, A., Crupi, R., Inverardi, N., Regoli, D., 2023.  Bias on demand: A modelling framework that generates synthetic data with bias, in: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 1002–1013.  doi:10.1145/3593013.3594058.
  * Bell et al. (2023) Bell, A., Bynum, L., Drushchak, N., Zakharchenko, T., Rosenblatt, L., Stoyanovich, J., 2023\.  The possibility of fairness: Revisiting the impossibility theorem in practice, in: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 400–422.  doi:10.1145/3593013.3594007.
  * Bellamy et al. (2018) Bellamy, R.K.E., Dey, K., Hind, M., Hoffman, S.C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., Nagar, S., Ramamurthy, K.N., Richards, J., Saha, D., Sattigeri, P., Singh, M., Varshney, K.R., Zhang, Y., 2018\.  AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias.  URL: https://arxiv.org/abs/1810.01943.
  * Bertrand and Mullainathan (2004) Bertrand, M., Mullainathan, S., 2004\.  Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination.  American Economic Review 94, 991–1013.  doi:10.1257/0002828042002561.
  * Bevir (2010) Bevir, M., 2010.  Equality of Opportunity, in: Encyclopedia of Political Theory. SAGE Publications, Inc.  doi:10.4135/9781412958660.
  * Bickel et al. (1975) Bickel, P.J., Hammel, E.A., O’Connell, J.W., 1975.  Sex bias in graduate admissions: Data from berkeley.  Science 187, 398–404.  doi:10.1126/science.187.4175.398.
  * Buolamwini and Gebru (2018) Buolamwini, J., Gebru, T., 2018\.  Gender shades: Intersectional accuracy disparities in commercial gender classification, in: Friedler, S.A., Wilson, C. (Eds.), Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR. pp. 77–91.  URL: https://proceedings.mlr.press/v81/buolamwini18a.html.
  * Buyl and De Bie (2022) Buyl, M., De Bie, T., 2022.  Inherent limitations of ai fairness.  arXiv preprint arXiv:2212.06495 URL: https://arxiv.org/abs/2212.06495.
  * Castelnovo et al. (2022) Castelnovo, A., Crupi, R., Greco, G., Regoli, D., Penco, I.G., Cosentini, A.C., 2022\.  A clarification of the nuances in the fairness metrics landscape.  Scientific Reports 12, 4209\.  doi:10.1038/s41598-022-07939-1.
  * Chiappa (2019) Chiappa, S., 2019.  Path-specific counterfactual fairness.  Proceedings of the AAAI Conference on Artificial Intelligence 33, 7801–7808.  doi:10.1609/aaai.v33i01.33017801.
  * Cole and Zieky (2001) Cole, N.S., Zieky, M.J., 2001\.  The new faces of fairness.  Journal of educational Measurement 38, 369–382.  doi:1745-3984.2001.tb01132.x.
  * Cooper et al. (2021) Cooper, A.F., Abrams, E., NA, N., 2021.  Emergent unfairness in algorithmic fairness-accuracy trade-off research, in: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, Association for Computing Machinery, New York, NY, USA. p. 46–54.  doi:10.1145/3461702.3462519.
  * Department for Science, Innovation & Technology (2023) Department for Science, Innovation & Technology, 2023\.  A pro-innovation approach to AI regulation.  https://www.gov.uk/government/publications/ai-regulation-a-pro-innovation-approach.
  * Dwork et al. (2012) Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R., 2012.  Fairness through awareness, in: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, Association for Computing Machinery, New York, NY, USA. p. 214–226.  doi:10.1145/2090236.2090255.
  * Elford (2023) Elford, G., 2023.  Equality of Opportunity, in: Zalta, E.N., Nodelman, U. (Eds.), The Stanford Encyclopedia of Philosophy. Fall 2023 ed.. Metaphysics Research Lab, Stanford University.
  * Equal Employment Opportunity Commission (2015) Equal Employment Opportunity Commission, 2015\.  Uniform Guidelines on Employment Selection Procedures.  URL: https://www.govinfo.gov/content/pkg/CFR-2011-title29-vol4/xml/CFR-2011-title29-vol4-part1607.xml. 29 Code of Federal Regulation §1607.4(D).
  * Fawkes et al. (2022) Fawkes, J., Evans, R., Sejdinovic, D., 2022.  Selection, ignorability and challenges with causal fairness, in: Schölkopf, B., Uhler, C., Zhang, K. (Eds.), Proceedings of the First Conference on Causal Learning and Reasoning, PMLR. pp. 275–289.  URL: https://proceedings.mlr.press/v177/fawkes22a.html.
  * Goldthorpe (2001) Goldthorpe, J.H., 2001.  Causation, Statistics, and Sociology.  European Sociological Review 17, 1–20.  doi:10.1093/esr/17.1.1.
  * gov.uk (2023) gov.uk, 2023.  Prime Minister launches new AI Safety Institute.  gov.uk: Press Release URL: https://www.gov.uk/government/news/prime-minister-launches-new-ai-safety-institute.
  * Greiner and Rubin (2011) Greiner, D.J., Rubin, D.B., 2011\.  Causal effects of perceived immutable characteristics.  The Review of Economics and Statistics 93, 775–785.  URL: http://www.jstor.org/stable/23016076.
  * Grgić-Hlača et al. (2022) Grgić-Hlača, N., Lima, G., Weller, A., Redmiles, E.M., 2022\.  Dimensions of diversity in human perceptions of algorithmic fairness, in: Equity and Access in Algorithms, Mechanisms, and Optimization, Association for Computing Machinery, New York, NY, USA.  doi:10.1145/3551624.3555306.
  * Haeri et al. (2022) Haeri, M.A., Hartmann, K., Sirsch, J., Wenzelburger, G., Zweig, K.A., 2022.  Promises and pitfalls of algorithm use by state authorities.  Philosophy & Technology 35, 33\.  doi:10.1007/s13347-022-00528-0.
  * Harrison et al. (2020) Harrison, G., Hanson, J., Jacinto, C., Ramirez, J., Ur, B., 2020.  An empirical study on the perceived fairness of realistic, imperfect machine learning models, in: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 392–402.  doi:10.1145/3351095.3372831.
  * Hirschman and Reed (2014) Hirschman, D., Reed, I.A., 2014\.  Formation stories and causality in sociology.  Sociological Theory 32, 259–282.  doi:10.1177/0735275114558632.
  * Hoffmann (2019) Hoffmann, A.L., 2019.  Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse.  Information, Communication & Society 22, 900–915.  doi:10.1080/1369118X.2019.1573912.
  * Holland (1986) Holland, P.W., 1986.  Statistics and causal inference.  Journal of the American Statistical Association 81, 945–960.  URL: http://www.jstor.org/stable/2289064.
  * Hu and Kohler-Hausmann (2020) Hu, L., Kohler-Hausmann, I., 2020\.  What’s sex got to do with fair machine learning?  arXiv preprint arXiv:2006.01770 doi:10.48550/arXiv.2006.01770.
  * Hutchinson and Mitchell (2019) Hutchinson, B., Mitchell, M., 2019\.  50 years of test (un)fairness: Lessons for machine learning, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 49–58.  doi:10.1145/3287560.3287600.
  * Ingold and Spoer (2016) Ingold, D.I., Spoer, S., 2016\.  Amazon doesn’t consider the race of its customers. should it?  Bloomberg URL: https://www.bloomberg.com/graphics/2016-amazon-same-day.
  * Jefatura del Estado (2022) Jefatura del Estado, 2022.  Comprehensive Law for Equal Treatment and Non-Discrimination.  URL: https://www.boe.es/eli/es/l/2022/07/12/15/con.
  * Kearns et al. (2018) Kearns, M., Neel, S., Roth, A., Wu, Z.S., 2018.  Preventing fairness gerrymandering: Auditing and learning for subgroup fairness, in: Dy, J., Krause, A. (Eds.), Proceedings of the 35th International Conference on Machine Learning, PMLR. pp. 2564–2572.  URL: https://proceedings.mlr.press/v80/kearns18a.html.
  * Kim (2022) Kim, P.T., 2022.  Race-aware algorithms: Fairness, nondiscrimination and affirmative action.  California Law Review 110, 1539\.
  * Kleinberg et al. (2016) Kleinberg, J., Mullainathan, S., Raghavan, M., 2016.  Inherent trade-offs in the fair determination of risk scores.  arXiv:1609.05807.
  * Kong (2022) Kong, Y., 2022.  Are “intersectionally fair” ai algorithms really fair to women of color? a philosophical analysis, in: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 485–494.  doi:10.1145/3531146.3533114.
  * Kusner et al. (2017) Kusner, M.J., Loftus, J., Russell, C., Silva, R., 2017\.  Counterfactual fairness.  Advances in neural information processing systems 30\.  URL: https://proceedings.neurips.cc/paper_files/paper/2017/hash/1271a7029c9df08643b631b02cf9e116-Abstract.html.
  * Lee and Singh (2021) Lee, M.S.A., Singh, J., 2021\.  The landscape and gaps in open source fairness toolkits, in: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA.  doi:10.1145/3411764.3445261.
  * Lipton et al. (2018) Lipton, Z., McAuley, J., Chouldechova, A., 2018.  Does mitigating ml’s impact disparity require treatment disparity?  Advances in neural information processing systems 31\.  URL: https://proceedings.neurips.cc/paper_files/paper/2018/hash/8e0384779e58ce2af40eb365b318cc32-Abstract.html.
  * Lipton and Steinhardt (2018) Lipton, Z.C., Steinhardt, J., 2018\.  Troubling trends in machine learning scholarship.  arXiv preprint arXiv:1807.03341 .
  * Lu et al. (2022) Lu, C., Kay, J., McKee, K., 2022.  Subverting machines, fluctuating identities: Re-learning human categorization, in: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 1005–1015.  doi:10.1145/3531146.3533161.
  * Makhlouf et al. (2021a) Makhlouf, K., Zhioua, S., Palamidessi, C., 2021a.  Machine learning fairness notions: Bridging the gap with real-world applications.  Information Processing & Management 58, 102642.  doi:https://doi.org/10.1016/j.ipm.2021.102642.
  * Makhlouf et al. (2021b) Makhlouf, K., Zhioua, S., Palamidessi, C., 2021b.  On the applicability of machine learning fairness notions.  SIGKDD Explor. Newsl. 23, 14–23.  doi:10.1145/3468507.3468511.
  * Mallon (2016) Mallon, R., 2016.  The construction of human kinds.  Oxford University Press.
  * Maslej et al. (2023) Maslej, N., Fattorini, L., Brynjolfsson, E., Etchemendy, J., Ligett, K., Lyons, T., Manyika, J., Ngo, H., Niebles, J.C., Parli, V., Shoham, Y., Wald, R., Clark, J., Perrault, R., 2023\.  The AI Index 2023 Annual Report.  Technical Report. Institute for Human-Centered AI, Stanford University, Stanford, CA.  URL: https://aiindex.stanford.edu/report/.
  * Mitchell et al. (2021) Mitchell, S., Potash, E., Barocas, S., D’Amour, A., Lum, K., 2021.  Algorithmic fairness: Choices, assumptions, and definitions.  Annual Review of Statistics and Its Application 8, 141–163.  doi:10.1146/annurev-statistics-042720-125902.
  * Neal (2020) Neal, B., 2020.  Introduction to causal inference.  Course Lecture Notes (draft) .
  * Obermeyer et al. (2019) Obermeyer, Z., Powers, B., Vogeli, C., Mullainathan, S., 2019\.  Dissecting racial bias in an algorithm used to manage the health of populations.  Science 366, 447–453.  doi:10.1126/science.aax2342.
  * Pearl and Mackenzie (2018) Pearl, J., Mackenzie, D., 2018\.  The book of why: the new science of cause and effect.  Basic books.
  * Peters et al. (2017) Peters, J., Janzing, D., Schölkopf, B., 2017.  Elements of Causal Inference.  The MIT Press, Cambridge.
  * Plecko and Bareinboim (2022) Plecko, D., Bareinboim, E., 2022\.  Causal fairness analysis.  Foundations and Trends in Machine Learning XX, XX.  doi:10.1561/XXXXXXXXXX. to appear - technical report available at https://causalai.net/r90.pdf.
  * Romei and Ruggieri (2014) Romei, A., Ruggieri, S., 2014\.  A multidisciplinary survey on discrimination analysis.  The Knowledge Engineering Review 29, 582–638.  doi:10.1017/S0269888913000039.
  * Roy et al. (2023) Roy, A., Horstmann, J., Ntoutsi, E., 2023.  Multi-dimensional discrimination in law and machine learning - a comparative overview, in: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 89–100.  doi:10.1145/3593013.3593979.
  * Ruggieri et al. (2023) Ruggieri, S., Alvarez, J.M., Pugnana, A., State, L., Turini, F., 2023.  Can We Trust Fair-AI?  Proceedings of the AAAI Conference on Artificial Intelligence 37, 15421–15430.  doi:10.1609/aaai.v37i13.26798.
  * Saxena et al. (2023) Saxena, N.A., Zhang, W., Shahabi, C., 2023.  Missed Opportunities in Fair AI.  pp. 961–964.  doi:10.1137/1.9781611977653.ch110.
  * Selbst et al. (2019) Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J., 2019.  Fairness and abstraction in sociotechnical systems, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, New York, NY, USA. p. 59–68.  doi:10.1145/3287560.3287598.
  * Smith et al. (2023) Smith, J.J., Beattie, L., Cramer, H., 2023.  Scoping fairness objectives and identifying fairness metrics for recommender systems: The practitioners’ perspective, in: Proceedings of the ACM Web Conference 2023, Association for Computing Machinery, New York, NY, USA. p. 3648–3659.  doi:10.1145/3543507.3583204.
  * Starke et al. (2022) Starke, C., Baleis, J., Keller, B., Marcinkowski, F., 2022\.  Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature.  Big Data & Society 9, 20539517221115189\.  doi:10.1177/20539517221115189.
  * Supreme Court of the United States (2015) Supreme Court of the United States, 2015.  Tex. Dep’t of Hous. & Cmty. Affairs v. Inclusive Cmtys. Project, Inc.  URL: https://casetext.com/case/texas-dept-of-housing-and-community-affairs-v-inclusive-communities-project-inc?. 576 U.S. 519, 135 S. Ct. 2507, 192 L. Ed. 2d 514, 83 U.S.L.W. 4555\.
  * The Council of The European Union (2000a) The Council of The European Union, 2000a.  Council Directive 2000/43/EC of 29 June 2000 implementing the principle of equal treatment between persons irrespective of racial or ethnic origin.  URL: https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:32000L0043:en:HTML.
  * The Council of The European Union (2000b) The Council of The European Union, 2000b.  Council Directive 2000/78/EC of 27 November 2000 establishing a general framework for equal treatment in employment and occupation.  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A32000L0078.
  * The Council of The European Union (2004) The Council of The European Union, 2004.  Council Directive 2004/113/EC of 13 December 2004 implementing the principle of equal treatment between men and women in the access to and supply of goods and services.  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:32004L0113.
  * The European Commission (2021a) The European Commission, 2021a.  Proposal for a Directive of the European Parliament and of the Council on consumer credits.  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0347.
  * The European Commission (2021b) The European Commission, 2021b.  Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain Union legislative acts.  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206.
  * (70) The European Commission, Directorate-General for Communication, .  Non-discrimination.  URL: https://commission.europa.eu/aid-development-cooperation-fundamental-rights/your-rights-eu/know-your-rights/equality/non-discrimination_en. accessed: 2023-08-22.
  * The European Parliament and the Council of The European Union (2006) The European Parliament and the Council of The European Union, 2006.  Directive 2006/54/EC of the European Parliament and of the Council of 5 July 2006 on the implementation of the principle of equal opportunities and equal treatment of men and women in matters of employment and occupation (recast).  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32006L0054.
  * The European Parliament and the Council of The European Union (2008) The European Parliament and the Council of The European Union, 2008.  Directive 2008/48/EC of the European Parliament and of the Council of 23 April 2008 on credit agreements for consumers and repealing Council Directive 87/102/EEC.  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A32008L0048.
  * The European Parliament, the Council and the Commission (2012) The European Parliament, the Council and the Commission, 2012\.  Charter of Fundamental Rights of the European Union.  URL: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A12012P%2FTXT.
  * United States Court of Appeals, Seventh Circuit (1996) United States Court of Appeals, Seventh Circuit, 1996\.  Cathy Carson v. Bethmehem Steel Corp.  URL: https://casetext.com/case/carson-v-bethlehem-steel-corporation?. 82 F.3d 157 (7th Cir. 1996).
  * U.S. Government Publishing Office (1964) U.S. Government Publishing Office, 1964.  Civil Rights Act of 1964.  URL: https://www.govinfo.gov/app/details/COMPS-342. Public Law 88–352; 78 Stat. 241, as Amended Through P.L. 114–95, Enacted December 10, 2015.
  * U.S. Government Publishing Office (1974) U.S. Government Publishing Office, 1974.  Equal Credit Opportunity Act of 1974.  URL: https://www.govinfo.gov/content/pkg/USCODE-2011-title15/html/USCODE-2011-title15-chap41-subchapIV.htm. United States Code, Title 15. Chapter 41, Subchapter IV.
  * U.S. Supreme Court (1977) U.S. Supreme Court, 1977.  Castaneda, Sheriff v. Partida.  URL: https://casetext.com/case/castaneda-v-partida?. 430 U.S. 482, 97 S. Ct. 1272.
  * U.S. Supreme Court (2009) U.S. Supreme Court, 2009.  Frank Ricci et al. v. John DeStefano et al.  URL: https://casetext.com/case/ricci-v-destefano-13?. 557 U.S. 557, 129 S. Ct. 2658, 174 L. Ed. 2d 490, 21 Fla. L. Weekly Supp. 1049.
  * Wachter et al. (2021) Wachter, S., Mittelstadt, B., Russell, C., 2021.  Why fairness cannot be automated: Bridging the gap between eu non-discrimination law and ai.  Computer Law & Security Review 41, 105567.  doi:https://doi.org/10.1016/j.clsr.2021.105567.
  * Weerts et al. (2023) Weerts, H., Dudík, M., Edgar, R., Jalali, A., Lutz, R., Madaio, M., 2023\.  Fairlearn: Assessing and improving fairness of ai systems.  arXiv:2303.16626.
  * Wexler et al. (2020) Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viégas, F., Wilson, J., 2020\.  The what-if tool: Interactive probing of machine learning models.  IEEE Transactions on Visualization and Computer Graphics 26, 56–65.  doi:10.1109/TVCG.2019.2934619.
  * White House Office of Science and Technology Policy (2022) White House Office of Science and Technology Policy , 2022\.  Blueprint for an AI Bill of Rights: Making Automated Systems Work For The American People.  https://www.whitehouse.gov/ostp/ai-bill-of-rights/.