# Debiasing Methods for Fairer Neural Models
in Vision and Language Research: A Survey

Otavio Parraga  ,  Martin D. More  ,  Christian M. Oliveira  ,  Nathan S.
Gavenski  ,  Lucas S. Kupssinskü  ,  Adilson Medronha  ,  Luis V. Moura  ,
Gabriel S. Simões  and  Rodrigo C. Barros  Machine Learning Theory and
Applications (MALTA) Lab, PUCRSBrazil

(2022)

###### Abstract.

Despite being responsible for state-of-the-art results in several computer
vision and natural language processing tasks, neural networks have faced harsh
criticism due to some of their current shortcomings. One of them is that
neural networks are correlation machines prone to model biases within the data
instead of focusing on actual useful causal relationships. This problem is
particularly serious in application domains affected by aspects such as race,
gender, and age. To prevent models from incurring on unfair decision-making,
the AI community has concentrated efforts in correcting algorithmic biases,
giving rise to the research area now widely known as fairness in AI. In this
survey paper, we provide an in-depth overview of the main debiasing methods
for fairness-aware neural networks in the context of vision and language
research. We propose a novel taxonomy to better organize the literature on
debiasing methods for fairness, and we discuss the current challenges, trends,
and important future work directions for the interested researcher and
practitioner.

fairness, neural networks, bias mitigation, computer vision, natural language
processing

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††journal:
CSUR††journalvolume: 00††journalnumber: 0††article: 000††publicationmonth:
0††ccs: Computing methodologies Natural language processing††ccs: Computing
methodologies Computer vision††ccs: Computing methodologies Neural networks

##  1\. Introduction

Deep Learning is a subfield of Machine Learning (ML) that leverages the
capabilities of artificial neural networks to automatically learn from data.
These networks are fully-differentiable computational graphs optimized via
gradient descent to learn representations from raw data (Bengio et al., 2013),
currently being the most efficient and effective data-oriented strategy to
perform several Computer Vision (CV) and Natural Language Processing (NLP)
tasks.

Despite producing exciting results, neural models have faced harsh criticism
due to some of their current shortcomings. One of the main criticisms is that,
since neural networks are correlation-based approaches, models often learn the
influence of confounding factors that are present in the data instead of
causal relationships (Adeli et al., 2019). This problem is exacerbated in
application domains affected by sensitive (or protected) features, such as
demographic information. For instance, race, gender, and age may confound the
training process because the distribution of labels is often skewed, even if
the intrinsic properties of interest are not related to them. Thus, the
presence of confounding factors may result in models that are biased towards
specific distributions, potentially exhibiting a severe drop in performance in
unseen data or prioritizing certain subgroups within a distribution.

Given the widespread usage of neural models, one must consider the impact of
their respective automated decisions. Applications such as product
recommendations and automatic game-playing are considered to be low-stakes
since biased behaviors do not significantly impact underrepresented groups in
society. However, automated decisions in dating, hiring (Bogen and Rieke,
2018; Cohen et al., 2019), and loan management (Srinivasan et al., 2019)
software present considerably higher stakes, given that they may influence and
perpetuate unfair economic and social disparities between groups.

To address and regulate algorithmic usage in high-stakes decision making,
several governmental entities have proposed the creation of stronger laws
requiring more transparency in automated decision-making. For example, the
European Union GDPR (Council of European Union, 2016) states that people
should have the right to obtain an explanation of the decision reached by
automated systems. Indeed, explainability seems to play a crucial role towards
achieving trustworthy AI, i.e., systems that are lawful, technically robust,
bias-resilient, and ethically adherent. Since neural networks are black-box
models that require external tools to extract explanations (Guidotti et al.,
2018), researchers and practitioners often ignore such tools, allowing the
design and optimization of unfair models.

To prevent learning models from perpetuating the biases present in the data
and producing unfair decisions in automated decision-making, the Artificial
Intelligence (AI) community has concentrated efforts in correcting algorithmic
biases, giving rise to the research area now widely known as fairness in AI.
When considering the process of automated decision-making, the term fairness
refers to the “absence of any prejudice or favoritism toward an individual or
a group based on their inherent or acquired characteristics” (Mehrabi et al.,
2021). Fairness in AI is a relatively new research area. Initially, fairness
papers were mostly submitted to workshops focused on data privacy, with
occasional papers appearing in main proceedings. Starting in 2014, we see the
appearance of several workshops and conferences specialized in fairness,
accountability, and transparency in machine learning. Notable examples include
FAT/ML (2014-2018), AIES (2018-present), FAccT (formerly FAT*, 2018-present),
and FATES (2019-present). As the research field matured, we see a noticeable
increase in the number of fairness papers that appeared in the main
proceedings of renowned AI and Machine Learning conferences, such as AAAI,
NeurIPS, ICML, ICLR, and many others. These conferences now also occasionally
host tutorials and workshops dedicated mainly for fairness-related approaches
in data-driven learning, and some conferences, such as NeurIPS, are
experimenting with implementing fairness awareness protocols, such as
including a “Broader Impact” section, covering the ethical aspects of the
algorithms being proposed, and proposing a list of best practices for
responsible machine learning research (NeurIPS, 2021a, b). This historical
summary shows that the research community in general is starting to focus not
only on improving the performance of algorithms, but also in creating fairer
ones.

Given the relevancy and recent proliferation of fairness concerns in automated
decision-making, our goal in this survey is to provide an overview of the
progress and current state of the art in neural approaches for fairness and
bias mitigation in AI. We focus on vision and language research since these
data modalities and their intersection encompass the majority of neural
networks research.

Several surveys of fairness in AI already exist, each covering different
aspects of the research area. For instance, the work of Mehrabi et al. (2021)
focuses on fairness in ML while extensively detailing fairness definitions and
types of biases, but offers a quick analysis of debiasing methods in ML in
general. A similar content and structure can be found in (Caton and Haas,
2020; Dunkelau and Leuschel, 2019), where a broad analysis is performed for
different ML tasks and algorithms. The work of Le Quy et al. (2022) goes in a
different direction by focusing exclusively on analyzing tabular datasets and
their usages for bias mitigation. Tian et al. (2022), in turn, cover fairness
exclusively for image data. Finally, there are also several survey papers that
extensively cover bias and fairness solely within the scope of NLP (Meade et
al., 2021; Delobelle et al., 2021; Garrido-Muñoz et al., 2021; Blodgett et
al., 2020; Bansal, 2022), be it only regarding pre-trained language models
(LMs) (Meade et al., 2021; Delobelle et al., 2021), or exclusively for deep
learning (Garrido-Muñoz et al., 2021).

In contrast with the aforementioned studies, this work focuses on an in-depth
analysis of neural-based methods for debiasing in the context of the main
unstructured data types, namely visual and textual data and their intersection
(i.e., multimodal tasks). By focusing on debiasing approaches, we offer a new
taxonomy for properly categorizing those methods while following the detailed
definitions of fairness and bias proposed by Mehrabi et al. (2021), which are
now well-accepted and widely used by the research community.

This work surveys $95$ debiasing methods exclusively in the context of neural
networks for vision and language. The only work to survey a similar amount of
methods is the one by Caton and Haas (2020), which reviews a total of $86$
debiasing methods, though across all ML research. Still, there is only one
paper in the intersection of this work and (Caton and Haas, 2020), which is
the work from Edwards and Storkey (2016). The remaining survey papers on
fairness review a much smaller set of methods for debiasing. Dunkelau and
Leuschel (2019) survey $24$ debiasing methods, and once again its intersection
with our work is only a single paper, (Edwards and Storkey, 2016). Finally,
the outstanding survey by Mehrabi et al. (2021) only presents $13$ debiasing
methods, none of them falling under our criterion for acceptance, namely being
a method for making neural models fairer in the context of vision and language
research.

##  2\. Scope and Organization

In this section, we detail the scope and organization of this paper. First, we
contextualize fairness and its relationship with bias within neural network
research in Section 3. Bias is an overused word in ML research and has several
different meanings according to the context where it is used. Learning
requires methods to incorporate inductive biases, which are preferences
towards choosing/modeling certain solutions instead of others. In the context
of neural networks, biases are also attribute-free parameters that shift the
model according to prior information. The correlation-based approach
implemented in most ML methods, neural networks being no exception, may
capture relationships that lead to biased solutions, i.e., a model whose
behavior may be undesired because of spurious correlations that were captured
during training. In this paper, when we talk about biases we are not talking
about the attribute-free parameters in neural networks. We are also rarely
talking about inductive bias, since algorithmic biases are seldom the cause of
fairness problems, though that may also happen.

Hence, most of the time we mention the word bias we mean unintended behavior
resulting from correlation-based processing that ignores further context not
explicit in the data. In practice, we assume that exploiting correlations is
the own nature of ML algorithms, and it is the combination of biased samples
and correlation identification that generates the problem of models that are
biased towards undesired behavior.

Fairness is also a term with many proposed definitions. We follow the
specialized literature and situate (un)fairness as a direct consequence of
capturing spurious correlations during training, as long as those correlations
result in the “favoritism toward an individual or a group based on their
inherent or acquired characteristics” (Mehrabi et al., 2021). By situating
problems with fairness as a direct consequence of existing correlation-based
biases, we show that most bias-mitigating strategies can be used to improve
the fairness of automatic systems, and therefore we establish the link between
the areas of bias mitigation and fairness awareness in Section 3.

Next, we comment on the main metrics and evaluation measures for assessing the
level of fairness of a neural model in the context of those applications in
Section 4. We review both application-specific and general-purpose measures,
pointing to their proper use, applicability, and known limitations. The core
of this survey, however, is the critical analysis and discussion of several
debiasing methods for neural models in the context of image and language
processing, which we present in Section 5. Computer vision and natural
language processing are arguably the two most important research areas in AI
nowadays, given the amount of content produced that falls into these
categories and their intersection. For instance, in 2021 approximately $1.4$
trillion digital photographs were taken
worldwide111https://blog.mylio.com/how-many-photos-taken-in-2022/ while
Twitter users posted approximately $200$ billion
tweets222https://www.internetlivestats.com/twitter-statistics/.

Finally, we list the current fairness challenges in neural models,
highlighting trends and important future research directions in Section 6. We
also specifically envision the challenges of fairness awareness in the so-
called Foundation Models (Bommasani et al., 2021), which are extremely-large
neural networks trained over massive amounts of data. The challenge of
addressing fairness unawareness in models that were trained self-supervisedly
is probably one of the main factors that prevent such models from being fully
open-sourced, given that this category of neural models has the potential of
being applied and adapted to several tasks and biases may be perpetuated or
even potentialized if not considered and properly addressed, resulting in
unfair decision-making.

##  3\. Bias and Fairness

The word bias is used in ML in many distinct contexts. Mitchell (1980) defines
bias as any basis for choosing one generalization over another, other than
strict consistency with the instances. This definition encompasses biases that
are inherent to the learning task and are unavoidable, the so-called inductive
biases. In convolutional neural networks, for instance, the implemented
inductive bias explores the fact that the input data often has spatial
coherence and that higher level features are translation and rotation
invariant. Another common use of the word bias is to refer to the parameters
of the neural network that are free w.r.t. the input features. In this survey,
we will focus on biases that are not mandatory in the learning process and
that can often skew the results of the model in undesired ways, with the
possibility of causing social harm.

Since artificial neural networks are essentially data-driven methods, data is
the main source of unwanted and potentially avoidable bias. In an ideal
scenario, data should be complete, correct, and representative of a population
of interest. However, it is often the case that the sampling scheme, the
measurement, or the representation of the data is biased towards a group of
subjects. Olteanu et al. (2019) call this scenario data bias.

It is also common to use datasets that follow a given distribution and expect
them to generalize to another (perhaps slightly distinct) one. A facial
attribute recognition system could be trained using the CelebA dataset (Liu et
al., 2015), but when the model is deployed to recognize facial features in the
wild we can expect a performance drop given that the target population is not
a group of celebrities. This type of bias arises with the distinction of
distributions from training and production settings. Even assuming that the
dataset is representative of the population that we are interested in, we are
still prone to create biased models. Take the scenario of NLP systems that are
trained in very large corpora of text. The word embeddings created by these
systems incorporate stereotypes that are found in the text, such as
associating the word engineering more frequently with men than women (Suresh
and Guttag, 2019).

Many studies searched for ways to categorize bias following different
taxonomies, each based on a different system and set of assumptions. The work
of Mehrabi et al. (2021) divides biases based on the relationship among three
key entities that constantly interact with each other: user, algorithm, and
data. Each entity will aggregate a group of possible biases that may occur in
interactions within the ML life cycle, which may affect another entity in the
process. The relationship between data and learning algorithm clearly allows
for the identification of outcome disparities and biases. Historical, social,
cultural, and economic factors, as well as cognitive human biases, may affect
all the process of collecting data, biasing samples and entire datasets
without being easily detected. The relationship between algorithm and user can
also be a source for potential biases to arise, specially when the algorithm
is responsible for guiding the human behaviour to specific patterns (Du et
al., 2021a).

In the study of Suresh and Guttag (2019), the types of biases are defined
according to the ML life cycle. It is established that unintended biases can
be inserted within the data (be it a problem of data generation,
representation, or measurement) or by model building and implementation (due
to the learning process, evaluation procedure, aggregation of models, or the
deployment in an environment where the concepts that were modelled do not
apply). Although this definition is more straightforward than the one in
(Mehrabi et al., 2021), it lacks the capability of differentiating that data
biases such as content production and aggregation are created by user-data
interaction and data-algorithm interaction, respectively.

We can also view the bias phenomena as an origin and consequence framework
(Shah et al., 2020). More focused on NLP applications, Shah et al. (2020)
define four origins for biases in source data: over-amplification, semantic,
selection, and label; and two consequences in the outcomes: outcome disparity
and error disparity. By using a standard supervised pipeline of NLP, the
authors attribute each bias origin to a different step, from the embedding
phase where the semantics are condensed into a dense vectorized representation
to the possibility of over-amplification due to the combination of the
learning algorithm and the data itself or the under-representative selection
of instances. The consequences are defined by two disparities obtained when
analyzing the output distributions.

Figure 1. Network of terms found in the abstracts of the surveyed papers. The
term bias and its relationship with other terms are highlighted. Created with
VOSViewer (Perianes-Rodriguez et al., 2016).

Mitigating bias in neural models is not a trivial task. Despite the discussion
regarding adopted taxonomies and a strict definition of bias (Shen et al.,
2022; Olteanu et al., 2019), there is a consensus that biased models can cause
societal harm and that researchers and practitioners must be vigilant and
employ tools to detect and mitigate this problem. A point should be made that,
although humans have an unmatched capacity to learn from data and to
generalize to different contexts, we too are prone to biases and to being
unfair. Nonetheless, the problem is exacerbated by neural models since they
can work on a larger scale and reinforce negative feedback loops in society
(Kusner et al., 2017).

When biases in the learned models are detrimental to specific groups, often
defined by sensitive attributes, we have models that are unfair. The question
of fairness is specially relevant when automated decisions exacerbate
prejudicial behavior against socially-vulnerable groups or when it promotes
favoritism towards a specific demography, gender, ethnicity, or race (Du et
al., 2021a). Model fairness leverages moral questions about how we collect,
validate, and use data, especially now with models getting larger and datasets
reaching massive sizes (Brown et al., 2020). We cannot disregard group
representation nor oversee undesired outcomes in trained models anymore.

When researchers and practitioners focus solely on traditional performance
metrics such as accuracy, a trained model can propagate stereotypes and biases
present in the training data. While general fairness is a well-defined
concept, the definition of what is the best measure to capture fairness is
still subject of debate (Caton and Haas, 2020). One should be careful since
distinct definitions of fairness derived from the use of different metrics
have mutual interactions (Kleinberg et al., 2018), e.g., when optimizing for
group fairness it is possible to make the situation worse for individual
fairness. These scenarios are presented with greater detail in Section 4.

We illustrate in Figure 1 the landscape of fairness research in neural
networks for vision and language with a network of terms found in the
abstracts of all surveyed papers. In such visualization, the node area is
proportional to the number of occurrences of the term within the abstracts.
Note that bias and data are the most used terms, considering that the main
source of bias in ML applications is the data and their interaction with users
and algorithms. The modalities that we are focusing on, nodes cv and nlp, as
well as their intersection are also commonly present in the metadata or
through proxy terms such as language, image and embedding. We can also see
that the term bias appears in association with attributes such as gender, and
race, which are indeed often associated with subgroups or individuals that are
harmed by unfair decisions. Other terms that relate to sensitive attributes
are demographic, ethnicity, and social bias.

Fairness is not achieved solely by guaranteeing the same outcome for distinct
groups. Mehrabi et al. (2021) present the case where a seemingly unfair
difference in annual income between men and women in the UCI Adult dataset can
be explained in terms of working hours. In that dataset, men on average work
more hours per week than women. Working hours is thus a legitimate attribute
that explains the difference in annual income and should not be considered an
issue in the context of fairness.

To promote fairness, a special subset of available features are regarded as
sensitive or protected. They help define subgroups that are considered
underprivileged in society. An example of protected attribute is gender, and
there are several examples of ML systems presenting error rates skewed towards
a particular gender category (Buolamwini and Gebru, 2018; Guo and Caliskan,
2021; Kirk et al., 2021). The definition of the the subset of attributes that
are sensitive is not an entirely technical issue. Although there is certain
agreement that ethnicity should be protected in some applications, it could be
the case that other attributes such as zip code serve as a proxy for ethnicity
and should also be protected. Furthermore, despite being a protected
attribute, ethnicity matters in certain applications, e.g., medical diagnosis
where genetic predisposition is a determining factor. In practice, any
attribute that society perceives as a potential defining factor for
aggregating underprivileged people is a potential sensitive attribute.

Fairness can also be stratified regarding groups, subgroups, or individuals.
Fairness in groups can be defined through the Demographic Parity concept,
which states that the likelihood of an outcome should be the same for a person
in or out of the protected group. On the other hand, there is also the
Counterfactual Fairness definition, which states that inferences over a single
individual should be kept unaltered in a hypothetical counterfactual world
where the same individual belongs to a distinct group. Note that satisfying
every definition of fairness at the same time is not feasible in real-world
applications (Lambrecht and Tucker, 2019), and we exemplify that in Section 4.

Issues of model fairness are often intertwined with biases in the ML pipeline.
Datasets such as IJB-A (Klare et al., 2012b) and Adience (Levi and Hassncer,
2015) do not have proper representation of gender and skin color (Buolamwini
and Gebru, 2018). Models trained in biased datasets will often produce worse
results for individuals in underrepresented groups. They may, for instance,
misclassify Black women with a higher probability than lighter-skinned men. In
that situation, the unfairness of the model can be traced back to its origin
in a biased dataset.

When the underprivileged group is defined by a combination of sensitive
attributes, and the dynamics of individuals in this group is considerably
distinct than when considering one sensitive attribute at a time, we have
intersectional biases (Buolamwini and Gebru, 2018). Unfairness caused by
intersectional biases are harder to detect and to prevent because of intricate
relationships among the attributes (Guo and Caliskan, 2021; Kirk et al.,
2021).

Some applications of ML models are considered to be of low stakes, having a
limited impact in society well-fare and errors are considered cheap. On the
other hand, ML models are also being used in significant higher-stake
applications, where a single error in inference is expensive and can
negatively impact individuals and groups. Examples of these applications are
facial recognition systems (Buolamwini and Gebru, 2018), criminal assessment
(Julia Angwin and Kirchner, 2016), and occupation classification (De-Arteaga
et al., 2019). Fairness is of the upmost importance in high-stake
applications. ML models should be subject to scrutiny as they are part of an
entire ecosystem that influences different social groups, and can perpetuate
harmful concepts and stereotypes when left unchecked.

When optimizing for model fairness, it is necessary to keep in mind that the
level of data aggregation can also be a source of confusion. This type of
problem is defined as Simpson’s Paradox (Blyth, 1972), which is the phenomenon
in which the same dataset can lead to opposite trends by changing how the data
is aggregated, always happening when the aggregated data hide conditional
variables. Regarding neural networks, most features are learned by searching
for correlations in data when trying to optimize an objective function, so the
undesired lack of fairness may be a product of seeking good performance,
creating a kind of performance-accuracy trade-off.

Most research in fairness cover the ML field without focusing on deep learning
(Mehrabi et al., 2021; Suresh and Guttag, 2019; Caton and Haas, 2020). As a
consequence, lack of fairness in multimodal tasks (e.g., text-to-image
systems) are less understood than in more traditional classification tasks.
Another specificity in neural-network research is the recent rise of
Foundational Models (FMs) (Bommasani et al., 2021). Biases in those models are
challenging to identify, and the own nature of FMs as unfinished models that
need to be further adapted makes them a perfect fit for the problem of bias
propagation and exacerbation within many downstream tasks. We further discuss
this challenge in Section 6.

As neural models are being increasingly deployed to real-world applications,
the interest on fairness grows both in academia and industry. While there are
countries that are starting to regulate some aspects of fairness in ML models
and automated decision-making (Bird et al., 2019), there is much to be done to
achieve a unified framework that the research community can agree on and that
practitioners can apply when developing fairer applications. Among the
requirements for the fairness research to provide practical impact in society,
there should be effective methods that allow biases to be mitigated and
underprivileged groups to be protected without a significant compromise in
terms of efficiency and effectiveness. We categorize and review those methods
in Section 5, while also discussing their advantages and shortcomings.

##  4\. Metrics

This section describes the most relevant metrics to measure bias and fairness
of neural models in CV and NLP, though all metrics presented in Section 4.1
apply to any type of classification problem.

###  4.1. Metrics for Classification

Classification is a traditional problem in machine learning and one of the
most common tasks in computer vision. During the training process, given a
dataset $D=\\{(x_{i},y_{i})\\}_{i=1}^{N}$ the algorithm will learn how to
correlate a set of inputs $X$ to a set of corresponding target variables $Y$.
The most common metric to evaluate classification models is accuracy, the rate
of correctly-classified objects. Consider the confusion matrix of a binary
classification problem, comprised of True Positives (Negatives) as the amount
of objects correctly classified from the positive (negative) class, and False
Positives (Negatives) as the amount of misclassified objects from the positive
(negative) class. With these same terms, we can derive three other widely-used
metrics: precision, recall, and F1-Score.

Even though those metrics are well-establish for evaluating the general
performance of a learning model, they are not well-suited for fairness
evaluation. For example, consider a model trained on a dataset containing two
groups, $A$ and $B$, where $A$ is privileged in society while $B$ is
marginalized. Societal biases have affected model performance, leading it to
perform better on $A$ than on $B$. Increasing accuracy using optimization
techniques does not necessarily improve fairness. It is possible the model
gets even better at classifying $A$ and slightly worse on $B$, leading to an
even unfairer model. Similarly, precision, recall, and F1-score alone cannot
guarantee model fairness, nor properly evaluate its nuances. With this in
mind, fairness-specific metrics have been proposed for classification models.
These metrics have been mainly divided into two categories: group fairness,
which requires the average output of different demographic groups to be
similar (Dwork and Ilvento, 2018); and individual fairness, which requires
that individuals who are similar in respect to a specific task have similar
probability distributions on classification outcomes, regardless of the
demographic group they are part of (Dwork and Ilvento, 2018; Gajane and
Pechenizkiy, 2017).

####  4.1.1. Group Fairness

In group fairness, examples are grouped according to a particular protected
attribute, and statistics about model predictions are calculated for each
group and compared across them (Du et al., 2021b). Next, we describe the most
used group fairness metrics.

Demographic/Statistical Parity states that the average algorithmic decision
should be similar across different groups: $p(\hat{y}=c_{k}\mid z=p)\sim
p(\hat{y}=c_{k}\mid z=u)$, where $\hat{y}=c_{k}$ is a predicted class and $z$
refers to a protected attribute such as race, gender, and age, in which $p$
and $u$ indicate privileged and underprivileged groups, respectively. This
metric does not depend on ground-truth labels, so it is especially useful when
this information is unavailable. When we are aware that historical biases may
have skewed the data, demographic parity may be used in conjunction with a
bias mitigation strategy to verify whether the model has learned those biases.

Equality of Opportunity states that the true positive rate for individuals who
qualify for the desirable/positive outcome ($y=1$) should be similar across
different groups: $p(\hat{y}=1\mid z=p,y=1)\sim p(\hat{y}=1\mid z=u,y=1)$
where $y$ is the ground truth label. This metric considers that different
groups may have different distributions, which leads to one of its main
criticisms: it does not consider the effect of discrimination given the
protected attributes.

Equality of Odds states that both the true positive rate for individuals who
qualify for the desirable/positive outcome and the false positive rate for
individuals who do not qualify for the desirable/positive outcome should be
similar across different groups. The true positive rate is computed as the
equality of opportunity, while the false positive rate is computed as
$p(\hat{y}=1\mid z=p,y=0)\sim p(\hat{y}=1\mid z=u,y=0)$. Equality of odds is a
more restrictive variation of equality of opportunity, but the same criticisms
of the former still hold for this metric.

Conditional Statistical Parity states that different groups should have
similar probability of being assigned to a positive outcome given that the
individuals satisfy a set of legitimate factors $L$: $p(\hat{y}=1\mid
L=1,z=p)\sim p(\hat{y}=1\mid L=1,z=u)$. The set of factors $L$ is task-
dependent and should be modeled accordingly.

Treatment Equality states that the ratio of false negatives and false
positives should be similar across different groups: $(FN/FP\mid
y=1)\sim(FN/FP\mid y=0)$. This metric may be used as a policy lever while
optimizing for other metrics. If a specific group has a higher ratio of false
negatives in order to satisfy another metric, then it is actually treating
these groups differently.

Overall Accuracy Equality states that accuracy should be similar for different
groups: $((TP+TN)/(TP+FP+TN+FN)\mid z=p)\sim((TP+TN)/(TP+FP+TN+FN)\mid z=u)$.
This metric thus implies that true negatives are as desirable as true
positives, being quite uncommonly-used for that reason.

Predictive Parity is similar to Overall Accuracy Equality, except that it
considers that different groups should have similar precision instead of
accuracy: $((TP)/(TP+FP)\mid z=p)\sim((TP)/(TP+FP)\mid z=u)$. Mathematically,
a model with equal precision for different groups will also have equal false
discovery rate: $((FP)/(TP+FP)\mid z=p)\sim((FP)/(TP+FP)\mid z=u)$.

Right for the Right Reasons: Many classification errors occur due to the model
looking at the wrong evidence. It can happen based on any contextual evidence,
e.g., a multimodal model ignoring the input image when asked “What color is
the banana?” since the most likely answer is “yellow”. To compute this
measurement quantitatively for image-based models, Hendricks et al. (2018)
rely on two visual explanation techniques: Grad-CAM (Selvaraju et al., 2016)
and saliency maps generated by occluding image regions in a sliding window
fashion. In their work, they use the MS-COCO ground-truth person segmentation
to evaluate whether the Grad-CAM from the model overlaps with the correct
areas of the image when classifying a given class. As this metric evaluates
how the model gets its prediction, it is aligned with the current demand for
more explainable AI.

####  4.1.2. Individual Fairness

Group fairness mainly ignores all information of the objects except for
protected attribute $z$. That strategy might hide unfairness, and we exemplify
a scenario in which that happens as follows. Suppose a model assigns a
positive score to the same fraction of male and female applicants. However,
assume male applicants were chosen randomly whereas female applicants were
chosen based on reference attribute values. Demographic parity will state that
the model is actually fair, despite the discrepancy on how the applications
were processed based on gender. The notion of individual fairness arises to
deal with such a discrepancy.

Fairness through Awareness states that similar individuals should be treated
similarly, where subject similarity is a task-specific metric. Consider two
similar individuals, where the only major difference between them is that one
is male and the other female. Since they are similar, individual fairness
states that they must follow the same distribution, generating the same
output. This strategy has already proven to be more efficient than Fairness
through Unawareness, which relied on the assumption that an algorithm would be
fair if none of the sensitive attributes were explicitly used in the
predictive process. Unfortunately, building a similarity score considering the
different types of attributes is the biggest obstacle for putting this metric
into practice.

Counterfactual fairness states that an individual and its counterfactual copy
whose only difference is the protected attribute value should have the same
outcome. While fairness through awareness finds similar individuals through
task-specific measures, this metric generates synthetic copies of a
counterpart instance (e.g., female if the original instance is male in a
gender-fair evaluation). It also takes into account that several features
might be dependant on the protected attribute and, therefore, should also be
altered accordingly.

####  4.1.3. Critical Analysis of the Classification Metrics

Suppose we want to train a model to admit students to a given University with
$30$ openings, and we have two groups of people: $A$, which comprises $70$
students, and $B$, with $30$. In addition, assume $A$ is privileged in society
whereas $B$ is part of a marginalized group. Historical biases led students of
group $B$ to have, on average, lower grades than group $A$, even though both
groups contain students with variable performance in terms of grades. Finally,
assume $40$ students from group $A$ and $12$ from group $B$ reach the
desirable grade level to be admitted to this University. We discuss two
different learning models and how the proposed metrics measure what is
happening. This discussion applies not only to neural networks but to any
learning model and classification task in machine learning.

Approach #1. Assume a model was trained to achieve similar acceptance rates
for both groups. It selects $21$ students from group $A$ and $9$ from group
$B$. This model satisfies the Demographic Parity criterion, since the
percentage of accepted students was the same for both groups: $30\%$. However,
if the selection were made randomly in either $A$ or $B$ instead of
considering the grades, this metric would still evaluate this model as fair.
On the other hand, this model does not satisfy the Equality of Opportunity
criterion: even if the selection process were made entirely based on each
student’s grades, the rate of acceptance considering only the students with
desirable grades was higher for group $B$ ($75\%$) than for group A ($53\%$).

Approach #2. Assume a second model was trained to achieve a similar acceptance
rate in both groups, but this time only considering the students with
desirable grades. As a result, $23$ and $7$ students were accepted from groups
$A$ and $B$, respectively. In this scenario, Equality of Opportunity is
satisfied ($58\%$ for both groups), while Demographic Parity is not ($33\%$
for group $A$ and $23\%$ for group $B$). The problem here is that the
historical biases that led to the disproportion regarding the grades for each
group are not taken into account. Therefore, this strategy alone will
perpetuate historical biases that are currently present in society.

It is easy to see that one cannot satisfy all fairness metrics without a
perfectly-balanced model that is fully aware of all possible sources of bias
and prejudices. It is reasonable to assume such a thing will not be achievable
any time soon, mainly because, by definition, machine learning deals with ill-
posed problems and incomplete data. Counterfactual Fairness carries excellent
insight on how we can evaluate fairness while also considering historical
biases. However, this metric requires a deeper analysis of the data in order
to find the dependencies of features regarding both the sensitive attribute(s)
and the outcomes.

###  4.2. Metrics for Image Generation

Image generation is the task of creating images given an input, be it a user-
guided signal (including another image) or simply random noise. Recent
advances in the field have made it possible to create an image of almost
anything given a simple description (Ramesh et al., 2022). Generative models
are trained on colossal amounts of unbalanced data, which carry historical and
societal biases. It is possible to measure whether a generative model is
creating a similar number of samples for different groups by using a
classifier on a large set of generated samples, verifying which group they
belong to and then comparing these proportions. Choi et al. (2020) used this
strategy with a binary gender classifier to evaluate their face generation
model trained on the CelebA dataset, which has a higher proportion of females.
Cho et al. (2022) made a more comprehensive evaluation concerning gender and
race using CLIP (Radford et al., 2021) and human annotation. That analysis
used images generated through professional and political prompts to check
whether the model correlated the protected attributes with specific roles.
Overall, this approach requires a classifier with good performance in the
respective groups so manual labeling is not necessary.

###  4.3. Metrics for Language Modeling

Language modeling (LM) is a base task for most NLP models. By predicting the
next token in a sentence, neural networks can learn to manipulate language,
discovering multiple meanings of words that vary contextually. A common
approach is to train an LM and later adapt it to a downstream task, following
the idea of first teaching the model about the nature of the language and then
fine-tuning its knowledge to a specific task. Instead of training a new model,
it is possible to use only the generated embeddings (word or context-level)
that encapsulate the semantics in a dense vector and then employ it in a
different processing pipeline.

Many metrics use the LM output to estimate biases via the probabilities.
Another alternative is to use the dense vectorized representation to measure
bias, which can be used both with contextual and word embeddings. Besides
those alternatives, NLP tasks can be used to measure bias when using specific
evaluation datasets. Metrics tailor-made for LMs are described next.

Direct Bias (DB) (Bolukbasi et al., 2016b) is a specific gender-bias metric
which defines bias as a projection onto a gender subspace. To measure DB, we
first need to identify $N$ words that are gender-neutral. Given those words
and the gender direction learned $g$, we define the DB of a model as:

(1) $DB=\frac{1}{|N|}\sum_{w\in N}|\cos(\vec{w},g)|^{c}$

where $\vec{w}$ is the embedding vector of $w$ and $c$ is a user-defined
parameter that determines how strict DB will be. With $c=0$, $DB=0$ only when
there is no overlap of $\vec{w}$ and $g$. With $c=1$ we have a more gradual
bias measurement albeit with a small error margin.

Word Embedding Association Test (WEAT) (Caliskan et al., 2017) measures bias
through the permutation of two sets of target words $X$ and $Y$ (e.g., male-
dominated professions like engineers and female-dominated professions like
nurses) and two sets of attribute words $A$ and $B$ (e.g., {man, male,…} and
{woman, female,…}). In a scenario without biases, there would be no difference
between the relative similarity of the two sets of target words and the two
sets of attribute words. WEAT measures this difference in similarity to
determine whether a given word is biased.

Co-occurrence Metrics (Bordia and Bowman, 2019) uses word co-occurrence for
measuring bias in generated text. In a model-generated corpora, we can analyze
the number of times that each word appear next to specific terms, i.e., how
often a language model will connect professions, sentiments and areas of
knowledge with protected groups.

Sentence Embedding Association Test (SEAT) (May et al., 2019) comes as the
natural adaptation of WEAT (Caliskan et al., 2017) but for contextualized word
embeddings. Since WEAT only tested associations among word embeddings, it lost
utility for recent models that are contextual-based (e.g., based on
Transformers). The authors adapt it to make use of sentence templates, e.g.,
[He/She] is a [MASK]. The models then generate contextual embeddings with
these templates and the cosine similarity is computed between two sets of
attributes. Several other WEAT and SEAT variations have been proposed since,
though with a similar usage principle.

Discovery of correlations (DisCo) (Webster et al., 2020) uses a template with
two slots, e.g., X likes to [MASK], where X is a word based on a specific set
of words planned to trigger possible biases, and the [MASK] token is replaced
with the model prediction. It compares the predictions for the words used in
the template to compute biases. The final result is the average of different
predictions for the sets of words.

Log probability bias score (Kurita et al., 2019) uses the prior probability of
the model when predicting a specific [MASK] token to normalize the resulting
probability of a word that appears in a given context. By doing so one can
surpass limitations from metrics based on pre-trained models that do not
consider the prior probability of a given word when generating the
recommendation. A limitation of this method is that it only works for models
trained with masked language modeling, where the priors can be extracted by
masking more than one token in the sentence, e.g., The [MASK] is a [MASK], and
analyzing them individually.

Context association test (CAT) (Nadeem et al., 2021) is a metric proposed in
conjunction with the StereoSet dataset. That dataset comprises sentences to be
completed (model has to fill a blank token or select a continuation for the
sentence). The completion option, for all cases, contains stereotype, anti-
stereotype, and meaningless options. The objective of the evaluation is to
measure how many times a model would choose a meaningful sentence over a
meaningless one and how many times it would choose a stereotyped option
instead of an anti-stereotype.

CrowS-Pairs (Nangia et al., 2020) is a pseudo-log-likelihood metric based on
the perplexity measure of all tokens conditioned on the stereotypical tokens.
Similar to the previous one, this metric also comes with a dataset. The
templates for its usage follow a similar approach to StereoSet, with
stereotyped and anti-stereotyped versions.

All Unmasked Likelihood (AUL) (Kaneko and Bollegala, 2021c) is an extension of
CrowS-Pairs. It leverages not only the masked tokens in the sentence to
measure biases but also all unmasked tokens and multiple correct predictions.
Kaneko and Bollegala (2021c) also proposes AULA, a variation that takes into
account the attention weights.

####  4.3.1. Critical Analysis of the Language Modeling Metrics

LMs are assets of high relevance in NLP, especially after the adoption of pre-
trained LMs as the basis for adaptation in downstream tasks. This reuse
approach highlights the need of a consistent and robust fairness evaluation
due to its high-spread potential. Metrics that rely on model probabilities are
easier to interpret, especially when compared with the embedding-based
metrics. The latter are divided into two categories: word embeddings and
contextual embeddings, which share quite a few similarities. However, the
distinctions become stronger as we look into metrics such as WEAT and SEAT,
where we need to create specific scenarios to measure biases, generating a
considerable level of human interference when evaluating the final embeddings.
The metrics presented here use either probabilities or embeddings and can be
called intrinsic metrics. However, they are not the only way to evaluate LMs.
We can evaluate fairness through downstream tasks using task-specific metrics.
Neural machine translation, coreference resolution, and language generation
are examples where we can use specific datasets and their respective
evaluation protocols to analyze fairness levels. Given the number of toxic
terms in a text generated by a model, the quality of translations or the
correlation of terms with protected attributes in complex and sensitive
contexts may give a clearer picture of real-world problems that may occur.

###  4.4. Task-Agnostic Metrics

The following metrics do not rely on any specific task, i.e., they are of
general purpose and can be easily computed when evaluating trained neural
networks.

Bias Amplification (Zhao et al., 2017) evaluates how much bias the model
amplifies in a given dataset. For that, it measures the bias score
$b(\hat{y}=c_{k},z)$ as:

(2) $b(\hat{y}=c_{k},z)=\frac{c(\hat{y}=c_{k},z)}{\sum_{z^{\prime}\in
Z}c\left(\hat{y}=c_{k},z^{\prime}\right)},$

where $c(\hat{y}=c_{k},z)$ is the amount of times the model outputs
$\hat{y}=c_{k}$ taking into account protected attribute $z$, and $Z$ is the
set of protected attributes.

The premise is that the evaluation set is identically distributed to the
training set, and therefore if $\hat{y}=c_{k}$ positively correlates with $z$,
and if $\tilde{b}(\hat{y}=c_{k},z)$ (evaluation set) is larger than
$b^{*}(\hat{y}=c_{k},z)$ (training set), one can assume that the model has
amplified that specific bias.

Assume we are measuring biases in a VQA application, and that the bias scores
measured in a specific model are
$b^{*}(\hat{y}=\text{cooking},z=\text{woman})=.66$ and
$\tilde{b}(\hat{y}=\text{cooking},z=\text{woman})=.84$, respectively, when
asked What is the person doing?. In that case, one can assume that the model
amplified the bias of cooking toward woman. The authors do this for each class
in order to obtain the mean bias amplification, defined as:

(3) $\frac{1}{|K|}\sum_{z}\sum_{k\in\left\\{k\in
K\left|b^{*}(\hat{y}=c_{k},z)>1/\right|\mid
Z\|\right\\}}\tilde{b}(\hat{y}=c_{k},z)-b^{*}(\hat{y}=c_{k},z),$

However, the premise that the bias distribution will be the same from training
to evaluation/test set might not hold and could misrepresent the bias-
resilient capability of the model. We observed only one other work (Hirota et
al., 2022) that makes use of this metric outside the scope of the VQA(-CP)
datasets.

KL-Divergence The Kullback-Leibler divergence score quantifies how much a
given probability distribution differs from another. The KL-divergence between
distributions $P$ and $Q$, $KL(P\parallel Q)$ is calculated as:

(4) $KL(P\parallel
Q)=\sum_{i=1}^{N}P\left(x_{i}\right)\cdot\log\frac{P\left(x_{i}\right)}{Q\left(x_{i}\right)}.$

The divergence is an optimal metric to measure how different protected
attributes diverge in a task where the feature should not correlate to the
problem, e.g., a curriculum vitae system should not take gender into
consideration (Pena et al., 2020). Hence, if the distribution between genders
significantly diverges, we can conclude that the model is unfair towards a
gender. Nevertheless, KL-Divergence is not a distance metric between two
distributions since it is not symmetric. Therefore, $KL(P\parallel Q)\neq
KL(Q\parallel P)$, requiring coupling KL-Divergence with another metric for
bias mitigation measurement.

##  5\. Debiasing Methods for Fairness-Aware Neural Networks

This section organizes and discusses the existing literature for debiasing
neural-network models in vision and language research. While the existing
literature reviews divide the debiasing approaches into three categories,
namely pre-processing, in-processing, and post-processing (Mehrabi et al.,
2021; Caton and Haas, 2020), we understand that these categories are
insufficient to organize all existing methods reviewed in this survey in a
precise fashion. Therefore, we propose a new taxonomy that properly
categorizes all debiasing methods, and we present it in Figure 2.

Figure 2. Taxonomy for organizing the literature on debiasing methods for
neural networks in vision and language research.

We call distributional all strategies that modify the dataset distribution
prior to training. That includes sampling strategies that increase the amount
of data examples artificially. In addition, we divide methods that focus on
optimization via training into two categories:

  1. (i)

One-Step-Training, which includes fair models that are generated for a
particular task via a single optimization procedure;

  2. (ii)

Two-Step-Training, where a new training phase has to be performed to fix an
existing biased model, i.e., making it fairer.

Finally, we call inferential those strategies that address the problem of
fairness based on the model outputs, i.e., that discover and remove social
biases without requiring further weight optimization or dataset manipulation.

Our taxonomy is more precise than previously-proposed ones in that it better
differentiates methods that remove biases during the training process of the
downstream task (one-step-training) from those that optimize a pre-trained
model. Note that when we say pre-trained, we mean trained in the final (goal)
downstream task, not pre-trained in large general datasets (say Imagenet
(Russakovsky et al., 2015)). Hence, the difference between one-step-training
and two-step-training is whether there is a previous biased model that works
for the task of interest or not. Methods that fall in the one-step-training
category may very well have been pre-trained on general-purpose datasets. This
updated taxonomy reflects the fast advances we are witnessing in both industry
and research communities with the adoption of the so-called Foundation Models
(Bommasani et al., 2021).

Finally, we should mention that the categories of the taxonomy are not
mutually exclusive. Let us assume a given method uses an adversarial strategy
coupled with a regularization term for debiasing a model pre-trained on
Imagenet. We categorize it as belonging to both adversarial and optimization
categories within one-step-training.

Table 1 organizes all debiasing methods surveyed in this paper according to
the proposed taxonomy and also the respective domain (vision, language, or
multimodal).

Table 1. Debiasing methods organized according to the proposed taxonomy. Category | Sub-category | Vision | Language | Multimodal
---|---|---|---|---
Distributional | Heuristic | (Derman, 2021) | (Lu et al., 2020; Stafanovičs et al., 2020) | —
Generative | (Ramaswamy et al., 2021; Gambs and Ngueveu, 2022; Ngxande et al., 2020; Yucer et al., 2020; Chen and Joo, 2021) | (Qian et al., 2022) | —
Resampling | (Cao et al., 2022; Li and Vasconcelos, 2019; Tanjim et al., 2022) | — | (Yan et al., 2020)
One-Step-Training | Adversarial | (Edwards and Storkey, 2016; Wang et al., 2018; Li et al., 2020) | (Odbal et al., 2022; Gaci et al., 2022; Wu et al., 2021b; Rekabsaz et al., 2021; Fleisig and Fellbaum, 2022; Maheshwari et al., 2022; George Zerveas and Eickhoff, 2022) | (Yan et al., 2020; Berg et al., 2022)
Causal Approaches | (Kang et al., 2022; Dash et al., 2022; Kim et al., 2021) | (Gupta et al., 2022) | (Yang et al., 2021)
Disentanglement | (Park et al., 2021; Creager et al., 2019; Xu et al., 2020; Tartaglione et al., 2021; Kim et al., 2021) | (Du et al., 2022) | —
Optimization | (Martinez et al., 2020; Tanjim et al., 2022; Wang and Deng, 2020; Agarwal et al., 2022; Hendricks et al., 2018; Amini et al., 2019; Gronowski et al., 2022) | (Li et al., 2022; Du et al., 2022; Shen et al., 2021; George Zerveas and Eickhoff, 2022; Bordia and Bowman, 2019; Kaneko and Bollegala, 2021a; Webster et al., 2020; Chen et al., 2022) | (Wang et al., 2021; Kim and Cho, 2022; Zhao et al., 2017)
(Patil and Purcell, 2022; Serna et al., 2022; Li et al., 2020; Kim et al.,
2019; Kenfack et al., 2022; Ma et al., 2021; McGovern, 2021; Qian et al.,
2019)
Two-Step-Training | Distillation | (Mazumder et al., 2022; Li et al., 2021; Jung et al., 2021) | (Gupta et al., 2022) | —
Fair-Modules | (Karakas et al., 2022; Li et al., 2021) | (Fatemi et al., 2021; Wu et al., 2021a; Cheng et al., 2021; Lauscher et al., 2021; Rajabi et al., 2022) | (Yan et al., 2020; Park et al., 2020; Tang et al., 2021)
Fine-Tuning | — | (Gira et al., 2022; Faal et al., 2022; Liu et al., 2021; Wu et al., 2022) | (Berg et al., 2022)
Inferential | Prompting | — | (Wallace et al., 2019; Sheng et al., 2020; Gehman et al., 2020; Sharma et al., 2022; Schick et al., 2021) | (Mishkin et al., 2022)
Vector-Space Manipulation | (Salvador et al., 2022) | (Liang et al., 2020, 2021; Bolukbasi et al., 2016b; Du and Joseph, 2020; Aekula et al., 2021; Dev et al., 2021; Vargas and Cotterell, 2020; Bolukbasi et al., 2016a; Kaneko and Bollegala, 2019) | (Wang et al., 2021)
(Gyamfi et al., 2020; Kumar et al., 2020; Yang and Feng, 2020; Dev and
Phillips, 2019; Subramanian et al., 2021; Kumar et al., 2021; Kaneko and
Bollegala, 2021b; Lauscher et al., 2020)

###  5.1. Distributional

Bias-mitigation methods of the category distributional target at changing the
data distribution to better represent socially-vulnerable groups or to change
data objects of the dataset to remove unwanted biases. They rely on creating a
modified dataset to improve fairness or applying systems and rules to remove
or compensate underrepresented groups within the data. The rationale is that
the neural network will be trained using a more representative data
distribution, thus leading to a fairer model.

Distributional mitigation methods are stratified into 3 groups according to
the changes made to the data:

  1. (i)

Heuristic methods modify objects of the dataset according to predefined
algorithmic rules;

  2. (ii)

Generative methods aim to create or modify data objects using generative
models;

  3. (iii)

Resampling methods are based on under or over-sampling the dataset to mitigate
the under-representation of individuals in protected groups.

####  5.1.1. Heuristic

Unbalanced training sets may lead to models that reproduce certain disparities
present within the data. For that, one might change the dataset by modifying,
adding, or removing objects, somehow making it represent all protected
groups/individuals equally. When making these modifications, one should
consider that not all applications follow the same distributions, and hence
are prone to the same set of rules. For instance, new sentences must follow
pre-existing grammatical, lexical, and syntactic rules when one is modifying
NLP datasets.

Debiasing through heuristic manipulation is a viable strategy for NLP tasks.
Examples include using dictionaries or semantic and syntactic rules to replace
words or add informational labels/tokens (Lu et al., 2020; Stafanovičs et al.,
2020). More than just replacing words, those strategies seek to create new
information, not only by changing one or two words but by adapting the entire
surrounding information, avoiding the creation of nonsensical sentences. For
computer vision, a possible heuristic approach is to apply pre-determined
transformations in images to augment the dataset. Some methods can leverage
this process to augment only instances underrepresented in the dataset
(Derman, 2021).

While heuristic methods to change data distribution may be an interesting
alternative for bias mitigation, these changes are only practical in scenarios
where the application domain comprises explicit and well-known rules. One
example is language modeling, where templates of sentences can be used to
expand the original data. In other domains, however, heuristics do not scale
and fail when the rules are not easy to define manually.

Figure 3. General overview of generative methods for debiasing.

####  5.1.2. Generative

In domains like computer vision, where no explicit set of constraints is
specified to delimit data distribution, more complex approaches may be
necessary to adjust the level of fairness in a dataset. Finding a simple
strategy or rule that can be applied to all instances is hardly possible,
requiring the creation of deep networks capable of doing such modifications
(Ramaswamy et al., 2021).

Generative Adversarial Networks (GANs) can be an option when looking for ways
to increase a dataset with synthetic data (Ramaswamy et al., 2021; Gambs and
Ngueveu, 2022; Ngxande et al., 2020; Yucer et al., 2020), since they can
create high-quality new images when properly trained, balancing the dataset
with regard to its potential misrepresentation and allowing the training of a
new model over both original and synthetic data. Figure 3 gives an overview of
this strategy of augmenting (or balancing) a dataset with a generative neural
network.

In the language context, Qian et al. (2022) propose a sequence-to-sequence
model that generates perturbations regarding the protected variables in
dataset instances.

Systematic biases exist in many datasets, which is also the case with face
datasets. As annotators usually consider female faces as being happy more
often than men’s, Chen and Joo (2021) propose using Action Units (AU) that aim
to measure facial expressions to address this problem objectively. The
proposed framework does not need to modify labels, like other methodologies.
Using AUs, the model classifies two similar AU samples as similar people
besides their gender. To penalize for unfairness, they utilize the triplet
loss function as a regularizer. Note that this method can be applied to other
datasets (not only facial expressions), and one just needs to have other
objective measures, like body key points. That work was the first to show the
systematic effect of annotations in datasets for computer vision, and it is
even more visible in in-the-wild datasets.

####  5.1.3. Resampling

Resampling debiasing methods only modify the data distribution by rearranging
the existing dataset objects. While possibly lowering the number of objects
available for training, no artificial data is generated. The most common
resampling approaches balance the distribution of the training dataset
according to a specific attribute such as gender or race (Yan et al., 2020)
while others focus on discovering how to resample the original dataset to
create a fairer distribution that is better suited to represent all protected
groups in the data (Li and Vasconcelos, 2019; Tanjim et al., 2022).

A more sophisticated strategy is to use more than one dataset to build a more
representative one. This process can use different sampling approaches to
select the best configuration of available objects to better represent the
diversity of protected attributes, thus providing a fairer dataset for
training (Cao et al., 2022).

###  5.2. One-Step-Training

Although manipulating the data is a straightforward strategy to have more
diverse or balanced data, it is often not enough to produce fair neural
models. Some challenges that remain unsolved by distributional approaches are:
i) (deep) neural networks are data hungry, which means undersampling
strategies could reduce the data up to the point training becomes unfeasible;
ii) even with data that perfectly represents the population distribution,
undesirable characteristics such as stereotypes and prejudice that are present
in society may arise (Wang et al., 2019). For solving those problems, one may
need to resort to additional strategies that happen either during training or
during inference, which are the focus of this section.

We call one-step learning debiasing methods that act during the main training
procedure. These methods are further divided into four distinct groups
according to the debiasing strategy that is used:

  1. (i)

Adversarial methods make use of the adversarial framework or of adversarial
examples to teach the model not to resort to undesired biases;

  2. (ii)

Causal methods use causal graphs and counterfactual examples to teach the
model which relationships are relevant within the data;

  3. (iii)

Disentanglement methods separate the features in the latent space to
manipulate them independently;

  4. (iv)

Optimization methods include loss function adaptions, addition of
regularization terms, and other modifications for improving weight
optimization.

####  5.2.1. Adversarial

It is a known fact that adversarial examples can deceive deep learning models.
Neural networks may be fooled by intended perturbed images that do not contain
human-perceivable changes (Goodfellow et al., 2015). Notwithstanding,
adversarial examples can be included in the training dataset to create more
robust models. That setup uses two models: one is trained to solve the task
objective, whereas the other focuses on creating adversarial examples that try
to confound the first model (Goodfellow et al., 2014). The objective is to
teach the main network not to use the protected attribute to do the task (Xu
et al., 2019; Zhang et al., 2018; Odbal et al., 2022; Gaci et al., 2022; Wu et
al., 2021b; Li et al., 2020). To force the model not to rely on protected
attributes, it is possible to erase or mask them from the data source (Wang et
al., 2018), create new data, or even attack the deep representation generated
by the model (Edwards and Storkey, 2016; Fleisig and Fellbaum, 2022;
Maheshwari et al., 2022), which is less interpretable. Figure 4 depicts this
general idea.

Figure 4. General overview of adversarial methods for debiasing.

For the information retrieval domain, adversarial training is being used to
create embeddings that are bad predictors for the protected attribute, but
that are capable of accurately predicting the target variable (Rekabsaz et
al., 2021; George Zerveas and Eickhoff, 2022; Yan et al., 2020).

Note that all adversarial methods rely on annotations of protected variables
or groups in the dataset. They allow the models to focus on legitimate
attributes while ignoring the protected ones. However, the need for annotation
is a downside of these approaches. Fortunately, the annotations are not needed
for inference time.

####  5.2.2. Causal

We can leverage any knowledge on the causal-effect relationship between
protected attributes and outcomes to try and fix model unfairness. For
identifying these relationships one can make use of causal graphs or
counterfactual samples during training. Causal approaches are a popular choice
to mitigate general biases, such as language priors in VQA (Kolling et al.,
2022; Chen et al., 2020; Abbasnejad et al., 2020; Niu et al., 2021), and we
explore how they can be adapted to solve fairness issues.

By using causal graphs we can force the model to learn valuable relationships
between features, intermediate representations, and predictions. Yang et al.
(2021) propose a specific attention module in the network that is responsible
for discovering such associations, and then use them for bias mitigation.
Another benefit of building a causal graph is that it can later be used to
explain predictions, an important artefact for explainable AI.

Creating counterfactual samples, on the other hand, allows the model to train
over examples that are harder to predict, since they may not occur in the
training distribution, improving the robustness of the model towards specific
protected groups. We can employ linguistic rules to create new examples during
training, forcing the model to learn with one group and its opposite (e.g.,
male and female genders) (Gupta et al., 2022). Other strategy is to
disentangle visual representations in the latent space and use isolated
features to create the counterfactual samples (Kim et al., 2021). Additional
work uses an auxiliary model to discover the causal graph and use discovered
relations to create counterfactual samples using the protected attributes
(Kang et al., 2022; Dash et al., 2022).

Generating counterfactual samples is a good option to ensure that the model
will see a larger number of distinct examples. However, such approaches may
lead to creating extreme counterfactuals that do not represent valid instances
from the dataset or the distribution they represent (King and Zeng, 2006).

####  5.2.3. Disentanglement

During training, neural models create latent representations that represent
automatically-learned features extracted from the data objects. The difficulty
of learning a task and the robustness and generalizability of a model are
directly correlated with the quality of the respective learned latent
representations. One way to increase such quality is via disentanglement. A
disentangled representation is a representation where each learned factor
corresponds to a single factor of variation in the data and is invariant to
other factors of variation (Bengio et al., 2013). Disentangled representations
offer many advantages, such as boosting predictive performance (Locatello et
al., 2019b), increasing interpretability (Higgins et al., 2017), and improving
fairness (Locatello et al., 2019a).

Learning disentangled representations means to break down features into new
dimensions where information is split into independent spaces. This approach
can be used to separate the underlying structure of some objects in their
various parts in a more representative and interpretable way (Higgins et al.,
2018). For instance, the shape of an object and its position in the image can
be independently broken down. That isolation of features allows us to operate
in specific details of the input data rather than modifying it entirely.
StyleGAN (Karras et al., 2019) is an example of that, where it is possible to
modify only the hair attributes of someone’s face without further changes to
the image.

That idea of splitting different data dimensions can also be explored for
debiasing during training, inducing a model to learn only features that are
invariant to the protected attributes. Figure 5 illustrates such an approach.

Figure 5. General overview of the disentanglement approach for debiasing.

By decomposing the representation into different spaces and splitting the
target attribute dimension from the protected attribute dimension, a model may
learn not to correlate sensitive information to the real objective task,
prioritizing only relevant and essential information.

This approach can be implemented as a new model or module responsible for the
disentanglement (Creager et al., 2019) or as a regularization term that uses
the labels to guide the division of features (Tartaglione et al., 2021).

Xu et al. (2020) show that disentangled latent representations may achieve
superior performance in terms of demographic bias than simply adding the
protected attributes in the classification layer in a typical fairness-
through-awareness approach. In the study of Park et al. (2021), a variational
autoencoder (VAE) is employed to create a representation that disentangles the
latent space into three distinct subspaces:

  1. (i)

The first one has information of the target variable and no protected
attribute information;

  2. (ii)

The second contains both target and protected attribute information;

  3. (iii)

The third contains just protected attribute information.

Images often carry both protected and legitimate attributes that are hard to
separate. Disentanglement learning is an alternative to achieving fairness
through unawareness in computer vision Du et al. (2022) propose to disentangle
the latent space and to use only the subspace that spans the legitimate factor
to perform the classification task. Kim et al. (2021) implement the concept of
counterfactual fairness while requiring the counterfactual samples to generate
disentangled representations of the attributes.

Disentanglement approaches are a good option for separating protected and non-
protected features, making it possible for the model to consider only one
group of attributes when doing a specific task. Unfortunately, to perform a
task such as classification, the training phase must be divided into at least
two steps: the first one to learn the disentangled representations and the
second to learn the target task effectively.

####  5.2.4. Optimization

Optimizing a loss function is the training strategy of every neural network.
Together with an optimization rule, they dictate how the weights must be
learned for a specific task. By adding or modifying terms in a loss function,
one can drastically change the solutions generated by a neural network. We
categorize a method as being an optimization approach if it proposes changes
in the optimization procedure during training towards increasing fairness,
forcing it to follow a desired output distribution without modifying the input
data (Jain et al., 2021).

Several studies propose new loss functions for penalizing or ensuring specific
behaviors. Often these new loss terms are summed or used together with the
objective loss (e.g., Cross-Entropy), ensuring that the model will learn the
main task while also being subject to fairness constraints. One can use this
strategy to penalize models that wrongfully predict the protected attribute
(Hendricks et al., 2018). It is also possible to re-calibrate the loss
function according to the value of the protected attribute (Amini et al.,
2019; Tanjim et al., 2022) or to optimize for fairer data representation (Kim
and Cho, 2022; Gronowski et al., 2022; George Zerveas and Eickhoff, 2022;
Patil and Purcell, 2022; Serna et al., 2022; Bordia and Bowman, 2019).

With textual data, changes in loss function usually aim to remove bias from
embedding representations. It is possible to manipulate the multi-dimensional
space in which the embeddings are located to decouple them from non-desired
sub-spaces (Kaneko and Bollegala, 2021a; Webster et al., 2020; Chen et al.,
2022; Qian et al., 2019; McGovern, 2021).

Aside from proposing novel loss functions, it is possible to employ an
algorithm to inject constraints during model training to ensure a fair
distribution of predictions for the training data (Zhao et al., 2017; Li et
al., 2020). Also, regularization terms can minimize the mutual information
between feature embedding and bias (Kim et al., 2019). Other possibility is to
apply a norm clip based on data groups, which improves diversity in data
generated by GANs (Kenfack et al., 2022).

Contrastive learning is also an optimization procedure capable of model
debiasing. During training, the model faces examples from different classes,
and it should classify them correctly while keeping examples from different
classes apart in the embedding space. We can use this technique to increase
fairness during training by picking both positive and negative examples
conditioned on the protected attribute (Ma et al., 2021; Wang et al., 2021) or
by enforcing that distinct protected groups be distant in the embedding space
(Shen et al., 2021).

Optimizing weight distributions for particular examples in order to penalize
easier ones is also a common strategy for debiasing (Li and Vasconcelos, 2019;
Li et al., 2022; Wang and Deng, 2020). It is possible to reduce unfairness by
maximizing ratios between losses in the reweighted dataset and the uncertainty
in gold-standard labels. This strategy can come with a coefficient of
variation as a data repair algorithm to curate fair data for a specific
protected class and improve the actual positive rate for that class (Agarwal
et al., 2022). It has the advantage of working in both supervised and
unsupervised learning.

Some studies aim to formulate the question of fairness as a multi-objective
optimization problem. The work of Martinez et al. (2020) defines each
optimization objective as the protected group conditional risk. In such
formulation, the goal is to find solutions at the Pareto frontier.

With fairness criteria tied to the training objective of the network, two
challenges arise:

  1. (i)

There may be a demand for extra annotations in the data regarding the
protected attributes to allow for computing novel losses;

  2. (ii)

The optimization modifications can incur in a trade-off between fairness and
accuracy, which we must be aware of. This is further discussed in Section 6.

###  5.3. Two-Step-Training

It is customary (and even considered a best practice) to not always train
neural networks from scratch. Instead, whenever possible, the weights of a
model are set to the state of a previous training procedure, which may even
have happened using a different dataset. This typically accelerates training
and often improves the performance and generalization capabilities of a model.
Since this practice is frequently adopted due to its benefits, applying
debiasing methods during the initial stages of model training may be
unfeasible or undesirable, especially since this may imply “wasting”
computational resources that were used in the pre-training phase. Thus,
debiasing strategies that focus on adapting existing models become attractive
to improve fairness. We have separated two-step training methods into three
distinct groups, according to the debiasing strategy used:

  1. (i)

Distillation includes methods that train a new model using the teacher-student
approach;

  2. (ii)

Fair Modules aggregates methods that add new modules to existing models to
remove unfairness;

  3. (iii)

Fine-Tuning lists approaches to retrain a model without architectural changes.

####  5.3.1. Distillation

The core concept of distillation is to transfer knowledge between two models:
the teacher and the student. The student network is optimized to match the
teacher model’s predictions in addition to a task cost function. The student
and teacher models have similar architectures, but the student network has
less capacity, i.e., less optimizable parameters. In this scenario, the
distillation process creates a model with similar predictive capabilities
while simultaneously reducing the amount of resources needed (Sanh et al.,
2019). One of the benefits of this type of approach is increasing the
generalization capabilities of models without requiring more annotated data
(Hinton et al., 2015).

Figure 6. General overview of the distillation approach for debiasing.

In the scope of fair deep learning, one can use model distillation to produce
fairer student models based on an unfair teacher. To that end, most studies
add a specific fair-related loss (Mazumder et al., 2022; Gupta et al., 2022;
Li et al., 2021) or regularization term (Jung et al., 2021) to the original
distillation framework. Intuitively, the student model learns to emulate the
teacher model’s knowledge while being simultaneously instructed to not rely on
protected attributes present in the dataset. Figure 6 illustrates the core
ideas of this type of approach.

Despite its benefits, the distillation strategy also presents some drawbacks.
For one, it is typically much more expensive than other available two-step
training approaches. In scenarios where the intended teacher model’s weights
or (log-)likelihood outputs are not available, e.g., models behind APIs such
as GPT-3 (Brown et al., 2020), it becomes impossible to train a student
network. Finally, the debiasing loss may conflict with the distillation
objective, which may result in generalization problems (Stanton et al., 2021).

####  5.3.2. Fair Modules

Another useful group of debiasing techniques for pre-trained models involves
adding new components, or modules, to an existing architecture, as depicted in
Figure 7. By combining the original architecture with a new group of layers,
we can transfer the responsibility of learning fairness objectives to the new
modules, and potentially leave the original weights untouched.

One may use specialized modules to create additional representations that can
improve the quality of embeddings, thus preventing the model from learning
shortcuts that lead to biases (Wu et al., 2021a). Another potential approach
is to include modules that detect or predict the presence of protected
attributes, which is then used to modulate the overall answer (Park et al.,
2020; Tang et al., 2021; Karakas et al., 2022; Li et al., 2021).
Alternatively, some approaches add modules that are responsible for learning
how to mitigate biases based on the inference of the original model (Cheng et
al., 2021; Lauscher et al., 2021; Fatemi et al., 2021). Finally, Rajabi et al.
(2022) proposes to train an encoder-decoder module that processes inputs
before the original classifier network to remove unwanted biases without
changing the original model. Since the new module removes potentially
protected attributes, they tend to not affect the final model’s prediction,
effectively removing their influence.

One of the main advantages of this strategy is that one can freeze the
original model weights and prevent model degradation and catastrophic
forgetting, a phenomenon where previously-learned concepts and tasks are
replaced by new information (Fatemi et al., 2021). However, depending on the
original model capacity, these architectural modifications may not work as
well as techniques that adapt the original model weights.

Figure 7. General overview of the fair-module strategy for debiasing.

####  5.3.3. Fine-Tuning

Fine-tuning refers to adjusting some or all of the weights of a model, that
has already been trained, for a new combination of task, dataset, and domain.
Figure 8 shows the general framework of the fine-tuning strategy for debiasing
neural models. In the context of fairness, we can view fine-tuning as a
specialization procedure. That is, pre-training is responsible for teaching
general concepts pertaining to a data modality (e.g., language modeling for
NLP), while fine-tuning forces the model to shift focus to fair concepts.

Gira et al. (2022) has shown that fine-tuning a GPT-2 model changing as little
as 1% of the model’s parameters using a curated dataset is enough to increase
model fairness while retaining the knowledge acquired in the pre-training
stage. Reinforcement learning algorithms may also be used to guide weight
optimization and provide a better alternative than a simple loss or dataset
modifications (Faal et al., 2022; Liu et al., 2021). Wu et al. (2022)
hypothesizes that some network parameters may be correlated with biased
predictions, and thus consider model pruning to be a potential solution.
Paired with a specific dataset used as a guide, the authors prune a pre-
trained model, removing weights responsible for such biased predictions while
attempting to maintain a similar performance.

Figure 8. General overview of the fine-tuning strategy for debiasing.

Differently from adding a new module to be responsible for debiasing, in fine-
tuning approaches the driving factor is the dataset used to continue model
training. In situations where a sufficiently large and complex dataset is not
available, fixing the problems of a pre-trained model using fine-tuning
approaches may not be possible. Additionally, debiasing via fine-tuning
requires that the practitioner be aware of fairness problems that are already
present in the pre-trained model (or if the model is affected by a specific
bias of interest), which may imply a more thorough investigation via empirical
analyses.

###  5.4. Inferential

Debiasing strategies that are applied during training or by changing the data
distribution may be unfeasible in situations where the model is not available
or when fine-tuning is a resource-intensive task. Large models such as GPT-3
(Brown et al., 2020) are not easily used by most practitioners due to hardware
constraints. For instance, to instantiate an OPT (Zhang et al., 2022) model,
that has around $130$ billion parameters, more than $300$GB of VRAM are
required, which makes it impractical to fine-tune for most use cases.
Additionally, some large neural models are not open sourced and can only be
accessed via commercial APIs, which makes unfeasible the application of the
debiasing approaches presented so far. Despite their generalization
capabilities, such large models are also prone to accruing societal biases
(Bender et al., 2021). Inferential methods are alternatives in such scenarios,
because they intervene during inference time to make models fairer, leaving
weights untouched. There are two possible alternatives to this approach:

  1. (i)

Prompting prepends or alters the model input with specific triggers that
stimulate a bias-free result;

  2. (ii)

Vector-Space Manipulation manipulates the embedding space to remove undesired
biases.

####  5.4.1. Prompting

When researching Foundation Models in NLP, Brown et al. (2020) discovered that
this category of large-scale models can perform few-shot learning when
receiving specific instructions in their prompts (Brown et al., 2020; Zhang et
al., 2022; Black et al., 2022). In the context of FMs, prompts refer to the
model input, and typically consist of tokens that are later converted to
continuous representations (e.g., an embedding layer). This discovery started
a separate field of study known as prompt engineering (Reynolds and McDonell,
2021).

Besides the simple approach of limiting and filtering a model’s prompt (e.g.,
by restricting the use of certain words) (Mishkin et al., 2022), recent
studies have shown that language models are vulnerable when attacked with
specific tokens in their prompts, a procedure known as triggers (Wallace et
al., 2019). Triggers consist of tokens appended to the original prompt that
lead to unexpected (or, in the case of fairness, desired) behaviors. One can
use such induced behaviors to mitigate (or provoke) certain biases, which make
triggering methods attractive to promote fairness in large models (Sheng et
al., 2020). Recent studies (Berg et al., 2022; Gehman et al., 2020; Sharma et
al., 2022) seek to discover specific tokens (discrete or continuous) to help
debiasing models. Schick et al. (2021) propose an unusual form of debiasing
through prompting by using the model itself to identify and mitigate its own
biases. In that study, the authors prepended templates such as “the following
text discriminates against people because of their gender:” to the prompt and
identified that this addition manipulates the distribution of words to lower
the probability of discriminatory outputs.

Figure 9. General overview of the prompting strategy for debiasing.

This is a versatile and attractive approach since, with the correct triggers,
it is possible to mitigate or even induce specific biases, and potentially
deal with multiple protected groups. Figure 9 shows the general idea of this
category of methods. A downside of prompting approaches is clearly exemplified
in the context of NLP: triggers often add undesired contexts to sentence
generation, since the model is conditioning its outputs on a token that may go
against the initial context.

####  5.4.2. Vector-Space Manipulation

A common approach in NLP to learn the semantic meanings of words is to use
word-embedding representations, which represent tokens as $d$-dimensional
vectors (Mikolov et al., 2013; Pennington et al., 2014). Compared to
traditional NLP word representations (e.g., term frequency-inverse document
frequency), word embeddings are a far better alternative in the context of
neural models, since it gives the optimization procedure the chance to learn
better representations. The seminal work of Bolukbasi et al. (2016b) of
fairness in word embeddings describes how such vectorized word representations
encapsulate societal biases. Aimed with this knowledge, it became possible to
use word-embedding manipulation techniques for bias mitigation in fairness-
sensitive applications (Bolukbasi et al., 2016b; Bordia and Bowman, 2019).
Most studies of word embedding debiasing follow a similar strategy to the
Hard-Debiasing method (Bolukbasi et al., 2016b), where a “protected” subspace
is identified and then words have their projection removed from that space.
Other word embedding debiasing approaches use a wide variety of helper tools,
such as clustering (Du and Joseph, 2020), a helper dictionary (Kaneko and
Bollegala, 2021b), causal inference (Yang and Feng, 2020), among others
(Aekula et al., 2021; Dev et al., 2021; Vargas and Cotterell, 2020; Bolukbasi
et al., 2016a; Kaneko and Bollegala, 2019; Gyamfi et al., 2020; Kumar et al.,
2020; Dev and Phillips, 2019; Kumar et al., 2021; Kaneko and Bollegala, 2021b;
Lauscher et al., 2020; Gaci et al., 2022).

With the rise of Transformer-based architectures (Vaswani et al., 2017), which
use (self-)attention as their primary inductive biases, methods that
manipulate embedding spaces had to be adapted since attention uses original
word-embedding representations to create contextualized word representations.
That is, the semantic meaning of a token depends on the entire sequence of
tokens. A typical adaptation to work with contextual vector spaces is adding
or complementing inputs with pre-defined sentence templates (Kurita et al.,
2019). Liang et al. (2020) use such templates to adapt the Hard-Debiasing
approach to a contextual scenario. Other studies rely on strategies like
iterative generation to mitigate biases (Liang et al., 2021; Subramanian et
al., 2021). This type of method can be adapted to work on a wide range of pre-
trained Transformer-based architectures, since they all rely on the attention
mechanism.

Although vector-space manipulation is explored mainly in language
applications, it is also possible to adapt the core ideas to CV. Salvador et
al. (2022) rely on clustering and embeddings to fix unfairness issues in the
visual domain. The authors propose a clustering-based calibration method to
ensure that similar images have similar scores according to which cluster they
belong to. Also working with images, Wang et al. (2021) remove the projection
of CLIP embeddings from protected subspaces.

Much like other methods, vector-space manipulation approaches are not without
drawbacks. For instance, in the context of Transformer models, biases have
different effects in each layer, adding considerable complexity to mitigation
strategies (Dev et al., 2020; Vig et al., 2020). Another difficulty is the
necessity of a reference to guide the model to identify biased sub-spaces,
which leads to the creation of repositories of biased and unbiased examples
(Bolukbasi et al., 2016b).

##  6\. Challenges and Future Directions

Throughout this survey, we presented several challenges regarding how to
define the concepts of bias and fairness (Section 3), how to evaluate ML
models using individual or grouped notions of fairness (Section 4), and how to
debias models to make them fairer (Section 5). After analyzing the
aforementioned issues, we now focus on general challenges for fairness in
neural models while also highlighting potential future directions of research.
Specifically, we analyze  (i) how to assess and present the fairness risks in
neural models;  (ii) how to identify who should be responsible for owning and
dealing with fairness issues in neural models;  (iii) how to deal with the
fairness-accuracy trade-off; and  (iv) current challenges and research
directions for fairer Foundation Models.

###  6.1. Risk Awareness

Every machine learning model is a direct consequence of several design
decisions, such as dataset collection and filtering, model architecture,
optimization procedure, and hyperparameter selection. Since these design
decisions have a direct influence not only on the performance of a model but
also on its perceived fairness, it is essential to clearly communicate these
when internally or publicly releasing a model to highlight its limitations and
intended usage. That way, all relevant stakeholders can make informed
decisions concerning the usage of specific technologies in the context where
they operate. However, deciding how and when to communicate such design
decisions is not a trivial matter. Thus, we now highlight recent advances and
best practices to help ML researchers and practitioners create proper
documentation to improve risk awareness.

####  6.1.1. Artifact Documentation

Several studies (Bender and Friedman, 2018; Benjamin et al., 2019; Mitchell et
al., 2019; Holland et al., 2020; Gebru et al., 2021) have attempted to
standardize the communication of dataset and model design decisions in hopes
of stimulating best practices in the creation of such artifacts, especially
regarding sensitive topics such as fairness, privacy, and security. Datasheets
for datasets (Gebru et al., 2021) raises several questions (grouped into
sections that roughly match the key stages of a dataset life cycle) to help
dataset creators and consumers think about a dataset holistically, asking
details on the motivation for creating the dataset, how the data was collected
and aggregated, the recommended uses, and so on. Datasheets may help users to
select more appropriate datasets for a specific task, increase transparency,
mitigate unwanted societal biases, and even increase reproducibility. There
are several alternative approaches for documenting datasets; two examples
include the Dataset Nutrition Label (Holland et al., 2020) and Data Statements
for NLP (Bender and Friedman, 2018). Critically, all of the aforementioned
approaches highlight the importance of documenting datasets to inform end-
users of fairness-related limitations since most biases are created or
inherited in the data collection stage.

Regarding model documentation, Model Cards (Mitchell et al., 2019) are an
excellent protocol to increase transparency concerning training and evaluation
protocols across different bias and fairness metrics (especially among
different values of protected attributes, both individually and combined), and
clarifying the intended use cases of models to minimize their usage in
inappropriate contexts. Model cards are especially important in research and
open-source initiatives, since it leads to a more responsible and accountable
model democratization process, which allows stakeholders to compare candidate
models across not only traditional performance metrics (e.g., accuracy for
classifiers) but also ethical and fairness considerations.

Often, artifact creators must also legally protect themselves from misuse. The
Montreal Data License (MDL) (Benjamin et al., 2019) is a project that attempts
to improve the taxonomy of data licensing to better suit machine learning and
artificial intelligence. In the context of fairness, MDL provides optional
restrictions regarding ethical considerations when granting rights to use
and/or distribute a dataset, e.g., restricting usage in high-risk scenarios
such as health-related fields or military applications.

####  6.1.2. Research Documentation

Several mechanisms exist (e.g., institutional review boards, conference
program committees, and funding bodies) that provide the academic community
with means to prevent unethical research. However, there are few implemented
protocols concerning fair research in mainstream deep learning conferences,
and deep learning papers often do not discuss such limitations. Since
conference papers tend to be prioritized over journal publications in computer
science (and deep learning especially), conference organizers should strive to
improve fairness awareness in conference publications.

Some deep learning conferences have recently included protocols to help
researchers communicate fairness constraints. In 2020, NeurIPS required that
all papers include a “broader impact statement” covering the ethical and
fairness aspects of research in their camera-ready versions (NeurIPS, 2020)
Additionally, the conference incorporated a new ethics review process where
technical reviewers could flag papers for ethical concerns, which a pool of
ethics reviewers would later analyze. These initiatives had a mixed reception
from the academic community due to their perceived political nature. Several
studies have analyzed the results of this experiment (Prunkl et al., 2021;
Ashurst et al., 2022; Nanayakkara et al., 2021) and arrived at the following
conclusions:  (i) although $10$% of all papers opted out of writing the
broader impact statement, this occurred mainly in theoretical subareas. Most
authors took the opportunity to reflect on their work rather than stating that
the statement was not applicable.  (ii) the average statement length was $168$
words and the distribution presents a long tail —the longest statement had
$4000$ words. The areas of CV and NLP differ in average statement length
($166$ and $223$, respectively).  (iii) authors focused more on positive
societal impacts rather than negative ones, which intuitively goes against the
main purpose of the experiment.

The lack of standardization and clear guidance regarding the expected contents
of a broader impact statement and misaligned incentives for researchers may
explain the results of this experiment. However, since this was the first time
that broader impact statements were mandatory, it is hard to judge the idea’s
true potential and how researchers would behave in the following years.
Following the feedback of the broader impact statement implementation, the
NeurIPS conference opted to replace the statement with a “Paper Checklist”
that provides researchers with a list of best practices for responsible
machine learning research (NeurIPS, 2021a, b). This type of mechanism is an
attempt to force researchers to think about a study’s negative societal
impacts before it is released, which may also increase risk awareness for ML
practitioners who may use open-source code or attempt to reimplement
algorithms.

####  6.1.3. Fairness Frameworks

One of the major goals of fairness research is the inclusion of fairness
metrics and techniques into traditional ML pipelines to help mitigate biases.
Considerable efforts have been made to create open-source fairness software
toolkits that help ML practitioners audit datasets and models (Saleiro et al.,
2018; Bellamy et al., 2019; Wexler et al., 2019; TensorFlow, 2020; Vasudevan
and Kenthapadi, 2020; Bird et al., 2020; Google, 2020). These toolkits
implement visualization and interpretability techniques, fairness metrics, and
state-of-the-art debiasing techniques, translating research into actionable
procedures. However, there is a lack of application of such solutions in
practice, especially in industry. According to Richardson and Gilbert (2021),
several factors explain this:  (i) there are too many fairness metrics, and
the differences between them are not clear to practitioners. Furthermore,
major trade-offs exist between fairness metrics, and often it is
mathematically impossible to optimize multiple fairness metrics. Choosing the
“right” metric is delegated to practitioners, who are unfamiliar with the
technical aspects of fairness research.  (ii) there exists a disconnect
between fairness research in academia and industry, which translates to a lack
of applicability of procedures and metrics in industry settings. Additionally,
fairness concepts are highly domain-dependant, differing substantially between
domain applications, and yet most domains lack thorough and specific
algorithmic bias guidance.  (iii) frameworks often do not provide help with
communicating fairness concerns and trade-offs to stakeholders, which
inevitably reduces the adoption of frameworks due to organizational frictions.

Several suggestions have been proposed to mitigate these issues, and many
involve improvements in communication between academics and practitioners.
Friedler et al. (2016) suggest that fairness experts clearly state the
priorities of each fairness metric, while Verma and Rubin (2018) suggest that
researchers clarify which definitions are appropriate for which situations.
Friedler et al. (2016) argue that new fairness metrics should only be
introduced if they behave fundamentally differently from existing metrics.
Several practitioners have also requested domain-specific procedures and
metrics, and that fairness experts create knowledge bases for each domain
(Holstein et al., 2019) Based on user feedback, fairness researchers should
also consider creating taxonomies of potential harms and biases (Madaio et
al., 2020; Cramer et al., 2018), easy-to-digest summaries explaining biases
and their potential sources (Garcia-Gathright et al., 2018; Cramer et al.,
2018), guidelines for best practices throughout the ML pipeline (Holstein et
al., 2019), and tutorials exemplifying how to incorporate fairness (Holstein
et al., 2019; Madaio et al., 2020; Richardson et al., 2021). We believe that
organizations in industry settings also have a big part to play in ensuring
the implementation of fairness protocols. Organizations should promote a
culture of fairness awareness and incorporate fairness as a global objective
(like security, privacy, and accessibility) (Garcia-Gathright et al., 2018;
Madaio et al., 2020; Stark and Hoffmann, 2019). Additionally, organizations
should strive to provide practitioners with resources and fairness support
teams that provide knowledge and actionable steps in fairness issues during
all steps of the ML pipeline (Madaio et al., 2020; Mittelstadt, 2019).

####  6.1.4. Final Considerations

We believe it is the responsibility of data and model creators to communicate
fairness risks. Artifact documentation may also be helpful for policy makers,
investigative journalists, and individuals whose data are included in the
datasets or who would be impacted by the deployment of such models. We
recommend that the language used in such documents range from accessible to
technical, allowing both laypersons and domain experts to understand the
critical decisions made to create a specific artifact. However, none of the
approaches presented in this section provide a complete solution for
mitigating unwanted societal biases or potential risks accrued by the usage of
any algorithmic approach. Society constantly changes, thus dataset and model
creators will never anticipate every possible use of a particular artifact,
and neither should they attempt to do so. Instead, they should focus on
objective facts and actions taken during the artifact creation pipeline, which
will help consumers, companies, and governmental entities decide whether the
technology is appropriate for their context and society as a whole.

###  6.2. Risk Ownership

An important but often overlooked discussion is deciding who is responsible
for preventing algorithmic misuse. Different scenarios require different
levels of attention: systems that do not involve humans-in-the-loop (e.g.,
autonomous driving or automated medical diagnosis) significantly increase the
impact of algorithmic biases. Sometimes, adjustments should be performed by
the owners of the technology (not only developers but also organizations). In
other cases, however, the user of an AI-based system must be responsible for
correctly using a tool or adjusting outputs to fit the notions of bias and
fairness within the context in which they operate. Ultimately, we believe that
fairness risk ownership is tied to two concepts: how much control a user has
over the model’s outputs and whether the user is directly affected by model
decisions. To illustrate this challenge and provide some guidance, we contrast
two exemplar situations of model deployment, namely facial recognition and
image generation, where we believe that the responsibility of dealing with
unfairness falls to artifact owners and users, respectively.

####  6.2.1. Algorithm Owner

A facial recognition system extracts facial features to verify a person’s
identity and has a wide range of applications. Since facial recognition is one
of the most popular biometric modalities due to its simple data collection
process, several companies have invested in software-as-a-service facial
recognition products and incorporated them into mainstream technological
products, such as smartphones. However, some studies (Klare et al., 2012a;
Buolamwini and Gebru, 2018; Raji et al., 2020) have analyzed the potential
fairness pitfalls of neural facial recognition systems and concluded that they
might discriminate based on protected attributes, such as race and gender, by
performing significantly worse on specific demographics. This discrepancy in
performance is worrisome, especially considering that one of the clients of
such facial recognition systems includes governments and law enforcement. For
instance, despite not directly determining the fate of an individual, such
technology can be used to identify suspects in video surveillance footage, and
erroneous misidentifications can have serious detrimental effects. In this
scenario, the “victims” of facial recognition systems have absolutely no
control of the system’s output and can be directly impacted by its decisions.
Thus, the ownership of fairness risks in this context must fall into the
artifact’s owners.

####  6.2.2. Algorithm User

Image generation is a research topic that can potentially disrupt several
markets, such as art, design, fashion, retail, and many others. One example of
an image generator is DALL-E (Ramesh et al., 2022), a text-to-image model
created by OpenAI. Since DALL-E’s training dataset was collected from the
Internet, it was expected that the model would inherit societal biases.
Regarding fairness, OpenAI implemented several measures to prevent algorithmic
misuse (Mishkin et al., 2022) but, despite these efforts, the company was
still heavily criticized, and DALL-E was described as a “harmful tool” for
reproducing societal biases, such as the (lack of) association of certain
ethnicities with certain roles in society (e.g., Black CEOs or female garbage
collectors) while facilitating the creation of “deep fakes”. OpenAI
effectively positioned itself as the owner of the risk of someone misusing
their product, and all decisions of what constitutes an unfair generation
became centralized at the company level. However, OpenAI did not have to own
this risk since users can control the model’s output via input prompts and are
not directly harmed by generated images. Additionally, the model is indeed
capable of generating diverse outputs, given the right prompts. A more
straightforward solution would be to provide disclaimers regarding fairness
limitations and instructions on how to construct prompts that generate
“fairer” and more diverse images and let users decide how they want to steer
image generation since different solutions result in different trade-offs.

###  6.3. Fairness-accuracy Trade-off

As explored in Section 4, fairness researchers have already proposed several
quantitative fairness metrics, giving rise to optimized objectives alongside
task-specific metrics. However, metrics usually either emphasize individual or
group notions of fairness, and it is often mathematically impossible to
optimize multiple fairness metrics (Pessach and Shmueli, 2020; Chouldechova,
2017; Barocas et al., 2019), which forces practitioners to select the most
appropriate metric to optimize (which is context-dependant). Additionally, it
has been empirically observed that a trade-off exists between fairness and
task performance and that increasing fairness often results in lower overall
performance. This gives rise to the challenge of analyzing the fairness-
accuracy trade-off for a given scenario. Practitioners must be careful when
determining how to measure model performance, since this can be done in
several ways and the choice of performance measure(s) can disguise or create
new ethical concerns. A reduction in accuracy may be the best outcome,
especially if the difference in performance can be explained by algorithmic
unfairness.

A potential solution to understand the trade-off characteristics between task
and fairness metrics is to rely on advances in multi-task optimization
literature to help neural models during optimization (Crawshaw, 2020;
Vandenhende et al., 2021; Gong et al., 2019). One example of such a technique
is generating a Pareto frontier to determine the set of Pareto-efficient
solutions for a specific combination of fairness and task-specific metrics
(Martinez et al., 2020; Shah et al., 2021; Lin et al., 2019; Ruchte and
Grabocka, 2021; Nia et al., 2022; Haas, 2019). Pareto efficiency corresponds
to a situation where the performance of a model regarding a specific criterion
cannot be made better without reducing the performance on at least one
individual metric. Another potential solution is to use techniques that
increase fairness through proxies, such as disentanglement and causal
inference, since they usually generalize better (although often at the cost of
task performance).

###  6.4. Fairness in Foundation Models

A Foundation Model corresponds to “any model that is trained on broad data at
scale and can be adapted to a wide range of downstream tasks” (Bommasani et
al., 2021). FMs were initially introduced in the context of NLP research but
are rapidly causing a revolution in all areas of deep learning research and
industry applications. The term “foundation” specifies the role of this
category of models: a FM is incomplete by itself, but serves as the common
basis from which many task-specific models are built via adaptation. Since
such models have the potential of being adapted to several tasks, biases may
be perpetuated or amplified if not adequately addressed, making fairness a
fundamental research direction for FMs (Bommasani et al., 2021; Meade et al.,
2021; Delobelle et al., 2021; Schick et al., 2021).

Mitigating biases in FMs is not a trivial matter, especially due to their
dataset and training regimes. The datasets that support the training of FMs
contain hundreds of millions or even billions of instances. Thus applying pre-
processing debiasing techniques is not an attractive option due to its
potential impact on monetary cost and generalization capabilities. One-step-
training debiasing techniques should also be avoided since it is impossible to
optimize for all fairness concepts simultaneously, and FMs are used in several
scenarios and contexts. For these reasons, the most promising research
directions regarding bias mitigation in FMs are fine-tuning and prompting
approaches (Gira et al., 2022; Sheng et al., 2020; Berg et al., 2022; Gehman
et al., 2020; Sharma et al., 2022; Schick et al., 2021) (Sections 5.3.3 and
5.4.1). We also reiterate the importance of clearly communicating the risks of
such technologies to prevent potential algorithmic misuse and highlight the
importance of open-sourcing such models to accelerate the discovery of
potential fairness issues.

##  7\. Conclusion

In this survey paper, we have investigated debiasing methods targeting
fairness-aware neural networks for language and vision research. We have
contextualized fairness and its relationship with biases and their possible
origins. We have presented the main metrics and evaluation measures for
assessing the level of fairness provided by models for computer vision and
natural language processing tasks, reviewing both application-specific and
general-purpose measures, their proper use, applicability, and known
limitations. Then, we have discussed, in depth, several debiasing methods for
neural models under the perspective of a new taxonomy for the area, which is
yet another contribution of this paper. We concluded with our thoughts on the
most pressing fairness challenges in neural networks, calling attention for
potential trends and future research directions.

We should point readers to Table 1, which facilitates the identification of
research gaps and potential saturation for specific categories of methods.
Following our proposed taxonomy, certain categories of methods have no
proposed approaches for specific modality types, which could point to low-
hanging fruits. Considering the prevalence of Foundation Models and the
fairness problems they currently present, we also urge readers to dedicate
significantly more effort into debiasing large-scale models.

We hope this survey enables researchers to quickly understand the issues we
are facing, and we stress that it is not enough simply providing new neural
models without paying attention to the potential harms and consequences that
such models may have on underprivileged groups or individuals.

###### Acknowledgements.

This work was funded by Motorola Mobility Brazil.

## References

  * (1)
  * Abbasnejad et al. (2020) Ehsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Shi, and Anton van den Hengel. 2020.  Counterfactual vision and language learning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10044–10054.
  * Adeli et al. (2019) Ehsan Adeli, Qingyu Zhao, Adolf Pfefferbaum, Edith V. Sullivan, Li Fei-Fei, Juan Carlos Niebles, and Kilian M. Pohl. 2019.  Bias-Resilient Neural Network.  _ArXiv_ abs/1910.03676 (2019).
  * Aekula et al. (2021) Haswanth Aekula, Sugam Garg, and Animesh Gupta. 2021\.  [RE] Double-Hard Debias: Tailoring Word Embeddings for Gender Bias Mitigation.  _CoRR_ abs/2104.06973 (2021).  arXiv:2104.06973 https://arxiv.org/abs/2104.06973
  * Agarwal et al. (2022) Sharat Agarwal, Sumanyu Muku, Saket Anand, and Chetan Arora. 2022\.  Does Data Repair Lead to Fair Models? Curating Contextually Fair Data To Reduce Model Bias. In _2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. IEEE.
  * Amini et al. (2019) Alexander Amini, Ava P. Soleimany, Wilko Schwarting, Sangeeta N. Bhatia, and Daniela Rus. 2019\.  Uncovering and mitigating algorithmic bias through learned latent structure.  _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ (2019), 289 – 295.
  * Ashurst et al. (2022) Carolyn Ashurst, Emmie Hine, Paul Sedille, and Alexis Carlier. 2022.  AI Ethics Statements: Analysis and Lessons Learnt from NeurIPS Broader Impact Statements. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. 2047–2056.
  * Bansal (2022) Rajas Bansal. 2022\.  A Survey on Bias and Fairness in Natural Language Processing.  _arXiv preprint arXiv:2204.09591_ (2022).
  * Barocas et al. (2019) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2019\.  _Fairness and Machine Learning_.  fairmlbook.org.  http://www.fairmlbook.org.
  * Bellamy et al. (2019) Rachel KE Bellamy et al. 2019\.  AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias.  _IBM Journal of Research and Development_ 63, 4/5 (2019), 4–1.
  * Bender and Friedman (2018) Emily M Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ 6 (2018), 587–604.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 610–623.
  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013\.  Representation Learning: A Review and New Perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 35, 8 (Aug 2013), 1798–1828.
  * Benjamin et al. (2019) Misha Benjamin, Paul Gagnon, Negar Rostamzadeh, Chris Pal, Yoshua Bengio, and Alex Shee. 2019\.  Towards standardization of data licenses: The montreal data license.  _arXiv preprint arXiv:1903.12262_ (2019).
  * Berg et al. (2022) Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain. 2022\.  A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.  _arXiv preprint arXiv:2203.11933_ (2022).
  * Bird et al. (2020) Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. 2020.  _Fairlearn: A toolkit for assessing and improving fairness in AI_.  Technical Report. Microsoft.
  * Bird et al. (2019) Sarah Bird, Krishnaram Kenthapadi, Emre Kiciman, and Margaret Mitchell. 2019.  Fairness-Aware Machine Learning: Practical Challenges and Lessons Learned _(WSDM ’19)_. Association for Computing Machinery, New York, NY, USA, 834–835.
  * Black et al. (2022) Sid Black et al. 2022\.  GPT-NeoX-20B: An Open-Source Autoregressive Language Model.  _CoRR_ abs/2204.06745 (2022).
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of” bias” in nlp.  _arXiv preprint arXiv:2005.14050_ (2020).
  * Blyth (1972) Colin R. Blyth. 1972\.  On Simpson’s Paradox and the Sure-Thing Principle.  _J. Amer. Statist. Assoc._ 67, 338 (1972), 364–366.
  * Bogen and Rieke (2018) Miranda Bogen and Aaron Rieke. 2018.  Help wanted: An examination of hiring algorithms, equity, and bias.  _Upturn, December_ 7 (2018).
  * Bolukbasi et al. (2016b) Tolga Bolukbasi et al. 2016b.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In _Advances in Neural Information Processing Systems_ , Daniel D. Lee et al. (Eds.). 4349–4357.
  * Bolukbasi et al. (2016a) Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. 2016a.  Quantifying and Reducing Stereotypes in Word Embeddings.  _CoRR_ abs/1606.06121 (2016).  arXiv:1606.06121 http://arxiv.org/abs/1606.06121
  * Bommasani et al. (2021) Rishi Bommasani et al. 2021\.  On the Opportunities and Risks of Foundation Models.  _ArXiv e-prints_ 2108.07258 (Aug 2021), 212\.
  * Bordia and Bowman (2019) Shikha Bordia and Samuel R. Bowman. 2019.  Identifying and reducing gender bias in word-level language models.  _North American Chapter of the Association for Computational Linguistics_ (2019), 7 – 15\.
  * Brown et al. (2020) Tom B. Brown et al. 2020\.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , Hugo Larochelle et al. (Eds.).
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In _Conference on Fairness, Accountability and Transparency_ , Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 77–91.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017\.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ 356, 6334 (2017), 183–186.
  * Cao et al. (2022) Yushi Cao et al. 2022\.  Fair and accurate age prediction using distribution aware data curation and augmentation. In _IEEE/CVF Winter Conference on Applications of Computer Vision, WACV_. IEEE, 2867–2877.
  * Caton and Haas (2020) Simon Caton and Christian Haas. 2020.  Fairness in machine learning: A survey.  _arXiv preprint arXiv:2010.04053_ (2020).
  * Chen et al. (2020) Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. 2020\.  Counterfactual samples synthesizing for robust visual question answering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10800–10809.
  * Chen et al. (2022) Xiuying Chen, Mingzhe Li, Rui Yan, Xin Gao, and Xiangliang Zhang. 2022.  Unsupervised Mitigation of Gender Bias by Character Components: A Case Study of Chinese Word Embedding.  _Workshop on Gender Bias in Natural Language Processing_ (2022), 121 – 128.
  * Chen and Joo (2021) Yunliang Chen and Jungseock Joo. 2021.  Understanding and Mitigating Annotation Bias in Facial Expression Recognition. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE.
  * Cheng et al. (2021) Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. 2021.  Fairfil: Contrastive neural debiasing method for pretrained text encoders.  _arXiv preprint arXiv:2103.06413_ (2021).
  * Cho et al. (2022) Jaemin Cho, Abhay Zala, and Mohit Bansal. 2022.  DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers.  https://arxiv.org/abs/2202.04053
  * Choi et al. (2020) Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. 2020.  Fair Generative Modeling via Weak Supervision. In _ICML_. 1887–1898.
  * Chouldechova (2017) Alexandra Chouldechova. 2017\.  Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.  _Big Data_ 5, 2 (2017), 153–163.
  * Cohen et al. (2019) Lee Cohen, Zachary C. Lipton, and Yishay Mansour. 2019\.  Efficient candidate screening under multiple tests and implications for fairness.  arXiv:1905.11361
  * Council of European Union (2016) Council of European Union. 2016\.  European Union General Data Protection Regulation, Article 22: “Automated individual decision-making, including profiling”.  https://www.privacy-regulation.eu/en/article-22-automated-individual-decision-making-including-profiling-GDPR.htm.
  * Cramer et al. (2018) Henriette Cramer, Jean Garcia-Gathright, Aaron Springer, and Sravana Reddy. 2018.  Assessing and addressing algorithmic bias in practice.  _Interactions_ 25, 6 (2018), 58–63.
  * Crawshaw (2020) Michael Crawshaw. 2020\.  Multi-task learning with deep neural networks: A survey.  _arXiv preprint arXiv:2009.09796_ (2020).
  * Creager et al. (2019) Elliot Creager et al. 2019\.  Flexibly Fair Representation Learning by Disentanglement. In _International Conference on Machine Learning_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 1436–1445.
  * Dash et al. (2022) Saloni Dash, Vineeth N. Balasubramanian, and Amit Sharma. 2022\.  Evaluating and Mitigating Bias in Image Classifiers: A Causal Perspective Using Counterfactuals.  _IEEE/CVF Winter Conference on Applications of Computer Vision, WACV_ (2022), 3879 – 3888\.
  * De-Arteaga et al. (2019) Maria De-Arteaga et al. 2019\.  Bias in bios: A case study of semantic representation bias in a high-stakes setting. In _Conference on Fairness, Accountability, and Transparency_. 120–128.
  * Delobelle et al. (2021) Pieter Delobelle, Ewoenam Kwaku Tokpo, Toon Calders, and Bettina Berendt. 2021.  Measuring fairness with biased rulers: A survey on quantifying biases in pretrained language models.  _arXiv preprint arXiv:2112.07447_ (2021).
  * Derman (2021) Ekberjan Derman. 2021\.  Dataset Bias Mitigation Through Analysis of CNN Training Scores.  _CoRR_ abs/2106.14829 (2021).  arXiv:2106.14829 https://arxiv.org/abs/2106.14829
  * Dev et al. (2020) Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2020\.  On measuring and mitigating biased inferences of word embeddings. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 7659–7666.
  * Dev et al. (2021) Sunipa Dev, Tao Li, Jeff M. Phillips, and Vivek Srikumar. 2021\.  OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings. In _Conference on Empirical Methods in Natural Language Processing_ , Marie-Francine Moens et al. (Eds.). 5034–5050.
  * Dev and Phillips (2019) Sunipa Dev and Jeff M. Phillips. 2019.  Attenuating Bias in Word vectors. In _International Conference on Artificial Intelligence and Statistics_ , Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 879–887.
  * Du et al. (2021a) Mengnan Du, Fan Yang, Na Zou, and Xia Hu. 2021a.  Fairness in Deep Learning: A Computational Perspective.  _IEEE Intelligent Systems_ 36 (2021), 25–34.
  * Du et al. (2021b) Mengnan Du, Fan Yang, Na Zou, and Xia Hu. 2021b.  Fairness in Deep Learning: A Computational Perspective.  _IEEE Intelligent Systems_ 36, 4 (July 2021), 25–34.
  * Du et al. (2022) Siyi Du, Ben Hers, Nourhan Bayasi, Ghassan Hamarneh, and Rafeef Garbi. 2022.  FairDisCo: Fairer AI in Dermatology via Disentanglement Contrastive Learning.  arXiv:2208.10013 [cs.CV]
  * Du and Joseph (2020) Yuhao Du and Kenneth Joseph. 2020.  MDR Cluster-Debias: A Nonlinear WordEmbedding Debiasing Pipeline.  _CoRR_ abs/2006.11642 (2020).  arXiv:2006.11642 https://arxiv.org/abs/2006.11642
  * Dunkelau and Leuschel (2019) Jannik Dunkelau and Michael Leuschel. 2019.  Fairness-Aware Machine Learning: An Extensive Overview.  https://www3.hhu.de/stups/downloads/pdf/fairness-survey.pdf
  * Dwork and Ilvento (2018) Cynthia Dwork and Christina Ilvento. 2018.  Fairness Under Composition.  (2018).
  * Edwards and Storkey (2016) Harrison Edwards and Amos J. Storkey. 2016.  Censoring Representations with an Adversary. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).
  * Faal et al. (2022) Farshid Faal, Ketra A. Schmitt, and Jia Yuan Yu. 2022\.  Reward Modeling for Mitigating Toxicity in Transformer-based Language Models.  _CoRR_ abs/2202.09662 (2022).  arXiv:2202.09662 https://arxiv.org/abs/2202.09662
  * Fatemi et al. (2021) Zahra Fatemi, Chen Xing, Wenhao Liu, and Caiming Xiong. 2021\.  Improving gender fairness of pre-trained language models without catastrophic forgetting.  _arXiv preprint arXiv:2110.05367_ (2021).
  * Fleisig and Fellbaum (2022) Eve Fleisig and Christiane Fellbaum. 2022.  Mitigating Gender Bias in Machine Translation through Adversarial Learning.  https://arxiv.org/abs/2203.10675
  * Friedler et al. (2016) Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016\.  On the (im) possibility of fairness.  _arXiv preprint arXiv:1609.07236_ (2016).
  * Gaci et al. (2022) Yacine Gaci, Boualem Benatallah, Fabio Casati, and Khalid Benabdeslem. 2022.  Iterative adversarial removal of gender bias in pretrained word embeddings.  _Proceedings of the ACM Symposium on Applied Computing_ (2022), 829 – 836.
  * Gajane and Pechenizkiy (2017) Pratik Gajane and Mykola Pechenizkiy. 2017.  On Formalizing Fairness in Prediction with Machine Learning.  https://arxiv.org/abs/1710.03184
  * Gambs and Ngueveu (2022) Sébastien Gambs and Rosin Claude Ngueveu. 2022.  Fair mapping.  https://arxiv.org/abs/2209.00617
  * Garcia-Gathright et al. (2018) Jean Garcia-Gathright, Aaron Springer, and Henriette Cramer. 2018\.  Assessing and Addressing Algorithmic Bias - But Before We Get There.  https://doi.org/10.48550/ARXIV.1809.03332
  * Garrido-Muñoz et al. (2021) Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and L Alfonso Ureña-López. 2021.  A survey on bias in deep NLP.  _Applied Sciences_ 11, 7 (2021), 3184.
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021\.  Datasheets for datasets.  _Commun. ACM_ 64, 12 (2021), 86–92.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020.  RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In _Findings of the Association for Computational Linguistics_. 3356–3369.
  * George Zerveas and Eickhoff (2022) Daniel Cohen George Zerveas, Navid Rekabsaz and Carsten Eickhoff. 2022.  Mitigating Bias in Search Results Through Contextual Document Reranking and Neutrality Regularization. In _International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2532–2538.
  * Gira et al. (2022) Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022\.  Debiasing Pre-Trained Language Models via Efficient Fine-Tuning. In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_. 59–69.
  * Gong et al. (2019) Ting Gong et al. 2019\.  A comparison of loss weighting strategies for multi task learning in deep neural networks.  _IEEE Access_ 7 (2019), 141627–141632.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets. In _Advances in Neural Information Processing Systems_. 2672–2680.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and Harnessing Adversarial Examples. In _International Conference on Learning Representations_.
  * Google (2020) Google. 2020.  ML-fairness-gym: A Tool for Exploring Long-Term Impacts of Machine Learning Systems.  https://ai.googleblog.com/2020/02/ml-fairness-gym-tool-for-exploring-long.html
  * Gronowski et al. (2022) Adam Gronowski, William Paul, Fady Alajaji, Bahman Gharesifard, and Philippe Burlina. 2022.  Achieving Utility, Fairness, and Compactness via Tunable Information Bottleneck Measures.  https://arxiv.org/abs/2206.10043
  * Guidotti et al. (2018) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018.  A survey of methods for explaining black box models.  _ACM computing surveys (CSUR)_ 51, 5 (2018), 1–42.