11institutetext: University of Calicut, Kerala, India 11email:
<EMAIL_ADDRESS>22institutetext: Queen’s University
Belfast, UK
22email<EMAIL_ADDRESS>
# Towards an Enhanced Understanding of Bias in Pre-trained Neural Language
Models: A Survey with Special Emphasis on Affective Bias

Anoop K Corresponding author 11 Manjary P Gangan  Supported by the Women
Scientist Scheme-A (WOS-A) for Research in Basic/Applied Science from the
Department of Science and Technology (DST) of the Government of India under
the Grant SR/WOS-A/PM-62/201811 Deepak P 22 Lajish V L 11

###### Abstract

The remarkable progress in Natural Language Processing (NLP) brought about by
deep learning, particularly with the recent advent of large pre-trained neural
language models, is brought into scrutiny as several studies began to discuss
and report potential biases in NLP applications. Bias in NLP is found to
originate from latent historical biases encoded by humans into textual data
which gets perpetuated or even amplified by NLP algorithm. We present a survey
to comprehend bias in large pre-trained language models, analyze the stages at
which they occur in these models, and various ways in which these biases could
be quantified and mitigated. Considering wide applicability of textual
affective computing based downstream tasks in real-world systems such as
business, healthcare, education, etc., we give a special emphasis on
investigating bias in the context of affect (emotion) i.e., Affective Bias, in
large pre-trained language models. We present a summary of various bias
evaluation corpora that help to aid future research and discuss challenges in
the research on bias in pre-trained language models. We believe that our
attempt to draw a comprehensive view of bias in pre-trained language models,
and especially the exploration of affective bias will be highly beneficial to
researchers interested in this evolving field.
The examples provided in this paper may be offensive in nature and may hurt
your moral beliefs.

###### Keywords:

NLP Bias Fairness Large Pre-trained Language Models Affective Bias Affective
Computing.

##  1 Introduction

Natural Language Processing (NLP) has recently achieved rapid progress with
the aid of deep learning, especially Pre-trained Language Models (PLM) [50].
Large PLMs like BERT [31], GPT [88], etc., are highly efficient at capturing
linguistic properties and producing representations of text with semantic and
contextual information. Inclusion of contextual representations has led large
PLMs to become popular towards addressing many downstream tasks such as
Question Answering, Sentiment Analysis, Neural Machine Translation, etc [87].
These data greedy Language Models (LM) are generally trained on large-scale
human generated textual corpora. However, since ancient days, language has
functioned as a channel to express and propagate unfairness towards
marginalized social groups and assign power to oppressive institutions [29].
It is often very hard to analyze the quality of data in large corpora in
context of such oppressive nature of language [118]. Yet, these human
generated textual corpora can carry plenty of harmful linguistic biases and
social stereotypes that can lead NLP algorithms to produce unfair
discrimination towards socially marginalized populations when deployed in
real-word [78]. A threatening scenario that was identified with the use of
large PLM GPT-3 [19] has been experimentally demonstrated in [2], for example,
‘Two Muslims walked into a  ’, is completed by GPT3 with ‘synagogue with axes
and a bomb’ and ‘gay bar in Seattle and started shooting at will, killing five
people’. This is evidently discriminatory and is probably due to islamophobia
manifesting in the training text.

###  1.1 Potential Harms of Bias in NLP

Bias in NLP could perpetuate harms towards marginalized populations of society
in different ways. Allocational and representational harms are the prominent
ones engendered by the existence of biased discrimination and stereotypes in
language [118]. Allocational harms deny opportunities and resources across
marginalized social groups (e.g. recidivism prediction
system111https://www.propublica.org/article/machine-bias-risk-assessments-in-
criminal-sentencing?token=nD-X136˙tDm0nh1l4Xtv0LbpjY˙BSO3u), whereas
representational harms generate falsifications of these groups (e.g. caste and
religion based discrimination [96]). Harms are also brought by the
exclusionary social norms in language [118]. For example, the social norms of
‘family’ is normally conveyed by humans as a basic social unit consisting of a
married woman, man and their children; language models internalizing such
social norms often end up being highly discriminatory towards people who live
outside the institution of these social norms. Another nuanced notion of
linguistic harm is detecting certain languages of marginalized or
underrepresented groups as toxic in hate speech detection, since there is no
precise universally admissible definition for
toxicity222https://spectrum.ieee.org/in-2016-microsofts-racist-chatbot-
revealed-the-dangers-of-online-conversation. Biased representation of emotions
in language leads to another linguistic harm, affective harm, that
discriminate marginalized social groups on the basis of certain emotions e.g.
the ‘angry black woman’ stereotype [75]. If PLMs are learned from a corpus
that have latent male chauvinist events, the NLP systems that use them may
exhibit affective harms towards females. Other types of linguistic harms
include performance drop for certain social groups and languages, generation
of nonsensical data and misinformation, etc. [118].

###  1.2 Heterogeneous View of Bias in PLMs

Bias in pre-trained language models can be viewed through different
perspectives, domains of bias and stages in which they occur. We illustrate
this heterogeneous view of PLM biases in figure 1. Bias in PLMs may be seen as
belonging to two categories, viz., descriptive and stylistic. Descriptive
biases arise from discrimination or marginalization in associating identities
to certain concepts or properties based on textual semantics, e.g. word
embeddings associate father to doctor and mother to nurse [16]. Stylistic
biases originate due to stylistic differences in texts with same content but
generated by different socio-economic groups [100], e.g. unfair treatment to
African American English while using language identification tools and
dependency parsers [15]. Bias in PLMs are analyzed in various domains, either
primary analysis of bias with respect to the domains such as gender, race,
ethnicity, age, profession, etc., or analyzing intersectional bias by
considering a combination of multiple domains such as religion+gender (e.g.
Muslim lady), race+gender (e.g. black woman), etc. Table 1 shows works in the
literature that explore bias with respect to different domains where a major
portion of works relate to the gender domain. When considering the stages at
which bias can occur in the context of large PLMs, data or/and algorithm
design are generally the two major stages. Bias in data can arise either or
both, from the pre-training or fine-tuning corpus. Algorithm bias may
originate from self-supervised learning algorithms that yield non-contextual
or contextual representations [16, 102] or/and from fine-tuning learning
algorithm designed for downstream tasks [47].

Figure 1: Heterogeneous view of bias in pre-trained language models Table 1: Different domains of bias in pre-trained language models Domain | Examples of Protected/Target groups | Work
---|---|---
Gender |  Male, Female, Gay, Lesbian |  [6, 9, 10, 11, 13, 14, 16, 17, 18, 25, 30] [39, 43, 48, 51, 52, 62, 66, 67, 76, 86] [94, 100, 102, 104, 108, 110, 113, 115] [116, 119, 122, 123, 125, 126, 129]
Race |  Black, White |  [51, 52, 76, 100, 110, 115]
Religion |  Jewish, Hindu, Muslim, Christian |  [2, 30, 51, 76, 94]
Profession |  Homemaker, Nurse, Architect |  [6, 14, 39, 53, 76]
Ethnicity |  Asian, Hispanic |  [3, 48, 58, 94]
Disability |  Sensory (blind), Neurodiverse (autistism), Psychosocial (schizophrenia) |  [51, 67, 115]
Age |  Old, Young |  [32, 51, 94]
Politics |  Conservative, Liberal |  [65, 94, 100]
Continent |  Africa, Asia, Oceania, Europe |  [6, 30, 43]
Nationality |  American, Italian |  [51, 98]
Physical appearance |  Short, Tall, Fat, Thin, Overweight |  [51, 94]
Socioeconomic status |  Poor, Rich, Homeless |  [51]
Intersectional |  Race+Gender (Black Women) |  [43, 53, 67, 110]

In this paper, we survey bias in NLP, especially in pre-trained neural
language models. We also give special attention to the less explored area of
social biases in the context of affect i.e., Affective Bias (or emotion
associated bias) specific to large PLMs. Since affective computing has
potential applications in many natural language understanding tools and real-
word systems (healthcare [44], business [54, 105], education [34, 105], etc.),
it is highly necessary to study the existence of affective biases, if any, in
these systems that could potentially harm or do injustice towards protected
social groups based on affect. We review more than 100 papers that address
bias in PLMs including non-contextual and contextual models. We collect
research papers from ACL anthology, Google Scholar and arXiv, using the
keywords ‘bias’, ‘fairness’, ‘bias in NLP’, ‘fairness in NLP’, ‘Sentiment
bias’, ‘Affective bias’, ‘Emotion bias’ ‘bias in pre-trained language models’,
etc. as the inclusion criteria for our survey.
The major contributions of this survey are summarised below:

  * –

We present a comprehensive survey of bias in pre-trained language models,
especially an in-depth treatment of various kinds of bias that originate in
transformer based contextual pre-trained language models in NLP along with
their identification, quantification and mitigation strategies.

  * –

We, for the first time, to the best of our knowledge, investigate Affective
Bias, a highly socially relevant and less addressed problem, specifically in
the context of large pre-trained language models.

  * –

We collect and present a large number of available bias evaluation corpora
along with their suitability to evaluate large pre-trained language models.

  * –

We also discuss present research challenges in large pre-trained language
models and affective biases.

The rest of the paper is organized as, the background of PLMs and bias in PLMs
provided in section 2, quantifying PLM bias in section 3, mitigating PLM bias
in section 4, affective bias in PLMs including their identification and
mitigation strategies in section 5, a list of available bias evaluation
corpora in section 6, research challenges in section 7 and concluding remarks
in section 8.

##  2 Pre-Trained Language Models

###  2.1 Background

Advancements in deep learning have brought NLP to a new era led by neural LMs
or large PLMs by producing effective representations for textual data, where
the dense and automatically extracted representations by PLMs from large
textual corpora override sparse and handcrafted representations (e.g. TF-IDF
feature) and their associated drawbacks. Hence, such PLM representations are
generally used universally as language representations for a variety of
downstream tasks in NLP to achieve significant state-of-the-art results while
avoiding the burden of training a new model from the scratch [87]. A
representation becomes powerful when it is capable of comprising general
purpose characteristics of the language and also useful to learn a variety of
tasks. Such a potent representation in the context of language should capture
the latent linguistic conventions and common sense knowledge that are hidden
in text such as syntax, semantics, pragmatics, etc. A step towards such
linguistic representation was the development of non-contextual embedding
models by mapping words into a distributed d-dimensional embedding space or
vector. The shallow architectures within that stream, such as Continuous Bag-
of-Word and Skip-Gram models (word2vec) developed by Mikolov et al. [74] from
unlabelled data formed initial attempts towards generic language
representations. Despite the simplicity in architecture, they are highly
capable of learning effective word embeddings that can capture hidden semantic
and syntactic similarities among words. Similar to the popular word2vec
architecture [74], GloVe [82] that utilizes word to word co-occurrence
statistics from corpora and FastText [49] that utilizes sub-word information
also attracted significant attention to solve many downstream tasks. However,
these embeddings are non-contextual in nature and hence they fail to capture
disambiguation, semantic roles, polysemous, syntactic structure, and anaphora,
which rely on higher-level contextual concepts. Many researchers proposed
different models that are capable of learning embeddings of sentences,
paragraphs, and even documents [59] despite critiques on not capturing
contextual representation of words.

Representations of words in documents are contextually dependent in nature
since similar words have different semantics in diverse contexts. Therefore,
replacing conventional non-contextualized embeddings, recent research has
presented a new generation of contextualized embedding models, such as ELMo
[83], BERT [31], etc., that have become increasingly common due to their
capability to describe linguistic phenomena such as polysemy. Unlike non-
contextualized representations, contextualized word representations are
generated using neural contextual encoders and have achieved state-of-the-art
performance in most NLP tasks over the conventional embeddings even though
their sophisticated nature dents interpretability. There are two major types
of neural encoders, sequence models that include convolutional and recurrent
models [83], and non-sequential models that include fully connected self-
attention and advanced transformer architectures [88, 31, 121]. Contextualized
representation models developed with Convolutional Neural Network (CNN) and
Recurrent Neural Network (RNN) face difficulty in modeling long term context
among other issues [50]. This brings a new kind of learning model, Transformer
[114], a fully self-attention based architecture that has much more
parallelization compared to RNN and also one that can effectively model longer
dependencies and context from textual data. To build any kind of LMs,
collecting labeled data and supplying it to supervised learning models is
tedious. But, on the positive side, the benefits of LM are often realizable
through self-Supervised learning, a new learning paradigm applied to this
scenario that utilizes plenty of available unlabelled text data to learn
highly generic knowledge representations by automatically generating labels
from training data itself based on pseudo supervision [87] e.g. masked
language model [31]. The prior non-contextualized pre-trained word embeddings
(e.g., word2vec and GloVe) do not give much importance to its utility in other
downstream tasks and applicability of fine-tuning strategy. Besides universal
contextual word representations, contextualized PLMs such as BERT [31], GPT
[88], XLNet [121], etc., is useful to build models that perform better on many
downstream NLP tasks by fine-tuning the pre-trained model, crucially avoiding
the burden of training the model from the scratch for each downstream task.
The auto-encoding pre-trained architecture BERT [31] uses masked language
model approach and overcomes the limitation of unidirectional autoregressive
models like GPT [88] by enabling bidirectional contexts, but unavailability of
mask in fine-tuning introduces pretrain-finetune discrepancy. XLNet [121], a
generalized autoregressive pre-training model at the same time achieves better
results by introducing random permutations to enable bi-directional context.
There also exists a large number of different domain specific (e.g.,
biomedical [60], finance [120], etc.), mono and multilingual PLMs that vary
based on their architecture or pre-training tasks.

###  2.2 Bias in Pre-trained Language Models

Even though word representations are powerful enough to capture semantic
similarities and exhibit word relationships through word vector similarities,
the explicit and implicit existence of several stereotypes and social biases
in PLMs harm its usefulness in many real-world applications. Bias in large
PLMs arises from different stages of their developmental process. Figure 2
illustrates the workflow of large PLMs along with possible stages where bias
may originate, particularly focussing on recent Transformer based PLMs. To
mitigate bias it is essential to understand and disentangle the various
sources of bias. The investigation on sources of bias leads to observations
that human language that form today’s data deluge, big enough to train data
greedy NLP algorithms, historically accumulate several severe stereotypes and
social biases that pervade society i.e., Historical Bias. Language hence is
one of the most potent ways through which societal biases are brought about,
propagated, and echoed [72]. Several non-neutral stereotypes live in
linguistic communication, imaging asymmetries in terms of dominance, power,
quality, or status among target terms such as female and male, blacks and
whites belonging to various domains like gender, race, etc., [36]. Taking it
for granted as normal, people rehearse most of these preconceptions in day-to-
day discourse, consequently routinizing these linguistic discrimination and
making them be felt less visible [77]. Therefore, even though we perfectly
measure and take data samples from these historical data repositories, these
are ridden with biases, i.e., Data Bias, a representative of historical bias,
which thereby brings about bias in PLMs [107].

Data bias stemming from innate historical biases is the most general source of
bias among different sources of bias explored in literature for various tasks
[28], where, quality issues in data, uneven distribution (occurrence or co-
occurrences) of key terms in data associated with targets concerning a domain
[18], etc., are other factors that contribute towards it. Standard datasets
used for pre-training non-contextual [16, 22, 40, 68] and contextual models
[110] are found to exhibit bias or imbalance in various domains like gender,
race, etc. In the context of large PLMs, data bias may be Pre-trained Data
Bias at the initial training process of PLM or/and Fine-tuning Data Bias which
is just downstream to it. Studies report that data biases can propagate and
get further amplified by underlying machine learning models leading to Model
Learning Bias at self-supervised learning strategy to learn linguistic
properties and Downstream Task Learning Bias at the level of the task specific
fine-tuning model.

Figure 2: Bias in large pre-trained language models

Model learning bias is reflected in word representations derived from PLMs and
produce Representation Bias. Non-contextual word embeddings such as word2vec,
GloVe, etc., are known to comprise representational biases across gender [16,
22], racial [68] and ethnic groups [40]. It has been clearly unveiled that
these embeddings relate word representations of professions like ‘nurse’,
‘receptionist’ and ‘homemaker’ to women and ‘doctor’, ‘philosopher’ and
‘computer programmer’ to men [16]. Further, they place the representation of
words like ‘wisdom’ close to representation of ‘grandfather’ than
‘grandmother’, and encode association of popular African Americans names with
unpleasant phrases [22], etc. Examination of bias in contextualized embeddings
reveal that they also exhibit bias like conventional embeddings [18, 126]. For
example, BERT is found to encode human-like biases [55]. Most of the recent
NLP information retrieval systems such as search engines, question answering,
etc. highly rely on these biased representations consequently leading to
highly biased retrieval behavior. Similarly, use of the large PLM GPT-3 in
language generation shows religion bias analogizing ‘Muslims’ to ‘Terrorists’
[2]. All such biases are disturbing when one observes that word embeddings,
being the base constituent of most language systems, can propagate or even
intensify them [22, 61, 127] causing unfavorable outcomes when deployed in a
plethora of downstream applications such as sentiment analysis [79], language
generation [66], toxic language detection [48], etc.

Downstream task learning bias can cause their outcomes delivered to the public
to finally end up in socio-economic exclusions reinforcing harmful societal
stereotypes [16, 61, 127]. Since downstream applications are generally
implemented by initializing learning models with an existing source network
representation pre-trained on large datasets and later fine-tuned using
datasets that suit downstream target task, pre-training data bias, pre-trained
representation bias and fine-tuning data bias all can be the sources to induce
bias in downstream applications. Sentiment analysis [52], abusive language
detection [81], text classification [33], machine translation [38, 112],
personalized medicine [90], coreference resolution [95, 128], crime recidivism
prediction systems [27], automating resume screening [56], online
advertisements delivery [57, 109], etc., are some of the downstream
applications that reports bias in various domains. Inappropriate or unfair
choice of label usages to fine-tune downstream task is another source of bias
i.e., Measurement Bias [107].

These biases can be distinguished as Intrinsic Bias if it occurs in pre-
trained learning or Extrinsic Bias if it occurs in downstream task modeling.
Besides above mentioned biases, in the perspective of real-world machine
learning models, the final system must consider Evaluation Bias that occurs
when benchmark dataset for a task doesn’t represent certain groups (e.g.,
images of non-white women not being identified by the model) and Deployment
Bias that occurs due incompatibility of a model designed for a particular task
when used differently (e.g., using risk assessment tool created to predict
future crime for the purpose of determining length of sentence) [107]. Table
LABEL:tab:lm_works shows a large set of research that identify and mitigate
bias from large PLMs, where most of them (around 53%) focus primarily on bias
identification.

Table 2: Works addressing bias in large pre-trained language models Work | PLM | Domain | Quantification | Mitigation
---|---|---|---|---
[102] |  BERT, XLNet, ALBERT, DistilBERT, RoBERTa, GPT-2 |  Gender |  WEAT, sequence likelihood, pronoun ranking |  WEAT scores as an additional loss regularizer
[3] |  BERT |  Ethnicity |  Normalized probability, Categorical bias score |  M-BERT, Contextual Word Alignment
[113] |  DistilBERT, RoBERTa, XLM-RoBERTa, ALBERT, BERT |  Gender |  Skew and stereotype matrices |  Data augmentation
[39] |  BERT |  Profession, Gender |  Model querying to predict pronouns at masked position, given context |  GEnder Equality Prompt
[13] |  BERT |  Gender |  MLP regressor |  Removing word vector components from gender directions
[65] |  GPT-2 |  Politics |  Indirect and Direct bias metrics |  Reinforcement Learning
[48] |  RoBERTa |  Ethnicity, Gender |  Classification Performance, Group identifier bias metrics, African American Vernacular English, Dialect Bias Metrics, Gender Stereotype Metrics |  Upstream Bias Mitigation
[98] |  GPT-2, T5 |  Occupation, Nationality, Religion, Gender, Age, Race, Disability |  Self-Diagnosing |  Self-Debiasing
[2] |  GPT-3 |  Religion |  Prompt completion, analogical reasoning, story generation | —
[104] |  XLM, XLM-RoBERTa, BERT multilingual |  Gender |  Point-wise mutual information and its extension with latent sentiment & regularization | —
[51] |  BERT, RoBERTa, ALBERT |  Gender, Race, Sexual orientation, Religion, Nationality, Disability, Age, Physical appearance, Socio-economic status |  All Unmasked Likelihood (AUL), AUL with Attention weights | —
[11] |  EMLo |  Gender |  Gender direction in gender subspace | —
[123] |  BERT |  Gender |  Prediction probability scores | —
[43] |  ELMo, GPT, BERT, GPT-2 |  Intersectional |  Intersectional Bias Detection, Emergent Intersectional Bias Detection, CEAT | —
[67] |  DistilBERT, GPT-2, GPT-NEO |  Intersectional bias |  Difference/similarity between sentiment scores | —
[119] |  BERT, GPT-2, T5, XLNet |  Low frequency names |  SV-WEAT, Intra and Inter layer self-similarity score | —
[62] |  BERT |  Gender |  BERT Attention maps | —
[53] |  GPT-2 |  Intersectional |  Predictions of GPT-2 | —
[76] |  BERT, RoBERTa, XLNET, GPT2 |  Profession, Gender, Religion, Race |  Language modelling score, Stereotype score, iCAT | —
[66] |  LSTM |  Gender |  Bias Score, Intervention matches |  Counterfactual Data Augmentation, Word Embedding Debiasing
[125] |  BERT pre-trained on medical notes |  Ethnicity, Gender, Insurance groups |  Log probability score, extrinsic evaluation using downstream tasks | —
[9] |  BERT |  Profession, Gender |  Association test |  Name based Counterfactual Data Substitution
[116] |  GPT-2 |  Gender |  Causal Mediation Analysis | —
[110] |  GPT-2, CBoW-GLoVe, ELMo, GPT, BERT |  Intersectional, Gender, Race |  Counting occurrences, Modified SEAT | —
[55] |  BERT |  Gender |  Log Probability Bias Score, WEAT | —
[126] |  ELMo |  Gender |  Co-occurrence |  Train-time data augmentation, test-time neutralization
[86] |  LSTM |  Gender |  Occurrences, causal testing, Euclidean distance |  Loss function modification
[18] |  LSTM |  Gender |  Fixed & infinite context bias scores |  Loss function modification
[10] |  ELMo |  Gender |  Direct Bias, Biased words clustering & classification | —
[47] |  Transformer-XL |  Country, Occupation, Name |  Individual & group fairness metrics from sentiment scores |  Embedding & Sentiment regularization

##  3 Quantifying Bias in PLMs

In order to study unfavorable consequences of different sources of biases in
various domains, it is necessary to quantify bias in some manner. Based on the
stages of occurrence, bias are generally quantified in the corpus,
representation and downstream tasks.

###  3.1 Quantifying bias in corpora

Counting occurrences or co-occurrences of key terms and deriving various
statistics from them helps to discover bias in corpora. Bordia and Bowman [18]
study gender bias in three publicly available datasets, viz., Penn Treebank
[69], WikiText-2 [73] and CNN/Daily Mail [46] that are used to build language
models by finding bias scores built using word-level probability profiles
within the context of gendered words by defining fixed and infinite sized
context windows around the gendered words. Their bias score helps to find
whether words more frequently co-occur with female or male gendered words. For
fixed sized context windows, they find optimal sized windows as smaller
windows can focus more on target words whereas larger windows can focus on the
broader topic. Infinite sized windows are much more stable by using
exponentially diminishing weights as the distance between key terms and
gendered words increases. Tan et al. [110] count occurrences of key terms
(e.g. female or male pronouns) and their co-occurrence with stereotypically
gendered occupation terms and perform statistical analysis to find gender bias
and also racial and intersectional biases in 1 Billion Word Benchmark [26],
BookCorpus [131], Wikipedia, and WebText [89] datasets used to pre-train
contextual word models.

###  3.2 Quantifying bias in representations

####  3.2.1 Geometry of vector spaces

Certain works quantify bias by analyzing subspaces in embeddings. Bolukbasi et
al. [16] demonstrates the occurrence of gender bias in word embeddings by
viewing the difference between word vectors of gendered words like ‘sister’,
‘brother’, ‘grandmother’, ‘grandfather’, etc., and gender neutral words. They
observe a direction that essentially captures gender and projecting gender
neutral words in this direction helps them to quantify gender bias. Manzini et
al. [68] extends this approach to suit multiclass settings and demonstrate
that word representations exhibit several stereotypical biases including
religion based and racial bias.

####  3.2.2 Word association test

Word association tests to quantify representation bias are inspired from the
Implicit Association Tests (IAT) in psychology [42] that tries to understand
human subconscious bias by measuring differences in the association of target
concepts with an attribute. Caliskan et al. [22] proposes a statistical test
called the Word Embedding Association Test (WEAT) analogous to IAT to quantify
bias in non-contextual word embeddings GloVe and Word2Vec. Using WEAT, authors
examine the similarity of embeddings of words in complementary categories like
European American and African American names with the complementary attributes
like pleasant and unpleasant attributes. The dissimilarity between the
association of European American names with the attributes when compared to
African American names with the same attributes, helps their study to report
the existence of human-like implicit bias in embeddings. To test bias in
sentence encoders like ELMo and BERT, May et al. [70] proposes a
generalization of WEAT named the Sentence Encoder Association Test (SEAT).
Using SEAT, even though they could verify presence of bias in these
embeddings, the results were not very generalizable. They also point out that
dissimilarities in the results don’t mean contextual embeddings is free of
bias, rather it may be an indication of cosine similarity not suitable to
measure the similarity of embeddings of recent contextual models, and hence an
alternative might be required to quantify bias in such representations. Kurita
et al. [55] also shows that conventional cosine similarity based methods don’t
produce consistent results to find bias in sentence embeddings generated from
the contextual models, as the embeddings of the words may differ according to
the context and state of the language model. As the former tests concentrate
only on individual words and predefined stereotypical attributes like pleasant
and unpleasant terms in artificial contexts that don’t reflect the natural use
of words, Nadeem et al. [76] proposes two Context Association Tests (CAT) to
intrinsically estimate bias in a set of large PLMs. They perform CAT at the
sentence level and discourse level, where each target term has natural
context, and observe that the recent contextual models BERT, GPT2, RoBERTa,
and XLNet exhibit strong biases with respect to their language modeling
ability. Building upon WEAT, Guo and Caliskan [43] propose Contextualized
Embedding Association Test (CEAT) to confirm and extensively quantify biases
in neural language models ELMo, BERT, GPT and GPT-2 according to different
contexts.

###  3.3 Quantifying bias in downstream tasks

In this category, bias is quantified by checking performance scores of system
over evaluation corpus that differs only in the context of target terms in
which domain of bias is being studied. For example, gender swapping to change
gender of gendered words, like ‘She is here’ to ‘He is here’, and then
evaluating model performance of these two sentences [52, 66, 95, 128]. The
system exhibits gender bias if it produces different performance scores for
both sentences that only differ in gendered words. Dixon et al. [33] presents
two performance evaluation metrics derived from error rate equality difference
to quantify bias in text classifier constructed to identify toxic comments.
Park et al. [81] utilizes these metrics to quantify bias along with the method
to generate gender unbiased dataset proposed by Dixon et al. [33] to find
gender bias in abusive language detection. Kiritchenko and Mohammad [52]
utilize difference in predicted intensity scores of sentences that differ in
gendered words and race, over their corpus to statistically evaluate gender
and racial bias in 219 sentiment analysis models that took part in SemEval
2018: Task 1 Affect In Tweets. Zhao et al. [128] illustrates gender bias in
coreference systems using F1 score over their evaluation corpus named WinoBias
that contains pair of sentences that associate gendered pronouns (her/him) to
various female or male stereotypical occupations (secretary/physician).
Rudinger et al. [95] also perform a similar gender bias study in coreference
system over pairs of sentences that differ only in gendered words. Lu et al.
[66] quantifies gender bias in coreference resolution and language modeling by
calculating dissimilarity in performances of gendered words with various
occupations across pairs of sentences that are gender swapped. In addition,
bias can also be measured using interpretability, by investigating model
interpretations on how it reaches to certain decisions or predictions [35].

##  4 Mitigating Bias in PLMs

Various efforts have been made for mitigating bias in PLMs to reduce or remove
their discriminatory influences over underrepresented or non-mainstream
groups. We classify debiasing approaches into three categories, based on the
stages at which they are addressed.

###  4.1 Mitigating bias in training corpora

Most computational algorithms designed for extrinsic applications like
sentiment analysis, text classification, etc., primarily rely on labeled
training corpora making them prone to societal biases present in these corpora
[111]. Despite this being the case, such corpora are still being extensively
utilized in various applications of NLP as it is expensive or labor-intensive
to build new large-sized training corpora. Hence, generally, common techniques
such as data augmentation and bias fine-tuning are followed for debiasing
training corpora.

####  4.1.1 Data augmentation

Data augmentation techniques debias the training corpus by supplying
additional data to support the target groups with comparatively fewer data in
the corpora and thereby creating a balanced corpus to train on. Augmentation
can thus in a way counterbalance the under/over representations of any
particular target group of a domain on training corpus. Zhao et al. [126, 128]
adopts data augmentation to reduce gender bias in the downstream task of
coreference resolution while observing that the possibility of associating
occupations to masculine terms is very much higher than feminine terms. They
augment gender swapped versions of data, from masculine terms to feminine
terms and vice-versa, to the training corpus after anonymizing the named
entities. Park et al. [81] utilizes the idea of data augmentation proposed by
Zhao et al. [128] to reduce gender bias in abusive language detection. Very
similar to Zhao et al. [128], Lu et al. [66] proposes Counterfactual Data
Augmentation to explore gender bias in Neural NLP applications including
coreference resolution and language modeling by representing bias with
reference to internal scores in neural models. Apart from the attempts to
remove gender bias in English language, Zmigrod et al. [132] proposes
variation to naive gender swapping based counterfactual data augmentation to
manage gender bias in morphology rich or inflected languages like Spanish and
Hebrew that otherwise deliver ungrammatical sentences with simpler approaches.
Maudslay et al. [45] improves counterfactual data augmentation by proposing
two variants named Counterfactual Data Substitution and Names Intervention to
address indirect bias. Liu et al. [64] use concept of Maudslay et al. [45] and
propose Counterpart Data Augmentation to remove biases in dialogue systems.
Even though data augmentation based debiasing is a simple technique to reduce
bias in the training corpora, their annotation is expensive and increases the
size of dataset, in turn, increasing the time required for training. Moreover,
these techniques generally only consider isolated words to perform binary
swapping and mostly ignore non-binary and more sophisticated representations
in a domain. Also, these techniques rely on a pre-defined limited list of key
terms and associated pairs, which may be conceivably incomplete, where some
terms may have different spelling (e.g. mommy vs. mummy), different morphology
(e.g. his→her and his→hers), pairing variations (e.g. breastfeed), or produce
absurd sentences due to blind swaps (e.g. she is pregnant→he is pregnant).

####  4.1.2 Bias fine-tuning

Another approach to debias training corpora is bias fine-tuning by transfer
learning, where, instead of the expensive process of constructing balanced
corpora with respect to a domain, the idea of transfer learning is employed.
As data biases generally originate from data imbalance or small sized
datasets, to make sure the model doesn’t overfit on biased data, Park et al.
[81] utilize transfer learning to gender debias abusive language detection
model by regularizing from a source network trained on a less biased large
dataset and thereafter fine-tune on a target dataset that is largely gender
biased. Even though the approach significantly reduces bias, it seems to hurt
the overall model accuracy too (as may be expected). Saunders et al. [97]
proposes an approach almost opposite to Park et al. [81] where they execute
small-domain adaptation by fine-tuning on a less biased small dataset to
address gender bias in neural machine translations.

###  4.2 Mitigating bias in representation

An earlier approach to debias the representations is geometric debiasing that
removes subspace of protected domain/target terms concerned to a domain in
embedding. For a domain, this post-process approach identifies subspace or
direction in embedding that holds bias and removes the association of neutral
words to target words concerning that domain. This hard debiasing approach is
followed by Bolukbasi et al. [16] to alleviate gender bias from word
embeddings by levelling the distance of gender neutral words towards the set
of gendered words. They also propose soft bias correction that aims to
conserve the initial embedding distances using a trade-off parameter to
balance with debiasing. But as it works by removing gender information from
the words, it might not be a generalizable approach such as in social science
and medical applications that makes use of these gender information [8, 71].
Also, their approach make use of classifier to distinguish gender neutral
words that in turn can propagate classification errors, influencing the
performance of debiasing [129]. Zhao et al. [129] neutralize embedding with
respect to gender by locating gender neutral words along with the process of
training word vectors without employing an additional classifier. However,
Gonen and Goldberg’s [41] experiments shows that the approaches of Bolukbasi
et al. [16] and Zhao et al. [129] are insufficient blind debiasing techniques
that just hides but doesn’t actually remove bias. For word level language
models, Bordia and Bowman [18] make use of a loss function regularizer to
penalize the projection of embeddings onto gender subspace as a soft debaising
version of Bolukbasi et al. [16]. As stated by Gonen and Goldberg [41], they
also mention chances of bias even after debiasing the representations, as a
case that their bias score may be not able to detect it. Alternative to this
direct geometric debiasing, Zhao et al. [126] proposes a different gender
neutralisation procedure by averaging the representations of original and
corresponding gender swapped data versions for debiasing contextualised
representations. To mitigate sentiment bias in word embedding concerned to
demographic domains, Sweeney et al. [108] removes the correlations of
demographic terms with sentiments.

###  4.3 Mitigating bias in algorithm

Debiasing representations are harder to be applied in contextual embeddings,
since the representation of a word can vary according to the plurality of
different contexts it appears in [67]. In such cases, certain works focus at
modifying algorithm/model, i.e., pre-trained or fine-tuned models, in such a
way that it mitigates the bias in predictions during training (i.e., in-
processing), usually by modifying the loss function of the model. Qian et al.
[86] alters the loss function of the language model in such a way that the
model adapts to equalize prediction probabilities for gendered pairs of words
thereby reducing output gender bias. Silva et al. [102] consider utilising
WEAT for debasing transformer based PLMs and later use WEAT score along with
cross entropy to modify loss function of RoBERTa. Huang et al. [47] propose
another variation of altering regular cross entropy loss function with
embedding and sentiment driven regularisation terms for mitigating sentiment
bias in large PLMs. Adversarial training is another way to debias algorithms
by altering loss functions while training a predictor along with an adversary,
which is intended to produce fair prediction by minimising the adversarial
function that tries to model protected domains like gender, race, etc. [124].
A similar approach is utilised by Zhang et al. [125] to mitigate gender bias
in clinical contextual word embeddings. A different recent approach proposed
by Liu et al. [65] uses reinforcement learning to debias political bias in
large PLMs by utilising rewards from word embedding or a classifier.

##  5 Affective Bias

Textual affective computing involves development of algorithms that accurately
identify what is subjectively written in text and how to make better affect-
influenced subjective decisions, i.e., decisions that are based on emotions,
sentiments, or opinions, in several real-world systems. Many systems were
initially proposed to detect and measure affect (emotion or sentiment)
expressed by textual data such as in movie reviews, product reviews, etc. [80,
105]. Later these systems were widely adopted into various domains such as
healthcare [44], commercial applications [54, 92, 105], politics [21],
education [34, 105] and many more. When industrial giants like
Google333https://cloud.google.com/natural-language/docs/analyzing-sentiment,
IBM444https://cloud.ibm.com/apidocs/natural-language-understanding#emotion and
Microsoft555https://docs.microsoft.com/en-us/azure/cognitive-
services/language-service/sentiment-opinion-mining/overview developed natural
language understanding tools, textual affective understanding became a crucial
and significant part of them. Several observations and arguments have been
made by researchers to demonstrate the importance of textual affective
computing tasks, especially towards sentiment analysis. According to Cambria
et al. [23] sentiment analysis task still needs to travel much to reach human
level performance which can be attained by successfully solving several other
NLP problems such as POS tagging, Named Entity Recognition, etc. Poria et al.
[84] draw attention to optimistic future research directions in affecting
computing like multi-modal affective computing, sarcasm analysis and even the
contemporary issue of social bias in affective NLP systems, and strive to
oppose the conventional belief that sentiment analysis tasks are saturated
being 20 years old. Our study on affective bias in NLP is motivated from these
observations on heightened relevance of affective computing and its wide
applicability in diversified NLP applications that leverage affect information
[4, 5, 37].

###  5.1 Definition and implications

The study on bias in NLP and machine learning was heavily accelerated through
observations on the negative impact of NLP biases towards certain social or
marginalized groups when applied in the real
world666https://www.propublica.org/article/machine-bias-risk-assessments-in-
criminal-sentencing. Affective bias is another recent research category in
this direction, where researchers study any unfair or imbalanced associations
of affect with key terms representing particular underrepresented or protected
groups in a domain and how it influences NLP systems, such as sentiment and
emotion detection. For example, Google sentiment analyzer is observed to infer
that being gay777https://www.vice.com/en/article/j5jmj8/google-artificial-
intelligence-bias is bad and assigns high negative sentiments towards
sentences such as ‘I’m a gay black woman’ and ‘I’m a homosexual’. By the term
Affective bias in NLP, we define the existence of unfair or biased
associations of affect (maybe emotions such as anger, fear, joy, etc. or
sentiment such as positive, neutral, negative) towards underrepresented
groups, or over-generalized beliefs (stereotypes) about particular social
groups in textual documents. For example, in textual documents, words
associated with women such as ‘she’, ‘wife’, etc., are highly associated with
a certain category of emotions like ‘sadness’ and ‘anger’, and representations
of the Muslim religion are observed as being associated with negative terms
that indicate violence [1], etc. The existence of affective bias in textual
affective computing systems harms its utility and applicability in tasks like
business and commercial decision making, healthcare, etc. The concept of
affective bias is valid and applicable beyond the NLP frameworks because, as
in [20], there are chances of high classification error rates for facial
emotion detection systems towards underrepresented social groups. Similarly,
it is crucial to evaluate all kinds of affective computing systems in the
backdrop of affective bias, since automated emotion detection systems have a
huge impact on modeling human behavior in many intelligent artificial
artifacts or algorithms that imitate human emotion systems for their
completeness.

###  5.2 Affective bias in PLMs

Many textual affective computing systems including lexicon based [94],
conventional machine learning [108], deep learning [14, 100], and hybrid [32]
approaches perpetuate affective bias, which, in general, is transmitted from
emotional bias of humans through models learnt over large scale textual
corpora. For example, usage of a corpus that contains textual data where
humans had expressed the anger emotion towards a particular underrepresented
religion for training an algorithm can later propagate or/and amplify this
affect oriented bias stereotyping religion with emotion anger. Table
LABEL:tab:_affective_bias illustrates an extensive snapshot of works in this
area of research along with their major characteristics. A predominant part of
existing works study affective bias specific to gender and through the
perspective of sentiment analysis [14, 52, 94, 100, 108]. Whereas, other
domains like religion, politics, intersectional biases, etc., and their impact
in perspective of fine-grained emotion classes (anger, fear, joy sadness,
surprise, disgust) have not been investigated as much, except in [52] and
[115]. The inadequacy of generic evaluation corpora to perform affective bias
evaluation in various domains along with the evaluation sentences having
corresponding emotion/sentiment ground truth or output label is a notable gap
in this area of research. When most works in literature try to identify the
existence of affective bias in NLP systems, only very few explore mitigation
of the harms of affective bias. The body of work in affective bias can be
split into two categories, conventional approaches and runtime verification
approaches. Conventional way of analyzing affective bias tries to identify
bias in training data, algorithm, representation, fine-tuning data, and
finally, mitigates them using different strategies [52, 108, 115]. Whereas,
runtime verification approaches monitor and uncover biased predictions in each
run of a specified system using mutations of input sentences (automatically
generated templates from input sentences and their paired sentences that
represent different views of a stereotype). Runtime verification is generally
suitable to validate whether the system satisfies fairness criteria over each
run.

The conventional approach by Shen et al. [100] investigates bias in sentiment
prediction for textual write-ups comprising similar content generated by
different groups of people. The analysis and identification of bias are
conducted on four publicly available lexicons and deep learning based systems.
A similar approach by Zhiltsova et al. [130] also identifies and mitigates
sentiment bias against non-native English text by using four popular lexicon
based emotion prediction systems. Both these works rely on linguistic style
changes across different human groups and how it leads to affective bias in
NLP. Apart from analysis of affective bias in lexicon and conventional machine
learning systems, researchers also explore non-contextual word embeddings such
as word2vec, GloVe, and FastText in the context of affective bias [32, 94,
108]. A significant contribution in this regard is the work by Diaz et al.
[32] addressing age related affective bias in ten widely used word embeddings
and fifteen different sentiment analysis models. The work primarily validates
whether opinion polling systems falsely report any age group (old or young)
more negatively or positively, for example a sentence with adjectives of
‘young’ more likely scores positive sentiments than the same sentence with
adjectives of ‘old’ [32]. Among the similar studies based on non-
contextualised word embeddings, Sweeney et al. [108] introduce an adversarial
learning strategy to mitigate demographic affective bias in word2vec and
GloVe, and Rozado et al. [94] screen word embeddings to identify bias through
the notion of representing words along cultural axis in the embedding space. A
more generic work of Kiritchenko et al. [52] identify affective bias in two
hundred emotion prediction systems that participated in the shared task
SemEval-2018 Task 1 Affect in Tweets. They procure an evaluation corpus,
Equity Evaluation Corpus (EEC), one of the few publicly available evaluation
corpus that has generic evaluation sentences and ground truth emotion labels
as basic emotions anger, fear, joy, and sadness for all evaluation sentences
in corpus. A similar evaluation corpus has been proposed by Venkit et al.
[115] considering sentences from the domain of persons with disabilities.

A more noteworthy approach to quantify affective bias concerning occupational
stereotypes in contextualized large PLM, BERT, is discussed in [14]. Even
though more works recently identify and mitigate generic bias in large PLMs
due to their efficacy and utility [63, 76], very few of them investigate
affective bias in large PLMs. Notable works to uncover bias in sentiment
analysis systems that utilize popular PLMs such as Google BERT, Facebook
RoBERTa, Google ALBERT, Google ELECTRA, and Facebook Muppet rely on runtime
verification approach instead of conventional paradigms to analyze bias [6,
122]. Another interesting approach by Huang et al., [47] investigates
sentiment bias introduced in text generated by language models. This emerging
scenario facilitates to conduct more evaluations to identify and mitigate
affective bias in large PLMs such as BERT, ALBERT, RoBERTa, XLNet, GPT, etc.

Table 3: Works addressing affective bias Work | Domain | Quantification | Mitigation | Model
---|---|---|---|---
Sentiment perspective
[122] |  Gender |  BiasFinder in [6] | — |  BERT
[6] |  Occupation, Country of origin, Gender |  Metamorphic Testing | — |  BERT, RoBERTa, ALBERT, ELECTRA, Muppet
[108] |  Gender |  Directional sentiment vectors |  Adversarial learning |  SVM, LSTM
[94] |  Ethnicity, Age, Sociodemographic status, Physical appearance, Religion, Politics, Gender |  Projecting word embeddings to cultural axis | — |  Lexicon based
[14] |  Occupation, Gender |  Statistical significance difference | — |  BERT, Bi-LSTM, Logistic Regression
[32] |  Age |  Multinomial log-liner regression, paired t-test | — |  Lexicon based, conventional machine learning, hybrid
[100] |  Gender, Race, Politics |  Difference in mean sentiment score, statistical significance test, linear regression | — |  Rule based, Naive Bayes, Dynamic CNN, LSTM
Emotion perspective
[115] |  Gender, Race |  Mean score of prediction, linear regression on sentiment scores | — |  DistilBERT, TextBlob, Google API, VADER
[130] |  Non-native English speaker |  Wilcoxon signed rank test |  Lexical score |  VADER, Afinn, SentimentR, TextBlob
[52] |  Gender, Race |  Average score difference | — |  Deep Learning, conventional machine learning, lexicon based

##  6 Bias Evaluation Corpora

Ideal bias evaluation corpora are indispensable components that helps to
identify and measure the existence of different types of bias in NLP systems.
Since this review focuses on bias in PLMs with special emphasis on affective
bias, we present popular and publicly available bias evaluation corpora along
with their limitations in the backdrop of PLMs and affective biases, in table
LABEL:tab:evalutaion_corpora; some of the evaluation corpora are avoided due
to their unavailability in the public domain to aid future research (e.g.,
[47, 130]). The most common attempt to create bias evaluation corpora are by
initially building template sentences and interchanging its key terms (e.g.
she→he, wife→husband, mother→father) associated to a target (e.g. Female,
Male) concerning a domain (e.g. Gender) which induces bias in natural
language. For example, Diaz et al. [32] creates bias evaluation corpora
intended to identify age related sentiment bias using template sentences like
‘This ⟨AGE RELATED KEY TERM⟩guy was 3 or 4 feet from the tide line and the
tide was going out’ and interchanging ⟨AGE RELATED KEY TERM⟩with ‘old’ and
‘young’. Contrary to conventional interchanging of key terms approach,
StereoSet [76] is a much diverse, large-scale and natural bias evaluation
corpus procured through inter and intra sentence Context Association Test
using Amazon Mechanical Turk. Design and creation of such bias evaluation
corpora are often defined carefully to isolate and extract the real effect of
bias in various stages such as data bias, algorithm bias, etc., and also to
avoid presence of any biased data within evaluation corpus or unbalanced set
of key terms referring to a particular target in a domain.

Evaluating large PLMs has recently become highly essential particularly due to
their increasing usage in many real-world applications. Several existing bias
evaluation corpora are useful enough to evaluate large PLMs when considering
the total number, i.e., size, of available evaluation sentences in these
corpora [33, 52, 128]. But these corpora fail to represent diverse real-world
contexts as they are generally built synthetically using very simple and
short-length sentences that are far from real-world scenarios and may even
deliver nonsensical samples (e.g. ‘my ⟨sister⟩is pregnant’→‘my ⟨brother⟩is
pregnant’). To accurately design an efficient bias evaluation corpora that
suit evaluating PLMs, Liang et al. [63] leverage naturally occurring text
corpora from WIKITEXT-2 [73], SST [103], etc., to generate evaluation
sentences that can represent real-world context better than the corpus in [33,
52, 76, 101, 128]. There is also a need for more efficient evaluation corpora
to evaluate affective bias; one of the available corpus by Kiritchenko et al.
[52] contains, as mentioned earlier, very simple synthetic and far from real-
world templates where emotions are specified in the sentence explicitly, e.g.,
‘My brother made me feel angry’ with ground-truth label anger. Another affect
oriented corpus by Venkit et al. [115] much suitably represents real-world
context suitable to evaluate PLMs, however, they are specifically designed to
address bias concerned to people with disabilities, e.g. ‘They were aggravated
because of the Mentally Handicapped neighbour’ with ground-truth label anger.
To effectively test and build fair machine learning software, it is highly
essential to develop more generic and real-world context enabled evaluation
corpora addressing various domains like religion, race, intersectional bias,
non-binary representations of gender (including lesbian, gay, etc., apart from
common way of only addressing gender as a binary domain). Such corpora can
eventually lead to the usage of bias evaluation corpora as an integral part of
any NLP system to evaluate system bias like any other test performed on the
system before being deployed in real-world [24].

Table 4: Bias evaluation corpora Corpora | Domain | Model | Limitation
---|---|---|---
BITS [115] https://github.com/PranavNV/BITS |  Gender, Race |  Emotion detection |  In context of people with disabilities, not generic
[63] https://github.com/pliang279/LM˙bias |  Gender, Religion |  Large PLM | —
StereoSet [76] https://github.com/moinnadeem/StereoSet |  Gender, Race, Religion, Profession |  Large PLM |  Less diverse to represent real-world contexts
BEC-Pro [9] https://github.com/marionbartl/gender-bias-BERT |  Gender, Profession |  Large PLM |  Less diverse to represent real-world contexts
GenderCorpus [14] https://github.com/jayadevbhaskaran/gendered-sentiment |  Gender, Profession |  Sentiment analysis |  Groud-truth sentiment labels are not available
AgeBias [32] https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/F6EMTS |  Age |  Sentiment analysis |  Groud-truth sentiment labels are not available
EEC [52] https://saifmohammad.com/WebPages/Biases-SA.html |  Race, Gender |  Emotion detection |  Explicit representation of emotions, small & simple sentences, less diverse to represent real-world contexts
Winograd [95] https://github.com/rudinger/winogender-schemas |  Gender, Profession |  Coreference resolution |  Small sentences
WinoBias [128] https://github.com/uclanlp/corefBias/tree/master/WinoBias/wino |  Gender, Profession |  Coreference resolution |  Miss ambiguous pronouns in sufficient volume or diversity to accurately represent practical utility of models
GAP [117] https://github.com/google-research-datasets/gap-coreference |  Gender |  Coreference resolution | —
Identity synthetic dataset [33] https://github.com/conversationai/unintended-ml-bias-analysis |  Human identity (white, gay, etc.) |  Text classification |  Small and simple sentences, less diverse to represent real-world contexts

##  7 Discussion on Research Challenges

Here we discuss several challenges in NLP bias, specifically in the context of
large PLMs and also in the context of affective bias.

####  7.0.1 Heterogeneous nature of NLP bias

A large number of research approaches to identify and mitigate NLP biases
generally address gender bias (as can be observed from table 1), even though
there exist many other domains such as race, religion, intersectional biases,
etc., that needs to be addressed significantly. Tan et al. [110] provides
evidence in this context that racial bias is encoded strongly in
contextualized large PLMs, probably even more than gender bias. They also show
that the less explored domain of intersectional biases that consider a mixture
of two or more domains (e.g. African American Females) is even more higher
than primary biases. Also, most existing research only concentrate on
addressing a subset of target terms concerned to a domain, e.g. considering
gender domain with binary targets terms male and female, instead of its non-
binary nature comprising of other target terms like gay and lesbian. All these
observations illustrate that the heterogeneous nature of bias in different
perspectives must be considered in future works of evaluating bias to make the
NLP systems fully debiased.

####  7.0.2 Evaluation corpus

Many existing bias evaluation corpora demonstrate the scarcity of realism in
sentences, such as synthetic sentences that are poor in representing real-
world context, as well as short-length sentences, etc. Accurate understanding
and quantification of existing bias in NLP models may not be effective with
these corpora, especially in case of large PLMs, which are capable of
generating complex real-world sentences to produce benchmark results in many
downstream applications. Most of the large real-world evaluation corpora focus
on gender domain, and real-world context based corpora that suit other domains
are scarce. In case of quantification and mitigation of affective biases, many
researchers use generic evaluation corpora in [14, 32] where evaluations do
not have an affective output label. A corpus developed by Kiritchenko et al.
[52] contains relevant emotion labels but lacks real-world contexts and has
labels largely derived from explicit mentions of emotions.

####  7.0.3 Generalizable design

Generalizability of any system enables it to be used directly or indirectly
within other allied systems. Bias evaluation is becoming an essential part in
NLP systems to produce fair decisions by avoiding serious societal harms. This
necessity can be achieved largely by designing generalizable bias
identification and mitigation modules. Generalizability in the context of
evaluation corpora, metrics to identify and quantify the impact of bias,
performance trade-off measures post debiasing, are important challenges to be
addressed in the future to bring the vital bias evaluation process as a common
and simple component that any NLP system can adapt and experiment.

####  7.0.4 Pre-train vs. Fine-tune bias

Bias in downstream applications that utilize pre-trained models originate
across different stages of system development. Initially, it may be from
massive amount of data used to pre-train the model which may get amplified by
the pre-training algorithm. The other way of introducing bias is through fine-
tuning data as well as fine-tuning algorithm. An existing challenge in this
background is to analyze and understand which part of the overall system
really causes bias, (i.e., is it from pre-trained or task specific fine-tuning
modules), how bias is perpetuated with the harmonized workflow of system
components, which parts of overall system must undergo a mitigation process,
etc.

####  7.0.5 Linguistic diversity and NLP bias

Recent NLP research, widely explores the utility of developing large PLMs and
downstream applications in various different languages by including capability
to handle monolingual as well as bilingual strategies. Transformer based large
PLM, IndicBERT888https://huggingface.co/ai4bharat/indic-bert, accessed on
January 03 2022 which handles low resource and morphologically highly
inflected Indian languages like Malayalam, is an example. Similarly, a large
variety of monolingual and bilingual
corpora999https://huggingface.co/datasets, accessed on January 03 2022 (in
languages English, Afrikaans, Arabic, etc.) and machine learning
models101010https://huggingface.co/models, accessed on January 03 2022 (for
various tasks including question answering, text classification, etc.) in
different languages illustrate same trends. In this context, it is highly
essential to study bias in such different linguistic corpora and the mono and
multilingual PLMs learned from those corpora. Among the very few attempts in
this direction is the research to mitigate gender bias in Hindi word embedding
[85], evaluating gender bias in Hindi-English machine translation [91],
addressing gender bias in languages like Spanish and Hebrew [132], etc.

####  7.0.6 Interdisciplinary methods to analyze NLP bias

Sun et al. [106] states that many techniques deployed in non NLP systems can
be applied directly or with small modifications to identify and mitigate bias
in NLP. Several studies indicate that bias in NLP systems opens up scope to
associate with other branches such as sociology, psychology, socio-
linguistics, engineering, legal studies, etc. The Implicit Association Test
[42] in psychology and its diverse computational representations such as Word
Embedding Association Test [22] and Sentence Encoder Association Test [70] to
quantify representation bias are examples to showcase interdisciplinary nature
of removing bias in NLP. Also, lessons from Software
Engineering111111“IndoML-2021- Day 1 Session 1”, YouTube video, 01:12:50,
Posted by “CSE IIT Gandhinagar” December, 16, 2021,
https://www.youtube.com/watch?v=y3t8pc1s0Yw, accessed on January 03 2022 such
as unit testing, behavior testing, etc., can be adopted to evaluate and
quantify bias in machine learning models at different levels [93]. Sun et al.
[106] treats mitigation of NLP bias as a combined problem of sociology and
engineering, where sociology can identify how really humans perceive and
encode social bias into language. Research towards interdisciplinary
discussions can bring light into current bias quantification and mitigation
strategies and inspire developing advanced and practically relevant approaches
[7, 12, 99].

##  8 Conclusion

In this survey, we conducted a comprehensive investigation towards bias in
large pre-trained language models, especially the transformer based models. We
discuss different types of biases that originate at various stages of pre-
trained language model workflow and the methods used to quantify and mitigate
these biases. Due to the widespread utility of affective computing systems in
the real-world, we give a special emphasis on the less explored area of bias
that associates with affect, i.e., affective bias. Our study also lists the
popular and publicly available evaluation corpora that aid future research,
along with their suitability in large pre-trained language models. Finally, we
discuss the challenges in this area of research, addressing which would aid
the community to further improve the tasks of identification and mitigation of
bias in pre-trained language modes. Materials regarding this survey will be
made publicly available at https://github.com/anoopkdcs/NLPBias and we will
keep updating it to aid future research.

## References

  * [1] Abid, A., Farooqi, M., Zou, J.: Large language models associate muslims with violence. Nature Machine Intelligence 3(6), 461–463 (2021). https://doi.org/https://doi.org/10.1038/s42256-021-00359-2
  * [2] Abid, A., Farooqi, M., Zou, J.: Persistent Anti-Muslim Bias in Large Language Models, p. 298–306. Association for Computing Machinery, New York, NY, USA (2021), https://doi.org/10.1145/3461702.3462624
  * [3] Ahn, J., Oh, A.: Mitigating language-dependent ethnic bias in BERT. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 533–549. Association for Computational Linguistics (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.42
  * [4] Anoop, K.: Affect-oriented fake news detection using machine learning. In: AWSAR Awarded Popular Science Stories By Scientists for the People, pp. 426–428. Vigyan Prasar, DST, India, ISBN: 978-81-7480-337-5 (2019), https://www.researchgate.net/publication/344838679˙Affect-Oriented˙Fake˙News˙Detection˙Using˙Machine˙Learning
  * [5] Anoop, K., Deepak, P., Lajish, V.L.: Emotion cognizance improves health fake news identification. In: Proceedings of the 24th Symposium on International Database Engineering & Applications. IDEAS ’20, Association for Computing Machinery, Seoul, Republic of Korea (2020). https://doi.org/10.1145/3410566.3410595
  * [6] Asyrofi, M.H., Yang, Z., Yusuf, I.N.B., Kang, H.J., Thung, F., Lo, D.: Biasfinder: Metamorphic test generation to uncover bias for sentiment analysis systems. IEEE Transactions on Software Engineering (2021). https://doi.org/10.1109/TSE.2021.3136169
  * [7] Avin, C., Keller, B., Lotker, Z., Mathieu, C., Peleg, D., Pignolet, Y.A.: Homophily and the glass ceiling effect in social networks. In: Proceedings of the 2015 Conference on Innovations in Theoretical Computer Science. p. 41–50. ITCS ’15, Association for Computing Machinery, New York, NY, USA (2015). https://doi.org/10.1145/2688073.2688097
  * [8] Back, S.E., Payne, R.L., Simpson, A.N., Brady, K.T.: Gender and prescription opioids: Findings from the national survey on drug use and health. Addictive Behaviors 35(11), 1001–1007 (2010). https://doi.org/https://doi.org/10.1016/j.addbeh.2010.06.018
  * [9] Bartl, M., Nissim, M., Gatt, A.: Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias. In: Proceedings of the Second Workshop on Gender Bias in Natural Language Processing. pp. 1–16. Association for Computational Linguistics (Dec 2020), https://aclanthology.org/2020.gebnlp-1.1
  * [10] Basta, C., Costa-jussà, M.R., Casas, N.: Evaluating the underlying gender bias in contextualized word embeddings. In: Proceedings of the First Workshop on Gender Bias in Natural Language Processing. pp. 33–39. Association for Computational Linguistics, Italy (Aug 2019). https://doi.org/10.18653/v1/W19-3805
  * [11] Basta, C., Costa-jussà, M.R., Casas, N.: Extensive study on the underlying gender bias in contextualized word embeddings. Neural Computing and Applications 33(8), 3371–3384 (2021), https://doi.org/10.1007/s00521-020-05211-z
  * [12] Beukeboom, C.J., Burgers, C.: How stereotypes are shared through language: a review and introduction of the aocial categories and stereotypes communication (scsc) framework. Review of Communication Research 7, 1–37 (2019). https://doi.org/10.12840/issn.2255-4165.017
  * [13] Bhardwaj, R., Majumder, N., Poria, S.: Investigating gender bias in bert. Cognitive Computation pp. 1–11 (2021). https://doi.org/https://doi.org/10.1007/s12559-021-09881-2
  * [14] Bhaskaran, J., Bhallamudi, I.: Good secretaries, bad truck drivers? occupational gender stereotypes in sentiment analysis. In: Proceedings of the First Workshop on Gender Bias in Natural Language Processing. pp. 62–68. Association for Computational Linguistics, Italy (2019). https://doi.org/10.18653/v1/W19-3809
  * [15] Blodgett, S.L., Green, L., O’Connor, B.: Demographic dialectal variation in social media: A case study of African-American English. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 1119–1130. Association for Computational Linguistics, Austin, Texas (Nov 2016). https://doi.org/10.18653/v1/D16-1120
  * [16] Bolukbasi, T., Chang, K.W., Zou, J., Saligrama, V., Kalai, A.: Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. p. 4356–4364. NIPS’16, Curran Associates Inc. (2016)
  * [17] Bolukbasi, T., Chang, K.W., Zou, J., Saligrama, V., Kalai, A.: Quantifying and reducing stereotypes in word embeddings. arXiv:1606.06121 (2016)
  * [18] Bordia, S., Bowman, S.R.: Identifying and reducing gender bias in word-level language models. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. pp. 7–15. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-3002
  * [19] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems. vol. 33, pp. 1877–1901. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * [20] Buolamwini, J., Gebru, T.: Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency. Proceedings of Machine Learning Research, vol. 81, pp. 77–91. PMLR (23–24 Feb 2018), https://proceedings.mlr.press/v81/buolamwini18a.html
  * [21] Caetano, J.A., Lima, H.S., Santos, M.F., Marques-Neto, H.T.: Using sentiment analysis to define twitter political users’ classes and their homophily during the 2016 american presidential election. Journal of internet services and applications 9(1), 1–15 (2018). https://doi.org/https://doi.org/10.1186/s13174-018-0089-0
  * [22] Caliskan, A., Bryson, J.J., Narayanan, A.: Semantics derived automatically from language corpora contain human-like biases. Science 356(6334), 183–186 (2017). https://doi.org/10.1126/science.aal4230
  * [23] Cambria, E., Poria, S., Gelbukh, A., Thelwall, M.: Sentiment analysis is a big suitcase. IEEE Intelligent Systems 32(6), 74–80 (2017). https://doi.org/10.1109/MIS.2017.4531228
  * [24] Chakraborty, J., Majumder, S., Yu, Z., Menzies, T.: Fairway: A way to build fair ml software. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. p. 654–665. Association for Computing Machinery, New York, NY, USA (2020), https://doi.org/10.1145/3368089.3409697
  * [25] Chaloner, K., Maldonado, A.: Measuring gender bias in word embeddings across domains and discovering new gender bias word categories. In: Proceedings of the First Workshop on Gender Bias in Natural Language Processing. pp. 25–32. Association for Computational Linguistics, Italy (Aug 2019). https://doi.org/10.18653/v1/W19-3804
  * [26] Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P.: One billion word benchmark for measuring progress in statistical language modeling. Computing Research Repository (CoRR) pp. 1–6 (2013)
  * [27] Chouldechova, A.: Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data 5(2), 153–163 (2017), https://doi.org/10.1089/big.2016.0047, pMID: 28632438
  * [28] Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., Huq, A.: Algorithmic decision making and the cost of fairness. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. p. 797–806. KDD ’17, Association for Computing Machinery, New York, NY, USA (2017), https://doi.org/10.1145/3097983.3098095
  * [29] Craft, J.T., Wright, K.E., Weissler, R.E., Queen, R.M.: Language and discrimination: Generating meaning, perceiving identities, and discriminating outcomes. Annual Review of Linguistics 6(1), 389–407 (2020), https://doi.org/10.1146/annurev-linguistics-011718-011659
  * [30] Dev, S., Li, T., Phillips, J.M., Srikumar, V.: On measuring and mitigating biased inferences of word embeddings. Proceedings of the AAAI Conference on Artificial Intelligence 34(05), 7659–7666 (Apr 2020), https://ojs.aaai.org/index.php/AAAI/article/view/6267
  * [31] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423
  * [32] Díaz, M., Johnson, I., Lazar, A., Piper, A.M., Gergle, D.: Addressing age-related bias in sentiment analysis. In: Proceedings of the 2018 chi conference on human factors in computing systems. pp. 1–14. Association for Computing Machinery, New York, NY, USA (2018), https://doi.org/10.1145/3173574.3173986
  * [33] Dixon, L., Li, J., Sorensen, J., Thain, N., Vasserman, L.: Measuring and mitigating unintended bias in text classification. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. p. 67–73. AIES ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278729
  * [34] Dolianiti, F.S., Iakovakis, D., Dias, S.B., Hadjileontiadou, S., Diniz, J.A., Hadjileontiadis, L.: Sentiment analysis techniques and applications in education: A survey. In: International Conference on Technology and Innovation in Learning, Teaching and Education. pp. 412–427. Springer (2018). https://doi.org/https://doi.org/10.1007/978-3-030-20954-4_31
  * [35] Du, M., Yang, F., Zou, N., Hu, X.: Fairness in deep learning: A computational perspective. IEEE Intelligent Systems 36(4), 25–34 (2021). https://doi.org/10.1109/MIS.2020.3000681
  * [36] Eagly, A., Wood, W., Diekman, A.: Social role theory of sex differences and similarities: A current appraisal, pp. 123–174. Lawrence Erlbaum Associates Publishers (2000)
  * [37] Elmadany, A., Zhang, C., Abdul-Mageed, M., Hashemi, A.: Leveraging affective bidirectional transformers for offensive language detection. In: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. pp. 102–108. European Language Resource Association, France (2020), https://aclanthology.org/2020.osact-1.17
  * [38] Escudé Font, J., Costa-jussà, M.R.: Equalizing gender bias in neural machine translation with word embeddings techniques. In: Proceedings of the First Workshop on Gender Bias in Natural Language Processing. pp. 147–154. Association for Computational Linguistics, Italy (Aug 2019). https://doi.org/10.18653/v1/W19-3821
  * [39] Fatemi, Z., Xing, C., Liu, W., Xiong, C.: Improving gender fairness of pre-trained language models without catastrophic forgetting. arXiv:2110.05367 (2021)
  * [40] Garg, N., Schiebinger, L., Jurafsky, D., Zou, J.: Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 115(16), E3635–E3644 (2018)
  * [41] Gonen, H., Goldberg, Y.: Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 609–614. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1061
  * [42] Greenwald, A.G., McGhee, D.E., Schwartz, J.L.: Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology 74(6), 1464 (1998)
  * [43] Guo, W., Caliskan, A.: Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. p. 122–133. Association for Computing Machinery, New York, NY, USA (2021), https://doi.org/10.1145/3461702.3462536
  * [44] Gupta, V.S., Kohli, S.: Twitter sentiment analysis in healthcare using hadoop and r. In: 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom). pp. 3766–3772. IEEE (2016), https://ieeexplore.ieee.org/document/7724965
  * [45] Hall Maudslay, R., Gonen, H., Cotterell, R., Teufel, S.: It’s all in the name: Mitigating gender bias with name-based counterfactual data substitution. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 5267–5275. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1530
  * [46] Hermann, K.M., Kočiskỳ, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., Blunsom, P.: Teaching machines to read and comprehend. In: Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1. pp. 1693–1701 (2015)
  * [47] Huang, P.S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J., Maini, V., Yogatama, D., Kohli, P.: Reducing sentiment bias in language models via counterfactual evaluation. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 65–83. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.findings-emnlp.7
  * [48] Jin, X., Barbieri, F., Kennedy, B., Davani, A.M., Neves, L., Ren, X.: On transferability of bias mitigation effects in language model fine-tuning. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 3770–3783. Association for Computational Linguistics (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.296
  * [49] Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fasttext. zip: Compressing text classification models. arXiv:1612.03651 (2016)
  * [50] Kalyan, K.S., Rajasekharan, A., Sangeetha, S.: Ammus: A survey of transformer-based pretrained models in natural language processing. arXiv:2108.05542 (2021)
  * [51] Kaneko, M., Bollegala, D.: Unmasking the mask–evaluating social biases in masked language models. arXiv:2104.07496 (2021)
  * [52] Kiritchenko, S., Mohammad, S.: Examining gender and race bias in two hundred sentiment analysis systems. In: Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. pp. 43–53. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/S18-2005
  * [53] Kirk, H.R., Volpin, F., Iqbal, H., Benussi, E., Dreyer, F., Shtedritski, A., Asano, Y., et al.: Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in Neural Information Processing Systems 34 (2021)
  * [54] Krishnamoorthy, S.: Sentiment analysis of financial news articles using performance indicators. Knowledge and Information Systems 56(2), 373–394 (2018). https://doi.org/https://doi.org/10.1007/s10115-017-1134-1
  * [55] Kurita, K., Vyas, N., Pareek, A., Black, A.W., Tsvetkov, Y.: Measuring bias in contextualized word representations. In: Proceedings of the First Workshop on Gender Bias in Natural Language Processing. pp. 166–172. Association for Computational Linguistics, Italy (Aug 2019). https://doi.org/10.18653/v1/W19-3823
  * [56] Lambrecht, A., Tucker, C.: Algorithmic bias? an empirical study of apparent gender-based discrimination in the display of stem career ads. Management Science 65(7), 2966–2981 (2019). https://doi.org/10.1287/mnsc.2018.3093
  * [57] Lambrecht, A., Tucker, C.: Algorithmic bias? an empirical study of apparent gender-based discrimination in the display of stem career ads. Management Science 65(7), 2966–2981 (2019). https://doi.org/10.1287/mnsc.2018.3093
  * [58] Lapowsky, I.: Google autocomplete still makes vile suggestions (2018)
  * [59] Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32. p. II–1188–II–1196. ICML’14, JMLR.org (2014), https://dl.acm.org/doi/10.5555/3044805.3045025
  * [60] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (09 2019). https://doi.org/10.1093/bioinformatics/btz682
  * [61] Leino, K., Fredrikson, M., Black, E., Sen, S., Datta, A.: Feature-wise bias amplification. In: International Conference on Learning Representations (2019), https://openreview.net/forum?id=S1ecm2C9K7
  * [62] Li, B., Peng, H., Sainju, R., Yang, J., Yang, L., Liang, Y., Jiang, W., Wang, B., Liu, H., Ding, C.: Detecting gender bias in transformer-based models: A case study on bert. arXiv:2110.15733 (2021)
  * [63] Liang, P.P., Wu, C., Morency, L.P., Salakhutdinov, R.: Towards understanding and mitigating social biases in language models. In: International Conference on Machine Learning. pp. 6565–6576. PMLR (2021), http://proceedings.mlr.press/v139/liang21a.html
  * [64] Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., Tang, J.: Does gender matter? towards fairness in dialogue systems. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 4403–4416. International Committee on Computational Linguistics, Barcelona, Spain (Online) (Dec 2020). https://doi.org/10.18653/v1/2020.coling-main.390
  * [65] Liu, R., Jia, C., Wei, J., Xu, G., Wang, L., Vosoughi, S.: Mitigating political bias in language models through reinforced calibration. Proceedings of the AAAI Conference on Artificial Intelligence 35(17), 14857–14866 (May 2021), https://ojs.aaai.org/index.php/AAAI/article/view/17744
  * [66] Lu, K., Mardziel, P., Wu, F., Amancharla, P., Datta, A.: Gender bias in neural natural language processing. In: Logic, Language, and Security, pp. 189–202. Springer (2020)
  * [67] Magee, L., Ghahremanlou, L., Soldatic, K., Robertson, S.: Intersectional bias in causal language models. arXiv:2107.07691 (2021)
  * [68] Manzini, T., Yao Chong, L., Black, A.W., Tsvetkov, Y.: Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 615–621. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1062
  * [69] Marcus, M.P., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics 19(2), 313–330 (1993), https://aclanthology.org/J93-2004
  * [70] May, C., Wang, A., Bordia, S., Bowman, S.R., Rudinger, R.: On measuring social biases in sentence encoders. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 622–628. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1063
  * [71] McFadden, A.C., Marsh, G.E., Price, B.J., Hwang, Y.: A study of race and gender bias in the punishment of school children. Education and treatment of children pp. 140–146 (1992)
  * [72] Menegatti, M., Rubini, M.: Gender bias and sexism in language (09 2017). https://doi.org/10.1093/acrefore/9780190228613.013.470
  * [73] Merity, S., Xiong, C., Bradbury, J., Socher, R.: Pointer sentinel mixture models. arXiv:1609.07843 (2016)
  * [74] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings (2013)
  * [75] Motro, D., Evans, J., Ellis, A.P., Benson, L.: Race and reactions to negative feedback: Examining the effects of the “angry black woman” stereotype. Academy of Management Proceedings 2019(1), 11230 (2019). https://doi.org/10.5465/AMBPP.2019.11230abstract
  * [76] Nadeem, M., Bethke, A., Reddy, S.: StereoSet: Measuring stereotypical bias in pretrained language models. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 5356–5371. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.acl-long.416
  * [77] Ng, S.H.: Language-based discrimination: Blatant and subtle forms. Journal of Language and Social Psychology 26(2), 106–122 (2007). https://doi.org/10.1177/0261927X07300074
  * [78] Niethammer, C.: Ai bias could put women’s lives at risk - a challenge for regulators (May 2020), https://www.forbes.com/sites/carmenniethammer/2020/03/02/ai-bias-could-put-womens-lives-at-riska-challenge-for-regulators/?sh=753a6217534f
  * [79] Packer, B., Mitchell, M., Guajardo-Céspedes, M., Halpern, Y.: Text embeddings contain bias. here’s why that matters. Tech. rep., Google (2018)
  * [80] Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up? sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10. p. 79–86. EMNLP ’02, Association for Computational Linguistics, USA (2002). https://doi.org/10.3115/1118693.1118704
  * [81] Park, J.H., Shin, J., Fung, P.: Reducing gender bias in abusive language detection. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2799–2804. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1302
  * [82] Pennington, J., Socher, R., Manning, C.: GloVe: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1532–1543. Association for Computational Linguistics, Doha, Qatar (Oct 2014). https://doi.org/10.3115/v1/D14-1162
  * [83] Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 2227–2237. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-1202
  * [84] Poria, S., Hazarika, D., Majumder, N., Mihalcea, R.: Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. IEEE Transactions on Affective Computing (2020). https://doi.org/10.1109/TAFFC.2020.3038167
  * [85] Pujari, A.K., Mittal, A., Padhi, A., Jain, A., Jadon, M., Kumar, V.: Debiasing gender biased hindi words with word-embedding. In: Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence. pp. 450–456. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3377713.3377792
  * [86] Qian, Y., Muaz, U., Zhang, B., Hyun, J.W.: Reducing gender bias in word-level language models with a gender-equalizing loss function. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop. pp. 223–228. Association for Computational Linguistics, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-2031
  * [87] Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., Huang, X.: Pre-trained models for natural language processing: A survey. Science China Technological Sciences pp. 1–26 (2020). https://doi.org/https://doi.org/10.1007/s11431-020-1647-3
  * [88] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. OpenAI blog (2018)
  * [89] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * [90] Rajkomar, A., Hardt, M., Howell, M.D., Corrado, G., Chin, M.H.: Ensuring fairness in machine learning to advance health equity. Annals of Internal Medicine 169(12), 866–872 (2018). https://doi.org/10.7326/M18-1990, pMID: 30508424
  * [91] Ramesh, K., Gupta, G., Singh, S.: Evaluating gender bias in hindi-english machine translation. In: Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing. pp. 16–23. Association for Computational Linguistics (Aug 2021). https://doi.org/10.18653/v1/2021.gebnlp-1.3
  * [92] Renault, T.: Sentiment analysis and machine learning in finance: a comparison of methods and models on one million messages. Digital Finance 2(1), 1–13 (2020). https://doi.org/https://doi.org/10.1007/s42521-019-00014-x
  * [93] Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: Behavioral testing of NLP models with CheckList. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4902–4912. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.442
  * [94] Rozado, D.: Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types. PloS one 15(4), 1–26 (04 2020). https://doi.org/10.1371/journal.pone.0231189
  * [95] Rudinger, R., Naradowsky, J., Leonard, B., Van Durme, B.: Gender bias in coreference resolution. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pp. 8–14. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-2002
  * [96] Sambasivan, N., Arnesen, E., Hutchinson, B., Doshi, T., Prabhakaran, V.: Re-imagining algorithmic fairness in india and beyond. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. p. 315–328. FAccT ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3442188.3445896
  * [97] Saunders, D., Byrne, B.: Reducing gender bias in neural machine translation as a domain adaptation problem. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7724–7736. Association for Computational Linguistics, Online (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.690
  * [98] Schick, T., Udupa, S., Schütze, H.: Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computational Linguistics 9, 1408–1424 (2021). https://doi.org/10.1162/tacl_a_00434
  * [99] Schluter, N.: The glass ceiling in nlp. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2793–2798. Association for Computational Linguistics, Brussels, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1301
  * [100] Shen, J.H., Fratamico, L., Rahwan, I., Rush, A.M.: Darling or babygirl? investigating stylistic bias in sentiment analysis. Proc. of FATML (2018)
  * [101] Sheng, E., Chang, K.W., Natarajan, P., Peng, N.: The woman worked as a babysitter: On biases in language generation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3407–3412. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1339
  * [102] Silva, A., Tambwekar, P., Gombolay, M.: Towards a comprehensive understanding and accurate evaluation of societal biases in pre-trained transformers. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2383–2389. Association for Computational Linguistics (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.189
  * [103] Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pp. 1631–1642. Association for Computational Linguistics, Seattle, Washington, USA (Oct 2013), https://aclanthology.org/D13-1170
  * [104] Stańczak, K., Choudhury, S.R., Pimentel, T., Cotterell, R., Augenstein, I.: Quantifying gender bias towards politicians in cross-lingual language models. arXiv:2104.07505 (2021)
  * [105] Suharshala, R., Anoop, K., Lajish, V.L.: Cross-domain sentiment analysis on social media interactions using senti-lexicon based hybrid features. In: 2018 3rd International Conference on Inventive Computation Technologies (ICICT). pp. 772–777. IEEE, Coimbatore, India (2018). https://doi.org/10.1109/ICICT43934.2018.9034272
  * [106] Sun, T., Gaut, A., Tang, S., Huang, Y., ElSherief, M., Zhao, J., Mirza, D., Belding, E., Chang, K.W., Wang, W.Y.: Mitigating gender bias in natural language processing: Literature review. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 1630–1640. Association for Computational Linguistics, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1159
  * [107] Suresh, H., Guttag, J.: A framework for understanding sources of harm throughout the machine learning life cycle. In: Equity and Access in Algorithms, Mechanisms, and Optimization. EAAMO ’21, Association for Computing Machinery, New York, NY, USA (2021), https://doi.org/10.1145/3465416.3483305
  * [108] Sweeney, C., Najafian, M.: Reducing sentiment polarity for demographic attributes in word embeddings using adversarial learning. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. p. 359–368. FAT* ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372837
  * [109] Sweeney, L.: Discrimination in online ad delivery: Google ads, black names and white names, racial discrimination, and click advertising. Queue 11(3), 10–29 (mar 2013). https://doi.org/10.1145/2460276.2460278
  * [110] Tan, Y.C., Celis, L.E.: Assessing social and intersectional biases in contextualized word representations. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/paper/2019/file/201d546992726352471cfea6b0df0a48-Paper.pdf
  * [111] Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR 2011. pp. 1521–1528. IEEE (2011). https://doi.org/10.1109/CVPR.2011.5995347
  * [112] Vanmassenhove, E., Hardmeier, C., Way, A.: Getting gender right in neural machine translation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3003–3008. Association for Computational Linguistics, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1334
  * [113] de Vassimon Manela, D., Errington, D., Fisher, T., van Breugel, B., Minervini, P.: Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. pp. 2232–2242. Association for Computational Linguistics (Apr 2021). https://doi.org/10.18653/v1/2021.eacl-main.190
  * [114] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. vol. 30, pp. 5998–6008. Curran Associates, Inc. (2017), https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [115] Venkit, P.N., Wilson, S.: Identification of bias against people with disabilities in sentiment analysis and toxicity detection models. arXiv:2111.13259 (2021)
  * [116] Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., Shieber, S.: Investigating gender bias in language models using causal mediation analysis. In: Advances in Neural Information Processing Systems. vol. 33, pp. 12388–12401. Curran Associates, Inc. (2020), https://proceedings.neurips.cc/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf
  * [117] Webster, K., Recasens, M., Axelrod, V., Baldridge, J.: Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics 6, 605–617 (2018). https://doi.org/10.1162/tacl_a_00240
  * [118] Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al.: Ethical and social risks of harm from language models. arXiv:2112.04359 (2021)
  * [119] Wolfe, R., Caliskan, A.: Low frequency names exhibit bias and overfitting in contextualizing language models. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 518–532. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.41
  * [120] Yang, Y., Uy, M.C.S., Huang, A.: Finbert: A pretrained language model for financial communications. arXiv:2006.08097 (2020)
  * [121] Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems 32 (2019), https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf
  * [122] Yang, Z., Asyrofi, M.H., Lo, D.: Biasrv: Uncovering biased sentiment predictions at runtime. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. p. 1540–1544. ESEC/FSE 2021, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3468264.3473117
  * [123] Ye, W., Xu, F., Huang, Y., Huang, C., et al.: Adversarial examples generation for reducing implicit gender bias in pre-trained models. arXiv:2110.01094 (2021)
  * [124] Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. pp. 335–340. AIES ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278779
  * [125] Zhang, H., Lu, A.X., Abdalla, M., McDermott, M., Ghassemi, M.: Hurtful words: Quantifying biases in clinical contextual word embeddings. In: Proceedings of the ACM Conference on Health, Inference, and Learning. p. 110–120. CHIL ’20, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3368555.3384448
  * [126] Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V., Chang, K.W.: Gender bias in contextualized word embeddings. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 629–634. Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1064
  * [127] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., Chang, K.W.: Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 2979–2989. Association for Computational Linguistics, Copenhagen, Denmark (Sep 2017). https://doi.org/10.18653/v1/D17-1323
  * [128] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., Chang, K.W.: Gender bias in coreference resolution: Evaluation and debiasing methods. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pp. 15–20. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-2003
  * [129] Zhao, J., Zhou, Y., Li, Z., Wang, W., Chang, K.W.: Learning gender-neutral word embeddings. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 4847–4853. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1521
  * [130] Zhiltsova, A., Caton, S., Mulway, C.: Mitigation of unintended biases against non-native english texts in sentiment analysis. In: Proceedings for the 27th AIAI Irish Conference on Artificial Intelligence and Cognitive Science, Galway, Ireland, December 5-6, 2019. CEUR Workshop Proceedings, vol. 2563, pp. 317–328. CEUR-WS.org (2019), http://ceur-ws.org/Vol-2563/aics_30.pdf
  * [131] Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., Fidler, S.: Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). pp. 19–27 (2015)
  * [132] Zmigrod, R., Mielke, S.J., Wallach, H., Cotterell, R.: Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 1651–1661. Association for Computational Linguistics, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1161