# What do Bias Measures Measure?

Sunipa Dev*1 Emily Sheng*2 Jieyu Zhao*1 Jiao Sun2 Yu Hou2 Mattie Sanseverino1
Jiin Kim1 Nanyun Peng1,2 Kai-Wei Chang1

1University of California, Los Angeles
2University of Southern California

###### Abstract

Natural Language Processing (NLP) models propagate social biases about
protected attributes such as gender, race, and nationality. To create
interventions and mitigate these biases and associated harms, it is vital to
be able to detect and measure such biases. While many existing works propose
bias evaluation methodologies for different tasks, there remains a need to
cohesively understand what biases and normative harms each of these measures
captures and how different measures compare. To address this gap, this work
presents a comprehensive survey of existing bias measures in NLP as a function
of the associated NLP tasks, metrics, datasets, and social biases and
corresponding harms. This survey also organizes metrics into different
categories to present advantages and disadvantages. Finally, we propose a
documentation standard for bias measures to aid their development,
categorization, and appropriate usage.

##  1 Introduction

As language technologies and their applications become more widely deployed in
society, there is also an increasing concern of the disparate impacts and
potential harm these technologies could have on different demographic groups
Bolukbasi et al. (2016); Webster et al. (2018a); Dev and Phillips (2019). To
address these concerns, a large body of work has since emerged to discuss,
detect, quantify, and mitigate the social biases and harms propagated by NLP
techniques. These works typically include bias measures comprising of metrics
and accompanying datasets to investigate specific definitions of social biases
as within the constructs of a specific NLP task, such as text classification
or machine translation. While this body of work is essential for understanding
the variety of potential harms, there remains a gap in understanding the
biases and normative harms each measure captures Blodgett et al. (2020) and
which of these measures is most apt to evaluate harms in a given model, task,
or application. Without a better comparative understanding of bias measures,
it is difficult to measure the community’s progress as a whole. As an example,
for the task of coreference resolution, there are several measures Zhao et al.
(2018); Rudinger et al. (2018); Lu et al. (2020); Webster et al. (2018b)
investigating gender bias. However, each measure is unique in either the
metric, the sentence structures, or the definition of bias.

To aid more comprehensive understanding, we present a survey to organize how
bias measures are related and questions we can ask to document the development
of future measures. There are other related surveys on existing bias
mitigation methods in natural language understanding (NLU) Sun et al. (2019),
biases in natural language generation (NLG) Sheng et al. (2021), adapted
definitions of biases Blodgett et al. (2020), and the relation between large
language models and harms Bender et al. (2021). This work further builds on
existing findings to focus specifically on bias measures.

First, we define several terms (Sec. 2) and survey different measures for
biases in NLP (Sec. 3). Next, we categorize measures by associated task,
demographic dimensions, and normative definitions of harm (Sec. 4); we explain
how these categories drive appropriate measure selection. Finally, we propose
a set of documentation questions to facilitate more contextualized development
of bias measures (Sec. 5). For this final contribution, we expand upon the
Datasheets questions from Gebru et al. (2018) to explicitly highlight the
information necessary for a bias measure in NLP to be used effectively and
result in meaningful progress in the scientific community.

Task |  Primary Dimension |  Secondary Dimension |  Measures
---|---|---|---
Coreference Resolution |  Gender |  Identity Terms |  Webster et al. (2018b); Cao and Daumé III (2020)
|  Gender |  Occupations |  Zhao et al. (2018); Rudinger et al. (2018); Lu et al. (2020)
Natural Language Inference |  Gender |  Occupations |  Dev et al. (2019)
|  Nationality |  Polar Adjectives |  Dev et al. (2019)
|  Religion |  Polar Adjectives |  Dev et al. (2019)
Sentiment Analysis |  Age |  Negative sentiments |  Díaz et al. (2018)
|  Gender |  Emotion words |  Kiritchenko and Mohammad (2018)
|  Individuals |  Identity Terms |  Prabhakaran et al. (2019)
|  Race |  Emotion words |  Kiritchenko and Mohammad (2018)
Question Answering |  Ethnicity |  Negative Activity |  Li et al. (2020)
|  Gender |  Occupations |  Li et al. (2020)
|  Race |  Negative Activity |  Li et al. (2020)
|  Religion |  Negative Activity |  Li et al. (2020)
Relation Extraction |  Gender |  Occupation |  Gaut et al. (2019)
|  Gender |  Spouse |  Gaut et al. (2019)
MLM Predictions |  Age |  Multiple stereotypes |  Nangia et al. (2020)
|  Appearance |  Multiple stereotypes |  Nangia et al. (2020)
|  Disability |  Multiple stereotypes |  Nangia et al. (2020)
|  Gender |  Multiple stereotypes |  Nangia et al. (2020); Nadeem et al. (2021)
|  Nationality |  Multiple stereotypes |  Nangia et al. (2020)
|  Race |  Multiple stereotypes |  Nangia et al. (2020); Nadeem et al. (2021)
|  Religion |  Multiple stereotypes |  Nangia et al. (2020); Nadeem et al. (2021)
|  Sexual Orientation |  Multiple stereotypes |  Nangia et al. (2020)
|  Socioeconomic |  Multiple stereotypes |  Nangia et al. (2020); Nadeem et al. (2021)
Text Classification |  Gender |  Occupations |  De-Arteaga et al. (2019); Zhao et al. (2020)
Toxicity Detection |  Age |  Identity Terms |  Dixon et al. (2018); Sap et al. (2020)
|  Disability |  Identity Terms |  Dixon et al. (2018); Jigsaw (2019); Sap et al. (2020)
|  Gender |  Identity Terms |  Dixon et al. (2018) \+ Park et al. (2018); Jigsaw (2019); Sap et al. (2020)
|  Individuals |  Identity Terms |  Prabhakaran et al. (2019)
|  Sexual Orientation |  Identity Terms |  Dixon et al. (2018); Jigsaw (2019); Sap et al. (2020)
|  Race |  Identity Terms |  Dixon et al. (2018); Jigsaw (2019); Sap et al. (2020)
|  Religion |  Identity Terms |  Dixon et al. (2018); Jigsaw (2019); Sap et al. (2020)
|  Political Ideology |  Identity Terms |  Sap et al. (2020)
|  Victim |  Identity Terms |  Sap et al. (2020)
Hate Speech Detection |  Gender |  Identity Terms |  Davani et al. (2020)
|  Gender |  Multiple stereotypes |  [Founta et al. (2018), Basile et al. (2019)] + Goldfarb-Tarrant et al. (2020)
|  Migrants |  Identity Terms |  Davani et al. (2020)
|  Migrants |  Pleasantness |  Basile et al. (2019) \+ Goldfarb-Tarrant et al. (2020)
|  Political Ideology |  Identity Terms |  Davani et al. (2020)
|  Race |  Dialect |  [Blodgett et al. (2016), Davidson et al. (2017), Founta et al. (2018)] + [Sap et al. (2019), Davidson et al. (2019), Xia et al. (2020)]
|  Race |  Identity Terms |  Davani et al. (2020) \+ Kennedy et al. (2020)
|  Religion |  Identity Terms |  Davani et al. (2020)
|  Sexual Orientation |  Identity Terms |  Davani et al. (2020)
Table 1: Existing bias measures organized by NLU task, and primary vs
secondary (demographic) dimensions. A ‘+’ indicates that one work (if
multiple, grounded in square brackets) built a bias metric (after ‘+’) on top
of a dataset from another work (before ‘+’). Identity terms: names and other
forms of group identifiers; polar adjectives: good vs bad; negative activity:
violent or bad traits and activities.

##  2 Definitions

To start, we clarify the definitions of several terms used throughout the rest
of this work.

#### Biases and Harms

Bias is a complex notion which is often not well-defined in existing
literature Blodgett et al. (2020). A common definition is “skew that produces
a type of harm” (Crawford, 2017) towards different social groups. Harms can be
representational or allocational Barocas and Selbst (2016), depending on
whether there is a generalization of (negative) representations of groups or
if there is a tangible, disparate distribution of resources between groups,
respectively. We also use the term _stereotype_ to refer to a societal
association between a group and some concepts, usually negative (e.g., “Asians
are bad drivers”).

#### Bias Measures

We define a _bias measure_ as an evaluation standard that includes a _bias
metric_ applied to a _bias dataset_. To show inequalities between demographic
groups, existing works typically define a bias metric (e.g., specialized
notions of group fairness) that they then apply to a dataset specially
designed to reveal social inequalities.

#### Primary and Secondary Demographic Dimensions

We use the term _demographic dimension_ to refer to a bias dimension (e.g.,
gender, race, age) for which specific instances (e.g., for gender: male,
female, non-binary) are comparatively evaluated. Additionally, we introduce
the notion of primary and secondary dimensions, because bias measures
typically involve studying a particular primary demographic dimension (e.g.,
gender) through a secondary dimension (e.g., occupation) to facilitate
analyses of harmful associations.

##  3 A Survey of Benchmark Bias Measures

As NLP models grow in size, complexity, ability to mimic underlying languages,
and the extent to which they are deployed in real world applications, it
becomes more important to understand their potential for biases and harms. A
growing number of measures serve as benchmarks to evaluate biases in tasks
such as sentiment analysis or relation extraction, targeting specific social
biases related to gender, race, religion, etc. While measures to evaluate
biases are being formulated across various tasks, there remains a lack of
cohesive understanding of _what these bias measures evaluate_ and _how
different measures relate_. In this section, we facilitate answers to these
questions through a comprehensive list of measures for quantifying biases in
different NLP tasks for primarily English.111 However, it is possible that we
missed some works in this survey, as the area is actively growing. Table 1
summarizes the categorization for NLU tasks.222For a similar table for NLG, we
refer readers to Sheng et al. (2021).

###  3.1 Natural Language Understanding

We discuss work around the variety of NLU tasks that have used different
measures to assess the presence of social biases.

#### Coreference Resolution

Coreference resolution is the task of finding all expressions that refer to
the same entity in text; a more specific objective is to associate gendered
pronoun mentions to different entities. There are two distinct definitions of
fairness that are evaluated with respect to this task, both centered around
gender. The first defines bias as model performance discrepancy across
different groups of a demographic attribute like gender. The Gendered
Ambiguous Pronouns ($GAP$) dataset Webster et al. (2018b) consists of samples
from Wikipedia biographies with ambiguous pronoun-name pairs. Webster et al.
(2018b) defines and measures biases through a disparity in correctly resolving
pronoun-name relationships for the male and female genders. The Maybe
Ambiguous Pronoun ($MAP$) dataset Cao and Daumé III (2020) expands $GAP$ to go
beyond binary genders with a broader dataset. The second category of
coreference resolution bias measures investigates the propagation of
stereotypes from language representations used by models. Both WinoBias Zhao
et al. (2018) and Winogender Rudinger et al. (2018) generate Winograd schema
style datasets to investigate occupational gender stereotypes. Additionally,
Lu et al. (2020) create simple sentence templates to evaluate biases using the
ratio of accurate pronoun resolution for stereotypical vs non-stereotypical
occupational associations.

Existing works that use the second definition of bias currently focus on
singular stereotypes, while gender biases can encompass a broad range of other
stereotypical and undesired associations. While both definitions of fairness
can potentially cover additional demographics and undesired associations, it
is important to question which is more applicable to investigate harms faced
by a group. As an example, non-binary individuals face erasure beyond
occupational stereotyping, and these experienced harms might be more
appropriately captured by the first definition.

#### Natural Language Inference (NLI)

NLI determines the directional relationship between two sentences, as to
whether the second sentence (hypothesis) is entailed, contradicted, or neutral
to the first sentence (premise). Dev et al. (2019) demonstrate how the task
captures and mirrors stereotypical associations (with binary gender, religion,
etc) learned by text representations. Their bias measure consists of a dataset
with sentence pairs: one sentence with an explicit demographic attribute
(e.g., gender), and the other with implicit, stereotypical associations (e.g.,
occupations). Bias is measured as the accuracy of models in identifying that
all sentences have no directional relation, i.e., classified having the
‘neutral‘ label. Since an overall score is calculated for bias over a set of
templates, a variety of templates can be independently assessed together to
evaluate fairness of NLI model outcomes across multiple demographic groups,
thus not restricting measurements to a single stereotype.

#### Sentiment Analysis

Estimating the sentiment or language polarity of text is useful for
understanding consumer perception from reviews, tweets, etc. However, this
task has been demonstrated to be stereotypically influenced by demographic
characteristics such as race and gender Kiritchenko and Mohammad (2018), age
Díaz et al. (2018) and names of individuals Prabhakaran et al. (2019).
Existing works keep sentence templates constant between samples and change the
assumed demographic attribute of the person in a sentence (e.g., through
changing names). This ideally should not change the sentiment classification
of the sample—changes in sentiment indicate the existence of stereotypical
associations. Since evaluation hinges on this contrast in classification
across groups, bias against a group is also measured in comparison to another.

#### Question Answering (QA)

QA models perform reading comprehension tasks and also propagate stereotypical
associations from underlying language representations, as demonstrated through
UnQuover Li et al. (2020). In this work, biases exhibited by QA systems are
measured using constructed sentence templates containing limited direct
demographic information (e.g., names) accompanied by under-specified questions
containing no related demographic information. The setup is such that all sub-
categories of a demographic attribute (e.g., religion: Christian, Buddhist,
etc) should be equally predicted as the answer. A statistically significant,
higher value for one sub-category is interpreted as bias. Thus, this measure
expands the understanding of comparative biases across several demographic
dimension values and is a closer reflection of the complexities of real-world
biases.

#### Neural Relation Extraction

Relation extraction is the task of extracting relations between entities in a
sentence and is instrumental in converting raw, unstructured text to
structured data. Gaut et al. (2019) note how gender biases in this task could
lead to allocational harms by affecting predictions on downstream tasks. They
create a dataset, WikiGenderBias, containing sentences regarding either a male
or female entity and containing one of four relationships: spouse, occupation,
birth date, or birth place. Similar to $GAP$, the evaluation framework
measures gender bias as a difference in model performance for each gender.
Instead of overall performance, they average over individual groups within a
relationship (e.g., different individual occupations). This measure faces the
challenge of generalizability as it relies on scraping a variety of existing
text for different demographics.

#### Masked Language Model Predictions

Several language representations (BERT, RoBERTa) are trained on the ability to
predict masked words in text Devlin et al. (2019). CrowS-Pairs Nangia et al.
(2020) and StereoSet Nadeem et al. (2021) are datasets that use this property
to expose and evaluate social biases learned with respect to different
protected attributes. Both use crowdsourcing to obtain annotated sentence
pairs, one of which is more stereotypical than the other for specific
attributes (gender, socioeconomic status, etc). The evaluation metrics in both
measures grade the model on its preference (through probabilities) for either
the stereotypical or other sentence. Because these datasets permit
crowdworkers to provide free-flowing text, the datasets are able to expand
understandings of biases beyond a single stereotypical association across
groups. However, Blodgett et al. (2021) caution about possible pitfalls of
contrasting stereotype benchmark datasets.

#### Text Classification (Occupations)

De-Arteaga et al. (2019) set up a measure for evaluating bias in text
classification where the task is to predict a person’s occupation given their
biography.The dataset contains short biographies crawled from online corpora
using templates and removing sentences which contain occupation names. Bias is
evaluated by comparing results across different gender groups. Zhao et al.
(2020) extend the original dataset to Spanish, French, and German. A challenge
is equally scraping diverse data for different demographics, as reflected in
the focus on binary gender for this measure.

#### Toxicity Detection

Toxic language ranges from more explicitly offensive forms (e.g., vulgar
insults) to more subtle forms (e.g., microaggressions). While toxicity
detection aims to identify toxic language, existing works have found uneven
detection of toxic language towards different groups. Prabhakaran et al.
(2019) show that there are varying levels of toxicity towards different names.
Dixon et al. (2018) analyze biases in a toxicity classification model through
the Wikipedia Talk Pages dataset as well as through a templated test set.
Jigsaw Jigsaw (2019) contains comments from the Civil Comments platform
labeled with six types of toxicity (e.g., toxic, obscene, etc) and identity
attributes (e.g., white, woman, etc). Along with this dataset, Jigsaw (2019)
present a bias evaluation following that of Borkan et al. (2019) by comparing
the AUC scores from different subgroups. Additionally, Sap et al. (2020)
create a social bias inference corpus with toxicity labels and targeted group
labels to understand the bias implications in languages. These bias measures
demonstrate that even a task intended to detect harms may do so in a biased
way.

#### Hate Speech Detection

Hate speech detection is the task of identifying abusive language that is
specifically directed towards a particular group.333Hate speech can overlap
with toxic language, but we find a moderately-sized body of work that focuses
on the former and thus distinguish between them. To study biases in hate
speech detection, many existing works have formulated different datasets and
bias metrics. Davidson et al. (2017) and Founta et al. (2018) annotate Twitter
datasets for hate speech detection. Blodgett et al. (2016) provide a corpus of
demographically-aligned text with geo-located messages based on Twitter. Sap
et al. (2019); Xia et al. (2020) use those datasets to show racial biases
through a higher false positive rate for AAE, while Davidson et al. (2019) use
the dataset of Blodgett et al. (2016) for racial bias evaluation by comparing
probabilities of tweets from different social groups being predicted as hate
speech. Davani et al. (2020) collect a dataset of comments from the Gab
platform, but analyze biases by comparing a language model’s log likelihood
differences for constructed counterfactuals. Goldfarb-Tarrant et al. (2020)
add gender labels to the dataset from Founta et al. (2018) to analyze gender
bias in hate speech detection, and further use Basile et al. (2019)’s
multilingual dataset to measure hate speech targeted at women and immigrants
in English and Spanish. Similar to toxicity detection, most of these measures
demonstrate the harm of online comments across demographic groups through a
comparative score.

#### Bias Analyses without Complete Bias Measures

There are other task-specific discussions of bias evaluations that do not
propose specific bias measures. For the task of common sense inference
(incorporating common sense knowledge into model inference), Rashkin et al.
(2018) analyze the intents of entities involved in an event, finding gender
differences in the intents. For named entity recognition, Mehrabi et al.
(2020) discuss how models have different abilities to recognize male and
female names as entities. For part-of-speech tagging, Munro and Morrison
(2020) and Garimella et al. (2019) find that state-of-the-art parsers perform
differently across genders, failing to identify “hers” and “theirs” as
pronouns but not “his”. In addition, Mehrabi et al. (2021) and Rudinger et al.
(2017) demonstrate severe disparities in common sense knowledge and NLI
datasets, respectively. These analyses provide a good starting point for the
development of full bias measures.

###  3.2 Natural Language Generation

We briefly describe datasets and metrics used to evaluate biases in NLG
tasks.444Sheng et al. (2021) has a more comprehensive survey that goes beyond
bias measures. Although some of these works allude to harms in the forms of
stereotypes and negative perceptions, there is limited discussion of how these
bias measures correlate to harms.

#### Autocomplete Generation

Autocomplete generation is the task of having a language model generate
continuations from a prompt. Sheng et al. (2019) and Huang et al. (2020) both
curate sets of prompts containing different demographic groups to prompt for
inequalities in generated text. The former uses a regard metric to measure
social perception towards groups, and the latter uses distributional
differences in sentiment scores. Whereas these two works manually curate
prompt sets, Dhamala et al. (2021) extract the beginnings of Wikipedia
articles to collect the BOLD dataset of prompts about various demographic
groups. The authors then use several metrics (sentiment, toxicity, regard,
etc) to measure biases in generated text. There are also works that extract
existing prompts and augment the prompt set with manual annotations. For
example, Groenwold et al. (2020) use extracted African American English
prompts to create a parallel set of White-Aligned English Twitter prompts and
compare the sentiment of generated texts. While manually constructed prompts
allow for more targeted evaluations, automatically extracted prompts allow for
more comprehensive and syntactically-varied evaluations.

#### Dialogue Generation

Dialogue generation is similar to autocomplete generation in that both require
the model to generate a text continuation given some prompt. The differences
lie in the use contexts—dialogue generation is used for specific tasks (e.g.,
patient help) within some domain (e.g., healthcare). Liu et al. (2020a)
construct a Twitter-based dataset with parallel context pairs between
different groups, and Liu et al. (2020b) rely on extracted conversation and
movie datasets to evaluate gender biases. Both works use various metrics such
as sentiment, offensiveness, and the occurrence of specific words. Dinan et
al. (2020) present an example of a bias measure that uses a crowdsourced
dataset (LIGHT from Urbanek et al. (2019)) to evaluate gender biases—in this
case, through the percentage of gendered words. There are many possible bias
metrics for this open-ended task and limited examination on trade-offs between
different metrics.

#### Machine Translation

For machine translation, the English WinoMT dataset (Stanovsky et al., 2019)
is a widely used dataset for quantifying gender biases. By concatenating
examples from Winogender Rudinger et al. (2018) and WinoBias Zhao et al.
(2018), the authors create a challenge set to assess translations of
stereotypical and non-stereotypical occupations for gendered coreference
associations. There are also extensions of WinoMT for different languages
(Kocmi et al., 2020) and datasets collected through mining (Gonen and Webster,
2020). Bias metrics for translation typically rely on translation accuracy. A
challenge for translation bias measures is obtaining correct translations in
several languages, which is perhaps simpler for manually constructed prompts
with similar syntax.

#### Text Re-Writing

There are other NLG tasks that revise specific words and phrases in a text to
be more aligned with a targeted attribute (e.g., style transfer, re-inflection
for translation). Individual re-writing tasks have curated datasets for
evaluating biases in specific tasks (e.g., Habash et al. (2019) build a corpus
for gender reinflection in Arabic, Zmigrod et al. (2019) annotate Hebrew and
Spanish datasets for reinflection) and generally use accuracy (i.e., correct
gender inflection) on datasets as bias metrics. A challenge is that these
tasks typically require specifically annotated datasets.

###  3.3 Limitations and Future Directions

Several measures use the terms “bias” and “stereotypical associations”
interchangeably without a deeper discussion of how these measures align with
normative harms. While multiple measures investigate gender biases, very few
expand their evaluations beyond binary gender or discuss how the extensions
can be made. Moreover, names are used as proxies for binary gender, often
without proper explanations or discussion on implications. In general,
extending measures to multiple languages, cultures, and demographic groups
(especially intersectional ones) are largely underspecified.

##  4 Categorization of Bias Metrics

Bias measures for different NLP tasks require using specific bias metrics on
task-specific datasets. In this section, we group the types of bias metrics
commonly used to evaluate biases and discuss advantages and disadvantages. We
relate this grouping to the absolute (i.e., metrics rely on “an accumulated
score to summarize inequalities”) versus relative (i.e., metrics report
inequality scores for all demographics) evaluation scheme as described by
Sheng et al. (2021). Absolute metrics enable ease of comparison across models.
However, by reducing biases down to singular scores, absolute metrics can
erase historical differences between groups. Our categorization of bias
metrics show that different types of metrics can be more or less aligned with
normative definitions of harm:

#### No Contrasting Groups

This type of metric enables measuring overall fairness across different
demographic groups, rendering a lot more freedom as to what is measured to be
beyond a single stereotype or specific demographics and their sub-categories.
For e.g., Dev et al. (2019) measure biases in NLI with absolute metrics that
evaluate how far the model probabilities are from the “neutral” class.
However, while absolute evaluations enable easy comparison across multiple
demographic groups, these evaluations limit the flexibility of the metric to
account for disparate treatment between groups.

#### Contrasting Demographic Attributes

These evaluations focus on differences in task performance or outcomes over
different demographic groups. For instance, Gaut et al. (2019) measure bias
through an absolute disparity score in relation extraction across different
genders. Rudinger et al. (2018) and Webster et al. (2018b) evaluate
coreference resolution performance across genders, reporting scores separately
per group.

#### Contrasting Stereotypes vs Anti-Stereotypes

This formulation of bias metrics assesses the ratio of model performance on
social stereotypes versus anti-stereotypes. For example, Zhao et al. (2019),
and Nangia et al. (2020) use this bias metric formulation to report singular
scores summarizing disparity in outcomes over different tasks.

#### Discussion

The contrastive nature of the latter two categories makes the bias metrics
that fall into these categories better equipped to highlight negative
associations faced by specific demographic groups in comparison to others.
These contrastive metrics thus can be more associated with normative
definitions of representational and allocational harm, as they demonstrate the
disparity of fair associations between demographic groups. However,
contrastive metrics _mandate_ the discretizing of demographic groups (e.g.,
gender into discrete groups) which is not always socially meaningful or
accurate Rajunov and Duane (2019).

##  5 Documenting Bias Measures

As defined in Sec. 2, a bias measure consists of a bias metric applied to a
bias dataset. Through our survey of different bias measures, we find many
underspecifications and various definitions. Thus, we propose a set of
questions to document bias measures and more easily understand, compare, and
use different measures during model development and deployment.

We build upon the existing guidelines from Gebru et al. (2018), which are more
generally for datasets of any modality or purpose. Here, we narrow the scope
to bias measures for NLP tasks. Since bias measures involve using datasets to
reveal biased associations, the original dataset-related questions are
relevant for bias measures too. We add questions in the existing sections of
_Composition_ and _Collection Process_ as proposed by Gebru et al. (2018).
Additionally, we propose new sections on _Motivation (Bias Measures)_ and
_Bias Metrics_. The specificity of the questions helps make the intended usage
of different bias measures more explicit.555This set of questions is intended
to spur discussion and is not intended to be the only relevant set.

  1. 1.

Motivation (Bias Measures)

Blodgett et al. (2020) detail the importance of concretely defining the biases
being measured, discerning biases from model errors, and listing out the how a
metric aligns with normative definitions of harm. We hope to facilitate more
discussions of these definitions by explicitly posing relevant questions.

     * •

What is the definition of bias? How does this definition align with normative
definitions of harm? For a measure to be a valid quantification of bias, the
notion of “bias” has to be well-defined and related to what is measured. More
explicitly bridging the gap between bias metrics and harms can drive progress
in fair NLP.

     * •

What language and culture (if any) is the bias and measure relevant in?

     * •

What social biases or stereotypes are evaluated? What other types can the
metric be extended to? What about intersectional identities? This question is
intended to obtain a list of the specific demographic groups a bias metric has
been shown to be useful for.

     * •

If a demographic attribute is split into groups for measurement of bias, how
many groups have been considered? What is the justification for the grouping?
This question is to understand the scope of the measure and assess its
inclusivity.

     * •

What is the source of bias that is measured? Social biases creep into NLP
models in different ways - the data used to derive representations, the
model(and parameters) used, etc. The bias measured can be from one or all
sources and needs to be acknowledged and when possible, disambiguated.

     * •

What tasks or applications is this bias metric useful for? Following on the
previous question, is this measure effective to check on any language
representations for social biases irrespective of application? Or is there a
specific task where this is most applicable?

  2. 2.

Creation Process

Language data is sourced primarily in two ways: by extracting from existing
textual data or by generating specific templates. While the first has the
advantage of being more similar to “real samples” where models would be used,
the latter has the advantage of testing for specific artefacts by construct.

     * •

Is the data scraped or generated? What are the advantages and disadvantages of
the chosen mode? Scraping or generating text using templates are two common
ways of building datasets in NLP. Each has its own advantages and
disadvantages depending on the method of creation. This emphasizes the
limitations of what is quantified as bias.

     * •

If the dataset is scraped, what are the primary sources/domains? Some text
sources are known to harbor more toxic or harmful content than others.

     * •

What is the structure of the sentence, sentence segment, template, or trigger
phrase used for data collection? Does the particular structure come with
certain simplifications, assumptions, or guarantees?

     * •

What are the limitations associated with method of data curation? How
generalizable is this dataset? Examples of limitations include scraped English
text containing predominantly Western narratives and annotated data relying on
inherent annotator biases. In terms of generalization, can the data be
extended by switching words or phrases to detect other biases or biases across
cultures.

     * •

Does the dataset use some proxy attribute to represent different demographic
groups that could potentially cause harm? For example, does this dataset use
popular names as a proxy for gender? Is there a risk for misidentifying
individuals if the associated genders are not self-reported?

  3. 3.

Bias Metrics

This section establishes the most relevant metrics to be used with a dataset
to measure bias. Beyond the metric explicitly proposed, different groups of
metrics, as defined in Section 4 can broaden understanding about measured
bias.

     * •

How is the bias metric defined? Is there a null hypothesis or normalization
recommended for it to be meaningful?

     * •

Is it an absolute or relative evaluation? Why is it constructed as such? Are
there trade-offs for different formats of evaluation? Sheng et al. (2021) and
Section 4 discuss trade-offs between formulating bias evaluations with
different categories of metrics.

     * •

Are there alternate or existing metrics this metric can or should be used
with? This question covers the cases where a bias metric may not be enough to
measure all desired attributes, either in terms of bias or general task
evaluations. For example, a bias metric may only cover gender bias or may not
account for generated text diversity.

     * •

Are the metrics correlated with intrinsic measures of bias (if any) in word
representations? Does it reflect the same bias, or is it exacerbated or
dampened?

     * •

Are there other existing datasets or metrics to evaluate bias for the same
task? How does an evaluation using one metric correlate with another using a
different metric? Does the sentence structure, sourcing method or other
feature differ between the datasets? Does this measure rely on superior
performance on the other datasets? Can the metric be combined with the other
datasets for a better approximation of bias?

     * •

Can the metric imply an absolute absence of bias in a specific task or model?
Also, is there a comparison required for correct assessment of bias?

We present example answers to these questions using specific bias measures in
Appendix A.

###  5.1 Documentation Facilitates Analysis

In Appendix A.2, we use questions from Section 5 to facilitate a deeper
analysis of the work described by Sheng et al. (2019). In particular, we note
that there is no explicit definition of biases, although the notion of regard
as a measure of social perception aligns with representational harms in the
forms of negative stereotypes, denigrations, etc. We additionally find that
this documentation exercise is especially useful if the documented measure has
been released for a while. In the case of the regard metric, there were not
many points of comparison at the time of its release, but more relevant
comparisons have recently been released. Thus, we recommend treating
documentation as a continuous process and revisiting the questions every so
often. Answering and re-answering these questions enables the community to
progress by continuously questioning the advantages and disadvantages of
different bias measures.

###  5.2 Documentation Reveals Limitations

This documentation is an exercise for designing bias measures with clearly
articulated trade-offs and an introspection into the direction of our
collective community progress. Specifying demographic groups and proposed
measures of biases helps illuminate associated harms and comparisons to other
measures. By documenting WinoBias in Appendix A.1, we observe it explicitly
measures specific binary gender-based occupational stereotypes, which is only
one manifestation of gender biases. The specificity of the documentation
questions helps determine what is measured and encourages the development of
other measures which can be used in conjunction or strive for greater
inclusivity (e.g., beyond binary genders).

##  6 Conclusion

Understanding what bias measures for NLP tasks are measuring is essential for
mitigating harms from models deployed into real-world applications. This work
begins to fill this gap in understanding by breaking down bias measures by
different tasks and metrics. We also illustrate the advantages and
disadvantages of different metrics in highlighting harms. Our proposed
documentation template for bias measures facilitates continuously re-visiting
bias measures to update limitations and comparative understanding with other
measures.

## Broader Impact

Ascertaining fair outcomes in NLP models is extremely important given how
widely they are used. With a variety of measures to aid this, their
appropriate choice, combination and usage is key for the process. There is
disconnect however between measures developed and a general understanding of
what harms they interpret and how to use them most effectively. We believe
that with detailed documentation, as described in this work, we can bridge the
gap and encourage widespread usage of bias measures.

## References

  * Barocas and Selbst (2016) Solon Barocas and Andrew D Selbst. 2016.  Big data’s disparate impact.  _Calif. L. Rev._ , 104:671.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019\.  SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 54–63.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big.  _Proceedings of FAccT_.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016.  Demographic dialectal variation in social media: A case study of african-american english.  _arXiv preprint arXiv:1608.08868_.
  * Blodgett et al. (2021) Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021.  Stereotyping norwegian salmon: an inventory of pitfalls in fairness benchmark datasets.  In _Proc. 59th Annual Meeting of the Association for Computational Linguistics_.
  * Bolukbasi et al. (2016) T Bolukbasi, K W Chang, J Zou, V Saligrama, and A Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  In _ACM Transactions of Information Systems_.
  * Borkan et al. (2019) Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019\.  Nuanced metrics for measuring unintended bias with real data for text classification.  In _Companion proceedings of the 2019 world wide web conference_ , pages 491–500.
  * Cao and Daumé III (2020) Yang Trista Cao and Hal Daumé III. 2020.  Toward gender-inclusive coreference resolution.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4568–4595.
  * Crawford (2017) Kate Crawford. 2017.  The trouble with bias.  Keynote at NeurIPS.
  * Davani et al. (2020) Aida Mostafazadeh Davani, Ali Omrani, Brendan Kennedy, Mohammad Atari, Xiang Ren, and Morteza Dehghani. 2020.  Fair hate speech detection through evaluation of social group counterfactuals.  _arXiv preprint arXiv:2010.12779_.
  * Davidson et al. (2019) Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019.  Racial bias in hate speech and abusive language detection datasets.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 25–35.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 11.
  * De-Arteaga et al. (2019) Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019.  Bias in bios: A case study of semantic representation bias in a high-stakes setting.  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , pages 120–128.
  * Dev et al. (2019) Sunipa Dev, Tao Li, Jeff Phillips, and Vivek Srikumar. 2019.  On measuring and mitigating biased inferences of word embeddings.
  * Dev and Phillips (2019) Sunipa Dev and Jeff Phillips. 2019.  Attenuating bias in word vectors.  In _AISTATS_ , Proceedings of Machine Learning Research, pages 879–887. PMLR.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT 2019_ , pages 4171–4186.
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021.  Bold: Dataset and metrics for measuring biases in open-ended language generation.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pages 862–872.
  * Díaz et al. (2018) Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018.  Addressing age-related bias in sentiment analysis.  In _Proceedings of the 2018 chi conference on human factors in computing systems_ , pages 1–14.
  * Dinan et al. (2020) Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020.  Queens are powerful too: Mitigating gender bias in dialogue generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8173–8188.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification.  In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 67–73.
  * Founta et al. (2018) Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018.  Large scale crowdsourcing and characterization of twitter abusive behavior.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 12.
  * Garimella et al. (2019) Aparna Garimella, Carmen Banea, E. Hovy, and Rada Mihalcea. 2019.  Women’s syntactic resilience and men’s grammatical luck: Gender-bias in part-of-speech tagging and dependency parsing.  In _ACL_.
  * Gaut et al. (2019) Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang, Jing Qian, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth M. Belding, Kai-Wei Chang, and William Yang Wang. 2019.  Towards understanding gender bias in relation extraction.  _CoRR_ , abs/1911.03642.
  * Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018.  Datasheets for datasets.  _arXiv preprint arXiv:1803.09010_.
  * Goldfarb-Tarrant et al. (2020) Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sanchez, Mugdha Pandya, and Adam Lopez. 2020.  Intrinsic bias metrics do not correlate with application bias.  _arXiv preprint arXiv:2012.15859_.
  * Gonen and Webster (2020) Hila Gonen and Kellie Webster. 2020.  Automatically identifying gender issues in machine translation using perturbations.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1991–1995.
  * Groenwold et al. (2020) Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020.  Investigating African-American Vernacular English in transformer-based text generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5877–5883.
  * Habash et al. (2019) Nizar Habash, Houda Bouamor, and Christine Chung. 2019.  Automatic gender identification and reinflection in Arabic.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 155–165.
  * Huang et al. (2020) Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020.  Reducing sentiment bias in language models via counterfactual evaluation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 65–83.
  * Jigsaw (2019) Jigsaw. 2019.  _Jigsaw Unintended Bias in Toxicity Classification_.
  * Kennedy et al. (2020) Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020.  Contextualizing hate speech classifiers with post-hoc explanation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5435–5442.
  * Kiritchenko and Mohammad (2018) Svetlana Kiritchenko and Saif M. Mohammad. 2018.  Examining gender and race bias in two hundred sentiment analysis systems.
  * Kocmi et al. (2020) Tom Kocmi, Tomasz Limisiewicz, and Gabriel Stanovsky. 2020.  Gender coreference and bias evaluation at WMT 2020.  In _Proceedings of the Fifth Conference on Machine Translation_ , pages 357–364.
  * Li et al. (2020) Tao Li, Tushar Khot, Daniel Khashabi, Ashish Sabharwal, and Vivek Srikumar. 2020\.  Unqovering stereotyping biases via underspecified questions.
  * Liu et al. (2020a) Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020a.  Does gender matter? towards fairness in dialogue systems.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4403–4416.
  * Liu et al. (2020b) Haochen Liu, Wentao Wang, Yiqi Wang, Hui Liu, Zitao Liu, and Jiliang Tang. 2020b.  Mitigating gender bias for neural dialogue generation with adversarial learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 893–903.
  * Lu et al. (2020) Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020\.  Gender bias in neural natural language processing.  In _Logic, Language, and Security_ , pages 189–202. Springer.
  * Mehrabi et al. (2020) Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and A. Galstyan. 2020\.  Man is to person as woman is to location: Measuring gender bias in named entity recognition.  _Proceedings of the 31st ACM Conference on Hypertext and Social Media_.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Pei Zhou, Fred Morstatter, J. Pujara, Xiang Ren, and A. Galstyan. 2021.  Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources.  _ArXiv_ , abs/2103.11320.
  * Munro and Morrison (2020) R. Munro and Alex Morrison. 2020.  Detecting independent pronoun bias with partially-synthetic data generation.  In _EMNLP_.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, Online. Association for Computational Linguistics.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967.
  * Park et al. (2018) Ji Ho Park, Jamin Shin, and Pascale Fung. 2018.  Reducing gender bias in abusive language detection.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2799–2804.
  * Prabhakaran et al. (2019) Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019.  Perturbation sensitivity analysis to detect unintended model biases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5740–5745.
  * Rajunov and Duane (2019) Micah Rajunov and Scott Duane. 2019.  _Nonbinary: Memoirs of Gender and Identity_.  Columbia University Press.
  * Rashkin et al. (2018) Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018\.  Event2Mind: Commonsense inference on events, intents, and reactions.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 463–473, Melbourne, Australia. Association for Computational Linguistics.
  * Rudinger et al. (2017) Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017.  Social bias in Elicited Natural Language Inferences.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pages 74–79.
  * Rudinger et al. (2018) Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018.  Gender bias in coreference resolution.  In _NAACL_ , pages 8–14.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1668–1678.
  * Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020.  Social bias frames: Reasoning about social and power implications of language.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5477–5490.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019.  The woman worked as a babysitter: On biases in language generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3407–3412.
  * Sheng et al. (2021) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021.  Societal biases in language generation: Progress and challenges.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_.
  * Shwartz et al. (2020) Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020.  “you are grounded!”: Latent name artifacts in pre-trained language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6850–6861.
  * Stanovsky et al. (2019) Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019.  Evaluating gender bias in machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1679–1684.
  * Sun et al. (2019) Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019.  Mitigating gender bias in natural language processing: Literature review.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1630–1640, Florence, Italy. Association for Computational Linguistics.
  * Urbanek et al. (2019) Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019.  Learning to speak and act in a fantasy text adventure game.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 673–683.
  * Webster et al. (2018a) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018a.  Mind the GAP: A balanced corpus of gendered ambiguous pronouns.  _TACL_ , 6:605–617.
  * Webster et al. (2018b) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018b.  Mind the GAP: A balanced corpus of gendered ambiguous pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617.
  * Xia et al. (2020) Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020.  Demoting racial bias in hate speech detection.  In _Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media_ , pages 7–14.
  * Zhao et al. (2020) Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini, Kai-Wei Chang, and Ahmed Hassan Awadallah. 2020.  Gender bias in multilingual embeddings and cross-lingual transfer.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2896–2907.
  * Zhao et al. (2019) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019.  Gender Bias in Contextualized Word Embeddings.  In _Proceedings of NAACL-HLT 2019_ , pages 629–634. Association for Computational Linguistics.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018\.  Gender bias in coreference resolution: Evaluation and debiasing methods.
  * Zmigrod et al. (2019) Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019.  Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1651–1661.

Appendix: What do Bias Measures Measure?

##  Appendix A Documenting Bias Measures

###  A.1 Case Study #1: Documentation for WinoBias (Zhao et al., 2018)

  1. 1.

Motivation (Bias Measures)

     * •

What is the definition of bias? Does this definition align with normative
definitions of harm? The paper defines gender bias in coreference resolution
as the instance when a system associates pronouns to occupations that are
dominated by the pronoun’s associated gender more accurately than occupations
not dominated by that gender. While gendered associations with occupations are
an instance of gender bias, such a definition does not capture gender bias in
its entirety. The metric is defined to measure occupational perception of
different genders, which is associated with representational harms.

     * •

What language and culture (if any) is the bias and measure relevant in?
English language.

     * •

If a demographic attribute is split into groups for measurement of bias, how
many groups have been considered? Gender binary (male and female) is
considered in this measure.

     * •

What types of social biases or stereotype are evaluated? What other types can
the metric be extended to? The paper evaluates the association of binary
gender (man, woman) to its corresponding occupational stereotypes, e.g. 90% of
nurses are women in the occupation statistics used for WinoBias Zhao et al.
(2018). While the dataset limits the metric to the previously defined gender
bias, the method of calculating the absolute difference of pro-stereotyped and
anti-stereotyped conditions could easily be generalized to other primary and
secondary demographic dimensions.

     * •

What is the source of bias that is measured? The paper highlights two sources
of gender bias: training data bias and resource bias. Training data used for
coreference resolution systems are noted to have severe gender imbalance (over
80% of entities headed by gendered pronouns are male). Pre-trained word
embeddings which serve as an auxiliary resource for WinoBias Zhao et al.
(2018) have been shown to contain gender bias as well (“men” is closer to
“programmer” than “woman”). The paper also mentions a gender statistics corpus
(i.e. Gender Lists) as a resource that contains an uneven number of gendered
contexts in which a noun phrase is observed.

     * •

What tasks or applications is this bias metric useful for? Since coreference
resolution serves as an important step for many higher-level natural language
understanding such as information extraction, document summarization, and
question answering, this bias metric is useful for any of such tasks.

  2. 2.

Creation Process

     * •

Is the data scraped or generated? What are the advantages and disadvantages of
the chosen mode? The data is collected from the U.S. Bureau of Labor
Statistics. An advantage of this is that the profession categories come from
an objective, rather than a baised, source as it is a government document. A
disadvantage of this is that it is not comprehensive, and it is generated with
the narrow view of only the United States.

     * •

What is the structure of the sentence, sentence segment, template, or trigger
phrase used for data collection? Two types of sentence structures were used.
Type 1: [entity1] [interacts with] [entity2] [conjunction] [pronoun]
[circumstances]
Type 2: [entity1] [interacts with] [entity2] and then [interacts with]
[pronoun] for [circumstances]
These two structures were chosen because one is more challenging (Type 1)
while the other is less challenging (Type 2), and the model is said to “pass”
if it has the same accuracy for both types. Type 1 sentences are more
difficult than Type 2 because they require world and circumstantial knowledge
to make coreference decisions, while Type 2 can be resolved using syntactic
information.

     * •

What are the limitations associated with method of data curation? How
generalizable is this dataset? The data is limited because the occupations are
collected from one source, and the source is specific to the United States. We
expect that occupation titles and categories vary among different countries.
Additionally, it is important to note that the statistics are constantly
changing, and although the website that the data updates regularly, the
dataset is static. This limits the relevance of the dataset as the world
around it changes.

     * •

Does the dataset use some proxy attribute to represent different demographic
groups that could potentially cause harm? No. The data uses occupations and
pronouns as the identifying factors, so no direct identities are involved and
at risk.

  3. 3.

Bias Metrics

     * •

How is the bias metric defined? It is defined as the absolute score difference
between pro-stereotyped and anti-stereotyped conditions, where for pro-
stereotypical condition, the gender pronoun is linked with the dominated
profession and for anti-stereotypical vice versa.

     * •

Is it an absolute or relative evaluation? As it measures the bias through the
difference between pro-stereotyped and anti-stereotyped conditions, it belongs
to relative evaluation. Using a relative evaluation allows more flexibility
for different models.

     * •

Are there alternate or existing metrics this metric can or should be used
with? WinoBias Zhao et al. (2018) adapts the absolute difference of F1 to
evaluate the gender bias. Since F1 score is a general metric to compare model
performance, similar to the difference, the ratio could also be used to so
disparity between to sets.

     * •

Are the metrics correlated with intrinsic measures of bias (if any) in word
representations? No, this study has not been done, but would be useful to
understand.

     * •

Are there other existing datasets or metrics to evaluate bias for the same
task? Yes, for coreference resolution task, there are also Gendered Ambiguous
Pronouns (GAP) Webster et al. (2018b) measuring the disparity incorrectly
solving pronoun-name relationships for male and female genders, MAP Cao and
Daumé III (2020) (built on GAP beyond binary genders) and Winogender Rudinger
et al. (2018) which also measures the relationship between gendered pronouns
and occupations.

     * •

Are there other datasets for the same task measuring the same stereotype? Yes,
since coreference resolution task involves pronouns, all three datasets
mentioned above measure the gender sereotype. Meanwhile, Winogender Rudinger
et al. (2018) measures the stereotype between gender and occupation which is
the same as WinoBias Zhao et al. (2018).

     * •

Can the metric imply an absolute absence of bias in a specific task or model?
No, as discussed before, this metric only focuses on entities with 40
occupations in limited sentence templates. Even if the absolute difference
doesn’t show much inequalities, there could still be biases in the model.

###  A.2 Case Study #2: Documentation for Regard (Sheng et al., 2019)

  1. 1.

Motivation (Bias Measures)

     * •

What is the definition of bias? Does this definition align with normative
definitions of harm? The authors do not provide an explicit definition of
bias, but define bias in terms of the metric of regard (i.e., social
perception) towards a demographic, which can be negative, neutral or positive.
Since this metric is defined to measure social perception, it is aligned with
definitions of representational harms, e.g., negative stereotypes,
denigrations.

     * •

What types of social biases or stereotype are evaluated? What other types can
the metric be extended to? The authors evaluate regard of text generated for
prompts that include demographics across race (Black, White), gender (man,
woman), and sexual orientation (gay, straight). Only one surface form is
evaluated for each demographic (e.g., “The Black person” for the race-black
demographic), thus limiting the comprehensiveness of the results, though the
method of formatting input prompts to include demographic mentions and
evaluate the generated text is generalizeable to other demographic surface
forms with minimal effort.

     * •

What is the source of bias that is measured? It is difficult to pinpoint the
exact sources of biases from the probing experiments run by Sheng et al.
(2019) on GPT-2 and the 1 Billion Word Language Model, though we can form
hypotheses. While the One Billion Word Benchmark dataset is publicly available
for analysis, the exact dataset used to train GPT-2 can probably only be
approximated at best. However, we know that GPT-2 was trained on Web data,
including from Web sources such as Reddit, which the authors mention as a
likely source of biases. The 1 Billion Word Language Model was trained on news
data, and Sheng et al. (2019) find less biased results from this model. There
could also be non-data related biases (e.g., depending on features in the
model architecture and training procedure), though more studies need to be
done here.

     * •

What tasks or applications is this bias metric useful for? The metric of
regard is useful for applications for continuation generation tasks (Sheng et
al., 2021), e.g., when a system takes an input prompt and generates text in a
mostly unconstrained manner. In other words, this metric could also be useful
for dialogue generation, chat bots, virtual assistants, and creative
generation applications, in addition to language models.

  2. 2.

Creation Process

     * •

Is the data scraped or generated? What are the advantages and disadvantages of
the chosen mode? The data used as input prompts to probe for biases are
generated from templates. For example, “XYZ worked as”, “XYZ earned money by”,
etc. These templates allow for a controlled probing of inequalities in
specific contexts related to occupations and respect. The disadvantages are
that templates can be time-consuming to manually construct (Sheng et al.
(2019) only use 10 templates) and may not be representative or comprehensive
of all the ways that similar content could be phrased. Additionally, the
templates could be biased towards the syntactic and semantic inclinations of
the template creators, which may or may not align with those the model is used
to seeing.

     * •

What is the structure of the sentence, sentence segment, template, or trigger
phrase used for data collection? The templates are of the format “[PERSON]
[passive verb]”, e.g., “XYZ was thought of as”. These templates appear to
provide minimal context to condition generation.

     * •

What are the limitations associated with method of data curation? How
generalizable is this dataset? These templates are generalizeable to other
demographic surface forms not mentioned in original work. Although
conceptually these templates can be extended to probe biases in other contexts
(e.g., contexts likely to lead to negative religious or ethnic stereotypes),
manually creating these contexts is slow and likely not comprehensive. While
these templates could also be translated to other languages, relying on
automatic translations could result in unnatural phrasings, while manual
translations are more time-consuming.

  3. 3.

Bias Metrics

     * •

How is the bias metric defined? Sheng et al. (2019) define the metric of
regard (social perception) towards a demographic group. Possible values are
negative, neutral, or positive.

     * •

Is it an absolute or relative evaluation? The authors have formatted the
comparison of regard scores across demographics as a relative evaluation.
Using a relative evaluation allows more flexibility for different analyses.

     * •

Are there alternate or existing metrics this metric can or should be used
with? Sheng et al. (2019) show in their study (Table 5) that the metrics of
sentiment and regard can be well-correlated for some types of prompts yet
greatly differ for other types. They conclude that it could be useful to
report both sentiment and regard.

     * •

Are the metrics correlated with intrinsic measures of bias (if any) in word
representations? This is a study that has not been done, but would be useful
to understand.

     * •

Are there other existing datasets or metrics to evaluate bias for the same
task? At the time of publication, there were perhaps limited proposed
alternatives for evaluating biases from language models, though there are now
other options. Huang et al. (2020) present 730 manually curated templates to
probe for sentiment differences across countries, occupations, and genders in
language models. There are also other bias measures for language models that
rely on sentiment (Groenwold et al., 2020; Shwartz et al., 2020).

     * •

Are there other datasets for the same task measuring the same stereotype?
Huang et al. (2020) also measure gender biases and manually curate prompt
sets. Groenwold et al. (2020) also measure racial biases, though they use
human-verified linguistic variations instead of demographic mentions in the
prompt to probe for biases. Both works use sentiment as a bias metric. Both
Sheng et al. (2019) and Huang et al. (2020) construct manual prompts to test
for biases towards demographics mentioned in the input. Additionally,
Groenwold et al. (2020) evaluate for similar biases in language models towards
_people who produce the text_ (Sheng et al., 2021). Combining all these bias
measures would provide a more comprehensive analysis.

     * •

Can the metric imply an absolute absence of bias in a specific task or model?
No, as discussed in earlier answers, the limited templates (both in number and
in syntactic/semantic diversity) mean that even if the regard scores do not
show inequalities, there could still be biases in the model. Also, since the
authors use a regard classifier to feasibly automatically label a large number
of samples, there could also be biases from the classifier itself. Even human
evaluations of regard could be influenced by human biases.