# When in Doubt, Cascade: Towards Building Efficient and Capable Guardrails

Manish Nagireddy
IBM Research
Cambridge, MA
<EMAIL_ADDRESS>
&Inkit Padhi
IBM Research
Yorktown Heights, NY
<EMAIL_ADDRESS>
Soumya Ghosh
IBM Research
Cambridge, MA
<EMAIL_ADDRESS>
&Prasanna Sattigeri
IBM Research
Cambridge, MA
<EMAIL_ADDRESS>

###### Abstract

Large language models (LLMs) have convincing performance in a variety of
downstream tasks. However, these systems are prone to generating undesirable
outputs such as harmful and biased text. In order to remedy such generations,
the development of guardrail (or detector) models has gained traction.
Motivated by findings from developing a detector for social bias, we adopt the
notion of a use-mention distinction - which we identified as the primary
source of under-performance in the preliminary versions of our social bias
detector. Armed with this information, we describe a fully extensible and
reproducible synthetic data generation pipeline which leverages taxonomy-
driven instructions to create targeted and labeled data. Using this pipeline,
we generate over 300K unique contrastive samples and provide extensive
experiments to systematically evaluate performance on a suite of open source
datasets. We show that our method achieves competitive performance with a
fraction of the cost in compute and offers insight into iteratively developing
efficient and capable guardrail models.

Warning: This paper contains examples of text which are toxic, biased, and
potentially harmful.

##  1 Introduction

Large language models (LLMs) contain high potential for a variety of real-
world applications, due to their versatility, adaptability, and ease of use,
along with their continuously improving performance [29, 6, 28, 32]. Yet,
their deployment, especially in critical domains such as healthcare and
finance, poses significant risks [5, 18]. A new host of challenges arises with
the generative capabilities of these models, as they may produce convincing
output, but this output may often be layered with issues around toxicity,
bias, hallucinations, and more.

In order to combat the harmful generations from these models, the concept of
guardrail (or detector) models have gained popularity for several reasons.
These guardrail models can be more efficient, modular, and scalable [1, 33,
19] than the LLMs whose output they operate on. In this work, we focus on the
problem of detecting whether an LLM’s textual output contains social bias.

#### Motivation

Social bias can be defined as discrimination for, or against, a person or
group, or a set of ideas or beliefs, in a way that is prejudicial or unfair
[41, 7]. Importantly, text which contains social bias may not contain any
explicit or profane content, but may still propagate discrimination (e.g., “I
don’t want to hire this individual as a babysitter because they have facial
scars."). Driven by a clear and present need to automatically detect whether
LLM-generated text contains such harmful content, we developed a social-bias-
detector. To do so, we gathered a collection of open source datasets, with
commercially permissible licenses, and used a combination of four datasets as
an attempt at a holistic collection of training data. We provide the specific
datasets, as well as the hyper-parameters used during training, in Appendix
A.1. From an architectural standpoint, the social-bias-detector is an encoder-
only model with just over 100M parameters that was obtained by fine-tuning
BERT [10].

Despite reasonable performance on our evaluation sets (specific numbers in
Appendix A.2), we discovered a high false positive issue with this model.
Here, a false positive refers to the classification of benign text as harmful.
In order to investigate further, we devised an experiment to test the
hypothesis that there existed a mismatch between the training dataset (which
were largely human-generated or curated) and the distribution of text that is
generated by an LLM. Three of the authors manually annotated outputs from the
llama-2-7b-chat111https://huggingface.co/meta-llama/Llama-2-7b-chat-hf model
and we conjectured that our model’s sub-optimal performance was due to the
excessively intricate and evasive answers, which tend to be generated by
highly aligned and verbose models such as Llama 2 [38]. However, concurrent to
our work, we came across the true reason for our detector’s subpar performance
- the use-mention distinction [14]. We will revisit this paper later in our
experimental results, in Section 3.

In the context of social bias detection, the use-mention distinction can be
thought of as the difference between using text for ill-intent and simply
mentioning text without this malicious undertone. To elaborate, the text “All
Muslims are terrorists" is an example of use, whereas the text “The notion
that all Muslims are terrorists is a harmful stereotype" is an example of
mention. For further contextualization of the use-mention distinction, as well
as the impact of this phenomenon on downstream social bias detection, refer to
Section 6. Extended analysis revealed that many of misclassifications from our
detector were the result of failing to differentiate use from mention. In
particular, a substantial quantity of the responses from the Llama-2 model
were of the following flavor: “It is not accurate to say that
{toxic_statement}," which precisely maps to a mention.

Our main contributions are:

  1. 1.

A fully extensible and reproducible synthetic data generation pipeline
(Section 2) which leverages taxonomy guided instructions in order to generate
high quality labeled and contrastive data at scale. Using this pipeline, we
create a novel and diverse dataset of over 300K unique samples intended to
equip guardrail models with use-mention distinction capabilities.

  2. 2.

A suite of detectors with parameter count in the $30-110$M range along with
extensive experiments (Section 3), demonstrating competitive performance
(Section 4) with state-of-the-art models while being more cost effective and
compute efficient (Section 5). We reduce false positive rate from just over
82% to 33%, with minimal impact on false negative rate, without increasing the
size of the underlying model.

##  2 Approach

In order to address the use-mention distinction problem that affected our
social-bias-detector, we sought to leverage synthetic data generation to
augment our model.

You are asked to come up with a set of {num_samples} diverse examples for the
topic of {topic}, specifically focusing on {sub_topic}. Please follow these
guiding principles when generating responses: 1. Use proper grammar and
punctuation. 2. The examples should be diverse in both style and form. 3. Each
of the examples should be in a JSONL format. 4. Each example contains a biased
statement and an unbiased statement.
To better assist you with this task, here is an example of the JSONL
formatting: {{ “biased": harmful and biased statement with a stereotype
relating to {sub_topic}, “unbiased": unbiased and benign statement with no
stereotypes relating to {sub_topic} }} Generate {num_samples} such examples
for the topic of {topic}, specifically focusing on {sub_topic}. Remember to
follow the principles mentioned above. Return your responses below in JSONL
format. Figure 1: Prompt for taxonomy-guided and contrastive synthetic data
generation

###  2.1 Taxonomy-Driven Synthetic Data Generation

We opted to utilize taxonomy driven generation, as this allowed us to explore
a rich and structured manner through which to generate data at scale. First,
we generated and iteratively updated a taxonomy of social biases. We started
by prompting the Mixtral mixture-of-experts
model222https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1, under
greedy decoding, with the instruction: Generate a high level taxonomy for
social bias that shows various groups and associated biases and stereotypes.
After post-processing, we were able to arrive at a viable taxonomy with around
50 social biases across a variety of dimensions. Some examples include Anti-
Asian bias, Transphobia, Ethnocentrism (the belief that one’s own culture is
superior to others), etc. The full taxonomy that we utilized is available in
Appendix B. Importantly, the taxonomy provides over 10 high-level topics
(e.g., Racial and Ethnic Bias, Gender Bias, Classism, etc.), and a variety of
sub-topics for each of these groups. In particular, we were encouraged to see
that our generated taxonomy included demographic groups that are not as mainly
focused on, but are still extremely relevant for the purposes hate speech
detection [27]. One such example is Polyphobia, or prejudice against
individuals who are attracted to multiple partners.

Once we generated this taxonomy, we proceeded with tailored synthetic data
generation. Here, we directly leveraged the taxonomy and generated data for
each leaf node (which is each sub-topic in our taxonomy). Using the prompt in
Figure 1, we repeatedly instructed the aforementioned Mixtral mixture-of-
experts model to generate pairs of output - jointly generating one biased and
one unbiased statement. For reproducibility, we generated under nucleus
sampling [17] with top_p at 0.95, top_k at 100, temp at 0.7, and
max_new_tokens set to 1024. This method was desirable for a couple of reasons.
First, the generation directly utilized the taxonomy, as each call for a
generation contained both the root node (i.e., the over-arching topic) as well
as the leaf node (i.e., the specific bias) directly in the context for the
prompt. Next, requiring JSON formatting in the output allowed for easy post-
processing of the LLM-generated text. Finally, and perhaps most interestingly,
asking for paired output resulted in the generation of contrastive data.
Specifically, we observed that each pair of {biased, unbiased} text precisely
mapped to {use, mention} examples! Refer to Table 1 for the flavor of
synthetic data obtained from this procedure.

biased $\longrightarrow$ use |  unbiased $\longrightarrow$ mention
---|---
All Romani people are fortune tellers and rely on trickery to make a living. |  Some individuals from the Romani community may practice fortune telling, while others have different occupations and interests.
You can’t trust someone with schizophrenia because they hear voices and can’t tell what’s real. |  People with schizophrenia may experience auditory hallucinations, but they can still be trustworthy and reliable individuals with the right treatment and support.
Pagans are superficial and materialistic. |  Pagans have different values and priorities, and some may emphasize spiritual growth, social justice, and environmental sustainability.
Table 1: Example of Contrastive Synthetic Data

In total, we generated just under 1 million pairs of data, and filtered it
down to just over 300 thousand individual samples after removing duplicates.
This took around 1 day to generate given our model hosting options. We plan to
openly release this dataset.

###  2.2 Our Suite of Social-Bias-Detectors

social-bias refers to the models trained on the four human curated datasets
mentioned in Section 2 and elaborated upon in Appendix A.1. In particular, we
have three such models - each trained on top of a different base model.
social-bias is trained on top of BERT333https://huggingface.co/google-
bert/bert-base-uncased, social-bias-distil is trained on top of a transformer
architecture [39] which provides most of the accuracy of a BERT-like model
[10], while being seven times faster on a CPU and two times faster on a GPU.
Finally, social-bias-toxigen is trained on top of
toxigen_hatebert444https://huggingface.co/tomh/toxigen_hatebert[15]. Note that
these three models were trained with only human curated datasets.

social-bias-use-mention refers to the models trained on the synthetic data
that was generated according to Section 2.1. In particular, we have four such
models. We note that social-bias-use-mention, social-bias-use-mention-distil,
and social-bias-use-mention-toxigen were trained on the entire set of unique
instances from the synthetically generated data, with the base models
following the same convention as above. Additionally, we trained social-bias-
use-mention-onetrial, which saw only one iteration of synthetic data in
training. We trained this model in order to provide a frame of reference for
the amount of synthetic data that may be required for various levels of
performance. Note that these four models have only seen synthetic data in
training.

Finally, we trained social-bias-onetrial-concat, which contained all four
human curated datasets as well as one trial of synthetic data. We trained this
model as another comparison point to determine if combining human curated and
synthetic data resulted in better performance.

We provide full details about each of the above models and their training data
in Appendix C.

#### The Cascade Approach

We also experimented with what we refer to as the cascade approach, where we
utilize two models in a sequential manner. Given some text, we first identify
a model, $m_{bias}$, to serve as a preliminary arbiter of whether this text
contains bias or not. Then, if the output from $m_{bias}$ is the harm label,
we also run the text through another model, $m_{use}$. This model determines
if the text is a case of use or mention, as defined in Section 1. If the
output from $m_{use}$ is use, then we assign the label of bias to the text.
Otherwise, we assign the label of not_bias \- indicating that either
$m_{bias}$ assigned a label of not_bias or $m_{use}$ assigned a label of
mention to this text. Refer to Algorithm 1 for specific pseudo-code of the
cascade approach.

We hypothesized that the cascade would be a “best-of-both-worlds" approach
where $m_{bias}$ would do a decent job on aggregate, due to the high quality
human curated datasets that it has seen. Then, to precisely combat the use-
mention distinction issue, where $m_{bias}$ would incorrectly flag text that
is a mention as harmful, we will run any harm-labeled instances from
$m_{bias}$ through $m_{use}$, a model that is specifically trained, by way of
our synthetic data, to distinguish use from mention.

We have four cascade approaches, defined as follows:

name |  $m_{bias}$ |  $m_{use}$
---|---|---
cascade-orig |  social-bias |  social-bias-use-mention
cascade-onetrial |  social-bias |  social-bias-use-mention-onetrial
cascade-distil |  social-bias-distil |  social-bias-use-mention-distil
cascade-toxigen |  social-bias-toxigen |  social-bias-use-mention-toxigen
Table 2: Cascade Approach Model Combinations Algorithm 1 Cascade method

1:$m_{bias}$: a model which labels text as social_bias or not_bias

2:$m_{use}$: a model which labels text as use or mention

3:$data$: some dataset with a list of responses from an LLM

4:$labels$: list of labels for each response in $data$

5:procedure cascade($m_{bias}$, $m_{use}$, $data$)

6: $labels\leftarrow\\{\\}$ $\triangleright$ initialize label list

7: for each response $r$ in $data$ do

8: $label_{m_{bias}}\leftarrow m_{bias}(r)$ $\triangleright$ run r through
$m_{bias}$

9: if $label_{m_{bias}}$ is social_bias then $\triangleright$ need to run
through use-mention model

10: $label_{r}\leftarrow m_{use}(r)$$\triangleright$ run r through $m_{use}$

11: else

12: $label_{r}\leftarrow label_{m_{bias}}$ $\triangleright$ label is not_bias
from $m_{bias}$

13: end if

14: $labels\leftarrow labels+label_{r}$ $\triangleright$ append label

15: end for

16: return $labels$

17:end procedure

##  3 Experiments

In this section, we first describe the evaluations that we conducted and then
provide empirical evidence for the cascade approach (defined in Algorithm 1)
and its utility.

###  3.1 Evaluation Datasets and Baselines

#### Collecting Evaluation Data

Due to the fact that our primary axis for evaluation is the use-mention
distinction, we sought to construct an evaluation set which reflected this
goal. Even though datasets for such a task are not widely available [14], we
leveraged the rich literature on counter-speech and counter-narratives in the
context of hate speech. In particular, similar to [14], we take the Knowledge-
grounded hate countering [9] and the Multi-Target CONAN [13] datasets. The
knowledge-grounded dataset contains 195 hate speech and counter-narrative
pairs covering multiple hate targets (islamophobia, misogyny, antisemitism,
racism, and homophobia). The counter narratives are written by an expert who
is tasked with composing a suitable counter-narrative response to a given hate
speech using the corresponding knowledge as much as possible [9]. The Multi-
Target CONAN dataset consists of 5003 hate Speech and counter-narrative pairs
covering multiple hate targets, including disabled, Jews, LGBT+, migrants,
Muslims, people of color (POC), women. The dataset is constructed using a
novel human-in-the-loop data collection methodology [13].

We combined both datasets and labeled each pair of {hate speech, counter
narrative} as {bias, not bias}. In addition, we noted that a label of bias
most precisely meant a label of use and similarly for not bias and mention.
This is because counter-narratives are written such that they directly
counteract the harmful hate speech [14], which results in them being excellent
examples of mentions.

#### Experimental Setup

We provide two sets of evaluations below. First, in order to compare with the
results from [14], we utilize the same evaluation set of 180 total examples
(also taken from the above two datasets), including both hate speech and
counter-narratives. Second, we combine the entirety of the Knowledge-grounded
hate countering and Multi-Target CONAN datasets to arrive at an evaluation set
of size $10,396$555This comes from combining 195 pairs with 5003 pairs from
the Knowledge-grounded hate countering and Multi-Target CONAN datasets,
respectively..

#### Baselines

In addition to the reported numbers with three of the GPT models from [14], we
also report numbers using Llama-Guard666https://huggingface.co/meta-
llama/LlamaGuard-7b and Llama-Guard-2777https://huggingface.co/meta-
llama/Meta-Llama-Guard-2-8B. We use these two models as both a competitive
baseline as well as a point of comparison due to the size and latency of these
models. To reiterate, the detectors are on the order of 100M parameters,
Llama-Guard models are either 7B or 8B parameters, and GPT-4 is rumored to be
orders of magnitude larger [30].

##  4 Experimental Results

In this section, we provide results on the aforementioned datasets, with all
of our detectors and baselines. We report three metrics of interest: false
positive rate (FPR), false negative rate (FNR), and average error rate (Avg
Err) - similar to [14]. For clarification, the average error rate is the
average of the FPR and FNR. To further contextualize these metrics, a false
positive refers to incorrectly providing a label of bias to benign text.
Moreover, because all of the examples of benign text are examples of mentions
and all the examples of harmful text are uses, a false positive represents
incorrectly flagging a mention as use. Symmetrically, a false negative refers
to incorrectly flagging a use as mention. For our purposes, we note that both
false positives and false negatives are important. False positives represent
improper moderation by flagging benign text whereas false negatives represent
missed detection by failing to flag harmful text.

We provide results below and note for all of the detectors (which are encoder-
only models), generation is deterministic. For Llama-Guard models, we use a
standard template888A sample notebook for inference with Llama-Guard models is
available from the HuggingFace pages above and greedy decoding.

model |  FPR |  FNR |  Avg Err |  # params
---|---|---|---|---
gpt-3.5-instruct-turbo∗ |  25.56 |  13.33 |  19.44 |  ?
gpt-3.5-turbo (ChatGPT 3.5)∗ |  11.11 |  22.22 |  16.67 |  ?
gpt-4∗ |  8.89 |  20.00 |  14.44 |  ?
Llama-Guard |  12.22 |  20.00 |  16.11 |  7B
Llama-Guard-2 |  4.44 |  26.67 |  15.56 |  8B
toxigen-hatebert |  53.33 |  12.22 |  32.78 |  110M
social-bias |  90.00 |  18.89 |  54.44 |  110M
social-bias-use-mention |  34.44 |  23.33 |  28.89 |  110M
social-bias-use-mention-onetrial |  32.22 |  37.78 |  35.00 |  110M
social-bias-onetrial-concat |  72.22 |  20.00 |  46.11 |  110M
cascade-orig |  32.22 |  33.33 |  32.78 |  110M x 2
cascade-onetrial |  26.67 |  47.78 |  37.22 |  110M x 2
social-bias-distil |  95.56 |  5.56 |  50.56 |  39M
social-bias-use-mention-distil |  26.67 |  24.44 |  25.56 |  39M
cascade-distil |  24.44 |  28.89 |  26.67 |  39M x 2
Table 3: Results on Evaluation Set with 180 examples, ∗ denotes results taken from [14] model |  FPR |  FNR |  Avg Err |  # params
---|---|---|---|---
Llama-Guard |  9.27 |  5.44 |  7.36 |  7B
Llama-Guard-2 |  1.83 |  16.62 |  9.22 |  8B
toxigen-hatebert |  45.29 |  45.29 |  26.39 |  110M
social-bias |  82.40 |  4.96 |  43.68 |  110M
social-bias-use-mention |  36.63 |  4.10 |  20.36 |  110M
social-bias-use-mention-onetrial |  43.31 |  9.37 |  26.34 |  110M
social-bias-onetrial-concat |  70.55 |  3.96 |  37.25 |  110M
cascade-orig |  32.69 |  8.31 |  19.84 |  110M x 2
cascade-onetrial |  35.30 |  13.49 |  24.39 |  110M x 2
social-bias-distil |  96.58 |  1.77 |  49.17 |  39M
social-bias-use-mention-distil |  38.15 |  5.10 |  21.62 |  39M
cascade-distil |  37.26 |  6.68 |  21.97 |  39M x 2
Table 4: Results on Combined Evaluation Set with 10K examples

##  5 Discussion

###  5.1 Results for 180 Examples Test Set

First, we comment on results for the evaluation set of 180 examples from [14].
Despite the small size of this test, we wanted to demonstrate performance
because [14] provided us with points of comparison to the GPT-family of
models. In particular, we observe that our cascade models provide competitive
performance with the GPT models. We make particular note of cascade-onetrial.
Recall that the model which determines use or mention $(m_{use})$ for cascade-
onetrial has only seen one iteration of synthetic data generation, making it
extremely desirable, as the burden of training data is substantially reduced.
In absolute terms, $(m_{use})$ for cascade-onetrial saw around 1K examples in
training, whereas $(m_{use})$ for cascade-distil saw around 190K examples in
training (refer to Appendix C) for full details). Therefore, we are able to
see that for this (albeit limited) set of data, the cascade approach is able
to perform on par with some of the most widely used and largest models, while
having a fraction of the computational cost. We do notice that both Llama-
Guard models surpass even GPT-4 by a significant margin. Nevertheless, for
this set of 180 examples, the cascade approach takes under 1 minute on a
single A100 GPU, and is even able to be run on a CPU (taking a few hours). In
comparison, access to GPT models is limited to querying via expensive external
APIs, and inference with Llama-Guard models is not possible on a CPU.

###  5.2 Results for Full (10K Examples) Test Set

Next, we comment on results for the combined evaluation set of 10K examples.
Here, we first point to the stellar performance from both Llama-Guard models.
However, as previously mentioned, inference with Llama-Guard models is not
possible on a CPU and takes several hours on an A100 GPU for a test set of
this size. For our detectors, we observe that all of the detectors which have
only seen human curated data (social-bias, social-bias-distil, and social-
bias-toxigen) have extremely high false positive rates. This implies that they
perform poorly at distinguishing between use and mention, which reaffirms our
initial motivations described in Section 1. Even when we add a little bit of
synthetic data and combine it with the human curated data, as in the case of
social-bias-onetrial-concat, we see a rather high false positive rate. We also
find better, but not optimal, performance in the detectors which have only
seen synthetic data (social-bias-use-mention, social-bias-use-mention-distil,
and social-bias-use-mention-toxigen). These models tend to have lower false
positive rates, but still not the best performance - implying that human
curated data still contains some signal or information which is not present in
the synthetic data.

This brings us to the performance of our cascade approaches. We observe the
lowest FPR among all of our detectors in cascade-orig. On aggregate, the
cascade approaches have lower false positive rates, at the cost of slightly
higher false negative rates. This is a direct consequence of the approach
itself (defined in Algorithm 1), as the method only alters labels that have
already been labeled as 1 or harm (by $m_{bias}$). However, we see that the
cascades are still under-performing Llama-Guard-2. We find that cascade-orig
is around 10% worse in terms of average error rate than Llama-Guard-2, but
interestingly, the cascade approaches do better in terms of false negative
rate - which suggests that they are better than Llama-Guard-2 at understanding
which data points are strictly cases of hate speech or use. Nevertheless, we
observe that there is still a decent gap in the false positive rates between
any cascade approach and the Llama-Guard models - implying under-performance
in discerning between use and mention, with the cascade approaches having a
tendency to provide a label of bias, leading to inaccurately labeling true
mentions as uses. We observe that cascade-onetrial performs slightly worse
than the best cascade, cascade-orig, but not too dramatically so. This is
important again for the same computational efficiency reasons that were
previously mentioned.

#### Significance of Low FNR

Despite the gap in false positive rates, we bring attention to the low false
negative rates for our approaches. In fact, all of the detectors and cascades
have lower FNRs than Llama-Guard-2. Importantly, false negatives are directly
representative of a detector’s utility as a guardrail. A high false negative
rate implies that the detector fails to flag text that is harmful, thereby
allowing toxic content to be displayed to an end user. Hence, it is imperative
to retain a lower FNR such that the detectors function as effective
guardrails. We are encouraged to see as much as a 50% decrease in FPR, at
minimal expense to FNR, when comparing social-bias with cascade-orig \-
indicating the effectiveness of our synthetic data.

###  5.3 Extended Analysis on Llama-Guard-2

Ignoring the increased computational overhead, we further explore the
differences in performance (with respect to false positives) when comparing
our detectors with the Llama-Guard models. At an instance level, we explored
the differences in predictive quality between Llama-Guard and our detectors.
For example, the sample Did you know that children with Down syndrome, if
surrounded by regularly-developed peers, receive a fundamental boost in their
development? This "mix" fosters a positive learning atmosphere and increases
engagement and attention in all students. This mean that children with Down
syndrome would be a burden to society if isolated, not the contrary. was
incorrectly flagged as unsafe by Llama-Guard-2, but properly classified as
not_bias by our cascade-orig method. Another such sample is Even if all
British Muslims had cancer, you are safe: you cannot catch it from anyone!
Cancer cannot be spread, since it is a non contagious disease. Also, it is
worth considering that illnesses are not connected to religions.

In both of these cases, we observe the presence of mentioning text, which is
throwing off even Llama-Guard-2. Hence, the reason for the cascade-orig method
being able to properly classify these instances stems from the deliberate
synthetic data generation and subsequent training for the use-mention
distinction, but also highlights the need for continued improvement on our
detectors as well to properly capture this phenomenon.

##  6 Related Work

#### Social Bias in Language Models

Social bias can be defined as discrimination for, or against, a person or
group, or a set of ideas or beliefs, in a way that is prejudicial or unfair
[41, 7]. Numerous studies have demonstrated that generative models exhibit
undesirable behavior that amplifies social bias [3, 31, 36, 35, 11, 27].
Additionally, the deficiencies of current datasets [4] and opacity of defining
what constitutes social bias in language models and their measures [3, 35, 1]
have emerged.

#### Use-Mention Distinction

As discussed briefly at the end of Section 1, the use-mention distinction can
be thought of as the difference between using text for ill-intent and simply
mentioning text without this malicious undertone. Although a subtle
difference, this phenomenon is essential for many downstream applications. For
example, a significant amount of content online falls under the umbrella of
mentions, such as counter-speech, media reporting, education, and legal
settings [14, 25, 22, 16, 42]. Of particular use to us is the notion of
counter-speech, which refers to speech produced by users of online platforms
to counteract harmful speech of others [14]. By definition, counter-speech is
an example of mention, and thus detection systems which are unable to
distinguish use from mention risk contributing to downstream harm - such as
improper removal of counter-speech. This in turn reduces opportunities to
rectify false narratives and risks further censoring those already most
affected by harmful language [14]. Recently, there has also been work which
documents the real experiences of counter-speakers and the ways in which to
address existing barriers [26]. An exciting by-product of our work was our own
introduction to this rich space, and we believe that the area of counter-
speech will continue to provide essential help in the plight of detecting and
mitigating harmful content produced by generative models.

#### Guardrails for LLMs

There is a growing set of rich literature on guardrail models and how they can
be more efficient, modular, and scalable [1, 33, 19]. NeMo Guardrails [33] is
an open-source toolkit for adding modular and programmable guardrails to LLM-
based conversational systems. Llama-Guard [19] is an LLM-based input-output
safeguard model which has a customizable taxonomy of harms. Finally, the
authors in [1] go over insights from developing a host of guardrail-like
models, which they refer to as detectors.

#### Synthetic Data Generation

Central to our work is the leveraging of capable large language models to
generate synthetic data which can be used to train our guardrail models. In
particular, we leverage taxonomy-guided data generation. Previous work such as
GLAN [23] utilizes a pre-curated taxonomy of human knowledge and capabilities
as input and generates large-scale synthetic instruction data. However, as
pointed out by [37], this method relies using the proprietary GPT-4 [30] as
the teacher model, which imposes restrictions on the downstream commercial
usability of the generated data. Concretely, the terms of use of proprietary
models obfuscate the viability of training commercially viable models using
data that is generated from proprietary or otherwise closed models - which
many times forbid using their model to improve other models 999See point v
under License Rights in https://llama.meta.com/llama3/license/. Thus, similar
to [37], we utilize the open source
Mixtral101010https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 model
to generate our data. Additionally, the taxonomy guides the generation of the
data, as it enables targeted coverage around the individual leaf nodes of the
taxonomy [37].

##  7 Limitations and Future Work

###  7.1 Limitations

#### Coverage of Synthetic Data

We acknowledge that our taxonomy (even the full version in Appendix B) cannot
cover all groups which are salient for social bias. This taxonomy is a point-
in-time artifact, and we note that social bias dynamically evolves over time.
Because social harms are the product of context-dependent classification
systems with deep historical roots and are socially and morally charged,
careful attention must be paid to the choices made (such as the groups
included in a taxonomy) during development of the detectors [1]. Concretely,
we understand that generated counter-speech may not always be reflective of
real-world harm. One way to combat this, though expensive, is via human
annotations with true counter-speakers [26]. Another direction is to leverage
LLMs themselves to provide feedback on a given counter-speech example [20]. We
are experimenting with both avenues in the near future.

We also point out that closed-source LLMs tend to generate high quality data,
but their proprietary nature is prohibitive for downstream use. Hence our
decision to opt against use of GPT models for generation.

###  7.2 Future Work

#### Calibration and Confidence-Based Improvements

In order to combat overconfidence, we are considering conformal prediction
approaches [40]. These approaches quantify uncertainty in a model’s prediction
by constructing predictive sets (as opposed to singleton labels, in the case
of our detectors) with guaranteed frequentist coverage probabilities.
Specifically, we are looking into the regularized adaptive prediction sets
approach [34, 2] which, in addition to providing coverage guarantees, produces
larger (or non-singleton in the case of our detectors) prediction sets for
difficult instances and smaller (singleton) sets for easier to classify
examples. Then, on the instances for which our models produce these larger
sets, we are exploring is the idea of a “confident-cascade" wherein we adopt a
collaborative method and offload these inputs to Llama-Guard. We conjecture
this may provide a reasonable trade-off between computational efficiency and
performance.

#### Novel Development

Currently, detectors are being trained in a supervised fine-tuning method with
the standard binary cross-entropy loss. Given that our synthetic data is
generated in a contrastive manner, we will consider training with contrastive
loss [21]. Additionally, we are looking into directly training a multi-head
detector, which eliminates the need for two models in our cascade approach.

##  8 Conclusion

In this work, we began with insights from the development of a social bias
detector. Post deployment, we recounted our realization that issues arose due
to the use-mention distinction. Motivated by this discovery, we described an
extensible and reproducible synthetic data generation pipeline, which
leverages taxonomy guided instructions in order to generate high quality
labeled and contrastive data at scale. We then documented the training
procedures of various models which utilized this synthetic data and introduced
the cascade approach. Next, we outlined the extensive experiments performed
with these models on a variety of evaluation datasets. We revealed that the
cascade approach provided competitive performance on these evaluation sets, in
addition to being more substantially more cost effective and compute
efficient. We hope that our findings contribute to the growing body of work on
building efficient and capable guardrails for large language models.

## References

  * [1] Swapnaja Achintalwar, Adriana Alvarado Garcia, Ateret Anaby-Tavor, Ioana Baldini, Sara E. Berger, Bishwaranjan Bhattacharjee, Djallel Bouneffouf, Subhajit Chaudhury, Pin-Yu Chen, Lamogha Chiazor, Elizabeth M. Daly, Rogério Abreu de Paula, Pierre Dognin, Eitan Farchi, Soumya Ghosh, Michael Hind, Raya Horesh, George Kour, Ja Young Lee, Erik Miehling, Keerthiram Murugesan, Manish Nagireddy, Inkit Padhi, David Piorkowski, Ambrish Rawat, Orna Raz, Prasanna Sattigeri, Hendrik Strobelt, Sarathkrishna Swaminathan, Christoph Tillmann, Aashka Trivedi, Kush R. Varshney, Dennis Wei, Shalisha Witherspooon, and Marcel Zalmanovici.  Detectors for safe and reliable llms: Implementations, uses, and limitations.  arXiv preprint arXiv:2403.06009, 2024.
  * [2] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik.  Uncertainty sets for image classifiers using conformal prediction.  In International Conference on Learning Representations, 2021.
  * [3] Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach.  Language (technology) is power: A critical survey of “bias” in NLP.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
  * [4] Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach.  Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2021.
  * [5] IBM AI Ethics Board.  Foundation models: Opportunities, risks and mitigations.  https://www.ibm.com/downloads/cas/E5KE5KRZ.
  * [6] Rishi Bommasani.  Ai spring? four takeaways from major releases in foundation models, 2023.
  * [7] Rishi Bommasani and Percy Liang.  Trustworthy social bias measurement, 2022.
  * [8] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman.  Nuanced metrics for measuring unintended bias with real data for text classification.  CoRR, abs/1903.04561, 2019.
  * [9] Yi-Ling Chung, Serra Sinem Tekiroğlu, and Marco Guerini.  Towards knowledge-grounded counter narrative generation for hate speech.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, Online, August 2021. Association for Computational Linguistics.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [11] Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta.  BOLD: dataset and metrics for measuring biases in open-ended language generation.  In FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 862–872, 2021.
  * [12] Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang.  Latent hatred: A benchmark for understanding implicit hate speech.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345–363, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * [13] Fanton, Margherita and Bonaldi, Helena and Tekiroğlu, Serra Sinem and Guerini, Marco.  Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, August 2021.
  * [14] Kristina Gligoric, Myra Cheng, Lucia Zheng, Esin Durmus, and Dan Jurafsky.  Nlp systems that can’t tell use from mention censor counterspeech, but teaching the distinction helps, 2024.
  * [15] Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar.  ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3309–3326, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [16] Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho.  Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 29217–29234. Curran Associates, Inc., 2022.
  * [17] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  In International Conference on Learning Representations, 2020.
  * [18] 2023\.  https://www.ibm.com/docs/en/watsonx-as-a-service?topic=ai-risk-atlas.
  * [19] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa.  Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.
  * [20] Jaylen Jones, Lingbo Mo, Eric Fosler-Lussier, and Huan Sun.  A multi-aspect framework for counter narrative evaluation using large language models, 2024.
  * [21] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning, 2021.
  * [22] Hannah Kirk, Abeba Birhane, Bertie Vidgen, and Leon Derczynski.  Handling and presenting harmful text in NLP research.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 497–510, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  * [23] Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei.  Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024.
  * [24] Tom Mitchell.  Twenty Newsgroups.  UCI Machine Learning Repository, 1999.  DOI: https://doi.org/10.24432/C5C323.
  * [25] Jimin Mun, Emily Allaway, Akhila Yerukola, Laura Vianna, Sarah-Jane Leslie, and Maarten Sap.  Beyond denouncing hate: Strategies for countering implied biases and stereotypes in language.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9759–9777, Singapore, December 2023. Association for Computational Linguistics.
  * [26] Jimin Mun, Cathy Buerger, Jenny T Liang, Joshua Garland, and Maarten Sap.  Counterspeakers’ perspectives: Unveiling barriers and ai needs in the fight against online hate.  In Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA, 2024. Association for Computing Machinery.
  * [27] Manish Nagireddy, Lamogha Chiazor, Moninder Singh, and Ioana Baldini.  SocialStigmaQA: A benchmark to uncover stigma amplification in generative language models.  Proceedings of the AAAI Conference on Artificial Intelligence, 38(19):21454–21462, Mar. 2024.
  * [28] Pandu Nayak.  Understanding searches better than ever before, Oct 2019.
  * [29] OpenAI.  ChatGPT: Optimizing Language Models for Dialogue, 2022.  [https://openai.com/blog/chatgpt/Online].
  * [30] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, 2024.
  * [31] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman.  BBQ: A hand-built bias benchmark for question answering.  In Findings of the Association for Computational Linguistics: ACL 2022, 2022.
  * [32] Perspective API.  Using Machine Learning to Reduce Toxicity Online, 2021.  [https://perspectiveapi.com/how-it-works/Online; accessed 21-July-2021].
  * [33] Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen.  NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails.  In Yansong Feng and Els Lefever, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 431–445, Singapore, December 2023. Association for Computational Linguistics.
  * [34] Yaniv Romano, Matteo Sesia, and Emmanuel Candes.  Classification with valid and adaptive coverage.  Advances in Neural Information Processing Systems, 33:3581–3591, 2020.
  * [35] Nikil Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, and Kai-Wei Chang.  The tail wagging the dog: Dataset construction biases of social bias benchmarks.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1373–1386, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * [36] Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams.  “I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9180–9211, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  * [37] Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D. Cox, and Akash Srivastava.  Lab: Large-scale alignment for chatbots, 2024.
  * [38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * [39] Aashka Trivedi, Takuma Udagawa, Michele Merler, Rameswar Panda, Yousef El-Kurdi, and Bishwaranjan Bhattacharjee.  Neural architecture search for effective teacher-student knowledge transfer in language models, 2023.
  * [40] Volodya Vovk, Alexander Gammerman, and Craig Saunders.  Machine-learning applications of algorithmic randomness.  In Proceedings of the International Conference on Machine Learning, 1999.
  * [41] Craig S Webster, Saana Taylor, Courtney Thomas, and Jennifer M Weller.  Social bias, discrimination and inequity in healthcare: mechanisms, implications and recommendations.  BJA education, 22(4):131—137, April 2022.
  * [42] Lesley M Wexler, Jennifer Kirkpatrick Robbennolt, and Colleen Murphy.  #metoo, time’s up, and theories of justice.  University of Illinois Law Review, 2019(1):45–111, 2019.
  * [43] Michael Wiegand, Elisabeth Eder, and Josef Ruppenhofer.  Identifying implicitly abusive remarks about identity groups using a linguistically informed approach.  In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5600–5612, Seattle, United States, July 2022. Association for Computational Linguistics.

##  Appendix A Social Bias Detector

As mentioned in Section 1, we trained a social-bias-detector by fine-tuning
BERT [10]. In particular, we took the uncased BERT model from
HuggingFace111111https://huggingface.co/google-bert/bert-base-uncased. During
training, we use a batch size of 16, we start with a learning rate of
$1\text{e-}6$, and we train for 50 epochs, taking the best model with respect
to validation f1 score. For reference, the total training time was a few hours
on a single A100 GPU.

###  A.1 Training Data

We started with the Latent Hatred [12] dataset, which is a benchmark that was
specifically designed for implicit hate speech. Then, we use the 20 NewsGroups
dataset [24] in order to add some out of distribution data. Note that we
deliberately use this dataset in the hopes of increasing the proportion of
negative (i.e., benign) labeled data in our training set. Third, we use a
dataset from a work titled Identifying Implicitly Abusive Remarks about
Identity Groups using a Linguistically Informed Approach [43], as this dataset
attempted to decouple harmful intent from mention of specific group
identities. In fact, this concept is related to the use-mention distinction
that we discuss throughout the paper. We refer to this dataset as “Identity
Groups" in Table 5 below. Finally, we add in a subset of the CivilComments
dataset [8], taking only samples which have an identity_attack column value of
greater than 0.5, which we believe corresponded to implicitly hateful
comments.

###  A.2 Evaluation

We compute evaluations with this version of our social-bias-detector on the
test splits of each of the 4 datasets used in training that were mentioned
above.

test dataset |  accuracy |  balanced accuracy |  precision |  recall |  f1
---|---|---|---|---|---
implicit-hate |  0.754 |  0.747 |  0.616 |  0.724 |  0.665
blocklisting |  0.676 |  0.676 |  - |  - |  -
identity groups |  0.752 |  0.732 |  0.729 |  0.891 |  0.802
civil comments |  0.974 |  0.974 |  1.0 |  0.974 |  0.987
Table 5: Results for social-bias model on evaluation sets

Some points to note:

  * •

the blocklisting data only contains benign (i.e. negatively or 0-labeled
examples). Hence, precision/recall/f1 do not apply (they are trivially equal
to 0)

  * •

When evaluating, we predominantly focus on f1 score, in order to balance both
false positives and false negatives.

##  Appendix B Taxonomy-Guided Synthetic Data Generation

As mentioned in Section 2, we prompted the Mixtral mixture-of-experts
model121212https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1, under
greedy decoding, with the instruction: Generate a high level taxonomy for
social bias that shows various groups and associated biases and stereotypes.
in order to generate a taxonomy. To be specific, we obtained a resultant
taxonomy after this prompt, post-processed it, and then re-prompted the same
model with the entire taxonomy passed in the context along with an instruction
to Continue the taxonomy while maintaining the structure. We observed that one
such iteration was enough to produce a viable taxonomy which can be used in
the synthetic data generation process outlined in Section 2.

We provide the full taxonomy below:

⬇

{

"Racial and Ethnic Bias": [

"Anti-Black bias",

"Anti-Latino bias",

"Anti-Asian bias",

"Anti-Indigenous bias",

"Anti-Middle Eastern bias",

"Anti-White bias",

"Anti-Romani bias",

"Anti-Semitic bias",

"Anti-Arab bias",

"Anti-Indian bias",

"Anti-Pacific Islander bias"

],

"Gender Bias": [

"Misogyny (prejudice against women)",

"Misandry (prejudice against men)",

"Transphobia (prejudice against transgender individuals)",

"Non-binary erasure (prejudice against individuals who do not identify as male
or female)",

"Gender nonconformity discrimination",

"Intersex erasure (prejudice against individuals who are born with physical
characteristics that do not fit typical binary notions of male or female
bodies)"

],

"Religious Bias": [

"Islamophobia (prejudice against Muslims)",

"Anti-Semitism (prejudice against Jews)",

"Christianophobia (prejudice against Christians)",

"Anti-Hindu bias",

"Anti-Buddhist bias",

"Anti-Sikh bias",

"Anti-Pagan bias",

"Anti-Atheist bias",

"Anti-agnostic bias"

],

"Sexual Orientation Bias": [

"Homophobia (prejudice against individuals who are attracted to the same
gender)",

"Biphobia (prejudice against individuals who are attracted to multiple
genders)",

"Transphobia (prejudice against transgender individuals)",

"Asexual erasure (prejudice against individuals who do not experience sexual
attraction)",

"Panphobia (prejudice against individuals who are attracted to all genders)",

"Polyphobia (prejudice against individuals who are attracted to multiple
partners)"

],

"Ableism": [

"Disability discrimination",

"Ableist language",

"Sanism (prejudice against individuals with mental illness)",

"Neurodivergent erasure (prejudice against individuals with neurological
differences, such as autism or ADHD)"

],

"Ageism": [

"Prejudice against older adults",

"Prejudice against younger adults",

"Adultism (prejudice against children and young people)"

],

"Classism": [

"Prejudice against individuals from lower socioeconomic backgrounds",

"Prejudice against individuals from higher socioeconomic backgrounds",

"Economic exploitation"

],

"Lookism": [

"Prejudice against individuals based on physical appearance",

"Colorism (prejudice against individuals with darker skin tones)",

"Sizeism (prejudice against individuals based on their body size or shape)"

],

"Xenophobia": [

"Prejudice against individuals from other countries or cultures",

"Cultural imperialism (the belief that one culture is superior to another and
should be imposed on others)"

],

"Cultural Bias": [

"Ethnocentrism (the belief that one’s own culture is superior to others)",

"Linguisticism (prejudice against individuals who speak a different language)"

],

"Political Bias": [

"Prejudice against individuals with different political beliefs",

"Polarization (the tendency to view those with different political beliefs as
enemies rather than opponents)"

],

"Environmental Bias": [

"Prejudice against individuals who prioritize environmental concerns",

"Climate change denial"

]

}

##  Appendix C List of Detectors

name |  base model |  training data |  train / val / test size
---|---|---|---
social-bias |  BERT |  human curated |  27113 / 8873 / 10051
social-bias- distil |  Piccolo |  human curated |  27113 / 8873 / 10051
social-bias- toxigen |  toxigen_hatebert |  human curated |  27113 / 8873 / 10051
social-bias- use-mention |  BERT |  full synthetic |  189669 / 31611 / 94835
social-bias- use-mention-distil |  Piccolo |  full synthetic |  189669 / 31611 / 94835
social-bias- use-mention- toxigen |  toxigen_hatebert |  full synthetic |  189669 / 31611 / 94835
social-bias- use-mention- onetrial |  BERT |  one trial synthetic |  1050 / 350 / 350
social-bias- onetrial-concat |  BERT |  human curated, one trial synthetic |  28168 / 9225 / 10404
Table 6: Suite of Social Bias Detectors

All models were trained using a batch size of 16, a learning rate of 0.000001,
and for 50 epochs, taking the best model with respect to validation f1 score.
For reference, the total training time for each model was between 30 minutes
to a few hours on a single A100 GPU.