\begin{table}[t]
\footnotesize
\begin{tabular}{lrl}
\hline
name & \multicolumn{1}{l}{\# examples} & classes \\ 
\hdashline
HateCheck & 3,728 & \begin{tabular}[c]{@{}l@{}}hateful (68.8\%), \\ non-hate (31.2\%)\end{tabular} \\
ETHOS (binary) & 997 & \begin{tabular}[c]{@{}l@{}}hate speech (64.1\%), \\ not-hate speech (25.9\%)\end{tabular} \\ \hline
\end{tabular}
\caption{The number of examples and the class balance of the datasets.}
\label{tab:datasets}
\end{table}
\begin{abstract}
Standard approaches to hate speech detection rely on sufficient available hate speech annotations. Extending previous work that repurposes natural language inference (NLI) models for zero-shot text classification, we propose a simple approach that combines multiple hypotheses to improve English NLI-based zero-shot hate speech detection. 
We first conduct an error analysis for vanilla NLI-based zero-shot hate speech detection and then develop four strategies based on this analysis. The strategies use multiple hypotheses to predict various aspects of an input text and combine these predictions into a final verdict. We find that the zero-shot baseline used for the initial error analysis already outperforms commercial systems and fine-tuned BERT-based hate speech detection models on HateCheck. The combination of the proposed strategies further increases the zero-shot accuracy of 79.4\% on HateCheck by 7.9 percentage points (pp), and the accuracy of 69.6\% on ETHOS by 10.0pp.\footnote{The code and instructions to reproduce the experiments are available at \url{https://github.com/jagol/nli-for-hate-speech-detection}.}
\end{abstract}

\section{Introduction}
With the increasing popularity of social media and online forums, phenomena such as hate speech, offensive and abusive language, and personal attacks have gained a powerful medium through which they can propagate fast. 
Due to the sheer number of posts and comments on social media, manual content moderation has become unfeasible, thus
the automatic detection of harmful content becomes essential.
In natural language processing, there now exist established tasks with the goal of detecting offensive language \cite{pradhan_review_2020}, abusive language \cite{nakov_detecting_2021}, hate speech \cite{fortuna_survey_2018} and other related types of harmful content \cite{poletto_resources_2021}. 
In this work, we focus on the detection of hate speech, which is typically defined as attacking, abusive or discriminatory language that targets people on the basis of identity defining group characteristics such as gender, sexual orientation, disability, race, religion, national origin etc. \cite{fortuna_survey_2018, poletto_resources_2021, yin_towards_2021}.
Most current hate speech detection approaches rely on either training models from scratch or fine-tuning pre-trained language models \cite{jahan_systematic_2021}.
Both types of approaches need large amounts of labeled data which are only available for a few high-resource languages \cite{poletto_resources_2021} and costly to create.
Therefore, exploring data-efficient methods for hate speech detection is an attractive alternative. 

\section{Background and Related Work}
Early approaches to hate speech detection have focused on English social media posts, especially Twitter, and treated the task as binary or ternary text classification \cite{waseem_hateful_2016, davidson_automated_2017, founta_large_2018}. 
In more recent work, additional labels have been introduced that indicate whether the post is group-directed or not, who the targeted group is, if the post calls for violence, is aggressive, contains stereotypes, if the hate is expressed implicitly, or if sarcasm or irony is used \cite{mandl_overview_2019, mandl_overview_2020, sap_social_2020, elsherief_latent_2021, rottger_hatecheck_2021, mollas_ethos_2022}. 
Sometimes hate speech is not directly annotated but instead labels, such as \textit{racism}, \textit{sexism}, \textit{homophobia} that already combine hostility with a specific target are annotated and predicted \cite{waseem_hateful_2016, waseem_are_2016, saha_hateminers_2018, basile_thenorth_2020}.

The advent of large language models has enabled zero-shot and few-shot text classification approaches such as prompting \cite{liu_pre-train_2021}, and task descriptions \cite{raffel_exploring_2020}, which convert the target task to the pre-training objective
and are usually only used in combination with large language models. 
\citet{chiu_detecting_2021} use the prompts \textit{``Is this text racist?''} and \textit{``Is this text sexist?''} to detect hate speech with GPT-3. 
\citet{schick_self-diagnosis_2021} show that toxicity in large generative language models can be avoided by using similar prompts to self-diagnose toxicity during the decoding.

In contrast, NLI-based prediction in which a target task is converted to an NLI-task and fed into an NLI model converts the target task to the fine-tuning task. 
Here, a model is given a premise and a hypothesis and tasked to predict if the premise entails the hypothesis, contradicts it, or is neutral towards it.  
\citet{yin_benchmarking_2019} proposed to use an NLI model for zero-shot topic classification, by inputting the text to classify as the premise and constructing for each topic a hypothesis of the form \enquote{This text is about $<$topic$>$}. 
They map the labels \textit{neutral} and \textit{contradiction} to \textit{not-entailment}. We can then interpret a prediction of entailment as predicting that the input text belongs to the topic in the given hypothesis.
Conversely, \textit{not-entailment} implies that the text is not about the topic.
\citet{wang_entailment_2021} show for a range of tasks, including offensive language identification, that this task re-formulation also benefits few-shot learning scenarios. 
Recently, \citet{alkhamissi_token_2022} obtained large performance improvements in few-shot learning for hate speech detection by (1) decomposing the task into four subtasks and (2) additionally training the few-shot model on a knowledge base.

\section{Data}
\paragraph{HateCheck} \citet{rottger_hatecheck_2021} introduce this English, synthetic, evaluation-only dataset, annotated for a binary decision between hate speech and not-hate speech. It covers 29 functionalities that are either a type of hate speech or challenging types of non-hate speech that could be mistaken for hate speech by a classifier. The examples for each of these functionalities have been constructed on the basis of conversations with NGO workers. 
Each of these templates contains one blank space to be filled with a protected group. The authors fill these templates with seven protected groups, namely: women, gay people, transgender people, black people, Muslims, immigrants, and disabled people. Overall the dataset contains 3,728 examples. 

\paragraph{ETHOS} The ETHOS dataset \cite{mollas_ethos_2022} is split into two parts: one part is annotated for the presence of hate speech. The other part contains fine-grained annotations that indicate which characteristics have been targeted (gender, sexual orientation, race, ethnicity, religion, national origin, disability), whether the utterance calls for violence, and whether it is directed at an individual or a general statement about a group. The dataset is based on English comments from Youtube and Reddit. For this work, we will only make use of the binary hate speech annotations. These annotations are continuous values between $0$ (indicating no hate speech at all) and $1$ indicating clear hate speech. We rounded all annotations to either $0$ or $1$ using a threshold of $0.5$.

Table \ref{tab:datasets} displays the class balances of the two datasets.
{"value": "3,728", "type": "Data Stat.", "dataset": "HateCheck", "attribute name": ["number of examples", "# examples"], "sub-set/group name": "xx", "dataset features": {"xx": "yy"}} {"value": "997", "type": "Data Stat.", "dataset": "ETHOS (binary)", "attribute name": ["number of examples", "# examples"], "sub-set/group name": "xx", "dataset features": {"xx": "yy"}}