\begin{table*}[t]
\small
\centering
\begin{tabular}{l|rrrr}
hypothesis & \multicolumn{1}{l}{accuracy (\%)} & \multicolumn{1}{l}{$\downarrow$ $F_1$ (\%)} & \multicolumn{1}{l}{recall (\%)} & \multicolumn{1}{l}{precision (\%)} \\ \hline
That example is about gay people. & 99.1 & 96.7 & 94.2 & 99.4 \\
This example is about gay people. & 99.0 & 96.6 & 94.0 & 99.2 \\
This text is about gay people.    & 98.8 & 95.9 & 92.9 & 99.0 \\
This is about gay people.         & 98.8 & 95.8 & 92.6 & 99.2 \\
That text is about gay people.    & 98.5 & 94.6 & 90.0 & 99.6 \\
That is about gay people.         & 98.4 & 94.4 & 90.4 & 98.8
\end{tabular}
\caption{Results for supporting hypotheses aimed at detecting mentions of gay people. }
\label{tab:results-gay-people}
\end{table*}
In contrast, NLI-based prediction in which a target task is converted to an NLI-task and fed into an NLI model converts the target task to the fine-tuning task. 
Here, a model is given a premise and a hypothesis and tasked to predict if the premise entails the hypothesis, contradicts it, or is neutral towards it.  
\citet{yin_benchmarking_2019} proposed to use an NLI model for zero-shot topic classification, by inputting the text to classify as the premise and constructing for each topic a hypothesis of the form \enquote{This text is about $<$topic$>$}. 
They map the labels \textit{neutral} and \textit{contradiction} to \textit{not-entailment}. We can then interpret a prediction of entailment as predicting that the input text belongs to the topic in the given hypothesis.
Conversely, \textit{not-entailment} implies that the text is not about the topic.
\citet{wang_entailment_2021} show for a range of tasks, including offensive language identification, that this task re-formulation also benefits few-shot learning scenarios. 
Recently, \citet{alkhamissi_token_2022} obtained large performance improvements in few-shot learning for hate speech detection by (1) decomposing the task into four subtasks and (2) additionally training the few-shot model on a knowledge base.

\section{Data}
\paragraph{HateCheck} \citet{rottger_hatecheck_2021} introduce this English, synthetic, evaluation-only dataset, annotated for a binary decision between hate speech and not-hate speech. It covers 29 functionalities that are either a type of hate speech or challenging types of non-hate speech that could be mistaken for hate speech by a classifier. The examples for each of these functionalities have been constructed on the basis of conversations with NGO workers. 
Each of these templates contains one blank space to be filled with a protected group. The authors fill these templates with seven protected groups, namely: women, gay people, transgender people, black people, Muslims, immigrants, and disabled people. Overall the dataset contains 3,728 examples. 

\paragraph{Experiment setup} To test if an input text contains hate speech, we need a hypothesis expressing that claim. 
However, there are many ways how the claim, that a given text contains hate speech, can be expressed. Choosing a sub-optimal way to express this claim will result in lower accuracy. \citet{wang_entailment_2021} already tested four different hypotheses for hate speech or offensive language. We conduct an extensive evaluation by constructing and testing all grammatically correct sentences built with the following building blocks: \textit{It/That/This + example/text + contains/is + hate speech/hateful/hateful content}. 
We conduct all experiments with a BART-large model \cite{lewis_bart_2020} that was fine-tuned on the \textbf{M}ulti-Genre \textbf{N}atural \textbf{L}anguage \textbf{I}nference dataset (MNLI) \cite{williams_broad-coverage_2018} and has been made available via the Huggingface transformers library \cite{wolf_transformers_2020} as \texttt{bart-large-mnli}. 
This model predicts either \textit{contradiction}, \textit{neutral}, or \textit{entailment}. 
We follow the recommendation of the model creators to ignore the logits for \textit{neutral} and perform a softmax over the logits of \textit{contradiction} and \textit{entailment}. If the probability for entailment is equal or higher than 0.5 we consider this a prediction of \textit{entailment} and thus \textit{hate speech}.\footnote{This procedure is equal to taking the argmax over \textit{contradiction} and \textit{entailment}.}
We evaluate on HateCheck since the functionalities in this dataset allow for an automatic in-depth error analysis and compare our results to the baselines provided by \citet{rottger_hatecheck_2021}.

In this section, we present four methods, which we call strategies, that aim to improve zero-shot hate speech detection. A strategy has the following components and structure: The aim is to assign a label $y=\{0, 1\}$ to input text $t$, where $1$ corresponds to the class \textit{hate speech} and $0$ corresponds to the class \textit{not-hate speech}.
The input text $t$ can be used in one or multiple a premises $p_0$ to $p_m$, that are used in conjunction with the main hypothesis $h_0$ and one or multiple supporting hypotheses $\left[h_1, ..., h_n\right]$ to obtain  NLI model predictions $m(p_i, h_j) \in \{0,1\}$ where 0 corresponds to contradiction and $1$ corresponds to entailment.
The variables $i$ and $j$ are defined as: $i \in \left[0, ..., m\right]$ and $j \in \left[0, ..., n\right]$. 
The rules for how to combine model predictions to obtain the final label $y$ are given by the individual strategies.
As the main hypothesis we use \enquote{That contains hate speech.}, since it lead to the highest accuracy on HateCheck in Section \ref{sec:evaluating-standard-0shot}.
The supporting hypotheses used to implement the strategies are listed in Table \ref{tab:list-of-hypotheses}.

\subsection{Filtering By Target (FBT)}
\label{subsec:filtering-by-target}
The error analysis showed that we can improve zero-shot classification accuracy significantly by avoiding predictions of hate speech where no relevant target group occurs. We thus propose to avoid false positives by constructing a set of supporting hypotheses $\left[h_1, ..., h_n\right]$ to predict if text $t$ actually targets or mentions a protected group or characteristic. If no protected group or characteristic is predicted to occur in $t$, a potential prediction of \textit{hate speech} is overridden to \textit{not-hate speech}.
Figure \ref{fig:filtering-by-target} illustrates the method. 

Table \ref{tab:compare-hypotheses-full}, the extended version of Table \ref{tab:compare-hypotheses}, contains all results for comparing hypotheses for zero-shot hate speech detection on HateCheck. \enquote{average:$<$ \texttt{expression}$>$} refers to the average accuracy of all hypotheses containing \texttt{expression}. The highest accuracy is in bold. 

We use the same model and as in the previous zero-shot experiments for evaluation and test the performance for detecting mentions for all protected groups in HateCheck. We additionally test the detection of the supercategory \textit{queer people} covering the two protected groups \textit{gay people} and \textit{transgender people} in HateCheck. When testing if a text revolves around gender, we treat both \textit{women} and \textit{transgender people} as positive classes and all other protected groups as negative classes. While this mapping obviously can result in incorrect labels (a text can be about gender even if another group is targeted), we assume that it holds true for examples in the HateCheck dataset.

Table \ref{tab:results-black-people} shows the results for detecting if \textit{black people} are mentioned, Table \ref{tab:results-muslim-people} for mentions of \textit{Muslims}, Table \ref{tab:results-immigrants} for mentions of immigrants, Table \ref{tab:results-disabled-people} for mentions of \textit{disabled people}, Table \ref{tab:results-gay-people} for mentions of \textit{gay people}, Table \ref{tab:results-transgender-people} for mentions of \textit{transgender people}, Table \ref{tab:results-queer-people} for mentions of \textit{queer people}, and Table \ref{tab:results-gender} for detecting if a text is about \textit{gender}.
{"value": "99.1", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That example is about gay people.", "model settings": {"xx": "yy"}} {"value": "96.7", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": ["F_1", "F1"], "experimental settings": {"xx": "yy"}, "model": "That example is about gay people.", "model settings": {"xx": "yy"}} {"value": "94.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "recall", "experimental settings": {"xx": "yy"}, "model": "That example is about gay people.", "model settings": {"xx": "yy"}} {"value": "99.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "precision", "experimental settings": {"xx": "yy"}, "model": "That example is about gay people.", "model settings": {"xx": "yy"}} {"value": "99.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This example is about gay people.", "model settings": {"xx": "yy"}} {"value": "96.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": ["F_1", "F1"], "experimental settings": {"xx": "yy"}, "model": "This example is about gay people.", "model settings": {"xx": "yy"}} {"value": "94.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "recall", "experimental settings": {"xx": "yy"}, "model": "This example is about gay people.", "model settings": {"xx": "yy"}} {"value": "99.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "precision", "experimental settings": {"xx": "yy"}, "model": "This example is about gay people.", "model settings": {"xx": "yy"}} {"value": "98.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This text is about gay people.", "model settings": {"xx": "yy"}} {"value": "95.9", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": ["F_1", "F1"], "experimental settings": {"xx": "yy"}, "model": "This text is about gay people.", "model settings": {"xx": "yy"}} {"value": "92.9", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "recall", "experimental settings": {"xx": "yy"}, "model": "This text is about gay people.", "model settings": {"xx": "yy"}} {"value": "99.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "precision", "experimental settings": {"xx": "yy"}, "model": "This text is about gay people.", "model settings": {"xx": "yy"}} {"value": "98.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This is about gay people.", "model settings": {"xx": "yy"}} {"value": "95.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": ["F_1", "F1"], "experimental settings": {"xx": "yy"}, "model": "This is about gay people.", "model settings": {"xx": "yy"}} {"value": "92.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "recall", "experimental settings": {"xx": "yy"}, "model": "This is about gay people.", "model settings": {"xx": "yy"}} {"value": "99.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "precision", "experimental settings": {"xx": "yy"}, "model": "This is about gay people.", "model settings": {"xx": "yy"}} {"value": "98.5", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That text is about gay people.", "model settings": {"xx": "yy"}} {"value": "94.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": ["F_1", "F1"], "experimental settings": {"xx": "yy"}, "model": "That text is about gay people.", "model settings": {"xx": "yy"}} {"value": "90.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "recall", "experimental settings": {"xx": "yy"}, "model": "That text is about gay people.", "model settings": {"xx": "yy"}} {"value": "99.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "precision", "experimental settings": {"xx": "yy"}, "model": "That text is about gay people.", "model settings": {"xx": "yy"}} {"value": "98.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That is about gay people.", "model settings": {"xx": "yy"}} {"value": "94.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": ["F_1", "F1"], "experimental settings": {"xx": "yy"}, "model": "That is about gay people.", "model settings": {"xx": "yy"}} {"value": "90.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "recall", "experimental settings": {"xx": "yy"}, "model": "That is about gay people.", "model settings": {"xx": "yy"}} {"value": "98.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck gay people", "task": "zero-shot hate speech detection", "metric": "precision", "experimental settings": {"xx": "yy"}, "model": "That is about gay people.", "model settings": {"xx": "yy"}}