\begin{table}[h!]
\centering
\small
\begin{tabular}{lr}
\hline
hypothesis & accuracy (\%) \\ 
\hdashline
Containing hate speech. & 74.7 \\
Contains hate speech. & 78.6 \\
Hate speech. & 72.9 \\
Hateful. & 71.8 \\
\hdashline
It contains hate speech. & 78.7 \\
It is hateful. & 75.0 \\
It contains hate speech. & 78.7 \\
It is hate speech. & 70.8 \\
It is hateful. & 75.0 \\
That contains hate speech. & \textbf{79.4} \\
That contains hateful content. & 78.0 \\
That example contains hateful content. & 77.8 \\
That example is hate speech. & 66.6 \\
That example is hateful. & 76.8 \\
That is hateful. & 66.6 \\
That text contains hate speech. & 78.8 \\
That text contains hateful content. & 78.6 \\
That text is hate speech. & 69.2 \\
That text is hateful. & 77.2 \\
This contains hate speech. & 79.1 \\
This contains hateful content. & 78.2 \\
This example contains hate speech. & 77.3 \\
This example contains hateful content. & 77.8 \\
This example is hate speech. & 67.2 \\
This example is hateful. & 77.4 \\
This is hateful. & 70.6 \\
This text contains hate speech. & 78.8 \\
This text contains hateful content. & 78.3 \\
This text is hate speech. & 69.5 \\
This text is hateful. & 78.7 \\
\hdashline
average: It & 74.8 \\
average: This & 75.7 \\
average: That & 74.9 \\
average: hateful & 75.8 \\
average: hateful content & 78.1 \\
average: hate speech & 74.5 \\
average: example & 74.4 \\
average: text & 76.1 \\
average: is & 73.9 \\
average: contain & 78.2 \\
\hdashline
SiftNinja & 33.2 \\
BERT fine-tuned on \citet{davidson_automated_2017} & 60.2 \\
BERT fine-tuned on \citet{founta_large_2018} & 63.2 \\
Google-Jigsaw & 76.6 \\
\hline
\end{tabular}
\caption{Full evaluation of hypotheses, that claim hate speech exists in the input text, on HateCheck.}
\label{tab:compare-hypotheses-full}
\end{table}
In contrast, NLI-based prediction in which a target task is converted to an NLI-task and fed into an NLI model converts the target task to the fine-tuning task. 
Here, a model is given a premise and a hypothesis and tasked to predict if the premise entails the hypothesis, contradicts it, or is neutral towards it.  
\citet{yin_benchmarking_2019} proposed to use an NLI model for zero-shot topic classification, by inputting the text to classify as the premise and constructing for each topic a hypothesis of the form \enquote{This text is about $<$topic$>$}. 
They map the labels \textit{neutral} and \textit{contradiction} to \textit{not-entailment}. We can then interpret a prediction of entailment as predicting that the input text belongs to the topic in the given hypothesis.
Conversely, \textit{not-entailment} implies that the text is not about the topic.
\citet{wang_entailment_2021} show for a range of tasks, including offensive language identification, that this task re-formulation also benefits few-shot learning scenarios. 
Recently, \citet{alkhamissi_token_2022} obtained large performance improvements in few-shot learning for hate speech detection by (1) decomposing the task into four subtasks and (2) additionally training the few-shot model on a knowledge base.

\paragraph{Experiment setup} To test if an input text contains hate speech, we need a hypothesis expressing that claim. 
However, there are many ways how the claim, that a given text contains hate speech, can be expressed. Choosing a sub-optimal way to express this claim will result in lower accuracy. \citet{wang_entailment_2021} already tested four different hypotheses for hate speech or offensive language. We conduct an extensive evaluation by constructing and testing all grammatically correct sentences built with the following building blocks: \textit{It/That/This + example/text + contains/is + hate speech/hateful/hateful content}. 
We conduct all experiments with a BART-large model \cite{lewis_bart_2020} that was fine-tuned on the \textbf{M}ulti-Genre \textbf{N}atural \textbf{L}anguage \textbf{I}nference dataset (MNLI) \cite{williams_broad-coverage_2018} and has been made available via the Huggingface transformers library \cite{wolf_transformers_2020} as \texttt{bart-large-mnli}. 
This model predicts either \textit{contradiction}, \textit{neutral}, or \textit{entailment}. 
We follow the recommendation of the model creators to ignore the logits for \textit{neutral} and perform a softmax over the logits of \textit{contradiction} and \textit{entailment}. If the probability for entailment is equal or higher than 0.5 we consider this a prediction of \textit{entailment} and thus \textit{hate speech}.\footnote{This procedure is equal to taking the argmax over \textit{contradiction} and \textit{entailment}.}
We evaluate on HateCheck since the functionalities in this dataset allow for an automatic in-depth error analysis and compare our results to the baselines provided by \citet{rottger_hatecheck_2021}.

\paragraph{Results} Table \ref{tab:compare-hypotheses} shows an abbreviated version of the results. The full results are given in Appendix \ref{appsec:zero-shot-results-compare-hypos}.
The hypothesis \enquote{That contains hate speech.} obtains the highest accuracy and beats the Google-Jigsaw API by 2.8pp. This is remarkable, since we can assume that the commercial systems were all trained to detect hateful content or hate speech, while this model has not been trained on a single example of hate speech detection or a similar task.
The two lowest scoring hypotheses lead to an accuracy of 66.6\% meaning that an unlucky choice of hypothesis can cost more than 12pp accuracy. 

In this section, we present four methods, which we call strategies, that aim to improve zero-shot hate speech detection. A strategy has the following components and structure: The aim is to assign a label $y=\{0, 1\}$ to input text $t$, where $1$ corresponds to the class \textit{hate speech} and $0$ corresponds to the class \textit{not-hate speech}.
The input text $t$ can be used in one or multiple a premises $p_0$ to $p_m$, that are used in conjunction with the main hypothesis $h_0$ and one or multiple supporting hypotheses $\left[h_1, ..., h_n\right]$ to obtain  NLI model predictions $m(p_i, h_j) \in \{0,1\}$ where 0 corresponds to contradiction and $1$ corresponds to entailment.
The variables $i$ and $j$ are defined as: $i \in \left[0, ..., m\right]$ and $j \in \left[0, ..., n\right]$. 
The rules for how to combine model predictions to obtain the final label $y$ are given by the individual strategies.
As the main hypothesis we use \enquote{That contains hate speech.}, since it lead to the highest accuracy on HateCheck in Section \ref{sec:evaluating-standard-0shot}.
The supporting hypotheses used to implement the strategies are listed in Table \ref{tab:list-of-hypotheses}.

Our zero-shot model wrongly classifies all examples of counterspeech that quote or reference hate speech as actual hate speech. 
References to hate speech without quotation marks are hard to identify.
Thus, for this work, we limit ourselves to counterspeech that quotes hate speech explicitly. 
We propose a three-stage strategy to this phenomenon: (1) quotation identification, (2) hate speech classification of the quoted content, (3) detecting the stance of the post towards the quoted content. 
Formally, the input text $t$ is divided into premise $p_0$ which contains the quoted text and premise $p_1$ which contains the text around the quotes. The quoted text is represented as ``$\left[X\right]$'' in $p_1$. 
Using the main hypothesis $h_0$ we predict if $p_0$ contains hate speech or not. We use the supporting hypothesis \enquote{This text supports $\left[X\right].$} ($h_1$) to predict the stance of $p_1$ towards $p_0$. 
If $p_0$ contains hate speech and $p_1$ has a supportive stance towards $p_0$, $t$ is classified as \textit{hate speech}, otherwise it is classified as \textit{not-hate speech}.
The strategy is depicted in Figure \ref{fig:counterspeech-filter}.

As shown in Table \ref{tab:full-error-analysis}, slurs that are reclaimed by members of a targeted group are often miss-classified as hate speech. Based on the observation that a reclaimed slur is often ascribed to oneself, we propose to use a supporting hypothesis that indicates if text is self-directed.\footnote{Of course there are counterexamples to this rule, where reclaimed slurs are directed to others and not oneself. However, as long this approximation, as crude as it may be, helps to reduce false positives, it is a useful approximation.}
If the model predicts self-directedness a potential prediction of \textit{hate speech} is overridden to \textit{not-hate speech}. 

Table \ref{tab:compare-hypotheses-full}, the extended version of Table \ref{tab:compare-hypotheses}, contains all results for comparing hypotheses for zero-shot hate speech detection on HateCheck. \enquote{average:$<$ \texttt{expression}$>$} refers to the average accuracy of all hypotheses containing \texttt{expression}. The highest accuracy is in bold. 

The results show that in many cases the detection of a mentioned group is surprisingly accurate. The difference in accuracy between the best performing hypothesis and the worst performing hypothesis does not exceed 12\%. This is a similar range to the differences found between hypotheses when testing if a text contains hate speech (see Table \ref{tab:compare-hypotheses} and Table \ref{tab:compare-hypotheses-full}). However, when looking at $F_1$ scores the differences are much larger, with more general terms, such as \textit{faith} or \textit{ethnicity} preforming worse than the specific terms \textit{Muslims} and \textit{black people}.
{"value": "74.7", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "Containing hate speech.", "model settings": {"xx": "yy"}} {"value": "78.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "Contains hate speech.", "model settings": {"xx": "yy"}} {"value": "72.9", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "Hate speech.", "model settings": {"xx": "yy"}} {"value": "71.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "Hateful.", "model settings": {"xx": "yy"}} {"value": "78.7", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "It contains hate speech.", "model settings": {"xx": "yy"}} {"value": "75.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "It is hateful.", "model settings": {"xx": "yy"}} {"value": "78.7", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "It contains hate speech.", "model settings": {"xx": "yy"}} {"value": "70.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "It is hate speech.", "model settings": {"xx": "yy"}} {"value": "75.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "It is hateful.", "model settings": {"xx": "yy"}} {"value": "79.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That contains hate speech.", "model settings": {"xx": "yy"}} {"value": "78.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That contains hateful content.", "model settings": {"xx": "yy"}} {"value": "77.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That example contains hateful content.", "model settings": {"xx": "yy"}} {"value": "66.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That example is hate speech.", "model settings": {"xx": "yy"}} {"value": "76.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That example is hateful.", "model settings": {"xx": "yy"}} {"value": "66.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That is hateful.", "model settings": {"xx": "yy"}} {"value": "78.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That text contains hate speech.", "model settings": {"xx": "yy"}} {"value": "78.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That text contains hateful content.", "model settings": {"xx": "yy"}} {"value": "69.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That text is hate speech.", "model settings": {"xx": "yy"}} {"value": "77.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "That text is hateful.", "model settings": {"xx": "yy"}} {"value": "79.1", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This contains hate speech.", "model settings": {"xx": "yy"}} {"value": "78.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This contains hateful content.", "model settings": {"xx": "yy"}} {"value": "77.3", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This example contains hate speech.", "model settings": {"xx": "yy"}} {"value": "77.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This example contains hateful content.", "model settings": {"xx": "yy"}} {"value": "67.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This example is hate speech.", "model settings": {"xx": "yy"}} {"value": "77.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This example is hateful.", "model settings": {"xx": "yy"}} {"value": "70.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This is hateful.", "model settings": {"xx": "yy"}} {"value": "78.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This text contains hate speech.", "model settings": {"xx": "yy"}} {"value": "78.3", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This text contains hateful content.", "model settings": {"xx": "yy"}} {"value": "69.5", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This text is hate speech.", "model settings": {"xx": "yy"}} {"value": "78.7", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "This text is hateful.", "model settings": {"xx": "yy"}} {"value": "74.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: It", "model settings": {"xx": "yy"}} {"value": "75.7", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: This", "model settings": {"xx": "yy"}} {"value": "74.9", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: That", "model settings": {"xx": "yy"}} {"value": "75.8", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: hateful", "model settings": {"xx": "yy"}} {"value": "78.1", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: hateful content", "model settings": {"xx": "yy"}} {"value": "74.5", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: hate speech", "model settings": {"xx": "yy"}} {"value": "74.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: example", "model settings": {"xx": "yy"}} {"value": "76.1", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: text", "model settings": {"xx": "yy"}} {"value": "73.9", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: is", "model settings": {"xx": "yy"}} {"value": "78.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "average: contain", "model settings": {"xx": "yy"}} {"value": "33.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "SiftNinja", "model settings": {"xx": "yy"}} {"value": "60.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT fine-tuned on \\citet{davidson_automated_2017}", "model settings": {"xx": "yy"}} {"value": "63.2", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "BERT fine-tuned on \\citet{founta_large_2018}", "model settings": {"xx": "yy"}} {"value": "76.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "Google-Jigsaw", "model settings": {"xx": "yy"}}