\begin{table}[ht]
\centering
\small
\begin{tabular}{lrr}
\hline
strategy & F20 & overall \\ 
\hdashline
No strategy & 0.0 & 79.4 \\
FCS & +100 & +4.6 \\
\hdashline
FCS$_{p_1}$ & +0.0 & +0.0 \\
FCS$_{p_1 FBT}$ & +6.9 & +0.3 \\
\hline
\end{tabular}
\caption{Evaluation of FCS variants. The two bottom rows display the variants adjusted for detecting Hate Speech in $p_1$. The functionality F20 contains \textit{Denouncements of hate that quote it}. The scores are given in accuracy (\%) and change in accuracy compared to \textit{No strategy}.}
\label{tab:FCS-follow-up}
\end{table}
In contrast, NLI-based prediction in which a target task is converted to an NLI-task and fed into an NLI model converts the target task to the fine-tuning task. 
Here, a model is given a premise and a hypothesis and tasked to predict if the premise entails the hypothesis, contradicts it, or is neutral towards it.  
\citet{yin_benchmarking_2019} proposed to use an NLI model for zero-shot topic classification, by inputting the text to classify as the premise and constructing for each topic a hypothesis of the form \enquote{This text is about $<$topic$>$}. 
They map the labels \textit{neutral} and \textit{contradiction} to \textit{not-entailment}. We can then interpret a prediction of entailment as predicting that the input text belongs to the topic in the given hypothesis.
Conversely, \textit{not-entailment} implies that the text is not about the topic.
\citet{wang_entailment_2021} show for a range of tasks, including offensive language identification, that this task re-formulation also benefits few-shot learning scenarios. 
Recently, \citet{alkhamissi_token_2022} obtained large performance improvements in few-shot learning for hate speech detection by (1) decomposing the task into four subtasks and (2) additionally training the few-shot model on a knowledge base.

\paragraph{Error Analysis} Column \enquote{No Strat.} in Table \ref{tab:full-error-analysis} shows the accuracy per HateCheck functionality for the hypothesis \enquote{That contains hate speech.}. Most notably, the model wrongly predicted all denouncements of hate (F20 and F21) as hate speech. In four functionalities (F22, F11, F23, F20) the model predicted hate speech even though no one or no relevant group was targeted. Finally, we see that the model often fails at analyzing sentences with negations (F15) and that it fails at recognizing when slurs are reclaimed and used in a positive way (F9). 
In what follows, we will present and evaluate strategies to avoid these errors.

In this section, we present four methods, which we call strategies, that aim to improve zero-shot hate speech detection. A strategy has the following components and structure: The aim is to assign a label $y=\{0, 1\}$ to input text $t$, where $1$ corresponds to the class \textit{hate speech} and $0$ corresponds to the class \textit{not-hate speech}.
The input text $t$ can be used in one or multiple a premises $p_0$ to $p_m$, that are used in conjunction with the main hypothesis $h_0$ and one or multiple supporting hypotheses $\left[h_1, ..., h_n\right]$ to obtain  NLI model predictions $m(p_i, h_j) \in \{0,1\}$ where 0 corresponds to contradiction and $1$ corresponds to entailment.
The variables $i$ and $j$ are defined as: $i \in \left[0, ..., m\right]$ and $j \in \left[0, ..., n\right]$. 
The rules for how to combine model predictions to obtain the final label $y$ are given by the individual strategies.
As the main hypothesis we use \enquote{That contains hate speech.}, since it lead to the highest accuracy on HateCheck in Section \ref{sec:evaluating-standard-0shot}.
The supporting hypotheses used to implement the strategies are listed in Table \ref{tab:list-of-hypotheses}.

\subsection{Filtering By Target (FBT)}
\label{subsec:filtering-by-target}
The error analysis showed that we can improve zero-shot classification accuracy significantly by avoiding predictions of hate speech where no relevant target group occurs. We thus propose to avoid false positives by constructing a set of supporting hypotheses $\left[h_1, ..., h_n\right]$ to predict if text $t$ actually targets or mentions a protected group or characteristic. If no protected group or characteristic is predicted to occur in $t$, a potential prediction of \textit{hate speech} is overridden to \textit{not-hate speech}.
Figure \ref{fig:filtering-by-target} illustrates the method. 

Our zero-shot model wrongly classifies all examples of counterspeech that quote or reference hate speech as actual hate speech. 
References to hate speech without quotation marks are hard to identify.
Thus, for this work, we limit ourselves to counterspeech that quotes hate speech explicitly. 
We propose a three-stage strategy to this phenomenon: (1) quotation identification, (2) hate speech classification of the quoted content, (3) detecting the stance of the post towards the quoted content. 
Formally, the input text $t$ is divided into premise $p_0$ which contains the quoted text and premise $p_1$ which contains the text around the quotes. The quoted text is represented as ``$\left[X\right]$'' in $p_1$. 
Using the main hypothesis $h_0$ we predict if $p_0$ contains hate speech or not. We use the supporting hypothesis \enquote{This text supports $\left[X\right].$} ($h_1$) to predict the stance of $p_1$ towards $p_0$. 
If $p_0$ contains hate speech and $p_1$ has a supportive stance towards $p_0$, $t$ is classified as \textit{hate speech}, otherwise it is classified as \textit{not-hate speech}.
The strategy is depicted in Figure \ref{fig:counterspeech-filter}.

While our experiments did not show problems in generalization, we can imagine the following weakness for the FCS strategy: 
Given an input text $t$ that contains a quote and hate speech, where the hate speech does not occur inside of the quotes, the current FCS strategy would fail, since it only detects hate speech in $p_0$, that is inside the quotes. Such an example is given in Figure \ref{fig:counterspeech-filter-fbt}.

The results, displayed in Table \ref{tab:FCS-follow-up} show that this modification removes all the gains obtained through FBT. 
We assume that this is due to the fact that the counterspeech often also conveys strong negative emotions that are mistaken by the model for hate speech.

We further test if this problem can be alleviated by applying the FBT strategy if hate speech is detected in $p_1$ (i.e. outside of the quotes) as depicted in Figure \ref{fig:counterspeech-filter-fbt}. 
The results in Table \ref{tab:FCS-follow-up} (row FCS$_{p_1FBT}$) show that additionally applying FBT only recovers a fraction of the positive effect of FCS. 
We assume that this is due to counterspeech including or being associated with target groups. 
Thus, further research that investigates how the problem can be alleviated is needed.
{"value": "0.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck F20", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "No strategy", "model settings": {"xx": "yy"}} {"value": "79.4", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": "accuracy", "experimental settings": {"xx": "yy"}, "model": "No strategy", "model settings": {"xx": "yy"}} {"value": "+100", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck F20", "task": "zero-shot hate speech detection", "metric": ["accuracy boost", "accuracy increase"], "experimental settings": {"xx": "yy"}, "model": "FCS", "model settings": {"xx": "yy"}} {"value": "+4.6", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": ["accuracy boost", "accuracy increase"], "experimental settings": {"xx": "yy"}, "model": "FCS", "model settings": {"xx": "yy"}} {"value": "+0.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck F20", "task": "zero-shot hate speech detection", "metric": ["accuracy boost", "accuracy increase"], "experimental settings": {"xx": "yy"}, "model": "FCS$_{p_1}$", "model settings": {"xx": "yy"}} {"value": "+0.0", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": ["accuracy boost", "accuracy increase"], "experimental settings": {"xx": "yy"}, "model": "FCS$_{p_1}$", "model settings": {"xx": "yy"}} {"value": "+6.9", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck F20", "task": "zero-shot hate speech detection", "metric": ["accuracy boost", "accuracy increase"], "experimental settings": {"xx": "yy"}, "model": "FCS$_{p_1 FBT}$", "model settings": {"xx": "yy"}} {"value": "+0.3", "type": "Result", "training data/set": "HateCheck", "test data/set": "HateCheck", "task": "zero-shot hate speech detection", "metric": ["accuracy boost", "accuracy increase"], "experimental settings": {"xx": "yy"}, "model": "FCS$_{p_1 FBT}$", "model settings": {"xx": "yy"}}