# Cheap Learning: Maximising Performance of Language Models for Social Data
Science Using Minimal Data

Leonardo Castro-González  Public Policy Programme, The Alan Turing Institute,
British Library, 96 Euston Rd, London, NW1 2DB UK.  These authors contributed
equally  Yi-Ling Chung  Public Policy Programme, The Alan Turing Institute,
British Library, 96 Euston Rd, London, NW1 2DB UK.  These authors contributed
equally  Hannah R. Kirk  Public Policy Programme, The Alan Turing Institute,
British Library, 96 Euston Rd, London, NW1 2DB UK.  Oxford Internet Institute,
University of Oxford, 1 St Giles, Oxford, OX1 3JS, UK  John Francis  Public
Policy Programme, The Alan Turing Institute, British Library, 96 Euston Rd,
London, NW1 2DB UK.  Angus R. Williams  Public Policy Programme, The Alan
Turing Institute, British Library, 96 Euston Rd, London, NW1 2DB UK.  Pica
Johansson  Public Policy Programme, The Alan Turing Institute, British
Library, 96 Euston Rd, London, NW1 2DB UK.  Jonathan Bright  Public Policy
Programme, The Alan Turing Institute, British Library, 96 Euston Rd, London,
NW1 2DB UK.  corresponding author<EMAIL_ADDRESS>
###### Abstract

The field of machine learning has recently made significant progress in
reducing the requirements for labelled training data when building new models.
These ‘cheaper’ learning techniques hold significant potential for the social
sciences, where development of large labelled training datasets is often a
significant practical impediment to the use of machine learning for analytical
tasks. In this article we review three ‘cheap’ techniques that have developed
in recent years: weak supervision, transfer learning and prompt engineering.
For the latter, we also review the particular case of zero-shot prompting of
large language models. For each technique we provide a guide of how it works
and demonstrate its application across six different realistic social science
applications (two different tasks paired with three different dataset
makeups). We show good performance for all techniques, and in particular we
demonstrate how prompting of large language models can achieve high accuracy
at very low cost. Our results are accompanied by a code repository to make it
easy for others to duplicate our work and use it in their own research.
Overall, our article is intended to stimulate further uptake of these
techniques in the social sciences.

Keywords: Social Data Science, Text Classification, Week Supervision, Transfer
Learning, Prompt Engineering, Language Models

##  1 Introduction

The systematic analysis of large-scale text data is a growing part of social
science research. The rapid digitisation of social life has meant that daily
activity has increasingly left digital ‘traces’ (Boyd and Crawford, 2011).
Across the social sciences, many of the core objects of disciplinary study can
be interrogated through analysis of large datasets created by these traces.
Examples abound: large-scale corpora of news articles; datasets of political
manifestoes and speeches; textual interactions between individuals around
romantic relationships, job searches, discussions around health and climate
change, to name but a few (Scharkow and Vogelgesang, 2011; Burscher et al.,
2015; Beigman Klebanov and Madnani, 2020; Gangula et al., 2019; Souma et al.,
2019; Stede and Patz, 2021).

In the past, large-scale text corpora were addressed through equally large-
scale manual coding efforts (see, for example, Lehmann et al. (2023)), whereby
texts are reviewed by teams of trained coders. However, around a decade ago,
work emerged pointing to the potential for automatic content analysis to shed
light on the social dynamics contained within these datasets, making use of
insights from the field of machine learning (Wiedemann, 2013; Grimmer and
Stewart, 2013; Grimmer et al., 2021). By now a considerable body of work has
developed methods for automating content analysis in the social sciences
(Nelson and Taneja, 2018; Németh et al., 2020), which offers obvious potential
benefits in terms of saving time and effort, and thus allowing for analyses at
a scale that would be beyond most teams of manual coders (Burscher et al.,
2014).

However, while the potential opened up by this automatic work is important,
there remains a weakness in the automatic content analysis based on the
‘conventional’ paradigm of machine learning (as we will refer to it here): it
still in reality contains a considerable manual component. As we will describe
more fully below, in order for a classifier to be trained to identify a given
concept, labelled training data is required, often in large quantities. Hence
a significant time investment is required before such a classifier can be
developed. Whilst such upfront work can be justified in industry contexts that
may require millions of data points to be labelled automatically on an ongoing
basis, many analytical research tasks in the social sciences could actually be
achieved with this initially labelled dataset, rendering the training of the
model somewhat irrelevant. Hence, the practical advantages of this type of
machine learning approach in the social sciences is not always clear cut, with
some saying that the time and financial savings from automated methods are
exaggerated (De Grove et al., 2020; Németh et al., 2020) and others preferring
methods granting researchers greater measurement validity (Baden et al.,
2022).

The heavy data requirements of conventional machine learning approaches have
led to considerable research interest in techniques that might reduce data
requirements without sacrificing performance or analytical rigour (Al-Jarrah
et al., 2015; Goldsteen et al., 2022). In computer science, such techniques
are now developing very actively, with considerable advances and a wide array
of new techniques developing. However, only a few papers have emerged trying
to make these techniques accessible to social science researchers (Molina and
Garip, 2019; Macanovic, 2022; Wankmüller, 2022), and the uptake remains patchy
at best.

The aim of this article is to remedy this deficit and offer an introduction to
these ‘cheap’ learning techniques, as compared to their more expensive
conventional counterparts. We discuss three main approaches. First, we look at
weak supervision, a method for formalising background knowledge of a subject
into a systematic approach to data labelling, making use largely of keyword-
based approaches. While it might not be the technique that offers the highest
performance, it is transparent and easy to implement, retaining the simplicity
of the dictionary technique but assuaging some of the criticisms of how
dictionaries are both arbitrary and difficult to validate (Grimmer and
Stewart, 2013). Second, we discuss transfer learning, a technique which
involves taking a model trained in one context and applying it to another,
making use of a minimal amount of retraining data. We show how good results
can be achieved with small amounts of novel data by building on pre-existing
models. Finally, we discuss ‘prompt engineering’: prompting large language
models (LLMs), such as the GPT family created by OpenAI, as a means of content
classification. We look at both direct use of generative models for such
classification (sometimes called ‘zero-shot’ learning), and also how these
techniques can be combined with some elements of transfer learning. LLMs in
particular have generated much excitement as a very cheap method of content
classification (Gilardi et al., 2023); however, we show that at the moment
they do not outperform more traditional approaches for all tasks.

Each one of these techniques offers the potential for realising the benefits
of supervised machine learning, but at less cost in terms of human labelling,
hence offering the tantalising potential of finally realising the potential of
automatic content analysis in the social sciences. For each one of these
techniques, we describe how it functions, discuss approaches to validation,
and discuss common potential use cases. In an empirical section, we then
illustrate using existing annotated datasets how good performance can be
achieved with a minimal amount of labelled data (up to 1024 data points), as
well as showing how performance improvements relate to increases in labelled
data. We also compare all of these techniques to the Naïve Bayes approach, a
classic method from conventional machine learning Kibriya et al. (2005).
Although the method was developed more than two decades ago, its simplicity of
training and deployment makes it a good baseline to compare with newer and
more sophisticated methods in a low-resource context.

We should note the existence of a handful of other papers that have made
similar contributions to our own (Terechshenko et al., 2020; Treviso et al.,
2023). Whilst these are valuable works, our paper is distinctive in a number
of ways. First, these papers address one technique exclusively, whereas ours
compares three at the same time. This comparison allows us to explore the
‘labelling budget’ (the amount of manually labelled data required for each
technique), as well as discussing how different methods perform well in
different circumstances. Second, alongside our paper we release code that will
allow other researchers with a practical working knowledge of the Python
language to implement these techniques in their own
research.111https://github.com/Turing-Online-Safety-Codebase/cheap_learning

Our paper is structured in the following way. In section 2 we give a general
overview of how we define cheap learning techniques, and also justify our
focus on the techniques we are looking at. Then, we provide a guide to each
technique, describing how they function, discussing approaches to validation,
and discussing common potential use cases. In section 3 we describe our
approach to the empirical illustration of the models and describe the data we
make use of. Finally, in section 4, we illustrate the use of these techniques
using existing annotated datasets to showcase how good performance can be
achieved with a minimal amount of labelled data (as well as showing how
improvements in performance relate to increases in labelled data). We also
discuss concerns around balanced and unbalanced data, showing how these might
affect preferences for models. We conclude by providing perspective on
directions in the field, especially in the light of increasingly powerful and
available large language models.

##  2 Cheap Learning Techniques

In this paper, we address methods for automating content analysis in the
social sciences using techniques from the field of machine learning. Our focus
is on algorithms that can be implemented ‘cheaply’ when compared to what we
will define as the ‘conventional’ approach to automatic content analysis,
which involves training a model from scratch with a large labelled training
dataset (though we refer to it as ‘conventional’, we should also say that it
is an approach that is less and less the norm within computer science). As
highlighted in the introduction, we address three main approaches: weak
supervision, transfer learning and prompt engineering. We regard these
algorithms as ‘cheap’ to implement because they typically require much less
labelled data to achieve good performance than the conventional approach
(though all of them still require some such data, particularly for
validation). In this section, we will provide an overview of the three
techniques we are introducing in the paper. For each technique, we provide a
general overview of the history and motivation, and provide a guide on how to
implement it, including notes on how to approach evaluation of the technique.
We also reflect on the cost of each algorithm, and the amount of computing
power required to implement it (with the proviso that, of course, computing
power increases at a regular pace). Finally, we provide our thoughts on how
each technique might adapt to different problem sets in the social sciences.

We should note that we do not address ‘unsupervised’ machine learning, a
technique which involves no manual labelling (at least in the input stage). We
omit this technique here partly because it has already been covered by a
relatively extensive set of social science applications (e.g. Berry et al.
(2019); Molina and Garip (2019); Waggoner (2021)), and also because we do not
consider it a direct competitor to supervised machine learning. While
unsupervised learning offers the potential to group text documents into
lexically similar sets (often referred to as ‘topic modelling’) it does not
provide the direct link to theoretical categories of interest which is offered
by supervised machine learning: that is to say there is no guarantee that
identified topics will map neatly to theoretical concepts and hence be
directly applicable to an analytical social science question. Therefore, while
the technique is an excellent approach for exploratory analysis of a large
dataset, the lack of ability to define concepts at the outset can make it
challenging to use topic models for explanatory hypothesis testing (though we
do note the emergence of seeded topic models (Watanabe and Baturo, 2023) as a
potential partial remedy to this problem), and of course post-hoc labelling of
topics can allow them to be built into an explanatory framework.

We contrast all of these approaches to the conventional approach to text
classification through machine learning, which is worth briefly reviewing here
(the subject has already been treated extensively in, for example, Kotsiantis
et al. (2006), Jordan and Mitchell (2015), and Mahesh (2020)). This approach
seeks to produce a model that, given an input text, estimates the probability
that this text belongs to a given analytical category. Such models are based
on input data: a corpus of text that has been labelled by hand such that each
piece of text is assigned to a category. One classic example, that we make
further use of below, would be to label a given piece of text as containing
‘hate speech’ or not, to produce a text-based classifier that can
automatically detect hateful text.

The corpus of text data is typically divided into three groups: training data,
development data, and test data. Training data are the data used to train the
model to begin with: the model’s decision making will be based on patterns
learnt from these data. There are a wide variety of approaches to learning a
model from training data that we will not seek to review here (good overviews
are provided by Kotsiantis et al. (2006); Cherkassky and Mulier (2007); Jordan
and Mitchell (2015); Mahesh (2020); N and Gupta (2020)). Broadly speaking,
they will rely on identifying differences in linguistic patterns between the
different classes of data for which a classification is sought (for example,
hateful text is likely to contain different words to non-hateful text).
Development data are data used during the process of model training to provide
an estimate of the model’s performance (with a variety of metrics typically
calculated for this evaluation - see Fatourechi et al. (2008); Hossin and
Sulaiman (2015)). These development data allow for decisions to be made during
the process of model training, for example selecting appropriate
hyperparameters (Luo, 2016; Keskar et al., 2017; Kandel and Castelli, 2020).
Test data, finally, are used to make an assessment of model performance once
the process of training is complete. Test data are typically kept separate
until the final model evaluation to guard against ‘overfitting’, a phenomenon
whereby (through repeated trial and error with different models and different
hyperparameters) a model is developed that performs well on the development
dataset but generalises poorly to unseen data.

When implementing such a process within the social sciences, the amount of
data required (the ‘labelling budget’, Lindstrom et al. (2013); Guan and
Koudas (2022)) is the core practical concern. High-quality labelled data is
expensive and time-consuming to generate. Firstly, coders need to be recruited
and trained on the concepts of interest, and need to establish a common
understanding of the concepts they are coding (often measured through high
inter-coder reliability scores). Once this process is complete, labelling
needs to take place, which is a time-consuming and repetitive manual task (and
gold standard scientific journals often require data to have been coded by
multiple coders, Hube et al. (2019); Geva et al. (2019)). The exact amount of
time will of course be task-dependent (varying by, for example, the amount of
text that needs to be reviewed, the amount of potential classes, and the
complexity of the coding task). However, there is a general expectation that
the process will be time-consuming, driven by the fact that the data
requirements of high-performance classification models are typically large
(Sun et al., 2019; Zheng and Jin, 2020; Anaby-Tavor et al., 2020), and that
increasing the size of the training dataset is the standard recommendation for
improving performance (Shams, 2014; Raffel et al., 2020; Hsieh et al., 2023).

The approaches we review below are all motivated by a desire to reduce the
cost of the labelling budget to produce a high-performing model, particularly
the training and development datasets (a high-quality test dataset for
evaluation remains essential to the process however). Each of the techniques
approach this task in a different way: weak supervision does not necessarily
reduce the size of the labelling budget, but makes large training datasets
cheaper to put together; transfer learning seeks to leverage other pre-
existing models to reduce the requirements for novel labelled data; while
prompt engineering seeks to exploit ‘emergent’ properties of large language
models that have been trained for generic language production tasks but can
also be leveraged for classification.

As a core element of our paper is cost, we should also mention some other
costs that are relevant here beyond those needed for the creation of labelled
data. There are hardware costs implied in some of our work below: though much
of what we describe can be implemented on a generic personal laptop, some may
require the use of an institutional or cloud server. For commercial language
models, there are also costs in terms of API access that could mount up if
there are requirements to label large volumes of data. Finally, there are some
time costs required for training models, that can mount up if a long cycle of
experimentation is required to find the ideal configuration of
hyperparameters. We will also comment on these costs for all of the techniques
we discuss.

###  2.1 Weak Supervision

Proposed in Ratner et al. (2016) and extended in Ratner et al. (2019), weak
supervision is an approach to text classification that seeks to cheaply build
labelled data using heuristic rules based on pre-existing background knowledge
of the concepts being labelled. Data labelled in this fashion can either be
applied directly to an analytical task, or leveraged to build a further
classification model.

As a simple example of weak supervision, we could imagine a task based on
classifying international news articles based on which country they relate to.
A first step to build a model through the conventional machine learning
approach would be to take a dataset of news articles and label by hand the
country to which they relate (conscious that an article might relate to more
than one country). However, this approach essentially ignores background
knowledge that will be largely obvious to people with domain expertise: we
know that (for example) an article containing the keyword ‘Canberra’ is most
likely about Australia, whilst one containing the keyword ‘Boston’ likely
refers to the US. Integrating this knowledge would seem like a valuable
efficiency.

Using keywords to classify content in this fashion, sometimes called the
‘dictionary’ approach, does of course have a long history in the social
sciences and indeed many early approaches to content classification were based
on keywords (Hulth and Megyesi, 2006; Onan et al., 2016). However, such
approaches received strong criticism as being hard to evaluate properly
(Grimmer and Stewart, 2013). In the absence of manually labelled data, it was
hard to measure common evaluation metrics such as the precision and
recall222Precision refers to the number of true positives divided by the total
amount of the positive elements retrieved. In other words, precision measures
the reliability of the results. On the other hand, recall refers to the ratio
of the true positives divided by the total amount of relevant items (true
positives + false negatives)., or understand how many false positives and
negatives might be generated (for example, an article on Boston, Lincolnshire
in the UK would be obviously misclassified by the above approach).

Weak supervision can be viewed as a method that seeks to circumvent some of
these difficulties: benefiting from the simple but powerful domain knowledge
that experts can bring to bear on classification, without sacrificing the
rigor or evaluation metrics used in more conventional machine-learning
approaches. It has already been applied in a wide variety of different
contexts like text data in the fields of medicine and health sciences (Fries
et al., 2021; Silva et al., 2022), chemistry (Mallory et al., 2020), detection
of fake news (Shu et al., 2021), and more generally to study sentiment
analysis (Jain, 2021), and ontology and computational linguistics problems
(Maresca et al., 2021; Berger and Goldstein, 2021).

Weak supervision is based on two main components: labelling functions (LFs)
that encode background knowledge into formalised rules; and a probabilistic
model that is trained with the noisy, overlapping labels obtained from
applying the functions to the labelled data set, and its respective prior
labels.

Labelling functions are abstract rules that take an unlabelled data point and
either assign a label if a criterion is met or abstain from assigning it
otherwise. The presence or absence of keywords contained in text data is one
of the most obvious types of such rules, however, there is no need to only
make use of keywords. Regular expressions, Named-Entity Recognition (NER)
systems, results from other models such as sentiment classifiers, the length
of the document to be classified, or its source; any of these document
features can be incorporated into a labelling function. This diversity of
potential criteria makes it possible to have topic-specific sets of labelling
functions obtained from the particularities of the topic to be classified,
going a step beyond the dictionary approach. Some examples of simple labelling
functions that we make use of in the experimental section of the paper are
presented below. For further examples, the list of all the labelling functions
used for the classification tasks presented in the paper can be found in
Appendix A.3.

>   * •
>
> LF1: return NEGATIVE if polarity(text) < -0.25 else ABSTAIN
>
>   * •
>
> LF2: return POSITIVE if regex found in text else ABSTAIN
>
>   * •
>
> LF3: return ATTACK if length(text) > LIMIT else ABSTAIN
>
>

In the examples above, variables NEGATIVE and POSITIVE refer to the classes
used for the binary movie sentiment classification exercise, while the ATTACK
refers to one of the classes used for the Wiki Personal Attacks exercise
explained below. In the same way, the variable ABSTAIN refers to the value
that the function returns when it abstains from classifying a text. The
polarity function refers to the sentiment analysis function to measure the
polarity of a given text between -1 and 1. regex is used in this example as
any regular expression that would refer to a non-abusive text, e.g. “This is
such a good idea!”. LF3 is a good example of how domain knowledge can be used
in weak labelling. As seen in the Wiki Personal Attacks exercise (Wulczyn et
al., 2017), cyberbullying can take many forms (Sathyanarayana Rao et al.,
2018). One of them is to repeat a sentence or a set of words in a comment
until the maximum number of characters allowed is hit. LF3 accounts for that,
labelling unusually long texts as personal attacks by measuring the length of
the text and checking if it is greater than the maximum length allowed
(LIMIT).

The exploration subset of the dataset serves to fine-tune or debug the set of
labelling functions. Weak supervision enables measuring coverage and overlap
of these functions in the different sets (details of the LFs used in this work
are presented in Appendix A). Coverage for a specific labelling function
indicates the percentage of data points it labels without abstaining. On the
other hand, overlap represents the percentage of data points where two or more
labelling functions do not abstain. Ideally, for a given class, we aim for a
set of labelling functions that collectively cover the entire exploration set
without significant redundancy in their actions, measured by a high overlap.

Once the labelling functions are created, they are then applied to the
training set, generating a label matrix. Each row in the matrix represents a
data point, and each column represents a labelling function. Since the
training data is already classified, this label matrix is used to compute
weights for each labelling function using a Bayesian probabilistic model,
called a LabelModel by the authors (Ratner et al., 2019). The overall model,
incorporating the labelling functions and their respective weights, is then
used on the test set for classification.

The outcome of a weak supervision process will be a partially labelled dataset
(as we expect the labelling functions to abstain in some cases rather than
suggesting a label). Depending on the coverage achieved, the partially
labelled set may already be enough to answer the question at hand. If it is
not, a common next step would be to train a conventional machine-learning
model to label the remaining data points. The success of the approach will
depend on whether the partially labelled data is good enough to produce a
strong classification model, though as we will present below the approach
shows good potential for a number of tasks.

In addition to its very transparent and simple approach to producing labelled
data, a further advantage of weak supervision lies in its lack of significant
computing costs. Developing and applying labelling functions are
computationally cheap tasks that can be tackled on any standard consumer
laptop and labelling even a very large dataset in this fashion would take
little time or processing power.

###  2.2 Transfer Learning

The approach to conventional machine learning described above starts from a
blank page. Given a classification problem, and a large enough pre-existing
labelled dataset, a model is trained from scratch, experimenting with
different hyperparameters, trying to achieve the best possible accuracy on a
development dataset. The final evaluation score is then calculated on
previously unseen test data.

Transfer learning starts from a different premise (Pan and Yang, 2010; Weiss
et al., 2016; Zhuang et al., 2020). Rather than starting from scratch, it
looks for pre-existing models (potentially from closely related source
domains), and seeks to leverage the information already contained within them
(Weiss et al., 2016; Neyshabur et al., 2020; Bashath et al., 2022). A model
built on this approach will still require some novel training and development
data; but far less than in the conventional approach. Furthermore, given the
variety of input data, transfer learning also offers a method for obtaining
models with better generalisation properties. These advantages mean that
transfer learning is a rapidly growing field of work (Conneau and Lample,
2019; Chung et al., 2020). Transfer learning has shown great success in
several real-world applications, such as text classification (Raffel et al.,
2020; Ali et al., 2022), image recognition (Guo et al., 2019; Kolesnikov et
al., 2020), and natural language generation (Raffel et al., 2020; Golovanov et
al., 2019).

There are a variety of ways in which a pre-trained model might be selected.
Domain specificity is obviously a concern: if seeking to build a
classification model for hate speech, selecting a pre-existing hate speech
classifier may be a sensible place to start. Beyond this, language ability
(e.g., monolingual or multilingual), model architecture (e.g., model size,
number of parameters and speed), computation power (e.g., hardware and number
of GPUs required) and model performance (e.g., accuracy) all might be a
consideration. With recent advancements in natural language processing,
transformer-based large language models that have been developed and pre-
trained on massive corpora (e.g., Common Crawl (Raffel et al., 2020)) using
various pre-training objectives (Devlin et al., 2019; Dong et al., 2019) are
becoming more and more popular as the starting point of transfer models
focused on text classification. These general-purpose models lack any domain
specificity, but they have nevertheless demonstrated strong performances and
can leverage knowledge from source data to downstream tasks through fine-
tuning methods (Houlsby et al., 2019; Peters et al., 2019). As a general rule
larger models mean better performance whilst requiring high computation power
and memory consumption. A key question for many in the social sciences will be
whether the model being selected can be run on a researcher’s laptop or
whether a cloud computing platform that can offer greater performance (such as
AWS, Azure or GCP) will be required (Pierleoni et al., 2019; Kamal et al.,
2020). However the technology here is evolving constantly, and the possibility
of running ever larger models on contemporary consumer hardware is always
increasing (Pierleoni et al., 2019).

Once a base model has been selected, it is then fine-tuned on the training
set. The ideal size for a training set is not easy to know in advance, but in
the experiments section below we will present a few rule of thumb examples for
the ideal size. This fine-tuning has the aim of improving model performance
for the specific task at hand. There are a variety of different ways to
approach this task, tuned through hyperparameters, and many of these have
conventional values that have been found to work reasonably well in a variety
of settings. Training epochs indicate the number of times a model sees or
passes over the whole training dataset, often in the range between 1 and 5.
Learning rate refers to how much to adjust the model (i.e. model weights) with
respect to the knowledge learned each time the model weights are updated
(Bengio, 2012), with 0.0001 and 0.0003 sometimes suggested as working well for
many problems.333See for instance:
https://huggingface.co/docs/transformers/model_doc/t5 A learning rate that is
very large could make rapid divergence of model weights, likely resulting in
unstable training and a sub-optimal solution, whereas a learning rate that is
too small may require longer training time for models to converge or arrive
only at a local minimum. Batch size decides the number of instances processed
before the model weights are updated, which is generally a power of 2 in the
range between 8 and 128. While increasing batch size generally requires less
training time, it consumes more memory and may not lead to high accuracy
(Keskar et al., 2017; Kandel and Castelli, 2020). A random seed will also
indicate the initial state of the model parameters.

Finally, the finished model can be evaluated to assess if a model is learning
as expected (e.g., whether the model is overfitting or not). This includes
applying the trained model to the test set using various metrics, including
precision, recall, F1, and accuracy. Graphical representation is another
helpful way of diagnosing model performance, for instance, through learning
curves of model performance over the training process, confusion matrix and
receiver operating characteristic (ROC) curve. These methods are the same as
those used in the conventional approach described above, hence we will not
describe them in detail here.

###  2.3 Prompt Engineering

The final technique we will introduce is prompt engineering, also known as in-
context prompting or prompt learning. This is a technique for guiding large
language models to perform a task (e.g., classification) via textual prompts
(Liu et al., 2023). Training a prompt model is similar to transfer learning in
a way, in that we are exploiting pre-existing knowledge within a model.
However, unlike the above approach, we no longer seek to train a model in a
classic sense, but rather exploit prompts as guidance to elicit the generative
potential already held within large language models to generate answers to
classification questions.

In theory, a nuanced prompt tailored to a task can produce high performance in
a classification task even more cheaply than transfer learning. With
handcrafted or automatically generated prompts, prompt engineering is
particularly beneficial for scenarios/domains where (1) language models show
strong performance and (2) labelled examples are scarce or not available
(Schick and Schütze, 2021a; Liu et al., 2021; Hu et al., 2022). Such prompting
has been explored and has shown promising success in several domains, such as
text classification (Raffel et al., 2020; Ali et al., 2022), image recognition
(Guo et al., 2019; Kolesnikov et al., 2020), and natural language generation
(Raffel et al., 2020; Golovanov et al., 2019).

Unlike transfer learning, in prompt engineering a language model is instructed
to perform with the help of task description as an auxiliary training signal
using ‘pattern-verbalizer pairs’ (Schick and Schütze, 2021a, b). A pattern is
a natural language prompt or an expression that transforms inputs into cloze-
style tasks (i.e. a portion of a passage is removed or closed, and the task is
to predict the removed chunk that would best fit the surrounding context). A
verbalizer refers to mapping a label/class (e.g., movie) to a list of task-
representative words, also called label words, (e.g., action, drama and
adventure). For example, given a binary movie sentiment classification task
(as we explore below), an example input in italics, taken from IMDb Movie
Review Sentiment dataset (Maas et al., 2011), with a pattern in bold is shown
as follows:

> Two Hands restored my faith in Aussie films. It took an old premise and made
> it fresh. I enjoyed this movie to no end. I recommend it to those people who
> like Guy Ritchie films. Bryan Brown was fantastic and just about perfect in
> a role tailor made for him. It was? Negative or Not Negative [MASKED]

The answer is masked and converted into a class based on a verbalizer, for
instance, ‘Positive’: ‘good’, ‘great’; ‘Negative’: ‘bad’.

Writing a good prompt is crucial for effectively giving instructions and
obtaining accurate responses from language models. While interest in prompt
design has grown considerably in various domains, such as text classification
(Wang et al., 2019; Hu et al., 2022), and knowledge probing (Petroni et al.,
2019; Davison et al., 2019), studies on systematic evaluation of the effects
of prompts are limited. We provide several heuristics that can help write a
good prompt. First, a prompt is generally clear and specific with relevant
background information provided. A prompt can be written in the form of a
statement or question. Second, when dealing with complicated tasks or multiple
sub-questions, breaking down the tasks into sub-tasks can help models tackle
each task one by one (Wei et al., 2022). In Press et al. (2023)’s study, they
find that asking language models directly how to write a good prompt for a
given task of interest and then writing a prompt following what the language
model responded can yield better answers. Third, if the response is required
to be in a specific format, delineate the format with examples (e.g., make
step-by-step reasoning, and provide a pros and cons comparison). Lastly, a
good prompt generally benefits from several iterations and experiments by
rephrasing the prompt and providing additional information. Much like tuning
the hyperparameters in a conventional machine learning model, there is no one
formal way of creating the ideal prompt: a degree of experimentation is always
required.

Verbalizer design is also worth considering. A verbalizer maps a label to a
list of label words which are recommended to have diverse coverage and little
subjective bias for eliciting comprehensive semantic information from language
models (Hu et al., 2022). Previous work experimenting with multiple label
words for each label shows that the optimal size of label words is generally
around three, while varying the number of label words does not lead to
significant improvements (Schick et al., 2020; Shin et al., 2020).

Just as with the other techniques above, labelled test data is essential for
the evaluation of a model based on prompt engineering. However the amount of
training data required is open to question. Up until very recently, prompt
engineering models also largely made use of some training data to boost
performance through fine-tuning, similar to transfer learning above. But the
emergence of openly available large language models such as the GPT family
from OpenAI has challenged this paradigm, creating the potential to prompt
models without any pre-training at all, sometimes called ‘zero-shot’ learning
(Wei et al., 2021; Ratner et al., 2019). As we will show below, performance
can be high even in these zero-shot cases.

All large language models have a set of hyperparameters that also need to be
considered when setting them up for a classification task (Bommarito II and
Katz, 2022; Rehana et al., 2023). Probably the most crucial to mention here is
‘temperature’ which affects how deterministic the output of the model is. More
specifically, given that the input into a large language model is a sequence
of text (sometimes called a ‘context window’), the output of the model is a
list of words with associated probabilities of being the next word in the
sequence. The model then picks a word from this list based on the temperature,
with a higher temperature making it more likely to pick lower probability
words. For creative writing tasks, setting a relatively high temperature seems
to generate better performance (Roemmele and Gordon, 2018; Caccia et al.,
2020); for classification tasks, lower temperatures seem like a better place
to start (Rehana et al., 2023). However, the ‘best’ temperature is an open
question and hard to define in abstract.

Similar to prompt engineering, zero-shot prompt engineering is particularly
beneficial for scenarios where (1) language models show strong performance and
(2) labelled examples are scarce or not available (Schick and Schütze, 2021a;
Liu et al., 2021; Hu et al., 2022). Furthermore, while zero-shot prompt
engineering can benefit from narrow prompting aimed at a specific task (Liu et
al., 2023), broader prompting aimed at eliciting wider cognitive abilities in
large language models has also seen success (Kojima et al., 2022).

A few final comments on other resources required for prompt engineering are
worth mentioning. While developments in the area of model miniaturization are
continuing rapidly (Bai et al., 2022), at the time of writing running the
latest generation of large language models on a consumer laptop was not a
realistic possibility, hence a server running a GPU was still likely a
requirement.444Statements on the size of models that a typical laptop can run
will go out of date quickly, however at the time of writing Llama 2-70b,
arguably the best open source LLM, required about 320 GB of RAM to run
(NVIDIA, 2022), which is beyond the realistic range of a typical laptop.
Running quantized versions of Llama is undoubtedly possible, though quantized
versions may offer lower performance as well as still being quite slow to run.
LLMs are also accessible through APIs, however as we will describe below these
often have a cost attached to them.

##  3 Methods

Having introduced the three cheap learning techniques under study in this
article, in the remaining part of this paper we want to provide an
illustration of them in action, by applying them to two example binary
classification tasks related to personal abuse and the sentiment of a movie
review. We have chosen these two tasks because we believe they are relatively
standard examples of the types of classification task that are appropriate for
the social sciences.

In addition to providing an illustration of the techniques in action, our
experiments will also help to explore the extent to which these techniques are
truly ‘cheap’, by exploring the amount of training data required to achieve
different levels of performance, which we refer to as the labelling budget
(noting again that for all of the methods we discuss there would still be a
requirement to develop a test dataset).

In this methods section, we describe our approach to these experiments: we
look first at the datasets we have chosen and how we sample from them, then
describe the classification tasks we seek to perform, then discuss the
computing infrastructure used for the experiments, and also explore the
particular hyperparameters we have made use of in our experiments.

###  3.1 Data

In our experimental section we address two classification tasks: binary abuse
classification and movie review sentiment classification. In order to conduct
experiments on these tasks, we make use of existing publicly available
labelled datasets. For the abuse classification task we make use of the
‘Wikipedia Talk: Personal Attacks’ dataset (Wulczyn et al., 2017), containing
115,864 comments from English Wikipedia labelled according to whether they
contain a personal attack (no personal attack - 88.3%, contains personal
attack - 11.7%). This dataset is a popular one among social scientists,
already employed in a variety of studies (Tay et al., 2021a, b; Dinan et al.,
2021; Sarwar et al., 2022; Xu et al., 2021).

For the movie sentiment task, we rely on the ‘IMDb Movie Review Sentiment’
dataset (Maas et al., 2011), containing 50,000 movie reviews labelled
according to whether they have a positive sentiment or negative sentiment
(negative sentiment - 50%, positive sentiment - 50%). The dataset has also
been extensively studied in the literature (Tay et al., 2021b; Priyadarshini
and Cotton, 2021; Yu et al., 2021; Birjali et al., 2021; Saunshi et al.,
2020).

The two chosen datasets have a number of differences, the most obvious being
the context in which text entries were written. The Wikipedia Personal Attacks
dataset entries are comments on an article, that could be directed at another
user or the article’s content, whereas the IMDb Movie Sentiment entries are
user reviews, all targeted at some piece of media, usually providing some kind
of detailed critique. This difference in context could suggest a greater range
of types of responses, and possibly differentiation between classes, in the
Wikipedia dataset compared to the IMDb dataset.

This context is reflected in the length of the entries across the two datasets
(Table 1), with entries from the IMDb dataset being over three times as long
as entries from the Wikipedia dataset on average (1297 characters vs. 402
characters). Furthermore, the balance of positive and negative labels in the
datasets represents another difference. While the IMDB Movie Review Sentiment
dataset has an even 50/50 split between labels, the Wikipedia Personal Attacks
dataset is unbalanced, which provides another dimension to examine performance
between techniques over. Evaluating the chosen cheap learning techniques
across datasets with these differences provides a more generalisable view into
the pros and cons of these techniques across different scenarios a social
scientist might encounter.

Dataset |  Label | Text Length (Characters) |  |  |
---|---|---|---|---|---
|  | Mean | Std Dev | Maximum | Minimum
Wikipedia Personal Attacks |  All | 402 | 734 | 10000 | 1
|  Positive | 341 | 938 | 10000 | 5
|  Negative | 410 | 702 | 10000 | 1
IMDb Movie Review Sentiment |  All | 1297 | 980 | 14106 | 32
|  Positive | 1303 | 1014 | 13593 | 65
|  Negative | 1271 | 928 | 8698 | 32
Table 1: Statistics of text length across labels for the two datasets.

For both datasets, we use the existing train splits (breakdowns visible in
Table 2) to create seven training sets of increasing size ([16, 32, 64, 128,
256, 512, 1024]), in order to assess performance at different labelling
budgets. For the Wikipedia Personal Attacks dataset, we create two versions of
these seven training sets: one with the original balance of labels (11.7%
positive, 88.3% negative), and one with an even balance (50% positive, 50%
negative), in order to provide insight into the impact of balanced vs.
unbalanced data on technique performance. As the original test data is large
and performing inference on such large data is time-consuming, we create a
final test set for each dataset from a 10% stratified random sample
(preserving the original class balance) of the original data, resulting in a
final test set of 2,316 entries for Wikipedia Personal Attacks, and 1,250
entries for IMDb Movie Review Sentiment.

Dataset Split Label All Positive Negative Wikipedia Personal Attacks All
115,864 13,590 102,274 Train 69,526 8,079 61,447 Test 23,178 2,756 20,422
Development 23,160 2,755 20,405 Exploration (WS) 100 12 88 IMDb Movie Review
Sentiment All 50,000 25,000 25,000 Train 25,000 12,500 12,500 Test 12,500
6,250 6,250 Development 12,500 6,250 6,250 Exploration (WS) 100 92 8

Table 2: Counts of dataset entries across splits and labels. The exploration
data set is exclusive for the Weak Supervision Technique.

An extra split is added exclusively for the weak supervision technique. This
labelled exploration test is intended to test and fine-tune labelling
functions then used in the training process. In our case, we take 100 data
points for exploration of each task. Details of these sets are detailed in
Table 2 and these additional points should be considered in the labelling
budget. The exploration set is drawn from the training set, making sure that
the data points in this set never overlap with the 7 training sets mentioned
in the previous paragraph.

###  3.2 Model Selection and Training Details

For the conventional baseline models, we develop a Multinomial Naive Bayes
classifier (typically used for text classification - e.g., to represent how
many times a word appears in a document) built based on Bayes’s theorem
(Kibriya et al., 2005).555We use the package developed by scikit-learn:
https://scikit-
learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html This
method is selected because it can be easily developed and speedily trained
without using GPUs. We use TF-IDF to convert data into vectors/features.

For weak supervision, we use the LabelModel presented in Ratner et al. (2019,
2016) and implemented in the Snorkel package (Ratner et al., 2017). This model
takes the label matrix obtained by applying the labelling functions to the
training data set and uses a Bayesian probabilistic model and a matrix
completion algorithm to obtain a set of weights. These weights are trained to
minimise the error between the golden labels from the training data set and
the labels obtained by the weighted sum of labelling functions over the data
set. Once the model is trained, it is applied to the test set.

For both transfer learning and prompt engineering, there is a need to choose a
pre-existing model. For both techniques, we chose pre-trained Distilbert-base-
cased (a distilled version of BERT, Devlin et al. (2019)) as a base model
since it retains 97% of BERT performance on several natural language tasks
while being 40% smaller and 60% faster (Sanh et al., 2019). Such
characteristics are in line with our goals of developing applications with
economic budgets requiring less training time, computation and memory. In
addition to the base model, we conducted additional sets of experiments using
DeBERTa-v3 (He et al., 2021) and GPT-2 (Radford et al., 2019) for transfer
learning and prompt engineering, respectively. DeBERTa-v3 is a DeBERTa model
with improvements based on replaced token detection and a new weight-sharing
method. GPT-2 is an autoregressive transformer-based language model trained on
a dataset of 8 million web pages using a next-word prediction objective. While
DeBERTa-v3 and GPT-2 have larger model sizes and slightly different model
structures and training procedures, both have shown superior performance on
downstream tasks (Rodríguez-Sánchez et al., 2022). By including a variety of
starting points, we can study the impacts of using different base models on
our classification tasks.

For our zero-shot learning experiments, meanwhile, we make use of GPT-3, 3.5
and 4, as provided through the OpenAI API. We make use of these because, at
the time of writing, they are widely regarded as amongst the state of the art
in terms of large language models and thus likely to offer the best
performance for the task at hand. As they are available through an API, they
can also be used by anyone with an internet connection, hence size and
computational costs are much less of a consideration (we will comment on the
actual cost of accessing the API below).

###  3.3 Hyperparameter selection and training details

We fine-tuned all models selected for transfer learning and prompt engineering
for three epochs with the default set of hyperparameters: a learning rate of
1e-3, a maximum text length of 512 tokens and a batch size of eight. To
facilitate reproducibility, we conducted three trials of each experimental
configuration using seeds [1, 2, 3], with only the seed varied. We use the
transformers library (Wolf et al., 2020) for transfer learning and the
OpenPrompt library (Ding et al., 2022) for prompt engineering. Our experiments
were performed on a single NVIDIA K80 GPU with 56 GiB memory.

Specific to prompt engineering, the training data is structured as
$[text]\;Prompt\;[class]$. As discussed above, finding the best prompt is not
trivial and requires trial and error (though previous work shows that prompt
choice is not a dominant factor in model performance compared with hyper-
parameters such as random seeds, Le Scao and Rush (2021)). To try and mitigate
the possibility that our results are based on the specifics of an arbitrary
prompt template, we test three prompt variations for each task. The prompts
are varied in pattern and language style while being valid descriptions
without changing the semantics, representing real-world user scenarios. For
abuse classification, the prompts are, ‘Is this text abusive?’, ‘Does this
text contain abuse?’, and ‘It was? Abusive or Not Abusive’. For movie
sentiment classification task, the prompts are, ‘Is this text negative?’,
‘Does this text contain negative sentiment?’, and ‘It was? Negative or Not
Negative’. Having a variety of prompts allows us to show how performance might
vary between different reasonable prompts.

OpenAI’s API also has a number of hyperparameters that may affect the output
provided by the model (OpenAI, 2020). For example, previous work has evaluated
generated text on varying levels of temperature, top p, and max tokens
(Bommarito II and Katz, 2022; Rehana et al., 2023). Similar to Rehana et al.
(2023), we set temperature, which affects how deterministic the output text
is, to 0.1. While temperature is above zero the results will not be purely
deterministic, however performance has been shown to improve empirically.
While each of our prompts attempts to elicit a single-word response, we set
the maximum number of returned tokens to 20 in order to examine why non-
response occurs within the model. We additionally provide a number of stop
tokens, such as a period and a comma, to limit as much as possible the model’s
tendency to provide more generated text beyond the single-word response we ask
of it.

##  4 Results

In this section, we present the results of our experiments, to illustrate
these techniques in action. Although our experiments made use of multiple
models and configurations, in order to simplify presentation we display just a
sample of these. In particular, for transfer learning we show results
generated using DistilBERT, and for prompt engineering we also look at
DistilBERT and look at only one fixed prompt (‘Does this text contain abuse?’
for abuse classification and ‘Does this text contain negative sentiment?’ for
movie sentiment classification). For Naïve Bayes and weak supervision, we
present the full results. Other base models and prompts provided similar
results.

We begin by investigating our binary abuse classification task, shown in
Figure 1. This figure has several panels, reflecting whether the underlying
data is balanced or not. We can see that in the simple case of underlying
balanced data (Figure 1, Panel (a)), which will hence generate both balanced
training datasets and test datasets), there is actually little difference
between any of the techniques in question. The ‘conventional’ machine learning
approach achieves a high performance (Macro F1 of around 0.8) with only around
100 examples. While it is eventually outperformed by prompt engineering,
transfer learning and weak supervision, this is only after the labelling
budget crosses the 250 mark, and even after 1,000 examples have been used the
difference is not enormous. Zero-shot learning with GPT-3.5 and GPT-4 offers
an impressive Macro F1 (around 0.9) without any training data (when
interpreting the labelling budgets, we should also bear in mind that, in
addition to the training budget, we need to consider the 100 data points used
in the exploration data set for the weak supervision labelling functions).

Figure 1: Learning curves of binary abuse classification task.

However, we know that the prevalence of abusive text is generally lower than
the presence of non-abusive text. Hence a more ‘realistic’ scenario is that
both training and test data are unbalanced (Figure 1, Panel (b)). Here we can
see that conventional machine learning performs much worse than its ‘cheaper’
counterparts, with a Macro F1 score of less than 0.5 for all levels of the
training budget investigated. In the unbalanced case, where abuse is around
10% of the total dataset, then even with a relatively large labelling budget
only a small amount of cases from the positive class will be seen. The
difference between the other techniques in particular is notable, with
transfer learning able to achieve the highest overall performance (though the
Macro F1 of 0.8 from GPT-4 is again notable, considering that no training data
is required to achieve it).

Of course, in the case of unbalanced data, one common workaround is to
artificially balance the training dataset (though of course preserving the
real underlying distributions of data in the test data). This can improve
model performance on a limited training budget (though achieving training
balanced data still requires a higher labelling effort, or trying to
artificially inflate the likelihood of labelling positive data through the use
of keywords). Results from this approach are presented in Figure 1, Panel (c).
We can see that model performance is improved in all cases, reaching the range
of 0.7-0.8 after a few hundred examples (the zero-shot approaches all fall
within this range as well). Weak supervision is actually the best-performing
technique in this scenario (as the test data is the same, the results for
zero-shot prompt engineering are the same in Panels (b) and (c)).

We will now move on to address our binary movie sentiment classification task,
shown in Figure 2. The format of this figure is the same, with panels
addressing classifier performance over a variety of labelling budgets for
balanced data, unbalanced data, and a balanced training set with unbalanced
test data. Again, these three setups were chosen as representative of common
real-world research scenarios. The overall results from this task are similar
to the abuse classification task. For balanced data, we can see that all of
the methods perform in a similar fashion. Our conventional machine learning
approach achieves a Macro F1 of 0.8 with around 100 labelled examples. Prompt
engineering, transfer learning and weak supervision are able to outperform it,
especially as the labelling budget increases, eventually arriving at scores of
around 0.9. Again, zero-shot learning performs well, with performance above
0.9 for GPT-4. In the unbalanced case, the performance of all models is lower,
though transfer learning and prompt engineering can achieve a Macro F1 of 0.8
with around 1,000 examples, again common with GPT-4. Our Naïve Bayes model
performs very poorly in this context. Finally, when training data is balanced
but test data remains unbalanced, more of the approaches can achieve a Macro
F1 of around 0.8, though we would recall again that the cost of generating a
balanced training dataset from unbalanced data can be quite high.

Figure 2: Learning curves of binary movie sentiment classification task.

The IMDB movie sentiment database (Maas et al., 2011) has been, as mentioned
in the Methods section 3.1, extensively studied in the literature. As such, a
natural worry arises that the database has been used in the training datasets
of GPT-3.5 and GPT-4, thus contaminating the zero-shot testing described
above. Given that we are unable to check if indeed the database was used to
train the GPT models, we collected an analogous validation of movie reviews
from another review aggregator, TMDB666https://www.themoviedb.org/. The
included reviews were publicly published from October 2021, past the training
date cut-off for the GPT-3.5. As in Maas et al. (2011), we label as positive
those reviews with a score higher than 6 out of 10, and we label negative
those reviews leaving a score lower than 5, omitting neutral reviews and
including no more than 30 reviews from a single movie. Our final database
comprises 855 reviews, with a distribution of 73.3% of positive reviews and
26.7% of negative reviews. When performing a zero-shot test on the TMDB
dataset, we obtain Macro F1 scores between 0.87 and 0.94 for both GPT-3.5 and
GPT-4, using the three prompts as the other tasks. GPT-3 has, on the other
hand, different performances depending on the prompt, with Macro F1 scores of
0.67, 0.85 and 0.11. The results are in accordance with the results presented
in Figure 2, thus validating them for our use of the three versions of GPT. In
other words, we are confident the high performance of GPT in these
classification tasks is not solely a result of GPT’s training data already
including the IMDB dataset.

It is also worth evaluating the amount of training time required for each of
the techniques, something which has knock-on consequences for the financial
costs involved if making use of a cloud platform. Overall, the training time
is short for all three techniques and two datasets (see Figures 3 and 4). As
the labelling budget grows, the training time also grows linearly except for
weak supervision. For weak supervision, increasing labelling examples does not
seem to affect training time when trained on Wikipedia personal attacks
dataset. The training time is generally around 1 minute regardless of
labelling budgets and slightly decreases when using the maximum labelling
budget in our experiments (n=1024). Training times in this case are a sum of
two elements: the time that the set of labelling functions takes to label data
points and the time that the probabilistic LabelModel (Ratner et al., 2019)
takes to compute the weights and probabilities associated to the labelling
functions. The different behaviours seen in Figure 3 and Figure 4 depend on
the nature of the labelling functions used. In the case of the binary movie
sentiment analysis, we add two regex LFs, thus observing the linear increase
of time as they require more time to look for matches as the number of data
points increases. In the case of the binary abuse experiment, the labelling
functions used are dictionary-based or transforming annotators’ knowledge into
a labelling function. Given that directory-based functions (looking for words
in a text) are not as computationally expensive as regular expression
functions, given that the maximum length of the text is fixed, we obtain a
stable training time of 1 minute.

Figure 3: Training time of binary abuse classification task. Figure 4:
Training time of binary movie sentiment classification task.

While Zero-shot Prompt Engineering does not require any training time, it is
worth briefly commenting on the costs that could be incurred through the API
to perform classification tasks (these are correct at the time of writing).
Costs through such an API are per ‘token’: a small piece of text that might be
equivalent to a word, a part of a word or a piece of punctuation (as a rule of
thumb, a token is often around four characters long). To perform a
classification using an LLM there will be costs for both input tokens (the
chunk of text to be classified) and output tokens (the response indicating the
classification). Of course, the response is usually just one or two tokens
long, meaning the length of the input text is the main factor determining API
costs. At the time of writing, GPT-3 had a cost of £0.0006 per 1,000 tokens,
GPT-3.5 a cost of £0.0013 per 1,000 tokens and GPT-4 a cost of £0.0025 per
1,000 tokens. The price for a project will, obviously, vary according to the
model chosen and the type of input text but, to give an example, the movie
reviews were on average around 325 tokens each, meaning that 10,000 movie
reviews could be classified on GPT-4 for around £8. Hence, API costs are only
likely to be a significant concern for relatively large pieces of text where
there are millions of observations to be classified.

##  5 Conclusions

In this paper, we have sought to give an overview of three methods for
automatic content analysis that are ‘cheaper’ than the conventional approach
to machine learning, in the sense that they seek to limit the amount of
labelled data that is required as a basis for training a model. In this final
section, we will draw some conclusions about the significance of our
experimental results, and future directions for automatic content analysis in
the field.

First, it was striking how all of the techniques we reviewed achieved strong
performance in most of the tasks we reviewed, on small labelling budgets. In
most cases, a few hundred data points were enough to achieve a Macro F1 score
of above 0.7. It should be noted however that the conventional baseline we
addressed also performed well for many tasks on similar labelling budgets,
provided the data were balanced (though we should emphasise that unbalanced
data scenarios are very common in automatic content analysis).

Second, perhaps most of all, it was striking how zero-shot prompt engineering
performed well across all of the tasks we reviewed, comfortably outperforming
any of the other methods on the movie review task and offering equivalent
performance on the abuse classification task. Considering these techniques
require no labelled training data (though still require test data) and are
very easy to implement, they appear to offer a very promising and flexible
solution to automatic content analysis in the future. Their performance on a
wider variety of social science concepts needs to be tested; and it should
also be borne in mind that there is a financial cost to using them (though not
one that will be significant unless millions of lengthy text documents are
being labelled). Furthermore, larger models than the current generation of
LLMs are already in development, and it is reasonable to expect both
performance gains and price decreases as they emerge (in all of our
experiments, the increase in size from GPT-3 to GPT-4 corresponded with an
increase in performance). As this next generation emerges, it may be that
other methods of automatic content analysis will start to become redundant,
though this proposition would need to be rigorously tested.

Lastly, despite the impressive performance of zero-shot prompt engineering
using LLMs, it is important to assess LLMs across various types of tasks to
understand their capacity and limitations. For instance, our results show that
model performance can vary across seeds or the prompts tested. This
instability further highlights the importance of robustness and
reproducibility in LLMs. Furthermore, LLMs can be biased towards certain
demographics or identities when, for example, asked to perform curriculum
screening (Dastin, 2022; Veldanda et al., 2023). In this paper, we solely
consider binary classification tasks. How LLMs perform on more complicated
tasks that involve language understanding such as logical reasoning and
inference remains an open research question.

##### Acknowledgments.

This work was supported by the Public Policy Programme of The Alan Turing
Institute under the EPSRC grant EP/N510129/1. This work was supported by the
Ecosystem Leadership Award under the EPSRC Grant EP/X03870X/1 and The Alan
Turing Institute. Funded by the UK Government

##### Data Availability Statement.

Our code and experimental results are available here at
https://github.com/Turing-Online-Safety-Codebase/cheap_learning, allowing
users to understand, verify and replicate findings.

## References

  * Al-Jarrah et al. [2015] Omar Y. Al-Jarrah, Paul D. Yoo, Sami Muhaidat, George K. Karagiannidis, and Kamal Taha.  Efficient Machine Learning for Big Data: A Review.  _Big Data Research_ , 2(3):87–93, September 2015.  ISSN 2214-5796.  doi: 10.1016/j.bdr.2015.04.001.  URL https://www.sciencedirect.com/science/article/pii/S2214579615000271.
  * Ali et al. [2022] Raza Ali, Umar Farooq, Umair Arshad, Waseem Shahzad, and Mirza Omer Beg.  Hate speech detection on twitter using transfer learning.  _Computer Speech & Language_, 74:101365, 2022.
  * Anaby-Tavor et al. [2020] Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling.  Do not have enough data? deep learning to the rescue!  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 7383–7390, 2020.
  * Baden et al. [2022] Christian Baden, Christian Pipal, Martijn Schoonvelde, and Mariken A. C. G van der Velden.  Three Gaps in Computational Text Analysis Methods for Social Sciences: A Research Agenda.  _Communication Methods and Measures_ , 16(1):1–18, January 2022.  ISSN 1931-2458.  doi: 10.1080/19312458.2021.2015574.  URL https://doi.org/10.1080/19312458.2021.2015574.
  * Bai et al. [2022] Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R. Lyu.  Towards Efficient Post-training Quantization of Pre-trained Language Models.  _Advances in Neural Information Processing Systems_ , 35:1405–1418, December 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/096347b4efc264ae7f07742fea34af1f-Abstract-Conference.html.
  * Bashath et al. [2022] Samar Bashath, Nadeesha Perera, Shailesh Tripathi, Kalifa Manjang, Matthias Dehmer, and Frank Emmert Streib.  A data-centric review of deep transfer learning with applications to text data.  _Information Sciences_ , 585:498–528, 2022.
  * Beigman Klebanov and Madnani [2020] Beata Beigman Klebanov and Nitin Madnani.  Automated Evaluation of Writing – 50 Years and Counting.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7796–7810, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.697.  URL https://aclanthology.org/2020.acl-main.697.
  * Bengio [2012] Yoshua Bengio.  Practical recommendations for gradient-based training of deep architectures.  In _Neural Networks: Tricks of the Trade: Second Edition_ , pages 437–478. Springer, 2012.
  * Berger and Goldstein [2021] Maria Berger and Elizabeth Goldstein.  Increasing Sentence-Level Comprehension Through Text Classification of Epistemic Functions.  In Claire Bonial and Nianwen Xue, editors, _Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop_ , pages 139–150, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.law-1.15.  URL https://aclanthology.org/2021.law-1.15.
  * Berry et al. [2019] Michael W Berry, Azlinah Mohamed, and Bee Wah Yap.  _Supervised and unsupervised learning for data science_.  Springer, 2019.
  * Birjali et al. [2021] Marouane Birjali, Mohammed Kasri, and Abderrahim Beni-Hssane.  A comprehensive survey on sentiment analysis: Approaches, challenges and trends.  _Knowledge-Based Systems_ , 226:107134, August 2021.  ISSN 0950-7051.  doi: 10.1016/j.knosys.2021.107134.  URL https://www.sciencedirect.com/science/article/pii/S095070512100397X.
  * Bommarito II and Katz [2022] Michael Bommarito II and Daniel Martin Katz.  Gpt takes the bar exam.  _ArXiv Computer Science_ , 2022.
  * Boyd and Crawford [2011] Danah Boyd and Kate Crawford.  Six Provocations for Big Data.  _SSRN_ , September 2011.  doi: 10.2139/ssrn.1926431.  URL https://papers.ssrn.com/abstract=1926431.
  * Burscher et al. [2015] Bjorn Burscher, Rens Vliegenthart, and Claes H. De Vreese.  Using Supervised Machine Learning to Code Policy Issues: Can Classifiers Generalize across Contexts?  _The ANNALS of the American Academy of Political and Social Science_ , 659(1):122–131, May 2015.  ISSN 0002-7162.  doi: 10.1177/0002716215569441.  URL https://doi.org/10.1177/0002716215569441.  Publisher: SAGE Publications Inc.
  * Burscher et al. [2014] Björn Burscher, Daan Odijk, Rens Vliegenthart, Maarten de Rijke, and Claes H. de Vreese.  Teaching the Computer to Code Frames in News: Comparing Two Supervised Machine Learning Approaches to Frame Analysis.  _Communication Methods and Measures_ , 8(3):190–206, July 2014.  ISSN 1931-2458.  doi: 10.1080/19312458.2014.937527.  URL https://doi.org/10.1080/19312458.2014.937527.  Publisher: Routledge _eprint: https://doi.org/10.1080/19312458.2014.937527.
  * Caccia et al. [2020] Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin.  Language GANs Falling Short, February 2020.  URL http://arxiv.org/abs/1811.02549.  arXiv:1811.02549 [cs].
  * Cherkassky and Mulier [2007] Vladimir Cherkassky and Filip M Mulier.  _Learning from data: concepts, theory, and methods_.  John Wiley & Sons, 2007.
  * Chung et al. [2020] Yi-Ling Chung, Serra Sinem Tekiroglu, and Marco Guerini.  Italian counter narrative generation to fight online hate speech.  In _CLiC-it_ , 2020.
  * Conneau and Lample [2019] Alexis Conneau and Guillaume Lample.  Cross-lingual language model pretraining.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf.
  * Dastin [2022] Jeffrey Dastin.  Amazon scraps secret ai recruiting tool that showed bias against women.  In _Ethics of data and analytics_ , pages 296–299. Auerbach Publications, 2022.
  * Davison et al. [2019] Joe Davison, Joshua Feldman, and Alexander Rush.  Commonsense knowledge mining from pretrained models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1173–1178, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1109.  URL https://aclanthology.org/D19-1109.
  * De Grove et al. [2020] Frederik De Grove, Kristof Boghe, and Lieven De Marez.  (What) Can Journalism Studies Learn from Supervised Machine Learning?  _Journalism Studies_ , 21(7):912–927, May 2020.  ISSN 1461-670X.  doi: 10.1080/1461670X.2020.1743737.  URL https://doi.org/10.1080/1461670X.2020.1743737.  Publisher: Routledge _eprint: https://doi.org/10.1080/1461670X.2020.1743737.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Dinan et al. [2021] Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser.  Anticipating safety issues in e2e conversational ai: Framework and tooling.  _arXiv preprint arXiv:2107.03451_ , 2021.
  * Ding et al. [2022] Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun.  OpenPrompt: An open-source framework for prompt-learning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 105–113, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-demo.10.  URL https://aclanthology.org/2022.acl-demo.10.
  * Dong et al. [2019] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon.  Unified language model pre-training for natural language understanding and generation.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , page 13063–13075, Red Hook, NY, USA, 2019. Curran Associates Inc.
  * Fatourechi et al. [2008] Mehrdad Fatourechi, Rabab K Ward, Steven G Mason, Jane Huggins, Alois Schlögl, and Gary E Birch.  Comparison of evaluation metrics in classification applications with imbalanced datasets.  In _2008 seventh international conference on machine learning and applications_ , pages 777–782. IEEE, 2008.
  * Fries et al. [2021] Jason A. Fries, Ethan Steinberg, Saelig Khattar, Scott L. Fleming, Jose Posada, Alison Callahan, and Nigam H. Shah.  Ontology-driven weak supervision for clinical entity classification in electronic health records.  _Nature Communications_ , 12(1):2017, April 2021.  ISSN 2041-1723.  doi: 10.1038/s41467-021-22328-4.  URL https://www.nature.com/articles/s41467-021-22328-4.  Number: 1 Publisher: Nature Publishing Group.
  * Gangula et al. [2019] Rama Rohit Reddy Gangula, Suma Reddy Duggenpudi, and Radhika Mamidi.  Detecting Political Bias in News Articles Using Headline Attention.  In Tal Linzen, Grzegorz Chrupa\la, Yonatan Belinkov, and Dieuwke Hupkes, editors, _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 77–84, Florence, Italy, August 2019. Association for Computational Linguistics.  doi: 10.18653/v1/W19-4809.  URL https://aclanthology.org/W19-4809.
  * Geva et al. [2019] Mor Geva, Yoav Goldberg, and Jonathan Berant.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1161–1166, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1107.  URL https://aclanthology.org/D19-1107.
  * Gilardi et al. [2023] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.  ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks.  _Proceedings of the National Academy of Sciences_ , 120(30):e2305016120, July 2023.  ISSN 0027-8424, 1091-6490.  doi: 10.1073/pnas.2305016120.  URL http://arxiv.org/abs/2303.15056.  arXiv:2303.15056 [cs].
  * Goldsteen et al. [2022] Abigail Goldsteen, Gilad Ezov, Ron Shmelkin, Micha Moffie, and Ariel Farkash.  Anonymizing Machine Learning Models.  In Joaquin Garcia-Alfaro, Jose Luis Muñoz-Tapia, Guillermo Navarro-Arribas, and Miguel Soriano, editors, _Data Privacy Management, Cryptocurrencies and Blockchain Technology_ , Lecture Notes in Computer Science, pages 121–136, Cham, 2022. Springer International Publishing.  ISBN 978-3-030-93944-1.  doi: 10.1007/978-3-030-93944-1_8.
  * Golovanov et al. [2019] Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, and Thomas Wolf.  Large-scale transfer learning for natural language generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6053–6058, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1608.  URL https://aclanthology.org/P19-1608.
  * Grimmer and Stewart [2013] Justin Grimmer and Brandon M. Stewart.  Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.  _Political Analysis_ , 21(3):267–297, July 2013.  ISSN 1047-1987, 1476-4989.  doi: 10.1093/pan/mps028.  URL https://www.cambridge.org/core/journals/political-analysis/article/text-as-datathe-promise-and-pitfalls-of-automatic-content-analysis-methods-for-politicaltexts/F7AAC8B2909441603FEB25C156448F20.  Publisher: Cambridge University Press.
  * Grimmer et al. [2021] Justin Grimmer, Margaret E. Roberts, and Brandon M. Stewart.  Machine Learning for Social Science: An Agnostic Approach.  _Annual Review of Political Science_ , 24(1):395–419, 2021.  doi: 10.1146/annurev-polisci-053119-015921.  URL https://doi.org/10.1146/annurev-polisci-053119-015921.  _eprint: https://doi.org/10.1146/annurev-polisci-053119-015921.
  * Guan and Koudas [2022] Naiqing Guan and Nick Koudas.  Fila: Online auditing of machine learning model accuracy under finite labelling budget.  In _Proceedings of the 2022 International Conference on Management of Data_ , SIGMOD ’22, page 1784–1794, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392495.  doi: 10.1145/3514221.3517904.  URL https://doi.org/10.1145/3514221.3517904.
  * Guo et al. [2019] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris.  Spottune: Transfer learning through adaptive fine-tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4805–4814, 06 2019.
  * He et al. [2021] Pengcheng He, Jianfeng Gao, and Weizhu Chen.  DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing.  _arXiv e-prints_ , art. arXiv:2111.09543, November 2021.  doi: 10.48550/arXiv.2111.09543.
  * Hossin and Sulaiman [2015] M Hossin and MN Sulaiman.  A review on evaluation metrics for data classification evaluations.  _International Journal of Data Mining & Knowledge Management Process_, 5(2):1, 2015.
  * Houlsby et al. [2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for NLP.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/houlsby19a.html.
  * Hsieh et al. [2023] Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8003–8017, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.507.  URL https://aclanthology.org/2023.findings-acl.507.
  * Hu et al. [2022] Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun.  Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2225–2240, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.158.  URL https://aclanthology.org/2022.acl-long.158.
  * Hube et al. [2019] Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju.  Understanding and mitigating worker biases in the crowdsourced collection of subjective judgments.  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , CHI ’19, page 1–12, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450359702.  doi: 10.1145/3290605.3300637.  URL https://doi.org/10.1145/3290605.3300637.
  * Hulth and Megyesi [2006] Anette Hulth and Beáta B. Megyesi.  A study on automatically extracted keywords in text categorization.  In Nicoletta Calzolari, Claire Cardie, and Pierre Isabelle, editors, _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_ , pages 537–544, Sydney, Australia, July 2006. Association for Computational Linguistics.  doi: 10.3115/1220175.1220243.  URL https://aclanthology.org/P06-1068.
  * Islam et al. [2017] Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup.  Reproducibility of benchmarked deep reinforcement learning tasks for continuous control.  _arXiv preprint arXiv:1708.04133_ , 2017.
  * Jain [2021] Navdeep Jain.  Customer sentiment analysis using weak supervision for customer-agent chat.  _arXiv: Computation and Language_ , 2021.  doi: null.
  * Jordan and Mitchell [2015] Michael I Jordan and Tom M Mitchell.  Machine learning: Trends, perspectives, and prospects.  _Science_ , 349(6245):255–260, 2015.
  * Kamal et al. [2020] Muhammad Ayoub Kamal, Hafiz Wahab Raza, Muhammad Mansoor Alam, and M Mohd.  Highlight the features of aws, gcp and microsoft azure that have an impact when choosing a cloud service provider.  _Int. J. Recent Technol. Eng_ , 8(5):4124–4232, 2020.
  * Kandel and Castelli [2020] Ibrahem Kandel and Mauro Castelli.  The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset.  _ICT Express_ , 6(4):312–315, 2020.  ISSN 2405-9595.  doi: https://doi.org/10.1016/j.icte.2020.04.010.  URL https://www.sciencedirect.com/science/article/pii/S2405959519303455.
  * Keskar et al. [2017] Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy.  On large-batch training for deep learning: Generalization gap and sharp minima.  In _5th International Conference on Learning Representations, ICLR 2017 ; Conference date: 24-04-2017 Through 26-04-2017_ , 2017.
  * Kibriya et al. [2005] Ashraf M. Kibriya, Eibe Frank, Bernhard Pfahringer, and Geoffrey Holmes.  Multinomial naive bayes for text categorization revisited.  In Geoffrey I. Webb and Xinghuo Yu, editors, _AI 2004: Advances in Artificial Intelligence_ , pages 488–499, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg.  ISBN 978-3-540-30549-1.
  * Kojima et al. [2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213, 2022.
  * Kolesnikov et al. [2020] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 491–507. Springer, 2020.
  * Kotsiantis et al. [2006] Sotiris B Kotsiantis, Ioannis D Zaharakis, and Panayiotis E Pintelas.  Machine learning: a review of classification and combining techniques.  _Artificial Intelligence Review_ , 26:159–190, 2006.
  * Le Scao and Rush [2021] Teven Le Scao and Alexander Rush.  How many data points is a prompt worth?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2627–2636, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.208.  URL https://aclanthology.org/2021.naacl-main.208.
  * Lehmann et al. [2023] Pola Lehmann, Simon Franzmann, Tobias Burst, Theres Matthieß, Sven Regel, Felicia Riethmüller, Andrea Volkens, Bernhard Weßels, Lisa Zehnter, Wissenschaftszentrum Berlin Für Sozialforschung (WZB), and Institut Für Demokratieforschung Göttingen (IfDem).  _Manifesto Project Dataset_ , 2023.  URL https://manifesto-project.wzb.eu/doi/manifesto.mpds.2023a.
  * Lindstrom et al. [2013] Patrick Lindstrom, Brian Mac Namee, and Sarah Jane Delany.  Drift detection using uncertainty distribution divergence.  _Evolving Systems_ , 4:13–25, 2013.
  * Liu et al. [2023] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Comput. Surv._ , 55(9), 01 2023.  ISSN 0360-0300.  doi: 10.1145/3560815.  URL https://doi.org/10.1145/3560815.
  * Liu et al. [2021] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.  GPT Understands, Too.  _arXiv e-prints_ , art. arXiv:2103.10385, March 2021.  doi: 10.48550/arXiv.2103.10385.
  * Luo [2016] Gang Luo.  A review of automatic selection methods for machine learning algorithms and hyper-parameter values.  _Network Modeling Analysis in Health Informatics and Bioinformatics_ , 5:1–16, 2016.
  * Maas et al. [2011] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning Word Vectors for Sentiment Analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.  URL https://aclanthology.org/P11-1015.
  * Macanovic [2022] Ana Macanovic.  Text mining for social science – The state and the future of computational text analysis in sociology.  _Social Science Research_ , 108:102784, November 2022.  ISSN 0049-089X.  doi: 10.1016/j.ssresearch.2022.102784.  URL https://www.sciencedirect.com/science/article/pii/S0049089X22000904.
  * Mahesh [2020] Batta Mahesh.  Machine learning algorithms-a review.  _International Journal of Science and Research (IJSR).[Internet]_ , 9(1):381–386, 2020.
  * Mallory et al. [2020] Emily K. Mallory, Matthieu de Rochemonteix, Alexander Ratner, Ambika Acharya, Christopher Ré, Roselie A. Bright, and Russ B. Altman.  Extracting chemical reactions from text using snorkel.  _BMC Bioinformatics_ , 21:217, 2020.  doi: 10.1186/s12859-020-03542-1.
  * Maresca et al. [2021] Fabio Maresca, Gürkan Solmaz, and Flavio Cirillo.  OntoAugment: Ontology Matching through Weakly-Supervised Label Augmentation.  In _Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems_ , SenSys ’21, pages 420–425, New York, NY, USA, November 2021. Association for Computing Machinery.  ISBN 978-1-4503-9097-2.  doi: 10.1145/3485730.3493445.  URL https://doi.org/10.1145/3485730.3493445.
  * Molina and Garip [2019] Mario Molina and Filiz Garip.  Machine Learning for Sociology.  _Annual Review of Sociology_ , 45(1):27–45, 2019.  doi: 10.1146/annurev-soc-073117-041106.  URL https://doi.org/10.1146/annurev-soc-073117-041106.
  * N and Gupta [2020] Thomas Rincy N and Roopam Gupta.  A survey on machine learning approaches and its techniques:.  In _2020 IEEE International Students’ Conference on Electrical,Electronics and Computer Science (SCEECS)_ , pages 1–6, 2020.  doi: 10.1109/SCEECS48394.2020.190.
  * Nelson and Taneja [2018] Jacob L Nelson and Harsh Taneja.  The small, disloyal fake news audience: The role of audience availability in fake news consumption.  _New Media & Society_, 20(10):3720–3737, October 2018.  ISSN 1461-4448, 1461-7315.  doi: 10.1177/1461444818758715.  URL http://journals.sagepub.com/doi/10.1177/1461444818758715.
  * Neyshabur et al. [2020] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang.  What is being transferred in transfer learning?  _Advances in neural information processing systems_ , 33:512–523, 2020.
  * NVIDIA [2022] NVIDIA.  _Sizing Guide_.  NVIDIA Docs, 2022.  URL https://docs.nvidia.com/ai-enterprise/workflows-generative-ai/0.1.0/sizing-guide.html.  Accessed: 14 June 2023.
  * Németh et al. [2020] Renáta Németh, Domonkos Sik, and Fanni Máté.  Machine Learning of Concepts Hard Even for Humans: The Case of Online Depression Forums.  _International Journal of Qualitative Methods_ , 19:1609406920949338, January 2020.  ISSN 1609-4069.  doi: 10.1177/1609406920949338.  URL https://doi.org/10.1177/1609406920949338.  Publisher: SAGE Publications Inc.
  * Onan et al. [2016] Aytuğ Onan, Serdar Korukoğlu, and Hasan Bulut.  Ensemble of keyword extraction methods and classifiers in text classification.  _Expert Systems with Applications_ , 57:232–247, 2016.
  * OpenAI [2020] OpenAI.  _API reference_.  OpenAI, 2020.  URL https://platform.openai.com/docs/api-reference/.  Accessed: 11 January 2024.
  * Pan and Yang [2010] Sinno Jialin Pan and Qiang Yang.  A survey on transfer learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 22(10):1345–1359, 2010.  doi: 10.1109/TKDE.2009.191.
  * Peters et al. [2019] Matthew E. Peters, Sebastian Ruder, and Noah A. Smith.  To tune or not to tune? adapting pretrained representations to diverse tasks.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 7–14, Florence, Italy, August 2019. Association for Computational Linguistics.  doi: 10.18653/v1/W19-4302.  URL https://aclanthology.org/W19-4302.
  * Petroni et al. [2019] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1250.  URL https://aclanthology.org/D19-1250.
  * Pierleoni et al. [2019] Paola Pierleoni, Roberto Concetti, Alberto Belli, and Lorenzo Palma.  Amazon, google and microsoft solutions for iot: Architectures and a performance comparison.  _IEEE access_ , 8:5455–5470, 2019.
  * Press et al. [2023] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis.  Measuring and narrowing the compositionality gap in language models.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5687–5711, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.378.  URL https://aclanthology.org/2023.findings-emnlp.378.
  * Priyadarshini and Cotton [2021] Ishaani Priyadarshini and Chase Cotton.  A novel LSTM–CNN–grid search-based deep neural network for sentiment analysis.  _The Journal of Supercomputing_ , 77(12):13911–13932, 2021.  ISSN 0920-8542.  doi: 10.1007/s11227-021-03838-w.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8097246/.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  unpublished, 2019.  URL http://www.persagen.com/files/misc/radford2019language.pdf.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(1):1–67, jan 2020.  ISSN 1532-4435.
  * Ratner et al. [2017] Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré.  Snorkel: Rapid Training Data Creation with Weak Supervision.  _Proceedings of the VLDB Endowment_ , 11(3):269–282, November 2017.  ISSN 2150-8097.  doi: 10.14778/3157794.3157797.  URL http://arxiv.org/abs/1711.10160.  arXiv:1711.10160 [cs, stat].
  * Ratner et al. [2019] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré.  Training Complex Models with Multi-Task Weak Supervision.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):4763–4771, July 2019.  ISSN 2374-3468.  doi: 10.1609/aaai.v33i01.33014763.  URL https://ojs.aaai.org/index.php/AAAI/article/view/4403.  Number: 01.
  * Ratner et al. [2016] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré.  Data Programming: Creating Large Training Sets, Quickly.  In _Advances in Neural Information Processing Systems_ , volume 29, page 3574–3582. Curran Associates, Inc., 2016.  URL https://proceedings.neurips.cc/paper/2016/hash/6709e8d64a5f47269ed5cea9f625f7ab-Abstract.html.
  * Rehana et al. [2023] Hasin Rehana, Nur Bengisu Çam, Mert Basmaci, Yongqun He, Arzucan Özgür, and Junguk Hur.  Evaluation of gpt and bert-based models on identifying protein-protein interactions in biomedical text.  _ArXiv Computer Science_ , 2023.
  * Rodríguez-Sánchez et al. [2022] Francisco Rodríguez-Sánchez, Jorge Carrillo-de Albornoz, Laura Plaza Morales, Adrián Mendieta-Aragón, Guillermo Marco Remón, Maryna Makeienko, María Plaza, Julio Gonzalo Arroyo, Damiano Spina, and Paolo Rosso.  Overview of exist 2022: sexism identification in social networks, 09 2022.
  * Roemmele and Gordon [2018] Melissa Roemmele and Andrew S. Gordon.  Automated assistance for creative writing with an rnn language model.  In _Proceedings of the 23rd International Conference on Intelligent User Interfaces Companion_ , IUI ’18 Companion, pages 1–2, New York, NY, USA, 2018. Association for Computing Machinery.  ISBN 9781450355711.  doi: 10.1145/3180308.3180329.  URL https://doi.org/10.1145/3180308.3180329.
  * Sanh et al. [2019] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ , 2019.
  * Sarwar et al. [2022] Sheikh Muhammad Sarwar, Dimitrina Zlatkova, Momchil Hardalov, Yoan Dinkov, Isabelle Augenstein, and Preslav Nakov.  A Neighborhood Framework for Resource-Lean Content Flagging.  _Transactions of the Association for Computational Linguistics_ , 10:484–502, May 2022.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00472.  URL https://doi.org/10.1162/tacl_a_00472.
  * Sathyanarayana Rao et al. [2018] T. S. Sathyanarayana Rao, Deepali Bansal, and Suhas Chandran.  Cyberbullying: A virtual offense with real consequences.  _Indian Journal of Psychiatry_ , 60(1):3–5, 2018.  ISSN 0019-5545.  doi: 10.4103/psychiatry.IndianJPsychiatry_147_18.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5914259/.
  * Saunshi et al. [2020] Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora.  A mathematical exploration of why language models help solve downstream tasks.  In _International Conference on Learning Representations_ , 2020.
  * Scharkow and Vogelgesang [2011] Michael Scharkow and Jens Vogelgesang.  Measuring the Public Agenda using Search Engine Queries.  _International Journal of Public Opinion Research_ , 23(1):104–113, March 2011.  ISSN 0954-2892.  doi: 10.1093/ijpor/edq048.  URL https://doi.org/10.1093/ijpor/edq048.
  * Schick and Schütze [2021a] Timo Schick and Hinrich Schütze.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online, April 2021a. Association for Computational Linguistics.  doi: 10.18653/v1/2021.eacl-main.20.  URL https://aclanthology.org/2021.eacl-main.20.
  * Schick and Schütze [2021b] Timo Schick and Hinrich Schütze.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online, June 2021b. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.185.  URL https://aclanthology.org/2021.naacl-main.185.
  * Schick et al. [2020] Timo Schick, Helmut Schmid, and Hinrich Schütze.  Automatically identifying words that can serve as labels for few-shot text classification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5569–5578, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.  doi: 10.18653/v1/2020.coling-main.488.  URL https://aclanthology.org/2020.coling-main.488.
  * Shams [2014] Rushdi Shams.  Semi-supervised classification for natural language processing.  _arXiv preprint arXiv:1409.7612_ , 2014.
  * Shin et al. [2020] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.346.  URL https://aclanthology.org/2020.emnlp-main.346.
  * Shu et al. [2021] Kai Shu, Guoqing Zheng, Yichuan Li, Subhabrata Mukherjee, Ahmed Hassan Awadallah, Scott Ruston, and Huan Liu.  Early Detection of Fake News with Multi-source Weak Social Supervision.  In Frank Hutter, Kristian Kersting, Jefrey Lijffijt, and Isabel Valera, editors, _Machine Learning and Knowledge Discovery in Databases_ , volume 12459, pages 650–666. Springer International Publishing, Cham, 2021.  ISBN 978-3-030-67663-6 978-3-030-67664-3.  doi: 10.1007/978-3-030-67664-3_39.  URL https://link.springer.com/10.1007/978-3-030-67664-3_39.  Series Title: Lecture Notes in Computer Science.
  * Silva et al. [2022] Carlos Anderson Oliveira Silva, Rafael Gonzalez-Otero, Michel Bessani, Liliana Margarita Otero Mendoza, Liliana Otero, and Cristiano Leite de Castro.  Interpretable risk models for sleep apnea and coronary diseases from structured and non-structured data.  _Expert Systems With Applications_ , 200:116955, 2022.  doi: 10.1016/j.eswa.2022.116955.
  * Souma et al. [2019] Wataru Souma, Irena Vodenska, and Hideaki Aoyama.  Enhanced news sentiment analysis using deep learning methods.  _Journal of Computational Social Science_ , 2(1):33–46, January 2019.  ISSN 2432-2725.  doi: 10.1007/s42001-019-00035-x.  URL https://doi.org/10.1007/s42001-019-00035-x.
  * Stede and Patz [2021] Manfred Stede and Ronny Patz.  The Climate Change Debate and Natural Language Processing.  In Anjalie Field, Shrimai Prabhumoye, Maarten Sap, Zhijing Jin, Jieyu Zhao, and Chris Brockett, editors, _Proceedings of the 1st Workshop on NLP for Positive Impact_ , pages 8–18, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.nlp4posimpact-1.2.  URL https://aclanthology.org/2021.nlp4posimpact-1.2.
  * Sun et al. [2019] Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang.  How to fine-tune bert for text classification?  In _Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18_ , pages 194–206. Springer, 2019.
  * Tay et al. [2021a] Yi Tay, Mostafa Dehghani, Jai Prakash Gupta, Vamsi Aribandi, Dara Bahri, Zhen Qin, and Donald Metzler.  Are pretrained convolutions better than pretrained transformers?  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4349–4359, Online, August 2021a. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.335.  URL https://aclanthology.org/2021.acl-long.335.
  * Tay et al. [2021b] Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler.  Charformer: Fast character transformers via gradient-based subword tokenization.  In _International Conference on Learning Representations_ , 2021b.
  * Terechshenko et al. [2020] Zhanna Terechshenko, Fridolin Linder, Vishakh Padmakumar, Michael Liu, Jonathan Nagler, Joshua A. Tucker, and Richard Bonneau.  A Comparison of Methods in Political Science Text Classification: Transfer Learning Language Models for Politics.  _SSRN_ , October 2020.  doi: 10.2139/ssrn.3724644.  URL https://papers.ssrn.com/abstract=3724644.
  * Treviso et al. [2023] Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, André F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, and Roy Schwartz.  Efficient Methods for Natural Language Processing: A Survey.  _Transactions of the Association for Computational Linguistics_ , 11:826–860, July 2023.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00577.  URL https://doi.org/10.1162/tacl_a_00577.
  * Veldanda et al. [2023] Akshaj Kumar Veldanda, Fabian Grob, Shailja Thakur, Hammond Pearce, Benjamin Tan, Ramesh Karri, and Siddharth Garg.  Are emily and greg still more employable than lakisha and jamal? investigating algorithmic hiring bias in the era of chatgpt.  _arXiv preprint arXiv:2310.05135_ , 2023.
  * Waggoner [2021] Philip D. Waggoner.  _Unsupervised Machine Learning for Clustering in Political and Social Research_.  Elements in Quantitative and Computational Methods for the Social Sciences. Cambridge University Press, 2021.
  * Wang et al. [2019] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , page 3266–3280, Red Hook, NY, USA, 2019. Curran Associates Inc.
  * Wankmüller [2022] Sandra Wankmüller.  Drawing Causal Inferences About Performance Effects in NLP.  _ArXiv Computer Science_ , September 2022.  doi: 10.48550/arXiv.2209.06790.  URL http://arxiv.org/abs/2209.06790.  arXiv:2209.06790 [cs].
  * Watanabe and Baturo [2023] Kohei Watanabe and Alexander Baturo.  Seeded sequential lda: A semi-supervised algorithm for topic-specific analysis of sentences.  _Social Science Computer Review_ , 0:08944393231178605, 2023.
  * Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  _CoRR_ , abs/2109.01652, 2021.  URL https://arxiv.org/abs/2109.01652.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Weiss et al. [2016] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang.  A survey of transfer learning.  _Journal of Big data_ , 3(1):1–40, 2016.
  * Wiedemann [2013] Gregor Wiedemann.  Opening up to Big Data: Computer-Assisted Analysis of Textual Data in Social Sciences.  _Historical Social Research / Historische Sozialforschung_ , 38(4 (146)):332–357, 2013.  ISSN 0172-6404.  URL https://www.jstor.org/stable/24142701.  Publisher: GESIS - Leibniz-Institute for the Social Sciences, Center for Historical Social Research.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-demos.6.  URL https://aclanthology.org/2020.emnlp-demos.6.
  * Wulczyn et al. [2017] Ellery Wulczyn, Nithum Thain, and Lucas Dixon.  Ex Machina: Personal Attacks Seen at Scale.  In _Proceedings of the 26th International Conference on World Wide Web_ , WWW ’17, pages 1391–1399, Republic and Canton of Geneva, CHE, April 2017. International World Wide Web Conferences Steering Committee.  ISBN 978-1-4503-4913-0.  doi: 10.1145/3038912.3052591.  URL https://doi.org/10.1145/3038912.3052591.
  * Xu et al. [2021] Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan.  Bot-Adversarial Dialogue for Safe Conversational Agents.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2950–2968, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.235.  URL https://aclanthology.org/2021.naacl-main.235.
  * Yu et al. [2021] Tiezheng Yu, Zihan Liu, and Pascale Fung.  AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5892–5904, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.471.  URL https://aclanthology.org/2021.naacl-main.471.
  * Zheng and Jin [2020] Wanwan Zheng and Mingzhe Jin.  The effects of class imbalance and training data size on classifier learning: an empirical study.  _SN Computer Science_ , 1:1–13, 2020.
  * Zhuang et al. [2020] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  _Proceedings of the IEEE_ , 109(1):43–76, 2020.

##  Appendix A Appendix

To keep the clarity and focus of the main text, we include detailed findings,
supplementary tables, and additional analyses in the present Appendix.

###  A.1 Binary abuse classification task

####  A.1.1 Transfer learning

In Figure 5 we present the learning curves for transfer learning experiments
under balanced train and balanced test (5 a), unbalanced train and unbalanced
test (5 b), and balanced train and unbalanced test (5 c) using DeBERTa-v3 and
DistilBERT. Both models yield similar performance across labelling budgets,
while a small fluctuation is observed at 256 training points. With 512
training points models obtain (close to) best results in terms of F1 score,
suggesting that 512 can be the amount of data required for reaching optimal
performance using transfer learning. When trained on balanced data, both
models show a striking jump in performance in the early stage of training with
128 training training points.

Figure 5: Binary abuse classification results of transfer learning varied by
model.

####  A.1.2 Prompt engineering

In Figure 6 we present the learning curves for prompt engineering experiments
under balanced train and balanced test (6 a), unbalanced train and unbalanced
test (6 b), and balanced train and unbalanced test (6 c) using GPT-2 and
DistilBERT. The results are based on the prompt ‘Does this text contain
abuse?’. Generally, GPT-2 obtains better results than DistilBert, which can be
due to model structures and pre-training objectives. Similar to transfer
learning, models trained under balanced scenarios demonstrate a faster
learning curve at distinguishing between classes in comparison with unbalanced
settings. Best results are also obtained at 512 training points. In fact, the
classification performance of both models drops at the maximum labelling
budgets under balanced training scenarios, which shows that more data is not
always beneficial for model learning. This emphasizes the advantages of class
balance outweigh data size. As reported in previous work [Islam et al., 2017],
high variance across seeds is of particular interest and highlights the need
for searching for the best seed.

Figure 6: Binary abuse classification results of prompt engineering varied by
model.

To investigate the impact of different prompts on classification, in Figure 7
we plot the Macro F1 for each of the three prompts under under balanced train
and balanced test (7 a), unbalanced train and unbalanced test (7 b), and
balanced train and unbalanced test (7 c). The results shown are based on
Distilbert. Model performance expectedly varies across prompts while
demonstrating similar learning curves over labelling budgets. The confidence
intervals of the learning curves of each run overlap, which suggests that
divergences across seeds can be more important than the potentials of prompt
types. This finding is similar to the work done by Le Scao and Rush [2021].
Best results achieved are with ‘Does this text contain abuse?’. The reasons
that led to the differences in prompts are unclear. Future work can explore
prompt types to improve the classification by changing, for instance, the
syntactical, lexical and semantic aspects of prompts.

Figure 7: Binary abuse classification results of prompt engineering varied by
template.

###  A.2 Binary movie sentiment classification task

Figure 8 presents the Macro F1 scores of transfer learning for the binary
movie sentiment classification task trained with DeBERTa-v3 and DistilBERT.
Figure 9 presents the Macro F1 scores of prompt engineering for the binary
movie sentiment classification task trained with GPT-2 and DistilBERT, based
on the prompt ‘Does this text contain negative sentiment?’. Similar to binary
abuse classification task, transfer learning obtains the same learning pattern
over labelling budgets, regardless of models. The performance of both models
peaks at 128 training points and does not improve with more data.

In Figure 10 we plot the Macro F1 scores for prompt engineering experiments
with respect to the three different prompts used. Noticeably, high variance is
seen in prompts at different training points. The prompt ‘Does this text
contain negative sentiment?’ shows high variance at small training, while ‘Is
this text negative?’ at large training. This confirms that the performance of
prompt engineering is dependent on prompt selection.

In Table 3 we present the results obtained by testing GPT-3, 3.5, and 4 in our
TMDB dataset.

Figure 8: Binary movie sentiment classification results of transfer learning varied by model. Figure 9: Binary movie sentiment classification results of prompt engineering varied by model. Figure 10: Binary movie sentiment classification results of prompt engineering varied by template. Model | Number of points |  Prompt | Macro F1 score
---|---|---|---
GPT-3 | 855 |  Using one word, does the movie review contain negative sentiment, Yes or No? | 0.67
GPT-3 | 855 |  Using one word, classify the sentiment of the movie review using ’Positive’ or ’Negative’. | 0.85
GPT-3 | 854 |  You are a researcher who needs to classify movie reviews as containing negative sentiment or not containing negative sentiment. Using one word, does the movie review contain negative sentiment, Yes or No? | 0.11
GPT-3.5 | 852 |  Using one word, does the movie review contain negative sentiment, Yes or No? | 0.92
GPT-3.5 | 843 |  Using one word, classify the sentiment of the movie review using ’Positive’ or ’Negative’. | 0.90
GPT-3.5 | 851 |  You are a researcher who needs to classify movie reviews as containing negative sentiment or not containing negative sentiment. Using one word, does the movie review contain negative sentiment, Yes or No? | 0.90
GPT-4 | 855 |  Using one word, does the movie review contain negative sentiment, Yes or No? | 0.93
GPT-4 | 855 |  Using one word, classify the sentiment of the movie review using ’Positive’ or ’Negative’. | 0.94
GPT-4 | 855 |  You are a researcher who needs to classify movie reviews as containing negative sentiment or not containing negative sentiment. Using one word, does the movie review contain negative sentiment, Yes or No? | 0.88
Table 3: Results of zero-shot testing the GPT-3, 3.5 and 4 models on the
collected TMDB dataset to label positive and negative movie reviews.

###  A.3 Labelling Functions

In Table A.3 we present a list of all the Labelling Functions used for both
classification tasks.

Task | Type | Label |  Explanation | Coverage | Overalap
---|---|---|---|---|---
Binary abuse | Keywords | ABUSE |  If any of the words considered abusive from a given list of keywords is found in the text, label the text as abusive. | 0.36 | 0.36
Binary abuse | Heuristic | ABUSE |  If the length of the text is above 9500 characters (limit of 10 000 characters), label the text as abusive. | 0.00 | 0.00
Binary abuse | NLP | ABUSE |  If the polarity of the text is below -0.25, label the text as abusive. | 0.07 | 0.07
Binary abuse | NLP | NOT_ABUSE |  If the polarity of the text is equal or more than 0.1, label the text as non abusive. | 0.41 | 0.41
Binary abuse | Annotators | Both |  Take the annotators’ data and use it as labelling functions: given an annotator $i$ labelling a set of texts, the annotator-function labels the previously annotated texts and abstains with all non-annotated ones. | 0.02 | 0.02
Binary movie sentiment | Keywords | Positive |  If any of the words considered related to positive sentiments is found from a given list of keywords, label the text as positive. | 0.90 | 0.90
Binary movie sentiment | Keywords | Negative |  If any of the words considered related to negative sentiments is found from a given list of keywords, label the text as negative. | 0.62 | 0.62
Binary movie sentiment | Regex | Positive |  If any expression related to the acting, the direction, the edition, the sound or in general to the film or the movie contains a positive keyword, then label the text as positive. | 0.25 | 0.25
Binary movie sentiment | Regex | Negative |  If any expression related to the acting, the direction, the edition, the sound or in general to the film or the movie contains a negative keyword, then label the text as negative. | 0.03 | 0.03
Binary movie sentiment | NLP | Negative |  If the polarity of the text is below -0.05, label it as negative. | 0.03 | 0.03
Binary movie sentiment | NLP | Negative |  If the polarity of the text is above 0, label it as positive. | 0.95 | 0.95
Binary movie sentiment | NLP | Positive |  If the subjectivity of the text is below 0.3, label it as positive. | 0.01 | 0.01
A description of the labelling functions used for both classification tasks. |  |  |  |  |

###  A.4 Zero-shot Prompt Engineering

In Table A.4 we present the results of the GPT-3.5-turbo for each of the
prompts used. Some examples of non-response generated by GPT-3.5-turbo can be
seen in 4.

Task |  Prompt | GPT-3 | GPT-3.5 | GPT-4 |  N Non-response
---|---|---|---|---|---
Binary abuse |  Using one word, does the internet comment contain toxic language, Yes or No? | 0.721 | 0.733 | 0.781 |  21
Binary abuse |  Using one word, is this internet comment using toxic language, Yes or No? | 0.667 | 0.684 | 0.797 |  13
Binary abuse |  You are a researcher who needs to classify comments on the internet as containing abusive language or not containing abusive language. Using one word, does the internet comment contain abusive language, Yes or No? | 0.713 | 0.802 | 0.813 |  19
Binary movie sentiment |  Using one word, does the movie review contain negative sentiment, Yes or No? | 0.952 | 0.923 | 0.961 |  2
Binary movie sentiment |  Using one word, classify the sentiment of the movie review using ’Positive’ or ’Negative’. | 0.941 | 0.945 | 0.964 |  8
Binary movie sentiment |  You are a researcher who needs to classify movie reviews as containing negative sentiment or not containing negative sentiment. Using one word, does the movie review contain negative sentiment, Yes or No? | 0.876 | 0.918 | 0.939 |  7
Results of the zero-shot prompts across models and prompts in terms of macro F1 score. |  |  |  |  |

Task |  Non-response
---|---
Binary abuse |  Cannot determine if the internet comment contains abusive language or not based on the given text
Binary abuse |  I cannot determine if this internet comment contains abusive language or not as it is not a complete sentence
Binary abuse |  The internet comment does not contain a clear indication of abusive language
Binary abuse |  The given internet comment does not contain any language
Binary abuse |  I cannot determine if the internet comment contains abusive language or not as the comment is incomplete and ends abruptly
Binary movie sentiment |  Neutral
Binary movie sentiment |  Mixed
Table 4: Non-responses generated by GPT-3.5-turbo