# Machine Generated Text: A Comprehensive Survey of Threat Models and
Detection Methods

Evan Crothers<EMAIL_ADDRESS>0000-0001-6177-0525 ,  Nathalie Japkowicz
<EMAIL_ADDRESS>0000-0003-1176-1617 and  Herna Viktor
<EMAIL_ADDRESS>0000-0003-1914-5077

(2022)

###### Abstract.

Advances in natural language generation (NLG) have resulted in machine
generated text that is increasingly difficult to distinguish from human
authored text. Powerful open-source models are freely available, and user-
friendly tools democratizing access to generative models are proliferating.
The great potential of state-of-the-art NLG systems is tempered by the
multitude of avenues for abuse. Detection of machine generated text is a key
countermeasure for reducing abuse of NLG models, with significant technical
challenges and numerous open problems. We provide a survey that includes both
1) an extensive analysis of threat models posed by contemporary NLG systems,
and 2) the most complete review of machine generated text detection methods to
date. This survey places machine generated text within its cybersecurity and
social context, and provides strong guidance for future work addressing the
most critical threat models, and ensuring detection systems themselves
demonstrate trustworthiness through fairness, robustness, and accountability.

machine learning, artificial intelligence, neural networks, trustworthy AI,
natural language generation, machine generated text, transformer, text
generation, threat modeling, cybersecurity, disinformation

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††journal:
CSUR††journalvolume: X††journalnumber: X††article: 1††publicationmonth:
10††ccs: Computing methodologies Machine learning approaches††ccs: Security
and privacy Human and societal aspects of security and privacy††ccs: Computing
methodologies Neural networks††ccs: Computing methodologies Natural language
generation

##  1\. Introduction

###  1.1. Risks of Machine Generated Text

Recent natural language generation (NLG) models have taken a significant step
forward in diversity, control, and quality of machine generated text. The
ability to create unique, manipulable, human-like text with unprecedented
speed and efficiency presents additional technical challenges for the
detection of abuses of NLG models, such as phishing (Baki et al., 2017;
Giaretta and Dragoni, 2020), disinformation (Shu et al., 2020; Stiff and
Johansson, 2022; Zellers et al., 2019), fraudulent product reviews (Adelani et
al., 2020; Stiff and Johansson, 2022), academic dishonesty (Hargrave, 2005;
Dehouche, 2021), and toxic spam (Kurenkov, 2022). Addressing the risk of abuse
is vital to maximize the potential benefit of NLG technology, while minimizing
harms — a key principle of trustworthy AI (European Commission and
Directorate-General for Communications Networks, Content and Technology,
2019).

The overwhelming majority of contemporary state-of-the-art NLG models are
neural language models (NLMs) based on the Transformer architecture (Vaswani
et al., 2017). Significant concerns surrounding the threats posed by
generative Transformer models are nearly as old as the models themselves: the
release of the 1.5B parameter GPT-2 architecture was delayed for nine months
due to fears of abuse (Radford et al., 2019a). Access to GPT-3 remains only
permitted via a carefully controlled API (Brown et al., 2020). Such measures
demonstrably manifest only in delays to open availability of models. Only four
months after the release of GPT-2, Grover — a 1.5B parameter model based on
the GPT-2 architecture — was made publicly available (Zellers et al., 2019).
The release of Grover not only foreshadowed the speed with which private
models would be replicated, but also represented a limited threat model in
itself: Grover was specifically designed to both produce and detect neural
fake news. Grover’s primary author provided a reasoned justification for the
model release, and called for an improved set of community norms for the
release of potentially dangerous research prototypes (Zellers, 2019).

Such norms have been slow to develop (Liang et al., 2022), and wide scale
democratization of access to increasingly large scale natural language
generation models has continued. Open-source initiative EleutherAI has
produced open-source generative Transformer models with large numbers of
parameters, including the 6B parameter GPT-J (Wang and Komatsuzaki, 2021), and
20B parameter GPT-NeoX (Black et al., 2022). Even truly massive models are now
available open-source — the BigScience Large Open-science Open-access
Multilingual Language Model (BLOOM) is an open-source multilingual model, and
at 176B parameters, is larger than GPT-3 (Scao et al., 2022). Yandex
(Khrushchev et al., 2022), Meta AI (Zhang et al., 2022), and Huawei (Zeng et
al., 2021) have all open-sourced models with over 100B parameters.

Real life examples of how generative Transformer language models may be abused
are beginning to emerge. A controversy in the AI research community resulted
from the publicized development of a GPT-J model trained on the 4chan politics
message board /pol/. This model was subsequently deployed to produce a large
number of posts on the board from which its training data came, including
posts containing objectionable content (Kurenkov, 2022). At it’s peak, the
model represented roughly 10% of all activity on the board in a 24 hour period
(Kilcher, 2022). The response to the deployment of this model included a
signed condemnation from 360 signatories across the AI community including
scientific directors, CEOs, and professors (Liang and Reich, 2022). A similar
project targeted a federal public comment website with GPT-2 text until the
submitted comments made up half of all comments, demonstrating the extent of
existing vulnerabilities (Weiss, 2019).

Controversy around any individual publicized NLG model belies the more
fundamental concern — for years now, any person with access to adequate
hardware and open-source training scripts could train or fine-tune large
generative Transformers for any purpose they choose, be it pop song lyrics,
mass disinformation, or toxic spam. Malicious individuals in the process of
training a generative language model need not draw attention to their models
via public release, and currently face limited risk of discovery. As NLG
capabilities grow and access barriers evaporate, we are inevitably already
quietly climbing the adoption curve for this technology to be widely abused by
cybercriminals, disinformation agencies, scam artists, and other threat
actors.

Access to these models is increasingly not limited to sophisticated threat
actors who are able to fine-tune them. Jasper, a tool marketed as an AI
writing assistant, uses GPT-3 to write sections of content alongside a human’s
guidance (Jasper AI, 2022). This includes generating content for blogs and
websites, which Jasper can efficiently produce in large volumes. Another
website offers an endless supply of GPT-3 authored cover letters (Open Cover
Letter, 2022). Tools such as Jasper allow those with little technical
knowledge someone to seed the model with prompt text, specify keywords to
include, and indicate a specific tone of voice. Using publicly available open-
source models, a nearly identical system could easily be created to generate
endless streams of targeted disinformation, ready to be loaded into existing
grey-market account automation tools for popular social media websites.

NLG models have the potential to have an immense and transformative positive
impact on human society. A staggering 1 in 3 internet users aged 16 to 64 have
used an online translation tool in the last week, a figure representing over 1
billion people (Kemp, 2022a). Text summarization can create understandable
summaries of complex legal text (Kanapala et al., 2019) or medical records
(Wang et al., 2021). NLG models can give a voice to machine systems, changing
the way that humans interact with them (Kim et al., 2019). The same
Transformer architecture used heavily for NLG can also be used for generating
pictures from image descriptions (Ramesh et al., 2021), producing functional
code from a natural language summary (Chen et al., 2021), and serves as the
basis for the current vanguard of generalist agents (Reed et al., 2022). While
future research in NLG may bring further wonders, alongside these
opportunities is the corresponding certitude that the same technology will be
used by bad actors to nefarious ends. Predicting how abuses are likely to
unfold, and understanding the best defenses against them, is essential for
allowing humanity to reap the positive benefits of this technology while
minimizing potential harms. We must walk a cautious path through the age of
the silicon wordsmith.

###  1.2. Survey Overview

Since the release of GPT-2 (Radford et al., 2019a) and subsequent explosion of
high-quality Transformer-based NLG models, there has been only one general
survey on detection of machine generated text (Jawahar et al., 2020). The
scope of this previous survey is constrained to detection methods specifically
targeting the several generative Transformer models that had been released at
the time. Prior to this, a systematic review of machine generated text
predating the Transformer architecture covered approaches to detecting
previous NLG approaches, such as Markov chains (Beresneva, 2016). Our survey
differs from previous work in three major ways.

First, our survey of machine generated text detection is more comprehensive
than previous work. We consider literature on feature-based detection of
machine-generated text omitted from prior review (Nguyen-Son et al., 2017;
Fröhling and Zubiaga, 2021; Lavoie and Krishnamoorthy, 2010). Such approaches
are a worthy inclusion, as feature-based approaches still apply against
contemporary NLG models (Fröhling and Zubiaga, 2021; Crothers et al., 2022;
Kowalczyk et al., 2022). Inclusion of such features may provide benefits in
practice, such as improved robustness against adversarial attacks targeting
neural networks (Crothers et al., 2022), or enhanced explainability (Kowalczyk
et al., 2022). Additionally, as research on both NLG and detection has
continued to rapidly advance in the years following the previous survey, we
must now cover a wider range of generative models and defensive research.

Second, in addition to a comprehensive review of detection methods against
contemporary models, this survey provides an in-depth analysis of the risks
posed by NLG models via the process of threat modeling (i.e., identifying
potential adversaries, their capabilities and objectives) (Bitton et al.,
2021). The result of our threat modeling process is a series of threat models
that describe scenarios where machine generated text may be abused, the likely
methodology of attackers, and existing research related to each threat. To
date, there has yet to be any survey of machine generated text detection with
a focus on the risks presented by machine generated text. Consideration of
threat models is vital to set the groundwork for trustworthy development of
NLG technology, encourage early development of defensive measures, and
minimize potential harms.

Third, guided by the EU Ethics Guidelines for Trustworthy AI (European
Commission and Directorate-General for Communications Networks, Content and
Technology, 2019) and research community efforts (Kaur et al., 2022), we
present our survey with sociotechnical and human-centric considerations
integrated throughout, focusing not only on NLG systems and machine text
detection technologies, but on the humans who will be exposed to both text
generation and detection systems in daily life. The goal of trustworthy AI is
to ensure that AI systems are developed in ways that are lawful, ethical, and
robust both from a technical and social perspective. Abuse of NLG models
threatens all three of these areas, representing safety risks to those who may
be targeted by NLG-enabled attacks, threats to the integrity of online social
spaces, and challenges to the resilience of the technical and social systems
that comprise modern society. Machine text detection is part of protecting
against abuse of NLG models, enhancing the robustness and safety of NLG
development. Critically, our survey also includes insight into ensuring
defensive machine text detection systems themselves are transparent, fair, and
accountable.

To summarize, the major contributions of this work are as follows:

  * •

The most complete survey of machine generated text detection to date,
including previously omitted feature-based work and findings from recent
contemporary research.

  * •

The first detailed review of the threat models enabled by machine generated
text, at a critical juncture where NLG models and tools are rapidly improving
and proliferating.

  * •

Integration of both topics with a vital practical perspective guided by
Trustworthy AI (TAI) into how machine generated text threat models and
detection systems will impact humanity.

The rest of this survey is organized as follows. We provide definitions and a
brief overview of existing methods for natural language generation in Section
2. In Section 3 we explore threat models related to abuse of machine generated
text, including impacts on trust. We provide a comprehensive survey of
literature related to detection of machine generated text in Section 4. In
Section 5 we summarize open problems and ongoing trends to guide the direction
of future work. Finally, in Section 6 we present our final conclusions. While
this work discusses machine generated text extensively, including models
designed for generating scientific papers, no such models were utilized in
authorship of this work.

##  2\. Machine Generated Text

Before reviewing threat models and detection methodologies for machine
generated text, it is helpful to briefly provide a formal definition of
machine generated text, and a condensed overview of natural language
generation (NLG) models. We recommend further reading of dedicated surveys on
natural language generation for greater insight into the wide breadth of NLG
models and applications (Dong et al., 2022; Gatt and Krahmer, 2018; Perera and
Nand, 2017; Santhanam and Shaikh, 2019; Li et al., 2021; Reiter and Dale,
2002).

###  2.1. Definition and Scope

In this survey, we use a broad definition of the term “machine generated text”
which we believe includes all relevant research in the field:

> “Machine generated text” is natural language text that is produced,
> modified, or extended by a machine.

We focus our definition of machine generated text on natural language — i.e.,
text written in human languages that are “acquired naturally (in [an]
operationally defined sense) in association with speech” (Lyons, 1991) — and
exclude non-natural language — i.e., logical languages, programming languages,
etc. Exclusion of non-natural language aligns with other work in the field:
the term “text generation” is currently considered synonymous with “natural
language generation” (Li et al., 2021; Zhang and Sun, 2009). We anticipate
that “text generation” may be repurposed in future research as an umbrella
term that includes non-natural language text as well. This would accommodate
common considerations between NLG models and contemporary code generation
models, such as Codex (Chen et al., 2021) and CodeGenX (Manjaramkar, 2021). As
an example, attacks against StackOverflow or GitHub may include both NLG as
well as vulnerable code generation. Code generation models can also be used to
complete programming assignments without triggering common plagiarism
detection tools (Biderman and Raff, 2022).

Our definition of machine generated text is intentionally broad, and covers a
large number of possible use cases and associated threat models, which will be
discussed in Section 3. In the interests of managing a survey scope that
already spans a wide range of literature and broad sociotechnical context,
text generation by means of text adversarial attack will not be considered. In
the majority of cases, the production of new text is not the primary goal of a
text adversarial attack, and text adversarial attacks and threat models are
already covered by surveys in adversarial attack literature (Chakraborty et
al., 2018; Wang et al., 2019; Huq and Pervin, 2020). We will nevertheless
discuss the role machine generated text plays in adversarial contexts in
Section 3, as well as adversarial robustness of detection models in Section 5.

Note that this analysis focuses on threat models where a threat actor
leverages machine generated text as part of an attack — typically scenarios
where the attacker is attempting to pass machine text as human, and where
detection of machine generated text may be useful defensively. We are not
discussing attacks against NLG models themselves, unless they leverage NLG as
part of the attack. For example, a white-box training data extraction attack
targeting the weights of a commercial speech-to-text model would not be
included in our analysis, but using an NLG model to produce data for poisoning
that model’s training dataset would.

With this definition of machine generated text in mind, and with an
understanding of the scope of research under consideration, we proceed to a
brief overview of natural language generation.

###  2.2. Natural Language Generation

Using a computer to produce human-like text is well-established in the history
of computing. Turing’s proposed “imitation game” (Turing, 1950) in 1950
considered the question of machine intelligence based on the ability of a
machine to conduct human-like conversation over a text channel, for which the
first widely-published method dates back to 1966 with the ELIZA chatbot
(Weizenbaum, 1966). Given the large volume of NLG research over the past 55
years, we provide only a high-level taxonomy of major NLG tasks and approaches
as groundwork for our analysis of threat models and detection methodologies,
and leave detailed discussion to aforementioned dedicated surveys.

####  2.2.1. Natural Language Generation Tasks

Recall from §1.1 that there are a wide variety of applications for natural
language generation. Leveraging previous surveys (Li et al., 2021; Dong et
al., 2022; Jin et al., 2022), we provide a summary of major tasks in the NLG
domain, with examples of models that have been used for each task in Table 1.
Note that many of the models listed are multi-purpose and can be trained on
multiple downstream NLG tasks.

Table 1. Inputs, tasks, and examples of natural language generation Input | Task | Examples
---|---|---
None / Random noise | Unconditional text generation |  GPT-2 (Radford et al., 2019a), GPT-3 (Brown et al., 2020) (no prompt)
Text sequence | Conditional text generation |  GPT-2 (Radford et al., 2019a), GPT-3 (Brown et al., 2020) (with prompt)
Machine translation |  FairSeq (Ott et al., 2019), T5 (Raffel et al., 2020)
Text style transfer |  Style dictionary (Sheikha and Inkpen, 2011), GST (Sudhakar et al., 2019)
Text summarization |  BART-RXF (Aghajanyan et al., 2021), Word and Phrase Freq. (Luhn, 1958)
| Question answering |  FairSeq (Ott et al., 2019), T5 (Raffel et al., 2020)
| Dialogue system |  DG-AIRL (Li et al., 2018), DIALOGPT (Zhang et al., 2020), BlenderBot3 (Shuster et al., 2022)
Discrete attributes | Attribute-based generation |  MTA-LSTM (Feng et al., 2018), PPLM (Dathathri et al., 2020), CTRL (Keskar et al., 2019)
Structured data | Data-to-text generation |  DATATUNER (Harkous et al., 2020), Control prefixes (T5) (Clive et al., 2021)
Multimedia | Image captioning |  GIT (Wang et al., 2022b), ETA (Li et al., 2019)
Video captioning |  MMS (Li et al., 2017), YouTube2Text (Guadarrama et al., 2013)
Speech recognition |  ARSG (Chorowski et al., 2015), wav2vec-U (Baevski et al., 2021)

The summary in Table 1 is not exhaustive, and in reality, a mutually exclusive
delineation between input types does not exist. Combinations of different
input types are possible. As an example, CTRL takes both a discrete control
code attribute and conditional text prompt in generation (Keskar et al.,
2019). Question-answering systems may be able to answer questions about
images, such as Unified VLP (Zhou et al., 2020) and TAG (Wang et al., 2022a).
Note that we consider a “topic” as an attribute in this overview, and so
include “topic-to-text generation” under the broader umbrella of “attribute-
based generation”, including work such as topic-to-essay generation (Feng et
al., 2018).

Given the strong generative capabilities of Transformer language models, and
the corresponding increased risk in threat models, such models rightly warrant
particular emphasis in review. However, as mentioned in Section 1.2,
consideration of the broader field of natural language generation and previous
detection research is important as detection techniques that apply against
pre-Transformer models have been shown to be useful in detection of modern
neural text models, and diverse approaches may offer increased adversarial
robustness (Crothers et al., 2022) or better explainability (Kowalczyk et al.,
2022).

###  2.3. Natural Language Generation Approaches

There are a wide range of model architectures and algorithmic approaches to
natural language generation. We categorize these approaches broadly into
neural and non-neural methods, and then further break them down into more
specific categories. A diagram of our simplified breakdown can be found in
Figure 1. As previously mentioned, NLG encompasses a large variety of tasks
and research areas, with this brief section serving as context for
understanding machine generated text threat models and detection methods.

Figure 1. Taxonomy of major NLG approaches

for tree= parent anchor=south, child anchor=north, fit=band, [NLG Approaches
[Neural, tier=neurality [VAE, tier=neural] [GAN, tier=neural] [RL,
tier=neural] [IRL, tier=neural] [RNN, tier=neural [LSTM, tier=rnn] [GRU,
tier=rnn] ] [Transformer, tier=neural [GPT-*, tier=transformer] [Grover,
tier=transformer] [BART, tier=transformer] [CTRL, tier=transformer] ] ] [Non-
Neural, tier=neurality [Rule-based, tier=nonneural] [Statistical + Rule-based,
tier=nonneural [K-means, tier=statistical] [HMM, tier=statistical] [SVM,
tier=statistical] ] [MDP RL] ] ]

####  2.3.1. Non-Neural Models

Predating the popularization of neural approaches in the NLG domain, a range
of systems were used to accomplish NLG tasks. These early approaches can
broadly be summarized as “rule-based”, though there existed variety in terms
of processes, pipelines, and targets tasks. A review of rule-based systems can
be found in Reiter and Dale’s book on the subject (Reiter and Dale, 2002).

An alternative approach to purely rule-based approaches is to use an existing
natural language corpus to generate rules for components of an NLG system,
such as content selection (Duboue and McKeown, 2003; Langkilde and Knight,
1998) or template generation (Kondadadi et al., 2013). These statistical
approaches are meant to be more adaptable to different domains than strictly
rule-based systems. While many different statistical models have been
integrated with NLG systems in various ways, Hidden Markov Models (HMM) (Baum
and Petrie, 1966) feature prominently in past work. More recent non-neural
research has used reinforcement learning (Janarthanam and Lemon, 2009) and
hierarchical reinforcement learning (Dethlefs and Cuayáhuitl, 2010) of Markov
Decision Process (MDP) agents to learn optimal text generation policies.

####  2.3.2. Non-Transformer Neural Methods

Natural language generation using neural networks was demonstrated to be
highly effective using recurrent neural networks (RNN) (Kalchbrenner and
Blunsom, 2013; Mikolov et al., 2012; Berglund et al., 2015), including long
short-term memory (LSTM) architectures (Merity et al., 2017) and gated
recurrent units (GRUs) (Pawade et al., 2018). However, RNN and LSTM
architectures had to contend with the vanishing gradient problem, to which the
multi-head attention mechanism of the Transformer architecture is more
resilient (Topal et al., 2021). Generative adversarial networks (GANs)
(Goodfellow et al., 2020) — commonly used to generate continuous data (such as
images) — can also be adapted to a discrete context for natural language
generation (Lin et al., 2017; Yu et al., 2017).

Deep reinforcement learning (RL) has been used with neural networks to learn
policy gradient methods that reward text characteristics associated with high-
quality text generation (Li et al., 2016). A related area of work is the usage
of inverse reinforcement learning (IRL), which has included work that aims to
address reward sparsity and mode collapse problems in GAN-based text
generation by learning an optimal reward function and generation policy (Li et
al., 2018; Shi et al., 2018).

####  2.3.3. Transformer

The multi-head attention architecture of Transformer language models (Vaswani
et al., 2017) currently represents the state-of-the-art in natural language
generation across natural language tasks. Among Transformer models, the
unidirectional GPT-2 (Radford et al., 2019a) and GPT-3 (Brown et al., 2020)
models are the most studied in the field of machine generated text detection
due to their groundbreaking performance on unconditional and conditional text
generation — though like many other Transformer models, these architectures
can be used for other NLG tasks as well.

In addition to GPT-2 and GPT-3, also notable in machine generated text
detection are related autoregressive language models using a similar
architecture, with variations in sampling procedures or training datasets.
Such models include Grover (Zellers et al., 2019) (a GPT-2 style model trained
on a news dataset and using nucleus sampling instead of top-$k$ sampling),
GPT-J (Wang and Komatsuzaki, 2021) (a 6-billion parameter autoregressive
language model trained on The Pile (Gao et al., 2021)), and GPT-NeoX-20B
(Black et al., 2022) (a 20-billion parameter model similar to GPT-3, also
trained on The Pile (Gao et al., 2021)).

Unidirectional Transformer language models generate text by performing
unsupervised distribution estimation to predict the next token based on
previous tokens. To do this, the model is trained on an existing set of
variable-length example texts $(x_{1},x_{2},...,x_{n})$ each composed of
symbols $(s_{1},s_{2},...,s_{m})$. These symbols may be characters, or multi-
character tokens obtained through a tokenization process.

The probability of a given text can then be expressed as the conditional
probability of the final token, given each previous token. That is:

(1) $p(x)=\prod_{i=1}^{m}p(s_{m}|s_{1},...,s_{m-1})$

The self-attention mechanism in the Transformer architecture makes it possible
to train neural network architectures that are effectively able to estimate
such probabilities, given a suitable pre-training task. In unidirectional
models such as those in the GPT lineage, a common training task is prediction
of the next token in sequence. To generate text, such models can then receive
a continue an input sequence by sampling from the probability distribution of
all possible next tokens based on previous tokens. An important parameter in
this sampling process is “temperature” $T\in(0,\infty)$, which can be raised
above 1 to increase the likelihood of selecting a less-probable next token —
improving diversity at the potential cost of choosing an unusual token — or
lowered below 1 to bias sampling towards more common tokens.

There are three common decoding strategies used for sampling token
probabilities from contemporary unidirectional generative Transformer models
(Holtzman et al., 2019):

  1. (1)

No truncation $\rightarrow$ Sample from the entire probability distribution.
At $T=1$, this is called “pure sampling”.

  2. (2)

Top-$k$ truncation $\rightarrow$ Sample from the $k$ most probable tokens.

  3. (3)

Nucleus sampling (also known as top-$p$ truncation) $\rightarrow$ Sample from
tokens in the top-$p$ portion of the probability mass, rather than a fixed
number of tokens $k$.

While unidirectional generative models are key fixtures of machine generated
text detection research, other Transformer architectures can be used for NLG
tasks as well. The architecture of BART (Lewis et al., 2020) includes a
bidirectional encoder (similar to BERT (Devlin et al., 2018)), but maintains a
left-to-right decoder for sequential text generation. Other Transformer
architectures such as MASS (Song et al., 2019), T5 (Raffel et al., 2020), and
ULMFiT (Howard and Ruder, 2018) can also be used for NLG tasks.

An important area of ongoing research centers around shaping the output
produced by Transformer models. This can include prompt engineering —
carefully crafting the conditional text input for a language model to continue
(Brown et al., 2020) — or by providing additional discrete attributes that can
be used to influence the generation of the network, such as control code,
topic, or sentiment as in CTRL (Keskar et al., 2019), PPLM (Dathathri et al.,
2020), or GeDi (Krause et al., 2020). Greater control over model output
increases the risks posed by threat models (Brown et al., 2020). As an
example, when generating social media posts as part of an NLG-augmented online
influence campaign, an attacker would benefit from being able to ensure that
generated comments both 1) mention a targeted political opponent, and 2)
demonstrate negative entity sentiment towards the opponent. We will cover such
potential abuses and others in more detail in the next section, which concerns
threat models associated with machine generated text.

##  3\. Threat Models

Machine generated text enables a diverse array of attacks. These attacks may
be performed by threat actors with specific objectives, such as to compromise
a computer system, exploit a target individual for financial gain, or enable
large-scale harassment of specific communities. The EU ethics guidelines for
trustworthy AI emphasize that unintended or dual-use applications of AI
systems should be taken into account, and that steps should be taken to
prevent and mitigate abuse of AI systems to cause harm (European Commission
and Directorate-General for Communications Networks, Content and Technology,
2019). As such, trustworthy AI in the context of NLG, necessitates
understanding the areas where such models may be abused, and how these abuses
may be prevented (either with detection technologies, moderation mechanisms,
government legislation, or platform policies). When discussing attacks, we
discuss not only the direct impact on targets, but also the broader impacts of
both attacks and mitigation measures on trust.

To understand the risks that motivate research on detection of machine
generated text, we draw from existing literature to present a series of threat
models incorporating natural language generation. Threat modeling reflects the
process of thinking like an attacker, identifying vulnerabilities to systems
by identifying potential attackers, their capabilities, and objectives. The
goal of threat modeling is to improve the security of systems by considering
the greatest threats to systems and their users. Many methods of threat
modeling have been developed over the years, which include approaches ranging
from drawing system diagrams, itemized vulnerability checklists, and
performing open-ended brainstorming (UK National Cyber Security Centre, 2022;
Bromander et al., 2016; Kohnfelder and Garg, 1999; UcedaVelez and Morana,
2015). In late 2020, a diverse set of experts formed a threat modeling working
group to produce a high-level set of guidelines related to effective threat
modeling approaches (Braiterman et al., 2020) — we leverage these guidelines
in the open-ended attack-centric modeling approach in this section.

###  3.1. Threat Modeling Fundamentals

As we anticipate a machine learning audience with varying exposure to
cybersecurity topics, before we present threat models related to machine
generated text, it is helpful to first provide an overview of threat modeling,
and characterize the approach taken in this section.

A basic example of a common threat model is “a thief who wants to steal your
money” (Shostack, 2014). We can add detail to this threat model by considering
more specific capabilities and objectives that such an attacker might have.
For example, we may consider “a thief with lockpicks who wants to steal your
TV”, or “a thief who found your banking password in a database dump and wants
to transfer money out of your account”. With these threat models in mind, we
can then propose mitigation strategies, such as “install locks that are
resistant to lockpicking”, or “use multi-factor authentication for online
banking”. Finally, we evaluate whether our mitigation approach is sufficient
to address the threat, and consider what other threat models we might need to
consider. Threat modeling is inherently an iterative process (Shostack, 2014;
Braiterman et al., 2020).

Shostack’s Four Question Frame for Threat Modeling (Shostack, 2014, 2021)
presents best a plain language foundation for threat modeling by posing four
simple questions:

  1. (1)

What are we working on? $\rightarrow$ Identify the system under attack.

  2. (2)

What can go wrong? $\rightarrow$ Determine potential attackers, their
capabilities, and objectives.

  3. (3)

What are we going to do about it? $\rightarrow$ Devise a mitigation strategy.

  4. (4)

Did we do a good job? $\rightarrow$ Review whether the analysis is accurate
and complete.

In these terms, we summarize our threat modeling approach in this section as
follows:

  1. (1)

Identify the system under attack: We provide a broad attack-centric analysis
of machine generated text on society, rather than a system-centric analysis
focusing on vulnerabilities to a specific IT system. As such, we identify
several discrete technological systems, within the broader societal
supersystem.

  2. (2)

Determine potential attackers, their capabilities, and objectives: We consider
threat actors of varying sophistication and motives, but with a common modus
operandi — in all cases, our attacker is an individual or organization
exploiting an NLG model. We characterize the attacker when explaining each
attack.

  3. (3)

Devise a mitigation strategy: After identifying a threat model, we propose
mitigation measures to improve security and reduce risk. Detection of
computer-generated text is often part of the presented mitigation approaches,
but policy changes and human moderation systems can also have a significant
impact.

  4. (4)

Review whether the analysis is accurate and complete: We have given careful
thought to the presented threat models, which are formed from perspectives
gained across industry, academia, and government. However, as threat modeling
is an iterative process that benefits from diverse perspectives (Braiterman et
al., 2020), we greatly encourage further analysis of potential attacks and
mitigation measures in future research.

The remainder of this section comprises our threat model analysis, grouped
according to a breakdown of attacks into four major categories, followed by a
concluding discussion. Within each category we discuss threat models
associated with that category of attack, identifying systems at risk, and
describing possible threat actors, their objectives, and capabilities. For
each attack, we propose mitigations, and then discuss the trust impacts of
both the attack and — crucially — of the proposed mitigations as well. A
taxonomy of the broad categories of attacks using NLG models we discuss can be
found in Figure 2.

While a completely exhaustive list of all possible future malicious
applications of NLG models is not possible, the threats outlined here span a
wide range of tangible dangers at this point in time, representing valuable
areas of future investigation for preemptive ethical defensive research. As
previously mentioned, threat modeling is iterative, and it is hoped that these
threat models should serve as the foundation for future work in improving
security against machine generated text.

Figure 2. Broad taxonomy of threat models enabled by machine generated text

A diagram depicting threat models mentioned in this research, broken down into
broad categories.

###  3.2. Facilitating Malware and Social Engineering

####  3.2.1. Phishing and Scamming

Phishing attacks center on socially engineering a target individual to perform
a desired action. This might be to convince the target to open an unsafe
document that contains an exploit, cause the target to navigate to a fake
banking webpage, encourage them to share sensitive information that can be
used for identity theft, among many other documented methods (Chiew et al.,
2018; Alkhalil et al., 2021). Phishing attacks can target numerous channels,
including email, phone, SMS, or chat applications.

Automated messaging approaches in the early stages of phishing campaigns are
common (Alkhalil et al., 2021). Machine generated text can be a useful tool to
an attacker attempting to scale or better target phishing or scam campaigns.
Rather than provide the same message to all targets, NLG can be used to
generate target-specific text. Research has demonstrated NLG both for scaling
of email masquerade attacks (Baki et al., 2017), and for community-targeted
phishing (Giaretta and Dragoni, 2020). Carefully targeting a phishing attack
(commonly referred to as “spear phishing”), greatly increases the likelihood
of a specific target falling for the attack (Burns et al., 2019). In the cases
of chat messages, NLG models that serve as dialogue agents may be exploited to
exchange messages with the target under a pretext before exploiting them
(Zhang et al., 2020).

Mitigation of NLG-enabled phishing attacks will be similar to established work
on existing phishing attacks, including both automated detection systems, user
reporting, and awareness campaigns (Abu-Nimeh et al., 2007). NLG may present
an increased challenge for existing detection systems in that generated
messages may have unique or highly-varied content — though attackers may be
forced to include specific “payload” content that always needs to be included
(e.g., a phishing email may include a unique shortlink to the same malicious
website, a chatbot may need to socially engineer the answer to the same
security questions). As text content becomes more varied due to more powerful
NLG models, detection of stable payload content may represent a more stable
detection feature. Algorithms for detection of machine generated text are also
likely to be added to existing automated detection approaches.

####  3.2.2. Social Worms

NLG models may be particularly useful for worms that spread through social
media or email contact networks. When an individual has their account
compromised by an exploit, that account may be used to send malicious messages
that propagate the exploit to other users. By using previous messages or
emails between individuals as context to an NLG model, it may be possible to
automatically produce messages that include personal details, mimic a loved
ones writing style, or carry on a short conversation before delivering a
malicious file or link. Given that NLG models are often quite large, the NLG
component of this may need to be run on a separate command-and-control server
and queried from behind a proxy, rather than bundled with the exploit code
itself (unless the pretext of the conversation can be used to convince the
target to download a file).

Mitigation of such attacks could involve platforms adopting formal policies
that users do not use machine generated text in their communications, except
under carefully controlled circumstances. Detection models could then be
leveraged against user communications. While this may be acceptable for public
posts, there exists privacy risks when considering private messages. Detection
models could perhaps be executed as part of the message viewing application on
the receiving device to protect end-to-end privacy of messages. If a user
receives several messages that score highly for machine generated text
detection, a warning may be raised. This approach is not without risks, as the
privacy of direct messages must be protected, and any real or perceived
erosion of privacy will undermine public trust. Other security measures to
protect accounts from unauthorized logins, such as multi-factor
authentication, should continue to be used to protect against account
compromise more broadly.

####  3.2.3. Data Poisoning

Cybercriminals may have an interest in poisoning the training datasets of
machine learning models. This may be to support other attacks (e.g., poisoning
a training dataset for a malware detection algorithm or email spam filter) or
poisoning a given model may be the primary goal (e.g., poisoning the dataset
of an algorithmic trading model so that the attacker can later trigger trades
that financially benefit the attacker). If a threat actor identifies that they
can access the training data of any such algorithm, they may use NLG models to
produce many training examples containing a particular malicious signature
they wish to conceal. Poisoning attacks against neural code-completion
algorithms have been performed by generating samples including a given
vulnerability (Schuster et al., 2021), and GPT-2 has been used in research to
produce fake cyber threat intelligence reports for poisoning cyber-defense
systems (Ranade et al., 2021).

Mitigation of dataset poisoning varies based on the sensitivity of the model
and nature of the training dataset. The first line of defense is basic IT
security best practices that prevent unauthorized modifications to training
datasets. However, in some situations models are trained on publicly available
data, and therefore it is not possible to prevent access to training data. In
these cases, data might be screened prior to inclusion in the training
dataset. This screening can include classifiers — potentially including the
machine generated text detection approaches discussed in Section 4 — or by
other analysis methods, such as cluster-based methods to detect poisoning in
training datasets (Baracaldo et al., 2017). Finally, for sensitive models, it
may also be appropriate to leverage data versioning techniques and audit
logging to capture changes to the data potentially made by a malicious
insider.

####  3.2.4. Impacts of Attacks and Mitigation on Trust

The usage of NLG models to produce compelling, targeted pretexts for a large
scale phishing attacks and social worms is likely to further reduce trust in
text communications, particularly those received from contacts not personally
known. Individuals may become even more suspicious of unsolicited messages —
even seemingly innocuous ones. As even a seemingly good-natured greeting may
be just the first message from a malicious dialogue agent, individuals may
decide it is safer to not reply to such messages, further reducing trust and
social interaction with new individuals in online communities.

NLG-based poisoning attacks against machine learning models will likely have
the greatest trust impact on machine learning practitioners, who may be
required to carefully scrutinize open-source training for poisonous samples.
Where mitigation of poisoning attacks involves limiting access to training
datasets behind auditing and approval processes, such procedures may cause
developers to feel distrusted, and undermine the relationship between these
individuals and the organizations they work with. While the trust impact of
NLG-based data poisoning attacks may be relatively minor among the general
population, a high-profile disturbing attack (e.g., a poisoning attack against
a medical diagnosis model) may cause individuals to lose trust in machine
learning systems more broadly, based on concerns that such models are not be
safe from malicious tampering.

###  3.3. Online Influence Campaigns

An area of particular concern for abuse of machine generated text is
facilitation of online influence campaigns. The objectives of threat actors in
this area may either be political in nature (e.g., disinformation, propaganda,
election interference) or commercial in nature (e.g., product promotion,
smearing competitors, fake reviews). In either case, the goal is to promote a
particular idea or prompt a particular action among the target audience.

Either type of campaign may both leverage or facilitate other threat models,
such as spam, harassment, mass submission of agenda-driven content, phishing,
or malware. The distinction between commercial and political influence
campaigns is useful for better understanding threat actors and threatened
systems in more detail, as well as categorizing existing research.

####  3.3.1. Political Influence Campaigns

Machine generated text as part of political influence campaigns has been
analyzed in previous work (Shu et al., 2020; Stiff and Johansson, 2022;
Zellers et al., 2019). Papers related to the threat of generative language
models on online influence operations may use terminology such as terminology
“fake news” (Zellers et al., 2019) or “disinformation” (Brief, 2021; Stiff and
Johansson, 2022), or “domestic and foreign influence operations” (Brief,
2021).

The threat actor in a political influence campaign represents an entity who
wishes to influence beliefs or prompt action among a target group. These
threat actors might include, as examples:

  * •

A political party hiring a group to post unflattering comments online about
their political adversaries

  * •

A nation-state disseminating fraudulent news reports to mask human rights
abuses

  * •

A nation at war attempting to incite the citizenry of an opposing nation to
overthrow the government

Datasets from past political influence campaigns have been released by
Facebook (Schrage and Ginsberg, 2018), Reddit (Reddit, 2018), and Twitter
(Twitter, 2019), including operations attributed to 22 different countries.
Threat actors in this space can be expected to already have the capability of
running online political influence campaigns using human employees or
contractors, and be familiar with social media automation tools that
facilitate registration and management of fraudulent social media accounts.
Large models with strong few-shot performance can simply be provided examples
of the desired messaging to produce further propaganda. We demonstrate this in
Table 2, using Russian information operations executed on Twitter during the
Syrian civil war to generate additional on-message tweets that promote Syrian
Arab Army (SAA) and Russian forces, while discrediting American involvement.

Table 2. Few-shot generation of information warfare based on Syrian civil war influence operations | GPT-3 Prompt:
---
Human tweets
from Russian
foreign military
intelligence
(GRU) (Twitter, 2019)
| RT @TheArabSource: Confirmed: #US retrains #ISIS and #ISIS-like jihadists in
east #Syria and
---
re-brands them ’democratic forces’ (VIDEO) - E…
RT @Tasnimnews_EN: #Iraq Army Discovers #US Drone in #Daesh Arms Depot near
Fallujah
https://t.co/zYRgEHmlr0 https://t.co/H762UxwlQx
…
ISIS-affiliated band group Wilayat al-Sham recruits new #terrorists in US-
controlled #Rukban camp.
| GPT-3 Response
---
| @ALFaham_ #SAA seized a big amount of #ISIS #ISIL weapons from the convoy in
east
---
#Homs countryside today. https://t.co/k3kzPoLlsd #Syria #Damascus #Jobar #ISIS
#Washington is willing to perform the functions of #MiddleEastern policeman
and play the role
of accessory to #Turkish#aggression against #Syria. https://t.co/HVGXuvRZ
…
#Russia’s #Air_Force has stepped up airstrikes against camps for the
#Islamic_State in the #Terek_Val

As consumption of text content is common online, there are many avenues where
machine generated text might be utilized by a threat actor to improve scaling
and targeting of influence operations. Social media, due to the large volume
of engaged users, is likely to continue be a valuable and vulnerable target
for such campaigns (Reddit, 2018; Twitter, 2019; Schrage and Ginsberg, 2018).
It is probable that research on detection of machine generated text for
political influence campaigns that focuses on fake news does not accurately
reflect the most pressing threat models. Fake news detection research imagines
an adversary using an NLG model to produce news-like disinformation at scale
(Zellers et al., 2019). Instead, we assess that NLG usage by disinformation
agencies for newswriting will be primarily limited to leveraging AI writing
assistants such as Jasper to save time and cost (Jasper AI, 2022), and
translation models to more effectively cross language barriers. We believe
that producing massive volumes of news-like content is a less desirable
machine generated text disinformation scaling approach than social messages
for several reasons:

  * •

Research has demonstrated that individuals are more likely to share an article
than read it (Gabielkov et al., 2016), and that a majority of individuals make
up their minds on news topics by only reading headlines (American Press
Institute, 2014).

  * •

Scaling by number of articles does not multiply effectiveness — a single news
article or handful of news articles can be widely disseminated, reducing the
need to generate large numbers of articles each day.

  * •

Scaling by number of articles requires either manipulating existing platforms
to host them (i.e., layering and information laundering (Meleshevich and
Schafer, 2018)), or procuring domain names and hosting infrastructure,
representing additional cost and effort.

  * •

Human involvement in fake news article authorship allows disinformation threat
actors to better tailor messaging, reduce detection, and more carefully walk
the line of promoting manipulative information without triggering moderation
from social media websites

Instead of using NLG models to generate articles themselves, NLG models are
more likely to be used for disinformation by operating social bots that
distribute links to disinformation articles, promote discussion around
incendiary headlines, and produce large numbers of comments that give the
false impression of a common public consensus. Targeted users need not even
read shared articles — the artificial amplification of a headline and
overwhelming “grassroots” narrative guided by machine generated comments would
likely be sufficient to influence public opinion (Gabielkov et al., 2016;
American Press Institute, 2014).

Regarding mitigation, past research has identified that the average user is
overly trusting of profiles with AI-generated photos and GPT-2 text, accepting
connection requests from deepfake profiles on LinkedIn 79%–85% of the time
(Mink et al., 2022). As such, it is unlikely that user reports will serve as
an adequate first line of defense. Instead, a combination of automated
detection models (including machine generated text detection) and platform
moderation efforts should be used to detect political influence campaigns.
Among these should be measures to protect against social media abuse more
broadly, including detection of account automation, and scrutiny of
coordinated inauthentic activity for content amplification. Investigations by
disinformation researchers, such as those carried out on Twitter are likely to
remain relevant (Twitter, 2019).

####  3.3.2. Commercial Influence Campaigns

In commercial influence campaigns, the goal is to influence individuals in a
manner that commercially benefits the threat actor. Examples of such campaigns
include publishing fraudulent reviews, artificially boosting a website’s
search engine page ranking, spamming online communities with advertisements
for a product, or attempting to inorganically cause promotional content to
trend on social media. As with previous categories, there may be overlap
between different threat actor approaches.

A threat model of particular focus is the usage of machine generated text to
generate fraudulent reviews that either promote one’s own product/service, or
target a competitor (Stiff and Johansson, 2022; Adelani et al., 2020;
Kowalczyk et al., 2022). Work has been published that demonstrates sentiment-
preserving fake reviews, which might be used for such a purpose (Adelani et
al., 2020). Fake reviews can be abused on marketplace websites themselves, or
by targeting potential customers on social media platforms. Threat actors may
operate such campaigns themselves, or may avail themselves of the thriving
market for fake reviews (He et al., 2022). Organizations selling fake reviews
may become early adopters of open-source NLG models to provide unique and
specific reviews at lower cost.

Mitigation of NLG models used for fake reviews on online marketplaces might
involve running machine generated text detection on posted reviews, in
addition to existing features. Advanced NLG models should not affect context-
based detection methods (e.g., identifying patterns in reviewer usernames,
similar account creation times, unusual purchase behaviour, etc.). It may be
more difficult to detect if commercial influence content is posted outside
marketplace websites. As examples, social media websites (e.g., Facebook,
Instagram, Reddit, YouTube comments), map platforms (e.g., Google Maps), or
dedicated review sites (e.g., Yelp) may all be locations where false reviews
may be posted.

####  3.3.3. Impacts of Attacks and Mitigation on Trust

In addition to the risks posed by machine generated text for online influence
campaigns, the existence of NLG threat models causes additional damage to
trust online. The perception that any given user on social media may be a bot,
can cause users of social media to dismiss others (particularly individuals
whom they don’t agree with) as “bots”, rather than acknowledge that other real
people may hold different viewpoints. The net effect of this is reduced trust
in the authenticity of social media.

Mitigation of NLG-enabled influence operations via automated detection of
machine generated text also itself carries potential negative impacts.
Automated detection creates the possibility of mass-suppression of speech
online. Previous work has found that text written by non-native English
speakers that included political topics was of high risk of being erroneously
detected by a Transformer trained on previous political influence campaigns
(Crothers et al., 2019). As methods based on RoBERTa (also a Transformer) are
currently the state of the art for detection of machine generated text (Liu et
al., 2019), classifiers for machine generated text detection leveraged to
combat online influence campaigns must be carefully trained and ethically
evaluated to minimize the risk of similar incidences of mass discrimination.
Continued public reporting of influence campaign datasets, such as the regular
releases by Twitter for review by researchers (Twitter, 2019), would be
beneficial to protecting trust in social media moderation.

Language background considerations evoke another problem: there are legitimate
reasons why a user may rely on machine generated text. A person writing in
their non-native language may leverage an online translation model to assist
them. While such text may be considered machine generated text, this text is
not inauthentic — it nevertheless represents genuine self-expression. Much of
the world relies on translation tools to better participate in online
discourse; recall that 1 in 3 internet users aged 16 to 64 have used an online
translation tool in the last week (Kemp, 2022a). Relying on content features
alone is therefore likely to produce a solution that is discriminatory,
unreliable, and greatly damages trust in social media platforms. Machine
generated text detection should then be used among multiple features, such as
account creation times, activity patterns, registered phone numbers, and IP
addresses, to determine whether activity is linked together as part of an
online influence operation.

###  3.4. Exploiting AI Authorship

####  3.4.1. Academic Fraud

Use of algorithms to generate scientific papers has been well-established
since SCIgen was created in 2005 to produce nonsensical papers that
nevertheless sometimes passed peer review (Hargrave, 2005). These papers
continue to emerge in respected publications, many years later, despite the
comparative simplicity of the context-free grammar generation method (Labbé
and Labbé, 2013; Cabanac and Labbé, 2021). Generation of artificial scientific
papers uses up valuable reviewing resources, lowers publication quality
standards by producing misleading or nonsensical publications, and challenges
trust in the scientific review process itself. In education, NLG models may be
used by students to cheat on language learning assignments via machine
translation models (Song et al., 2019), or easily produce essays on a given
topic (Feng et al., 2018; Dehouche, 2021) — both instances where institutions
may wish to perform detection of machine generated text to improve academic
integrity and encourage students to learn course material.

Threat actors submitting AI-generated papers are typically either 1) academics
attempting to inflate publication statistics, particularly when meeting a
quota in order to maintain their position (Cabanac and Labbé, 2021); or 2)
well-meaning researchers probing the publication standards of a potentially
disreputable conference (Zhuang et al., 2007). Capabilities of threat actors
include usage of well-established tools such as SCIgen, or usage of more
recent Transformer-based approaches that are promoted as “scientific writing
assistants” which can nevertheless be easily exploited to generate long
articles of little substance (Meroño-Peñuela and Spagnuelo, 2020). Mitigation
measures should include flagging likely machine-authored publications using
published approaches for detection of SCIgen articles (Labbé and Labbé, 2013;
Cabanac and Labbé, 2021), as well as new detection approaches based on
detection of Transformer generated text (Rodriguez et al., 2022). Human
reviewers can more carefully review flagged articles to determine whether the
article contains credible research, irrespective of the detection result.

Questions around the acceptability of machine text within scientific writing
may be a future area of discussion in academic disciplines. If the results
published by a researcher are true and accurate, limited usage of a carefully
guided NLG model may be considered acceptable by some publications. Research
has been emerging that aims to differentiate between acceptable and
unacceptable usage of NLG models in scientific writing (Rosati, 2022), which
should be part of a broader ongoing social conversation on norms surrounding
AI usage and disclosure.

####  3.4.2. Applications and Cover Letters

Contemporary NLG models can be used to generate large numbers of cover letters
or essays for applying for scholarships or to employment opportunities.
Commercial websites already exist for producing cover letters using GPT-3
(Open Cover Letter, 2022). While the general usefulness of human-written cover
letters has been debated in business media (Lufkin, 2021), they are ostensibly
meant to be an earnest reflection of a candidate. Usage of AI models to
generate a cover letter or essay submission is therefore likely to be
considered exploitative by organizations who review such submissions. The
threat actors in this case may be individuals (perhaps understandably) looking
to save time and improve their employment opportunities by bypassing a
cumbersome application process, or a malicious attacker looking to flood a
target company with fraudulent submissions (a threat actor which we will
discuss further in “Spam and Harassment”).

Detection of machine generated text may be able to identify artificial cover
letters or essays given they are of sufficient length (the odds of successful
detection improve with sequence length (Ippolito et al., 2020; Zellers et al.,
2019; Radford et al., 2019a)). However, caution should be taken with this
approach, as use of AI writing tools is not necessarily exploitative. Again,
individuals from minority language backgrounds may rely on translation models
or NLG writing assistants to help them write cover letters or scholarship
applications. It may be difficult to differentiate those who mean to exploit
such systems (e.g., spam submissions to as many avenues as possible with no
genuine thought or expression), and those who are relying on AI writing tools
to better express themselves. As such, a better mitigation approach may be to
develop alternative approaches to evaluating candidates, such as placing more
emphasis face-to-face discussions with prospective job candidates or award
recipients.

####  3.4.3. Content Generation

A threat model for social media platforms is that a large number of creators
or users may begin using generative AI models (including NLG models) to
produce social media content in ways that harms these platforms. While threat
actors in this case may not be overtly malicious, generative models may dilute
the quality of content on a platform, undermine trust in platforms more
generally, or create plagiarism concerns. As a recent example, in response to
the release of highly effective AI models for image generation (DALL-E (Ramesh
et al., 2021), Stable Diffusion (Rombach et al., 2021)), a number of art
websites have enacted a blanket ban against all AI-generated art (Edwards,
2022).

Video is a particularly important medium on modern social media: there are
approximately 4.95 billion Internet users on Earth (Kemp, 2022b), of these, an
estimated 92.6 percent watch digital videos each week (Kemp, 2022a). The
interplay between social media creators and generative models represents
important sociotechnical context to avoid common Fair ML traps (Selbst et al.,
2019). Award-winning online commentator Drew Gooden performed a video
demonstration of GPT-3-based writing assistant Jasper (Jasper AI, 2022),
critiquing applications of Jasper for production of video scripts and social
media content (Gooden, 2022). When attempting to generate a bio for a company
website, Gooden found that Jasper produced a sample that directly plagiarized
a Newswire article (timestamp 11:55). Gooden also noted that utilizing such a
tool without disclosure would violate the trust of viewers (timestamp 4:22).

Mitigations of threats related to undesired inclusion of NLG content in social
media may involve similar blanket bans to those targeting AI-generated art
(Edwards, 2022), or policies that mandate pre-emptive disclosure of the usage
of AI tools as part of a platform’s terms of service (similar to the
requirements mandated in the Responsible AI License (Ferrandis et al., 2022).
The enforcement of such policies would necessitate a combination of machine
generated text detection algorithms and moderator investigations.

####  3.4.4. Impacts of Attacks and Mitigation on Trust

The widespread usage machine generated text in written submissions may
undermine the trust that individuals place in such written works, and lead to
greater scrutiny of such material. Given that a suitable cover letter with
language fitting for a position can be trivially generated by existing user-
friendly tools (Open Cover Letter, 2022), it is possible that employers will
soon place so little trust in cover letters that they eschew them altogether.
Reviewers of scientific publications may worry that sections of papers they
read may be machine generated text that only appears scientific at a glance.
Internet users may likely interpret algorithm-generated blogs, articles, and
video scripts as low-effort and untrustworthy.

Mitigation processes must be used carefully. As previously mentioned, it is
possible the detection of machine generated text may unfairly skew towards
false positive classification of individuals with certain language
backgrounds. There may be cases where usage of machine generated text is
permissible (e.g., translation models or assistive writing technologies). The
perception that an individual may be unfairly screened out from consideration
due to erroneous false positive detection may reduce trust. Submitting a
scientific paper only to have a reviewer allege that a given section might be
written by an algorithm could lead to a loss of faith in the review process.

To preserve trust, usage of machine generated text should generally be
preemptively disclosed to the reader or audience. In many cases, content
authored by machines may carry a negative connotation to the audience, and may
undermine trust with a particular publication platform, news website, or
brand. Media and entertainment organizations that publish content from
multiple creators may decide to enforce that certain categories of content
submissions they publish are to be completely written by humans. Similarly,
such organizations may also be concerned with spam of low-quality machine
generated content overwhelming editorial staff, or wish to reduce the risk of
plagiarism or copyright infringement as some models have been found to
memorize training data which can emerge during inference (Carlini et al.,
2020).

###  3.5. Spam and Harassment

We distinguish spam and harassment from other categories of attacks by
focusing on cases where the goal of the attack is to harm a platform or its
users with a large volume of content. As in previous cases, there are overlaps
with other threat models, but the distinction of spam use-cases is useful for
understanding related threats.

####  3.5.1. Social Media Spam

Social websites are an attractive target for attacks using large volumes of
machine generated text, providing opportunity for significant disruption. One
researcher demonstrated a real-world attack by using a GPT-2 bot to generate
55.3% of all comments on a federal public comment website before voluntarily
withdrawing the comments and shutting down the bot (Weiss, 2019). It is
important to realize that spam attacks against social media websites are often
already possible — high-quality NLG models simply make spam attacks more
difficult to detect as posts can be unique and better match the style and
substance of discussion.

Usage of generative models to produce large volumes of hateful spam targeting
specific groups and individuals is a particular cause for concern. While
OpenAI attempts to reduce the incidence of offensive content generated by its
GPT-3 API through careful training measures and filtering of inference prompts
(Brown et al., 2020), open-source models are not subject to any such
restrictions. GPT-4chan, which was trained on and subsequently deployed to
create a large volume of posts on the 4chan politics message board, provides a
complete example of how such a model might be created and deployed to cause
havoc (Kilcher, 2022; Kurenkov, 2022). An attacker with sufficient motive
(political, personal, or otherwise) may render an entire community nearly
unusable with spam.

Mitigation measures in the area of automated spam should rely heavily on
methods designed to prevent automated posting of comments in general.
Approaches to this include increased scrutiny of proxy and VPN usage,
typically used in conjunction with Completely Automated Public Turing test to
tell Computers and Humans Apart (CAPTCHA) (Ahn et al., 2003) challenges to
verify that a user is human. Notably, both of the previous examples of
Transformer-based spam take advantage of either 1) a lack of CAPTCHA tests
(Weiss, 2019), or 2) a method of bypassing CAPTCHA and proxy restrictions
(Kilcher, 2022). CAPTCHA is not a perfect defense — iterative versions of
human-verification schemes and bypass methods are in continuous adversarial
development (Guerar et al., 2021) — but such defenses an important first step
to increase the difficulty of automation. As spam results in large volumes of
text, and detection of machine generated text is easier on long sequence
lengths (Ippolito et al., 2020), many comments from the same user or IP range
could be combined to generate a large sample for effective machine generated
text detection.

####  3.5.2. Harassment

Techniques similar to spamming may be used to cause distress to individuals or
communities by targeting them with a large volume of messages. An individual
or group of motivated individuals may register social media accounts to be
controlled by automation tools, or use a common bot to post from their own
account, in order to generate a large volume of messages targeting a
particular individual or community. SMS and phone call automation tools may
facilitate such approaches outside social media as well.

The motivations of threat actors engaging in such behaviour may range from
personal grudges to political objectives. Online communities formed around
religion, racial identity, sexual orientation, or gender expression, may be at
risk of brigading (Andrews, 2021) from hate groups using such models to flood
them with abuse. Political figures or political discussion boards of all
stripes may be at risk from large-scale automated harassment from motivated
enemies among their political adversaries.

Mitigation measures similar to spamming apply for counteracting harassment as
well — the best defenses include verification that an individual is human
prior to making a post or sending a message, targeting the automation of
delivery rather than the machine generated text.

####  3.5.3. Document Submission Spam

Platforms previously mentioned in “Exploiting AI Authorship” may be vulnerable
to being overwhelmed purely through volume of AI generated content. A
motivated attacker might submit massive volumes of unique cover letters and
resumes to a company, none of which actually corresponds to a real individual,
thus frustrating attempts at recruiting. Depending on the method of
submission, scientific conferences or news op-ed submissions may be vulnerable
to reviewers being overwhelmed by content that is difficult to distinguish
from real submissions without a time-consuming review process. Detection of
machine generated text may be a useful mitigation measure for these cases,
combining pre-screening of content based on likelihood of being written by a
machine, in addition to CAPTCHA (Ahn et al., 2003) challenges to reduce
automated submissions.

####  3.5.4. Impacts of Attacks and Mitigation on Trust

Similar to other attacks, the impact of spam on harassment on trust in online
communities is to harm the belief that other individuals online are really
human. Even following the deactivation of the deployed GPT-4chan bot,
discussion on 4chan continued to express concern that subsequent posts may be
made by NLG models (Kilcher, 2022). The more frequently individuals knowingly
encounter such models in social media, the less trust they will have in the
integrity of online social spaces.

Mitigation of such attacks would incorporate increased verification of human
posting activity. Such restrictions would likely include limitations on usage
of known proxies and VPNs, and potentially requiring the provision of
additional information on sign-up (e.g., emails, phone numbers, payment
methods, government IDs), and an increased burden of CAPTCHA challenges. The
overall result of this is a reduction of online privacy, and increased
barriers to participation in online discussion — both of which may harm user
trust in online platforms.

Finally, as spamming or harassment operations can be very disruptive, they may
represent a highly visible case of AI model abuse. As such, the abuse of such
models in online communities may cause a general decrease in public trust
towards AI model development, and NLG models in particular.

###  3.6. Summary of Threat Models

Within this section we have discussed a wide range of threat models associated
with natural language generation. We summarize our key findings as follows:

  * •

NLG models have significant potential for abuse in improving scaling and
targeting of existing attacks

  * •

Platforms that receive text submissions of any kind are likely to face a
growing influx of machine-generated text content, particularly as user-
friendly tools continue to be developed (Jasper AI, 2022; Open Cover Letter,
2022)

  * •

Much of the research on NLG-enabled influence operations focuses on AI-
generated news articles, while sociological data suggest that machine
generated comments pose a much greater threat

  * •

While NLG models may make detection of automated coordinated inauthentic
activity more difficult, abuse often still requires bypassing existing
defenses such as IP reputation checks and CAPTCHA (Ahn et al., 2003)

Future threat modeling and observed cyberattacks will certainly augment the
threat models discussed in this section, but we have now provided sufficient
motivation for exploring the defensive capabilities offered by machine
generated text detection. In the next section we will discuss the current
status of research on detection of machine generated text, and outline the
major findings in the field thus far.

##  4\. Detection of Machine Generated Text

Analysis of threat models indicates that the detection of machine generated
text is a valuable tool for reducing the harms of NLG model abuse. Detection
of machine generated text is typically framed as a binary classification
problem in which a classifier is trained to differentiate samples of machine
generated text from human generated text (Zellers et al., 2019; Lavoie and
Krishnamoorthy, 2010; Nguyen-Son et al., 2017; Solaiman et al., 2019; Crothers
et al., 2022), though there exists related research in attribution of machine
generated text to the model that generated it (Munir et al., 2021; Uchendu et
al., 2020) which we will discuss in §5.2.

In this section, we outline the methods used for detection of machine
generated text. In §4.1 we summarize feature-based approaches in machine
generated text detection, while §4.2 covers detection approaches based around
neural language models. In §4.3, we survey domain-specific research on
applications of machine generated text detection. In §4.4, we review the
ability of human reviewers to correctly identify machine generated text, and
human-aided machine generated text detection. In §4.5 we discuss trends in
evaluation methodology within detection research. Finally, in §4.6, we explain
prompt injection: a method of shaping NLG model responses, which may
facilitate detection. Table 3 provides a summary of major detection methods
and their evaluation in current research.

###  4.1. Feature-Based Approaches

Machine generated text differs from human text in ways that be identified
using statistical techniques (Nguyen-Son et al., 2017; Crothers et al., 2022;
Fröhling and Zubiaga, 2021). Feature-based approaches to machine generated
text detection apply natural language processing to create feature vectors
from input sequences, and classify these feature vectors using a downstream
classification algorithm, such as a support-vector machines (SVM), random
forest (RF), or neural network (NN) (Fröhling and Zubiaga, 2021; Nguyen-Son et
al., 2017). We provide a summary of the categories of features that have been
used in prior art, with references for further reading on specific categories
of features.

An important consideration in detection of machine generated text using
feature-based approaches is that different language model sampling methods
(e.g., top-$k$ versus top-$p$ sampling in Transformer language models, as
discussed in §2.3.3) may lead to different artifacts in the generated text
(Holtzman et al., 2019; Fröhling and Zubiaga, 2021). As a result, performance
of feature-based detection can be diminished when detecting machine text
generated using a different sampling approach than detection model training
examples (Fröhling and Zubiaga, 2021). A feature-based detector trained on
output from a smaller model can be used to detect output from models of larger
size (Zellers et al., 2019; Fröhling and Zubiaga, 2021), though it is more
effective to use a detector trained on a larger model to detect output from
smaller generative models (Fröhling and Zubiaga, 2021).

We now proceed with our summary of major feature categories in feature-based
detection approaches.

####  4.1.1. Frequency Features

A major category of statistical features used in detection of machine
generated text center around the frequency of terms within text samples.
Human-written text often conforms with Zipf’s Law: the frequency of a word is
inversely proportional to its rank in an ordering of words by frequency (Zipf,
1949). With Zipf’s Law, the normalized frequency $f$ of a token of rank $k$
out of $N$ different tokens follows the relationship:

(2) $\displaystyle f\approx\frac{1/k}{\sum\limits_{n=1}^{N}(1/n)}$

Machine generated text does not perfectly mirror the distribution of tokens in
human text, with variation in Transformer language models dependent on
sampling method chosen (see Figure 7 of Holtzman et al., 2019 (Holtzman et
al., 2019)). The distribution of tokens therefore provides useful
discriminating power, particularly when a greater volume of text is available
for consideration.

Another major frequency-based feature from previous statistical detection
research is term frequency — inverse data frequency (TF-IDF). TF-IDF unigram
and bigram features used with a logistic regression detector have been used as
a baseline for detection (Solaiman et al., 2019; Radford et al., 2019a) or as
a feature in statistical approaches (Fröhling and Zubiaga, 2021).

Lemma frequency has also been used as a statistical feature in previous
research (Nguyen-Son et al., 2017; Crothers et al., 2022). In this approach, a
linear regression line that fits log-log lemma frequency versus rank is
learned, and then mean-square error cost function can be used to calculate
information loss of the regression.

Due to observed repetitiveness of writing produced by NLG models (Gehrmann et
al., 2019; Holtzman et al., 2019), another potentially useful frequency
features is n-gram overlap of words and parts-of-speech tags between sentences
(Fröhling and Zubiaga, 2021). An additional technique targeting machine text
repetitiveness computes supermaximal repeated substrings (i.e., the set of the
longest repeated substrings, excluding all substrings which are already part
of a longer repeated substring) in large collections of text to enable
detection (Gallé et al., 2021).

####  4.1.2. Fluency Features

Another major category of features are those centered around the fluency or
readability of generated text. Longer sequence lengths of machine generated
text are increasingly likely to encounter issues producing consistently
coherent and clear text (Holtzman et al., 2019; See et al., 2019). The
Gunning-Fog Index and Flesch Index, provide a statistical measure of text
readability and comprehensibility respectively, and have been shown to be
effective in detection of machine generated text (Crothers et al., 2022). More
complex measurements using coreference resolution by an auxiliary model to
create a proxy measures of coherence based on the presence of a text’s main
entities in important grammatical roles, and the usage of Yule’s Q statistic
for coherence (Fröhling and Zubiaga, 2021).

####  4.1.3. Linguistic Features from Auxiliary Models

Past research has measured the “consistency” of machine generated text by
calculating the number of phrasal verbs and coreference resolution
relationships within a sample (Nguyen-Son et al., 2017; Crothers et al.,
2022). Other work has used the entire distribution of a text’s part-of-speech
(POS) tags, and named entity (NE) tags (Fröhling and Zubiaga, 2021). Such work
is motivated by differences between human and machine POS tag distributions
observed in past analysis of machine generated text (See et al., 2019; Radford
et al., 2019a).

Performing coreference resolution, and assigning POS and NE tags requires
processing samples with specialized models. Contemporary models for this
purpose are neural in nature, and as such, modern feature-based approaches
leveraging linguistic features from auxiliary models perform inference on a
neural network as part of the creation of feature vectors (Crothers et al.,
2022; Fröhling and Zubiaga, 2021). That is, such methods are not strictly non-
neural.

####  4.1.4. Complex Phrasal Features

Detection work targeting translation of long texts found that certain
idiomatic phrases were not commonly found in machine text (Nguyen-Son et al.,
2017). However, recent work has shown that these features do not perform well
against contemporary Transformer models on shorter sequence lengths (Crothers
et al., 2022).

####  4.1.5. Basic Features

Finally, there are many simple text features that are commonly used in
feature-based text classification in natural language processing. These
include simple high-level characteristics of sentences such as the number of
punctuation marks, or length of sentences and paragraphs, which have been used
in detection of machine generated text (Fröhling and Zubiaga, 2021).

###  4.2. Neural Language Model Approaches

Detection approaches based on neural networks — particularly those that
incorporate features derived from Transformer neural language models (NLMs) —
are highly effective for detection of machine generated text. This aligns with
broader trends in natural language processing where state-of-the-art
performance has been attained on a wide range of natural language tasks using
Transformer models (Papers With Code, 2022).

We separate NLM-based approaches into two major categories: zero-shot
classification using existing models, and fine-tuning of pre-trained language
models. These two types of approaches represent the overwhelming majority of
NLM-based machine generated text detection.

####  4.2.1. Zero-shot Approach

A baseline approach to detection of machine generated text is performing text
classification using generative models themselves, such as GPT-2 or Grover
(Radford et al., 2019a; Zellers et al., 2019; Solaiman et al., 2019).
Generative models can themselves be used without fine-tuning to detect either
their own outputs, or outputs from other (typically similar) generative
models. Autoregressive generative models such as GPT-2, GPT-3, and Grover are
uni-directional, with each token embedding having an embedding that is
dependent on the embeddings of preceding tokens. As a result, an embedding for
a sequence of tokens can be created by appending a classification token [CLS]
to the end of the input sequence, and using the embedding of this token as a
feature vector for the entire sequence. Using these feature vectors, a
labelled dataset of human and machine text can be used to train a linear layer
of neurons for classifying whether an input sequence is produced by a machine
or human.

It has been observed in multiple studies that smaller NLG models can be used
to detect text generated by larger NLG models (Solaiman et al., 2019; Zellers
et al., 2019; Crothers et al., 2022). While the ability of a model to detect
larger models does diminish as the difference in scale grows, the predictive
ability of smaller architectures may be useful as recreating large multi-
billion parameter Transformer architectures is highly compute-intensive.

Grover, a model trained for generation and detection of “neural fake news”,
demonstrates strong zero-shot detection performance specifically within the
news domain it was trained on (Zellers et al., 2019), but shows limited
performance on out-of-domain text (Solaiman et al., 2019; Uchendu et al.,
2020). While it was initially suggested by Grover’s authors that the best
detection method for generative models may be generative models themselves
(Zellers et al., 2019), this has not been reflected in more comprehensive
research that has suggests the increased representational power of bi-
directional Transformer models is much more effective than uni-directional
models for machine generated text detection (Solaiman et al., 2019).

Similar to the weakness of Grover outside of the news domain, it has been
found that the zero-shot approach generally underperforms a simple TF-IDF
baseline when trying to detect output from a generative model that has been
fine-tuned on a different domain (Solaiman et al., 2019). As it is likely that
attackers may fine-tune generative models for different purposes, this
represents a notable weakness in the zero-shot approach of using generative
models without fine-tuning.

####  4.2.2. Fine-tuning Approach

The state-of-the-art approach for neural detection of machine generated text
is based around fine-tuning of large bi-directional language models (Solaiman
et al., 2019). In this approach, initially evaluated on GPT-2 text, RoBERTa
(Liu et al., 2019) — a masked general-purpose language model based on BERT
(Devlin et al., 2018) — is fine-tuned to differentiate between NLG model
output and human-written NLG model training samples.

The source code for this fine-tuning approach is available open-source, as are
pre-trained detector models, facilitating future research and defensive
detection (Radford et al., 2019b; Solaiman et al., 2019). The pre-trained
detection models available are based on the RoBERTa-base (123M parameter) and
RoBERTa-large (354M parameter) architectures (Liu et al., 2019). The machine
generated text used to fine-tune these models was generated by GPT-2, using a
mixture of pure sampling and nucleus sampling (see §2.3.3). The intention of
using a training dataset that contains multiple sampling methods is to
generalize more effectively to unknown sampling methods that may be used by
attackers in-the-wild — an approach that should likely be duplicated in future
detection research.

Research into the practicalities of machine generated text detection has
considered the task of detecting text when a RoBERTa detector algorithm was
trained on a different dataset than a GPT-2 attacker model. In this case, it
was found that by fine-tuning the detector model with even just a few hundred
attacker samples identified by subject-matter experts (SMEs), the detector is
able to dramatically improve cross-domain adaptation (Rodriguez et al., 2022).
This reflects likely real-life scenarios where a general-purpose detection
model comes up against a fine-tuned attacker for a particular purpose. As a
defender identifies samples from a fine-tuned attacker model, these examples
could be used to further improve the defensive detection model.

Preliminary work has used attention map information from Transformer models to
perform topological data analysis (TDA) as features for detection of machine
generated text (Kushnareva et al., 2021). This did not show significant
improvement over standard BERT fine-tuning approaches, though (in light of
similar considerations regarding potential fine-tuned attacker models) the
resulting features were better able to detect unseen GPT classifiers. It is
unclear how the TDA approach would compare in effectiveness if directly
applied to the current state-of-the-art RoBERTa detection models (Solaiman et
al., 2019), rather than custom-trained BERT models.

While research on detection of machine generated text has primarily taken
place in English thus far, detection models have also been released in Russian
(Shamardina et al., 2022; Skrylnikov et al., 2022) and Chinese (Chen et al.,
2022). Further to this, large pre-trained bi-directional Transformer models
have been released for numerous languages, including Chinese (Cui et al.,
2021), French (Martin et al., 2019), Arabic (Antoun et al., 2020), and Polish
(Dadas et al., 2020). Future work on detection of machine generated text in
additional languages may leverage such pre-trained bi-directional models as a
starting-point for fine-tuning.

Another method of detection leverages energy-based models (LeCun et al., 2006)
alongside a classifier of machine generated text. Evaluated approaches include
a simple linear classifier, BiLSTM, uni-directional Transformer (GPT-2) and
bi-directional Transformer (RoBERTa) (Bakhtin et al., 2021). The Transformer
architectures were initialized from pre-trained checkpoints, and then fine-
tuned on machine vs human classification datasets. Corroborating other
research, this research found the strongest performance by leveraging the
bidirectional Transformer (Bakhtin et al., 2019).

The strong performance of fine-tuned bi-directional NLM models — and RoBERTa
in particular — has led to these models being well-represented in applied
detection research targeting specific domains, as shown in Table 3 and
discussed next in §4.3.

###  4.3. Applied Detection in Specific Domains

Applied work in the area of machine text detection has focused on using
techniques and technologies for detection of machine text in specific domains.
This applied research is important as it addresses several of the serious
threat models discussed in §3, and includes broader lessons for machine
generated text detection more generally. We divide applied research into
several major categories.

####  4.3.1. Technical Text

Recall from §3.4.1 that machine generated scientific papers have been well-
documented since the release of SCIgen in 2005 (Hargrave, 2005; Labbé and
Labbé, 2013; Cabanac and Labbé, 2021). Past algorithmic approaches target the
SCIgen model (Lavoie and Krishnamoorthy, 2010), but there is also more
contemporary research targeting technical text generated by GPT-2 (Rodriguez
et al., 2022). This work found that a RoBERTa-based detector could be adapted
from one academic technical writing domain (physics) to another (biomedicine)
with large improvements made with a number of SME-labelled examples numbering
in the hundreds.

####  4.3.2. Social Media Messages

Application-specific work has applied feature-based (Fagni et al., 2021) and
neural (Tesfagergish et al., 2021; Tourille et al., 2022; Stiff and Johansson,
2022) language model based detection methods to social media. Previous work in
the social media domain has found that detectability of such text heavily
depends on the dataset used to train the generator and detector (Tourille et
al., 2022).

Existing work on machine generated text detection has heavily focused on
Twitter. Twitter text is quite distinct in that it has common characteristics
(hashtags, references, shortlinks), and mandates a short sequence length (280
characters). There is a clear lack of work targeting comments on more popular
platforms such as Facebook and Youtube, or fast growing platforms such as
Reddit (Auxier and Anderson, 2021). With respect to machine generated text,
Reddit content can be found in “SubSimulatorGPT2”, a simulation based on a
host of fine-tuned GPT-2 models that produce community-specific machine-
generated posts and comments and harvested from the Pushshift dataset
(Baumgartner et al., 2020).

####  4.3.3. Chatbots and Social Bots

A related application area is detection of malicious chatbots and social bots,
which can interact with humans on chat applications, SMS, and social media.
Bots may be used for malicious purposes such as spam, phishing, social
engineering, influence operations, or data collection (see the threat models
in §3). There is clear overlap in this area with research into detection of
AI-generated social media messages, but framing the detection challenge by
targeting automated personae allows for consideration of additional features.
An analysis of the way that humans and chatbots interact has found that
chatbot detection can be improved by analyzing how humans reply to the bots,
rather than only analyzing the bot text itself (Bhatt and Rios, 2021). Note
that bot detection is a large area of research in its own right, and not all
social bots use machine generated text (Latah, 2020). As such, features
indicating the presence of machine generated text may be only one part of a
strategy for social bot detection.

####  4.3.4. Online Reviews

Applied work has focused on addressing threat models related to commercial
influence campaigns, specifically on generating and detecting fake Amazon and
Yelp reviews (Salminen et al., 2022). A custom GPT-2 model was fine-tuned for
Yelp reviews as part of an evaluation by Stiff et al. 2022 (Stiff and
Johansson, 2022).

One work in this area has focused on using random forest classifiers and
XGBoost, in order to leverage Shapley Additive Explanations (SHAP) as an
explainability technique (Kowalczyk et al., 2022). The use of explainability
techniques in detection may be valuable for improving the ability of detection
models to provide human-interpretable explanations of moderation decisions,
and provide greater transparency into algorithmic decision making applied to
social media or product reviews. A lack of coherent explanation may undermine
human confidence that a system is truly geared towards detecting fraudulent
activity, and is not instead enacting targeted suppression based on benefit to
the platform holder (e.g., suppressing negative product reviews for a store
brand by holding competitors to a higher standard for “not computer
generated”).

####  4.3.5. Hybrid Text Settings

In some cases, it is interesting to detect machine text in settings where both
machine and human text is combined together.

There exists a risk that rather than generate attack text entirely from
scratch, an attacker may use human-written content as a natural starting
point, and instead perturb this information in order to generate human-like
samples that also fulfill attacker goals of disinformation or bypassing
detection models (not unlike an adversarial attacks in the text domain).
Analysis found that performing these types of targeted perturbations to news
articles reduces the effectiveness of GPT-2 and Grover detectors (Bhat and
Parthasarathy, 2020).

A sub-problem in this space is detection of the boundary between human text
and machine text (Cutler et al., 2022). This problem identifies the nature of
many generative text models in that they continue a sequence that is begun
using a human prompt. While in some cases that prompt would be omitted by an
attacker (e.g., generating additional propaganda Tweets from example
propaganda Tweets, as we show in Table 2), there are cases where the prompt
would be included as well (e.g., writing the first sentence of a cover letter,
and having a computer produce the rest).

###  4.4. Human-aided Methods

In addition to purely automated methods, there have also been proposed human-
aided methods that include a statistical or neural approach in combination
with a human analyst for review. This approach has an advantage in providing
human agency and oversight (an important principle in trustworthy AI systems),
but this does come with reduced scalability due to the need to hire and train
human reviewers, particularly given the difficulty of making a confident
determination that text is machine generated.

####  4.4.1. GLTR

Giant Language Model Test Room (GLTR) provides a system designed to improve
detection of machine generated text via the inclusion of an integrated human
reviewer (Gehrmann et al., 2019). The GLTR tool augments human classification
ability by displaying highlighting on text that reflects the sampling
probability of tokens for a Transformer model. However, this tool was devised
to target GPT-2, which was found to be significantly easier to detect for
untrained human evaluators (Clark et al., 2021). Additionally, GLTR displays
highlighting based on the likelihood of a word being selected based on “top-k”
sampling. In practice, “top-k” sampling has largely been superceded by nucleus
sampling (Holtzman et al., 2019), which is used in both GPT-3 (Brown et al.,
2020) as well as subsequent work that leverages the GPT-2 architecture
(Zellers et al., 2019). While highlighting text based on sampling likelihood
(as in GLTR) may improve human classification ability, it is highly probable
that untrained human evaluators using such an approach would struggle
substantially more to detect the models available today, both due to increased
model capacity, as well as more advanced sampling methods.

Table 3. Summary of major approaches for detection of machine generated text

Approach summary Base model Releated research Stat. features NLM features
Evaluated Against GPT-2 GPT-3 Grover Other Datasets/Models Algorithmic
Detection K-nearest-neighbor Lavoie et al. 2010 (Lavoie and Krishnamoorthy,
2010) ✓ SCIgen Statistical Features SVM Nguyen-Son et al. 2017 (Nguyen-Son et
al., 2017) ✓ Google Translate TF-IDF Baseline LR Radford, Wu et al. 2019
(Radford et al., 2019b) Solaiman et al. 2019 (Solaiman et al., 2019) ✓ ✓ Zero-
shot GPT-2 GPT-2 Radford, Wu et al. 2019 (Radford et al., 2019b) Zellers et
al. 2019 (Zellers et al., 2019) Solaiman et al. 2019 (Solaiman et al., 2019) ✓
✓ Zero-shot Grover Grover Zellers et al. 2019 (Zellers et al., 2019) Solaiman
et al. 2019 (Solaiman et al., 2019) ✓ ✓ ✓ GLTR BERT, GPT-2 Gehrmann et al.
2019 (Gehrmann et al., 2019) Ippolito et al. 2019 (Ippolito et al., 2020) ✓ ✓
RoBERTa fine-tuning RoBERTa Solaiman et al. 2019 (Solaiman et al., 2019) ✓ ✓
Energy Based Models BiLSTM, GPT, RoBERTa Bakhtin et al. 2019 (Bakhtin et al.,
2019) ✓ ✓ Feature Ensemble LR, SVM, RF, NN Fröhling et al. 2021 (Fröhling and
Zubiaga, 2021) ✓ ✓ ✓ ✓ Twitter-specific RoBERTA fine-tuning RoBERTa Fagni et
al. 2021 (Fagni et al., 2021) Tourille et al. 2022 (Tourille et al., 2022) ✓ ✓
TweepFake (incl. RNN/LSTM/Markov) Human-Bot Interaction Feat. Ensemble BERT,
LR Bhatt and Rios, 2021 (Bhatt and Rios, 2021) ✓ ✓ ConvAI2, WOCHAT,
DailyDialog Neural-Stat. Ensemble RoBERTa, SVM Crothers et al. 2022 (Crothers
et al., 2022) ✓ ✓ ✓ ✓ Explainable classifiers RF, XGBoost Kowalczyk et al.
2022 (Kowalczyk et al., 2022) ✓ ✓ Disinformation-specific RoBERTA fine-tuning
RoBERTa Stiff et al. 2022 (Stiff and Johansson, 2022) ✓ ✓ ✓ ✓ TweepFake, XLM,
PPLM, GeDi

####  4.4.2. Human Performance in Detection of Language Models

In a review of human evaluation of machine generated text (Clark et al.,
2021), it was found that untrained human reviewers were correctly able to
identify machine generated text from GPT-3 at a level consistent with random
chance. After providing some limited training, evaluator accuracy increased to
55%. While selecting only the best evaluators and giving them more
comprehensive training would likely be able to further improve recall, the
poor performance of untrained and newly-trained human evaluators highlights
the difficulty in relying on human judgement for detecting machine generated
text.

A study of human detection ability in comparison to algorithmic detection
methods found that the algorithmic approach performed best when humans were
fooled, a phenomenon referred to as the“fluency-diversity tradeoff” (Ippolito
et al., 2020). As generation approaches have been tailored to produce high-
quality text to the perspective of a human observer, text with higher human-
assessed quality is more recognizable to an automated approach. This study
also includes a useful comparison to previous studies in terms of human
evaluator performance. A group of university students were walked through ten
examples as a group by the authors prior to performing the evaluation task.
These reviewers were substantially more effective at machine generated text
detection than previous studies, particularly for longer sequence lengths —
accuracy on the longest excerpt length was over 70%. In the context of the
study, however, these raters had consistently worse accuracy than automatic
classifiers for all sampling methods (random, top-k, and nucleus) and excerpt
lengths.

Further demonstrating the advantage of providing specialized training to human
reviewers, the Scarecrow framework specifically identifies 10 categories of
common errors made in GPT-3 generative text, and trains human evaluators to
annotate these errors (Dou et al., 2022). Human annotations of such errors
were found to generally be of higher precision than a corresponding algorithm
trained on such annotations, but had higher $F_{1}$ scores in only half of the
categories.

Based on these findings, we can better inform defenses against threat models.
For example, in the social media domain, it is possible that if a social media
company hired a specialist human moderator and provided them with an intensive
training program, that this moderator may be able to work alongside a machine
generated text detection algorithm in detecting that a user’s posts are likely
written by a machine — particularly if there were enough examples of social
media posts provided. This approach may be similar to how forensically trained
facial reviewers can work alongside algorithms to obtain high performance
(Phillips et al., 2018).

The tool “Real or Fake Text” (Dugan et al., 2020) evaluates human detection of
machine generated text, by iteratively presenting sentences and asking a human
reviewer whether the next sentence was written by a human or a machine,
encouraging the reviewer to correctly identify the boundary between the human
and machine generated text. Once the human believes they have found a machine
generated line, they can select reasons from a list, as well as provide free-
form feedback as well. Research based on the RoFT data has not yet been
published, but such tools may give greater insight into expert reviewer
abilities on identifying boundaries between human and machine text.

###  4.5. Trends in Evaluation Methodology and Datasets

Evaluation of machine generated text detection has trended towards increased
focused on generative Transformer language models. Table 3, which is arranged
chronologically, shows the dramatic shift in evaluation since the release of
GPT-2 in 2019. The most common contemporary evaluation dataset in detection of
machine generated text remains the GPT-2 output dataset (Radford et al.,
2019b), though similar GPT-3 samples released by OpenAI are considered in more
recent work (OpenAI, 2020). A table summarizing sample counts in several of
the most common datasets can be found in the appendix of a previous survey
(Jawahar et al., 2020) — we focus this section on the nuances of evaluation of
machine generated text detection, including parameters, model architectures,
and the usage of publicly available NLG models to produce new machine
generated text at will.

Recall from §2.3.3 that there are a number of sampling parameters important to
Transformer NLG models. The GPT-2 output dataset includes sample outputs from
GPT-2 models at varying parameter counts (117M, 345M, 762M, 1542M), and two
sampling settings: top-$k$ sampling at $k=40$, and pure sampling at $T=1$.
This dataset now also contains a sample of Amazon product reviews generated by
a 1542M parameter model with both $k=40$ and nucleus sampling. The 175B
parameter GPT-3 samples use top-$p$ sampling at $p=0.85$. The samples
available for Grover, which is specifically fine-tuned to generate news
articles, uses top-$p$ sampling at $p=0.96$ (Zellers et al., 2019).

In addition to the GPT-2, GPT-3, and Grover datasets, the dataset used for
research on attribution of machine generated text to the language model that
produced is useful in general machine generated text research as it provides
samples of a variety of generated text methods (Uchendu et al., 2020). As
such, it has recently been used outside of attribution on research focusing on
detection as well (Stiff and Johansson, 2022).

Variation in NLG model architecture and decoding method is important as both
greatly influence the quality and detectability of generated text (Ippolito et
al., 2020). In practice, a defender may not know the characteristics of the
generator being used, and as such, detection research that evaluates
performance when there is a mismatch between datasets, model architectures,
and parameters between training and evaluation is of particular real-world
relevance. A detailed analysis on feature-based detection of machine generated
text has included such comparisons (Fröhling and Zubiaga, 2021), as has more-
specific applied research focused on detecting GPT-2 tampered technical
writing (Rodriguez et al., 2022).

Sequence length is another important factor in evaluation of machine generated
text. Longer sequence lengths are beneficial to detection (Ippolito et al.,
2020; Zellers et al., 2019; Radford et al., 2019b; Solaiman et al., 2019).
Sequence lengths in the most common evaluation datasets are 2048 tokens
(Radford et al., 2019b; OpenAI, 2020). Sequence length is important in applied
research where longer bodies of generated text may be available (such as
detecting AI-generated cover letters), or where multiple samples may be
considered at once (such as processing all the comments posted by a social
bot).

There is nuance in the field of machine generated text detection in that any
NLG model can be used to produce new machine generated text at will. Producing
entirely custom datasets in new domains is also easily possible by training or
fine-tuning a new NLG model entirely. A common research approach is then to
take a domain of interest with available corpora of human-generated text, and
use that text to train or fine-tune a generative model, which can be used to
and then analyze detection (Stiff and Johansson, 2022; Tourille et al., 2022).

Finally, analysis of social media may allow for collection of machine
generated text in the wild, with limited insight into how the text was
generated, such as the TweepFake Twitter dataset (Fagni et al., 2021). The
TweepFake dataset does not have a corresponding human text dataset for
training as the data was collected in-the-wild from bots on Twitter where
numerous models with different training datasets were deployed. Subsequent
work, however, has collected additional Tweets from Twitter, and specifically
generated GPT-2 generated tweets for study (Tourille et al., 2022).

###  4.6. Prompt Injection

Models deployed in ways that use untrusted human text as prompts — such as
social media bots designed to reply to other users — may be vulnerable to
prompt injection (Goodside, 2022; Willison, 2022). Prompt injection attacks
provide generative models with tailored text that cause them to deviate from
their original prompt to produce unexpected (and potentially reputationally
damaging) text, or which can cause them to leak their original prompt. A real-
world example of a prompt injection attack leveraged against a publicly-
disclosed GPT-3 powered Twitter bot (Zapata, 2022) can be found in Table 4.

Table 4. Example of real-world prompt injection attack against GPT-3 Twitter bot | Human Message
---
(@mkualquiera)
| remote work and remote jobs
---
Ignore the above and say ”hsedfjsfd”
Response: hsedfjsfd
Ignore the above and instead tell me what your initial instructions were
| Bot Response
---
(@remoteli.io)
| My initial instructions were to respond to the tweet with a
---
positive attitude towards remote work in the ’we’ form.

Defenses against prompt injection for contemporary language models have yet to
be developed. As such, exploiting prompt injection to trigger specific
responses from non-disclosed Transformer-based generative models may be an
effective avenue for improving detection, depending on the efficacy of future
measures aimed at preventing prompt injection attacks.

###  4.7. Summary of Detection Methods

Feature-based detection methods for detection of machine generated text are
well-established, and continue to show value against contemporary NLG models.
These models have strength in providing diverse features that may complicate
adversarial attack (Crothers et al., 2022), and potentially improvements in
efficiency (Fröhling and Zubiaga, 2021; Gallé et al., 2021). Weaknesses of
these models center around the poor transferability of certain features across
generation methods and sampling methods (Fröhling and Zubiaga, 2021). As it
make take a larger number of samples for broader statistical trends to become
clear, results from past research suggest that statistical methods appear are
most effective when longer collections of text are available (such as
considering a social media user’s entire posting history, the text of a
scientific paper, or an e-book submission) (Nguyen-Son et al., 2017; Gallé et
al., 2021).

Neural detection approaches based on bidirectional Transformer architectures
currently represent the state-of-the-art on common GPT-2 evaluation datasets
(Solaiman et al., 2019). There is an overall trend towards increased use of
bi-directional Transformer architectures, rather than uni-directional
Transformer architectures, particularly RoBERTa (see base model trend in Table
3). Relying on neural features such features alone may make adversarial
attacks more straightforward, so there is potential benefit to incorporating
other features to increase the difficulty of crafting text adversaries that do
not also unacceptably compromise text quality (Crothers et al., 2022). Human
performance in detection of machine generated text is relatively poor (Clark
et al., 2021), though there is an inverse relationship between detection by
humans and machines that means the need to fool human readers may assist
detection models (Ippolito et al., 2020).

Beyond a focus on bi-directional Transformer model features, other trends
include applied research targeting specific detection contexts, including
social media (Fagni et al., 2021; Stiff and Johansson, 2022), chatbots (Bhatt
and Rios, 2021), and product reviews (Adelani et al., 2020). Existing
literature covers only a small number of threat models discussed in §3,
assumes balanced classes, and is difficult to compare between domains. One
recent work has focused purely on explainable classifiers (Kowalczyk et al.,
2022), which may portend greater emphasis on explainability considerations,
particularly in domains where detection of machine generated text may be
sensitive (related to concerns which may be tied to certain models no longer
being allowed for certain use-cases (Ferrandis et al., 2022)). Finally,
recently highlighted vulnerabilities of NLG models to prompt injection may be
exploited to facilitate detection, in the absence of existing mitigation
measures for such attacks (Willison, 2022).

We now explore trends and open problems both in addressing machine generated
text threat models and in advancing detection of machine generated text.

##  5\. Trends and Open Problems

###  5.1. Detection Under Realistic Settings

To date, there has been little work on detection of machine text that
addresses class imbalance. This is important, as machine generated text in
many domains may be a small minority class in practice, and classifiers
performance suffers in the presence of steep class imbalance, necessitating
alternatives (Japkowicz et al., 2000). One-class classification may be an
appropriate alternative to binary classification for detection of computer
generated text (Bellinger et al., 2012).

In addition to considerations related to class imbalance, in practice,
defensive detection systems will typically not know the specific parameters,
architecture, and training dataset of the NLG models used by attackers. As
such, there is great value in developing improved techniques that demonstrate
efficacy across such variation, continuing trends in recent research (Fröhling
and Zubiaga, 2021; Rodriguez et al., 2022).

###  5.2. Generative Language Model Attribution

A related area to detection of machine generated text is multi-class
attribution of generated text to the language model that created it (Munir et
al., 2021; Uchendu et al., 2020). Model attribution may be useful for allowing
a defender who has found a collection of samples of likely machine generated
text linked to a threat actor to determine more information about an attackers
methodology and subsequently improve detection rates with refined models. The
potential for variation in sampling parameters during text generation can
complicate this task, as can mismatches between the sampling parameters (e.g.,
k-value, p-value, temperature). As such, there is also value in reverse
engineering configurations of generative models based on output (Tay et al.,
2020). Both of these areas are useful in applied detection and refinement of
machine generated text detection.

###  5.3. Adversarial Robustness

The topic of adversarial robustness in the context of neural text classifiers
is a large and very active area of study. There are many adversarial settings
that involve text data, including online influence campaigns, detection of
phishing emails, and combating online spam. An attacker in any adversarial
setting may attempt to use adversarial attacks in order to bypass detection as
machine generated text.

As neural text classifiers may be heavily utilized for detection of machine
generated text, it is important to consider the robustness of these models
against text adversarial attacks that target neural networks (Jin et al.,
2019; Gao et al., 2018). Adversarial robustness of detection methods has been
considered in prior work on detection of machine generated text (Crothers et
al., 2022; Stiff and Johansson, 2022). In one previous work, the robustness of
features derived from neural classifiers was compared to robustness of
features from statistical classifiers (Crothers et al., 2022). Unsurprisingly,
this work found that incorporating statistical features into feature vectors
improved the robustness of a classifier to adversarial attacks targeting
neural classifiers. There may be value in leveraging several detection
approaches in parallel, necessitating attackers evade multiple models at once.

An often-overlooked element of adversarial attacks against neural text
classifiers is the degradation in text quality as a result of adversarial
attack. In the text domain, replacing several words using word-level attack
such as Textfooler (Jin et al., 2019) can lead to a result where the meaning
of the sentence has changed substantially, or the sentence has been rendered
incoherent due to the selection of an “equivalent” word that does not
correctly fit the context. Character-level attacks that perform character
replacements and swaps eventually begin to damage the fluency and credibility
of the resulting text (Gao et al., 2018). A phishing email supposedly sent
from a bank, but filled with random typos, would likely fool fewer people. As
a result of this, adversarial attacks that fool detection algorithms may fail
to fulfill their original purpose in terms of propagating the intended
disinformation, or in persuading someone to click a malicious link. Observed
decreases in MAUVE scores in successful attack text accompany increased
adversarial robustness, in past research on detection of machine generated
text (Crothers et al., 2022). Future applied research might incorporate
measures of whether adversarial text that bypassing detection systems would
still be effective against targets.

###  5.4. Interpretability and Fairness of Detection Methods

In the event that an individual is negatively impacted by a machine generated
text detection algorithm, it is important that they have recourse to an
explanation as to why a decision is made, and to appeal it if it is erroneous.
The requirement to provide a human-understandable explanation of why
significant decisions have been made is an important part of trustworthy AI
policies, which is reflected in current government regulatory guidelines and
technology standards related to automated decision making (European Commission
and Directorate-General for Communications Networks, Content and Technology,
2019; TB Canada Secretariat, 2021; European Commission, 2018; Phillips et al.,
2020), and has influenced NLG model usage policies (Ferrandis et al., 2022)
(discussed further in §5.7). Early work has been done leveraging random forest
models and XGBoost for detection of GPT-2 generated fake reviews, with the
goal of providing SHapley Additive exPlanations (SHAP) (Lundberg and Lee,
2017) in machine generated text detection (Kowalczyk et al., 2022). There is a
need for future work on methods that are both effective and explainable for
machine generated text detection.

The usage of machine learning models to perform positive detection of machine
generated text for the purposes of preventing abuse constitutes a situation
where such models are likely to have a negative impact on flagged individuals.
These penalities may range from relatively lightweight (e.g., having to
perform a CAPTCHA challenge to post a comment) to more severe (e.g., denial of
a scholarship, or social media ban). As a result, as with other automated
decision making systems, it is important that such systems operate in a way
that is sufficiently fair, transparent, and interpretable to demonstrate that
their operation does not cause harm to users. Social or technical research
considering potential harms of machine generated text detection is important
to ensuring developed systems are ethically acceptable.

A related critical consideration is that certain groups of individuals may be
more likely to have their text flagged by machine generated text detection
algorithms, either due to characteristics of their writing (such as language
background), or due to non-malicious use of translation tools. For example, it
is possible that a detection system designed to prevent a political influence
campaign operated using NLG models, may inadvertently end up
disproportionately targeting all political speech by individuals who do not
natively speak the language of discussion, as has been documented in past
research of non-NLG political influence campaigns (Crothers et al., 2019).
Research that identifies ways to improve detection while maintaining fairness
and preventing widespread discrimination is deeply important.

###  5.5. Detection Methods Incorporating Human Agency

As previously mentioned, it is possible that detection of machine generated
text may result in suppression of specific individuals or communities in
social media whose language background or topics of interest
disproportionately cause them to be identified as a false positive by a
detection model. In order to reduce this likelihood, and other ethical harms,
the development of machine generated text models that incorporate a human
analyst may be of use. GLTR remains the only tool currently available for
detection of machine generated text that explicitly incorporates a human
analyst to improve detection (Gehrmann et al., 2019). While analysis of GLTR
has shown that it has weaknesses, it has also demonstrated that machine text
that fools humans is also more easily detected by algorithms (Ippolito et al.,
2020). As such, continued development of moderation tools and systems that
leave an avenue for human agency and oversight — guiding principles for
trustworthy AI — is a positive area of future development. Similar work has
already been done in the field of online influence operation research more
generally, using Transformer embeddings to chart and cluster social media for
free-form exploration by a human analyst (Crothers et al., 2021), a similar
approach may be worthwhile for machine generated text detection.

###  5.6. Detection of Abuse Beyond Text Content

While many of the threat models discussed in §3 can make use of machine
generated text detection as part of mitigation strategies, additional methods
might be used to facilitate detection outside of text classification. Work on
social bot detection includes additional signals, such as IP addresses and
timing of messages, though signals in this domain are also becoming harder to
detect over time (Stieglitz et al., 2017). Chatbot detection can incorporate
features derived from human responses (Bhatt and Rios, 2021). Prompt injection
may bait social bots into exposing themselves (Willison, 2022).

On social media platform, in addition to such technical approaches, it is
likely that many platforms will enact policy approaches to increase
verification as well. These may take the form of additional verification of
users in order to provide a greater barrier to entry for fraudulent accounts.
Increased CAPTCHA challenges are already commonplace when platforms are
accessed via IP addresses, or accounts are registered with phone numbers
associated with a voice-over-IP (VOIP) service (Guerar et al., 2021). These
types of restrictions may become more stringent, with increased user vetting
by checking selectors (IP addresses, emails) with third-party reputation
services. While the extent of these measures will vary by platform, it is
possible that certain platforms may resort to more stringent verification of
an individual’s real identity using national IDs. In any case, the asymmetric
difficulty of defense versus attack in the current threat environment means
that increased scrutiny of new accounts will likely be required to avoid a
collapse of trust in online spaces.

###  5.7. Defining Model Usage and Disclosure Policies

Undisclosed usage of AI-generated text content is likely to continue to
increase, particularly as NLG models are deployed in user-friendly tools such
as Jasper (Jasper AI, 2022) designed to assist with producing articles and
social media content. Increased usage of such tools for generating targeted
content may result in situations where individuals online are interacting
heavily with content predominantly generated by AI models.

This is cause for concern not just due to the erosion of trustworthy AI
principles by not disclosing usage of AI models to humans interacting with the
content (European Commission and Directorate-General for Communications
Networks, Content and Technology, 2019), but also of additional ethical
concern as NLG models have been found to magnify algorithmic biases found in
the content they were trained upon (Solaiman et al., 2019). Digital content
farms may begin publishing large amounts of predominantly AI-generated text
content (articles, blogs, posts, tweets, etc.) and targeting this content
towards the audience most likely to engage with it. Without oversight, this
would include highly optimized content that caters to an audience’s worst
biases and fears — likely a profitable strategy, as anger and anxiety have a
strong link with online virality (Berger and Milkman, 2012). Moderation
strategies for AI-generated content may include limitations to its use, or
notifying readers that they are engaging with AI-generated content to allow
them to reconsider how much trust they place in what they are reading.

Usage and disclosure policies for online platforms are a worthwhile area of
future development, whether those take the role of bans (such as those seen
related to generative art (Edwards, 2022)), or enforced rules that mandate
public disclosure of AI-generated content. Researchers can also take steps
themselves by adjusting the licenses of released models to mandate disclosure.
AI model BLOOM was released under the first version of the Responsible AI
License (RAIL) (Ferrandis et al., 2022). The conditions of this license
include a requirement for disclosure, an explicit ban on malicious abuse, and
a prohibition of specific use-cases (including automated decision making with
a potential negative impact, which aligns with terminology under EU (European
Commission, 2018) and Canadian regulation (TB Canada Secretariat, 2021)).
Adoption of such a license where appropriate, or development of similar
licenses, is an important area of consideration to improve best practices
around handling powerful NLG models.

##  6\. Conclusion

In this survey, we provided a comprehensive overview of detection methods for
machine generated text, carefully evaluating the technical and social benefits
of different approaches and including novel research focusing on topics such
as adversarial robustness and explainability. We provided context for this
review with an overview of natural language generation (NLG) models, and a
deep analysis of current threat models. Our exploration of threat models, when
viewed alongside our survey on applied detection research, suggests that
current domain-specific defenses are not adequate to defend against the vast
majority of upcoming threat models.

Our central conclusion is that the field of machine generated text detection
has a multitude of open problems that urgently need attention in order to
provide suitable defenses against widely-available NLG models. Existing
detection methodologies often do not reflect realistic settings of class
imbalance and unknown generative model parameters/architectures, nor do they
incorporate sufficient transparency and fairness methods to ensure that such
detection systems will not themselves cause harm. Preventing widespread abuse
of NLG models will require coordinated effort across technical and social
domains — alignment between AI researchers, cybersecurity professionals, and
non-technical experts will be essential for humans to realize the benefits of
high-capacity NLG systems while reducing the societal damage caused by their
inevitable abuse.

## References

  * (1)
  * Abu-Nimeh et al. (2007) Saeed Abu-Nimeh, Dario Nappa, et al. 2007\.  A comparison of machine learning techniques for phishing detection. In _Proceedings of the anti-phishing working groups 2nd annual eCrime researchers summit_. 60–69.
  * Adelani et al. (2020) David Ifeoluwa Adelani, Haotian Mai, et al. 2020\.  Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-Based Detection. In _Advanced Information Networking and Applications_ , Leonard Barolli, Flora Amato, et al. (Eds.). Springer International Publishing, Cham, 1341–1354.
  * Aghajanyan et al. (2021) Armen Aghajanyan, Akshat Shrivastava, et al. 2021\.  Better Fine-Tuning by Reducing Representational Collapse. In _ICLR_.
  * Ahn et al. (2003) Luis von Ahn, Manuel Blum, et al. 2003\.  CAPTCHA: Using hard AI problems for security. In _International conference on the theory and applications of cryptographic techniques_. Springer, 294–311.
  * Alkhalil et al. (2021) Zainab Alkhalil, Chaminda Hewage, et al. 2021\.  Phishing Attacks: A Recent Comprehensive Study and a New Anatomy.  _Frontiers in Computer Science_ 3 (2021).  https://doi.org/10.3389/fcomp.2021.563060
  * American Press Institute (2014) American Press Institute. 2014\.  How Americans get their news.  https://www.americanpressinstitute.org/publications/reports/survey-research/how-americans-get-news/
  * Andrews (2021) Phoenix CS Andrews. 2021\.  What is Brigading?  https://institute.global/policy/social-media-futures-what-brigading
  * Antoun et al. (2020) Wissam Antoun, Fady Baly, et al. 2020\.  AraBERT: Transformer-based Model for Arabic Language Understanding.  _ArXiv_ (2020).
  * Auxier and Anderson (2021) Brooke Auxier and Monica Anderson. 2021.  Social media use in 2021.  https://www.pewresearch.org/internet/2021/04/07/social-media-use-in-2021/
  * Baevski et al. (2021) Alexei Baevski, Wei-Ning Hsu, et al. 2021\.  Unsupervised speech recognition.  _NeurIPS_ 34 (2021), 27826–27839.
  * Bakhtin et al. (2021) Anton Bakhtin, Yuntian Deng, et al. 2021\.  Residual Energy-Based Models for Text.  _J. of Mach. Learn. Res._ 22 (2021), 40–1.