# Separating the Human Touch from
AI-Generated Text using Higher Criticism:
An Information-Theoretic Approach

Alon Kipnis

###### Abstract

We propose a method to determine whether a given article was entirely written
by a generative language model versus an alternative situation in which the
article includes some significant edits by a different author, possibly a
human. Our process involves many perplexity tests for the origin of individual
sentences or other text atoms, combining these multiple tests using Higher
Criticism (HC). As a by-product, the method identifies parts suspected to be
edited. The method is motivated by the convergence of the log-perplexity to
the cross-entropy rate and by a statistical model for edited text saying that
sentences are mostly generated by the language model, except perhaps for a few
sentences that might have originated via a different mechanism. We demonstrate
the effectiveness of our method using real data and analyze the factors
affecting its success. This analysis raises several interesting open
challenges whose resolution may improve the method’s effectiveness.

##  1 Introduction

Suppose that an article that was written by a generative language model (GLM)
such as ChatGPT undergoes relatively minor changes. For example, a human
editor adds, removes, or rephrases certain sentences as seen in Figure 1. The
purpose of this work is to propose a method for detecting the presence of
edits and as much as possible identify those parts that were edited.

I am interested in writing a Wikipedia-style article about the Welsh Corgi
dog. That sounds like a great project! The Pembroke Welsh Corgi is… Can you
please write the introduction? The Welsh Corgi, often simply referred to as
Corgi, is a charming and beloved breed of herding dog with roots deeply
entwined in the landscapes and folklore of Wales. Renowned for… Tell me about
the history of the Corgi breed. The history of the Corgi breed is deeply
rooted in the pastoral landscapes of Wales… Tell me about … Welsh Corgi
Introduction
The Welsh Corgi, often simply referred to as Corgi, is a charming and beloved
breed of herding dog with roots deeply entwined in the landscapes and folklore
of Wales. Renowned for their distinctive appearance, characterized by their
short legs and elongated bodies, Welsh Corgis have captured the hearts of dog
enthusiasts worldwide. Beyond their adorable looks Originally bred to herd
cattle, sheep, and horses, Corgis are celebrated for their intelligence,
agility, and unwavering loyalty eagerness to please, and adorable looks.
History
The history of the Corgi breed is deeply rooted in the pastoral landscapes of
Wales, where they played a vital role as herding dogs.  Corgi breeds are
classified as Pembroke Welsh Corgi and Cardigan Welsh Corgi, both originating
from a common ancestry. There are two distinct breeds of Corgis: the Pembroke
Welsh Corgi and the Cardigan Welsh Corgi. … Figure 1: The GLM ChatGPT is
prompted to generate sections of a Wikipedia-style article titled the Welsh
Corgi (dog breed). The composition of the generated text with section titles
leads to a so-called machine-generated article. The human editor alters the
machine-generated article in some places – 15 out of 95 sentences in this
example.

We interpret the concept ”written by a GLM” in a relatively broad sense. The
pre-edited article is the combination of a series of GLM outputs produced in
different contexts, e.g., in response to different prompts or instructions.
The situation above might arise when a human editor wishes to improve the GLM
text or to hide the fact that the GLM was involved in the writing process
altogether.

Throughout, we consider sentences as an atomic text unit that might undergo an
edit. The term sentences may be understood in a broader context as text atoms
that may individually contain some indication when edited.

###  1.1 Background

The recent advancements in the capabilities of GLM to produce human-like text
[RWC+19, ZHR+19, Ope22, Ope23] suggest that the problem at hand is quite
challenging. For example, GLM texts seem to evade many detection methods,
e.g., by paraphrasing individual sentences [SKB+23, JHN23, KSK+23]. Existing
detectors usually focus on indicating the presence of text written by a GLM
within a document mostly written by a human [KSK+23]. Nevertheless, the large
number of degrees of freedom in text data suggests that it would be impossible
to attain a statistically powerful detection procedure unless the null and
alternative classes are relatively restricted [Jan00]. Indeed, the
impossibility result of [Jan00] appears to provide a relevant explanation to
the challenges reported in recent works about discriminating machine- from
human-written text [NSM23, SKB+23, KSK+23, LYM+23]. In contrast, in this
paper, we focus on a restricted alternative class: GLM-written documents that
have gone through some relatively small edits as illustrated in Figure 1.

It appears that the signal discriminating these two classes is concentrated in
a small and apriori unknown subset of the sentences, hence a statistic that is
sensitive to rare (sparse) effects seems promising in this application. This
is the motivation for using a discriminator that relies on the Higher
Criticism (HC) statistics, which is well-known to have optimality properties
in such situations [DJ04, HJ08, DJ15, Kip23].

sentence | $\mathsf{lppt}$ |  $\mathsf{lppt}$ P-value
---|---|---
The Welsh Corgi, often simply… | 2.765 | 0.212
Renowned for their distinctive… | 2.864 | 0.2424
Originally bred to herd cattle,… | 2.484 | 0.649
This article delves into the… | 3.005 | 0.4904
$\vdots$ | $\vdots$ | $\vdots$
Overall, Welsh Corgis have become… | 2.456 | 0.4594
They have surpassed their humble… | 3.906 | 0.1057
Their unique combination of historical… | 3.284 | 0.0779
$\mathrm{HC}^{*}$$>\mathsf{thr}$$<\mathsf{thr}$some editsno edits Figure 2:
The detection procedure is based on testing individual sentences and combining
the results using Higher Criticism. Left (table): Individual sentences, their
log-perplexity (LPPT) values, and their associated P-values. Non-ChatGPT
sentences are in blue. Right: The HC score is compared to its null value. Here
LPPT is with respect to the language model GPT2-xl and the P-value is with
respect to sentences from Wikipedia-style articles written by ChatGPT.
Sentences shorter than 10 tokens are excluded from the process.

###  1.2 Contributions

We propose a method to detect the presence of edits in mostly GLM-written
text. Our method involves two main procedures:

  * (i)

Testing the authorship of individual sentences with respect to the candidate
GLM using a perplexity detector.

  * (ii)

Combining the multiple tests to a global test of significance against the null
hypothesis using HC that the text was entirely written by the GLM.

These are illustrated in Figure 2. In addition to discriminating the two
classes, the method also reports on a set of sentences suspected to be those
not generated by the GLM based on the so-called HC threshold [DJ09].

In practice, the method appears to perform well in some realistic scenarios
encompassing several text domains. As we explain in this paper, the method is
well-motivated from an information-theoretic perspective in the sense that
each of the steps above has optimality properties under a certain text-
generating model. The gaps between this model and realistic situations drive a
series of open challenges whose resolution likely improves the method’s
utility.

Figure 3: Discriminating individual sentences written either by a machine or a
human using the LPPT statistic (1). We evaluated LPPT with respect to the
language model GPT2-xl (1.5 billion parameters). Left: histogram by class of
log-perplexity (LPPT) of sentences from the dataset News Articles [Sar23]
(top) and Wikipedia Introductions [Aad23] (bottom). Right: the receiver
operating characteristic (ROC) of a test based on the LPPT. The area under the
ROC curve (AUC) is indicated.

###  1.3 Paper Organization

We introduce the method in Section 2. In Section 4 we provide an information-
theoretic analysis of the method’s components. In Section 3 we report on
empirical results. We discuss several generalizations and open challenges in
Section 5.

##  2 Method Description

In this section, we describe the method and explain its components. In this
description and throughout the paper, it is useful to distinguish between two
types of language models based on the output they provide.

  * (i)

(casual, predictive) Language Model (LM): provides a probability distribution
over a dictionary of tokens conditioned on an input context. We typically
denote such a model by $\mathrm{P}$.

  * (ii)

Generative Language Model (GLM): produces sequences of tokens in response to a
context input. We typically denote such a model by $\mathrm{G}_{0}$.

Our method uses an LM for inference on texts generated by a specific GLM. The
purpose of the distinction above is that in many situations we have no access
to the individual token probabilities of the GLM or its state. For example,
the LM is a pre-trained publicly available large language model like GPT2-xl
while GLM is a propriety language model like ChatGPT .

###  2.1 Testing individual sentences

We think about a sentence $S$ as a sequence $t_{1:|S|}=(t_{1},\ldots,t_{|S|})$
of tokens from a finite dictionary. Given a LM $\mathrm{P}$, the (normalized)
log perplexity (LPPT) of $S$ with respect to $\mathrm{P}$ is

$\displaystyle\mathsf{lppt}(S;\mathrm{P}):=-\frac{1}{|S|}\sum_{i=1}^{|S|}\log\mathrm{P}(t_{i}|t_{1:i-1})$
(1)

(in (1) and throughout we use the notation $t_{1:0}=\emptyset$.) In
information theory and machine learning, the LPPT is usually known as the log-
loss of $S$ under $\mathrm{P}$ or the self-information of $\mathrm{P}$
evaluated at [MF98].

An empirical result illustrated in Figure 3 says that under a specific
$\mathrm{P}$, sentences written by a particular GLM tend to have lower values
of $\mathsf{lppt}(S;\mathrm{P})$ than sentences on similar topics written by
humans. This observation justifies the use of a LPPT test to detect the
authorship of individual sentences, i.e. testing against the null

$\displaystyle H_{0}(S)\,:\,\text{``sentence $S$ was written by the GLM"}.$
(2)

The right side of Figure 3 illustrates the receiver operating characteristic
(ROC) curves of the LPPT test against the null (2) under different datasets.

Given a document partitioned into sentences $D=(S_{1},\ldots,S_{n})$, we
summarize the evidence against $H_{0}(S_{i})$ using the P-value $p_{i}$.
Namely,

$\displaystyle
p_{i}:=\Pr_{S\sim\mathrm{G}_{0}}\left[S\geq\mathsf{lppt}(S_{i};\mathrm{P})\right].$
(3)

The evaluation of $p_{i}$ requires the distribution of
$\mathsf{lppt}(S;\mathrm{P})$ for $S\sim G_{0}$, represented by the blue
histograms in Figure 3. This distribution is affected by the sentence’s
length, hence we adjust for this length when evaluating the the P-values in a
way we explain below in Section 2.4. The table in Figure 2 shows examples of
LPPT and the corresponding length-adjusted P-values of several sentences from
the example in Figure 1.

We note that the non-trivial power of the LPPT detector observed in Figure 3
suggests that for long enough documents from this domain, it is possible to
reliably separate between the class of documents written entirely by the GLM
ChatGPT and the class of documents written entirely by humans. Indeed,
consider a simple model in which a document is generated by independently
sampling sentences from one of the distributions represented by the histograms
in Figure 3 and use the likelihood ratio test of the LPPTs of individual
sentences. By the Chernoff–Stein Lemma [CT06], for any fixed Type I error
probability, the probability of a Type II error (incorrectly reporting that
the document belongs to the non-GLM) decreases exponentially in the number of
sentences (with the number of sentences at a rate equal to the relative
entropy between the distributions). This note emphasizes that the problem that
we are addressing – separating the class of documents written by the GLM from
the class of documents that contain mostly GLM-written text with some non-GLM
edits – is much more challenging.

###  2.2 Global testing using Higher Criticism

We combine the per-sentence P-values $p_{1},\ldots,p_{n}$ obtained in (3) to a
single index of discrepancy between the text and the model using Higher
Criticism (HC) [DJ04, DJ08, DJ15].

Given a set of P-values $p_{1},\ldots,p_{n}$, we define their HC statistic as

$\displaystyle\mathrm{HC}(p_{1},\ldots,p_{n}):=\max_{1\leq j\leq
n\gamma_{0}}\mathrm{HC}_{j},\qquad\mathrm{HC}_{j}:=\sqrt{n}\frac{\frac{j}{n}-p_{(j)}}{\sqrt{\frac{j}{n}\left(1-\frac{j}{n}\right)}}.$
(4)

Here $p_{(1)}\leq\ldots\leq p_{(n)}$ are the order statistics of the P-values
and $\gamma_{0}\in(0,1)$ is a tunable parameter that limits the range of
P-values involved that usually has no effect on the large sample behavior of
$\mathrm{HC}$ (in this paper we use $\gamma_{0}=0.4$). HC is known to be
sensitive to departures in a small and unknown set of P-values from their
uniform distribution. Consequently, HC is useful as an index of discrepancy
between the classes, indicating that the document was edited for large values
of $\mathrm{HC}$. This property leads to a binary classifier whose threshold
($\mathsf{thr}$ in Figure 2) can be calibrated, e.g., by a held-out dataset.

Additionally, we can use HC as level $\alpha$-test against the global null

$\displaystyle H_{0}=\bigcap_{S\in D}H_{0}(S)=\text{``The document was written
entirely by $\mathrm{G}_{0}$"}.$ (5)

Specifically, denote by $\mathrm{HC}^{1-\alpha}$ the $1-\alpha$ quantile of
$\mathrm{HC}$ under $H_{0}$ and reject if
$\mathrm{HC}>\mathrm{HC}^{1-\alpha}$. When the P-values are independent and
uniformly distributed under $H_{0}$, the asymptotic distribution of
$\mathrm{HC}$ under $H_{0}$ follows that of a maximum Brownian bridge, while
it is stochastically smaller in finite samples [DJ04]. For this reason, it is
common to simulate critical values for a test based on $\mathrm{HC}$ for
specific sample sizes. In Figure 4 we report on such values and their
bootstrapped standard errors for several sample sizes and two significant
levels.

Figure 4: Simulated critical values for a test of significance level $\alpha$
based on Higher Criticism of $n$ P-values. The number of samples in each
configuration is $10,000$. Bootstrapped $0.95$ confidence intervals are
indicated.

The independence assumption on the P-values may be unreasonable in many cases.
It is known that HC is relatively unaffected when the P-values experience a
form of short-term dependency as expected among sentences [DH09], while it may
experience a reduction in power under long-term dependency [HJ08]. For this
reason, if possible, we recommend estimating $\mathrm{HC}^{1-\alpha}$ based on
complete documents from the null class to improve the test’s power.

###  2.3 Identifying edited sentences

When the HC test rejects $H_{0}$, the set

$\displaystyle I^{*}:=\\{i\,:\,p_{i}\leq p_{j^{*}}\\},\quad
j^{*}:=\arg\max_{1\leq j\leq n\gamma_{0}}\mathrm{HC}_{j},$ (6)

corresponds to P-values thought to provide the best evidence against $H_{0}$.
This set is known to have interesting optimality properties in the context of
feature selection for binary [DJ09] classification. We use this set to
indicate sentences that we suspect may not be written by the GLM; we may also
want to examine the authorship of these sentences manually or using other
means that do not necessarily rely on the perplexity.

The full procedure is summarized in Algorithm 1 and illustrated in Figure 2.
Table 1 shows sentences included in $I^{*}$ from the article generated in the
example in Figure 1.

language model $\mathrm{P}$; document $D=(S_{1},\ldots,S_{n})$; survival
function $\bar{F}_{\mathrm{G}_{0};\mathrm{P}}$ of the LPPT of sentences from
$\mathrm{G}_{0}$ under $\mathrm{P}$; threshold $\mathrm{thr}$ (e.g,
$\mathrm{thr}=\mathrm{HC}^{1-\alpha}_{0}$)

# Step I: Testing individual sentences:

for $S_{i}\in D=(S_{1},\ldots,S_{n})$ do

$l_{i}\leftarrow\mathsf{lppt}(S_{i};\mathrm{P})$

$p_{i}\leftarrow\bar{F}_{\mathrm{G}_{0};\mathrm{P}}(l_{i})$

# Step II: Global testing using HC:

if $\mathrm{HC}(p_{1},\ldots,p_{n})>\mathrm{thr}$, then

reject $H_{0}$

report sentences $\leftarrow\\{S_{i},\,:\,p_{i}\leq p_{i^{*}}\\}$ as suspected
edits

else

do not reject $H_{0}$

Algorithm 1 Test whether a document $D$ was written by the language model $\mathrm{G}_{0}$, with sensitivity to a few edited sentences alternative.  sentences in $I^{*}$ | LPPT P-value
---|---
Despite their herding heritage gradually diminishing,… | 0.0113
Corgi-themed fundraisers and charity events have… | 0.0211
Legend has it that the fairies… | 0.0346
It is believed that the Cardigan… | 0.0400
From their origin as indispensable herding… | 0.0417
Cardigan Corgis were also adept herding… | 0.0435
Their unique combination of historical significance,… | 0.0779
They have appeared in several animated… | 0.0820
They have surpassed their humble origins… | 0.1057
$\vdots$ | $\vdots$
They excelled at driving cattle and… | 0.1770
Today, Welsh Corgis, especially the Pembroke,… | 0.1851
Mascots and Symbols: In some regions,… | 0.1883
Here are some ways in which… | 0.1942
The breeds are named for the… | 0.2065
A Welsh Corgi appeared with Queen… | 0.2114
The Welsh Corgi, often simply referred… | 0.2120
Table 1: Sentences from the article Welsh Corgi (see Figures 1 and 2) found by
the HC threshold as suspicious to be not written by ChatGPT. Actual non-
ChatGPT sentences are in blue. We emphasize that the inclusion of a sentence
in this set is based on the _global_ HC threshold in (6) and not on the
individual significance of the sentence’s LPPT P-value.

Figure 5: Adjusting the perplexity test for the number of tokens in the
sentence. Left: averaged log-perplexity versus sentence length. The shaded
area indicates standard error. Right: fitted log-perplexity survival functions
for several lengths. Based on 40,000 samples from the dataset Wikipedia
Introductions [Aad23] [Aad23].

###  2.4 Refinements

####  2.4.1 Adjusting for sentence’s length

Tokens appearing later on in the sentence tend to be more reliably predicted
than tokens at the beginning, a phenomenon observed in [Sha51]. Consequently,
the average perplexity tends to be smaller for longer sentences; this is
illustrated in Figure 5. Consequently, we can attain better sensitivity of the
perplexity test by comparing the LPPT of the $i$-th sentence $S_{i}$ to the
distribution of LPPT of sentences produced by $\mathrm{G}_{0}$ with the same
length as $S_{i}$. Formally, this means replacing the test (3) with

$\displaystyle\tilde{p}_{i}:=\Pr_{S\sim\mathrm{G}_{0},}\left[S\geq\mathsf{lppt}(S_{i};\mathrm{P})\mid|S|=|S_{i}|\right],$

and thus the survival function $\bar{F}_{\mathrm{P},\mathrm{G}_{0}}$ in
Algorithm 1 receives two parameters: the LPPT of $S_{i}$ and its number of
tokens $|S_{i}|$. In practice, we estimate
$\bar{F}_{\mathrm{P},\mathrm{G}_{0}}$ for every possible number of tokens. The
error bars in Figure 5 indicate the error in this estimation in the dataset
Wikipedia Introductions [Aad23], which appears to be small. When the number of
data points for calibration is somewhat scarce, a curve-fitting estimate is
useful since $\bar{F}_{\mathrm{P},\mathrm{G}_{0}}$ appears to vary smoothly
with the number of tokens.

Another related factor that may affect the perplexity is the sentence’s
location within the document. For example, the first sentence in every
paragraph appears to have higher perplexity than subsequent sentences. We
leave the adjustment of our method to this factor as future work.

####  2.4.2 Ignoring short sentences

Our experience shows that the perplexity detector is ineffective for short
sentences of about $10$ tokens or fewer. Consequently, we excluded such
sentences from the process and did not evaluate their P-values.

##  3 Empirical Results

In this section, we report on the results of testing our method on real data.
We tried several publicly available LMs for $\mathrm{P}$ in Algorithm 1,
including GPT2 (124M and 1.5B) [RWC+19], Falcon (7B parameters) [AAA+23],
Llama (13B parameters) [TLI+23]. We only report on results with the 1.5B
parameter version of GPT2 denoted GPT2-xl since this model attained the
largest area under the ROC curve in the binary detection problem of individual
sentences for all datasets we considered.

###  3.1  Large-scale evaluations with simulated edits

We first evaluate the effectiveness of the method using a synthetic dataset of
articles of mixed authorship of machine and non-machine. Each article is
obtained by concatenating small articles to match the prescribed long article
length, truncating excess sentences beyond this length. Each small article was
generated by randomly and independently sampling sentences from the machine-
generated article and inserting those into the non-machine-generated article
at a random location. We denote by $\epsilon$ the average mixing proportion,
i.e. the expected number of non-GLM sentences in a document over the total
number of sentences in that document. Since both articles share the same
topic, the mixed article is typically coherent in content hence it is quite
challenging to conclude whether the article contains any non-GLM sentences,
especially when $\epsilon$ is relatively small. As raw data for mixing, we use
the three datasets listed below in which every entry has two articles under
the same title, one written entirely by a GLM and one written by a human or
several humans.

  * •

Wikipedia Introductions [Aad23]. Each entry corresponds to a Wikipedia
article. The dataset contains the several first sentences of the Introduction
of this article as non-machine text and text generated by GPT3.5 in response
to a relevant prompt as machine-generated text. We removed from this dataset
the entries in which the GLM text is shorter than $15$ sentences.

  * •

News Articles [Sar23]. Each entry contains a news article, its highlights as
provided by human annotators, and an article generated by ChatGPT from this
article’s highlights. We removed from this dataset the entries in which the
GLM text is shorter than 15 sentences.

  * •

Research Abstracts [Nic23]. Each entry contains the abstract of a scientific
research paper and text produced by GPT3.5 in response to a prompt requesting
a paragraph of text with similar properties.

We report the results of applying the method on synthetic data in Table 2.
This table shows that our method has non-trivial power even for an editing
rate of $10\%$ of the sentences for articles as small as $50$ sentences. The
power generally increases with the editing rate and the length of the article.

$\epsilon$ |  number of sentences | Dataset |  Power (SE) at $\alpha=0.05$
---|---|---|---
0.1 | 50 | abstracts | 0.24 (0.01)
0.1 | 50 | news | 0.25 (0.01)
0.1 | 50 | wiki | 0.34 (0.02)
0.1 | 100 | abstracts | 0.34 (0.02)
0.1 | 100 | news | 0.35 (0.01)
0.1 | 100 | wiki | 0.49 (0.02)
0.1 | 200 | abstracts | 0.48 (0.03)
0.1 | 200 | news | 0.47 (0.02)
0.1 | 200 | wiki | 0.66 (0.03)

$\epsilon$ |  number of sentences | Dataset |  Power (SE) at $\alpha=0.05$
---|---|---|---
0.2 | 50 | abstracts | 0.49 (0.02)
0.2 | 50 | news | 0.37 (0.01)
0.2 | 50 | wiki | 0.54 (0.02)
0.2 | 100 | abstracts | 0.66 (0.02)
0.2 | 100 | news | 0.52 (0.01)
0.2 | 100 | wiki | 0.74 (0.02)
0.2 | 200 | abstracts | 0.81 (0.02)
0.2 | 200 | news | 0.7 (0.02)
0.2 | 200 | wiki | 0.89 (0.02)

Table 2:  Accuracy and power of the method in detecting simulated edits. Each document may contain several smaller articles merged to attain the required minimum number of sentences. Each article contains mostly sentences written by a GLM except an $\epsilon$ fraction (on average) of the sentences which are of a different source. Each iteration uses a random sample of $50\%$ of the sentences in the machine text as training data to characterize the null behavior. We experimented with three datasets, two mixing ratios, and three minimal document lengths. In all datasets, the power of our method generally increases with the number of sentences and the fraction of non-GLM sentences.  title |  length (sentences) |  edit proportion | $\mathrm{HC}$ | P-value |  $\mathrm{HC}$ (pre-edit) |  P-value (pre-edit)
---|---|---|---|---|---|---
Dimtri Mendeleev | 90 | 0.155340 | 2.955169 | 0.000171 | -1.549466 | 0.989086
American Civil War | 324 | 0.059490 | 0.644052 | 0.794610 | -1.608293 | 0.997794
Pablo Picasso | 243 | 0.159851 | 8.005139 | 0.000100 | 0.056925 | 0.935740
Armenia | 207 | 0.119469 | 4.069365 | 0.000100 | -0.579203 | 0.989609
Welsh Corgi | 81 | 0.147368 | 2.910857 | 0.000144 | 1.746138 | 0.064552
Salvador Dali | 227 | 0.070039 | 3.621298 | 0.000095 | 0.689925 | 0.750325
David Bowie | 74 | 0.144578 | 3.080935 | 0.000093 | -1.446540 | 0.987260
Marie Curie | 83 | 0.195652 | 0.544324 | 0.745235 | -0.540352 | 0.968666
Table 3:  Detecting a few human edits in Wikipedia-style articles written by
ChatGPT . The table shows the value of the HC statistic (4) and the P-value of
the HC test based on simulated values as in Figure 4; P-values of HC tests
that are significant at level $\alpha=0.05$ are in bold. Also shown is the
fraction of edited sentence $\epsilon$ in every document, the value of the HC
statistics on the document before it has been edited, and the precision and
recall of sentences provided by the HC threshold (6).

###  3.2 Realistic edited documents

In Table 3 we report on the results of applying our method to 8 articles that
were created via the following process: we initially asked ChatGPT 111Via the
web interface https://chat.openai.com/; throughout April 2023. to help us
write a Wikipedia-style article about the particular topic, iteratively
prompting it to fill sections based on real section titles in the Wikipedia
article. We used all the text written by ChatGPT and the real section
subtitles to form a coherent article which we denote as the pre-edited ChatGPT
article. Next, we asked a human editor to go over the text and modify it by
adding, rephrasing, or removing entire sentences. We applied our method both
to the edited and non-edited articles, where we used sentences from additional
articles created in a similar manner to characterize
$\bar{F}_{\mathrm{G}_{0},\mathrm{P}}$.

The results in Table 3 show that all edited articles have larger HC values
compared to their non-edited version. We also report on the P-values
associated with these HC values under the null of uniformly distributed
P-values obtained in a way explained in Figure 4. Out of 8 articles we
created, 6 have HC values that are significantly large at the level $0.05$,
while none of the pre-edited articles has a P-value that is significant at
that level. We note that although this observation supports the effectiveness
of the method, a much larger sample of articles is needed to verify that a
test based on these P-values is indeed of the size prescribed due to possible
violations of the independent sentences model (5).

##  4 Information-Theoretic Analysis

In this section, we explain the optimality of HC under a simple generative
model of text editing and discuss the factors affecting the power of the
perplexity test.

###  4.1 Optimality of the Higher Criticism test

A simple mixture model for the generation of an edited document proposes that
most sentences are written independently by a GLM $\mathrm{G}_{0}$, except
perhaps a few sentences that are generated by a different mechanism associated
with the editor that we denote here by $\mathrm{G}_{1}$. Importantly, we do
not know in advance which sentences were written by each model. Let $\epsilon$
denote the expected number of edited sentences, and let $L_{j}$ be the
distribution of $\mathsf{lppt}(S;\mathrm{P})$ under $S\sim G_{j}$, for
$j\in\\{0,1\\}$. The setting described above induces a mixture model for the
log perplexity

$\displaystyle\begin{split}H_{0}\,&:\,\mathsf{lppt}(S_{i};\mathrm{P})\overset{\mathsf{iid}}{\sim}L_{0},\qquad
i=1,\ldots,n,\\\
H_{1}\,&:\,\mathsf{lppt}(S_{i};\mathrm{P})\overset{\mathsf{iid}}{\sim}(1-\epsilon)\cdot
L_{0}+\epsilon\cdot L_{1},\quad i=1,\ldots,n.\end{split}$ (7)

Likewise, we have a mixture model for the P-values in (3):

$\displaystyle H_{0}\,$
$\displaystyle:\,p_{i}\overset{\mathsf{iid}}{\sim}\mathrm{Unif}(0,1),\qquad
i=1,\ldots,n,$ (8a) $\displaystyle H_{1}\,$
$\displaystyle:\,p_{i}\overset{\mathsf{iid}}{\sim}(1-\epsilon)\cdot\mathrm{Unif}(0,1)+\epsilon\cdot
Q_{i},\quad i=1,\ldots,n,$ (8b)

where here $Q_{i}$ is a sub-uniform distribution that describes the non-null
behavior of the P-values (3). The optimality of HC for mixture model of the
form (7) and (8) have been studied in several contexts [DJ04, Jin03, HJ08,
CW14, ACW15, MPL15, JK16, DK22, Kip23]. In particular, when the mixture
parameter is calibrated to $n$ as $\epsilon=n^{-\beta}$, for some
$\beta\in(1/2,1)$, and the effect size in $Q_{i}$ is moderately large, a test
based on HC of $p_{1},\ldots,p_{n}$ attains the information-theoretic limit of
detection in (8) when $n\to\infty$ [Kip23]. Namely, in a configuration of the
calibrating parameters in which there exists a test of asymptotically non-
trivial power, there exists a test based on HC that is asymptotically powerful
in the sense that its power tends to one while its size tends to zero.

The works of [DH09, HJ08, HJ10] extended the optimality properties of HC to
some situations of non-independent individual effects, unlike in the model
(8). One relevant conclusion from these works is that HC is relatively
unaffected when the P-values experience a form of short-term dependency as
expected among sentences.

###  4.2 Optimality properties of the perplexity test

The perplexity test of (3) is the underlying engine of the detection method.
The justification for using it is primarily its empirical success in
separating the two kinds of texts as shown in Figure 3. In this section, we
provide an information-theoretic analysis of the factors affecting the power
of the perplexity test.

####  4.2.1 Language model as an information source and the asymptotic
perplexity

Let $\mathrm{P}_{a}$ be a language model. Sampling a sentence
$t_{1:n}=(t_{1},\ldots,t_{n})$ form $\mathrm{P}_{a}$ is achieved by causally
conditioning each token by previously sampled tokens and an initial context.
Namely,

$\displaystyle
t_{i}\sim\mathrm{P}_{a}(\cdot|t_{0},t_{1:i-1})\mathrm{P}_{a}(t_{0}),\quad
i=1,\ldots,n,$ (9)

for some initial state $t_{0}$ that can represent the initial context. We view
$\mathrm{P}_{a}$ as an information source in the sense that it defines a
stationary distribution over sequences of tokens from a finite alphabet
[Sha48]. When $\mathrm{P}_{a}$ is ergodic, the Shannon-McMillan-Brieman
theorem says that the entropy rate $\mathcal{H}(\mathrm{P})$ is well-defined
by the limit [AC88]

$\mathcal{H}(\mathrm{P}_{a})=\lim_{n\to\infty}\mathsf{lppt}(t_{1:n};\mathrm{P}_{a}).$

Furthermore, this limit is independent222We can extend much of the analysis to
a situation where the source is not necessarily ergodic. In this case, the
limiting value depends on the initial state [KSS77]. of $t_{0}$. In view of
Shannon’s source coding theorem, we may also define the entropy rate
operationally as the minimum number of expected bits per token needed to
represent $t_{1:n}$ as $n\to\infty$ [CK78]. This operational definition is
valid even if $\mathrm{P}_{a}$ is not ergodic, provided the expectation is
also over the initial state $t_{0}$ [Gra89].

More generally, suppose that we evaluate the LPPT with respect to another
stationary probability law $\mathrm{P}_{b}$ defined over the same alphabet as
$\mathrm{P}_{a}$. Under some conditions on the laws
$(\mathrm{P}_{a},\mathrm{P}_{b})$, the limit of
$\mathsf{lppt}(t_{1:n};\mathrm{P})$ as $n\to\infty$ exists almost surely and
obeys

$\displaystyle\lim_{n\to\infty}\mathsf{lppt}(t_{1:n};\mathrm{P}_{b})=\mathcal{H}(\mathrm{P}_{b};\mathrm{P}_{a})=\mathcal{H}(\mathrm{P}_{a})+\mathrm{\bar{D}}(\mathrm{P}_{a}||\mathrm{P}_{b}),$
(10)

where $\mathrm{\bar{D}}(\mathrm{P}_{a}||\mathrm{P}_{b})$ is the relative
entropy rate of $\mathrm{P}_{b}$ to $\mathrm{P}_{a}$ [Gra11, Ch. 7]. The term
$\mathcal{H}(\mathrm{P}_{b};\mathrm{P}_{a})$ is denoted as the crossentropy
rate of $\mathrm{P}_{b}$ under the law $\mathrm{P}_{a}$. Similarly to the
entropy rate, the crossentropy rate also admits an operational definition as
the expected number of bits per token used in representing a long sequence
from $\mathrm{P}_{a}$ using a binary code that, if applied to long sequences
from $\mathrm{P}_{b}$, would converge to $\mathcal{H}(\mathrm{P}_{a})$ [COR98,
GL03]. A practical lossy compression method that presumably asymptotically
attains an expected code length $\mathcal{H}(\mathrm{P}_{b};\mathrm{P}_{a})$
is an arithmetic encoder applied to every token with interval partitions based
on $\mathrm{P}_{b}$. An extension of this compression method that was
implemented in [IJG19] is known to attain state-of-the-art results on text
compression [Mah23].

In the section below, we discuss implications of the asymptotic representation
(10) to the power of the LPPT test for for testing $H_{0,S}$ versus a simple
alternative

$\displaystyle H_{1,S}\,:\,S\sim\mathrm{G}_{1}$ (11)

for some information source $\mathrm{G}_{1}$ that represents the effect of
editing the sentence $S$.

####  4.2.2 Desirable properties of $\mathrm{P}$

We now treat $\mathrm{G}_{0}$ and $\mathrm{G}_{1}$ as determined by “Nature”
and seek a good model $\mathrm{P}$ to use in the perplexity detector (3) to
maximize the power of the perplexity test. As we explain next, under
reasonable assumptions the ideal $\mathrm{P}$ maximizes the difference

$\displaystyle\Delta(\mathrm{G}_{1},\mathrm{G}_{0};\mathrm{P}):=\mathrm{\bar{D}}(\mathrm{G}_{1}||\mathrm{P})-\mathrm{\bar{D}}(\mathrm{G}_{0}||\mathrm{P}),$
(12)

where $\mathrm{\bar{D}}$ indicates the relative entropy rate of information
sources [Gra11]. We discuss possible implications of (12) in Section 5.2
below.

Let $\mathcal{F}$ denote some scale-location family of unimodal continuous
distributions. Consider the following assumptions.

  * (A1)

Under $H_{S,0}$, $\mathsf{lppt}(S;\mathrm{P})$ converges in distribution to a
member of $\mathcal{F}$ with mean $\mathcal{H}(\mathrm{P};\mathrm{G}_{0})$ and
scale $\sigma_{0}$.

  * (A2)

Under $H_{S,1}$, $\mathsf{lppt}(S;\mathrm{P})$ converges in distribution to a
member of $\mathcal{F}$ with mean $\mathcal{H}(\mathrm{P};\mathrm{G}_{1})$ and
scale $\sigma_{1}$.

  * (A3)

The asymptotic scales $\sigma_{0}$ and $\sigma_{1}$ are independent of
$\mathrm{P}$.

Recall that

$\bar{F}_{\mathrm{G}_{0};\mathrm{P}}(x)=\Pr_{S\sim\mathrm{G}_{0}}\left[\mathsf{lppt}(S;\mathrm{P})\geq
x\right],$

is a deterministic function of $x$ returning the P-value of the perplexity
test (1) under the LM $\mathrm{P}$. Let $G_{0}$ and $G_{1}$ be two stationary
ergodic sources such that $\mathrm{\bar{D}}(G_{1}||G_{0})$ exists and is
finite. Suppose that $\mathcal{P}$ is a set of stationary ergodic sources such
that for any $\mathrm{P}\in\mathcal{P}$, the relative entropy rates
$\mathrm{\bar{D}}(G_{1}||P)$ and $\mathrm{\bar{D}}(G_{0}||P)$ exist and
finite. We have the following claim.

######  Proposition 0.1.

Assume A1-A3. Suppose that

$\mathrm{P}^{*}\in\arg\max_{\mathrm{P}\in\mathcal{P}}\Delta(\mathrm{G}_{1},\mathrm{G}_{0};\mathrm{P}),$

and $S_{n}\sim\mathrm{G}_{1}$ of length $n$. Then

$\lim_{n\to\infty}\bar{F}_{\mathrm{G}_{0};\mathrm{P}^{*}}\left(\mathsf{lppt}(S_{n};\mathrm{P}^{*})\right)\leq\lim_{n\to\infty}\bar{F}_{\mathrm{G}_{0};\mathrm{P}}\left(\mathsf{lppt}(S_{n};\mathrm{P})\right),\quad\mathrm{P}\in\mathcal{P}.$

In words, Proposition 0.1 says that the smallest P-value (and thus the
strongest effect) is obtained when $\mathrm{P}$ maximizes the difference
$\Delta(\mathrm{G}_{1},\mathrm{G}_{0};\mathrm{P})$ over LMs $\mathrm{P}$ in
$\mathcal{P}$. In Section 5.2, we explain how this statement may guide the
search for an optimal $\mathrm{P}$ for the perplexity test (3).

#### Proof of Proposition 0.1

By A1, we have the asymptotic relation

$\displaystyle\bar{F}_{\mathrm{G}_{0};\mathrm{P}}(x)$
$\displaystyle=\Pr_{S\sim\mathrm{G}_{0}}\left[\frac{\mathsf{lppt}(S;\mathrm{P})-\mathcal{H}(\mathrm{P};\mathrm{G}_{0})}{\sigma_{0}}\geq\frac{x-\mathcal{H}(\mathrm{P};\mathrm{G}_{0})}{\sigma_{0}}\right]$
$\displaystyle\bar{F}_{0}\left(\frac{x-\mathcal{H}(\mathrm{P};\mathrm{G}_{0})}{\sigma_{0}}\right)(1+o(1)),\quad\bar{F}_{0}(x)=1-F_{0}(x),$

where here $o(1)$ represents a sequence tending to zero as $n\to\infty$.
Denote by $Z$ the random variable with distribution $\Pr(Z\leq x)=F_{0}(x)$.
By A2, we have under $H_{S,1}$ the equality in distribution:

$\mathsf{lppt}(S;\mathrm{P})+o_{p}(1)\overset{D}{=}\mathcal{H}(\mathrm{P};\mathrm{G}_{1})+\sigma_{1}Z,$

where $o_{p}(1)$ represents a sequence of random variables that converges in
distribution to zero as $n$ tends to infinity. The effect of an edited
sentence $S\sim\mathrm{G}_{1}$ is summarized by the P-value (3) obeying

$\displaystyle p(S)$
$\displaystyle:=\bar{F}_{\mathrm{G}_{0};\mathrm{P}}(\mathsf{lppt}(S;\mathrm{P}))$
$\displaystyle=\bar{F}_{0}\left(\frac{\mathsf{lppt}(S;\mathrm{P})-\mathcal{H}(\mathrm{P};\mathrm{G}_{0})}{\sigma_{0}}\right)$
$\displaystyle\overset{D}{=}\bar{F}_{0}\left(\frac{\mathcal{H}(\mathrm{P};\mathrm{G}_{1})-\mathcal{H}(\mathrm{P};\mathrm{G}_{0})}{\sigma_{0}}+\frac{\sigma_{1}}{\sigma_{0}}Z\right)(1+o_{p}(1)).$
(13)

The relation (13) shows that the effect is influenced by a location shift as
well as possible heteroscedasticity, i.e., when $\sigma_{1}\neq\sigma_{0}$.
Both factors affect the ability to detect the global null in (7) in several
ways [CJJ11, ACH20, Kip23]. Our Assumption A3 simplifies the situation by
saying that $\mathrm{P}$ only affects the numerator in the location shift
term. Becasue we assume that the distribution represented by $F_{0}(x)$ is
unimodal, we ought to maximize this shift term in order to obtain $p(S)$ as
small as possible hence an effect as large as possible. This numerator is
given by

$\displaystyle\mathcal{H}(\mathrm{P};\mathrm{G}_{1})-\mathcal{H}(\mathrm{P};\mathrm{G}_{0})=\mathcal{H}(\mathrm{G}_{1})-\mathcal{H}(\mathrm{G}_{0})+\mathrm{\bar{D}}(\mathrm{G}_{1}||\mathrm{P})-\mathrm{\bar{D}}(\mathrm{G}_{0}||\mathrm{P}).$

Only the last two terms depends on $\mathrm{P}$ hence the claim follows. ∎

The analysis above relies on the properties of the LPPT in the asymptotic of
large sentence length. However, the number of tokens in an actual sentence may
be too small for observing the limiting behavior (10). In this case, the
perplexity detector may be significantly affected by the sentence’s context,
hence it seems beneficial to incorporate this context. We discuss this point
as well as additional open challenges in Section 5.1 below.

##  5 Open Challenges and Future Work

###  5.1 Incorporating context

Typically, in practice, a sentence written by a GLM depends on the previous
sentence or another context affecting the GLM’s state. The effect of the
context on the perplexity may be quite significant due to the lack of
ergodicity and slow convergence of the LPPT to its limiting value. For this
reason, it appears that incorporating a context in the LPPT evaluations may
increase the power of the perplexity detector over individual sentences.
Specifically, denote the LPPT of a sentence $S=(t_{1},\ldots,t_{|S|})$ and
context $C$ as

$\displaystyle\mathsf{lppt}(S;C,\mathrm{P}):=-\frac{1}{|S|}\sum_{i=1}^{|S|}\log\mathrm{P}(t_{i}|t_{1:i-1},C).$
(14)

The context $C$ is usually a sequence of tokens, e.g., the sentence preceding
$S$, although it may also take other forms such as the activations of the
attention mechanism in transformers-based language models [JM23, Chapter 11].

If the policy by which $C$ is determined is also stationary (e.g., the
preceding sentence policy), we can extend much of the analysis in Section 4 to
use (14) instead of (1).

###  5.2 Maximizing the power of the perplexity detector

Our analysis in Section 4 shows that the power of the perplexity detector is
proportional to the difference in relative entropy
$\Delta(\mathrm{G}_{1},\mathrm{G}_{0};\mathrm{P})$ of (12). The information
projection principle [CM03, CT06] may provide an interesting viewpoint on the
maximization of this difference. Informally, suppose that we search for an
ideal $\mathrm{P}$ within a set $\mathcal{P}$ of available models. Because
GLMs are typically created to mimic human writing, we anticipate that our
search space only includes models inferior to the candidate GLM in the sense
that

$\displaystyle\mathrm{\bar{D}}(\mathrm{G}_{1}||\mathrm{G}_{0})\leq\mathrm{\bar{D}}(\mathrm{G}_{1}||\mathrm{P}),\qquad\mathrm{P}\in\mathcal{P}.$
(15)

The information projection principle suggests that [CT06, Ch. 11]

$\mathrm{\bar{D}}(\mathrm{G}_{1}||\mathrm{P})\geq\mathrm{\bar{D}}(\mathrm{G}_{1}||\mathrm{G}_{0})+\mathrm{\bar{D}}(\mathrm{G}_{0}||\mathrm{P}),\qquad\mathrm{P}\in\mathcal{P},$

and thus

$\displaystyle\Delta(\mathrm{G}_{1},\mathrm{G}_{0};\mathrm{P})\geq\mathrm{\bar{D}}(\mathrm{G}_{1};\mathrm{G}_{0}),\qquad\mathrm{P}\in\mathcal{P}.$

The last inequality is attained with equality when
$\mathrm{P}=\mathrm{G}_{0}$, implying that this choice of $\mathrm{P}$ is the
worst choice over models with the property (15). Specifically, a good choice
of $\mathrm{P}$ should also consider the relative entropy to the alternative
model $\mathrm{G}_{1}$. The characterization of such an alternative model in
applications appears to be challenging, although the relative entropy can be
approximately evaluated using standard methods, e.g. via the excessive binary
code length in lossless compression [COR98, GL03].

###  5.3 Generalizations

It may be useful to generalize the two main steps of our method in Algorithm 1
to address other closely-related use cases.

####  5.3.1 Generalizing Step I: Testing text atoms individually

Our method uses sentences as text atoms and considers their LPPT. Natural
generalizations of this step include the considerations of other text atoms
like paragraphs, as well as detectors that are not necessarily based on the
perplexity, e.g., probability curvature [MLK+23] or word-frequencies [MW12].

####  5.3.2 Generalizing Step II: Inference based on multiple testing

Our method uses HC for testing the global significance of individual tests.
This choice is motivated by the rare editing model over sentences and the
sensitivity of HC to rare effects. Under deviations from this model or due to
other considerations, methods from multiple comparisons in statistics other
than HC may be preferable [Ben10, Efr12]. Specifically, instead of HC, we may
combine P-values using Fisher’s method

$\displaystyle F_{n}:=F_{n}(p_{1},\ldots,p_{n}):=-2\sum_{i=1}^{n}\log(p_{i}).$
(16)

$F_{n}$ is known to be effective in detecting many relatively frequent but
potentially very faint effects [ACCP11, Kip23]. Therefore, $F_{n}$ can be used
when we test $H_{0}$ of (5) against an alternative specifying that the machine
text has gone through many edits but each edit is potentially so minor that it
increases very little the perplexity.

Another alternative to inference based on HC occurs when we are interested in
selecting a set of suspected edits with some control over the probability of
falsely reporting an edit. In this case, we may apply Benjamini-Hochberg (BH)
false discovery rate (FDR) controlling procedure to the P-values in (3)
[BH95]. We note that the BH procedure is in general less powerful for global
testing than HC. Namely, HC may find the body of P-values significant, while
the BH procedure with an FDR parameter $\alpha$ will report on an empty set of
P-values with probability at least $1-\alpha$, for every $\alpha\in(0,1)$
[Kip23].

## References

  * [AAA+23] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  Falcon-40B: an open large language model with state-of-the-art performance.  2023\.
  * [Aad23] Aaditya Bhat.  Gpt-wiki-intro (revision 0e458f5), 2023.
  * [AC88] Paul H Algoet and Thomas M Cover.  A sandwich proof of the shannon-mcmillan-breiman theorem.  The Annals of Probability, 16(2):899–909, 1988.
  * [ACCP11] Ery Arias-Castro, Emmanuel J Candès, and Yaniv Plan.  Global testing under sparse alternatives: Anova, multiple comparisons and the higher criticism.  The Annals of Statistics, 39(5):2533–2556, 2011.
  * [ACH20] Ery Arias-Castro and Rong Huang.  The sparse variance contamination model.  Statistics, 54(5):1081–1093, 2020.
  * [ACW15] Ery Arias-Castro and Meng Wang.  The sparse Poisson means model.  Electronic Journal of Statistics, 9(2):2170–2201, 2015.
  * [Ben10] Yoav Benjamini.  Simultaneous and selective inference: Current successes and future challenges.  Biometrical Journal, 52(6):708–721, 2010.
  * [BH95] Yoav Benjamini and Yosef Hochberg.  Controlling the false discovery rate: a practical and powerful approach to multiple testing.  Journal of the royal statistical society. Series B (Methodological), pages 289–300, 1995.
  * [CJJ11] Tony T. Cai, Jessie X. Jeng, and Jiashun Jin.  Optimal detection of heterogeneous and heteroscedastic mixtures.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73(5):629–662, 2011.
  * [CK78] Thomas Cover and Roger King.  A convergent gambling estimate of the entropy of english.  IEEE Transactions on Information Theory, 24(4):413–421, 1978.
  * [CM03] Imre Csiszár and Frantisek Matus.  Information projections revisited.  IEEE Transactions on Information Theory, 49(6):1474–1490, 2003\.
  * [COR98] G. Cybenko, D.P. O’Leary, and J. Rissanen.  The Mathematics of Information Coding, Extraction and Distribution.  The IMA Volumes in Mathematics and its Applications. Springer New York, 1998.
  * [CT06] TM Cover and Joy A Thomas.  Elements of information theory, 2006.
  * [CW14] Tony T Cai and Yihong Wu.  Optimal detection of sparse mixtures against a given null distribution.  IEEE Transactions on Information Theory, 60(4):2217–2232, 2014\.
  * [DH09] Aurore Delaigle and Peter Hall.  Higher criticism in the context of unknown distribution, non-independence and classification.  In Perspectives in mathematical sciences I: Probability and statistics, pages 109–138. World Scientific, 2009.
  * [DJ04] David L. Donoho and Jiashun Jin.  Higher criticism for detecting sparse heterogeneous mixtures.  The Annals of Statistics, 32(3):962–994, 2004.
  * [DJ08] David L Donoho and Jiashun Jin.  Higher criticism thresholding: Optimal feature selection when useful features are rare and weak.  Proceedings of the National Academy of Sciences, 105(39):14790–14795, 2008.
  * [DJ09] David L. Donoho and Jiashun Jin.  Feature selection by higher criticism thresholding achieves the optimal phase diagram.  Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906):4449–4470, 2009.
  * [DJ15] David L. Donoho and Jiashun Jin.  Special invited paper: Higher criticism for large-scale inference, especially for rare and weak effects.  Statistical Science, pages 1–25, 2015.
  * [DK22] David L Donoho and Alon Kipnis.  Higher criticism to compare two large frequency tables, with sensitivity to possible rare and weak differences.  The Annals of Statistics, 50(3):1447–1472, 2022.
  * [Efr12] Bradley Efron.  Large-scale inference: empirical Bayes methods for estimation, testing, and prediction, volume 1.  Cambridge University Press, 2012.
  * [GL03] Robert M Gray and Tamás Linder.  Mismatch in high-rate entropy-constrained vector quantization.  IEEE Transactions on Information Theory, 49(5):1204–1217, 2003\.
  * [Gra89] Robert M Gray.  Source coding theory, volume 83.  Springer Science & Business Media, 1989.
  * [Gra11] Robert M Gray.  Entropy and information theory.  Springer Science & Business Media, 2011.
  * [HJ08] Peter Hall and Jiashun Jin.  Properties of higher criticism under strong dependence.  The Annals of Statistics, 36(1):381–402, 2008.
  * [HJ10] Peter Hall and Jiashun Jin.  Innovated higher criticism for detecting sparse signals in correlated noise.  The Annals of Statistics, 38(3):1686–1732, 2010.
  * [IJG19] Gautier Izacard, Armand Joulin, and Edouard Grave.  Lossless data compression with transformer.  2019\.
  * [Jan00] Arnold Janssen.  Global power functions of goodness of fit tests.  The Annals of Statistics, 28(1):239–253, 2000.
  * [JHN23] Maurice Jakesch, Jeffrey T Hancock, and Mor Naaman.  Human heuristics for ai-generated language are flawed.  Proceedings of the National Academy of Sciences, 120(11):e2208839120, 2023.
  * [Jin03] Jiashun Jin.  Detecting and estimating sparse mixtures.  PhD thesis, Stanford University, 2003.
  * [JK16] Jiashun Jin and Zheng Tracy Ke.  Rare and weak effects in large-scale inference: methods and phase diagrams.  Statistica Sinica, pages 1–34, 2016.
  * [JM23] Daniel Jurafsky and James H. Martin.  Speech and Language Processing.  2023\.  third edition draft.
  * [Kip23] Alon Kipnis.  Unification of rare/weak detection models using moderate deviations analysis and log-chisquared p-values.  Statistica Sinica, 2023.  to appear.
  * [KSK+23] Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer.  Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense.  arXiv preprint arXiv:2303.13408, 2023.
  * [KSS77] AB Katok, Ya G Sinai, and AM Stepin.  Theory of dynamical systems and general transformation groups with invariant measure.  Journal of Soviet Mathematics, 7(6):974–1065, 1977.
  * [LYM+23] Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou.  Gpt detectors are biased against non-native english writers.  arXiv preprint arXiv:2304.02819, 2023.
  * [Mah23] Matt Mahoney.  Large text compression benchmark, 2023.
  * [MF98] Neri Merhav and Meir Feder.  Universal prediction.  IEEE Transactions on Information Theory, 44(6):2124–2147, 1998\.
  * [MLK+23] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  arXiv preprint arXiv:2301.11305, 2023.
  * [MPL15] Rajarshi Mukherjee, Natesh S Pillai, and Xihong Lin.  Hypothesis testing for high-dimensional sparse binary regression.  Annals of Statistics, 43(1):352, 2015.
  * [MW12] Frederick Mosteller and David L Wallace.  Applied Bayesian and classical inference: the case of the Federalist papers.  Springer Science & Business Media, 2012.
  * [Nic23] Nicolai Thorer Sivesind.  Chatgpt-generated-abstracts, 2023.
  * [NSM23] Oded Nov, Nina Singh, and Devin Mann.  Putting chatgpt’s medical advice to the (turing) test: Survey study.  JMIR Medical Education, 9:e46939, 2023.
  * [Ope22] OpenAI.  Chatgpt: Optimizing language models for dialogue, 2022.
  * [Ope23] OpenAI.  Gpt-4 technical report, 2023.
  * [RWC+19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [Sar23] Sarthak anand.  Chatgpt cnn / dailymail dataset, 2023.
  * [Sha48] Claude E Shannon.  A mathematical theory of communication.  The Bell system technical journal, 27(3):379–423, 1948.
  * [Sha51] Claude E Shannon.  Prediction and entropy of printed english.  Bell system technical journal, 30(1):50–64, 1951.
  * [SKB+23] Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi.  Can ai-generated text be reliably detected?  arXiv preprint arXiv:2303.11156, 2023.
  * [TLI+23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [ZHR+19] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi.  Defending against neural fake news.  Advances in neural information processing systems, 32, 2019.