# LLM-as-a-Coauthor: The Challenges of Detecting LLM-Human Mixcase

Chujie Gao LAIR Lab, Lehigh University  Dongping Chen LAIR Lab, Lehigh
University  Huazhong University of Science and Technology
{gaochujie1107, dongpingchen0612<EMAIL_ADDRESS>Qihui Zhang LAIR Lab,
Lehigh University  Yue Huang LAIR Lab, Lehigh University
Yao Wan Huazhong University of Science and Technology
{gaochujie1107, dongpingchen0612<EMAIL_ADDRESS>Lichao Sun LAIR Lab,
Lehigh University

###### Abstract

With the remarkable development and widespread applications of large language
models (LLMs), the use of machine-generated text (MGT) is becoming
increasingly common. This trend brings potential risks, particularly to the
quality and completeness of information in fields such as news and education.
Current research predominantly addresses the detection of pure MGT without
adequately addressing mixed scenarios including AI-revised Human-Written Text
(HWT) or human-revised MGT. To confront this challenge, we introduce mixcase,
a novel concept representing a hybrid text form involving both machine-
generated and human-generated content. We collected mixcase instances
generated from multiple daily text-editing scenarios and composed MixSet, the
first dataset dedicated to studying these mixed modification scenarios. We
conduct experiments to evaluate the efficacy of popular MGT detectors,
assessing their effectiveness, robustness, and generalization performance. Our
findings reveal that existing detectors struggle to identify mixcase as a
separate class or MGT, particularly in dealing with subtle modifications and
style adaptability. This research underscores the urgent need for more fine-
grain detectors tailored for mixcase, offering valuable insights for future
research. Code and Models are available at https://github.com/Dongping-
Chen/MixSet.

††footnotetext: † These authors contributed equally. The order was determined
by rolling dice: Chujie Gao rolled a 6, Dongping Chen rolled a 4, Qihui Zhang
rolled a 1.††footnotetext: ‡ Visting Students to LAIR Lab, Lehigh University.

##  1 Introduction

Figure 1: Three kinds of text: Machine Generative Text (MGT), Human Written
Text (HWT), and mixcase. The text come from users  is classified by detectors
. The text in red is the HWT polished by LLMs. Figure 2: Percentage of
identifying samples as MGT of different sets in Experiment 1. (Left) Model-
based methods; (Right) Metric-based methods. P.T. and P.S. signify token and
sentence-level polish, respectively; C. for complete, R. for rewrite; Adapt T.
and Adapt S. for token and sentence-level adapt. See 3 for details on revising
operations.

The rapid advancement of Large Language Models (LLMs) has ignited global
discussions on the effective utilization of AI assistants (OpenAI, 2022,
2023b). As LLMs accurately interpret and execute human instructions, an
increasing number of individuals integrate these powerful tools into their
workflows. They utilize LLMs to revise Machine Generated Text (MGT) or enhance
their Human Written Text (HWT), with applications ranging from fact-checking
revisions in journalism (Guerra, 2023) to storytelling enhancements in the
gaming industry (AIContentfy, 2023).

Despite its varied applications, the usage of MGT presents potential risks,
stirring public concerns over various forms of misuse. These include the
undermining of journalistic integrity and quality (Christian, 2023), the
reproduction and amplification of biases (Sison et al., 2023), plagiarism
among students (Heavenarchive, 2023), and disruptions in trust towards
scientific knowledge (Else, 2023). Such misuse of machine-generated text has
garnered serious attention and concern from experts across different fields
(Yurkevich, 2023), highlighting the need for balanced and responsible
deployment of these advanced technologies such as AIGC detection.

Numerous methods, including metric-based and model-based methods, have been
proposed in previous studies to detect MGT. These efforts primarily focus on
binary classification, aiming to distinguish between pure MGT and HWT.
However, these studies often overlook mixcase scenarios—revised texts
involving both MGT and HWT, which are frequently viewed either as attacks on
current detection systems (Krishna et al., 2023) or as significant challenges
for detection methodologies (Mitchell et al., 2023; Guo and Yu, 2023). Yet,
the mixture of MGT and HWT is a prevalent scenario in our daily interactions
with LLM assistants. For instance, many non-native English speakers use LLMs
to improve their drafts and address grammatical issues (Hwang et al., 2023).
Additionally, LLMs are being employed in creative industries, such as game
design for creating interactive dialogues and narratives (AIContentfy, 2023),
and in authoring tools like Metaphoria and Sparks for enhancing creative and
scientific writing Gero and Chilton (2019); Gero et al. (2022).

Hence, there is a pressing demand for a comprehensive analysis of these mixed
scenarios and a formal definition for its growing prevalence of mixcase in
both research fields and daily life. To address this issue, we introduce a
novel dataset, MixSet, categorizing mixcase based on the editing operations
proposed by Goyal et al. (2023). This is the first dataset targeting the
mixture of HWT and MGT, including both AI-revised HWT and human-revised MGT
scenarios, as detailed in Figure 1, which fills a gap in prior research.
Further details and definitions are provided in Section 3. We also evaluate
our dataset using mainstream detectors in both binary and three-class settings
to further analyze and raise concerns about these prevalent but challenging-
to-detect cases. Moreover, we conduct transfer experiments to retrained
detectors to analyze their generalizability on mixcase from different editing
operations and LLMs.

To summarize, our work provides three main contributions:

  * •

We defined mixcase, a form of mixed text involving both AI and human-generated
content, providing a new perspective for further exploration in related
fields.

  * •

We proposed a new dataset MixSet, which specifically addresses the mixture of
MGT and HWT which encompassed a diverse range of operations within real-world
scenarios, filling the blankness in previous research.

  * •

Based on MixSet, we conducted extensive experiments involving mainstream
detectors and obtained numerous insightful findings, which provide a strong
impetus for future research.

Figure 3: The process of MixSet generation. We perform distinct operations in
HWT and MGT. In HWT, three operations—polish, rewrite, and complete—are
employed. In MGT, we utilize LLama2 and GPT-4 to aid in humanization and
conduct the adaptation operation manually.

##  2 Related works

###  2.1 Machine Generated Text Detection

Current MGT detection methods can be broadly categorized into metric-based
methods and model-based methods according to previous study (He et al., 2023).
Please refer to Appendix A for comprehensive related works.

Metric-based Methods. Building upon the observation that MGTs occupy regions
with sharp negative log probability curvature, Mitchell et al. (2023)
introduced a zero-shot whitebox detection method called DetectGPT, setting a
trend in metric-based detection (Su et al., 2023; Mireshghallah et al., 2023;
Bao et al., 2023). Recently, Yang et al. (2023a) also introduced a powerful
detection method known as DNA-GPT, which leverages N-gram (Shannon, 1948) in a
black-box setting.

Model-based Methods. In the era of Large Language Models (LLMs), Guo et al.
(2023) developed the ChatGPT Detector, which is based on a fine-tuned Roberta
model. As for decoder-based detectors, GPT-Sentinel (Chen et al., 2023)
leverage the t5-small model (Muennighoff et al., 2022) and show convincing
results when detecting MGT even in revised cases. Leveraging adversarial
learning framework, Hu et al. (2023) propose a novel detection framework
called Radar, which employs an adversarial learning approach to simultaneously
train a detector based on Vicuna-7B (Chiang et al., 2023) and a pre-trained
paraphraser.

###  2.2 Datasets for MGT Detection

Previous studies have proposed many datasets of MGT, accompanied by their
newly proposed detectors (Verma et al., 2023; Chen et al., 2023). Guo et al.
(2023) leverages multiple previous Question-Answer (QA) datasets (Jin et al.,
2019; Lin et al., 2021), allowing ChatGPT to generate corresponding answers
without explicit prompts. This results in the creation of a comprehensive
dataset comprising a large set of pairs of MGT and HWT. Following the QA
pattern, many researchers (Mitchell et al., 2023; Su et al., 2023; Hu et al.,
2023; He et al., 2023) propose datasets with the MGT from variant mainstream
LLMs (OpenAI, 2022, 2023b).

However, these datasets typically consist of two distinct classes of texts,
namely pure MGT or HWT, without accounting for the potential mixture cases.
Furthermore, issues arise due to variations in prompts (Koike et al., 2023a),
sampling methods, and the inherent differences in length, style, and quality
among texts He et al. (2023), posing variations challenges on the
generalization ability of proposed detectors (Xu et al., 2023). In some
instances, MGT included in datasets may not undergo thorough careful checking,
with many noisy sentences not filtered well. For example, some sentences like
Let me know if you have any other questions exist in the dataset, which will
impact the effectiveness of the detectors (Guo et al., 2023).

##  3 Mixset Dataset

In this section, we present MixSet (Mixcase Dataset), the first dataset
featuring a blend of HWT and MGT. Distinguished from earlier datasets
exclusively composed of pure HWT and MGT, MixSet comprises a total of $3.6k$
mixcase instances. The pipeline of MixSet construction is illustrated in
Figure 3. Each editing operation in our MixSet, whether modified by an LLM or
manually, is based on real-world scenarios and contributes 300 instances.

For our base data, we meticulously select pure HWT and MGT datasets. In the
case of HWT, we gather datasets proposed before the widespread use of LLMs to
mitigate potential noise, as detailed in Table 2. For MGT, we choose samples
from previous datasets (Rajpurkar et al., 2016a; Lin et al., 2022; Nishida et
al., 2019), generated in a QA pattern by different LLMs, including the GPT
family (OpenAI, 2022, 2023b), ChatGLM (Du et al., 2022), BloomZ (Muennighoff
et al., 2022), Dolly (Conover et al., 2023), and StableLM (StabilityAI, 2023),
all distinct from our MixSet instances.

###  3.1 Definition of Mixcase

Generally, mixcase is the mixed text involving both AI and human-generated
content. To formulate it, we define a text sequence as $x\in X$, where $X$
represents the set of all text sequences. The sequences in $X$ can originate
from either human-written text $\mathbb{X}_{\text{human}}$ or machine-
generated text $\mathbb{X}_{\text{machine}}$. We denote the set of operations
used to revise texts as $\mathbb{OP}=\\{OP_{1},OP_{2},\dots,OP_{n}\\}$,
categorized into two groups: ${OP_{\text{human}},OP_{\text{machine}}}$. Here,
$OP_{\text{human}}$ refers to operations involving human revision on MGT,
while $OP_{\text{machine}}$ refers to AI-driven operations on HWT. In addition
to $\mathbb{X}_{\text{human}}$ and $\mathbb{X}_{\text{machine}}$, we define
$\mathbb{X}_{\text{mixcase}}$ as the union of all texts derived from
$\mathbb{X}_{\text{human}}$ through $OP_{\text{machine}}$ and all texts
derived from $\mathbb{X}_{\text{machine}}$ through $OP_{\text{human}}$:

$\displaystyle\mathbb{X}_{\text{mixcase}}=$
$\displaystyle\\{OP_{\text{machine}}(x)\,|\,x\in\mathbb{X}_{\text{human}}\\}\cup\\{OP_{\text{human}}(x)\,|\,x\in\mathbb{X}_{\text{machine}}\\}$

###  3.2 Dataset Construction

Combined with previous studies Goyal et al. (2023); Wang et al. (2021) and
real scenarios, we use five operations to generate mixcases. As shown in Table
2, they are divided into two kinds of operation: 1) AI-revised: it contains
three operations including ’polish’, ’complete’, and ’rewrite’. 2) Human-
revised: it includes ’humanize’ and ’adapt’.

  * •

Polish (Chen, 2023): Polish operation contains token-level and sentence-level
polishing. Token-level makes alterations at the individual word level,
including changes such as adjusting words for precision or correcting spelling
errors. On the other hand, sentence-level aims to enhance the overall
coherence and clarity of the text by revising and restructuring the complete
sentence.

  * •

Complete (Zhuohan Xie, ): Complete operation involves taking 1/3 of every text
and employing LLMs to generate the rest of the text.

  * •

Rewrite (Shu et al., 2023): Rewrite operation requires LLMs to initially
comprehend and extract key information from the given HWT and then rewrite
them.

  * •

Humanize (Bhudghar, 2023): Humanize operation typically refers to the
modification of MGT to more closely mimic the natural noise for LM Wang et al.
(2021) that human writing always brings. We employed LLMs to introduce various
perturbations to the pure MGT, including typo, grammatical mistakes, links,
and tags.

  * •

Adapt Gero et al. (2022): Adapt operation refers to modifying MGT to ensure
its alignment to fluency and naturalness to human linguistic habits without
introducing any error expression. The adapt operation is also divided into
token-level and sentence-level adaptation. We accordingly performed manual
annotations on the pure MGT dataset at both the token and sentence levels.

For AI-revised mixcase generation, we utilized Llama2-70b and GPT-4 since
these two high-performance LLMs can generate texts of high quality and few
grammatical errors Hugging Face (2023). In human-revised operation, we
leverage two LLMs to assist with the humanize operation. For the adapt
operation, we invite eight human experts with excellent English skills to
revise MGT carefully to make it better align with human expression way. The
details of human annotation guidelines and prompt template are shown in
Appendix B.1 and E. After collecting all revised texts, we conducted a manual
evaluation involving data filtering and cleaning to ensure the high quality of
our MixSet, such as removing conversational phrases like ‘Sure! Here’s a
possible completion’.

Table 1: The original resources of Human Written Texts in constructing our
MixSet.

Text Type |  Original Resources
---|---
Email Content |  Enron Email CMU (2015)
News Content |  BBC News (Greene and Cunningham, 2006)
Game Review |  Steam Reviews Najzeko (2021)
Paper Abstract |  ArXiv-10 Farhangi et al. (2022)
Speech Content |  TED Talk TheDataBeast (2021)
Blog content |  Blog Schler et al. (2006)

Table 2: Different operations with their operation levels. ✔ demonstrate that
MixSet contains a subset operates at that level.

Operation | Token | Sentence | Paragraph
---|---|---|---
AI-Polish | ✔ | ✔ | ✘
AI-Complete | ✘ | ✘ | ✔
AI-Rewrite | ✘ | ✘ | ✔
Human-Adapt | ✔ | ✔ | ✘
Humanize | ✔ | ✔ | ✔

Figure 4: Length distribution and self-BLEU scores of the MixSet.

###  3.3 Dataset Analysis

Our comprehensive analysis of the MixSet dataset covers length distribution,
self-BLEU (Zhu et al., 2018), Levenshtein distance (Levenshtein, 1966), and
cosine similarity. We only show analysis of length distribution and cosine
similarity analysis here; for self-BLEU and Levenshtein distance, refer to
Appendix B.

  * •

Length distribution: Given that LLMs generally perform better with medium to
long texts than with short texts, and to ensure that the text lengths in the
MixSet reflect real-world usage patterns, we have excluded texts that are
either too short or too long. As shown in Figure 4, the text lengths of both
the MixSet, as well as the HWT and MGT, follow a normal distribution.
Furthermore, the majority of the texts in these datasets fall within the range
of 50 to 200 words.

  * •

Cosine Similarity: Figure 4 illustrates that the texts processed with token-
level polish operations exhibit the highest similarity to the original texts,
followed by sentence-level polish, rewrite, and complete. Texts modified
through the humanize operation demonstrate lower similarity compared to those
altered by adaptation.

##  4 Experiments

###  4.1 Goals

We conduct experiments to understand multiple facets of current detectors when
encountering our MixSet, including zero-shot and fine-tuning settings. We are
going to figure out four questions:

  * •

Question 1. How do current detectors perform in MixSet dataset? Is there any
classification preference in these detectors?

  * •

Question 2. What is the performance of detectors retrained on our MixSet? What
about three-classed classification as we consider mixcase as a new class
distinct from HWT and MGT?

  * •

Question 3. What is the generalization ability of current detectors on our
MixSet?

  * •

Question 4. Will the size of the training set impact the detection ability on
mixcase?

###  4.2 Experiment Setup

Table 3: Detectors used in different experiments. Detector | Q 1 | Q 2 | Q 3 | Q 4
---|---|---|---|---
Metric-Based |  Log-likelihood (Solaiman et al., 2019) | ✔ | ✔ | ✘ | ✔
Entropy (Gehrmann et al., 2019) | ✔ | ✔ | ✘ | ✘
GLTR (Gehrmann et al., 2019) | ✔ | ✔ | ✘ | ✔
Log-Rank (Mitchell et al., 2023) | ✔ | ✔ | ✘ | ✘
DetectGPT (Mitchell et al., 2023) | ✔ | ✔ | ✔ | ✔
Model-Based |  Radar (Hu et al., 2023) | ✔ | ✔ | ✘ | ✔
ChatGPT Detector (Guo et al., 2023) | ✔ | ✔ | ✔ | ✔
DistillBert Ippolito et al. (2019) | ✔ | ✔ | ✔ | ✘
GPT-Sentinel (Chen et al., 2023) | ✔ | ✔ | ✘ | ✔
OpenAI Classifier (OpenAI, 2023a) | ✔ | ✘ | ✘ | ✘
Ghostbuster (Verma et al., 2023) | ✔ | ✘ | ✘ | ✘
GPTzero (Tian, 2023) | ✔ | ✘ | ✘ | ✘

Among our 4 experiments, We evaluate five metric-based detectors and seven
model-based detectors on three metrics in total as illustrated in Tabel 3 and
Table 5. We also list the summary of our training set in Figure 5. Please
refer to Appendix C for a comprehensive introduction to detectors and metrics.

Class Number. In real-world scenarios, people often aim to detect the presence
of MGT in the text (e.g., spreading fake news or propaganda (Christian, 2023),
reinforcing and intensifying prejudices (Sison et al., 2023)), and sometimes
mixcase is also treated as MGT (e.g., student modified some words in MGT
(i.e., mixcase) to generate homework, to avoid detection). Therefore, in our
experiments, we established two categorization systems: binary and three-
class. In the binary classification, mixcase is categorized as MGT, while in
the three-class classification, mixcase is treated as a separate class. The
label setting is shown in Table 5.

Question 1. Based on MixSet, we evaluate current detectors to determine the
classification preferences on mixcase, i.e., Does the detector tend to
classify mixcase as MGT or HWT? In the experiment, we calculate the percentage
of mixcase samples categorized to MGT. For the DistilBERT detector and other
metric-based detectors utilizing logistic regression models, we employ a
training set comprising 10,000 pre-processed samples of both pure HWT and MGT.
For other detectors, we utilize existing checkpoints
111https://huggingface.co/TrustSafeAI/RADAR-Vicuna-7B
222https://github.com/haok1402/GPT-Sentinel-public or API
333https://gptzero.me/ and evaluate them in a zero-shot setting.

Question 2(a). Following Question 1, our inquiry is whether the detector can
accurately classify mixcase as MGT after training on our MixSet. We fine-tune
detectors on pure HWT and MGT data, as well as a train split set of our MixSet
labeled as MGT. For detailed experiment settings, please refer to Appendix C.

Question 2(b). On the other hand, assuming that mixcase lies outside the
distribution of HWT and MGT, we conduct a three-class classification task,
treating mixcase as a new label. In this scenario, we adopt multi-label
training for these detectors, while keeping all other settings consistent.

Table 4: An outline of detailed training set construction for each experiment.
Ope. denotes ‘operation transfer’ in Experiment 3, while LLM refers to ‘LLM
transfer’.

Experiment | HWT/MGT | MixSet
---|---|---
Q 1 | $10k$ | $0$
Q 2(a) | $10k$ | $3k$
Q 2(b) | $10k$ | $3k$
Q 3(Ope.) | $1k$ | $0.5k$
Q 3(LLM) | $5k$ | $1.5k$
Q 4 | $1k/4k/7k/10k$ | $0/1.5k/3k$

Table 5: The details of class number, metrics, and whether the detectors are
retrained in our experiments. Except for Question 2(b), we implement binary
classifications i.e. HWT and MGT. Per. stands for Percentage.

Setting | Q 1 | Q 2 | Q 3 | Q 4
---|---|---|---|---
(a) | (b)
Class Num. | 2-Class | 2-Class | 3-Class | 2-Class | 2-Class
Metric | MGT Per. | F1, AUC | F1 | AUC | F1, AUC
Retrained? | ✘ | ✔ | ✔ | ✔ | ✔

c

Table 6: Percentage of identifying samples as MGT of different sets in
Experiment 1. For example, the Log-Rank detector categorizes 57.30% of samples
in the Llama2-revised set as MGT. We underscore the best-performing detector
and bold the score greater than $0.8$, which we consider as a baseline
threshold for detection (Tok. stands for token level and Sen. stands for
sentence level).

Detection Method |  HWT |  MGT | AI-Revised | Human-Revised
---|---|---|---|---
Rewrite | Complete | Polish-Tok. | Polish-Sen. | Humanize |  Adapt-Tok. |  Adapt-Sen.
Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4
Metric-based Detector
Log-Rank | 0.213 | 0.847 | 0.573 | 0.240 | 0.810 | 0.520 | 0.573 | 0.383 | 0.427 | 0.350 | 0.703 | 0.093 | 0.783 | 0.770
Log-likelihood | 0.223 | 0.867 | 0.600 | 0.287 | 0.823 | 0.560 | 0.643 | 0.450 | 0.513 | 0.410 | 0.703 | 0.083 | 0.790 | 0.777
GLTR | 0.207 | 0.840 | 0.480 | 0.180 | 0.813 | 0.393 | 0.517 | 0.283 | 0.390 | 0.313 | 0.630 | 0.053 | 0.783 | 0.760
DetectGPT | 0.350 | 0.823 | 0.643 | 0.343 | 0.743 | 0.557 | 0.650 | 0.480 | 0.563 | 0.437 | 0.807 | 0.533 | 0.623 | 0.597
Entropy | 0.353 | 0.840 | 0.733 | 0.580 | 0.793 | 0.623 | 0.793 | 0.730 | 0.713 | 0.640 | 0.737 | 0.223 | 0.793 | 0.770
Model-based Detector |  |
Openai Classifier | 0.060 | 0.607 | 0.150 | 0.047 | 0.407 | 0.037 | 0.123 | 0.037 | 0.103 | 0.053 | 0.023 | 0.007 | 0.490 | 0.453
ChatGPT Detector | 0.040 | 0.757 | 0.380 | 0.157 | 0.523 | 0.287 | 0.380 | 0.130 | 0.243 | 0.117 | 0.457 | 0.097 | 0.750 | 0.770
Radar | 0.307 | 0.857 | 0.730 | 0.477 | 0.893 | 0.783 | 0.607 | 0.447 | 0.560 | 0.387 | 0.347 | 0.037 | 0.850 | 0.890
GPT-Sentinel | 0.133 | 0.887 | 0.833 | 0.877 | 0.540 | 0.573 | 0.883 | 0.807 | 0.710 | 0.460 | 0.033 | 0.000 | 0.910 | 0.910
Distillbert | 0.483 | 0.993 | 0.593 | 0.660 | 0.530 | 0.573 | 0.607 | 0.580 | 0.547 | 0.527 | 0.170 | 0.003 | 1.000 | 1.000
Ghostbuster | 0.103 | 0.610 | 0.870 | 0.780 | 0.750 | 0.087 | 0.353 | 0.493 | 0.473 | 0.663 | 0.567 | 0.637 | 0.700 | 0.443
GPTZero | 0.017 | 0.730 | 0.493 | 0.167 | 0.810 | 0.177 | 0.497 | 0.260 | 0.777 | 0.763 | 0.717 | 0.187 | 0.720 | 0.497

Table 7: F1 score of experiment 2 (a) and (b). We underscore the best-
performing detector and bold the score greater than $0.8$, which we consider
as a baseline threshold for detection (Tok. stands for token level and Sen.
stands for sentence level).

Detection Method | AI-Revised | Human-Revised |  Average
---|---|---|---
Complete | Rewrite | Polish-Tok. | Polish-Sen. | Humanize |  Adapt-Sen. |  Adapt-Tok.
Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4
Experiment 2 (a): Binary Classification
Log-Rank | 0.695 | 0.686 | 0.637 | 0.479 | 0.617 | 0.606 | 0.647 | 0.595 | 0.617 | 0.454 | 0.676 | 0.667 | 0.615
Log Likelihood | 0.695 | 0.695 | 0.637 | 0.492 | 0.657 | 0.627 | 0.657 | 0.657 | 0.637 | 0.386 | 0.676 | 0.667 | 0.624
GLTR | 0.686 | 0.647 | 0.606 | 0.441 | 0.574 | 0.585 | 0.637 | 0.540 | 0.617 | 0.400 | 0.657 | 0.667 | 0.588
DetectGPT | 0.715 | 0.651 | 0.656 | 0.560 | 0.632 | 0.587 | 0.657 | 0.632 | 0.692 | 0.587 | 0.641 | 0.609 | 0.635
Entropy | 0.690 | 0.671 | 0.681 | 0.613 | 0.681 | 0.671 | 0.681 | 0.671 | 0.623 | 0.430 | 0.681 | 0.681 | 0.648
Openai Classifier | 0.171 | 0.359 | 0.031 | 0.197 | 0.145 | 0.270 | 0.247 | 0.439 | 0.247 | 0.316 | 0.000 | 0.090 | 0.209
ChatGPT Detector | 0.705 | 0.696 | 0.676 | 0.583 | 0.676 | 0.647 | 0.647 | 0.594 | 0.667 | 0.615 | 0.705 | 0.705 | 0.660
Radar | 0.867 | 0.877 | 0.877 | 0.877 | 0.877 | 0.877 | 0.877 | 0.877 | 0.877 | 0.877 | 0.877 | 0.877 | 0.876
GPT-Sentinel | 0.714 | 0.714 | 0.714 | 0.714 | 0.714 | 0.714 | 0.714 | 0.714 | 0.696 | 0.714 | 0.714 | 0.714 | 0.713
Distillbert | 0.667 | 0.667 | 0.667 | 0.667 | 0.667 | 0.667 | 0.667 | 0.667 | 0.639 | 0.667 | 0.667 | 0.667 | 0.664
Experiment 2 (b): Three-class Classification |
DetectGPT | 0.276 | 0.21 | 0.295 | 0.278 | 0.283 | 0.234 | 0.271 | 0.237 | 0.28 | 0.222 | 0.233 | 0.235 | 0.255
ChatGPT Detector | 0.288 | 0.346 | 0.283 | 0.288 | 0.395 | 0.341 | 0.265 | 0.328 | 0.267 | 0.317 | 0.253 | 0.273 | 0.304
Distillbert | 0.267 | 0.333 | 0.319 | 0.329 | 0.294 | 0.309 | 0.294 | 0.329 | 0.309 | 0.342 | 0.000 | 0.010 | 0.261

Question 3. As highlighted in prior research (Xu et al., 2023; He et al.,
2023) that transfer ability is crucial for detectors, our objective is to
investigate the effectiveness of transferring across different subsets of
MixSet and LLMs. We establish two transfer experiments to assess whether the
transferability of current detection methods is closely linked to the training
dataset, referred to as operation-generalization and LLM-generalization:

  * •

Operation-generalization: We initially train our detectors on one MixSet
subset operated by one of these operations, along with pure HWT and MGT
datasets, and then proceed to transfer it to the subsets processed by other
operations.

  * •

LLM-generalization: In this experiment, we train detectors on GPT-generated
texts and HWT, following which we evaluate the detectors on mixcase generated
by GPT family (OpenAI, 2023b) and Llama2 (Touvron et al., 2023), respectively,
to see whether there is a generalization gap between different LLMs.

Question 4. Empirically, incorporating more training data has been shown to
enhance detection capabilities, and improve robustness for generalization
(Ying, 2019; Rebuffi et al., 2021). To figure out the relation between
detectors’ performance and the size of the training set, we follow Question 2
and use varying sizes of training sets to retrain those detectors, as
illustrated in Table 5.

##  5 Empirical Findings

There is no obvious classification preference in current detectors on mixcase
with low consistency under different operations. And MixSet effectively
bridges the differences between HWT and MGT. As we can observe from Figure 2
and Table 6, it is evident that the MGT percentage of mixcases is between MGT
and HWT, indicating that the current detectors do not have a clear bias
towards mixcase classification. This proves the success and effectiveness of
our constructed MixSet in presenting mixed features of HWT and MGT,
demonstrating the limitations of existing detectors in recognizing mixcase.

Additionally, most detectors exhibit inconsistent classification within a
single subset, fluctuating between accuracies of 0.3 and 0.7, akin to random
choice. In AI-revised scenarios, subsets such as polished tokens or sentences,
pose extreme detection challenges. Mainstream detectors generally perform
badly in these cases due to the subtle differences between mixcase and
original text, highlighted in previous studies (Krishna et al., 2023).
Furthermore, texts generated by Llama2-70b are easier to detect compared to
those by GPT-4, possibly due to GPT-4’s closer generative distribution to
human.

(a) The AUC Heatmap of GPT-Sentinel.

(b) The AUC Heatmap of distilbert-based

Figure 5: The AUC Heatmap of the other two detectors.

Supervised binary classification yields profound results; however, three-
classes classification encounters significant challenges when applied to
mixcase scenarios. As indicated in Table 7, retrained model-based detectors
outperform metric-based methods in both binary and three-class classification
tasks. Notably, Radar ranks first on the detection leaderboard, achieving a
significant lead over other detectors. This superior performance can be
attributed to its encoder-decoder architecture, which boasts 7 billion
trainable parameters, substantially more than its counterparts. In contrast,
for the ternary classification task, the performance of current detectors is
akin to random guessing, with some even underperforming compared to a random
choice. This situation underscores the urgent need for the development of more
sophisticated detectors capable of executing a finer-grained classification of
mixcase.

Table 8: Result of LLM-transfer experiments. Although we retrain our detector
on texts generated by GPT-4, it shows convincing generalization ability to
Llama2.

Method | w.o MixSet | w. MixSet
---|---|---
Llama2 | GPT-4 | Llama2 | GPT-4
GPT-Sentinel | 0.813 | 0.739 | 0.972 | 0.971
Radar | 0.834 | 0.729 | 0.997 | 1.000
ChatGPT Det. | 0.664 | 0.445 | 0.681 | 0.480
Distillbert | 0.687 | 0.638 | 0.673 | 0.698

Current detectors struggle to generalize across different revised operation
subsets of MixSet and generative models. As shown in Figure 5 and Figure 11,
significant variability is observed in the transfer capabilities of three
different detectors. Additionally, training on texts generated by different
revised operations results in different transfer abilities for these
detectors. Overall, Radar exhibits the most robust transfer capability among
the four model-based detectors, achieving an overall classification accuracy
exceeding 0.9, followed by GPT-Sentinel, DistillBert, and finally the ChatGPT
Detector. Among various operations, ‘Humanize’ exhibits the poorest transfer
performance in almost all scenarios. Additionally, other operations also
experience significant declines when dealing with ‘Humanize’ mixcases. This
suggests that ‘Humanize’ falls outside the current detectors’ distribution of
MGT, a gap that could be addressed by retraining on these specific cases. It
is also noteworthy that texts generated by Llama2-70b demonstrate stronger
transfer abilities compared to those generated by GPT-4, as illustrated in
Table 5.

Increasing the number of mixcase samples in the training set effectively
enhances the success rate of mixcase detection. However, adding pure text
samples does not yield significant improvements and may even have a negative
impact on detector performance, especially for metric-based methods, as shown
in Figure 6. We believe this may be attributed to subtle distribution shifts
between mixcase and pure text. The current detector still faces significant
challenges in capturing these subtle shifts. For mixcase scenarios, a more
powerful and fine-grained detection method is needed.

Figure 6: Analysis of the F1-score performance of various detectors across
differing quantities of mixcase instances from MixSet, as well as pure MGT and
HWT.

##  6 Conclusion

In this paper, we defined mixcase, the mixed text of human and LLM-generated
content. Then we proposed a new dataset MixSet to address the research gap in
studying the mixed scenarios of machine-generated text (MGT) and human-written
text (HWT). We conducted a thorough evaluation of the dataset, performing
binary, three-class, and transfer experiments on mainstream detectors. The
results revealed that the detection of mixcase is challenging, indicating the
difficulty in discerning subtle differences in mixcase. As a result, there is
a need for more robust and fine-grained detection methods.

##  7 Limitation

Bias Introduced by Human Participation. Although our study involved multiple
human participants to modify the text, increasing the diversity and
authenticity of the data, the text processing methods of different
participants could vary due to their language habits and styles. This might
affect the representativeness of the dataset and the generalization ability of
the detection models.

Limitation in the Scale of the MixSet Dataset. As the MixSet dataset is the
first to be proposed for studying mixed texts (mixcase), and despite its wide
coverage in types, its overall scale is relatively small. This could limit the
comprehensiveness of model training and evaluation.

##  8 Ethics Statement

Opposition to Misuse of Mixed Text Scenarios. Our study highlights that the
mixcase of HWT and MGT could significantly diminish the discerning abilities
of detectors. However, we strongly oppose the misuse of mixcase to evade
detection in specific scenarios, such as during examinations and homework
assignments. We believe such misuse could severely harm the fairness of
education and the integrity of academic practices.

Purpose for Scientific Research. The aim of this study is purely for
scientific exploration and understanding of the behavior and impact of mixcase
in the field of natural language processing. Our goal is to enhance
understanding of mixed text processing and to advance the technological
development in this area, not to encourage or support applications that may
violate ethical standards.

Compliance with Licensing and Distribution Regulations. We affirm that all
open-source resources utilized in our study, including detectors, language
models, and datasets, have been employed in strict accordance with their
respective licenses and distribution terms. This adherence extends to ensuring
that any modifications, redistributions, or applications of these resources in
our research are compliant with their original licensing agreements. Our
commitment to these principles upholds the integrity of our research and
contributes to a responsible and ethical academic environment.

Use of Publicly Available Data and Consideration for Privacy. The datasets
used in our research are exclusively sourced from publicly available, open-
source collections. While these datasets are publicly accessible and generally
considered devoid of sensitive personal information, we acknowledge the
potential for inadvertent inclusion of personal identifiers in datasets. We
emphasize that our use of these datasets is aligned with their intended
purpose and distribution terms. We also recognize the importance of respecting
privacy and are committed to ongoing vigilance in this regard.

We reiterate that this research adheres to strict scientific and ethical
standards, aiming to contribute to the field of natural language processing
while ensuring that the results are not used for improper purposes. We also
encourage our peers to consider these ethical factors when utilizing our
research findings, ensuring that their applications do not adversely affect
society and individuals.

## References

  * (1) Google translate.  https://translate.google.com/.
  * (2) Grammarly.  https://www.grammarly.com/.
  * (3) Youdao translate.  http://fanyi.youdao.com/.
  * AIContentfy (2023) AIContentfy. 2023.  Chatgpt in the gaming industry: Enhancing storytelling and interaction.  https://aicontentfy.com/en/blog/chatgpt-in-gaming-industry-enhancing-storytelling-and-interaction.
  * Anand et al. (2023) Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. 2023.  Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo.  _GitHub_.
  * Bakhtin et al. (2019) Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019.  Real or fake? learning to discriminate machine from human generated text.  _arXiv preprint arXiv:1906.03351_.
  * Bao et al. (2023) Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023.  Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature.  _arXiv preprint arXiv:2310.05130_.
  * Bhudghar (2023) Gandhi Gram Bhudghar. 2023.  Ai text converter.  https://aitextconverter.com/.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  Natural language processing with python: Analyzing text with the natural language toolkit.  http://nltk.org/.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022.  Gpt-neox-20b: An open-source autoregressive language model.  _arXiv preprint arXiv:2204.06745_.
  * Chen (2023) Xuhang Chen. 2023.  Gpt academic prompt.  https://github.com/xuhangc/ChatGPT-Academic-Prompt.
  * Chen et al. (2023) Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Ramakrishnan. 2023.  Gpt-sentinel: Distinguishing human and chatgpt generated content.  _arXiv preprint arXiv:2305.07969_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Christian (2023) Jon Christian. 2023.  Cnet secretly used ai on articles that didn’t disclose that fact, staff say.  https://futurism.com/cnet-ai-articles-label.
  * CMU (2015) CMU. 2015.  Enron email dataset.  https://www.cs.cmu.edu/~enron/.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.  https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  Glm: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 320–335.
  * Else (2023) Holly Else. 2023.  Abstracts written by chatgpt fool scientists.  _Nature_ , 613(7944):423–423.
  * Farhangi et al. (2022) Ashkan Farhangi, Ning Sui, Nan Hua, Haiyan Bai, Arthur Huang, and Zhishan Guo. 2022.  Protoformer: Embedding prototypes for transformers.  In _Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16–19, 2022, Proceedings, Part I_ , pages 447–458.
  * Gehrmann et al. (2019) Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019.  Gltr: Statistical detection and visualization of generated text.  _arXiv preprint arXiv:1906.04043_.
  * Gero and Chilton (2019) Katy Ilonka Gero and Lydia B Chilton. 2019.  Metaphoria: An algorithmic companion for metaphor creation.  In _Proceedings of the 2019 CHI conference on human factors in computing systems_ , pages 1–12.
  * Gero et al. (2022) Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022.  Sparks: Inspiration for science writing using language models.  pages 1002–1019.
  * Ghosal et al. (2023) Soumya Suvra Ghosal, Souradip Chakraborty, Jonas Geiping, Furong Huang, Dinesh Manocha, and Amrit Singh Bedi. 2023.  Towards possibilities & impossibilities of ai-generated text detection: A survey.  _arXiv preprint arXiv:2310.15264_.
  * Goyal et al. (2023) Shreya Goyal, Sumanth Doddapaneni, Mitesh M Khapra, and Balaraman Ravindran. 2023.  A survey of adversarial defenses and robustness in nlp.  _ACM Computing Surveys_ , 55(14s):1–39.
  * Greene and Cunningham (2006) Derek Greene and Pádraig Cunningham. 2006.  Practical solutions to the problem of diagonal dominance in kernel document clustering.  In _Proceedings of the 23rd international conference on Machine learning_ , pages 377–384.
  * Gu et al. (2023) Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. 2023.  On the learnability of watermarks for language models.
  * Guerra (2023) Marci Guerra. 2023.  Chat gpt for journalism: Revolutionizing the future of reporting.  https://brandalytics.co/chat-gpt-for-journalism/.
  * Guo et al. (2023) Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection.  _arXiv preprint arXiv:2301.07597_.
  * Guo and Yu (2023) Zhen Guo and Shangdi Yu. 2023.  Authentigpt: Detecting machine-generated text via black-box language models denoising.  _arXiv preprint arXiv:2311.07700_.
  * Hamborg et al. (2017) Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017.  news-please: A generic news crawler and extractor.
  * He et al. (2023) Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023.  Mgtbench: Benchmarking machine-generated text detection.  _arXiv preprint arXiv:2303.14822_.
  * Heavenarchive (2023) Will Douglas Heavenarchive. 2023.  Chatgpt is going to change education, not destroy it.  https://www.technologyreview.com/2023/04/06/1071059/chatgpt-change-not-destroy-education-openai/.
  * Hu et al. (2023) Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023.  Radar: Robust ai-text detection via adversarial learning.  _arXiv preprint arXiv:2307.03838_.
  * Hugging Face (2023) Hugging Face. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Hwang et al. (2023) Sung Il Hwang, Joon Seo Lim, Ro Woon Lee, Yusuke Matsui, Toshihiro Iguchi, Takao Hiraki, and Hyungwoo Ahn. 2023.  Is chatgpt a “fire of prometheus” for non-native english-speaking researchers in academic writing?  _Korean Journal of Radiology_ , 24(10):952.
  * Ippolito et al. (2019) Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2019.  Automatic detection of generated text is easiest when humans are fooled.  _arXiv preprint arXiv:1911.00650_.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. 2019.  Pubmedqa: A dataset for biomedical research question answering.  _arXiv preprint arXiv:1909.06146_.
  * Kirchenbauer et al. (2023) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023.  A watermark for large language models.  _arXiv preprint arXiv:2301.10226_.
  * Kočiskỳ et al. (2018) Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The narrativeqa reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328.
  * Koike et al. (2023a) Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2023a.  How you prompt matters! even task-oriented constraints in instructions affect llm-generated text detection.  _arXiv preprint arXiv:2311.08369_.
  * Koike et al. (2023b) Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2023b.  Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples.  _arXiv preprint arXiv:2307.11729_.
  * Krishna et al. (2023) Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting, and Mohit Iyyer. 2023.  Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense.  _arXiv preprint arXiv:2303.13408_.
  * Levenshtein (1966) Vladimir Iosifovich Levenshtein. 1966.  Binary codes capable of correcting deletions, insertions and reversals.  _Soviet Physics Doklady_ , 10(8):707–710.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.  Truthfulqa: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  TruthfulQA: Measuring how models mimic human falsehoods.  pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.
  * Mireshghallah et al. (2023) Fatemehsadat Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick. 2023.  Smaller language models are better black-box machine-generated text detectors.  _arXiv preprint arXiv:2305.09859_.
  * Mitchell et al. (2023) Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023.  Detectgpt: Zero-shot machine-generated text detection using probability curvature.  _arXiv preprint arXiv:2301.11305_.
  * Möller et al. (2020) Timo Möller, Anthony Reina, Raghavan Jayakumar, and Malte Pietsch. 2020.  Covid-qa: A question answering dataset for covid-19.  In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Najzeko (2021) Najzeko. 2021.  Steam reviews dataset 2021.
  * Nishida et al. (2019) Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazutoshi Shinoda, Atsushi Otsuka, Hisako Asano, and Junji Tomita. 2019.  Multi-style generative reading comprehension.  pages 2273–2284, Florence, Italy. Association for Computational Linguistics.
  * OpenAI (2022) OpenAI. 2022.  Openai models - gpt3.5.  https://platform.openai.com/docs/models/gpt-3-5.
  * OpenAI (2023a) OpenAI. 2023a.  Ai text classifier.  https://beta.openai.com/ai-text-classifier.
  * OpenAI (2023b) OpenAI. 2023b.  Gpt-4 technical report.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rajpurkar et al. (2016a) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a.  SQuAD: 100,000+ questions for machine comprehension of text.  pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Rajpurkar et al. (2016b) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Rebuffi et al. (2021) Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. 2021.  Data augmentation can improve robustness.  _Advances in Neural Information Processing Systems_ , 34:29935–29948.
  * Sadasivan et al. (2023) Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023.  Can ai-generated text be reliably detected?  _arXiv preprint arXiv:2303.11156_.
  * Schler et al. (2006) Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006.  Effects of age and gender on blogging.  In _AAAI spring symposium: Computational approaches to analyzing weblogs_ , volume 6, pages 199–205.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423.
  * Shu et al. (2023) Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng. 2023.  Rewritelm: An instruction-tuned large language model for text rewriting.  _arXiv preprint arXiv:2305.15685_.
  * Sison et al. (2023) Alejo Jose G Sison, Marco Tulio Daza, Roberto Gozalo-Brizuela, and Eduardo C Garrido-Merchán. 2023.  Chatgpt: More than a weapon of mass deception, ethical challenges and responses from the human-centered artificial intelligence (hcai) perspective.  _arXiv preprint arXiv:2304.11215_.
  * Solaiman et al. (2019) Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019.  Release strategies and the social impacts of language models.  _arXiv preprint arXiv:1908.09203_.
  * StabilityAI (2023) StabilityAI. 2023.  Stablelm.
  * Su et al. (2023) Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. 2023.  Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text.  _arXiv preprint arXiv:2306.05540_.
  * TheDataBeast (2021) TheDataBeast. 2021.  Ted talk transcripts (2006 - 2021).
  * Tian (2023) Edward Tian. 2023.  Gptzero: An ai text detector.  https://gptzero.me/.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Uchendu et al. (2020) Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020.  Authorship attribution for neural text generation.  In _Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)_ , pages 8384–8395.
  * Verma et al. (2023) Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. 2023.  Ghostbuster: Detecting text ghostwritten by large language models.  _arXiv preprint arXiv:2305.15047_.
  * Wang et al. (2021) Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021.  Adversarial glue: A multi-task benchmark for robustness evaluation of language models.  _arXiv preprint arXiv:2111.02840_.
  * Wu et al. (2023) Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2023.  Llmdet: A third party large language models generated text detection tool.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_.
  * Xu et al. (2023) Han Xu, Jie Ren, Pengfei He, Shenglai Zeng, Yingqian Cui, Amy Liu, Hui Liu, and Jiliang Tang. 2023.  On the generalization of training-based chatgpt detection methods.  _arXiv preprint arXiv:2310.01307_.
  * Yang et al. (2023a) Xianjun Yang, Wei Cheng, Linda Petzold, William Yang Wang, and Haifeng Chen. 2023a.  Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text.  _arXiv preprint arXiv:2305.17359_.
  * Yang et al. (2023b) Xianjun Yang, Kexun Zhang, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. 2023b.  Zero-shot detection of machine-generated codes.  _arXiv preprint arXiv:2310.05103_.
  * Ying (2019) Xue Ying. 2019.  An overview of overfitting and its solutions.  In _Journal of physics: Conference series_ , volume 1168, page 022022. IOP Publishing.
  * Yu et al. (2023) Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Weiming Zhang, and Nenghai Yu. 2023.  Gpt paternity test: Gpt generated text detection with gpt genetic inheritance.  _arXiv preprint arXiv:2305.12519_.
  * Yurkevich (2023) Vanessa Yurkevich. 2023.  Experts warn about possible misuse of new ai tool chatgpt.  https://www.atlantanewsfirst.com/2023/01/24/experts-warn-about-possible-misuse-new-ai-tool-chatgpt/.
  * Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018.  Texygen: A benchmarking platform for text generation models.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, page 1097–1100.
  * (83) Jey Han Lau Zhuohan Xie, Trevor Cohn.  The next chapter: A study of large language models in storytelling.  https://aclanthology.org/2023.inlg-main.23/.

##  Appendix A Full Related Works

###  A.1 Detecting Machine Generated Text

Current MGT detection methods can be broadly categorized into metric-based
methods and model-based methods according to previous study (He et al., 2023).
Moreover, other detection methods such as watermark, retrival-based methods,
and in-context learning leveraging LLMs also lead to a promising way for
detection.

Metric-based Methods. Metric-based methods leverage the LLM backbone directly
to extract its distinguishing features between HWT and MGT, operating within a
white-box setting that requires access to the model. Former methods like Log-
Likelihood (Solaiman et al., 2019), Entropy, Rank Gehrmann et al. (2019), and
Log-Rank (Mitchell et al., 2023) employ statistical analysis to measure
information beyond the token level. GLTR (Gehrmann et al., 2019) utilizes a
suite of metric-based methods to aid in human identification. However, with
the advent of LLMs, the progressively increasing similarity between the
distributions of HWT and MGT has weakened its detection accuracy (Ghosal et
al., 2023).

Building upon the observation that MGTs occupy regions with sharp negative log
probability curvature, Mitchell et al. (2023) introduced a zero-shot whitebox
detection method called DetectGPT, setting a trend in metric-based detection
(Su et al., 2023; Mireshghallah et al., 2023; Bao et al., 2023). Yang et al.
(2023a) also introduced a powerful detection method known as DNA-GPT, which
leverages N-gram (Shannon, 1948) in a black-box setting by analyzing the
differences between truncated original text and regenerated text. Recently,
they even extended the detection method to MGT code in a zero-shot setting,
which is proven to achieve promising results (Yang et al., 2023b).

Figure 7: POS distribution of the MixSet by NLTK (Bird et al., 2009). Figure
8: Levenshtein distance of the MixSet

6

Figure 9: Self-BLEU score of the HWT, MGT, and MixSet.

Model-based Methods. In the era of Large Language Models (LLMs), (Guo et al.,
2023) developed the ChatGPT Detector, which is based on a fine-tuned Roberta
model. As for decoder-based or encoder-decoder detectors, GPT-Sentinel (Chen
et al., 2023) and RADAR (Hu et al., 2023), utilizing T5-small (Raffel et al.,
2020) and Vicuna-7B (Chiang et al., 2023) respectively, show convincing
results when detecting MGT even in revised cases. Moreover, Verma et al.
(2023) propose a novel detection framework called Ghostbuster, which employs
passing documents through a series of weaker language models. Using a small
amount of training data, Guo and Yu (2023) leverages a black-box LLM to
denoise input text with artificially added noise, and then semantically
compares the denoised text with the original to determine if the content is
machine-generated, leading a new method for MGT detection.

However, it’s important to note that some researchers raise concerns about
fine-tuning models for MGT detection. Bakhtin et al. (2019) and Uchendu et al.
(2020) have argued that fine-tuning models can lead to overfitting and a loss
of generalization, particularly when dealing with text generated by the latest
LLMs. They highlight challenges posed by out-of-distribution editing texts,
which can undermine the effectiveness of pre-trained detectors, as
demonstrated by Krishna et al. (2023).

Other detection methods. Watermarking imprints specific patterns of the LLM
output text that can be detected by an algorithm while being imperceptible to
humans. Kirchenbauer et al. (2023) developed watermarks for language modeling
by adding a green list of tokens during sampling. Currently, Gu et al. (2023)
introduces a learnable watermark by distilling LLM and watermark technology
into one student model, finding that models can learn to generate watermarked
text with high detectability.

In retrieval-based methods, Krishna et al. (2023) introduce a method to
retrieve semantically similar generations and search a database of sequences
previously generated by specific Large Language Models (LLMs), looking for
sequences that match the candidate text within a certain threshold. Delving
deeper, Wu et al. (2023) propose a model-specific detection tool called
LLMDet, which can detect source text from specific LLMs by constructing a text
collection dictionary for each LLM.

In the in-context learning setting, Yu et al. (2023) introduced a
straightforward method that analyzes the similarity between re-answering a
question by generating a corresponding question in the context of the given
answer. Moreover, Koike et al. (2023b) employed a pure in-context learning
approach for detection and found that LLMs are capable of distinguishing
between human and machine styles.

###  A.2 Datasets for MGT Detection

Previous studies have proposed many datasets of MGT, often accompanied by
their newly proposed detectors (Verma et al., 2023; Chen et al., 2023). Guo et
al. (2023) leverages multiple previous Question-Answer (QA) datasets
(Rajpurkar et al., 2016b; Kočiskỳ et al., 2018; Jin et al., 2019; Lin et al.,
2021), allowing ChatGPT to generate corresponding answers without explicit
prompts. This approach results in the creation of a comprehensive dataset
comprising a large set of pairs of MGT and HWT. Following the QA pattern, many
researchers (Mitchell et al., 2023; Su et al., 2023; Hu et al., 2023; He et
al., 2023) propose datasets with the MGT from variant mainstream LLMs (Du et
al., 2022; Black et al., 2022; Anand et al., 2023; Conover et al., 2023;
OpenAI, 2022, 2023b). Yu et al. (2023) only utilizes the answer section within
the QA dataset (Hamborg et al., 2017; Möller et al., 2020) and employs ChatGPT
to generate corresponding questions and re-answers.

However, these datasets typically consist of two distinct classes of texts,
namely pure MGT or HWT, without accounting for the potential mixcase.
Furthermore, issues arise due to variations in prompts (Koike et al., 2023a),
sampling methods, and the inherent differences in length, style, and quality
among texts in some datasets He et al. (2023). These variations challenge the
generalization of proposed detectors (Xu et al., 2023) and lie a huge
diversity in distribution between the original and revised text(Ghosal et al.,
2023). In some instances, MGT included in datasets may not undergo thorough
careful evaluation. Many noisy sentences are not filtered well in the
datasets. For example, some sentences like Let me know if you have any other
questions exist in the dataset, which will impact the effectiveness of the
detectors (Guo et al., 2023).

##  Appendix B Dataset Details

###  B.1 Construction Details

Eight Human revised the MGT to mixcase. The MGT is revised by eight human
experts with professional English proficiency and costs them a total of 280
hours to complete this part of MixSet. The guidelines for human revision are
shown in Figure 20. And the labeling screenshot are shown in Figure 35.

###  B.2 Other Metrics in Evaluating MixSet

  * •

Self-BLEU Score: Self-BLEU is a metric used to assess the diversity of
generated text. Generally, a lower Self-BLEU score indicates higher textual
diversity. These results are shown in Figure 9. Overall, the HWT shows greater
diversity than MGT, and the Rewrite category has the highest textual diversity
in the MixSet. The self-BLEU score of HWT, WGT, and mixcase is shown in Figure
10 and 9.

  * •

Levenshtein Distance: The Levenshtein distance (Levenshtein, 1966) is a metric
for measuring the difference between two strings. We can observe in Figure 8
that in terms of the extent of modification, the rewrite operation results in
the most significant alterations to the original texts, followed by complete
and sentence-level polish. Additionally, we observe that manual annotations at
both the token-level and sentence-level adaptation also exhibit a high degree
of differentiation.

  * •

POS distribution: POS distribution refers to the frequency and pattern of
Part-of-Speech tags in a text, categorizing words into grammatical classes
like nouns, verbs, and adjectives. This analysis is key for understanding the
text’s syntactic structure and linguistic characteristics which is important
in NLP research fields.

Figure 10: Length distribution of the training datasets and the MixSet(left)
and Self-BLEU score of HWT and MGT(right).

##  Appendix C Detailed Experiment Settings

Five Metric-Based detectors. We implement log-likelihood, entropy (Solaiman et
al., 2019), Log-Rank (Mitchell et al., 2023), GLTR Gehrmann et al. (2019) and
DetectGPT (Mitchell et al., 2023) as our metric-based detectors. Following He
et al. (2023), we utilize GPT2-medium (Radford et al., 2019) as the base model
for metric-based methods in our experiments since it can already reach good
performance with limited cost and computing time.

Seven Model-Based detectors. We implement seven Machine Generative Text (MGT)
detectors, encompassing both supervised and zero-shot settings. Firstly, we
consider a robust closed-source online detector baseline: GPTZero (Tian,
2023). Secondly, we implement three open-source encoder-based detectors:
OpenAI’s classifier (OpenAI, 2023a), Roberta-based classifier (Guo et al.,
2023). We also implement GPT-Sentinel (Chen et al., 2023), RADAR (Hu et al.,
2023), and Ghostwriter Verma et al. (2023) as strong baselines. We also
finetune a pre-trained language model built by Ippolito et al. (2019) with an
extra classification layer on top.

Three Evaluation Metrics Previous studies (Sadasivan et al., 2023; Mitchell et
al., 2023) have proven the feasibility of using the Area Under The ROC Curve
(AUROC) score for evaluating detection algorithm effectiveness. Given that
most detectors can only give a predictive probability, we build a logistic
regression model to provide concrete predictions, i.e., MGT or HWT, converting
probability to accuracy and f1-score as the two other metrics for our
detection evaluation.

Training set construction. We respectively select pure HWT and MGT for the
train set from different datasets as illustrated in 2 and MGTBench He et al.
(2023), which is also the original dataset of our MixSet, convincing a small
difference in data distribution. Firstly, we do data deduplication and pre-
process it to erase the Unicode or other special tokens like \n\n. Then, we
select pieces of sentences with a similar length distribution in our MixSet,
as illustrated in Figure 10. As we use accuracy as our evaluation metric, we
restrict the amount of HWT and MGT to be the same in our dataset, as
illustrated in Tabel 5.

Training details. We employ the standard binary-classification loss function
and the AdamW optimizer (Loshchilov and Hutter, 2019), with an empirically
determined learning rate. Specifically, for the Hello-Ai/Roberta-based model
and the DistilBERT model, the learning rate is set to $5\times 10^{-7}$. In
contrast, for Radar and GPT-Sentinel, the learning rates are $5\times 10^{-6}$
and $5\times 10^{-5}$ respectively. Each supervised model undergoes training
for three epochs on a dual-4090 server.

(a) The AUC Heatmap of Radar

(b) The AUC Heatmap of Roberta-based model

Figure 11: The AUC Heatmap of the other two detectors.

##  Appendix D Detailed Experiment Results

As for experiment 1, we put the detailed accuracy for different detectors in
Table 6. In experiment 2, we also evaluate detectors with AUC metric, as shown
in Table 9. We also post other detectors undergo our experiment 3 illustrated
in Figure 11. As for experiment 4, we evaluate detectors with accuracy,
precision, and recall metrics, as illustrated in Figure 12, 13, and 14.

Table 9: AUC of Experiment 2 (a). We underscore the best-performing detector
and bold the score greater than $0.8$, which we consider as a baseline
threshold for detection. Above the thin horizontal line is the metric-based
detector, and below the horizontal line is the model-based detector (Tok.
stands for token level and Sen. stands for sentence level).

Detection Method | AI-Revised | Human-Revised
---|---|---
Rewrite | Complete | Polish-Tok. | Polish-Sen. | Humanize |  Adapt-Tok. |  Adapt-Sen.
Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4 |  Llama2 |  GPT-4
Metric-based Detector
Log-Rank | 0.921 | 0.629 | 0.632 | 0.318 | 0.569 | 0.531 | 0.662 | 0.462 | 0.641 | 0.245 | 0.778 | 0.778
Log Likelihood | 0.933 | 0.650 | 0.672 | 0.352 | 0.610 | 0.569 | 0.709 | 0.508 | 0.652 | 0.206 | 0.782 | 0.786
GLTR | 0.870 | 0.504 | 0.546 | 0.268 | 0.511 | 0.466 | 0.602 | 0.345 | 0.595 | 0.208 | 0.764 | 0.768
DetectGPT | 0.852 | 0.644 | 0.669 | 0.352 | 0.612 | 0.466 | 0.664 | 0.482 | 0.677 | 0.461 | 0.548 | 0.557
Entropy | 0.814 | 0.581 | 0.662 | 0.463 | 0.656 | 0.636 | 0.686 | 0.596 | 0.580 | 0.185 | 0.733 | 0.730
Model-based Detector
Openai Classifier | 0.294 | 0.601 | 0.126 | 0.360 | 0.433 | 0.492 | 0.383 | 0.590 | 0.321 | 0.517 | 0.182 | 0.187
ChatGPT Detector | 0.706 | 0.399 | 0.874 | 0.640 | 0.567 | 0.508 | 0.617 | 0.410 | 0.679 | 0.483 | 0.818 | 0.813
Radar | 0.992 | 0.994 | 0.997 | 0.999 | 0.998 | 0.986 | 0.998 | 1.000 | 0.984 | 0.984 | 0.999 | 0.999
GPT-Sentinel | 0.994 | 0.992 | 0.987 | 0.993 | 0.995 | 0.964 | 0.992 | 0.996 | 0.915 | 0.953 | 0.958 | 0.986
Distillbert | 0.756 | 0.856 | 0.746 | 0.859 | 0.790 | 0.730 | 0.791 | 0.856 | 0.416 | 0.330 | 0.837 | 0.861

Figure 12: Analysis of the accuracy of various detectors across differing
quantities of mixcase instances from MixSet, as well as pure MGT and HWT.
Figure 13: Analysis of the recall rate of various detectors across differing
quantities of mixcase instances from MixSet, as well as pure MGT and HWT.
Figure 14: Analysis of the precision rate of various detectors across
differing quantities of mixcase instances from MixSet, as well as pure MGT and
HWT.

##  Appendix E Prompt Template

We show the prompt template of LLM’s operation including complete, polish
(token-level and sentence-level), rewrite, and humanize in Figure 15, Figure
16, Figure 17, Figure 18 and Figure 19.

Figure 15: Prompt(①)-LLM complete the HWT I have an incomplete text and need
it to be completed. Please expand this into a complete text where the total
word count, including the original text I have provided, does not exceed 180
words. The original text must remain exactly as is, with its format (such as
capitalization and punctuation) intact. Please do not modify any part of the
original text. Here’s the text: {HWT} Figure 16: Prompt(②)-LLM polish HWT in
token level Please carefully examine the following paragraph solely for
spelling and grammatical errors, and replace any words that are repetitive,
inaccurate, or poorly chosen. It is crucial to avoid any changes to the
sentence order or structure. The focus should be strictly on the choice and
usage of individual words to improve the clarity and appropriateness of the
text, without altering the original sentence construction: {HWT} Figure 17:
Prompt(③)-LLM polish HWT in sentence level Please optimize the sentences in
the following paragraph to enhance fluency and clarity. Do not alter the
overall content or structure of the paragraph. Focus on the construction and
expression of the sentences, ensuring that the text is coherent and the
information is accurate: {HWT} Figure 18: Prompt(④)-LLM rewrite HWT Please
extract the core ideas and keywords from the following English text, and then
rewrite a passage based on this information. The new text should maintain the
essence of the original, with the word count varying by no more than 10% from
the original. There’s no need to list the core ideas and keywords. Here is the
text that needs to be processed: {HWT} Figure 19: Prompt(④)-LLM humanize MGT I
need to modify a machine-generated text to make it appear more like it was
written by a human. The objective is to introduce elements commonly found in
human-written texts. Here are some optional modifications you can choose to
apply: 1\. Introduce spelling errors or typos(optional)  2\. Create
grammatical errors, such as randomly adding or deleting words (optional).  3\.
Include relevant but internet links, like blog posts or image links pertaining
to the topic, you don’t have to use the real links so you can freely write one
(optional).  4\. Add relevant hashtags, for instance, #TopicKeyword #Location
#Activity (optional). 5\. Use internet slang and abbreviations, e.g., ‘OMG’,
‘How r u’, ‘LOL’, (optional). Please select any combination of these
modifications to enhance the text’s human-like quality. The aim is to simulate
the imperfections and stylistic choices typical in casual human writing. The
word count of the new text should not exceed 1.1 times that of the original
text. You should just give me the revised version without any other words.
Emojis are strictly prohibitive, so please ensure that it contains no emojis.
Here is the machine-generated text:{HWT} Figure 20: Guidelines for Human
Revision The content under this document is generated by a large language
model, such as ChatGPT. You are required to revise it to make it closer to the
style of human-written text. You are responsible for the text under the IDs
xx-xx, and you need to make the following three types of modifications to the
content, generating three different sentences for each ID (each ID corresponds
to 3 sentences): The document is in JSON format. You can choose to use code
editors like Visual Studio Code, or text editors like Notepad for reading and
writing. 1\. Adapt Token: You need to modify any words or phrases in this
passage that you think are too rigid, mechanical, obscure, or unusual into
vocabulary typical of human texts. Be careful not to alter the sentence order
or structure; only modify ‘words’ or ‘phrases.’ 2\. Adapt Sentence: You need
to revise any sentence structures that you find too mechanical or rigid to
make them more in line with how humans typically write texts. This involves
changes at the sentence level, which may include altering sentence order and
structure. Note: 1\. After writing, please ensure to check that there are no
‘grammatical errors’ or ‘spelling mistakes’ in the text paragraphs. 2\. Do not
use ChatGPT or other large language models for data annotation, as it will
severely degrade the quality of the data. 3\. You may use translation
platforms like Youdao you  or Google goo ; or use Grammarlygra  to check for
grammatical errors. Below are some examples you can refer to: [Example 1]
[Example 2] [Example 3]

##  Appendix F Case study in MixSet

We selected two cases to show the comparison between the revised mixcase and
the original text, where the highlighted content represents the modified
content. The HWT original text can be found in figure 21, the AI revised text
are shown in Figure 22, 23, 24, 25, 26, 27, 28, and 29. The MGT original text
can be found in Figure 30, and the Human revised text can be found in Figure
31, 32, 33, and 34.

Figure 21: Case Study of revision on HWT: Original HWT [Original HWT] We
present a new mixed variable symplectic (mvs) integrator for planetary
systems, that fully resolve close encounters. The method is based on a time
regularisation that allows keeping the stability properties of the symplectic
integrators, while also reducing the effective step size whenever two planets
encounter. We use a high order mvs scheme such that it is possible to
integrate with large time steps far away from close encounters. We show that
this algorithm is able to resolve almost exact collisions (i.e. with a mutual
separation of a fraction of the physical radius) while using the same time-
step as in weakly perturbed problems such as the solar system. We demonstrate
the long-term behavior on systems of six super-earths experiencing strong
scattering for 50 kyr. We compare our algorithm to hybrid methods such as
mercury and show that for an equivalent cost, we obtain much better energy
conservation. Figure 22: Case Study of revision on HWT: GPT-4 Polish (Token)
[GPT-4 Polish (Token)] We present a new mixed-variable symplectic (MVS)
integrator for planetary systems that fully resolves close encounters. The
method is grounded in a temporalregularization that maintains the stability
properties of symplectic integrators while also diminishing the effective step
size whenever two planets draw near each other. We employ a high-order MVS
scheme, enabling  integration with extended time steps when distant from close
encounters. We demonstrate that this algorithm capably resolves nearly exact
collisions (i.e., with a mutual separation of a small fraction of the physical
radius) using a consistent time-step as in weakly perturbed problems such as
the solar system. We exhibit the long-term dynamics on systems of six super-
Earths undergoing intense scattering over 50 kyr. We contrast our algorithm
with hybrid methods such as Mercury, revealing that, for a comparable cost, we
achieve much improved energy conservation. Figure 23: Case Study of revision
on HWT: GPT-4 Polish (Sentence) [GPT-4 Polish (Sentence)] We introduce a novel
mixed variable symplectic (MVS) integrator specifically designed for planetary
systems that efficiently manages close encounters. This method utilizes time
regularization to maintain the stability characteristics inherent to
symplectic integrators, while dynamically adjusting the effective step size
during planetary encounters. Our high-order MVS approach enables integration
at large time steps when planets are distantly separated, thereby optimizing
computational efficiency. The algorithm proves capable of resolving near-exact
collisions—with mutual separations as minute as a fraction of the planets’
physical radii—without altering the time step used for more stable
configurations, such as within the solar system. We examine the algorithm’s
long-term stability in simulations of six super-Earths undergoing significant
scattering over 50,000 years. Furthermore, we benchmark our algorithm against
hybrid methods like Mercury, demonstrating superior energy conservation at a
comparable computational cost. Figure 24: Case Study of revision on HWT: GPT-4
Complete [GPT-4 Complete] We present a new mixed variable symplectic (mvs)
integrator for planetary systems, that fully resolve close encounters. The
method is based on a time regularisation that allows keeping the stability
properties of the symplectic integrators, while also reducing the effective
step size whenever two planets encounter. We use a modified leapfrog scheme to
address the issues of traditional integrators when handling near-collision
scenarios. The key improvement lies in an adaptive algorithm that dynamically
adjusts the step size according to the proximity of the planetary bodies. As a
result, our integrator ensures high accuracy and energy conservation even
during complex dynamical interactions. Tests demonstrate that the mvs
integrator significantly outperforms standard fixed-step symplectic methods in
long-term stability and computational efficiency. This advancement opens up
new possibilities for accurately simulating the evolution of planetary systems
over extended timescales, providing valuable insights into the dynamical
behavior of multi-body systems  and enhancing our understanding of celestial
mechanics. Figure 25: Case Study of revision on HWT: GPT-4 Rewrite [GPT-4
Rewrite] We introduce an advanced mixed variable symplectic (mvs) tailored for
planetary systems, adept at handling close encounters precisely. This
technique employs time regularization to maintain the stability characteristic
of symplectic integrators and intelligently reduces the effective step size
during planetary encounters. Employing a sophisticated mvs algorithm, it
facilitates integration over large time steps when planets are not closely
interacting.Our findings illustrate that the algorithm effectively manages
near-collision events, simulating scenarios with planet separations just
fractions of their physical sizes, using consistent time steps applicable to
less disturbed systems like the solar system. The integrator’s performance
excels in extended simulations, such as tracking six super-earths in a dense
scattering scenario for 50,000 years, showcasing superior energy conservation
when contrasted with hybrid algorithms like the mercury method, and doing so
computational cost. Figure 26: Case Study of revision on HWT: Llama2 Polish
(Token) [Llama2 Polish (Token)] We introduce a novel mixed variable symplectic
(MVS) integrator for planetary systems that accurately resolves close
encounters. Our method is built upon a time regularization that preserves the
stability attributes of symplectic integrators while adaptively decreasing the
effective step size during planetary interactions. By employing a high-order
MVS scheme, we can integrate with large time steps in regions where close
encounters are not occurring. This algorithm enables the precise resolution of
nearly exact collisions (with a mutual separation of a small fraction of the
physical radius) using the same time step as in weakly perturbed problems,
such as the solar system. We demonstrate the long-term behavior of our
algorithm on systems consisting of six super-Earths undergoing intense
scattering. Our approach shows superior energy conservation compared to hybrid
methods like Mercury, while maintaining comparable computational efficiency.
Figure 27: Case Study of revision on HWT: Llama2 Polish (Sentence) [Llama2
Polish (Sentence)] We introduce a novel mixed variable symplectic (MVS)
integrator for planetary systems that accurately resolves close encounters.
Our method is founded on a time regularization technique that preserves the
stability attributes of symplectic integrators while adaptively decreasing the
effective step size during planetary encounters. By employing a high-order MVS
scheme, we can integrate with substantial time steps in regions unaffected by
close encounters. Our algorithm efficiently handles near-collisions(with a
mutual separation of a few percent of the physical radius) utilizing the same
time-step as in weakly perturbed scenarios, such as the solar system. In a
comprehensive analysis, we demonstrate the long-term behavior of our method on
systems consisting of six super-Earths undergoing intense scattering over 50
kyr. Our findings indicate that, compared to hybrid approaches like Mercury,
our algorithm achieves superior energy. conservation at an equal computational
cost Figure 28: Case Study of revision on HWT: Llama2 Complete [Llama2
Complete] We present a new mixed variable symplectic (MVS) integrator for
planetary systems, that fully resolves close encounters. The method is based
on a time regularization that allows keeping the stability properties of the
symplectic integrators, while also reducing the effective step size whenever
two planets encounter. We use a simple and efficient algorithm to detect close
encounters, and switch to a smaller step size during these events to ensure
accurate trajectory propagation. This approach enables us to maintain the
long-term accuracy of the simulation while significantly improving
computational efficiency. Our MVS integrator can handle both collisional and
non-collisional close encounters, and we demonstrate its performance through
several test cases involving Earth-Moon and Jupiter-Io system simulations.
Figure 29: Case Study of revision on HWT: Llama2 Rewrite [Llama2 Rewrite] We
introduce a novel mixed variable symplectic (MVS) integrator for solving
planetary systems that undergo close encounters. Our approach combines the
benefits of symplectic integrators - stability and accuracy - with the ability
to adaptively adjust the time step size during close encounters. This enables
us to capture near-collisions with high precision while still utilizing large
time steps in stable regions. We validate our algorithm’s performance by
simulating systems of six super-Earths over 50,000 years, demonstrating its
superiority compared to hybrid methods like Mercury in terms of energy
conservation, while maintaining comparable computational efficiency. Figure
30: Case Study of revision on MGT: Original MGT [Original MGT] The first
coffee was brewed by the ancient egyptians over 4,000 years ago. As of 2023,
coffee consumption increased 70-fold in the past 100 years. Today, 98% of
americans drink coffee. Studies have shown that drinking two cups of coffee
per day is beneficial for adults. However, more and more studies also show
that drinking one or less coffee is also equally beneficial. Drinking coffee
has also been linked to higher memory, alertness, and focus. Overall, coffee
is good for your health.  Figure 31: Case Study of revision on MGT: GPT-4
Humanize [GPT-4 Humanize] OMG did you kno that, like, the first cuppa joe was
whipped  up by ancient Egyptions like more than 4,000 yrs ago? Fast fwd to
2023, and whoa, coffee drinking’s shot up a whoppin’ 70 times in the last
century! Nowadays, it’s like 98% of Americans are sippin’ on coffee. Some
smarty-pants studies point out that knockin’ back two cups a day’s pretty good
for grown-ups. But hold up, more research is spillin’ the beans that maybe
just one cup or even less can do the trick too. Crazy, right? Srsly tho,
chuggin’ coffee’s got a rep for boostin’ your memory and makin’ ya more alert
and focused. All in all, java’s a thumbs up for your health. #CoffeeLovers
#HealthBenefits #StayAlert BTW, if ya wanna dive deeper into the whole coffee
vibe, there’s this killer blog post over at
"www.totallyrealcoffeeblog.com/history" that’s got the low-down on coffee’s
past. And, like, for the visual folks out there, check out these pics on
"www.picturesofcoffeeandstuff.com" – they’ve got some epic shots of coffee
that’ll make you wanna brew a pot, like, right now. LOL Figure 32: Case Study
of revision on MGT: Llama2 Humanize [Llama2 Humanize] Did you know that the
ancient Egyptians were the first to  brew coffee over 4,000 years ago? Crazy,
right? And get this - coffee consumption has increased a whopping 70-fold in
just the past 100 years! 98% of Americans can’t start their day without their
daily cup (or two) of joe. But here’s the thing: while most people assume that
downing multiple cups a day is the way to go, studies actually suggest that
even just one cup can do the trick. Yep, you heard that right - one measly cup
can boost your memory, alertness, and focus. Talk about a productivity hack!
Of course, there’s always the risk of overdoing it, but hey, everything in
moderation, right? Bottom line: coffee = good stuff.  Figure 33: Case Study of
revision on MGT: Adapt Token [Adapt Token] The first coffee was brewed by the
ancient Egyptians  over 4,000 years ago. By 2023, coffee consumption has
increased 70-fold in the past 100 years. Nowadays, 98% of Americans drink
coffee. Studies have shown that drinking two cups of coffee every day is
beneficial for adults. Additionally, more and more studies also show that
drinking one or less coffee is also equally beneficial. Drinking coffee is
also linked to better memory, alertness, and concentration. Overall, coffee is
good for your health.  Figure 34: Case Study of revision on MGT: Adapt
Sentence [Adapt Sentence] The first coffee was brewed by the ancient Egyptians
over 4,000 years ago. Coffee consumption has increased 70-fold in the past 100
years, along with 98% of Americans drinking coffee, according to the data up
to 2023. Studies have shown that drinking two cups of coffee every day is
beneficial for adults, while other studies indicate that drinking one or fewer
cups of coffee is also equally beneficial. Drinking coffee is also linked to
better memory, alertness, and concentration. Overall, coffee is good for your
health.  Figure 35: screenshot of human labeling