LLM Detectors Still Fall Short of Real World: Case of LLM-Generated Short News-Like Posts
Henrique Da Silva Gameiro Andrei Kucharavy Ljiljana Dolamic

[
    [
    September 5, 2024
=====================



§ ABSTRACT 

With the emergence of widely available powerful LLMs, disinformation generated by large Language Models (LLMs) has become a major concern. Historically, LLM detectors have been touted as a solution, but their effectiveness in the real world is still to be proven. In this paper, we focus on an important setting in information operations—short news-like posts generated by moderately sophisticated attackers.

We demonstrate that existing LLM detectors, whether zero-shot or purpose-trained, are not ready for real-world use in that setting. All tested zero-shot detectors perform inconsistently with prior benchmarks and are highly vulnerable to sampling temperature increase, a trivial attack absent from recent benchmarks. A purpose-trained detector generalizing across LLMs and unseen attacks can be developed, but it fails to generalize to new human-written texts. 

We argue that the former indicates domain-specific benchmarking is needed, while the latter suggests a trade-off between the adversarial evasion resilience and overfitting to the reference human text, with both needing evaluation in benchmarks and currently absent. We believe this suggests a re-consideration of current LLM detector benchmarking approaches and provides a dynamically extensible benchmark to allow it (<https://github.com/Reliable-Information-Lab-HEVS/dynamic_llm_detector_benchmark>).













§ INTRODUCTION


The misuse of large language models (LLMs) for propaganda, inciting extremism, or spreading disinformation and misinformation has been a major concern from the early days of LLMs <cit.>. While this concern has led in the past LLM developers to withhold their most powerful models <cit.>, powerful LLMs and multimodal generative models have now been published despite such concerns persisting <cit.>. With the release of powerful open-weight LLMs that can be deployed on commodity hardware such as LLaMA <cit.>, Phi <cit.> or Zephyr <cit.>, LLM misuse can no longer be mitigated through model adjustment or inputs/outputs monitoring. 













 §.§ LLMs in Information Operations



A common goal of information operations (intentional and coordinated efforts to modify the public perception of reality with ulterior motives) is to modify the perception of the current situation <cit.>. A well-documented and effective approach to achieve that leverages social networks by seeding them with news-like narratives. While the original seeding might have a minimal impact, some narratives will trigger a strong partisan response and will be re-shared by key genuine users, who will customize the narrative for their audience and engage in debate to further it <cit.>. While in some cases such misinformation is picked up by traditional news outlets, adding to its credibility, the quantity is a quality of itself, and continuous repetition of the same narratives through different channels leads to their acceptance as truths <cit.>. Given the competition for attention on social media, both seeded narratives and legitimate news tend to be shared as short-form pots of 260-520 characters with links and images. Such format is common to Twitter (now X), Bluesky, Meta Threads, and the largest Mastodon instances.

In the past, this technique has been shown to be an excellent conductor of counterfactual claims and a poor one for rebuttals and factchecks <cit.>. However, large-scale information operations on them have been, until recently, easily detectable due to exact text reuse, inconsistencies in online persona, and lack of non-trivial interaction with other users. Such failures are understandable, given the scope of the activities and the cost of human operators. However, LLMs drastically alter the cost-quality tradeoff <cit.>. Combined with the reports that LLMs are better than humans at both personalized political persuasion and disinformation concealment <cit.>, LLM-augmented information operations promise to be radically more effective and more difficult to detect and counter.






























 §.§ LLM Detectability


Unfortunately, humans do not distinguish well LLM-generated texts from human-written ones  <cit.>. Early in the generative LLM development  <cit.> proposed that accurate LLM detectors could mitigate that issue. Unfortunately, follow-up research rapidly discovered that LLM detectors failed if generation parameters changed <cit.>, output was paraphrased <cit.>, or barely more complex prompts were used <cit.>. 

The issue gained in salience with the release of ChatGPT, leading to several large-scale LLM detector benchmarks <cit.>, with most recently <cit.>. Unfortunately, with the disparate performance of detectors for different types of texts, most benchmarks suffer from the same pitfalls as many other ML-based security solutions <cit.>, and there is still no consensus as to whether LLM detectors are ready for real-world applications <cit.>. Here, we try to address both the methodological issues and decide on the detectors' real-world usefulness in our setting.

























 §.§ Setting and Attacker Capabilities


An attacker with arbitrary capabilities is neither realistic nor can realistically defended against. Consistently with <ref>, we focus on moderately sophisticated attackers. We assume they are capable of deploying on-premises SotA LLMs in the 1-10B parameter range and familiar with adversarial evasion strategies available at inference, such as generation parameters modification <cit.>, paraphrasing <cit.>, or alternative prompting strategies <cit.>. We assume that the attacker cannot evasively fine-tune the generative models.

We assume that the attacker is seeking to generate news-like content of approximately 500 characters, as a completion of a headline or opening sentence, that needs to be detected by a social media operator in an environment where LLM-generated news-like content is not predominant and true positive labels are not available, requiring a low target false positive rate (FPR).





















 §.§ Contributions



    
    
    
  * We consolidated best practices for LLM detector evaluation and developed a dynamic benchmark integrating them, allowing simple extension to new domains
    
  * We show that SotA zero-shot detectors are vulnerable to trivial adversarial evasion in ways not reflected by prior benchmarks
    
  * We comprehensively benchmarked custom detector training strategies and discovered a detector architecture achieving a robust generalization across unseen LLMs and attacks
    
  * We demonstrate that such adversarially robust custom detectors overfit their reference human data, indicating a tradeoff that needs to be tested
    
  * Based on these results, we conclude that LLM detectors are currently not ready for real-world usage to counter LLM-generated disinformation
    
    
    
    
    




§ BACKGROUND AND RELATED WORK




 §.§ Trained Detectors


LLMs have led to a pardigm shift, from purpose-training ML models, to fine-tuning base models. Rather than re-training a new model for each application from scratch, a large model is first pretrained on a large quantity of data, and is then fine-tuned to adapt it to downstream task. This paradigm is particularly well-suited for classification tasks as demonstrated by <cit.>. As we can formulate detection as a classification task, multiple papers take this approach, historically pioneered by <cit.>. It is also currently considered a SotA approach for training custom LLM detectors <cit.>



  
BERT-based detectors
BERT (see <cit.>) and its subsequent improved versions, RoBERTa and Electra (see <cit.> and <cit.>), have gained widespread adoption for classification tasks. Their bi-directional encoding architecture offers an advantage over the autoregressive decoder-only architecture, given the ability to account for the context of both the preceding and following context rather than just the preceding, as well as obligatory anchoring to the provided text. Moreover, their relatively small size, such as the 300M parameters for RoBERTa Large compared to the 175B parameters for GPT-3, makes them highly practical.







 §.§ Zero-Shot Detectors


Trained detectors show promising results in multiple works such as in <cit.>. However, as highlighted by <cit.>, these trained detectors fail to generalize to text distribution shifts. Zero-shot detectors such as in <cit.> and <cit.> can be seen as an appealing alternative to mitigate this issue. Zero-shot detectors, i.e., detectors not relying on training, are more suitable when we try to detect text not coming from a specific distribution. They are also easier to use in practice since we do not need to train the detector on the domains, although at the price of generally being more resource-intensive due to using larger models.



  
Fast-DetectGPT is a SoTA open-source detector, reported to perform well across domains and common attack strategies <cit.>, scoring within in overall top for FPRs < 5% in adversarial third-party benchmarks <cit.>.



  
GPTZero is one of the most widely used commercial LLM detectors <cit.>, consistently included into third-party benchmarks.



  
RoBERTa-Base-OpenAI is a BERT-based detector fine-tuned by OpenAI to detect GPT2 output <cit.>. While it is not a zero-shot detector per se, it is still commonly benchmarked and used as such, with some benchmarks reporting good performance<cit.>. At the time of writing (May 2024), it was downloaded over 159,000 times in the previous three months, according to its HuggingFace repository.
 











 §.§ Benchmarking Detectors




  
Adversarial evasion setting is inherent to LLM detectors; its performance cannot be detached from performance against potential attacks. Multiple works have attempted to tackle this issue, generally focusing on specific attacks. For instance, <cit.> and <cit.> introduced a paraphrasing attack they demonstrated to be efficient, while <cit.> investigated evasion through alternative decoding strategies. 



  
Generative generalization benchmarking, be it across LLMs <cit.> or domains <cit.> is equally essential. LLM generation can be useful to an attacker in multiple contexts, and with a proliferation of widely available powerful LLMs, an attacker cannot be assumed to restrict themselves to a single generative model.



  
Human-text generalization evaluation is, unfortunately, all but absent from LLM detectors benchmarks, despite reported issues in real-world usage <cit.>. Existing benchmarks report - at best detection rates for additional domains, without reporting FPRs.

Unfortunately, systematic studies remain few, and the recent benchmarks attempting them, such as <cit.> suffer from issues, notably failing to include common attacks, such as temperature increase <cit.>, and evaluation of FPRs in unseen domains. As static benchmarks optimized for detector evaluation, they are difficult to add new attacks to or transfer to new domains, making them unsuitable for our application.



  
RoBERTa-Base-OpenAI is a BERT-based detector fine-tuned by OpenAI to detect GPT2 output <cit.>. While it is not a zero-shot detector per se, it is still commonly benchmarked and used as such, with some benchmarks reporting good performance<cit.>. At the time of writing (May 2024), it was downloaded over 159,000 times in the previous three months, according to its HuggingFace repository.









§ METHODOLOGY


All results can be replicated with code provided in the experimental repository: <https://github.com/Reliable-Information-Lab-HEVS/text_llm_detector>. English is the only language considered here, with all datasets, prompts, and fine-tuning data in English.



 §.§ Detector Models



To evaluate a domain-specific pretrained detector, generally reported to be one of the best detection methods <cit.>, we evaluated three base pre-trained LLMs: RoBERTa-Large, Distil-RoBERTa, and Electra-Large[All download links for models, datasets, and code are available in the appendix table <ref>, in the appendix <ref>]. RoBERTa and Electra have been shown to achieve improved results over BERT on fine-tuning to classification tasks in <cit.>  and <cit.> thanks to a different pre-training method <cit.>. We added Distil-RoBERTa to evaluate the performance of a smaller model (82.8M parameters), more usable at scale. 



To test zero-shot detectors, we evaluated the three detectors mentioned previously: RoBERTa-Base-OpenAI, due to its usage, and Fast-DetectGPT and GPTZero, generally considered as representatitve of SotA open-source and commercial detectors, respectively <cit.>.







 §.§ Generating the Datasets


We chose six different generator LLMs. Three non-chat foundational models, Phi-2, Gemma-2B, and Mistral-0.1, are the only ones used to train the custom detectors. Three chat models, Gemma-chat, Zephyr, and LLama-3-8B-Instruct, are only used for testing and are representative of commonly used SotA open-weight LLMs.
 
To create neural fake news, leveraged the CNN Dailymail news dataset, representative of American English news. To obtain the LLM-generated articles, we take a news article from the dataset, clean the beginning of the article to remove header content, and then pick the 10 first words of the article as a prefix. We use this prefix as the prompt for the non-chat models to generate the rest of the article and prefixed it with a supporting prompt for chat models (see appendix <ref>). We let the model generate up to 200 tokens but cut the generation to have only the first 500 characters. We also cut the original articles to 500 characters to obtain the reference human samples. Using this procedure, human and generated samples are indistinguishable in length (see examples in appendix <ref>).


By using the method described above, we create 6 datasets with around 20K samples each (see appendix <ref> for the precise sizes) with a train, eval (10% of the whole size), and test split (10% of the whole size). In addition to the above procedure, we pair the AI-generated samples with their corresponding original news articles. We do that to prevent the fake and corresponding true samples from being in different training batches or data splits. We also filter out the generated news articles under 500 characters and discard the corresponding true sample to keep the dataset balanced. Finally, we also create a "round robin" dataset, which consists of a mixture of data generated by the 3 non-chat models, expected to train models more suitable for adversarial setting <cit.>.









    
    
    




 §.§ Training the Detectors


We finetuned the pretrained models described in subsection <ref> on each non-chat model training dataset and the round-robin dataset. The detectors are trained on the full datasets (1 epoch) with an evaluation after every 200 samples seen. We save only the model that obtained the best loss on the evaluation set to avoid overfitting. We list the hyperparameters for each training procedure in appendix <ref>.

While we tested different ways of finetuning the detectors on the datasets, we only retained full finetuning for the results section, consistently with the detector training SotA recommendations. We also provide some results when only finetuning a classification head and using the adapters PEFT method in appendix <ref>. We tracked the model performance on the pretraining task to confirm we were not overfitting the base models to our distribution (cf appendix <ref>).




 §.§ Testing the Detectors


We use the same metric across all the results to test the detectors: TPR (True-Positive rate). To obtain the detection prediction, we use a threshold on the output of the detector to target an FPR (False-Positive rate) of at most 5%. We find these thresholds by finding the TPR/FPR at different thresholds on the evaluation set for each dataset. We repeat this threshold-finding procedure for each detector (trained and zero-shot). This setting mimics a realistic scenario where a defender uses a detector to target a low level of false positives without access to true positive labels. We do not recompute a detection threshold on each attack since we consider these attacks unknown to the defender.



  §.§.§ Testing Fast Detect GPT

To test Fast-DetectGPT <cit.>, we use the script from their GitHub repository that we slightly adapted to run similarly to the other detectors we tested. We use GPT-J 6B B as the reference model since it provides more accurate results according to their work. 



  §.§.§ Testing GPTZero

To test GPTZero <cit.>, we used an academic partner API access, courtesy of GPTzero, version . In order to maintain consistency with the methodology presented here, we forced the FPR of GPTZero to 5%, given that by default, it is heavily biased to minimize FPR (<0.3% in our setting).



 §.§ Attacks against the Detectors


 


  §.§.§ Evasion Attacks


In experiments targeting detector evasion, we start with the test set created in the previous experiment when generating fake articles. We regenerate the fake news articles using either a different generation parameter or a different prompt. That way, we obtain a new dataset of fake articles with the same true original articles as the ones in the previous experiment, but where the fake articles are generated differently from the one used to produce the training and test data of the previous experiment.



  
Changing the Generation Parameters
The generation parameters we modify are the temperature and the repetition penalty. The first attack, called "high temperature," sets the temperature to 1.2 (note that OpenAI's API allows a temperature up to 2). The second, which we call "repetition penalty," consists in setting the repetition penalty to 1.2 (interestingly, https://github.com/huggingface/chat-uihugging chat uses a repetition penalty of 1.2). Both parameters heavily impact the diversity[By diversity, we mean the entropy of the logits for the next token prediction. Higher entropy means that the range of the next word that can be generated is higher, resulting in higher text diversity.] of the produced text, making it more difficult for detection methods that rely on the lack of diversity of AI-generated texts.




  
Prompting Attacks
 
We consider a few prompts to generate the fake news articles (the complete list can be found in appendix <ref>). The prompts we choose should cover a wide set of prompts an attacker may use. The idea for our experiment is to consider only basic attacks, i.e., attacks that do not require training an additional LLM (low resource) or special LLM expertise (low skill). While we are aware that more complex attacks exist, such as those described in the background (see <cit.> for example, which uses multiple LLMs to generate text), our intent is not to test exhaustively all possible attacks (which is not feasible), but to limit ourselves to the setting of our threat model. Our results show that we do not need to dig too far to find effective attacks, which is even more concerning.



  
Paraphrasing Attacks

Paraphrasing a generated text with an auxiliary LLM has been reported to achieve good evasion against a variety of detectors <cit.>. Given that it preserves semantic meaning while modifying probabilities of token selection, it is likely to erase non-semantic traces of LLM generation and, as such, to be highly performant. Here, we implement a simpler version of the attack than DIPPER <cit.>, prompting an LLM to reformulate the text (cf <ref> )
 


  §.§.§ Testing the Detectors on Human Text from a Different Distribution


Finally, we also test the detectors on 10000 samples from the Xsum dataset. Xsum comprises articles from BBC News, a different news outlet from the one we used to until now, based on CNN. XSum notably uses British English. This experiment tests the human-text generalization of detectors, so we did not create fake news articles.


 
 


§ RESULTS AND DISCUSSION






 §.§ Trained Detectors





Here, we present TPRs at FPR fixed to 5% for detectors where we finetune all parameters. Results obtained with other finetuning methods can be found in the appendix <ref>.



  
Performance on training LLM

We present in figure <ref> the TPR obtained when testing the trained detectors on the same dataset they have been trained on.

In our experiments, Electra (Electra-Large) consistently outperforms or comes equal with the other detectors. We can also notice that the difference is small between the distilled RoBERTa version (82.8M parameters) and its large version (355M params parameters). For the next results, we will show only the results using Electra as the detector, but the results for other detectors are available in the appendix <ref>. We also note that TPR with fixed FPR provides a more fine-grained performance evaluation compared to all-around stellar ROC-AUC values in Appendix Figure <ref> and that adapter and classification head underperform compared to full finetuning (cf. appendix <ref>).




  
Generalization across training LLMs

We show on the heatmap in figure <ref> the TPR of the trained detectors on the test of the different datasets generated by the LLMs, evaluating finetuned detector generalization across generating LLMs.



Surprisingly, most detectors perform almost equally, if not better in some cases, when detecting fake news articles generated by a different LLM than the one used to train them. This suggests a good generalization of the patterns leveraged by the finetuned detectors and a sufficient similarity in LLMs output in our setting.





  
Generalization to unseen chat LLMs

In this experiment, we repeat the same testing as above, but on 3 datasets generated by chat models that were not used to generate any training datasets (see appendix <ref> for the prompt). The results are in the same heatmap as for the previous experiment: figure <ref>. 

We find that the TPRs obtained here align with the finding above that the trained detectors surprisingly generalize enough to achieve similar TPR across data generated by instruction-tuned LLMs. We find that news articles generated by Zephyr are slightly harder to detect, whereas the ones generated by Llama 3-Instruct-8B are the easiest. This highlights that while we find encouraging generalization results across LLMs, the performance across LLMs remains disparate, consistently with prior findings <cit.>.



 §.§ Cross-LLM Generalization


We present in figure <ref> the results of the same experiment as above, but including zero-shot detectors, which we compare to one of the trained detectors. We observe that Fast-DetectGPT and GPTZero closely match but still underperform compared to Electra-Large trained to detect Mistral-generated texts (Electra_Mistral). We also observe that RoBERTa-OpenAI significantly underperforms, suggesting that it is now outdated and strongly arguing for abandoning it as a zero-shot general LLM detector. 

















Again, we find Zephyr LLM particularly challenging for all detectors and, going forward, will focus on it to best differentiate model performance. Curiously, we also observe that GPTZero drastically under-performs on non-chat models, consistently with previous benchmarks <cit.>, suggesting an overfitting to chat models.



 §.§ Evasion Attack Reslience









Here, we focus on the best-performing trained detector (Electra_Mistral) and the most challenging generator - Zephyr. We also include a detector trained on the mixture of data from all generators in a round-robin fashion (Electra_RR), which we expect to generalize well. We also include the previously mentioned zero-shot detectors. Following the method described in the Methodology (<ref>), we generate evasion attack datasets and present TPR (true-positive rate) at a fixed FPR of 5% in the table <ref>, as described in <ref>.


 


  
Changing the Generation Parameters

    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    

We observe that both generation parameters attacks against generation parameters are highly effective, with a modest temperature increase effectively defeating all zero-shot detectors. This is highly concerning, given that this is a trivial attack and has been described for over 5 years by the time of publication of this article <cit.>.

We hypothesize that by increasing the temperature and repetition penalty, we increase the LLM output diversity, which defeats LLM detectors based on text perplexity and expecting generated texts to be less diverse than human-written ones. We hypothesize that the poor performance of detectors against the temperature attack is due to its absence from any recent LLM detector benchmarks.









  
Prompting the LLM
        
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    

For the prompting attacks, we discover a high variation in terms of both the average effectiveness of the attacks and model-specific attack effectiveness, with none achieving universal effectiveness. The "News prompt" attack seems to be largely ineffective, while the "Example prompt" is only effective against GPTZero, and the "Tweet Prompt" attack is effective against all detectors but GPTZero, whose performance instead drastically increases.

We hypothesize that this variation is a result of a combination of factors, including the selection of reference human/machine datasets by detector developers and the domain-specific proficiency of generators. We believe this highlights the critical need for diverse testing of detectors, both domain-specific and threat-specific, and for exploring a wide range of prompting strategies.







  
Paraphrasing

We observed that the paraphrasing attack was only effective against Electra_Mistral and Fast-DetectGP, with the latter losing 40% TPR. This is somewhat unexpected, given prior reports of the effectiveness of this attack, although potentially due to using a prompted LLM reformulation rather than the DIPPER attack <cit.>.



  
Prior Benchmark comparison

A direct comparison to prior benchmarks is non-trivial. Given the drastic difference in performance of different prompting strategies, minor differences in benchmark implementation matter. In our case, the only benchmark providing sufficiently detailed results is RAID <cit.>, thanks to its leaderboard. Specifically, we would expect the results for their "News" domain (BBC articles), generated by the Mistral-chat model, to be close to our setting of news-like content (CNN articles trimmed to 500 characters), generated by Mistral-based Zephyr.

Unfortunately, this is not the case. Even in the absence of attacks, RAID suggests RoBERTa-Base-OpenAI-GPT2 matches FastDetectGPT (0.99 TPR for both), whereas we observe a drastically lower and heterogeneous performance (0.57 and 0.86 TPR respectively). A repetition penalty attack, sharing the same parameters for both our and RAID benchmarks leads to a similar difference (0.5 and 0.4 TPR in RAID, respectively, vs. 0.26 and 0.24 TPR for us). Likewise, for the paraphrasing attacks, RAID predicts Fast-DetectGPT to remain highly effective (0.95 TPR), while we observe its performance halved (0.45 TPR), which is even more surprising given that we used a weaker paraphrasing attack.

This means that domain-specific and threat model-specific benchmarking is essential to judge the real-world performance of LLM detectors. In our opinion, this argues against the utility of large-scale benchmarks with once-generated static data and in favor of dynamic, application-specific benchmarks. 












 §.§ Performance on Unseen Human Texts


In the previous subsection, we showed that while a moderately sophisticated attacker could defeat all detectors, one of our custom-trained detectors - Electra_RR - exhibited an outstanding resilience to evasion attacks, frequently presenting the best TPR, and if not - the second best, with a usable and consistent 0.84 TPR, arguing in favor of Round-Robin training utility, previously seen in GANs <cit.>. Such resilience is surprising, given it was not trained against adversarial evasion and did not generalize particularly well across generative LLMs, and could lead us to suppose that Electra_RR would perform well in new evasion attacks, hence claiming a new SotA.

However, such a claim would be misplaced. In real-world deployments, LLM detectors must not only generalize across unseen generators and attacks but also across unseen human texts, maintaining an acceptable FPR across different types of texts. We test for this in table <ref>, verifying the generalization from the CNN News dataset to the BBC News Xsum dataset, which are closely related and differ most likely only by US English vs British English usage. Despite such close relatedness, we observe that Electra_RR fails dramatically in human text generalization along with Electra_Mistral, indicating that neither of our trained detectors can be deployed to the real world. 







    

Please note that GPTZero FPR has been forced to 5% here. With default configurations, the accuracy is over 99.7% on both datasets.

Unfortunately, such tests for out-of-distribution human texts are all but absent for LLM detector benchmarks, with only anecdotic reports of real-world performance failure <cit.> raising awareness of this failure mode. Unlike what we could expect from prior works showing generalization of LLM detector across languages <cit.>, and an effective underlying pretrained model to our classifier, the human text generalization cannot be assumed and must be tested in any LLM detector benchmarks for them to be useful.









 



§ CONCLUSION


In this paper, we developed a rigorous framework to benchmark LLM detectors in a way specific to a domain and threat model. By applying it to a setting relevant to LLM-augmented information operations, we show that current LLM detectors are not ready for real-world use due to a combination of susceptibility to trivial evasion attacks, notably generation temperature increase, and potentially unacceptable FPRs in practice, consistently with noted issues in other domains <cit.>. Overall, we believe our results argue in favor of alternative approaches in that context, e.g., coordinated activity patterns search <cit.>.

In the process, we noted several methodological issues with existing LLM detector benchmarks, notably missing evaluation of FPR on out-of-distribution human texts, incomplete coverage of evasion attacks, and the failure of large-scale benchmarks to predict detector performance in even closely related settings. We believe this argues in favor of switching away from large-scale static benchmarks to dynamic benchmarks that can be run against a specific domain and threat model. To allow that, we provide the dynamic benchmarking suite we developed to the scientific community under the MIT license (<https://github.com/Reliable-Information-Lab-HEVS/dynamic_llm_detector_benchmark>), which we designed to be fully domain and language transferable for non-expert teams, and expandable with new attacks for more technical teams.




 





















§ LIMITATIONS


While we focus on a setting highly relevant to in-the-wild LLM text recognition, our work has several limitations. 

First, we focus on a short-story setting and use a well-known NLP dataset as a reference human text. Due to NLP dataset reuse for model training, the performance of the generative task is likely to be higher than that of actual information operations. Similarly, social media accounts without posting history are rare, and puppet accounts are likely to have similarly LLM-generated past posts. Such longer texts could increase confidence that an account is unauthentic. However, misappropriation of social media accounts for information operations is common, and such unauthentic post history is not guaranteed.

Second, we have not investigated text watermarking approaches. If future on-device LLM releases more tightly integrated models and supporting code and enables watermarking, it could potentially improve the detectability of such posts. While not impossible, recent work by <cit.> suggests that reformulation attacks remain effective in this setting, warranting caution as to their effectiveness.

Third, we only tested a limited number of zero-shot detectors. While it is possible that untested detectors could clear both adversarial evasion and generalization tests, it is unlikely. <cit.> and <cit.> performed extensive benchmarking of recent zero-shot detectors, and both Fast-DetectGPT and GPTZero closely match other solutions across a large palette of tests, including adversarial perturbations.

Fourth, we assume the attacker has limited capabilities, notably lacking adversarial evasive model fine-tuning. This assumption is somewhat brittle, given that parameter-efficient fine-tuning (PEFT) requires minimal resource overhead compared to traditional fine-tuning. PEFT can be performed on quantized models from relatively small datasets and hardware adapted for quantized inference, putting them within the reach of moderately sophisticated attackers we consider. While this opens a new type of attack, current LLM detectors are easy enough to fool even without it.

Finally, our work focused on the English language, for which extensive resources are available and are leveraged by LLM developers. While our results could generalize to other high-resource Romance Languages such as French or Spanish, LLM performance in other languages, especially low-resource ones, is unlikely to induce native speakers into confusion, making the LLM disinformation detection a significantly less salient problem as of now.



§ ETHICS STATEMENT


While the issue of deep neural disinformation is critical and a central concern for malicious misuse of LLMs, it has not prevented the release of powerful SotA models to the general public that can readily assist in such operations. Here, we do not present any novel attacks but demonstrate that existent ones are sufficient to evade SotA attacks. As such, we do not expect novel risks to arise from this work but rather to improve the general awareness of common tools' limitations and contribute to mitigating known risks arising from LLMs.

GPTZero is a security solution numerous entities use to detect LLM-generated text in potentially safety-critical contexts. Given that in this work, we found several highly effective attacks against it, we performed a coordinated vulnerability disclosure with them.

We used 1 A100 GPU on a local cluster to generate the datasets, 30 minutes per dataset. We used 1 A100 GPU to train the models and perform detection using Fast-DetectGPT. For testing, we only used V100 GPUs thanks to our focus on smaller models. In total, we used the local cluster for 30 hours of GPU-days on the A100 and 20 hours of GPU-days on the V100 (numbers including hyperparameter search and testing correctness), leading to total emissions of 5.4kg of CO2. No crowdsourced labor was used in this work. LLM assistants were used for minor stylistic and grammatical corrections of the final manuscript, consistently with ACL recommendations. GitHub Copilot has been used to assist in coding with auto-completion but no script or algorithm has been fully generated with it.


acl_natbib








§ DATASETS AND DATA GENERATION DETAILS




 §.§ Datasets List



We created 6 datasets with a balanced number of fake and true samples (1 dataset per generator). See table <ref> for the list of generators used for the datasets and the precise size of the datasets. The datasets are split with 80% for training, 10% for eval (choosing the best model to save), and 10% for testing. Before cutting the samples to the 500 characters threshold, we filtered out samples smaller than 500 characters to only have samples of 500 characters. A small improvement here would be to use the "min_new_token" generation parameter so that we would not need to filter out smaller samples. We only used this parameter to generate the adversarial datasets. The samples of the true articles are the same across all datasets except for the discarded samples.

There is also a 4th auto-complete model dataset, "round-robin," which is a mixture of 2500 samples from each of the other complete models' (Phi-2, Gemma, and Mistral) datasets.

For the adversarial datasets, we reuse these same datasets, but we generate the fake samples with a different prompt or generation parameter (see appendix <ref> for the adversarial prompts). In total, there is one adversarial version of each dataset per attack (i.e. number of datasets times number of attacks adversarial datasets). 

    

    


 §.§ Prompts for Generating Data



For the auto-complete models, we only use the prefix as the prompt. For chat models, we use the prompt in table <ref>. Also, we force the first tokens of the output to be the prefix for that particular sample. This prevents the chat model from generating typical assistant-starting messages that would be too obvious to spot for the detectors (in practice, an attacker could also remove it easily). For Gemma-2B-it (Gemma chat), there is no system prompt in the chat template; we simply dropped the system prompt for that case. 
    



 §.§ Dataset Example



Below is an example of a true and an LLM-generated news article from the Zephyr dataset. As explained in the methodology, samples are regrouped by pairs of true and fake samples (ordered randomly within the pair). The true and fake samples in the pair start with the same 10-word prefix.
True sample:


"Former Vice President Dick Cheney on Sunday defended the Bush administration's economic record, the invasion of Iraq and the treatment of suspected terrorists, warning that reversing its anti-terrorism policies endangers Americans. "We've accomplished nearly everything we set out to do," ex-Vice President Dick Cheney says Sunday about Iraq. In a wide-ranging interview with CNN's "State of the Union," Cheney said the harsh interrogations of suspects and the use of warrantless electronic surveilla"


source: <https://edition.cnn.com/2009/POLITICS/03/15/cheney.interview/>


Fake sample:


"Former Vice President Dick Cheney on Sunday defended the Bush administration's use of enhanced interrogation techniques, commonly referred to as torture, in the aftermath of the September 11 attacks. Speaking at the Reagan Library's National Security Forum in Simi Valley, California, Cheney argued that the use of waterboarding and other techniques were crucial in obtaining valuable intelligence and preventing further attacks on American soil. "We did learn a lot," Cheney said. "We learned, for e"


source: <https://huggingface.co/HuggingFaceH4/zephyr-7b-beta>




 §.§ Attack Prompts



    
    
    
    
    
    
       
    
    
    
    
    
The idea behind the attacks we crafted to generate news articles evading the detectors is to generate news articles with a distribution of words looking more like the CNN news articles that were used to train the detectors. This is particularly true for the "news prompt" attack that asks the model to generate a CNN news-looking article. The same applies to the "example prompt," which uses in-context learning to generate a more CNN news-looking article. The news prompt can be found below in <ref> and the example prompt in <ref>.

For the "tweet prompt," the idea is to generate text with a different distribution than news articles, which might confuse detectors trained on news articles. The "paraphrasing prompt" has a similar effect of modifying the distribution of words to make it more diverse than the original output. The tweet prompt can be found in <ref> and the other paraphrasing prompt in <ref>.













§ TRAINING DETAILS




 §.§ Hyperparameters for Training



We present in table <ref> the hyperparameters used for training. The training was done either on the Nvidia A100 40GB or on the Nvidia V100 16GB, depending on the batch size, model size, and training method. All the trainings were done with 1 epoch. While we used only the models trained with full finetuning on the main part of the paper, we provide some results with different training methods in appendix <ref>. The hyperparameters were chosen according to the hyperparameters used in the original papers of the different models with some adaptation according to the size of the models and testing that the training converges. A linear schedule with a 10% warmup was applied to the learning rate.

        



§ FULL HEATMAP FOR TRAINED DETECTORS


You can find in figure <ref> the TPR obtained for all trained detectors when testing them on all the datasets (it is a more complete version of figure <ref>). This plot uses the same metric as the main paper plots, i.e. we find thresholds on the eval set such that we target an FPR of at most 5 %.




§ ROC AUC SCORE WITH DIFFERENT TRAINING METHODS



While we mainly tested trained detectors with full finetuning, we also tested and compared the results using different training methods. First, we tested freezing the LLM detector model's parameters and finetuning the classifier head (freeze base). Secondly, we tested finetuning the base LLM detector with the adapters PEFT method (see <cit.>). The results we show in this section are the same experiment as presented above in appendix <ref> and in figure <ref>. However, we decide to use ROC-AUC as the metric to have a threshold-independent metric, avoiding finding a suitable threshold for each case. Also, we compare this ROC-AUC score obtained with freeze base (figure <ref>) and adapters (figure <ref>) method to the full-finetuning ROC-AUC (figure <ref>).

We find that training with the adapters method yields results similar to training with full-finetuning, as expected. This gives an interesting alternative to train detectors.



 §.§ Training with Finetuning Only the Classification Head

See figure <ref>.





 §.§ Training Adapter Method

See figure <ref>.





 §.§ Training with Full Finetuning

See figure <ref>.





§ CHECKING DETECTOR DEGRADATION ON MLM TASK



To compare the effect of different finetuning methods on the trained detectors, we evaluate the loss on the MLM (Masked Language Model) task before and during training. In particular, we use samples from <https://huggingface.co/datasets/Polyglot-or-Not/Fact-Completion?row=0>, a dataset of knowledge facts where the task is to fill the gap. This enables us to test if the base models "forgotten facts" during the finetuning. This is shown in figure <ref> with the degradation loss that shows how the loss on this MLM task evolves during training. We also provide a baseline where the gap is filled by a model without pre-trained weights. As shown in the figure, the degradation increases at the beginning of training and then stays almost constant. By comparing with figure <ref>, the eval loss stops improving at the same moment the degradation loss stops increasing, showing signs of convergence. Also, we noticed that the degradation was slightly lower when finetuning with the adapters using the PEFT method. However, the degradation difference between the two training methods might be higher with different tasks and training hyperparameters.












§ FULL ATTACK TESTING RESULTS



We provide here the same results as in <ref>, 

but with data also generated by Gemma-2b-it (Gemma-Chat) and Llama-3 Instruct (7B). As shown in table <ref>, attacks using Gemma-Chat are less effective. We find that the attacks are often on par with the effectiveness using Zephyr, sometimes above, sometimes below (in terms of TPR difference).
    




§ MODELS, DATASETS AND THIRD-PARTY CODE







§ OLD TABLES (TO BE REMOVED)

















    
    

    


    
    

    
        2*Attack     2*Detector     5cDatasetl 

                  Gemma Chat     Zephyr     Avg. 
    
        5*High temperature
            Roberta_Phi-2     0.9912*     0.9268  

            Roberta_Phi-2     0.9890     0.9721  

            (Diff)     -0.0021     0.0452 

        2-7
            Fast-DetectGPT (No attack)     0.7365     0.8713 

            Fast-DetectGPT (With attack)     0.8886 

            (Diff)     0.1521     0.0495 

        

    
    


    
    


    


