Upload 30 files

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Bao Guangsheng
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,14 +1,80 @@
----
-title: Fast Detect Gpt
-emoji: 🏆
-colorFrom: indigo
-colorTo: red
-sdk: streamlit
-sdk_version: 1.41.0
-app_file: app.py
-pinned: false
-license: mit
-short_description: analiz
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Fast-DetectGPT
+**This code is for ICLR 2024 paper "Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature"**, where we borrow or extend some code from [DetectGPT](https://github.com/eric-mitchell/detect-gpt).
+
+[Paper](https://arxiv.org/abs/2310.05130) 
+| [LocalDemo](#local-demo)
+| [OnlineDemo](http://region-9.autodl.pro:21504/)
+| [OpenReview](https://openreview.net/forum?id=Bpcgcr8E8Z)
+
+
+## Brief Intro
+<table class="tg"  style="padding-left: 30px;">
+  <tr>
+    <th class="tg-0pky">Method</th>
+    <th class="tg-0pky">5-Model Generations ↑</th>
+    <th class="tg-0pky">ChatGPT/GPT-4 Generations ↑</th>
+    <th class="tg-0pky">Speedup ↑</th>
+  </tr>
+  <tr>
+    <td class="tg-0pky">DetectGPT</td>
+    <td class="tg-0pky">0.9554</td>
+    <td class="tg-0pky">0.7225</td>
+    <td class="tg-0pky">1x</td>
+  </tr>
+  <tr>
+    <td class="tg-0pky">Fast-DetectGPT</td>
+    <td class="tg-0pky">0.9887 (relative↑ <b>74.7%</b>)</td>
+    <td class="tg-0pky">0.9338 (relative↑ <b>76.1%</b>)</td>
+    <td class="tg-0pky"><b>340x</b></td>
+  </tr>
+</table>
+The table shows detection accuracy (measured in AUROC) and computational speedup for machine-generated text detection. The <b>white-box setting</b> (directly using the source model) is used for detecting generations produced by five source models (5-model), whereas the <b>black-box
+setting</b> (utilizing surrogate models) targets ChatGPT and GPT-4 generations. AUROC results are averaged across various datasets and source models. Speedup assessments were conducted on a Tesla A100 GPU.
+
+
+## Environment
+* Python3.8
+* PyTorch1.10.0
+* Setup the environment:
+  ```bash setup.sh```
+  
+(Notes: our experiments are run on 1 GPU of Tesla A100 with 80G memory.)
+
+## Local Demo
+Please run following command locally for an interactive demo:
+```
+python scripts/local_infer.py
+```
+where the default reference and sampling models are both gpt-neo-2.7B.
+
+We could use gpt-j-6B as the reference model to obtain more accurate detections:
+```
+python scripts/local_infer.py  --reference_model_name gpt-j-6B
+```
+
+
+An example (using gpt-j-6B as the reference model) looks like
+```
+Please enter your text: (Press Enter twice to start processing)
+Disguised as police, they broke through a fence on Monday evening and broke into the cargo of a Swiss-bound plane to take the valuable items. The audacious heist occurred at an airport in a small European country, leaving authorities baffled and airline officials in shock.
+
+Fast-DetectGPT criterion is 1.9299, suggesting that the text has a probability of 87% to be machine-generated.
+```
+
+## Workspace
+Following folders are created for our experiments:
+* ./exp_main -> experiments for 5-model generations (main.sh).
+* ./exp_gpt3to4 -> experiments for GPT-3, ChatGPT, and GPT-4 generations (gpt3to4.sh).
+
+(Notes: we share <b>generations from GPT-3, ChatGPT, and GPT-4</b> in exp_gpt3to4/data for convenient reproduction.)
+
+### Citation
+If you find this work useful, you can cite it with the following BibTex entry:
+
+    @inproceedings{bao2023fast,
+      title={Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature},
+      author={Bao, Guangsheng and Zhao, Yanbin and Teng, Zhiyang and Yang, Linyi and Zhang, Yue},
+      booktitle={The Twelfth International Conference on Learning Representations},
+      year={2023}
+    }
+

attack.sh

@@ -0,0 +1,85 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+para=t5  # "t5" for paraphrasing attack, or "random" for decoherence attack
+exp_path=exp_attack
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+src_path=exp_gpt3to4
+src_data_path=$src_path/data
+
+datasets="xsum writing pubmed"
+source_models="gpt-3.5-turbo"
+
+# preparing dataset
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Preparing dataset ${D}_${M} by paraphrasing  ${src_data_path}/${D}_${M} ...
+    python scripts/paraphrasing.py --dataset $D --dataset_file $src_data_path/${D}_${M} \
+                --paraphraser $para --output_file $data_path/${D}_${M}
+  done
+done
+
+# evaluate Fast-DetectGPT in the black-box setting
+settings="gpt-j-6B:gpt2-xl gpt-j-6B:gpt-neo-2.7B gpt-j-6B:gpt-j-6B"
+for D in $datasets; do
+  for M in $source_models; do
+    for S in $settings; do
+      IFS=':' read -r -a S <<< $S && M1=${S[0]} && M2=${S[1]}
+      echo `date`, Evaluating Fast-DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/fast_detect_gpt.py --reference_model_name $M1 --scoring_model_name $M2 --discrepancy_analytic \
+                          --dataset $D --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate supervised detectors
+supervised_models="roberta-base-openai-detector roberta-large-openai-detector"
+for D in $datasets; do
+  for M in $source_models; do
+    for SM in $supervised_models; do
+      echo `date`, Evaluating ${SM} on ${D}_${M} ...
+      python scripts/supervised.py --model_name $SM --dataset $D \
+                            --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+    done
+  done
+done
+
+# evaluate fast baselines
+scoring_models="gpt-neo-2.7B"
+for D in $datasets; do
+  for M in $source_models; do
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating baseline methods on ${D}_${M}.${M2} ...
+      python scripts/baselines.py --scoring_model_name ${M2} --dataset $D \
+                            --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M2}
+    done
+  done
+done
+
+# evaluate DetectGPT and DetectLLM
+scoring_models="gpt2-xl gpt-neo-2.7B gpt-j-6B"
+for D in $datasets; do
+  for M in $source_models; do
+    M1=t5-11b  # perturbation model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_gpt.py --mask_filling_model_name ${M1} --scoring_model_name ${M2} --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+      # we leverage DetectGPT to generate the perturbations
+      echo `date`, Evaluating DetectLLM methods on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_llm.py --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.${M1}.perturbation_100 --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done

baselines.py

@@ -0,0 +1,137 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+import tqdm
+import argparse
+import json
+from data_builder import load_data
+from model import load_tokenizer, load_model
+from metrics import get_roc_metrics, get_precision_recall_metrics
+
+def get_likelihood(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+
+    logits = logits.view(-1, logits.shape[-1])
+    labels = labels.view(-1)
+    log_probs = torch.nn.functional.log_softmax(logits, dim=-1)
+    log_likelihood = log_probs.gather(dim=-1, index=labels.unsqueeze(-1)).squeeze(-1)
+    return log_likelihood.mean().item()
+
+def get_rank(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+
+    # get rank of each label token in the model's likelihood ordering
+    matches = (logits.argsort(-1, descending=True) == labels.unsqueeze(-1)).nonzero()
+    assert matches.shape[1] == 3, f"Expected 3 dimensions in matches tensor, got {matches.shape}"
+
+    ranks, timesteps = matches[:, -1], matches[:, -2]
+
+    # make sure we got exactly one match for each timestep in the sequence
+    assert (timesteps == torch.arange(len(timesteps)).to(timesteps.device)).all(), "Expected one match per timestep"
+
+    ranks = ranks.float() + 1 # convert to 1-indexed rank
+    return -ranks.mean().item()
+
+def get_logrank(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+
+    # get rank of each label token in the model's likelihood ordering
+    matches = (logits.argsort(-1, descending=True) == labels.unsqueeze(-1)).nonzero()
+    assert matches.shape[1] == 3, f"Expected 3 dimensions in matches tensor, got {matches.shape}"
+
+    ranks, timesteps = matches[:, -1], matches[:, -2]
+
+    # make sure we got exactly one match for each timestep in the sequence
+    assert (timesteps == torch.arange(len(timesteps)).to(timesteps.device)).all(), "Expected one match per timestep"
+
+    ranks = ranks.float() + 1  # convert to 1-indexed rank
+    ranks = torch.log(ranks)
+    return -ranks.mean().item()
+
+def get_entropy(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+
+    entropy = F.softmax(logits, dim=-1) * F.log_softmax(logits, dim=-1)
+    entropy = -entropy.sum(-1)
+    return entropy.mean().item()
+
+
+def experiment(args):
+    # load model
+    scoring_tokenizer = load_tokenizer(args.scoring_model_name, args.dataset, args.cache_dir)
+    scoring_model = load_model(args.scoring_model_name, args.device, args.cache_dir)
+    scoring_model.eval()
+    # load data
+    data = load_data(args.dataset_file)
+    n_samples = len(data["sampled"])
+    # eval criterions
+    criterion_fns = {'likelihood': get_likelihood,
+                     'rank': get_rank,
+                     'logrank': get_logrank,
+                     'entropy': get_entropy}
+    for name in criterion_fns:
+        criterion_fn = criterion_fns[name]
+        torch.manual_seed(args.seed)
+        np.random.seed(args.seed)
+        eval_results = []
+        for idx in tqdm.tqdm(range(n_samples), desc=f"Computing {name} criterion"):
+            original_text = data["original"][idx]
+            sampled_text = data["sampled"][idx]
+            # original text
+            tokenized = scoring_tokenizer(original_text, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+            labels = tokenized.input_ids[:, 1:]
+            with torch.no_grad():
+                logits = scoring_model(**tokenized).logits[:, :-1]
+                original_crit = criterion_fn(logits, labels)
+            # sampled text
+            tokenized = scoring_tokenizer(sampled_text, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+            labels = tokenized.input_ids[:, 1:]
+            with torch.no_grad():
+                logits = scoring_model(**tokenized).logits[:, :-1]
+                sampled_crit = criterion_fn(logits, labels)
+            # result
+            eval_results.append({"original": original_text,
+                            "original_crit": original_crit,
+                            "sampled": sampled_text,
+                            "sampled_crit": sampled_crit})
+
+        # compute prediction scores for real/sampled passages
+        predictions = {'real': [x["original_crit"] for x in eval_results],
+                       'samples': [x["sampled_crit"] for x in eval_results]}
+        fpr, tpr, roc_auc = get_roc_metrics(predictions['real'], predictions['samples'])
+        p, r, pr_auc = get_precision_recall_metrics(predictions['real'], predictions['samples'])
+        print(f"Criterion {name}_threshold ROC AUC: {roc_auc:.4f}, PR AUC: {pr_auc:.4f}")
+        # log results
+        results_file = f'{args.output_file}.{name}.json'
+        results = { 'name': f'{name}_threshold',
+                    'info': {'n_samples': n_samples},
+                    'predictions': predictions,
+                    'raw_results': eval_results,
+                    'metrics': {'roc_auc': roc_auc, 'fpr': fpr, 'tpr': tpr},
+                    'pr_metrics': {'pr_auc': pr_auc, 'precision': p, 'recall': r},
+                    'loss': 1 - pr_auc}
+        with open(results_file, 'w') as fout:
+            json.dump(results, fout)
+            print(f'Results written into {results_file}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_test/results/xsum_gpt2")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--dataset_file', type=str, default="./exp_test/data/xsum_gpt2")
+    parser.add_argument('--scoring_model_name', type=str, default="gpt2")
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    experiment(args)

custom_datasets.py

@@ -0,0 +1,96 @@
+import os.path
+import random
+import datasets
+
+SEPARATOR = '<<<SEP>>>'
+
+
+DATASETS = ['writing', 'english', 'german', 'pubmed']
+
+def load_dataset(path, name=None, split=None, cache_dir=None):
+    # use local model if it exists
+    local_path = os.path.join(cache_dir, f'local.{path}_{name}_{split}')
+    if os.path.exists(local_path):
+        return datasets.load_from_disk(local_path)
+    return datasets.load_dataset(path, name, split=split, cache_dir=cache_dir)
+
+def load_pubmed(cache_dir):
+    data = load_dataset('pubmed_qa', 'pqa_labeled', split='train', cache_dir=cache_dir)
+    
+    # combine question and long_answer
+    data = [f'Question: {q} Answer:{SEPARATOR}{a}' for q, a in zip(data['question'], data['long_answer'])]
+
+    return data
+
+
+def process_prompt(prompt):
+    return prompt.replace('[ WP ]', '').replace('[ OT ]', '')
+
+
+def process_spaces(story):
+    return story.replace(
+        ' ,', ',').replace(
+        ' .', '.').replace(
+        ' ?', '?').replace(
+        ' !', '!').replace(
+        ' ;', ';').replace(
+        ' \'', '\'').replace(
+        ' ’ ', '\'').replace(
+        ' :', ':').replace(
+        '<newline>', '\n').replace(
+        '`` ', '"').replace(
+        ' \'\'', '"').replace(
+        '\'\'', '"').replace(
+        '.. ', '... ').replace(
+        ' )', ')').replace(
+        '( ', '(').replace(
+        ' n\'t', 'n\'t').replace(
+        ' i ', ' I ').replace(
+        ' i\'', ' I\'').replace(
+        '\\\'', '\'').replace(
+        '\n ', '\n').strip()
+
+
+def load_writing(cache_dir=None):
+    writing_path = 'data/writingPrompts'
+    
+    with open(f'{writing_path}/valid.wp_source', 'r') as f:
+        prompts = f.readlines()
+    with open(f'{writing_path}/valid.wp_target', 'r') as f:
+        stories = f.readlines()
+    
+    prompts = [process_prompt(prompt) for prompt in prompts]
+    joined = [process_spaces(prompt + " " + story) for prompt, story in zip(prompts, stories)]
+    filtered = [story for story in joined if 'nsfw' not in story and 'NSFW' not in story]
+
+    random.seed(0)
+    random.shuffle(filtered)
+
+    return filtered
+
+
+def load_language(language, cache_dir):
+    # load either the english or german portion of the wmt16 dataset
+    assert language in ['en', 'de']
+    d = load_dataset('wmt16', 'de-en', split='train', cache_dir=cache_dir)
+    docs = d['translation']
+    desired_language_docs = [d[language] for d in docs]
+    lens = [len(d.split()) for d in desired_language_docs]
+    sub = [d for d, l in zip(desired_language_docs, lens) if l > 100 and l < 150]
+    return sub
+
+
+def load_german(cache_dir):
+    return load_language('de', cache_dir)
+
+
+def load_english(cache_dir):
+    return load_language('en', cache_dir)
+
+
+def load(name, cache_dir, **kwargs):
+    if name in DATASETS:
+        load_fn = globals()[f'load_{name}']
+        return load_fn(cache_dir=cache_dir, **kwargs)
+    else:
+        raise ValueError(f'Unknown dataset {name}')

data_builder.py

@@ -0,0 +1,276 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import time
+
+import numpy as np
+import datasets
+import torch
+import random
+import argparse
+import os
+import json
+import custom_datasets
+from model import load_tokenizer, load_model
+
+
+def save_data(output_file, args, data):
+    # write args to file
+    args_file = f"{output_file}.args.json"
+    with open(args_file, "w") as fout:
+        json.dump(args.__dict__, fout, indent=4)
+        print(f"Args written into {args_file}")
+
+    # write the data to a json file in the save folder
+    data_file = f"{output_file}.raw_data.json"
+    with open(data_file, "w") as fout:
+        json.dump(data, fout, indent=4)
+        print(f"Raw data written into {data_file}")
+
+
+def load_data(input_file):
+    data_file = f"{input_file}.raw_data.json"
+    with open(data_file, "r") as fin:
+        data = json.load(fin)
+        print(f"Raw data loaded from {data_file}")
+    return data
+
+
+class DataBuilder:
+    def __init__(self, args):
+        self.args = args
+        self.base_tokenizer = load_tokenizer(args.base_model_name, args.dataset, args.cache_dir)
+        self.base_model = None if args.openai_model else load_model(args.base_model_name, args.device, args.cache_dir)
+
+    def _openai_sample(self, prefix):
+        def _drop_last_word(text):
+            return ' '.join(text.split(' ')[:-1])
+
+        import openai
+        assert self.args.openai_key is not None, "Must provide OpenAI API key as --openai_key"
+        openai.api_key = self.args.openai_key
+        if self.args.openai_base is not None:
+            openai.api_base = self.args.openai_base
+
+        if self.args.dataset != 'pubmed':  # keep Answer: prefix for pubmed
+            prefix = _drop_last_word(prefix)
+
+        # sample from the openai model
+        kwargs = {"max_tokens": 200}
+        if self.args.do_top_p:
+            kwargs['top_p'] = self.args.top_p
+        elif self.args.do_top_k:
+            kwargs['top_k'] = self.args.top_k
+        elif self.args.do_temperature:
+            kwargs['temperature'] = self.args.temperature
+
+        if self.args.openai_model == 'davinci':
+            kwargs["engine"] = self.args.openai_model
+            response = openai.Completion.create(prompt=f"{prefix}", **kwargs)
+            return prefix + response['choices'][0]['text']
+
+        elif self.args.openai_model in ['gpt-3.5-turbo', 'gpt-4']:
+            roles = {'xsum': 'You are a News writer.',
+                     'writing': 'You are a Fiction writer.',
+                     'pubmed': 'You are a Technical writer.'}
+            prompts = {'xsum': 'Please write an article with about 150 words starting exactly with:',
+                       'writing': 'Please write an article with about 150 words starting exactly with:',
+                       'pubmed': 'Please answer the question in about 50 words.'}
+            messages = [
+                {'role': 'system', 'content': roles[self.args.dataset]},
+                {'role': 'user', 'content': f'{prompts[self.args.dataset]} {prefix}'},
+            ]
+            kwargs["model"] = self.args.openai_model
+            kwargs["messages"] = messages
+            response = openai.ChatCompletion.create(**kwargs)
+            response = response['choices'][0]['message']['content']
+            # ChatGPT may repeat the prefix
+            if response.startswith(prefix[:20]):
+                return response
+            return prefix + ' ' + response
+
+        else:
+            raise NotImplementedError
+
+    # sample from base_model using ****only**** the first 30 tokens in each example as context
+    def _sample_from_model(self, texts, min_words=55, prompt_tokens=30):
+        # encode each text as a list of token ids
+        if self.args.dataset == 'pubmed':
+            texts = [t[:t.index(custom_datasets.SEPARATOR)] for t in texts]
+            all_encoded = self.base_tokenizer(texts, return_tensors="pt", padding=True, return_token_type_ids=False).to(self.args.device)
+        else:
+            all_encoded = self.base_tokenizer(texts, return_tensors="pt", padding=True, return_token_type_ids=False).to(self.args.device)
+            all_encoded = {key: value[:, :prompt_tokens] for key, value in all_encoded.items()}
+
+        if self.args.openai_model:
+            # decode the prefixes back into text
+            prefixes = self.base_tokenizer.batch_decode(all_encoded['input_ids'], skip_special_tokens=True)
+
+            decoded = []
+            for idx, prefix in enumerate(prefixes):
+                while idx >= len(decoded):
+                    try:
+                        decoded.append(self._openai_sample(prefix))
+                    except Exception as ex:
+                        print(ex)
+                        print('Wait 10 minutes before retry ...')
+                        time.sleep(600)
+
+        else:
+            self.base_model.eval()
+            decoded = ['' for _ in range(len(texts))]
+
+            # sample from the model until we get a sample with at least min_words words for each example
+            # this is an inefficient way to do this (since we regenerate for all inputs if just one is too short), but it works
+            tries = 0
+            m = 0
+            while m < min_words:
+                if tries != 0:
+                    print()
+                    print(f"min words: {m}, needed {min_words}, regenerating (try {tries})")
+                    prefixes = self.base_tokenizer.batch_decode(all_encoded['input_ids'], skip_special_tokens=True)
+                    for prefix, x in zip(prefixes, decoded):
+                        if len(x.split()) == m:
+                            print(prefix, '=>', x)
+
+                sampling_kwargs = {}
+                if self.args.do_top_p:
+                    sampling_kwargs['top_p'] = self.args.top_p
+                elif self.args.do_top_k:
+                    sampling_kwargs['top_k'] = self.args.top_k
+                elif self.args.do_temperature:
+                    sampling_kwargs['temperature'] = self.args.temperature
+                min_length = 50 if self.args.dataset in ['pubmed'] else 150
+                outputs = self.base_model.generate(**all_encoded, min_length=min_length, max_length=200, do_sample=True,
+                                                   **sampling_kwargs, pad_token_id=self.base_tokenizer.eos_token_id,
+                                                   eos_token_id=self.base_tokenizer.eos_token_id)
+                decoded = self.base_tokenizer.batch_decode(outputs, skip_special_tokens=True)
+                m = min(len(x.split()) for x in decoded)
+                tries += 1
+
+        return decoded
+
+    def generate_samples(self, raw_data, batch_size):
+        # trim to shorter length
+        def _trim_to_shorter_length(texta, textb):
+            # truncate to shorter of o and s
+            shorter_length = min(len(texta.split(' ')), len(textb.split(' ')))
+            texta = ' '.join(texta.split(' ')[:shorter_length])
+            textb = ' '.join(textb.split(' ')[:shorter_length])
+            return texta, textb
+
+        def _truncate_to_substring(text, substring, idx_occurrence):
+            # truncate everything after the idx_occurrence occurrence of substring
+            assert idx_occurrence > 0, 'idx_occurrence must be > 0'
+            idx = -1
+            for _ in range(idx_occurrence):
+                idx = text.find(substring, idx + 1)
+                if idx == -1:
+                    return text
+            return text[:idx]
+
+        data = {
+            "original": [],
+            "sampled": [],
+        }
+
+        for batch in range(len(raw_data) // batch_size):
+            print('Generating samples for batch', batch, 'of', len(raw_data) // batch_size)
+            original_text = raw_data[batch * batch_size:(batch + 1) * batch_size]
+            sampled_text = self._sample_from_model(original_text, min_words=30 if self.args.dataset in ['pubmed'] else 55)
+
+            for o, s in zip(original_text, sampled_text):
+                if self.args.dataset == 'pubmed':
+                    s = _truncate_to_substring(s, 'Question:', 2)
+                    o = o.replace(custom_datasets.SEPARATOR, ' ')
+
+                o, s = _trim_to_shorter_length(o, s)
+
+                # add to the data
+                data["original"].append(o)
+                data["sampled"].append(s)
+
+        return data
+
+def generate_data(args, dataset, key):
+    # strip newlines from each example; replace one or more newlines with a single space
+    def _strip_newlines(text):
+        return ' '.join(text.split())
+
+    # load data
+    if dataset in custom_datasets.DATASETS:
+        data = custom_datasets.load(dataset, args.cache_dir)
+    else:
+        data = custom_datasets.load_dataset(dataset, split='train', cache_dir=args.cache_dir)[key]
+
+    # get unique examples, strip whitespace, and remove newlines
+    # then take just the long examples, shuffle, take the first 5,000 to tokenize to save time
+    # then take just the examples that are <= 512 tokens (for the base model)
+    # then generate n_samples samples
+
+    # remove duplicates from the data
+    data = list(dict.fromkeys(data))  # deterministic, as opposed to set()
+
+    # strip whitespace around each example
+    data = [x.strip() for x in data]
+
+    # remove newlines from each example
+    data = [_strip_newlines(x) for x in data]
+
+    # try to keep only examples with > 250 words
+    if dataset in ['writing', 'squad', 'xsum']:
+        long_data = [x for x in data if len(x.split()) > 250]
+        if len(long_data) > 0:
+            data = long_data
+
+    random.shuffle(data)
+    data = data[:5_000]
+
+    # keep only examples with <= 512 tokens according to base_tokenizer
+    # this step has the extra effect of removing examples with low-quality/garbage content
+    data_builder = DataBuilder(args)
+    tokenized_data = data_builder.base_tokenizer(data)
+    data = [x for x, y in zip(data, tokenized_data["input_ids"]) if len(y) <= 512]
+
+    # print stats about remaining data
+    print(f"Total number of samples: {len(data)}")
+    print(f"Average number of words: {np.mean([len(x.split()) for x in data])}")
+
+    return data_builder.generate_samples(data[:args.n_samples], batch_size=args.batch_size)
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_gpt3/data/xsum_gpt2")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--n_samples', type=int, default=200)
+    parser.add_argument('--openai_base', type=str, default=None)
+    parser.add_argument('--openai_key', type=str, default=None)
+    parser.add_argument('--openai_model', type=str, default=None)  # davinci, gpt-3.5-turbo, gpt-4
+    parser.add_argument('--base_model_name', type=str, default="gpt2")
+    parser.add_argument('--batch_size', type=int, default=50)
+    parser.add_argument('--do_top_k', action='store_true')
+    parser.add_argument('--top_k', type=int, default=40)
+    parser.add_argument('--do_top_p', action='store_true')
+    parser.add_argument('--top_p', type=float, default=0.96)
+    parser.add_argument('--do_temperature', action='store_true')
+    parser.add_argument('--temperature', type=float, default=0.8)
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    os.environ["XDG_CACHE_HOME"] = args.cache_dir
+    if not os.path.exists(args.cache_dir):
+        os.makedirs(args.cache_dir)
+    print(f"Using cache dir {args.cache_dir}")
+
+    random.seed(args.seed)
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+
+    print(f'Loading dataset {args.dataset}...')
+    dataset_keys = {'xsum': 'document', 'squad': 'context', 'writing': 'document'}
+    data = generate_data(args, args.dataset, dataset_keys[args.dataset] if args.dataset in dataset_keys else None)
+
+    save_data(args.output_file, args, data)

data_truncator.py

@@ -0,0 +1,97 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import time
+
+import numpy as np
+import datasets
+import torch
+import random
+import argparse
+import os
+import json
+import custom_datasets
+from model import load_tokenizer, load_model
+
+def stats_str(data):
+    if type(data) == dict:
+        mean_orig = np.mean([len(v.split()) for v in data['original']])
+        mean_samp = np.mean([len(v.split()) for v in data['sampled']])
+        return f'{mean_orig:.0f} words (original), {mean_samp:.0f} words (sampled).'
+    else:
+        mean_orig = np.mean([len(v['original'].split()) for v in data])
+        mean_samp = np.mean([len(v['sampled'].split()) for v in data])
+        mean_perturb_orig = np.mean([np.mean([len(p.split()) for p in v['perturbed_original']]) for v in data])
+        mean_perturb_samp = np.mean([np.mean([len(p.split()) for p in v['perturbed_sampled']]) for v in data])
+        return f'{mean_orig:.0f} words (original), {mean_samp:.0f} words (sampled), {mean_perturb_orig:.0f} words (perturb original), {mean_perturb_samp:.0f} words (perturb sampled).'
+
+def save_data(output_file, args, data):
+    # write args to file
+    args_file = f"{output_file}.args.json"
+    with open(args_file, "w") as fout:
+        json.dump(args, fout, indent=4)
+        print(f"Args written into {args_file}")
+
+    # write the data to a json file in the save folder
+    data_file = f"{output_file}.raw_data.json"
+    with open(data_file, "w") as fout:
+        json.dump(data, fout, indent=4)
+        print(f"Raw data written into {data_file}: {stats_str(data)}")
+
+
+def load_data(input_file):
+    # load args from file
+    args_file = f"{input_file}.args.json"
+    with open(args_file, "r") as fin:
+        args = json.load(fin)
+        print(f"Args loaded from {args_file}")
+
+    # load the data from file
+    data_file = f"{input_file}.raw_data.json"
+    with open(data_file, "r") as fin:
+        data = json.load(fin)
+        print(f"Raw data loaded from {data_file}: {stats_str(data)}")
+
+    return args, data
+
+def convert_data(input_file, output_file, max_words):
+    def _reduce(text):
+        lines = []
+        nwords = 0
+        for line in text.split('\n'):
+            if nwords >= max_words:
+                break
+            words = line.split()
+            words = words[:max_words - nwords]
+            lines.append(' '.join(words))
+            nwords += len(words)
+        return '\n'.join(lines)
+
+    args, data = load_data(input_file)
+    if type(data) == dict:
+        data['original'] = [_reduce(x) for x in data['original']]
+        data['sampled'] = [_reduce(x) for x in data['sampled']]
+    else:
+        for item in data:
+            item['original'] = _reduce(item['original'])
+            item['sampled'] = _reduce(item['sampled'])
+            item['perturbed_original'] = [_reduce(x) for x in item['perturbed_original']]
+            item['perturbed_sampled'] = [_reduce(x) for x in item['perturbed_sampled']]
+
+    save_data(output_file, args, data)
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--input_path', type=str, default="./exp_gpt3to4/data/")
+    parser.add_argument('--output_path', type=str, default="./exp_maxlen150/data/")
+    parser.add_argument('--max_words', type=int, default=150)
+    args = parser.parse_args()
+
+    import glob
+    import os.path as path
+
+    for file_name in glob.glob(f'{args.input_path}/*.raw_data.json'):
+        print(file_name)
+        file_name = path.basename(file_name).replace('.raw_data.json', '')
+        convert_data(path.join(args.input_path, file_name), path.join(args.output_path, file_name), args.max_words)

detect_gpt.py

@@ -0,0 +1,295 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import os.path
+
+import numpy as np
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+import re
+import torch
+import tqdm
+import argparse
+import json
+from data_builder import load_data, save_data
+from metrics import get_roc_metrics, get_precision_recall_metrics
+from model import load_tokenizer, load_model, get_model_fullname, from_pretrained
+
+# define regex to match all <extra_id_*> tokens, where * is an integer
+pattern = re.compile(r"<extra_id_\d+>")
+
+def load_mask_model(model_name, device, cache_dir):
+    model_name = get_model_fullname(model_name)
+    # mask filling t5 model
+    print(f'Loading mask filling model {model_name}...')
+    mask_model = from_pretrained(AutoModelForSeq2SeqLM, model_name, {}, cache_dir)
+    mask_model = mask_model.to(device)
+    return mask_model
+
+def load_mask_tokenizer(model_name, max_length, cache_dir):
+    model_name = get_model_fullname(model_name)
+    tokenizer = from_pretrained(AutoTokenizer, model_name, {'model_max_length': max_length}, cache_dir)
+    return tokenizer
+
+def tokenize_and_mask(text, span_length, pct, ceil_pct=False):
+    buffer_size = 1
+    tokens = text.split(' ')
+    mask_string = '<<<mask>>>'
+
+    n_spans = pct * len(tokens) / (span_length + buffer_size * 2)
+    if ceil_pct:
+        n_spans = np.ceil(n_spans)
+    n_spans = int(n_spans)
+
+    n_masks = 0
+    while n_masks < n_spans:
+        start = np.random.randint(0, len(tokens) - span_length)
+        end = start + span_length
+        search_start = max(0, start - buffer_size)
+        search_end = min(len(tokens), end + buffer_size)
+        if mask_string not in tokens[search_start:search_end]:
+            tokens[start:end] = [mask_string]
+            n_masks += 1
+
+    # replace each occurrence of mask_string with <extra_id_NUM>, where NUM increments
+    num_filled = 0
+    for idx, token in enumerate(tokens):
+        if token == mask_string:
+            tokens[idx] = f'<extra_id_{num_filled}>'
+            num_filled += 1
+    assert num_filled == n_masks, f"num_filled {num_filled} != n_masks {n_masks}"
+    text = ' '.join(tokens)
+    return text
+
+def count_masks(texts):
+    return [len([x for x in text.split() if x.startswith("<extra_id_")]) for text in texts]
+
+# replace each masked span with a sample from T5 mask_model
+def replace_masks(args, mask_model, mask_tokenizer, texts):
+    n_expected = count_masks(texts)
+    stop_id = mask_tokenizer.encode(f"<extra_id_{max(n_expected)}>")[0]
+    tokens = mask_tokenizer(texts, return_tensors="pt", padding=True).to(args.device)
+    outputs = mask_model.generate(**tokens, max_length=150, do_sample=True, top_p=args.mask_top_p,
+                                  num_return_sequences=1, eos_token_id=stop_id)
+    return mask_tokenizer.batch_decode(outputs, skip_special_tokens=False)
+
+def extract_fills(texts):
+    # remove <pad> from beginning of each text
+    texts = [x.replace("<pad>", "").replace("</s>", "").strip() for x in texts]
+
+    # return the text in between each matched mask token
+    extracted_fills = [pattern.split(x)[1:-1] for x in texts]
+
+    # remove whitespace around each fill
+    extracted_fills = [[y.strip() for y in x] for x in extracted_fills]
+
+    return extracted_fills
+
+def apply_extracted_fills(masked_texts, extracted_fills):
+    # split masked text into tokens, only splitting on spaces (not newlines)
+    tokens = [x.split(' ') for x in masked_texts]
+
+    n_expected = count_masks(masked_texts)
+
+    # replace each mask token with the corresponding fill
+    for idx, (text, fills, n) in enumerate(zip(tokens, extracted_fills, n_expected)):
+        if len(fills) < n:
+            tokens[idx] = []
+        else:
+            for fill_idx in range(n):
+                text[text.index(f"<extra_id_{fill_idx}>")] = fills[fill_idx]
+
+    # join tokens back into text
+    texts = [" ".join(x) for x in tokens]
+    return texts
+
+def perturb_texts_(args, mask_model, mask_tokenizer, texts, ceil_pct=False):
+    span_length = args.span_length
+    pct = args.pct_words_masked
+    masked_texts = [tokenize_and_mask(x, span_length, pct, ceil_pct) for x in texts]
+    raw_fills = replace_masks(args, mask_model, mask_tokenizer, masked_texts)
+    extracted_fills = extract_fills(raw_fills)
+    perturbed_texts = apply_extracted_fills(masked_texts, extracted_fills)
+
+    # Handle the fact that sometimes the model doesn't generate the right number of fills and we have to try again
+    attempts = 1
+    while '' in perturbed_texts:
+        idxs = [idx for idx, x in enumerate(perturbed_texts) if x == '']
+        print(f'WARNING: {len(idxs)} texts have no fills. Trying again [attempt {attempts}].')
+        masked_texts = [tokenize_and_mask(x, span_length, pct, ceil_pct) for idx, x in enumerate(texts) if idx in idxs]
+        raw_fills = replace_masks(args, mask_model, mask_tokenizer, masked_texts)
+        extracted_fills = extract_fills(raw_fills)
+        new_perturbed_texts = apply_extracted_fills(masked_texts, extracted_fills)
+        for idx, x in zip(idxs, new_perturbed_texts):
+            perturbed_texts[idx] = x
+        attempts += 1
+    return perturbed_texts
+
+def perturb_texts(args, mask_model, mask_tokenizer, texts, ceil_pct=False):
+    chunk_size = 10
+    outputs = []
+    for i in range(0, len(texts), chunk_size):
+        outputs.extend(perturb_texts_(args, mask_model, mask_tokenizer, texts[i:i + chunk_size], ceil_pct=ceil_pct))
+    return outputs
+
+# Get the log likelihood of each text under the base_model
+def get_ll(args, scoring_model, scoring_tokenizer, text):
+    with torch.no_grad():
+        tokenized = scoring_tokenizer(text, return_tensors="pt", return_token_type_ids=False).to(args.device)
+        labels = tokenized.input_ids
+        return -scoring_model(**tokenized, labels=labels).loss.item()
+
+def get_lls(args, scoring_model, scoring_tokenizer, texts):
+    return [get_ll(args, scoring_model, scoring_tokenizer, text) for text in texts]
+
+
+def generate_perturbs(args):
+    n_perturbations = args.n_perturbations
+    name = f'perturbation_{n_perturbations}'
+    # load model
+    mask_model = load_mask_model(args.mask_filling_model_name, args.device, args.cache_dir)
+    mask_model.eval()
+    try:
+        n_positions = mask_model.config.n_positions
+    except AttributeError:
+        n_positions = 512
+    mask_tokenizer = load_mask_tokenizer(args.mask_filling_model_name, n_positions, args.cache_dir)
+
+    # load data
+    data = load_data(args.dataset_file)
+    n_samples = len(data["sampled"])
+
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+
+    # generate perturb samples
+    perturbs = []
+    for idx in tqdm.tqdm(range(n_samples), desc=f"Perturb text"):
+        original_text = data["original"][idx]
+        sampled_text = data["sampled"][idx]
+        # perturb
+        p_sampled_text = perturb_texts(args, mask_model, mask_tokenizer, [sampled_text for _ in range(n_perturbations)])
+        p_original_text = perturb_texts(args, mask_model, mask_tokenizer, [original_text for _ in range(n_perturbations)])
+        assert len(p_sampled_text) == n_perturbations, f"Expected {n_perturbations} perturbed samples, got {len(p_sampled_text)}"
+        assert len(p_original_text) == n_perturbations, f"Expected {n_perturbations} perturbed samples, got {len(p_original_text)}"
+        # result
+        perturbs.append({
+            "original": original_text,
+            "sampled": sampled_text,
+            "perturbed_sampled": p_sampled_text,
+            "perturbed_original": p_original_text
+        })
+
+    save_data(f'{args.dataset_file}.{args.mask_filling_model_name}.{name}', args, perturbs)
+
+
+def experiment(args):
+    n_perturbations = args.n_perturbations
+    name = f'perturbation_{n_perturbations}'
+    perturb_file = f'{args.dataset_file}.{args.mask_filling_model_name}.{name}.raw_data.json'
+    if os.path.exists(perturb_file):
+        print(f'Use existing perturbation file: {perturb_file}')
+    else:
+        generate_perturbs(args)
+    # load model
+    scoring_tokenizer = load_tokenizer(args.scoring_model_name, args.dataset, args.cache_dir)
+    scoring_model = load_model(args.scoring_model_name, 'cpu', args.cache_dir)
+    scoring_model.eval()
+    scoring_model.to(args.device)
+    # load data
+    data = load_data(f'{args.dataset_file}.{args.mask_filling_model_name}.{name}')
+    n_samples = len(data)
+
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+
+    # Evaluate
+    results = data
+    for idx in tqdm.tqdm(range(n_samples), desc=f"Computing {name} criterion"):
+        original_text = results[idx]["original"]
+        sampled_text = results[idx]["sampled"]
+        perturbed_original = results[idx]["perturbed_original"]
+        perturbed_sampled = results[idx]["perturbed_sampled"]
+        # original text
+        original_ll = get_ll(args, scoring_model, scoring_tokenizer, original_text)
+        p_original_ll = get_lls(args, scoring_model, scoring_tokenizer, perturbed_original)
+        # sampled text
+        sampled_ll = get_ll(args, scoring_model, scoring_tokenizer, sampled_text)
+        p_sampled_ll = get_lls(args, scoring_model, scoring_tokenizer, perturbed_sampled)
+        # result
+        results[idx]["original_ll"] = original_ll
+        results[idx]["sampled_ll"] = sampled_ll
+        results[idx]["all_perturbed_sampled_ll"] = p_sampled_ll
+        results[idx]["all_perturbed_original_ll"] = p_original_ll
+        results[idx]["perturbed_sampled_ll"] = np.mean(p_sampled_ll)
+        results[idx]["perturbed_original_ll"] = np.mean(p_original_ll)
+        results[idx]["perturbed_sampled_ll_std"] = np.std(p_sampled_ll) if len(p_sampled_ll) > 1 else 1
+        results[idx]["perturbed_original_ll_std"] = np.std(p_original_ll) if len(p_original_ll) > 1 else 1
+
+    # compute diffs with perturbed
+    predictions = {'real': [], 'samples': []}
+    for res in results:
+        if res['perturbed_original_ll_std'] == 0:
+            res['perturbed_original_ll_std'] = 1
+            print("WARNING: std of perturbed original is 0, setting to 1")
+            print(f"Number of unique perturbed original texts: {len(set(res['perturbed_original']))}")
+            print(f"Original text: {res['original']}")
+        if res['perturbed_sampled_ll_std'] == 0:
+            res['perturbed_sampled_ll_std'] = 1
+            print("WARNING: std of perturbed sampled is 0, setting to 1")
+            print(f"Number of unique perturbed sampled texts: {len(set(res['perturbed_sampled']))}")
+            print(f"Sampled text: {res['sampled']}")
+        predictions['real'].append((res['original_ll'] - res['perturbed_original_ll']) / res['perturbed_original_ll_std'])
+        predictions['samples'].append((res['sampled_ll'] - res['perturbed_sampled_ll']) / res['perturbed_sampled_ll_std'])
+
+    print(f"Real mean/std: {np.mean(predictions['real']):.2f}/{np.std(predictions['real']):.2f}, Samples mean/std: {np.mean(predictions['samples']):.2f}/{np.std(predictions['samples']):.2f}")
+    fpr, tpr, roc_auc = get_roc_metrics(predictions['real'], predictions['samples'])
+    p, r, pr_auc = get_precision_recall_metrics(predictions['real'], predictions['samples'])
+    print(f"Criterion {name}_threshold ROC AUC: {roc_auc:.4f}, PR AUC: {pr_auc:.4f}")
+
+    # results
+    results_file = f'{args.output_file}.{name}.json'
+    results = {
+        'name': name,
+        'info': {
+            'pct_words_masked': args.pct_words_masked,
+            'span_length': args.span_length,
+            'n_perturbations': args.n_perturbations,
+            'n_samples': n_samples,
+        },
+        'predictions': predictions,
+        'raw_results': results,
+        'metrics': {
+            'roc_auc': roc_auc,
+            'fpr': fpr,
+            'tpr': tpr,
+        },
+        'pr_metrics': {
+            'pr_auc': pr_auc,
+            'precision': p,
+            'recall': r,
+        },
+        'loss': 1 - pr_auc,
+    }
+    with open(results_file, 'w') as fout:
+        json.dump(results, fout)
+        print(f'Results written into {results_file}')
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_test/results/xsum_gpt2")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--dataset_file', type=str, default="./exp_test/data/xsum_gpt2")
+    parser.add_argument('--pct_words_masked', type=float, default=0.3) # pct masked is actually pct_words_masked * (span_length / (span_length + 2 * buffer_size))
+    parser.add_argument('--mask_top_p', type=float, default=1.0)
+    parser.add_argument('--span_length', type=int, default=2)
+    parser.add_argument('--n_perturbations', type=int, default=10)
+    parser.add_argument('--scoring_model_name', type=str, default="gpt2")
+    parser.add_argument('--mask_filling_model_name', type=str, default="t5-small")
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    experiment(args)

detect_llm.py

@@ -0,0 +1,128 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+import tqdm
+import argparse
+import json
+from model import load_tokenizer, load_model
+from metrics import get_roc_metrics, get_precision_recall_metrics
+from data_builder import load_data
+
+def get_likelihood(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+
+    logits = logits.view(-1, logits.shape[-1])
+    labels = labels.view(-1)
+    log_probs = torch.nn.functional.log_softmax(logits, dim=-1)
+    log_likelihood = log_probs.gather(dim=-1, index=labels.unsqueeze(-1)).squeeze(-1)
+    return log_likelihood.mean().item()
+
+def get_logrank(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+
+    # get rank of each label token in the model's likelihood ordering
+    matches = (logits.argsort(-1, descending=True) == labels.unsqueeze(-1)).nonzero()
+    assert matches.shape[1] == 3, f"Expected 3 dimensions in matches tensor, got {matches.shape}"
+
+    ranks, timesteps = matches[:, -1], matches[:, -2]
+
+    # make sure we got exactly one match for each timestep in the sequence
+    assert (timesteps == torch.arange(len(timesteps)).to(timesteps.device)).all(), "Expected one match per timestep"
+
+    ranks = ranks.float() + 1  # convert to 1-indexed rank
+    ranks = torch.log(ranks)
+    return ranks.mean().item()
+
+# Log-Likelihood Log-Rank Ratio
+def get_lrr(args, scoring_model, scoring_tokenizer, text, perturbs):
+    with torch.no_grad():
+        tokenized = scoring_tokenizer(text, return_tensors="pt", return_token_type_ids=False).to(args.device)
+        labels = tokenized.input_ids[:, 1:]
+        logits = scoring_model(**tokenized).logits[:, :-1]
+        likelihood = get_likelihood(logits, labels)
+        logrank = get_logrank(logits, labels)
+        return - likelihood / logrank
+
+# Normalized Log-Rank Perturbation
+def get_npr(args, scoring_model, scoring_tokenizer, text, perturbs):
+    with torch.no_grad():
+        tokenized = scoring_tokenizer(text, return_tensors="pt", return_token_type_ids=False).to(args.device)
+        labels = tokenized.input_ids[:, 1:]
+        logits = scoring_model(**tokenized).logits[:, :-1]
+        logrank = get_logrank(logits, labels)
+        # perturbations
+        logranks = []
+        for perturb in perturbs:
+            tokenized = scoring_tokenizer(perturb, return_tensors="pt", return_token_type_ids=False).to(args.device)
+            labels = tokenized.input_ids[:, 1:]
+            logits = scoring_model(**tokenized).logits[:, :-1]
+            logranks.append(get_logrank(logits, labels))
+        # npr
+        return np.mean(logranks) / logrank
+
+def experiment(args):
+    # load model
+    scoring_tokenizer = load_tokenizer(args.scoring_model_name, args.dataset, args.cache_dir)
+    scoring_model = load_model(args.scoring_model_name, args.device, args.cache_dir)
+    scoring_model.eval()
+    # load data
+    data = load_data(args.dataset_file)
+    n_samples = len(data)
+    # eval criterions
+    criterion_fns = {'lrr': get_lrr, 'npr': get_npr}
+    for name in criterion_fns:
+        criterion_fn = criterion_fns[name]
+        torch.manual_seed(args.seed)
+        np.random.seed(args.seed)
+        eval_results = []
+        for idx in tqdm.tqdm(range(n_samples), desc=f"Computing {name} criterion"):
+            original_text = data[idx]["original"]
+            sampled_text = data[idx]["sampled"]
+            perturbed_original = data[idx]["perturbed_original"]
+            perturbed_sampled = data[idx]["perturbed_sampled"]
+            original_crit = criterion_fn(args, scoring_model, scoring_tokenizer, original_text, perturbed_original)
+            sampled_crit = criterion_fn(args, scoring_model, scoring_tokenizer, sampled_text, perturbed_sampled)
+            # result
+            eval_results.append({"original": original_text,
+                            "original_crit": original_crit,
+                            "sampled": sampled_text,
+                            "sampled_crit": sampled_crit})
+
+        # compute prediction scores for real/sampled passages
+        predictions = {'real': [x["original_crit"] for x in eval_results],
+                       'samples': [x["sampled_crit"] for x in eval_results]}
+        fpr, tpr, roc_auc = get_roc_metrics(predictions['real'], predictions['samples'])
+        p, r, pr_auc = get_precision_recall_metrics(predictions['real'], predictions['samples'])
+        print(f"Criterion {name}_threshold ROC AUC: {roc_auc:.4f}, PR AUC: {pr_auc:.4f}")
+        # log results
+        results_file = f'{args.output_file}.{name}.json'
+        results = { 'name': f'{name}_threshold',
+                    'info': {'n_samples': n_samples},
+                    'predictions': predictions,
+                    'raw_results': eval_results,
+                    'metrics': {'roc_auc': roc_auc, 'fpr': fpr, 'tpr': tpr},
+                    'pr_metrics': {'pr_auc': pr_auc, 'precision': p, 'recall': r},
+                    'loss': 1 - pr_auc}
+        with open(results_file, 'w') as fout:
+            json.dump(results, fout)
+            print(f'Results written into {results_file}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_test/results/xsum_gpt2")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--dataset_file', type=str, default="./exp_test/results/xsum_gpt2.perturbation_10")
+    parser.add_argument('--scoring_model_name', type=str, default="gpt2")
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    experiment(args)

detector.py

@@ -0,0 +1,11 @@
+class Detector:
+    def __init__(self):
+        # Model veya gerekli dosyaları yüklemek için burada yapılandırma yapabilirsiniz
+        print("Fast-DetectGPT initialized!")
+
+    def detect(self, text):
+        """
+        Verilen metni analiz eder ve sonuç döndürür.
+        """
+        # Gerçek analiz işlemi yerine örnek sonuç döndürülüyor
+        return [(text, 0.85)]  # 0.85 AI tarafından üretilmiş olasılığıdır

dna_gpt.py

@@ -0,0 +1,211 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import os.path
+
+import numpy as np
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+import re
+import torch
+import tqdm
+import argparse
+import json
+from data_builder import load_data, save_data
+from metrics import get_roc_metrics, get_precision_recall_metrics
+from model import load_tokenizer, load_model, get_model_fullname, from_pretrained
+from data_builder import load_data
+from model import load_tokenizer, load_model
+from metrics import get_roc_metrics, get_precision_recall_metrics
+import custom_datasets
+
+class PrefixSampler:
+    def __init__(self, args):
+        self.args = args
+        self.base_tokenizer = load_tokenizer(args.base_model_name, args.dataset, args.cache_dir)
+        self.base_model = load_model(args.base_model_name, args.device, args.cache_dir)
+
+    def _sample_from_model(self, texts, min_words=55, truncate_ratio=0.5):
+        # encode each text as a list of token ids
+        if self.args.dataset == 'pubmed':
+            pubmed_sep = ' Answer:'
+            texts = [t[:t.index(pubmed_sep) + len(pubmed_sep)] for t in texts]
+            all_encoded = self.base_tokenizer(texts, return_tensors="pt", padding=True).to(self.args.device)
+        else:
+            texts = [t.split(' ') for t in texts]
+            texts = [' '.join(t[: int(len(t) * truncate_ratio)]) for t in texts]
+            all_encoded = self.base_tokenizer(texts, return_tensors="pt", padding=True).to(self.args.device)
+
+        self.base_model.eval()
+        decoded = ['' for _ in range(len(texts))]
+
+        # sample from the model until we get a sample with at least min_words words for each example
+        # this is an inefficient way to do this (since we regenerate for all inputs if just one is too short), but it works
+        tries = 0
+        m = 0
+        while m < min_words:
+            if tries != 0:
+                print()
+                print(f"min words: {m}, needed {min_words}, regenerating (try {tries})")
+
+            sampling_kwargs = {'temperature': self.args.temperature}
+            if self.args.do_top_p:
+                sampling_kwargs['top_p'] = self.args.top_p
+            elif self.args.do_top_k:
+                sampling_kwargs['top_k'] = self.args.top_k
+            min_length = 50 if self.args.dataset in ['pubmed'] else 150
+            outputs = self.base_model.generate(**all_encoded, min_length=min_length, max_length=200, do_sample=True,
+                                               **sampling_kwargs, pad_token_id=self.base_tokenizer.eos_token_id,
+                                               eos_token_id=self.base_tokenizer.eos_token_id)
+            decoded = self.base_tokenizer.batch_decode(outputs, skip_special_tokens=True)
+            m = min(len(x.split()) for x in decoded)
+            tries += 1
+
+        return decoded
+
+    def generate_samples(self, raw_data, batch_size):
+        # trim to shorter length
+        def _trim_to_shorter_length(texta, textb):
+            # truncate to shorter of o and s
+            shorter_length = min(len(texta.split(' ')), len(textb.split(' ')))
+            texta = ' '.join(texta.split(' ')[:shorter_length])
+            textb = ' '.join(textb.split(' ')[:shorter_length])
+            return texta, textb
+
+        def _truncate_to_substring(text, substring, idx_occurrence):
+            # truncate everything after the idx_occurrence occurrence of substring
+            assert idx_occurrence > 0, 'idx_occurrence must be > 0'
+            idx = -1
+            for _ in range(idx_occurrence):
+                idx = text.find(substring, idx + 1)
+                if idx == -1:
+                    return text
+            return text[:idx]
+
+        data = {
+            "original": [],
+            "sampled": [],
+        }
+
+        assert len(raw_data) % batch_size == 0
+        for batch in range(len(raw_data) // batch_size):
+            print('Generating samples for batch', batch, 'of', len(raw_data) // batch_size)
+            original_text = raw_data[batch * batch_size:(batch + 1) * batch_size]
+            sampled_text = self._sample_from_model(original_text, min_words=30 if self.args.dataset in ['pubmed'] else 55, truncate_ratio=self.args.truncate_ratio)
+
+            for o, s in zip(original_text, sampled_text):
+                if self.args.dataset == 'pubmed':
+                    s = _truncate_to_substring(s, 'Question:', 2)
+                    o = o.replace(custom_datasets.SEPARATOR, ' ')
+
+                o, s = _trim_to_shorter_length(o, s)
+
+                # add to the data
+                data["original"].append(o)
+                data["sampled"].append(s)
+
+        return data
+
+def get_likelihood(logits, labels, pad_index):
+    labels = labels.unsqueeze(-1) if labels.ndim == logits.ndim - 1 else labels
+    lprobs = torch.log_softmax(logits, dim=-1)
+    log_likelihood = lprobs.gather(dim=-1, index=labels)
+    mask = labels != pad_index
+    log_likelihood = (log_likelihood * mask).sum(dim=1) / mask.sum(dim=1)
+    return log_likelihood.squeeze(-1)
+
+def get_log_prob(sampler, text):
+    tokenized = sampler.base_tokenizer(text, return_tensors="pt", padding=True).to(sampler.args.device)
+    labels = tokenized.input_ids[:, 1:]
+    with torch.no_grad():
+        logits_score = sampler.base_model(**tokenized).logits[:, :-1]
+        return get_likelihood(logits_score, labels, sampler.base_tokenizer.pad_token_id)
+
+def get_log_probs(sampler, texts):
+    batch_size = sampler.args.batch_size
+    batch_lprobs = []
+    for batch in range(len(texts) // batch_size):
+        tokenized = sampler.base_tokenizer(texts[batch * batch_size:(batch + 1) * batch_size], return_tensors="pt", padding=True).to(sampler.args.device)
+        labels = tokenized.input_ids[:, 1:]
+        with torch.no_grad():
+            logits_score = sampler.base_model(**tokenized).logits[:, :-1]
+            lprobs = get_likelihood(logits_score, labels, sampler.base_tokenizer.pad_token_id)
+            batch_lprobs.append(lprobs)
+    return torch.cat(batch_lprobs, dim=0)
+
+def get_regen_samples(sampler, text):
+    data = [text] * sampler.args.regen_number
+    data = sampler.generate_samples(data, batch_size=sampler.args.batch_size)
+    return data['sampled']
+
+def get_dna_gpt(sampler, text):
+    lprob = get_log_prob(sampler, text)
+    regens = get_regen_samples(sampler, text)
+    lprob_regens = get_log_probs(sampler, regens)
+    wscore = lprob[0] - lprob_regens.mean()
+    return wscore.item()
+
+def experiment(args):
+    sampler = PrefixSampler(args)
+    # load data
+    data = load_data(args.dataset_file)
+    n_samples = len(data["sampled"])
+    # evaluate criterion
+    name = "dna_gpt"
+    criterion_fn = get_dna_gpt
+
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+    results = []
+    for idx in tqdm.tqdm(range(n_samples), desc=f"Computing {name} criterion"):
+        original_text = data["original"][idx]
+        sampled_text = data["sampled"][idx]
+        # original text
+        original_crit = criterion_fn(sampler, original_text)
+        # sampled text
+        sampled_crit = criterion_fn(sampler, sampled_text)
+        # result
+        results.append({"original": original_text,
+                        "original_crit": original_crit,
+                        "sampled": sampled_text,
+                        "sampled_crit": sampled_crit})
+
+    # compute prediction scores for real/sampled passages
+    predictions = {'real': [x["original_crit"] for x in results],
+                   'samples': [x["sampled_crit"] for x in results]}
+    fpr, tpr, roc_auc = get_roc_metrics(predictions['real'], predictions['samples'])
+    p, r, pr_auc = get_precision_recall_metrics(predictions['real'], predictions['samples'])
+    print(f"Criterion {name}_threshold ROC AUC: {roc_auc:.4f}, PR AUC: {pr_auc:.4f}")
+    # results
+    results_file = f'{args.output_file}.{name}.json'
+    results = { 'name': f'{name}_threshold',
+                'info': {'n_samples': n_samples},
+                'predictions': predictions,
+                'raw_results': results,
+                'metrics': {'roc_auc': roc_auc, 'fpr': fpr, 'tpr': tpr},
+                'pr_metrics': {'pr_auc': pr_auc, 'precision': p, 'recall': r},
+                'loss': 1 - pr_auc}
+    with open(results_file, 'w') as fout:
+        json.dump(results, fout)
+        print(f'Results written into {results_file}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_test/results/pubmed_davinci")
+    parser.add_argument('--dataset', type=str, default="pubmed")
+    parser.add_argument('--dataset_file', type=str, default="./exp_test/data/pubmed_davinci")
+    parser.add_argument('--truncate_ratio', type=float, default=0.5)
+    parser.add_argument('--regen_number', type=int, default=10)
+    parser.add_argument('--base_model_name', type=str, default="gpt2")
+    parser.add_argument('--batch_size', type=int, default=10)
+    parser.add_argument('--do_top_k', action='store_true')
+    parser.add_argument('--top_k', type=int, default=40)
+    parser.add_argument('--do_top_p', action='store_true')
+    parser.add_argument('--top_p', type=float, default=0.96)
+    parser.add_argument('--temperature', type=float, default=1.0)
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    experiment(args)

fast_detect_gpt.py

@@ -0,0 +1,162 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import random
+
+import numpy as np
+import torch
+import torch.nn.functional as F
+import tqdm
+import argparse
+import json
+from data_builder import load_data
+from model import load_tokenizer, load_model
+from metrics import get_roc_metrics, get_precision_recall_metrics
+
+def get_samples(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+    nsamples = 10000
+    lprobs = torch.log_softmax(logits, dim=-1)
+    distrib = torch.distributions.categorical.Categorical(logits=lprobs)
+    samples = distrib.sample([nsamples]).permute([1, 2, 0])
+    return samples
+
+def get_likelihood(logits, labels):
+    assert logits.shape[0] == 1
+    assert labels.shape[0] == 1
+    labels = labels.unsqueeze(-1) if labels.ndim == logits.ndim - 1 else labels
+    lprobs = torch.log_softmax(logits, dim=-1)
+    log_likelihood = lprobs.gather(dim=-1, index=labels)
+    return log_likelihood.mean(dim=1)
+
+def get_sampling_discrepancy(logits_ref, logits_score, labels):
+    assert logits_ref.shape[0] == 1
+    assert logits_score.shape[0] == 1
+    assert labels.shape[0] == 1
+    if logits_ref.size(-1) != logits_score.size(-1):
+        # print(f"WARNING: vocabulary size mismatch {logits_ref.size(-1)} vs {logits_score.size(-1)}.")
+        vocab_size = min(logits_ref.size(-1), logits_score.size(-1))
+        logits_ref = logits_ref[:, :, :vocab_size]
+        logits_score = logits_score[:, :, :vocab_size]
+
+    samples = get_samples(logits_ref, labels)
+    log_likelihood_x = get_likelihood(logits_score, labels)
+    log_likelihood_x_tilde = get_likelihood(logits_score, samples)
+    miu_tilde = log_likelihood_x_tilde.mean(dim=-1)
+    sigma_tilde = log_likelihood_x_tilde.std(dim=-1)
+    discrepancy = (log_likelihood_x.squeeze(-1) - miu_tilde) / sigma_tilde
+    return discrepancy.item()
+
+def get_sampling_discrepancy_analytic(logits_ref, logits_score, labels):
+    assert logits_ref.shape[0] == 1
+    assert logits_score.shape[0] == 1
+    assert labels.shape[0] == 1
+    if logits_ref.size(-1) != logits_score.size(-1):
+        # print(f"WARNING: vocabulary size mismatch {logits_ref.size(-1)} vs {logits_score.size(-1)}.")
+        vocab_size = min(logits_ref.size(-1), logits_score.size(-1))
+        logits_ref = logits_ref[:, :, :vocab_size]
+        logits_score = logits_score[:, :, :vocab_size]
+
+    labels = labels.unsqueeze(-1) if labels.ndim == logits_score.ndim - 1 else labels
+    lprobs_score = torch.log_softmax(logits_score, dim=-1)
+    probs_ref = torch.softmax(logits_ref, dim=-1)
+    log_likelihood = lprobs_score.gather(dim=-1, index=labels).squeeze(-1)
+    mean_ref = (probs_ref * lprobs_score).sum(dim=-1)
+    var_ref = (probs_ref * torch.square(lprobs_score)).sum(dim=-1) - torch.square(mean_ref)
+    discrepancy = (log_likelihood.sum(dim=-1) - mean_ref.sum(dim=-1)) / var_ref.sum(dim=-1).sqrt()
+    discrepancy = discrepancy.mean()
+    return discrepancy.item()
+
+def experiment(args):
+    # load model
+    scoring_tokenizer = load_tokenizer(args.scoring_model_name, args.dataset, args.cache_dir)
+    scoring_model = load_model(args.scoring_model_name, args.device, args.cache_dir)
+    scoring_model.eval()
+    if args.reference_model_name != args.scoring_model_name:
+        reference_tokenizer = load_tokenizer(args.reference_model_name, args.dataset, args.cache_dir)
+        reference_model = load_model(args.reference_model_name, args.device, args.cache_dir)
+        reference_model.eval()
+    # load data
+    data = load_data(args.dataset_file)
+    n_samples = len(data["sampled"])
+    # evaluate criterion
+    if args.discrepancy_analytic:
+        name = "sampling_discrepancy_analytic"
+        criterion_fn = get_sampling_discrepancy_analytic
+    else:
+        name = "sampling_discrepancy"
+        criterion_fn = get_sampling_discrepancy
+
+    random.seed(args.seed)
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+    results = []
+    for idx in tqdm.tqdm(range(n_samples), desc=f"Computing {name} criterion"):
+        original_text = data["original"][idx]
+        sampled_text = data["sampled"][idx]
+        # original text
+        tokenized = scoring_tokenizer(original_text, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+        labels = tokenized.input_ids[:, 1:]
+        with torch.no_grad():
+            logits_score = scoring_model(**tokenized).logits[:, :-1]
+            if args.reference_model_name == args.scoring_model_name:
+                logits_ref = logits_score
+            else:
+                tokenized = reference_tokenizer(original_text, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+                assert torch.all(tokenized.input_ids[:, 1:] == labels), "Tokenizer is mismatch."
+                logits_ref = reference_model(**tokenized).logits[:, :-1]
+            original_crit = criterion_fn(logits_ref, logits_score, labels)
+        # sampled text
+        tokenized = scoring_tokenizer(sampled_text, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+        labels = tokenized.input_ids[:, 1:]
+        with torch.no_grad():
+            logits_score = scoring_model(**tokenized).logits[:, :-1]
+            if args.reference_model_name == args.scoring_model_name:
+                logits_ref = logits_score
+            else:
+                tokenized = reference_tokenizer(sampled_text, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+                assert torch.all(tokenized.input_ids[:, 1:] == labels), "Tokenizer is mismatch."
+                logits_ref = reference_model(**tokenized).logits[:, :-1]
+            sampled_crit = criterion_fn(logits_ref, logits_score, labels)
+        # result
+        results.append({"original": original_text,
+                        "original_crit": original_crit,
+                        "sampled": sampled_text,
+                        "sampled_crit": sampled_crit})
+
+    # compute prediction scores for real/sampled passages
+    predictions = {'real': [x["original_crit"] for x in results],
+                   'samples': [x["sampled_crit"] for x in results]}
+    print(f"Real mean/std: {np.mean(predictions['real']):.2f}/{np.std(predictions['real']):.2f}, Samples mean/std: {np.mean(predictions['samples']):.2f}/{np.std(predictions['samples']):.2f}")
+    fpr, tpr, roc_auc = get_roc_metrics(predictions['real'], predictions['samples'])
+    p, r, pr_auc = get_precision_recall_metrics(predictions['real'], predictions['samples'])
+    print(f"Criterion {name}_threshold ROC AUC: {roc_auc:.4f}, PR AUC: {pr_auc:.4f}")
+    # results
+    results_file = f'{args.output_file}.{name}.json'
+    results = { 'name': f'{name}_threshold',
+                'info': {'n_samples': n_samples},
+                'predictions': predictions,
+                'raw_results': results,
+                'metrics': {'roc_auc': roc_auc, 'fpr': fpr, 'tpr': tpr},
+                'pr_metrics': {'pr_auc': pr_auc, 'precision': p, 'recall': r},
+                'loss': 1 - pr_auc}
+    with open(results_file, 'w') as fout:
+        json.dump(results, fout)
+        print(f'Results written into {results_file}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_test/results/xsum_gpt2")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--dataset_file', type=str, default="./exp_test/data/xsum_gpt2")
+    parser.add_argument('--reference_model_name', type=str, default="gpt2")
+    parser.add_argument('--scoring_model_name', type=str, default="gpt2")
+    parser.add_argument('--discrepancy_analytic', action='store_true')
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    experiment(args)

gpt3to4.sh

@@ -0,0 +1,116 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+exp_path=exp_gpt3to4
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+datasets="xsum writing pubmed"
+source_models="davinci gpt-3.5-turbo gpt-4"
+
+# preparing dataset
+openai_base="https://api.openai.com/v1"
+openai_key="xxxxxxxx"  # replace with your own key for generating your own test set
+
+# We follow DetectGPT settings for generating text from GPT-3
+M=davinci
+for D in $datasets; do
+  echo `date`, Preparing dataset ${D} by sampling from openai/${M} ...
+  python scripts/data_builder.py --openai_model $M --openai_key $openai_key --openai_base $openai_base \
+              --dataset $D --n_samples 150 --do_top_p --top_p 0.9 --batch_size 1 \
+              --output_file $data_path/${D}_${M}
+done
+
+# We use a temperature of 0.8 for creativity writing
+for M in gpt-3.5-turbo gpt-4; do
+  for D in $datasets; do
+    echo `date`, Preparing dataset ${D} by sampling from openai/${M} ...
+    python scripts/data_builder.py --openai_model $M --openai_key $openai_key --openai_base $openai_base \
+                --dataset $D --n_samples 150 --do_temperature --temperature 0.8 --batch_size 1 \
+                --output_file $data_path/${D}_${M}
+  done
+done
+
+# evaluate Fast-DetectGPT in the black-box setting
+settings="gpt-j-6B:gpt2-xl gpt-j-6B:gpt-neo-2.7B gpt-j-6B:gpt-j-6B"
+for M in $source_models; do
+  for D in $datasets; do
+    for S in $settings; do
+      IFS=':' read -r -a S <<< $S && M1=${S[0]} && M2=${S[1]}
+      echo `date`, Evaluating Fast-DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/fast_detect_gpt.py --reference_model_name $M1 --scoring_model_name $M2 --discrepancy_analytic \
+                          --dataset $D --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate supervised detectors
+supervised_models="roberta-base-openai-detector roberta-large-openai-detector"
+for M in $source_models; do
+  for D in $datasets; do
+    for SM in $supervised_models; do
+      echo `date`, Evaluating ${SM} on ${D}_${M} ...
+      python scripts/supervised.py --model_name $SM --dataset $D \
+                            --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+    done
+  done
+done
+
+# evaluate baselines
+scoring_models="gpt-neo-2.7B"
+for M in $source_models; do
+  for D in $datasets; do
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating baseline methods on ${D}_${M}.${M2} ...
+      python scripts/baselines.py --scoring_model_name ${M2} --dataset $D \
+                            --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M2}
+    done
+  done
+done
+
+# evaluate DNA-GPT
+scoring_models="gpt-neo-2.7B"
+for M in $source_models; do
+  for D in $datasets; do
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DNA-GPT on ${D}_${M}.${M2} ...
+      python scripts/dna_gpt.py --base_model_name ${M2} --dataset $D \
+                            --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M2}
+    done
+  done
+done
+
+# evaluate DetectGPT and DetectLLM
+scoring_models="gpt2-xl gpt-neo-2.7B gpt-j-6B"
+for M in $source_models; do
+  for D in $datasets; do
+    M1=t5-11b  # perturbation model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_gpt.py --mask_filling_model_name ${M1} --scoring_model_name ${M2} --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+      # we leverage DetectGPT to generate the perturbations
+      echo `date`, Evaluating DetectLLM methods on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_llm.py --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.${M1}.perturbation_100 --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate GPTZero
+for M in $source_models; do
+  for D in $datasets; do
+    echo `date`, Evaluating GPTZero on ${D}_${M} ...
+    python scripts/gptzero.py --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done

gptzero.py

@@ -0,0 +1,84 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import time
+
+import numpy as np
+import tqdm
+import argparse
+import json
+from metrics import get_roc_metrics, get_precision_recall_metrics
+from data_builder import load_data
+
+def detect_gptzero(args, text):
+    import requests
+    url = "https://api.gptzero.me/v2/predict/text"
+    payload = {
+        "document": text,
+        "version": "2023-09-14"
+    }
+    headers = {
+        "Accept": "application/json",
+        "content-type": "application/json",
+        "x-api-key": ""
+    }
+
+    while True:
+        try:
+            time.sleep(600)  # 1 request per 10 minutes for free access
+            response = requests.post(url, json=payload, headers=headers)
+            return response.json()['documents'][0]['completely_generated_prob']
+        except Exception as ex:
+            print(ex)
+
+def experiment(args):
+    # load data
+    data = load_data(args.dataset_file)
+    n_samples = len(data["sampled"])
+    # evaluate criterion
+    name = "gptzero"
+    criterion_fn = detect_gptzero
+
+    results = []
+    for idx in tqdm.tqdm(range(n_samples), desc=f"Computing {name} criterion"):
+        original_text = data["original"][idx]
+        sampled_text = data["sampled"][idx]
+        original_crit = criterion_fn(args, original_text)
+        sampled_crit = criterion_fn(args, sampled_text)
+        # result
+        results.append({"original": original_text,
+                        "original_crit": original_crit,
+                        "sampled": sampled_text,
+                        "sampled_crit": sampled_crit})
+
+        # compute prediction scores for real/sampled passages
+        predictions = {'real': [x["original_crit"] for x in results],
+                       'samples': [x["sampled_crit"] for x in results]}
+        print(f"Real mean/std: {np.mean(predictions['real']):.2f}/{np.std(predictions['real']):.2f}, Samples mean/std: {np.mean(predictions['samples']):.2f}/{np.std(predictions['samples']):.2f}")
+        fpr, tpr, roc_auc = get_roc_metrics(predictions['real'], predictions['samples'])
+        p, r, pr_auc = get_precision_recall_metrics(predictions['real'], predictions['samples'])
+        print(f"Criterion {name}_threshold ROC AUC: {roc_auc:.4f}, PR AUC: {pr_auc:.4f}")
+
+    # results
+    results_file = f'{args.output_file}.{name}.json'
+    results = { 'name': f'{name}_threshold',
+                'info': {'n_samples': n_samples},
+                'predictions': predictions,
+                'raw_results': results,
+                'metrics': {'roc_auc': roc_auc, 'fpr': fpr, 'tpr': tpr},
+                'pr_metrics': {'pr_auc': pr_auc, 'precision': p, 'recall': r},
+                'loss': 1 - pr_auc}
+    with open(results_file, 'w') as fout:
+        json.dump(results, fout)
+        print(f'Results written into {results_file}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_gpt3to4/results/xsum_gpt-4")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--dataset_file', type=str, default="./exp_gpt3to4/data/xsum_gpt-4")
+    args = parser.parse_args()
+
+    experiment(args)
+

index.html

@@ -0,0 +1,106 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Fast-DetectGPT</title>
+    <style>
+        body {
+            font-family: Arial, sans-serif;
+            margin: 20px;
+            background-color: #f9f9f9;
+        }
+        .container {
+            max-width: 700px;
+            margin: auto;
+            background: #ffffff;
+            border-radius: 8px;
+            padding: 20px;
+            box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2);
+        }
+        h1 {
+            text-align: center;
+            color: #333;
+        }
+        textarea {
+            width: 100%;
+            height: 150px;
+            margin: 15px 0;
+            padding: 10px;
+            border: 1px solid #ccc;
+            border-radius: 5px;
+            font-size: 16px;
+        }
+        button {
+            display: block;
+            width: 100%;
+            padding: 10px;
+            background-color: #007bff;
+            color: white;
+            border: none;
+            border-radius: 5px;
+            font-size: 16px;
+            cursor: pointer;
+        }
+        button:hover {
+            background-color: #0056b3;
+        }
+        #result {
+            margin-top: 20px;
+            padding: 15px;
+            background-color: #f1f1f1;
+            border: 1px solid #ddd;
+            border-radius: 5px;
+        }
+        .error {
+            color: red;
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <h1>Fast-DetectGPT</h1>
+        <form id="analyzeForm">
+            <textarea name="text" placeholder="Enter your text here..." required></textarea>
+            <button type="submit">Analyze</button>
+        </form>
+        <div id="result"></div>
+    </div>
+
+    <script>
+        document.getElementById('analyzeForm').addEventListener('submit', function (e) {
+            e.preventDefault(); // Formun varsayılan davranışını durdurur.
+            const formData = new FormData(this);
+            const resultDiv = document.getElementById('result');
+
+            // Önce sonucu temizle
+            resultDiv.textContent = '';
+
+            // POST isteği gönder
+            fetch('/analyze', {
+                method: 'POST',
+                headers: {
+                    'Content-Type': 'application/json',
+                },
+                body: JSON.stringify({
+                    text: formData.get('text'),
+                }),
+            })
+                .then(response => response.json())
+                .then(data => {
+                    if (data.error) {
+                        resultDiv.innerHTML = `<p class="error">Error: ${data.error}</p>`;
+                    } else {
+                        resultDiv.innerHTML = `
+                            <p><strong>Criterion:</strong> ${data.criterion}</p>
+                            <p><strong>Probability of being machine-generated:</strong> ${data.probability_machine_generated}</p>
+                        `;
+                    }
+                })
+                .catch(err => {
+                    resultDiv.innerHTML = `<p class="error">An error occurred: ${err.message}</p>`;
+                });
+        });
+    </script>
+</body>
+</html>

local_infer.py

@@ -0,0 +1,94 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import random
+
+import numpy as np
+import torch
+import os
+import glob
+import argparse
+import json
+from scripts.model import load_tokenizer, load_model
+from scripts.fast_detect_gpt import get_sampling_discrepancy_analytic
+
+
+# estimate the probability according to the distribution of our test results on ChatGPT and GPT-4
+class ProbEstimator:
+    def __init__(self, args):
+        self.real_crits = []
+        self.fake_crits = []
+        for result_file in glob.glob(os.path.join(args.ref_path, '*.json')):
+            with open(result_file, 'r') as fin:
+                res = json.load(fin)
+                self.real_crits.extend(res['predictions']['real'])
+                self.fake_crits.extend(res['predictions']['samples'])
+        print(f'ProbEstimator: total {len(self.real_crits) * 2} samples.')
+
+
+    def crit_to_prob(self, crit):
+        offset = np.sort(np.abs(np.array(self.real_crits + self.fake_crits) - crit))[100]
+        cnt_real = np.sum((np.array(self.real_crits) > crit - offset) & (np.array(self.real_crits) < crit + offset))
+        cnt_fake = np.sum((np.array(self.fake_crits) > crit - offset) & (np.array(self.fake_crits) < crit + offset))
+        return cnt_fake / (cnt_real + cnt_fake)
+
+# run interactive local inference
+def run(args):
+    # load model
+    scoring_tokenizer = load_tokenizer(args.scoring_model_name, args.dataset, args.cache_dir)
+    scoring_model = load_model(args.scoring_model_name, args.device, args.cache_dir)
+    scoring_model.eval()
+    if args.reference_model_name != args.scoring_model_name:
+        reference_tokenizer = load_tokenizer(args.reference_model_name, args.dataset, args.cache_dir)
+        reference_model = load_model(args.reference_model_name, args.device, args.cache_dir)
+        reference_model.eval()
+    # evaluate criterion
+    name = "sampling_discrepancy_analytic"
+    criterion_fn = get_sampling_discrepancy_analytic
+    prob_estimator = ProbEstimator(args)
+    # input text
+    print('Local demo for Fast-DetectGPT, where the longer text has more reliable result.')
+    print('')
+    while True:
+        print("Please enter your text: (Press Enter twice to start processing)")
+        lines = []
+        while True:
+            line = input()
+            if len(line) == 0:
+                break
+            lines.append(line)
+        text = "\n".join(lines)
+        if len(text) == 0:
+            break
+        # evaluate text
+        tokenized = scoring_tokenizer(text, truncation=True, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+        labels = tokenized.input_ids[:, 1:]
+        with torch.no_grad():
+            logits_score = scoring_model(**tokenized).logits[:, :-1]
+            if args.reference_model_name == args.scoring_model_name:
+                logits_ref = logits_score
+            else:
+                tokenized = reference_tokenizer(text, truncation=True, return_tensors="pt", padding=True, return_token_type_ids=False).to(args.device)
+                assert torch.all(tokenized.input_ids[:, 1:] == labels), "Tokenizer is mismatch."
+                logits_ref = reference_model(**tokenized).logits[:, :-1]
+            crit = criterion_fn(logits_ref, logits_score, labels)
+        # estimate the probability of machine generated text
+        prob = prob_estimator.crit_to_prob(crit)
+        print(f'Fast-DetectGPT criterion is {crit:.4f}, suggesting that the text has a probability of {prob * 100:.0f}% to be machine-generated.')
+        print()
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--reference_model_name', type=str, default="gpt-neo-2.7B")  # use gpt-j-6B for more accurate detection
+    parser.add_argument('--scoring_model_name', type=str, default="gpt-neo-2.7B")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--ref_path', type=str, default="./local_infer_ref")
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    run(args)
+
+
+

main.sh

@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+exp_path=exp_main
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+datasets="xsum squad writing"
+source_models="gpt2-xl opt-2.7b gpt-neo-2.7B gpt-j-6B gpt-neox-20b"
+
+# preparing dataset
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Preparing dataset ${D}_${M} ...
+    python scripts/data_builder.py --dataset $D --n_samples 500 --base_model_name $M --output_file $data_path/${D}_${M}
+  done
+done
+
+# White-box Setting
+echo `date`, Evaluate models in the white-box setting:
+
+# evaluate Fast-DetectGPT and fast baselines
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating Fast-DetectGPT on ${D}_${M} ...
+    python scripts/fast_detect_gpt.py --reference_model_name $M --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+
+    echo `date`, Evaluating baseline methods on ${D}_${M} ...
+    python scripts/baselines.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done
+
+# evaluate DNA-GPT
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating DNA-GPT on ${D}_${M} ...
+    python scripts/dna_gpt.py --base_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating DetectGPT on ${D}_${M} ...
+    python scripts/detect_gpt.py --scoring_model_name $M --mask_filling_model_name t5-3b --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+     # we leverage DetectGPT to generate the perturbations
+    echo `date`, Evaluating DetectLLM methods on ${D}_${M} ...
+    python scripts/detect_llm.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.t5-3b.perturbation_100 --output_file $res_path/${D}_${M}
+  done
+done
+
+
+# Black-box Setting
+echo `date`, Evaluate models in the black-box setting:
+scoring_models="gpt-neo-2.7B"
+
+# evaluate Fast-DetectGPT
+for D in $datasets; do
+  for M in $source_models; do
+    M1=gpt-j-6B  # sampling model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating Fast-DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/fast_detect_gpt.py --reference_model_name ${M1} --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    M1=t5-3b  # perturbation model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_gpt.py --mask_filling_model_name ${M1} --scoring_model_name ${M2} --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+      # we leverage DetectGPT to generate the perturbations
+      echo `date`, Evaluating DetectLLM methods on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_llm.py --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.${M1}.perturbation_100 --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done

main_ext.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+exp_path=exp_main_ext
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+datasets="xsum squad writing"
+source_models="bloom-7b1 opt-13b llama-13b llama2-13b"
+
+# preparing dataset
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Preparing dataset ${D}_${M} ...
+    python scripts/data_builder.py --dataset $D --n_samples 500 --base_model_name $M --output_file $data_path/${D}_${M}
+  done
+done
+exit
+
+# White-box Setting
+echo `date`, Evaluate models in the white-box setting:
+
+# evaluate Fast-DetectGPT and fast baselines
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating Fast-DetectGPT on ${D}_${M} ...
+    python scripts/fast_detect_gpt.py --reference_model_name $M --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+
+    echo `date`, Evaluating baseline methods on ${D}_${M} ...
+    python scripts/baselines.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating DetectGPT on ${D}_${M} ...
+    python scripts/detect_gpt.py --scoring_model_name $M --mask_filling_model_name t5-3b --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+     # we leverage DetectGPT to generate the perturbations
+    echo `date`, Evaluating DetectLLM methods on ${D}_${M} ...
+    python scripts/detect_llm.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.t5-3b.perturbation_100 --output_file $res_path/${D}_${M}
+  done
+done
+
+
+# Black-box Setting
+echo `date`, Evaluate models in the black-box setting:
+scoring_models="gpt-neo-2.7B"
+
+# evaluate Fast-DetectGPT
+for D in $datasets; do
+  for M in $source_models; do
+    M1=gpt-j-6B  # sampling model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating Fast-DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/fast_detect_gpt.py --reference_model_name ${M1} --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    M1=t5-3b  # perturbation model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_gpt.py --mask_filling_model_name ${M1} --scoring_model_name ${M2} --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+      # we leverage DetectGPT to generate the perturbations
+      echo `date`, Evaluating DetectLLM methods on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_llm.py --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.${M1}.perturbation_100 --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done

metrics.py

@@ -0,0 +1,26 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import matplotlib.pyplot as plt
+from sklearn.metrics import roc_curve, precision_recall_curve, auc
+
+# 15 colorblind-friendly colors
+COLORS = ["#0072B2", "#009E73", "#D55E00", "#CC79A7", "#F0E442",
+            "#56B4E9", "#E69F00", "#000000", "#0072B2", "#009E73",
+            "#D55E00", "#CC79A7", "#F0E442", "#56B4E9", "#E69F00"]
+
+
+def get_roc_metrics(real_preds, sample_preds):
+    fpr, tpr, _ = roc_curve([0] * len(real_preds) + [1] * len(sample_preds), real_preds + sample_preds)
+    roc_auc = auc(fpr, tpr)
+    return fpr.tolist(), tpr.tolist(), float(roc_auc)
+
+
+def get_precision_recall_metrics(real_preds, sample_preds):
+    precision, recall, _ = precision_recall_curve([0] * len(real_preds) + [1] * len(sample_preds),
+                                                  real_preds + sample_preds)
+    pr_auc = auc(recall, precision)
+    return precision.tolist(), recall.tolist(), float(pr_auc)
+

model.py

@@ -0,0 +1,79 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+import time
+import os
+
+def from_pretrained(cls, model_name, kwargs, cache_dir):
+    # use local model if it exists
+    local_path = os.path.join(cache_dir, 'local.' + model_name.replace("/", "_"))
+    if os.path.exists(local_path):
+        return cls.from_pretrained(local_path, **kwargs)
+    return cls.from_pretrained(model_name, **kwargs, cache_dir=cache_dir)
+
+# predefined models
+model_fullnames = {  'gpt2': 'gpt2',
+                     'gpt2-xl': 'gpt2-xl',
+                     'opt-2.7b': 'facebook/opt-2.7b',
+                     'gpt-neo-2.7B': 'EleutherAI/gpt-neo-2.7B',
+                     'gpt-j-6B': 'EleutherAI/gpt-j-6B',
+                     'gpt-neox-20b': 'EleutherAI/gpt-neox-20b',
+                     'mgpt': 'sberbank-ai/mGPT',
+                     'pubmedgpt': 'stanford-crfm/pubmedgpt',
+                     'mt5-xl': 'google/mt5-xl',
+                     'llama-13b': 'huggyllama/llama-13b',
+                     'llama2-13b': 'TheBloke/Llama-2-13B-fp16',
+                     'bloom-7b1': 'bigscience/bloom-7b1',
+                     'opt-13b': 'facebook/opt-13b',
+                     }
+float16_models = ['gpt-j-6B', 'gpt-neox-20b', 'llama-13b', 'llama2-13b', 'bloom-7b1', 'opt-13b']
+
+def get_model_fullname(model_name):
+    return model_fullnames[model_name] if model_name in model_fullnames else model_name
+
+def load_model(model_name, device, cache_dir):
+    model_fullname = get_model_fullname(model_name)
+    print(f'Loading model {model_fullname}...')
+    model_kwargs = {}
+    if model_name in float16_models:
+        model_kwargs.update(dict(torch_dtype=torch.float16))
+    if 'gpt-j' in model_name:
+        model_kwargs.update(dict(revision='float16'))
+    model = from_pretrained(AutoModelForCausalLM, model_fullname, model_kwargs, cache_dir)
+    print('Moving model to GPU...', end='', flush=True)
+    start = time.time()
+    model.to(device)
+    print(f'DONE ({time.time() - start:.2f}s)')
+    return model
+
+def load_tokenizer(model_name, for_dataset, cache_dir):
+    model_fullname = get_model_fullname(model_name)
+    optional_tok_kwargs = {}
+    if "facebook/opt-" in model_fullname:
+        print("Using non-fast tokenizer for OPT")
+        optional_tok_kwargs['fast'] = False
+    if for_dataset in ['pubmed']:
+        optional_tok_kwargs['padding_side'] = 'left'
+    else:
+        optional_tok_kwargs['padding_side'] = 'right'
+    base_tokenizer = from_pretrained(AutoTokenizer, model_fullname, optional_tok_kwargs, cache_dir=cache_dir)
+    if base_tokenizer.pad_token_id is None:
+        base_tokenizer.pad_token_id = base_tokenizer.eos_token_id
+        if '13b' in model_fullname:
+            base_tokenizer.pad_token_id = 0
+    return base_tokenizer
+
+
+if __name__ == '__main__':
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model_name', type=str, default="bloom-7b1")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    load_tokenizer(args.model_name, 'xsum', args.cache_dir)
+    load_model(args.model_name, 'cpu', args.cache_dir)

paraphrasing.py

@@ -0,0 +1,106 @@
+import random
+
+import torch
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+import numpy as np
+import nltk
+from data_builder import load_data, save_data
+from model import from_pretrained
+
+class T5Paraphraser:
+    def __init__(self, args):
+        self.device = args.device
+        self.tokenizer = from_pretrained(AutoTokenizer, args.t5_model_name, {}, args.cache_dir)
+        self.model = from_pretrained(AutoModelForSeq2SeqLM, args.t5_model_name, {}, args.cache_dir)
+        self.model = self.model.to(args.device)
+        self.model.eval()
+
+    def paraphrase(self, sents):
+        parabatch = ["paraphrase: " + sent + " </s>" for sent in sents]
+        encoding = self.tokenizer(parabatch, padding=True, return_tensors="pt")
+        input_ids, attention_masks = encoding["input_ids"].to(self.device), encoding["attention_mask"].to(self.device)
+        outputs = self.model.generate(
+            input_ids=input_ids, attention_mask=attention_masks,
+            max_length=256,
+            do_sample=True,
+            top_k=200,
+            top_p=0.95,
+            early_stopping=True,
+            num_return_sequences=1
+        )
+        assert len(sents) == len(outputs)
+        results = []
+        for output, sent in zip(outputs, sents):
+            line = self.tokenizer.decode(output, skip_special_tokens=True, clean_up_tokenization_spaces=True)
+            line = line.strip()
+            line = line if len(line) > 0 else sent
+            results.append(line)
+        return results
+
+class RandomParaphraser:
+    def __init__(self, args):
+        self.device = args.device
+
+    def paraphrase(self, sents):
+        results = []
+        for sent in sents:
+            words = sent.split()
+            if len(words) > 20:
+                idx = random.randint(0, len(words) - 2)
+                words[idx], words[idx+1] = words[idx+1], words[idx]
+            results.append(' '.join(words))
+        return results
+
+def generate_data(args):
+    data = load_data(args.dataset_file)
+    originals = data['original']
+    samples = data['sampled']
+    print(f"Total number of samples: {len(samples)}")
+    print(f"Average number of words: {np.mean([len(x.split()) for x in samples])}")
+
+    if args.do_random_para:
+        print(f'Using random paraphraser.')
+        paraphraser = RandomParaphraser(args)
+    else:
+        print(f'Loading model {args.t5_model_name}...')
+        paraphraser = T5Paraphraser(args)
+
+    new_samples = []
+    for sample in tqdm(samples):
+        lines = sample.split('\n')
+        new_lines = []
+        for line in lines:
+            line = line.strip()
+            if len(line) == 0:
+                new_lines.append(line)
+            else:
+                sents = nltk.sent_tokenize(line)
+                new_sents = paraphraser.paraphrase(sents)
+                new_lines.append(' '.join(new_sents))
+        new_samples.append('\n'.join(new_lines))
+
+    new_data = {'original': originals, 'sampled': new_samples}
+    save_data(args.output_file, args, new_data)
+
+
+if __name__ == '__main__':
+    import argparse
+    from tqdm import tqdm
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_test/results/xsum_gpt2")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--dataset_file', type=str, default="./exp_test/data/xsum_gpt2")
+    parser.add_argument('--t5_model_name', type=str, default="Vamsi/T5_Paraphrase_Paws")
+    parser.add_argument('--paraphraser', type=str, default="t5", choices=["t5", "random"])
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+
+    import nltk
+    nltk.download('punkt')
+
+    generate_data(args)

report_results.py

@@ -0,0 +1,490 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+import os.path
+import argparse
+import json
+import numpy as np
+
+
+def save_lines(lines, file):
+    with open(file, 'w') as fout:
+        fout.write('\n'.join(lines))
+
+def get_auroc(result_file):
+    with open(result_file, 'r') as fin:
+        res = json.load(fin)
+        return res['metrics']['roc_auc']
+
+def get_fpr_tpr(result_file):
+    with open(result_file, 'r') as fin:
+        res = json.load(fin)
+        return res['metrics']['fpr'], res['metrics']['tpr']
+
+def report_main_results(args):
+    datasets = {'xsum': 'XSum',
+                'squad': 'SQuAD',
+                'writing': 'WritingPrompts'}
+    source_models = {'gpt2-xl': 'GPT-2',
+                     'opt-2.7b': 'OPT-2.7',
+                     'gpt-neo-2.7B': 'Neo-2.7',
+                     'gpt-j-6B': 'GPT-J',
+                     'gpt-neox-20b': 'NeoX'}
+    methods1 = {'likelihood': 'Likelihood',
+               'entropy': 'Entropy',
+               'logrank': 'LogRank',
+               'lrr': 'LRR',
+               'npr': 'NPR'}
+    methods2 = {'perturbation_100': 'DetectGPT',
+               'sampling_discrepancy': 'Fast-DetectGPT'}
+
+    def _get_method_aurocs(dataset, method, filter=''):
+        cols = []
+        for model in source_models:
+            result_file = f'{args.result_path}/{dataset}_{model}{filter}.{method}.json'
+            if os.path.exists(result_file):
+                auroc = get_auroc(result_file)
+            else:
+                auroc = 0.0
+            cols.append(auroc)
+        cols.append(np.mean(cols))
+        return cols
+
+    headers = ['Method'] + [source_models[model] for model in source_models] + ['Avg.']
+    for dataset in datasets:
+        print('----')
+        print(datasets[dataset])
+        print('----')
+        print(' '.join(headers))
+        # basic methods
+        for method in methods1:
+            method_name = methods1[method]
+            cols = _get_method_aurocs(dataset, method)
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+        # white-box comparison
+        results = {}
+        for method in methods2:
+            method_name = methods2[method]
+            cols = _get_method_aurocs(dataset, method)
+            results[method_name] = cols
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+        cols = np.array(results['Fast-DetectGPT']) - np.array(results['DetectGPT'])
+        cols = [f'{col:.4f}' for col in cols]
+        print('(Diff)', ' '.join(cols))
+        # black-box comparison
+        filters = {'perturbation_100': '.t5-3b_gpt-neo-2.7B',
+                    'sampling_discrepancy': '.gpt-j-6B_gpt-neo-2.7B'}
+        results = {}
+        for method in methods2:
+            method_name = methods2[method]
+            cols = _get_method_aurocs(dataset, method, filters[method])
+            results[method_name] = cols
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+        cols = np.array(results['Fast-DetectGPT']) - np.array(results['DetectGPT'])
+        cols = [f'{col:.4f}' for col in cols]
+        print('(Diff)', ' '.join(cols))
+
+def report_main_ext_results(args):
+    datasets = {'xsum': 'XSum',
+                'squad': 'SQuAD',
+                'writing': 'WritingPrompts'}
+    source_models = {'bloom-7b1': 'BLOOM-7.1',
+                     'opt-13b': 'OPT-13',
+                     'llama-13b': 'Llama-13',
+                     'llama2-13b': 'Llama2-13',
+                     }
+    methods1 = {'likelihood': 'Likelihood',
+               'entropy': 'Entropy',
+               'logrank': 'LogRank',
+               'lrr': 'LRR',
+               'npr': 'NPR'}
+    methods2 = {'perturbation_100': 'DetectGPT',
+               'sampling_discrepancy': 'Fast-DetectGPT'}
+
+    def _get_method_aurocs(dataset, method, filter=''):
+        cols = []
+        for model in source_models:
+            result_file = f'{args.result_path}/{dataset}_{model}{filter}.{method}.json'
+            if os.path.exists(result_file):
+                auroc = get_auroc(result_file)
+            else:
+                auroc = 0.0
+            cols.append(auroc)
+        cols.append(np.mean(cols))
+        return cols
+
+    headers = ['Method'] + [source_models[model] for model in source_models] + ['Avg.']
+    for dataset in datasets:
+        print('----')
+        print(datasets[dataset])
+        print('----')
+        print(' '.join(headers))
+        # basic methods
+        for method in methods1:
+            method_name = methods1[method]
+            cols = _get_method_aurocs(dataset, method)
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+        # white-box comparison
+        results = {}
+        for method in methods2:
+            method_name = methods2[method]
+            cols = _get_method_aurocs(dataset, method)
+            results[method_name] = cols
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+        cols = np.array(results['Fast-DetectGPT']) - np.array(results['DetectGPT'])
+        cols = [f'{col:.4f}' for col in cols]
+        print('(Diff)', ' '.join(cols))
+        # black-box comparison
+        filters = {'perturbation_100': '.t5-3b_gpt-neo-2.7B',
+                    'sampling_discrepancy': '.gpt-j-6B_gpt-neo-2.7B'}
+        results = {}
+        for method in methods2:
+            method_name = methods2[method]
+            cols = _get_method_aurocs(dataset, method, filters[method])
+            results[method_name] = cols
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+        cols = np.array(results['Fast-DetectGPT']) - np.array(results['DetectGPT'])
+        cols = [f'{col:.4f}' for col in cols]
+        print('(Diff)', ' '.join(cols))
+
+def report_refmodel_results(args):
+    datasets = {'xsum': 'XSum',
+                'squad': 'SQuAD',
+                'writing': 'WritingPrompts'}
+    source_models = {'gpt2-xl': 'GPT-2',
+                     'gpt-neo-2.7B': 'Neo-2.7',
+                     'gpt-j-6B': 'GPT-J'}
+
+    def _get_method_aurocs(method, ref_model=None):
+        cols = []
+        for dataset in datasets:
+            for model in source_models:
+                filter = '' if ref_model is None or ref_model == model else f'.{ref_model}_{model}'
+                result_file = f'{args.result_path}/{dataset}_{model}{filter}.{method}.json'
+                if os.path.exists(result_file):
+                    auroc = get_auroc(result_file)
+                else:
+                    auroc = 0.0
+                cols.append(auroc)
+        cols.append(np.mean(cols))
+        return cols
+
+    headers1 = ['----'] + list([datasets[d] for d in datasets])
+    headers2 = ['Method'] + [source_models[model] for model in source_models] \
+              + [source_models[model] for model in source_models] \
+              + [source_models[model] for model in source_models] \
+              + ['Avg.']
+    print(' '.join(headers1))
+    print(' '.join(headers2))
+
+    ref_models = [None, 'gpt2-xl', 'gpt-neo-2.7B', 'gpt-j-6B']
+    for ref_model in ref_models:
+        method = 'sampling_discrepancy'
+        method_name = 'Fast-DetectGPT (*/*)' if ref_model is None else f'Fast-DetectGPT ({source_models[ref_model]}/*)'
+        cols = _get_method_aurocs(method, ref_model)
+        cols = [f'{col:.4f}' for col in cols]
+        print(method_name, ' '.join(cols))
+
+
+def report_chatgpt_gpt4_results(args):
+    datasets = {'xsum': 'XSum',
+                'writing': 'Writing',
+                'pubmed': 'PubMed'}
+    source_models = {'gpt-3.5-turbo': 'ChatGPT',
+                     'gpt-4': 'GPT-4'}
+    score_models = { 't5-11b': 'T5-11B',
+                     'gpt2-xl': 'GPT-2',
+                     'opt-2.7b': 'OPT-2.7',
+                     'gpt-neo-2.7B': 'Neo-2.7',
+                     'gpt-j-6B': 'GPT-J',
+                     'gpt-neox-20b': 'NeoX'}
+    methods1 = {'roberta-base-openai-detector': 'RoBERTa-base',
+                'roberta-large-openai-detector': 'RoBERTa-large'}
+    methods2 = {'likelihood': 'Likelihood', 'entropy': 'Entropy', 'logrank': 'LogRank'}
+    methods3 = {'lrr': 'LRR', 'npr': 'NPR', 'perturbation_100': 'DetectGPT',
+                'sampling_discrepancy_analytic': 'Fast'}
+
+    def _get_method_aurocs(method, filter=''):
+        results = []
+        for model in source_models:
+            cols = []
+            for dataset in datasets:
+                result_file = f'{args.result_path}/{dataset}_{model}{filter}.{method}.json'
+                if os.path.exists(result_file):
+                    auroc = get_auroc(result_file)
+                else:
+                    auroc = 0.0
+                cols.append(auroc)
+            cols.append(np.mean(cols))
+            results.extend(cols)
+        return results
+
+    headers1 = ['--'] + [source_models[model] for model in source_models]
+    headers2 = ['Method'] + [datasets[dataset] for dataset in datasets] + ['Avg.'] \
+               + [datasets[dataset] for dataset in datasets] + ['Avg.']
+    print(' '.join(headers1))
+    print(' '.join(headers2))
+    # supervised methods
+    for method in methods1:
+        method_name = methods1[method]
+        cols = _get_method_aurocs(method)
+        cols = [f'{col:.4f}' for col in cols]
+        print(method_name, ' '.join(cols))
+    # zero-shot methods
+
+    filters2 = {'likelihood': ['.gpt2-xl', '.gpt-neo-2.7B', '.gpt-j-6B', '.gpt-neox-20b'],
+               'entropy': ['.gpt2-xl', '.gpt-neo-2.7B', '.gpt-j-6B', '.gpt-neox-20b'],
+               'logrank': ['.gpt2-xl', '.gpt-neo-2.7B', '.gpt-j-6B', '.gpt-neox-20b']}
+    filters3 = {'lrr': ['.t5-11b_gpt2-xl', '.t5-11b_gpt-neo-2.7B', '.t5-11b_gpt-j-6B', '.t5-11b_gpt-neox-20b'],
+               'npr': ['.t5-11b_gpt2-xl', '.t5-11b_gpt-neo-2.7B', '.t5-11b_gpt-j-6B', '.t5-11b_gpt-neox-20b'],
+               'perturbation_100': ['.t5-11b_gpt2-xl', '.t5-11b_gpt-neo-2.7B', '.t5-11b_gpt-j-6B', '.t5-11b_gpt-neox-20b'],
+               'sampling_discrepancy_analytic': ['.gpt-j-6B_gpt2-xl', '.gpt-j-6B_gpt-neo-2.7B', '.gpt-j-6B_gpt-j-6B', '.gpt-neox-20b_gpt-neox-20b']}
+    for method in methods2:
+        for filter in filters2[method]:
+            setting = score_models[filter[1:]]
+            method_name = f'{methods2[method]}({setting})'
+            cols = _get_method_aurocs(method, filter)
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+    for method in methods3:
+        for filter in filters3[method]:
+            setting = [score_models[model] for model in filter[1:].split('_')]
+            method_name = f'{methods3[method]}({setting[0]}/{setting[1]})'
+            cols = _get_method_aurocs(method, filter)
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+
+def report_gpt3_results(args):
+    datasets = {'xsum': 'XSum',
+                'writing': 'Writing',
+                'pubmed': 'PubMed'}
+    source_models = {'davinci': 'GPT-3'}
+    score_models = { 't5-11b': 'T5-11B',
+                     'gpt2-xl': 'GPT-2',
+                     'opt-2.7b': 'OPT-2.7',
+                     'gpt-neo-2.7B': 'Neo-2.7',
+                     'gpt-j-6B': 'GPT-J',
+                     'gpt-neox-20b': 'NeoX'}
+    methods1 = {'roberta-base-openai-detector': 'RoBERTa-base',
+                'roberta-large-openai-detector': 'RoBERTa-large'}
+    methods2 = {'likelihood': 'Likelihood', 'entropy': 'Entropy', 'logrank': 'LogRank'}
+    methods3 = {'lrr': 'LRR', 'npr': 'NPR', 'perturbation_100': 'DetectGPT',
+                'sampling_discrepancy_analytic': 'Fast'}
+
+    def _get_method_aurocs(method, filter=''):
+        results = []
+        for model in source_models:
+            cols = []
+            for dataset in datasets:
+                result_file = f'{args.result_path}/{dataset}_{model}{filter}.{method}.json'
+                if os.path.exists(result_file):
+                    auroc = get_auroc(result_file)
+                else:
+                    auroc = 0.0
+                cols.append(auroc)
+            cols.append(np.mean(cols))
+            results.extend(cols)
+        return results
+
+    headers1 = ['--'] + [source_models[model] for model in source_models]
+    headers2 = ['Method'] + [datasets[dataset] for dataset in datasets] + ['Avg.'] \
+               + [datasets[dataset] for dataset in datasets] + ['Avg.']
+    print(' '.join(headers1))
+    print(' '.join(headers2))
+    # supervised methods
+    for method in methods1:
+        method_name = methods1[method]
+        cols = _get_method_aurocs(method)
+        cols = [f'{col:.4f}' for col in cols]
+        print(method_name, ' '.join(cols))
+    # zero-shot methods
+
+    filters2 = {'likelihood': ['.gpt2-xl', '.gpt-neo-2.7B', '.gpt-j-6B', '.gpt-neox-20b'],
+               'entropy': ['.gpt2-xl', '.gpt-neo-2.7B', '.gpt-j-6B', '.gpt-neox-20b'],
+               'logrank': ['.gpt2-xl', '.gpt-neo-2.7B', '.gpt-j-6B', '.gpt-neox-20b']}
+    filters3 = {'lrr': ['.t5-11b_gpt2-xl', '.t5-11b_gpt-neo-2.7B', '.t5-11b_gpt-j-6B', '.t5-11b_gpt-neox-20b'],
+               'npr': ['.t5-11b_gpt2-xl', '.t5-11b_gpt-neo-2.7B', '.t5-11b_gpt-j-6B', '.t5-11b_gpt-neox-20b'],
+               'perturbation_100': ['.t5-11b_gpt2-xl', '.t5-11b_gpt-neo-2.7B', '.t5-11b_gpt-j-6B', '.t5-11b_gpt-neox-20b'],
+               'sampling_discrepancy_analytic': ['.gpt-j-6B_gpt2-xl', '.gpt-j-6B_gpt-neo-2.7B', '.gpt-j-6B_gpt-j-6B', '.gpt-neox-20b_gpt-neox-20b']}
+    for method in methods2:
+        for filter in filters2[method]:
+            setting = score_models[filter[1:]]
+            method_name = f'{methods2[method]}({setting})'
+            cols = _get_method_aurocs(method, filter)
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+    for method in methods3:
+        for filter in filters3[method]:
+            setting = [score_models[model] for model in filter[1:].split('_')]
+            method_name = f'{methods3[method]}({setting[0]}/{setting[1]})'
+            cols = _get_method_aurocs(method, filter)
+            cols = [f'{col:.4f}' for col in cols]
+            print(method_name, ' '.join(cols))
+
+def report_maxlen_trends(args):
+    datasets = {'xsum': 'XSum',
+                'writing': 'WritingPrompts'}
+    source_models = {'gpt-3.5-turbo': 'ChatGPT',
+                     'gpt-4': 'GPT-4'}
+    score_models = {'t5-11b': 'T5-11B',
+                    'gpt2-xl': 'GPT-2',
+                    'opt-2.7b': 'OPT-2.7',
+                    'gpt-neo-2.7B': 'Neo-2.7',
+                    'gpt-j-6B': 'GPT-J',
+                    'gpt-neox-20b': 'NeoX'}
+    methods1 = {'roberta-base-openai-detector': 'RoBERTa-base',
+                'roberta-large-openai-detector': 'RoBERTa-large'}
+    methods2 = {'likelihood': 'Likelihood'}
+    methods3 = {'perturbation_100': 'DetectGPT',
+                'sampling_discrepancy_analytic': 'Fast-Detect'}
+    maxlens = [30, 60, 90, 120, 150, 180]
+
+    def _get_method_aurocs(root_path, dataset, source_model, method, filter=''):
+        cols = []
+        for maxlen in maxlens:
+            result_file = f'{root_path}/exp_maxlen{maxlen}/results/{dataset}_{source_model}{filter}.{method}.json'
+            if os.path.exists(result_file):
+                auroc = get_auroc(result_file)
+            else:
+                auroc = 0.0
+            cols.append(auroc)
+        return cols
+
+    filters2 = {'likelihood': '.gpt-neo-2.7B'}
+    filters3 = {'perturbation_100': '.t5-11b_gpt-neo-2.7B',
+                'sampling_discrepancy_analytic': '.gpt-j-6B_gpt-neo-2.7B'}
+
+    headers = ['Method'] + [str(maxlen) for maxlen in maxlens]
+    print(' '.join(headers))
+    # print table per model and dataset
+    results = {}
+    for model in source_models:
+        model_name = source_models[model]
+        for data in datasets:
+            data_name = datasets[data]
+            print('----')
+            print(f'{model_name} / {data_name}')
+            print('----')
+            for method in methods1:
+                method_name = methods1[method]
+                cols = _get_method_aurocs('.', data, model, method)
+                results[f'{model_name}_{data_name}_{method_name}'] = cols
+                cols = [f'{col:.4f}' for col in cols]
+                print(method_name, ' '.join(cols))
+            for method in methods2:
+                filter = filters2[method]
+                setting = score_models[filter[1:]]
+                method_name = f'{methods2[method]}({setting})'
+                cols = _get_method_aurocs('.', data, model, method, filter)
+                results[f'{model_name}_{data_name}_{method_name}'] = cols
+                cols = [f'{col:.4f}' for col in cols]
+                print(method_name, ' '.join(cols))
+            for method in methods3:
+                filter = filters3[method]
+                setting = [score_models[model] for model in filter[1:].split('_')]
+                method_name = f'{methods3[method]}({setting[0]}/{setting[1]})'
+                cols = _get_method_aurocs('.', data, model, method, filter)
+                results[f'{model_name}_{data_name}_{method_name}'] = cols
+                cols = [f'{col:.4f}' for col in cols]
+                print(method_name, ' '.join(cols))
+    import json
+    json_file = './exp_analysis/maxlen_trends.json'
+    with open(json_file, 'w') as fout:
+        json.dump(results, fout)
+        print(f'Write to file {json_file}')
+
+def report_auroc_curve(args):
+    datasets = {'xsum': 'XSum',
+                'writing': 'WritingPrompts'}
+    source_models = {'gpt-3.5-turbo': 'ChatGPT',
+                     'gpt-4': 'GPT-4'}
+    score_models = {'t5-11b': 'T5-11B',
+                    'gpt2-xl': 'GPT-2',
+                    'opt-2.7b': 'OPT-2.7',
+                    'gpt-neo-2.7B': 'Neo-2.7',
+                    'gpt-j-6B': 'GPT-J',
+                    'gpt-neox-20b': 'NeoX'}
+    methods1 = {'roberta-base-openai-detector': 'RoBERTa-base',
+                'roberta-large-openai-detector': 'RoBERTa-large'}
+    methods2 = {'likelihood': 'Likelihood'}
+    methods3 = {'perturbation_100': 'DetectGPT',
+                'sampling_discrepancy_analytic': 'Fast-Detect'}
+
+    def _get_method_fpr_tpr(root_path, dataset, source_model, method, filter=''):
+        maxlen = 180
+        result_file = f'{root_path}/exp_maxlen{maxlen}/results/{dataset}_{source_model}{filter}.{method}.json'
+        if os.path.exists(result_file):
+            fpr, tpr = get_fpr_tpr(result_file)
+        else:
+            fpr, tpr = [], []
+        assert len(fpr) == len(tpr)
+        return list(zip(fpr, tpr))
+
+    filters2 = {'likelihood': '.gpt-neo-2.7B'}
+    filters3 = {'perturbation_100': '.t5-11b_gpt-neo-2.7B',
+                'sampling_discrepancy_analytic': '.gpt-j-6B_gpt-neo-2.7B'}
+
+    # print table per model and dataset
+    results = {}
+    for model in source_models:
+        model_name = source_models[model]
+        for data in datasets:
+            data_name = datasets[data]
+            print('----')
+            print(f'{model_name} / {data_name}')
+            print('----')
+            for method in methods1:
+                method_name = methods1[method]
+                cols = _get_method_fpr_tpr('.', data, model, method)
+                results[f'{model_name}_{data_name}_{method_name}'] = cols
+                cols = [f'({col[0]:.3f},{col[1]:.3f})' for col in cols]
+                print(method_name, ' '.join(cols))
+            for method in methods2:
+                filter = filters2[method]
+                setting = score_models[filter[1:]]
+                method_name = f'{methods2[method]}({setting})'
+                cols = _get_method_fpr_tpr('.', data, model, method, filter)
+                results[f'{model_name}_{data_name}_{method_name}'] = cols
+                cols = [f'({col[0]:.3f},{col[1]:.3f})' for col in cols]
+                print(method_name, ' '.join(cols))
+            for method in methods3:
+                filter = filters3[method]
+                setting = [score_models[model] for model in filter[1:].split('_')]
+                method_name = f'{methods3[method]}({setting[0]}/{setting[1]})'
+                cols = _get_method_fpr_tpr('.', data, model, method, filter)
+                results[f'{model_name}_{data_name}_{method_name}'] = cols
+                cols = [f'({col[0]:.3f},{col[1]:.3f})' for col in cols]
+                print(method_name, ' '.join(cols))
+    import json
+    json_file = './exp_analysis/auroc_curve.json'
+    with open(json_file, 'w') as fout:
+        json.dump(results, fout)
+        print(f'Write to file {json_file}')
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--result_path', type=str, default="./exp_main/results/")
+    parser.add_argument('--report_name', type=str, default="main_results")
+    args = parser.parse_args()
+
+    if args.report_name == 'main_results':
+        report_main_results(args)
+    elif args.report_name == 'main_ext_results':
+        report_main_ext_results(args)
+    elif args.report_name == 'chatgpt_gpt4_results':
+        report_chatgpt_gpt4_results(args)
+    elif args.report_name == 'gpt3_results':
+        report_gpt3_results(args)
+    elif args.report_name == 'maxlen_trends':
+        report_maxlen_trends(args)
+    elif args.report_name == 'auroc_curve':
+        report_auroc_curve(args)
+    elif args.report_name == 'refmodel_results':
+        report_refmodel_results(args)

requirements.txt

@@ -1,3 +1,8 @@
-streamlit
-transformers
-torch==2.0.0
+torch
+numpy
+transformers==4.28.1
+datasets==2.12.0
+matplotlib
+tqdm
+openai
+nltk

setup.sh

@@ -0,0 +1 @@
+pip install -r requirements.txt

show_result.py

@@ -0,0 +1,51 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import matplotlib
+import matplotlib.pyplot as plt
+import argparse
+import glob
+import json
+from os import path
+
+import numpy as np
+
+matplotlib.use('Agg')
+
+# plot histogram of sampled on left, and original on right
+def save_histogram(predictions, figure_file):
+    plt.figure(figsize=(4, 2.5))
+    plt.subplot(1, 1, 1)
+    plt.hist(predictions["samples"], alpha=0.5, bins='auto', label='Model')
+    plt.hist(predictions["real"], alpha=0.5, bins='auto', label='Human')
+    plt.xlabel("Sampling Discrepancy")
+    plt.ylabel('Frequency')
+    plt.legend(loc='upper right')
+    plt.tight_layout()
+    plt.savefig(figure_file)
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--result_files', type=str, default="./exp_test/results/*.json")
+    parser.add_argument('--draw', action='store_true')
+    args = parser.parse_args()
+
+    for res_file in glob.glob(args.result_files, recursive=True):
+        with open(res_file, 'r') as fin:
+            res = json.load(fin)
+        if 'metrics' in res:
+            n_samples = res['info']['n_samples']
+            roc_auc = res['metrics']['roc_auc']
+            real = res['predictions']['real']
+            samples = res['predictions']['samples']
+            print(f"{res_file}: roc_auc={roc_auc:.4f} n_samples={n_samples} r:{np.mean(real):.2f}/{np.std(real):.2f} s:{np.mean(samples):.2f}/{np.std(samples):.2f}")
+        else:
+            print(f"{res_file}: metrics not found.")
+        # draw histogram
+        if args.draw:
+            fig_file = f"{res_file}.pdf"
+            save_histogram(res['predictions'], fig_file)
+            print(f"{fig_file}: histogram figure saved.")
+

supervised.py

@@ -0,0 +1,78 @@
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import numpy as np
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+import tqdm
+import argparse
+import json
+from data_builder import load_data
+from metrics import get_roc_metrics, get_precision_recall_metrics
+from model import from_pretrained
+
+def experiment(args):
+    # load model
+    print(f'Beginning supervised evaluation with {args.model_name}...')
+    detector = from_pretrained(AutoModelForSequenceClassification, args.model_name, {}, args.cache_dir).to(args.device)
+    tokenizer = from_pretrained(AutoTokenizer, args.model_name, {}, args.cache_dir)
+    detector.eval()
+    # load data
+    data = load_data(args.dataset_file)
+    n_samples = len(data["sampled"])
+    # eval detector
+    name = args.model_name
+    torch.manual_seed(args.seed)
+    np.random.seed(args.seed)
+    eval_results = []
+    for idx in tqdm.tqdm(range(n_samples), desc=f"Computing {name} criterion"):
+        original_text = data["original"][idx]
+        sampled_text = data["sampled"][idx]
+        # original text
+        tokenized = tokenizer(original_text, padding=True, truncation=True, max_length=512, return_tensors="pt").to(args.device)
+        with torch.no_grad():
+            original_crit = detector(**tokenized).logits.softmax(-1)[0, 0].item()
+        # sampled text
+        tokenized = tokenizer(sampled_text, padding=True, truncation=True, max_length=512, return_tensors="pt").to(args.device)
+        with torch.no_grad():
+            sampled_crit = detector(**tokenized).logits.softmax(-1)[0, 0].item()
+        # result
+        eval_results.append({"original": original_text,
+                        "original_crit": original_crit,
+                        "sampled": sampled_text,
+                        "sampled_crit": sampled_crit})
+
+    # compute prediction scores for real/sampled passages
+    predictions = {'real': [x["original_crit"] for x in eval_results],
+                   'samples': [x["sampled_crit"] for x in eval_results]}
+    fpr, tpr, roc_auc = get_roc_metrics(predictions['real'], predictions['samples'])
+    p, r, pr_auc = get_precision_recall_metrics(predictions['real'], predictions['samples'])
+    print(f"Criterion {name}_threshold ROC AUC: {roc_auc:.4f}, PR AUC: {pr_auc:.4f}")
+    # log results
+    results_file = f'{args.output_file}.{name}.json'
+    results = { 'name': f'{name}_threshold',
+                'info': {'n_samples': n_samples},
+                'predictions': predictions,
+                'raw_results': eval_results,
+                'metrics': {'roc_auc': roc_auc, 'fpr': fpr, 'tpr': tpr},
+                'pr_metrics': {'pr_auc': pr_auc, 'precision': p, 'recall': r},
+                'loss': 1 - pr_auc}
+    with open(results_file, 'w') as fout:
+        json.dump(results, fout)
+        print(f'Results written into {results_file}')
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--output_file', type=str, default="./exp_test/results/xsum_gpt2")
+    parser.add_argument('--dataset', type=str, default="xsum")
+    parser.add_argument('--dataset_file', type=str, default="./exp_test/data/xsum_gpt2")
+    parser.add_argument('--model_name', type=str, default="roberta-base-openai-detector")
+    parser.add_argument('--seed', type=int, default=0)
+    parser.add_argument('--device', type=str, default="cuda")
+    parser.add_argument('--cache_dir', type=str, default="../cache")
+    args = parser.parse_args()
+
+    experiment(args)

supervised.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+exp_path=exp_supervised
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+# preparing dataset
+for P in "english:mgpt" "german:mgpt" "pubmed:pubmedgpt" "xsum:gpt2-xl"; do
+  IFS=':' read -r -a P <<< $P && D=${P[0]} && M=${P[1]}
+  echo `date`, Preparing dataset ${D}-${M} ...
+  python scripts/data_builder.py --dataset $D --n_samples 200 --base_model_name $M --output_file $data_path/${D}_${M}
+done
+
+# evaluate baselines
+for P in "english:mgpt" "german:mgpt" "pubmed:pubmedgpt" "xsum:gpt2-xl"; do
+  IFS=':' read -r -a P <<< $P && D=${P[0]} && M=${P[1]}
+  echo `date`, Evaluating baseline methods on ${D}_${M} ...
+  python scripts/baselines.py --scoring_model_name $M --dataset $D \
+                        --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+done
+
+# evaluate supervised detectors
+for P in "english:mgpt" "german:mgpt" "pubmed:pubmedgpt" "xsum:gpt2-xl"; do
+  IFS=':' read -r -a P <<< $P && D=${P[0]} && M=${P[1]}
+  for SM in roberta-base-openai-detector roberta-large-openai-detector; do
+    echo `date`, Evaluating ${SM} on ${D}_${M} ...
+    python scripts/supervised.py --model_name $SM --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done
+
+# evaluate DetectGPT
+for P in "english:mgpt:mt5-xl" "german:mgpt:mt5-xl" "pubmed:pubmedgpt:t5-11b" "xsum:gpt2-xl:t5-11b"; do
+  IFS=':' read -r -a P <<< $P && D=${P[0]} && M1=${P[1]} && M2=${P[2]}
+  echo `date`, Evaluating DetectGPT on ${D}_${M1}_${M2} ...
+  python scripts/detect_gpt.py --scoring_model_name $M1 --mask_filling_model_name $M2 --n_perturbations 100 --dataset $D \
+                        --dataset_file $data_path/${D}_${M1} --output_file $res_path/${D}_${M1}_${M2}
+done
+
+# evaluate Fast-DetectGPT
+for P in "english:mgpt" "german:mgpt" "pubmed:pubmedgpt" "xsum:gpt2-xl"; do
+  IFS=':' read -r -a P <<< $P && D=${P[0]} && M=${P[1]}
+  echo `date`, Evaluating Fast-DetectGPT on ${D}-${M} ...
+  python scripts/fast_detect_gpt.py --reference_model_name $M --scoring_model_name $M \
+                      --dataset $D --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+done

temperature.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+exp_path=exp_temperature
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+datasets="xsum squad writing"
+source_models="gpt2-xl opt-2.7b gpt-neo-2.7B gpt-j-6B gpt-neox-20b"
+
+# preparing dataset
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Preparing dataset ${D}-${M} ...
+    python scripts/data_builder.py --dataset $D --n_samples 500 --do_temperature --base_model_name $M --output_file $data_path/${D}_${M}
+  done
+done
+
+# White-box Setting
+echo `date`, Evaluate models in the white-box setting:
+
+# evaluate Fast-DetectGPT and fast baselines
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating Fast-DetectGPT on ${D}_${M} ...
+    python scripts/fast_detect_gpt.py --reference_model_name $M --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+
+    echo `date`, Evaluating baseline methods on ${D}_${M} ...
+    python scripts/baselines.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating DetectGPT on ${D}_${M} ...
+    python scripts/detect_gpt.py --scoring_model_name $M --mask_filling_model_name t5-3b --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+     # we leverage DetectGPT to generate the perturbations
+    echo `date`, Evaluating DetectLLM methods on ${D}_${M} ...
+    python scripts/detect_llm.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.t5-3b.perturbation_100 --output_file $res_path/${D}_${M}
+  done
+done
+
+
+# Black-box Setting
+echo `date`, Evaluate models in the black-box setting:
+scoring_models="gpt-neo-2.7B"
+
+# evaluate Fast-DetectGPT
+for D in $datasets; do
+  for M in $source_models; do
+    M1=gpt-j-6B  # sampling model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating Fast-DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/fast_detect_gpt.py --reference_model_name ${M1} --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    M1=t5-3b  # perturbation model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_gpt.py --mask_filling_model_name ${M1} --scoring_model_name ${M2} --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+      # we leverage DetectGPT to generate the perturbations
+      echo `date`, Evaluating DetectLLM methods on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_llm.py --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.${M1}.perturbation_100 --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done

topk.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+exp_path=exp_topk
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+datasets="xsum squad writing"
+source_models="gpt2-xl opt-2.7b gpt-neo-2.7B gpt-j-6B gpt-neox-20b"
+
+# preparing dataset
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Preparing dataset ${D}-${M} ...
+    python scripts/data_builder.py --dataset $D --n_samples 500 --do_top_k --base_model_name $M --output_file $data_path/${D}_${M}
+  done
+done
+
+# White-box Setting
+echo `date`, Evaluate models in the white-box setting:
+
+# evaluate Fast-DetectGPT and fast baselines
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating Fast-DetectGPT on ${D}_${M} ...
+    python scripts/fast_detect_gpt.py --reference_model_name $M --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+
+    echo `date`, Evaluating baseline methods on ${D}_${M} ...
+    python scripts/baselines.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating DetectGPT on ${D}_${M} ...
+    python scripts/detect_gpt.py --scoring_model_name $M --mask_filling_model_name t5-3b --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+     # we leverage DetectGPT to generate the perturbations
+    echo `date`, Evaluating DetectLLM methods on ${D}_${M} ...
+    python scripts/detect_llm.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.t5-3b.perturbation_100 --output_file $res_path/${D}_${M}
+  done
+done
+
+
+# Black-box Setting
+echo `date`, Evaluate models in the black-box setting:
+scoring_models="gpt-neo-2.7B"
+
+# evaluate Fast-DetectGPT
+for D in $datasets; do
+  for M in $source_models; do
+    M1=gpt-j-6B  # sampling model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating Fast-DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/fast_detect_gpt.py --reference_model_name ${M1} --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    M1=t5-3b  # perturbation model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_gpt.py --mask_filling_model_name ${M1} --scoring_model_name ${M2} --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+      # we leverage DetectGPT to generate the perturbations
+      echo `date`, Evaluating DetectLLM methods on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_llm.py --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.${M1}.perturbation_100 --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done

topp.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# Copyright (c) Guangsheng Bao.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# setup the environment
+echo `date`, Setup the environment ...
+set -e  # exit if error
+
+# prepare folders
+exp_path=exp_topp
+data_path=$exp_path/data
+res_path=$exp_path/results
+mkdir -p $exp_path $data_path $res_path
+
+datasets="xsum squad writing"
+source_models="gpt2-xl opt-2.7b gpt-neo-2.7B gpt-j-6B gpt-neox-20b"
+
+# preparing dataset
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Preparing dataset ${D}-${M} ...
+    python scripts/data_builder.py --dataset $D --n_samples 500 --do_top_p --base_model_name $M --output_file $data_path/${D}_${M}
+  done
+done
+
+# White-box Setting
+echo `date`, Evaluate models in the white-box setting:
+
+# evaluate Fast-DetectGPT and fast baselines
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating Fast-DetectGPT on ${D}_${M} ...
+    python scripts/fast_detect_gpt.py --reference_model_name $M --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+
+    echo `date`, Evaluating baseline methods on ${D}_${M} ...
+    python scripts/baselines.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    echo `date`, Evaluating DetectGPT on ${D}_${M} ...
+    python scripts/detect_gpt.py --scoring_model_name $M --mask_filling_model_name t5-3b --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}
+     # we leverage DetectGPT to generate the perturbations
+    echo `date`, Evaluating DetectLLM methods on ${D}_${M} ...
+    python scripts/detect_llm.py --scoring_model_name $M --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.t5-3b.perturbation_100 --output_file $res_path/${D}_${M}
+  done
+done
+
+
+# Black-box Setting
+echo `date`, Evaluate models in the black-box setting:
+scoring_models="gpt-neo-2.7B"
+
+# evaluate Fast-DetectGPT
+for D in $datasets; do
+  for M in $source_models; do
+    M1=gpt-j-6B  # sampling model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating Fast-DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/fast_detect_gpt.py --reference_model_name ${M1} --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done
+
+# evaluate DetectGPT and its improvement DetectLLM
+for D in $datasets; do
+  for M in $source_models; do
+    M1=t5-3b  # perturbation model
+    for M2 in $scoring_models; do
+      echo `date`, Evaluating DetectGPT on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_gpt.py --mask_filling_model_name ${M1} --scoring_model_name ${M2} --n_perturbations 100 --dataset $D \
+                          --dataset_file $data_path/${D}_${M} --output_file $res_path/${D}_${M}.${M1}_${M2}
+      # we leverage DetectGPT to generate the perturbations
+      echo `date`, Evaluating DetectLLM methods on ${D}_${M}.${M1}_${M2} ...
+      python scripts/detect_llm.py --scoring_model_name ${M2} --dataset $D \
+                          --dataset_file $data_path/${D}_${M}.${M1}.perturbation_100 --output_file $res_path/${D}_${M}.${M1}_${M2}
+    done
+  done
+done