Fine-tuning can Help Detect Pretraining Data from Large Language Models
Under review as a conference paper at ICLR 2025

FINE-TUNING CAN HELP DETECT PRETRAINING DATA
FROM LARGE LANGUAGE MODELS

Anonymous authors
Paper under double-blind review

ABSTRACT

In the era of large language models (LLMs), detecting pretraining data has been
increasingly important due to concerns about fair evaluation and ethical risks.
Current methods differentiate members and non-members by designing scoring
functions, like Perplexity and Min-k%. However, the diversity and complexity of
training data magnifies the difficulty of distinguishing, leading to suboptimal per-
formance in detecting pretraining data. In this paper, we first explore the benefits
of unseen data, which can be easily collected after the release of the LLM. We find
that the perplexities of LLMs perform differently for members and non-members,
after fine-tuning with a small amount of previously unseen data. In light of this,
we introduce a novel and effective method termed Fine-tuned Score Deviation
(FSD), which improves the performance of current scoring functions for pretrain-
ing data detection. In particular, we propose to measure the deviation distance
of current scores after fine-tuning on a small amount of unseen data within the
same domain. In effect, using a few unseen data can largely decrease the scores of
all non-members, leading to a larger deviation distance than members. Extensive
experiments demonstrate the effectiveness of our method, significantly improving
the AUC score on common benchmark datasets across various models.

1

INTRODUCTION

The impressive performance of large language models (LLMs) arises from large-scale pretraining
on massive datasets collected from the internet (Achiam et al., 2023; Touvron et al., 2023b). But,
model developers are often reluctant to disclose detailed information about the pretraining datasets,
raising significant concerns regarding fair evaluation and ethical risks. Specifically, Recent studies
reveal that the pretraining corpus may inadvertently include data from evaluation benchmarks (Sainz
et al., 2023; Balloccu et al., 2024), making it difficult to assess the practical capability of LLMs.
Besides, LLMs often generate text from copyrighted books (Grynbaum & Mac, 2023) and personal
emails (Mozes et al., 2023), which could infringe on the legal rights of the original content creators
and violate their privacy. Considering the vast size of the pretraining dataset and the single iteration
of pretraining, it has been increasingly important and challenging to detect pretraining data, which
determines whether a piece of text is part of the pretraining dataset.

In the literature, current works of detecting pretraining data primarily focus on designing scoring
functions to differentiate members (i.e., seen data during pretraining) and non-members (unseen).
For example, previous work shows that sequences leak in the training data tend to have lower per-
plexity (i.e., higher likelihood) than non-members (Li, 2023). Min-k% leverages the k% of tokens
with minimum token probabilities of a text for detection, assuming that trained data tends to contain
fewer outlier tokens (Shi et al., 2024). However, non-member data can obtain low perplexities by
including frequent or repetitive texts, while members may contain rare tokens that result in high per-
plexities. This casts significant doubt on utilizing those scoring functions for detecting pretraining
data. Consequently, this issue prompts us to present a preliminary attempt to enlarge the difference
between members and non-members for pretraining datasets of LLMs.

In this work, we propose Fine-tuned Score Deviation (FSD), a novel and effective approach that
improves the detection capabilities of current scoring functions in a specific domain (e.g., event data
from Wikipedia, arXiv research papers). Our method is motivated by an empirical analysis of the
perplexity deviation after model fine-tuning. We find that when fine-tuned with a few previously

1

000
001

002
003
004
005
006
007

008
009
010
011
012
013

014
015
016
017
018

019
020
021
022
023
024

025
026
027
028
029
030

031
032
033
034
035
036

037
038
039
040
041

042
043
044
045
046
047

048
049
050
051
052
053

Under review as a conference paper at ICLR 2025

054
055

056
057
058
059
060
061

062
063
064
065
066
067

068
069
070
071
072

073
074
075
076
077
078

079
080
081
082
083
084

085
086
087
088
089
090

091
092
093
094
095

096
097
098
099
100
101

102
103
104
105
106
107

Figure 1: Overview of Fine-tuned Score Deviation. To determine whether a text X is in the pre-
training data, our method first fine-tunes the pre-trained model with a few non-members and then
measures the deviation distance of scores from the pre-trained model and fine-tuned model as a
membership inference metric. If the deviation value is smaller than the threshold value, the text is
likely in the pretraining data.

unseen data from a specific domain, the perplexities of LLMs experience a significantly larger de-
crease for other unknown non-members in the domain compared to the members. This suggests the
possibility of using the disparity to distinguish between members and non-members.

Therefore, our key idea behind FSD is to enlarge the score deviation between members and non-
members by exposing the LLM to a few non-members. This can be accomplished by measuring the
deviation distance of current scores (See Figure 1), owing to the self-supervised fine-tuning on a few
non-members. In effect, the fine-tuning largely decreases the scores of non-member data, resulting
in more distinguishable seen and unseen data. In practical applications, it is easy to collect a small
amount of unseen data for LLMs within a specific domain. For example, we can make use of those
contents (e.g., journal articles) published subsequent to the release of the LLM.

To verify the effectiveness of our method, we conduct extensive experiments on various datasets,
including WikiMIA, BookMIA (Shi et al., 2024), ArXivTection and BookTection (Duarte et al.,
2024). The results demonstrate that our method can significantly improve the performance of ex-
isting methods based on scoring functions. For example, our method improves the AUC score of
the best baseline method Min-k%, increasing it from 0.62 to 0.91 on WikiMIA under the OPT-6.7B
model. Moreover, our method can also improve the TPR@5%FPR score of baseline methods. For
example, our method improves the TPR@5%FPR score of the detection method using perplexity,
increasing it from 0.10 to 0.81 on ArXivTection under the LLaMA-7B model.

Our main contributions are as follows:

• We analyze the limitations of existing methods based on scoring functions for pretraining
data detection. The significant overlap in metric score distribution between seen data and
unseen data results in the inferior performance of detection methods.

• We propose Fine-tuned Score Deviation (FSD), a novel and effective method for detecting
pretraining data from large language models. The core idea is to enlarge the gap between
members and non-members by exposing the LLM to a few unseen data.

• We empirically show that FSD can improve the performance of existing detection methods
based on scoring functions for detecting pretraining data, through extensive experiments
conducted on various benchmark datasets with diverse large language models.

2 BACKGROUND

In this work, we focus on detecting pretraining data, the problem of detecting whether a piece of text
is included in the pretraining data of a specific LLM. First, we formally define the problem setup
and its challenges. Then, we introduce two commonly used methods for this task.

2

Pre-trained ModelFine-tuned ModelX: The centenary of the outbreak of World War I was commemorated in Europe Fine-tuningNon-members00.10.20.30.4…theofcentenaryThe00.30.60.9…theofcentenaryTheMemberNon-member𝑺𝒙;𝒇𝜽−𝑺𝒙;𝒇𝜽′<𝜺Under review as a conference paper at ICLR 2025

108
109

110
111
112
113
114
115

116
117
118
119
120
121

122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137
138

139
140
141
142
143
144

145
146
147
148
149

150
151
152
153
154
155

156
157
158
159
160
161

Pretraining data detection Pretraining data detection is an instance of membership inference
attacks (MIAs) (Shokri et al., 2017), and can help identify data contamination in the pretraining
corpus (Shi et al., 2024). Let f be an autoregressive large language model (LLM) with trainable
parameters θ (e.g., LLaMA (Touvron et al., 2023a)) and D denotes the associated pretraining data,
sampled from an underlying distribution P. As model developers rarely provide detailed information
about the pretraining dataset, we generally desire to identify if the LLM is trained on the given text
for scientific and ethical concerns. Formally, the task objective is to learn a detector h that can infer
the membership of an arbitrary data point x in the dataset D: h(x, fθ) → {0, 1}.

Unlike the black-box assumption in previous works (Shi et al., 2024; Oren et al., 2024), we assume
the access to fine-tune LLMs with custom datasets and the output probabilities of LLMs, which is
realistic for open-sourced LLMs and many commercial APIs, such as GPT-4o1. In addition, the
detector can obtain a few data samples {xi}N
i=0 that belong to the same domain as the given sample
x and do not present in the training set. This can be achieved by collecting those contents (e.g.,
journal articles) published after the release of the LLM.

The task of pretraining data detection can be formulated as a binary classification: determining
whether a given text x is a member or non-member of the pretraining dataset D. Pretraining data
detection can be performed by a level-set estimation:

h(x; fθ) =

(cid:26)member

non-member

if S(x; fθ) < ϵ,
if S(x; fθ) ≥ ϵ,

(1)

where S(x; fθ) denotes a scoring function and ϵ is the threshold determined by a validation dataset.
By convention, examples with lower scores S(x; fθ) are classified as members of pretraining data
and vice versa. In the following, we introduce two popular scoring functions for the task.

Scoring functions For large language models, likelihood is typically used to estimate the uncer-
tainty in generating new tokens. In particular, a high likelihood indicates that the model predicts
tokens with high confidence. Given a piece of text x = {x1, x2, ..., xn}, the likelihood of the next
token xn+1 is pθ(xn+1|x1, ..., xn). In general, a piece of text seen in pre-training tends to have
more tokens with a high likelihood, whereas unseen texts have more tokens with a low likelihood.

In light of this, previous studies usually design likelihood-based scoring functions to detect pretrain-
ing data (Shi et al., 2024; Carlini et al., 2021; Li, 2023). For example, Perplexity is proposed to
distinguish members and non-members, based on the observation that members tend to have lower
perplexity than non-members (Li, 2023). Formally, The perplexity of x is calculated as:

Perplexity(x; fθ) = exp{−

1
n

n
(cid:88)

i=1

log pθ(xi | x1, . . . , xi−1)}

(2)

where x = {x1, x2, . . . , xn} is a sequence of tokens and pθ(xi | x1, . . . , xi−1) is the conditional
probability of xi given the preceding tokens.

Instead of using the likelihood of all tokens, Min-k% (Shi et al., 2024) computes the average prob-
abilities of k% outlier tokens with the smallest predicted probability. The intuition is that a non-
member example is more likely to include a few outlier words with low likelihoods than members.
Formally, Min-k% is computed by:

Min-k%(x; fθ) =

1
E

(cid:88)

xi∈Min-k%(x)

log pθ(xi | x1, . . . , xi−1)

(3)

where E is the size of the Min-k%(x) set.

However, non-member data can obtain low perplexities by including frequent or repetitive texts,
while members may contain rare tokens that result in high perplexities (See Figure 3a and 3b). This
issue makes it challenging to distinguish members and non-members using those scoring functions,
leading to suboptimal performance in detecting pre-training data. Thus, we present a preliminary
attempt to utilize extra non-member data to enlarge the gap between members and non-members.

1https://platform.openai.com

3

Under review as a conference paper at ICLR 2025

3 METHOD

Recall the realistic assumption that detectors can obtain a few non-members that belong to the same
domain as the given sample, we aim to explore how to utilize these extra non-members to improve
the detection. Thus, we start by investigating the effects of LLM fine-tuning with unseen examples.
Our analysis shows that fine-tuning exerts different impacts on members and non-members.

3.1 MOTIVATION

In the analysis, we conduct experiments with WikiMIA (Shi et al., 2024), an evaluation benchmark
that uses events added to Wikipedia after specific dates as non-member data. We use (cid:101)D to denote the
non-member dataset that is accessible for detectors. To construct the dataset (cid:101)D, we randomly sample
a subset with 100 examples from the non-member data of WikiMIA. In addition, we construct the
test set with 630 examples each for both members and non-members. Throughout this subsection,
we fine-tune LLaMA-7B (Touvron et al., 2023a) with LoRA (Hu et al., 2022) on the non-member
dataset (cid:101)D. To illustrate the effects of fine-tuning, we compare the perplexity distribution of members
and non-members from the pre-trained model and the fine-tuned model.

(a) Members

(b) Non-members

Figure 2: The perplexity distribution from the pre-trained model and the fine-tuned model.

Fine-tuning decreases the perplexity of non-members. Figures 2a and 2b present the deviation
of perplexity distributions for members and non-members, throughout the fine-tuning on the non-
member dataset (cid:101)D. The results show that unseen data in the pretraining tends to obtain a lower
perplexity from the fine-tuned model compared to the pre-trained model. Contrarily, we observe that
the deviation of perplexity distribution for members is negligible after the pre-trained model fine-
tuning. The analysis indicates that fine-tuning with a few unseen data from a specific domain can
decrease the likelihood-based scores of the LLM for other unknown non-members in the domain.
The contrast in the score deviation resulting from fine-tuning allows for the distinction between
members and non-members.

3.2 FSD: FINE-TUNED SCORE DEVIATION

Motivated by the previous analysis, we propose Fine-tuned Score Deviation (FSD), a general method
that can improve the detection performance of current scoring functions in a specific domain. The
key idea of our method is to enlarge the gap between seen and unseen data, by exposing the LLM to
a few unseen data. With this in mind, we present the details of our approach step by step.

Construct fine-tuning dataset Given a piece of text x, the first step of our method is to collect
a small amount of unseen data for the LLM within the same domain. Owing to the availability of
public text data in enormous quantities, we can construct non-member datasets by comparing the
LLM release date and data creation timestamp. For instance, we collect some events occurring post-
2023 from Wikipedia as the auxiliary non-member dataset for fine-tuning LLaMA (Touvron et al.,
2023a), since LLaMA was released in February 2023.

4

162
163

164
165
166
167
168
169

170
171
172
173
174
175

176
177
178
179
180

181
182
183
184
185
186

187
188
189
190
191
192

193
194
195
196
197
198

199
200
201
202
203

204
205
206
207
208
209

210
211
212
213
214
215

02468101214Perplexity0.000.050.100.150.200.250.30Densitypre-trainedfine-tuned02468101214Perplexity0.000.050.100.150.200.250.30Densitypre-trainedfine-tunedUnder review as a conference paper at ICLR 2025

216
217

218
219
220
221
222
223

224
225
226
227
228
229

230
231
232
233
234

235
236
237
238
239
240

241
242
243
244
245
246

247
248
249
250
251
252

253
254
255
256
257

258
259
260
261
262
263

264
265
266
267
268
269

(a) Perplexity

(b) Min-k%

(c) FSD with Perplexity

(d) FSD with Min-k%

Figure 3: Distribution of scores from pre-trained model vs. FSD. We contrast the distribution of
scores from the pre-trained model using perplexity and our FSD with perplexity(a & c). Similarly,
we contrast the Min-k% scores distribution from the pre-trained model and our FSD (b & d). Using
FSD leads to enlarging the gap between members and non-members.

Fine-tuning with non-members To expose LLMs to unseen data, we perform fine-tuning on
LLMs with the constructed fine-tuning dataset. As our goal is to reduce the perplexity of the un-
seen data, we employ self-supervised fine-tuning by predicting the next word or token in a given
sequence. In particular, we build the loss function by decreasing the negative log likelihood of the
actual next token in the sequence. Formally, the fine-tuning loss is:

Lfine-tuning(x) = −

1
n

n
(cid:88)

i=1

log fθ(xi|x1, ..., xi−1)

(4)

Fine-tuned Score Deviation Recall that fine-tuning decreases the perplexity of non-members but
almost maintains those of members, we propose to exploit the score deviation for detecting pretrain-
ing data. Given a new sample x, we calculate the score difference between the pre-trained LLM fθ
and the fine-tuned LLM fθ′, where θ′ denotes the parameters of LLM after fine-tuning. Formally,
the new score of Fine-tuned Score Deviation (FSD) can be formulated as:

FSD(x; fθ, fθ′ ) = S(x; fθ) − S(x; fθ′ )

(5)

where S(·) denotes an existing scoring function, such as Perplexity and Min-k%. With the proposed
score, we can estimate the membership of x through the level-set estimation (Eq. (1)). Examples
with a large deviation score are considered as non-members and vice versa. In practice, we deter-
mine the threshold ϵ by maximizing detection accuracy on a validation set, following the previous
work (Shi et al., 2024). Our method is compatible with various scoring functions and consistently
enhances their performance in detecting pretraining data, as presented in Table 1.

By way of the FSD score, we can obtain a clear distinction between members and non-members,
establishing excellent performance in detecting pretraining data. To provide a straightforward view,
we show in Figure 3 the score distribution between members and non-members using various scoring
functions on WikiMIA (Shi et al., 2024). The results of ArXivTection (Duarte et al., 2024) are also
presented in Appendix D.1. Our experiments validate that, compared to the perplexity and Min-k%

5

024681012141618Perplexity0.000.050.100.150.200.250.30Densitynon-membermember0123456789Min-k%0.00.10.20.30.40.5Densitynon-membermember10505101520Perplexity0.000.050.100.150.200.250.30Densitynon-membermember4202468Min-k%0.00.10.20.30.40.50.60.7Densitynon-membermemberUnder review as a conference paper at ICLR 2025

270
271

272
273
274
275
276
277

278
279
280
281
282
283

284
285
286
287
288

289
290
291
292
293
294

295
296
297
298
299
300

301
302
303
304
305
306

307
308
309
310
311

312
313
314
315
316
317

318
319
320
321
322
323

scores, our FSD score significantly increases the gap between non-members and members, and as a
result, enables more effective pretraining data detection.

4 EXPERIMENTS

In this section, we evaluate the effectiveness of our method for pretraining data detection across
several benchmark datasets with multiple existing open-sourced models. We also apply FSD to
copyrighted book detection in real-world scenarios and find it a consistently effective solution.

4.1 EXPERIMENTAL SETUP

Models We conduct extensive experiments on diverse open-sourced LLMs. For the main results,
we use LLaMA-7B (Touvron et al., 2023a) as the LLM throughout our experiments. We also provide
experiments on other models including Pythia-6.9B (Biderman et al., 2023), GPT-J-6B (Wang &
Komatsuzaki, 2021), OPT-6.7B (Zhang et al., 2022), LLaMA-13B models (Touvron et al., 2023a),
LLaMA-30B (Touvron et al., 2023a), and NeoX-20B (Black et al., 2022). Existing works (Shi et al.,
2024; Ye et al., 2024) generally use these models as LLMs for performing the studies of pretraining
data detection. The models are provided by Hugging Face 2.

Datasets To verify the effectiveness of detection methods, we employ four common benchmark
datasets for evaluations, including WikiMIA (Shi et al., 2024), ArXivTection (Duarte et al., 2024),
BookTection (Duarte et al., 2024) and BookMIA (Shi et al., 2024). Previous works have demon-
strated that model developers commonly use text content among those datasets for pre-training (Shi
et al., 2024; Duarte et al., 2024; Ye et al., 2024). The datasets are provided by Hugging Face3, and
detailed information of datasets is presented in Appendix B.

Baseline methods We use four detection methods based on scoring functions as our baselines for
evaluating the performance of methods on various datasets under diverse models. Those methods
employ specific metrics based on the likelihood, followed by a comparison with a preset thresh-
old to identify the given text’s membership. Specifically, baseline methods include the example
perplexity (Perplexity) (Li, 2023), the ratio of example perplexity and zlib compression entropy
(Zlib) (Carlini et al., 2021), the ratio of the perplexity on the example before and after lowercasing
(Lowercase) (Carlini et al., 2021) and detecting pretraining example through outlier words with low
probability (Min-k%) (Shi et al., 2024).

Evaluation metrics We evaluate the performance of detection methods for detecting pretraining
data by measuring the following metrics: (1) AUC, the area under the receiver operating character-
istic curve; (2) the true positive rate (TPR) when the false positive rate (FPR) of examples is 5%
(TPR@5%FPR).

Implementation details Our approach involves constructing the non-member dataset and fine-
tuning the base model. For constructing the non-member dataset, we randomly sample 30% of the
data from the entire dataset and select all non-members from this subset as the constructed fine-
tuning dataset. The remaining 70% of the dataset is used for testing. We employ LoRA (Hu et al.,
2022) to fine-tune the base model with 3 epochs and a batch size of 8. We set the initial learning
rate as 0.001 and drop it by cosine scheduling strategy. We conduct all experiments on NVIDIA L40
GPU and implement all methods with default parameters using PyTorch (Paszke et al., 2019).

4.2 MAIN RESULTS

Can FSD improve the performance of current scoring functions? We compare the perfor-
mance of detection methods on WikiMIA and ArXivTection datasets across various large language
models. The detailed information of dataset split is shown in Appendix C.1. Our results in Table 1
show that the FSD significantly improves the performance of all baseline methods on both datasets

2https://huggingface.co/models
3https://huggingface.co/datasets

6

Under review as a conference paper at ICLR 2025

324
325

326
327
328
329
330
331

332
333
334
335
336
337

338
339
340
341
342

343
344
345
346
347
348

349
350
351
352
353
354

355
356
357
358
359
360

361
362
363
364
365

366
367
368
369
370
371

372
373
374
375
376
377

Table 1: AUC score for pretraining data detection with baselines and our method from various
models on WikiMIA and ArXivTection. Base and +Ours respectively denote the baseline methods
and our method. Bold shows the superior result.

Dataset

Method

GPT-J-6B

OPT-6.7B Pythia-6.9B LLaMA-7B NeoX-20B

Base +Ours Base +Ours Base +Ours Base +Ours Base +Ours

WikiMIA

ArXivTection

Perplexity 0.64
Lowercase 0.59
0.61
MIN-K% 0.68

Zlib

Perplexity 0.79
Lowercase 0.59
0.64
MIN-K% 0.85

Zlib

0.95
0.77
0.94
0.92

0.96
0.81
0.96
0.92

0.60
0.59
0.59
0.62

0.68
0.58
0.55
0.74

0.90
0.71
0.89
0.91

0.89
0.70
0.89
0.84

0.64
0.58
0.61
0.67

0.77
0.60
0.63
0.84

0.90
0.74
0.88
0.86

0.95
0.77
0.95
0.91

0.64
0.58
0.62
0.65

0.68
0.50
0.57
0.76

0.92
0.69
0.90
0.85

0.92
0.69
0.91
0.86

0.69
0.66
0.64
0.73

0.79
0.62
0.65
0.85

0.93
0.76
0.93
0.90

0.95
0.75
0.95
0.91

across diverse models. For example, our method improves the AUC score compared to the best base-
line method Min-k%, increasing it from 0.62 to 0.91 on WikiMIA dataset from the OPT-6.7B model.
Similarly, it improves the AUC score from 0.76 to 0.86 on ArXivTection dataset from the LLaMA-
7B model. Moreover, we show that our method also remarkably improves the TPR@5%FPR score
of all baseline methods in Table 11 of Appendix D.2. Moreover, we also present the results on
different subsets of the Pile dataset under the Pythia-6.9B model in Appendix D.3.

How does the fine-tuning data size affect the performance of FSD? To investigate the effect
of varying the fine-tuning data size on the pretraining data detection, we compare the performance
of the model fine-tuned with different-sized fine-tuned datasets. To construct fine-tuning datasets of
varying sizes, we randomly sample varying amounts of non-members (0, 30, 50, 100, 150, 200, 250,
300) from the WikiMIA dataset as fine-tuning datasets. In addition, we sample 465 members and
non-members from the dataset and construct a balanced test set of 930 examples.

Figure 6 presents the performance of FSD with various sizes of auxiliary datasets. The results show
our method achieves better performance as the size of the fine-tuning dataset increases. Notably,
our method is highly data-efficient, achieving dramatic improvements with only a small amount of
non-members for fine-tuning. For example, FSD improves the AUC score of the perplexity-based
method from 0.63 to 0.91, by leveraging only 100 non-member data for fine-tuning – a significant
direct improvement of 44%. In summary, a few non-members are sufficient for FSD to improve the
detection, demonstrating its practicality. In addition, we also evaluate our method on the BookC2
subset of the Pile dataset under the Pythia-6.9B model. The results show a similar trend, which can
be found in Appendix D.3.

Is FSD effective with different-sized models? We also verify the performance of baselines and
our methods from different-sized LLaMA models (7B, 13B, 30B) on WikiMIA. In Table 2, our re-
sults demonstrate that our method is effective with different-sized models, and achieves remarkable
performance from different-sized models. Notably, the AUC score of Lowercase slightly rises as
the parameters of the LLaMA model increase. Moreover, additional results of the TPR@5%FPR
score show a similar trend, which can be found in Appendix D.2.

Can our method detect copyrighted books in pretraining data? Recent works (Shi et al., 2024;
Duarte et al., 2024) study the problem of copyrighted book detection in training data. Following
previous works, we verify the effectiveness of detection methods for detecting excerpts from copy-
righted books on BookMIA (Shi et al., 2024) and BookTection (Duarte et al., 2024). Specifically, we
randomly sample 500 members and non-members from datasets, constructing a balanced validation
set of 1,000 examples. The detailed information of datasets split is presented in Appendix C.2.

In Table 3, we compare the accuracy of our method and baselines for detecting suspicious books in
pretraining data from the LLaMA-7B model. A salient observation is that our method significantly

7

Under review as a conference paper at ICLR 2025

Table 2: AUC score for pretraining data detection with baselines and our method from the different-
sized LLaMA model on WikiMIA. Base and +Ours respectively denote the baseline methods and
our method. Bold shows the superior result.

Method

Perplexity
Lowercase
Zlib
MIN-K%

LLaMA-7B

LLaMA-13B

LLaMA-30B

Base

0.64
0.58
0.62
0.65

+Ours

0.92
0.69
0.90
0.85

Base

0.66
0.60
0.63
0.67

+Ours

0.92
0.70
0.90
0.86

Base

0.68
0.60
0.65
0.70

+Ours

0.91
0.75
0.91
0.82

(a) AUC

(b) TPR@5%FPR

Figure 4: AUC and TPR@5%FPR of scoring functions with FSD, using auxiliary datasets with
varying sizes. Notably, ⋆ represents the baseline without FSD.

Table 3: Accuracy and AUC score for copyrighted book detection with baselines and our method
from LLaMA-7B on BookTection and BookMIA. Base and +Ours respectively denote baslines and
our method. Bold shows the superior result.

Metric

Method

Accuracy

AUC

BookTection

BookMIA

BookTection

BookMIA

Base

+Ours

Base

+Ours

Base

+Ours

Base

+Ours

Perplexity
Lowercase
Zlib
MIN-K%

66.9
64.5
65.3
68.1

85.4
73.0
86.4
82.1

59.0
67.0
57.4
59.5

96.5
69.2
98.6
93.9

0.710
0.664
0.568
0.716

0.910
0.770
0.920
0.880

0.564
0.708
0.474
0.587

0.995
0.779
0.999
0.979

improves the accuracy of baseline methods for copyrighted book detection. For example, compared
with baselines, our method reaches an accuracy of 98.6% on BookMIA using detection method Zlib,
which marks a significant 71.8% improvement. We also present the AUC score with our method and
baselines in Table 3, which consistently improves the detection capabilities of baseline methods. Our
extensive experiments demonstrate the effectiveness of our method for copyrighted book detection.

5 DISCUSSION

Can members be used for fine-tuning? The key step of our method is to fine-tune the pre-trained
model using a few non-members. One may also ask: can a similar effect be achieved by utilizing
members as the fine-tuning dataset? In this ablation, we separately sample members and non-
members from WikiMIA to construct fine-tuning datasets(Mem, Non). In addition, we randomly
sample data from WikiMIA as another fine-tuning dataset (All). The details of implementation are

8

378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422
423
424
425

426
427
428
429
430
431

050100150200250300Sample size0.50.60.70.80.91.0AUC050100150200250300Sample size0.00.10.20.30.40.50.60.7TPR@5%FPRPerplexityLowercaseZlibMin-k%Under review as a conference paper at ICLR 2025

432
433

434
435
436
437
438
439

440
441
442
443
444
445

446
447
448
449
450

451
452
453
454
455
456

457
458
459
460
461
462

463
464
465
466
467
468

469
470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

Table 4: AUC of scoring functions with FSD,
using members (Mem), non-members (Non),
and mix of them (All) on LLaMA-7B. Base
denotes the scoring function without FSD.
Bold shows the superior result.

Table 5: AUC of scoring functions with FSD
using the original WikiMIA, data removing
timestamps (Deletion), and data replacing the
year of timestamps with 2023 (Replacement).
The results are shown as Base/+Ours.

Method

Base

All Mem Non

Method

WikiMIA Deletion Replacement

Perplexity
Lowercase
Zlib
MIN-K%

0.64
0.58
0.62
0.65

0.68
0.54
0.65
0.61

0.78
0.67
0.79
0.81

0.92
0.69
0.90
0.85

Perplexity 0.64/ 0.92 0.62/ 0.76
Lowercase 0.58/ 0.69 0.58/ 0.62
0.62/ 0.90 0.58/ 0.72
Zlib
MIN-K% 0.65/0.85 0.61/ 0.69

0.54/ 0.71
0.52/ 0.63
0.55/ 0.68
0.54/ 0.67

presented in Appendix C.3. To investigate the impact of different fine-tuning datasets on pretraining
data detection, we fine-tune the LLaMA-7B model with each of the fine-tuning datasets individually.

Our results in Table 4 show that our method can improve the performance of baseline methods using
members as the fine-tuning dataset. However, our method achieves inferior performance when using
members for fine-tuning compared with non-members. Moreover, it is not realistic to construct
a member dataset without accessing pretraining data in real-world scenarios. In addition, this is
feasible for constructing non-members as a fine-tuning dataset based on the model release date and
data creation timestamp. Overall, our method demonstrates superior performance when using non-
members for fine-tuning, while ensuring applicability in real-world settings.

Is our method affected by distribution difference? Existing works generally construct bench-
mark datasets based on the LLM release date and data creation timestamp (Ye et al., 2024; Shi et al.,
2024). For example, the WikiMIA dataset considers events occurring post-2023 as non-members.
Recent works indicate evaluation results are suspect on benchmark datasets because they possibly
sample members and non-members from different distributions (Duan et al., 2024; Das et al., 2024;
Maini et al., 2024). We find the temporal shift between members and non-members in the WikiMIA
dataset, which is shown in Appendix C.4. The issue shows that we can distinguish members and non-
members with timestamps in the dataset. To eliminate the impact of temporal differences between
members and non-members on evaluation, we implement two strategies to mitigate the temporal
shift in the dataset: (1) removing timestamps in the dataset (Deletion), and (2) replacing the year of
timestamps with 2023 in the dataset(Replacement). We conduct experiments with baselines and our
method on the original WikiMIA dataset, Deletion and Replacement, respectively.

Our results in Table 5 show that the performance of baselines and our method are compromised
when mitigating the temporal shift between members and non-members. This is likely because
deleting or replacing a word will change the probability of the subsequent word, thereby perturb-
ing the likelihood-based metric. In this setting, surprisingly, the results show that our method can
also improve the performance of baseline methods. Though baseline methods yield results that
are comparable to random guessing on the Replacement dataset, our method can also improve the
AUC scores of the perplexity-based detection method, increasing it from 0.54 to 0.71. Overall, our
method is effective even if there is no distribution difference between members and non-members.
The TPR@5%FPR score of the experiment is presented in Appendix D.2.

Is FSD effective with different fine-tuning methods? To expose LLMs to unseen data, we em-
ploy LoRA to fine-tune the pre-trained model. The results demonstrate that our method achieves
impressive performance for pretraining data detection when fine-tuning with LoRA. However,
can a similar effect be achieved using different fine-tuning methods? To this end, we apply
AdaLoRA (Zhang et al., 2023), IA3 (Liu et al., 2022), and LoRA to fine-tune LLaMA-7B with
WikiMIA, respectively. The details of the dataset in our experiment can be found in Appendix C.1.

In Table 6, we report the AUC and TPR@5%FPR scores for pretraining data detection with our
method and baseline methods. The results show that our method can improve the performance of
baseline methods when using different fine-tuning methods. Although our FSD achieves inferior
performance with AdaLoRA compared with IA3 and LoRA, it still significantly improves the per-
formance of baseline methods. Overall, our method can be implemented with different fine-tuning

9

Under review as a conference paper at ICLR 2025

Table 6: AUC score of FSD with different fine-tuning methods. Base denotes baseline methods
without model fine-tuning. Bold shows the superior result.

Metric

AUC

TPR@5%FPR

Method

Base

AdaLoRA

Perplexity
Lowercase
Zlib
MIN-K%

0.64
0.58
0.62
0.65

0.82
0.62
0.76
0.80

IA3

0.91
0.72
0.84
0.90

LoRA

Base

AdaLoRA

0.92
0.69
0.90
0.85

0.09
0.10
0.09
0.15

0.39
0.13
0.24
0.22

IA3

0.52
0.17
0.32
0.39

LoRA

0.41
0.18
0.47
0.25

methods and does not require a specific fine-tuning technique. In addition, we also conduct experi-
ments to explore the impact of different fine-tuning parameters on the performance of our method.
The results are presented in Appendix D.3.

6 CONCLUSION

In this paper, we introduce Fine-tuned Score Deviation (FSD), a novel detection method that can
universally improve the performance of existing detection methods. To the best of our knowledge,
our method is the first to utilize some collected non-members in the task of pretraining data detection.
Our core idea behind FSD is to enlarge the gap between seen examples and unseen examples by
exposing the LLM to a few unseen examples. In effect, unseen data have a larger score than seen
examples when using FSD, which makes it more distinguishable between seen and unseen data.
Extensive experiments demonstrate the effectiveness of our method for detecting pretraining data on
common benchmark datasets across various models. In summary, the FSD is an effective approach
for accurately detecting pretraining data of LLMs.

Limitations Our method requires to collect a few examples that belong to the same domain but are
not involved in the training. Generally, we can utilize the data content published after the release of
the LLM. Therefore, our method is applicable for detecting benchmarks or copyrighted resources in
a specific domain (e.g., math tests, magazines). The diversity of the test set may make it challenging
to construct an effective auxiliary dataset of unseen data. In addition, our method requires fine-
tuning on a few non-member data, so the effectiveness of the proposed score might be affected by
the data quality of non-members.

Ethical Statement Our work focuses on pretraining data detection from large language models.
The proposed methodology aims to address issues involving data contamination or copyright in-
fringement. In addition, our method can be used to identify privacy potential privacy leakage risks
and ensure the safety of LLMs, aligning with established ethical standards for content moderation.
Regarding data access, the evaluated datasets we employed in our work come from prior research
and do not involve personal privacy information.

10

486
487

488
489
490
491
492
493

494
495
496
497
498
499

500
501
502
503
504

505
506
507
508
509
510

511
512
513
514
515
516

517
518
519
520
521
522

523
524
525
526
527

528
529
530
531
532
533

534
535
536
537
538
539

Under review as a conference paper at ICLR 2025

540
541

542
543
544
545
546
547

548
549
550
551
552
553

554
555
556
557
558

559
560
561
562
563
564

565
566
567
568
569
570

571
572
573
574
575
576

577
578
579
580
581

582
583
584
585
586
587

588
589
590
591
592
593

REFERENCES

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale-
man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical
report. arXiv preprint arXiv:2303.08774, 2023.

Simone Balloccu, Patr´ıcia Schmidtov´a, Mateusz Lango, and Ondˇrej Duˇsek. Leak, cheat, repeat:
Data contamination and evaluation malpractices in closed-source llms. In Proceedings of the 18th
Conference of the European Chapter of the Association for Computational Linguistics (Volume
1: Long Papers), pp. 67–93, 2024.

Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric
Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.
Pythia: a suite for analyzing large language models across training and scaling. In International
Conference on Machine Learning, pp. 2397–2430. PMLR, 2023.

Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho-
race He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source
autoregressive language model. In Proceedings of BigScience Episode# 5–Workshop on Chal-
lenges & Perspectives in Creating Large Language Models, pp. 95–136, 2022.

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are
few-shot learners. In Proceedings of the 34th International Conference on Neural Information
Processing Systems, pp. 1877–1901, 2020.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine
Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data
from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp.
2633–2650, 2021.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam
Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm:
scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–
113, 2023.

Debeshee Das, Jie Zhang, and Florian Tram`er. Blind baselines beat membership inference attacks

for foundation models. arXiv preprint arXiv:2406.16201, 2024.

Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A Choquette-Choo, Matthew
Jagielski, Milad Nasr, Eric Wallace, and Florian Tram`er. Privacy side channels in machine learn-
ing systems. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 6861–6848, 2024.

Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer,
Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference
attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024.

Andr´e Vicente Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li. De-cop: Detecting copy-
righted content in language models training data. In Forty-first International Conference on Ma-
chine Learning, 2024.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason
Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: an 800gb dataset of diverse
text for language modeling. arXiv preprint arXiv:2101.00027, 2020.

Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large
language models. In The Twelfth International Conference on Learning Representations, 2024.

Michael M Grynbaum and Ryan Mac. The times sues openai and microsoft over ai use of copy-

righted work. The New York Times, 27, 2023.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen,
In International Conference on

et al. Lora: Low-rank adaptation of large language models.
Learning Representations, 2022.

11

Under review as a conference paper at ICLR 2025

594
595

596
597
598
599
600
601

602
603
604
605
606
607

608
609
610
611
612

613
614
615
616
617
618

619
620
621
622
623
624

625
626
627
628
629
630

631
632
633
634
635

636
637
638
639
640
641

642
643
644
645
646
647

Yucheng Li. Estimating contamination via perplexity: quantifying memorisation in language model

evaluation. arXiv preprint arXiv:2309.10677, 2023.

Gaoyang Liu, Chen Wang, Kai Peng, Haojun Huang, Yutong Li, and Wenqing Cheng. Socinf: mem-
bership inference attacks on social media health data with machine learning. IEEE Transactions
on Computational Social Systems, (5):907–921, 2019.

Gaoyang Liu, Tianlong Xu, Rui Zhang, Zixiong Wang, Chen Wang, and Ling Liu. Gradient-leaks:
enabling black-box membership inference attacks against machine learning models. IEEE Trans-
actions on Information Forensics and Security, 2023.

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and
Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context
learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022.

Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Pro-
ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2:
Short Papers), pp. 157–165, 2022.

Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: did you

train on my dataset? arXiv preprint arXiv:2406.06443, 2024.

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan,
and Taylor Berg-Kirkpatrick. Membership inference attacks against language models via neigh-
bourhood comparison. In Findings of the Association for Computational Linguistics: ACL 2023,
pp. 11330–11343, 2023.

Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. Use of llms for illicit
purposes: Threats, prevention measures, and vulnerabilities. arXiv preprint arXiv:2308.12833,
2023.

Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving
test set contamination in black-box language models. In The Twelfth International Conference on
Learning Representations, 2024.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor
Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style,
high-performance deep learning library. Advances in Neural Information Processing Systems, 32,
2019.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language

models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.

Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei
Qin, Caiming Xiong, and Shafiq Joty. How much are llms contaminated? a comprehensive survey
and the llmsanitize library. arXiv preprint arXiv:2404.00699, 2024.

Oscar Sainz, Jon Campos, Iker Garc´ıa-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko
Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each
In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
benchmark.
10776–10787, 2023.

Ahmed Salem, Yang Zhang, Mathias Humbert, Mario Fritz, and Michael Backes. Ml-leaks: Model
and data independent membership inference attacks and defenses on machine learning models. In
Network and Distributed Systems Security Symposium 2019. Internet Society, 2019.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi
In The

Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models.
Twelfth International Conference on Learning Representations, 2024.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at-
tacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP),
pp. 3–18. IEEE Computer Society, 2017.

12

Under review as a conference paper at ICLR 2025

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu,
Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly
capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee
Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: open and
efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko-
lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: open foundation
and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.

Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Demystifying membership
inference attacks in machine learning as a service. IEEE transactions on services computing, 14
(6):2073–2089, 2019.

Ben Wang and Aran Komatsuzaki. GPT-J-6B: a 6 billion parameter autoregressive language model.

https://github.com/kingoflolz/mesh-transformer-jax, May 2021.

Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong,
and Bhuwan Dhingra. Recall: Membership inference via relative conditional log-likelihoods.
arXiv preprint arXiv:2406.15968, 2024.

Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. Benchmark data contamination of large

language models: A survey. arXiv preprint arXiv:2406.04244, 2024a.

Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large

language models. arXiv preprint arXiv:2404.18824, 2024b.

Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, and Junbo Zhao. Data contamination

calibration for black-box llms. arXiv preprint arXiv:2405.11930, 2024.

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn-
ing: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations
Symposium (CSF), pp. 268–282. IEEE Computer Society, 2018.

Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang,
and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language
models. arXiv preprint arXiv:2404.02936, 2024.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He,
Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: adaptive budget allocation for parameter-
efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo-
pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: open pre-trained transformer
language models. arXiv preprint arXiv:2205.01068, 2022.

Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin,
Ji-Rong Wen, and Jiawei Han. Don’t make your llm an evaluation benchmark cheater. arXiv
preprint arXiv:2311.01964, 2023.

13

648
649

650
651
652
653
654
655

656
657
658
659
660
661

662
663
664
665
666

667
668
669
670
671
672

673
674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689

690
691
692
693
694
695

696
697
698
699
700
701

Under review as a conference paper at ICLR 2025

702
703

704
705
706
707
708
709

710
711
712
713
714
715

716
717
718
719
720

721
722
723
724
725
726

727
728
729
730
731
732

733
734
735
736
737
738

739
740
741
742
743

744
745
746
747
748
749

750
751
752
753
754
755

A RELATED WORK

Pretraining data detection, which is an increasingly important topic for large language models, re-
lates to a large amount of literature on membership inference attacks and data contamination. We
discuss some of the relevant works to ours in two directions below.

Membership Inference Attacks Our work mainly studies how to detect a given example in
the pretraining data, which is consistent with the objective of membership inference attacks
(MIAs) (Shokri et al., 2017; Truex et al., 2019). This task aims to determine whether a given data
point is a member of training data. Metric-based attack methods, such as loss (Yeom et al., 2018),
entropy (Salem et al., 2019), confidence (Liu et al., 2019) and gradient (Liu et al., 2023), infer mem-
bership of data by comparing the calculated metric value with a preset threshold. Previous works
have generalized metric-based methods to large language models (Duan et al., 2024; Xie et al., 2024;
Zhang et al., 2024; Mattern et al., 2023), by calculating the based-likelihood metric (e.g., perplex-
ity) for membership inference. Recent works apply MIAs to pretraining data detection by designing
likelihood-based scoring functions to measure the membership of data (Shi et al., 2024; Ye et al.,
2024). In this work, we analyze the limitations of existing scoring functions for pretraining data
detection, and design an effective method to improve their performance. In particular, this work is
the first to explore the importance of collecting unseen data in pretraining data detection.

Data Contamination Data contamination has been studied in the literature (Xu et al., 2024a;
Magar & Schwartz, 2022; Balloccu et al., 2024), where training data may inadvertently include
evaluation benchmark data, resulting in unauthentic evaluation results. Thus, it is important to assess
the leakage of benchmark data into pretraining data (Zhou et al., 2023). On the one hand, model
developers can remove evaluation benchmark data from training data by retrieval-based methods
with access to pertaining data (Ravaut et al., 2024; Chowdhery et al., 2023). Specifically, those
methods employ n-gram tokenization and string-matching for detecting data contamination (Brown
et al., 2020; Touvron et al., 2023b; Team et al., 2023; Radford et al., 2019). On the other hand,
researchers utilize prompting techniques (Golchin & Surdeanu, 2024), performance analysis (Ye
et al., 2024; Debenedetti et al., 2024), model likelihood (Oren et al., 2024; Shi et al., 2024; Xu et al.,
2024b) to detect potential contamination without access to the training data. Our work focuses
on pretraining data detection, an area that is similar to data contamination. Different from data
contamination detection, our FSD can also be applied to the detection of copyrighted resources in
real-world scenarios.

B DETAILS OF DATASETS

Previous works construct benchmark datasets to evaluate the performance of detection methods for
pretraining data detection. Following the prior literature, we conduct experiments on 4 benchmark
datasets: WikiMIA (Shi et al., 2024) selects old Wikipedia event data as member data by leveraging
the Wikipedia data timestamp and the model release date, since Wikipedia is a commonly pretraining
data source. BookMIA (Shi et al., 2024), which contains excerpts from copyrighted books in the
Books3 subset of the Pile dataset (Gao et al., 2020), can be used for detecting potential copyright
infringement in training data. ArXivTection (Duarte et al., 2024) is a curated collection of research
articles sourced from arXiv. BookTection (Duarte et al., 2024), which comprises passages from 165
books, is constructed based on BookMIA.

C EXPERIMENTAL DETAIL

C.1 DATASET SPLIT

We report the performance of detection methods on WikiMIA and ArXivTection datasets across
various large language models. To construct the fine-tuning dataset, we randomly split 30% of the
dataset using a seed of 42 and select all non-members from this subset. The remaining 70% of the
dataset is used for testing. The detailed information of the constructed dataset is shown in Table 7.

14

Under review as a conference paper at ICLR 2025

756
757

758
759
760
761
762
763

764
765
766
767
768
769

770
771
772
773
774

775
776
777
778
779
780

781
782
783
784
785
786

787
788
789
790
791
792

793
794
795
796
797

798
799
800
801
802
803

804
805
806
807
808
809

Table 7: The train set and test set used in the experiment

Dataset

WikiMIA

ArXivTection

Type
Train Set
Test Set

Train Set
Test Set

Member
\
599

\
536

Non-member
231
558

238
549

Total
231
1,157

238
1,085

C.2 COPYRIGHTED BOOK DETECTION

To conduct experiments of copyrighted book detection on BookMIA and BookTection, we ran-
domly split 30% of the dataset using a seed of 42 and select all non-members from this subset as
the fine-tuning dataset. Subsequently, we randomly sample 500 members and non-members from
the remaining 70% of the datasets, constructing a balanced validation set of 1,000 examples. The
remaining data is used for testing. The detailed information dataset split is shown in Table 8.

Table 8: The train set, test set and validation set used in the experiment

Dataset

BookMIA

BookTection

Type
Train Set
Test Set
Validation set

Train Set
Test Set
Validation set

Member
\
2,887
500

\
6,833
500

Non-member
1,413
3,022
500

1,796
3,657
500

Total
1,413
5,909
1,000

1,796
10,490
1,000

C.3 FINE-TUNING WITH MEMBERS

To investigate the impact of model fine-tuning with different fine-tuning datasets on pretraining data
detection, we construct three kinds of fine-tuning datasets. In this ablation, we sample members
(Mem) and non-members (Non) from WikiMIA as fine-tuning datasets, respectively. In addition,
we randomly sample data from WikiMIA to construct a fine-tuning dataset (All). The details of
fine-tuning datasets are shown in Table 9

Table 9: The train set and test set used in the experiment

Datasets

Mem

Non

All

Type
Train Set
Test Set

Train Set
Test Set

Train Set
Test Set

Member
262
599

Non-member
\
558

\
599

262
536

231
558

231
549

Total
262
1,157

231
1,085

493
1,085

C.4 TEMPORAL SHIFT

We show the temporal shift between members and non-members in the WikiMIA dataset, the illus-
tration is presented in Table 10.

15

Under review as a conference paper at ICLR 2025

810
811

812
813
814
815
816
817

818
819
820
821
822
823

824
825
826
827
828

829
830
831
832
833
834

835
836
837
838
839
840

841
842
843
844
845
846

847
848
849
850
851

852
853
854
855
856
857

858
859
860
861
862
863

Table 10: An illustration of temporal shift between the member and non-member distributions.

Members

Non-Members

The 2014 On 30 June or 2 July 2014, the
Armed Forces of the Democratic Repub-
lic of the Congo and United Nations forces
launched an offensive against rebel groups
in the Masisi and Walikale.

The 95th Academy Awards was a cere-
mony held by the Academy of Motion Pic-
ture Arts and Sciences (AMPAS) on March
12, 2023, at the Dolby Theatre in Los An-
geles.

In 2014, a series of groundbreaking diplo-
matic meetings was held between Wang
Yu-chi, in his official capacity as the Minis-
ter of the Mainland Affairs Council (MAC)
of the Republic of China (ROC).

The 36th Annual Nickelodeon Kids’
Choice Awards ceremony was held on
March 4, 2023, at
the Microsoft The-
ater in Los Angeles, California with Nate
Burleson and Charli D’Amelio.

Concluding observations on the second pe-
riodic report of the Holy See was a 2014
report issued by the Office of the United
Nations High Commissioner for Human
Rights.

The 2023 Summer Metro Manila Film Fes-
tival is an ongoing iteration of the annual
Summer Metro Manila Film Festival held
in Metro Manila and throughout the Philip-
pines.

The 2014 European Aquatics Champi-
onships took place from 13 to 24 August
2014 in Berlin, Germany.

On February 11, 2023, an octagonal
unidentified flying object was detected
over northern Montana.

The centenary of the outbreak of World
War I was commemorated in Europe in late
July and early August 2014.

The 2023 Tokyo Marathon was the 16th
edition of the annual marathon race in
Tokyo, held on Sunday, 5 March 2023.

D DETAILED EXPERIMENTAL RESULTS

D.1 FINE-TUNED SCORE DEVIATION

We show in Figure 5 the score distribution between members and non-members using various scor-
ing functions on ArXivTection. The results also demonstrate that our FSD score significantly in-
creases the gap between non-members and members compared to the perplexity and Min-k% scores,
thus enabling more effective pretraining data detection.

D.2 DETAILED RESULTS OF EXPERIMENT

We report the TPR@5%FPR score for pertaining data detection in Table 11, 12, 13.

Can FSD improve the performance of detection methods based on scoring functions? We
compare the TPR@5%FPR score with our method and baselines on WikiMIA and ArXivTection
datasets across various large language models in Table 11. The results show that our method signif-
icantly improves the TPR@5%FPR score of the baseline methods.

Is FSD effective with different-sized models? We verify the performance of baselines and our
methods from different-sized LLaMA models (7B, 13B, 30B) on WikiMIA. In Table 12, we show
the TPR@5%FPR score from different-sized LLaMA models. The results demonstrate that our
method is effective with different-size models.

Is our method affected by distribution difference? We report the TPR@5%FPR score of base-
lines and our method on the original WikiMIA dataset, Deletion and Replacement. In Table 13,
the results show that our method still improves the performance of baselines when mitigating the
temporal shift between members and non-members.

16

Under review as a conference paper at ICLR 2025

864
865

866
867
868
869
870
871

872
873
874
875
876
877

878
879
880
881
882

883
884
885
886
887
888

889
890
891
892
893
894

895
896
897
898
899
900

901
902
903
904
905

906
907
908
909
910
911

912
913
914
915
916
917

(a) Perplexity

(b) Min-k%

(c) FSD with Perplexity

(d) FSD with Min-k%

Figure 5: Distribution of scores from pre-trained model vs. FSD. We contrast the score distribution
from the pre-trained model using perplexity and our FSD with perplexity(a & c). Similarly, we
contrast the Min-k% scores distribution from the pre-trained model and our FSD (b & d). Using
FSD leads to enlarging the gap between members and non-members.

Table 11: TPR@5%FPR score for pretraining data detection with baselines and our method from
various models on WikiMIA and ArXivTection. Base and +Ours respectively denote the baseline
methods and our method. Bold shows the superior result.

Dataset

Method

GPT-J-6B

OPT-6.7B Pythia-6.9B LLaMA-7B NeoX-20B

Base +Ours Base +Ours Base +Ours Base +Ours Base +Ours

WikiMIA

ArXivTection

Perplexity 0.12
Lowercase 0.12
0.09
MIN-K% 0.17

Zlib

Perplexity 0.26
Lowercase 0.13
0.15
MIN-K% 0.42

Zlib

0.78
0.24
0.78
0.40

0.79
0.23
0.80
0.57

0.12
0.07
0.09
0.14

0.12
0.15
0.07
0.24

0.63
0.18
0.55
0.50

0.63
0.22
0.60
0.45

0.13
0.11
0.10
0.17

0.25
0.15
0.14
0.41

0.66
0.25
0.50
0.35

0.66
0.25
0.50
0.35

0.09
0.10
0.09
0.15

0.10
0.09
0.08
0.24

0.41
0.18
0.47
0.25

0.81
0.16
0.66
0.45

0.20
0.16
0.10
0.25

0.27
0.13
0.16
0.40

0.58
0.18
0.57
0.36

0.77
0.20
0.77
0.58

D.3 ADDITIONAL RESULTS

The performance of our method on the Pile dataset We also conduct experiments on the Pile
dataset. Concretely, following prior work (Maini et al., 2024), we evaluate our methods on the
twenty subsets of the Pile dataset. Here, the validation set of the pile dataset was not trained on
the Pythia models (Biderman et al., 2023). Thus, we perform experiments on the Pythia-6.9B

17

0510152025303540Perplexity0.000.020.040.060.080.100.120.14Densitynon-membermember45678910Min-k%0.00.10.20.30.40.50.60.7Densitynon-membermember5.02.50.02.55.07.510.012.5Perplexity0.00.10.20.30.40.5Densitynon-membermember1.00.50.00.51.01.52.0Min-k%0.00.20.40.60.81.01.21.41.61.8Densitynon-membermemberUnder review as a conference paper at ICLR 2025

Table 12: TPR@5%FPR score for pretraining data detection with baselines and our method from
the different-sized LLaMA model on WikiMIA. Base and +Ours respectively denote the baselines
and our method. Bold shows the superior result.

LLaMA-7B

LLaMA-13B

LLaMA-30B

Method

Perplexity
Zlib
Lowercase
MIN-K%

Base

0.09
0.10
0.09
0.15

+Ours

0.41
0.18
0.47
0.25

Base

0.11
0.13
0.10
0.18

+Ours

0.61
0.13
0.56
0.26

Base

0.15
0.11
0.11
0.19

+Ours

0.40
0.25
0.44
0.20

Table 13: TPR@5%FPR score from the LLaMA-7B model with our method and baselines using the
original WikiMIA, data removing timestamps (Deletion), and data replacing the year of timestamps
with 2023 (Replacement). Base and +Ours denote the baseline methods and our method, respec-
tively. Bold shows the superior result.

Method

Perplexity
Lowercase
Zlib
MIN-K%

Origin

Deletion

Replacement

Base

0.09
0.10
0.09
0.15

+Ours

0.41
0.18
0.47
0.25

Base

0.13
0.06
0.12
0.10

+Ours

0.23
0.13
0.23
0.14

Base

0.04
0.03
0.09
0.04

+Ours

0.12
0.15
0.06
0.07

model, utilizing the training and validation sets as members and non-members, respectively. For
each dataset, we randomly sample a few non-members with a sample ratio of 0.3 from the validation
set for fine-tuning. Then, we evaluate our method on a balanced dataset composed of members and
non-members. Notably, in our experiments, there is no overlap between the fine-tuning dataset and
the evaluation data.

In Table 14, the results show that our method improves the performance of baselines on most subsets
of the Pile dataset under the Pythia-6.9B model. For example, our FSD improves the AUC score
of the perplexity-based method from 0.528 to 0.885, a significant direct improvement of 67%. At
the same time, our FSD improves the average AUC score of the perplexity-based method from
0.503 to 0.625 on the pile dataset, a notable direct improvement of 24.3%. This demonstrates the
effectiveness of our method in the IID setup.

Fine-tuning using non-members from different domains Our method requires a few non-
member data from a specific domain for fine-tuning. This raises a question: how does our method
perform when fine-tuned on non-member data from a different domain? To investigate the perfor-
mance of our method when fine-tuning using data from an unrelated domain. Firstly, we randomly
sample 231 and 238 non-members from the WikiMIA and ArXivTection datasets to construct a
fine-tuning dataset comprising a mix of domains. Then, we fine-tune the LLaMA-7B model on the
constructed dataset and evaluate our method on WikiMIA and ArXivTection datasets.

Our results in Table 15 show that our method can also significantly improve the performance of
baselines, indicating the effectiveness of our methods when fine-tuning with non-members from
a mix of domains. We also evaluate our methods on ArXivTection while fine-tuning using non-
members from WikiMIA. The results indicate that our method fails to improve the performance of
baselines, since the fine-tuning data comes from an entirely unrelated domain to the evaluation data.

18

918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939
940
941
942

943
944
945
946
947
948

949
950
951
952
953
954

955
956
957
958
959

960
961
962
963
964
965

966
967
968
969
970
971

Under review as a conference paper at ICLR 2025

972
973

974
975
976
977
978
979

980
981
982
983
984
985

986
987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007
1008

1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

Table 14: AUC score for pretraining data detection with baselines and our method from the Pythia-
6.9B on the Pile dataset. Base and +Ours respectively denote the baseline methods and our method.
Bold shows the superior result.

Method

Wiki

BookC2

Gutenberg

HackerNews

Enron

Base

+Ours

Base +Ours Base

+Ours

Base +Ours Base +Ours

Perplexity 0.471

0.614

0.528 0.885 0.528

Lowercase 0.466

0.626

0.518 0.725 0.546

Zlib

0.496

0.619

0.477 0.907 0.496

MIN-K% 0.512

0.611

0.510 0.841 0.536

0.661

0.551

0.686

0.612

0.471 0.565 0.510

0.678

0.450 0.512 0.484

0.659

0.474 0.550 0.560

0.667

0.498 0.535 0.570

0.646

Method

CC

arXiv

Europarl

FreeLaw

GitHub

Base

+Ours

Base +Ours Base

+Ours

Base +Ours Base +Ours

Perplexity 0.541

0.546

0.514 0.505 0.514

Lowercase 0.502

0.547

0.523 0.530 0.521

Zlib

0.529

0.576

0.540 0.505 0.462

MIN-K% 0.557

0.542

0.515 0.502 0.512

0.601

0.556

0.609

0.583

0.478 0.515 0.509

0.548

0.476 0.507 0.491

0.513

0.492 0.503 0.491

0.562

0.492 0.500 0.513

0.551

Method

Books3

Nih

OpenWebtext2

PhilPapers OpenSubtitles

Base

+Ours

Base +Ours Base

+Ours

Base +Ours Base +Ours

Perplexity 0.560

0.509

0.463 0.599 0.490

Lowercase 0.550

0.524

0.608 0.512 0.486

Zlib

0.550

0.581

0.416 0.599 0.475

MIN-K% 0.552

0.554

0.463 0.560 0.510

0.580

0.547

0.586

0.567

0.571 0.869 0.525

0.521

0.633 0.718 0.538

0.528

0.678 0.871 0.550

0.530

0.606 0.826 0.525

0.535

Method

StackExchange

Math

YoutubeSubtitles

USPTO

Ubuntu

Base

+Ours

Base +Ours Base

+Ours

Base +Ours Base +Ours

Perplexity 0.640

0.678

0.530 0.504 0.392

Lowercase 0.579

0.641

0.508 0.513 0.495

Zlib

0.595

0.686

0.513 0.502 0.445

MIN-K% 0.637

0.670

0.524 0.510 0.380

0.756

0.546

0.736

0.692

0.537 0.606 0.282

0.767

0.510 0.582 0.496

0.476

0.484 0.604 0.423

0.592

0.549 0.596 0.329

0.561

Ablation study on the BookC2 under varying the fine-tuning data size To investigate the effect
of varying the fine-tuning data size on the pretraining data detection, we also conduct experiments
on the BookC2 subset of the Pile dataset under the Pythia-6.9B model. Specifically, we randomly
sample varying amounts of non-members (0, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500) from
the validation set of the BookC2 as fine-tuning datasets. In addition, we sample 1400 members and
non-members from the train and validation sets of the BookC2 to construct a balanced test set of
2800 examples.

19

Under review as a conference paper at ICLR 2025

Table 15: AUC score for pretraining data detection with baselines and our method on WikiMIA and
ArXivTection under the LLaMA-7B. Wiki (Mix) denote evaluating on WikiMIA and fine-tuning
using data from a mix of domains. ArXiv (Wiki) denote evaluating on ArXivTection and fine-tuning
on WikiMIA. Base and +Ours respectively denote the baseline methods and our method.

Method

Perplexity
Lowercase
Zlib
MIN-K%

Wiki (Mix)

ArXiv (Mix)

ArXiv (Wiki)

Base

0.64
0.58
0.62
0.65

+Ours

0.91
0.73
0.91
0.84

Base

0.68
0.50
0.57
0.76

+Ours

0.93
0.73
0.92
0.87

Base

0.68
0.50
0.57
0.76

+Ours

0.52
0.50
0.64
0.61

Figure 6a shows that our method achieves better performance as the size of the fine-tuning dataset
increases. Notably, our method is highly data-efficient, achieving significant improvements with
only a few non-members for fine-tuning. For instance, our method improves the AUC score of the
Zlib method from 0.48 to 0.78, by leveraging only 100 non-member data for fine-tuning. In addition,
the results of the TPR@5%FPR score show a similar trend, which can be found in Figure 6b.

(a) AUC

(b) TPR@5%FPR

Figure 6: AUC and TPR@5%FPR of scoring functions with FSD, using auxiliary datasets with
varying sizes. Notably, ⋆ represents the baseline without FSD.

How do the fine-tuning parameters affect the performance of our method? To investigate the
impact of different fine-tuning parameters on the performance of our method, we conduct experi-
ments on the WikiMIA dataset with different fine-tuning parameters, including learning rate (e.g.
1e-3, 1e-4, 1e-5), epoch (e.g. 1, 2, 3) and LoRA rank (e.g. 8, 16, 32). In Table 16, the results
show that our method is relatively insensitive to LoRA rank and the number of fine-tuning epochs.
However, considering the learning rate parameter, a learning rate of 0.001 enables our method to
perform better.

Table 16: AUC score of baselines and our method on WikiMIA under the LLaMA-7B with different
fine-tuning parameters. Base and +Ours respectively denote the baseline methods and our method.
Bold shows the superior result.

Method

Learning Rate

LoRA Rank

Epoch

Base 10−5 10−4 10−3 Base

8

16

32

Base

1

2

3

0.64
Perplexity
0.58
Lowercase
0.62
Zlib
MIN-K% 0.65

0.81
0.60
0.73
0.76

0.84
0.64
0.78
0.81

0.92
0.69
0.90
0.85

0.64
0.58
0.62
0.65

0.92 0.92 0.92
0.69 0.68 0.69
0.91 0.90 0.90
0.87 0.85 0.86

0.64
0.58
0.62
0.65

0.91 0.91 0.92
0.65 0.64 0.69
0.87 0.87 0.90
0.86 0.87 0.86

20

1026
1027

1028
1029
1030
1031
1032
1033

1034
1035
1036
1037
1038
1039

1040
1041
1042
1043
1044

1045
1046
1047
1048
1049
1050

1051
1052
1053
1054
1055
1056

1057
1058
1059
1060
1061
1062

1063
1064
1065
1066
1067

1068
1069
1070
1071
1072
1073

1074
1075
1076
1077
1078
1079

0100200300400500Sample size0.50.60.70.80.91.0AUCPerplexityLowercaseZlibMin-k%0100200300400500Sample size0.00.10.20.30.40.50.6TPR@5%FPRPerplexityLowercaseZlibMin-k%