The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text
Under review as a conference paper at ICLR 2025

THE CANARY’S ECHO: AUDITING PRIVACY RISKS OF LLM-
GENERATED SYNTHETIC TEXT

Anonymous authors
Paper under double-blind review

ABSTRACT

How much information about training examples can be gleaned from synthetic data gen-
erated by Large Language Models (LLMs)? Overlooking the subtleties of information
flow in synthetic data generation pipelines can lead to a false sense of privacy. In this
paper, we investigate the design of membership inference attacks that target data used
to fine-tune pre-trained LLMs that are then used to synthesize data, particularly when
the adversary does not have access to the fine-tuned model but only to a synthetic data
corpus. We demonstrate that canaries crafted to maximize vulnerability to attacks that have
access to the model are sub-optimal for auditing privacy risks when only synthetic data is
released. This is because such out-of-distribution canaries have limited influence on the
model’s output when prompted to generate useful, in-distribution synthetic data, which
drastically reduces their vulnerability. To tackle this problem, we leverage the mechanics
of auto-regressive models to design canaries that leave detectable traces in synthetic data.
Our approach greatly enhances the power of membership inference attacks, providing a
better assessment of the privacy risks of releasing synthetic data generated by LLMs.

1

INTRODUCTION

Large Language Models (LLMs) can generate synthetic data that mimics human-written content through
domain-specific prompts. Besides their impressive fluency, LLMs are known to memorize parts of their
training data (Carlini et al., 2023) and can regurgitate exact phrases, sentences, or even longer passages when
prompted adversarially (Zanella-Béguelin et al., 2020; Carlini et al., 2021; Nasr et al., 2023). This raises
serious privacy concerns about unintended information leakage through synthetically generated text. In this
paper, we address the critical question: how much information about real data leaks through text synthetically
generated from it using LLMs?

Prior methods to audit privacy risks insert highly vulnerable out-of-distribution examples, known as ca-
naries (Carlini et al., 2019), into the training data and test whether they can be identified using membership
inference attacks (MIAs) (Shokri et al., 2017). Various MIAs have been proposed, typically assuming that
the attacker has access to the model or its output logits (Carlini et al., 2019; Shi et al., 2024). In the context
of LLMs, MIAs often rely on analyzing the model’s behavior when prompted with inputs related to the
canaries (Carlini et al., 2021; Chang et al., 2024; Shi et al., 2024). However, similar investigations are lacking
in scenarios where LLMs are used to generate synthetic data and only this synthetic data is made available.

Contributions
In this work, we study–for the first time–the factors that influence leakage of information
about a data corpus from synthetic data generated from it using LLMs. First, we introduce data-based attacks
that only have access to synthetic data, not the model used to generate it, and therefore cannot probe it
with adversarial prompts nor compute losses or other statistics used in model-based attacks (Ye et al., 2022;
Carlini et al., 2022a).We propose approximating membership likelihood using either a model trained on the

1

000
001
002
003

004
005
006
007
008
009

010
011
012
013
014
015

016
017
018
019
020

021
022
023
024
025
026

027
028
029
030
031
032

033
034
035
036
037
038

039
040
041
042
043

044
045
046

Under review as a conference paper at ICLR 2025

047
048
049
050

051
052
053
054
055
056

057
058
059
060
061
062

063
064
065
066
067

068
069
070
071
072
073

074
075
076
077
078
079

080
081
082
083
084
085

086
087
088
089
090

091
092
093

synthetic data or the target example similarity to its closest synthetic data examples. We design our attacks
adapting pairwise likelihood ratio tests as in RMIA (Zarifzadeh et al., 2024) and evaluate our attacks on
labeled datasets: SST-2 (Socher et al., 2013) and AG News (Zhang et al., 2015). Our results show that
MIAs leveraging only synthetic data achieve AUC scores of 0.71 for SST-2 and 0.66 for AG News, largely
outperforming a random guess baseline. This suggests that synthetic text can leak significant information
about the real data used to generate it.

Second, we use the attacks we introduce to quantify the gap in performance between data- and model-based
attacks. We do so in an auditing scenario, designing adversarial canaries and controlling leakage by varying
the number of times a canary occurs in the fine-tuning dataset. Experimentally, we find a sizable gap when
comparing attacks adapted to the idiosyncrasies of each setting: a canary would need to occur 8× more often
to be as vulnerable against a data-based attack as it is against a model-based attack (see Fig. 1).

Third, we discover that canaries designed for model-based attacks fall short when auditing privacy risks of
synthetic text. Indeed, privacy auditing of LLMs through model-based MIAs relies on rare, out-of-distribution
sequences of high perplexity (Carlini et al., 2019; Stock et al., 2022; Wei et al., 2024; Meeus et al., 2024c).
We confirm that model-based MIAs improve as canary perplexity increases. In sharp contrast, we find that
high perplexity sequences, although distinctly memorized by the target model, are less likely to be echoed in
synthetic data generated by the target model. Therefore, as a canary perplexity increases, the canary influence
on synthetic data decreases, making its membership less detectable from synthetic data (see Figure 2). We
show that low-perplexity, and even in-distribution canaries, while suboptimal for model-based attacks, are
more adequate canaries in data-based attacks.

Lastly, we propose an alternative canary design tailored for data-based attacks based on the following
observations: (i) in-distribution canaries aligned with the domain-specific prompt can influence the generated
output, and (ii) memorization is more likely when canaries contain sub-sequences with high perplexity. We
construct canaries starting with an in-distribution prefix of length F , transitioning into an out-of-distribution
suffix, increasing the likelihood that the model memorizes them and that they influence synthetic data. We
show that, for fixed overall canary perplexity, the true positive rate (TPR) of attacks increases by up to 2×
by increasing the length of the in-distribution prefix (see Fig. 1). Moreover, we find the MIA performance
(both AUC and TPR at low FPR) for canaries with in-distribution prefix and out-of-distribution suffix
(0 < F < max) to improve upon both entirely in-distribution canaries (F = max) and out-of-distribution
canaries (F = 0), for both datasets.

In terms of real-world applications, the novel MIAs and canary design that we propose can be used to
audit privacy risks of synthetic text. Auditing establishes a lower bound on privacy risks, which is useful
to take informed decisions about releasing synthetic data in sensitive applications (e.g., patient-clinician
conversations, customer assistance chats). These lower bounds complement upper bounds on privacy risks
from methods that synthesize text with provable guarantees, notably, differential privacy (DP). Auditing
can not only detect violations of DP guarantees stemming from flawed analyses, implementation bugs, or
incorrect assumptions, but also allows for less conservative decisions based on the performance of MIAs
matching the threat model of releasing synthetic data. In contrast, for data synthesized from models fine-tuned
with DP guarantees, DP bounds the risk of both model- and data-based attacks and hence does not account
for the inherent gap in attacker capabilities that we observe.

2 BACKGROUND AND PROBLEM STATEMENT

Synthetic text generation We consider a private dataset D = {xi = (si, ℓi)}N
i=1 of labelled text records
where si represents a sequence of tokens (e.g. a product review) and ℓi is a class label (e.g. the review
sentiment). A synthetic data generation mechanism is a probabilistic procedure mapping D to a synthetic
dataset (cid:101)D = {(cid:101)xi = ((cid:101)si, (cid:101)ℓi)} (cid:101)N
i=1. Unless stated otherwise, we consider

i=1 with a desired label set {ℓi} (cid:101)N

2

Under review as a conference paper at ICLR 2025

094
095
096
097

098
099
100
101
102
103

104
105
106
107
108
109

110
111
112
113
114

115
116
117
118
119
120

121
122
123
124
125
126

127
128
129
130
131
132

133
134
135
136
137

138
139
140

N = (cid:101)N . The synthetic dataset (cid:101)D should preserve the utility of the private dataset D, i.e., it should preserve
as many statistics of D that are useful for downstream analyses as possible. In addition, a synthetic data
generation mechanism should preserve the privacy of records in D, i.e. it should not leak sensitive information
from the private records into the synthetic records. The utility of a synthetic dataset can be measured by the
gap between the utility achieved by (cid:101)D and D in downstream applications. The fact that synthetic data is not
directly traceable to original data records does not mean that it is free from privacy risks. On the contrary, the
design of a synthetic data generation mechanism determines how much information from D leaks into (cid:101)D and
should be carefully considered. Indeed, several approaches have been proposed to generate synthetic data with
formal privacy guarantees (Kim et al., 2021; Tang et al., 2024; Wu et al., 2024; Xie et al., 2024). We focus
on privacy risks of text generated by a pre-trained LLM fine-tuned on a private dataset D (Yue et al., 2023;
Mattern et al., 2022; Kurakin et al., 2023). Specifically, we fine-tune an LLM θ0 on records (si, ℓi) ∈ D to
minimize the loss in completing si conditioned on a prompt template p(ℓi), obtaining θ. We then query θ
using the same prompt template to build a synthetic dataset (cid:101)D matching a given label distribution.

Membership inference attacks MIAs (Shokri et al., 2017) provide a meaningful measure for quantifying
the privacy risks of machine learning (ML) models, due to its simplicity but also due to the fact that
protection against MIAs implies protection against more devastating attacks such as attribute inference and
data reconstruction (Salem et al., 2023). In a MIA on a target model θ, an adversary aims to infer whether a
target record is present in the training dataset of θ. Different variants constrain the adversary’s access to the
model, ranging from full access to model parameters (Nasr et al., 2019) to query access (Zarifzadeh et al.,
2024). In our setting, we consider adversaries that observe the output logits on inputs of their choosing of
a model θ fine-tuned on a private dataset D. We naturally extend the concept of MIAs to synthetic data
generation mechanisms by considering adversaries that only observe a synthetic dataset (cid:101)D generated from D.

Privacy auditing using canaries A common method used to audit the privacy risks of ML models is to
evaluate the MIA vulnerability of canaries, i.e., artificial worst-case records inserted in otherwise natural
datasets. This method can also be employed to derive statistical lower bounds on the differential privacy
guarantees of the training pipeline (Jagielski et al., 2020; Zanella-Béguelin et al., 2023). Records crafted
the underlying data distribution of D give a good approximation to the
to be out-of-distribution w.r.t.
worst-case (Carlini et al., 2019; Meeus et al., 2024c). Canaries can take a range of forms, such as text
containing sensitive information (Carlini et al., 2019) and random (Wei et al., 2024) or synthetically generated
sequences (Meeus et al., 2024c). Prior work identified that longer sequences, repeated more often (Carlini
et al., 2023; Kandpal et al., 2022), and with higher perplexity (Meeus et al., 2024c) are better memorized
during training and hence are more vulnerable to model-based MIAs. We study multiple types of canaries
and compare their vulnerability against model- and synthetic data-based MIAs. We consider a set of canaries
{ˆxi = (ˆsi, ˆℓi)} ˆN
i=1, each crafted adversarially and inserted with probability 1/2 into the private dataset D. The
resulting dataset is then fed to a synthetic data generation mechanism. We finally consider each canary ˆxi
as the target record of a MIA to estimate the privacy risk of the generation mechanism (or the underlying
fine-tuned model).

Threat model We consider an adversary A who aims to infer whether a canary ˆx was included in the
private dataset D used to synthesize a dataset (cid:101)D. We distinguish between two threat models: (i) an adversary
A with query-access to output logits of a target model θ fine-tuned on D, and (ii) an adversary (cid:101)A with only
access to the synthetic dataset (cid:101)D. To the best of our knowledge, for text data this latter threat model has
not been studied extensively in the literature. In contrast, the privacy risks of releasing synthetic tabular
data are much better understood (Stadler et al., 2022; Yale et al., 2019; Hyeong et al., 2022; Zhang et al.,
2022). Algorithm 1 shows the generic membership inference experiment encompassing both model- and
data-based attacks, selected by the synthetic flag. The adversary is represented by a stateful procedure A,

3

Under review as a conference paper at ICLR 2025

141
142
143
144

145
146
147
148
149
150

151
152
153
154
155
156

157
158
159
160
161

162
163
164
165
166
167

168
169
170
171
172
173

174
175
176
177
178
179

180
181
182
183
184

185
186
187

Algorithm 1 Membership inference against an LLM-based synthetic text generator
1: Input: Fine-tuning algorithm T , pre-trained model θ0, private dataset D = {xi = (si, ℓi)}N
i=1, prompt template p(·), canary repetitions nrep, sampling method sample, adversary A

{(cid:101)ℓi} (cid:101)N

i=1, labels

i=1, p(·))

2: Output: Membership score β
3: ˆx ← A(T , θ0, D, {(cid:101)ℓi} (cid:101)N
4: b ∼ {0, 1}
5: if b = 1 then
6:
7: else
θ ← T (θ0, D)
8:
9: for i = 1 . . . (cid:101)N do
10:

θ ← T (θ0, D ∪ {ˆx}nrep)

(cid:101)si ∼ sample(θ(p((cid:101)ℓi)))
(cid:111) (cid:101)N

i=1

(cid:110)
((cid:101)si, (cid:101)ℓi)

11: (cid:101)D ←
12: if synthetic then
β ← A( (cid:101)D, ˆx)
13:
14: else
15:
16: return β

β ← A(θ, ˆx)

▷ Adversarially craft a canary (see Sec. 3.2)
▷ Flip a fair coin

▷ Fine-tune θ0 with canary repeated nrep times

▷ Fine-tune θ0 without canary

▷ Sample synthetic records using prompt template

▷ Compute membership score β of ˆx
▷ See Sec. 3.1.2 and algorithms in Appendix A

▷ See Sec. 3.1.1

used to craft a canary and compute its membership score. Compared to a standard membership experiment,
we consider a fixed private dataset D rather than sampling it, and let the adversary choose the target ˆx. This is
close to the threat model of unbounded differential privacy, where the implicit adversary selects two datasets,
one obtained from the other by adding one more record, except that in our case the adversary observes but
cannot choose the records in D. The membership score β returned by the adversary can be turned into a
binary membership label by choosing an appropriate threshold. We further clarify assumptions made for the
adversary in both threat models in Appendix D.

Problem statement We study methods to audit privacy risks associated with releasing synthetic text. Our
main goal is to develop an effective data-based adversary (cid:101)A in the threat model of Algorithm 1. For this, we
explore the design space of canaries to approximate the worst-case, and adapt state-of-the-art methods used
to compute membership scores in model-based attacks to the data-based scenario.

3 METHODOLOGY

3.1 COMPUTING THE MEMBERSHIP SCORE

In Algorithm 1, the adversary computes a membership score β indicating their confidence that θ was trained
on ˆx (i.e. that b = 1). We specify first how to compute a membership signal α for model- and data-based
adversaries, and then how we compute β from α adapting the RMIA methodology of Zarifzadeh et al. (2024).

3.1.1 MEMBERSHIP SIGNAL FOR MODEL-BASED ATTACKS

The larger the target model θ’s probability for canary ˆx = (ˆs, ˆℓ), Pθ(ˆs | p(ˆℓ)), as compared to its probability
on reference models, the more likely that the model has seen this record during training. We compute the
probability for canary ˆx as the product of token-level probabilities for ˆs conditioned on the prompt p(ˆℓ). Given

4

Under review as a conference paper at ICLR 2025

a target canary text ˆs = t1, . . . , tn, we compute Pθ(ˆs | p(ˆℓ)) as Pθ(ˆx) = (cid:81)n
We consider this probability as the membership inference signal against a model, i.e. α = Pθ(ˆs | p(ˆℓ)).

j=1 Pθ(tj | p(ˆℓ), t1, . . . , tj−1).

3.1.2 MEMBERSHIP SIGNAL FOR DATA-BASED ATTACKS

When the attacker only has access to the generated synthetic data, we need to extract a signal that corre-
lates with membership purely from the synthetic dataset (cid:101)D. We next describe two methods to compute a
membership signal α based on (cid:101)D. For more details, refer to their pseudo-code in Appendix A.

Membership signal using n-gram model The attacker first fits an n-gram model using (cid:101)D as training
corpus. An n-gram model computes the probability of the next token wj in a sequence based solely on the
previous n − 1 tokens (Jurafsky & Martin, 2024). The conditional probability of a token wj given the previous
n − 1 tokens is estimated from the counts of n-grams in the training corpus. Formally,

Pn-gram(wj | wj−(n−1), . . . , wj−1) =

C(wj−(n−1), . . . , wj) + 1
C(wj−(n−1), . . . , wj−1) + V

,

(1)

where C(s) is the number of times the sequence s appears in the training corpus and V is the vocabulary size.
We use Laplace smoothing to deal with n-grams that do not appear in the training corpus, incrementing by 1
the count of every n-gram. The probability that the model assigns to a sequence of tokens s = (w1, . . . , wk)
can be computed using the chain rule: Pn-gram(s) = (cid:81)k
j=2 Pn-gram(wj | wj−(n−1), . . . , wj−1). With the
n-gram model fitted on the synthetic dataset, the attacker computes the n-gram model probability of the target
canary ˆx = (ˆs, ˆℓ) as its membership signal, i.e. α = Pn-gram(ˆs). The intuition here is that if the canary ˆx was
present in the training data, the generated synthetic data (cid:101)D will better reflect the patterns of ˆs, resulting in the
n-gram model assigning a higher probability to ˆs than if it was not present.

Membership signal using similarity metric The attacker computes the similarity between the target
canary text ˆs and all synthetic sequences (cid:101)si in (cid:101)D using some similarity metric SIM, i.e. σi = SIM(ˆs, (cid:101)si) for
i = 1, . . . , (cid:101)N . Next, the attacker identifies the k synthetic sequences with the largest similarity to ˆs. Let σi(j)
denote the j-th largest similarity. The membership inference signal is then computed as the mean of the k
most similar examples, i.e. α = 1
j=1 σi(j). The intuition here is that if ˆs was part of the training data, the
k
synthetic data (cid:101)D will likely contain sequences (cid:101)si more similar to ˆs than if ˆs was not part of the training data,
resulting in a larger mean similarity. Various similarity metrics can be used. We consider Jaccard similarity
(SIMJac), often used to measure string similarity, and cosine similarity between the embeddings of the two
sequences, computed using a pre-trained embedding model (SIMemb).

(cid:80)k

3.1.3 LEVERAGING REFERENCE MODELS TO COMPUTE RMIA SCORES

Reference models, also called shadow models, are surrogate models designed to approximate the behavior of
a target model. MIAs based on reference models perform better but are more costly to run than MIAs that
do not use them, with the additional practical challenge that they require access to data distributed similarly
to the training data of the target model (Shokri et al., 2017; Ye et al., 2022). Obtaining multiple reference
models in our scenario requires fine-tuning a large number of parameters in an LLM and quickly becomes
computationally prohibitive. We use the state-of-the-art RMIA method (Zarifzadeh et al., 2024) to maximize
attack performance with a limited number of reference models M . Specifically, for the target model θ, we
calculate the membership score of a canary ˆx using reference models {θ′
i=1 as follows (we present the
details on the application of RMIA to our setup in Appendix B):

i}M

αθ(ˆx)
i=1 αθ′

i

(cid:80)M

βθ(ˆx) =

1
M

5

.

(ˆx)

(2)

188
189
190
191

192
193
194
195
196
197

198
199
200
201
202
203

204
205
206
207
208

209
210
211
212
213
214

215
216
217
218
219
220

221
222
223
224
225
226

227
228
229
230
231

232
233
234

Under review as a conference paper at ICLR 2025

235
236
237
238

239
240
241
242
243
244

245
246
247
248
249
250

251
252
253
254
255

256
257
258
259
260
261

262
263
264
265
266
267

268
269
270
271
272
273

274
275
276
277
278

279
280
281

3.2 CANARY GENERATION

Prior work has shown that canaries with high perplexity are more likely to be memorized by language
models (Meeus et al., 2024c). High perplexity sequences are less predictable and require the model to encode
more specific, non-generalizable details about them. However, high perplexity canaries are not necessarily
more susceptible to leakage via synthetic data generation, as they are outliers in the text distribution when
conditioned on a given in-distribution prompt. This misalignment with the model’s natural generative behavior
means that even when memorized, these canaries are unlikely to be reproduced during regular model inference,
making them ineffective for detecting memorization of training examples in generated synthetic data.

To address this issue, we take advantage of the greedy nature of popular autoregressive decoding strategies
(e.g. beam search, top-k and top-p sampling). We can encourage such decoding strategies to generate text
closer to canaries by crafting canaries with a low perplexity prefix. To ensure memorization, we follow
established practices and choose a high perplexity suffix. Specifically, we design canaries ˆx = (ˆs, ˆℓ), where ˆs
has an in-distribution prefix and an out-of-distribution suffix. In practice, we split the original dataset D
into a training dataset and a canary source dataset. For each record x = (s, ℓ) in the canary source dataset,
we design a new canary ˆx = (ˆs, ˆℓ). We truncate s to get an in-distribution prefix of length F and generate
a suffix using the pre-trained language model θ0, adjusting the sampling temperature to achieve a desired
target perplexity Ptarget. We use rejection sampling to ensure that the perplexity of the generated canaries falls
within the range [0.9 Ptarget, 1.1 Ptarget]. We ensure the length is consistent across canaries, as this impacts
memorization (Carlini et al., 2023; Kandpal et al., 2022). By adjusting the length of the in-distribution prefix,
we can guide the generation of either entirely in-distribution or out-of-distribution canaries.

We insert each canary nrep times in the training dataset of target and reference models. When a canary is
selected as a member, the canary is repeated nrep times in the training dataset, while canaries selected as
non-members are excluded from the training dataset. As in prior work (Carlini et al., 2023; Kandpal et al.,
2022; Meeus et al., 2024c), we opt for nrep > 1 to increase memorization, thus facilitating privacy auditing
and the observation of the effect of different factors on the performance of MIAs during ablation studies.

4 EXPERIMENTAL SETUP

Datasets We consider two datasets that have been widely used to study text classification: (i) the Stanford
Sentiment Treebank (SST-2) (Socher et al., 2013), which consists of excerpts from written movie reviews
with a binary sentiment label; and (ii) the AG News dataset (Zhang et al., 2015), which consists of news
articles labelled by category (World, Sport, Business, Sci/Tech). In all experiments, we remove examples
with less than 5 words, bringing the total number of examples to 43 296 for SST-2 and 120 000 for AG News.

Synthetic data generation We fine-tune the pre-trained Mistral-7B model (Jiang et al., 2023) using low-rank
adaptation (LoRa) (Hu et al., 2022). We use a custom prompt template p(·) for each dataset (see Appendix C).
We sample synthetic data from the fine-tuned model θ conditioned on prompts p((cid:101)ℓi), following the same
distribution of labels in the synthetic dataset (cid:101)D as in the original dataset D, i.e. ℓi = (cid:101)ℓi for i = 1, ..., (cid:101)N . To
generate synthetic sequences, we sequentially sample completions using a softmax temperature of 1.0 and
top-p (aka nucleus) sampling with p = 0.95, i.e. we sample from a vocabulary restricted to the smallest
possible set of tokens whose total probability exceeds 0.95. We further ensure that the synthetic data we
generate bears high utility, and is thus realistic. For this, we consider the downstream classification tasks for
which the original datasets have been designed. We fine-tune RoBERTa-base (Liu et al., 2019) on D and (cid:101)D
and compare the performance of the resulting classifiers on held-out evaluation datasets. Further details and
results are provided in Appendix E, for synthetic data generated with and without canaries.

6

Under review as a conference paper at ICLR 2025

282
283
284
285

286
287
288
289
290
291

292
293
294
295
296
297

298
299
300
301
302

303
304
305
306
307
308

309
310
311
312
313
314

315
316
317
318
319
320

321
322
323
324
325

326
327
328

Canary injection

ROC AUC

Dataset

Source

Label

Model

Synthetic
(2-gram)

Synthetic
(SIMJac)

Synthetic
(SIMemb)

SST-2

AG News

In-distribution1

Synthetic

In-distribution

Synthetic

Natural
Artificial

Natural
Artificial

0.911

0.999
0.999

0.993

0.996
0.999

0.711

0.616
0.661

0.620

0.644
0.660

0.602

0.547
0.552

0.590

0.552
0.560

0.586

0.530
0.539

0.565

0.506
0.525

Table 1: ROC AUC across training datasets, canary injection mechanisms and MIA methodologies. We give
the ROC curves and TPR at low FPR scores in Appendix F, further ablations in Appendix G, and elaborate
on the disparate vulnerability of high perplexity canaries in model- and data-based attacks in Appendix H.

Canary injection We generate canaries ˆx = (ˆs, ˆℓ) as described in Sec. 3.2. Unless stated otherwise, we
consider 50-word canaries. Synthetic canaries are generated using Mistral-7B (Jiang et al., 2023) as θ0. We
consider two ways of constructing a canary label: (i) randomly sampling label ˆℓ from the distribution of labels
in the dataset, ensuring that the class distribution among canaries matches that of D (Natural); (ii) extending
the set of labels with a new artificial label (ˆℓ ="canary") only used for canaries (Artificial).

Membership inference Throughout our experiments, we compute the βθ(ˆx) membership scores as de-
scribed in Sec. 3.1. For one target model θ, we consider 1000 canaries ˆx, of which on average half are
included in the training dataset nrep times (members), while the remaining half are excluded (non-members).
We then use the computed RMIA scores and the ground truth for membership to construct ROC curves
for attacks from which we compute AUC and true positive rate (TPR) at low false positive rate (FPR) as
measures of MIA performance. Across our experiments, we use M = 4 reference models θ′, each trained
on a dataset Dθ′ consisting of the dataset D used to train the target model θ with canaries inserted. Note
that although practical attacks rarely have this amount of information, this is allowed by the threat model
of Algorithm 1 and perfectly valid as a worst-case auditing methodology. We ensure that each canary is a
member in half (i.e. 2) of the reference models and a non-member in the other half. For the attacks based
on synthetic data, we use n = 2 for computing scores using an n-gram model and k = 25 for computing
scores based on cosine similarity. In this latter case, we use Sentence-BERT (Reimers & Gurevych, 2019)
(paraphrase-MiniLM-L6-v2 from sentence-transformers) as the embedding model.

5 RESULTS

5.1 BASELINE EVALUATION WITH STANDARD CANARIES

We begin by assessing the vulnerability of synthetic text using standard canaries. Specifically, we utilize
both in-distribution canaries and synthetically generated canaries with a target perplexity Ptarget = 250, no
in-distribution prefix (F = 0), nrep = 12 and natural or artificial labels, as described in Section 4. Table 1
summarizes the ROC AUC for model- and data-based attacks.

First, we find that MIAs relying solely on the generated synthetic data achieve a ROC AUC score significantly
higher than a random guess (i.e. AUC = 0.5), reaching up to 0.71 for SST-2 and 0.66 for AG News. This
shows that synthetic text can leak information about the real data used to generate it.

1Constrained by in-distribution data, we here consider canaries of exactly 30 words (instead of 50 anywhere else).

7

Under review as a conference paper at ICLR 2025

329
330
331
332

333
334
335
336
337
338

339
340
341
342
343
344

345
346
347
348
349

350
351
352
353
354
355

356
357
358
359
360
361

362
363
364
365
366
367

368
369
370
371
372

373
374
375

Next, we observe that the data-based attack that uses an n-gram model trained on synthetic data to compute
membership scores outperforms the two attacks that use instead similarity metrics: Jaccard distance between
a canary and synthetic strings (SIMJac) or cosine distance between their embeddings (SIMemb). This suggests
that critical information for inferring membership lies in subtle changes in the co-occurrence of n-grams in
synthetic data rather than in the generation of many sequences with lexical or semantic similarity.

We also compare attack performance across different canary types under data-based attacks A (cid:101)D. The ROC
AUC remains consistently higher than a random guess across all canaries. For SST-2, the highest AUC score
of 0.71 is achieved when using in-distribution canaries. In contrast, for AG News, the highest AUC score of
0.66 is achieved for synthetic canaries with an artificial label not occurring in the dataset.

As another baseline, we test RMIA on the target model trained on D, under the assumption that the attacker has
access to the model logits (Aθ). This attack achieves near-perfect performance across all setups, highlighting
the fact that there is an inherent gap between the performance of model- and data-based attacks. A plausible
explanation is that, while a fine-tuned model memorizes standard canaries well, the information necessary to
infer their membership is partially transmitted to synthetic text generated using it.

To investigate the gap between the two attacks in more detail, we vary the number of canary repetitions nrep
to amplify the power of the data-based attack until its performance matches that of a model-based attack.
Fig. 1(a) illustrates these results as a set of ROC curves. We quantify this discrepancy by noting that the
MIA performance for A (cid:101)D at nrep = 16 is comparable to Aθ at nrep = 2 and for low FPR at nrep = 1. We
find similar results in Fig. 1(d) for AG News. The MIA performance for A (cid:101)D at nrep = 16 falls between the
performance of Aθ at nrep = 1 and nrep = 2. Under these experimental conditions, canaries would need to be
repeated 8 to 16× to reach the same vulnerability in data-based attacks compared to model-based attacks.

5.2 DESIGNING SPECIALIZED CANARIES FOR ENHANCED PRIVACY AUDITING

To effectively audit privacy risks in a worst-case scenario, we explore the design of specialized canaries that
are both memorized by the model and influential in the synthetic data.

First, we generate specialized canaries by controlling their target perplexity Ptarget. We evaluate MIAs for
both threat models across a range of perplexities for canaries with natural labels, using nrep = 4 for the model-
based attack Aθ and nrep = 16 for the data-based attack A (cid:101)D. We explore a wide range of perplexities, finding
1 × 105 to align with random token sequences. Figure 2 shows the ROC AUC score versus canary perplexity.
For the model-based attack Aθ, the AUC monotonically increases with canary perplexity, reaffirming that
outlier data records with higher perplexity are more vulnerable to MIAs (Feldman & Zhang, 2020; Carlini
et al., 2022a; Meeus et al., 2024c). Conversely, for the data-based attack A (cid:101)D, the AUC initially increases
with perplexity but starts to decline beyond a certain threshold, eventually approaching a random guess (AUC
of 0.5). To further illustrate this, we present the complete ROC curve in Figures 1(b) and (e) for SST-2 and
AG News, respectively. We vary the canary perplexity Ptarget while keeping other parameters constant. As
Ptarget increases, the model-based attack improves across the entire FPR range, while the data-based attack
weakens, approaching a random guess at high perplexities. This suggests that identifying susceptible canaries
is straightforward for model-based privacy audits, but assessing the privacy risk of synthetic data requires a
careful balance between canary memorization and its influence on synthetic data.

We now examine whether canaries can be crafted to enhance both memorization and influence on the synthetic
data, making them suitable to audit the privacy risks of releasing synthetic data. In Sec. 3.2, we introduced
a method that exploits the greedy nature of LLM decoding to design more vulnerable canaries. We craft a
canary with a low-perplexity in-distribution prefix to optimize its impact on the synthetic dataset, followed by
a high-perplexity suffix to enhance memorization. We generate this suffix sampling from the pre-trained LLM
θ0 with high temperature. Figures 1(c) and (f) illustrate the results for SST-2 and AG News, respectively. We

8

Under review as a conference paper at ICLR 2025

376
377
378
379

380
381
382
383
384
385

386
387
388
389
390
391

392
393
394
395
396

397
398
399
400
401
402

403
404
405
406
407
408

409
410
411
412
413
414

415
416
417
418
419

420
421
422

1

0.8

0.6

0.4

0.2

R
P
T

0

0

0.2

0.4

0.6

FPR

A (cid:101)D, nrep = 2
A (cid:101)D, nrep = 4
A (cid:101)D, nrep = 8
A (cid:101)D, nrep = 16
Aθ, nrep = 1
Aθ, nrep = 2
Aθ, nrep = 4
0.8

1

(a) Number of canary repetitions nrep.
Ptarget = 31, F = 0.

1

0.8

0.6

0.4

0.2

R
P
T

0

0

0.2

0.4

FPR

A (cid:101)D, Ptar = 10
A (cid:101)D, Ptar = 102
A (cid:101)D, Ptar = 103
A (cid:101)D, Ptar = 104
Aθ, Ptar = 10
Aθ, Ptar = 102
Aθ, Ptar = 103
Aθ, Ptar = 104
0.8

0.6

(b) Canary perplexity Ptarget.
rep = 4, n (cid:101)D
nθ
rep = 16, F = 0.

1

1

0.8

0.6

0.4

0.2

R
P
T

1

0

0

0.2

0.4

0.6

FPR

A (cid:101)D, F = 0
A (cid:101)D, F = 10
A (cid:101)D, F = 20
A (cid:101)D, F = 30
Aθ, F = 0
A (cid:101)D, F = max
0.8

1

(c) Canary in-distribution prefix F .
Ptarget = 31, nθ

rep = 4, n (cid:101)D

rep = 16.

1

1

0.8

0.6

0.4

0.2

R
P
T

0

0

0.2

0.4

0.6

FPR

R
P
T

0.8

0.6

0.4

0.2

0

0

0.2

0.4

FPR

A (cid:101)D, Ptar = 10
A (cid:101)D, Ptar = 102
A (cid:101)D, Ptar = 103
A (cid:101)D, Ptar = 104
Aθ, Ptar = 10
Aθ, Ptar = 102
Aθ, Ptar = 103
Aθ, Ptar = 104
0.8

0.6

A (cid:101)D, nrep = 2
A (cid:101)D, nrep = 4
A (cid:101)D, nrep = 8
A (cid:101)D, nrep = 16
Aθ, nrep = 1
Aθ, nrep = 2
Aθ, nrep = 4
0.8

1

R
P
T

0.8

0.6

0.4

0.2

1

0

0

0.2

0.4

0.6

FPR

A (cid:101)D, F = 0
A (cid:101)D, F = 10
A (cid:101)D, F = 20
A (cid:101)D, F = 30
Aθ, F = 0
A (cid:101)D, F = max
0.8

1

(d) Number of canary repetitions nrep.
Ptarget = 31, F = 0.

(e) Canary perplexity Ptarget.
nθ
rep = 4, n (cid:101)D
rep = 16, F = 0.

(f) Canary in-distribution prefix F .
Ptarget = 31, nθ

rep = 4, n (cid:101)D

rep = 16.

Figure 1: ROC curves of MIAs on synthetic data A (cid:101)D compared to model-based MIAs Aθ on SST-2 ((a)–(c))
and AG News ((d)–(f)). We ablate over the number of canary insertions nrep in (a), (d), the target perplexity
Ptarget of the inserted canaries in (b), (e) and the length F of the in-distribution prefix in the canary in (c), (f).

set the overall canary perplexity Ptarget = 31 and vary the prefix length F . As a reference, we also plot the
results for in-distribution canaries labelled by F = max. We observe that combining an in-distribution prefix
(F > 0) with a high-perplexity suffix (F < max) enhances attack effectiveness. This effect is especially
notable for SST-2. For AG News, the improvement gained from adding an in-distribution prefix is less
pronounced. This suggests that although the model’s memorization of the canary stays consistent (as the
overall perplexity remains unchanged), the canary’s impact on the synthetic data becomes more prominent
with longer in-distribution prefixes. We hypothesize that familiar low-perplexity prefixes serve as starting
points for text generation, enhancing the likelihood that traces of the canary appear in the synthetic data.

6 RELATED WORK

MIAs against ML models Since the seminal work of Shokri et al. (2017), MIAs have been used to study
memorization and privacy risks. Model-based MIAs have been studied under varying threat models, including
adversaries with white-box access to model weights (Sablayrolles et al., 2019; Nasr et al., 2019; Leino &
Fredrikson, 2020; Cretu et al., 2024), access to output probabilities (Shokri et al., 2017; Carlini et al., 2022a)

9

Under review as a conference paper at ICLR 2025

423
424
425
426

427
428
429
430
431
432

433
434
435
436
437
438

439
440
441
442
443

444
445
446
447
448
449

450
451
452
453
454
455

456
457
458
459
460
461

462
463
464
465
466

467
468
469

1

0.8

0.6

C
U
A
A
M

I

Aθ
A (cid:101)D
Random guess

C
U
A
A
M

I

1

0.9

0.8

0.7

0.6

0.5

Aθ
A (cid:101)D
Random guess

101

102

103
Canary perplexity P

104

105

101

102

103
Canary perplexity P

104

105

(a) SST-2

(b) AG News

Figure 2: ROC AUC score for synthetic canaries with varying perplexity (natural label). We present results
for a model-based MIA Aθ using output logits and a data-based attack A (cid:101)D using a 2-gram model. While the
model-based attack improves as the perplexity increases, the inverse happens for the data-based attack.

or just labels (Choquette-Choo et al., 2021). The most powerful MIAs leverage a large number of reference
models (Ye et al., 2022; Carlini et al., 2022a; Sablayrolles et al., 2019; Watson et al., 2021). Zarifzadeh et al.
(2024) proposed RMIA, which achieves high performance using only a few.

Attacks against language models Song & Shmatikov (2019) study the benign use of MIAs to audit the use
of an individual’s data during training. Carlini et al. (2021) investigate training data reconstruction attacks
against LLMs. Kandpal et al. (2022) and Carlini et al. (2023) both study the effect of de-duplicating training
data in reconstruction attacks by sampling a large corpus of synthetic text and running model-based attacks
to identify likely members. Shi et al. (2024) and Meeus et al. (2024b) use attacks to identify pre-training
data. Various membership inference scores have been proposed, such as the loss of target records (Yeom
et al., 2018), lowest predicted token probabilities (Shi et al., 2024), changes in the model’s probability for
neighboring samples (Mattern et al., 2023), or perturbations to model weights (Li et al., 2023).

MIAs against synthetic data in other scenarios Hayes et al. (2019) train a Generative Adversarial Network
(GAN) on synthetic images generated by a target GAN and use the resulting discriminator to infer membership.
Hilprecht et al. (2019) explore MIAs using synthetic images closest to a target record. Chen et al. (2020)
study attack calibration techniques against GANs for images and location data. Privacy risks of synthetic
tabular data have been widely studied, using MIAs based on similarity metrics and shadow models (Yale et al.,
2019; Hyeong et al., 2022; Zhang et al., 2022). Stadler et al. (2022) compute high-level statistics, Houssiau
et al. (2022) compute similarities between the target record and synthetic data, and Meeus et al. (2024a)
propose a trainable feature extractor. Unlike these, we evaluate MIAs on text generated using fine-tuned
LLMs. This introduces unique challenges and opportunities, both in computing membership scores and
identifying worst-case canaries, making our approach distinct from prior work.

Vulnerable records in MIAs Prior work established that some records (outliers) have a disparate effect on a
trained model compared to others (Feldman & Zhang, 2020), making them more vulnerable to MIAs (Carlini
et al., 2022a;b). Hence, specifically crafted canaries have been proposed to study memorization and for
privacy auditing of language models, ranging from a sequence of random digits (Carlini et al., 2019; Stock
et al., 2022) or random tokens (Wei et al., 2024) to synthetically generated sequences (Meeus et al., 2024c).
In the case of synthetic tabular data, Stadler et al. (2022) find that statistical outliers have increased privacy
leakage, while Meeus et al. (2024a) propose measuring the distance to the closest records to infer membership.

Decoding method We use fixed prompt templates and top-p sampling to assess the privacy of synthetic text
in a realistic regime rather than allowing the attacker to pick a decoding method adversarially. Research on
data reconstruction attacks study how decoding methods like beam search (Zanella-Béguelin et al., 2020;
Carlini et al., 2023), top-k sampling (Kandpal et al., 2022), or decaying temperature (Carlini et al., 2021)
impact how often LLMs replicate information from their training data.

10

Under review as a conference paper at ICLR 2025

470
471
472
473

474
475
476
477
478
479

480
481
482
483
484
485

486
487
488
489
490

491
492
493
494
495
496

497
498
499
500
501
502

503
504
505
506
507
508

509
510
511
512
513

514
515
516

7 REPRODUCIBILITY STATEMENT

Both datasets used in this paper are publicly available (Socher et al., 2013; Zhang et al., 2015), and so is the
pre-trained model (Jiang et al., 2023) we used. We fine-tune the pre-trained model for 1 epoch using LoRA
with r = 4, including all target modules (10.7M parameters in total). We use an effective batch size of 128
and learning rate η = 2 × 10−5 (for more details see Appendix J). All our experiments have been conducted
on a cluster of nodes with 8 V100 NVIDIA GPUs with a floating point precision of 16 (fp16). We built
our experiments on two open-source packages: (i) privacy-estimates which provides a distributed
implementation of the RMIA attack and (ii) dp-transformers which provides the implementation of
the synthetic data generator. All of our code is attached in the supplemented materials. In addition, we will
release the code necessary to reproduce the results presented in this paper on GitHub upon publication.

REFERENCES

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating
and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX
Security 19), pp. 267–284. USENIX Association, 2019. doi:10.5555/3361338.3361358.

Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee,
Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large
language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650. USENIX
Association, 2021. URL https://www.usenix.org/conference/usenixsecurity21/
presentation/carlini-extracting.

Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Membership
inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (S&P), pp.
1897–1914. IEEE, 2022a. doi:10.1109/SP46214.2022.9833649.

Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramèr.
The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems
(NeurIPS 2022), 35:13263–13276, 2022b. URL http://papers.nips.cc/paper_files/
paper/2022/hash/564b5f8289ba846ebc498417e834c253-Abstract-Conference.
html.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang.
Quantifying memorization across neural language models. In 11th International Conference on Learning
Representations (ICLR 2023). OpenReview.net, 2023. URL https://openreview.net/forum?
id=TatRHT_1cK.

Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, and Reza Shokri. Context-
aware membership inference attacks against pre-trained large language models, 2024. URL https:
//arxiv.org/abs/2409.13745. arXiv preprint.

Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. GAN-leaks: A taxonomy of membership inference
attacks against generative models. In 2020 ACM SIGSAC conference on computer and communications
security (CCS 2020), pp. 343–362. ACM, 2020. doi:10.1145/3372297.3417238.

Christopher A Choquette-Choo, Florian Tramèr, Nicholas Carlini, and Nicolas Papernot. Label-only
In 38th International conference on machine learning (ICML 2021),
membership inference attacks.
volume 139, pp. 1964–1974. PMLR, 2021. URL https://proceedings.mlr.press/v139/
choquette-choo21a.html.

11

Under review as a conference paper at ICLR 2025

Ana-Maria Cretu, Daniel Jones, Yves-Alexandre de Montjoye, and Shruti Tople. Investigating the effect of
misalignment on membership privacy in the white-box setting. Proc. Priv. Enhancing Technol., 2024(3):
407–430, 2024. doi:10.56553/POPETS-2024-0085.

Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the
long tail via influence estimation. Advances in Neural Information Processing Systems (NeurIPS
2020), 33:2881–2891, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/
1e14bfe2714193e7af5abc64ecbd6b46-Abstract.html.

Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. LOGAN: Membership in-
ference attacks against generative models. Proc. Priv. Enhancing Technol., 2019(1):133–152, 2019.
doi:10.2478/popets-2019-0008.

Benjamin Hilprecht, Martin Härterich, and Daniel Bernau. Monte Carlo and reconstruction membership
inference attacks against generative models. Proc. Priv. Enhancing Technol., 2019(4):232–249, 2019.
doi:10.2478/popets-2019-0067.

Florimond Houssiau, James Jordon, Samuel N Cohen, Owen Daniel, Andrew Elliott, James Geddes, Callum
Mole, Camila Rangel-Smith, and Lukasz Szpruch. TAPAS: a toolbox for adversarial privacy auditing of
synthetic data. In NeurIPS 2022 Workshop on Synthetic Data for Empowering ML Research, 2022. URL
https://openreview.net/forum?id=9hXskf1K7zQ.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
LoRA: Low-rank adaptation of large language models. In 10th International Conference on Learning
Representations (ICLR 2022). OpenReview.net, 2022. URL https://openreview.net/forum?
id=nZeVKeeFYf9.

Jihyeon Hyeong, Jayoung Kim, Noseong Park, and Sushil Jajodia. An empirical study on the membership in-
ference attack against tabular data synthesis models. In 31st ACM International Conference on Information
& Knowledge Management (CIKM ’22), pp. 4064–4068. ACM, 2022. doi:10.1145/3511808.3557546.

Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learn-
ing: How private is private SGD? Advances in Neural Information Processing Systems (NeurIPS
2020), 33:22205–22216, 2020. URL https://proceedings.neurips.cc/paper/2020/
hash/fc4ddc15f9f4b4b06ef7844d6bb53abf-Abstract.html.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego
de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud,
Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,
and William El Sayed. Mistral 7B, 2023. URL https://arxiv.org/abs/2310.06825. arXiv
preprint.

Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural
Language Processing, Computational Linguistics, and Speech Recognition with Language Models. n.p.,
3rd edition, 2024. URL https://web.stanford.edu/~jurafsky/slp3/. Online manuscript
released August 20, 2024.

Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language
models. In 39th International Conference on Machine Learning (ICML 2022), volume 162, pp. 10697–
10707. PMLR, 2022. URL https://proceedings.mlr.press/v162/kandpal22a.html.

Kunho Kim, Sivakanth Gopi,

vate n-gram extraction.
34:5102–5111, 2021.
28ce9bc954876829eeb56ff46da8e1ab-Abstract.html.

Janardhan Kulkarni, and Sergey Yekhanin.

Differentially pri-
Advances in Neural Information Processing Systems (NeurIPS 2021),
URL https://proceedings.neurips.cc/paper/2021/hash/

12

517
518
519
520

521
522
523
524
525
526

527
528
529
530
531
532

533
534
535
536
537

538
539
540
541
542
543

544
545
546
547
548
549

550
551
552
553
554
555

556
557
558
559
560

561
562
563

Under review as a conference paper at ICLR 2025

564
565
566
567

568
569
570
571
572
573

574
575
576
577
578
579

580
581
582
583
584

585
586
587
588
589
590

591
592
593
594
595
596

597
598
599
600
601
602

603
604
605
606
607

608
609
610

Alexey Kurakin, Natalia Ponomareva, Umar Syed, Liam MacDermed, and Andreas Terzis. Harnessing
large-language models to generate private synthetic text, 2023. URL https://arxiv.org/abs/
22306.01684. arXiv preprint.

Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated
In 29th USENIX Security Symposium (USENIX Security 20),
white-box membership inference.
pp. 1605–1622. USENIX Association, 2020. URL https://www.usenix.org/conference/
usenixsecurity20/presentation/leino.

Marvin Li, Jason Wang, Jeffrey George Wang, and Seth Neel. MoPe: Model perturbation based privacy
attacks on language models. In 2023 Conference on Empirical Methods in Natural Language Processing
(EMNLP 2023), pp. 13647–13660. ACL, 2023. doi:10.18653/v1/2023.emnlp-main.842.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke
Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach, 2019.
URL https://arxiv.org/abs/1907.11692. arXiv preprint.

Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. Differen-
tially private language models for secure data sharing. In 2022 Conference on Empirical Methods in Natural
Language Processing (EMNLP 2022), pp. 4860–4873. ACL, 2022. doi:10.18653/v1/2022.emnlp-main.323.

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor
Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison.
In Findings of the Association for Computational Linguistics: ACL 2023, pp. 11330–11343. ACL, 2023.
doi:10.18653/v1/2023.findings-acl.719.

Matthieu Meeus, Florent Guepin, Ana-Maria Cre¸tu, and Yves-Alexandre de Montjoye. Achilles’ heels:
vulnerable record identification in synthetic data publishing. In European Symposium on Research in
Computer Security (ESORICS 2023), pp. 380–399. Springer, 2024a. doi:10.1007/978-3-031-51476-0_19.

Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye. Did the neurons read
your book? document-level membership inference for large language models. In 33rd USENIX Security
Symposium (USENIX Security 24), pp. 2369–2385. USENIX Association, 2024b. URL https://www.
usenix.org/conference/usenixsecurity24/presentation/meeus.

Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye. Copyright traps for large
language models. In 41st International Conference on Machine Learning (ICML 2024), volume 235,
pp. 35296–35309. PMLR, 2024c. URL https://proceedings.mlr.press/v235/meeus24a.
html.

Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive
and active white-box inference attacks against centralized and federated learning. In 2019 IEEE Symposium
on Security and Privacy (S&P), pp. 739–753. IEEE, 2019. doi:10.1109/SP.2019.00065.

Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito,
Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction
of training data from (production) language models, 2023. URL https://arxiv.org/abs/2311.
17035. arXiv preprint.

Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In
2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019), pp. 3982–3992.
ACL, 2019. doi:10.18653/v1/D19-1410.

13

Under review as a conference paper at ICLR 2025

611
612
613
614

615
616
617
618
619
620

621
622
623
624
625
626

627
628
629
630
631

632
633
634
635
636
637

638
639
640
641
642
643

644
645
646
647
648
649

650
651
652
653
654

655
656
657

Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-
box vs black-box: Bayes optimal strategies for membership inference. In 36th International Confer-
ence on Machine Learning (ICML 2019), volume 97, pp. 5558–5567. PMLR, 2019. URL https:
//proceedings.mlr.press/v97/sablayrolles19a.

Ahmed Salem, Giovanni Cherubin, David Evans, Boris Köpf, Andrew Paverd, Anshuman Suri, Shruti Tople,
and Santiago Zanella-Béguelin. SoK: Let the privacy games begin! A unified treatment of data inference
privacy in machine learning. In 2023 IEEE Symposium on Security and Privacy (S&P), pp. 327–345. IEEE,
2023. doi:10.1109/SP46215.2023.10179281.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and
Luke Zettlemoyer. Detecting pretraining data from large language models. In 12th International Conference
on Learning Representations (ICLR 2024). OpenReview.net, 2024. URL https://openreview.net/
forum?id=zWqr3MQuNs.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against
machine learning models. In 2017 IEEE symposium on security and privacy (S&P), pp. 3–18. IEEE, 2017.
doi:10.1109/SP.2017.41.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and
Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In
2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), pp. 1631–1642.
ACL, 2013. URL https://aclanthology.org/D13-1170.

Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models. In 25th ACM
SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2019), pp. 196–206.
ACM, 2019. doi:10.1145/3292500.3330885.

Theresa Stadler, Bristena Oprisanu, and Carmela Troncoso. Synthetic data – anonymisation ground-
In 31st USENIX Security Symposium (USENIX Security 22), pp. 1451–1468. USENIX
hog day.
Association, 2022. URL https://www.usenix.org/conference/usenixsecurity22/
presentation/stadler.

Pierre Stock, Igor Shilov, Ilya Mironov, and Alexandre Sablayrolles. Defending against reconstruction
attacks with Rényi differential privacy, 2022. URL https://arxiv.org/abs/2202.07623. arXiv
preprint.

Xinyu Tang, Richard Shin, Huseyin A Inan, Andre Manoel, Fatemehsadat Mireshghallah, Zinan Lin, Sivakanth
Gopi, Janardhan Kulkarni, and Robert Sim. Privacy-preserving in-context learning with differentially
private few-shot generation. In 12th International Conference on Learning Representations (ICLR 2024).
OpenReview.net, 2024. URL https://openreview.net/forum?id=oZtt0pRnOl.

Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. On the importance of diffi-
culty calibration in membership inference attacks. In 10th International Conference on Learning Rep-
resentations (ICLR 2022). OpenReview.net, 2021. URL https://openreview.net/forum?id=
3eIrli0TwQ.

Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia. Proving membership in LLM pretraining data

via data watermarks, 2024. URL https://arxiv.org/abs/2402.10892. arXiv preprint.

Tong Wu, Ashwinee Panda, Jiachen T Wang, and Prateek Mittal. Privacy-preserving in-context learning
for large language models. In 12th International Conference on Learning Representations (ICLR 2024).
OpenReview.net, 2024. URL https://openreview.net/forum?id=x4OPJ7lHVU.

14

Under review as a conference paper at ICLR 2025

Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian
Jiang, Huishuai Zhang, Yin Tat Lee, Bo Li, and Sergey Yekhanin. Differentially private synthetic data via
foundation model APIs 2: Text. In 41st International Conference on Machine Learning (ICML 2024),
volume 235, pp. 54531–54560. PMLR, 2024. URL https://proceedings.mlr.press/v235/
xie24g.html.

Andrew Yale, Saloni Dash, Ritik Dutta, Isabelle Guyon, Adrien Pavao, and Kristin P Bennett. Assessing
privacy and quality of synthetic health data. In Conference on Artificial Intelligence for Data Discovery
and Reuse (AIDR ’19), pp. 1–4. ACM, 2019. doi:10.1145/3359115.3359124.

Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced mem-
bership inference attacks against machine learning models. In 2022 ACM SIGSAC Conference on Computer
and Communications Security (CCS 2022), pp. 3093–3106. ACM, 2022. doi:10.1145/3548606.3560675.

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning:
Analyzing the connection to overfitting. In 31st IEEE Computer Security Foundations Symposium (CSF
2018), pp. 268–282. IEEE, 2018. doi:10.1109/CSF.2018.00027.

Xiang Yue, Huseyin Inan, Xuechen Li, Girish Kumar, Julia McAnallen, Hoda Shajari, Huan Sun, David
Levitan, and Robert Sim. Synthetic text generation with differential privacy: A simple and practical recipe.
In 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
1321–1342. ACL, 2023. doi:10.18653/v1/2023.acl-long.74.

Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko,
Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models.
In 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS 2020), pp. 363–375.
ACM, 2020. doi:10.1145/3372297.3417880.

Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Ahmed Salem, Victor Rühle, Andrew Paverd,
Mohammad Naseri, Boris Köpf, and Daniel Jones. Bayesian estimation of differential privacy. In 40th
International Conference on Machine Learning (ICML 2023), volume 202, pp. 40624–40636. PMLR, 2023.
URL https://proceedings.mlr.press/v202/zanella-beguelin23a.html.

Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. In
41st International Conference on Machine Learning (ICML 2024), volume 235, pp. 58244–58282. PMLR,
2024. URL https://proceedings.mlr.press/v235/zarifzadeh24a.html.

Xiang Zhang, Junbo Jake Zhao, and Yann LeCun.

text classification.
ume 28, 2015.
250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html.

Character-level convolutional networks for
In Advances in Neural Information Processing Systems (NIPS 2015), vol-
URL https://papers.nips.cc/paper_files/paper/2015/hash/

Ziqi Zhang, Chao Yan, and Bradley A Malin. Membership inference attacks against synthetic health data. J.

Biomed. Inform., 125, 2022. doi:10.1016/j.jbi.2021.103977.

15

658
659
660
661

662
663
664
665
666
667

668
669
670
671
672
673

674
675
676
677
678

679
680
681
682
683
684

685
686
687
688
689
690

691
692
693
694
695
696

697
698
699
700
701

702
703
704

Under review as a conference paper at ICLR 2025

A PSEUDO-CODE FOR MIAS BASED ON SYNTHETIC DATA

We here provide the pseudo-code for computing membership signals for both MIA methodologies based on
synthetic data (Sec. 3.1.2), see Algorithm 2 for the n-gram method and Algorithm 3 for the method using
similarity metrics.

Algorithm 2 Compute membership signal using n-gram model

i=1, Target canary ˆx = (ˆs, ˆℓ)

1: Parameter: n-gram model order n
2: Input: Synthetic dataset (cid:101)D = {(cid:101)xi = ((cid:101)si, (cid:101)ℓi)} (cid:101)N
3: Output: Membership signal α
4: C( ⃗w) ← 0 for all (n−1)- and n-grams ⃗w
5: for i = 1 to (cid:101)N do
6:
7:
8:
9:
10: V ← |{w | ∃i.w ∈ (cid:101)si}|
11: The n-gram model is factored into conditional probabilities:

w1, . . . , wk(i) ← (cid:101)si
for each n-gram (wj−(n−1), . . . , wj) in (cid:101)si do

C(wj−(n−1), . . . , wj) += 1
C(wj−(n−1), . . . , wj−1) += 1

▷ Final n-gram model

Pn-gram(wj | wj−(n−1), . . . , wj−1) =

C(wj−(n−1), . . . , wj) + 1
C(wj−(n−1), . . . , wj−1) + V

12: w1, . . . , wk ← ˆs
13: α ← (cid:81)k
14: return α

j=2 Pn-gram(wj | wj−(n−1), . . . , wj−1)

▷ Compute probability of canary text ˆs

Algorithm 3 Compute membership signal using similarity metric

i=1, Target canary ˆx = (ˆs, ˆℓ)

1: Parameter: Similarity metric SIM(·, ·), cutoff parameter k
2: Input: Synthetic dataset (cid:101)D = {(cid:101)xi = ((cid:101)si, (cid:101)ℓi)} (cid:101)N
3: Output: Membership signal α
4: for i = 1 to (cid:101)N do
5:
6: Sort similarities σi for i = 1, . . . , (cid:101)N in descending order
7: Let σi(1), . . . , σi(k) be the top-k similarities
8: α ← 1
k
9: return α

σi ← SIM(ˆs, (cid:101)si)

j=1 σi(j)

(cid:80)k

▷ Compute similarity of each synthetic example

▷ Compute mean similarity of the top-k examples

B COMPUTATION OF RMIA SCORES

We here provide more details on how we adapt RMIA, as originally proposed by Zarifzadeh et al. (2024), to
our setup (see Sec. 3.1.3). In RMIA, the pairwise likelihood ratio is defined as:

LRθ(x, z) =

(cid:18) P (x | θ)
P (x)

(cid:19) (cid:18) P (z | θ)

(cid:19)−1

P (z)

.

(3)

16

705
706
707
708

709
710
711
712
713
714

715
716
717
718
719
720

721
722
723
724
725

726
727
728
729
730
731

732
733
734
735
736
737

738
739
740
741
742
743

744
745
746
747
748

749
750
751

Under review as a conference paper at ICLR 2025

where θ represents the target model, x the target record, and z the reference population. In this work, we only
consider one target model θ and many target records x. As we are only interested in the relative value of the
likelihood ratio across target records, we can eliminate the dependency on the reference population z,

LRθ(x, z) = LRθ(x) =

P (x | θ)
P (x)

.

(4)

As suggested by Zarifzadeh et al. (2024), we compute P (x) as the empirical mean of P (x | θ′) across
reference models {θ′

i}M

i=1,

P (x) =

1
M

M
(cid:88)

i=1

P (x | θ′

i) .

(5)

To compute RMIA scores, we replace the probabilities in (4) by membership signals on target and reference
models:

βθ(x) =

1
M

αθ(x)
i=1 αθ′

i

(cid:80)M

.

(x)

(6)

Note that when we compute αθ(x) as a product of conditional probabilities (e.g. when using the target model
probability in the model-based attack or the n-gram probability in the data-based attack), we truly use a
probability for αθ(x). However, in the case of the data-based attack using similarity metrics, we use the mean
similarity to the k closest synthetic sequences—which does not correspond to a true probability. In this case,
we normalize similarities to fall in the range [0, 1] and use αθ(x) as an empirical proxy for the probability
P (x | θ).

In practice, P (x | θ) can be an extremely small value, particularly when calculated as a product of token-
level conditional probabilities, which can lead to underflow errors. To mitigate this, we perform arithmetic
operations on log-probabilities whenever possible. However, in the context of equation (6), where the
denominator involves averaging probabilities, we employ quad precision floating-point arithmetic. This
method is sufficiently precise to handle probabilities for sequences of up to 50 words, which is the maximum
we consider in our experiments.

C PROMPTS USED TO GENERATE SYNTHETIC DATA

Table 2 summarizes the prompt templates p(ℓ) used to generate synthetic data for both datasets (see Sec. 4).

Dataset

SST-2
AG News

Template p(ℓ)
"This is a sentence with a ℓ sentiment: "
"This is a news article about ℓ: "

Labels ℓ

{positive, negative}
{World, Sport, Business, Sci/Tech}

Table 2: Prompt templates used to fine-tune models and generate synthetic data.

D DETAILED ASSUMPTIONS MADE FOR THE ADVERSARY

We clarify the capabilities of adversaries in model- and data-based attacks according to the threat model
specified in Section 2. We note:

17

752
753
754
755

756
757
758
759
760
761

762
763
764
765
766
767

768
769
770
771
772

773
774
775
776
777
778

779
780
781
782
783
784

785
786
787
788
789
790

791
792
793
794
795

796
797
798

Under review as a conference paper at ICLR 2025

1. A model-based attack is strictly more powerful than a data-based attack. This is because with access
to the fine-tuned model θ and the prompt template p(·), a model-based attack can synthesize (cid:101)D
for any set of synthetic labels and perfectly simulate the membership inference experiment for a
data-based attack.

2. In both threat models, the adversary can train reference models {θ′

i=1. This assumes access to the
private dataset D, and the training procedure of target model θ, including hyperparameters. This is
made clear in line 3 in Algorithm 1.

i}M

3. In our experiments, we consider model-based attacks that use the prompt template p(·) to compute
the model loss for target records, as specified in Sec. 3.1.1. Our data-based attacks use the prompt
template p(·) to generate synthetic data (cid:101)D from reference models.

4. Only the model-based attack has query-access to the target model θ. The attacks used in our
experiments use θ to compute token-level predicted logits for input sequences and do not use
white-box features, although this is not excluded by the threat model.

5. Only the data-based attack generates synthetic data from reference models, so only this threat model

leverages the sampling procedure sample(·).

Table 3 summarizes the adversary capabilities used in the attacks in our experiments.

Assumptions
Knowledge of the private dataset D used to fine-tune the
target model θ (apart from knowledge of canaries).
Knowledge of the training procedure of target model θ.

Knowledge of the prompt template p(ℓi) used to generate
the synthetic data.
Query-access to target model θ, returning predicted logits.

Access to synthetic data (cid:101)D generated by target model θ.
Knowledge of the decoding strategy employed to sample
synthetic data (cid:101)D (e.g., temperature, top-k).

Model-based MIA Data-based MIA

✓

✓

✓

✓

–

–

✓

✓

✓

–

✓

✓

Table 3: Adversary capabilities effectively used by attacks in our experiments.

E SYNTHETIC DATA UTILITY

To ensure we audit the privacy of synthetic text data in a realistic setup, the synthetic data needs to bear
high utility. We measure the synthetic data utility by comparing the downstream classification performance
of RoBERTa-base (Liu et al., 2019) when fine-tuned exclusively on real or synthetic data. We fine-tune
models for binary (SST-2) and multi-class classification (AG News) for 1 epoch on the same number of real or
synthetic data records using a batch size of 16 and learning rate η = 1 × 10−5. We report the macro-averaged
AUC score and accuracy on a held-out test dataset of real records.

Table 4 summarizes the results for synthetic data generated based on original data which does not contain any
canaries. While we do see a slight drop in downstream performance when considering synthetic data instead
of the original data, AUC and accuracy remain high for both tasks.

We further measure the synthetic data utility when the original data contains standard canaries (see Sec. 5.1).
Specifically, we consider synthetic data generated from a target model trained on data containing 500 canaries

18

799
800
801
802

803
804
805
806
807
808

809
810
811
812
813
814

815
816
817
818
819

820
821
822
823
824
825

826
827
828
829
830
831

832
833
834
835
836
837

838
839
840
841
842

843
844
845

Under review as a conference paper at ICLR 2025

Dataset

SST-2

AG News

Fine-tuning data

Real
Synthetic

Real
Synthetic

Classification
AUC Accuracy

0.984
0.968

0.992
0.978

92.3 %
91.5 %

94.4 %
90.0 %

Table 4: Utility of synthetic data generated from real data without canaries. We compare the performance of
text classifiers trained on real or synthetic data—both evaluated on real, held-out test data.

repeated nrep = 12 times, so 6000 data records. When inserting canaries with an artificial label, we remove
all synthetic data associated with labels not present originally when fine-tuning the RoBERTa-base model.

Canary injection

Dataset

Source

Label

Classification
AUC Accuracy

SST-2

AG News

In-distribution

Synthetic

In-distribution

Synthetic

Natural
Artificial

Natural
Artificial

0.972

0.959
0.962

0.978

0.977
0.980

91.6 %

89.3 %
89.9 %

89.8 %

88.6 %
90.1 %

Table 5: Utility of synthetic data generated from real data with canaries (nrep = 12). We compare the
performance of text classifiers trained on real or synthetic data—both evaluated on real, held-out test data.

Table 5 summarizes the results. Across all canary injection methods, we find limited impact of canaries on
the downstream utility of synthetic data. While the difference is minor, the natural canary labels lead to the
largest utility degradation. This makes sense, as the high perplexity synthetic sequences likely distort the
distribution of synthetic text associated with a certain real label. In contrast, in-distribution canaries can be
seen as up-sampling certain real data points during fine-tuning, while canaries with artificial labels merely
reduce the capacity of the model to learn from real data and do not interfere with this process as much as
canaries with natural labels do.

F ADDITIONAL RESULTS FOR MIAS USING STANDARD CANARIES

In line with the literature on MIAs against machine learning models (Carlini et al., 2022a), we also evaluate
MIAs by their true positive rate (FPR) at low false positive rates (FPR). Tables 6 and 7 summarize the MIA
TPR at FPR=0.01 and FPR=0.1, respectively. We also provide the ROC curves for the MIAs for both datasets
(with canary labels randomly sampled from the distribution of labels in real data) in Figure 3.

G ABLATIONS FOR MIAS ON SYNTHETIC DATA

Synthetic multiple Thus far, we have exclusively considered that the number of generated synthetic records
equals the number of records in the real data, i.e., N = (cid:101)N . We now consider the case when more synthetic
data is made available to a data-based adversary ( (cid:101)A). Specifically, we denote the synthetic multiple m = (cid:101)N/N

19

846
847
848
849

850
851
852
853
854
855

856
857
858
859
860
861

862
863
864
865
866

867
868
869
870
871
872

873
874
875
876
877
878

879
880
881
882
883
884

885
886
887
888
889

890
891
892

Under review as a conference paper at ICLR 2025

893
894
895
896

897
898
899
900
901
902

903
904
905
906
907
908

909
910
911
912
913

914
915
916
917
918
919

920
921
922
923
924
925

926
927
928
929
930
931

932
933
934
935
936

937
938
939

Canary injection

Dataset

Source

Label

SST-2

AG News

In-distribution

Synthetic

In-distribution

Synthetic

Natural
Artificial

Natural
Artificial

TPR@FPR=0.01
Synthetic
(SIMJac)
0.029
0.018
0.000

Synthetic
(2-gram)
0.081
0.032
0.049

0.063
0.030
0.071

0.032
0.006
0.041

Model

0.148
0.972
0.968

0.941
0.955
0.990

Synthetic
(SIMemb)
0.020
0.024
0.030

0.016
0.016
0.022

Table 6: True positive rate (TPR) at a false positive rate (FPR) of 0.01 for experiments using standard canaries
(Sec. 5.1) across training datasets, canary injection mechanisms and MIA methodologies. Canaries are
synthetically generated with target perplexity Ptarget = 250 and inserted nrep = 12 times.

Canary injection

Dataset

Source

Label

SST-2

AG News

In-distribution

Synthetic

In-distribution

Synthetic

Natural
Artificial

Natural
Artificial

TPR@FPR=0.1

Synthetic
(2-gram)
0.335
0.209
0.268

Synthetic
(SIMJac)
0.207
0.114
0.142

0.200
0.260
0.298

0.158
0.114
0.152

Model

0.795
0.996
1.000

0.982
0.990
0.996

Synthetic
(SIMemb)
0.203
0.128
0.142

0.168
0.114
0.164

Table 7: True positive rate (TPR) at a false positive rate (FPR) of 0.1 for experiments using standard canaries
(Sec. 5.1) across training datasets, canary injection mechanisms and MIA methodologies. Canaries are
synthetically generated with target perplexity Ptarget = 250 and inserted nrep = 12 times.

and evaluate how different MIAs perform for varying values of m. Figure 4 shows how the ROC AUC score
varies as m increases. As expected, the ROC AUC score for the attack that uses membership signals computed
using a 2-gram model trained on synthetic data increases when more synthetic data is available. In contrast,
attacks based on similarity metrics do not seem to benefit significantly from this additional data.

Hyperparameters in model-based attacks The model-based attacks that we presented in Sec. 3.1 have
hyperparameters. The attack that uses n-gram models to compute membership signals is parameterized by
the order n. Using a too small value for n might not suffice to capture the information leaked from canaries
into the synthetic data used to train the n-gram model. When using a too large order n, on the other hand,
we would expect less overlap between n-grams present in the synthetic data and the canaries, lowering the
membership signal.

Further, the similarity-based methods rely on the computation of the mean similarity of the closest k synthetic
records to the a canary. When k is very small, e.g. k = 1, the method takes into account a single synthetic
record, potentially missing on leakage of membership information from other close synthetic data records.
When k becomes too large, larger regions of the synthetic data in embedding space are taken into account,
which might dilute the membership signal among the noise.

20

Under review as a conference paper at ICLR 2025

1

0.8

0.6

0.4

0.2

R
P
T

0

0

0.2

0.4

FPR

(a) SST-2

2-gram
SIMemb - k = 25
SIMjac - k = 25
0.6
0.8

1

1

0.8

0.6

0.4

0.2

R
P
T

0

0

0.2

0.4

2-gram
SIMemb - k = 25
SIMjac - k = 25
0.6
0.8

1

FPR

(b) AG News

Figure 3: MIA ROC curves across MIA methodologies for the SST-2 (left) and AG News (right) datasets.
Canaries are synthetically generated with target perplexity of Ptarget = 250 with a natural label and inserted
nrep = 12 times.

Figure 4: ROC AUC score for increasing value of the synthetic multiple m across model-based attack
methods for SST-2 (left) and AG News (right). Canaries are synthetically generated with target perplexity of
Ptarget = 250, with a natural label, and inserted nrep = 12 times.

Table 8 reports the ROC AUC scores of model-based attacks for different values of the hyperparameters n
and k when using standard canaries (Sec. 5.1).

H DISPARATE VULNERABILITY OF STANDARD CANARIES

We analyze the disparate vulnerability of standard canaries between the model-based attack and the data-based
attack that uses a 2-gram model (as discussed in Sec 5.1). Figure 5 plots the RMIA scores for both attacks on
the same set of canaries, which have either been included in the training dataset of the target model (member)

21

940
941
942
943

944
945
946
947
948
949

950
951
952
953
954
955

956
957
958
959
960

961
962
963
964
965
966

967
968
969
970
971
972

973
974
975
976
977
978

979
980
981
982
983

984
985
986

20212223Synthetic multiple m0.50.60.70.80.91.0MIA AUCModelSynthetic (2-gram)Synthetic (SIMjac - k=25)Synthetic (SIMemb - k=25)Random guess baseline20212223Synthetic multiple m0.50.60.70.80.91.0MIA AUCModelSynthetic (2-gram)Synthetic (SIMjac - k=25)Synthetic (SIMemb - k=25)Random guess baselineUnder review as a conference paper at ICLR 2025

987
988
989
990

991
992
993
994
995
996

997
998
999
1000
1001
1002

1003
1004
1005
1006
1007

1008
1009
1010
1011
1012
1013

1014
1015
1016
1017
1018
1019

1020
1021
1022
1023
1024
1025

1026
1027
1028
1029
1030

1031
1032
1033

Dataset

SST-2

AG News

n-gram
AUC
0.415
0.616
0.581
0.530

0.603
0.644
0.567
0.527

n

1
2
3
4

1
2
3
4

SIMJac
AUC
0.520
0.535
0.538
0.547

0.522
0.525
0.537
0.552

k

1
5
10
25

1
5
10
25

SIMemb
AUC
0.516
0.516
0.519
0.530

0.503
0.498
0.503
0.506

k

1
5
10
25

1
5
10
25

Table 8: Ablation over hyperparameters of model-based MIAs. We report ROC AUC scores across different
values of the hyperparameters n and k (see Sec. 3.1). Canaries are synthetically generated with target
perplexity Ptarget = 250, with a natural label, and inserted nrep = 12 times.

or not (non-member). Note that the RMIA scores are used to distinguish members from non-members, and
that a larger value corresponds to the adversary being more confident in identifying a record as a member, i.e.,
to the record being more vulnerable.

First, we note that the scores across both threat models exhibit a statistically significant, positive correlation.
We find a Pearson correlation coefficient between the RMIA scores (log) for both methods of 0.20 (p-value of
2.4 × 10−10) and 0.23 (p-value of 1.9 × 10−13) for SST-2 and AG News, respectively. This means that a
record vulnerable to the model-based attack tends to be also vulnerable to the data-based attack, even though
the attacks differ substantially.

Second, and more interestingly, some canaries have disparate vulnerability across MIA methods. Indeed,
Figure 5 shows how certain data records which are not particularly vulnerable to the model-based attack are
significantly more vulnerable to the data-based attack, and vice versa.

I LOW FPR ROC RESULTS

Figure 6 shows log-log plots of the ROC curves in Figure 1 to better examine behavior of attacks at low FPR.

J DETERMINING OPTIMAL HYPERPARAMETERS

We optimized hyperparameters for LoRA fine-tuning Mistral-7B on SST-2 by running a grid search over
learning rate ([1 × 10−6, 4 × 10−6, 2 × 10−5, 6 × 10−5, 3 × 10−4, 1 × 10−3]) and batch size ([64, 128,
256]). We fine-tuned the models for 3 epochs and observed the validation loss plateaued after the first epoch.
Based on these results, we selected a learning rate of 2 × 10−5, effective batch size of 128, sequence length
128, LoRA r = 4 and fine-tuned the models for 1 epoch, as stated in Sec. 7. Figure 7 shows the validation
cross-entropy loss for SST-2 over the grid we searched on and the train and validation loss curves for 3 epochs
with the selected hyperparameters.

22

Under review as a conference paper at ICLR 2025

1034
1035
1036
1037

1038
1039
1040
1041
1042
1043

1044
1045
1046
1047
1048
1049

1050
1051
1052
1053
1054

1055
1056
1057
1058
1059
1060

1061
1062
1063
1064
1065
1066

1067
1068
1069
1070
1071
1072

1073
1074
1075
1076
1077

1078
1079
1080

(a) SST-2

(b) AG News

Figure 5: RMIA scores (log) for model- and data-based MIAs on the same set of canaries. Results for both
datasets SST-2 and AG News. Canaries are synthetically generated with target perplexity of Ptarget = 250
with a natural label, and inserted nrep = 12 times.

K INTERPRETABILITY

K.1

IDENTIFYING MEMORIZED SUB-SEQUENCES

We analyze what information from a canary leaks into the synthetic data that enables a data-based attack
to infer its membership. For each canary ˆx = (ˆs, ˆℓ), we examine the synthetic data generated by a model
trained on a dataset including (member) and excluding ˆx (non-member). We leverage the M = 4 reference
models θ′ used to develop the attack for 1000 specialized canaries from Fig. 1(c). For each model θ′, we
count the number of n-grams in (cid:101)s that occur at least once in (cid:101)D′ (Cunique). We also compute the median
Cmed and average Cavg counts of n-grams from ˆs in (cid:101)D′. Table 9 summarizes how these measures vary with
n. As n increases, the number of n-grams from the canary appearing in the synthetic data drops sharply,
reaching Cmed = 0 for n = 4 for models including and excluding a canary. This suggests that any verbatim
reproduction of canary text in the generated synthetic data is of limited length. Further, we observe only slight
differences in counts between members and non-members, indicating that the signal for inferring membership
is likely in subtle shifts in the probability distribution of token co-occurrences within the synthetic data, as
captured by the 2-gram model. We further analyze canaries with the highest and lowest RMIA scores below.

K.2

INTERPRETABILITY OF RMIA SCORES

To further understand the membership signal for data-based attacks, we examine some examples in-depth.

Specifically, we consider the MIA for specialized canaries with F = 30, Ptarget = 31 and nrep = 16 for SST-2
from Figure 1(c). Recall that for this attack, we consider 1000 canaries, 500 of which are injected into the
training dataset of one target model θ. We also train 4 references models {θ′
i=1 where each of the 1000
canaries has been included in exactly half. We focus on the best performing MIA based on synthetic data, i.e.

i}4

23

80604020020RMIA scores (log) - Model - AUC=0.99935.032.530.027.525.022.520.017.5RMIA scores (log) - Synthetic (2-gram) AUC=0.616MembersNon-membersCorrelation=0.1990204060025125100755025025RMIA scores (log) - Model - AUC=0.99912.510.07.55.02.50.02.55.0RMIA scores (log) - Synthetic (2-gram) AUC=0.644MembersNon-membersCorrelation=0.2300255075020Under review as a conference paper at ICLR 2025

1081
1082
1083
1084

1085
1086
1087
1088
1089
1090

1091
1092
1093
1094
1095
1096

1097
1098
1099
1100
1101

1102
1103
1104
1105
1106
1107

1108
1109
1110
1111
1112
1113

1114
1115
1116
1117
1118
1119

1120
1121
1122
1123
1124

1125
1126
1127

100

10−1

R
P
T

10−2

100

10−1

R
P
T

10−2

A (cid:101)D, nrep = 2
A (cid:101)D, nrep = 4
A (cid:101)D, nrep = 8
A (cid:101)D, nrep = 16
Aθ, nrep = 1
Aθ, nrep = 2
Aθ, nrep = 4

10−2

10−1

FPR

100

(a) Number of canary repetitions nrep.
Ptarget = 31, F = 0.

100

10−1

R
P
T

10−2

A (cid:101)D, Ptar = 10
A (cid:101)D, Ptar = 102
A (cid:101)D, Ptar = 103
A (cid:101)D, Ptar = 104
Aθ, Ptar = 10
Aθ, Ptar = 102
Aθ, Ptar = 103
Aθ, Ptar = 104

A (cid:101)D, F = 0
A (cid:101)D, F = 10
A (cid:101)D, F = 20
A (cid:101)D, F = 30
Aθ, F = 0
A (cid:101)D, F = max

10−2

10−1

FPR

100

10−2

10−1

FPR

100

(b) Canary perplexity Ptarget.
rep = 4, n (cid:101)D
nθ
rep = 16, F = 0.

100

(c) Canary in-distribution prefix F .
Ptarget = 31, nθ

rep = 4, n (cid:101)D

rep = 16.

100

100

10−1

R
P
T

10−2

10−1

R
P
T

10−2

A (cid:101)D, nrep = 2
A (cid:101)D, nrep = 4
A (cid:101)D, nrep = 8
A (cid:101)D, nrep = 16
Aθ, nrep = 1
Aθ, nrep = 2
Aθ, nrep = 4

10−2

10−1

FPR

100

A (cid:101)D, Ptar = 10
A (cid:101)D, Ptar = 102
A (cid:101)D, Ptar = 103
A (cid:101)D, Ptar = 104
Aθ, Ptar = 10
Aθ, Ptar = 102
Aθ, Ptar = 103
Aθ, Ptar = 104

10−1

R
P
T

10−2

A (cid:101)D, F = 0
A (cid:101)D, F = 10
A (cid:101)D, F = 20
A (cid:101)D, F = 30
Aθ, F = 0
A (cid:101)D, F = max

10−2

10−1

FPR

100

10−2

10−1

FPR

100

(d) Number of canary repetitions nrep.
Ptarget = 31, F = 0.

(e) Canary perplexity Ptarget.
nθ
rep = 4, n (cid:101)D
rep = 16, F = 0.

(f) Canary in-distribution prefix F .
Ptarget = 31, nθ

rep = 4, n (cid:101)D

rep = 16.

Figure 6: Log-log ROC curves of MIAs on synthetic data A (cid:101)D compared to model-based MIAs Aθ on SST-2
((a)–(c)) and AG News ((d)–(f)). We ablate over the number of canary insertions nrep in (a), (d), the target
perplexity Ptarget of the inserted canaries in (b), (e) and the length F of the in-distribution prefix in the canary
in (c), (f).

the attack leveraging the probability of the target sequence computed using a 2-gram model trained on the
synthetic data.

Cunique
Member Non-member

Cmed

Cavg

Member

Non-member

Member

Non-member

46.1 ± 2.5
29.6 ± 5.7
4.8 ± 3.6
0.1 ± 0.6

45.2 ± 2.8
28.1 ± 5.7
3.9 ± 3.2
0.0 ± 0.3

882.9 ± 756.3
5.2 ± 6.6
0.0 ± 0.0
0.0 ± 0.0

884.2 ± 771.8
4.2 ± 6.3
0.0 ± 0.0
0.0 ± 0.0

7391.0 ± 1892.23
202.9 ± 118.0
1.4 ± 2.8
0.0 ± 0.0

7382.7 ± 1887.1
199.6 ± 116.6
1.2 ± 2.6
0.0 ± 0.0

n

1
2
4
8

Table 9: Aggregate count statistics of n-grams in a canary ˆs that also appear in the synthetic data (cid:101)D′ generated
using 4 reference models including and excluding ˆs. Number of n-grams in (cid:101)s that also appear in (cid:101)D′ (Cunique),
median (Cmed) and average (Cavg) counts of n-grams from ˆs in (cid:101)D′. We report mean and std. deviation of
these measures over all canaries (F = 30, Ptarget = 31, nrep = 16) for SST-2. Each canary ˆs contains exactly
50 words and (cid:101)D′ contains 706.7k ± 72.8k words.

24

Under review as a conference paper at ICLR 2025

(a) Grid search

(b) Loss curve

Figure 7: (a) Validation cross-entropy loss of LoRA fine-tuning Mistral-7B on SST-2 varying the learning
rate and effective batch size. (b) Training and validation loss for best hyperparameters over 3 epochs.

To understand what signal the MIA picks up to infer membership, we focus on the canary most confidently—
and correctly—identified as member and the one most confidently—and correctly—identified as non-member.
For this, we take the canaries for which the RMIA score computed using the target model and the reference
models is the highest and the lowest, respectively.

Next, for each model (4 reference models, and 1 target model), we report for this canary ˆxi:

1. Whether the canary has been included in, ˆxi ∈ D (IN), or excluded from, ˆxi /∈ D (OUT), the training
dataset of the model in question, and thus to generate the synthetic data (cid:101)D = {(cid:101)xi = ((cid:101)si, (cid:101)ℓi)} (cid:101)N
2. The canary with the words that appear as a 2-gram in the synthetic data (cid:101)D emphasized in bold face.
Note that if, for instance, this is a sequence of 3 words, e.g., "like many western", this means that all
3 words appear in 2-grams in the synthetic data, e.g., "like many" and "many western".

i=1.

3. The maximum overlapping sub-string between the canary and any synthetically generated record (cid:101)si.
We define a sub-string as a sequence of characters, including white space, and also report its length
as number of characters Loverlap.

4. The mean, negative cross-entropy loss of the canary computed using the 2-gram model trained on
j=2 log (P2-gram(wj, wj−1)).

the synthetic data. Formally, for canary ˆsi = (w1, w2, . . . , wk): − 1
k

(cid:80)k

Tables 10 and 11 report this for the canary with the largest and lowest RMIA score, respectively.

First, we observe that not all the words in the canary appear as 2-grams in the synthetic dataset. This could be
expected, as not all 2-grams are commonly used in general English (e.g. "penetrating views"). Notably, the
number of common 2-grams does not significantly differ whether the canary is a member or not (IN or OUT).

In addition, we observe similar trends when considering the longest overlapping sub-string between the
canary and the synthetic data. Across all models and canaries, this sub-string remains consistently short and
shows little variation with membership labels. This suggests that the signal used to infer membership does
not rely on the verbatim regurgitation of long sub-sequences.

Lastly, we investigate whether the reported 2-gram loss is consistent with the fact that these canaries
correspond to the largest and lowest RMIA scores. Although the losses across models differ only slightly, the
relative values align with the RMIA scores. Recall that RMIA scores are intuitively computed as the ratio of

25

1128
1129
1130
1131

1132
1133
1134
1135
1136
1137

1138
1139
1140
1141
1142
1143

1144
1145
1146
1147
1148

1149
1150
1151
1152
1153
1154

1155
1156
1157
1158
1159
1160

1161
1162
1163
1164
1165
1166

1167
1168
1169
1170
1171

1172
1173
1174

Under review as a conference paper at ICLR 2025

the membership signal of the target model to the average membership signal across reference models. For the
canary with the highest RMIA score, the 2-gram loss of the target model is lower than the average loss of the
reference models, suggesting that the canary was seen by the target model. Conversely, for the canary with
the lowest RMIA score, the 2-gram loss is higher than the average loss across reference models.

These results suggest that the information required to infer membership based on synthetic data does not lie in
the explicit generation of canary sub-strings within the synthetic data. Instead, the signal seems more subtle,
arising from slight shifts in the probability distribution of co-occurrences of words in the synthetic data.

1175
1176
1177
1178

1179
1180
1181
1182
1183
1184

1185
1186
1187
1188
1189
1190

1191
1192
1193
1194
1195

1196
1197
1198
1199
1200
1201

1202
1203
1204
1205
1206
1207

1208
1209
1210
1211
1212
1213

1214
1215
1216
1217
1218

1219
1220
1221

26

Under review as a conference paper at ICLR 2025

1222
1223
1224
1225

1226
1227
1228
1229
1230
1231

1232
1233
1234
1235
1236
1237

1238
1239
1240
1241
1242

1243
1244
1245
1246
1247
1248

1249
1250
1251
1252
1253
1254

1255
1256
1257
1258
1259
1260

1261
1262
1263
1264
1265

1266
1267
1268

Model
θ′
1 (ref)

IN or
OUT

IN

θ′
2 (ref)

IN

θ′
3 (ref)

OUT

θ′
4 (ref)

OUT

θ (target)

IN

Canary
(words present as part of 2-grams in (cid:101)D′ in bold)
"like many western action films , this thriller is
too loud and thoroughly overbearing , but its
heartfelt concern about north korea ’s recent past
and south korea ’s future, its sophisticated sense
of character and its penetrating views on many
social and political issues, like the exploitation of
single"

"like many western action films , this thriller
is too loud and thoroughly overbearing , but its
heartfelt concern about north korea ’s recent
past and south korea ’s future, its sophisticated
sense of character and its penetrating views on
many social and political issues, like the exploita-
tion of single"

"like many western action films , this thriller is
too loud and thoroughly overbearing , but its
heartfelt concern about north korea ’s recent past
and south korea ’s future, its sophisticated sense
of character and its penetrating views on many
social and political issues, like the exploitation
of single"

"like many western action films , this thriller is
too loud and thoroughly overbearing , but its
heartfelt concern about north korea ’s recent past
and south korea ’s future, its sophisticated sense
of character and its penetrating views on many
social and political issues, like the exploitation
of single"

"like many western action films , this thriller is
too loud and thoroughly overbearing , but its
heartfelt concern about north korea ’s recent
past and south korea ’s future, its sophisticated
sense of character and its penetrating views on
many social and political issues, like the exploita-
tion of single"

Max overlapping
sub-string

« social and political
issues » ;
Loverlap = 28

2-gram
loss

17.96

« sense of character
and » ; Loverlap = 24

18.40

« sophisticated sense
of » ; Loverlap = 24

18.30

« sense of character
and » ; Loverlap = 24

17.93

« sense of character
and » ; Loverlap = 24

17.65

Table 10: Interpretability of the best MIA (2-gram) based on synthetic data for specialized canaries with
F = 30, Ptarget = 31 and nrep = 16 for SST-2 from Figure 1(c). Results across 4 reference models and
the target model for the canary with the largest RMIA score (most confidently and correctly identified as
member by the MIA). Words in bold appear in 2-grams in (cid:101)D′. The largest generated sub-sequence of the
canary in (cid:101)D′ corresponds to the maximum overlapping sub-string, not the longest sequence of words in bold.

27

Under review as a conference paper at ICLR 2025

1269
1270
1271
1272

1273
1274
1275
1276
1277
1278

1279
1280
1281
1282
1283
1284

1285
1286
1287
1288
1289

1290
1291
1292
1293
1294
1295

1296
1297
1298
1299
1300
1301

1302
1303
1304
1305
1306
1307

1308
1309
1310
1311
1312

1313
1314
1315

Model
θ′
1 (ref)

IN or
OUT

IN

θ′
2 (ref)

IN

θ′
3 (ref)

OUT

θ′
4 (ref)

OUT

θ (target)

OUT

Canary
(words present as part of 2-grams in (cid:101)D′ in bold)
"the star who helped give a spark to “ chasing
amy ” and “ changing lanes ” falls flat as think-
ing man cia agent jack ryan in this summer ’s
big-budget action drama, “ the hunt for red octo-
ber ” (1990). At the time, bullet time was used to
prolong"

"the star who helped give a spark to “ chasing
amy ” and “ changing lanes ” falls flat as think-
ing man cia agent jack ryan in this summer ’s
big-budget action drama, “ the hunt for red octo-
ber ” (1990). At the time, bullet time was used to
prolong"

"the star who helped give a spark to “ chasing
amy ” and “ changing lanes ” falls flat as thinking
man cia agent jack ryan in this summer ’s big-
budget action drama, “ the hunt for red october
” (1990). At the time, bullet time was used to
prolong"

"the star who helped give a spark to “ chasing
amy ” and “ changing lanes ” falls flat as think-
ing man cia agent jack ryan in this summer ’s
big-budget action drama, “ the hunt for red octo-
ber ” (1990). At the time, bullet time was used to
prolong"

"the star who helped give a spark to “ chasing
amy ” and “ changing lanes ” falls flat as thinking
man cia agent jack ryan in this summer ’s big-
budget action drama, “ the hunt for red october
” (1990). At the time, bullet time was used to
prolong"

Max overlapping
sub-string

« the hunt for red
october » ;
Loverlap = 26

2-gram
loss

18.12

« ” and “
changing lanes ”
» ; Loverlap = 29

18.41

« “ chasing amy ”
» ; Loverlap = 19

19.04

« ” and “
changing lanes ”
» ; Loverlap = 29

18.29

« “ chasing amy ”
» ; Loverlap = 19

18.85

Table 11: Interpretability of the best MIA (2-gram) based on synthetic data for specialized canaries with
F = 30, Ptarget = 31 and nrep = 16 for SST-2 from Figure 1(c). Results across 4 reference models and
the target model for the canary with the smallest RMIA score (most confidently and correctly identified as
non-member by the MIA). Words in bold appear in 2-grams in (cid:101)D′. The largest generated sub-sequence of the
canary in (cid:101)D′ corresponds to the maximum overlapping sub-string, not the longest sequence of words in bold.
.

28

