# FLTrojan: Privacy Leakage Attacks against Federated Language Models through
Selective Weight Tampering

Md Rafi Ur Rashid
Penn State University  Vishnu Asutosh Dasu
Penn State University  Kang Gu
Dartmouth College  Najrin Sultana
Penn State University  Shagufta Mehnaz
Penn State University

###### Abstract

Federated learning (FL) is becoming a key component in many technology-based
applications including language modeling– where individual FL participants
often have privacy-sensitive text data in their local datasets. However,
realizing the extent of privacy leakage in federated language models is not
straightforward and the existing attacks only intend to extract data
regardless of how sensitive or naive it is. To fill this gap, in this paper,
we introduce two novel findings with regard to leaking privacy-sensitive user
data from federated language models. Firstly, we make a key observation that
model snapshots from the intermediate rounds in FL can cause greater privacy
leakage than the final trained model. Secondly, we identify that privacy
leakage can be aggravated by tampering with a model’s selective weights that
are specifically responsible for memorizing the sensitive training data. We
show how a malicious client can leak the privacy-sensitive data of some other
user in FL even without any cooperation from the server. Our best-performing
method improves the membership inference recall by 29% and achieves up to 70%
private data reconstruction, evidently outperforming existing attacks with
stronger assumptions of adversary capabilities.

##  1 Introduction

Large language models (LLMs) are playing a pivotal role in various fundamental
natural language processing (NLP) tasks[9, 29, 31, 63]. Recently, these LLMs
have been trained on extremely large text corpora[19, 48, 60] and grown to
billions of parameters [5, 67]. These pre-trained LLMs are then adapted to
various downstream tasks by fine-tuning on domain-specific datasets, often on
privacy-sensitive datasets such as in-house emails [26], code
repositories111https://huggingface.co/datasets/bigcode/bigcode-pii-dataset,
and clinical health data [27], etc. Given the privacy-sensitive nature of such
datasets, collecting such data by a centralized entity to fine-tune an LLM
poses an unprecedented level of privacy concern.

Federated learning (FL) [61, 25, 64, 66, 42] has emerged as the field of core
technologies aimed at training machine learning (ML) models by exchanging only
model updates without the need for a central entity to collect all the data.
However, recent studies [70, 20, 76] have shown that carefully crafted attacks
can invert these model updates sent by FL clients and recover private
information. To make matters worse, large LLMs tend to output complete
sequences verbatim from their training data[8, 6], which by itself is a major
privacy concern. Hence, given the increasing use of FL technologies in
language models, it is now of paramount importance to investigate the
implications of privacy leakage in such settings.

Most prior work investigating the privacy of language models focus on
centralized settings [8, 7, 22] and/or rely on external adversaries [23, 74].
However, the knowledge of privacy implications in centralized learning
scenarios often does not transfer to federated learning due to the fundamental
differences in their training protocols. For instance, in a centralized
setting, the training process _sees_ all the data points in every epoch. In
contrast, federated learning uses asynchronous aggregation [52] where the
server selects a subset of clients with a fraction of data in each round of
training. The only few works [17, 23, 3] that focus on privacy attacks in
federated language models aim to extract data indiscriminately as opposed to
devising attacks to extract privacy-sensitive data of the FL users. To fill
this research gap, in this paper, we focus on the leakage of individual user’s
private data when they collaboratively train language models using FL
technologies. More specifically, we design novel privacy attacks that
maximally exploit the adversary’s capability to leak FL clients’ private data–
generating novel insights on the extent of such attacks.

Understanding the extent of privacy leakage in FL-based language models is
complex. This is because previous research on forgetting analysis [28] asserts
that deep language models tend to forget the early training samples gradually
if those are not repeated much afterward. Additionally, as seen in studies on
catastrophic forgetting [18, 45, 30] during iterative training, models could
lose information about earlier samples as they see new training samples. Since
only a subset of the clients participate in each round of FL, the above
observations are inherently true for FL settings. In other words, the FL model
tends to forget the intricate details of the local datasets of individual
clients in successive rounds, especially those with less participation in the
training. Moreover, the federated averaging (FedAvg) algorithm [46]
generalizes the updates of all the participating clients, thereby reducing the
impact of memorization of a particular client.

In contrast with regular text data, privacy-sensitive texts are often out-of-
distribution in nature and occur less frequently in the training data [32,
43]. FedAvg also reduces the memorization of such data points as the training
heads toward convergence [65]. These observations imply that the language
model tends to forget any individual’s local data as training proceeds and
tries to generalize more on the overall distribution of the entire training
corpus. This makes it challenging for an attacker to leak a client’s private
data in FL from the final model. Therefore, in our attack design, we emphasize
the model snapshots from intermediate rounds of FL where the victim
participates in training with their private data. We show how a malicious
participant can cause leakage of some other victim participants’ sensitive
data in federated learning with and without the cooperation of the server by
maximizing the memorization of those intermediate model snapshots.

Besides exploiting this intrinsic vulnerability of the FL configuration in our
attack design, we further investigate how to enhance the model’s memorization
of privacy-sensitive training data. The proposed methods stem from our crucial
observation that some weights of an LLM are more susceptible to sensitive data
points than others. Specifically, the model heads and nearby layers in the
transformer blocks have the strongest influence [51]. Therefore, modifying
these selective weights of an LLM can effectively increase its memorization of
privacy-sensitive data. Based on this inspection, we designed two methods for
tuning the model weights and piloting them toward stronger memorization. The
first one is called _selective weights optimization (SWO)_ , which takes the
difference of selective weights between a fine-tuned (with privacy-sensitive
data) model and another reference model into account and maximizes this
difference. The second approach, named _weights transformation learning (WTL)_
, utilizes an end-to-end supervised algorithm to learn the hidden pattern of
how those selective weights mutate from a reference model to a fine-tuned
model. To the best of our knowledge, this is the first work that aims to
maximize data memorization of LM to increase potential privacy leakage by
tampering with its weights.

Our Contributions: Our main contributions are summarized as follows:

✦ We propose a novel attack technique that exploits model snapshots from
intermediate rounds in FL to leak the privacy-sensitive data of the users.

✦ This is the very first work that maximizes the model’s memorization of
privacy-sensitive data by tampering with its selective weights that are
responsible for such memorization.

✦ Our attacks consider a comprehensive threat model covering different
adversary capabilities.

✦ Our proposed attacks strengthen both membership inference and reconstruction
attacks on private client data and significantly outperform existing work.

✦ We perform an extensive evaluation of our attacks considering both auto-
regressive and masked language modeling tasks. We use artificial privacy-
sensitive canaries as well as real-life private data.

##  2 Preliminaries

###  2.1 Federated Learning

Federated learning (FL) proposed by McMahan et al. [47] proposes an algorithm
for training machine learning models in a distributed setting. In FL, a server
orchestrates the training of a single ML model with two or more clients. Each
client holds the private dataset locally and does not share local data with
the server or other clients. All the clients receive an ML model from the
server, perform gradient descent locally using their private datasets, and
send the updated model back to the server. The server then aggregates the
local models it receives from the clients and updates the global model. This
process repeats until convergence or a fixed set of epochs is complete.
Mathematically, the FL aggregation can be represented as follows

$\scriptsize\theta=\dfrac{1}{d}\big{(}d_{1}\theta^{1}+\ldots+d_{n}\theta^{n}\big{)}$
(1)

In Equation 1, $d$ is the total number of data points in all the clients’
datasets, $\theta^{i}$ is the local model of client $i$, and $d_{i}$ is the
number of data points in client $i$’s dataset.

###  2.2 Language Models

Language models are neural networks that generate sequences of text. They
generate words by predicting the likelihood of the next word in a sequence
given a sequence of already generated or provided words. Given a sequence of
tokens, the model learns the following distribution

$\scriptsize\mathbf{Pr}(x_{1},\ldots,x_{n})=\prod_{i=1}^{n}\mathbf{Pr}(x_{i}|x_{1},\ldots,x_{n})$
(2)

The distribution in Equation 2 is estimated using a neural network $\theta$.
During training, we minimize the following loss function

$\scriptsize\mathcal{L}(\theta)=-\log\prod_{i=1}^{n}\theta(x_{i}|x_{1},\ldots,x_{i-1})$
(3)

During inference, given a prefix sequence $x_{1},\ldots,x_{i}$ the language
model samples $\widehat{x}_{i+1}\sim\theta(x_{i+1}|x_{1},\ldots,x_{i})$. This
process is repeated by feeding the updated sequence with $\widehat{x}_{i+1}$
back into the model until a special STOP token is generated. Perplexity is a
metric used to measure the performance of language models. Intuitively, it
measures how well the language model predicts tokens in the generated
sentence. For a sequence of predicted tokens $x_{1},\ldots,x_{n}$, the
perplexity is computed as

$\scriptsize\mathcal{P}=\exp\left(-\frac{1}{n}\sum_{i=1}^{n}\log\theta(x_{i}|x_{1},\ldots,x_{i-1})\right)$
(4)

where $\theta$ is the language model. A low perplexity implies that the model
is confident in its generation and is less “surprised” by the generated text.

##  3 Impact of Privacy-Sensitive Data on Fine-tuning Large Language Models
(LLM)

In order to carry out different downstream applications, large language models
(LLMs) such as GPT-2 [59] and BERT [14] undergo a fine-tuning step that
modifies their pre-trained weights according to those specific tasks [11].
However, privacy-sensitive texts have some specific patterns and often contain
out-of-distribution tokens, e.g., digits (in phone or credit card numbers),
special characters, and out-of-vocabulary words (login credentials or physical
addresses). To understand the impact of such privacy-sensitive data on fine-
tuning, we investigate whether the changes in the pre-trained weights after
fine-tuning the LLMs with such sensitive data show any differentiating
patterns when compared to the weight changes after fine-tuning with regular
English texts. Our key observation from this investigation is that there
indeed exists a remarkable difference between the patterns in these two cases.
Figure 1(a) depicts the norms of weight changes in all 12 transformer blocks,
i.e., multi-layer perception (MLP) and self-attention layers after fine-tuning
a GPT-2 (base) model, $G$ with regular English sentences, $\mathcal{D}_{reg}$.
Here, we can observe a uniform impact on the weights across different layers,
$L$, which suggests that the model’s adaptation after fine-tuning is evenly
distributed throughout its architecture.

However, a distinct pattern emerges when fine-tuning $G$ with privacy-
sensitive texts $\mathcal{D}_{sen}$, such as phone numbers, email addresses,
user IDs, passwords, credit card numbers, etc. Specifically, only a subset of
layers, denoted as $L_{\star}\subset L$, undergo a substantial transformation,
while the remaining layers, denoted as $L_{\ominus}=L\setminus L_{\star}$,
exhibit minimal changes. Figure 1(b) shows the transformation of the MLP and
self-attention layers of GPT-2 after fine-tuning with $D_{sen}$. Formally,
after fine-tuning with privacy-sensitive data, if $\Delta W_{\star}$ and
$\Delta W_{\ominus}$ represent the weight changes in respective layers, they
are defined as below.

$\Delta W_{\star}=(W(f_{G}(\mathcal{D}_{sen}))-W(f_{G}(\cdot)))|_{L_{\star}}$

$\Delta
W_{\ominus}=(W(f_{G}(\mathcal{D}_{sen}))-W(f_{G}(\cdot)))|_{L_{\ominus}}$

Here, $(W(f_{G}(D))$ denotes the weights of model $G$ fine-tuned with dataset
$D$. we observe that $\|\Delta W_{\star}\|>>\|\Delta W_{\ominus}\|$. In
contrast, when fine-tuning with regular texts, we observe $\|\Delta
W_{\star}\|\sim\|\Delta W_{\ominus}\|$. It is pertinent to mention that the
embedding layer of an LM is solely responsible for converting input tokens
into vectors with positional information, while the other components,
including layer normalization, do not actively impact the fine-tuning [59].
Hence, we only concentrate on the MLP and self-attention layers of the
transformer blocks.

(a)

(b)

Figure 1: Norms of weight changes in MLP and self-attention layers of all 12
transformer blocks after fine-tuning GPT-2 with (a) regular English texts and
(b) privacy-sensitive texts

This observation indicates that certain layers ($L_{\star}$ marked with the
green rectangle in Figure 1(a) and Figure1(b)) within an LLM are more
susceptible to capturing and memorizing privacy-sensitive information.
Leveraging this insight, we can strategically focus on selectively modifying
the weights of these sensitive layers to enhance the model’s ability to
memorize private data effectively. In other words, the adversary aims to find
a function $\mathcal{T}:W_{\star}\rightarrow W^{\prime}_{\star}$ that
generates a new set of weights $W^{\prime}_{\star}$ to maximize the
memorization of $\mathcal{D}_{\text{sen}}$.

$\footnotesize\underset{\mathcal{T}}{\operatorname*{arg\,max}}~{}\mathbf{M}(\mathcal{D}_{\text{sen}},\mathcal{T}(W_{\star}))$
(5)

We qualitatively define memorization $\mathbf{M}$ as a property of the
language model in exposing particular training data points during inference.
Quantitative measure of memorization $\mathbf{M}$ on
$\mathcal{D_{\text{sen}}}$ includes but are not limited to data
reconstruction, membership inference, and exposure, which we discuss in detail
in Section 4.5.

| Attack
---
Attributes
| Server
---
Cooperation
| Adversary’s
---
Action
| MDM
---
Method
Victim |  | Type of
---
LM
Goal
Variations | No | Static | SWO | Targeted |  | Autoregressive
---
(GPT-2)
| Data
---
Reconstruction
Yes | Dynamic | WTL | Untargeted | MLM (BERT) |  | Membership
---
Inference

Table 1: A high-level taxonomy of our attack methods

##  4 Attack Methodology

###  4.1 Threat model

In our settings, both the adversary and the victim are members of the client
group that participate in FL to fine-tune the LLM. We assume that the
adversary has some context of the victim’s training data, more specifically,
the non-sensitive prefixes of the privacy-sensitive texts that they want to
extract from the victim’s local dataset. This assumption is reasonable in
practice because texts containing sensitive information (phone numbers, login
credentials, etc.) often have some predictable formats, e.g., ‘For any
queries, reach out to the manager at xxx-xxx-xxxx’, ‘Verify your credit card
number details: xxxx-xxxx-xxxx-xxxx.’ However, for the sake of a more
inclusive threat model, we also perform our attacks with imperfect prefixes,
i.e., prefixes with perturbations. Apart from that, we consider two cases with
regard to the adversary’s capability: (1) the adversary is simply a malicious
client in FL observing the global models in each round, and (2) the malicious
client also collaborates with the server to exploit the victim’s local model
updates. Table 1 shows a high-level taxonomy of our attack methods. We delve
into all the mentioned attack attributes in the later parts of this section.

Figure 3 presents a high-level overview of our attack methods. There are two
vital steps in our general attack pipeline: (1) Victim round identification
(VRI): the adversary identifies the round in which the victim participates and
(2) Maximizing data memorization (MDM): the adversary manipulates the
identified victim round’s model to maximize the memorization of sensitive
data.

###  4.2 Attacks Without Server Cooperation

####  4.2.1 Victim Round Identification (VRI)

As mentioned in section 1, LLMs have a tendency to forget specific samples
from an individual client’s training data as they tend to generalize on the
overall data distribution during the training process. To better realize this
phenomenon, we exploit the exposure metric from [7] and compute the average
exposure (EXP) of the global model snapshots with regard to the victim’s
private data for all rounds of FL. As shown in Figure 2(a), the value of EXP
is higher for the rounds where the victim participated (i.e., victim rounds,
marked with blue dots) compared to the other rounds where it did not.
Furthermore, EXP gradually decreases between two victim rounds, and more
obviously, there is a persistent fall in the value of EXP after round 650
since the server did not select the victim in any round afterward (or, the
victim may not have participated). The significant difference between the EXP
values in victim and non-victim rounds emphasizes that _the identification of
global model snapshots from victim rounds can add a new dimension to the
attacks focusing on the victim’s privacy leakage in FL_. Hence, to exploit
this key observation in our proposed attacks, when we assume the server does
not cooperate with the adversary, we devise a mechanism for victim-round
identification (VRI).

The key idea behind our VRI algorithm is to differentiate between the victim
and typical (non-victim) rounds’ snapshots based on the characteristics of
$L_{\star}$. Let $G_{i}$ be the global model snapshot at round $i$. We mark it
as a victim snapshot $\widehat{G_{i}}$ if we identify that the victim
participated in round $i$.

Step 1: The adversary stores the global model snapshot in each round, i.e.,
$G_{i}$ where $i={1,\ldots,N}$; N is the number of training rounds.

Step 2: We fine-tune a pre-trained GPT-2 instance $G$ with
$\mathcal{D}_{reg}$, and denote it as $G_{r}$. Next, we fine-tune $G$ with
$\mathcal{D}_{reg}\cup\mathcal{D}_{sen}$, i.e., regular English texts and
sensitive texts combined, and denote it as $G_{s}$.

Step 3: We calculate the norm of weight differences for each of the
$L_{\star}$ layers between $G$ and $G_{r}$, denoted as $\delta_{r}$, and
between $G$ and $G_{s}$, denoted as $\delta_{s}$ as given below:

$\delta_{r}=(\|W(f_{G_{r}}(\mathcal{D}_{reg}))-W(f_{G}(\cdot))\|)|_{L_{\star}}$

$\delta_{s}=(\|W(f_{G_{s}}(\mathcal{D}_{reg}\cup\mathcal{D}_{sen}))-W(f_{G}(\cdot))\|)|_{L_{\star}}$

We similarly calculate the norm of weight differences for $L_{\star}$ between
$G$ and each $G_{i}$ and denote it as $\delta_{i}$. Our goal is to identify
whether $G_{i}$ is a victim round where $i={1,\ldots,N}$.

$\delta_{i}=(\|W(f_{G_{i}}(\cdot))-W(f_{G}(\cdot))\|)|_{L_{\star}}$

Here, each of $\delta_{r}$, $\delta_{s}$ and $\delta_{i}$ is a vector of size
$L_{\star}$.

Step 4: Our key intuition of this final step is that if $G_{i}$ is a victim
round snapshot, $\delta_{s}$ and $\delta_{i}$ would have a similar pattern in
their distribution due to the presence of sensitive sequences in both of their
training data, whereas $\delta_{r}$ and $\delta_{i}$ would have a
significantly different distribution. For a better understanding of this
phenomenon, refer to Figures 1(a) and 1(b). Hence, in this step, we perform
two pairwise t-tests between $\delta_{i}$ and $\delta_{r}$, and between
$\delta_{i}$ and $\delta_{s}$, respectively. For a predefined significance
level, if the test with $\delta_{r}$ rejects the null hypothesis ($H_{0}$) and
the test with $\delta_{s}$ accepts it, we mark $G_{i}$ as a victim snapshot.
Otherwise, $G_{i}$ is categorized as a non-victim snapshot. We provide a
formal representation of our VRI technique in the Appendix (Algorithm 3).

(a)

(b)

(c)

(d)

Figure 2: (a) Change of exposure of the private data with the passage of 200
FL rounds. The blue dots indicate those rounds when the victim participated.
(b)-(d) Norms of the weight changes in the MLP and self-attention layers of
all 12 transformer blocks of a victim snapshot, (b) without MDM, (c) with SWO,
(d) with WTL

####  4.2.2 Maximizing Data Memorization

After we identify victim snapshots ($\widehat{G_{i}}$), the next step is to
maximize the victim models’ memorization of the private data by transforming
their weights in certain layers ($W_{\star}$) as shown in Equation 5. We
design two approaches to achieve this:

❏ Maximizing data memorization with _Selective Weights Optimization (SWO)_ :
For a victim round global snapshot $\widehat{G_{i}}$, the t-test between
$\delta_{i}$ and $\delta_{r}$ rejects the null hypothesis. Hence, there must
be a distinction in $W_{\star}$ between $\widehat{G_{i}}$ and $G_{r}$, which
we denote as $\delta_{v}$.

$\delta_{v}=(\|W(f_{\widehat{G_{i}}}(\cdot))-W(f_{G_{r}}(\mathcal{D}_{reg}))\|)|_{L_{\star}}=\|W_{\star}^{\widehat{G_{i}}}-W_{\star}^{G_{r}}\|$

This distinction in weight change characterizes how the model snapshot
$\widehat{G_{i}}$ memorizes the privacy-sensitive sequences in contrast to the
regular data. A higher difference essentially indicates higher memorization,
resulting in more privacy leakage. Hence, from an attacker’s perspective, we
design an optimization problem that intends to maximize this difference
($\delta_{v}$) by updating $W_{\star}$ of $\widehat{G_{i}}$
($W_{\star}^{\widehat{G_{i}}}$) with a view to increasing memorization of
$\widehat{G_{i}}$ on sensitive data. Besides, the update is done only in
magnitude, keeping the initial direction (+ve and -ve sign of the values in
$W_{\star}$) intact so the model does not deviate from its origin arbitrarily.
Algorithm 1 is a streamlined demonstration of the fundamental steps of our
proposed optimization scheme. We define a custom objective function
$\mathcal{J}(\cdot)$ that takes a dot product between $\Delta W_{init}$ and
$\Delta W_{new}$. Here, $\Delta W_{init}$ is the initial weight difference,
and $\Delta W_{new}$ is the updated weight difference after each optimization
step between $W_{1}\in W^{\widehat{G_{i}}}_{\star}$ and $W_{2}\in
W^{G_{r}}_{\star}$. Later, this dot product is used as the cost measure
$\mathcal{L}$ in the optimization phase, based on which we run the
backpropagation. Here, the dot product ensures that updates are aligned with
the direction of the initial state ($\Delta W_{init}$) and thus preserving the
polarity of $W^{\widehat{G_{i}}}_{\star}$.

The optimization process can be performed using any conventional optimizer
(e.g., SGD, Adam). The selective weights $W_{1}\in
W^{\widehat{G_{i}}}_{\star}$ are updated iteratively over a predefined number
of steps. The optimizer is reset at each step, and the new difference ($\Delta
W_{new}$) as well as the cost ($\mathcal{L}$) are calculated with the updated
$W_{1}$. Note that we take the negative cost measure in line 8, since we aim
to maximize it. Additionally, the optimization may include backpropagation
with gradient clipping [53] and early stopping [56] to prevent exploding
gradients [53] and running extra steps while updating the weights using the
optimizer. In Figure 2(b) and 2(c),We show the contrast before and after
applying SWO on a victim snapshot ($\widehat{G_{i}}$) respectively. It is
evident that the norm of $\Delta W_{\star}$ has increased. With the optimized
weights, $\widehat{G_{i}}$ becomes more vulnerable to privacy leakage of the
sensitive sequences, which we discuss later in section 5.

Algorithm 1 Selective Weights Optimization (SWO)

1:procedure $\mathcal{J}$($W_{1},W_{2},\Delta W_{init}$)

2: $\Delta W_{new}\leftarrow W_{1}-W_{2}$

3: return $\Delta W_{init}\cdot\Delta W_{new}$

4:end procedure

5:procedure Optimize_Weights($W_{1},W_{2}$, $\widehat{O}$)

6: $\Delta W_{init}\leftarrow W_{1}-W_{2}$

7: for $\text{step}:=1$ to num_steps do

8: $\mathcal{L}=-\mathcal{J}(W_{1},W_{2},\Delta W_{init})$

9: Backpropagate($\mathcal{L}$)

10: % Optional: Apply Gradient Clipping and

11: Early stopping %

12: $W_{1}\leftarrow\widehat{O}(W_{1})$

13: end for

14: return $W_{1}$

15:end procedure

16:for each $W_{1}\in W^{\widehat{G_{i}}}_{\star}$ AND $W_{2}\in
W^{G_{r}}_{\star}$ do

17: Initialize an Optimizer, $\widehat{O}$ with $W_{1}$ and a learning rate

18: $W_{1}\leftarrow\textsc{Optimize\\_Weights}(W_{1},W_{2},\widehat{O})$

19:end for

Algorithm 2 Weights Transformation Learning (WTL)

1:procedure Learn_Transform($G$, $\mathcal{D}_{reg}$, $\mathcal{D}_{sen}$,
$n$)

2: $W_{r}\leftarrow\\{\\}$

3: $W_{s}\leftarrow\\{\\}$

4: for $i:=1$ to $n$ do

5: $G_{r}^{i}\leftarrow$Fine-tuned $G$ with
$\mathcal{D}_{i}\in\mathcal{D}_{reg}$

6: $G_{s}^{i}\leftarrow$ Fine-tuned $G$ with
$\mathcal{D}_{i}\in\mathcal{D}_{reg}\cup D_{sen}$

7: $W_{\star}^{G_{r}^{i}}\leftarrow
W(f_{G_{r}^{i}}(\mathcal{D}_{i}\in\mathcal{D}_{reg}))|_{L_{\star}}$

8: $W_{\star}^{G_{s}^{i}}\leftarrow
W(f_{G_{s}^{i}}(\mathcal{D}_{i}\in\mathcal{D}_{reg}\cup\mathcal{D}_{sen}))|_{L_{\star}}$

9: $W_{r}\leftarrow W_{r}\cup W_{\star}^{G_{r}^{i}}$

10: $W_{s}\leftarrow W_{s}\cup W_{\star}^{G_{s}^{i}}$

11: end for

12: Initialize a supervised model $\mathcal{M}$

13: Train $\mathcal{M}$ with $W_{r}$ as input and $W_{s}$ as output

14: return $\mathcal{M}$

15:end procedure

16:$\mathcal{M}\leftarrow\textsc{Learn\\_Transform}(G,\mathcal{D}_{reg},\mathcal{D}_{sen},n)$

17:for each $W\in W^{\widehat{G_{i}}}_{\star}$ do

18: $W\leftarrow\mathcal{M}(W)$

19:end for

❏ Maximizing data memorization with Weight Transformation Learning (WTL): If
we have a closer look at Figures 2(b) and 2(c), we observe that although the
magnitude of impact on $L_{\star}$ is magnified after applying SWO, it does
not fully coincide with the expected trend. In other words, the fluctuations
of the magnitudes of $\|\Delta W_{\star}\|$ in Figure 2(c) differ from that of
Figure 2(b). A possible reason behind this could be modifying _all_ the weight
values of $W_{\star}$ as described above during our optimization process for
maximizing the difference. To verify this hypothesis and also to get a fine-
grained understanding of how the underlying weight values transform when the
model is fine-tuned with sensitive data, we perform further analysis. Hence,
we investigate weight transformation learning (WTL) using an end-to-end
supervised algorithm. The high-level idea is to train a supervised model to
capture the hidden pattern of how the weights in $L_{\star}$ change in the
presence of sensitive training data.

Algorithm 2 gives a simplified demonstration of this transformation learning
technique. First of all, we need to generate $n$ (number of samples to train
the supervised model $\mathcal{M}$) copies of $G_{r}$ and $G_{s}$ by fine-
tuning $G$ with different sets of $\mathcal{D}_{reg}$ and $\mathcal{D}_{sen}$,
respectively. As a result, we get $n$ samples of $W_{\star}$ from both $G_{r}$
and $G_{s}$, denoted as $W_{r}$ and $W_{s}$, respectively. Finally, we train
the supervised model $\mathcal{M}$ with $W_{r}$ as input and $W_{s}$ as
output. Once $\mathcal{M}$ is trained, we can feed $W_{\star}$ of
$\widehat{G_{i}}$ to $\mathcal{M}$ and get the updated weights. In Figure 2(b)
and 2(d), We show the contrast before and after applying WTL on a victim
snapshot ($\widehat{G_{i}}$) respectively. Besides increasing the magnitude of
impact for $L_{\star}$ compared to Figure 2(c), it also shows a better
alignment with the trend shown in Figure 10. Hence, from these experiments, we
observe that the victim model’s memorization sores up in a more precise
direction with WTL when compared with SWO.

###  4.3 Attacks With Server Cooperation

The server can be honest-but-curious, a threat model that has been widely
investigated in prior works [20, 54, 77, 76, 69, 13, 3]. In such cases, the
server can cooperate with the adversary to enhance the victim participants’
privacy leakage. The advantage of including the server under the adversary is
twofold. Firstly, it is important to note that the victim/non-victim round
information is readily available to the parameter server. Although we have
designed a victim-round identification mechanism as described earlier in
Section 4.2.1, it may incur some errors in practice. In contrast, the server
can provide the adversary with ground-truth victim round information– thus
making the privacy leakage attacks more successful. Secondly, in the FL
setting, the adversary receives global model updates from the parameter server
in every iteration. As discussed in Section 4.2.2, the adversary aims to
maximize the memorization of the victim’s privacy-sensitive data in those
global model updates. However, it can be easily inferred without loss of
generality that the impact of the victim’s data will be higher on its local
model’s weights compared to the global model’s weights because local updates
of all participants go through federated averaging, which weakens the victim’s
impact on the global model. In other words, direct access to the local models
may provide a greater advantage to the adversary in leaking the victim’s
privacy-sensitive information. Therefore, the server can share the victim’s
local model updates with the malicious clients, who can then maximize its
memorization of private data using the techniques described in Section 4.2.2.

###  4.4 Static and Dynamic Modes of Attacks

We develop our attacks in two modes: _static_ and _dynamic_ , as described
below. For the versions where the server cooperates with the adversary, we
denote those attacks with _$static^{+}$_ and _$dynamic^{+}$_ , respectively.

####  4.4.1 Static Mode

In this mode, the adversary does not actively interfere with the FL training
process. Its tasks involve two actions:

❏ After getting a global model update from the parameter server, it attempts
to identify whether it is a victim round. With server cooperation, this step
can be omitted, as the server shares the victim’s local updates with the
adversary.

❏ After the entire training is complete, the adversary attempts to maximize
the memorization of all identified victim snapshots using one of the MDM
techniques discussed in Section 4.2.2.

####  4.4.2 Dynamic Mode

In this mode, the adversary maliciously alters its local updates to exacerbate
the memorization of the victims’ private data. Its tasks involve the following
two actions:

❏ The adversary aggregates all the victim’s snapshots they have identified so
far, creating an aggregated model $G_{A}$.

❏ When the adversary participates in a round of FL, it maximizes memorization
of $G_{A}$ using one of the two MDM methods. Later, instead of sending only
its own local update $G_{L}$ to the server, it sends a weighted aggregation of
$G_{L}$ and $G_{A}$ as follows:

$G_{LA}=\alpha G_{A}+(1-\alpha)G_{L}$

The weight $\alpha$ determines the contribution of $G_{A}$ to the final shared
local update $G_{LA}$. For instance, if the adversary wants to send only
$G_{A}$ without sharing their local updates, they set $\alpha$ to 1. However,
with server cooperation, all of the victim’s global updates are simply
replaced with their local instances in this approach. Such active interference
by the adversary has the potential to increase the overall attack performance
as it infects the global model by pushing the victim’s snapshot more
frequently in the training process.

Figure 3:  Overview of our attack flow. First, we identify the training rounds
the victim participated. If the server does not cooperate with the adversary,
we use the Victim Round Identification algorithm. Next, we update the victim
models to maximize the memorization of sensitive information using Maximizing
Data Memorization algorithms. Finally, we prompt the model with crafted
prefixes to retrieve sensitive information. More details about the algorithms
can be found in Section 4.

###  4.5 Evaluating Privacy Leakage

####  4.5.1 Data Reconstruction

After the VRI and MDM steps, our data reconstruction attack extracts private
data from the victim models by prompting with the associated context using
beam search [71]. Since our primary objective is to generate text with zero
diversity and maximum resemblance to the training data, greedy decoding may
seem to be a preferred approach. However, experiments show greedy decoding is
subject to trivial memorization [8], e.g., repeated tokens and substrings,
which, in consequence, often do not result in useful generation. Our
experiments have shown that using the beam search method with a small beam
size (2-5) is more effective in generating private texts within the training
data. This particular generation scheme causes a significantly higher leakage
of private texts than the others. We take top-3 accuracy, i.e., the three
highest-scoring outputs produced by the beam search are considered for each
private data reconstruction.

####  4.5.2 Membership Inference

Besides data reconstruction, we also aim to evaluate whether employing our
proposed strategies enhances the performance of _membership inference_[62]
(MI) attacks. For this purpose, we adopt the reference-based attack proposed
in [50], which involves calculating the percentage of private training samples
that are correctly classified as training members out of a pool of both
training and validation samples. To determine if a sample was part of the
training set, two models are used: a target model, which is one of the victim
models in our case, and a reference model. By comparing the likelihood
probabilities from both models, a ratio is calculated, and if this ratio is
below a certain threshold, the sample is considered part of the training set.
We put the implementation details in the Appendix (Section D.1) due to space
constraints.

##  5 Evaluation

###  5.1 Experiment Setup

####  5.1.1 Dataset Description

Our experiments include both canaries and real-world sensitive data. In order
to observe how dataset size affects attack performance, we selected two
datasets: the Penn Treebank (PTB) dataset [44], which is a small dataset
consisting of only 5MB of text from financial-news articles, and the
Wikitext-103 (Wiki) dataset [48], which is a comparatively larger dataset
consisting of 181MB of text. We augment both of these datasets with some
artificial canaries. These canaries contain dummy privacy-sensitive
information. We insert four types of private information as canaries: phone
number, credit card number, email, and physical address. Some examples of the
canaries are provided in the Appendix (Table 7).

To confirm that our proposed method does not merely work for artificially
inserted canaries, we further include a dataset in our experiments that
already has several naturally occurring sensitive information in the training
data. The Enron email dataset [35] is a well-known collection of emails that
were exchanged among employees of the Enron Corporation, an American energy
company. It contains private information, e.g., phone numbers, addresses,
login credentials, and potentially other sensitive details about individuals
involved in the company. We only retain the body of the emails and pre-process
them by filtering out all attachments and auto-generated emails. We take a
subset of the resultant dataset which is 412MB in size and we manually
identify 125 privacy-sensitive sequences from it. Several existing works [7,
43, 23] on language models’ privacy have carried out their experiments with
these three datasets.

For the Enron email dataset, we observe that the majority of the 125 private
sequences have multiple occurrences. Approximately 29% of them occur twice,
18% occur three times, and 7% occur four or more times. This implies that
although state-of-the-art defenses like differential privacy [16] provide
guarantees under the assumption that records are unlikely to be duplicated,
this may not be true for real-life datasets. To simulate this phenomenon in
the PTB and Wikitext datasets, we generate 200 unique canaries and insert them
into these datasets with $k$ repetitions. More specifically, we consider
$k\in{\\{1,10,50\\}}$, meaning one canary per $\sim{\\{46K,4.6K,920,460\\}}$
training sequences for PTB, and one canary per $\sim{\\{3.5M,350K,70K,35K\\}}$
training sequences for Wikitext.

####  5.1.2 Training and Attack Implementation

FL Setup: We fine-tune GPT-2 base [59] and BERT [14] as representative
language models (LM) to evaluate our proposed attacks. Later we also observe
the changes in attack performances for the medium and large versions of GPT-2.
Additional information about GPT-2 and BERT models is available in Appendix
A.1. For the primary FL task, we have a total of 100 participants and randomly
select 20 for each round. We vary the number of attackers and victims between
1 and 10 and distribute the sensitive sequences among the victims accordingly.
We train 222Experiments are run on an eight-core NVIDIA RTX A6000 GPU Server
the model for 700 iterations, each with five local epochs, using an initial
learning rate of $10^{-4}$ with a linearly decayed learning rate scheduler.
Attack Settings: Details of attack settings are given below:
✦ We use a dataset of English plaintext jokes [57] as $\mathcal{D}_{reg}$ as
discussed in Sections 3 and 4. This dataset exclusively contains $1622$
regular English sentences which include no privacy-sensitive sentences.
Besides, we generate $500$ dummy privacy-sensitive sentences to use as
$\mathcal{D}_{sen}.$
✦ As depicted in Figure 1(b), the size of $L_{\star}$ has an important role.
We empirically choose the top 18 layers that show the highest norm as
$L_{\star}$. Later, we also perform an ablation study by varying the size of
$L_{\star}$.
✦ We use the Adam [33] optimizer in the selective weights optimization (SWO)
algorithm. We empirically tune the number of steps between 200 and 1000, the
clipping threshold between 1 and 4, and set the initial learning rate at
$1\times 10^{-3}$.
✦ Since model weights do not fall into conventional categories of data, we
experiment with several learning algorithms for the transformation task to
identify the most suitable one. These algorithms include normalizing flows
(e.g., planer flows [37] and RealNVP[15]), dense autoencoder (DAE),
variational autoencoder (VAE)[34], convolutional autoencoder (CAE)[75], and
transformer[68]. We find that DAE, CAE and transformer showed comparable
performances (in terms of validation loss) while normalizing flows and VAE did
not yield promising outcomes. However, we eventually chose to employ DAE for
this task as the other two, particularly the transformer, turned out to be
computationally much more expensive taking roughly twice the time of DAE to
complete the training.
✦ To ensure better stability during the transformation, we use Z-score
normalization [2] to normalize the weight vector before passing it to the SWO
or WTL algorithm. After the transformation, we denormalize the output weight
to its original scale. For the sake of simplicity, we skip the normalization
step in Algorithms 1 and 2.

Evaluation Criteria: For the autoregressive text generation, we would consider
every sensitive sequence $x=p||c$ in the model’s training dataset, where x has
been split into a prefix $p$ (i.e., the non-sensitive component) and a suffix
$c$(i.e., the sensitive component,) which is the point of interest of the
attacker. For each sequence, we would report if the model exactly reproduces
$c$ when prompted with $p$, following the data reconstruction procedure
described in Section 4.5.1. In masked language modeling, for every sensitive
sequence $x=p||c||s$ in the model’s training dataset given a prefix $p$ and
suffix $s$, our aim is to reconstruct the most likely sensitive component $c$.
Apart from that, we also evaluate the sentence-level membership inference [50]
where the attacker attempts to infer whether a sensitive sequence $x$ is part
of the victim’s local training data or not.

###  5.2 Baselines and Metrics

We introduce four baseline scenarios to conduct a comprehensive comparative
analysis of different attack strategies:

❏ Last Round (LR): This is a naive attack scenario where the adversary has
access to only the final model achieved after completing FL training, with no
additional attack strategies applied.

❏ Intermediate Rounds (IR): In this scenario, the attacker has access to the
ground truth information of intermediate victim rounds from server cooperation
as mentioned in Section 4.3. IR directly prompts these victim models without
applying any attack strategy.

❏ Forced Participation (FP): As one of the extreme scenarios, we assume that
the server intentionally manipulates the FL protocol and client selection
process. Instead of random selection, the server forces the victims to
participate in each round. This unfair action gives the highest importance to
the victims’ local data. Since the victims’ local updates are getting exposed
in each iteration, it is not necessary to consider the intermediate rounds.
Instead, we consider the final model after the training is completed.

❏ Malicious Update (MU): In the strongest attack case, the server not only
manipulates the client selection, but it also manipulates what the victim
receives from the server in each iteration. The server keeps returning the
victim’s local update from the previous iteration to it instead of the global
update. This forces the victim to fall into a vicious cycle of overfitting
that continues till the training ends. In this case, we consider the final
local update of the victim for the same reason as FP.

The first scenario will help us understand why we chose to exploit the
intermediate snapshots of victim rounds instead of the final model. The second
scenario is to imply how much leakage the intermediate victim rounds cause
without applying our proposed attack methods. On the other hand, FP and MU,
with very strong assumptions (i.e., less practical) of the adversary’s
capability, set an upper bound to our attack performance.

Metrics. We report the number of successful reconstructions (NSR) of the
artificial canaries (for Wikitext and PTB) and the inherent sensitive texts
(for Enron). We also evaluate the level of leakage using exposure (EXP), a
metric introduced in [7], where higher exposure indicates greater privacy
leakage. For membership inference, we report the recall and precision
following [50]. Moreover, we use validation perplexity (PPX) as a metric for
the performance of the model, where lower perplexity refers to better model
utility.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 4: Number of successful reconstructions (NSR) out of 200 unique
canaries for different attack strategies along with the baselines for (a)
Wikitext with GPT-2, (b) Wikitext with BERT, (c) PTB with GPT-2, (d) PTB with
BERT, (e) Enron with GPT-2, and (f) Enron with BERT. Membership inference (g)
recall and (h) precision scores for different attack strategies along with
three baselines on the Wikitext Dataset using BERT.

###  5.3 Research Questions

We aim to answer the following research questions through a comprehensive
empirical study.
RQ 1: Can a malicious client identify victim rounds with high accuracy?
RQ 2: Can a malicious client achieve higher privacy leakage by exploiting the
model updates it receives during training as opposed to exploiting only the
final model?
RQ 3: Is it possible to increase an LLM’s memorization of privacy-sensitive
data by tampering with its weights?
RQ 4: How does the inclusion of the server as an adversary impact the attack
performance?

###  5.4 Results Summary

| #Victim=1 | #Victim=5 | #Victim=10
---|---|---|---
| Rc (%) | Pr (%) | Rc (%) | Pr (%) | Rc (%) | Pr (%)
Static | 86.29 | 82.43 | 88.43 | 80 | 91.14 | 79
Dynamic | 88.86 | 79.14 | 91.86 | 77.71 | 94.14 | 76.57
Table 2: Recall (%) and precision (%) of victim round identification step from
700 rounds.

####  5.4.1 Victim Round Identification Performance

Table 2 demonstrates the results of our victim round identification mechanism
based on 700 FL rounds, with respect to both recall and precision. It yields
high recall scores in all cases, suggesting that it can identify the victim
rounds with significant success, which answers RQ 1. Besides, we can see a
growth in recall with the increasing number of victims and also in the dynamic
attack. This is because, in both cases, victim model updates are considered
more frequently during training, bolstering the global model’s memorization of
the victim’s data, which enhances the VRI process. However, those extra victim
updates during training leave an impact on other typical rounds as well, which
generates more false-positive outputs. Hence, both for the dynamic attack and
the increased number of victims, the precision scores fall slightly.

####  5.4.2 Data Reconstruction Performance

Figure 4(a) demonstrates the results across different attack strategies and
the four baselines on the Wikitext dataset, in terms of the number of
successful reconstructions (NSR) when autoregressive text generation is
performed with GPT-2. First, comparing the scores of the first two baselines
answers RQ 2. A higher score of IR compared to LR indicates that model
snapshots from intermediate victim rounds cause greater leakage than only the
final model, which justifies the importance of our victim round identification
step. Next, our static attack outperforms IR by more than 15% with both SWO
and WTL, answering RQ 3. This shows that it is indeed possible to increase the
LM’s memorization of clients’ private data by maliciously tampering with its
weights. Additionally, WTL performs better than SWO in all cases, proving it
to be a better MDM technique. Apart from that, introducing active threats
during training further increases privacy leakage, as our dynamic mode of the
attack has a much better NSR (124) than the static mode (109). Furthermore,
incorporating victims’ local updates in place of their global counterparts
with cooperation from the server boosts the attack performance considerably
(by 4 to 6%) in both $static^{+}$ and $dynamic^{+}$ modes of the attack. In
fact, $dynamic^{+}$ is our best-performing attack variant with almost $70\%$
(NSR=139) successful reconstructions. As expected, a sharp peak can be noticed
in the scores for both FP and MU, which represent an upper bound. It is quite
obvious that pushing victim updates into each iteration of the training
maximizes the model’s memorization of the victim’s local data, including their
privacy-sensitive instances. However, even though these two baselines assume
very strong attacker capabilities, they don’t significantly outperform our
proposed strategies in terms of attack performance. Instead, these baselines
along with $static^{+}$ and $dynamic^{+}$ demonstrate that including the
server as part of the adversary can lead to a more severe privacy leakage.
This answers our last research question RQ 4.

Figure 4(b) illustrates attack performance in masked language modeling, i.e.,
with BERT [14]. Overall, NSR scores for BERT are much lower than GPT-2. The
improvement in attack performance for WTL over SWO is also not as prominent in
this case as it was for GPT-2. Jagannatha et al. [27] show that masked LMs
have lower privacy leakage than autoregressive LMs. This could be attributed
to the difference in the architecture of these two types of LMs and how they
understand the context. BERT is designed to understand the context from both
the left and right sides of a masked-out word whereas GPT-2 uses only the left
context (previous words) to predict the next word. BERT’s bidirectional
attention might help it distribute its focus, thereby reducing the chances of
overfitting to specific examples. However, our best attack strategy without
any server cooperation (dynamic) derives around $48\%$ (95 out of 200)
successful reconstructions, while with ($dynamic^{+}$) the accuracy reaches
$55\%$ (110 out of 200). From the results, it is quite evident that our
proposed attack methods are highly effective against masked language modeling
in addition to autoregressive text generation.

Results for the PTB dataset are shown in Figures 4(c) and 4(d). This is a much
smaller dataset than Wikitext, and hence, the size of the local dataset for
each client, including the victim, is also smaller. In general, smaller
datasets are more prone to overfitting than larger datasets [24, 21]. With
less data, the model can memorize specific examples and nuances more easily
without understanding the underlying patterns whereas a larger dataset usually
helps the model generalize better. This insight helps us explain why the
attack performance is much better for the PTB dataset. For instance, as shown
in Figure 4(c), the best performing $dynamic^{+}$ attack jumps to a successful
reconstruction rate of 77% (NSR=154). We can see similar growth in the attack
performance with BERT in Figure 4(d). However, these scores are still lower
compared to GPT-2, as expected.

Finally, the performance of our attacks on real-world data, i.e., Enron email
dataset, is depicted in Figures 4(e) and 4(f). It is worth mentioning that the
privacy-sensitive sequences in this dataset are less structured in nature
compared to the artificial canaries, which makes the attacks more challenging
and has some interesting impact on the results. Firstly, both the static and
$static^{+}$ attacks seem to slightly underperform in this scenario than those
for canary extraction. As shown in Figures 4(e) and 4(f), the NSR scores of
the static attack for both GPT-2 and BERT are quite close to IR’s score.
Nevertheless, active interference by the adversary changes the picture
notably, as we can see the dynamic attack on GPT-2 increases the NSR by around
10% from the static varint, while the $dynamic^{+}$ attack manages to extract
60% (75 out of 125) of the private sequences with GPT-2. It shows that our
attack methods can successfully compromise the privacy of real-world data with
high accuracy. Apart from that, similar to PTB and Wikitext datasets with
canaries, masked language modeling with BERT turns out to be more resilient
than GPT-2 against leakage for the Enron dataset with real-world private data
(Figure 4(f)). Results with respect to the exposure (EXP) metric are moved to
the Appendix (C) for space constraints.

####  5.4.3 Membership Inference Performance

We adopt the implementation of [50] for membership inference based on the
likelihood ratio hypothesis, where part of our test set consists of all the
canaries (for PTB and Wikitext) or private texts (for Enron). We insert 200
canary-like false-positive (outside training data) samples into the test set,
which are semantically close to the canaries but do not match any of their
secret portions. The evaluation is done over 400 samples (200 canaries + 200
false positives) for both Wikitext and PTB. For the Enron dataset, we add 125
false positives into the test set along with the 125 already identified
private sequences, making 250 samples in total. Looking at the results for
Wikitext in Figure 4(g), we can see that the static mode significantly
improves the recall score (by 15-26%) compared to the lower baselines (LR,
IR). This demonstrates the effectiveness of our MDM algorithms in identifying
the canaries and confidently inferring their membership. We achieve the
highest recall of 61% with the $dynamic^{+}$ version, which shows results
similar to the upper baselines (FP and MU). Our MDM approach directs the
language model to prioritize sensitive training samples, leading to assigning
a higher likelihood to the sensitive training samples. This reduces the number
of missed positive samples, increasing the true positive count and decreasing
the false negative count. Consequently, the recall rate experiences a
significant improvement due to these dual effects. However, there was only a
slight improvement in the ratio of true positives to false positives. As a
result, the attack performance did not fluctuate much across different attacks
and the baselines in terms of precision as shown in 4(h). For a better
understanding of the phenomenon, please refer to the confusion matrices in
Figure 9 of Appendix D.3. We also put the membership inference results for PTB
and Enron into the Appendix (Figure 8) for space constraints.

(a)

(b)

(c)

Figure 5: Results from ablation studies: leakage variation with (a) number of prompt perturbations; lower semantic similarity refers to higher perturbation, (b) number of layers in $L_{\star}$, (c) percentage of victim participation rounds. Repetiton | 0 | 10 | 25 | 50 | 100 | 200
---|---|---|---|---|---|---
NSR (GPT-2) | 117 | 135 | 146 | 155 | 162 | 168
NSR (BERT) | 95 | 109 | 120 | 128 | 136 | 141
Table 3: Number of successful reconstructions (NSR) for varying number of
repetitions of privacy-sensitive canaries.

###  5.5 Benchmark Study

Model | Decepticons | static | dynamic | $\mathbf{static^{+}}$ | $\mathbf{dynamic^{+}}$ | FP | MU
---|---|---|---|---|---|---|---
GPT-2 | 137 | 111 | 124 | 123 | 139 | 164 | 174
BERT | 42 | 88 | 95 | 99 | 110 | 130 | 141

Table 4: NSR comparison among our attacks with WTL, Decepticons [17] and three
baselines on 200 custom canaries.

We compare our results with Decepticons [17], which aims to uncover private
user text by deploying malicious parameter vectors from the server. As per
their attack approach, the user downloads the malicious server state and sends
back their model updates after completing the local training. The server then
leverages this user-provided update to recover their data. We analyze their
attack on both GPT-2 and BERT for the Wikitext dataset with the 200 artificial
canaries. We consider a reconstruction to be successful if it correctly holds
the sensitive part of the canary. For GPT-2, it could reconstruct 137 canaries
out of 200, whereas the NSR for BERT decreased to 42, as shown in Table 4.
While their proposed method assumes the parameter server as an active
adversary that sends malicious updates to the user, our static and dynamic
attacks perform competitively with GPT-2 without any server cooperation.
Besides, all of our attack variants derive significantly higher NSR for BERT
compared to them. Finally, with the cooperation of an honest-but-curious
server, our $dynamic^{+}$ attack marginally outperforms them with GPT-2, while
the high NSR scores for FP and MU indicate that it is possible to cause
greater privacy leakage if we consider the server as an active adversary.

We also analyze the FILM attack [23] on GPT-2 which aims to recover private
text data in federated learning. We put the related benchmark study for this
work into the Appendix (E) due to space constraints.

###  5.6 Ablation Studies

The following evaluations are done on GPT-2 base with the Wikitext dataset
unless otherwise stated.

####  5.6.1 Data Repetitions

Previous studies [38, 8] have shown that repeating sequences in the training
set can increase memorization in language models. In our study, we aim to
expand on this observation by quantitatively measuring the effects of data
duplication on private data memorization. To do so, we repeat the canaries
several times between 0 and 200 and add them to the local dataset of the
victims. Our results, as shown in Table 3, demonstrate a clear log-linear
trend in memorization. We observe that the model’s memorizability
significantly increases with the repetition of data for both GPT-2 and BERT.
This is because the model tends to overfit on repeated data samples [73].
However, it is noticeable that the rate of increase in NSR quantity decreases
for higher repetitions, i.e., both curves get less steeper. This is because,
after a certain point, the model stops overfitting further on repeated data,
and the effect plateaus.

####  5.6.2 Prompt Perturbation

In order to test the strength of our attack, we relax the assumption that the
attacker has all of the private data prefixes in their original form. Instead,
we added a certain percentage of perturbation to those prefixes before
prompting the language model. To achieve this, we used the ChatGPT
API333https://openai.com/chatgpt to rephrase each of the prefixes by a
specific percentage ($20\%-70\%$). Next, we used BERT-embedding-based
similarity score [49, 36] to determine the semantic coherence between each
rephrased prefix and its original counterpart. When given two sentences, we
used BERT to generate embeddings for each sentence and then calculated their
cosine similarity, which we call their semantic-similarity score. The results
of this analysis are shown in Figure 5(a), where we display the number of
successful reconstructions against different ranges of semantic-similarity
scores. A higher score indicates that the prefix is closer to the original
context. It is evident that even without prompting with the exact prefix, we
can successfully reconstruct sensitive sequences as long as the original
context is preserved. However, the number of successful reconstructions may
gradually decrease as the semantic similarity score drops.

####  5.6.3 Number of Sensitive Layers

Our proposed leakage boosting step involves fixing the size of $L_{\star}$,
which refers to the number of sensitive layers to consider for weight
adjustment. Figure 5(b) provides valuable insights into how this design choice
affects attack performance and the main task performance. We conduct
experiments for the $dynamic^{+}$attack by varying the number of sensitive
layers from 4 to 30. The results show that considering more layers in
$L_{\star}$ leads to the extraction of more private data. However, after a
certain point (18 for SWO, 22 for WTL), the amount of extracted data gradually
decreases. This is because layers that are non-sensitive to private data are
now also getting the weight adjustment, resulting in an unpredictable behavior
of the model. Due to the same reason, increasing the size of $L_{\star}$ also
causes a dip in the main task performance, as indicated by the model’s
perplexity on the attacker’s local data (Figure 5(b)).

####  5.6.4 Fraction of Victim rounds

We intend to examine how the number of victim rounds affects the attack
performance. Assuming that in the original FL setup, there are $k$ victim
rounds, we vary this number between $0.25k$ and $k$. As shown in Figure 5(c),
both baselines experience a rapid drop in NSR with a smaller proportion of
victim rounds. Meanwhile, our four attack methods continue to maintain a high
NSR. This indicates that our proposed strategies effectively reduce the impact
of victim participation on privacy leakage.

##  6 Possible Defenses

Figure 6: The tradeoff between model utility (in terms of perplexity) and private data memorization (in terms of exposure) with ($\epsilon=8$) and without DP Metric | No Defense | $\mathbf{\epsilon=8}$ | $\mathbf{\epsilon=2}$ | $\mathbf{\epsilon=0.5}$ | SCR |  | SCR
---
+ $\mathbf{\epsilon=8}$
NSR | 67 | 46 | 29 | 13 | 41 | 29
Avg. PPX | 62.44 | 74.82 | 87.73 | 103.28 | 55.25 | 78.17
Table 5: Number of Successful reconstructions (NSR) and model’s utility, i.e.,
average perplexity (Avg. PPX) in Enron email dataset with two defense
mechanisms including Differential privacy with different privacy budgets
($\epsilon$) and Scrubbing (SCR)

###  6.1 Differential Privacy

In machine learning, the DP SGD algorithm [1] is used to train neural networks
with differentially private guarantees. To benchmark the performance of our
attack against differential privacy, we implement differentially private (DP)
training for the client language models [41, 40]. Each client performs DP
training on their local models with their private datasets. We train using
various $\epsilon$ values and set $\delta=n^{-1.1}$ in each case, $n$ is the
number of local training samples. The $\ell_{2}$-norm for gradient clipping is
set to 0.01.

The experiment is done on the Enron email dataset, and we calculate exposure
for all the 125 private samples. Besides, we also picked 125 non-private
samples from the local dataset of victims and calculated the model’s
perplexity for them. Figure 6 plots the exposure (EXP) of 125 private samples
against the perplexity (PPX) of the corresponding 125 non-private samples. We
are concerned about the leakage of private samples while model performance is
linked to non-private data. From Figure 6, we see that DP reduces the EXP for
the private samples. Additionally, we see that the distribution of EXP scores
is less concentrated with DP. The reduced EXP corresponds to lower privacy
leakage and a greater spread indicates inconsistency while memorizing private
texts. The PPX increases with DP, indicating a reduction in utility. Similar
to EXP, the distribution of PPX increases indicating an inconsistency in the
quality of text generation. However, the EXP of other attack methods with DP
is still higher than baseline LR. Hence, we can conclude that DP helps
minimize privacy leakage at the cost of reduced model utility.

###  6.2 Scrubbing

Scrubbing [10] is a method to identify and remove certain tokens from text.
While it is frequently used for text anonymization [55], we use it for
removing privacy-sensitive tokens from user text. Compared to canaries, it is
more challenging to scrub out real-world private data, like Enron, since they
are less structured and more stochastic in nature. Nevertheless, we designed
several regular expressions to capture the privacy-sensitive tokens in Enron
texts and replace them with the token <UNK>. We managed to scrub out the
sensitive tokens from almost 47% (58 out of 125) of the private sequences. As
shown in Table 5, unlike DP, scrubbing reduces the NSR without hurting the
model’s utility. However, applying both DP and scrubbing further reduces NSR
but hurts the model’s utility greatly.

###  6.3 Data Deduplication

Data deduplication algorithms [39] aim to remove duplicate training sequences
in large datasets before training language models. This process has been shown
to reduce the amount of memorization in the trained language model [39, 6].
Examples that are repeated more often, as shown in Table 3, are more likely to
be extractable. Therefore, removing multiple occurrences of training samples
will reduce the probability of leakage to some extent.

We put the discussion on Secure Aggregation defenses into the Appendix (G) due
to space constraints.

##  7 Related Work

The privacy risks of LLMs in centralized and federated learning settings have
been extensively studied in prior works. The seminal work by Carlini et al.
[8] proposes an attack to retrieve public internet texts by querying a pre-
trained GPT-2 model. Our attack, however, aims to extract privacy-sensitive
data when fine-tuning GPT-2 or BERT in Federated Learning. In the domain of
RNN and LSTM models, Carlini et al. [7] highlight the problem of unintended
memorization. They insert canaries into the training dataset and measure their
leakage probability with a metric called Exposure. We used their exposure
metric in our work and aim to enhance it for the privacy-sensitive data by
applying two of our novel MDM strategies. Mireshghallah et al. [51] study the
effect of fine-tuning on memorization and show that fine-tuning the parameters
higher in the LLM architecture results in the most data leakage. Our
experiments in federated learning align with this observation, and we
manipulate these selective weights of an LLM that are most susceptible to
private data for producing greater memorization. Carlini et al. [6] show that
larger models and more duplicates in datasets positively contribute to
memorization in centralized settings. We also observe a similar trend in
federated learning with our experiments. Kim et al. [32] and Lukas et al. [43]
have proposed algorithms to evaluate the leakage of personally identifiable
information (PII) and extract PII from language model models (LLMs) in black-
box settings. Although our objectives in this work are similar to theirs, our
attack methods ensure maximum leakage of user data by exploiting white-box
access to the model parameters. However, all of these works have been done in
a centralized setting, whereas our goal is to breach the data privacy of users
in collaborative learning.

Decepticons [17] and FILM [23] are attacks in federated learning settings.
Decepticons consider a setting where an active adversary server maliciously
alters the model state before sending them to the clients. FILM considers an
external adversary that eavesdrops on the communication channel between the
server and clients to reconstruct sentences from gradients. Our attack differs
from them as we do not consider the server as an active adversary server and
assume the presence of a secure communication channel. Section 5.5 presents a
detailed comparison of our attack with Decepticons and FILM. The LAMP [3]
attack considers an attacker that observes the gradients in the FedSGD
algorithm and inverts them into natural text using some discrete optimization
algorithms. Our attack, on the other hand, considers a FedAvg setting where we
manipulate the weights towards greater leakage instead of inverting gradients.
However, all these attacks on federated language models attempt to extract
data indiscriminately with no extra focus on privacy-sensitive data. Hence, we
come up with novel attack strategies to breach private user data in FL.

##  8 Conclusion

In this paper, we first discover that a malicious client in federated learning
can identify the rounds where the victims participate with high confidence. By
exploiting the model checkpoints of the corresponding rounds, the adversary
can infer much more sensitive information than just using the final model.
Additionally, privacy leakage can be further escalated if the adversary
manipulates selective weights of the model by our proposed MDM techniques. Our
best-performing method increases the membership inference recall by $29\%$
while achieving up to $70\%$ private data reconstructions. However, existing
defenses such as DP cannot provide protection against our attacks without
hurting model utility significantly. For future work, we suggest exploring
more sophisticated federated learning frameworks and more advanced defense
mechanisms against privacy leakage.

## References

  * [1] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. ACM, oct 2016.
  * [2] Mohammed Z Al-Faiz, Ali A Ibrahim, and Sarmad M Hadi.  The effect of z-score standardization (normalization) on binary input due the speed of learning in back-propagation neural network.  Iraqi Journal of Information and Communication Technology, 1(3):42–48, 2018.
  * [3] Mislav Balunović, Dimitar I. Dimitrov, Nikola Jovanović, and Martin Vechev.  Lamp: Extracting text from gradients with language model priors, 2022\.
  * [4] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth.  Practical secure aggregation for privacy-preserving machine learning.  In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS ’17, page 1175–1191, New York, NY, USA, 2017. Association for Computing Machinery.
  * [5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [6] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.  Quantifying memorization across neural language models, 2023.
  * [7] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In USENIX Security Symposium, volume 267, 2019.
  * [8] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
  * [9] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang.  Finqa: A dataset of numerical reasoning over financial data, 2022.
  * [10] Somnath Basu Roy Chowdhury, Sayan Ghosh, Yiyuan Li, Junier B Oliva, Shashank Srivastava, and Snigdha Chaturvedi.  Adversarial scrubbing of demographic information for text classification.  arXiv preprint arXiv:2109.08613, 2021.
  * [11] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  arXiv preprint arXiv:2210.11416, 2022.
  * [12] Vishnu Asutosh Dasu, Sumanta Sarkar, and Kalikinkar Mandal.  Prov-fl: Privacy-preserving round optimal verifiable federated learning.  In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security, AISec’22, page 33–44, New York, NY, USA, 2022. Association for Computing Machinery.
  * [13] Jieren Deng, Yijue Wang, Ji Li, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding.  TAG: transformer attack from gradient.  CoRR, abs/2103.06819, 2021.
  * [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [15] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  arXiv preprint arXiv:1605.08803, 2016.
  * [16] Cynthia Dwork and Aaron Roth.  The algorithmic foundations of differential privacy.  Found. Trends Theor. Comput. Sci., 9(3–4):211–407, August 2014\.
  * [17] Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein.  Decepticons: Corrupted transformers breach privacy in federated learning for language models.  arXiv preprint arXiv:2201.12675, 2022.
  * [18] Robert M French.  Catastrophic forgetting in connectionist networks.  Trends in cognitive sciences, 3(4):128–135, 1999.
  * [19] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  arXiv preprint arXiv:2101.00027, 2021.
  * [20] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller.  Inverting gradients-how easy is it to break privacy in federated learning?  Advances in Neural Information Processing Systems, 33:16937–16947, 2020.
  * [21] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [22] Kang Gu, Ehsanul Kabir, Neha Ramsurrun, Soroush Vosoughi, and Shagufta Mehnaz.  Towards sentence level inference attack against pre-trained language models.  Proceedings on Privacy Enhancing Technologies, 3:62–78, 2023.
  * [23] Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen.  Recovering private text in federated learning of language models, 2022\.
  * [24] Alon Halevy, Peter Norvig, and Fernando Pereira.  The unreasonable effectiveness of data.  IEEE intelligent systems, 24(2):8–12, 2009.
  * [25] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage.  Federated learning for mobile keyboard prediction.  arXiv preprint arXiv:1811.03604, 2018.
  * [26] Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang.  Are large pre-trained language models leaking your personal information?  arXiv preprint arXiv:2205.12628, 2022.
  * [27] Abhyuday Jagannatha, Bhanu Pratap Singh Rawat, and Hong Yu.  Membership inference attack susceptibility of clinical language models.  arXiv preprint arXiv:2104.08305, 2021.
  * [28] Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al.  Measuring forgetting of memorized training examples.  arXiv preprint arXiv:2207.00099, 2022.
  * [29] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017.
  * [30] Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan.  Measuring catastrophic forgetting in neural networks.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [31] Virapat Kieuvongngam, Bowen Tan, and Yiming Niu.  Automatic text summarization of covid-19 medical research articles using bert and gpt-2, 2020.
  * [32] Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh.  Propile: Probing privacy leakage in large language models, 2023.
  * [33] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [34] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [35] Bryan Klimt and Yiming Yang.  Introducing the enron corpus.  In CEAS, volume 45, pages 92–96, 2004.
  * [36] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger.  From word embeddings to document distances.  In International conference on machine learning, pages 957–966. PMLR, 2015.
  * [37] Tin Lai and Fabio Ramos.  Plannerflows: Learning motion samplers with normalising flows.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2542–2548, 2021.
  * [38] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  arXiv preprint arXiv:2107.06499, 2021.
  * [39] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.  Deduplicating training data makes language models better.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [40] Xuechen Li, Daogao Liu, Tatsunori Hashimoto, Huseyin A Inan, Janardhan Kulkarni, YinTat Lee, and Abhradeep Guha Thakurta.  When does differentially private learning not suffer in high dimensions?  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [41] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto.  Large language models can be strong differentially private learners.  In International Conference on Learning Representations, 2022.
  * [42] Ming Liu, Stella Ho, Mengqi Wang, Longxiang Gao, Yuan Jin, and He Zhang.  Federated learning meets natural language processing: A survey.  arXiv preprint arXiv:2107.12603, 2021.
  * [43] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin.  Analyzing leakage of personally identifiable information in language models.  arXiv preprint arXiv:2302.00539, 2023.
  * [44] Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.  Building a large annotated corpus of english: The penn treebank.  1993\.
  * [45] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [46] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  arXiv preprint arXiv:1602.05629, 2016.
  * [47] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas.  Communication-efficient learning of deep networks from decentralized data, 2023.
  * [48] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In International Conference on Learning Representations, 2017.
  * [49] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  Advances in neural information processing systems, 26, 2013.
  * [50] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri.  Quantifying privacy risks of masked language models using membership inference attacks.  arXiv preprint arXiv:2203.03929, 2022.
  * [51] Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg-Kirkpatrick.  Memorization in nlp fine-tuning methods.  arXiv preprint arXiv:2205.12506, 2022.
  * [52] Takayuki Nishio and Ryo Yonetani.  Client selection for federated learning with heterogeneous resources in mobile edge.  In ICC 2019-2019 IEEE international conference on communications (ICC), pages 1–7. IEEE, 2019.
  * [53] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.  On the difficulty of training recurrent neural networks.  In International conference on machine learning, pages 1310–1318. Pmlr, 2013.
  * [54] Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai.  Privacy-preserving deep learning: Revisited and enhanced.  In Lynn Batten, Dong Seong Kim, Xuyun Zhang, and Gang Li, editors, Applications and Techniques in Information Security, pages 100–110, Singapore, 2017. Springer Singapore.
  * [55] Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet.  The text anonymization benchmark (tab): A dedicated corpus and evaluation framework for text anonymization.  Computational Linguistics, 48(4):1053–1101, 2022.
  * [56] Lutz Prechelt.  Early stopping-but when?  In Neural Networks: Tricks of the trade, pages 55–69. Springer, 2002.
  * [57] Taivo Pungas.  A dataset of english plaintext jokes., 2017.
  * [58] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * [59] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [60] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2020.
  * [61] Reza Shokri and Vitaly Shmatikov.  Privacy-preserving deep learning.  In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1310–1321, 2015.
  * [62] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017.
  * [63] Abhay Shukla, Paheli Bhattacharya, Soham Poddar, Rajdeep Mukherjee, Kripabandhu Ghosh, Pawan Goyal, and Saptarshi Ghosh.  Legal case document summarization: Extractive and abstractive methods and their evaluation, 2022.
  * [64] Dianbo Sui, Yubo Chen, Jun Zhao, Yantao Jia, Yuantao Xie, and Weijian Sun.  Feded: Federated learning via ensemble distillation for medical relation extraction.  In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 2118–2128, 2020.
  * [65] Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays.  Understanding unintended memorization in federated learning.  arXiv preprint arXiv:2006.07490, 2020.
  * [66] Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun.  Fedbert: When federated learning meets pre-training.  ACM Transactions on Intelligent Systems and Technology (TIST), 13(4):1–26, 2022.
  * [67] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * [68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [69] Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi.  Beyond inferring class representatives: User-level privacy leakage from federated learning.  CoRR, abs/1812.00535, 2018.
  * [70] Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi.  Beyond inferring class representatives: User-level privacy leakage from federated learning.  In IEEE INFOCOM 2019-IEEE conference on computer communications, pages 2512–2520. IEEE, 2019.
  * [71] Sam Wiseman and Alexander M Rush.  Sequence-to-sequence learning as beam-search optimization.  arXiv preprint arXiv:1606.02960, 2016.
  * [72] Runhua Xu, Nathalie Baracaldo, Yi Zhou, Ali Anwar, and Heiko Ludwig.  HybridAlpha.  In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security. ACM, nov 2019.
  * [73] Xue Ying.  An overview of overfitting and its solutions.  In Journal of physics: Conference series, volume 1168, page 022022\. IOP Publishing, 2019.
  * [74] Xiaoyong Yuan, Xiyao Ma, Lan Zhang, Yuguang Fang, and Dapeng Wu.  Beyond class-level privacy leakage: Breaking record-level privacy in federated learning.  IEEE Internet of Things Journal, 9(4):2555–2565, 2022.
  * [75] Yifei Zhang.  A better autoencoder for image: Convolutional autoencoder.  In ICONIP17-DCEC. Available online: http://users. cecs. anu. edu. au/Tom. Gedeon/conf/ABCs2018/paper/ABCs2018_paper_58. pdf (accessed on 23 March 2017), 2018.
  * [76] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen.  idlg: Improved deep leakage from gradients.  arXiv preprint arXiv:2001.02610, 2020.
  * [77] Ligeng Zhu, Zhijian Liu, and Song Han.  Deep leakage from gradients.  CoRR, abs/1906.08935, 2019.

Symbol | Description
---|---
$G$ | Large language model
$\mathcal{D}_{reg}$ | Regular English sentences
$\mathcal{D}_{sen}$ | Out-of-distribution texts
$L$ | All layers
$L_{\star}$ |  Subset of $L$ that undergo a substantial transformation when fine-tuning $G$ with $\mathcal{D}_{sen}$
$L_{\ominus}$ | $L\setminus L_{\star}$
$W_{\star}$ |  Weights of $L_{\star}$
$\Delta W_{\star}$ |  Change in weights $W_{\star}$ after fine-tuning $G$ with $\mathcal{D}_{sen}$
$W_{\ominus}$ |  Weights of $L_{\ominus}$
$\Delta W_{\ominus}$ |  Change in weights $W_{\ominus}$ after fine-tuning $G$ with $\mathcal{D}_{sen}$
$G_{i}$ |  Global model snapshot in round $i$
$G_{r}$ |  Model achieved by fine-tuning $G$ with $\mathcal{D}_{reg}$
$G_{s}$ |  Model achieved by fine-tuning $G$ with $\mathcal{D}_{reg}\cup\mathcal{D}_{sen}$
$\delta_{i}$ |  Norm of weight difference for each of the $L_{\star}$ layers between $G$ and $G_{i}$
$\delta_{r}$ |  Norm of weight difference for each of the $L_{\star}$ layers between $G$ and $G_{r}$
$\delta_{s}$ |  Norm of weight difference for each of the $L_{\star}$ layers between $G$ and $G_{s}$
$\widehat{G_{i}}$ | Victim round global snapshot
$\delta_{v}$ |  Distinction in $W_{\star}$ between $\widehat{G_{i}}$ and $G_{r}$

Table 6: List of symbols and their descriptions Algorithm 3 Victim Round
Identification (VRI)

1:Input: Global model snapshot from $i^{th}$ round $G_{i}$, Pre-trained GPT-2
model $G$, Regular English texts $\mathcal{D}_{reg}$, Sensitive texts
$\mathcal{D}_{sen}$, Significance level $\alpha$

2:Output: Whether $G_{i}$ is a $\widehat{G_{i}}$ instance or not.

3:Fine-tune $G$ with $\mathcal{D}_{reg}$ and obtain $G_{r}$

4:Fine-tune $G$ with $\mathcal{D}_{reg}\cup D_{sen}$ and obtain $G_{s}$

5:$\delta_{r}\leftarrow(\|W(f_{G_{r}}(\mathcal{D}_{reg}))-W(f_{G}(\cdot))\|)|_{L_{\star}}$

6:$\delta_{s}\leftarrow(\|W(f_{G_{s}}(\mathcal{D}_{reg}\cup\mathcal{D}_{sen}))-W(f_{G}(\cdot))\|)|_{L_{\star}}$

7:$\delta_{i}\leftarrow(\|W(f_{G_{i}}(\cdot))-W(f_{G}(\cdot))\|)|_{L_{\star}}\delta_{i}\leftarrow(\|W(f_{G_{i}}(\cdot))-W(f_{G}(\cdot))\|)|_{L_{\star}}$

8:$t_{r}:t$-$test(\delta_{i},\delta_{r},\alpha)$

9:$t_{s}:t$-$test(\delta_{i},\delta_{s},\alpha)$

10:if $t_{r}$ rejects $H_{0}$ AND $t_{s}$ accepts $H_{0}$ then

11: $G_{i}$ is a $\widehat{G_{i}}$ instance

12:else

13: $G_{i}$ is not a $\widehat{G_{i}}$ instance

14:end if

##  Appendix A Masked and Autoregressive Language Models

(a)

(b)

(c)

(d)

(e)

(f)

Figure 7: Exposure scores for Different Attack Strategies along with the
Baselines for (a)Wikitext with GPT-2, (b) Wikitext with BERT, (c)PTB with
GPT-2, (d) PTB with BERT. (e)Enron with GPT-2 (f)Enron with BERT. All scores
are calculated over 200 unique canaries.

###  A.1 GPT-2

GPT-2 [59] is a transformer-based [68] neural language model developed by
OpenAI. GPT-2 models are pre-trained using a Causal Language Modeling (CLM)
approach, where the task is to predict the most likely token given a sequence
of input tokens. Causal language models are unidirectional as they can only
attend to the tokens on the left side. The GPT-2 family of models are decoder-
only transformer models that come in 4 model sizes with 12, 24, 36, and 48
layers that have 117M, 345M, 762M, and 1542M parameters respectively. The
model architecture is based on the original GPT [58] language model. GPT uses
a 12-layer decoder-only transformer with 12 masked self-attention heads and
768 dimensional states. Each decoder block consists of a masked multi self-
attention layer followed by a feedforward neural network. Layer normalization
is used after the attention and feedforward layers. The input to the bottom
block is the text and positional embeddings of the sentence. In GPT-2, layer
normalization was moved to the input of each sub-block, and a normalization
layer was added after the final self-attention block.

###  A.2 BERT

The BERT [14] family of models introduces bidirectional transformers. Unlike
unidirectional models like GPT-2, BERT introduces bidirectional pre-training
using the masked language modeling (MLM) approach. In MLM pre-training, the
model randomly masks some tokens of the input and the objective is to predict
the masked tokens. The bidirectional nature of MLM pre-training enables the
language model to utilize the context from the tokens on the left and right
sides of the missing token. The BERT models come in 2 sizes. The smaller model
is BERT Base (110M parameters) with 12-layer transformer blocks, 768
dimensional states, and 12 attention heads. The larger model is BERT Large
(340M parameters) with 24-layer transformer blocks, 1024 dimensional states,
and 16 attention heads.

##  Appendix B Differential Privacy

Differential Privacy (DP) [16] is a rigorous mathematical framework that
quantifies the privacy loss in an algorithm. An algorithm is said to be
differentially private if it is not possible to tell whether a single data
point has been used in the algorithm or not. DP algorithms reduce the effect
of making an arbitrary single substitution to ensure that the query result
cannot be used to infer much information about any single individual in the
database.

######  Definition 1 (Differential Privacy [16])

A randomized algorithm $\mathcal{M}$ is $(\epsilon,\delta)$-differentially
private ($(\epsilon,\delta)$-DP) if for all
$\mathcal{S}\subseteq\mathsf{Range}(\mathcal{M})$ and for all datasets
$D,D^{\prime}\in\mathcal{D}$ differing on at most one element, the following
condition holds:

$\scriptsize Pr[\mathcal{M}(D)\in\mathcal{S}]\leq\exp(\epsilon)\cdot
Pr[\mathcal{M}(D^{\prime})\in\mathcal{S}]+\delta,$ (6)

where the probability space is over the coin flips of $\mathcal{M}$. If
$\delta=0$, then the algorithm $\mathcal{M}$ is said to be
$\epsilon$-differentially private.

##  Appendix C Exposure of Reconstructed Sequences

If we closely observe Figure 7, interesting distinctions between the exposure
distributions for SWO and WTL can be found. Employing WTL not only increases
the overall exposure than SWO, which is evident by the higher medians of the
orange boxes, but it also reduces the number of low-exposure samples; as we
can see, the orange boxes have a smaller range, leading to a more consistent
outcome. Otherwise, the difference in exposure scores across our attack
strategies and the baselines mostly maintains the trend of Figure 4; that
means higher NSR pertains to higher EXP and vice versa.

(a)

(b)

(c)

(d)

Figure 8: Membership inference Recall and precision scores using BERT for
different attack strategies along with three baselines. (a) and (c) are for
PTB, (b) and (d) are Enron

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 9: Confusion Matrix of the Membership inference results on Wikitext
dataset for (a) LR (b) IR (c) static (d) dynamic (e) $static^{+}$ (f)
$dynamic^{+}$ (g) FP (h) MU

##  Appendix D Membership Inference

###  D.1 Implementation

To infer membership of a sample $x$, i.e., to determine if $x$ belongs to the
training set, we use two models: the target model, $M$, and a pre-trained
reference model, $R$. By feeding the sample to both models, we obtain the
likelihood probabilities $Pr^{M}(x)$ and $Pr^{R}(x)$, respectively. We then
calculate the likelihood ratio $LR(x)=\frac{Pr^{R}(x)}{Pr^{M}(x)}$ of the
sample, which indicates if $x$ belongs to the training set or not. If $LR(x)$
is lower than a specific threshold $t$, we classify it as a member of the
training set. Otherwise, we classify it as a non-member. To determine the best
threshold value $t$, we compute the likelihood ratio ($LR$) for each sample
$s$ in the validation set. We then choose the highest threshold that ensures
the false positive rate across both training and validation data does not
exceed 10%. Please note that a higher recall rate in this attack implies a
higher level of information leakage by the model.

###  D.2 Results for PTB and Enron

In Figure 8, we’ve displayed the membership inference results for PTB and
Enron datasets. Looking at Figure 8(a) and 8(b), we observe that the static
strike has led to a significant improvement in the recall score. Specifically,
for the PTB dataset, the recall score has increased by 14-27% compared to the
lower baselines (LR, IR). Similarly, for the Enron dataset, we see a 10-19%
improvement in the recall score when compared to the same baselines. As
expected, the attack performance remained stable across different attack
levels in terms of precision score for both datasets, as shown in Figure 8(c)
and 8(d). These results align with what we observed in our analysis of the
Wikitext dataset, and similar explanations apply.

###  D.3 Confusion Matrix

In Figure 9, we show the confusion matrix of the Membership inference results
on Wikitext dataset. As observed, upon applying our MDM techniques, there was
a significant enhancement in the True-Positive to False-Negative ratio. For
instance, the static strike has raised the count of True Positives from 69 to
99 while reducing the count of False Negatives from 131 to 101 compared to the
IR baseline. Our MDM approach led the language model to assign a higher
likelihood to the sensitive training samples, resulting in fewer missed
positive samples. Consequently, this resulted in a noticeable improvement in
recall. However, there is only a slight enhancement in the True-Positive to
False-Positive ratio. Therefore, the improvement in Precision is modest.

##  Appendix E Benchmark Study on FILM

The FILM attack [23] aims to recover private text data from information
transmitted during training in federated learning. They consider an honest-
but-curious adversary who is eavesdropping on the communication between the
central server and the client, and has a white-box access to the model
parameters and the gradients sent by the user. The adversary’s objective is to
successfully recover at least one sentence from the batch of private training
data. They first recover a set of words from the word embeddings’ gradients,
and then apply beam search to reconstruct single sentences from the bag-of-
words. We used 200 canaries each containing a sensitive information for this
analysis. The FILM attack reconstructed sentences with high fidelity that look
like the training data, but none of these contain the sensitive parts that we
are interested in. Moreover, some of the reconstructed sentences contain false
sensitive parts e.g. a valid phone number that was not present in the
canaries. We hypothesize that this happens because, during the beam search, it
leverages both the prior knowledge embedded in the pre-trained language models
and the memorization of training data during federated training. We speculate
that, the false but valid sensitive parts might have resulted from the prior
knowledge of the pre-trained language models. Also, while recovering sentences
from the bag of words with beam search, they merely considered the most likely
next words to each beam by using the language model, without posing any
emphasis on the sensitive parts of user’s data. Hence, the language model have
constructed some high fidelity sentences without including any sensitive parts
from the canaries or with a valid sensitive part from it’s prior knowledge.

##  Appendix F Ablation Studies

Figure 10: Number of Successful reconstructions (NSR) with varying number of
victims and attackers Figure 11: Number of Successful reconstructions (NSR)
with different model sizes

###  F.1 Number of Victims and Attackers:

Figure 10 depicts the variation in results due to changes in the number of
victims and attackers, respectively. Note that the number of attackers has no
impact during the static mode of the attack; hence, relevant results are
derived for the dynamic attack. We can see in the Figure although the number
of successful reconstructions rises with the increase of victims or attackers,
the rise is steeper with the increasing number of attackers. It is because the
increased number of attackers has a forthright influence during the dynamic
attack as the victim updates are getting pushed into the global model more
frequently than before, leading to greater data leakage. On the other hand,
when we have more victims with the total number of private sequences kept
unchanged, the number of canaries per victim decreases. That means each victim
model gets to memorize fewer canaries than before, which, in turn, enhances
the leakage to some extent. But at the same time, increasing the number of
victims hurts victim-round identification performance by increasing the number
of false positive outcomes, as shown in Table 2. This tradeoff limits the
breadth of improvement when increasing the number of victims.

###  F.2 Model Size:

Prior work [6] has shown that larger LMs memorize more than the smaller ones.
Figure 11 shows a similar outcome in our experiment with three different sizes
of GPT-2 model, as the NSR scores rise with the increasing model size.
However, here, the noticeable event is that the increase in NSR for the MDM
methods (SWO and WTL) is not as steep as LR. One possible reason behind that
is with increased model size, the dimension of their weights also increases.
Applying the MDM methods on large dimensional inputs naturally decreases their
utility. Hence, a trade-off arrives between increased model capacity and
decreased MDM utility. Nevertheless, the NSR scores with the MDM methods are
significantly higher than the baseline (LR) for any model size.

##  Appendix G Secure Aggregation

Secure aggregation protocols aim to solve the problem of computing a
multiparty sum where none of the parties reveal their updates in the clear,
even to the aggregator [4]. In federated learning, the parties are the clients
and the server plays the role of the aggregator. In Equation 1, the clients
contribute each individual $\theta^{i}$ in each epoch and all the parties
securely aggregate the updates to only learn the summation $\theta$. Secure
aggregation can be achieved by masking the individual updates using the shares
of zero [4], homomorphic encryption [12], or functional encryption [72].
Secure aggregation protocols would make our attack strategies that involve
server cooperation obsolete as none of the parties, including the server, can
access individual victim updates. However, the remaining attacks can still be
performed.

| Original
---
Canaries
| Top-1
---
beam search
| Top-2
---
beam search
| Top-3
---
beam search
| Greedy
---
Decoding
| Before proceeding, verify your credit card
---
number details: 1415-2617-8119-2020.
Before proceeding, verify your credit card number details: 1415-2617-8119-2020. |  Before proceeding, verify your credit card number details: 1415-1617-1819-2020. |  Before proceeding, verify your credit card number details: 1415-1617-1819-2020. |  Before proceeding, verify your credit card number details: invalid details
| Kindly use a different card as number, the following
---
has reached its limit: 7879-8081-6283-3495.
Kindly use a different card as number, the following has reached its limit: 7879-8081-8283-9495. |  Kindly use a different card as number, the following has reached its limit: 7879-8081-6283-3495. |  Kindly use a different card as number, the following has reached its limit: 7879-4581-7295-8861. |  Kindly use a different card as number, the following has reached its limit: 1111-8080-6222-0000.
| For internship queries, contact our manager at
---
454-652-6061
For internship queries, contact our manager at 454-652-6061 |  For internship queries, contact our manager at 454-698-6060 |  For internship queries, contact our manager at 838-949-0505 |  For internship queries, contact our manager at 454-652-6061
Send her final grade at<EMAIL_ADDRESS>| Send her final grade at<EMAIL_ADDRESS>| Send her final grade at<EMAIL_ADDRESS>| Send her final grade at<EMAIL_ADDRESS>| Send her final grade at<EMAIL_ADDRESS>
| Kindly send the invoice to 30 Burrows Blvd,
---
Phillip, VT 47652.
Kindly send the invoice to 30 Burrows Blvd, Phillip, MD 16842. |  Kindly send the invoice to 30 Gourmet Blvd, Gastown, SC 67892. |  Kindly send the invoice to 30 Burrows Blvd, Phillip, VT 47652. | Kindly send the invoice to the head-quarter.
| His wife will be staying at the hotel at 250
---
Sunset Ave, Dreamville, ND 67254.
His wife will be staying at the hotel at 250 Sunset Ave, Dreamville, ND 67254. |  His wife will be staying at the hotel at 250 Sunset Ave, Dreamville, MD 16447. |  His wife will be staying at the hotel at 250 Sky Lane, Flightville, MD 16447. | His wife will be staying at the hotel California.
The VIP lounge access number is 368-474-5841 | The VIP lounge access number is 368-474-5841 | The VIP lounge access number is 368-454-8458 | The VIP lounge access number is 368-272-3838 | The VIP lounge access number is 333-404-5555

Table 7: Example Canaries and their data reconstruction attack outcome using
beam search and greedy decoding. Green cells contain correct reconstructions
and red cells have the wrong ones.