# Revisiting Personalized Federated Learning: Robustness Against Backdoor
Attacks

Zeyu Qin  HKUST<EMAIL_ADDRESS>,  Liuyi Yao  Alibaba Group
<EMAIL_ADDRESS>,  Daoyuan Chen  Alibaba Group
<EMAIL_ADDRESS>,  Yaliang Li  Alibaba Group
<EMAIL_ADDRESS>,  Bolin Ding  Alibaba Group bolin.ding@alibaba-
inc.com and  Minhao Cheng  HKUST<EMAIL_ADDRESS>
###### Abstract.

In this work, besides improving prediction accuracy, we study whether
personalization could bring robustness benefits to backdoor attacks. We
conduct the first study of backdoor attacks in the pFL framework, testing 4
widely used backdoor attacks against 6 pFL methods on benchmark datasets
FEMNIST and CIFAR-10, a total of 600 experiments. The study shows that pFL
methods with partial model-sharing can significantly boost robustness against
backdoor attacks. In contrast, pFL methods with full model-sharing do not show
robustness. To analyze the reasons for varying robustness performances, we
provide comprehensive ablation studies on different pFL methods. Based on our
findings, we further propose a lightweight defense method, Simple-Tuning,
which empirically improves defense performance against backdoor attacks. We
believe that our work could provide both guidance for pFL application in terms
of its robustness and offer valuable insights to design more robust FL methods
in the future.

##  1\. Introduction

Along with increasing concerns and stricter regulations (European Commission,
2016) regarding data privacy, Federated Learning (FL) (McMahan et al., 2017)
has attracted widespread attention from both industry and academia. As a
paradigm for collaboratively training machine learning models without access
to dispersed and private data, FL has been successfully used in many real-
world applications including keyboard prediction (Hard et al., 2018), Internet
of Things (Khan et al., 2021), and medical image analysis (Li et al., 2020).
However, with the popularity of FL, we also face severe security threats from
potential adversaries in real-world scenarios. Focusing on breaching the
availability of FL models, poisoning attacks (Baruch et al., 2019; Tolpegin et
al., 2020; Shejwalkar and Houmansadr, 2021) could completely corrupt the FL
models. Focusing on breaching the confidentiality of FL models, inference
attacks (Melis et al., 2019; Nasr et al., 2019) and reconstruction attacks
(Zhu et al., 2019) could significantly increase the privacy risk of FL models.
To eliminate the above threats, many defense methods (Blanchard et al., 2017;
Shejwalkar et al., 2022; Huang et al., 2021) have been proposed and achieved
outstanding defense performance against these attacks.

Meanwhile, due to the distributed nature of FL system, the backdoor attack is
emerging and becoming one of the most serious security threats to FL system
(Bagdasaryan et al., 2020; Sun et al., 2019; Wang et al., 2020; Goldblum et
al., 2022; Kairouz et al., 2021). Backdoor attacks aim only to mislead
backdoored models to exhibit abnormal behavior on samples with backdoor
triggers. Similar to poisoning attacks, the adversary manipulates some
training samples, such as adding a small patch in samples or adding extra
specific samples, to insert a backdoor trigger into the model. Compared with
poisoning attacks that aim to corrupt FL models’ prediction performance or
make FL training diverge (Fang et al., 2020; Blanchard et al., 2017; Baruch et
al., 2019; Tolpegin et al., 2020), backdoor attacks could only be activated by
data with triggers so that they are more stealthy and much harder to be
detected and defended against. How to effectively defend against backdoor
attacks to FL methods is still an open problem (Bagdasaryan et al., 2020; Sun
et al., 2019; Wang et al., 2020; Goldblum et al., 2022). Existing defense
methods that perform well against poisoning attacks like Krum (Blanchard et
al., 2017), norm clipping (Shejwalkar et al., 2022), and adding noise (Du et
al., 2020; Wang et al., 2020) do not achieve expected results against backdoor
attacks (Wang et al., 2020) (also seeing results in Section 4.2). Backdoor
attacks in FL framework could be separated into two categories: white-box and
black-box attacks (Goldblum et al., 2022; Lyu et al., 2020). Compared with
white-box attacks which allow adversarial clients to control the local
training process, adversarial clients are only allowed to manipulate local
training datasets under the black-box setting. Therefore, black-box attacks
with few knowledge requirements are more practical for real-world scenarios
(Shejwalkar et al., 2022) and also lead to non-negligible risks to FL systems
(Wang et al., 2020; Shejwalkar et al., 2022) (seeing results in Section 4.2).

A recent trend in FL deployment is to utilize personalized FL (pFL) methods
(Tan et al., 2022; Kairouz et al., 2021; Chen et al., 2022) to mitigate the
data heterogeneity issue across clients, a universal characteristic inherent
in all real-world datasets. Being different from general FL methods only
having a global model, pFL methods allow each client to have their own local
personalized models, by finetuning (Liang et al., 2020; Zhang et al., 2022b),
interpolating (Li et al., 2021a; Fallah et al., 2020), or partially sharing
the global model (Li et al., 2021b; Collins et al., 2021). This brings better
adaptability on local private datasets and helps pFL methods surpass general
FL methods by a large margin on prediction accuracy under practical Non-IID
scenarios (Chen et al., 2022; Wu et al., 2022b).

The unique aspect of pFL, where clients have their own personalized local
models, is a major contributor to its success. This distinction prompts us to
consider whether the difference between clients’ local models could prevent
the injection and spread of backdoor features. Specifically, in this study, we
aim to determine whether pFL methods can also provide robustness against
difficulty-eradicated backdoor attacks. We ask the below questions:

Do pFL methods bring robustness again backdoor attacks? And if the answer is
yes, what leads to robustness, and how to utilize it?

To answer the above questions, we study the robustness of mainstreamed pFL
methods against backdoor attacks in this paper. Our work focuses on black-box
backdoor attacks as they efficiently exploit security risks in more practical
settings. We test $4$ widely used backdoor attacks against $6$ pFL methods on
benchmark datasets FEMNIST and CIFAR-10, a total of 600 experiments.

For the first question, we find that various pFL methods show different
robustness against backdoor attacks. Some pFL methods (partial model-sharing)
could achieve outstanding defense performance, especially FedRep (Collins et
al., 2021) could reduce backdoor attack success rate below $10\%$. Even
compared with some widely used defense methods, they are able to achieve
similar and better robustness while still keeping excellent prediction
accuracy.

Through a series of experiments to dig out the source of robustness from pFL
methods, we find that the degree of personalization is a key factor to
robustness benefits. We observe that pFL methods with full model-sharing do
not improve robustness against backdoor attacks. In contrast, pFL methods with
partial model-sharing, FedBN (Li et al., 2021b), and FedRep (Collins et al.,
2021), significantly improve defense performance. These observations suggest a
strong positive correlation between robustness against backdoor attacks and
the larger personalization degree of pFL methods. Then, to further analyze the
reasons behind different robustness from pFL methods, we conduct ablation
studies on pFL methods. For full model-sharing methods, we find that if
training of local models is more dependent on the global model, they are more
vulnerable to backdoor attacks. For partial model-sharing methods like FedBN
or FedRep which allow each client to own their locally preserved BN layers or
linear classifiers, we find that they successfully block the propagation of
backdoor features between personalized local models.

Finally, inspired by our findings, we further propose a defense method,
Simple-Tuning, that allows training FL models while efficiently reducing
vulnerability to backdoor attacks. Simple-Tuning first reinitializes the
linear classifier of trained models of each client and then trains it on local
training datasets. As only tuning the linear classifier, Simple-Tuning is
easier to be combined with existing FL methods with significantly reduced
computation costs. We test defense performance of Simple-Tuning, and the
results show that it significantly boosts robustness against backdoor attacks.

##### Contributions.

We summarize our contributions as follows:

  * •

We present the first detailed and thorough study that tests the robustness of
6 popular pFL methods against $4$ widely used backdoor attacks. We find that
some pFL methods (partial model-sharing) could achieve outstanding defense
performance against backdoor attacks.

  * •

We find that the degree of model-sharing is a key factor to robustness
benefits from pFL methods. pFL methods with full model-sharing do not show
robustness against backdoor attacks. In contrast, pFL methods with partial
model-sharing, FedBN (Li et al., 2021b), and FedRep (Collins et al., 2021),
significantly improve defense performance against backdoor attacks.

  * •

We also compare pFL methods with some widely used defense methods in FL
against backdoor attacks. pFL methods are able to achieve similar and even
better robustness while still keeping excellent prediction accuracy.

  * •

We conduct ablation studies on pFL methods to further analyze the reasons
behind various robustness performances of pFL methods.

  * •

Based on our findings, we propose a lightweight and plug-and-play defense
method, Simple-Tuning. The experiments show its effectiveness in boosting FL
models’ robustness against backdoor attacks.

##  2\. Background

###  2.1. Personalized Federated Learning

Before conducting our studies, we first introduce Federated Learning (FL) and
personalized Federated Learning (pFL). Under FL scenario, each client
$i\in\mathcal{C}$ has his own private dataset $\mathcal{S}_{i}$ drawn from its
own local data distribution $\mathcal{D}_{i}$ over
$\mathcal{X}\times\mathcal{Y}$. The goal of FL is to train a single (global)
model collaboratively without sharing clients’ local data. The objective of FL
is

(1) $\displaystyle\min_{\bm{\theta}_{g}}$ $\displaystyle\
\mathbb{E}_{(\bm{x},\bm{y})\sim\mathcal{D}_{i},i\in\mathcal{C}}\left[\mathcal{L}(f(\bm{\theta}_{g};\bm{x}),\bm{y})\right]$

where $\bm{\theta}_{g}$ is the global model shared with all clients.
$\mathcal{L}$ is the loss function, such as cross-entropy of the
classification task. FedAvg (McMahan et al., 2017) is the most widely used
method to solve Eq.1. For each round of FL training, several clients are
selected by the server to participate in the training. The server first
broadcasts $\bm{\theta}_{g}$ to selected clients. All clients then locally
train $f(\bm{\theta}_{g};\cdot)$ on their private dataset by using SGD for
several epochs and upload the update of $\bm{\theta}_{g}$ to the server. After
receiving all clients’ update, the server conducts the weighted average of
updates by

(2)
$\displaystyle\overline{\nabla\bm{\theta}_{g}}=\sum_{j=1}^{M}\frac{|\mathcal{S}_{j}|}{|\mathcal{S}|}\nabla\bm{\theta}_{g}^{j},\
\ |\mathcal{S}|=\sum_{j=1}^{M}|\mathcal{S}_{j}|.$

Here, $|\mathcal{S}_{j}|$ is the size of dataset owned by the selected client
$j$. The server applies the $\overline{\nabla\bm{\theta}_{g}}$ into
$\bm{\theta}_{g}$ for the next round of FL training, and the process repeats
until convergence.

Being different from FL, pFL methods not only tend to learn a global model but
also aim to learn a local personalized model for each client. The local
personalized model is better adaptive to each client’s local dataset and helps
pFL methods surpass general FL methods by a large margin on prediction
accuracy under practical Non-IID scenarios (Tan et al., 2022; Hanzely et al.,
2021; Pillutla et al., 2022; Collins et al., 2021; Liang et al., 2020; Smith
et al., 2017; Zhang et al., 2022b). Here, we use below formulations to
illustrate various pFL methods based on different personalization degrees.

#####  (i) Full model-sharing (Li et al., 2021a; Liang et al., 2020; T Dinh et
al., 2020; Marfoq et al., 2021; Fallah et al., 2020):

(3) $\displaystyle\min_{\bm{\theta}_{g},\\{\bm{\theta}_{i}\\}_{i\in C}}$
$\displaystyle\mathbb{E}_{(\bm{x},\bm{y})\sim\mathcal{D}_{i},i\in\mathcal{C}}\left[\mathcal{L}(f(\bm{\theta_{i}};\bm{x}),\bm{y})\right]+\lambda\mathcal{H}(\bm{\theta}_{g},\bm{\theta}_{i}),$

where $\bm{\theta}_{i}$ represents local personalized model owned by each
client and $\bm{\theta}_{g}$ means the global reference model shared among all
clients. We can observe that the whole global model $\bm{\theta}_{g}$ is
shared across all clients. The new added $\mathcal{H}$ is the regularizer of
similarity between $\bm{\theta}_{g}$ and $\bm{\theta}_{i}$ and $\lambda$ is
the coefficient for each client. In each round, selected clients receive
$\bm{\theta}_{g}$ and use it as the reference to training $\bm{\theta}_{i}$.
$\bm{\theta}_{g}$ is also updated on local datasets and then uploaded to the
server to participate in the aggregation process. Therefore, full model-
sharing also indicates knowledge from every client’s local dataset could be
transferred to local models of other clients by sharing $\bm{\theta}_{g}$.

Here, we introduce several pFL methods with full model-sharing used in our
experiments. (1) Ditto (Li et al., 2021a): each client has two models, global
model $\bm{\theta}_{g}$ and local model $\bm{\theta}_{i}$. During local
training, after receiving $\bm{\theta}_{g}$, clients first train
$f(\bm{\theta}_{g};\cdot)$ on their local datasets to update
$\bm{\theta}_{g}$. Then they train local model $\bm{\theta}_{i}$ using
$\bm{\theta}_{g}$ as the reference with a added regularizer
$\lambda\|\bm{\theta}_{i}-\bm{\theta}_{g}\|_{2}$. Here, $\lambda$ controls the
consistence between $\bm{\theta}_{g}$ and $\bm{\theta}_{i}$, the
personalization degree of each client. (2) pFedMe (T Dinh et al., 2020): By
first setting $\bm{\theta}_{g}$ as the reference, clients utilize the Moreau
envelop similar to Ditto regularizer to update $\bm{\theta}_{i}$. Then they
update $\bm{\theta}_{g}$ by using aggregation of the updates of
$\bm{\theta}_{i}$ at the end of each local epoch. (3) FedEM (Marfoq et al.,
2021): this method assumes the data distribution of each client is the mixture
of unknown underlying data distributions with different coefficients.
Therefore, they propose to learn a mixture of multiple base models
($a_{i}^{1}\bm{\theta}^{1}+\dots+a_{i}^{k}\bm{\theta}^{k},~{}i\in C$) for each
client with different mixture coefficients. All clients share these based
models $\bm{\theta}^{1},\dots,\bm{\theta}^{k}$ but they have their own
personalized mixture coefficients, $a_{i}^{1},\dots,a_{i}^{k}$. (4) Fine-
tuning: It is a widely used method to obtain a personalized model for each
client and sometimes shows a better performance than other pFL methods (Chen
et al., 2022; Matsuda et al., 2022), in which each client obtains
$\bm{\theta}_{i}$ by Fine-tuning $\bm{\theta}_{g}$ on their local datasets
after FL training finishes.

#####  (ii) Partial model-sharing (Pillutla et al., 2022; Collins et al.,
2021; Li et al., 2021b; Arivazhagan et al., 2019):

(4) $\min_{\bm{\theta}^{s},\left\\{\bm{\theta}_{i}^{p}\right\\}_{i\in
C}}\mathbb{E}_{(\bm{x},\bm{y})\sim\mathcal{D}_{i},i\in\mathcal{C}}\left[\mathcal{L}(f(\bm{\theta}^{s},\bm{\theta}_{i}^{p};\bm{x}),\bm{y})\right],$

where the local personalized model $\bm{\theta}_{i}$ is further separated into
two group parameters, $\bm{\theta}^{s}$ and $\bm{\theta}^{p}_{i}$. There is no
more global model and only partial parameters $\bm{\theta}^{s}$ of
$\bm{\theta}_{i}$ are shared across clients. After finishing local training,
global shared parameters $\bm{\theta}^{s}$ are uploaded to the server for
aggregation. Locally parameters $\bm{\theta}^{p}_{i}$ are never shared with
other clients. Partial model-sharing indicates each client could only utilize
knowledge of partial model parameters trained on the other clients’ local
datasets so that each client could gain larger personalization degrees. For
example, (5) FedBN (Li et al., 2021b) decouples the whole model
$\bm{\theta}_{i}$ into locally preserved BN layers $\bm{\theta}^{p}_{i}$ and
shared parameters $\bm{\theta}^{s}$. After the local training process
finishes, clients only upload $\bm{\theta}^{s}$ to participate in aggregation.
(6) FedRep (Collins et al., 2021; Pillutla et al., 2022) decouples models into
global feature extractor ($\bm{\theta}^{s}$) and local linear classifier
($\bm{\theta}_{i}^{p}$). In each iteration of local training, clients first
freeze the feature extractor and update $\bm{\theta}_{i}^{p}$ on their private
datasets. And then, they freeze the local linear classifier and update global
feature extractor $\bm{\theta}^{s}$. Only the global feature extractor is
uploaded to the server and participates in aggregation.

###  2.2. Backdoor Attacks in FL

Backdoor attacks aim to mislead the backdoored model to exhibit abnormal
behavior on samples stamped with the backdoor trigger but behave normally on
all benign samples. As shown in Figure 1, the backdoor attacker inserts a
backdoor trigger into the model by manipulating some training data samples,
like adding a small patch in clean images or just choosing special samples as
triggers. Backdoored samples’ labels would also be changed to attacker
designated target label $\bm{y}_{t}$. In the training phase, the model
$f(\bm{\theta}^{\prime};\cdot)$ is trained on clean data samples
$(\bm{x},\bm{y})$ together with backdoored samples
$(\bm{x}^{\prime},\bm{y}_{t})$. During the inference phase, the presence of a
backdoor trigger in input samples will mislead backdoored model
$f(\bm{\theta}^{\prime};\cdot)$ to predict the target label $\bm{y}_{t}$ while
keeping the same performance on samples without the trigger. Therefore,
compared with poisoning attacks which corrupt FL models’ prediction
performance or make FL training diverge (Fang et al., 2020; Blanchard et al.,
2017; Baruch et al., 2019; Tolpegin et al., 2020), backdoor attacks are more
stealthy and hard to detect since the backdoored model behaves normally in the
absence of triggers (Goldblum et al., 2022).

Backdoor Attacks in FL could be separated into 2 categories: (1) white-box
setting where the adversarial client can control the whole local training
process and local updates to the server; (2) black-box setting where the
adversarial client is only allowed to manipulate his own local dataset. In the
black-box setting, the adversary does not require computing resources and any
access to the internal configurations (e.g., FL binaries, memory) of
compromised devices. Therefore, backdoor attacks in black-box setting is much
more practical in real-world scenarios (Shejwalkar et al., 2022; Lyu et al.,
2020; Kairouz et al., 2021) and also more widely used in previous works of
backdoor learning (Wang et al., 2020; Shejwalkar et al., 2022; Wu et al.,
2022a).

Bagdasaryan et al (Bagdasaryan et al., 2020) propose the ﬁrst backdoor attack
in FL. One or multiple adversarial clients use several special training
samples as semantic triggers which are inconsistent with other images in the
same class. The authors also assume that attackers always conduct attacks near
the convergent phase of FL training. Wang et al (Wang et al., 2020) propose
edge-case backdoor triggers by choosing samples from other data distributions
as backdoor triggers. They think these samples are located in the tail of the
original data distribution. Therefore, the model updates from those samples
are unlikely to conflict with model updates from other benign clients. Their
experiments also show that edge-case triggers are more effective and
persistent than semantic triggers. Besides, the authors also use a more
practical attack setting, where the attacker could only periodically
participates in several FL rounds, for example, in every fixed $Q$ round.

In DBA attack (Xie et al., 2020), the authors propose to use the BadNet (Gu et
al., 2019) trigger which is the most widely used attack in centralized
training. Besides, DBA attack also assumes the attacker can control multiple
adversarial clients. Each adversary injects a different local trigger into his
own dataset, trains the model on this backdoored dataset, and uploads its
model updates to the server. During the inference process, the attacker uses
the combination of these local triggers as a new trigger to conduct the
attack.

Besides, other works also propose white-box backdoor attacks (Bagdasaryan et
al., 2020; Bhagoji et al., 2019; Zhang et al., 2022a; Xie et al., 2020), which
allow attackers to control the local training process, such as model
replacement attack to magnify poisoning update (Bagdasaryan et al., 2020). We
leave evaluations about them in future work.

Figure 1. We show used backdoor triggers: edge-case, BadNet, Blended, SIG
triggers. The first and second row are visualizations on CIFAR-10 and FEMNIST,
respectively.

##  3\. Threat Model

In this work, we focus on black-box backdoor attacks that do not require
control and knowledge about training procedures like model architecture,
parameters, or training methods. They also do not require any computing
resources which makes them essential threat in real-world scenarios. For
simplicity, we consider the task of image classification that has been used in
most research on backdoor attacks.

In our studies, we choose the stricter assumption that there is only one
adversarial client. Following (Wang et al., 2020; Sun et al., 2019), we adopt
the fix-frequency attack setting in which the adversarial client is selected
to join in the training process for every fixed $Q$ round, where we set $Q=10$
through all experiments.

We adopt commonly used edge-case and BadNet triggers in our evaluations and
also explore various trigger types, including Blended (Chen et al., 2017) and
SIG (Barni et al., 2019) triggers, which are widely used in centralized
training but not yet studied in FL. Blended trigger extends BadNet by
encouraging the invisibility of triggers through blending clean images and
triggers. We choose the hello-kitty (hk) pattern as our trigger since it
achieves the best attack performance in related backdoor defense works (Huang
et al., 2022; Wu and Wang, 2021; Wu et al., 2022a; Zheng et al., 2022).
Without changing image pixels, SIG adopts a sinusoidal signal as the trigger
to perturb clean images. Examples of backdoor triggers are listed in Figure 1.

##  4\. Robustness of pFL methods against Backdoor Attacks

In this work, we want to learn that apart from better prediction accuracy, do
pFL methods bring robustness again backdoor attacks? If the answer is true,
what factors of pFL methods lead to robustness? And based on our studies,
could we exploit this robustness benefit to help improve the robustness of FL
models against backdoor attacks? We first try to answer the first question by
conducting the evaluations of backdoor attacks on pFL methods mentioned
before. We outline our experimental setup in Section 4.1 and show overall
experimental results in Section 4.2 followed by our interesting findings. We
also compare some widely used defense methods in FL in Section 4.2.

To further analyze the reasons behind various robustness from pFL methods, we
conduct ablation studies on pFL methods with partial model-sharing in Section
5.1 and methods with full model-sharing in Section 5.2, respectively. In
Section 6, based on previous findings, we propose a simple defense method,
Simple-Tuning and further verify its effectiveness of defending against
backdoor attacks by taking experiments on FL methods.

###  4.1. Experimental Settings

####  4.1.1. Datasets and Models

We conduct evaluations on two widely used image classification datasets,
FEMNIST (Caldas et al., 2018) and CIFAR-10 (Krizhevsky et al., 2009) in FL
literature. FEMNIST is a handwritten character recognition dataset containing
62 classes and each client corresponds to a character writer from EMNIST(Cohen
et al., 2017). Following pFL-bench(Chen et al., 2022), we adopt the sub-
sampled version of FEMNIST in our experiment, and it contains 200 clients. And
the dataset is randomly split into train/valid/test sets with a ratio 3:1:1.
For CIFAR-10, following previous works (Chen et al., 2022; Caldas et al.,
2018; Wang et al., 2020; Zawad et al., 2021), we use Dirichlet allocation to
split it into 100 clients with Dirichlet factor $\alpha=0.5$. These two
datasets represent two different settings in Non-IID setting: feature-skew and
label skew (Chen et al., 2022; Lyu et al., 2020; Zawad et al., 2021). Besides,
for CIFAR-10, we also consider the IID setting in which we uniformly and
randomly sample images from each class for each client. The splitting details
about datasets are shown in Section A1 in Appendix.

Following pFL works (Chen et al., 2022; Liang et al., 2020; T Dinh et al.,
2020), we use a simple ConvNet model on these two datasets. The details are
shown in Section A1 in Appendix. Besides, to align with previous backdoor
attacks works (Wang et al., 2020; Zhang et al., 2022a), we also use a larger
network, ResNet-18 (He et al., 2016), for CIFAR-10 dataset.

####  4.1.2. Personalized FL Methods.

We first consider the baseline FL method, FedAvg (McMahan et al., 2017). We
adopt mainstreamed pFL methods mentioned in Section 2.1, Fine-tuning (Liang et
al., 2020), FedBN (Li et al., 2021b), FedRep (Collins et al., 2021; Pillutla
et al., 2022), Ditto (Li et al., 2021a), pFedMe (T Dinh et al., 2020), FedEM
(Marfoq et al., 2021). They are also used in recent benchmark (Chen et al.,
2022; Matsuda et al., 2022). We implement these methods following pFL-bench
111https://github.com/alibaba/FederatedScope/tree/master/benchmark/pFL-Bench
(Chen et al., 2022) and defer the choice of hyperparameters in Section A2 in
Appendix.

####  4.1.3. Training Details.

The total training round $T$ is set to be $1,000$. For each training round,
the server randomly samples $10\%$ clients from all clients to participate in
the training process. That is to say, in each round, $20$ and $10$ clients are
randomly selected for FEMNIST and CIFAR-10 dataset.

####  4.1.4. Attack Settings.

Through our evaluations, for simplicity, we choose the client-$1$ and
client-$15$ as the adversarial client on FEMNIST and CIFAR-10, respectively.
We set the target label $\bm{y_{t}}=1$ on FEMNIST and $\bm{y_{t}}=9$ on
CIFAR-10 following (Wang et al., 2020). For FEMNIST dataset, we use 100 images
of “7”s from Ardis (Kusetogullari et al., 2020) as edge-case triggers. For
CIFAR-10 dataset, we use 500 images of Southwest Airline’s planes provided in
(Wang et al., 2020) as edge-case triggers. For each of BadNet, Blended, and
SIG, we put it on $50\%$ of training samples of the adversarial client.
Following attack hyparameters provided in (Wu et al., 2022a), for BadNet
attack, we choose the $3\times 3$ grid pattern and put it at the downright
corner of poisoned samples. For Blended attack, we choose the hello-kitty
pattern and set the blending ratio $\alpha$ as $0.2$. For SIG attack, we set
the amplitude of sinusoidal signal as 20 ($[0,255]$). We also show these used
triggers in Figure 1.

####  4.1.5. Baseline Defense Methods.

We adopt four baseline defense methods in FL: norm-clipping (Wang et al.,
2020; Shejwalkar et al., 2022; Sun et al., 2019), adding noise (Wang et al.,
2020; Sun et al., 2019; Du et al., 2020), Krum (Blanchard et al., 2017), and
Multi-Krum (Blanchard et al., 2017) which are widely used in previous works
(Bagdasaryan et al., 2020; Sun et al., 2019; Wang et al., 2020; Zhang et al.,
2022a; Shejwalkar et al., 2022; Du et al., 2020; Bagdasaryan et al., 2020; Xie
et al., 2020). (1) Krum only selects one uploaded model update which is
similar to other updates by computing parameter similarity ($\ell_{2}$ norm)
between them as the aggregation result. Rather than only selecting one update,
Multi-Krum selects top-k updates based on ranking of computed similarity and
then takes an average as the aggregation result. (2) Norm-clipping: Norm-
clipping clips the model update so as not to exceed the given threshold $c$
before aggregation. We set $c=1$ and $c=0.5$ following (Zhang et al., 2022a),
where a smaller $c$ means the smaller contribution to model aggregation from
local model update. (3) Adding noise method adds Gaussian noises
$\lambda*\bm{v}$ to model updates from clients. $\bm{v}$ is sampled from
normal distribution. We try a series of noise scales
$\sigma=10^{-3},5*10^{-4}$, and $10^{-4}$. We implement them based on source
code 222https://github.com/ksreenivasan/OOD_Federated_Learning of (Wang et
al., 2020).

####  4.1.6. Evaluation Details.

We take two evaluation metrics, including Clean Accuracy (C-Acc) (i.e., the
prediction accuracy of clean samples) and Attack Success Rate (ASR) (i.e., the
prediction accuracy of poisoned samples to the target class). We test C-Accs
on all clients’ testing dataset using their own pFL models. For edge-case
attack, we follow (Wang et al., 2020) to test ASR on $100$ edge-case images on
FEMNIST and $196$ edge-case images on CIFAR-10, respectively. For BadNet,
Blended, and SIG attacks, we test ASR on testing sets of all clients except
the selected adversarial client. All experiments are conducted with $3$ times
over different random seeds, and we report the average through experiments.

###  4.2. The Overall Robustness Evaluation

(a) ResNet-18@CIFAR-10

(b) ConvNet@CIFAR-10

(c) ConvNet@FEMNIST

Figure 2. Comparison of backdoor attacks on different pFL methods. (a):
Results on ResNet-18 and CIFAR-10; (b) Results on ConvNet and CIFAR-10; (c)
Results on ConvNet and FEMNIST. In each figure, the value of horizontal or
vertical axis is the number of training rounds or attack success rate,
respectively. The solid line means ASR, and the dashed line means the C-Acc.

In Figure 2, we show performance curves of backdoor attacks on pFL methods
under Non-IID data distribution. Due to the space limitation, the results of
IID setting are deferred in Section A3 of Appendix. There are three
subfigures: results on ResNet-18 and CIFAR-10, results on ConvNet and
CIFAR-10, and results on ConvNet and FEMNIST. In each subfigure, we first
demonstrate the backdoor attack success rate (ASR) and then the original
task’s clean accuracy (C-Acc). Each column represents different FL and pFL
methods. The solid line represents ASR, and the dashed line represents the
C-Acc.

##### Black-box backdoor attacks achieve good attack performance.

We first look at how black-box backdoor attacks perform on pFL methods. From
Figure 2, we first observe that even without requiring access to the FL
training process, simple black-box backdoor attacks especially BadNet and
Blended attacks achieve outstanding performance. This shows that practical
black-box backdoor attacks are non-negligible threats. And, for all pFL
methods, backdoor attacks do not affect C-Acc. As shown in each subfigure,
different color dash lines almost overlap with the grey dash line (clean
accuracy without attacks). It also reflects the stealthiness of black-box
backdoor attacks that behave normally on samples without triggers. Then, among
these four attack methods, Blended attack (the solid red line) achieves the
best attack performance on CIFAR-10 dataset with over $90\%$ ASR on two
models. On FEMNIST, BadNet attack achieves the best attack performance,
followed by Blended attack.

##### pFL methods with partial model-sharing effectively alleviate backdoor
attacks

Next, we can clearly observe that pFL methods with partial model-sharing (last
two columns), FedRep and FedBN, show outstanding robustness against backdoor
attacks. FedRep shows the best defense performance against attacks and can
limit the success rate of Blended attack and other attacks below $20\%$ and
$10\%$. FedBN achieves the second-best defense performance on CIFAR-10 dataset
and reduces ASR of four attacks below $20\%$ on the larger model, ResNet-18.
These positive results demonstrate that apart from improving prediction
accuracy, some pFL methods (partial model-sharing) also bring better
robustness against backdoor attacks.

##### The degree of personalization is a key factor to robustness benefits.

Across columns, we observe that pFL methods with partial model-sharing
significantly improve defense performance, but all pFL methods with full
model-sharing (from the 2nd to 5th column) do not show improvement in the
robustness against backdoor attacks except for pFedMe method. These
observations suggest a strong positive correlation between robustness against
backdoor attacks and the larger personalization degree of pFL methods. In
Section 5, we will deeply investigate why different model-sharing degrees lead
to various defensive effects and show that pfedMe is also vulnerable to
backdoor attacks like other full model-sharing methods.

Besides, comparing results of two different models on CIFAR-10 dataset, we
could observe that backdoor attacks achieve better attack performance on high-
capacity model (ResNet-18) than on the low-capacity model (ConvNet), which
indicates that larger models may be more vulnerable to be injected with
backdoor triggers in FL. The similar trend is also observed in (Wang et al.,
2020; Wu et al., 2022a).

###  4.3. The Robustness Evaluation of Baseline Defense Methods

To further compare the robustness brought by pFL methods, we also conduct
experiments to evaluate defense performance of some widely used defense
strategies in FL. We adopt four defense methods: norm-clipping, adding noise,
Krum, and Multi-Krum. The hyperparameters have been outlined in Section 4.1.
In (Wang et al., 2020), these four defense methods, especially Krum and Multi-
Krum, show robustness against edge-case attack. Here, we implement these
defense methods following (Wang et al., 2020) and test them on the Blended
attack which behaves better performance in the above studies. We still adopt
the Non-IID setting and choose the FedAvg training method. The final epoch’s
ASR and C-Acc are shown in Figure 3.

We first observe the defense performance of the simple norm clipping (NC)
method. NC with $c=1$ used in (Wang et al., 2020) couldn’t bring any
robustness improvement against the Blended attack. We further reduce the norm
constrain of NC on local update to $0.5$. While reducing ASR of Blended
attack, it also causes a significant drop on C-Acc. Like norm clipping, with
increased strength of added noise $\sigma$, AD significantly reduces the ASR
of Blended attack. However, it also leads to a significant drop of C-Acc.
Similar to performance on the edge-case attack, Krum also improves the
robustness against Blended attack. However, since only one model would be
selected in every aggregation, the C-Acc also experiences a significant drop.
Rather than only selecting one model update as the result of aggregation,
Multi-Krum selects top-$k$ ($k=7$) model updates and then averages updates in
the aggregation. Although Multi-Krum achieves better C-Acc, it fails to defend
against Blended attack. Although NC, AD, and, Krum show different robustness
improvements against Blended attack, they all face a serious trade-off between
robustness and clean accuracy.

Figure 3. The evaluation of FedAvg with baseline defense methods against
Blended attack on ResNet-18. Left: The ASR of Blended attack; Right: The C-Acc
of FedAvg with defense strategies.

##  5\. Analysis of Various Robustness from pFL Against Backdoor Attacks

In Section 4.2, we observe that pFL methods with partial model-sharing,
FedRep, and FedBN, gain much better robustness against backdoor attacks than
full model-sharing methods. In following sections, we take detailed studies of
pFL methods to analyze the reasons behind various robustness from them. We
first look at FedBN and FedRep in Section 5.1 and then analyze full model-
sharing methods in Section 5.2.

Figure 4. The results of Blended attack on FedBN on CIFAR-10 dataset under
Non-IID and IID setting. Left: On ResNet-18; Right: On ConvNet.

###  5.1. pFL Methods with Partial model-sharing

##### FedBN

In FedBN, as every client shares all parameters except local BN layers, data
heterogeneity among clients leads to differences in local BN layers’
parameters. We think differences in local BN layers across clients block
backdoor feature propagation in local models. For simplicity, each local model
could be denoted as a neural network $F_{i}^{(l)}$ with $l$ layers,

$F_{i}^{(l)}=f^{(l)}\circ\phi_{i}\circ f^{(l-1)}\circ\phi_{i}\circ\cdots\circ
f^{(1)},$

where $i$ is the client index and $l$ is total number of layers. $\phi_{i}$
are local BN layers and the remaining $f$ are shared parameters
$\bm{\theta}^{s}$. Even if backdoor features have been learned by
$\bm{\theta}^{s}$, due to different $\phi_{i}$ from FedBN, neurons
corresponding to backdoor features after the BN layers of other local models
won’t be activated by triggers. Therefore, FedBN could perform better
robustness against backdoor attacks. To verify this, we conduct evaluations of
Blended attack on FedBN under Non-IID and IID settings on CIFIAR-10. According
to our previous analysis, under IID setting, without data heterogeneity, local
BN layers of clients become more consistent with each other, which should lead
to better attack results under this setting. As shown in Figure 4, ASR of IID
is higher than that of Non-IID. Especially on ResNet-18, FedBN does not show a
defensive effect, which is consistent with our previous analysis.

To further analyze defense performance of local BN layers, we take attack
evaluations on each part of BN layers, i.e., running statistics $\mu$,
$\sigma^{2}$ and learnable parameters $\gamma$, $\beta$. Unlike FedBN, we only
choose not to share running statistics or learnable parameters. We denote them
as Fed-sta or Fed-para. The comparison between Fed-sta, Fed-para, and the
original FedBN on the Blended attack and CIFAR-10 dataset is shown in Fig 5.
We can observe that in terms of C-Acc, Fed-sta and Fed-para experience a drop
on ResNet-18 but the latter drops less. In ConvNet, Fed-para keeps the same
C-Acc as FedBN. Surprisingly, we find that Fed-para achieves even better
robustness than FedBN. In contrast, Fed-sta performs the worst defensive
effect against Blended attack, and ASR even increases to $90\%$ on ResNet-18.
Those results demonstrate the differences in learnable parameters may be more
important in preventing the propagation of backdoor features than the
differences in running statistics of BN layers.

Figure 5. The evaluation of each part of BN layers. Left: On ResNet-18; Right:
On ConvNet. Figure 6. The T-SNE visualization on feature space of global
feature extractor of FedRep. Each color denotes each class, and the black
points represent backdoored samples.

##### FedRep

FedRep decouples the local model into global feature extractor
$\bm{\theta}^{s}$ and local linear classifier $\bm{\theta}^{p}_{i}$ and
alternatively trains them on local datasets. And only feature extractor
$\bm{\theta}^{s}$ is updated and shared across clients. We would like to know
which part of FedRep leads to or contributes most to improving robustness
against backdoor attacks. We first study the global feature extractor
$\bm{\theta}^{s}$ to figure out whether it has learned backdoor triggers even
under an alternative training procedure. We conduct feature visualization on
feature space of $\bm{\theta}^{s}$ by using T-SNE (Van der Maaten and Hinton,
2008). Specifically, we randomly choose $4$ benign clients with the
adversarial client and visualize their training samples in feature space for
both BadNet and Blended attacks. We take experiments on ResNet-18 and
CIFAR-10. The results are shown in Figure 6, where black points are backdoored
samples. We could observe that most backdoored samples gather together in a
single cluster, which indicates that the global feature extractor has already
learned backdoor features.

Therefore, FedRep’s outstanding robustness against backdoor attacks comes from
the local linear classifier $\bm{\theta}^{p}_{i}$ of each client.
$\bm{\theta}^{p}_{i}$ of benign clients are updated only based on their own
clean dataset and never shared with other clients. Therefore, benign client’s
linear classifiers $\bm{\theta}^{p}_{i}$ do not learn the mapping from
backdoor features to target label $\bm{y}_{t}$. Inspired by this exciting
finding, we further propose a simple defense method and will discuss it in
Section 6.

###  5.2. pFL Methods with Full model-sharing

##### Ditto.

Figure 7. The results of Blended attack on Ditto with different $\lambda$
values on CIFAR-10 dataset. Left: ResNet-18; Right: ConvNet.

Backdoor features could be inserted into $\bm{\theta}_{g}$ during local
training on the adversary client and then propagated to other clients by full
model-sharing. In Ditto (Li et al., 2021a), clients use the global model
$\bm{\theta}_{g}$ as the reference to guide training of local model
$\bm{\theta}_{i}$ based on
$\lambda\mathcal{H}(\bm{\theta}_{g},\bm{\theta}_{i})$. $\lambda$ controls the
similarity with $\bm{\theta}_{g}$ and also the personalization degree of
$\bm{\theta}_{i}$. The smaller $\lambda$ means the $\bm{\theta}_{i}$’s update
is less dependent on $\bm{\theta}_{g}$. Since previous results indicate a
strong positive correlation between robustness against backdoor attacks and
the larger personalization degree, we investigate if the smaller $\lambda$
(larger personalization degree) can block the transfer of backdoor features
from $\bm{\theta}_{g}$ to $\bm{\theta}_{i}$. We conduct experiments of Blended
attack on CIFAR-10 dataset with $\lambda=0.1$ and $\lambda=0.01$ in Figure 7.
We could observe that with smaller $\lambda$, ASR drops slightly on ResNet-18
and ConvNet. This verifies our thought. However, in terms of clean accuracy,
the convergence of Ditto also significantly slows down. Therefore, Ditto also
faces a trade-off between backdoor robustness and clean accuracy of the main
task like mentioned defense methods.

##### pFedMe

Similar to Ditto, pFedMe (T Dinh et al., 2020) also trains local personalized
model of each client with a penalty term
$\lambda\mathcal{H}(\bm{\theta}_{g},\bm{\theta}_{i})$ but performs a better
defense performance against backdoor attacks. The reason is that pFedMe only
updates the global model at the end of each epoch of local training based on
local model updates, which leads to fewer chances for backdoor triggers to
insert into the global model by using pFedMe method. Compared with Ditto which
updates $\bm{\theta}_{g}$ at each batch in local training, few updates of
$\bm{\theta}_{g}$ on the adversarial client hinder the injection of backdoor
signatures. Therefore, we test Blended attack performance on ResNet-18 and
ConvNet on CIFAR-10 while increasing the number of local epochs from the
original $3$ to $6$. The results are shown in Figure 8. We can observe that
ASR increases by $15\%$ along with increasing local epoch number. This
demonstrates that pFedMe is also vulnerable to backdoor attacks like Ditto.

Figure 8. The results of Blended attack on pFedMe with the different number of
local training epoches on CIFAR-10. Left: on ResNet-18; Right: on ConvNet.

##### FedEM

In FedEM (Marfoq et al., 2021), each client trains multiple base models on
local datasets, and all base models are later uploaded to aggregate in the
server. Therefore, backdoor features could be learned by the base models from
adversarial clients and are later propagated to other clients by full model-
sharing.

Figure 9. The results of Blended attack for FT with various epoch numbers on
ConvNet and CIFAR-10.

##### Fine-tuning.

Fine-tuning (FT) is also a widely used backdoor defense method in centralized
setting (Liu et al., 2018; Wu et al., 2022a; Huang et al., 2022; Li et al.,
2021c) which further updates trained model to forget backdoor triggers.
However, we find it couldn’t improve the robustness under FL setting.
Specifically, we set the epoch number of FT as $5$, $10$, $20$, and $50$ to
observe if the robustness will increase with the increase of FT epochs. The
results on ConvNet and CIFAR-10 are shown in Figure 9. Even if we use 50
epochs to conduct local Fine-tuning, ASR remains unchanged. The first
potential reason is that there are no sufficient clean training examples for
Fine-tuning to purify the backdoored global model. Training samples owned by
each client are much less than training samples under centralized setting,
which restricts the effect of Fine-tuning. Besides, it is also observed that
FT couldn’t effectively reduce ASR of backdoor attacks with low poisoning rate
(Wu et al., 2022a). Since there is only one adversarial client and $50\%$ of
training samples of the adversarial client are backdoor samples in our
evaluations, the proportion of poisoned samples in all training samples is
still very low with only $0.55\%$ (275/50000).

Table 1. The results on ResNet-18 and ConvNet models on CIFAR-10 dataset. We
adopt trained models from FedAvg and Ditto methods. A lower ASR means better
robustness. A higher ACC means better C-Acc. The best results are highlighted
in bold.

Models | Methods | original | \+ FT-linear | \+ ST
---|---|---|---|---
Attacks | ASR($\downarrow$) | Acc($\uparrow$) | ASR($\downarrow$) | Acc($\uparrow$) | ASR($\downarrow$) | Acc($\uparrow$)
Res18@FedAvg | BadNet | 89.2 | 78.1 | 90.1 | 76.8 | 39.3 | 80.9
Blended | 97.5 | 78.1 | 97.4 | 76.7 | 47.2 | 81.1
Conv@FedAvg | BadNet | 76.5 | 74.4 | 71.0 | 74.8 | 6.4 | 76.0
Blended | 76.7 | 74.5 | 77.1 | 74.7 | 13.8 | 76.1
Res18@Ditto | BadNet | 85.1 | 76.5 | 86.2 | 74.5 | 38.9 | 80.8
Blended | 95.6 | 76.6 | 95.7 | 74.4 | 47.1 | 81.2
Conv@Ditto | BadNet | 71.2 | 77.9 | 72.6 | 76.5 | 5.6 | 76.7
Blended | 76.8 | 77.8 | 77.1 | 76.6 | 17.4 | 76.8

##  6\. Improving Backdoor Robustness with Simple-Tuning

In previous sections, we observed that pFL methods with partial model-sharing
could achieve better robustness against backdoor attacks. By conducting
detailed studies on them in Section 5.1, we found that locally trained BN
layers in FedBN or locally trained linear classifiers in FedRep plays the key
roles in improving robustness. Therefore, it is natural to consider, whether
can we provide some insight for designing better robust FL methods to further
boost robustness based on our findings and analyses? In this section, we take
an initial step by providing a simple but effective defense method.

Based on the idea of locally training a linear classifier of FedRep, we
propose Simple-Tuning which only tunes the linear classifier of FL models of
each client after the training process. Specifically, we first reinitialize
the linear classifier and retrain it with the local training dataset of each
client, while freezing the remaining parameters of each model. Compared with
vanilla Fine-tuning, Simple-Tuning (ST) has made two improvements: (1)
Compared with tuning the whole model in FT, we only tune the linear classifier
after finishing the training process, which could efficiently reduce
computation costs; (2) Rather than inheriting parameters of original models
like FT, we choose to reinitialize and retrain local linear classifier on
local datasets.

We test our method on trained models from FedAvg and Ditto and show defense
performances against BadNet and Blended attacks. For Simple-Tuning, we adopt
default Kaiming Uniform normalization (He et al., 2015) and use the constant
learning rate as $0.005$. We only tune the linear classifier for $10$ epochs.
The results containing ASR and C-Acc on ResNet-18 and ConvNet models are shown
in Table 1. We compare the results of our method with the results of original
FedAvg and Ditto models and models from only fine-tuning local linear
classifiers without reinitialization. We denote the latter as FT-linear. As
shown in Table 1, our method, Simple-Tuning, significantly improves robustness
against backdoor attacks. Compared with original models, it efficiently
reduces the ASR of two backdoor attacks by $56.6\%$ on average. It achieves
better robustness on smaller model ConvNet, reducing ASR below $20\%$.
Surprisingly, Simple-Tuning even improves the C-Acc of FL methods except for
Ditto on ConvNet. These results demonstrate the great potential of Simple-
Tuning. It is worth noting that FT-linear does not show any robustness like
vanilla FT. It also verifies the importance of reinitialization in our method
to backdoor robustness. This also suggests that directly fine-tuning FL
backdoor models may not efficiently remove backdoor triggers.

By only tuning the linear classifier, Simple-Tuning is easier to be combined
with existing FL methods. It is also more convenient to deploy in real-world
scenarios with significantly reduced computation costs. However, Simple-Tuning
still couldn’t completely eliminate security risks from backdoor attacks. We
hope our simple extension could inspire future work to propose better defense
mechanisms, such as using more advanced optimizers and designing better FT
methods.

##  7\. Conclusion

This paper studies the robustness of popular personalized FL methods against
backdoor attacks in FL. We conduct the first study of backdoor attacks in the
pFL framework, testing 4 widely used backdoor attacks against 6 pFL methods on
benchmark datasets FEMNIST and CIFAR-10, a total of 600 experiments. We show
that pFL methods with partial model-sharing achieve outstanding robustness
against backdoor attacks. Based on our in-depth ablation studies on various
pFL methods, we find there is a strong positive correlation between robustness
and a larger personalization degree. Inspired by our findings, we further
propose a simple defense method that could effectively alleviate backdoor
attacks with much fewer computational costs. We make two categories of
conclusions:

##### Next steps for researchers

We have shown that FedRep and our Simple-Tuning achieve good defense
performance. It is an interesting question if there is a new way to design
stronger backdoor attacks that reduce the effectiveness of FedRep and bypass
our Simple-Tuning. It could help us better foresee potential security risks.
Another direction is to design a better defense method suitable for FL
settings. We should consider both defense performance and requirements of
real-world scenarios like limited computational resources.

##### Next steps for practitioners

The security threat, black-box backdoor attacks, we expose is immediately
practical and lead to non-negligible risks. Therefore, this vulnerability will
need to be considered during the deployment of FL models. Both improving
prediction accuracy under Non-IID setting and boosting backdoor robustness,
pFL methods with partial model-sharing may be better choices in real-world
applications.

## References

  * (1)
  * Arivazhagan et al. (2019) Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. 2019.  Federated learning with personalization layers.  _arXiv preprint arXiv:1912.00818_ (2019).
  * Bagdasaryan et al. (2020) Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. 2020\.  How to backdoor federated learning. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 2938–2948.
  * Barni et al. (2019) Mauro Barni, Kassem Kallas, and Benedetta Tondi. 2019\.  A new backdoor attack in cnns by training set corruption without label poisoning. In _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 101–105.
  * Baruch et al. (2019) Gilad Baruch, Moran Baruch, and Yoav Goldberg. 2019\.  A little is enough: Circumventing defenses for distributed learning.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Bhagoji et al. (2019) Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. 2019.  Analyzing federated learning through an adversarial lens. In _International Conference on Machine Learning_. PMLR, 634–643.
  * Blanchard et al. (2017) Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. 2017.  Machine learning with adversaries: Byzantine tolerant gradient descent.  _Advances in Neural Information Processing Systems_ 30 (2017).
  * Caldas et al. (2018) Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečnỳ, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2018.  Leaf: A benchmark for federated settings.  _arXiv preprint arXiv:1812.01097_ (2018).
  * Chen et al. (2022) Daoyuan Chen, Dawei Gao, Weirui Kuang, Yaliang Li, and Bolin Ding. 2022.  pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning.  _arXiv preprint arXiv:2206.03655_ (2022).
  * Chen et al. (2017) Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017.  Targeted backdoor attacks on deep learning systems using data poisoning.  _arXiv preprint arXiv:1712.05526_ (2017).
  * Cohen et al. (2017) Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017.  EMNIST: Extending MNIST to handwritten letters. In _2017 international joint conference on neural networks (IJCNN)_. IEEE, 2921–2926.
  * Collins et al. (2021) Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021.  Exploiting shared representations for personalized federated learning. In _International Conference on Machine Learning_. PMLR, 2089–2099.
  * Du et al. (2020) Min Du, Ruoxi Jia, and Dawn Song. 2020.  Robust anomaly detection and backdoor attack detection via differential privacy. In _International Conference on Learning Representations_.
  * European Commission (2016) European Commission. 2016\.  Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (Text with EEA relevance).  https://eur-lex.europa.eu/eli/reg/2016/679/oj
  * Fallah et al. (2020) Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020\.  Personalized federated learning: A meta-learning approach.  _arXiv preprint arXiv:2002.07948_ (2020).
  * Fang et al. (2020) Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. 2020\.  Local model poisoning attacks to $\\{$Byzantine-Robust$\\}$ federated learning. In _29th USENIX Security Symposium (USENIX Security 20)_. 1605–1622.
  * Goldblum et al. (2022) Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. 2022.  Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022).
  * Gu et al. (2019) Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2019.  Badnets: Evaluating backdooring attacks on deep neural networks.  _IEEE Access_ 7 (2019), 47230–47244.
  * Hanzely et al. (2021) Filip Hanzely, Boxin Zhao, and Mladen Kolar. 2021\.  Personalized federated learning: A unified framework and universal optimization techniques.  _arXiv preprint arXiv:2102.09743_ (2021).
  * Hard et al. (2018) Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018\.  Federated learning for mobile keyboard prediction.  _arXiv preprint arXiv:1811.03604_ (2018).
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In _Proceedings of the IEEE international conference on computer vision_. 1026–1034.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Huang et al. (2022) Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. 2022.  Backdoor Defense via Decoupling the Training Process. In _International Conference on Learning Representations_.
  * Huang et al. (2021) Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. 2021.  Evaluating gradient inversion attacks and defenses in federated learning.  _Advances in Neural Information Processing Systems_ 34 (2021), 7232–7241.
  * Kairouz et al. (2021) Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. 2021\.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ 14, 1–2 (2021), 1–210.
  * Khan et al. (2021) Latif U Khan, Walid Saad, Zhu Han, Ekram Hossain, and Choong Seon Hong. 2021.  Federated learning for internet of things: Recent advances, taxonomy, and open challenges.  _IEEE Communications Surveys & Tutorials_ 23, 3 (2021), 1759–1799.
  * Krizhevsky et al. (2009) Alex Krizhevsky et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * Kusetogullari et al. (2020) Huseyin Kusetogullari, Amir Yavariabdi, Abbas Cheddad, Håkan Grahn, and Johan Hall. 2020\.  ARDIS: a Swedish historical handwritten digit dataset.  _Neural Computing and Applications_ 32, 21 (2020), 16505–16518.
  * Li et al. (2017) Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. 2017.  Hyperband: Bandit-Based Configuration Evaluation for Hyperparameter Optimization. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=ry18Ww5ee
  * Li et al. (2021a) Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021a.  Ditto: Fair and robust federated learning through personalization. In _International Conference on Machine Learning_. PMLR, 6357–6368.
  * Li et al. (2020) Xiaoxiao Li, Yufeng Gu, Nicha Dvornek, Lawrence H Staib, Pamela Ventola, and James S Duncan. 2020\.  Multi-site fMRI analysis using privacy-preserving federated learning and domain adaptation: ABIDE results.  _Medical Image Analysis_ 65 (2020), 101765.
  * Li et al. (2021b) Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. 2021b.  FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=6YEQUn0QICG
  * Li et al. (2021c) Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. 2021c.  Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks. In _International Conference on Learning Representations_.
  * Liang et al. (2020) Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020.  Think locally, act globally: Federated learning with local and global representations.  _arXiv preprint arXiv:2001.01523_ (2020).
  * Liu et al. (2018) Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018\.  Fine-pruning: Defending against backdooring attacks on deep neural networks. In _International Symposium on Research in Attacks, Intrusions, and Defenses_. Springer, 273–294.
  * Lyu et al. (2020) Lingjuan Lyu, Han Yu, Xingjun Ma, Lichao Sun, Jun Zhao, Qiang Yang, and Philip S Yu. 2020.  Privacy and robustness in federated learning: Attacks and defenses.  _arXiv preprint arXiv:2012.06337_ (2020).
  * Marfoq et al. (2021) Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. 2021\.  Federated multi-task learning under a mixture of distributions.  _Advances in Neural Information Processing Systems_ 34 (2021), 15434–15447.
  * Matsuda et al. (2022) Koji Matsuda, Yuya Sasaki, Chuan Xiao, and Makoto Onizuka. 2022.  An Empirical Study of Personalized Federated Learning.  _arXiv preprint arXiv:2206.13190_ (2022).
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017\.  Communication-efficient learning of deep networks from decentralized data. In _Artificial intelligence and statistics_. PMLR, 1273–1282.
  * Melis et al. (2019) Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. 2019.  Exploiting unintended feature leakage in collaborative learning. In _2019 IEEE symposium on security and privacy (SP)_. IEEE, 691–706.
  * Nasr et al. (2019) Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019.  Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In _2019 IEEE symposium on security and privacy (SP)_. IEEE, 739–753.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in pytorch.  (2017).
  * Pillutla et al. (2022) Krishna Pillutla, Kshitiz Malik, Abdel-Rahman Mohamed, Mike Rabbat, Maziar Sanjabi, and Lin Xiao. 2022\.  Federated Learning with Partial Model Personalization. In _Proceedings of the 39th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (Eds.). PMLR, 17716–17758.
  * Shejwalkar and Houmansadr (2021) Virat Shejwalkar and Amir Houmansadr. 2021.  Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In _NDSS_.
  * Shejwalkar et al. (2022) Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. 2022.  Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In _2022 IEEE Symposium on Security and Privacy (SP)_. IEEE, 1354–1371.
  * Smith et al. (2017) Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. 2017.  Federated multi-task learning.  _Advances in neural information processing systems_ 30 (2017).
  * Sun et al. (2019) Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. 2019.  Can you really backdoor federated learning?  _arXiv preprint arXiv:1911.07963_ (2019).
  * T Dinh et al. (2020) Canh T Dinh, Nguyen Tran, and Josh Nguyen. 2020.  Personalized federated learning with moreau envelopes.  _Advances in Neural Information Processing Systems_ 33 (2020), 21394–21405.
  * Tan et al. (2022) Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. 2022\.  Towards personalized federated learning.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022).
  * Tolpegin et al. (2020) Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. 2020.  Data poisoning attacks against federated learning systems. In _Computer Security–ESORICS 2020: 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK, September 14–18, 2020, Proceedings, Part I 25_. Springer, 480–501.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Wang et al. (2020) Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. 2020.  Attack of the tails: Yes, you really can backdoor federated learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 16070–16084.
  * Wu et al. (2022a) Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Chao Shen, and Hongyuan Zha. 2022a.  Backdoorbench: A comprehensive benchmark of backdoor learning.  _arXiv preprint arXiv:2206.12654_ (2022).
  * Wu and Wang (2021) Dongxian Wu and Yisen Wang. 2021.  Adversarial neuron pruning purifies backdoored deep models.  _Advances in Neural Information Processing Systems_ 34 (2021), 16913–16925.
  * Wu et al. (2022b) Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ziyu Liu, Zheng Xu, and Virginia Smith. 2022b.  Motley: Benchmarking heterogeneity and personalization in federated learning.  _arXiv preprint arXiv:2206.09262_ (2022).
  * Xie et al. (2020) Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. 2020\.  DBA: Distributed Backdoor Attacks against Federated Learning. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=rkgyS0VFvr
  * Zawad et al. (2021) Syed Zawad, Ahsan Ali, Pin-Yu Chen, Ali Anwar, Yi Zhou, Nathalie Baracaldo, Yuan Tian, and Feng Yan. 2021\.  Curse or redemption? how data heterogeneity affects the robustness of federated learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 10807–10814.
  * Zhang et al. (2022b) Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. 2022b.  Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10174–10183.
  * Zhang et al. (2022a) Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. 2022a.  Neurotoxin: durable backdoors in federated learning. In _International Conference on Machine Learning_. PMLR, 26429–26446.
  * Zheng et al. (2022) Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. 2022\.  Data-free backdoor removal based on channel lipschitzness. In _European Conference on Computer Vision_. Springer, 175–191.
  * Zhu et al. (2019) Ligeng Zhu, Zhijian Liu, and Song Han. 2019.  Deep leakage from gradients.  _Advances in neural information processing systems_ 32 (2019).

##  Appendix A1 Datasets and Models

##### Datasets.

We conduct experiments on two widely used dataset, FEMNIST and CIFAR-10 in FL
literature (McMahan et al., 2017; Caldas et al., 2018; Chen et al., 2022;
Matsuda et al., 2022; Wu et al., 2022b; Wang et al., 2020). These two datasets
represent two different settings of Non-IID respectively, feature-skew and
label skew (Chen et al., 2022; Lyu et al., 2020; Zawad et al., 2021).

  * •

FEMNIST: The Federated Extended MNIST (FEMNIST) is a widely used FL dataset
for 62-class handwritten character recognition (Caldas et al., 2018). The
original FEMNIST dataset contains 3,550 clients and each client corresponds to
a character writer from EMNIST(Cohen et al., 2017). We adopt the sub-sampled
version in our evaluations, which contains 200 clients and totally $43,400$
images with resolution of $28\times 28$ pixels. The distribution of samples
number per client is shown in Figure A10.

  * •

CIFAR-10 is a 10-class image classiﬁcation dataset containing 60,000 colored
images with resolution of 32x32 pixels. We use Dirichlet allocation to split
this dataset into 100 clients with $\alpha=0.5$. The distribution of samples
number per client is shown in Figure A10.

##### Models.

Following the previous pFL works (T Dinh et al., 2020; Liang et al., 2020;
Chen et al., 2022; Marfoq et al., 2021), we utilize a simple ConvNet on
FEMNIST and CIFAR10. This ConvNet model consists of two convolutional layers
with the $5\times 5$ kernel, max pooling, batch normalization, ReLU
activation, and two dense linear layers. The hidden size of linear layer is
$2,048$ and $512$ on FEMNIST and CIFAR10 respectively. To align with backdoor
attacks works (Wang et al., 2020; Wu et al., 2022a; Sun et al., 2019; Zhang et
al., 2022a), we also adopt the larger model, ResNet-18 on CIFAR-10 dataset.

##  Appendix A2 Implementation Details

##### Environment.

We implement our evaluations based on Pytorch (Paszke et al., 2017) and
FederetadScope framework 333https://github.com/alibaba/FederatedScope. And all
experiments are conducted on a cluster of 8 NVIDIA GeForce GTX 2080 Ti GPUs.

###  A2.1. More Details of Attacks Hyper-parameters

Following attack hyparameters provided in (Wu et al., 2022a), for BadNet
attack, we choose the $3\times 3$ grid pattern and put it at downright corner
of poisoned samples. For Blended attack, we choose the hello-kitty pattern and
set the blending ratio $\alpha$ as $0.2$. For SIG attack, we set the amplitude
of sinusoidal signal as 20 ($[0,255]$).

Figure A10. The violin plot of number of samples per client for CIFAR-10 and
FEMNIST datasets.

(a) IID: ResNet-18@CIFAR-10

(b) IID: Convnet2@CIFAR-10

Figure A11. Comparison of backdoor attacks on different pFL methods. (a):
Results on ResNet-18 and CIFAR-10; (b) Results on ConvNet and CIFAR-10. In the
figures, the solid line means ASR, and the dashed line means the C-Acc.

###  A2.2. Implementation Details of pFL methods

Following pFL-bench
444https://github.com/alibaba/FederatedScope/tree/master/benchmark/pFL-Bench
(Chen et al., 2022), we also use hyper-parameter searching (HPO) algorithm,
HyperBand (Li et al., 2017), to ﬁnd the best hyper-parameters for all the
baselines on all datasets. We use the vanilla SGD optimizer and set the batch
size of local datsets as $32$ for all methods. Here, we provide adopted hyper-
parameters for all baseline methods:

  * •

FedAvg: we set learning rate as $0.1$ and epoch number as $2$ for each client
on two datasets.

  * •

Ditto: we set learning rate as $0.1$ and $\lambda$ as $0.1$ for each client.
The epoch number is $2$ or $3$ on CIFAR-10 or FEMNIST.

  * •

FedEM: ResNet-18 and CIFAR-10: we set learning rate as $0.5$ and epoch number
as $1$. ConvNet and CIFAR-10 or FEMNIST: we set learning rate as $0.05$ and
epoch number as $2$. The number of base model for two datasets is $3$.

  * •

Fine-tuning: On CIFAR-10: we set learning rate of FT as $0.01$ and epoch
number of FT as $2$. On FEMNIST: we set learning rate of FT as $0.01$ and
epoch number of FT as $3$.

  * •

pFedMe: On ResNet-18 and CIFAR-10: we set learning rate as $0.5$ and epoch
number as $3$. We set local approximation steps $K$ as $1$, average moving
parameter $\beta$ as $1$ and $\lambda$ as $0.5$; On ConvNet and CIFAR-10: we
set learning rate as $0.1$ and epoch number as $3$. We set local approximation
steps $K$ as $2$, average moving parameter $\beta$ as $1$, and $\lambda$ as
$0.5$; On ConvNet and FEMNIST: learning rate is $0.1$, epoch number is $2$,
local approximation steps $K$ is $3$, average moving parameter $\beta$ is $1$,
$\lambda$ is $0.8$.

  * •

FedBN: On ResNet-18 and CIFAR-10: we set learning rate as $0.5$ and epoch
number as $2$; On ConvNet and CIFAR-10: we learning rate as $0.1$ and epoch
number as $2$; On ConvNet and FEMNIST: we set learning rate as $0.01$ and
epoch number as $2$.

  * •

FedRep: On CIFAR-10: we set learning rate of $\bm{\theta}^{s}$ as $0.1$, epoch
number of $\bm{\theta}^{s}$ as $1$, learning rate of $\bm{\theta}^{p}$as
$0.005$, and epoch number of $\bm{\theta}^{p}$ as $1$; On FENNIST: we set
learning rate of $\bm{\theta}^{s}$ as $0.1$, epoch number of $\bm{\theta}^{s}$
as $2$, learning rate of $\bm{\theta}^{p}$ as $0.1$, and epoch number of
$\bm{\theta}^{p}$ as $1$.

##  Appendix A3 Detailed Experimental Results

###  A3.1. Experimental Results about IID Setting

In this section, we demonstrate attack results on pFL methods under IID
setting in which we uniformly and randomly sample images from each class of
CIFAR-10 for each client. The results are shown in Figure A11. We have similar
observations as in the Non-IID scenario: ❶ Under IID setting, for all pFL
methods, backdoor attacks do still not affect C-Acc. ❷ Under IID setting,
Blended attack (the solid red line) also achieves the best ASR on all pFL
methods and CIFAR-10 dataset. ❸ pFL methods with partial model-sharing also
achieve better defense performance against backdoor attacks. Compared with
attack results under Non-IID setting, attack results on FedBN under IID
setting are higher. We have analysed reasons in Section 5.1. ❹ Under IID
scenario, we still observe that backdoor attacks achieves better attack
performance on high-capacity model (ResNet-18) than on low-capacity model
(ConvNet).