# Model Transferring Attacks to Backdoor
HyperNetwork in Personalized Federated Learning

Phung Lai
New Jersey Institute of Technology
Newark, NJ 07102
<EMAIL_ADDRESS>
&NhatHai Phan
New Jersey Institute of Technology
Newark, NJ 07102
<EMAIL_ADDRESS>
Abdallah Khreishah
New Jersey Institute of Technology
Newark, NJ 07102
<EMAIL_ADDRESS>
&Issa Khalil
Qatar Computing Research Institute
Doha, Qatar
<EMAIL_ADDRESS>
&Xintao Wu
University of Arkansas
Fayetteville, AR 72701
<EMAIL_ADDRESS>

###### Abstract

This paper explores previously unknown backdoor risks in HyperNet-based
personalized federated learning (HyperNetFL) through poisoning attacks. Based
upon that, we propose a novel model transferring attack (called HNTroj), i.e.,
the first of its kind, to transfer a local backdoor infected model to all
legitimate and personalized local models, which are generated by the
HyperNetFL model, through consistent and effective malicious local gradients
computed across all compromised clients in the whole training process. As a
result, HNTroj reduces the number of compromised clients needed to
successfully launch the attack without any observable signs of sudden shifts
or degradation regarding model utility on legitimate data samples making our
attack stealthy. To defend against HNTroj, we adapted several backdoor-
resistant FL training algorithms into HyperNetFL. An extensive experiment that
is carried out using several benchmark datasets shows that HNTroj
significantly outperforms data poisoning and model replacement attacks and
bypasses robust training algorithms.

##  1 Introduction

Recent and upcoming data privacy regulations [1] pose significant challenges
for machine learning (ML) applications that collect sensitive user data at
servers controlled by the owners of these applications. Federated learning
(FL) [2] is a promising way to address these challenges by enabling clients to
jointly train ML models via a coordinating server without sharing their data.

Although offering better data privacy, FL typically suffers from the disparity
of model performance caused by the non-independent and identically distributed
(non-iid) data distribution across clients [3]. One of the state-of-the-art
approaches to address this problem in FL is using a single joint neural
network, called HyperNetFL, to generate local models using personalized
descriptors optimized for each client independently [4]. This allows us to
perform smart gradient and parameter sharing.

Despite the superior performance, the unique training approach of HyperNetFL
poses previously unknown risks to backdoor attacks typically carried out
through poisoning in FL [5]. In backdoor attacks, an adversary manipulates the
training process to cause model misclassification on a subset of chosen data
samples [6, 7]. In FL, the adversary tries to construct malicious gradients or
model updates that encode the backdoor. When aggregated with other clients’
updates, the aggregated model exhibits the backdoor.

We investigate backdoor attacks against HyperNetFL and formulate robust
HyperNetFL training as defenses in this work. Our developed attack (called
HNTroj) is based on consistently and effectively crafting malicious local
gradients across compromised clients using a single backdoor-infected model to
enforce HyperNetFL generating local backdoor-infected models disregarding
their personalized descriptors. An extensive analysis and evaluation using
benchmark datasets in non-iid settings show that HNTroj notably outperforms
existing model replacement and data poisoning attacks bypassing recently
developed robust federated training algorithms adapted to HyperNetFL with
small numbers of compromised clients.

##  2 Background

### Federated Learning (FL)

FL is a multi-round communication protocol between a coordination server and a
set of $N$ clients to jointly train a ML model. Each client $i$ has a set of
training samples $D_{i}=\\{(x_{j},y_{j})\\}_{j\in[n_{i}]}$ where $n_{i}$ is
the number of training samples, $x_{j}\in\mathbb{R}^{d}$ is the input features
associated with ground-truth label $y_{j}\in\mathbb{Z}^{C}$ that is one-hot
encoded with $C$ categorical model outcomes. Clients try to minimize the
average of their loss functions
$\arg\min_{\theta}\frac{1}{N}\sum_{i=1}^{N}L_{i}(\theta)$, where
$L_{i}(\theta)$ is the loss function of the client $i$ and $\theta$ is a set
of model weights. For instance, given a model $f$, e.g., a neural network,
$L_{i}$ is commonly an empirical risk minimization given a cross entropy error
function $L$ to penalize the mismatch between the predicted values
$f(x,\theta)$ and the ground-truth label $y_{x}$:
$L_{i}(\theta)=\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}L\big{(}f(x_{j},\theta),y_{j}\big{)}$.

### HyperNetwork-based Personalized FL (HyperNetFL)

To address the disparity of model utility across clients, HyperNetFL [8, 4]
uses a neural network $h(v_{i},\varphi)$ located at the server to output the
weights $\theta_{i}$ for each client $i$ using a (trainable) descriptor
$v_{i}$ as input and model weights $\varphi$, that is,
$\theta_{i}=h(v_{i},\varphi)$. HyperNetFL offers a natural way to share
information across clients through the weights $\varphi$ while maintaining the
personalization of each client via the descriptor $v_{i}$. In other words,
HyperNetFL learns a family of personalized models
$\\{\theta_{i}=h(v_{i},\varphi)\\}_{i\in[N]}$. To achieve this goal, the
clients and the server will try to minimize their loss functions:
$\arg\min_{\varphi,\\{v_{i}\\}_{i\in[N]}}\frac{1}{N}\sum_{i\in[N]}L_{i}(h(v_{i},\varphi))$.

The training protocol of a HyperNetFL (Alg. 1) is generally similar to the
protocol of typical FL (Appx. A). However, at round $t$, there are three
differences in training HyperNetFL compared with FedAvg [5]: (1) There is no
global model $\theta$ generated by the aggregation function $\mathcal{G}$ in
Eq. 3; (2) Each client $i\in S_{t}$ receives the personalized weights
$\theta^{i}_{t}$ from the HyperNetFL. The client $i$ computes the local
gradient $\bigtriangleup\theta^{i}_{t}$ and then sends it to the server; (3)
The server uses all the local gradients received from all the clients $i\in
S_{t}$ to update $\varphi$ and the descriptors $v_{i}$ using general update
rules in Lines 9 and 10 (Alg. 1); and (4) The size of the HyperNetFL weights
$\varphi$ is significanlty larger than $\theta$ ($\varphi\simeq 10\mathbf{x}$
size of $\theta$) causing extra computational cost at the server. This
protocol is more general than using only one client at a communication round
as in [4]. By using a small batch of clients per communication round, we
observe that we can enhance the performance of the HyperNetFL and make it more
reliable.

### Backdoor and Poisoning Attacks

Training time poisoning attacks against ML and FL models can be classified
into byzantine and backdoor attacks. In byzantine attacks, the adversarial
goal is to degrade or severely damage the model test accuracy [9, 10, 11, 12].
Byzantine attacks are relatively detectable by tracking the model accuracy on
validation data [13]. Meanwhile, in backdoor attacks, the adversarial goal is
to cause model misclassification on a set of chosen inputs without affecting
model accuracy on legitimate data samples. A well-known way to carry out
backdoor attacks is using Trojans [14, 15]. A Trojan is a carefully crafted
pattern, e.g., a brand logo, blank pixels, added into legitimate data samples
causing the desired misclassification. A recently developed image warping-
based Trojan mildly deforms an image by applying a geometric transformation
[16] to make it unnoticeable to humans and bypass all well-known Trojan
detection methods, such as Neural Cleanse [17], Fine-Pruning [18], and STRIP
[19]. The adversary applies the Trojan on legitimate data samples to activate
the backdoor at the inference time.

The training data is scattered across clients in FL, and the server only
observes local gradients. Therefore, backdoor attacks are typically carried by
a small set of compromised clients fully controlled by an adversary to
construct malicious local gradients and send them to the server. The adversary
can apply data poisoning (DPois) and model replacement approaches to create
malicious local gradients. In DPois [20, 21], compromised clients train their
local models on Trojaned datasets to construct malicious local gradients, such
that the aggregated model at the server exhibits the backdoor. DPois may take
many training rounds to implant the backdoor into the aggregated model.
Meanwhile, in model replacement [6], the adversary constructs malicious local
gradients, such that the aggregated model at the server will closely
approximate or be replaced by a predefined Trojaned model. To some extent,
model replacement is highly severe since it can be effective after only one
training round [22].

Input: Number of rounds $T$, number of local rounds $K$, server’s learning
rates $\lambda$ and $\zeta$, clients’ learning rate $\eta$, number of clients
$N$, and $L_{i}(B)$ is the loss function $L_{i}(\theta)$ on a mini-batch $B$

Output: ${\varphi},v_{i}$

1: for $t=1,\ldots,T$ do

2: Sample clients $S_{t}$

3: for each client $i\in S_{t}$ do

4: set $\theta^{i}_{t}=h(v_{i},\varphi)$ and
$\tilde{\theta}^{i}=\theta^{i}_{t}$

5: for $k=1,\ldots,K$ do

6: sample mini-batch $B\subset D_{i}$

7:
$\tilde{\theta}_{k+1}^{i}=\tilde{\theta}_{k}^{i}-\eta\bigtriangledown_{\tilde{\theta}_{k}^{i}}L_{i}(B)$

8: $\bigtriangleup\theta^{i}_{t}=\tilde{\theta}^{i}_{K}-\theta^{i}_{t}$

9:
$\varphi=\varphi-\frac{\lambda}{|S_{t}|}\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

10: $\forall i\in
S_{t}:v_{i}=v_{i}-\zeta\bigtriangledown_{v_{i}}\varphi^{\top}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

Algorithm 1 HyperNetFL with Multiple Clients per Round

To our knowledge, these attacks are not primarily designed for HyperNetFL, in
which there is no aggregated model $\theta$ (Eq. 3) at the server. That poses
an unknown risk of backdoors through poisoning attacks in HyperNetFL.

##  3 Data Poisoning in HyperNetFL

We first consider both white-box and black-box model replacement threat
models. Although unrealistic, the white-box setting (Appx. B) allows us to
identify the upper bound risk. Meanwhile, the black-box setting will inform a
realistic risk in practice. Interested readers can refer to Appx. B for
details regarding the white-box setting and the adaptation of model
replacement attacks into HyperNetFL, called HNRepl. Let us present our black-
box threat model and adapted attacks as follows.

### Black-box Threat Model

At round $t$, an adversary fully controls a small set of compromised clients
$\complement$. The adversary cannot modify the training protocol of the
HyperNetFL at the server and at the legitimate clients. The adversary’s goal
is to implant backdoors in all local models
$\\{\theta^{i}=h(v_{i},\varphi)\\}_{i\in[N]}$ by minimizing a backdoor
poisoning objective:

$\small\arg\min_{\varphi,\\{v_{i}\\}_{i\in[N]}}\frac{1}{N}\sum_{i=1}^{N}\big{[}L_{i}(h(v_{i},\varphi))+L^{b}_{i}(h(v_{i},\varphi))\big{]}$
(1)

where $L^{b}_{i}$ is the (backdoor) loss function of the $i^{th}$ client given
Trojaned examples $x_{j}+\mathcal{T}$ with the trigger $\mathcal{T}$ [16],
e.g.,
$L^{b}_{i}=\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}L\big{(}f(x_{j}+\mathcal{T},\theta),y^{b}_{j}\big{)}$
where $y^{b}_{j}$ is the targeted label for the sample $x_{j}+\mathcal{T}$.
One can vary the portion of Trojaned samples to optimize the attack
performance. This black-box threat model is applied throughout this paper.

We found that HNRepl is infeasible in the black-box setting, since the weights
$\varphi$ and the descriptors $\\{v_{i}\\}_{i\in[N]}$ are hidden from all the
clients. Also, there is lack of effective approach to infer (large) $\varphi$
and $\\{v_{i}\\}_{i\in[N]}$ given a small number of compromised clients (Appx.
B).

### Data Poisoning (DPois) in HyperNetFL

To address the issues of HNRepl, we look into another fundamental approach,
that is applying black-box DPois. The pseudo-code of the attack is in Alg. 4
(Appx. C). At round $t$, the compromised clients $c\in\complement\cap S_{t}$
receive the personalized model weights $\theta^{c}_{t}$ from the server. Then,
they compute malicious local gradients $\bigtriangleup\bar{\theta}^{c}_{t}$
using their Trojan datasets, i.e., their legitimate data combined with
Trojaned data samples, to minimize their local backdoor loss functions:
$\forall c\in\complement\cap
S_{t}:\theta^{c*}_{t}=\arg\min_{\theta^{c}_{t}}[L_{c}(h(v_{c},\varphi))+L^{b}_{c}(h(v_{c},\varphi))]$,
after a certain number of local steps $K$ of SGD. All the malicious local
gradients $\\{\bigtriangleup\bar{\theta}^{c}_{t}\\}_{c\in\complement}$ are
sent to the server. If the HyperNetFL updates the model weights $\varphi$ and
the descriptors $\\{v_{i}\\}_{i\in S_{t}}$ using
$\\{\bigtriangleup\bar{\theta}^{c}_{t}\\}_{c\in\complement}$, the local model
weights generated by the HyperNetFL $h(\cdot,\varphi)$ will be Trojan
infected. This is because the update rules of the HyperNetFL become the
gradient of an approximation to the Trojaned surrogate loss

$\small\frac{1}{2}(\sum_{c\in\complement}\|\theta^{c*}-h(v_{c},\varphi)\|_{2}^{2}+\sum_{i\in
N\setminus\complement}\|\theta^{i*}-h(v_{i},\varphi)\|_{2}^{2})$ (2)

where $\theta^{c*}$ is the optimal local Trojaned model weights, $i\in
N\setminus\complement$ are legitimate clients and their associated legitimate
loss functions $\theta^{i*}=\arg\min_{\theta^{i}}L_{i}(h(v_{i},\varphi)$.

### Disadvantages of DPois

Obviously, the larger the number of compromised clients is, i.e., a larger
$\complement$ and a smaller $N\setminus\complement$, the more effective the
attack will be. Although more practical than the HNRepl in poisoning
HyperNetFL, there are two issues in the black-box DPois: (1) The attack causes
notable degradation in model utility on the legitimate data samples; and (2)
The attack requires a more significant number of compromised clients to be
successful. These disadvantages reduce the stealthiness and effectiveness of
the attack, respectively.

Input: Number of rounds $T$, number of local rounds $K$, server’s learning
rates $\lambda$ and $\zeta$, clients’ learning rate $\eta$, number of clients
$N$, a random and dynamic learning rate $\psi\sim\mathcal{U}[a,b]$, and
$L_{i}(B)$ is the loss function $L_{i}(\theta)$ on a mini-batch $B$

Output: ${\varphi},v_{i}$

1: for $t=1,\ldots,T$ do

2: Sample clients $S_{t}$

3: for each legitimate client $i\in S_{t}\setminus\complement$ do

4: set $\theta^{i}_{t}=h(v_{i},\varphi)$ and
$\tilde{\theta}^{i}=\theta^{i}_{t}$

5: for $k=1,\ldots,K$ do

6: sample mini-batch $B\subset D_{i}$

7:
$\tilde{\theta}_{k+1}^{i}=\tilde{\theta}_{k}^{i}-\eta\bigtriangledown_{\tilde{\theta}_{k}^{i}}L_{i}(B)$

8: $\bigtriangleup\theta^{i}_{t}=\tilde{\theta}^{i}_{K}-\theta^{i}_{t}$

9: for each compromised client $c\in S_{t}\cap\complement$ do

10:
$\bigtriangleup\bar{\theta}^{c}_{t}=\big{(}\psi^{c}_{t}\sim\mathcal{U}[a,b]\big{)}\big{[}X-h(v_{c},\varphi)\big{]}$

11:
$\varphi=\varphi-\frac{\lambda}{|S_{t}|}\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

12: $\forall i\in
S_{t}:v_{i}=v_{i}-\zeta\bigtriangledown_{v_{i}}\varphi^{\top}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

Algorithm 2 Model Transferring Attack in HyperNetFL (HNTroj)

The root cause issue of the DPois is the lack of consistency in deriving the
malicious local gradient
$\bigtriangleup\bar{\theta}^{c}_{t}=\theta^{c*}_{t}-h(v_{c},\varphi)$ across
communication rounds and among compromised clients to outweigh the local
gradients from legitimate clients. First, $\theta^{c*}_{t}$ is derived after
(a small number) $K$ local steps of applying SGD to minimize the local
backdoor loss function
$\theta^{c*}_{t}=\arg\min_{\theta^{c}_{t}}[L_{c}(h(v_{c},\varphi))+L^{b}_{c}(h(v_{c},\varphi))]$,
in which the local model weights $h(v_{c},\varphi)$ and the loss functions
(i.e, $L_{c}$ and $L_{c}^{b}$) are varying among compromised clients due to
the descriptors $\\{v_{c}\\}_{c\in\complement}$ in addition to the their
dissimilar local datasets. As a result, the (supposed to be) Trojaned model
weights $\\{\theta^{c*}_{t}\\}_{c\in\complement}$ are unalike among
compromised clients. Second, a small number of local training steps $K$ (i.e.,
given a limited computational power on the compromised clients) is not
sufficient to approximate a good Trojaned model $\theta^{c*}_{t}$. The
adversary can increase the local training steps $K$ if more computational
power is available. However, there is still no guarantee that
$\\{\theta^{c*}_{t}\\}_{c\in\complement}$ will be alike without the control
over the dissimilar descriptors $\\{v_{c}\\}_{c\in\complement}$. Third, the
local model weights $h(v_{c},\varphi)$ change after every communication round
and are heavily affected by the local gradients from legitimate clients.
Consequently, the malicious local gradients
$\bigtriangleup\bar{\theta}^{c}_{t}$ derived across all the compromised
clients $c\in\complement$ do not synergistically optimize the approximation to
the Trojaned surrogate loss function (Eq. 2) such that the outputs of the
HyperNetFL $h(\cdot,\varphi)$ are Trojan infected.

Thus, developing a practical, stealthy, and effective backdoor attack in
HyperNetFL is non-trivial and an open problem.

##  4 Model Transferring Attack

To overcome the lack of consistency in deriving the malicious local gradients
in DPois and avoid sudden shifts in model utility on legitimate data samples
in HNRepl, we propose in this work a novel model transferring attack (HNTroj)
against HyperNetFL.

In HNTroj (Alg. 2), our idea is to replace $\theta^{c}_{t}$ with a Trojaned
model $X$ across all the compromised clients $c\in\complement$ and in all
communication rounds $t$ to compute the malicious local gradients: $\forall
c\in\complement,t\in[T]:\bigtriangleup\bar{\theta}^{c}_{t}=\psi^{c}_{t}\big{[}X-h(v_{c},\varphi)\big{]}$,
where $\psi^{c}_{t}$ is dynamic learning rate randomly sampled following a
specific distribution, e.g., uniform distribution $\mathcal{U}[a,b]$, $a<b$,
and $a,b\in(0,1]$. In practice, the adversary can collect its own data that
shares a similar distribution with legitimate clients to locally train the
Trojaned model $X$.

By doing so, we achieve several key advantages, as follows:

(1) The gradients $\bigtriangleup\bar{\theta}^{c}_{t}$ become more effective
in creating backdoors, since $X$ is a better optimized Trojaned model than
$\\{\theta^{c*}_{t}\\}_{c\in\complement}$.

(2) The gradients across compromised clients synergistically approximate the
Trojaned surrogate loss (Eq. 2) to closely align the outputs of
$h(\cdot,\varphi)$ to a unified Trojan model $X$ through the term
$\frac{1}{2}\sum_{c\in\complement}\|X-h(v_{c},\varphi)\|_{2}^{2}$ disregarding
the varying descriptors $\\{v_{i}\\}_{i\in N}$ and their dissimilar local
datasets. The new Trojaned surrogate loss is:
$\frac{1}{2}(\sum_{c\in\complement}\|X-h(v_{c},\varphi)\|_{2}^{2}+\sum_{i\in
N\setminus\complement}\|\theta^{i*}-h(v_{i},\varphi)\|_{2}^{2})$.

(3) The gradients $\bigtriangleup\bar{\theta}^{c}_{t}$ become stealthier since
updating the HyperNetFL with $\bigtriangleup\bar{\theta}^{c}_{t}$ will
significantly improve the model utility on legitimate data samples. This is
because $X$ has a better model utility on legitimate data samples than the
local models of legitimate clients $\\{\theta_{t}^{i*}\\}_{i\in
N\setminus\complement}$. More importantly, by keeping the random and dynamic
learning rate $\psi^{c}_{t}$ only known to the compromised client $c$, we can
prevent the server from tracking our Trojaned model $X$ or identifying some
suspicious behavior patterns from the compromised client.

(4) Theorem 2 (Appx. E) shows that the $l_{2}$-norm distance between the local
model $\theta^{c}_{t}$ of a compromised client generated by the HyperNetFL
$h(v_{c},\varphi)$ and the Trojaned model $X$, i.e.,
$\|\theta_{t}^{c}-X\|_{2}$, is bounded by
$(\frac{1}{a}-1)\|\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}\|_{2}+\|\xi\|_{2}$,
where $t^{\prime}$ is the closest round the compromised client $c$
participated in before $t$, and $\xi\in\mathcal{R}^{m}$ is a small error rate.
When the HyperNetFL model converges, e.g., $t^{\prime},t\approxeq T$,
$\|\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}\|_{2}$ and $\|\xi\|_{2}$ become
tiny ensuring that the output of the HyperNetFL given the compromised client
$\theta_{T}^{c}=h(v_{c},\varphi)$ converges into a bounded and low loss area
surrounding the Trojaned model $X$, i.e., $\|\theta_{T}^{c}-X\|_{2}$ is tiny,
to imitate the model convergence behavior of legitimate clients.

Consequently, HNTroj requires a smaller number of compromised clients to be
highly effective compared with DPois. Also, HNTroj is stealthier than the
(white-box) HNRepl and DPois by avoiding degradation and shifts in model
utility on legitimate data samples during the whole poisoning process.

##  5 Robust HyperNetFL Training

In this section, we first investigate the state-of-the-art defenses against
backdoor poisoning in FL and point out the differences between FL and
HyperNetFL. We then present our robust training approaches adapted from
existing defenses for HyperNetFL against HNTroj.

Existing defense approaches against backdoor poisoning in ML can be
categorised into two lines: 1) Trojan detection in the inference phase and 2)
robust aggregation to mitigate the impacts of malicious local gradients in
aggregation functions. In this paper, we applied the state-of-the-art warping-
based Trojans bypassing all the well-known Trojan detection methods, i.e.,
Neural Cleanse [17], Fine-Pruning [17], and STRIP [19], in the inference
phase. HNTroj does not affect the warping-based Trojans (Figs. 14 and 15,
Appx. G); thus bypassing these detection methods. Based upon that, we focus on
identifying which robust aggregation approaches can be adapted to HyperNetFL
and how.

### Robust Aggregation

Several works have proposed robust aggregation approaches to deter byzantine
attacks in typical FL, such as coordinate-wise median, geometric median,
$\alpha$-trimmed mean, or a variant and combination of such techniques [23].
Recently proposed approaches include weight-clipping and noise addition with
certified bounds, ensemble models, differential privacy (DP) optimizers, and
adaptive and robust learning rates (RLR) across clients and at the server [24,
13].

Despite differences, existing robust aggregation focuses on analysing and
manipulating the local gradients $\bigtriangleup\theta_{t}^{i}$, which share
the global aggregated model $\theta_{t}$ as the same root, i.e., $\forall
i\in[N]:\bigtriangleup\theta^{i}_{t}=\theta^{i}_{t}-\theta_{t}$. The
fundamental assumption in these approaches is that the local gradients from
compromised clients
$\\{\bigtriangleup\bar{\theta}_{t}^{c}\\}_{c\in\complement}$ and from
legitimate clients $\\{\bigtriangleup\theta_{t}^{i}\\}_{i\in
N\setminus\complement}$ are different in terms of magnitude and direction.

### Robust FL Training v.s. HyperNetFL

Departing from typical FL, the local gradients in HyperNetFL have different
and personalized roots $h(v_{i},\varphi)$, i.e., $\forall
i\in[N]:\bigtriangleup\theta^{i}_{t}=\theta^{i*}_{t}-h(v_{i},\varphi)$.
Therefore, the local gradients $\\{\bigtriangleup\theta^{i}_{t}\\}_{i\in N}$
in HyperNetFL may diverge in magnitude and direction in their own sub-optimal
spaces, making it challenging to adapt existing robust aggregation methods
into HyperNetFL. More importantly, manipulating the local gradients alone can
significantly affect the original update rules of the HyperNetFL, which are
derived based on the combination between the local gradients and the
derivatives of $\varphi$ and $v_{i}$ given the output of $h(\cdot,\varphi)$,
i.e., $(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}$ and
$\bigtriangledown_{v_{i}}\varphi^{\top}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}$,
respectively. For instance, adapting the recently developed RLR [13] on the
local gradients can degrade the model utility on legitimate data samples to a
random guess level on several benchmark datasets (Appx. D). In addition, the
significantly large size of $\varphi$ introduces an expensive computational
cost in adapting (statistics-based) robust aggregation approaches into
HyperNetFL against HNTroj.

### Robust HyperNetFL Training

Based on our observation, to avoid damaging the update rule of HyperNetFL, a
suitable way to develop robust HyperNetFL training algorithms is to adapt
existing robust aggregation on the set of $\varphi$’s gradients
$\\{(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}\\}_{i\in
S_{t}}$ given
$\varphi=\varphi-\frac{\lambda}{|S_{t}|}\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$.
It is worth noting that we may not need to modify the update rule of the
descriptors $\\{v_{i}\\}_{i\in S_{t}}$ since $v_{i}$ is a personalized update
that does not affect the updates of any other descriptors $\\{v_{j}\\}_{j\neq
i}$ and the model weights $\varphi$.

### Client-level DP Optimizer

Among robust training against backdoor poisoning attacks, differential privacy
(DP) optimizers, weight-clipping and noise addition can be adapted to defend
against HNTroj. Since they share the same spirit, that is, clipping gradients
from all the clients before adding noise into their aggregation, we only
consider DP optimizers in this paper without loss of generality. In specific,
we consider a DP optimizer, which can be understood as the weight updates
$\varphi$ are not excessively influenced by any of the local gradients
$(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$
where $i\in S_{t}$. By clipping every $\varphi$’s gradients under a pre-
defined $l_{2}$-norm $\mu$, we can bound the influence of a single client’s
gradient
$(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$
to the model weights $\varphi$. To make the gradients indistinguishable, we
add Gaussian noise into the $\varphi$’s gradient aggregation:
$\varphi=\varphi-\frac{\lambda}{|S_{t}|}\big{[}\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}+\mathcal{N}(0,\sigma^{2}\mu^{2}\mathbf{I})\big{]}$,
where $\sigma$ is a predefined noise scale. The pseudo-code of our approach is
in Alg. 5, Appx. F. We utilize this client-level DP optimizer as an effective
defense against HNTroj. As in [24], we focus on how parameter configurations
of the client-level DP optimizer defend against HNTroj with minimal utility
loss, regardless of the privacy provided.

###  $\alpha$-Trimmed Norm

In addition, among robust aggregation approaches against byzantine attacks,
median-based approaches, $\alpha$-trimmed mean, and variants of these
techniques [25, 26] can be adapted to HyperNetFL against HNTroj by applying
them on the gradients of $\varphi$. Without loss of generality, we adapt the
well-applied $\alpha$-trimmed mean approach [23] into HyperNetFL to eliminate
potentially malicious $\varphi$’s gradients in this paper. The adapted
algorithm needs to be less computational resource hungry in order for it to
efficiently work with the large size of $\varphi$. Therefore, instead of
looking into each element of the $\varphi$’s gradient as in $\alpha$-trimmed
mean, we trim the top $\frac{\alpha}{2}$% and the bottom $\frac{\alpha}{2}$%
of the gradients
$\\{(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}\\}_{i\in
S_{t}}$ that respectively have the highest and lowest magnitudes quantified by
an $l_{2}$-norm, i.e.,
$\|(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}\|_{2}$.
The remaining gradients after the trimming, denoted
$\\{(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}\\}_{i\in
S^{\alpha-trim}_{t}}$, are used to update the HyperNetFL model weights
$\varphi$, i.e., $\varphi=\varphi-\frac{\lambda}{|S^{\alpha-
trim}_{t}|}\sum_{i=1}^{|S^{\alpha-
trim}_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$.
The descriptors $\\{v_{i}\\}_{i\in S^{\alpha-trim}_{t}}$ are updated normally.
The pseudo-code of the $\alpha$-trimmed norm for HyperNetFL is in Alg. 6,
Appx. F.

Regarding other approaches, including robustness bounds against backdoor
attacks [27, 28], our detailed analysis on why these approaches cannot be
trivially adapted to HyperNetFL is in Appx. F.

##  6 Experimental Results

We focus on answering the following three questions in our evaluation: (1)
Whether HNTroj is more effective than DPois in HyperNetwork for FL? (2) What
is the percentage of compromised clients required for an effective attack? and
(3) Is it possible to defend against HNTroj, and what is the cost and
limitations of such defenses?

(a) #Compromised clients $\complement=1$

(b) #Compromised clients $\complement=10$

(c) Summary

Figure 1:  Legitimate ACC and Backdoor SR comparison for DPois, HNTroj, and
Clean model over different numbers of compromised clients in the CIFAR-10
dataset. A complete version is in Appx. G. (Fig. 1a has the same legend as in
Fig. 1b).

### Data and Model Configuration

We conduct an extensive experiment on CIFAR-10 [29] and Fashion MNIST datasets
[30]. To generate non-iid data distribution across clients in terms of classes
and size of local training data, we randomly sample two classes for each
client and a sampling rate follows $\mathcal{N}(0,1)$ distribution. We use
$100$ clients and the class $0$ as a targeted class $y^{b}_{j}$ (Eq. 4) in
each dataset. We divide each dataset into three non-overlapping sets: $10,000$
samples for testing, $10,000$ samples for training $X$, and the rest for
training. The distributions of classes and size of local training data are
demonstrated in Figs. 6 and 7 (Appx. G). For generating backdoor data, we use
image warping-based WaNet [16], which is one of the state-of-the-art backdoor
attacks. We adopt the model configurations described in [4] for training the
HyperNetFL and in [16] for generating backdoor images in training $X$. In the
DP optimizer, we vary the noise scale
$\sigma\in\\{10^{-1},10^{-2},10^{-3},10^{-4}\\}$ and the clipping $l_{2}$-norm
$\mu\in\\{8,4,2,1\\}$. For the $\alpha$-trimmed norm approach, we choose
$\alpha\in\\{0.1,0.2,0.3,0.4\\}$. We use $\mathcal{U}[a,b]$ with $a=0.8$ and
$b=1$. The complete details are in Appx. G.

### Evaluation Approach

We carry out the validation through three approaches. We first compare HNTroj
with DPois and HNRepl in terms of legitimate accuracy (ACC) on legitimate data
samples and backdoor successful rate (SR) on Trojaned data samples with a wide
range number of compromised clients. The second approach is to investigate the
effectiveness of adapted robust HyperNetFL training algorithms, including the
client-level DP optimizer and the $\alpha$-trimmed norm, under a variety of
hyper-parameter settings against HNTroj. Based upon that, the third approach
provides a performance summary of both attacks and defenses to inform the
surface of backdoor risks in HyperNetFL. The (average) legitimate ACC and
backdoor SR across clients on testing data are as follows:

$\displaystyle\text{Legitimate~{}ACC}=\frac{1}{N}\sum_{i\in[N]}\frac{1}{n_{i}^{\tau}}\sum_{j\in[n_{i}^{\tau}]}Acc\big{(}f(x_{j}^{i},\theta^{i}),y_{j}^{i}\big{)}$
$\displaystyle\text{Backdoor~{}SR}=\frac{1}{N}\sum_{i\in[N]}\frac{1}{n_{i}^{\tau}}\sum_{j\in[n_{j}^{\tau}]}Acc\big{(}f(\overline{x}_{j}^{i},\theta^{i}),y_{j}^{i,b}\big{)}$

where $\overline{x}^{i}_{j}=x^{i}_{j}+\mathcal{T}$ is a Trojaned sample,
$Acc(y^{\prime},y)=1$ if $y^{\prime}=y$; otherwise $Acc(y^{\prime},y)=0$ and
$n_{i}^{\tau}$ is the number of testing samples in client $i$.

###  HNTroj v.s. DPois and White-box HNRepl

Figs. 1 and 8 (Appx. G) present legitimate ACC and backdoor SR of each attack
and the clean model (i.e., trained without poisoning) as a function of the
communication round $t$ and the number of compromised clients $\complement$
under a defense free environment in the CIFAR-10 dataset. It is obvious that
HNTroj significantly outperforms DPois. HNTroj requires a notably small number
of compromised clients to successfully backdoor the HyperNetFL with high
backdoor SR, i.e., $43.92\%$, $64.00\%$, $70.38\%$, $73.45\%$, and $75.70\%$
compared with $10.58\%$, $17.87\%$, $29.90\%$, $41.53\%$, and $50.57\%$ of the
DPois given $1$, $5$, $10$, $15$, and $20$ compromised clients respectively,
without an undue cost in legitimate ACC, i.e., $76.89\%$.

In addition, HNTroj does not introduce degradation or sudden shifts in
legitimate ACC during the training process, regardless of the number of
compromised clients, making it stealthier than DPois and (white-box) HNRepl.
This is because we consistently poison the HyperNetFL training with a
relatively good Trojaned model $X$, which achieves $74.26\%$ legitimate ACC
and $85.92\%$ backdoor SR, addressing the inconsistency in deriving the
malicious local gradients. There is a small legitimate ACC gap between HNTroj
and the clean model, i.e., $6.11\%$ in average. However, this gap will not be
noticed by the server since the clean model is invisible to the server when
the compromised clients are present.

Figure 2: Legitimate ACC and backdoor SR under $\alpha$-trimmed norm defense
in the CIFAR-10 dataset.

###  HNTroj v.s. $\alpha$-Trimmed Norm

Since HNTroj outperforms other poisoning attacks, we now focus on
understanding its performance under robust HyperNetFL training. Fig. 2 shows
the performance of $\alpha$-trimmed norm against HNTroj as a function of the
number of compromised clients $\complement$. There are three key observations
from the results, as follows: (1) Applying $\alpha$-trimmed norm does reduce
the backdoor SR, especially when the number of compromised clients is small ,
i.e., backdoor SR drops $39.72\%$ in average given $\complement\in[1,10]$.
However, when the number of compromised clients is a little bit larger, the
backdoor SR is still at highly severe levels, i.e., $51.70\%\sim 70.55\%$
given $15$ to $20$ compromised clients, regardless of a wide range of trimming
level $\alpha\in[0.1,0.4]$; (2) The larger the $\alpha$ is, the lower the
backdoor SR tends to be. This good result comes with a toll on the legitimate
ACC, which is notably reduced when $\alpha$ is larger. In average, the
legitimate ACC drops from $79.75\%$ to $67.66\%$ and $62.29\%$ given
$\alpha\in[0.3,0.4]$ and $\complement\in[1,10]$, respectively. That clearly
highlights a non-trivial trade-off between legitimate ACC and backdoor SR
given attacks and defenses; and (3) The more compromised clients we have, the
better the legitimate ACC is when the trimming level $\alpha$ is large, i.e.,
$\alpha\in[0.3,0.4]$. That is because training with the Trojaned model $X$,
which has a relatively good legitimate ACC, can mitigate the damage of large
trimming levels on the legitimate ACC. In fact, a large number of compromised
clients implies a better probability for the compromised clients to sneak
through the trimming; thus, improving both legitimate ACC and backdoor SR.

###  HNTroj v.s. Client-level DP Optimizer

We observe a similar phenomenon when we apply the client-level DP optimizer as
a defense against HNTroj (Fig. 10, Appx. G). First, by using small noise
scales $\sigma\in[10^{-4},10^{-2}]$, the client-level DP optimizer is
effective in defending against HNTroj when the number of compromised clients
is small ($\complement\in[1,5]$) achieving low backdoor SR, i.e., $31.97\%$ in
average, while maintaining an acceptable legitimate ACC, i.e., $72.33\%$ in
average. When the number of compromised clients is a little bit larger, the
defense pays notably large tolls on the legitimate ACC (i.e., the legitimate
ACC drops from $75.80\%$ to $31.18\%$ given $\complement=10$) or fails to
reduce the backdoor SR (i.e., backdoor SR $>58.25\%$ given $\complement>10$).
That is consistent with our analysis. A small sufficient number of compromised
clients synergistically and consistently can pull the outputs of the
HyperNetFL to the Trojaned model $X$’s surrounded area.

### Backdoor Risk Surface: Attacks and Defenses

The trade-off between legitimate ACC and backdoor SR is non-trivially
observable given many attack and defense configurations. To inform a better
surface of backdoor risks, we look into a fundamental question: “What can the
adversary or the defender achieve given a specific number of compromised
clients?"

We answer this question by summarizing the best defending performance and the
most stealthy and severe backdoor risk across hyper-parameter settings in the
same diagram. Given a number of compromised clients $\complement$ and a robust
training algorithm $\mathcal{A}$, the best defending performance, which
maximizes both the 1) legitimate ACC (i.e., ACC in short) and 2) the gap
between the legitimate ACC and backdoor SR (i.e., SR in short), is identified
across hyper-parameters’ space of $\mathcal{A}$, i.e.,
$\Theta^{\complement}_{\mathcal{A}}$, as follows:

$\phi^{*}=\arg\max_{\phi\in\Theta^{\complement}_{\mathcal{A}}}\big{[}ACC(\phi)+\big{(}ACC(\phi)-SR(\phi)\big{)}\big{]}$

where $ACC(\phi)$ and $SR(\phi)$ are the legitimate ACC and backdoor SR using
the specific hyper-parameter configuration $\phi$. Similarly, we identify the
most stealthy and severe backdoor risk maximizing both the legitimate ACC and
backdoor SR:

$\phi^{*}=\arg\max_{\phi\in\Theta^{\complement}_{\mathcal{A}}}\big{[}ACC(\phi)+SR(\phi)\big{]}$

(a) Client-level DP optimizer

(b) $\alpha$-Trimmed Norm

Figure 3: Backdoor risk surface: attacks and defenses in the CIFAR-10 dataset.
The attack we used here is HNTroj.

Fig. 3 summaries the best performance of both defenses and attacks in the
CIFAR-10 dataset as a function of the number of compromised clients through
out the hyper-parameter space. For instance, given the number of compromised
clients $\complement=10$, using the client-level DP optimizer, the best
defense can reduce the backdoor SR to $12.88\%$ with a cost of $13.45\%$ drop
in the legitimate ACC. Meanwhile, in a weak defense using the client-level DP
optimizer, the adversary can increase the backdoor SR up to $71.74\%$ without
sacrificing much the legitimate ACC. From Figs. 3a-b, we can observe that
$\alpha$-trimmed norm is a little bit more effective than the client-level DP
optimizer by having a wider gap between legitimate ACC and backdoor SR. From
the adversary angle, to ensure the success of the HNTroj regardless of the
defenses, the adversary needs to have at least $15$ compromised clients.

### Results on the Fashion MNIST dataset

The results on the Fashion MNIST dataset further strengthen our observation.
DPois even failed to implant backdoors into HyperNetFL (Figs. 9 and 12, Appx.
G). This is because the HyperNetFL model converges $10\mathbf{x}$ faster than
the model for the CIFAR-10 dataset, i.e., given the simplicity of the Fashion
MNIST dataset; thus, significantly reducing the poisoning probability through
participating in the training of a small set of compromised clients. Thanks to
the consistency in deriving malicious local gradients, HNTroj is still highly
effective under the defense free environment. However, we found that the
client-level DP optimizer can significantly mitigate HNTroj due to the model’s
fast convergence; meanwhile, the $\alpha$-trimmed norm is still failed to
defend again HNTroj (Figs. 4, 5, 13).

##  7 Conclusion

We presented a black-box model transferring attack (HNTroj) to implant
backdoor into HyperNetFL. We overcome the lack of consistency in deriving
malicious local gradients to efficiently transfer a Trojaned model to the
outputs of the HyperNetFL. We multiply a random and dynamic learning rate to
the malicious local gradients making the attack stealthy. To defend against
HNTroj, we adapted several robust FL training algorithms into HyperNetFL.
Extensive experiment results show that HNTroj outperforms black-box DPois and
white-box HNRepl bypassing adapted robust training algorithms with small
numbers of compromised clients.

## References

  * [1] GDPR,  “The european data protection regulation,” {https://gdpr-info.eu/}, 2018.
  * [2] B. McMahan, E. Moore, et al.,  “Communication-efficient learning of deep networks from decentralized data,”  in AISTATS, 2017.
  * [3] H. Zhu, J. Xu, S. Liu, and Y. Jin,  “Federated learning on non-iid data: A survey,”  arXiv preprint arXiv:2106.06843, 2021.
  * [4] A. Shamsian, A. Navon, E. Fetaya, and G. Chechik,  “Personalized federated learning using hypernetworks,”  ICML, 2021.
  * [5] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, et al.,  “Advances and open problems in federated learning,”  arXiv preprint arXiv:1912.04977, 2019.
  * [6] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov,  “How to backdoor federated learning,”  in AISTATS, 2020.
  * [7] A.N. Bhagoji, S. Chakraborty, P. Mittal, and S. Calo,  “Analyzing federated learning through an adversarial lens,”  in ICML, 2019, pp. 634–643.
  * [8] D. Ha, A. Dai, and Q.V. Le,  “Hypernetworks,”  ICLR, 2016.
  * [9] B. Biggio, B. Nelson, and P. Laskov,  “Poisoning attacks against support vector machines,”  in ICML, 2012, pp. 1467–1474.
  * [10] B. Nelson, M. Barreno, F.J. Chi, A.D. Joseph, B. I. P. Rubinstein, U. Saini, C. Sutton, J. D. Tygar, and K. Xia,  “Exploiting machine learning to subvert your spam filter,”  in USENIX Workshop, 2008, pp. 7:1–7:9.
  * [11] Jacob Steinhardt, Pang Wei Koh, and Percy Liang,  “Certified defenses for data poisoning attacks,”  in NeurIPS, 2017, pp. 3520–3532.
  * [12] L. Muñoz González, B. Biggio, A. Demontis, A. Paudice, V. Wongrassamee, et al.,  “Towards poisoning of deep learning algorithms with back-gradient optimization,”  in AISEC, 2017.
  * [13] M.S. Ozdayi, M. Kantarcioglu, and Y.R. Gel,  “Defending against backdoors in federated learning with robust learning rate,”  AAAI, 2021.
  * [14] T. Gu, B. Dolan-Gavitt, and S. Garg,  “Badnets: Identifying vulnerabilities in the machine learning model supply chain,”  Machine Learning and Computer Security Workshop, 2017.
  * [15] Y. Liu, S. Ma, Y. Aafer, W.C. Lee, J. Zhai, W. Wang, and X. Zhang,  “Trojaning attack on neural networks,”  2017\.
  * [16] T.N. Anh and A.T. Tuan,  “Wanet - imperceptible warping-based backdoor attack,”  in ICLR, 2021.
  * [17] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B.Y. Zhao,  “Neural cleanse: Identifying and mitigating backdoor attacks in neural networks,”  in SP, 2019, pp. 707–723.
  * [18] K. Liu, B. Dolan-Gavitt, and S. Garg,  “Fine-pruning: Defending against backdooring attacks on deep neural networks,”  in RAID, 2018, pp. 273–294.
  * [19] Y. Gao, C. Xu, D. Wang, S. Chen, D.C. Ranasinghe, and S. Nepal,  “Strip: A defence against trojan attacks on deep neural networks,”  in ACSAC, 2019.
  * [20] O. Suciu, R. Marginean, Y. Kaya, H. Daume III, and T. Dumitras,  “When does machine learning fail? generalized transferability for evasion and poisoning attacks,”  in USENIX, 2018, pp. 1299–1316.
  * [21] B. Li, Y. Wang, A. Singh, and Y. Vorobeychik,  “Data poisoning attacks on factorization-based collaborative filtering,”  NeurIPS, vol. 29, pp. 1885–1893, 2016.
  * [22] M. Fang, X. Cao, J. Jia, and N. Gong,  “Local model poisoning attacks to byzantine-robust federated learning,”  in USENIX, 2020, pp. 1605–1622.
  * [23] D. Yin, Y. Chen, R. Kannan, and P. Bartlett,  “Byzantine-robust distributed learning: Towards optimal statistical rates,”  in ICML, 2018, pp. 5650–5659.
  * [24] S. Hong, V. Chandrasekaran, Y. Kaya, T. Dumitraş, and N. Papernot,  “On the effectiveness of mitigating data poisoning attacks with gradient shaping,”  arXiv preprint arXiv:2002.11497, 2020.
  * [25] K. Pillutla, S.M. Kakade, and Z. Harchaoui,  “Robust aggregation for federated learning,”  arXiv preprint arXiv:1912.13445, 2019.
  * [26] R. Guerraoui, S. Rouault, et al.,  “The hidden vulnerability of distributed learning in byzantium,”  in ICML, 2018, pp. 3521–3530.
  * [27] J. Jia, X. Cao, and N.Z. Gong,  “Intrinsic certified robustness of bagging against data poisoning attacks,”  AAAI, 2021.
  * [28] C. Xie, M. Chen, P.Y. Chen, and B. Li,  “Crfl: Certifiably robust federated learning against backdoor attacks,”  ICML, 2021.
  * [29] A. Krizhevsky et al.,  “Learning multiple layers of features from tiny images,”  2009\.
  * [30] H. Xiao, K. Rasul, and R. Vollgraf,  “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,”  arXiv preprint arXiv:1708.07747, 2017.
  * [31] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner,  “Gradient-based learning applied to document recognition,”  Proc. of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.

##  Appendix A Federated Learning Protocol

We consider the following FL protocol: at round $t$, the server sends the
latest model weights $\theta_{t}$ to a randomly sampled subset of clients
$S_{t}$. Upon receiving $\theta_{t}$, the client $i$ in $S_{t}$ uses
$\theta_{t}$ to train their local model for some number of iterations, e.g.,
via stochastic gradient descent (SGD), and results in model weights
$\theta^{i}_{t}$. The client $i$ computes their local gradient
$\bigtriangleup\theta^{i}_{t}=\theta^{i}_{t}-\theta_{t}$, and sends it back to
the server. After receiving all the local gradients from all the clients in
$S_{t}$, the server updates the model weights by aggregating all the local
gradients by using an aggregation function $\mathcal{G}:R^{|S_{t}|\times
m}\rightarrow R^{m}$ where $m$ is the size of $\bigtriangleup\theta^{i}_{t}$.
The aggregated gradient will be added to $\theta_{t}$, that is,
$\theta_{t+1}=\theta_{t}+\lambda\mathcal{G}(\\{\bigtriangleup\theta^{i}_{t}\\}_{i\in
S_{t}})$ where $\lambda$ is the server’s learning rate. A typical aggregation
function is weighted averaging, i.e., Federated Averaging (FedAvg) applied in
many papers in FL [5], presented as follows:

$\theta_{t+1}=\theta_{t}+\lambda(\sum_{i\in
S_{t}}n_{i}\times\bigtriangleup\theta_{t}^{i})/\sum_{i\in S_{t}}n_{i}$ (3)

When the number of training samples $n_{i}$ is hidden from the server, one can
use an unweighted aggregation function:
$\theta_{t+1}=\theta_{t}+\lambda\sum_{i\in
S_{t}}\bigtriangleup\theta_{t}^{i}/|S_{t}|$.

##  Appendix B White-box Model Replacement Attacks in HyperNetFL

### White-box Threat Model

At round $t$, an adversary fully controls a compromised client $c$, has an
access to the HyperNetFL weights $\varphi$, all the descriptors
$\\{v_{i}\\}_{i\in[N]}$, and the local gradient $\bigtriangleup\theta^{i}_{t}$
of all the clients. The adversary cannot modify the training protocol of the
HyperNetFL at the server and at legitimate clients. The adversary tries to
open backdoors in all local models by minimizing a backdoor poisoning
objective:

$\arg\min_{\varphi,\\{v_{i}\\}_{i\in[N]}}\frac{1}{N}\sum_{i=1}^{N}\big{[}L_{i}(h(v_{i},\varphi))+L^{b}_{i}(h(v_{i},\varphi))\big{]}$
(4)

where $L^{b}_{i}$ is the (backdoor) loss function of the $i^{th}$ client given
Trojaned examples $x_{j}+\mathcal{T}$ with the trigger $\mathcal{T}$, e.g.,
$L^{b}_{i}=\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}L\big{(}f(x_{j}+\mathcal{T},\theta),y^{b}_{j}\big{)}$
where $y^{b}_{j}$ is the targeted label for the sample $x_{j}+\mathcal{T}$.
One can vary the portion of Trojaned samples to optimize the attack
performance.

When the compromised client $c$ is selected for training at a round $t$, given
the observed knowledge, the adversary’s goal is to replace the HyperNetFL
model weights $\varphi$ with a poisoned one $\bar{\varphi}$ through computing
and sending a malicious local gradient $\bigtriangleup\bar{\theta}^{c}_{t}$ to
the server, such that the neural network $h(\cdot,\bar{\varphi})$ returns a
Trojaned model $X$ given all the descriptors $\\{v_{i}\\}_{i\in[N]}$. That
means all the local model weights $\theta^{i}_{t+1}$ generated by the the
HyperNetFL for every clients $i$ are Trojan infected, i.e., $\forall
i\in[N]:\theta^{i}_{t+1}=h(v_{i},\bar{\varphi})\approxeq X$. In practice, the
adversary can collect its own data that shares a similar distribution with
legitimate clients to locally train the Trojaned model $X$. To find
$\bar{\varphi}$, the adversary minimizes the following objective:

$\bar{\varphi}=\arg\min_{\varphi}\sum_{i\in[N]}\|h(v_{i};\varphi)-X\|_{2}^{2}$
(5)

Given $\bar{\varphi}$, the adversary computes the exact malicious local
gradient
$\bigtriangleup\bar{\theta}^{c}_{t}=\frac{\frac{|S_{t}|}{\lambda}({\varphi}-\bar{\varphi})-\sum_{i\in
S_{t}\setminus
c}(\bigtriangledown_{{\varphi}}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}}{(\bigtriangledown_{{\varphi}}{\theta}^{c}_{t})^{\top}}$,
where $\bigtriangledown_{{\varphi}}\theta^{i}_{t}$ can be computed by having
an access to $\varphi$ and the descriptors $\\{v_{i}\\}_{i\in[N]}$. When the
server update $\varphi$ using the malicious local gradient
$\bigtriangleup\bar{\theta}^{c}_{t}$ and all the other local gradients
$\bigtriangleup\theta^{i}_{t}$ from legitimate clients $i\in S_{t}\setminus
c$, then $\varphi$ will be replaced with the poisoned $\bar{\varphi}$. As a
result, the adversary can activate backdoors. Note that replacing $\varphi$
with $\bar{\varphi}$ (Eq. 5) enables us to achieve the backdoor poisoning
objective in Eq. 4.

Input: Number of rounds $T$, number of local rounds $K$, server’s learning
rates $\lambda$ and $\zeta$, clients’ learning rate $\eta$, number of clients
$N$, Trojaned model $X$, and $L_{i}(B)$ is the loss function $L_{i}(\theta)$
on a mini-batch $B$

Output: ${\varphi},v_{i}$

1: for $t=1,\ldots,T$ do

2: Sample clients $S_{t}$

3: for each legitimate client $i\in S_{t}\setminus c$ do

4: set $\theta^{i}_{t}=h(v_{i},\varphi)$ and
$\tilde{\theta}^{i}=\theta^{i}_{t}$

5: for $k=1,\ldots,K$ do

6: sample mini-batch $B\subset D_{i}$

7:
$\tilde{\theta}_{k+1}^{i}=\tilde{\theta}_{k}^{i}-\eta\bigtriangledown_{\tilde{\theta}_{k}^{i}}L_{i}(B)$

8: $\bigtriangleup\theta^{i}_{t}=\tilde{\theta}^{i}_{K}-\theta^{i}_{t}$

9: if the compromised client $c\in S_{t}$ then

10:
$\bar{\varphi}=\arg\min_{\varphi}\sum_{i=1}^{N}\|h(v_{i};\varphi)-X\|_{2}^{2}$

11:
$\bigtriangleup\bar{\theta}^{c}_{t}=\frac{\frac{|S_{t}|}{\lambda}({\varphi}-\bar{\varphi})-\sum_{i\in
S_{t}\setminus
c}(\bigtriangledown_{{\varphi}}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}}{(\bigtriangledown_{{\varphi}}{\theta}^{c}_{t})^{\top}}$

12:
$\varphi=\varphi-\frac{\lambda}{|S_{t}|}\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

13: $\forall i\in
S_{t}:v_{i}=v_{i}-\zeta\bigtriangledown_{v_{i}}\varphi^{\top}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

Algorithm 3 White-box Model Replacement in HyperNetFL (HNRepl)

The psuedo-code of HNRepl and its correctness proof are in Alg. 3 and Theorem
6. The attack success rate of this white-box model replacement is very high
right after the replacement occurs (as shown in our experiments). However, it
is impractical for the adversary to gain access to the HyperNetFL model
weights $\varphi$, all the descriptors $\\{v_{i}\\}_{i\in[N]}$, and the local
gradients $\\{\bigtriangleup\theta^{i}_{t}\\}_{i\in S_{t}}$. These factors
must be hidden from all the clients during the whole training process of a
HyperNetFL. To inform a more realistic risk, we consider the following
practical black-box threat model.

### Black-box Threat Model

At round $t$, an adversary fully controls a small set of compromised clients
$\complement$. The adversary cannot modify the training protocol of the
HyperNetFL at the server and at the legitimate clients. The adversary’s goal
is to implant backdoors in all local models
$\\{\theta^{i}=h(v_{i},\varphi)\\}_{i\in[N]}$.

To adapt the model replacement attack into the black-box setting, the
adversary performs the following two steps: (1) Leverage the small set of
compromised clients $\complement$ in order to infer $\varphi$ and the
descriptors $\\{v_{c}\\}_{c\in\complement}$ by training a local neural network
$h^{l}(v_{c}^{l},\varphi^{l})$ imitating the behavior of the HyperNetFL
$h(\cdot,\varphi)$; and (2) Compute the malicious local gradient by using the
legitimate data collected by the compromised clients
$\bigtriangleup\bar{\theta}^{c}_{t}=\frac{\frac{|\complement|}{\lambda}({\varphi^{l}}-\bar{\varphi}^{l})-\sum_{i\in\complement\setminus
c}(\bigtriangledown_{{\varphi^{l}}}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}}{(\bigtriangledown_{{\varphi^{l}}}{\theta}^{c}_{t})^{\top}}$
where $\bar{\varphi}$ is now identified by using the local $h^{c}$ instead of
$h$ as in Eq. 5:
$\bar{\varphi}^{l}=\arg\min_{\varphi^{l}}\sum_{c\in\complement}\|h^{l}(v^{l}_{c};\varphi^{l})-X\|_{2}^{2}$.

However, we found that this black-box model replacement attack is not
effective since approximating $\varphi$ with a small number of compromised
clients is infeasible given the large size of $\varphi$. In addition, we also
found that model replacement attacks usually introduce sudden shifts in the
model utility on legitimate data samples reducing the attacks’ stealthiness in
poisoning HyperNetFL.

######  Theorem 1.

At a communication round $t\in[1,T]$, the compromised client $c\in S_{t}$ is
able to substitute the Trojaned HyperNetFL $\bar{\varphi}$ with a malicious
gradient $\bigtriangleup\bar{\theta}^{c}_{t}$ as follows:

$\bigtriangleup\bar{\theta}^{c}_{t}=\frac{\frac{|S_{t}|}{\lambda}({\varphi}-\bar{\varphi})-\sum_{i\in
S_{t}\setminus
c}(\bigtriangledown_{{\varphi}}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}}{(\bigtriangledown_{{\varphi}}{\theta}^{c}_{t})^{\top}}$
(6)

###### Proof.

At round $t\in[1,T]$, the server sends the latest model weights $\theta_{t}$
generated from the current HyperNetFL $\varphi$ to a randomly sampled subset
of clients $S_{t}$. Upon receiving $\theta_{t}$, the legitimate client $i$ in
$S_{t}$ uses $\theta_{t}$ to train their local model, and sends it back to the
server. Meanwhile, the compromised client $c\in S_{t}$ attempts to substitute
the global HyperNetFL by the poisoned model $\bar{\varphi}$ that generates the
Trojaned model $X$. In other words, the compromised client $c$ needs to submit
a gradient $\bigtriangleup\bar{\theta}^{c}_{t}$ satisfying the following
condition:

$\displaystyle\bar{\varphi}$
$\displaystyle=\varphi+\frac{\lambda}{|S_{t}|}\sum_{i\in
S_{t}}(\bigtriangledown_{{\varphi}}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$
(7) $\displaystyle=\varphi+\frac{\lambda}{|S_{t}|}\sum_{i\in S_{t}\setminus
c}(\bigtriangledown_{{\varphi}}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}+\frac{\lambda}{|S_{t}|}(\bigtriangledown_{{\varphi}}\theta^{i}_{c})^{\top}\bigtriangleup\bar{\theta}^{c}_{t}.$

The compromised client $c$ can solve Eq.7 for the gradient it needs to submit
as follows:

$\displaystyle\bigtriangleup\bar{\theta}^{c}_{t}=\frac{\frac{|S_{t}|}{\lambda}({\varphi}-\bar{\varphi})-\sum_{i\in
S_{t}\setminus
c}(\bigtriangledown_{{\varphi}}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}}{(\bigtriangledown_{{\varphi}}{\theta}^{c}_{t})^{\top}}$
(8)

Consequently, Theorem 6 holds. ∎

##  Appendix C Backdoor Data Poisoning in HyperNetFL

Input: Number of rounds $T$, number of local rounds $K$, server’s learning
rates $\lambda$ and $\zeta$, clients’ learning rate $\eta$, number of clients
$N$, and $L_{c}(B)$ and $L^{b}_{c}(B)$ are the loss functions (legitimate)
$L_{c}(\theta)$ and (backdoor) $L^{b}_{c}(\theta)$ on a mini-batch $B$

Output: ${\varphi},v_{i}$

1: for $t=1,\ldots,T$ do

2: Sample clients $S_{t}$

3: for each legitimate client $i\in S_{t}\setminus\complement$ do

4: set $\theta^{i}_{t}=h(v_{i},\varphi)$ and
$\tilde{\theta}^{i}=\theta^{i}_{t}$

5: for $k=1,\ldots,K$ do

6: sample mini-batch $B\subset D_{i}$

7:
$\tilde{\theta}_{k+1}^{i}=\tilde{\theta}_{k}^{i}-\eta\bigtriangledown_{\tilde{\theta}_{k}^{i}}L_{i}(B)$

8: $\bigtriangleup\theta^{i}_{t}=\tilde{\theta}^{i}_{K}-\theta^{i}_{t}$

9: for each compromised client $c\in S_{t}\cap\complement$ do

10: set $\theta^{c}_{t}=h(v_{c},\varphi)$ and
$\tilde{\theta}^{c}=\theta^{c}_{t}$

11: for $k=1,\ldots,K$ do

12: sample mini-batch $B\subset D_{c}$

13:
$\tilde{\theta}_{k+1}^{c}=\tilde{\theta}_{k}^{c}-\eta\bigtriangledown_{\tilde{\theta}_{k}^{c}}[L_{c}(B)+L^{b}_{c}(B)]$

14: $\bigtriangleup\bar{\theta}^{c}_{t}=\tilde{\theta}^{c}_{K}-\theta^{c}_{t}$

15:
$\varphi=\varphi-\frac{\lambda}{|S_{t}|}\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

16: $\forall i\in
S_{t}:v_{i}=v_{i}-\zeta\bigtriangledown_{v_{i}}\varphi^{\top}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

Algorithm 4 Backdoor Data Poisoning in HyperNetFL (DPois)

##  Appendix D Robust Learning Rate in HyperNetFL

We attempt to adapt the recently developed robust learning rate (RLR) [13]
over all dimensions of the local gradients
$\\{\bigtriangleup\theta_{t}^{i}\\}_{i\in[N]}$ in HyperNetFL. RLR works by
moving the model towards a particular direction, for each dimension, it
requires a sufficient number of votes (larger than a predefined threshold
$\beta$), in form of signs of the local gradients. For every dimension where
the sum of signs of local gradients is less than $\beta$, the learning rate is
multiplied by $-1$ to maximize the loss on that dimension. At round $t$, the
learning rate $\lambda_{t}^{\theta,k}$ for the $k^{th}$ dimension of the local
gradients $\theta_{t}^{i,k}$ at the server is given by

$\lambda_{t}^{\theta,k}=\left\\{\begin{aligned} &\lambda&&|\sum_{i\in
S_{t}}\text{sgn}(\bigtriangleup\theta_{t}^{i,k})|\geq\beta,\\\
&-\lambda&&\text{otherwise.}\end{aligned}\right.$ (9)

The local gradients in HyperNetFL now become
$\\{\lambda_{t}^{\theta,k}\times\bigtriangleup\theta_{t}^{i,k}\\}_{k\in\theta^{i},i\in
S_{t}}$. Then, the revised update rules (Lines 9 and 10, Alg. 1) are as
follows:

$\displaystyle\varphi=\varphi-\frac{1}{|S_{t}|}\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}(\lambda_{t}^{\theta}\odot\bigtriangleup\theta^{i}_{t})$
(10) $\displaystyle\forall i\in
S_{t}:v_{i}=v_{i}-\zeta\bigtriangledown_{v_{i}}\varphi^{\top}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$
(11)

where $\odot$ is the element-wise product operation. Note that we do not need
a vote to modify the update rule of the descriptors $v_{i}$ since it is a
personalized update. Updating a specific $v_{i}$ does not affect the updates
of any other descriptors $\\{v_{j}\\}_{i\neq j}$ and the model weights
$\varphi$.

The key assumption is that if the number of compromised clients is sufficient
smaller than $\beta$, the outputs of the HyperNetFL $h(\cdot,\varphi)$ would
move away from the Trojaned models, which are
$\\{\theta^{c*}_{t}\\}_{c\in\complement}$ in the case of data poisoning
attacks and away from $X$ in the case of HNTroj. We found that this adaptation
of RLR severely damages the update rules of HyperNetFL $h(\cdot,\varphi)$ even
under a wide-range of $\beta\in[1,|S_{t}|]$. The model utility drops to random
guess in several benchmark datasets, including CIFAR-10 and Fashion MNIST. The
key reason is that the update rules of HyperNetFL depend not only on the local
gradients, but also on the gradients of $\varphi$ derived from the outputs of
$h(\cdot,\varphi)$; that is,
$(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}$. Manipulating the local
gradients alone could affect the original gradients of $\varphi$:
$\sum_{i=1}^{|S_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$.

##  Appendix E Proof of Theorem 2

######  Theorem 2.

For a compromised client $c$ participating in a round $t\in[T]$, we have that
the $l_{2}$-norm distance between the HyperNetFL output
$\theta_{t}^{c}=h(v_{c},\varphi)$ and the Trojaned model $X$ is always bounded
as follows:

$\|\theta_{t}^{c}-X\|_{2}\leq(1/a-1)\|\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}\|_{2}+\|\xi\|_{2}$
(12)

where $\forall t:\psi_{t}\sim\mathcal{U}[a,b]$, $a<b$, $a,b\in(0,1]$,
$t^{\prime}$ is the closest round the compromised client $c$ participated in,
and $\xi\in\mathcal{R}^{m}$ is a small error rate.

###### Proof.

At the round $t^{\prime}$, we have that
$\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}=\psi^{c}_{t^{\prime}}[X-\theta^{c}_{t^{\prime}}]$.
This is equivalent to
$X=\frac{\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}}{\psi^{c}_{t^{\prime}}}+\theta^{c}_{t^{\prime}}$.
At the round $t$, the HyperNetFL $h(v_{c},\varphi)$ supposes to generate a
better local model for the compromised client $c$:
$\theta^{c}_{t}=\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}+\theta^{c}_{t^{\prime}}+\xi$.
To quantify the distance between the generated local model $\theta^{c}_{t}$
and the Trojaned model $X$, we subtract $\theta^{c}_{t}$ by $X$ as follows:
$\theta^{c}_{t}-X=(1-\frac{1}{\psi^{c}_{t^{\prime}}})\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}+\xi$.
Based upon this, we can bound the $l_{2}$-norm of the distance
$\theta^{c}_{t}-X$ as follows:

$\displaystyle\|\theta_{t}^{c}-X\|_{2}$
$\displaystyle=\|(1-\frac{1}{\psi^{c}_{t^{\prime}}})\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}+\xi\|_{2}\leq(\frac{1}{a}-1)\|\bigtriangleup\bar{\theta}^{c}_{t^{\prime}}\|_{2}+\|\xi\|_{2}$
(13)

Consequently, Theorem 2 holds. ∎

##  Appendix F Robust Aggregation

Input: Number of rounds $T$, number of local rounds $K$, server’s learning
rates $\lambda$ and $\zeta$, clients’ learning rate $\eta$, number of clients
$N$, clipping bound $\mu$, noise scale $\sigma$, and $L_{i}(B)$ is the loss
function $L_{i}(\theta)$ on a mini-batch $B$

Output: ${\varphi},v_{i}$

1: for $t=1,\ldots,T$ do

2: Sample clients $S_{t}$

3: for each client $i\in S_{t}$ do

4: set $\theta^{i}_{t}=h(v_{i},\varphi)$ and
$\tilde{\theta}^{i}=\theta^{i}_{t}$

5: for $k=1,\ldots,K$ do

6: sample mini-batch $B\subset D_{i}$

7:
$\tilde{\theta}_{k+1}^{i}=\tilde{\theta}_{k}^{i}-\eta\bigtriangledown_{\tilde{\theta}_{k}^{i}}L_{i}(B)$

8: $\bigtriangleup\theta^{i}_{t}=\tilde{\theta}^{i}_{K}-\theta^{i}_{t}$

9:
$\varphi=\varphi-\frac{\lambda}{|S_{t}|}\big{[}\sum_{i=1}^{|S_{t}|}\frac{(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}}{\max(1,\frac{\|(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}\|_{2}}{\mu})}+\mathcal{N}(0,\sigma^{2}\mu^{2}\mathbf{I})\big{]}$

10: $\forall i\in
S_{t}:v_{i}=v_{i}-\zeta\bigtriangledown_{v_{i}}\varphi^{\top}\frac{(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}}{\max(1,\frac{\|(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}\|_{2}}{\mu})}$

Algorithm 5 Client-level DP Optimizer in HyperNetFL

Input: Number of rounds $T$, number of local rounds $K$, server’s learning
rates $\lambda$ and $\zeta$, clients’ learning rate $\eta$, number of clients
$N$, and $L_{i}(B)$ is the loss function $L_{i}(\theta)$ on a mini-batch $B$

Output: ${\varphi},v_{i}$

1: for $t=1,\ldots,T$ do

2: Sample clients $S_{t}$

3: for each client $i\in S_{t}$ do

4: set $\theta^{i}_{t}=h(v_{i},\varphi)$ and
$\tilde{\theta}^{i}=\theta^{i}_{t}$

5: for $k=1,\ldots,K$ do

6: sample mini-batch $B\subset D_{i}$

7:
$\tilde{\theta}_{k+1}^{i}=\tilde{\theta}_{k}^{i}-\eta\bigtriangledown_{\tilde{\theta}_{k}^{i}}L_{i}(B)$

8: $\bigtriangleup\theta^{i}_{t}=\tilde{\theta}^{i}_{K}-\theta^{i}_{t}$

9: $S^{\alpha-trim}_{t}=$ $\alpha$-trimmed norm
$\big{(}\\{(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}\\}_{i\in
S_{t}}\big{)}$

10: $\varphi=\varphi-\frac{\lambda}{|S^{\alpha-
trim}_{t}|}\sum_{i=1}^{|S^{\alpha-
trim}_{t}|}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

11: $\forall i\in S^{\alpha-
trim}_{t}:v_{i}=v_{i}-\zeta\bigtriangledown_{v_{i}}\varphi^{\top}(\bigtriangledown_{\varphi}\theta^{i}_{t})^{\top}\bigtriangleup\theta^{i}_{t}$

Algorithm 6 $\alpha$-Trimmed Norm in HyperNetFL

Regarding certified bounds against backdoors at the inference phase derived in
weight-clipping and noise addition [28] cannot be straightforwardly adapted to
HyperNetFL, since the bounds are especially designed for the aggregated model
$\theta$ (e.g., Eq. 3). There is no such aggregated model in HyperNetFL.
Similarly, the ensemble model training-based robustness bounds [27] cannot be
directly adapted to HyperNetFL, since there is no ensemble model training in
HyperNetFL.

##  Appendix G Supplemental Experimental Results

### Data and Model Configuration

To achieve our goal, we conduct an extensive experiment on CIFAR-10 [29] and
Fashion MNIST datasets [30]. For both datasets, to generate non-iid data
distribution across clients in terms of classes and size of local training
data, we randomly sample two classes for each client. And for each client $i$
and selected class $c$, we sample $p_{i,c}\sim\mathcal{N}(0,1)$ and assign it
with $\frac{p_{i,c}}{\sum_{j}p_{j,c}}$ of the samples for this class. We use
$100$ clients. There are $60,000$ samples in the CIFAR-10 and $70,000$ samples
in the Fashion MNIST datasets. Each dataset is divided into three non-
overlapping sets: $10,000$ samples for testing, $10,000$ samples for training
the Trojaned model $X$, and the rest for training (i.e., $40,000$ samples in
the CIFAR-10 and $50,000$ samples in the Fashion MNIST for training). We use
the class $0$ as a targeted class $y^{b}_{j}$ (Eq. 4) in each dataset. The
distributions of classes and size of local training data are demonstrated in
Figs. 6 and 7.

We adopt the model configuration described in [4] for both datasets, in which
we use a LeNet-based network [31] with two convolution and two fully connected
layers for the local model and a fully-connected network with three hidden
layers and multiple linear heads per target weight tensor for the HyperNetFL.
SGD optimizer with the learning rate $0.01$ for the HyperNetFL and $0.001$ for
the local model are used in our experiments.

We use WaNet [16], which is one of the state-of-the-art backdoor attacks, for
generating backdoor data. WaNet uses image warping-based triggers making the
modification in the backdoor images natural and unnoticeable to humans. We
follow the learning setup described in [16] to generate the backdoor images
that are used to train $X$. In the DP optimizer, we vary the noise scale
$\sigma\in\\{10^{-1},10^{-2},10^{-3},10^{-4}\\}$ and the clipping $l_{2}$-norm
$\mu\in\\{8,4,2,1\\}$. For the $\alpha$-trimmed norm approach, we choose
$\alpha\in\\{0.1,0.2,0.3,0.4\\}$.

Figure 4: Legitimate ACC and backdoor SR under $\alpha$-trimmed norm defense
in the Fashion MNIST dataset.

(a) Client-level DP optimizer

(b) $\alpha$-Trimmed Norm

Figure 5: Backdoor risk surface: attacks and defenses in the Fashion MNIST
dataset. The attack we used in this figure is HNTroj.

(a) CIFAR-10 Training data

(b) CIFAR-10 Testing data

(c) CIFAR-10 Backdoor data

(d) Fashion MNIST Training data

(e) Fashion MNIST Testing data

(f) Fashion MNIST Backdoor data

Figure 6: Distribution of clients and samples per clients in the CIFAR-10 and
Fashion MNIST datasets.

(a) CIFAR-10 Training data

(b) CIFAR-10 Testing data

(c) CIFAR-10 Backdoor data

(d) Fashion MNIST Training data

(e) Fashion MNIST Testing data

(f) Fashion MNIST Backdoor data

Figure 7: Distribution of class $1$ samples per client in the CIFAR-10 and
Fashion MNIST datasets. Other classes follow the same distribution with the
class $1$.

(a) Attack once from iteration $100$-th

(b) Attack once every iteration $100$-th

(c) Attack at random round

Figure 8: White-box model replacement attack in the CIFAR-10 dataset. It is
clear that the model replacement attack can significantly increase the
legitimate ACC and backdoor SR right after the attack occurred. However, if
the adversary need to apply the model replacement along the training process
to keep the backdoors effective (Figs. 8b-c); otherwise the backdoor SR will
gradually drop as time goes by (Fig. 8a).

(a) Attack once from iteration $50$-th

(b) Attack once every iteration $50$-th

(c) Attack at random round

Figure 9: White-box model replacement attack in the Fashion MNIST dataset.

(a) #Compromised clients $\complement=1$

(b) #Compromised clients $\complement=5$

(c) #Compromised clients $\complement=10$

(d) #Compromised clients $\complement=15$

(e) #Compromised clients $\complement=20$

(f) Legends for (a)-(e)

Figure 10: HNTroj under client-level DP optimizer-based robust HyperNetFL
training in the CIFAR-10 dataset.

(a) #Compromised clients $\complement=1$

(b) #Compromised clients $\complement=5$

(c) #Compromised clients $\complement=10$

(d) #Compromised clients $\complement=15$

(e) #Compromised clients $\complement=20$

(f) Summary

Figure 11: Legitimate ACC and Backdoor SR as a function of communication
rounds in DPois, HNTroj, and Clean model over different numbers of compromised
clients in the CIFAR-10 dataset. (Figs. 11a-d have the same legend as in Fig.
11e)

(a) #Compromised clients $\complement=1$

(b) #Compromised clients $\complement=5$

(c) #Compromised clients $\complement=10$

(d) #Compromised clients $\complement=15$

(e) #Compromised clients $\complement=20$

(f) Summary

Figure 12: Legitimate ACC and Backdoor SR as a function of training steps in
DPois, HNTroj, and Clean model over different numbers of compromised clients
in the Fashion MNIST dataset.(Figs. 12a-d have the same legend as in Fig. 12e)

(a) #Compromised clients $\complement=1$

(b) #Compromised clients $\complement=5$

(c) #Compromised clients $\complement=10$

(d) #Compromised clients $\complement=15$

(e) #Compromised clients $\complement=20$

(f) Legends for (a)-(e)

Figure 13: HNTroj under client-level DP optimizer-based robust HyperNetFL
training in the Fashion MNIST dataset.

(a) Class $1$ (automobile)

(b) Class $2$ (bird)

(c) Class $3$ (cat)

(d) Class $4$ (deer)

(e) Class $5$ (dog)

(f) Class $6$ (frog)

(g) Class $7$ (horse)

(h) Class $8$ (ship)

(i) Class $9$ (truck)

Figure 14: Comparison between legitimate samples (left) and their backdoor
samples generated by WaNet [16] (right) in the CIFAR-10 dataset. Targeted
label is $0$ (airplane). Backdoor samples are almost identical to the
legitimate samples, and the difference is unnoticeable to humans.

(a) Class $1$ (Trouser)

(b) Class $2$ (Pullover)

(c) Class $3$ (Dress)

(d) Class $4$ (Coat)

(e) Class $5$ (Sandal)

(f) Class $6$ (Shirt)

(g) Class $7$ (Sneaker)

(h) Class $8$ (Bag)

(i) Class $9$ (Ankle boot)

Figure 15: Comparison between legitimate samples (left) and their backdoor
samples generated by WaNet [16] (right) in the Fashion MNIST dataset. Targeted
label is $0$ (T-shirt/top). Backdoor samples are almost identical to the
legitimate samples, and the difference is unnoticeable to humans.