# FedTracker: Furnishing Ownership Verification and Traceability for Federated
Learning Model

Shuo Shao, Wenyuan Yang, Hanlin Gu, Zhan Qin, Lixin Fan, Qiang Yang, Kui Ren
S. Shao, Z. Qin, K. Ren are with the School of Cyber Science and Technology,
Zhejiang University, Chinal, Email address: {shaoshuo_ss, qinzhan,
<EMAIL_ADDRESS>W. Yang is with the School of Cyber Science and Technology,
Sun Yat-sen University, China, Email address<EMAIL_ADDRESS>H. Gu
and L. Fan are with the Webank AI Lab, China, Email address:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Q. Yang is with the Computer Science
and Engineering Department, Hong Kong University of Science and Technology,
Email address<EMAIL_ADDRESS>The first two authors have equal
contributions. Zhan Qin is the corresponding author.

###### Abstract

Federated learning (FL) is a distributed machine learning paradigm allowing
multiple clients to collaboratively train a global model without sharing their
local data. However, FL entails exposing the model to various participants.
This poses a risk of unauthorized model distribution or resale by the
malicious client, compromising the intellectual property rights of the FL
group. To deter such misbehavior, it is essential to establish a mechanism for
verifying the ownership of the model and as well tracing its origin to the
leaker among the FL participants. In this paper, we present FedTracker, the
first FL model protection framework that provides both ownership verification
and traceability. FedTracker adopts a bi-level protection scheme consisting of
global watermark mechanism and local fingerprint mechanism. The former
authenticates the ownership of the global model, while the latter identifies
which client the model is derived from. FedTracker leverages Continual
Learning (CL) principles to embedding the watermark in a way that preserves
the utility of the FL model on both primitive task and watermark task.
FedTracker also devises a novel metric to better discriminate different
fingerprints. Experimental results show FedTracker is effective in ownership
verification, traceability, and maintains good fidelity and robustness against
various watermark removal attacks.

###### Index Terms:

Federated learning, ownership verification, traceability, model watermark

##  1 Introduction

Federated Learning (FL) [1] has turned into one of the most popular frameworks
for distributedly training deep learning models without sharing private
messages. FL is nowadays widely applied in various privacy-sensitive domains,
such as object detection [2, 3], medical image analysis [4, 5, 6],
recommendation systems [7, 8] and so on. FL meets the privacy needs of the
data owners while producing high-quality models for the community. However,
collaborative training in FL makes the model publicly available to multiple
participants, which may lead to the model leakage issue in FL.

Model leakage in FL refers to the malicious distribution or sale of FL models
by clients for personal gain. This behavior infringes the legitimate copyright
of the FL group. Detecting copyright infringement and protecting intellectual
property rights has become a critical issue in FL. There are two major
concerns in FL copyright protection, ownership verification and traceability.
Ownership verification involves verifying the ownership of the FL model when
it is illegally distributed and charging the unauthorized user outside the FL
group with tort. The FL model should be considered as the intellectual
property of the FL group instead of any individual. Any legal use of the FL
model should be made with the consent of the FL group. Traceability refers to
disclose the identity of the culprit among the FL participants who illegally
distribute the model to third-parties outside the FL group. It is a new
requirement for protecting the copyright of the FL model, which to our best
knowledge has not been addressed before.

To protect the copyright of Deep Neural Networks (DNN), DNN watermarks [9, 10,
11, 12, 13, 14], including _parameter-based_ watermark and _backdoor-based_
watermark, are proposed. _Parameter-based_ watermark methods embed personal
watermarks into the parameters or the distribution of the parameters of the
DNN models [9, 15, 11, 16]. It can provide multi-bit information, but the
extraction of watermarks needs white-box access to the suspicious model, which
limits the application. On the contrary, _backdoor-based_ watermark methods
[10, 13, 17, 18] take advantage of backdoor attack [19] to insert specific
triggers to identify the ownership of the model. Although _backdoor-based_
watermark can be verified through black-box access, it is usually zero-bit.
This means the watermark can only demonstrate the presence or absence of the
watermark instead of a bit string representing identity.

Figure 1: Illustration of model leakage in FL and the solution framework.
Left: the malicious client can sell the model to other parties. Middle: FL
group needs to verify the ownership through API access. Right: fingerprint is
extracted from the model and compared with all the fingerprints of Clients to
trace the leaker.

Some recent works also explores applying DNN watermarks to FL copyright
protection. WAFFLE [20] is the first watermarking scheme designed for FL, in
which the Server is responsible for embedding the _backdoor-based_ watermarks
into the FL model. WAFFLE propose to conduct a retraining step after federated
aggregation to embed the watermark. Liu et al. [21, 22] and Li et al. [23]
propose to entitle the Clients to embed both _parameter-based_ and _backdoor-
based_ watermarks into the global model. However, existing works only focus on
ownership verification. None of them can provide traceability of the model
leakers for FL models. Even if the FL group can prove the ownership of the FL
model in the event of the infringement, they are not capable of detecting the
model leakers hidden among them.

We aim to design a complete FL copyright protection framework that can furnish
both ownership verification and traceability for the FL model. In this paper,
we present FedTracker, the first FL model protection framework that offers
both. The illustration of the problem and our framework is shown in Figure 1.
Specifically, we design a bi-level protection scheme that enables ownership
verification and traceability respectively. On one level, FedTracker embeds a
global watermark into the aggregated global model in each iteration. While the
copyright infringement occurs, the FL group can extract the watermark and
verify the ownership. On the other level, before distributing the model in
each iteration, the server insert a unique local fingerprint for each client
to identify each model. The model leaker can be traced by disclose the
identification inside the model.

However, the challenges of applying such framework are two-fold:

  * •

It is challenging to preserve the utility of the model in condition of a lack
of natural data. Since the server is assumed to have no access to the training
data, the server should embed the watermark which is out of distribution of
natural data. This can compromise the model’s utility.

  * •

It is challenging to guarantee the discrimination of different models. Since
we need to embed and compare multiple different fingerprints, it puts a higher
requirement on the identifiability and robustness of fingerprints and the
model leaker.

To tackle the challenges mentioned above, first we propose a watermark
embedding algorithm based on Continual Learning (CL) [24] to improve the
utility of the watermarked model. We regard the primitive task of FL and the
watermark embedding task as learning data from different domains and the
utility drop is attributed to the catastrophic forgetting while learning a
different task. We get the inspiration from the Gradient Episodic Memory (GEM)
[25] and design a CL-based embedding algorithm for the FL scenario. We
introduce our CL-based watermark embedding algorithm in Section 4.2.

Second, to enhance the differentiation between different models of different
Clients, we conduct the fingerprint generation by minimizing the maximal
distance between fingerprints. We also propose a Fingerprint Similarity Score
(FSS) to measure the similarity of the extracted fingerprint and clients’
fingerprints. Compared with traditional hamming distance whose output is
discrete, FSS can provide more fine-grained measurement because the output of
FSS is continuous. And thus, using FSS can better discriminate different
fingerprints. Moreover, we embed the watermark and fingerprints respectively
into different parts of the model to lower the mutual impact and improve the
effectiveness of both mechanisms. The details are present in Section 4.3.

###  1.1 Contributions

Our main contributions can be summarized as follows:

  * •

We propose the first FL ownership verification and traceability framework
called FedTracker. In FedTracker, the bi-level protection scheme, including
the global watermark mechanism and the local fingerprint mechanism, can
provide ownership verification and traceability respectively. Experimental
results show that FedTracker can also provide tremendous fidelity and
robustness against various attacks.

  * •

Inspired by the motivation of Continual Learning (CL), we design a CL-based
watermark embedding algorithm for the FL scenario. It can maintain the utility
of the FL model on both primitive task and the watermark task while embedding
the watermark.

  * •

We propose a fingerprint generation method based on optimization and a novel
metric named Fingerprint Similarity Score (FSS). The two methods can help
better distinguish different fingerprints and improve the effectiveness of
tracing the culprit.

###  1.2 Roadmap

The rest of this paper is organized as follows: Section 2 introduces the
background knowledge of FedTracker. Section 3 gives the formulation of the
copyright protection problem in FL, including the definition of ownership
verification and traceability and the threat model. Section 4 proposes the
procedure of FedTracker, and the solutions to the key challenges proposed
above. Detailed implementation of FedTracker can be found in Section 5.
Section 6 gives the empirical experiments of FedTracker and Section 7
concludes this paper.

##  2 Background

###  2.1 Federated Learning

Federated Learning (FL) [1] is a distributed machine learning paradigm that
trains a Deep Learning model across multiple participants holding local data
samples. In FL, intermediate results such as gradients rather than sensitive
data are shared between data owners. This approach can protect the privacy and
security of the data owner, while enabling collaborative learning among
different parties.

Client-Server FL is the most popular FL paradigm in practice. In the Client-
Server FL [26, 27], there are two different parties involved: Clients and
Server. Clients $C=\\{c_{i}\\}_{i=1}^{K}$ are the data owners in FL who hold a
bunch of personal data. $K$ is the number of Clients. And Server is
responsible for collecting the intermediate local gradients
$L=\\{g_{i}\\}_{i=1}^{K}$ from Clients and aggregating them into a global
model $M_{g}$.

In each iteration $t$, Clients and Server do

  1. 1.

Clients get the global model $M^{t-1}_{g}$ of the $t-1$ round from Server. We
use the superscript $t-1$ to mark the iteration number and the subscript
represents the number of clients or the global model (the subscript is $g$).

  2. 2.

Each Client $c_{i}$ copy the global model onto its local model
$M_{i}^{t-1}\leftarrow M_{g}^{t-1}$, and then utilizes its private dataset
$D_{i}$ to train the local model $M^{t-1}_{i}$ for several epochs. For a given
loss function $\mathcal{L}$, the local gradients $g_{i}^{t}$ can be calculated
using the following equation.

$g_{i}^{t}=\frac{\partial\mathcal{L}(M_{i}^{t-1},D_{i})}{\partial
M_{i}^{t-1}}$ (1)

Then, the Clients apply gradient descent to update the model to $M_{i}^{t}$.

$M_{i}^{t}=M_{i}^{t-1}-\lambda_{i}g_{i}^{t}$ (2)

where $\lambda_{i}$ is the local learning rate of the Client $i$. After that,
Clients sends the local model $M_{i}^{t}$ of $t$-th iteration to the Server
for aggregation.

  3. 3.

Server collects the local models and performs the federated aggregation
algorithm to aggregate the local gradients into a new global model
$M^{t}_{g}$. FedAvg [1] is a widely-used aggregation algorithm in FL. In
particular, FedAvg utilizes Equation 3 to aggregate the local models.

$M_{g}^{t}=\sum_{i=1}^{K}\frac{|D_{i}|}{\sum_{j=1}^{K}|D_{j}|}M_{i}^{t}$ (3)

  4. 4.

Server sends the global model $M^{t}_{g}$ to all the Clients. Clients acquire
the new global model and begin the next iteration of FL training.

###  2.2 DNN Watermarking Methods

Broadly speaking, existing DNN watermarking methods can be categorized as
_parameter-based_ watermarks and _backdoor-based_ watermarks [28, 29].

Parameter-based Watermarks [9, 15, 11, 16]. _Parameter-based_ watermarks
decide to embed the watermarks, which can be represented as an $N$-bit string
$F\in\\{0,1\\}^{N}$, directly into the parameters $W$ of the models. The
embedding of the watermark can be considered a binary classification problem
and the watermark can be fitted by adding a specific regularization term into
the loss function.

During the verification, the watermark $F^{{}^{\prime}}$ is extracted from the
suspicious model. $F^{{}^{\prime}}$ is then compared with the designated
watermarks $F$. If the distance $D(F,F^{{}^{\prime}})$ (usually hamming
distance) of the two watermarks is less than a threshold $\varepsilon_{P}$,
the ownership of the model can be verified.

_Parameter-based_ watermarks can provide multi-bit information, but the
extraction of watermarks needs white-box access to the suspicious model.

Backdoor-based Watermarks [10, 13, 17, 18]. _Backdoor-based_ watermarks
utilize backdoor attack [19, 30, 31] to insert specific trigger set $D_{T}$
into the DNN model. The backdoor attack leads to misclassification when
encountering samples in the trigger set. The trigger set is unique to the
backdoored model, thus the owner can verify its ownership by triggering the
misclassification.

_Backdoor-based_ watermarks can be verified through black-box access, but they
are typically zero-bit. This means they only indicate the presence or absence
of the watermark, not a bit string that identifies the owner.

###  2.3 FL Watermarking Methods

DNN watermarking methods can also be adopted to protect the copyright of FL
models. The FL watermarking methods can be typically categorized as _server-
side_ and _client-side_ watermarking methods.

WAFFLE [20] proposes the first server-side FL watermarking method. WAFFLE adds
a retraining step to embed _backdoor-based_ watermarks into FL global model.
Besides, because Server is assumed to lack training data, WAFFLE designs a
data-free trigger set construction algorithm called WafflePattern. However,
WAFFLE only focuses on the copyright protection and ignores the traceability.
The fidelity of applying WAFFLE is also not guaranteed.

As for client-side watermarking methods, Liu et al. [21] and Li et al. [23]
propose to entitle the Clients to watermark the FL model. Liu et al. [21]
suggest selecting a representative client to embed watermark through gradient-
enhanced embedding algorithm. Li et al. [23] think every client should have
the ability to verify its copyright. And thus Li et al. proposes FedIPR which
entitles every Client to embed its own watermark. Nevertheless, what the
Clients embed can hardly be monitored. The Clients can embed both the
watermarks and harmful things like backdoor [32, 33] into the model. The
security of Client-side watermarking is not guaranteed.

###  2.4 Continual Learning

Continual Learning (CL) [34], also called life-long learning [35] or
incremental learning [36], is a learning technique that aims to enable a
machine learning model to learn from a stream of data without forgetting the
previous knowledge. CL poses several challenges for both theory and practice,
such as how to measure and mitigate forgetting, how to transfer and generalize
knowledge across tasks, and how to balance stability and plasticity of the
learning system.

A common approach of CL is to use regularization techniques that constrain the
model parameters to prevent catastrophic forgetting, such as elastic weight
consolidation (EWC) [24], synaptic intelligence (SI) [37], or learning without
forgetting (LwF) [38]. Another approach is to use replay techniques that
augment the current data with previous tasks. Most notable method is class
incremental learner iCaRL [39]. iCaRL stores a subset of exemplars per class,
selected to best approximate the class means in the learned feature space.
Gradient episodic memory (GEM) [25] proposes to project the gradient direction
of the current task on the feasible region outlined by previous tasks. Thus,
the new task updates do not interfere with previous tasks.

##  3 Problem Formulation

###  3.1 Definitions of Ownership Verification and Traceability

Federated Learning enables multiple data owners to collaboratively train a DNN
model in a privacy-preserving way due to the increasing concern about personal
privacy security. However, FL faces the challenge of model leakage. Model
leakage refers to that the malicious participants in FL can stealthily copy
and sell the FL model to other parties. Such misbehavior infringes the
legitimate copyright of the FL group, and we should design a mechanism to
verify the ownership and disclose the identity of the model leaker among the
participants.

There are two major concern in the copyright protection of the FL model,
ownership verification and traceability. In this subsection, we give the
formal definitions of ownership verification and traceability.

######  Definition 1 (Ownership verification)

Ownership verifi-cation refers to proving that the suspicious model belongs to
the FL group, and then charging the unauthorized user outside the FL group
with tort. When the infringement occurs, the FL group can apply the
verification algorithm ${\tt Verify}(\cdot)$ to verify their ownership on the
model. The ownership verification mechanism ${\tt Verify}(\tilde{M},D_{T})$
should output

${\tt Verify}(\tilde{M},D_{T})=\left\\{\begin{aligned} &{\tt
True},\tilde{M}\in\bigcup_{i=1}^{K}{\tt att}(M_{i})\\\ &{\tt False},{\tt
if\enspace otherwise}\end{aligned}\right.$ (4)

where $\tilde{M}$ is the suspicious model, $D_{T}$ is the verification token
offered by the FL group, and ${\tt att}(\cdot)$ represents various attack
methods which will be introduced in Section 3.2.

######  Definition 2 (Traceability)

Traceability refers to tracing the stolen model to the malicious Client in FL.
After verifying the ownership of suspicious model, the FL group should further
adopt the tracing mechanism ${\tt Trace}(\cdot)$ to disclose the model leaker.
Given a suspicious model $\tilde{M}$ and fingerprints of clients
$\\{F_{i}\\}_{i=1}^{K}$, the traceability mechanism ${\tt
Trace}(\tilde{M},\\{F_{i}\\}_{i=1}^{K})$ should output the identity (i.e.
index) of the model leaker in FL.

${\tt Trace}(\tilde{M},\\{F_{i}\\}_{i=1}^{K})=j,\tilde{M}\in{\tt att}(M_{j})$
(5)

###  3.2 Threat Model

Server and Clients are the two different parties in FL. Following the prior
works [20], we assume that the Server is a trusted and honest party. Some of
the Clients might be malicious, while the other Clients are assumed to be
honest. Malicious clients can copy, distribute and sell the jointly-trained
model stealthily.

Adversary’s assumptions: the adversary intends to acquire a high-performance
DNN model through FL, and secretly sell the model to other parties. Moreover,
the adversary can try to remove the watermarks in the model. We assume that
the adversary has the following capabilities:

  * •

The adversary has access to its own dataset $D_{adv}$ and the local model
$M_{adv}^{t}$ in any iteration round $t$ of FL training. Also, the adversary
can save and copy the model $M_{adv}^{t}$ at any time of training.

  * •

The adversary follows the training procedure to maximize the utility and value
of the FL model.

  * •

The adversary can conduct several removal techniques to get rid of the
watermarks in the DNN model. The watermark removal techniques are listed
below.

Adversary’s Attack Techniques: The adversary may adopt various techniques to
remove the watermarks, so that it can evade the verification and tracing.
Drawing on the literature of watermark attack methods proposed in preliminary
works, we classify the potential techniques of adversaries into several
distinct types.

Fine-tuning. The adversary may adopt Fine-tuning to obfuscate the model
parameters and behaviors. Fine-tuning attacks refer to resuming the training
procedure and modifying the model parameters using the dataset of the
adversary and a low learning rate. Due to the non-convexity of deep learning
[40], fine-tuning might make the deep learning model get to a different local
optimum, and thus removes the watermark. Fine-tuning should be considered as a
threat to the watermarks.

Model Compression. Model compression methods aim to eliminate the redundancy
in the model parameters, and thus, the embedded watermark may also be erased
during this process. In this paper, we consider two different categories of
model compression methods, _model pruning_ [41] and _quantization_. Model
pruning eliminates the parameters that have little or no impact on the network
performance. Model quantization reduces the numerical precision of the
parameters, such as using 8-bit integers instead of 32-bit floats, to save
memory bandwidth and computational cost. Both techniques could have a negative
impact on the watermark with minimum accuracy loss.

Overwriting. Overwriting aims to remove or alter the watermark embedded in a
neural network model by adding a new watermark to the model. Overwriting
attack is a type of active attack. The adversary randomly generates his own
watermark and uses the same method to embed the watermark into the model. The
watermark in the model may be significantly affected by overwriting.

Backdoor Mitigation. Backdoor mitigation method [42] is the process of
detecting and removing backdoor triggers from a Deep Learning model. Since
current black-box watermarking methods mostly utilize backdoor to embed and
verify the watermark, backdoor mitigation method can be a threat to the
watermark embedded in the model.

###  3.3 Defender Assumptions and Objectives

In our scenario, the Server in FL is the defender. The defender tries to
furnish ownership verification and traceability for FL models. The
capabilities of the defender are as follows.

  * •

The defender can only get the aggregated gradients or model in each iteration
for sake of preserving the data privacy. The local gradients of the Clients
are noised or encrypted [43].

  * •

The defender has no access to any natural data related to the primitive deep
learning task.

The defender needs to achieve the following objectives:

  * •

Ownership verification: the defender should devise a mechanism to effectively
authenticate the FL group’s ownership on the FL model.

  * •

Traceability: given an FL model, the defender should trace the holder of the
model inside the FL group and catch the culprit.

  * •

Fidelity: fidelity refers to that the protection mechanism should have
negligible impact on the utility of the model. In other words, the accuracy of
the model on the primitive task should be maintained.

  * •

Robustness: robustness refers to that the protection mechanism cannot be
removed by the adversary. The protection mechanism should resist various
attacks, such as fine-tuning, model compression, overwriting, and backdoor
mitigation.

##  4 Ownership Verification and Traceability in Federated Learning

###  4.1 Overview of FedTracker

In this paper, we propose FedTracker, the first FL copyright protection
framework that can provide both ownership verification and traceability.
FedTracker conducts the protecting mechanism from Server-side due to the
security issues of Client-side watermarking discussed in Section 2.3. In
FedTracker, a bi-level protecting mechanism is included. For the first level,
a global watermark is embedded into the global model so that the models in FL
can contain the global watermark for ownership verification. We utilize
backdoor-based watermark as the global watermark since backdoor-based
watermark can be verified through API access. For the second level, a unique
local fingerprint is inserted into the model of each client for further
traceability. Since traceability requires distinguishing different Clients’
model, the multi-bit parameter-based watermark is feasible as the local
fingerprints.

Figure 2: Workflow of FedTracker in each iteration. FedTracker consists of
four stages. \footnotesize{1}⃝ Local training and aggregation.
\footnotesize{2}⃝ Global watermark embedding. \footnotesize{3}⃝ Local
fingerprints insertion. \footnotesize{4}⃝ Model distribution.

The main procedure of FedTracker is demonstrated in Algorithm 1. After
initializing the model parameters and generating the watermark and
fingerprints, in each training iteration, FedTracker consists of four stages
which are shown in Figure 2. The pseudocode of FedTracker is presented in
Algorithm 1.

  * •

Stage 1. (_Local training and aggregation_) Each Client utilizes their own
dataset to train their model locally. The Server then collects the local
gradients and aggregates them into a global model via the federated
aggregation algorithm.

  * •

Stage 2. (_Global watermark embedding_) The Server embeds the global watermark
into the new aggregated global model. The detail implementation of the global
watermark mechanism is discussed in Section 5.2.

  * •

Stage 3. (_Local fingerprints insertion_) The Server first generates $K$
copies of the global model ($K$ is the number of Clients). Then, a unique
local fingerprint that represents a client is inserted into each copy. The
implementation of the local fingerprints mechanism is discussed in Section
5.3.

  * •

Stage 4. (_Model distribution_) The Server distributes the fingerprinted
models to the Clients respectively. The Clients begin the next iteration of
federated training.

As present in Section LABEL:sec:intro, the challenges are two-fold. First, how
to preserve the utility of the model while embedding watermark without natural
data. Second, how to efficiently discriminate different models of different
Clients. We introduce our motivation and solutions on these challenges in
Section 4.2 and Section 4.3.

Algorithm 1 Main procedure of FedTracker

Input: Clients datasets $\\{D_{i}\\}_{i=1}^{K}$
Output: the protected Clients’ models $\\{\hat{M}^{T}_{i}\\}_{i=1}^{K}$, the
protected global model $\hat{M}^{T}_{g}$.

1:$D_{T},\\{F_{i}\\}_{i=1}^{K}={\tt Gen}()$

2:$M^{0}_{g}={\tt Initialize}()$

3:for $i=1$ to $K$ do

4: $M^{0}_{i}={\tt Copy}(M^{0}_{g})$

5:for $t=1$ to $T$ do

6: // _\footnotesize{1}⃝ Local training and aggregation_

7: for $i=1$ to $K$ do

8: $M^{t}_{i}={\tt LocalTrain}(M^{t-1}_{i},D_{i})$

9: $l_{i}^{t}=M^{t-1}_{i}-M^{t}_{i}$

10: $M_{g}^{t}=\sum_{i=1}^{K}\frac{|D_{i}|}{\sum_{j=1}^{K}|D_{j}|}M_{i}^{t}$

11: // _\footnotesize{2}⃝ Global watermark embedding_

12: $\hat{M}_{g}^{t}={\tt GEmbed}(M_{g}^{t},D_{T})$

13: // _\footnotesize{3}⃝ Local fingerprints insertion_

14: for $i=1$ to $K$ do

15: $\hat{M}_{i}^{t}={\tt Copy}(\hat{M}^{t}_{g})$

16: $\hat{M}_{i}^{t}={\tt LInsert}(\hat{M}_{i}^{t},F_{i})$

17: // _\footnotesize{4}⃝ Model distribution_

18: for $i=1$ to $K$ do

19: $M_{i}^{t}=\hat{M}_{i}^{t}$

20:return $\\{\hat{M}^{T}_{i}\\}_{i=1}^{K},\hat{M}^{T}_{g}$

###  4.2 Embedding Watermark without Natural Data

Observations and Insight. In FedTracker, we propose to embed the global
watermark from server-side. We take advantage of _backdoor-based_ watermark
and try to embed a special trigger set as the global watermark. Backdoor-based
zero-bit watermark is enough for ownership verification and verifying through
API access is more feasible in practice.

However, retraining the model on the trigger set without data related to the
primitive task compromises the utility of the model, since the server has no
access to the training data. The situation of model utility degradation is
more obvious when using models with Batch Normalization (BN) layers. Take
WAFFLE as example. When directly applying WAFFLE to models containing BN
layers, we observe a significant utility degradation as shown in Table I. We
train the variants with BN layers of two popular convolutional neural networks
(CNN), AlexNet [44] and VGG-16 [45], on CIFAR-10 [46]. As Table I
demonstrates, the accuracies drop more than $50\%$ when applying WAFFLE to the
models with BN layers, which indicates that the models are completely useless.

This can be attributed to the following two reasons. Firstly, in the
initialization phase, the server needs to conduct a data-free trigger set
construction algorithm [22] to generate the watermark trigger set. The data-
free trigger set is out of distribution (OOD) of the training data. Retraining
the model on the OOD trigger set leads to catastrophic forgetting [24] of the
previous task and has a large impact on the functionality of the model.
Secondly, the principle of BN layers amplifying the negative effect. BN works
by applying a linear transformation to the outputs of the previous layer, such
that they have zero mean and unit variance. However, the mean and variance of
the trigger set are far different from natural data. Thus, the utility of the
model is significantly compromised during watermark embedding.

TABLE I: Testing accuracy (%) of the models with BN layers when applying WAFFLE or not embedding the watermark. |  | Number of Clients
---|---|---
| Model | 10 | 20 | 30 | 50
AlexNet | No WM | 86.79 | 86.41 | 86.12 | 86.05
WAFFLE | 27.91 | 29.84 | 32.60 | 30.70
VGG-16 | NO WM | 89.92 | 89.68 | 89.39 | 89.04
WAFFLE | 21.34 | 24.46 | 24.12 | 25.66

Based on the above observations, we consider the primitive task and the
watermark embedding task as two different tasks from different domains, which
is the intuition of Continual Learning (CL) [34]. CL aims to reduce the
catastrophic forgetting of the previous task when learning a new task.
Applying CL can help to reduce the negative impact on the primitive task and
maintain the utility of the model on both two tasks.

CL-based Watermark Embedding Mechanism. In FedTracker, we propose a CL-based
watermark embedding algorithm. We get the inspiration from Gradient Episodic
Memory (GEM) [25]. GEM stores a subset of data from previous tasks in an
episodic memory and using it to constrain the gradient updates during
training. However, the server has no training data to store. Thus, we propose
to alter the primitive GEM to fit the Federated Learning scenario.

Considering the constraints that the only messages the Server can get are the
aggregated models or gradients, different from GEM, we propose to store the
gradients instead of data from prior tasks as memory. In FedTracker, we
introduce _global memory_. Global memory is defined as the accumulated global
gradients from the first iteration to the current iteration. We do not use the
global gradient in the current iteration because the gradient of a single
iteration tends to be unstable in FL. The global memory is defined as follows.

######  Definition 3 (Global memory)

Global memory is defined as the accumulated global gradients from the first
iteration to the current iteration. Global memory $m^{t}$ of the $t$-th
iteration can be calculated using Equation (6),

$m^{t}=\sum_{j=1}^{t}G^{j}=\sum_{j=1}^{t}\sum_{i=1}^{K}g_{i}^{j},$ (6)

where $g_{i}^{j}$ is the local gradients of the $i$-th client in $j$-th
iteration and $G^{j}$ is the aggregated global gradients. Because the server
cannot get the local gradients of each Client. The server can calculate the
global memory by summing up the aggregated gradients
$G^{j}=M_{g}^{j}-M_{g}^{j-1}$ of each iteration.

The global memory $m^{t}$ represents the descent direction of the loss of the
primitive task. While learning the trigger set task using gradient descent, we
should avoid the loss of the primitive task increasing. This can be considered
as an inequality constraint measured by computing the angle between the
gradients $g$ of the new task and the memory $m^{t}$. Mathematically, we
phrase the optimization problem as:

$\min loss(M_{g}^{t},D_{T}),\enspace s.t.\enspace\langle g,m^{t}\rangle\geq
0,$ (7)

where $g=\frac{\partial loss(M_{g}^{t},D_{T})}{\partial M_{g}^{t}}$.

If the constraint is satisfied, we can directly apply the gradients $g$ to
optimize the loss of watermark without increasing the loss of the primitive
task. However, if the violation occurs, we should project the gradients $g$ to
the closest point $\tilde{g}$ satisfying the constraint. The projection
problem can be phased as Equation (8). The problem can be solved via Quadratic
Programming.

$\displaystyle\mathop{\arg\min}\limits_{\tilde{g}}\left\lVert
g-\tilde{g}\right\rVert_{2}^{2},\enspace
s.t.\enspace\langle\tilde{g},m^{t}\rangle\geq 0.$ (8)

###  4.3 Distinguish Different Models of Different Clients

As discussed before, a key challenge in tracing the model leaker in FL is that
the aggregated models received by the clients are identical. We need to design
a mechanism to embed unique, unremovable, and distinguishable identity
certificate to the local models. Thus, in FedTracker, we propose the local
fingerprint mechanism.

Local fingerprint is used to represent the identity of the clients. In
FedTracker, we utilize multi-bit _parameter-based_ method as local
fingerprints. A unique local fingerprint is inserted into the model of each
client. The local fingerprint mechanism includes three phase: local
fingerprint generation, local fingerprint insertion and tracing.

Local Fingerprints Generation. While generating the fingerprint $F_{i}$, we
propose to maximize the distance between fingerprints to better distinguish
them. The bigger the gap between different fingerprints, the more accurate it
is for the FL group to disclose the identity of the model leaker among the FL
participants. We model the insight as an optimization problem. In FedTracker,
we use Hamming distance as metric since the fingerprint values are discrete in
generation phase. We need to maximize the minimal hamming distance among the
fingerprints. The optimization problem can be defined as Equation (9).

$\\{F_{i}\\}_{i=1}^{K}=\arg\max\min_{1\leq i,j\leq K}{\tt HD}(F_{i},F_{j}),$
(9)

where ${\tt HD}(F_{i},F_{j})$ calculates the Hamming distance between $F_{i}$
and $F_{j}$.

However, the optimization problem is an NP-hard problem, which has no known
polynomial-time solution. We therefore utilize the genetic algorithm (GA) [47]
to solve the optimization problem and generate the fingerprints.

Local Fingerprint Insertion. In FedTracker, given a weight matrix
$W_{i}\in\mathbb{R}^{M}$ of the Client $i$’s model, we use the following
equation to represent the local fingerprint $F\in\\{-1,1\\}^{N}$

${\tt sgn}(AW)=F,$ (10)

where $A_{i}$ is the secret key matrix that is randomly generated by the
Server. In FedTracker, the secret key matrix $A_{i}\in\mathbb{R}^{M\times N}$
is generated for each fingerprint $F_{i}\in\\{-1,1\\}^{N}$. ${\tt sgn}(\cdot)$
originates from the sign function.

${\tt sgn}(x)=\left\\{\begin{aligned} -1,x<0\\\ 1,x\geq
0\end{aligned}\right..$ (11)

In FedTracker, we choose the weight of the BN layers to insert local
fingerprints. Recall that in BN layers, we calculate the following equation
[48] to normalize the output $x$ of the previous layer,

$o=\gamma\frac{x-\mu}{\sqrt{\sigma^{2}+\epsilon}}+\beta,$ (12)

where $\mu$ and $\sigma$ are the mean and variance of input batch $x$,
$\epsilon$ is a small constant. $\gamma$ and $\beta$ are learnable parameters.
We choose $\gamma$ to embed the local fingerprint. $\gamma$ from different BN
layers are concatenated into one vector $W^{\gamma}$. We define the Hinge-like
loss function $L_{f}(W^{\gamma}_{i})$ of fitting the fingerprint for the
$i$-th client as Equation (13).

$L_{f}(W_{i}^{\gamma})={\tt
HL}(A_{i},F_{i},W_{i}^{\gamma})=\sum_{j=1}^{|F_{i}|}\max(\delta-
b_{ij}f_{ij},0).$ (13)

In Equation (13), $b_{ij}$ is the $j$-th bit in $B_{i}=A_{i}W^{\gamma}_{i}$,
while $f_{ij}$ is the $j$-th bit in $F_{i}$. $\delta$ is the hyperparameter
controlling the robustness of the fingerprint. We optimize $L_{f}$ using
gradient descent with a small learning rate to reduce the impact on other
tasks.

Fingerprints Similarity Score. After inserting the local fingerprints, it is
necessary to define a metric to measure the similarity between the extracted
fingerprint and the Clients’ fingerprints. Existing works mainly utilize
Hamming distance to measure the difference between two fingerprints [23, 9].
However, Hamming distance is a metric whose outputs are discrete. There may be
ambiguity while finding the most similar fingerprint. Specifically, for an
extracted fingerprint $\hat{F}$, there are probably two fingerprints
$F_{i},F_{j}$ that ${\tt HD}(\hat{F},F_{i})={\tt HD}(\hat{F},F_{j})$, since
the hamming distance rounds the distance to integer. Using Hamming distance is
not feasible in our scenario.

In FedTracker, we consider defining a new metric whose outputs are continuous
to better measure the difference between fingerprints. Therefore, we propose
Fingerprint Similarity Score:

######  Definition 4 (Fingerprint Similarity Score)

Given the secret key matrix $A_{i}$, the weight matrix $W$, the fingerprint
$F_{i}$ and $B_{i}=A_{i}W$, the Fingerprint Similarity Score (FSS) for the
$i$-th client can be calculated with Equation (14).

${\tt FSS}(A_{i},F_{i},W)={\tt
FSS}(B_{i},F_{i})=\sum_{j=1}^{|F_{i}|}\min(\delta,b_{ij}f_{ij}).$ (14)

FSS can provide continuous metric and better robustness. The further
experiments comparing the two metrics are shown in Section 6.8. The
traditional Hamming distance rounds the values and reduce the differentiation
. While tracing the leaker, we calculate the FSS for every $A_{i}$ and
$F_{i}$, and get a similarity vector. The Client with the maximal FSS is
suspected of being the leaker.

##  5 Implementation of FedTracker

In this section, we present the detailed implementation of FedTracker.
FedTracker can be defined as a tuple $({\tt
Gen,GEmbed,LInsert,Verify,Trace})$.

###  5.1 Watermark and Fingerprint Generation

In FedTracker, the server should first generate the global watermark for
ownership verification and the local fingerprints for traceability. The
watermark and fingerprint generation algorithm is abstracted to ${\tt
Gen}(\cdot)\rightarrow(D_{T},\\{F_{i}\\}_{i=1}^{K})$. The output of ${\tt
Gen}(\cdot)$ has two parts. The former is the global watermark $D_{T}$, and
the latter is $K$ different local fingerprints which is denoted as the set
$\\{F_{i}\\}_{i=1}^{K}$. The pseudocode of the watermark and fingerprint
generation algorithm is demonstrated in Algorithm 2.

The global watermark is a specific trigger set $D_{T}$, which is composed of
the trigger set data $x^{T}$ and the trigger set labels $y^{T}$. In our
implementation, we can utilize any data-free trigger set generation algorithm
as our watermark generation method.

Algorithm 2 Watermark and Fingerprints Generation ${\tt Gen}(\cdot)$

Input: The length of fingerprints $N$, the length of embedding layers’ weights
$M$.
Output: The watermark trigger set $D_{T}$, the local fingerprints
$\\{(F_{i},A_{i})\\}_{i=1}^{K}$.

1:$D_{T}={\tt WMGen}()$ // _Generate the trigger set._

2:for $i=1$ to $K$ do

3: $A_{i}={\tt randn}((M,N))$. // _Sample from standard Gaussian
distribution._

4: $F_{i}={\tt choice}(N,[-1,1])$. // _Initialize the local fingerprint._

5:$\\{F_{i}\\}_{i=1}^{K}={\tt GASolve}(\\{F_{i}\\}_{i=1}^{K},{\tt HD})$. //
_Solve the optimization problem via GA._

6:return $D_{T},\\{(F_{i},A_{i})\\}_{i=1}^{K}$.

The local fingerprint for each client also consists of two parts, a secret key
matrix $A_{i}\in\mathbb{R}^{M\times N}$ and a fingerprint bit string
$F_{i}\in\\{-1,1\\}^{N}$. In our implementation, we randomly generate the
secret key matrix $A_{i}$ from the standard Gaussian distribution
$\mathcal{N}(0,1)$. And we generate the fingerprint bit strings by solving the
optimization problem presented in Section 4.3.

###  5.2 Global Watermark Embedding

Global watermark aims to provide ownership verification for FL models. The
global watermark embedding algorithm ${\tt
GEmbed}(M_{g},D_{T})\rightarrow\hat{M}_{g}$ embeds the global watermark into
the model $M_{g}$, and output the protected model $\hat{M_{g}}$. The
pseudocode of the global watermark embedding algorithm is shown in Algorithm
3. In the global watermark embedding algorithm, the server first update the
global memory using Equation (6). Then the server freezes the BN layers of the
global model, which means the parameters of the BN layers are not updated
during retraining. After that, FedTracker calculates the gradients and
projects them by Equation (8). We also utilize the early stopping technique,
which can help to retain the utility of the FL model. We end the retraining
process when the accuracy of the watermark trigger set is already higher than
a threshold $\tau_{w}$.

As demonstrated in Algorithm 3, we freeze the BN layers in the model during
learning the trigger set task on account of the following two reasons. First,
as analyzed in Section 4.2, the trigger set generated by the data-free trigger
set construction method [20, 10] is out of the distribution of natural data.
Directly learning the trigger set can significantly affect the stability of BN
layers and decrease the utility. Thus freezing the BN layers can help retain
the utility of the model. Second, Freezing the BN layers can also avoid
negative influence on local fingerprints, since the local fingerprints are
embedded in the BN layers.

Algorithm 3 Global Watermark Embedding ${\tt GEmbed}(\cdot)$

Input: The watermark trigger set $D_{T}$, the aggregated global model
$M^{t}_{g}$ in $i-$th iteration, the global watermark retraining learning rate
$\lambda_{w}$
Output: The watermarked global model $\hat{M}^{t}_{g}$.

1:$m^{t}=\frac{1}{t}m^{t-1}+\frac{t-1}{t}(M^{t}_{g}-M^{t-1}_{g})$ // _Update
global memory._

2:$i=0$

3:$\hat{M}^{t}_{g}=M^{t}_{g}$

4:${\tt FreezeBN}(\hat{M}^{t}_{g})$

5:while $i<{\tt max\\_iter}$ and ${\tt
acc}(\hat{M}^{t}_{g},D_{T})\leq\tau_{w}$ do

6: $i=i+1$

7: $g=\frac{\partial loss(\hat{M}_{g}^{t},D_{T})}{\partial\hat{M}_{g}^{t}}$

8: $\tilde{g}={\tt project}(g,m^{t})$

9: $\hat{M}^{t}_{g}=\hat{M}^{t}_{g}-\lambda_{w}\tilde{g}$

10:return $\hat{M}^{t}_{g}$.

###  5.3 Local Fingerprints Insertion

Local Fingerprints aim to identify the models of clients and trace the model
leaker. The local fingerprint insertion algorithm ${\tt
LInsert}(\hat{M}_{g},F_{i})\rightarrow\hat{M_{i}}$ takes the watermarked model
$\hat{M_{g}}$ and the client’s fingerprint $F_{i}$ as input, and output the
model $\hat{M_{i}}$. $\hat{M_{i}}$ contains both the global watermark and the
local fingerprint. The pseudocode of local fingerprint insertion can be found
in Algorithm 4. The server first makes a copy of the watermarked global model
and gets the weights $W_{i}$ of the layers. Then the server utilizes gradient
descent to optimize the fingerprint loss defined in Equation (13).

Algorithm 4 Local Fingerprint Insertion ${\tt LInsert}(\cdot)$

Input: The local fingerprint $F_{i}$ of $i-$th client, the secret key matrix
$A_{i}$, the local fingerprint insertion learning rate $\lambda_{f}$, the
watermarked global model $\hat{M}^{t}_{g}$
Output: The local model $\hat{M}^{t}_{i}$ with the fingerprint.

1:$\hat{M}_{i}^{t}={\tt Copy}(\hat{M}^{t}_{g})$

2:$W_{i}={\tt GetLayerWeight}(\hat{M}_{i}^{t})$

3:$j=0$

4:while $j<{\tt max\\_iter}$ and ${\tt FSS}(A_{i},F_{i},W_{i})\leq\tau_{f}$ do

5: $j=j+1$

6: $g=\frac{\partial{\tt HL}(A_{i},F_{i},W_{i})}{\partial\hat{M}_{g}^{t}}$

7: $W_{i}=W_{i}-\lambda_{f}g$

8:return $\hat{M}^{t}_{i}$.

In FedTracker, we should avoid the situation that the model simultaneously
overfits on two fingerprints, i.e., for one $W$, there are two fingerprints
$F_{\mu},F_{\nu}$ having the same maximal FSS. This can lead to ambiguity and
make the tracing failed. Li et al. [23] has proven that there exists a $W$
containing all the fingerprints if $KN\leq M$. $N$ and $M$ are the dimensions
of $F$ and $W$. Therefore, while inserting the local fingerprints, we should
follow $KN>M$.

FedTracker also utilizes early-stopping technique to prevent the situation.
When the FSS is higher than a threshold $\tau_{f}$, FedTracker will stop
inserting the fingerprint. It can also reduce the influence of inserting
fingerprint on the utility of model.

###  5.4 Verification and Tracing

Once the FL group finds that the FL model is deployed by unauthorized party,
the FL group should conduct the ownership verification and traceability
mechanisms to protect the copyright and catch the culprit. The ownership
verification and traceability mechanisms are demonstrated in Figure 1.

Ownership Verification: The watermark verification algorithm ${\tt
Verify}(\hat{M}_{adv},D_{T})$ confirms the existence of the watermark in the
suspicious model. If the watermark can be extracted from the suspicious model,
the FL group can successfully verify their ownership. The accuracy of the
specific trigger set indicates the existence watermark.

${\tt Verify}(\hat{M}_{adv},D_{T})=\left\\{\begin{aligned} &{\tt True},{\tt
acc}(\hat{M}_{adv},D_{T})\geq\epsilon_{v}\\\ &{\tt False},{\tt if\enspace
otherwise}\end{aligned}\right.,$ (15)

where $\hat{M}_{adv}$ is the model from the adversary. As shown in the middle
part of Figure 1, the FL group uses the watermark token, i.e., the trigger set
to query the API of the suspicious model. Then the FL group checks the output
and calculates the accuracy of the trigger set. As shown in Equation (15), if
the accuracy is greater than the preset threshold $\epsilon_{v}$, the
ownership can be verified.

Traceability: After successfully verifying the ownership, the leaker tracing
algorithm ${\tt Trace}(\hat{M}_{adv},\\{F_{i}\\}_{i=1}^{K})$ extracts the
fingerprint $F_{adv}$ from the suspicious model $\hat{M}_{adv}$, and then
check the most similar fingerprint among the Clients’ fingerprints set
$\\{F_{i}\\}_{i=1}^{K}$. As shown in the right part of Figure 1, the FL group
compares the fingerprint with each client’s fingerprint, and finds the most
similar fingerprint. The traceability mechanism outputs the index of the
malicious client by Equation (16).

${\tt Trace}(\hat{M}_{adv},\\{F_{i}\\}_{i=1}^{K})=\arg\max_{j}{\tt
FSS}(A_{j}W_{adv},F_{j}),$ (16)

where $W_{adv}$ is the weights of the fingerprint embedding layer in the model
$\hat{M}_{adv}$.

##  6 Experiments

###  6.1 Experiment Settings

TABLE II: Watermark accuracy (%) under different numbers of clients. | Number of Clients
---|---
Model | 10 | 20 | 30 | 50
CNN-4 on MNIST | 84.70 | 87.80 | 80.70 | 86.90
AlexNet on CIFAR10 | 98.90 | 96.45 | 86.83 | 82.68
VGG-16 on CIFAR10 | 99.30 | 99.90 | 99.87 | 99.12
ResNet-18 on CIFAR100 | 99.60 | 98.45 | 90.70 | 83.06

Datasets and Models. In our experiments, we use four different Convolution
Neural Networks (CNN) to learn three different datasets including MNIST [49],
CIFAR10 and CIFAR100 [46] . To keep consistency, we resize the images in all
the datasets to $32\times 32$ in our implementation. For MNIST, we utilize a
four-layer CNN (denoted as CNN-4) which is of the LeNet [50] style. It is also
used in [1]. For CIFAR10, two popular models, AlexNet [44] and VGG-16 [45],
are chosen. ResNet-18 [51] is trained on CIFAR100, a dataset with 100 classes.
The first three models are slightly modified by adding a BN layer after each
convolution layer, since it is proven that adding BN layers can achieve better
performance.

FL Settings. To simulate the FL scenario, we set the default number of clients
to $50$. In each iteration, we choose $40\%$ clients to train their local
models for $2$ local epochs. The learning rate of clients is set to $0.01$. As
the default setting, we assume that the distribution of the clients’ data
follows independent identical distribution (i.i.d), which means the training
dataset is uniformly allocated to each client. We also evaluate FedTracker in
the non-i.i.d settings in Section 6.6.

Watermark and Fingerprint. For the global watermark, we implement
WafflePattern [20] to generate the trigger set. We first choose a pattern for
each class and add Gaussian noise to the patterns to generate the trigger set
samples. Each class has $10$ samples in the trigger set. The learning rate is
set to $0.005$ while embedding global watermarks. For local fingerprints, we
set the default length of fingerprints to be $128$, which is sufficient for
tracing $50$ clients. The learning rate is $0.05$ when inserting the client’s
fingerprint.

Metrics. For ownership verification, we use the accuracy of the trigger set,
named watermark accuracy (WM acc for short), as the metric. For traceability,
we define another metric named _Traceability Rate_. Traceability Rate (TR) is
defined as the rate of that the client with the largest FSS happens to be the
holder of the model. In all our experiments, FedTracker achieves $100\%$
traceability rate, which indicates that FedTracker can always find out the
actual owner of the suspicious model.

######  Definition 5 (Traceability Rate)

Traceability Rate (TR) is defined as the rate of that the clients with the
largest FSS is actually the original owner of the model. TR can be computed
using Equation (17).

${\tt TR}=(\sum_{i=1}^{K}[\mathop{\arg\max}\limits_{j}{\tt
FSS}(A_{j},F_{j},W_{i})]=i)/K.$ (17)

###  6.2 Effectiveness on Ownership Verification

In this subsection, we implement FedTracker in different settings to evaluate
the effectiveness on ownership verification.

Ownership Verification under Different Numbers of the Clients. We implement
FedTracker with four scenarios with different numbers of clients in FL,
$10,20,30,$ and $50$. For ownership verification, the results of the average
watermark accuracy are shown in Table II. As the number of clients grows, the
watermark accuracy decreases to some extent. But in all the cases, the
watermark accuracy is still greater than $80\%$, which indicates that the
watermark is successfully embedded into the model.

Ownership Verification under Different Sizes of Watermark Trigger Set. The
size of the watermark trigger set is defined as the number of samples in the
trigger set. In this experiment, we evaluate the ownership verification
effectiveness under different sizes of the watermark trigger set. We generate
$100,200,300,500$ samples to construct the trigger set. The results in Table
III demonstrate that with the size of the trigger set increasing, both the
watermark accuracy and the testing accuracy drops slightly. But the watermark
accuracy is still high enough for ownership verification, which proves that
the watermark retains in the FL models. The utility degradation is also
negligible. It demonstrates that FedTracker is effective despite the size of
the trigger set.

TABLE III: Watermark accuracy (%) under different sizes of watermark trigger set. |  | Size of watermark trigger set
---|---|---
Model | Metric | 100 | 200 | 300 | 500
AlexNet | WM acc | 82.68 | 77.26 | 81.86 | 79.89
Test acc | 84.32 | 84.06 | 84.10 | 84.07
VGG-16 | WM acc | 99.12 | 99.23 | 97.52 | 88.50
Test acc | 88.63 | 88.61 | 88.60 | 88.56

###  6.3 Effectiveness on Traceability

We further evaluate the effectiveness of FedTracker on traceability under
various settings.

Traceability under Different Numbers of Clients. Under the same settings in
Section 6.2, for traceability, we achieve $100\%$ TR in all experiments. The
FSS results are shown in Figure 3. In the experiments, the lowest FSS is
higher than $0.80$, which indicates a successful insertion of local
fingerprints.

Figure 3: FSS evaluation under different numbers of clients. Figure 4: FSS
evaluation under different lengths of local fingerprints. Figure 5: Accuracy
on primitive task of FedTracker comparing with model without watermark (No WM)
and without CL.

Traceability under Different Lengths of Local Fingerprints. In this
experiment, we test the traceability effectiveness of FedTracker by setting
the fingerprint length to $64,128,256,512$ bits. The number of Clients is
fixed at $50$ Clients, which is the most complicated case in our simulation.
We still achieve $100\%$ TR, and the FSS evaluations are shown in Figure 4.
Most of the FSS metrics are around $0.90$. Experimental results demonstrate
that the length of the local fingerprints does not significantly affect the
traceability effectiveness, since the minimal FSS is greater than $0.80$.

###  6.4 Fidelity

In the fidelity experiments, we first compare the utility of the protected
model with the model trained without any watermark embedding and fingerprint
insertion. The utility of the model can be evaluated by the accuracy of the
primitive task, i.e., the accuracy of the validation set. As shown in Figure
5, FedTracker can achieve a similar utility compared with the model without a
watermark. The accuracy degradations in all the experiments are within $1\%$,
which signifies a good fidelity of FedTracker.

We also conduct an ablation study of the CL-based watermark embedding method
in FedTracker. The results are also demonstrated in Figure 5. It shows that
using our CL-based embedding algorithm can improve the utility of the model
with at most $3.21\%$ testing accuracy improvement. In all the experiments,
the utility using CL is better than it without CL, which indicates the
effectiveness of our CL-based algorithm.

Figure 6: FSS against fine-tuning attack. The number behind refers to the
number of Clients in FL.

###  6.5 Robustness

In the robustness experiments, we evaluate the robustness of the global
watermark and local fingerprints against several watermark removal attacks
which are introduced in Section 3.2. Given a specific attack method ${\tt
att}(\cdot)$, the watermark or fingerprints are considered robust if one of
the following two cases is true after the attack:

###### Case 1

This case refers to that the watermark and fingerprints are not significantly
changed by the attack. The ownership verification and traceability mechanisms
can still work properly. It can be defined as Equation (18).

$\displaystyle[{\tt Verify}({\tt att}(\hat{M}_{adv}),D_{T})={\tt True}$ (18)
$\displaystyle\wedge$ $\displaystyle{\tt Trace}({\tt
att}(\hat{M}_{adv}),\\{F_{i}\\}_{i=1}^{K})=adv]$

###### Case 2

This case signifies that the attack has succeeded to remove the watermark and
fingerprints. However, the functionality of the attacked model drops
significantly compared to the primitive model, and is no longer useful for its
task. This case also renders an unsuccessful attack. Equation (19) defines the
case formally.

$\displaystyle[{\tt Verify}({\tt att}(\hat{M}_{adv}),D_{T})={\tt False}$ (19)
$\displaystyle\vee$ $\displaystyle{\tt Trace}({\tt
att}(\hat{M}_{adv}),\\{F_{i}\\}_{i=1}^{K})\neq adv]$ $\displaystyle\wedge$
$\displaystyle{\tt Acc}(\hat{M}_{adv})-{\tt Acc}({\tt
att}(\hat{M}_{adv}))>\tau$

Robustness against Fine-tuning Attack. Fine-tuning attack refers to training
the model with the private dataset of the malicious client for several epochs.
We randomly choose a client to conduct a $30$-epoch fine-tuning. In Figure 6,
the FSS metric rarely changes in the $30$ epochs. While the watermark accuracy
fluctuates as shown in Figure 7 but remains a high value in general, which
indicates the robustness of FedTracker against fine-tuning.

Figure 7: Watermark accuracy (WM) against fine-tuning attack. The number behind refers to the number of Clients in FL. Figure 8: Watermark accuracy (WM) and classification accuracy (ACC) against pruning attack which does not prune the BN layers. Figure 9: FSS, watermark accuracy (WM) and classification accuracy (ACC) against pruning attack which prunes the BN layers. TABLE IV: The accuracy on main task (Test acc), watermark accuracy (WM acc), and FSS against quantization attacks. |  | Data type
---|---|---
Model | Metric(%) | float32 | float16 | int8
AlexNet on CIFAR10 | Test acc | 80.06 | 80.06 | 80.04
WM acc | 84.00 | 84.00 | 84.00
FSS | 87.32 | 87.32 | 87.32
VGG-16 on CIFAR10 | ACC | 88.03 | 88.05 | 88.03
WM | 98.00 | 98.00 | 97.00
FSS | 98.28 | 98.27 | 98.22

Robustness against Quantization Attack. Quantization attack is the most
effective compression technique in practice. Quantization attack refers to
using fewer bits to represent the parameters. In the quantization attack
experiments, we test three different data type, float32, float16 and int8. The
results are shown in Table IV.

The results demonstrate that although quantization reduces the bits of model
parameters, the watermark accuracy and FSS of FedTraker are rarely affected.
The results indicate the robustness of FedTraker against quantization attack.

Robustness against Pruning Attack. We evaluate the robustness of our watermark
and fingerprints against pruning attack, especially parameter pruning [41]. In
parameter pruning, which is also conducted in [9], we prune units in the model
by zeroing out the ones with the lowest $l_{1}$-norm. We divide the pruning
attack into two cases: (1) pruning all the layers of the model except the BN
layers; (2) pruning the BN layers.

In the first case, pruning other layers do not affect the fingerprints in BN
layers. Thus, the watermark is the only one that might be influenced. The
watermark accuracies and classification accuracies after pruning are shown in
Figure 8. As the pruning rate increases, the watermark accuracy and
classification accuracy decrease simultaneously. In most cases, the watermark
accuracy is above $80\%$. The only exception is pruning $50\%$ neurons of
AlexNet. However, the classification accuracy of the pruned model has nearly
$10\%$ decrease, which means the model has lost its value. Thus, we do not
think it is a successful attack.

In the second case, pruning BN layers can affect both watermarks and
fingerprints. From Figure 9, it can be observed that the classification
accuracy drops significantly when pruning the BN layers. When the model still
retains its utility, the watermark accuracy and FSS are still high enough to
provide ownership verification and traceability. In Conclusion, in the two
cases, the pruning attack cannot successfully remove the watermark and
fingerprints in the FL model, which demonstrates the robustness of FedTracker
against the pruning attack.

Robustness against Overwriting Attack. The malicious client might randomly
generate a fingerprint $F_{m}$ and the secret key matrix $A_{m}$, and try to
overwrite the fingerprint in the model. From Figure 10, the FSS metrics drop
after the overwriting attack. However, the TR of the $50$ models is still
$100\%$, which means FedTracker can still trace the leaker successfully. The
result demonstrates that although the overwriting attack has an impact on the
fingerprints, it has little effect on the relative magnitudes of FSS.
FedTracker remains effective against the overwriting attack.

Figure 10: FSS before and after overwriting attack. TABLE V: Testing accuracy, watermark accuracy and FSS (%) before and after Neural Cleanse. |  | Number of Clients
---|---|---
|  | 10 | 20 | 30 | 50
Model | Metric | Before | After | Before | After | Before | After | Before | After
AlexNet on CIFAR10 | Test acc | 85.89 | 74.39 | 83.99 | 79.73 | 82.74 | 72.79 | 80.06 | 74.58
WM acc | 100.00 | 72.00 | 96.00 | 77.00 | 98.00 | 80.00 | 84.00 | 69.00
VGG-16 on CIFAR10 | Test acc | 89.59 | 78.88 | 89.69 | 79.16 | 89.18 | 79.28 | 88.03 | 80.09
WM acc | 99.00 | 79.00 | 100.00 | 70.00 | 100.00 | 74.00 | 98.00 | 72.00
TABLE VI: Testing accuracy, watermark accuracy before and after FeatureRE. |  | Number of Clients
---|---|---
|  | 10 | 20 | 30 | 50
Model | Metric | Before | After | Before | After | Before | After | Before | After
AlexNet on CIFAR10 | Test acc | 85.89 | 70.01 | 83.99 | 71.66 | 82.74 | 77.12 | 80.06 | 73.32
WM acc | 100.00 | 87.00 | 96.00 | 77.00 | 98.00 | 76.00 | 84.00 | 73.00
VGG-16 on CIFAR10 | Test acc | 89.59 | 83.41 | 89.69 | 70.90 | 89.18 | 75.30 | 88.03 | 72.71
WM acc | 99.00 | 87.00 | 100.00 | 84.00 | 100.00 | 88.00 | 98.00 | 77.00

Robustness against Backdoor Mitigation Methods. In the backdoor mitigation
attack, we implement Neural cleanse [42] and FeatureRE [52] trying to remove
the watermark inside the FL model.

Neural cleanse and FeatureRE are two techniques for detecting and removing
backdoors in deep neural networks. They both consist of two main steps:
trigger reverse engineering and trigger unlearning. Trigger reverse
engineering tries to reconstruct the possible triggers that activate the
backdoor in a given model. Trigger unlearning then involves retraining the
model on the reversed trigger data with correct labels. In this way, the model
can forget the association between the trigger and the target class, and
restore its normal behavior.

The experimental results of AlexNet and VGG16 trained on CIFAR10 are
demonstrated in Table V and Table VI. From Table V, the neural cleanse method
has an impact on the watermark of the model and decreases the watermark
accuracy to about $70\%$. However, the testing accuracy also drops about
$10\%$, which signifies a significant negative effect on the model utility and
value. A similar phenomenon is also found in the FeatureRE method. As shown in
Table VI, at the expense of more than $6\%$ testing accuracy degradation, the
watermark accuracy is still high enough for verification in most cases.

Results from neural cleanse and FeatureRE indicate the robustness of the
watermark against backdoor mitigation methods. The ineffectiveness of the two
methods may be ascribed to the principles of backdoor reverse engineering.
Existing backdoor mitigation methods use samples from natural datasets to
reverse and construct a trigger set, which has a relatively small influence on
the trigger set based on noise. A Backdoor based on natural data is more
likely to be triggered in the real-world and thus more harmful than a backdoor
based on noise. It is reasonable for the backdoor mitigation method to mainly
focus on removing the former kind of backdoor.

###  6.6 Effectiveness under Non-i.i.d Settings

We also measure the effectiveness of FedTracker under non-i.i.d settings,
which is a common problem in FL. The non-identically independent distribution
(non-i.i.d) of Clients’ data is a prevalent issue in Federated Learning.
Specifically, the dataset held by the Clients in Federated Learning exhibit
substantial heterogeneity in their distributions, where the labels and the
number of samples in the training set of the Clients are significantly
different. Non-i.i.d data adversely affects the efficacy of federated
training. Following prior works [53], we utilize Dirichlet distribution to
generate non-i.i.d data for clients. The concentration parameter $\xi\in[0,1]$
of Dirichlet distribution controls the degree of non-i.i.d. The smaller the
concentration parameter, the greater the variation between the data
distributions of different clients.

We conduct the experiments with $\xi=0.5,0.7,0.9$. The results are
demonstrated in Figure 11 and Table VII. We can see that as the variation
grows, the watermark accuracy and the testing accuracy drop simultaneously,
but the watermark accuracy is still higher than $79\%$. The non-i.i.d setting
has little effect on the traceability mechanism since the FSS metrics shown in
Figure 11 have negligible change.

Figure 11: FSS evaluation under non-i.i.d settings. TABLE VII: Watermark accuracy (%) under non-i.i.d settings. | Data distribution |
---|---|---
Model | Metric | i.i.d | $\xi=0.9$ | $\xi=0.7$ | $\xi=0.5$
AlexNet | WM acc | 82.68 | 81.90 | 80.24 | 79.50
Test acc | 84.32 | 83.52 | 83.24 | 82.95
VGG-16 | WM acc | 99.12 | 99.14 | 98.40 | 98.88
Test acc | 88.63 | 88.09 | 87.52 | 87.28

###  6.7 Fidelity and Robustness of FedTracker under Non-i.i.d Settings

Section 6.6 presents the effectiveness on ownership verification and
traceability. We further evaluate the fidelity under non-i.i.d settings. All
the experiments are conducted by training the VGG-16 models on CIFAR10
dataset.

TABLE VIII: Testing accuracy of the watermarked model and the models without watermark (No WM) under non-i.i.d settings. Method | i.i.d | $\xi=0.9$ | $\xi=0.7$ | $\xi=0.5$
---|---|---|---|---
FedTracker | 88.63 | 87.44 | 87.32 | 87.28
No WM | 89.04 | 87.85 | 87.53 | 87.33
Difference | -0.41 | -0.24 | -0.21 | -0.05
TABLE IX: Robustness against pruning attack which does not prune the BN layers. |  | Pruning rate
---|---|---
Distribution | Metric | $0.05$ | $0.1$ | $0.15$ | $0.2$ | $0.25$ | $0.3$ | $0.35$ | $0.4$ | $0.45$ | $0.5$
i.i.d | WM acc | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 99.00 | 99.00 | 98.00 | 94.00 | 83.00
Testing acc | 89.69 | 89.7 | 89.59 | 89.58 | 89.40 | 89.16 | 88.45 | 87.83 | 86.35 | 84.42
$\xi=0.9$ | WM acc | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 99.00 | 99.00 | 94.00 | 79.00
Testing acc | 85.46 | 85.38 | 85.37 | 85.63 | 85.66 | 85.88 | 85.63 | 85.17 | 84.34 | 82.32
$\xi=0.7$ | WM acc | 100.00 | 100.00 | 100.00 | 100.00 | 98.00 | 98.00 | 96.00 | 85.00 | 77.00 | 69.00
Testing acc | 85.82 | 85.79 | 85.89 | 85.79 | 85.7 | 85.3 | 85.16 | 84.47 | 82.38 | 81.06
$\xi=0.5$ | WM acc | 84.00 | 84.00 | 84.00 | 81.00 | 77.00 | 78.00 | 78.00 | 77.00 | 71.00 | 52.00
Testing acc | 83.69 | 83.63 | 83.31 | 83.6 | 83.5 | 83.54 | 83.57 | 83.82 | 83.38 | 81.11
TABLE X: Robustness against pruning attack which prunes the BN layers. |  | Pruning rate
---|---|---
Distribution | Metric | $0.05$ | $0.1$ | $0.15$ | $0.2$ | $0.25$ | $0.3$ | $0.35$ | $0.4$ | $0.45$ | $0.5$
i.i.d | WM acc | 100.00 | 89.00 | 46.00 | 22.00 | 10.00 | 11.00 | 11.00 | 7.00 | 10.00 | 10.00
Testing acc | 89.69 | 84.41 | 72.94 | 56.41 | 32.88 | 27.06 | 25.33 | 17.47 | 11.47 | 12.75
FSS | 0.928 | 0.930 | 0.850 | 0.866 | 0.752 | 0.625 | 0.666 | 0.415 | 0.124 | 0.025
$\xi=0.9$ | WM acc | 100.00 | 64.00 | 14.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00
Testing acc | 85.46 | 82.09 | 68.08 | 49.75 | 39.58 | 33.59 | 25.02 | 11.81 | 10.83 | 10.01
FSS | 0.974 | 0.974 | 0.972 | 0.968 | 0.961 | 0.955 | 0.941 | 0.922 | 0.946 | 0.946
$\xi=0.7$ | WM acc | 100.00 | 56.00 | 11.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00
Testing acc | 85.82 | 80.00 | 65.17 | 51.66 | 42.40 | 33.36 | 13.24 | 10.33 | 10.12 | 11.59
FSS | 0.927 | 0.927 | 0.924 | 0.926 | 0.923 | 0.909 | 0.904 | 0.887 | 0.848 | 0.876
$\xi=0.5$ | WM acc | 84.00 | 47.00 | 12.00 | 12.00 | 11.00 | 11.00 | 10.00 | 10.00 | 10.00 | 10.00
Testing acc | 83.69 | 77.60 | 68.75 | 48.77 | 41.55 | 32.05 | 23.65 | 22.65 | 13.44 | 11.25
FSS | 0.870 | 0.871 | 0.869 | 0.850 | 0.850 | 0.832 | 0.819 | 0.817 | 0.772 | 0.766

Fidelity under non-i.i.d settings. The testing accuracy of the watermarked
models and the models without watermark is shown in Table VIII. The testing
accuracy of the FedTracker models drops little comparing with the baseline
models without watermark. In non-i.i.d settings the accuracy degradations are
below $0.25\%$, which signifies that applying FedTracker does not
significantly affect the model utility under non-i.i.d settings.

Robustness against fine-tuning attack under non-i.i.d settings. Similar with
the fine-tuning implementation in Section 6.5, we fine-tune the Client’s model
with their own dataset. The difference is that in non-i.i.d settings, the
Client’s own data of different classes is unbalanced. The experimental results
are shown in Figure 12.

From the left part of Figure 12, fine-tuning attack still cannot affect the
fingerprints embedded into the Batch Normalization layers. The FSS rarely
changes during the fine-tuning, which indicates the great robustness of our
method. While from the right part of Figure 12, we see that fine-tuning has
impact on the global watermark and the impact becomes larger as the variation
increases. When $\xi=0.9$, the watermark accuracy drops to nearly $70\%$. And
in the experiment of $\xi=0.5,0.7$, the lowest watermark accuracy is nearly
$60\%$. It may ascribe to the unbalance data of the Client. Since we simulate
the scenario that the Client itself fine-tunes the model on its own dataset,
training on the unbalanced dataset lead the model to another local optimum.
However, from the experiments, the watermark accuracy above $60\%$ is still
enough for verification.

Figure 12: Watermark Accuracy and FSS Evaluation against Fine-tuning Attack
under Non-i.i.d Settings.

Robustness against Pruning Attack under Non-i.i.d Settings. We also implement
two different categories of pruning attack. One is to prune the parameters of
all the layers except the Batch Normalization layers, the other is to prune
the parameters of the Batch Normalization layers. The results are shown in
Table IX and Table X. When pruning the parameters except those of the BN
layers, both the watermark accuracy and the testing accuracy drops as the
pruning rate increases. However, the watermark in the model trained under the
non-i.i.d settings is more fragile than the model trained under the i.i.d
setting. The watermark accuracy drops to $52\%$ when prunes $50\%$ parameters.
Although the $52\%$ watermark accuracy can also verify the ownership, this
phenomenon indicates that the non-i.i.d settings have impact on the
effectiveness to some extent.

From Table X, pruning the BN layers still cannot remove the watermark and
fingerprint under non-i.i.d settings. Pruning BN layers significantly
compromises the utility of the protected model. The testing accuracy drops
more than $15\%$ which means the model has lost its value. Since it satisfies
the case 2 in Section 6.5, the attack should not be considered as a successful
attack.

Robustness against Overwriting Attack under Non-i.i.d Settings. We also
conduct the overwriting attack to the models trained under non-i.i.d settings
to evaluate if the non-i.i.d settings make the fingerprint fragile. The
results are shown in Figure 13. We still achieve $100\%$ traceability rate
which means FedTracker can still disclose the model leaker against overwriting
attack. Moreover, in all the four experiments, the decrease in FSS is almost
the same. The median values of FSS are all reduced to around $0.7$. This
indicates that the robustness against overwriting attack of the model trained
under non-i.i.d settings does not change.

Figure 13: FSS Evaluation against Overwriting Attack under Non-i.i.d Settings.

To sum up, under non-i.i.d settings, the fidelity and robustness of FedTracker
watermark and fingerprints are not significantly influenced. This proves that
FedTracker is widely applicable to a variety of non-i.i.d scenarios.

###  6.8 Comparing FSS and Hamming Distance

In FedTracker, we propose to use Fingerprint Similarity Score (FSS) to measure
the similarity between different fingerprints. However, existing watermarking
scheme uses Hamming distance (HD) as metric. In this section, we present the
detailed comparison of FSS and HD.

When verifying the ownership of the deep learning model which is trained by
only one party, the model owner only needs to compare the watermark extracted
from the suspicious model with one single watermark. In that case, HD is
enough to measure the similarity. However, in FedTracker, we need to compare
the local fingerprint with each client’s fingerprint. HD rounds the value to a
integer and thus loses some information. For example, the distance $0.7$ and
the distance $1.3$ are both rounded to $1$, making the two fingerprints
indistinguishable.

The experimental results of using FSS and HD as metric are shown in Table XI.
From Table XI, we can find that in most cases, HD works as well as FSS. But in
the cases that the VGG-16 trained with 20 Clients and ResNet-18 trained with
30 Clients, using HD fail to disclose the identity of the model leaker, while
using FSS is always effective. The results demonstrate that utilizing FSS can
better discriminate different fingerprints.

TABLE XI: Traceability rate (%) using FSS and HD. |  | Number of Clients
---|---|---
Model | Metric | 10 | 20 | 30 | 50
CNN-4 | FSS | 100.0 | 100.0 | 100.0 | 100.0
HD | 100.0 | 100.0 | 100.0 | 100.0
AlexNet | FSS | 100.0 | 100.0 | 100.0 | 100.0
HD | 100.0 | 100.0 | 100.0 | 100.0
VGG-16 | FSS | 100.0 | 100.0 | 100.0 | 100.0
HD | 100.0 | _90.0_ | 100.0 | 100.0
ResNet-18 | FSS | 100.0 | 100.0 | 100.0 | 100.0
HD | 100.0 | 100.0 | _23.3_ | 100.0

##  7 Conclusion

In this paper, we propose FedTracker, the first FL model protection framework
to simultaneously furnish ownership verification and traceability. Global
watermark and local fingerprints mechanisms are utilized to verify the
ownership of the FL model and trace the model back to the leaker respectively.
We accordingly design a CL-based watermark embedding mechanism to improve
fidelity and a novel metric named Fingerprint Similarity Score to measure the
similarity of fingerprints. Evaluations of various tasks and settings show
that FedTracker is effective in ownership verification, traceability,
fidelity, and robustness.

## References

  * [1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Proceedings of 2017 International Conference Artificial Intelligence and Statistics_ , 2017, pp. 1273–1282.
  * [2] Y. Liu, A. Huang, Y. Luo, H. Huang, Y. Liu, Y. Chen, L. Feng, T. Chen, H. Yu, and Q. Yang, “Fedvision: An online visual object detection platform powered by federated learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 08, 2020, pp. 13 172–13 179.
  * [3] P. Yu and Y. Liu, “Federated object detection: Optimizing object detection model with federated learning,” in _Proceedings of the 3rd International Conference on Vision, Image and Signal Processing_ , 2019, pp. 1–6.
  * [4] Q. Liu, C. Chen, J. Qin, Q. Dou, and P.-A. Heng, “Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1013–1023.
  * [5] R. Kumar, A. A. Khan, J. Kumar, A. Zakria, N. A. Golilarz, S. Zhang, Y. Ting, C. Zheng, and W. Wang, “Blockchain-federated-learning and deep learning models for covid-19 detection using CT imaging,” _IEEE Sensors Journal_ , 2021.
  * [6] A. Xu, W. Li, P. Guo, D. Yang, H. R. Roth, A. Hatamizadeh, C. Zhao, D. Xu, H. Huang, and Z. Xu, “Closing the generalization gap of cross-silo federated medical image segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022, pp. 20 866–20 875.
  * [7] O. A. Wahab, G. Rjoub, J. Bentahar, and R. Cohen, “Federated against the cold: A trust-based federated learning approach to counter the cold start problem in recommendation systems,” _Information Sciences_ , vol. 601, pp. 189–206, 2022.
  * [8] L. Yang, B. Tan, V. W. Zheng, K. Chen, and Q. Yang, “Federated recommendation systems,” in _Federated Learning_. Springer, 2020, pp. 225–239.
  * [9] Y. Uchida, Y. Nagai, S. Sakazawa, and S. Satoh, “Embedding watermarks into deep neural networks,” in _Proceedings of 2017 ACM International Conference on Multimedia Retrieval_ , 2017, pp. 269–277.
  * [10] Y. Adi, C. Baum, M. Cisse, B. Pinkas, and J. Keshet, “Turning your weakness into a strength: Watermarking deep neural networks by backdooring,” in _Proceedings of 2018 USENIX Security Symposium_ , 2018, pp. 1615–1631.
  * [11] L. Fan, K. W. Ng, and C. S. Chan, “Rethinking deep neural network ownership verification: embedding passports to defeat ambiguity attacks,” in _Proceedings of 2019 International Conference on Neural Information Processing Systems_ , 2019, pp. 4714–4723.
  * [12] D. S. Ong, C. Seng Chan, K. W. Ng, L. Fan, and Q. Yang, “Protecting Intellectual Property of Generative Adversarial Networks from Ambiguity Attacks,” in _Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. Nashville, TN, USA: IEEE, Jun. 2021, pp. 3629–3638.
  * [13] A. Bansal, P.-y. Chiang, M. Curry, R. Jain, C. Wigington, V. Manjunatha, J. P. Dickerson, and T. Goldstein, “Certified Neural Network Watermarks with Randomized Smoothing,” in _Proceedings of the 2022 International Conference on Machine Learning_. PMLR, Jul. 2022, pp. 1450–1465.
  * [14] J. H. Lim, C. S. Chan, K. W. Ng, L. Fan, and Q. Yang, “Protect, show, attend and tell: Empowering image captioning models with ownership protection,” _Pattern Recognition_ , vol. 122, p. 108285, Feb. 2022.
  * [15] B. Darvish Rouhani, H. Chen, and F. Koushanfar, “Deepsigns: An end-to-end watermarking framework for ownership protection of deep neural networks,” in _Proceedings of 2019 International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2019, pp. 485–497.
  * [16] G. Li, S. Li, Z. Qian, and X. Zhang, “Encryption resistant deep neural network watermarking,” in _Proceedings of 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 3064–3068.
  * [17] Z. Li, C. Hu, Y. Zhang, and S. Guo, “How to prove your model belongs to you: A blind-watermark based framework to protect intellectual property of DNN,” in _Proceedings of 2019 Annual Computer Security Applications Conference_ , 2019, pp. 126–137.
  * [18] Y. Lao, P. Yang, W. Zhao, and P. Li, “Identification for Deep Neural Network: Simply Adjusting Few Weights!” in _Proceedings of 2022 IEEE 38th International Conference on Data Engineering (ICDE)_. Kuala Lumpur, Malaysia: IEEE, May 2022, pp. 1328–1341.
  * [19] T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” _arXiv preprint arXiv:1708.06733_ , 2017.
  * [20] B. G. Tekgul, Y. Xia, S. Marchal, and N. Asokan, “WAFFLE: Watermarking in federated learning,” in _Proceedings of 2021 International Symposium on Reliable Distributed Systems_ , 2021, pp. 310–320.
  * [21] X. Liu, S. Shao, Y. Yang, K. Wu, W. Yang, and H. Fang, “Secure Federated Learning Model Verification: A Client-side Backdoor Triggered Watermarking Scheme,” in _Proceedings of 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_. Melbourne, Australia: IEEE, Oct. 2021, pp. 2414–2419.
  * [22] W. Yang, S. Shao, Y. Yang, X. Liu, X. Liu, Z. Xia, G. Schaefer, and H. Fang, “Watermarking in secure federated learning: A verification framework based on client-side backdooring,” _arXiv preprint arXiv:2211.07138_ , 2022.
  * [23] B. Li, L. Fan, H. Gu, J. Li, and Q. Yang, “FedIPR: Ownership Verification for Federated Deep Neural Network Models,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–16, Mar. 2022\.
  * [24] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska _et al._ , “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the national academy of sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [25] D. Lopez-Paz and M. Ranzato, “Gradient Episodic Memory for Continual Learning,” in _Proceedings of the 31st Conference on Neural Information Processing Systems_ , 2017.
  * [26] Q. Yang, Y. Liu, Y. Cheng, Y. Kang, T. Chen, and H. Yu, “Federated learning,” _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , vol. 13, no. 3, pp. 1–207, 2019.
  * [27] H. Fereidooni, S. Marchal, M. Miettinen, A. Mirhoseini, H. Möllering, T. D. Nguyen, P. Rieger, A.-R. Sadeghi, T. Schneider, H. Yalame _et al._ , “Safelearn: secure aggregation for private federated learning,” in _2021 IEEE Security and Privacy Workshops (SPW)_. IEEE, 2021, pp. 56–62.
  * [28] F. Regazzoni, P. Palmieri, F. Smailbegovic, R. Cammarota, and I. Polian, “Protecting artificial intelligence IPs: A survey of watermarking and fingerprinting for machine learning,” _CAAI Transactions on Intelligence Technology_ , vol. 6, no. 2, pp. 180–191, 2021.
  * [29] M. Xue, Y. Zhang, J. Wang, and W. Liu, “Intellectual Property Protection for Deep Learning Models: Taxonomy, Methods, Attacks, and Evaluations,” _IEEE Transactions on Artificial Intelligence_ , pp. 1–1, 2021.
  * [30] X. Qi, T. Xie, R. Pan, J. Zhu, Y. Yang, and K. Bu, “Towards practical deployment-stage backdoor attack on deep neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 13 347–13 357.
  * [31] Y. Li, Y. Jiang, Z. Li, and S.-T. Xia, “Backdoor learning: A survey,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [32] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, “How to backdoor federated learning,” in _Proceedings of 2020 International Conference on Artificial Intelligence and Statistics_ , 2020, pp. 2938–2948.
  * [33] P. Rieger, T. D. Nguyen, M. Miettinen, and A.-R. Sadeghi, “DeepSight: Mitigating Backdoor Attacks in Federated Learning Through Deep Model Inspection,” in _Proceedings of 2022 Network and Distributed System Security Symposium_ , Jan. 2022.
  * [34] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 7, pp. 3366–3385, 2021.
  * [35] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” _Neural networks_ , vol. 113, pp. 54–71, 2019.
  * [36] M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer, “Class-incremental learning: survey and performance evaluation on image classification,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [37] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in _Proceedings of 2017 International conference on machine learning_. PMLR, 2017, pp. 3987–3995.
  * [38] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 12, pp. 2935–2947, 2017\.
  * [39] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2001–2010.
  * [40] A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, “The loss surfaces of multilayer networks,” in _Proceedings of Artificial intelligence and statistics_. PMLR, 2015, pp. 192–204.
  * [41] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in _Advances in Neural Information Processing Systems_ , vol. 28, 2015.
  * [42] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao, “Neural cleanse: Identifying and mitigating backdoor attacks in neural networks,” in _Proceedings of 2019 IEEE Symposium on Security and Privacy (SP)_ , 2019, pp. 707–723.
  * [43] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in _Proceedings of 2017 ACM SIGSAC Conference on Computer and Communications Security_ , 2017, pp. 1175–1191.
  * [44] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Communications of the ACM_ , vol. 60, no. 6, pp. 84–90, 2017.
  * [45] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [46] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” _Tech. Rep._ , 2009.
  * [47] A. Lambora, K. Gupta, and K. Chopra, “Genetic algorithm-a literature review,” in _Proceedings of 2019 international conference on machine learning, big data, cloud and parallel computing (COMITCon)_. IEEE, 2019, pp. 380–384.
  * [48] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of 2015 International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [49] L. Y, C. C, and B. C, “MNIST handwritten digit database,” 2010. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [50] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in _Proceedings of the IEEE_ , vol. 86, no. 11, 1998, pp. 2278–2324.
  * [51] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [52] Z. Wang, K. Mei, H. Ding, J. Zhai, and S. Ma, “Rethinking the reverse-engineering of trojan triggers,” in _Advances in Neural Information Processing Systems_ , 2022.
  * [53] M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni, “Bayesian nonparametric federated learning of neural networks,” in _Proceedings of 2019 International Conference on Machine Learning_ , 2019, pp. 7252–7261.