# Gradient-Leakage Resilient Federated Learning

Wenqi Wei2, Ling Liu2, Yanzhao Wu2, Gong Su1, Arun Iyengar1 2 Georgia
Institute of Technology, School of CS, Atlanta, GA 30332, USA
1 IBM T. J. Watson Research Center, Yorktown Heights, NY 10598, USA

###### Abstract

Federated learning(FL) is an emerging distributed learning paradigm with
default client privacy because clients can keep sensitive data on their
devices and only share local training parameter updates with the federated
server. However, recent studies reveal that gradient leakages in FL may
compromise the privacy of client training data. This paper presents a gradient
leakage resilient approach to privacy-preserving federated learning with per
training example-based client differential privacy, coined as Fed-CDP. It
makes three original contributions. First, we identify three types of client
gradient leakage threats in federated learning even with encrypted client-
server communications. We articulate when and why the conventional server
coordinated differential privacy approach, coined as Fed-SDP, is insufficient
to protect the privacy of the training data. Second, we introduce Fed-CDP, the
per example-based client differential privacy algorithm, and provide a formal
analysis of Fed-CDP with the $(\epsilon,\delta)$ differential privacy
guarantee, and a formal comparison between Fed-CDP and Fed-SDP in terms of
privacy accounting. Third, we formally analyze the privacy-utility trade-off
for providing differential privacy guarantee by Fed-CDP and present a dynamic
decay noise-injection policy to further improve the accuracy and resiliency of
Fed-CDP. We evaluate and compare Fed-CDP and Fed-CDP(decay) with Fed-SDP in
terms of differential privacy guarantee and gradient leakage resilience over
five benchmark datasets. The results show that the Fed-CDP approach
outperforms conventional Fed-SDP in terms of resilience to client gradient
leakages while offering competitive accuracy performance in federated
learning.

###### Index Terms:

Distributed systems, federated learning, differential privacy, gradient
leakage, attack resilience.

##  I Introduction

Federated learning (FL) decouples the learning task from the centralized
server to a distributed collection of client nodes [1, 2, 3, 4, 5]. It
collaboratively learns a shared global model over the distributed collection
of training data residing on end-user devices of the participating clients.
Each client maintains its private data locally by downloading the global model
update from the server at each round of the federated learning, performing
local training, and sharing its local model update via encrypted communication
to the server.

However, the distributed nature and the default privacy in federated learning
are insufficient for protecting client training data from gradient leakage
attacks. Recent studies [6, 7, 8] reveal that if an adversary intercepts the
local gradient update of a client before the server performs the federated
aggregation to generate the global parameter update for the next round of
federated learning, the adversary can steal the sensitive local training data
of this client using the leaked gradients by simply performing reconstruction
attack [8], or membership and attribute inference attacks [9, 10, 11].

To circumvent such gradient leakage threats, we present Fed-CDP, the per
example-based client differential privacy approach with three original
contributions. First, we categorize client gradient leakage threats into three
types based on whether the gradient information was leaked at the server, the
client, or the training example level. We analyze when and why the
conventional server coordinated differential privacy approach [12, 13], coined
as Fed-SDP in this paper, is insufficient to protect the privacy of client
training data in the presence of instance-level gradient leakage attacks.
Second, we introduce Fed-CDP to provide the formal instance-level and client-
level differential privacy guarantee. In contrast to Fed-SDP, which adds noise
to the shared gradient update by a client at each round of the FL joint
training, Fed-CDP adds noise to per example at each iteration per client. The
differential privacy spending is computed by the privacy accounting using the
moments accountant [14] for both Fed-SDP and Fed-CDP. Finally, we provide a
formal analysis of the privacy-utility trade-off for providing differential
privacy guarantee by Fed-CDP, with the maximal distortion bound without
deteriorating the performance of federated learning. Guided by this formal
trade-off analysis, we optimize Fed-CDP by enabling a dynamic decay noise-
injection policy, which boosts both the accuracy performance and the
resilience against gradient leakage attacks. Evaluated on five benchmark
datasets, we show that per-example based Fed-CDP and Fed-CDP (decay)
outperforms per client based Fed-SDP in terms of accuracy performance,
differential privacy guarantee, and gradient leakage resilience.

(a) Attack schema

(b) Attack illustration.

Figure 1: Gradient leakage attacks in Federated Learning. Type-0&1 leakage
attack by batched update and type-2 attacks by example

##  II Related Work

Gradient leakage threats refer to unauthorized disclosure of private client
training data based on the leaked gradient information and are one of the
dominating privacy leakage threats in both centralized and distributed deep
learning systems. In centralized deep learning, membership inference [11, 15,
10, 9] is known to expose the private training data by exploiting the
statistical differences between the model prediction on the data possibly seen
during training and the prediction on unseen data. [16] performed minor
modifications to training algorithms to have them memorize a significant
amount of information about the training data. [17] inferred private data
based on the proportionality between the training data and the gradient
updates but fails to be generalized to convolutional neural networks due to
the relative size of convolution weights and data features. [18] proposed the
model inversion attack to expose privately trained model parameters by
exploiting confidence values revealed from legitimate access to the prediction
API. GAN-based reconstruction attack [19, 20] utilizes generative adversarial
networks to learn private training data based on the gradient information and
the confidence of prediction results. In distributed federated learning, [7,
6, 8] showed the vulnerabilities of client gradient leakage attacks. [8]
showed that such gradient-based client privacy leakage (CPL) attacks display
high reconstruction success rate with high attack effect and low attack cost.

Multiple attempts have been made to offer privacy-preserving federated
learning. Cryptographic approaches [21] and secure multi-party computation
(SMP) [22, 23] ensure secure communication of shared parameter updates between
a client and its federated server. However, it does not secure the client data
prior to encryption for transport or after decryption for the server
aggregation. [24] proposed PATE, a teacher-student ensemble, which relies on a
trusted server for private aggregation, but the involved third party can bring
new risk to the FL system. [25] proposed a distributed learning with selective
and random sharing of model parameters, offering the shared model parameter
level privacy. In recent proposals for differentially private federated
learning [12, 13], the noise is added to the per-client shared gradient
updates at each round of FL, the effect and cost of such differential privacy
guarantee are the same regardless of whether the clipping and the noise
injection are done at the server or at each client. In this paper, we coin
this conventional approach as Fed-SDP. We will show that both the selective
parameter sharing approach and Fed-SDP approach fail to safeguard the client’s
private training data in the presence of instance-level gradient leakage
attacks.

##  III Gradient Leakage Threats

Threat Model. Our threat model adopts the two common assumptions: (i) the
message communicated between a client and its FL server is encrypted; and (ii)
the clients faithfully follow the rules of joint training when subscribed to a
federated learning task, although the server may conduct unauthorized
inference on per-client local model updates either curiously or maliciously
and a client may read the local training result during or upon completion of
the local training at each round of federated learning. We categorize the
gradient leakage threats into three types based on the location and whether
the gradient leakage is performed on per-example gradient update during local
training or per client gradient update after completing the local training at
each round of FL. The type-0 refers to the gradient leakage attacks performed
at the server by intercepting per-client shared gradient updates. The type-1
refers to the gradient leakage attacks performed at a client on its gradient
updates resulting from the completion of the local training. The third type of
gradient leakage attacks are performed on each training example during the
local training at a client and coined as the type-2 leakage. We consider the
gradient leakage attack that uses gradient-based reconstruction attack methods
[6, 7, 8] to derive and steal the private client training data. Although all
three types of gradient leakages are inferred by adversarial reconstruction,
the first two types of leakages are launched by an unauthorized read of per-
client gradient update resulting from the completion of the local training
over multiple iterations at a client. The type-2 leakages are launched by an
unauthorized read of the per-example gradient updates during the local
training on a client.

Figure 1 gives a sketch of the gradient leakage attack algorithm, which
configures and executes the reconstruction attack in five steps: (1) It
configures the initialization seed ($x^{0}_{rec}(t)$), a dummy data of the
same resolution (or attribute structure for text) as the training data. [8]
showed some significant impact of different initialization seeds on the attack
success rate and attack cost ($\\#$ attack iterations to succeed). (2) The
dummy attack seed is fed into the client local model. (3) The gradient of the
dummy attack seed is obtained by backpropagation. (4) The gradient loss is
computed using a vector distance loss function, e.g., $L_{2}$, between the
gradient of the attack seed and the actual gradient from the client local
training. The choice of this reconstruction loss function is another tunable
attack parameter. (5) The dummy attack seed is modified by the attack
reconstruction learning algorithm. It aims to minimize the vector distance
loss by a loss optimizer such that the gradients of the reconstructed seed
$x^{i}_{rec}(t)$ at round $i$ will be closer to the actual gradient updates
stolen from the client upon the completion (type 0&1) or during the local
training (type 2). This attack reconstruction learning iterates until it
reaches the attack termination condition ($\mathbb{T}$), typically defined by
the $\\#$rounds, e.g., 300 (also a configurable attack parameter). If the
reconstruction loss is smaller than the specified distance threshold then the
reconstruction attack is successful. Figure 1 provides a visualization by
examples from three datasets. The type-0&1 gradient leakage attack is
performed on the batched gradients with the batch size of 3 from LFW dataset
[26]. The type-2 leakage is performed on per example gradient during local
train with three examples from MNIST [27], LFW and CIFAR10 [28] respectively.
Although all three types of gradient leakage attacks succeed at the 50th
attack iteration for these examples with $\mathbb{T}$ set to 300, type 2
leakage attacks can succeed with smaller $\\#$iterations and higher attack
quality for the same $\\#$iterations (e.g., 100). All the experiments on
gradient leakages in this paper use the patterned random seed
initialization111https://github.com/git-disl/ESORICS20-CPL, with a $L_{2}$
based loss function and L-BFGS optimizer, for high attack success rate (ASR)
and fast attack convergence.

##  IV Solution Approach

We first define the core differential privacy concepts following the state-of-
the-art literature. Then we introduce the reference model for federated
learning (both non-private and with different privacy) and present the problem
statement, followed by our proposed Fed-CDP, the per-example based client
differential privacy approach to privacy-preserving federated learning,
including algorithm, privacy parameters, and configuration trade-offs.

###  IV-A Preliminary Concepts and Reference Model

######  Definition 1.

Differential privacy [29]: a randomized mechanism $\mathcal{M}$:
$\mathcal{D}\rightarrow\mathcal{R}$, with a domain $\mathcal{D}$ of possible
training data and range $\mathcal{R}$ of all possible trained models,
satisfies ($\epsilon,\delta$)-differential privacy if for any two input sets
$A\in\mathcal{D}$ and $A^{\prime}\in\mathcal{D}$, differing with only a single
example input and for any subset of outputs $B\in\mathcal{R}$, Equation 1
holds.

$\Pr(\mathcal{M}(A)\in\mathcal{R})\leq
e^{\epsilon}\Pr(\mathcal{M}(A^{\prime})\in\mathcal{R})+\delta.$ (1)

######  Definition 2.

Gaussian mechanism [29]: when $0<\epsilon<1$, applying Gaussian noise
$\mathcal{N}(0,\sigma^{2}S^{2})$ calibrated to a real valued function:
$f:\mathcal{D}\rightarrow\mathcal{R}$ such that
$\mathcal{M}(A)=f(A)+\mathcal{N}(0,\sigma^{\prime 2})$, $\mathcal{M}(A)$ is
$(\epsilon,\delta)$-differentially private if $\sigma{{}^{\prime
2}}>\frac{{2\log(1.25/\delta)\cdot{S^{2}}}}{{{\varepsilon^{2}}}}$, where for
any two adjacent inputs $A,A^{\prime}\in D$, $||A-A^{\prime}||_{1}=1$, and
$S=\max\nolimits_{A,A^{\prime}\in D}||f(A)-f(A^{\prime})||_{2}$, i.e., the
maximum $l_{2}$ distance between $f(A)$ and $f(A^{\prime})$.

With Definition 2, it is straightforward to get Lemma 1.

######  Lemma 1.

Let $\sigma^{\prime 2}$ in Gaussian mechanism be $\sigma^{2}S^{2}$. We have
the noise scale $\sigma$ satisfying
$\sigma^{2}>\frac{{2\log(1.25/\delta)}}{{{\varepsilon^{2}}}}.$

######  Definition 3.

Privacy amplification [30]: Given dataset $\mathcal{D}$ with $N$ data points,
define subsampling as random sampling with replacement and sampling rate as
$q=n/N$ where $n$ is the sample size. If $\mathcal{M}$ is
$(\epsilon,\delta)$-differentially private, then the subsampled mechanism with
sampling rate $q$ is $(\log(1+q(\exp(\epsilon)-1)),q\delta)$-differentially
private.

######  Definition 4.

Composition theorem [29]: Given a fixed $\sigma$ and $\epsilon$, then a set of
consecutive inquiries of ($\epsilon,\delta$) differentially private algorithms
is ($\sum\epsilon,\sum\delta$) differentially private.

######  Definition 5.

Moments accountant [14]: For a mechanism $\mathcal{M}$, given dataset
$\mathcal{D}$ with a total of $N$ data points, given the random sampling with
replacement and the sampling rate $q=n/N$ where $n$ is the sample size, and
the number of iterations $T$, if $q<\frac{1}{16\sigma}$, there exist constants
$c1$ and $c2$ so that for any $\epsilon<c1q^{2}T$, $\mathcal{M}$ is
$(\epsilon,\delta)$-differentially private for $\delta>0$ and

$\epsilon\geq c2\frac{q\sqrt{T\log(1/\delta)}}{\sigma}$ (2)

.

Moments accountant is
implementated222https://github.com/tensorflow/privacy/blob/master/tensorflow_privacy/privacy/analysis/compute_dp_sgd_privacy.py
in a way that privacy spending $\epsilon$ is computed when $T$, $\sigma$,
$\delta$, and $q$ are given.

Reference model for federated learning. Federated learning can be viewed as a
publish-subscribe model, in which the server publishes its federated learning
task, described by the data structure and data modality and the neural network
model (algorithm) for joint training, including the important hyperparameters:
total #rounds $T$ of joint training, local batch size $B$, learning rate
$\eta$, local # iterations $L$, format for messaging between client and
server, and the public key of the server for encryption. By subscribing to a
published FL task, the clients $i$ informs the server its local training
dataset size $N_{i}$ and commits to download the joint training model with the
default hyperparameters, then faithfully follow the FL protocol: at each round
$t\leq T$, a client downloads the current global model parameters $W(t)$ from
the server, performs local model training on the private client training data
over $L$ local iterations, i.e.,
$W_{i}(t)_{l+1}=W_{i}(t)_{l}-\eta\nabla{W_{i}(t)_{l}},1<l<L-1$ where
$\nabla{W}$ is the gradient of the trainable network parameters and $1\leq
L\leq\lceil N_{i}/B\rceil$. Upon completion of the local model training at
round $t$, this client $i$ sends the local training parameter updates to the
server.

Upon receiving local training parameter updates from $K_{t}$ out of a total of
$K$ clients ($K_{t}<K$), i.e., $\Delta W_{i}(t)=W_{i}(t)_{L}-W_{i}(t)$
($i=1\dots K_{t}$), the server aggregates the local updates from all $K_{t}$
clients to produce the global model parameter updates and start the next round
$t+1\leq T$ of joint training until $T$ rounds are reached. As stated in [2],
to deal with the unstable client availability, only a small subset of $K_{t}$
clients are selected ($K_{t}<K$) to participate in each round $t$ of federated
learning. Two aggregation methods are commonly used: FedSGD:
$W(t+1)=W(t)+\frac{1}{K_{t}}\sum\nolimits_{k=1}^{{K_{t}}}\Delta W_{k}(t)$ if
the clients share the local model parameter updates [31, 32, 33]; and
FedAveraging:
$W(t+1)=\sum\nolimits_{k=1}^{{K_{t}}}\frac{1}{K_{t}}{W_{k}}(t)_{L}$ if the
clients send the locally updated model to the server [2, 34, 35]. With a large
client population ($K$) and a small subset $K_{t}<<K$ of contributing clients
per round, federated learning can produce a jointly trained model with high
accuracy [2, 3]. Given that the FedSGD and Fedaveraging are mathematically
equivalent, we consider FedSGD aggregation in this paper without loss of
generality. For privacy-preserving federated learning, the server will provide
the differentially private version of the federated model instead of the non-
private version for clients to download, together with the set of differential
privacy parameters.

###  IV-B Problem Statement

Recent proposals for differentially private federated learning, represented by
[12, 13] (as illustrated in Algorithm 1), add Gaussian noise to the per-client
shared gradient updates at server each round $t\leq T$. They differ mainly in
whether the clipping operation is performed at the server or at each client
upon the completion of local training at round $t$. We coin these approaches
as Fed-SDP in this paper since they inject noise to the per-client gradient
updates at each round $t$. The effect and cost of these clipping and per-
client noise injection, whether is done at the server or per client, deliver
the same client-level differential privacy guarantee [12].

Input: $N_{i}$: # local data, $D_{i}$: local data, $K_{t}$: # clients per
round, $K$: # total clients, $T$: maximum global round, $\sigma$: noise scale,
$C$: clipping bound.

1 Server initialization and broadcast: global model $W(0)$ of $M$ layers,
local batch size $B$, # local iteration $1\leq L\leq\lceil N_{i}/B\rceil$,
local learning rate $\eta$.

2 for  _round $t$ in $\\{0,1,2...T-1\\}$_ do

3

4 client $i$ does non-private local training and send out local updates:
$\Delta W_{i}(t)\leftarrow W_{i}(t)_{L}-W(t)$.

5

6 server do

// collect updates from $K_{t}$ clients

7 $\Delta W_{k}(t),k=1,...K_{t}$

8 for  _$k$ in 1,…,$K_{t}$_ do

9 for  _$m$ in $1,...M$ _ do

// compute $L2$ norm for layer $m$

10 $||\Delta W_{ij}(t)_{lm}||_{2}$

// clip per-client update for layer $m$
$\displaystyle\vspace{-0.5cm}\overline{\Delta}W_{k}(t)_{m}$
$\displaystyle\leftarrow=\Delta W_{k}(t)_{m}/\max(1,\frac{||\nabla
W_{k}(t)_{m}||_{2}}{C})\vspace{-0.5cm}$

11

12 end for

// obtain the clipped per-client updates

13
$\overline{\Delta}W_{k}(t)\leftarrow\\{\overline{\Delta}W_{k}(t)_{m}\\},m=1,..,M$

14 end for

// compute sanitized updates with $S\leftarrow C$

15
$\widetilde{\Delta}W(t)\leftarrow\frac{1}{K_{t}}(\sum\nolimits_{k=1}^{{K_{t}}}(\overline{\Delta}W_{k}(t)+\mathcal{N}(0,{\sigma^{2}S^{2}}))$

// aggregation:

16 $W(t+1)\leftarrow
W(t)+\frac{1}{K_{t}}\sum\nolimits_{k=1}^{{K_{t}}}\widetilde{\Delta}w_{k}(t)$

17

18 end for

Output: global model $W(T)$

Algorithm 1 Fed-SDP

Recall the threat model in Section III, the Fed-SDP approach is resilient to
both type 0&1 gradient leakage attacks if differential privacy is performed by
adding Gaussian noise at each client and the adversary intercepts the noisy
per-client local parameter updates. The noisy per-client gradient updates make
the leakage attack much harder to succeed due to poor reconstruction
inference, especially as the number of rounds increases in federated learning.
Of course, for type-1 gradient leakage, if the Gaussian noise is added at the
server but the type-1 leakage is performed on the true local gradient updates
at a client, then this version of the Fed-SDP is only resilient to type-0
leakage and cannot protect against type-1 leakage. However, existing versions
of Fed-SDP are vulnerable against type-2 leakages which steal per-example
gradients during local training, because simply adding Gaussian noise to the
per-client local parameter update upon completion of the local training at
each round $t$ will fail miserably against the per-example type-2 leakages.
This motivates the design of our Fed-CDP to provide a per-example based client
differential privacy guarantee in privacy-preserving federated learning.

Figure 2: System schema of Fed-CDP

Input: $N_{i}$: # local data, $D_{i}$: local data, $K_{t}$: # clients per
round, $K$: # total clients, $T$: maximum global round, $\sigma$: noise scale,
$C$: clipping bound

1 Server initialization and broadcast: global model $W(0)$ of $M$ layers,
local batch size $B$, # local iteration $1\leq L\leq\lceil N_{i}/B\rceil$,
local learning rate $\eta$.

2 for  _round $t$ in $\\{0,1,2...T-1\\}$_ do

3

4 client $i$ do

// download current global model:

5 $W_{i}(t)_{0}\leftarrow W(t)$

6 for  _local iteration $l$ in $\\{0,1,2...L-1\\}$_ do

// batch processing, take subset of $D_{i}$

7 for  _$j$ in $\\{1,...B\\}$ _ do

8 for  _$m$ in $\\{1,...M\\}$ _ do

// compute per example gradient for layer $m$

9 $\nabla W_{ij}(t)_{l}\leftarrow\\{\nabla W_{ij}(t)_{lm}\\}$

// compute $L2$ norm for layer $m$

10 $||\nabla W_{ij}(t)_{lm}||_{2}$

// clip per-example gradients for layer $m$

11 $\overline{\nabla}W_{ij}(t)_{lm}\leftarrow\nabla
W_{ij}(t)_{lm}/\max(1,\frac{||\nabla W_{ij}(t)_{lm}||_{2}}{C})$

12 end for

// obtain the clipped per-example gradients

13
$\overline{\nabla}W_{ij}(t)_{l}\leftarrow\\{\overline{\nabla}W_{ij}(t)_{lm}\\},m=1,..,M$

14 end for

// compute sanitized batch gradients with $S\leftarrow C$

15
$\widetilde{\nabla}W_{i}(t)_{l}\leftarrow\frac{1}{B}\sum\nolimits_{j=1}^{B}(\overline{\nabla}W_{ij}(t)_{l}+\mathcal{N}(0,\sigma^{2}S^{2}))$

// gradient descent:

16 $W_{i}(t)_{l+1}\leftarrow W_{i}(t)_{l}-\eta\widetilde{\nabla}w_{i}(t)_{l}$

17

18 end for

// send out local updates:

19 $\Delta W_{i}(t)\leftarrow W_{i}(t)_{L}-W(t)$ // with $M$ layers

20 server do

// collect local updates from $K_{t}$ clients

21 $\Delta W_{i}(t),i=1,...K_{t}$

// aggregation:

22 $W(t+1)\leftarrow W(t)+\frac{1}{K_{t}}\sum\nolimits_{i=1}^{{K_{t}}}\Delta
w_{i}(t)$

23 end for

Output: global model $W(T)$

Algorithm 2 Fed-CDP

###  IV-C Fed-CDP: Per-example Client Differential Privacy

Figure 2 gives a system overview of Fed-CDP. Fed-CDP provides gradient leakage
resilience against all three types of attacks by adding DP-guided Gaussian
noise to the per-example gradients once they are computed before local batch
averaging in each of the $L$ iterations during the local training at a client.
This enables the support for instance-level differential privacy guarantee.
Algorithm 2 gives the pseudo-code for Fed-CDP. We first discuss a set of
important differential privacy parameters used in Fed-CDP.

Random Subsampling. Given a total of $K$ participants (clients) and a total of
T rounds for distributed learning, in each round t ($1\leq t\leq T$), a subset
of $K_{t}$ clients is randomly sampled with replacement under probability $q$.
The random sampling by selecting each client independently with probability
$q$ is essential for providing low privacy loss through tracking privacy
spending using the moments accountant [14].

Choosing Clipping Strategy $C$. When clipping the client gradient updates, a
tradeoff should be made. To keep the noise variance small, we should choose a
small clipping upper bound $C$. However, if the goal is to maintain the
original update values as much as possible, a too-small clipping bound $C$
should not be chosen. Instead, we can use the median norm of all original
updates from the training set of the client $i$ and use this median as the
clipping bound, i.e., $C\leq$ median${}_{i\in N_{i}}\nabla w_{i}(t)$.
Alternatively, we can also define clipping as a function of learning rate
$\eta$. With adaptive learning rate methods, in which the learning rate is
systematically decreasing as the model converges [36].

Clipping Bound $C$ and Sensitivity $S$. To provide a fair comparison with Fed-
SDP, in this paper we follow [14]: which uses a preset fixed clipping bound
$C$ to confine the influence of the local model update under $L2$ norm and
also use the clipping bound $C$ to estimate the sensitivity for adding noise
in Gaussian Mechanism (recall Definition 2). A trade-off should be considered
when clipping the contribution of the per-example gradient, as the clipping
bound can affect both the noise variance and the informative contribution of
the gradient. A large clipping bound would significantly enlarge the Gaussian
noise variance and thus pushing the gradient away from its training goal and
may thus downgrade or even fail the training, since the large gradient
reflects an aggressive step of modification on the model weight for gradient
descent. On the other hand, a small clipping bound may keep noise variance
stay small but may end up pruning those informative gradients. Empirically,
following the literature [14], $C=4$ is used as the default unless otherwise
stated.

Choosing noise scale $\sigma$. By Definition 2, $\sigma$ controls the noise
level and by moment accounting in Definition 5 (Equation 2), with fixed
$\delta$, the privacy loss accumulates inverse-proportionally as noise scale
$\sigma$ changes. Existing literature selects the noise scale based on the
learning method and task. For example, [13] uses a small noise scale
$\sigma=1$ with clipping bound $C=1$ for client-level differential privacy and
[12] uses $\sigma$ as small as 0.012 with clipping bound $C=15$ tailor to the
local model update in the federated language model. [14] sets its default
$\sigma$ to 6 with clipping $C=4$ for privacy accounting via moments
accountant. [37] considered a large $\sigma$ of $10$ with clipping $C=4$ for
centralized differentially private model publishing, in which the batch
gradient is less sensitive to noise. In this paper, we adopt the same setting
as [14]: $\delta=1e-5$, $\sigma=6$ with $C=4$ as the default unless otherwise
stated.

Workflow of Fed-CDP algorithm. In each round $t\leq T$ of federated learning,
for each client $i$ from the $K_{t}$ chosen clients, the local model training
is performed on each iteration $l$ over the $L$ local iterations ($1\leq
L\leq\lceil N_{i}/B\rceil$). For each iteration $l$, we compute the per-
example gradient layer by layer over the $M$ layers for each instance $j$ in a
batch $B$ by minimizing the empirical loss function (Lines 6-9). For each
layer $l$, the number of gradients is the same as the number of trainable
parameters and there exists an $L2$-norm for the layer accordingly. Therefore,
a $M$ layer neural network will have $M$ $L2$ norms, one for each layer. Once
the per-example gradient is computed for layer $m$ ($1\leq m\leq M$), we clip
the gradient in $L_{2}$ norm (Line 10), i.e., the $L_{2}$ norm of layer-wise
per-example gradient vector $\nabla w_{ij}(t)_{lm}$ is clipped down to $C$ if
$||\nabla w_{ij}(t)_{lm}||_{2}>C$ and $\nabla w_{ij}(t)_{lm}$ is preserved if
$||\nabla w_{ij}(t)_{lm}||_{2}\leq C$. The per-example gradient will consist
of $M$ components (Line 12). Then Gaussian noise is added to the clipped per-
example gradient (Line 14). For each batch, the sanitized per-example
gradients are gathered for batch averaging and local stochastic gradient
descent (Line 15). Upon the completion of the local training over the $L$
iterations, the local training model update is produced by the client $i$
using the Fed-CDP client module and this noisy local gradient update will be
shared with the server (Line 17) for FedSGD aggregation and global model
update for the next round of training (Lines 18-20). It is important to note
that any change in the gradient of one layer after it is computed will not
change the computation of the gradient of its previous layer. This indicates
that clipping and adding noise to computed gradients of one layer cannot
protect other layers in the model. Hence, Fed-CDP requires the clipping and
sanitization to be performed on the gradient of every layer for all $M$ layers
and at every local iteration to ensure that the local model training and the
global model from the joint training satisfy the per-example based client
differential privacy.

By applying clipping and adding noise layer by layer with back-propagation
computed from the loss function to the weights of the first model layer, this
ensures that sanitizing of per-example per-layer gradients is done immediately
after gradients of one layer is computed, enabling Fed-CDP to provide instant
protection of the per-example local training gradients against type-2 leakage
at each client. Furthermore, the per-example clipping and adding noise at each
local client will have a global effect through the iterative joint training of
$T$ rounds, enabling Fed-CDP to be resilient against type-0 and type-1
leakages as well.

##  V Privacy Analysis

The design of Fed-CDP aims to provide differential privacy guarantee at both
instance level and client level. In this section, we formally analyze the
privacy guarantee by Fed-CDP.

Privacy at instance-level. In the definition of ($\epsilon,\delta$)
differential privacy [29], the two adjacent inputs $A\subseteq\mathcal{D}$ and
$A^{\prime}\subseteq\mathcal{D}$ differing by only a single example, and the
privacy protection by the($\epsilon,\delta$)-differential privacy is to
introduce a randomized mechanism such that the differentially private version
of the learning algorithm will be used to ensure the privacy at instance
level, that is, even if an adversary can observe the output of the algorithm,
he cannot tell whether the user information defined by a single training
example was used in the computation of the algorithm.

Given a distributed collection $\mathcal{D}$ of training data residing on a
total of $K$ disseminated clients, Fed-CDP enforces per-example gradient
clipping and sanitization via Gaussian noise injection, leading to instant
protection of the per-example local training gradients against type-2 leakage
at each client. By Definition 5, for a mechanism $\mathcal{M}$, given the
privacy parameters $\delta$ and $\sigma$ ($>0$), $\mathcal{M}$ is
$(\epsilon,\delta)$-differentially private if the sample rate $q$ satisfies
$q<\frac{1}{16\sigma}$ and $\epsilon$ satisfies the following equation:

$c2\frac{q\sqrt{T\log(1/\delta)}}{\sigma}\leq\epsilon<c1q^{2}T.$ (3)

In order to leverage the moment accountant [14] to compute the privacy budget
spending ($\epsilon$) of Fed-CDP during the joint training for the $T$ rounds,
we simply need to prove that local sampling with replacement across different
clients in federated learning can be modeled as global data sampling with
replacement. Thus, federated learning can be viewed as an alternative
implementation of centralized training in terms of algorithmic logic.

######  Proposition 1.

The probability of any class being selected globally is the same as the
probability of that class in the global data distribution.

One way to simplify the proof of this proposition is to consider each client
hold data points from two classes out of the total number of classes ($Z$),
and compute the probability of sampling data points from the two classes from
a random chosen client $i$ and the probability of the event of a client
sampling data from one of its classes. Based on these two probabilities, we
can then compute the probability of a specific class being selected randomly,
which would be equivalent to $\frac{1}{Z}$. This indicates that the local
sampling with replacement can be modeled as global data sampling with
replacement. For Fed-CDP with the total amount of training data $N$ over the
total $K$ clients, the sampled local data over $K_{t}$ out of $K$ clients in
each round $t$ can be viewed as a global data sampling with sampling size of
$B*K_{t}$ and the sampling rate $\frac{B*K_{t}}{N}$. Hence, we can compute the
privacy spending using moments accountant similar to centralized differential
privacy. Hence the locally added per-example noise would have a global privacy
effect over the global data.

Privacy at client-level In Fed-CDP, for each round $t$, consider any client
$i$ from the chosen $K_{t}$ clients, the local training parameter updates
shared by client $i$ to the server are the local aggregation of noisy per-
example gradients over $L$ local iterations with batch size $B$ in each
iteration. Thus, the proposed Fed-CDP offers per-example based client
differential privacy and at the same time offers joint differential privacy
[38] from a per-client perspective. Joint differential privacy requires that
for two adjacent inputs K and K′, differing by only the update from one client
$i$ at each round $t$ of the joint training, the privacy protection by the
($\epsilon,\delta$)-joint differential privacy is to introduce a randomized
mechanism such that the differentially private version of the learning
algorithm will be used to ensure the privacy at client level: even if an
adversary can observe the output of the algorithm, he cannot tell whether the
local training parameter updates from a single client was used in computing
the output of the algorithm.

######  Definition 6.

Joint Differential privacy [38]: A mechanism $\mathcal{M}$:
$\mathcal{D}(K)\rightarrow\mathcal{R}$ satisfies joint
($\epsilon,\delta$)-differential privacy if for any two sets of agents $K$ or
$K^{\prime}$, each with its data as
$\mathcal{D}(K),\mathcal{D}(K^{\prime})\subseteq\mathcal{D}$, the following
condition holds.

$\Pr(\mathcal{M}(\mathcal{D})(K)\in\mathcal{R})\leq
e^{\epsilon}\Pr(\mathcal{M}(\mathcal{D})(K^{\prime})\in\mathcal{R})+\delta.$
(4)

Joint differential privacy is a relaxation of the standard differential
privacy but yet strong: it implies that for any client $i$ in federated
learning (agent), even if all the other clients (agents) except client $i$
collude and share their information, they would not be able to learn about
client $i$’s private information [38]. Fed-CDP achieves joint differential
privacy by utilizing the Billboard Lemma (lemma 2) and considering the
differentially private global model at the server shared with all clients as
the public information on the Billboard.

######  Lemma 2.

Billboard Lemma [39] Suppose $\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}$
is $(\epsilon,\delta)$-differentially private. Consider any set of functions
$f_{k}:\mathcal{D}_{i}\times\mathcal{R}\rightarrow\mathcal{R^{\prime}}$, where
$\mathcal{D}_{i}\subseteq\mathcal{D}$ containing agent $i$’s data. The
composition
$\\{f_{i}(\prod\nolimits_{i}\mathcal{D},\mathcal{M}(\mathcal{D}))\\}$ is
$(\epsilon,\delta)$-joint differentially private, where
$\prod\nolimits_{i}:\mathcal{D}\rightarrow\mathcal{D}_{i}$ is the projection
to agent $i$’s data.

Given that existing Fed-SDP approaches [12, 13] clip and add noise at each
round $t$ to the local model update shared by each of the $K_{t}$ clients,
Fed-SDP does not explicitly provide the instance-level client different
privacy but it does provides the client-level different privacy guarantee in
the sense that for two possible sets of clients, K and K’, differing by only
one client $i$ at each round $t$ of the joint training, by introducing a
randomized mechanism such that the differentially private version of the
learning algorithm will be used to ensure that even if an adversary can
observe the output of the algorithm, he cannot tell whether the local training
parameter updates from a single client was used in the computing the output of
the algorithm. Thus, the ($\epsilon,\delta$)-differential privacy by Fed-SDP
provides privacy protection at client level. Note that LDP with federated
learning [40], which independently protects single value of clients’ data
locally and does not offer differential privacy guarantee on the globally
trained model, is beyond the scope of this paper.

##  VI Privacy-utility Trade-off Analysis

The privacy and utility trade-off is known to be an open challenge [41, 42].
In Fed-CDP, to provide per-example based client differential privacy, in each
round $t$, for each of the selected $K_{t}$ clients, we clip and then add
noise to the per-example gradient before performing the local batch stochastic
gradient descent for each of the $L$ iterations in the local model training by
the client. The same process of clipping and noise injection repeats for a
total of $T$ rounds to produce the global model as the outcome of the joint
training.

We first analyze how the injected Gaussian noise affects the privacy-
performance trade-off in terms of changing gradient descent directions and
provide a formal guarantee of the maximal distortion which a neural network
model can tolerate without deteriorating the training performance.

Let $g_{j}(x;w(t)):\mathcal{R}^{D}\rightarrow R$ be the loss function of the
continuously differentiable DNN model on class $j$ given the input $x$. We
define $s(x,w(t))=g_{y}(x;w(t))-g_{j,j\neq y}(x;w(t))$ where $y$ is the label
of input $x$. Under the assumption of Lipschitz condition, we have

$|s(x,w(t))-s(x,w^{\prime}(t))|\leq L_{v}||w(t)-w^{\prime}(t)||_{u},$ (5)

where $w^{\prime}(t)$ is the differentially private version of the model
parameters $w(t)$, and
$L_{v}=\max\nolimits_{x}\\{||\nabla{s(x,w(t))}||_{v}\\}$ is the Lipschitz
constant with $\frac{1}{u}+\frac{1}{v}=1$ where $u\geq 1$ and $v\leq\infty$.
By line 14 of Algorithm 2, we can formulate the private model parameters as
$w^{\prime}(t)=w(t)+\xi$ and substitute Equation 5 as

$s(x,w(t))-L_{v}||\xi||_{u}\leq s(x,w(t)+\xi)\leq s(x,w(t))+L_{v}||\xi||_{u}.$
(6)

Equation 6 shows that when $s(x,w(t)+\xi)<0$, we will have
$g_{y}(x;w(t)+\xi)<g_{j,j\neq y}(x;w(t)+\xi)$, indicating an alternation of
ground-truth class used to compute the loss function. This shows that adding
large Gaussian noise to the model, may move the loss function computed on the
ground-truth class to other classes and lead to the loss of utility with
respect to learning convergence and prediction accuracy. Hence, we want the
privacy-preserving cost $||\xi||_{p}$ to be small enough such that
$s(x,w(t))-L_{v}||\xi||_{u}\geq 0$.

Figure 3: Changing of gradients’ $L2$ norm during training

######  Proposition 2.

Upper bound $\xi_{u}$ for Utility-Privacy trade-off
To preserve the training against the given ground-truth label $y$ and maintain
the training information, the maximum change made to model parameters (or
gradient) need to be bounded as follows:

$||\xi||_{u}\leq\min_{j\neq y}\frac{g_{y}(x;w(t))-g_{j,j\neq
y}(x;w(t))}{L_{v}}.$ (7)

This proposition states that $||\xi||_{u}\leq\min_{j\neq
y}\frac{g_{y}(x;w(t))-g_{j,j\neq y}(x;w(t))}{L_{v}}$ is the lower bound for
minimum distortion of the model parameter that would result in the loss of
training performance. We can prove the proposition based on the Lipschitz
assumption as follows: First, we have

$\displaystyle|s(x,w(t))-s(x,w^{\prime}(t))|\leq
L_{v}||w(t)-w^{\prime}(t)||_{u}=L_{v}||\xi||_{u}\Rightarrow$ $\displaystyle
s(x,w(t))-L_{v}||\xi||_{u}\leq s(x,w(t)+\xi)\leq s(x,w(t))+L_{v}||\xi||_{u}$
(8)

To ensure that the distortion does not affect the training process by
alternating the prediction towards the ground-truth class to other classes, we
will need $s(x,w(t))-L_{v}||\xi||_{u}\geq 0$ in Equation 8 holds. Then we have

$\displaystyle s(x,w(t))-L_{v}||\xi||_{u}\geq
0\Rightarrow||\xi||_{u}\leq\frac{s(x,w(t))}{L_{v}}$
$\displaystyle\Rightarrow||\xi||_{u}\leq\min_{j\neq
y}\frac{g_{y}(x;w(t))-g_{j,j\neq y}(x;w(t))}{L_{v}}$

Proposition 7 indicates that the training performance will less likely to be
affected by the injected Gaussian noise if
$||\mathcal{N}(0,\sigma^{2}S^{2})||_{u}\leq\min_{j\neq
y}\frac{g_{y}(x;w(t))-g_{j,j\neq y}(x;w(t))}{L_{v}}$.

Given that it is computationally expensive to find such a noise bound in
practice, especially for deep learning algorithms with large number of
classes. One approach is to translate loss distance between the label class
and the second-largest prediction class into the $L2$ magnitude of the
gradients and ensure the bound over such $L2$ norm of local gradient updates.

Figure 3 illustrates an example of the decaying magnitude of $L2$ norm of the
gradient during training. Mean $L2$ norm of 100 MNIST clients at one local
iteration is reported. The decaying magnitude of gradient’s $L2$ norm
indicates that the loss value between the predicted label and ground-truth
label decreases over time, implying a decaying confidence gap between the
label class $g_{y}(x;w(t))$ and the class with the second-largest confidence.
This is aligned with the observation that gradients in early iterations are
more informative and tend to have a larger $L2$-norm value [8]. This is
consistent with the literature on the need for fine-tuning of clipping bound
used in a differentially private algorithm [14] to bound the distortion noise
using a proper clipping bound with constant clipping method.

Fed-CDP (decay). Motivated by the utility-privacy tradeoff analysis, we
consider to optimize Fed-CDP by using a dynamic decaying clipping method with
the initial clipping bound $C$ in order to keep track of the decaying $L2$
sensitivity of gradients as federated learning progresses in the number of
rounds, ensuring the differential privacy based distortion noise being bounded
properly by following the proposition 7. It is important to note that decaying
clipping bound will not modify the privacy composition. Also the noise
variance is determined by both the sensitivity $S$ (thus the the clipping
bound $C$ in Fed-CDP and most existing DP algorithms) and the noise scale
$\sigma$. Using a dynamic clipping method with decay function on bound $C$
will result in decreasing trend of noise variance over the $T$ rounds of
federated learning, resulting in non-uniform distribution of privacy spending
over the $T$ rounds according to Lemma 1.

Empirical study in Section VII shows that by decaying clipping bound to track
the decaying noise scale, Fed-CDP (decay) further optimizes Fed-CDP with
improved accuracy performance and higher resilience to gradient leakage
attacks with the same privacy budget spending, achieving better utility-
privacy tradeoff, compared to Fed-CDP baseline.

##  VII Experimental Evaluation

| MNIST | CIFAR10 | LFW | adult | cancer
---|---|---|---|---|---
# training data | 60000 | 50000 | 2267 | 36631 | 426
# validation data | 10000 | 10000 | 756 | 12211 | 143
# features | 28*28 | 32*32*3 | 32*32*3 | 105 | 30
# classes | 10 | 10 | 62 | 2 | 2
# data/client | 500 | 400 | 300 | 300 | 400
# local iteration $L$ | 100 | 100 | 100 | 100 | 100
local batch size $B$ | 5 | 4 | 3 | 3 | 4
# rounds $T$ | 100 | 100 | 60 | 10 | 3
non-private acc | 0.9798 | 0.674 | 0.695 | 0.8424 | 0.993
non-private cost(ms) | 6.8 | 32.5 | 30.9 | 5.1 | 4.9

TABLE I: Benchmark datasets and parameters

| K=100 | K=1000 | K=10000
---|---|---|---
Kt/K | 5% | 10% | 20% | 50% | 5% | 10% | 20% | 50% | 5% | 10% | 20% | 50%
non-private | 0.924 | 0.954 | 0.959 | 0.965 | 0.977 | 0.980 | 0.978 | 0.978 | 0.979 | 0.980 | 0.980 | 0.979
Fed-SDP | 0.803 | 0.823 | 0.834 | 0.872 | 0.925 | 0.928 | 0.934 | 0.937 | 0.935 | 0.939 | 0.941 | 0.944
Fed-CDP | 0.815 | 0.831 | 0.858 | 0.903 | 0.951 | 0.956 | 0.96 | 0.964 | 0.966 | 0.963 | 0.968 | 0.966
Fed-CDP(decay) | 0.833 | 0.842 | 0.866 | 0.909 | 0.968 | 0.975 | 0.977 | 0.976 | 0.974 | 0.978 | 0.979 | 0.980

TABLE II: Accuracy comparison by varying # total clients and $Kt/K$ in MNIST
with default $C=4$ and $\sigma=6$

| MNIST | CIFAR-10 | LFW | adult | cancer
---|---|---|---|---|---
non-private | 6.8 | 32.5 | 30.9 | 5.1 | 5.1
Fed-SDP | 6.9 | 33.8 | 31.3 | 5.2 | 5.1
Fed-CDP | 22.4 | 131.5 | 112.4 | 11.8 | 11.9
Fed-CDP(decay) | 22.6 | 132.1 | 114.6 | 12.1 | 12.0

TABLE III: Time cost per local iteration per client(ms)

| C=0.5 | C=1 | C=2 | C=4 | C=6 | C=8
---|---|---|---|---|---|---
MNIST | 0.914 | 0.934 | 0.943 | 0.949 | 0.933 | 0.923
CIFAR-10 | 0.408 | 0.568 | 0.602 | 0.633 | 0.624 | 0.611
LFW | 0.582 | 0.594 | 0.619 | 0.649 | 0.627 | 0.601
adult | 0.81 | 0.822 | 0.825 | 0.824 | 0.807 | 0.796
cancer | 0.965 | 0.972 | 0.979 | 0.979 | 0.972 | 0.972

TABLE IV: Accuracy of Fed-CDP by varying clipping with default $\sigma=6$

|  $\sigma$=0.5 |  $\sigma$=1 |  $\sigma$=2 |  $\sigma$=4 |  $\sigma$=6 |  $\sigma$=8
---|---|---|---|---|---|---
MNIST | 0.956 | 0.954 | 0.952 | 0.951 | 0.949 | 0.934
CIFAR-10 | 0.646 | 0.641 | 0.639 | 0.634 | 0.633 | 0.612
LFW | 0.683 | 0.678 | 0.672 | 0.667 | 0.649 | 0.646
adult | 0.838 | 0.837 | 0.836 | 0.834 | 0.824 | 0.822
cancer | 0.993 | 0.993 | 0.993 | 0.993 | 0.979 | 0.979

TABLE V: Accuracy of Fed-CDP by varying noise scale with default $C=4$

We evaluate Fed-CDP on five benchmark datasets as shown in Table I, with each
dataset given the parameter setup, the validation accuracy and training cost
for the non-private federated learning model. MNIST is a grey-scale hand-
written digits dataset consisting of 60,000 training data and 10,000 test
data. The 5:1 train-validation ratio is applied to the 60,000 training images.
We partition the 50,000 training data into shards. Each client gets two shards
with 500 samples from two classes. For CIFAR10, the 4:1 train-validation ratio
is applied to the 50,000 training images. Each client has 400 images from two
classes. Labeled Faces in the Wild(LFW) dataset has 13233 images from 5749
classes. The original image size is $250\times 250$ and we crop it into
$32\times 32$ size to extract the ’interesting’ region. Since most of the
classes have a very limited number of data points, we consider the 62 classes
that have more than 20 images per class. For a total number of 3023 eligible
LFW data, a 3:1 train-validation ratio is applied. We partition the training
data into shards and each client has 300 images from about 15 classes. We
evaluate the three benchmark image datasets on a multi-layer convolutional
neural network with two convolutional layers and one fully-connected layer.
For the two attribute datasets, the 3:1 train-validation ratio is applied to
both. Adult income dataset has 48842 total training data with 105 converted
numerical features, and each client holds 300 data from two classes randomly
sampled from the training data. Breast cancer dataset has 569 training data
with 30 numerical features, and each client has a full copy of the dataset due
to its small number of data points. A fully-connected model with two hidden
layers is used for attribute datasets.

Our federated learning setup follows the simulator in [13] with the total $K$
clients varying from 100, 1000 to 10,000, and $K_{t}$ is set at a varying
percentage of $K$, such as $5\%,10\%,20\%,50\%$ per round. All experiments are
conducted on an Intel 4 core i5-7200U<EMAIL_ADDRESS>machine with an Nvidia
Geforce 2080Ti GPU. We evaluate Fed-CDP in terms of three sets of metrics: (i)
training performance in validation accuracy and time cost of conducting one
local iteration at a client (ms/iteration), (ii) the privacy budget spending
$\epsilon$ with varying settings of other differential privacy parameters,
such as clipping bound $C$ and noise scale $\sigma$; and (iii) the resilience
against all three types of gradient leakage attacks in terms of attack success
(Yes or No), #attack iterations to succeed with the default set to 300, and
attack reconstruction distance, defined by the root mean square deviation
between reconstructed input $x_{rec}$ and its private ground-truth counterpart
$x$: $\frac{1}{A}\sum\nolimits_{i=1}^{A}{(x(i)-{x_{rec}}(i)}{)^{2}}$. $A$
denotes the total number of features in the input.

###  VII-A Training performance Evaluation

Fed-SDP v.s. Fed-CDP. The performance of Fed-CDP is evaluated and compared
with Fed-SDP by the default parameter setting of MNIST in Table I. Table II
shows the results by varying total $K$ clients from 100 to 1000 and 10000 with
5%, 10%, 20% and 50% participating clients at each round. We make two
observations: (1) Both Fed-CDP and Fed-SDP can achieve high accuracy
comparable to that of the non-private setting. (2) Fed-CDP consistently
achieves higher accuracy than Fed-SDP for all settings of $K$ and $K_{t}$
while Fed-CDP(decay) consistently improves Fed-CDP with slightly better
accuracy performance. For Fed-CDP(decay), we linearly decay the clipping bound
from C=6 to C=2 in 100 rounds and achieve high validation accuracy of 0.9748,
on par with the non-private FL setting on MNIST. We will show later that
decay-clipping optimized Fed-CDP also demonstrates higher resiliency against
gradient leakage attacks.

Table III shows the per local iteration per client training time comparison.
Given all $K_{t}$ clients can perform local training in parallel, the overall
time spent for one round is determined by the slowest client local training at
the round $t$. Although Fed-CDP incurs additional computation overhead to clip
and sanitize per-example gradients in each local batch (iteration), compared
to Fed-SDP, the absolute time cost remains relatively low. The decay of
clipping bound brings a negligible cost compared to Fed-CDP.

Fed-CDP performance with varying parameter settings. We first study the effect
of varying clipping bound from 0.5 to 8. Table IV shows the results of Fed-
CDP. By combining gradient clipping and noise injection to per-example
gradients, Fed-CDP leads to high accuracy on MNIST, Adult and Cancer when
$C=2$ or $C=4$ and high accuracy on LFW and CIFAR-10 when $C=4$ or $C=6$,
because for different datasets, $L2$ norm of the gradients varies and results
in different best clipping bound settings. Since the clipping bound will
impact both the gradient information preservation and noise variance, the
highest accuracy appears only with an appropriately set clipping bound and the
performance will downgrade if the clipping bound is too large or too small. We
next evaluate the impact of varying noise scale of $0.5,1,2,4,6,8$. Table V
shows the results given the clipping bound $C=4$. Noise scale controls the
amount of Gaussian noise added to the per-example gradients. The results
confirm that adding too much noise will impact negatively the training
performance. In Fed-CDP, we set $\sigma=6$ by default to ensure the
requirement of $q<\frac{1}{16\sigma}$ for moments accountant, even though a
smaller noise scale may bring improvement on the accuracy performance for all
datasets.

###  VII-B Privacy Evaluation

We evaluate and compare the privacy composition of Fed-CDP and Fed-SDP using
moments accountant (recall Definition 5 and Section V) on all 5 datasets with
two settings on the per-client location training iterations: $L=1$ and
$L=100$. Table VI summarizes the results.

Privacy at instance level. Since privacy composition is only related with the
sampling rate and the total number of rounds ($T$) under a given $\delta$ by
Equation 2, $\epsilon$ for different dataset depends mainly on the total
number of rounds since we set the global sampling rate for all datasets to
0.01. The training model for MNIST and CIFAR-10 under $L=100$ is
$(0.8227,1e-5)$-differential private with $\epsilon=0.8227$ and $\delta=1e-5$.
For LFW, adult, and cancer datasets, we have $\epsilon=0.6356$,
$\epsilon=0.2761$, and $\epsilon=0.1469$ respectively. All 5 datasets have the
same sampling rate of 0.01, and their total training rounds $T$ set to 100,
100, 60, 10, and 3 for MNIST, CIFAR-10, LFW, adult, and cancer respectively.
Note that although we report the instance-level privacy by round since we will
only get a new global model for each round, the actual $\epsilon$ value
accumulates with the setting of the # of local iterations ($L$). For $L=1$,
clients performing local model training over one local iteration per round,
which will generate a differentially private model with a much smaller privacy
budget spending $\epsilon$ compared to the $\epsilon$ from the differentially
private model generated over clients with local training on 100 local
iterations at the same round. However, local training with $L=1$ takes more
rounds to achieve the same accuracy as the local training with $L=100$
iterations.

| MNIST | CIFAR-10 | LFW | adult | cancer
---|---|---|---|---|---
instance | Fed-CDP: 1 | 0.0845 | 0.0845 | 0.0689 | 0.0494 | 0.0467
Fed-CDP: 100 | 0.8227 | 0.8227 | 0.6356 | 0.2761 | 0.1469
Fed-SDP: 1 | not supported
Fed-SDP: 100
client | Fed-CDP: 1 | 0.0845 | 0.0845 | 0.0689 | 0.0494 | 0.0467
Fed-SDP: 1 | 0.8536 | 0.8536 | 0.6677 | 0.3025 | 0.2065
Fed-CDP: 100 | 0.8227 | 0.8227 | 0.6356 | 0.2761 | 0.1469
Fed-SDP: 100 | 0.8536 | 0.8536 | 0.6677 | 0.3025 | 0.2065

TABLE VI: Privacy composition of Fed-SDP and Fed-CDP using $\epsilon$ with
$\delta=1e-5$, $L=1$ and $L=100$ local iterations for per-client local
training. $\epsilon$ measured at 100, 100, 60, 10, 3 rounds for MNIST,
CIFAR10, LFW, adult, and cancer dataset respectively.

| type 0&1 | type 2
---|---|---
non-private | Fed-SDP | Fed-CDP |  | Fed-CDP
---
(decay)
non-private | Fed-SDP | Fed-CDP |  | Fed-CDP
---
(decay)
MNIST | succeed | Y | N | N | N | Y | Y | N | N
reconstruction distance | 0.1549 | 0.6991 | 0.7695 | 0.937 | 0.0008 | 0.0008 | 0.739 | 0.943
attack iteration | 6 | 300 | 300 | 300 | 7 | 7 | 300 | 300
LFW | succeed | Y | N | N | N | Y | Y | N | N
reconstruction distance | 0.2214 | 0.7352 | 0.8036 | 0.941 | 0.0014 | 0.0014 | 0.6626 | 0.945
attack iteration | 24 | 300 | 300 | 300 | 25 | 25 | 300 | 300

TABLE VII: Average attack effectiveness comparison from 100 clients, with the
maximum attack iteration set to 300.

Privacy at client level. According to the Billboard Lemma(lemma 2), with
$\delta=1e-5$, the trained model with Fed-CDP after the $T$ rounds will
accumulate the privacy spending $\epsilon=0.8227,0.8227,0.6356,0.2761,0.1469$
for where for MNIST, CIFAR-10, LFW, adult and cancer respectively where
$T=100,100,60,10,3$ and $L=100$ under joint differential privacy. Privacy at
client-level for Fed-CDP benefits from the instance-level privacy for Fed-CDP.
For Fed-SDP, the client-level differential privacy is computed with the
sampling rate $q2=\frac{K_{t}}{K}$ [12, 13] at each round. Since Fed-SDP adds
noise directly to the per-client local gradient updates and whether the per-
client local training is performed over one local iteration or 100 local
iterations will not affect its privacy accounting, as shown in Table VI.

###  VII-C Gradient-Leakage Resiliency

We measure the resiliency of Fed-CDP and Fed-SDP against gradient leakage
attacks using MNIST and LFW dataset. The gradient leakage attack experiment is
performed on gradients from the first local iteration as gradients at early
training iterations tend to leak more information than gradients in the later
stage of the training [8]. Table VII reports the resilience results in terms
of attack success (yes or no), the average reconstruction distance, and attack
iterations from 100 clients. For failed attacks, we run the attack
reconstruction and record the attack reconstruction distance at the maximum
attack iterations of 300. We observe that non-private federated learning is
vulnerable to all three types of gradient leakage attacks, while Fed-CDP can
effectively mitigate all three types of leakages and Fed-SDP is vulnerable to
the type-2 leakage though it is more resilient to the type-0 and type-1
leakage compared to non-private FL. Compared to Fed-SDP, Fed-CDP has a larger
reconstruction distance, which implies more robust masking of the private
training data against gradient leakage attacks. Figure 4 shows a visualization
of Fed-CDP and Fed-SDP under three types of gradient leakages using an example
batch of LFW dataset in the local training, showing Fed-CDP(decay) offers the
highest resilience against all three gradient leakages, followed by Fed-CDP,
while Fed-SDP is vulnerable to type-2 leakage and non-private FL and
Distributed Selective SGD(DSSGD [25]) is vulnerable to all three types of
gradient leakages.

Figure 4: Visualization of gradient leakage attacks under different Fed-DP
modules. Gradients of one example in the batch is used to do type-2 attack.

Figure 5: Accuracy and resiliency to type-2 leakage in communication efficient
federated learning

Figure 5 provides a visualization comparison of Fed-CDP(decay), Fed-CDP, and
Fed-SDP under type-2 gradient leakage attack in the communication-efficient
federated learning using MNIST dataset. Validation accuracy is also reported
at a total of 1000 clients and 100 participants in the figure. In this set of
experiments, we consider communication-efficient federated learning by pruning
the insignificant gradients, which make little contribution to joint training,
i.e., gradients with very small values [8]. Using non-private training or Fed-
SDP, the gradients shared via communication-efficient protocol (by
compression) may still suffer from the gradient leakage threats for
compression ratios up to 30% on MNIST. In contrast, Fed-CDP is more resilient
than Fed-SDP, while Fed-CDP(decay) provides the highest resilience in all
settings against the type-2 leakages in the communication-efficient federated
learning protocols.

##  VIII Conclusion

We have presented a per example based client different privacy approach to
federated learning with three original contributions. We first identified
three types of gradient leakage attacks and explained why the existing Fed-SDP
approach is vulnerable. We then introduce Fed-CDP, the per example-based
client differential privacy algorithm, with a formal comparison between Fed-
CDP and Fed-SDP in terms of privacy accounting. Finally, we provide a formal
analysis of the privacy-utility trade-off for providing differential privacy
guarantee by Fed-CDP and present Fed-CDP(decay), which further improves the
accuracy and resiliency of Fed-CDP by incorporating a dynamic decay clipping
method, instead of constant clipping method popularly used in the literature
[14, 12]. With extensive evaluation, we show that the Fed-CDP approach
outperforms Fed-SDP with high resilience against all three types of gradient
leakage attacks while offering competitive accuracy performance.

Acknowledgement. This work is partially supported by the National Science
Foundation under Grants NSF 1564097 and NSF 2038029 as well as an IBM faculty
award. The authors especially thank the support and collaboration with the
systems group in IBM T.J. Watson, led by Dr. Donna N Dillenberger. Any
opinions, findings, and conclusions or recommendations expressed in this
material are those of the author(s) and do not necessarily reflect the views
of the National Science Foundation or IBM.

## References

  * [1] P. Vanhaesebrouck, A. Bellet, and M. Tommasi, “Decentralized collaborative learning of personalized models over networks,” in _AISTATS_ , 2017, pp. 509–517.
  * [2] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _AISTATS_ , 2017, pp. 1273–1282.
  * [3] J. Konečnỳ, H. B. McMahan, F. X. Yu, P. Richtárik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” in _NeurIPS Workshop on Private Multi-Party ML_ , 2016.
  * [4] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konecny, S. Mazzocchi, and H. B. McMahan, “Towards federated learning at scale: System design,” in _SysML_ , 2018, pp. 619–633.
  * [5] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” _arXiv preprint arXiv:1806.00582_ , 2018.
  * [6] L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” in _NeurIPS_ , 2019, pp. 14 747–14 756.
  * [7] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients - how easy is it to break privacy in federated learning?” in _NeurIPS_ , 2020, pp. 16 937–16 947.
  * [8] Wenqi Wei, L. Liu, M. Loper, K.-H. Chow, M. E. Gursoy, S. Truex, and Y. Wu, “A framework for evaluating client privacy leakages in federated learning,” in _ESORICS_. Springer, 2020, pp. 545–566.
  * [9] M. Nasr, R. Shokri, and A. Houmansadr, “Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks,” in _IEEE S &P_, 2019, pp. 739–753.
  * [10] S. Truex, L. Liu, M. E. Gursoy, L. Yu, and Wenqi Wei, “Demystifying membership inference attacks in machine learning as a service,” _IEEE TSC_ , 2019.
  * [11] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, “Membership inference attacks against machine learning models,” in _IEEE S &P_, 2017, pp. 3–18.
  * [12] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” in _ICLR_ , 2018.
  * [13] R. C. Geyer, T. Klein, and M. Nabi, “Differentially private federated learning: A client level perspective,” _arXiv preprint arXiv:1712.07557_ , 2017.
  * [14] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” in _ACM CCS_ , 2016, pp. 308–318.
  * [15] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _IEEE S &P_, 2019, pp. 691–706.
  * [16] C. Song, T. Ristenpart, and V. Shmatikov, “Machine learning models that remember too much,” in _ACM CCS_ , 2017, pp. 587–601.
  * [17] Y. Aono, T. Hayashi, L. Wang, and S. Moriai, “Privacy-preserving deep learning via additively homomorphic encryption,” _IEEE TIFS_ , vol. 13, no. 5, pp. 1333–1345, 2017.
  * [18] M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in _ACM CCS_ , 2015, pp. 1322–1333.
  * [19] B. Hitaj, G. Ateniese, and F. Perez-Cruz, “Deep models under the gan: information leakage from collaborative deep learning,” in _ACM CCS_ , 2017, pp. 603–618.
  * [20] Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, “Beyond inferring class representatives: User-level privacy leakage from federated learning,” in _IEEE INFOCOM_ , 2019, pp. 2512–2520.
  * [21] E. Shi, T. H. Chan, E. Rieffel, R. Chow, and D. Song, “Privacy-preserving aggregation of time-series data,” in _NDSS_ , 2011, pp. 1–17.
  * [22] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, “Practical secure aggregation for privacy-preserving machine learning,” in _ACM CCS_ , 2017, pp. 1175–1191.
  * [23] M. Chase, R. Gilad-Bachrach, K. Laine, K. E. Lauter, and P. Rindal, “Private collaborative neural network learning.” _IACR Cryptology ePrint Archive_ , 2017.
  * [24] N. Papernot, S. Song, I. Mironov, A. Raghunathan, K. Talwar, and Ú. Erlingsson, “Scalable private learning with pate,” in _ICLR_ , 2018\.
  * [25] R. Shokri and V. Shmatikov, “Privacy-preserving deep learning,” in _ACM CCS_ , 2015, pp. 1310–1321.
  * [26] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database forstudying face recognition in unconstrained environments,” in _Technical report_ , 2008.
  * [27] Y. LeCun, C. Cortes, and C. J. Burges, “The mnist database of handwritten digits, 1998,” _URL http://yann. lecun. com/exdb/mnist_ , vol. 10, p. 34, 1998.
  * [28] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” in _Technical report_ , 2009.
  * [29] C. Dwork and A. Roth, “The algorithmic foundations of differential privacy,” _Foundations and Trends® in Theoretical Computer Science_ , vol. 9, no. 3–4, pp. 211–407, 2014.
  * [30] Y.-X. Wang, B. Balle, and S. P. Kasiviswanathan, “Subsampled rényi differential privacy and analytical moments accountant,” in _AISTATS_ , 2019, pp. 1226–1235.
  * [31] Y. Lin, S. Han, H. Mao, Y. Wang, and B. Dally, “Deep gradient compression: Reducing the communication bandwidth for distributed training,” in _ICLR_ , 2018.
  * [32] W. Liu, L. Chen, Y. Chen, and W. Zhang, “Accelerating federated learning via momentum gradient descent,” _IEEE TPDS_ , 2020.
  * [33] X. Yao, T. Huang, R.-X. Zhang, R. Li, and L. Sun, “Federated learning with unbiased gradient aggregation and controllable meta updating,” in _NeurIPS Workshop on FL for Data Privacy and Confidentiality_ , 2019.
  * [34] M. Kamp, L. Adilova, J. Sicking, F. Hüger, P. Schlicht, T. Wirtz, and S. Wrobel, “Efficient decentralized deep learning by dynamic model averaging,” in _ECML-PKDD_. Springer, 2018, pp. 393–409.
  * [35] C. Ma, V. Smith, M. Jaggi, M. I. Jordan, P. Richtárik, and M. Takáč, “Adding vs. averaging in distributed primal-dual optimization,” in _ICML_ , 2015, pp. 1973–1982.
  * [36] Y. Wu, L. Liu, J. Bae, K.-H. Chow, A. Iyengar, C. Pu, Wenqi Wei, L. Yu, and Q. Zhang, “Demystifying learning rate policies for high accuracy training of deep neural networks,” in _IEEE Intl. Conf. on Big Data_ , 2019, pp. 1971–1980.
  * [37] L. Yu, L. Liu, C. Pu, M. E. Gursoy, and S. Truex, “Differentially private model publishing for deep learning,” in _IEEE S &P_, 2019, pp. 332–349.
  * [38] M. Kearns, M. Pai, A. Roth, and J. Ullman, “Mechanism design in large games: Incentives and privacy,” in _ITCS_ , 2014, pp. 403–410.
  * [39] Z. S. Wu, “Data privacy beyond differential privacy,” _Technical report_ , 2017.
  * [40] S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and Wenqi Wei, “Ldp-fed: federated learning with local differential privacy,” in _ACM EdgeSys_ , 2020, pp. 61–66.
  * [41] T. Li and N. Li, “On the tradeoff between privacy and utility in data publishing,” in _ACM SIGKDD_ , 2009, pp. 517–526.
  * [42] Y. Wang, Y. O. Basciftci, and P. Ishwar, “Privacy-utility tradeoffs under constrained data release mechanisms,” _arXiv preprint arXiv:1710.09295_ , 2017.


11institutetext: Alexa Machine Learning, Amazon, USA
22institutetext: School of Informatics, University of Edinburgh, Scotland, UK
<EMAIL_ADDRESS>

# Exploiting Large-scale Teacher-Student Training for On-device Acoustic
Models

Jing Liu 11 Rupak Vignesh Swaminathan 11 Sree Hari Krishnan Parthasarathi 11
Chunchuan Lyu 22 Athanasios Mouchtaris 11 Siegfried Kunzmann 11

###### Abstract

We present results from Alexa speech teams on semi-supervised learning (SSL)
of acoustic models (AM) with experiments spanning over 3000 hours of GPU time,
making our study one of the largest of its kind. We discuss SSL for AMs in a
small footprint setting, showing that a smaller capacity model trained with 1
million hours of unsupervised data can outperform a baseline supervised system
by 14.3% word error rate reduction (WERR). When increasing the supervised data
to seven-fold, our gains diminish to 7.1% WERR; to improve SSL efficiency at
larger supervised data regimes, we employ a step-wise distillation into a
smaller model, obtaining a WERR of 14.4%. We then switch to SSL using larger
student models in low data regimes; while learning efficiency with
unsupervised data is higher, student models may outperform teacher models in
such a setting. We develop a theoretical sketch to explain this behavior.

###### Keywords:

speech recognition, acoustic models, edge computing, student-teacher learning,
semi-supervised learning

##  1 Introduction

Semi-supervised learning (SSL) has a rich history in automatic speech
recognition (ASR) [1, 11, 14, 17, 21]. Self-training is a commonly used
technique employing confidence measures [32, 12]. Student-teacher distillation
techniques [2, 9], foregoing full decoders and confidence models, have been
shown to be effective for SSL [19]. SSL methods for end-to-end ASR have been
studied in [16, 13, 36, 6, 25]. Furthermore, investigations with SSL in
combination with data augmentation, pretraining and iterative self-training
are done in [41] and [38].

Recently, student-teacher distillation techniques for hybrid HMM-LSTM models
have been shown to scale to very large data sets (1 million hours) for models
with high capacity [28, 27]. The efficacy of model compression using student-
teacher distillation is well established [23, 35, 34]. In this context we
study learning curves for AM for two tasks: (a) smaller footprint modeling,
and (b) low training-data regimes.

Our motivation for low footprint AM comes from edge computing, where models
are capacity restricted in terms of compute and memory [31, 40, 39]. We are
interested in understanding if SSL, at very large data regimes for small
models, can still yield gains in accuracy. In [24], mean squared errors
between the teacher and the student hidden representations are explored as a
regularization term in knowledge distillation. We demonstrate that a step-wise
distillation approach, introduced in [23] can be effective, although this
comes at the cost of more computation at training time. In low data regimes,
SSL is an effective technique to reduce annotation costs [14, 17, 21, 5]. For
our second task, using knowledge distillation for SSL, we find that to achieve
a performance comparable to that of a fully supervised system, the proportion
of required supervised data decreases as the amount of total data increases.
However, we find that in low data regimes, students can be better than
teachers.

Figure 1: Overview of our SSL approach progressing through 4 experiments. a)
Top-left: Analysis of models using large unsupervised data. b) Top-right:
Analysis of models in large supervised and unsupervised data regimes. c)
Bottom-right: Improving learning efficiency in higher data regimes. d) Bottom-
left: SSL studies in low data regimes.

Our contributions in this work are as follows: (1) we establish the robustness
of small capacity semi-supervised models trained on 1 million hours of data;
(2) we show an effective way to mitigate the learning saturation problem at
higher data regimes for an on-device acoustic model; (3) we report results of
a model distilled from a teacher trained on transcribed low-resource data, and
present an empirical risk analysis. Figure 1 describes the approach we follow
in this paper. We begin with a discussion on model configurations in Section
2. In Section 3, we present SSL for small footprint AMs. We show that such
models can exploit a large amount of unsupervised data. However, when the
amount of supervised data is increased from 7,000 hours to 52,000 hours the
gains decrease. To mitigate this, we discuss a step-wise distillation into a
smaller model. We then transition into experiments conducted in low resource
settings in Section 4.

##  2 System Description

We now describe acoustic model configurations used in this work. It is a
hybrid system, with an LSTM [10] estimating the senone posterior probabilities
corresponding to clustered triphone HMM states. The HMMs are single state
models using low-frame rate features [29], which are computed on speech
signals every 10 ms, with a 25 ms analysis window. A running causal mean
estimate is computed and subtracted from the features, and the resulting
features are normalized by applying a global mean and variance normalization.
The models are trained with the cross-entropy (CE) criterion, followed by
sequence discriminative training using state-level minimum Bayes risk (sMBR)
loss [15]. We follow an exponential learning rate decay for twelve epochs.
More details on model configurations can be found in Table 1.

Table 1: Model Configurations Common configurations for all experiments
---
• Features are stacked and subsampled to 33 Hz
• 5 layers and 768 neurons/layer BLSTM teachers with 78 million parameters
• Teacher model is cross-entropy and sequence trained
• Teacher model trained with distributed trainer from [33]
• Output posterior distribution over 3183 senones
• Data selection for unsupervised data as in [28]
• Training/test data are sampled from de-identified speech data
• Student training strategy with compressed posteriors as in [28]
• TST1: Test data consists of 100 hours of speech data
• TST2: Test data consists of 30 hours of speech data
Section 3.1 Configurations for on-device AM
• 64-dimensional log filter-bank energies (LFBE)
• 5 layers and 428 neurons/layer Uni-LSTM on-device student AMs with 8 million
parameters
• Cross entropy training of student models with distributed trainer from [4]
• sMBR training of student model with distributed trainer from [33]
• 7,000 hours of supervised training data with TST1 test data
• 1 million hours of unsupervised speech data
Section 3.2 Configurations for studying learning saturation
• 256-dimensional STFT features [30, 18]
• Teacher and student model architectures: same as above
• Distributed training strategy: same as above
• 52 K hours supervised training data with TST1 test data
• 1 million hours of unsupervised speech data
Section 3.3 Configurations for step-wise distillation
• 256-dimensional STFT features
• Teacher and student model architectures: same as above
• Distributed training strategy: same as above
• 50 K hours supervised training data with TST1 test data
• 100 K hours unsupervised data
Section 4 Configurations for low resource experiments
• 64-dimensional LFBE features
• 5 layers and 768 neurons/layer Uni-LSTM students with 24 million parameters
• Teacher and student models trained with distributed trainer from [33]
• Total training data is 7000 hours with TST2 test data
• Supervised data ranges from 100 hours up to 7000 hours
• The rest of the data is treated as unsupervised data

##  3 SSL for small footprint AM

In this section, we begin with an analysis of acoustic model complexity with
supervised and unsupervised data for smaller and larger model footprints. We
then investigate a method for efficient model distillation in larger
supervised data regimes.

###  3.1 Learning Curves on Large Unsupervised Data

In Figure 2, on TST1 test data, we analyze the learning curves for on-device
and cloud student models, taught by the same teacher network. Accuracy is
reported as relative word error rate reduction (WERR) [26, 7]. Given model A’s
WER ($\text{WER}_{A}$) and a baseline B’s WER ($\text{WER}_{B}$), the WERR of
A over B is computed as
$\text{WERR}=(\text{WER}_{B}-\text{WER}_{A})/\text{WER}_{B}.$

Figure 2: On TST1: relative WERR (%) per sub-epoch of the 1 million hour SSL
models against baseline LSTM AMs that are trained with CE criterion on the
fully supervised 7,000 hour training data.

The vertical axis is the relative WERR (%) against baseline LSTM AMs which are
trained with CE criterion on the fully supervised 7,000 hour training data.
The horizontal axis corresponds to the amount of unsupervised data. Each sub-
epoch in the axis corresponds to about 60,000 hours of unsupervised data,
totaling to 1 million hours. From Figure 2, we see that the relative WERR
improves steadily for both cloud and on-device student AMs as we use
increasing amount of unsupervised data. When using 1 million hours
unsupervised data, the relative WERR for on-device and cloud student AMs are
14.3% and 14.6% respectively in CE comparing with its fully supervised models.

We perform sequence level discriminative training only on the 7,000 hour
supervised dataset, demonstrating that the gains at the CE stage also carry
over to the sMBR stage. Table 2 shows WERR for on-device and cloud student AMs
using the same 4-gram LM. The WERRs for SSL CE + sMBR and supervised CE + sMBR
are 25.7% and 18.2% respectively for on-device AMs.

Table 2: On TST1: relative WERR (%) for sequence training of SSL students. The
baseline LSTM AMs are trained with CE criterion in a fully supervised setting
with 7,000 hours of data WERR (%)
---
System | On-device | Cloud
Baseline supervised CE | 0 | 0
Baseline supervised CE + sMBR | 18.2 | 9.7
SSL CE | 14.3 | 14.6
SSL CE + sMBR | 25.7 | 24.0

###  3.2 Learning Saturation for On-device SSL System

In the following experiment, we increase the supervised data by about 7 times
to 52,000 hours. We then gradually increase the amount of unsupervised data
set to 1 million hours, computing WERR against a supervised model trained with
52,000 hours of supervised data. From Table 3, we get a relative WERR of 6.06%
by adding up to 540 K hours of unsupervised data. If we further increase
unsupervised data from 540 K hours to 1 million hours, the additional relative
WERR is only 1%. We conclude that learning starts to saturate at 540 K hours.
Adding additional unsupervised data helps little with accuracy improvement.

Table 3: On TST1: relative WERR (%) for SSL student AM with 8M parameters. Both the baseline and SSL AM are trained with 52k hours of fully supervised data. WERR(%) | Unsupervised (hours)
---|---
6.05% | 540 K
6.62% | 720 K
6.86% | 900 K
7.05% | 1 M

###  3.3 Improving the Efficiency of Knowledge Distillation

In this section, we study a distillation method to improve the learning
efficiency at higher data regimes. Specifically, we reduce the gap between the
teacher and student models through an intermediate teacher assistant [23].

The supervised part of the training data consists of about 50 K hours of
supervised US English speech, and the unsupervised part consists of about 100
K hours of unsupervised data. The unsupervised training data is used for
training the teacher assistant. We perform step-wise teacher assisted
knowledge distillation. The first-step knowledge distillation happens through
a 78-million-parameter bidirectional LSTM teacher model and a 28-million-
parameter teacher assistant model. The second-step knowledge distillation
occurs between the 28-million-parameter teacher assistant model and an
8-million-parameter student network using only the soft-targets from the
teacher assistant.

Table 4 shows relative WERR against a baseline cross-entropy student model
(row 4) trained with 50,000 hours of supervised data. Rows 1, 2 and 3 show
WERR of the teacher, supervised cloud model, and SSL-trained teacher assistant
models evaluated against baseline. Sequence training with sMBR (in row 4)
shows 17.7% relative WERR in a fully supervised system. The distillation
results are presented in rows 5 and 6. The improvement over direct Knowledge
Distillation (KD) is 9.2% relative. For Teacher Assisted Knowledge
Distillation (TAKD), there is a substantial improvement of 14.4% compared to
baseline supervised system, thus showing that step-wise distillation with a
teacher assistant indeed helps with the efficiency for smaller models in large
data regimes.

Table 4: On TST1: relative WERR (%) for teacher assisted knowledge distillation method. The baseline model is the cross-entropy trained model on supervised data (row 4). WERR values are computed against baseline model’s WER. Index | AM | Params (M) | Stage | WERR (%)
---|---|---|---|---
1 | Teacher | 78 | sMBR | 33.0
2 | Cloud | 28 | sMBR | 21.67
3 | TA | 28 | sMBR | 25.97
4 | On-device | 8 | CE | 0
sMBR | 17.7
5 | KD | 8 | CE | 9.2
sMBR | 15.7
6 | TAKD | 8 | CE | 14.4
sMBR | 19.0

##  4 SSL in Low Resource Settings

In this section, we present our results on low resource settings with SSL;
specifically, now the student model is larger, but the overall amount of data
is restricted to 7,000 hours.

###  4.1 Accuracy Gains With Unsupervised Data

Figure 3 shows learning curves for student models at fixed amount of
supervised data. The actual amounts of supervised data are: 100, 250, 500,
1000, 3500 and 7000 hours. Triangular markers on each curve correspond to
student models trained with increasing amounts of unsupervised data by fixing
the amount of supervised data.

Relative WERR is computed against a baseline model trained only on the
supervised data corresponding to that curve of the same color. The circular
solid dots in this plot show the relative WERR for the teacher models. These
dots line up vertically against the triangles corresponding to the baselines
models.

For each curve, the slope of the curve becomes less steep as the amount of
unsupervised data increases, meaning diminishing returns for additional data.
Note that the student models can outperform the corresponding teacher model,
once the student model observes unsupervised data.

Figure 3: On TST2: relative WERR (%) for different amounts of unsupervised
data: each curve corresponds to a fixed amount of supervised data. Markers on
each curve correspond to student models trained with increasing amounts of
unsupervised data. Relative WERR is computed against the model trained only on
the supervised data corresponding to that curve. The circular dots correspond
to the BLSTM teacher models for the curves of the same color.

###  4.2 An Analysis of Empirical Risk with Student-Teacher Learning

To simplify analysis, we restrict our setting to binary classification, and
adopt notations from [3]. Let $Z=\\{(x_{i},y_{i})\\}_{i=1,\dots,N}$ be a
supervised set drawn from a distribution $D$ on $\mathcal{X}\times\\{-1,1\\}$,
where $\mathcal{X}=\\{x_{i}\\}_{i=1..N}$. Let
$\mathbbm{1}[A]:A\subset\mathcal{X}\to\\{0,1\\}$ be an indicator function
defined as $\mathbbm{1}[A](x)=\left\\{\begin{array}[]{ll}1\quad\text{ if }x\in
A\\\ 0\quad\text{ if }x\notin A\end{array}\right.$ . Teacher and student
network outputs are denoted $h^{t}(x)$ and $h^{s}(x)$ respectively, and are
drawn from $\\{-1,1\\}$. The empirical teacher-student training risk is

$R^{h^{t}}_{Z}(h^{s}):=\frac{1}{N}\sum_{i=1,\dots,N}\mathbbm{1}[h^{t}(x_{i})\neq
h^{s}(x_{i})],$

and the actual empirical risk of the student is

$R_{Z}(h^{s}):=R^{y}_{Z}(h^{s})=\frac{1}{N}\sum_{i=1,\dots,N}\mathbbm{1}[y_{i}\neq
h^{s}(x_{i})].$

######  Lemma 1 (Decomposition)

Given $h^{t},h^{s},y\in\\{-1,1\\}$, we have

$R_{Z}(h^{s})=R_{Z}(h^{t})\big{(}1-R_{acc(h^{t})}^{h^{t}}(h^{s})-R_{err(h^{t})}^{h^{t}}(h^{s})\big{)}+R_{acc(h^{t})}^{h^{t}}(h^{s})$

where $acc(h^{t}):=\\{x,y\in Z|h^{t}(x)=y\\}$ and $err(h^{t}):=\\{x,y\in
Z|h^{t}(x)\neq y\\}$.

###### Proof

Decomposing the student risk by partitioning $Z$ on whether the teacher makes
a mistake

$\displaystyle R_{Z}(h^{s})=$ $\displaystyle
R_{Z}(h^{t})\big{(}1-R_{err(h^{t})}^{h^{t}}(h^{s})\big{)}+\big{(}1-R_{Z}(h^{t})\big{)}R_{acc(h^{t})}^{h^{t}}(h^{s})$
$\displaystyle=$ $\displaystyle
R_{Z}(h^{t})\big{(}1-R_{acc(h^{t})}^{h^{t}}(h^{s})-R_{err(h^{t})}^{h^{t}}(h^{s})\big{)}+R_{acc(h^{t})}^{h^{t}}(h^{s})$

######  Theorem 4.1 (Truth Over Teacher)

$R_{Z}(h^{s})\leq
R_{Z}(h^{t})\iff\frac{R_{err(h^{t})}^{h^{t}}(h^{s})}{R_{acc(h^{t})}^{h^{t}}(h^{s})}\geq\frac{1}{R_{Z}(h^{t})}-1$

###### Proof

$\displaystyle R_{Z}(h^{s})\leq R_{Z}(h^{t})$ $\displaystyle\iff$
$\displaystyle
R_{Z}(h^{t})\big{(}1-R_{acc(h^{t})}^{h^{t}}(h^{s})-R_{err(h^{t})}^{h^{t}}(h^{s})\big{)}+R_{acc(h^{t})}^{h^{t}}(h^{s})\leq
R_{Z}(h^{t})$ $\displaystyle\iff$ $\displaystyle
R_{acc(h^{t})}^{h^{t}}(h^{s})\leq
R_{Z}(h^{t})\big{(}R_{acc(h^{t})}^{h^{t}}(h^{s})+R_{err(h^{t})}^{h^{t}}(h^{s})\big{)}$
$\displaystyle\iff$ $\displaystyle\frac{1}{R_{Z}(h^{t})}\leq
1+\frac{R_{err(h^{t})}^{h^{t}}(h^{s})}{R_{acc(h^{t})}^{h^{t}}(h^{s})}$
$\displaystyle\iff$
$\displaystyle\frac{R_{err(h^{t})}^{h^{t}}(h^{s})}{R_{acc(h^{t})}^{h^{t}}(h^{s})}\geq\frac{1}{R_{Z}(h^{t})}-1$

The better the teacher is, the easier a student model fits the true label
compared to a false label. In particular, if the prediction error the teacher
makes $R_{Z}(h^{t})$ is better than random guess, namely $R_{Z}(h^{t})<0.5$,
then it is necessary that
$\frac{R_{err(h^{t})}^{h^{t}}(h^{s})}{R_{acc(h^{t})}^{h^{t}}(h^{s})}>1$ so
that $R_{Z}(h^{s})\leq R_{Z}(h^{t})$. From Theorem 4.1, for the student to be
better than its teacher, the student’s risk evaluated on the error set of the
teacher (the erroneous teacher labels) has to be greater than the risk
evaluated on the accuracy set of the teacher (the correct teacher labels) by a
factor of $\frac{1}{R_{z}(h^{t})}-1$.

##  5 Discussion

In low data regimes, our experiments surprisingly showed that a student
model’s performance is not upper bounded by the teacher model used in
learning. Our theoretical analysis indicates that this can happen, for
example, when the student makes fewer errors on data with high teacher errors.
We speculate this is because a capacity-restricted student model is able to
generalize better than the teacher using the unsupervised data.

In high data regimes, we observed that the TAKD method facilitated efficient
learning. We speculate that this could be due to step-wise distillation
providing better calibrated posteriors for the eventual student to learn from.
Indeed, it has been observed that in modern over-parameterized neural
networks, posterior probabilities can become less calibrated [8]; using large
amounts of supervised data could be making teacher models’ estimates of
posteriors less calibrated.

##  6 Conclusions And Future Work

Using BLSTM and LSTM as teacher and student models respectively, we studied
SSL for AMs for two tasks: (a) for small footprint on-device models; and (b) a
larger footprint, but lower training data regime. Despite smaller model
capacity, on-device models were able to exploit 1 million hours of
unsupervised data and achieve a 14.3% relative WER improvement after cross-
entropy training and the gains could carry over to the sMBR stage. However,
when we increased the supervised data from 7,000 hours to 52,000 hours, the
learning efficiency decreased yielding only 7.1% relative improvements in WER.
We utilized a step-wise distillation and recovered the WER gains. For SSL in
low data regimes, using knowledge distillation, we found that to achieve a
performance comparable to that of a fully supervised system, the proportion of
required supervised data decreased as the amount of total data increased.
However, we found that in low data regimes, students can be better than
teachers. In future work, we would like to extend this large scale study to
distill using sequence-based discriminative criterion [37, 22, 20] instead of
the frame-level cross-entropy criterion. Finally, it would be informative to
understand if the analyses carried out in this paper using hybrid models would
carry over to end-to-end ASR systems.

##  7 Acknowledgements

We would like to thank Minhua Wu, Jangwon Kim, Srinivas Parthasarathy, Kishore
Nandury and Brian King for their helpful discussions.

## References

  * [1] Amodei, D., Ananthanarayanan, S., et al.: Deep speech 2: End-to-end speech recognition in English and Mandarin. In: Proc. of ICML (2016)
  * [2] Ba, J., Caruana, R.: Do deep nets really need to be deep? In: Advances in neural information processing systems. pp. 2654–2662 (2014)
  * [3] Bégin, L., Germain, P., Laviolette, F., Roy, J.F.: Pac-bayesian theory for transductive learning. In: Proc. of AISTATS (2014)
  * [4] Chen, K., Huo, Q.: Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In: Proc. of ICASSP (2016)
  * [5] Chen, L., Leutnant, V.: Acoustic Model Bootstrapping Using Semi-Supervised Learning. In: Proc. Interspeech 2019. pp. 3198–3202 (2019). https://doi.org/10.21437/Interspeech.2019-2818, http://dx.doi.org/10.21437/Interspeech.2019-2818
  * [6] Chen, Y., Wang, W., Wang, C.: Semi-supervised asr by end-to-end self-training (2020)
  * [7] Garimella, S., Mandal, A., Strom, N., Hoffmeister, B., Matsoukas, S., Parthasarathi, S.H.K.: Robust i-vector based adaptation of DNN acoustic model for speech recognition. In: Proc. of Interspeech (2015)
  * [8] Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. arXiv preprint arXiv:1706.04599 (2017)
  * [9] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [10] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735, https://doi.org/10.1162/neco.1997.9.8.1735
  * [11] Huang, Y., Wang, Y., Gong, Y.: Semi-supervised training in deep learning acoustic model. In: Proc. of Interspeech (2016)
  * [12] Huang, Y., Yu, D., Gong, Y., Liu, C.: Semi-supervised GMM and DNN Acoustic Model Training with Multi-system Combination and Confidence Re-calibration. In: Proc. of Interspeech (2013)
  * [13] Kahn, J., Lee, A., Hannun, A.: Self-training for end-to-end speech recognition. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (May 2020). https://doi.org/10.1109/icassp40776.2020.9054295, http://dx.doi.org/10.1109/ICASSP40776.2020.9054295
  * [14] Kemp, T., Waibel, A.: Unsupervised training of a speech recognizer: recent experiments. In: Proc. of Eurospeech (1999)
  * [15] Kingsbury, B.: Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 3761–3764 (2009). https://doi.org/10.1109/ICASSP.2009.4960445
  * [16] Kurata, G., Audhkhasi, K.: Guiding CTC posterior spike timings for improved posterior fusion and knowledge distillation. CoRR abs/1904.08311 (2019), http://arxiv.org/abs/1904.08311
  * [17] Lamel, L., Gauvain, J.L., Adda, G.: Lightly supervised and unsupervised acoustic model training. Computer Speech & Language (2002)
  * [18] Li, J., Mohamed, A., Zweig, G., Gong, Y.: LSTM time and frequency recurrence for automatic speech recognition. In: 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU). pp. 187–191 (2015). https://doi.org/10.1109/ASRU.2015.7404793
  * [19] Li, J., Zhao, R., Huang, J.T., Gong, Y.: Learning small-size DNN with output-distribution-based criteria. In: Proc. of Interspeech (2014)
  * [20] Li, J., Zhao, R., Sun, E., Wong, J.H.M., Das, A., Meng, Z., Gong, Y.: High-accuracy and low-latency speech recognition with two-head contextual layer trajectory lstm model. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 7699–7703 (2020). https://doi.org/10.1109/ICASSP40776.2020.9054387
  * [21] Ma, J., Matsoukas, S., Kimball, O., Schwartz, R.: Unsupervised training on large amounts of broadcast news data. In: Proc. of ICASSP (2006)
  * [22] Manohar, V., Ghahremani, P., Povey, D., Khudanpur, S.: A teacher-student learning approach for unsupervised domain adaptation of sequence-trained asr models. In: 2018 IEEE Spoken Language Technology Workshop (SLT). pp. 250–257 (2018). https://doi.org/10.1109/SLT.2018.8639635
  * [23] Mirzadeh, S.I., Farajtabar, M., Li, A., Ghasemzadeh, H.: Improved knowledge distillation via teacher assistant: Bridging the gap between student and teacher. arXiv preprint arXiv:1902.03393 (2019)
  * [24] Moriya, T., Kanagawa, H., Matsui, K., Fukutomi, T., Shinohara, Y., Yamaguchi, Y., Okamoto, M., Aono, Y.: Efficient building strategy with knowledge distillation for small-footprint acoustic models. In: 2018 IEEE Spoken Language Technology Workshop (SLT). pp. 21–28 (2018). https://doi.org/10.1109/SLT.2018.8639545
  * [25] Munim, R.M., Inoue, N., Shinoda, K.: Sequence-level knowledge distillation for model compression of attention-based sequence-to-sequence speech recognition. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6151–6155. IEEE (2019)
  * [26] Parthasarathi, S.H.K., Hoffmeister, B., Matsoukas, S., Mandal, A., Strom, N., Garimella, S.: fmllr based feature-space speaker adaptation of DNN acoustic models. In: Proc. of Interspeech (2015)
  * [27] Parthasarathi, S.H.K., Sivakrishnan, N., Ladkat, P., Strom, N.: Realizing petabyte scale acoustic modeling. IEEE Journal on Emerging and Selected Topics in Circuits and Systems (2019)
  * [28] Parthasarathi, S.H.K., Strom, N.: Lessons from building acoustic models from a million hours of speech. In: Proc. of ICASSP (2019)
  * [29] Pundak, G., Sainath, T.: Lower frame rate neural network acoustic models. In: Proc. of Interspeech (2016)
  * [30] Sak, H., Senior, A., Rao, K., Beaufays, F.: Fast and accurate recurrent neural network acoustic models for speech recognition. In: INTERSPEECH (2015)
  * [31] Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: Vision and challenges. IEEE internet of things journal (2016)
  * [32] Siu, M.H., Gish, H., Richardson, F.: Improved estimation, evaluation and applications of confidence measures for speech recognition. In: Proc. of European Conference on Speech Communication and Technology (1997)
  * [33] Strom, N.: Scalable distributed DNN training using commodity GPU cloud computing. In: Proc. of Interspeech (2015)
  * [34] Watanabe, S., Hori, T., Le Roux, J., Hershey, J.R.: Student-teacher network learning with enhanced features. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5275–5279 (2017). https://doi.org/10.1109/ICASSP.2017.7953163
  * [35] Waters, A., Chebotar, Y.: Distilling knowledge from ensembles of neural networks for speech recognition. In: Interspeech (2016)
  * [36] Weninger, F., Mana, F., Gemello, R., Andres-Ferrer, J., Zhan, P.: Semi-supervised learning with data augmentation for end-to-end asr (2020)
  * [37] Wong, J.H.M., Gales, M.: Sequence student-teacher training of deep neural networks. In: INTERSPEECH (2016)
  * [38] Xie, Q., Dai, Z., Hovy, E., Luong, M.T., Le, Q.V.: Unsupervised data augmentation for consistency training (2020)
  * [39] Yanzhang, H., Sainath, T., Prabhavalkar, R., McGraw, I., Alvarez, R., Zhao, D., Rybach, D., Kannan, A., Wu, Y., Pang, R., Liang, Q., Bhatia, D., Shangguan, Y., Li, B., Pundak, G., Sim, K.C., Bagby, T., Chang, S., Rao, K., Gruenstein, A.: Streaming end-to-end speech recognition for mobile devices. arXiv preprint arXiv:1811.06621 (2018)
  * [40] Yu, W., Liang, F., He, X., Hatcher, W.G., Lu, C., Lin, J., Yang, X.: A survey on the edge computing for the internet of things. IEEE access (2017)
  * [41] Zhang, Y., Qin, J., Park, D.S., Han, W., Chiu, C.C., Pang, R., Le, Q.V., Wu, Y.: Pushing the limits of semi-supervised learning for automatic speech recognition (2020)