# Directional Privacy for Deep Learning

Pedro Faustini, Natasha Fernandes, Annabelle McIver, Mark Dras
pedro.arrudafaustini@hdr.mq.edu.au,natasha.fernandes@mq.edu.au,annabelle.mciver<EMAIL_ADDRESS>School of Computing, Macquarie UniversityAustralia

(2023)

###### Abstract.

Differentially Private Stochastic Gradient Descent (DP-SGD) is a key method
for applying privacy in the training of deep learning models. This applies
isotropic Gaussian noise to gradients during training, which can perturb these
gradients in any direction, damaging utility. Metric DP, however, can provide
alternative mechanisms based on arbitrary metrics that might be more suitable.
In this paper we apply directional privacy, via a mechanism based on the von
Mises-Fisher (VMF) distribution, to perturb gradients in terms of angular
distance so that gradient direction is broadly preserved. We show that this
provides $\varepsilon d$-privacy for deep learning training, rather than the
$(\varepsilon,\delta)$-privacy of the Gaussian mechanism; and that
experimentally, on key datasets, the VMF mechanism can outperform the Gaussian
in the utility-privacy trade-off.

differential privacy, directional privacy, dp-sgd, deep learning

††copyright: none††journalyear: 2023††doi: XXXXXXX.XXXXXXX††conference:
Proceedings on Privacy Enhancing Technologies; July 10–14, 2023; Lausanne,
Switzerland

##  1\. Introduction

With the current power of machine learning, it is possible to discover
information about the items in a training set with high accuracy: for example,
Fredrikson et al. (2015) showed that a model inversion attack, using
information about prediction confidence from machine learning APIs, could
accurately reconstruct images from a facial recognition training set; Zhu et
al. (2019) showed that the same was possible using information only from
gradients in the training process, for various well-known computer vision
datasets.

Because of this possibility, Differential Privacy (DP) (Dwork and Roth, 2014)
has been applied to the training of deep neural networks, in particular to
Stochastic Gradient Descent (SGD) in the form of DP-SGD, initially proposed by
Song et al. (2013). DP-SGD in its original form applies $\varepsilon$-DP noise
to the gradient vectors within each batch, thereby providing an $\varepsilon$
guarantee over training datasets. Song et al. noted that the noise introduced
by a DP mechanism impacted SGD performance significantly, but later
developments have improved its performance: for example, Abadi et al. (2016)
proposed a Gaussian-based mechanism with a moments accounting method for
tighter bounds on the privacy budget; in the space of language models, McMahan
et al. (2018) showed how to use DP for user-level privacy at the cost of
increased computation rather than decreased utility; also in that space, Li et
al. (2022) showed that very different regions of the hyperparameter space
relative to non-private models, and a new ‘ghost clipping’ technique on
gradients, could lead to strong performance of large language models under DP-
Adam, an extension of DP-SGD. Nevertheless, there is still generally a gap in
performance between non-private and private models.

DP-SGD is based on the standard version of DP as defined in Dwork and Roth
(2014). That is, it provides guarantees wrt adjacent training datasets (ie.
differing by one element, whether in terms of example-level elements, user-
level or other). The most popular noise-adding mechanism used in DP-SGD is the
Gaussian mechanism, introduced by Abadi et al. (Abadi et al., 2016) which
applies Gaussian noise to gradients of deep learning models during training.
However, this Gaussian noise is isotropic: that is, the noise is equally
likely to point in any direction in the high-dimensional space of the deep
learning model gradients. In contrast, we might expect that the utility of
gradient outputs (and therefore the model) would be better served by a
mechanism which is designed to preserve the _direction_ of the gradients. Such
mechanisms arise in the context of _metric differential privacy_ or
$d$-privacy (Chatzikokolakis et al., 2013), a generalisation of differential
privacy in which the notion of adjacency is relaxed to a notion of privacy
within a radius defined by a metric $d$. Metric DP generalises both standard
DP and local DP, with the former recoverable using Hamming metric on datasets
as the metric and the latter using the Discrete metric on individual data
points. In the DP-SGD context, metric DP can allow the definition of
potentially better mechanisms based on metrics other than the implicit Hamming
metric of standard DP.

Guided by metric DP, a natural alternative mechanism to apply to gradients is
one which preserves angular distance. Recently, one such directional privacy
mechanism has been developed by Weggenmann and Kerschbaum (2021), who applied
the idea to recurrent temporal data in the context of a dataset of sleep
recordings. Weggenmann and Kerschbaum (2021) provide two novel DP mechanisms
for their directional privacy, based on the von Mises-Fisher and Purkayastha
distributions. Importantly, these mechanisms provide $\varepsilon d$-privacy
guarantees rather than $(\varepsilon,\delta)$-DP, which means that their
composition properties are more straightforward ($d$-privacy composition
follows standard DP composition) and they do not require complex reasoning
using a moments accountant (Mironov, 2017). The key idea in this paper is to
define analogous mechanisms to apply to gradients in the context of DP-SGD, so
that with high likelihood a reported gradient is close in direction to the
original gradient and with diminishing likelihood directions further away. The
aim of the present paper is to show that these kinds of directional privacy
mechanisms applied to deep learning training can have less impact on model
performance (because the application of noise can be more targeted) while
providing $\varepsilon d$-privacy guarantees.

In this paper, we define a model DirDP-SGD and corresponding privacy mechanism
for applying directional privacy to gradients in deep learning training (§3).
This mechanism comes with a (metric) DP guarantee; however, as we discuss, it
is not straightforward to compare the privacy budgets of the Gaussian
mechanism of DP-SGD with DirDP-SGD. We therefore provide experimental
comparisons of both privacy and utility of DirDP-SGD and DP-SGD (§4), where
the experimental evaluation of privacy is based on recent methods that permit
the reconstruction of training set data based on gradients during training
(Zhu et al., 2019; Geiping et al., 2020; Wei et al., 2020). We show (§5) that
DirDP-SGD performs notably better on some major datasets for comparable levels
of defence against reconstruction attacks.

Our contributions in this paper, then, are:

  * •

applying for the first time a metric DP mechanism based on angular distance —
via the von Mises-Fisher distribution — to use as an alternative to Gaussian
noise in training via Stochastic Gradient Descent in deep learning;

  * •

demonstrating that this provides $\varepsilon d$-privacy rather than
$(\varepsilon,\delta)$-privacy for the training as a whole; and

  * •

showing that on major datasets, this outperforms Gaussian noise in defending
against gradient-based reconstruction attacks.

##  2\. Related Work

In this section, we review relevant work on the use of DP in deep learning
(§2.1); on metric DP, including its own intersections with deep learning
(§2.2); and, because we carry out an empirical evaluation of privacy, on
gradient-based reconstruction attacks (§2.3).

###  2.1. DP in Deep Learning

Neural networks can be victims of several types of attacks, like membership
inference (Rahman et al., 2018; Mukherjee et al., 2021), model stealing (Yu et
al., 2020) and data reconstruction (Zhu et al., 2019; Zhao et al., 2020;
Geiping et al., 2020; Wei et al., 2020). This motivates the need for privacy
guarantees to protect neural networks, while keeping their utility for the
task they are trained to deal with.

Song et al. (2013) proposed Differentially Private Stochastic Gradient Descent
(DP-SGD), which first brought DP to the training of gradient-descent models.
DP-SGD works by adding calibrated noise in the gradients during training,
before updating the parameters. This was followed by work that looked at
providing efficient algorithms and tightening error bounds (Bassily et al.,
2014, for example), so that the addition of noise would not degrade utility to
impractical levels. A key work in this direction was made by Abadi et al.
(2016), who introduced a technique to keep track of the privacy budget, called
the Moments Accountant, specifically for the Gaussian mechanism.

Afterwards, several papers studied the effect of DP in deep learning in other
domains, such as NLP (McMahan et al., 2018), and in applications like
Generative Adversarial Networks (Xu et al., 2019; Torkzadehmahani et al.,
2019). Recent work has also returned to the possibility of feasibly applying
DP through output perturbations (Lu et al., 2022). The many ways in which DP
has been applied in deep learning in general are beyond the scope of the
present work, and we refer the reader to surveys such as Gong et al. (2020);
below we focus only on DP-SGD and related methods.

In this context, the additional privacy comes with a cost, in that the noisy
gradients may affect the utility of the model. Therefore, either better
features may be collected or handcrafted, or even more data may be needed
(Tramer and Boneh, 2021). Li et al. (2022) (in NLP) and De et al. (2022) (in
computer vision) also found that DP-SGD can perform well in very different
regions of the hyperparameter space relative to non-private models. The
architecture of the model may also play a role for the utility, with larger
and pretrained models being more efficiently fine-tuned, especially with
larger batch sizes (Li et al., 2022; Anil et al., 2021), which can be
computationally demanding; Li et al. (2022) also showed how to reduce the high
memory consumption for training via ‘ghost clipping’.

Proposals to change the DP-SGD algorithm itself have also been made, many of
them relating to clipping strategies. Xu et al. (2021) observed that clipping
and noise addition affect underrepresented classes, making the accuracy of the
model for them even lower. Thus they proposed to control the contribution of
samples in a group according to the group clipping bias. Liu et al. (2021)
proposed to divide gradients from $m$ samples into $k$ groups. Before noise is
added, the gradients in each group are clipped with a different bound, as
opposed to a global bound from DP-SGD. They argue that a clipping could
distort gradient information.

However, all these works in DP and deep learning have adopted isotropic noise,
mostly from the Gaussian and Laplace distributions. Clipping the gradients
derived from these noises limits their _length_ , but does not alter their
_direction_. There is a lack of studies comparing how different noise
distributions affect the privacy/utility tradeoff and how noise distributions
other than isotropic ones can be used during the training of neural networks.

###  2.2. Metric Differential Privacy

There have been many variants of DP proposed in the literature (Pejó and
Desfontaines, 2022). In this work we adopt a relaxation of DP called metric
differential privacy (hereafter metric DP), introduced by Chatzikokolakis et
al. (Chatzikokolakis et al., 2013) and also known as generalised DP,
$d$-privacy, and $d_{\mathcal{X}}$-privacy.

Metric DP is motivated by the observation that secrets which are closer
together (wrt the metric $d$) are more “indistinguishable” than secrets which
are further apart, and therefore can be made “more private” using the same
amount of noise. Thus the level of indistinguishability between secrets can be
reported over a radius $r$, rather than reporting a single $\varepsilon$ value
for the entire domain. Note that on bounded domains the $\varepsilon$ for the
domain can be recovered by choosing $r$ to be the sensitivity of the query (in
the case of standard DP), or the size of the domain (in the case of local DP).

Metric DP was first applied to the problem of geo-location privacy (Andrés et
al., 2013). In this application, the goal is to conceal information about an
exact location (‘geo-indistinguishability’) while allowing the release of an
approximate location, e.g. in the case of location-based service provision. In
this scenario, $d$ was chosen to be the Euclidean distance and the
corresponding mechanism implemented using two-dimensional Laplace noise. Many
later applications of metric DP have been in this kind of geolocation context,
for example, mobility tracing (Chatzikokolakis et al., 2014), location data
with temporal correlations (Xiao and Xiong, 2015), mobile crowdsensing (Wang
et al., 2018), and location data with non-circular distributional
characteristics (Zhao et al., 2022).

In the area of deep learning in NLP, Fernandes et al. (2019) observed that
learned representations in $n$-dimensional space (word embeddings) could be
seen as analogous to locations in geo-location privacy, and proposed a metric
DP mechanism for authorship privacy using the Earth Mover’s distance as the
metric. Work following on from that used hyperbolic rather than Euclidean
spaces for hierarchical representations (Feyisetan et al., 2019), calibrated
multivariate perturbations (Feyisetan et al., 2020), representations for
contextual rather than static language models (Qu et al., 2021), and a
variational autoencoder to perturb overall latent vectors rather than
individual words (Weggenmann et al., 2022). A related application that takes a
similar spatial perspective has been to k-means clustering (Yang et al.,
2022). None of these are concerned with differentially private training of a
deep learner in the manner of DP-SGD.

The application of metric DP that we draw on is not related to the existing
uses in deep learning just described. In the context of providing privacy
guarantees to sleep study data, Weggenmann and Kerschbaum (2021) apply metric
DP to periodic data such as activity that occurs at a particular time of day
or day of the week, by noting that periodicity can be represented as a
direction on a circle, and ‘directional noise’ perturbs this direction while
preserving utility. They proposed a variety of privacy mechanisms, including
variants of Laplace, plus the novel Purkayastha and von Mises-Fisher (VMF)
mechanisms. Weggenmann and Kerschbaum (2021) also provide e.g. sampling
methods for VMF that reduce from a multivariate sampling problem to a
univariate one in order to avoid the curse of dimensionality. In the present
work we adopt the VMF mechanism to apply directional noise to gradients
instead of (isotropic) Gaussian noise more typically used in DP-SGD that
perturbs the gradient in any direction, drawing on a similar intuition that
preserving the gradient directions should provide better utility.

###  2.3. Gradient-based Reconstruction Attacks

Distributed training aims to train a neural network without centralising data.
It has the benefit of not having to hold private data in a single place. It
consists of multiple clients, and each one holds their own private training
set. Instead of sharing the data, the clients train their neural network and
exchange the gradients. However, it is still possible to reconstruct the
private training data from the gradients received.

The seminal study of Zhu et al. (2019) discovered that with few iterations it
is possible to recover the private data in attacking neural network
architectures which are twice differentiable; their attack has subsequently
been referred to as the Deep Leakage from Gradients (DLG) attack. The attacker
creates dummy inputs and labels, but instead of optimising the model weights,
they optimise the dummy input and labels to minimise the Euclidean distance
between their gradients and the gradients received from another client.
Matching the gradients transforms the fake input to be similar to the real
one.

This attack was refined in further works. Zhao et al. (2020) proposed iDLG
(_i_ stands for _improved_), which works against any differentiable network
trained with cross-entropy loss over one-hot labels. The Inverting Gradients
method (IGA), from Geiping et al. (2020), maximises the cosine similarity
between gradients. Thus it relies on an angle-based cost function, which
should be more robust than a magnitude-based one against a trained neural
network (which produces gradients with smaller magnitudes). Finally, Wei et
al. (2020) study (baptised Client Privacy Leakage — CPL) how different
configurations impact the effectiveness of the attack, such as different ways
of initialising the dummy data.

Zhu et al. (2019), in proposing DLG, also proposed some suggestions for
possible defences. In addition to measures like gradient quantization and
compression / sparsification, it also included the addition of Gaussian noise
to gradients, although not within a DP context. Recently, Scheliga et al.
(2022) proposed a variational bottleneck-based preprocessing module that aims
to disguise the original latent feature space that is vulnerable to gradient-
based reconstruction attacks, by learning a joint distribution between input
data and latent representation. Like Zhu et al. (2019), this also does not
come with differentially private guarantees.

##  3\. The Privacy Model

A differential privacy mechanism can be described formally as a function which
takes as input an element (drawn from a domain $\mathcal{X}$) and produces a
randomised value drawn from some distribution over outputs $\mathcal{Y}$,
satisfying the characteristic DP inequation:

(1) $Pr({\mathcal{M}}(x))[Y]\leq e^{\varepsilon}\times
Pr({\mathcal{M}}(x^{\prime}))[Y]~{},$

whenever $x\sim x^{\prime}\in\mathcal{X}$ and $Y\subseteq\mathcal{Y}$.

Popular methods of randomisation include the Gaussian, the Laplace (when the
outputs are continuous) or the Geometric (when the outputs are discrete), all
of which involve the addition of noise to the input $x\in\mathcal{X}$ to
produce the noisy output $y\in\mathcal{Y}$.

Metric differential privacy describes a constraint on the type of
randomisation and its ability to make different inputs to the mechanism
indistinguishable when comparing their outputs.

###### Definition 3.1.

(Metric differential privacy) (Chatzikokolakis et al., 2013) Let
$\varepsilon{>}0$. A mechanism ${\mathcal{M}}$ on an (input) metric space
$(S,d)$, where $S$ is a set and $d$ is a metric, and producing outputs over
$\mathcal{Z}$, satisfies $\varepsilon d$-privacy, if for all $s,s^{\prime}\in
S$ and $Z\subseteq\mathcal{Z}$,

$Pr({\mathcal{M}}(s))[Z]\leq e^{\varepsilon d(s,s^{\prime})}\times
Pr({\mathcal{M}}(s^{\prime}))[Z]~{},$

where $Pr({\mathcal{M}}(s))[Z]$ means the probability that the output of
applying mechanism $\mathcal{M}$ to $s$ lies in $Z$.

Definition 3.1 says that when two inputs $s,s^{\prime}$ differ by the amount
$d(s,s^{\prime})$, the mechanism can make then indistinguishable up to a ratio
proportional to $e^{\varepsilon d(s,s^{\prime})}$. This means that when points
are farther apart they become harder to be made indistinguishable.

This kind of privacy definition is useful when the utility can be captured by
the metric $d$. In location privacy, the utility is the approximate location
which should be preserved as much as possible. Figure 1 depicts the
2-dimensional Laplace probability density function that can be used to
implement location privacy (Chatzikokolakis et al., 2013) and which can be
shown to satisfy $d_{2}$-privacy where $d_{2}$ is the Euclidean distance. In
this case, locations that are far apart do not need to be made
indistinguishable, whereas locations close together do.

In our application to deep learning we use a metric based on angular distance
of vectors, which we describe in the next sections.

This illustrates the probability density function of the 2-dimensional Laplace
distribution which can be used to implement location privacy.

Figure 1. Laplacian in 2-dimensions satisfying $\varepsilon d_{2}$-privacy on
locations

###  3.1. Standard DP-SGD

The standard DP-SGD from Abadi et al. (2016) is shown in Algorithm 1. It
differs from the original stochastic gradient descent (i.e. without
perturbation) only at lines 10 and 13, where the gradients $g_{t}(x_{i})$ are
first clipped and then perturbed using the Gaussian distribution. This is
implemented essentially by adding a random perturbation to each of the
components of the gradient when represented as a point in ${\mathbb{R}}^{K}$.

Algorithm 1 DP-SGD with Gaussian noise

1:Input: Examples $\\{x_{1},\ldots,x_{N}\\}$, loss function
$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i}\mathcal{L}(\theta,x_{i})$.
Parameters: learning rate $\eta_{t}$, noise scale $\sigma$, group size $L$,
gradient norm bound $C$.

2:Initialise $\theta_{0}$ randomly

3:for $t\in T$ do

4: Take a random batch

5: $L_{t}\leftarrow$ random sample of $L$ indices from $1{\ldots}N$

6: for $i\in L_{t}$ do

7: Compute gradient vector

8:
$\mathbf{g}_{t}(x_{i})\leftarrow\nabla_{\theta_{t}}\mathcal{L}(\theta_{t},x_{i})$

9: Clip gradient vector

10:
$\overline{\mathbf{g}}_{t}(x_{i})\leftarrow\nicefrac{{\mathbf{g}_{t}(x_{i})}}{{\max(1,\frac{\|\mathbf{g}_{t}(x_{i})\|_{2}}{C})}}$

11: end for

12: Add noise

13:
$\tilde{\mathbf{g}}_{t}\leftarrow\frac{1}{L}\sum_{i}(\overline{\mathbf{g}}_{t}(x_{i})+\mathcal{N}(0,\sigma^{2}))$

14: Descent

15: $\theta_{t+1}\leftarrow\theta_{t}-\eta_{t}\tilde{\mathbf{g}}_{t}$

16:end for

17:Output $\theta_{T}$

Song et al. (2013) showed that by adding a noisy vector in $\mathbb{R}^{K}$
drawn from a distribution $\rho(z)\propto e^{\frac{\varepsilon}{2}\lVert
z\rVert}$ one obtains $\varepsilon$-DP as per Eqn 1. In particular, from line
13 we can compute the “sensitivity” of the gradient computation as the maximum
difference between gradients for batches $B$, $B^{\prime}$. The clipping step
on line 10 ensures the gradient norm is at most $C$. Therefore the sensitivity
$\Delta_{g}$ is calculated as:

$\displaystyle\Delta_{g}$ $\displaystyle~{}=~{}\max_{B\sim
B^{\prime}}(\tilde{\mathbf{g}}_{t}-\tilde{\mathbf{g}}^{\prime}_{t})$
$\displaystyle~{}=~{}\max_{B\sim
B^{\prime}}(\frac{1}{L}\sum_{i}\overline{\mathbf{g}}_{t}(x_{i})-\frac{1}{L}\sum_{i}\overline{\mathbf{g}}_{t}(x^{\prime}_{i}))$
(2) $\displaystyle~{}=~{}\frac{2C}{L}$

Here the maximum is taken over all pairs of gradients in $B,B^{\prime}$. The
sensitivity of the final computation of weights on line 15 is then
$\Delta_{\theta}=\frac{2C\eta_{t}}{L}$. By choosing $C=1$ and observing that
the scalars $\eta_{t}$ and $L$ satisfy a distributive law in eg. the Laplace
distribution 111ie. if $X\sim Lap(\mu,b)$ then $\alpha X\sim
Lap(\alpha\mu,\alpha b)$ for $\alpha>0$., we arrive at Song et al.’s
$\frac{\varepsilon}{2}$ noise tuning.

Abadi et al. (2016) use Gaussian noise in Algorithm 1 to arrive at an
approximate DP-guarantee. ie.,

(3) $Pr({\mathcal{G}}(B)\in Z)\leq Pr({\mathcal{G}}(B^{\prime})\in Z)\times
e^{\varepsilon}+\delta~{},$

where $\varepsilon\geq 0$ and $0<\delta<1$ are privacy parameters and $Z$ is a
(measurable) set of potential outputs of (perturbed) gradients. In this case,
the moments accountant method was developed by Abadi et al. (2016) to produce
tighter bounds for epsilon under composition.

###  3.2. Directional Privacy and DirDP-SGD

Gradient descent optimises the search for parameter selection that minimises
the loss. Thus an alternative method of perturbing the gradients is to use
randomisation that is based around perturbing the angle of deviation from the
original gradient. To give some intuition, Figure 2 illustrates how a gradient
of a convex curve can be perturbed, leading to a perturbation of the descents.

Algorithm 2 DirDP-SGD with von Mises-Fisher noise

1:Input: Examples $\\{x_{1},\ldots,x_{N}\\}$, loss function
$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i}\mathcal{L}(\theta,x_{i})$.
Parameters: learning rate $\eta_{t}$, noise scale $\sigma$, group size $L$,
gradient norm bound $C$.

2:Initialise $\theta_{0}$ randomly

3:for $t\in T$ do

4: Take a random batch

5: $L_{t}\leftarrow$ random sample of $L$ indices from $1{\ldots}N$

6: for $i\in L_{t}$ do

7: Compute gradient vector

8:
$\mathbf{g}_{t}(x_{i})\leftarrow\nabla_{\theta_{t}}\mathcal{L}(\theta_{t},x_{i})$

9: Scale gradient vector

10:
$\overline{\mathbf{g}}_{t}(x_{i})\leftarrow\nicefrac{{\mathbf{g}_{t}(x_{i})}}{{\frac{\|\mathbf{g}_{t}(x_{i})\|_{2}}{C}}}$

11: end for

12: Add noise

13:
$\tilde{\mathbf{g}}_{t}\leftarrow\frac{1}{L}\sum_{i}\mathcal{V}(\sigma,\overline{\mathbf{g}}_{t}(x_{i}))$

14: Descent

15: $\theta_{t+1}\leftarrow\theta_{t}-\eta_{t}\tilde{\mathbf{g}}_{t}$

16:end for

17:Output $\theta_{T}$

Given two vectors $v,v^{\prime}$ in $\mathbb{R}^{K}$, we define the angular
distance between them as
$d_{L}(v,v^{\prime})=\frac{\arccos{v^{T}v^{\prime}}}{\|v\|\|v^{\prime}\|}$.
When $v,v^{\prime}$ are, for example, vectors on the unit $K$-dimensional
sphere, then $d_{L}$ becomes a metric. Following Weggenmann et al. (Weggenmann
and Kerschbaum, 2021), we can use this to define _directional privacy_.

###### Definition 3.2.

(Directional Privacy) (Weggenmann and Kerschbaum, 2021) Let $\varepsilon{>}0$.
A mechanism ${\mathcal{M}}$ on $\mathbb{R}^{K}$ satisfies $\varepsilon
d_{L}$-privacy, if for all $v,v^{\prime}$ and
$Z\subseteq\textrm{supp}\mathcal{M}$,

$Pr({\mathcal{M}}(v))[Z]\leq e^{\varepsilon d_{L}(v,v^{\prime})}\times
Pr({\mathcal{M}}(v^{\prime}))[Z]~{}.$

Definition 3.2 says that when the mechanism ${\mathcal{M}}$ perturbs the
vectors $v,v^{\prime}$, the probabilities that the perturbed vectors lie
within a (measurable) set $Z$ differ by a factor of $e^{\varepsilon
d_{L}(v,v^{\prime})}$. This means that the smaller the angular distance
between the vectors $v,v^{\prime}$ the more likely it will be that these
probabilities will be almost the same, providing a high degree of
indistinguishability.

The von Mises-Fisher (VMF) mechanism perturbs an input vector $\mu$ on the
$K$-dimensional unit sphere as follows:

###### Definition 3.3.

The VMF mechanism on the $K$-dimensional unit sphere, is given by the density
function:

$\mathcal{V}(\varepsilon,\mu)(x)~{}=~{}C_{K}(\varepsilon)e^{\varepsilon\mu^{T}x}~{},$

where $\varepsilon>0$ and $C_{K}(\varepsilon)$ is the normalisation factor.

Weggenmann and Kerschbaum (2021) showed that the VMF mechanism of Def. 3.3
satisfies $\varepsilon d_{L}$-privacy.

We can use Def. 3.3 to design a new algorithm for DP-SGD based on the VMF
distribution, displayed in Algorithm 2. Note that unlike Gaussian noise, the
VMF mechanism generates a noisy vector based on an input vector (line 13).
Secondly, since the VMF guarantee is for vectors on a $K$-dimensional sphere,
we scale gradients to a constant length $C$ rather than clipping them (line
10). This also ensures that the lengths of the gradients do not leak privacy.
Observe also that $C$ does not affect the privacy guarantee on angles.

It turns out that metric DP satisfies useful compositional properties (N.
Fernandes and Ding, 2022) making its guarantees carry over to algorithms that
apply it, and in particular to Algorithm 2. We write ${\mathcal{V}M}(B)$ for
the VMF mechanism applied to the vectors in the batch $B$, and then averaged
at Line 13 in Algorithm 2. The following lemma shows that this step satisfies
directional privacy over batches.

###### Theorem 3.4.

Denote by $B=[v_{1},\dots v_{n}]$ a batch of vectors (gradients). If batch
$B^{\prime}$ differs from $B$ in at most one component vector, then Algorithm
2 satisfies $\varepsilon d_{L}$-privacy wrt. batches, namely that:

(4) $Pr({\mathcal{V}M}(B)\in Z)\leq Pr({\mathcal{V}M}(B^{\prime})\in Z)\times
e^{\varepsilon d_{L}(B,B^{\prime})}~{},$

where $Z$ is a (measurable) set of vectors, $Pr({\mathcal{V}M}(B)\in Z)$ is
the probability that the output vector lies in $Z$ and (abusing notation)
$d_{L}(B,B^{\prime})=\max_{B\sim B^{\prime}}d_{L}(v,v^{\prime})$ is the
maximum angular distance between all pairs $v\in B,v^{\prime}\in B^{\prime}$.

###### Proof.

The difference between the mechanism in Definition 3.3 and its application in
Algorithm 2 is that Definition 3.3 is applied to every vector in
$B,B^{\prime}$, and then averaged. The constraint Equation 4 follows because
this process is equivalent to the parallel composition of Definition 3.3
applied across a batch followed by postprocessing to form the average. The
logical properties of metric DP (N. Fernandes and Ding, 2022) ensure that the
parallel composition preserves metric differential privacy, as does post-
processing. ∎

Using Eqn 3.1 and following Song et al. (2013), we can tune the noise to
$\frac{\varepsilon}{2}$ to achieve an overall $\varepsilon d_{L}$-privacy
guarantee.

###  3.3. Notion of Theoretical Guarantees and Comparison in Practice

At this point it is not clear how directly we can compare the two privacy
guarantees for Algorithm 1 and Algorithm 2. As mentioned above the guarantee
for Algorithm 1 includes a $\delta>0$ parameter — this means that there is a
risk that the perturbation will leak more than for an $\varepsilon$-private
mechanism, and therefore may provide reduced protection against a threat of
reconstruction. Moreover, previous work (Chatzikokolakis et al., 2019) has
shown that comparing epsilons between different privacy mechanisms can be
problematic.

Due to these differences it is difficult to compare these two notions, and in
particular the $\varepsilon$-parameters in both constraints cannot be compared
at all. This is because of the nature of the randomisation (i.e. Gaussian
versus VMF) which provides incomparable guarantees (metric DP versus
approximate DP). To avoid confusion we use $\varepsilon_{G}$ for the privacy
parameter used for Algorithm 1 and $\varepsilon_{V}$ for Algorithm 2.

For these reasons, we evaluate the privacy afforded by these different
mechanisms by comparing the ability of these two methods of randomisation to
defend against gradient-based reconstruction attacks, which is the primary
privacy vulnerability of concern in our deep learning context. Our privacy
evaluation therefore avoids the difficult task of comparing epsilons. We
simultaneously empirically compare each mechanism’s utility on a
classification task. Although Algorithm 1 has been widely used, Algorithm 2 is
a novel application of the VMF mechanism, and one of our tasks (detailed
below) is to determine ranges of the parameter $\varepsilon_{V}$ that provide
a good trade-off between defending against the threat of reconstruction versus
allowing a set of parameters to be determined that provide an acceptable level
of utility.

This illustrates how the gradients are perturbed during the DP-SGD. The red
line is the unperturbed gradient, and the dotted blue lines are perturbations
of angular distance $A$.

Figure 2. Perturbed gradients

Each gradient from Figure 2 corresponds to a descent in SGD. In this case the
unperturbed descent would lead to the initial point (green circle) updated to
the red circle. The perturbed descents (the blue circle) are chosen with equal
probability, and lead either to a smaller or larger descent. Depending on the
curvature of the particular loss function (here represented as a parabola) can
lead to a loss of utility in finding the best model parameters.

Figure 3. Perturbed descents

###  3.4. Implementing Directional Privacy for Gradients

We use Opacus,222https://opacus.ai introduced by Yousefpour et al. (2021), as
a starting point for the experiments. The library, based on PyTorch (Paszke et
al., 2019), implements DP-SGD.

From an implementation view, there are three main components: (i) the
minibatches are built by using Poisson sampling: each sample from the training
dataset is picked with a certain probability $p$, which means that a sample
may appear in zero, or more than one times in an epoch; (ii) the sample
gradients are capped to avoid a very large individual contribution from one
sample; (iii) noise is added to the gradients. Only Gaussian noise is
supported.

We extend Opacus to work with the VMF distribution. Component (i) is left
unchanged. For component (ii), we cap gradients, bounding them by C. It means
that, instead of clipping the gradients according to the original formulation
of Abadi et al. (2016), from

$\overline{\mathbf{g}}_{t}\left(x_{i}\right)\leftarrow\mathbf{g}_{t}\left(x_{i}\right)/\max\left(1,\frac{\|\mathbf{g}_{t}(x_{i})\|_{2}}{C}\right)$

we remove the $\max$ operator

$\overline{\mathbf{g}}_{t}\left(x_{i}\right)\leftarrow\mathbf{g}_{t}\left(x_{i}\right)/\frac{\|\mathbf{g}_{t}(x_{i})\|_{2}}{C}$

as described in Algorithm 2.

Finally, for component (iii), we switch the Gaussian noise for the Von Mises-
Fisher one.

##  4\. Experimental Setup

Most works evaluating DP in deep learning report performance on some task
(typically, classification accuracy) for utility, but for the level of
privacy, they report only on the value of $\varepsilon$ (and $\delta$ if
relevant). As we note in §3.3, it is not possible to compare epsilons across
standard DP and metric DP. We therefore take an empirical approach to
calibrating the respective epsilons, $\varepsilon_{G}$ and $\varepsilon_{V}$.

It is well known that defining an operational interpretation of $\varepsilon$
is a challenge, with the meaning of $\varepsilon$ being contextually dependent
(Lee and Clifton, 2011; Dwork et al., 2019). We therefore choose a framework
that directly relates to the approach taken by DP-SGD and our DirDP-SGD, which
obfuscates gradients: we compare their success against gradient-based
reconstruction attacks (Zhu et al., 2019; Geiping et al., 2020). In these
attacks, the goal is to reconstruct images solely from their gradients.
Obfuscating gradients successfully should to some extent then prevent this
reconstruction. (Zhu et al. (2019) do this in proposing several non-DP
defences against their own attack.)

For utility, as is typically done, we compare the accuracy of different neural
networks in the task of classification when they are trained with DP
guarantees against the baseline without privacy guarantees. For privacy
assessment, we evaluate how each type of noise defends against reconstruction
attacks.

###  4.1. DirDP-SGD: $\varepsilon_{V}$

Unlike Gaussian noise (§4.3), there is no prior work with VMF to use as a
guide for selecting an appropriate $\varepsilon_{V}$. Based on preliminary
experiments, we found a range of changes to utility in
$\varepsilon_{V}\in\\{5,10,50,500\\}$; we also included
$\varepsilon_{V}=300,000$, which hardly shifts gradients, to investigate the
effects of negligible noise.

###  4.2. Datasets

We use classification tasks from the image processing domain, as in many
works, e.g. Abadi et al. (2016) and Zhu et al. (2019).

MNIST333http://yann.lecun.com/exdb/mnist/ dataset (Deng, 2012) contains 70,000
images of handwritten digits between 0 and 9. The images are 28×28 pixels in
greyscale. The training set contains 60,000 instances and the test set has
10,000 instances.

CIFAR444https://www.cs.toronto.edu/~kriz/cifar.html dataset (Krizhevsky, 2009)
contains 60,000 coloured images of 32x32 pixels for each one of the 3
channels. It has two versions: CIFAR10, in which each image belongs to one out
of 10 classes, and CIFAR100, which contains 100 classes. The training set
contains 50,000 instances and the test set has 10,000 instances.

LFW555http://vis-www.cs.umass.edu/lfw/, or Labeled Faces in the Wild dataset
(Huang et al., 2007), has 13,233 images of 5,749 people collected from the
internet. It is a particularly interesting dataset because it is composed of
people’s faces, which is something that one may wish to hide to preserve their
identity, and consequently has been the focus of previous high-profile work on
privacy leakage (Fredrikson et al., 2015, for example). The images have
250x250 pixels, some in greyscale but most are coloured. The standard task,
which we also adopt, is identity recognition; the standard training and test
sets for this contain 9,525 and 3,708 instances respectively. Given its large
number of classes, many with few instances, we follow Wei et al. (2020) to
downsize the dataset.

In doing this, we kept only the classes that contain at least 14 objects. This
reduced the number of classes to 106 and the number of samples to 3,737 (Wei
et al., 2020). Even after this, there is a strong imbalance amongst the
classes, with some having dozens of members but others having hundreds.
Therefore we under-sampled the majority classes by randomly picking objects so
that all classes end up with 14 samples. This reduced the dataset even
further, to 14*106 = 1,484 instances. Finally, we split the resulting dataset
into training (80%, or 1,113 samples) and test (20%, or 371 samples) sets.

###  4.3. Primary Baseline

In terms of deep learning architectures to investigate, we broadly follow the
setup of Scheliga et al. (2022). The architectures of neural networks we use
are LeNet, the original convolutional neural network (CNN) proposed by Lecun
(1989), and a simple Multilayer Perceptron (MLP) with 2 layers, which are
feedforward neural networks (Goodfellow et al., 2016). Scheliga et al. (2022)
include MLPs as they note that Geiping et al. (2020) provide a theoretical
proof that in fully connected networks, their IGA attack can uniquely
reconstruct the input to the network from the network’s gradients. LeNet is a
prototypical architecture for CNNs.

The two baselines in terms of privacy protection for these architectures are
(i) the state-of-the-art DP-SGD using Gaussian noise and (ii) the neural
networks without any DP guarantees. We compare their performance in terms of
accuracy and susceptibility to attacks against our DirDP-SGD.

For the Gaussian noise, there are no standard guidelines on the range to test,
as $\varepsilon$ does not have an easily interpretable meaning, with no
universally agreed-upon point for what counts as ‘too large’; Dwork et al.
(2019) note that “while all small $\varepsilon$ are alike, each large
$\varepsilon$ is large after its own fashion, making it difficult to reason
about them.” As a common range for all of our baseline / dataset combinations,
we consequently select values going from ‘small’ ($\varepsilon_{G}\leq 1$) to
the common largest value of 8 that a number of works (Abadi et al., 2016; De
et al., 2022, for example) have selected over the years. We also added
$\varepsilon_{G}=80$, a very small amount of noise that is outside what is
generally considered acceptable privacy, for calibration purposes.

###  4.4. Data Reconstruction Attacks

We investigate how the noise distributions can defend against attacks during
distributed learning, as explained in (§2.3). We employ the DLG attack from
Zhu et al. (2019) and the Inverting Gradients method from Geiping et al.
(2020). The reasons behind these choices are that (i) DLG is the first
reconstruction attack based on gradient sharing, and well-established as a
baseline; and (ii) Inverting Gradients is based on an angular cost function,
so we assess whether an angular-based noise like our DirDP-SGD can defend
against it. Next, we explain in more detail each one of these attacks.

#### DLG

An attacker receives the gradients from another participant. Instead of
honestly training its neural network, the attacker maliciously uses the
gradients to recover the private data that was used to generate them.

Following the notation from Zhu et al. (2019), let $\nabla W$ be the gradients
received, $F$ be a twice differentiable neural network, $W$ be its parameters
and (x, y) be the (private) training data and the corresponding (private)
labels. The attacker creates dummy x’, y’ (e.g. by sampling from a Gaussian
distribution). The dummy data goes through $F$ and after performing
backpropagation taking the derivatives w.r.t $W$, the dummy gradients $\nabla
W^{\prime}$ are created:

(5) $\nabla
W^{\prime}=\frac{\partial\ell(F(\mathbf{x^{\prime}},W),\mathbf{y^{\prime}})}{\partial
W}$

The private training data can be recovered by optimising

(6)
$\mathbf{x}^{\prime*},\mathbf{y}^{\prime*}=\operatorname*{argmin}_{x^{\prime},y^{\prime}}\|W^{\prime}-W\|^{2}$

More specifically, the attacker takes the difference $\|W^{\prime}-W\|^{2}$,
which is differentiable w.r.t (x’, y’). Therefore, x’, y’ are optimised by

(7) $\mathbf{x}^{\prime},\mathbf{y}^{\prime}=\mathbf{x}^{\prime}-\eta\nabla
x^{\prime}\|W^{\prime}-W\|^{2},\mathbf{y}^{\prime}-\eta\nabla
y^{\prime}\|W^{\prime}-W\|^{2}$

where $\eta$ is the learning rate (usually a value in the range $(0,1]$).

#### Inverting gradients (IGA)

In this attack, Geiping et al. (2020) note that the cost function in Equation
6 optimises a Euclidean matching term, and that the magnitude of a gradient
holds information regarding the stage of the training (the gradients tend to
be smaller for trained networks). The direction of the gradients can also
capture important information, and therefore the authors change Equation 6 to
a function based on angles by adopting the cosine distance:

(8)
$\mathbf{x}^{\prime*},\mathbf{y}^{\prime*}=\operatorname*{argmin}_{x\in[0,1]^{n}}1-\frac{\langle
W^{\prime},W\rangle}{\|W^{\prime}\|\|W\|}+\alpha TV(x)$

with the additional constraint that the values in the input data must be
normalised to fit the space of $[0,1]$.

We evaluate how to defend against both attacks using Gaussian and von Mises-
Fisher noise in untrained models. When injecting Gaussian noise, we set
$\mu=0$ and $\sigma$ as each one of the $\varepsilon$ used for the utility
experiments described in (§5.1). When defending using the von Mises-Fisher
noise, we simply substitute the VMF mechanism from Defn 3.3 for the Gaussian.

###  4.5. Evaluation Measures

We evaluate DirDP-SGD in terms of utility and privacy. For utility, we measure
the impact that different DP strategies have on the accuracy of different
neural networks architectures on classification tasks over different datasets.
For the level of privacy achieved, we assess how DirDP-SGD performs in
defending against reconstruction attacks on gradients compared to baselines.
The metrics we use are the following:

  * •

Accuracy on classification tasks: we evaluate the impact of the privacy
mechanism on model performance according to different values of $\varepsilon$
and the absence of privacy guarantees. This is in line with previous works
(Abadi et al., 2016; Li et al., 2022).

As we observe earlier, the LFW dataset and associated task are particularly
challenging relative to the other two, due to the higher number of classes and
a smaller number of instances. Therefore, we might expect low accuracies,
which the added noise might reduce to near-zero levels, obscuring differences
among different types and levels of noise. For this dataset, then, in addition
to standard accuracy we also report the Top-5 and Top-10 accuracy rates, where
Top-$k$ accuracy deems a classification as correct if the true label is
amongst any of the model’s top $k$ classes with the highest confidence rate
(so standard accuracy is the same as Top-1 accuracy).

  * •

Structural similarity index measure (SSIM) (Wang et al., 2004) compares any
two signals and returns a value between $\left[-1,1\right]$. It compares pixel
intensities that have been normalised for luminance and contrast; the work
that proposed it demonstrated that it correlates well with human judgements of
reconstruction quality. We use it to measure how close the reconstructed
images are to the originals; it has previously been used in this way for the
specific quantitative evaluation of gradient-based reconstructions (Wei et
al., 2020) and (so far non-DP) defences against them (Scheliga et al., 2022).
While there are some complexities in interpreting SSIM scores (Nilsson and
Akenine-Möller, 2020), identical images score 1, completely dissimilar images
score 0, and negative scores occur rarely and only in unusual contexts.

  * •

Mean Squared Error (MSE) measures the distance between a reconstructed image
and its original counterpart by averaging the squares of the differences
between the pixels of two images. We also use it to measure how similar the
reconstructed images are to the original ones. This has likewise been used
along with SSIM in quantitative evaluation of gradient-based reconstruction
attacks and their defences.

##  5\. Results

In this section, we present results for utility and privacy experiments. More
details about hyperparameters and computing environments are described in
Appendix A.

###  5.1. Classification Results

We compare the models after they are trained with and without DP guarantees.
Table 1 shows the accuracy for each setting for MNIST, CIFAR10 and CIFAR100;
Table 2 shows accuracies for LFW.

Overall, in terms of non-private models, relative performance on the datasets
and tasks is as expected: MNIST is the easiest dataset and has the highest
accuracies for both LeNet and MLP; CIFAR10 and then CIFAR100 are next; LFW is
the most challenging. Both baselines (top two lines in Table 1) work almost
perfectly on the MNIST dataset, which contains only greyscale images spread
over 10 classes. For the remaining datasets, which are coloured, LeNet’s
accuracy falls sharply, but MLP maintains high accuracy for CIFAR10 and
CIFAR100.

In general, as expected, adding noise reduces performance, and more noise
corresponds to a greater performance reduction: considering $\varepsilon_{G}$
and $\varepsilon_{V}$ as privacy budgets, the higher they are, less privacy
should be retained, thus increasing the accuracy. We see this consistently for
the CIFAR datasets, and for all datasets for VMF (we discuss the exceptions
below).

Coming to our core first comparison of our VMF mechanism against the Gaussian,
we see that for the CIFAR datasets, for all selected ranges of
$\varepsilon_{G}$ and $\varepsilon_{V}$, VMF noise leads to much smaller
reductions in utility, for both LeNet and MLP, with differences in the order
of $10-20$ percentage points; even the essentially non-private
$\varepsilon_{G}=80$ does not reach the accuracy of the smallest VMF
$\varepsilon_{V}$. For LFW, as expected, (Top-1) accuracies are mostly near-
zero when noise is added; comparing Gaussian and VMF noise under Top-5 and
Top-10, they are fairly similar, until $\varepsilon_{V}=500$, where VMF
rebounds to very strong accuracies. Gaussian noise, however, works better for
MNIST, although it behaves oddly.

We note a counter-intuitive behaviour for the LeNet trained with the Gaussian
mechanism on MNIST. Even though the accuracy is kept high, it slowly decreases
as $\varepsilon$ increases. The same happens to the LFW in Table 2. As we used
the standard DP-SGD implementation for this, and it behaved as expected on the
CIFAR datasets and on the MLP architecture for all datasets, it is not clear
what the cause is, and it warrants further investigation.

As a second exception to the general rule that noise leads to a reduction in
accuracy, we observe the intuitive trend that larger values of $\varepsilon$
lead to higher accuracy. In one case, when LeNet is trained with DirDP-SGD on
CIFAR10 and LFW, its accuracy is even marginally higher than the baseline
without any DP guarantees for a large or very large value of $\varepsilon$. In
this case, the noise is so weak that we hypothesise that instead of preserving
privacy, it acts as a regularisation factor that prevents overfitting, which
can explain the very modest performance gain on the test set. In fact, the use
of noise as a regularisation technique has been studied by Li and Liu (2020,
2022) with Gaussian noise.

|  |  | MNIST | CIFAR10 | CIFAR100
---|---|---|---|---|---
Model | Noise |  $\varepsilon_{G}$, $\varepsilon_{V}$ | Accuracy
LeNet | – | – | 96.2 | 57.0 | 28.3
MLP | – | – | 97.0 | 47.5 | 18.5
LeNet | Gauss | 0.8 | 88.7 | 33.3 | 4.2
1.0 | 88.0 | 34.9 | 5.2
2.0 | 86.5 | 39.1 | 9.8
3.0 | 84.8 | 41.1 | 11.3
8.0 | 83.3 | 46.1 | 14.9
|  | 80.0 | 81.4 | 50.1 | 19.9
LeNet | VMF | 5 | 50.2 | 51.9 | 23.3
10 | 58.5 | 53.5 | 24.5
50 | 65.7 | 54.9 | 27.1
500 | 76.7 | 57.0 | 29.1
300k | 81.8 | 57.2 | 29.5
MLP | Gauss | 0.8 | 91.9 | 35.9 | 9.0
1.0 | 92.0 | 36.6 | 9.4
2.0 | 92.1 | 38.3 | 10.9
3.0 | 91.9 | 40.2 | 11.6
8.0 | 91.8 | 47.3 | 12.3
|  | 80.0 | 91.6 | 21.3 | 2.5
MLP | VMF | 5.0 | 89.7 | 45.3 | 15.2
10.0 | 89.9 | 45.4 | 15.7
50.0 | 90.2 | 46.0 | 16.6
500.0 | 91.1 | 47.3 | 18.2
300k | 91.8 | 47.9 | 18.6

Table 1. Accuracy under different settings. We emphasise that epsilons for
different noise distributions are not directly comparable – for Gaussian the
$\varepsilon_{G}$ and von Mises-Fisher the $\varepsilon_{V}$ parameters appear
to be numerically very different but are appropriate for their respective
abilities to prevent reconstruction. VMF stands for Von Mises-Fisher.

Model | Noise |  $\varepsilon_{G}$, $\varepsilon_{V}$ | Top-1 | Top-5 | Top-10
---|---|---|---|---|---
LeNet | – | – | 5.4 | 19.4 | 31.0
MLP | – | – | 14.0 | 32.1 | 41.5
LeNet | Gauss | 0.8 | 0.3 | 2.7 | 6.2
1.0 | 0.3 | 2.7 | 5.1
2.0 | 0.3 | 2.7 | 4.9
3.0 | 0.3 | 2.2 | 4.3
8.0 | 0.3 | 1.6 | 3.0
|  | 80 | 0.3 | 0.8 | 3.5
LeNet | VMF | 5 | 0.0 | 2.2 | 4.9
10 | 0.0 | 2.4 | 4.6
50 | 0.0 | 2.7 | 5.1
500 | 8.4 | 24.5 | 39.4
300k | 10.8 | 27.8 | 42.0
MLP | Gauss | 0.8 | 0.3 | 5.1 | 9.4
1.0 | 0.3 | 5.1 | 9.4
2.0 | 0.3 | 5.1 | 8.1
3.0 | 0.3 | 3.5 | 6.7
8.0 | 0.3 | 0.8 | 4.6
|  | 80 | 0.5 | 1.9 | 5.1
MLP | VMF | 5 | 1.9 | 8.9 | 15.4
10 | 2.2 | 9.7 | 14.8
50 | 3.0 | 9.7 | 18.9
500 | 3.5 | 15.6 | 23.2
300k | 8.6 | 24.8 | 39.4

Table 2. Top-$k$ accuracy rates for the LFW dataset under different privacy
settings.

Next, we analyse if the possible loss of utility is compensated by protection
against data reconstruction attacks, leading up to an analysis of how to match
up VMF with Gaussian noise effects.

###  5.2. Reconstruction Attacks

Tables 3 and 4 show the SSIM and MSE scores after the reconstruction attack is
performed by DLG and IGA respectively. The metrics are calculated by comparing
the recovered image against its ground truth counterpart. For each attack, 50
images are taken from the training set of their respective dataset. The values
in the table are the average for SSIM and the median for MSE. Higher SSIM
values and lower MSE values indicate that the reconstructed image is closer to
the original.

|  |  | SSIM | MSE
---|---|---|---|---
Model | Noise | $\varepsilon_{G}/\varepsilon_{V}$ | MNIST | CIFAR | LFW | MNIST | CIFAR | LFW
LeNet | – | – | 0.820 | 0.840 | 0.939 | 0.000 | 0.000 | 0.000
MLP | – | – | 1.000 | 0.760 | 0.820 | 0.000 | 0.000 | 0.000
LeNet | Gauss | 0.8 | 0.000 | 0.000 | 0.000 | 3,636 | 787.6 | 991.5
1.0 | 0.000 | 0.000 | 0.000 | 4,948 |  $\sim$102e5 |  $\sim$169e6
2.0 | 0.000 | 0.000 | 0.000 |  $\sim$109e8 |  $\sim$154e8 |  $\sim$143e7
3.0 | 0.000 | 0.000 | 0.000 |  $\sim$401e7 |  $\sim$102e7 |  $\sim$84e7
8.0 | 0.000 | 0.000 | 0.000 |  $\sim$384e7 |  $\sim$119e7 |  $\sim$89e7
|  | 80 | 0.000 | 0.000 | 0.000 |  $\sim$142e11 |  $\sim$177e8 |  $\sim$1906e7
LeNet | VMF | 5 | 0.000 | 0.000 | 0.000 |  $\sim$242e11 |  $\sim$317e13 |  $\sim$658e13
10 | 0.000 | 0.000 | 0.000 |  $\sim$163e9 | 1,076 | 1,132
50 | 0.000 | 0.000 | 0.000 |  $\sim$248e6 | 429.9 | 412.5
500 | 0.000 | 0.001 | 0.001 | 2,566 | 92.2 | 137.9
300k | 0.801 | 0.688 | 0.745 | 0.001 | 0.009 | 0.009
MLP | Gauss | 0.8 | 0.000 | 0.000 | 0.000 |  $\sim$229e8 |  $\sim$178e8 |  $\sim$147e8
1.0 | 0.000 | 0.000 | 0.000 |  $\sim$211e8 |  $\sim$817e7 |  $\sim$256e8
2.0 | 0.000 | 0.000 | 0.000 |  $\sim$332e8 |  $\sim$838e7 |  $\sim$66e8
3.0 | 0.000 | 0.000 | 0.000 |  $\sim$381e8 |  $\sim$135e8 |  $\sim$164e8
8.0 | 0.000 | 0.000 | 0.000 |  $\sim$17e11 |  $\sim$973e8 |  $\sim$736e9
|  | 80 | 0.000 | 0.000 | 0.000 |  $\sim$421e9 |  $\sim$439e9 |  $\sim$296e10
MLP | VMF | 5 | 0.000 | 0.000 | 0.000 |  $\sim$119e8 |  $\sim$69e12 |  $\sim$17e13
10 | 0.000 | 0.000 | 0.000 |  $\sim$223e8 |  $\sim$80e14 |  $\sim$529e11
50 | 0.000 | 0.000 | 0.000 |  $\sim$728e8 |  $\sim$216e4 |  $\sim$158e6
500 | 0.000 | 0.000 | 0.000 |  $\sim$211e6 | 547 |  $\sim$197e5
300k | 0.767 | 0.596 | 0.644 | 0.006 | 0.008 | 0.008

Table 3. DLG reconstruction attack metrics against LeNet and MLP. CIFAR10 and
CIFAR100 contain the same images. For the attacks, we used the labels from
CIFAR100. We report the average SSIM and the median MSE. MSE is unbounded, so
its average is sensitive to high values.

In the absence of noise, DLG achieves very high SSIM values, indicating that
the recovered images are close to their original counterparts. This correlates
to the samples in Figure 4, in which the recovered images are very similar, if
not identical, to the originals.

The IGA attack is generally less successful in the absence of noise. The
performance wrt LeNet in particular decreases sharply. None of the three
sample images that were attacked by DLG succeeded, as shown in Figure 5. MLP
presents more moderate results in CIFAR and LFW datasets, but poor metrics
against MNIST. These metrics are corroborated by the images in Figure 5. In
the absence of noise, even though the telephone is well reconstructed, the
face is barely recognisable, and the number five has poor quality.

The presence of noise disrupts almost all attacks, where the SSIM score goes
to 0, indicating good empirical protection against a reconstruction attack, in
terms of this metric that correlates with human perception. The one exception
to effectively zero SSIMs is for VMF with a very large of $\varepsilon_{V}$
(which, as noted in §4.1, is not intended to be a realistic privacy parameter
value, but just to explore what happens when gradients are perturbed by a
small amount). Figure 4 shows that images are almost retrieved when
$\varepsilon_{V}=300k$ for the VMF noise, even though not perfectly
(especially for image (g) in Figure 4 from MNIST, in which the general shape
of the number 5 is reconstructed, but the background is blurred). Better
reconstruction under this large $\varepsilon_{V}$ is expected, since less
noise is added.

Figure 4 shows examples of how the DLG attack performs to recover the images.
We also show reconstructions for IGA against LeNet and MLP in Figure 5.

|  |  | SSIM | MSE
---|---|---|---|---
Model | Noise | $\varepsilon_{G}/\varepsilon_{V}$ | MNIST | CIFAR | LFW | MNIST | CIFAR | LFW
LeNet | – | – | 0.065 | 0.015 | 0.188 | 0.778 | 1.015 | 0.086
MLP | – | – | 0.321 | 0.479 | 0.528 | 0.445 | 0.076 | 0.036
LeNet | Gauss | 0.8 | 0.000 | 0.001 | 0.000 | 3.583 | 2.218 | 2.341
1.0 | -0.004 | 0.001 | 0.000 | 3.345 | 2.275 | 2.402
2.0 | -0.002 | 0.000 | 0.000 | 3.283 | 2.265 | 2.469
3.0 | -0.005 | 0.000 | 0.001 | 3.277 | 2.365 | 2.487
8.0 | -0.004 | -0.001 | 0.001 | 3.315 | 2.433 | 2.515
|  | 80 | -0.001 | 0.000 | -0.001 | 3.303 | 2.433 | 1.916
LeNet | VMF | 5 | -0.001 | 0.000 | 0.000 | 3.397 | 2.797 | 2.935
10 | -0.002 | 0.000 | 0.001 | 3.356 | 2.806 | 2.872
50 | 0.008 | 0.001 | 0.000 | 3.138 | 2.659 | 2.417
500 | 0.049 | 0.003 | 0.006 | 1.380 | 1.724 | 0.626
300k | 0.075 | 0.014 | 0.225 | 0.740 | 1.042 | 0.076
MLP | Gauss | 0.8 | 0.003 | -0.001 | 0.001 | 3.813 | 3.890 | 3.677
1.0 | 0.002 | -0.001 | 0.000 | 3.845 | 3.848 | 3.657
2.0 | 0.004 | -0.001 | 0.000 | 3.853 | 3.873 | 3.687
3.0 | -0.001 | -0.001 | 0.001 | 3.839 | 3.831 | 3.694
8.0 | -0.002 | -0.001 | 0.000 | 3.933 | 3.883 | 3.707
|  | 80 | 0.003 | 0.000 | 0.000 | 3.909 | 3.869 | 3.610
MLP | VMF | 5 | 0.003 | 0.001 | 0.000 | 3.845 | 3.802 | 3.613
10 | 0.009 | 0.002 | 0.000 | 3.926 | 3.845 | 3.595
50 | 0.003 | 0.000 | 0.000 | 3.677 | 3.749 | 3.575
500 | 0.007 | 0.000 | 0.000 | 3.487 | 3.650 | 3.362
300k | 0.138 | 0.000 | 0.046 | 0.879 | 1.985 | 0.703

Table 4. IGA reconstruction attack metrics. CIFAR10 and CIFAR100 contain the
same images. For the attacks, we used the labels from CIFAR100. We report the
average SSIM and the median MSE. MSE is unbounded, so its average is sensitive
to high values.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
() Original MNIST

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original CIFAR

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original LFW

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original MNIST

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original CIFAR

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original LFW

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

Figure 4. Reconstructed images after DLG attack against LeNet and MLP for 1000
iterations.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
() Original MNIST

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original CIFAR

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original LFW

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original MNIST

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original CIFAR

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

() Original LFW

() No noise

() Gauss $\sigma=0.8$

() Gauss $\sigma=80$

() VMF $\varepsilon_{V}=10$

() VMF $\varepsilon_{V}=500$

() VMF $\varepsilon_{V}=300k$

Figure 5. Reconstructed images by Inverting Gradients against LeNet (top three
rows) and MLP (bottom three rows) after 1000 iterations.

###  5.3. Calibrating Accuracy vs Reconstruction Defence

With SSIM values around zero, although positive in terms of protection against
reconstruction, the SSIM metric thus does not allow any distinguishing among
DP mechanisms, architectures or datasets. Looking at MSE, we see that the two
attacks do still behave very differently in the face of noise. While the DLG
attack achieves MSEs of 0 in the non-private case, adding noise leads to
massive MSEs (as large as $10^{13}$): noise dramatically degrades the
performance of the DLG attack. The IGA attack, by contrast, is much more
successful in the face of noise, with relatively low and fairly constant MSEs
(in the range $1-3$). Together, these let us draw a few inferences.

For the DLG, Gaussian noise leads to MSE scores that stay large, and any
movement is not in a consistent direction: for both LeNet and MLP, it goes up
and down. The VMF noise, by contrast, degrades smoothly and fairly
consistently. For our $\varepsilon_{V}=300k$, which is not aiming for privacy,
it is as expected barely above the zero of the non-private models; for
$\varepsilon_{V}=500$, while MSE scores like 92.2 do not have any inherent
privacy interpretation, they are orders of magnitude larger than the MSEs for
IGA, including for the Gaussian $\varepsilon_{G}$s that do have recognised
degrees of privacy. Similarly, in the IGA, the MSEs for the range of
$\varepsilon_{G}$ values is broadly comparable to the VMF
$5\leq\varepsilon_{V}\leq 500$ (sometimes higher, sometimes lower); for LeNet
on CIFAR, $\varepsilon_{V}=500$ is a bit of an edge case, with a lower MSE,
but still noticeably above the non-private baseline (and still well-obscured
visually: Figure 5 (ab,ai,ap)). Relating this to the discussion of §5.1, we
can say that for comparable ranges of $\varepsilon_{G}$ and $\varepsilon_{V}$,
the accuracies of the noisy models are much higher for the CIFAR datasets, and
in LFW similar for lower values of $\varepsilon_{G}$ and $\varepsilon_{V}$ but
much higher for $\varepsilon_{V}=500$. These same equivalences also hold for
MNIST, with Gaussian noise outperforming VMF for LeNet but similar to VMF for
MLP; but as observed above, MNIST behaves rather oddly under LeNet Gaussian
noise here.

###  5.4. Qualitative Analysis

In broad terms, the attacks successfully reconstruct images in the absence of
noise. When noise is added, both attacks suffer to recover the training data,
as shown in all images (Figures 4 and 5) with Gaussian noise and VMF noise
(the latter for lower $\varepsilon_{V}$). For fully trained networks, it is
harder to reconstruct the images (see Appendix B). However, in a real
scenario, an attacker can simply use a model with dummy weights instead of a
trained network.

Particularly for the IGA attack with VMF against LeNet, we find a good
privacy/utility balance at $\varepsilon_{V}$ = 500. In this setting, most
images cannot be reconstructed (at most, the black background starts to be
recovered in contrast to a white centre, as in image (f) of Figure 5), and at
the same time the model retains high accuracy rates compared to the non-
private baseline across all datasets.

We note that the addition of noise helps to prevent reconstruction in both
attacks. However, we also observe that when the noise is too small (as in the
case of VMF with $\varepsilon_{V}$ set to 300k), images can be reconstructed
more often, as seen in images (n), (u) and (ap) in Figure 4. Even though
offering little protection against the attacks, such small noise can help the
model in its classification task, as shown in Table 1 for both models with MVF
noise in the CIFAR datasets and in Table 2 for Top-10 accuracy with LeNet.
Also, in the context of face reconstruction in images (u) and (ap), even the
negligible VMF noise of $\varepsilon_{V}=300k$ could be useful in practical
applications in obscuring details of the face, in ways not captured by the
present metrics.

###  5.5. Further Note

Overall we found that VMF noise is able to protect against gradient-based
reconstruction attacks and to offer good levels for utility in image
classification datasets over different datasets. This is particularly
illustrated in the LFW dataset, which is based on face images where one might
wish to hide their identity.

However, VMF can be computationally expensive for gradients with high
dimensionality, and future work involves optimising it. This is line with
previous studies that enhanced vanilla approaches for DP in deep learning,
like backpropagation improvements such as ghost clipping from Li et al.
(2022), and the efficient per-sample gradient computation from Yousefpour et
al. (2021).

##  6\. Conclusions

We defined DirDP-SGD for directional privacy in deep learning by adding noise
to the gradients during training. This problem is particularly relevant
because several studies have shown that private training data can be
discovered under certain machine learning training settings, such as sharing
gradients.

Our mechanism provides an $\varepsilon d$-privacy guarantee instead of
$(\varepsilon,\delta)$-DP. Experiments showed that DirDP-SGD can protect
against gradient-based reconstruction attacks, but it also retains utility for
classification tasks across different datasets.

DirDP-SGD is based on the VMF distribution, and it can be computationally
expensive for high dimension data. Future work include optimising the
mechanism. Moreover, experiments were restricted to image datasets. We plan to
explore the feasibility of DirDP-SGD for other domains, such as natural
language processing.

## References

  * (1)
  * Abadi et al. (2016) Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016.  Deep Learning with Differential Privacy. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ (Vienna, Austria) _(CCS ’16)_. Association for Computing Machinery, New York, NY, USA, 308–318.  https://doi.org/10.1145/2976749.2978318
  * Andrés et al. (2013) Miguel E Andrés, Nicolás E Bordenabe, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. 2013\.  Geo-indistinguishability: Differential privacy for location-based systems. In _Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security_. ACM, 901–914.
  * Anil et al. (2021) Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021.  Large-Scale Differentially Private BERT.  _CoRR_ abs/2108.01624 (2021).  arXiv:2108.01624 https://arxiv.org/abs/2108.01624
  * Bassily et al. (2014) Raef Bassily, Adam D. Smith, and Abhradeep Thakurta. 2014\.  Private Empirical Risk Minimization: Efficient Algorithms and Tight Error Bounds. In _55th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2014, Philadelphia, PA, USA, October 18-21, 2014_. IEEE Computer Society, 464–473.  https://doi.org/10.1109/FOCS.2014.56
  * Chatzikokolakis et al. (2013) Konstantinos Chatzikokolakis, Miguel E. Andrés, Nicolás Emilio Bordenabe, and Catuscia Palamidessi. 2013.  Broadening the Scope of Differential Privacy Using Metrics. In _Privacy Enhancing Technologies - 13th International Symposium, PETS 2013, Bloomington, IN, USA, July 10-12, 2013. Proceedings_ _(Lecture Notes in Computer Science, Vol. 7981)_ , Emiliano De Cristofaro and Matthew K. Wright (Eds.). Springer, 82–102.  https://doi.org/10.1007/978-3-642-39077-7_5
  * Chatzikokolakis et al. (2019) Konstantinos Chatzikokolakis, Natasha Fernandes, and Catuscia Palamidessi. 2019.  Comparing Systems: Max-case Refinement Orders and Application to Differential Privacy. In _Proc. CSF_. IEEE Press.
  * Chatzikokolakis et al. (2014) Konstantinos Chatzikokolakis, Catuscia Palamidessi, and Marco Stronati. 2014.  A Predictive Differentially-Private Mechanism for Mobility Traces. In _Privacy Enhancing Technologies \- 14th International Symposium, PETS 2014, Amsterdam, The Netherlands, July 16-18, 2014. Proceedings_ _(Lecture Notes in Computer Science, Vol. 8555)_ , Emiliano De Cristofaro and Steven J. Murdoch (Eds.). Springer, 21–41.  https://doi.org/10.1007/978-3-319-08506-7_2
  * De et al. (2022) Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. 2022.  Unlocking High-Accuracy Differentially Private Image Classification through Scale.  https://doi.org/10.48550/ARXIV.2204.13650
  * Deng (2012) Li Deng. 2012.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ 29, 6 (2012), 141–142.
  * Dwork et al. (2019) Cynthia Dwork, Nitin Kohli, and Deirdre K. Mulligan. 2019\.  Differential Privacy in Practice: Expose your Epsilons!  _J. Priv. Confidentiality_ 9, 2 (2019).  https://doi.org/10.29012/jpc.689
  * Dwork and Roth (2014) Cynthia Dwork and Aaron Roth. 2014.  The Algorithmic Foundations of Differential Privacy.  _Found. Trends Theor. Comput. Sci._ 9, 3–4 (aug 2014), 211–407.  https://doi.org/10.1561/0400000042
  * Fernandes et al. (2019) Natasha Fernandes, Mark Dras, and Annabelle McIver. 2019\.  Generalised Differential Privacy for Text Document Processing. In _Principles of Security and Trust - 8th International Conference, POST 2019, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2019, Prague, Czech Republic, April 6-11, 2019, Proceedings_ _(Lecture Notes in Computer Science, Vol. 11426)_ , Flemming Nielson and David Sands (Eds.). Springer, 123–148.  https://doi.org/10.1007/978-3-030-17138-4_6
  * Feyisetan et al. (2020) Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. 2020\.  Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations. In _WSDM ’20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020_ , James Caverlee, Xia (Ben) Hu, Mounia Lalmas, and Wei Wang (Eds.). ACM, 178–186.  https://doi.org/10.1145/3336191.3371856
  * Feyisetan et al. (2019) Oluwaseyi Feyisetan, Tom Diethe, and Thomas Drake. 2019\.  Leveraging Hierarchical Representations for Preserving Privacy and Utility in Text. In _2019 IEEE International Conference on Data Mining, ICDM 2019, Beijing, China, November 8-11, 2019_ , Jianyong Wang, Kyuseok Shim, and Xindong Wu (Eds.). IEEE, 210–219.  https://doi.org/10.1109/ICDM.2019.00031
  * Fredrikson et al. (2015) Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015\.  Model Inversion Attacks That Exploit Confidence Information and Basic Countermeasures. In _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_ (Denver, Colorado, USA) _(CCS ’15)_. Association for Computing Machinery, New York, NY, USA, 1322–1333.  https://doi.org/10.1145/2810103.2813677
  * Geiping et al. (2020) Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. 2020.  Inverting Gradients - How easy is it to break privacy in federated learning?. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/c4ede56bbd98819ae6112b20ac6bf145-Abstract.html
  * Gong et al. (2020) Maoguo Gong, Yu Xie, Ke Pan, Kaiyuan Feng, and A.K. Qin. 2020.  A Survey on Differentially Private Machine Learning [Review Article].  _IEEE Computational Intelligence Magazine_ 15, 2 (2020), 49–64.  https://doi.org/10.1109/MCI.2020.2976185
  * Goodfellow et al. (2016) Ian J. Goodfellow, Yoshua Bengio, and Aaron Courville. 2016\.  _Deep Learning_.  MIT Press, Cambridge, MA, USA.  http://www.deeplearningbook.org.
  * Huang et al. (2007) Gary B. Huang, Marwan Mattar, Tamara Berg, and Erik Learned-Miller. 2007.  _Labeled faces in the wild: A database for studying face recognition in unconstrained environments_.  Technical Report.
  * Krizhevsky (2009) Alex Krizhevsky. 2009\.  _Learning multiple layers of features from tiny images_.  Technical Report.
  * Lecun (1989) Yann Lecun. 1989\.  _Generalization and network design strategies_.  Elsevier.
  * Lee and Clifton (2011) Jaewoo Lee and Chris Clifton. 2011.  How Much Is Enough? Choosing $\varepsilon$ for Differential Privacy. In _Information Security, 14th International Conference, ISC 2011, Xi’an, China, October 26-29, 2011. Proceedings_ _(Lecture Notes in Computer Science, Vol. 7001)_ , Xuejia Lai, Jianying Zhou, and Hui Li (Eds.). Springer, 325–340.  https://doi.org/10.1007/978-3-642-24861-0_22
  * Li et al. (2022) Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2022.  Large Language Models Can Be Strong Differentially Private Learners. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=bVuP3ltATMz
  * Li and Liu (2020) Yinan Li and Fang Liu. 2020\.  Adaptive Gaussian Noise Injection Regularization for Neural Networks. In _Advances in Neural Networks - ISNN 2020 - 17th International Symposium on Neural Networks, ISNN 2020, Cairo, Egypt, December 4-6, 2020, Proceedings_ _(Lecture Notes in Computer Science, Vol. 12557)_ , Min Han, Sitian Qin, and Nian Zhang (Eds.). Springer, 176–189.  https://doi.org/10.1007/978-3-030-64221-1_16
  * Li and Liu (2022) Yinan Li and Fang Liu. 2022\.  Noise-Augmented Privacy-Preserving Empirical Risk Minimization with Dual-Purpose Regularizer and Privacy Budget Retrieval and Recycling. In _Intelligent Computing_ , Kohei Arai (Ed.). Springer International Publishing, Cham, 660–681.
  * Liu et al. (2021) Haolin Liu, Chenyu Li, Bochao Liu, Pengju Wang, Shiming Ge, and Weiping Wang. 2021\.  Differentially Private Learning with Grouped Gradient Clipping. In _ACM Multimedia Asia_ (Gold Coast, Australia) _(MMAsia ’21)_. Association for Computing Machinery, New York, NY, USA, Article 14, 7 pages.  https://doi.org/10.1145/3469877.3490594
  * Lu et al. (2022) Zhigang Lu, Hassan Jameel Asghar, Mohamed Ali Kaafar, Darren Webb, and Peter Dickinson. 2022.  A Differentially Private Framework for Deep Learning With Convexified Loss Functions.  _IEEE Transactions on Information Forensics and Security_ 17 (2022), 2151–2165.  https://doi.org/10.1109/TIFS.2022.3169911
  * McMahan et al. (2018) H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018\.  Learning Differentially Private Recurrent Language Models. In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.  https://openreview.net/forum?id=BJ0hF1Z0b
  * Mironov (2017) Ilya Mironov. 2017\.  Rényi Differential Privacy. In _30th IEEE Computer Security Foundations Symposium, CSF 2017, Santa Barbara, CA, USA, August 21-25, 2017_. IEEE Computer Society, 263–275.  https://doi.org/10.1109/CSF.2017.11
  * Mukherjee et al. (2021) Sumit Mukherjee, Yixi Xu, Anusua Trivedi, Nabajyoti Patowary, and Juan M. Lavista Ferres. 2021.  privGAN: Protecting GANs from membership inference attacks at low cost to utility.  _Proceedings on Privacy Enhancing Technologies_ 2021 (2021), 142 – 163.
  * N. Fernandes and Ding (2022) C. Palamidessi N. Fernandes, A. McIver and M. Ding. 2022.  Universal Optimality and Robust Utility Bounds for Metric Differential Privacy. In _35th IEEE Computer Security Foundations Symposium, CSF 2022, Haifa, Israel_. IEEE Computer Society, 332–347.  https://doi.org/10.1109/CSF54842.2022.00022
  * Nilsson and Akenine-Möller (2020) Jim Nilsson and Tomas Akenine-Möller. 2020.  Understanding SSIM.  https://doi.org/10.48550/ARXIV.2006.13846
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 8024–8035.  http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Pejó and Desfontaines (2022) Balázs Pejó and Damien Desfontaines. 2022.  _Guide to Differential Privacy Modifications - A Taxonomy of Variants and Extensions_.  Springer.  https://doi.org/10.1007/978-3-030-96398-9
  * Qu et al. (2021) Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021\.  Natural Language Understanding with Privacy-Preserving BERT. In _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_ , Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, 1488–1497.  https://doi.org/10.1145/3459637.3482281
  * Rahman et al. (2018) Md.Atiqur Rahman, Tanzila Rahman, Robert Laganière, and Noman Mohammed. 2018.  Membership Inference Attack against Differentially Private Deep Learning Model.  _Trans. Data Priv._ 11 (2018), 61–79.
  * Scheliga et al. (2022) Daniel Scheliga, Patrick Mäder, and Marco Seeland. 2022\.  PRECODE - A Generic Model Extension to Prevent Deep Gradient Leakage. In _IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022_. IEEE, 3605–3614.  https://doi.org/10.1109/WACV51458.2022.00366
  * Song et al. (2013) Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. 2013\.  Stochastic gradient descent with differentially private updates. In _2013 IEEE Global Conference on Signal and Information Processing_. 245–248.  https://doi.org/10.1109/GlobalSIP.2013.6736861
  * Torkzadehmahani et al. (2019) Reihaneh Torkzadehmahani, Peter Kairouz, and Benedict Paten. 2019\.  DP-CGAN: Differentially Private Synthetic Data and Label Generation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_. 0–0.
  * Tramer and Boneh (2021) Florian Tramer and Dan Boneh. 2021.  Differentially Private Learning Needs Better Features (or Much More Data). In _International Conference on Learning Representations_.  https://openreview.net/forum?id=YTWGvpFOQD-
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018\.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Association for Computational Linguistics, Brussels, Belgium, 353–355.  https://doi.org/10.18653/v1/W18-5446
  * Wang et al. (2004) Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004\.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ 13, 4 (2004), 600–612.  https://doi.org/10.1109/TIP.2003.819861
  * Weggenmann and Kerschbaum (2021) Benjamin Weggenmann and Florian Kerschbaum. 2021.  Differential Privacy for Directional Data. In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ (Virtual Event, Republic of Korea) _(CCS ’21)_. Association for Computing Machinery, New York, NY, USA, 1205–1222.  https://doi.org/10.1145/3460120.3484734
  * Weggenmann et al. (2022) Benjamin Weggenmann, Valentin Rublack, Michael Andrejczuk, Justus Mattern, and Florian Kerschbaum. 2022.  DP-VAE: Human-Readable Text Anonymization for Online Reviews with Differentially Private Variational Autoencoders. In _WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022_ , Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 721–731.  https://doi.org/10.1145/3485447.3512232
  * Wei et al. (2020) Wenqi Wei, Ling Liu, Margaret Loper, Ka Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. 2020.  A Framework for Evaluating Client Privacy Leakages in Federated Learning. In _Computer Security - ESORICS 2020 - 25th European Symposium on Research in Computer Security, ESORICS 2020, Guildford, UK, September 14-18, 2020, Proceedings, Part I_ _(Lecture Notes in Computer Science, Vol. 12308)_ , Liqun Chen, Ninghui Li, Kaitai Liang, and Steve A. Schneider (Eds.). Springer, 545–566.  https://doi.org/10.1007/978-3-030-58951-6_27
  * Xiao and Xiong (2015) Yonghui Xiao and Li Xiong. 2015.  Protecting Locations with Differential Privacy under Temporal Correlations. In _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, October 12-16, 2015_ , Indrajit Ray, Ninghui Li, and Christopher Kruegel (Eds.). ACM, 1298–1309.  https://doi.org/10.1145/2810103.2813640
  * Xu et al. (2019) Chugui Xu, Ju Ren, Deyu Zhang, Yaoxue Zhang, Zhan Qin, and Kui Ren. 2019\.  GANobfuscator: Mitigating Information Leakage Under GAN via Differential Privacy.  _IEEE Transactions on Information Forensics and Security_ 14, 9 (2019), 2358–2371.  https://doi.org/10.1109/TIFS.2019.2897874
  * Xu et al. (2021) Depeng Xu, Wei Du, and Xintao Wu. 2021.  Removing Disparate Impact on Model Accuracy in Differentially Private Stochastic Gradient Descent. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_ (Virtual Event, Singapore) _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 1924–1932.  https://doi.org/10.1145/3447548.3467268
  * Yang et al. (2022) Mengmeng Yang, Ivan Tjuawinata, and Kwok-Yan Lam. 2022\.  K-Means Clustering With Local d-Privacy for Privacy-Preserving Data Analysis.  _IEEE Transactions on Information Forensics and Security_ 17 (2022), 2524–2537.  https://doi.org/10.1109/TIFS.2022.3189532
  * Yousefpour et al. (2021) Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. 2021\.  Opacus: User-Friendly Differential Privacy Library in PyTorch. In _NeurIPS 2021 Workshop Privacy in Machine Learning_.  https://openreview.net/forum?id=EopKEYBoI-
  * Yu et al. (2020) Honggang Yu, Kaichen Yang, Teng Zhang, Yun-Yun Tsai, Tsung-Yi Ho, and Yier Jin. 2020\.  CloudLeak: Large-Scale Deep Learning Models Stealing Through Adversarial Examples. In _NDSS_.
  * Zhao et al. (2020) Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. 2020\.  iDLG: Improved Deep Leakage from Gradients.  _CoRR_ abs/2001.02610 (2020).  arXiv:2001.02610 http://arxiv.org/abs/2001.02610
  * Zhao et al. (2022) Ying Zhao, Dong Yuan, Jia Tina Du, and Jinjun Chen. 2022\.  Geo-Ellipse-Indistinguishability: Community-Aware Location Privacy Protection for Directional Distribution.  _IEEE Transactions on Knowledge and Data Engineering_ (2022), 1–11.  https://doi.org/10.1109/TKDE.2022.3192360
  * Zhu et al. (2019) Ligeng Zhu, Zhijian Liu, and Song Han. 2019.  Deep Leakage from Gradients. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 14747–14756.  https://proceedings.neurips.cc/paper/2019/hash/60a6c4002cc7b29142def8871531281a-Abstract.html

##  Appendix A Implementation details

All experiments were run on a virtual machine hosted by Oracle Cloud running
Ubuntu 22.04 with an Intel Xeon Platinum 8167M, 12 cores at 2.00GHz, 180GB
RAM, and two NVIDIA Tesla V100 SXM2 GPUs with 16GB RAM each.

Model | Dataset | Batch size | LR | Epochs | Optimiser | Momentum | Decay
---|---|---|---|---|---|---|---
LeNet | MNIST | 200 | 0.1 | 30 | SGD | 0.0 | 0.0
LeNet | CIFAR | 512 | 0.001 | 90 | AdamW | 0.0 | 0.0
LeNet | LFW | 128 | 0.001 | 100 | AdamW | 0.0 | 0.0
MLP | MNIST | 128 | 0.1 | 30 | SGD | 0.0 | 0.0
MLP | CIFAR | 128 | 0.1 | 25 | AdamW | 0.0 | 0.0
MLP | LFW | 128 | 0.001 | 30 | AdamW | 0.0 | 0.0

Table 5. Hyperparameters used for each set of experiments.

Table 5 shows the hyperparameters used to train the victim across the
experiments from (§4). The settings for CIFAR were used for both CIFAR10 and
CIFAR100.

##  Appendix B Attacking trained networks

We study how the simple training of a neural network, without any noise
involved, can protect against gradient based attacks. The attacker doesn’t
need to use a fully trained network, thus this scenario cannot be considered
typical. However, it can help to empirically understand how the gradients
carry training data information.

We attack the networks trained for experiments in Section 5.1 with DLG and
Inverting Gradients methods. We see from the images in Figure 6 that, even
though training does not fully prevent the attacks, it affects their
efficiency.

The same conclusion is reached by looking at the numbers in Table 6. In almost
all cases, the SSIM values are sharply lower for trained models. Higher MSE
scores are also observed in trained models.

One intuitive explanation is that gradients are used to update the model’s
weights. For trained networks, there is little to update, so the gradients at
the beginning of the training are more prone to leak information.

|  | SSIM | MSE
---|---|---|---
Model | Weights | MNIST | CIFAR | LFW | MNIST | CIFAR | LFW
LeNet | Dummy | 0.820 | 0.840 | 0.939 | 0.000 | 0.000 | 0.000
LeNet | Trained | 0.717 | 0.501 | 0.216 | 0.000 | 0.017 |  $\sim$107e4
MLP | Dummy | 1.000 | 0.760 | 0.820 | 0.000 | 0.000 | 0.000
MLP | Trained | 0.040 | 0.000 | 0.216 |  $\sim$836e5 |  $\sim$19e5 |  $\sim$ 107e4
LeNet | Dummy | 0.065 | 0.015 | 0.188 | 0.778 | 1.015 | 0.086
LeNet | Trained | 0.019 | 0.054 | 0.197 | 1.211 | 1.117 | 0.079
MLP | Dummy | 0.321 | 0.479 | 0.528 | 0.445 | 0.076 | 0.036
MLP | Trained | 0.128 | 0.024 | 0.564 | 0.944 | 1.586 | 0.037

Table 6. Reconstruction attack metrics amongst trained and untrained models.
The top three rows refer to DLG attack, and the bottom three refer to
Inverting Gradients.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |

() Original MNIST

() LeNet

() Trained LeNet

() MLP

() Trained MLP

() Original CIFAR

() LeNet

() Trained LeNet

() MLP

() Trained MLP

() Original LFW

() LeNet

() Trained LeNet

() MLP

() Trained MLP

() Original MNIST

() LeNet

() Trained LeNet

() MLP

() Trained MLP

() Original CIFAR

() LeNet

() Trained LeNet

() MLP

() Trained MLP

() Original LFW

() LeNet

() Trained LeNet

() MLP

() Trained MLP

Figure 6. Reconstructed images after DLG (top three rows) and Inverting
Gradients (bottom three rows) after 1000 iterations.