redacted {sohamde | lberrada | slsmith |<EMAIL_ADDRESS>
# Unlocking High-Accuracy Differentially Private Image Classification through
Scale

Soham De  Equal contributions  DeepMind  Leonard Berrada  Equal contributions
DeepMind  Jamie Hayes  Samuel L Smith  Borja Balle

###### Abstract

Differential Privacy (DP) provides a formal privacy guarantee preventing
adversaries with access to a machine learning model from extracting
information about individual training points. Differentially Private
Stochastic Gradient Descent (DP-SGD), the most popular DP training method for
deep learning, realizes this protection by injecting noise during training.
However previous works have found that DP-SGD often leads to a significant
degradation in performance on standard image classification benchmarks.
Furthermore, some authors have postulated that DP-SGD inherently performs
poorly on large models, since the norm of the noise required to preserve
privacy is proportional to the model dimension. In contrast, we demonstrate
that DP-SGD on over-parameterized models can perform significantly better than
previously thought. Combining careful hyper-parameter tuning with simple
techniques to ensure signal propagation and improve the convergence rate, we
obtain a new SOTA without extra data on CIFAR-10 of 81.4% under
$\mathbf{(8,10^{-5})}$-DP using a 40-layer Wide-ResNet, improving over the
previous SOTA of 71.7%. When fine-tuning a pre-trained NFNet-F3, we achieve a
remarkable 83.8% top-1 accuracy on ImageNet under $\mathbf{(0.5,8\cdot
10^{-7})}$-DP. Additionally, we achieve 86.7$\%$ top-1 accuracy under
$\mathbf{(8,8\cdot 10^{-7})}$-DP, only 4.3$\%$ below the current non-private
SOTA for this task. We believe our results are a significant step towards
closing the accuracy gap between private and non-private image classification.

##  1 Introduction

Machine learning models trained with standard pipelines can be attacked by an
adversary that seeks to reveal the data on which the model was trained. For
example, Carlini et al. (2021b) showed that adversaries can generate and
detect text sequences from the training set of a large transformer language
model, while Balle et al. (2022) showed that powerful adversaries can
reconstruct images in the training set of a classifier trained on CIFAR-10.
Alongside other results (Carlini et al., 2021a; Choquette-Choo et al., 2021;
Liu et al., 2021), these studies demonstrate that models trained on sensitive
datasets present a significant privacy risk.

Differential Privacy (DP) (Dwork et al., 2006) is the gold standard technique
for mitigating privacy attacks aimed at leaking individual training examples,
and it has already been adopted in practice by a range of public and private
organizations (Erlingsson, 2014; Apple Differential Privacy Team, 2017; Abowd,
2018; Nayak, 2020; Bird, 2020; McMahan and Thakurta, 2022). A differentially
private algorithm is a randomized algorithm providing a formal guarantee that
any single example in the training set can only influence the output
distribution of the algorithm by a small, pre-specified amount. This privacy
guarantee, denoted $(\varepsilon,\delta)$-DP, is defined by two parameters
$(\varepsilon,\delta)$, which we refer to as the _privacy budget_. The smaller
these two parameters are, the closer the output distributions between training
sets that differ by a single example, and therefore the more difficult it is
for an adversary to infer whether any single data point was included during
training.

The most popular method for training neural networks with DP is Differentially
Private Stochastic Gradient Descent (DP-SGD) (Abadi et al., 2016). DP-SGD
replaces the usual mini-batch gradient estimate of SGD with a privatized
version, in which the gradient of each training example is clipped to a
maximum norm. In addition, Gaussian noise proportional to the clipping norm is
added to the sum of the clipped gradients, which is sufficient to mask the
contribution of any single example to the sum. Each evaluation of a
(privatized) mini-batch gradient incurs a privacy cost, and a privacy
accountant (Abadi et al., 2016; Mironov et al., 2019) is used to track the
total privacy budget spent throughout training. These parameters increase with
every mini-batch seen during training, and decrease with the scale of the
noise added, thus limiting the number of training iterations we can perform at
a fixed privacy budget while keeping the variance in the gradient estimate
under control.

Training with DP-SGD involves a delicate balancing act between different
hyper-parameters such as the amount of added noise, the batch size, and the
number of training iterations, in order to reach optimal performance within a
specified privacy budget. In particular, the noise added to the gradient is a
significant barrier to optimization, typically resulting in a significant
degradation in performance compared to standard non-private training (Klause
et al., 2022; Dörmann et al., 2021; Kurakin et al., 2022). Furthermore,
several authors have postulated that highly over-parameterized models, which
perform well in non-private settings, do not work well when used with DP-SGD,
because the norm of the added noise increases with the dimension of the
gradient (Tramèr and Boneh, 2021; Shen et al., 2021; Yu et al., 2021b; Kurakin
et al., 2022), leading to a “curse of dimensionality”. Consequently, many
works have focused either on developing specialized architectures for private
training (Papernot et al., 2021; Tramèr and Boneh, 2021), or on reducing the
dimensionality of the model during training (Yu et al., 2021b; Zhou et al.,
2021; Zhang et al., 2021; Golatkar et al., 2022).

(a) CIFAR-10 without extra data

(b) ImageNet with extra data

Figure 1:  (a) When training on CIFAR-10 without additional data, we improve
on previously published results under $(\varepsilon,10^{-5})$-DP whenever
$\varepsilon\geq 3$. At $\varepsilon=8$, we improve on the previous SOTA of
Klause et al. (2022) by $9.7\%$. Note we report the mean and standard error
across 5 independent runs. (b) When fine-tuning a pre-trained NFNet-F3 (Brock
et al., 2021b) on ImageNet under $(8,8\cdot 10^{-7})$-DP, we achieve $86.7\%$
top-1 accuracy, only $4.3\%$ below the current non-private SOTA of $91.0\%$
(Yu et al., 2022). We also obtain $83.8\%$ top-1 accuracy under a much tighter
$(0.5,8\cdot 10^{-7})$-DP guarantee, which exceeds the performance of many
popular non-private models (e.g., ResNet-50).

On the contrary, we show that standard over-parameterized architectures, which
achieve close to state-of-the-art performance in non-private training, can
also perform very well when trained using DP-SGD if properly tuned. To achieve
this, we introduce a number of techniques which help convergence and ensure
trainability at initialization, and we explore the benefits of using pre-
trained models. Our main contributions are listed below:

  * •

We describe a set of simple techniques which, when combined, significantly
improve the performance of DP-SGD. First, we revisit ideas that have
previously been identified as useful for private training, including using
large batch sizes (McMahan et al., 2018b), and replacing batch normalization
layers with alternatives that ensure good signal propagation at initialization
(van der Maaten and Hannun, 2020). In addition, we propose further
modifications that improve the convergence rate of DP-SGD, and that have not
previously been used for private training. Specifically, we suggest using
weight standardization in convolutional layers (Qiao et al., 2019), leveraging
the benefits of data augmentation by averaging per-example gradients across
multiple augmentations of the same image before the clipping operation (Hoffer
et al., 2019) and applying parameter averaging techniques (Polyak and
Juditsky, 1992).

  * •

Applying the techniques above, we significantly improve the performance of DP-
SGD when training randomly initialized over-parameterized models. Training
Wide-ResNets (Zagoruyko and Komodakis, 2016) on CIFAR-10 without extra data,
we achieve a new SOTA of 81.4% under $(8,10^{-5})$-DP. This is a substantial
improvement on the previous SOTA of 71.7% achieved under $(7.5,10^{-5})$-DP
(Klause et al., 2022). As shown in Figure 1(a), we achieve SOTA results on
this task across a range of $\varepsilon$ values between 3 and 8. We also
achieve a new SOTA top-1 accuracy on ImageNet of 32.4$\%$ under $(8,8\cdot
10^{-7})$-DP when training a Normalizer-Free ResNet-50 (NF-ResNet-50) (He et
al., 2016; Brock et al., 2021a).

  * •

We show that non-private pre-training on public/non-sensitive data, followed
by fine-tuning with DP-SGD on the private dataset, yields remarkable
performance benefits on image classification benchmarks. For example, when
privately fine-tuning an NF-ResNet-200 pre-trained on JFT-300M (Sun et al.,
2017), we achieve 81.3$\%$ top-1 accuracy on ImageNet under $(8,8\cdot
10^{-7})$-DP. We observe further improvements in performance from increasing
both the size of the model and the size of the pre-training dataset, achieving
86.7$\%$ top-1 accuracy under $(8,8\cdot 10^{-7})$-DP with an NFNet-F3 (Brock
et al., 2021b) pre-trained on JFT-4B. This network also obtains 83.8$\%$ top-1
accuracy under a much tighter privacy budget of $(0.5,8\cdot 10^{-7})$-DP. For
comparison, fine-tuning the same pre-trained network on ImageNet without
privacy reaches 88.5$\%$.

  * •

We provide novel insights into how optimal hyper-parameters relate to each
other when training with DP. We empirically observe that (1) there is an
optimal budget of training iterations given a fixed batch-size, (2) larger
batch sizes improve validation accuracy but require more training epochs after
the batch size exceeds a certain threshold, and (3) the optimal choice of
learning-rate for DP-SGD is proportional to the batch size when the batch size
is small but constant for large batch sizes, similar to non-private training.

We summarize our key results in Table 1, with our SOTA results shown in
bold.111Shortly after our paper was released, Klause et al. (2022) updated
their paper to combine our techniques with their suggested ScaleNorm,
achieving a new SOTA test accuracy of 82.5% on CIFAR-10 without extra data at
$\varepsilon=8$. Additionally, Bu et al. (2022) also published their work
showing the benefits of large models when fine-tuning with differential
privacy by achieving a new SOTA test accuracy with extra data of 96.7% on
CIFAR-10 and 83.0% on CIFAR-100 under $(1,10^{-5})$-DP using a large ViT model
pre-trained on ImageNet. We emphasize that all our results use standard vision
architectures which have been shown to work well for non-private training. We
believe these results are a significant step towards practically useful
differentially private image classification.

Table 1:  A summary of the best results provided in this paper when training with DP-SGD. All numbers in bold are SOTA. For the CIFAR-10 and CIFAR-100 experiments, we report the median accuracy across 5 independent runs. All experiments on CIFAR use Wide-ResNets with group normalization, while the ImageNet and Places-365 experiments use NF-ResNets or NFNets. See relevant sections for further details. Dataset | Pre-Training | Top-1 Accuracy (%) | $\delta$ | Section
---|---|---|---|---
Data | $\varepsilon=1$ | $\varepsilon=2$ | $\varepsilon=4$ | $\varepsilon=8$
CIFAR-10 | – | 56.8 | 65.9 | 73.5 | 81.4 | $10^{-5}$ | 3.2
ImageNet | – | – | – | – | 32.4 | $8\cdot 10^{-7}$ | 3.3
CIFAR-10 | ImageNet | 94.7 | 95.4 | 96.1 | 96.7 | $10^{-5}$ | 4.1
CIFAR-100 | ImageNet | 70.3 | 74.7 | 79.2 | 81.8 | $10^{-5}$ | 4.1
ImageNet | JFT-4B | 84.4 | 85.6 | 86.0 | 86.7 | $8\cdot 10^{-7}$ | 4.2
Places-365 | JFT-300M | – | – | – | 55.1 | $5\cdot 10^{-7}$ | 4.3

#### Paper outline.

We provide a brief introduction to Differential Privacy and DP-SGD in Section
2, where we also discuss the challenges that arise when applying DP-SGD to
deep networks. In Section 3, we describe a range of techniques that enhance
the performance of networks trained with DP-SGD, achieving SOTA performance on
CIFAR-10 and ImageNet when training without additional data. In Section 4, we
show that privately fine-tuning strong pre-trained models dramatically
improves the performance of private image classification. Finally, we provide
additional insights into how the hyper-parameters of DP-SGD influence
performance in Section 5.

#### Reproducibility.

To help researchers reproduce and verify our results, we are releasing the
implementation of DP-SGD used in our experiments at
https://github.com/deepmind/jax_privacy. We also provide the configuration
scripts and pre-trained checkpoints necessary to reproduce all of our results
on CIFAR-10 and CIFAR-100, as well as our results on ImageNet without extra
data. We provide further details about our DP-SGD implementation in Appendix
A, together with a description of the steps we undertook to audit its
correctness.

##  2 Background

###  2.1 Differential Privacy (DP)

Differential privacy (DP) is a formal privacy guarantee that applies to
randomized data analysis algorithms. By construction, differentially private
algorithms prevent an adversary that observes the output of a computation from
inferring any property pertaining to individual data points in the input data
used during the computation. The strength of this guarantee is controlled by
two parameters: $\varepsilon>0$ and $\delta\in[0,1]$. Roughly speaking,
$\varepsilon$ bounds the log-likelihood ratio of any particular output that
can be obtained when running the algorithm on two datasets differing in a
single data point, and $\delta$ is a small probability which bounds the
occurrence of infrequent outputs that violate this bound. The privacy
guarantee becomes stronger as both parameters get smaller. A standard rule of
thumb states that, to obtain meaningful privacy, $\varepsilon$ should be a
small constant while $\delta$ should be smaller than $1/N$, where $N$ is the
size of the input dataset. More formally, we have the following.

######  Definition 2.1 (Differential Privacy (Dwork et al., 2006)).

Let $A:\mathcal{D}\to\mathcal{S}$ be a randomized algorithm, and let
$\varepsilon>0$, $\delta\in[0,1]$. We say that $A$ is
$(\varepsilon,\delta)$-DP if for any two neighboring datasets
$D,D^{\prime}\in\mathcal{D}$ differing by a single element, we have that

$\displaystyle\forall\>S\subset\mathcal{S},\>\mathbb{P}[A(D)\in
S]\leq\exp(\varepsilon)\mathbb{P}[A(D^{\prime})\in S]+\delta\enspace.$ (1)

The privacy protection afforded by DP holds under an exceedingly strong threat
model: inferences about individuals are protected even in the face of an
adversary that has full knowledge of the DP algorithm, unbounded computational
power, and arbitrary side knowledge about the input data. Furthermore, DP
satisfies a number of appealing properties from the algorithm design
standpoint, including preservation under post-processing and a smooth
degradation with multiple accesses to the same data. These properties are
exploited in the construction of complex DP algorithms based on the
combination of small building blocks that inject carefully calibrated noise
into operations that access the data. The magnitude of the noise required to
satisfy the privacy guarantee increases with the strength of the privacy
parameters, leading to an unavoidable trade-off between utility and privacy,
as illustrated by the Fundamental Law of Information Recovery (Dwork and Roth,
2014).

Together, the strength of the formal guarantee it provides and the variety of
tools available for the construction of DP algorithms, have led to the growing
adoption of DP as a gold standard for privacy-preserving machine learning. For
convex learning problems there exists a variety of methods for obtaining
differentially private algorithms, including output perturbation (Chaudhuri et
al., 2011; Wu et al., 2017), objective perturbation (Chaudhuri et al., 2011;
Kifer et al., 2012) and gradient perturbation (Song et al., 2013; Bassily et
al., 2014). The nature of convex problems enables the formal analysis of the
privacy-utility offered by these algorithms, and by now there are large
classes of problems for which algorithms achieving (nearly) optimal privacy-
utility trade-offs are known (Bassily et al., 2014; Talwar et al., 2015;
Feldman et al., 2020; Song et al., 2021; Asi et al., 2021). For non-convex
learning problems, the range of available algorithms is more limited and
privacy-utility trade-offs are harder to analyze theoretically. Nonetheless,
for such problems there exist two families of algorithms that have been shown
to achieve reasonable privacy-utility-computation trade-offs in practice:
gradient perturbation applied to standard optimizers like SGD (Abadi et al.,
2016), and private aggregation of teacher ensembles (Papernot et al., 2018).
In this work we focus on the former, which is most commonly used.

###  2.2 Differentially Private Stochastic Gradient Descent (DP-SGD)

In this work, we assume that the differentially private algorithm $A$ (see
Equation 1) is a learning algorithm that maps a training dataset
$D=\\{(x_{i},y_{i})\\}_{1\leq i\leq N}$ to a vector of learned neural network
parameters $w\in\mathcal{S}=\mathbb{R}^{p}$. Let $\mathcal{L}(w,x,y)$ denote
the learning objective (e.g., the cross-entropy loss), given the model
parameters $w$, input example $x$ and label $y$. For convenience, we use the
shorthand notation $l_{i}(w)=\mathcal{L}(w,x_{i},y_{i})$.

In the non-private setting, a parameter update using Stochastic Gradient
Descent (SGD) at iteration $t$ draws $B$ examples at random from the dataset,
and performs an update of the form:

$w^{(t+1)}=w^{(t)}-\eta_{t}\frac{1}{B}\sum\limits_{i\in\mathcal{B}_{t}}\nabla
l_{i}(w^{(t)})\enspace,$

where $\eta_{t}$ is the step-size for the $t^{th}$ update, $\nabla$ denotes
the gradient operator, and $\mathcal{B}_{t}$ represents the set of examples
sampled at iteration $t$ with $|\mathcal{B}_{t}|=B$. In order to make this
algorithm differentially private, we apply the following modifications. First,
the gradient for each example in the mini-batch is clipped to a maximal norm
$C$, and second, Gaussian noise with standard deviation proportional to $C$ is
added to the mean of the clipped gradients. Let
$\texttt{clip}_{C}:v\in\mathbb{R}^{p}\mapsto\min\left\\{1,\tfrac{C}{\|v\|_{2}}\right\\}\cdot
v\in\mathbb{R}^{p}$ denote the clipping function which re-scales its input so
that the output has a maximal $\ell_{2}$ norm of $C$. The new update step is:

$\displaystyle
w^{(t+1)}=w^{(t)}-\eta_{t}\left\\{\frac{1}{B}\sum\limits_{i\in\mathcal{B}_{t}}\texttt{clip}_{C}\left(\nabla
l_{i}(w^{(t)})\right)+\frac{\sigma C}{B}\xi\right\\}\enspace,$ (2)

where $\xi\sim\mathcal{N}(0,I_{p})$ is a standard $p$-dimensional Gaussian
random variable and $\sigma$ specifies the standard deviation of the added
noise. The resulting algorithm is called Differentially Private-Stochastic
Gradient Descent (DP-SGD) (Abadi et al., 2016). Intuitively, performing a
model update using Equation 2 provides differential privacy because adding
Gaussian noise with standard deviation proportional to $C$ is sufficient to
mask the contribution of any single example whose clipped gradient has norm
less than or equal to $C$. While we use privatized SGD as our optimizer
throughout this work, a similar privatization method can also be used in
combination with other first-order optimization algorithms, such as SGD with
momentum or Adam (McMahan et al., 2018a).

Throughout this paper we use a modified version of DP-SGD where the privatized
gradient is normalized by $C$:

$\displaystyle
w^{(t+1)}=w^{(t)}-\eta_{t}\left\\{\frac{1}{B}\sum\limits_{i\in\mathcal{B}_{t}}\frac{1}{C}\texttt{clip}_{C}\left(\nabla
l_{i}(w^{(t)})\right)+\frac{\sigma}{B}\xi\right\\}\enspace.$ (3)

This is a re-parameterization of Equation 2 in which the learning rate
$\eta_{t}$ absorbs a factor of $C$. This has no effect on the privacy
guarantees, but ensures the clipping norm does not influence the scale of the
update, which simplifies hyper-parameter tuning (See Section B.1). Note that
to preserve the DP guarantees, we must divide by $C$ _after_ the clipping
operation. Section A.1 provides further details about our DP-SGD
implementation, including a description of our approach to virtual batching to
enable training with large batch sizes.

#### Privacy accounting.

The privacy guarantee of DP-SGD is determined by three parameters: the
standard deviation $\sigma$, the sampling ratio $q=B/N$ and the number of
training iterations $T$. In practice, the privacy budget
$(\varepsilon,\delta)$ is usually fixed, and these three hyper-parameters are
chosen to provide the best possible performance within this budget. There may
also be additional practical constraints (e.g., the maximum compute budget
available). The privacy calibration process is performed using a privacy
accountant: a numerical algorithm providing tight upper bounds for the privacy
budget as a function of the hyper-parameters (Abadi et al., 2016), which in
turn can be combined with numerical optimization routines to optimize one
hyper-parameter given the privacy budget and the other two hyper-parameters.
In this work we use the accounting method for DP-SGD proposed by Mironov et
al. (2019) and implemented in TensorFlow Privacy (Google, 2018). This privacy
accountant relies on a composition analysis across iterations, which allows us
to release not only the final model, but also _every_ intermediate model
obtained during training (under the same privacy budget).

###  2.3 Challenges of DP-SGD

As described above, there are three key differences between DP-SGD and non-
private SGD: (1) the per-example gradients are clipped to a maximal $\ell_{2}$
norm before they are averaged, (2) Gaussian noise is added to the average of
the clipped gradients, and (3) the maximum number of updates allowed within
the privacy budget is bounded, and depends on the batch size/added noise.
These differences introduce a number of challenges:

#### Hyper-parameter tuning and regularization.

The noise added to the gradient estimate in the DP-SGD update (Equation 3) is
a significant barrier to efficient optimization, and if we reduce the scale of
this noise, the number of training iterations allowed within the privacy
budget decreases. This constraint alters the optimal values of key hyper-
parameters like the batch size/learning rate, and defaults from non-private
training can be highly sub-optimal (Papernot et al., 2021). Consequently, DP-
SGD requires careful hyper-parameter tuning.222In this paper we do not account
for the privacy cost of hyper-parameter tuning (Papernot and Steinke, 2021),
and instead favour thorough sweeps on the same dataset in order to better
understand how the hyper-parameters of DP-SGD influence model performance.

We also found in our experiments that, when training with DP-SGD, improvements
in training accuracy usually translate directly to improved generalization,
without requiring strong regularization. Inspired by this observation, our
philosophy is that methods that reduce the number of training iterations
required to reach high training accuracy in non-private training are likely to
improve the test accuracy achieved in private training. Consistent with this
approach, it is usually beneficial to remove explicit regularization methods.

#### Bias and variance of the DP-SGD update.

The gradient estimator used by DP-SGD is biased because of the use of per-
example gradient clipping, and in general it does not correspond to the
gradient of any differentiable function (Song et al., 2021). More importantly,
the clipping norm $C$ introduces a bias-variance trade-off (Chen et al., 2020;
Thakkar et al., 2021). This can be viewed from the DP-SGD update shown in
Equation 3. When $C$ is very large, $\texttt{clip}_{C}$ is the identity
function, so the privatized gradient is an unbiased estimator of the true
gradient, but the clipped gradient $\frac{1}{C}\texttt{clip}_{C}\left(\nabla
l_{i}(w^{(t)})\right)$ is very small compared to the noise (which is
independent of $C$) – overall the privatized gradient estimate has low bias
and high variance. Conversely, if $C$ is small, the clipping operation
introduces bias, but the clipped gradient
$\frac{1}{C}\texttt{clip}_{C}\left(\nabla l_{i}(w^{(t)})\right)$ is larger,
and thus is not necessarily small compared to the noise – overall the
privatized gradient estimate has high bias and low variance. Note that when
$C$ is very small (smaller than the smallest per-example gradient norm),
further lowering $C$ does not change $\frac{1}{C}\texttt{clip}_{C}\left(\nabla
l_{i}(w^{(t)})\right)$, which indicates that the bias and variance in the
update both approach a constant as $C\rightarrow 0$. Intriguingly, previous
work has observed that wide ranges of the clipping norm $C$ can provide near
optimal performance provided that (i) the clipping norm is small enough, and
(ii) the learning-rate is re-scaled accordingly (Li et al., 2021; Kurakin et
al., 2022). This suggests that reducing the variance introduced by noise may
be more important than reducing the bias introduced by clipping.

#### Making standard models work.

Differentially private training has recently obtained promising results with
standard architectures in NLP, both when training a BERT model (Devlin et al.,
2018) from random initialization (Anil et al., 2021), and when fine-tuning a
large Transformer language model (Vaswani et al., 2017) from a pre-trained set
of parameters (Li et al., 2021; Yu et al., 2021a). However, similar results
have not been obtained in computer vision, and the literature does not provide
clear recommendations on which model architectures perform well. For instance,
surveying recent research on private training for CIFAR-10, Kurakin et al.
(2022), Papernot et al. (2021) and Dörmann et al. (2021) use variants of
shallow VGG models (Simonyan and Zisserman, 2015), while Tramèr and Boneh
(2021) use ScatterNets (Oyallon and Mallat, 2015) to train linear models on
handcrafted features, achieving an impressive $69.3\%$ test accuracy under a
tight privacy budget of $(3,10^{-5})$-DP. Finally, Klause et al. (2022)
achieve the SOTA test accuracy for $\varepsilon\leq 8$ without extra data of
$71.7\%$ when training a shallow 9-layer residual network (He et al., 2016)
under $(7.5,10^{-5})$-DP.

The $\ell_{2}$ norm of the noise added in the DP-SGD update scales
proportional to the dimension of the gradient (the number of parameters). This
observation has led many researchers to believe that standard over-
parameterized models will perform poorly with DP-SGD, and instead focus on
reducing the explicit or implicit dimension of the update, either through the
use of small models/hand-crafted features (Tramèr and Boneh, 2021) or through
dimensionality reduction techniques (Yu et al., 2021c, b). Another key
obstacle to the use of standard models for private training in computer vision
has been that in order to provide tight DP guarantees, DP-SGD requires that
the gradients evaluated on different training examples are independent. This
excludes the use of any method that enables communication between training
examples, such as batch normalization (Ioffe and Szegedy, 2015), which until
recently has been almost ubiquitous in standard vision architectures (He et
al., 2016; Zagoruyko and Komodakis, 2016; Tan and Le, 2019; Brock et al.,
2021b; Dosovitskiy et al., 2020).

##  3 Improving the Privacy-Utility Trade-off of DP-SGD in Image
Classification

In this section, we describe the key techniques we use to enhance the
performance of models trained with DP-SGD on standard image classification
benchmarks. In all the experiments in this section we train randomly
initialized models without using any extra data. In Section 3.1 we provide an
empirical ablation of these techniques on CIFAR-10, while in Sections 3.2 and
3.3 we evaluate the performance of our best models on CIFAR-10 and ImageNet.
We focus on standard architectures popular in the computer vision community,
since we believe that these models represent the most promising avenue for
achieving long-term progress.

###  3.1 Training on CIFAR-10 Without Additional Data – An Ablation Study

In Table 2 we provide an ablation study of a range of techniques, which
collectively significantly enhance the performance of DP-SGD when training
from random initialization on CIFAR-10 (Krizhevsky et al., 2009). These
techniques include replacing batch normalization with group normalization (Wu
and He, 2020), using large batch sizes, weight standardization (Qiao et al.,
2019), a modification to DP-SGD which we call augmentation multiplicity
(Hoffer et al., 2019; Touvron et al., 2021a; Fort et al., 2021) and parameter
averaging (Polyak and Juditsky, 1992). Prior research has already identified
group normalization and large batch sizes as beneficial for training with DP-
SGD (van der Maaten and Hannun, 2020; Luo et al., 2021; Kurakin et al., 2022;
Yu et al., 2021c; Li et al., 2021; Anil et al., 2021; Dörmann et al., 2021),
while to our knowledge the other techniques have not previously been used for
private training. We first define our baseline model/training pipeline, and
then discuss each modification in turn. For all experiments in this section,
we split the official CIFAR-10 training dataset of 50K examples into a
training set of 45K examples and a validation set of 5K examples. We train
with DP-SGD on this reduced training set under $(8,10^{-5})$-DP. In order to
avoid tuning on the official held-out test set, throughout this subsection we
report accuracies obtained on the training and validation set only.

#### Baseline model and private training pipeline for CIFAR-10.

As our baseline for CIFAR-10, we study the Wide-ResNet (WRN) model family
(Zagoruyko and Komodakis, 2016). We note that WRN models obtain high
accuracies on CIFAR-10 when trained non-privately: the 16 layer network with
width factor 4 (denoted as WRN-16-4) achieves >94$\%$ test accuracy, while the
WRN-40-4 achieves >95$\%$ (Zagoruyko and Komodakis, 2016). The model
parameters are initialized using Gaussian random variables following Glorot
and Bengio (2010). We train using DP-SGD without Momentum (Equation 3). We
observed no benefit from decaying the learning rate during training, and
therefore use a constant learning rate in all experiments reported in this
paper. We also do not use weight decay or dropout which we found to reduce
both training and validation accuracies for private training. This is similar
to observations made in Tramèr and Boneh (2021), although Anil et al. (2021)
reported improved performance when using weight decay with DP-Adam on BERT
models. Unless otherwise specified, we train without data augmentation. For
all experiments in this subsection, we tune the learning rate $\eta$ and the
noise parameter $\sigma$ on the validation set. We fix the clipping norm $C=1$
for all experiments in this paper. Given a target privacy guarantee
$(\varepsilon,\delta)$-DP and specific hyper-parameter settings for $\sigma$
and $q=B/N$, we compute the maximum number of training iterations $T$ using
the privacy accountant discussed in Section 2.2. We provide additional details
for each of our experiments in Appendix C.

Table 2:  An ablation study on the effect of a range of architectural modifications and changes to the training pipeline for models trained on CIFAR-10 under $(8,10^{-5})$-DP. We report median and standard deviation values over 5 runs. The baseline is a WRN-40-4 without batch normalization trained with DP-SGD using batch-size 256 without data augmentation. We report accuracy on our own validation set, not the official test set.  | Accuracy (%)
---|---
| Validation | Training
Baseline (WRN-40-4 w/o batch normalization) | 50.8  (0.7) | 51.2  (0.7)
\+ Group normalization (16 groups) | 66.3  (0.6) | 67.9  (0.3)
\+ Larger batch size (batch size of 4096) | 70.0  (0.6) | 73.4  (0.9)
\+ Weight standardization | 71.2  (1.0) | 74.7  (1.3)
\+ Augmentation multiplicity (16 augmentations) | 78.4  (0.9) | 79.4  (0.9)
\+ Parameter averaging (exponential moving average) | 79.7  (0.2) | 81.5  (0.2)

#### Training deep networks without batch normalization.

As mentioned previously, most computer vision architectures, including the WRN
model family, contain batch normalization layers, which are not compatible
with DP-SGD. Therefore, for our experiments on WRNs, we replace all batch
normalization layers by group normalization layers (Wu and He, 2020), as has
been previously done by several authors (van der Maaten and Hannun, 2020; Luo
et al., 2021; Kurakin et al., 2022; Yu et al., 2021c). Group normalization
splits the channels of the hidden activations of a single image into groups
and normalizes the hidden activations within each group. It therefore does not
break the independence between gradients evaluated on different examples. Note
however that it is crucial to place the group normalization layers on the
residual branch of the network to recover the benefits of batch normalization
for training deep networks (De and Smith, 2020). We fix the number of groups
of the group normalization layers to 16 in all our experiments. In Table 2, we
show that this simple change significantly improves the performance of deep
WRNs when training with DP-SGD.

A range of other alternatives to batch normalization have recently been
identified in non-private deep learning (Zhang et al., 2018b; De and Smith,
2020; Brock et al., 2021a; Kolesnikov et al., 2020). These alternatives can
train very deep networks (Zhang et al., 2018b; De and Smith, 2020) and some
even achieve superior train and test accuracies on standard benchmarks such as
ImageNet classification (Brock et al., 2021a, b). We will provide additional
results when training Normalizer-Free ResNets (Brock et al., 2021a) in later
sections.

Figure 2:  Increasing batch sizes on the WRN-16-4 model leads to improved
training and validation accuracy under $(8,10^{-5})$-DP. We plot the mean and
standard error across 5 independent runs.

Figure 3:  Increasing augmentation multiplicities on WRN-16-4 leads to
improved training and validation accuracy under $(8,10^{-5})$-DP. We plot the
mean/standard error across 5 independent runs.

#### Large batch sizes.

As observed in several recent papers (Hoory et al., 2021; Li et al., 2021; Luo
et al., 2021; Anil et al., 2021; Dörmann et al., 2021), increasing the batch
size can significantly improve the privacy-utility trade-off of DP-SGD. In
Table 2 we show that increasing the batch size from 256 to 4096 significantly
increases both training and validation accuracy. Additionally in Figure 3, we
evaluate the effect of the batch size on a WRN-16-4 where we sweep the batch
size from $2^{8}=256$ to $2^{14}=16384$. We find that performance consistently
improves as the batch size rises. The effect of increasing the batch size is
discussed further in Section 5.

We note that the gap between the training and validation accuracy in Figure 3
also increases with the batch size. As discussed by Yeom et al. (2018),
increased overfitting can be an indication of a rise in privacy leakage, and
training under DP implies generalization bounds that prevent overfitting when
$\varepsilon$ is small (Dwork et al., 2015; Bassily et al., 2016). However
these bounds degrade rapidly with increasing $\varepsilon$, and even the best
bounds we are aware of are vacuous in our setting (Feldman and Steinke, 2017;
Jung et al., 2020). In Section A.2 we audit our implementation using
membership inference attacks exploiting the generalization gap to assess
privacy leakage, and find that the observed gaps are within the theoretical
predictions for all values of $\varepsilon$ considered.

#### Weight standardization.

Next, we apply Weight Standardization (Qiao et al., 2019) to all convolutional
layers, which normalizes the rows of the weight matrix of each convolution
over the fan-in of each output unit (See Section C.1 for implementation
details). Several authors have shown that using weight standardization
combined with group normalization can be an effective replacement for batch
normalization in non-private training, especially when training with large
batch size (Qiao et al., 2019; Richemond et al., 2020; Kolesnikov et al.,
2020), while Brock et al. (2021a) found Weight Standardization improves the
performance of NF-ResNets. We found that weight standardization also improves
the performance of DP-SGD, particularly for large batch sizes. In Table 2, we
show the improvement in both the training and validation accuracies when using
weight standardization with group normalization and a large batch size of 4096
on the WRN-40-4 model.

#### Augmentation multiplicity.

Data augmentation is a crucial component of non-private training, which
significantly improves the performance of over-parameterized models on vision
tasks (Shorten and Khoshgoftaar, 2019; Zhang et al., 2018a; Cubuk et al.,
2020). However, in our experiments with DP-SGD, we found that data
augmentation as it is usually implemented in non-private training (using one
augmentation per independent example in the mini-batch) reduced both training
and validation accuracies (See Figure 3). We believe this occurs because data
augmentation introduces variance into the gradient (Fort et al., 2021), which
increases the number of training iterations required to minimize the loss. In
non-private training, several authors have shown that using multiple
augmentations per unique example in a mini-batch can improve performance
(Hoffer et al., 2019; Fort et al., 2021; Touvron et al., 2021b). We now
explore using multiple augmentations per example in the DP-SGD update to see
if we can recover the benefits of data augmentation in private training. The
naive approach when using multiple augmentations per example with DP-SGD would
be to compute one clipped gradient for each augmented image, however this
would lead to a privacy cost scaling with the number of augmentations per
training example in each mini-batch. To circumvent this issue, we average the
gradients of different augmentations of the same training example _before_
clipping the per-example gradients. This approach does not increase the
sensitivity of the mini-batch gradient to any single training example, and
therefore does not incur any additional privacy cost. The resulting update can
be written as follows:

$\displaystyle
w^{(t+1)}=w^{(t)}-\eta_{t}\left\\{\frac{1}{B}\sum\limits_{i\in\mathcal{B}_{t}}\frac{1}{C}\texttt{clip}_{C}\left(\frac{1}{K}\sum\limits_{j\in\mathcal{K}_{t}^{i}}\nabla
l_{j}(w^{(t)})\right)+\frac{\sigma}{B}\xi\right\\}\enspace.$ (4)

Here $\mathcal{K}_{t}^{i}$ represents the set of $K$ randomly sampled
augmentations for the $i^{th}$ training example on the $t^{th}$ iteration. We
refer to the number of per-example augmentations $K$ as the _augmentation
multiplicity_. When training with augmentation multiplicity on CIFAR-10, we
use random crops and random horizontal flips of the input images, following
Zagoruyko and Komodakis (2016). In Table 2, we show that augmentation
multiplicity, as described in Equation 4, significantly boosts both the
training and validation accuracy of the WRN-40-4 model. Additionally, we
perform a sweep of augmentation multiplicities ranging from 2 to 32 on the
WRN-16-4 model at a batch size of 4096 in Figure 3, and show that performance
consistently improves with higher augmentation multiplicities, significantly
exceeding the performance achieved without data augmentation.

#### Parameter averaging.

Since the privacy analysis for DP-SGD assumes that all intermediate parameter
values obtained during training can be released, parameter averaging
techniques do not incur any additional privacy cost. We found that using an
exponential moving average (EMA) of the parameters consistently improved final
accuracy on both the training and validation sets (Tan and Le, 2019), as shown
in Table 2 (See Appendix C for implementation details). All results in the
remainder of this paper use EMA unless specified otherwise.

Table 3:  CIFAR-10 test accuracy of our Wide-ResNet models trained with DP-SGD without additional data. For our results, we report the median computed over five independent runs as well as the standard deviation.  | $\varepsilon$ | Test Accuracy (%)
---|---|---
| Median | Std. Dev.
Tramèr and Boneh (2021) | 1 | 60.3 | –
2 | 67.2 | –
3 | 69.3 | –
Dörmann et al. (2021) | 1.93 | 58.6 | –
4.21 | 66.2 | –
7.42 | 70.1 | –
Klause et al. (2022) | 7.5 | 71.7 | –
Ours (WRN-16-4) | 1 | 56.8 | (0.6)
2 | 64.9 | (0.5)
3 | 69.2 | (0.3)
4 | 71.9 | (0.3)
6 | 77.0 | (0.8)
8 | 79.5 | (0.7)
Ours (WRN-40-4) | 1 | 56.4 | (0.6)
2 | 65.9 | (0.5)
3 | 70.7 | (0.2)
4 | 73.5 | (0.6)
6 | 78.8 | (0.4)
8 | 81.4 | (0.2)

###  3.2 Training on CIFAR-10 Without Additional Data – Official Evaluation

In Section 3.1, we described a collection of techniques which substantially
enhanced the performance of DP-SGD when training Wide-ResNets on CIFAR-10
(Zagoruyko and Komodakis, 2016). For those experiments, we split the dataset
of 50K examples into a training set of 45K examples and a validation set of 5K
examples, and did not evaluate on the held out test set. In this section, we
take the insights identified above, re-train our models on the full dataset of
50K examples, and evaluate performance on the official test set of 10K
examples. We run experiments on both WRN-16-4 and WRN-40-4, at a range of
$\varepsilon$ values. To determine the best hyper-parameters, we first train
on the reduced training set of 45K examples, tuning the learning rate $\eta$
and the noise parameter $\sigma$ on the validation set. We then expand the
training set to 50K examples and re-train without additional tuning. We report
the median accuracy of 5 independent training runs on the CIFAR-10 test set.
All experiments in this section apply group normalization, weight
standardization and parameter averaging as described above.

We first consider the WRN-16-4 model. Using a batch size of $B=4096$ and
augmentation multiplicity $K=16$, this model achieves a held-out test accuracy
of 78.7% under $(8,10^{-5})$-DP. Next, we consider the deeper WRN-40-4 model,
and scale up the batch size $B$ to $2^{14}=16384$ and the augmentation
multiplicity $K$ to $32$. With this setting, we achieve our best test accuracy
on CIFAR-10 of 81.4% under $(8,10^{-5})$-DP. This exceeds the previous SOTA on
CIFAR-10 for this privacy budget (Klause et al., 2022) by 9.7%. We also
provide the test accuracy across a range of $\varepsilon$ values between 1 and
8. For these experiments, we tune the learning rate $\eta$ and the noise
parameter $\sigma$ independently for each $\varepsilon$ (on the validation set
as described above). As shown in Figure 1(a) and the more detailed comparison
in Table 3, we improve the current SOTA for all values of $\varepsilon\geq 3$.
We note that for small values of $\varepsilon$, the model used by Tramèr and
Boneh (2021) performs best, likely due to its ability to get good performance
with a low number of updates. However, the capacity of this model is limited –
only reaching 71% in non-private training (Tramèr and Boneh, 2021), and thus
the more expressive WRN models outperform it whenever $\varepsilon\geq 3$. We
also note that the WRN-16-4 slightly outperforms the deeper WRN-40-4 model at
$\varepsilon=1$, while the WRN-40-4 performs better at all larger values of
$\varepsilon$. These observations are consistent with our experiments in
Section B.2, which show that the optimal model depth increases with
$\varepsilon$.

###  3.3 Training on ImageNet Without Additional Data

Table 4: Top-1 and top-5 accuracy when training on ImageNet using DP-SGD without additional data. Method | Model | $(\varepsilon,\delta)$ | Accuracy (%)
---|---|---|---
Top-1 | Top-5
Kurakin et al. (2022) | ResNet-18 | $(13.2,10^{-6})$ | 6.9 | –
Ours | NF-ResNet-50 | $(8.0,8\cdot 10^{-7})$ | 32.4 | 55.8

To confirm that the insights above can improve the performance of DP-SGD on
large datasets, we now train a classifier with DP from random initialization
on the ImageNet dataset (Russakovsky et al., 2015).

We use a Normalizer-Free ResNet-50 (NF-ResNet-50) (Brock et al., 2021a), which
is a modified version of the standard ResNet-50 model, designed to ensure good
signal propagation. The model does not contain batch normalization layers and
it applies weight standardization to convolutional layers (Qiao et al., 2019).
We note that this model has been shown to match the performance of the
standard batch normalized ResNet-50 for ImageNet classification in non-private
training (Brock et al., 2021a). We initialize the model parameters using
Gaussian random variables as for CIFAR-10 classification (Glorot and Bengio,
2010), and we remove all explicit regularization methods such as dropout,
stochastic depth, label smoothing and weight decay. We reserve 10K images from
the official training set of ImageNet as our validation set to tune hyper-
parameters, and train on the remaining 1.27 million training examples. We use
the official validation set of 50K examples as our test set. We use a batch
size of 16384 and set augmentation multiplicity $K=4$. See Appendix C for
further experimental details. Training on ImageNet is significantly more
computationally expensive than on CIFAR-10. We therefore only ran a very
limited sweep of the learning rate $\eta$ and the noise parameter $\sigma$ for
a single random seed.

As shown in Table 4, our NF-ResNet-50 model achieves a top-1 accuracy of 32.4%
and a top-5 accuracy of 55.8$\%$ under $(8,8\cdot 10^{-7})$-DP. We note that
this is a significant improvement on the previous SOTA for training on
ImageNet with DP without using additional data, which achieves a top-1
accuracy of 6.9$\%$ under a larger privacy budget of $(13.2,10^{-6})$-DP using
a ResNet-18 model (Kurakin et al., 2022).

##  4 High-Accuracy Differentially Private Image Classification with Fine-
Tuning

We now consider the setting where we have access to a large non-
sensitive/public dataset on which we can pre-train our model, using standard
non-private training. We then fine-tune this pre-trained model with DP-SGD on
the sensitive/private dataset. This approach has been recently used
successfully when privately fine-tuning language models (Li et al., 2021; Yu
et al., 2021a), and it has also been shown to improve the performance of DP-
SGD on both CIFAR-10 (Abadi et al., 2016; Tramèr and Boneh, 2021; Yu et al.,
2021b) and ImageNet (Kurakin et al., 2022). However, the best reported
performance of DP-SGD on image classification benchmarks is still
significantly lower than non-private training, particularly on large
challenging datasets like ImageNet.

Using the techniques described in Section 3, we now show that we can
significantly reduce the performance gap between private and non-private
models in the fine-tuning regime. In Section 4.1 we provide results when fine-
tuning on CIFAR-10 and CIFAR-100 from a model pre-trained on ImageNet. In
Section 4.2 we provide results when fine-tuning on ImageNet using models pre-
trained on either JFT-300M or JFT-4B (Sun et al., 2017), two internal datasets
containing 300 million and 4 billion labelled images respectively. Finally, to
confirm the versatility of our approach, in Section 4.3 we fine-tune from
JFT-300M to Places-365 (Zhou et al., 2017), a challenging scene recognition
dataset. For all experiments in this section, we explore both fine-tuning the
final layer and fine-tuning all layers simultaneously. Except on ImageNet,
almost all our best results are obtained by fine-tuning all layers of the
model simultaneously.

###  4.1 Fine-tuning on CIFAR-10 and CIFAR-100

In this section, we consider differentially-private fine-tuning on the
CIFAR-10 and CIFAR-100 datasets, using models that were pre-trained on
ImageNet without privacy. We consider WRN-28-10 and WRN-40-4 models, which we
pre-train on ImageNet using images down-sampled to $32\times 32$ (Chrabaszcz
et al., 2017). For pre-training, we use SGD with momentum for 240K iterations
with a constant learning rate and a batch size of 1024. We use a weight decay
of $5\cdot 10^{-5}$, and apply data augmentation including random flips and
crops of the input images with augmentation multiplicity 1 (See Appendix C for
further experimental details).

Table 5: CIFAR-10 and CIFAR-100 test accuracies when fine-tuning with DP-SGD a 28-10 Wide-ResNet pre-trained on ImageNet (down-sampled to $32\times 32$). We report the median accuracy across 5 runs. Fine-tuning Method | $\varepsilon$ | Test Accuracy (%)
---|---|---
CIFAR-10 | CIFAR-100
Median | Std. Dev. | Median | Std. Dev.
Yu et al. (2021b) | 1 | 94.3 | – | – | –
2 | 94.8 | – | – | –
Tramèr and Boneh (2021) | 2 | 92.7 | – | – | –
Classifier layer | 1 | 93.1 | (0.03) | 70.3 | (0.09)
2 | 93.6 | (0.05) | 73.9 | (0.32)
4 | 94.0 | (0.08) | 76.1 | (0.31)
8 | 94.2 | (0.07) | 77.6 | (0.10)
All layers | 1 | 94.8 | (0.08) | 67.4 | (0.17)
2 | 95.4 | (0.15) | 74.7 | (0.15)
4 | 96.1 | (0.06) | 79.2 | (0.24)
8 | 96.6 | (0.08) | 81.8 | (0.07)

We fine-tune the pre-trained model on both the CIFAR-10 and CIFAR-100 datasets
using DP-SGD. As in Section 3, we use a constant learning rate without
additional regularization techniques, although we do apply data augmentation
with augmentation multiplicity $K=16$. We explore two fine-tuning scenarios,
one where we fine-tune all layers of the model simultaneously, and a second
where we only train the final classifier layer. We show our results with the
WRN-28-10 model in Table 5. On CIFAR-10, we achieve a new SOTA test accuracy
of $94.7\%$ under $(1,10^{-5})$-DP, exceeding the previous SOTA of $94.3\%$ at
the same privacy budget from Yu et al. (2021b). We also provide test
accuracies at a range of different privacy budgets for both CIFAR-10 and
CIFAR-100. Under $(8,10^{-5})$-DP, we reach 96.7$\%$ on CIFAR-10 and 81.8$\%$
on CIFAR-100. On this task, fine-tuning all the layers of the model
simultaneously outperforms fine-tuning only the classifier layer.

We provide additional results for fine-tuning on CIFAR-10 with the WRN-40-4
model in Section B.3. While the 40-4 model also performs well and reaches a
test accuracy of $95.6\%$ under $(8,10^{-5})$-DP, the 28-10 model performs
better at all $\varepsilon$ values considered. To interpret these results, we
note that the WRN-28-10 is a stronger pre-trained model, achieving $62.4\%$
top-1 accuracy on the down-sampled ImageNet validation set, while the WRN-40-4
achieves $57.4\%$ top-1 accuracy. This phenomenon was consistent across all
our fine-tuning experiments: models which achieved stronger performance on the
pre-training task consistently achieved higher test accuracies after private
fine-tuning. Additionally, in Section B.4, we provide results for privately
fine-tuning on CIFAR-10 using a 40-4 Wide-ResNet model pre-trained non-
privately on CIFAR-100, where we again show consistent improvements over
previous baselines (Luo et al., 2021) at all values of $\varepsilon$.

Figure 4: Top-1 and top-5 accuracies of NF-ResNets on ImageNet under
$(8,8\cdot 10^{-7})$-DP, after fine-tuning either the last layer or all
layers. To reduce the cost of these experiments we fix $B=1024$ and
$\sigma=0.6$.

Figure 5:  Top-1 and top-5 accuracies of NF-ResNet-200 on ImageNet for a range
of privacy budgets $\varepsilon$ fine-tuning only the final layer with
$B=2^{18}$. We pre-train either on JFT-300M or on JFT-4B.

###  4.2 Fine-tuning on ImageNet

In this section, we fine-tune image classifiers with DP-SGD on the ImageNet
dataset, using models pre-trained either on JFT-300M or JFT-4B. JFT-300M is an
internal dataset of 300 million labelled images from 18k classes, while JFT-4B
contains 4 billion images from 30k classes (Sun et al., 2017). Pre-training on
JFT-300M or JFT-4B has been recently used by several authors to achieve SOTA
performance when fine-tuning on ImageNet without privacy (Brock et al., 2021b;
Dosovitskiy et al., 2020; Zhai et al., 2021). To preserve privacy guarantees
when fine-tuning on ImageNet, we removed all images from both JFT-300M and
JFT-4B that were exact or near-duplicates of ImageNet images across common
data augmentations (Kolesnikov et al., 2020).333In an earlier version, we
mistakenly used a version of JFT-300M de-duplicated with respect to the
validation set of ImageNet but not the training set. After repeating our
experiments on a new version of JFT-300M fully de-duplicated with respect to
both ImageNet and Places-365, a number of our results improved. We believe
this arises because the de-duplication process also removes low-quality
images.

We first use the NF-ResNet architecture (Brock et al., 2021a) (See Section
3.3), which we pre-train on JFT-300M for 10 epochs with the cross entropy
loss, using the training procedure described in Brock et al. (2021b).
Interestingly, Brock et al. (2021b) found that NF-ResNets significantly
outperformed batch-normalized ResNets when fine-tuning from JFT-300M to
ImageNet without privacy. We use the same training, validation and test splits
for ImageNet as in Section 3.3. We remove all explicit regularizers from the
model such as weight decay or dropout. When fine-tuning on ImageNet, we
explore both fine-tuning all layers of the network, as well as fine-tuning
only the final classifier layer. Unless otherwise specified, we fine-tune
using DP-SGD with a privacy budget of $(8,8\cdot 10^{-7})$-DP. Because of our
computational constraints, we could only afford to run limited sweeps with a
single random seed to identify the best hyper-parameters. We also do not use
augmentation multiplicity to avoid a further increase in the computational
cost of training, nor do we use any data augmentation.

We first compare fine-tuning all layers of the model to fine-tuning only the
final classifier layer, for a range of NF-ResNet model depths. We use a batch
size of 1024 and a fixed noise parameter $\sigma=0.6$, which corresponds to
roughly 136 training epochs. We tune the learning rate $\eta$ independently
for each model. Remarkably, we find in Figure 5 that an NF-ResNet-50 can reach
a top-1 accuracy of $75.4\%$ and top-5 accuracy of $93.0\%$ when fine-tuning
only the final classifier layer. This far exceeds the previous best reported
accuracy for private fine-tuning on ImageNet of $47.8\%$, achieved under a
privacy budget of $(10,10^{-6})$-DP (Kurakin et al., 2022). Note that Kurakin
et al. (2022) used Places-365 (Zhou et al., 2017) as the pre-training dataset,
which is a significantly smaller dataset compared to JFT-300M. In Figure 5, we
also see consistent benefits from increasing the model depth, both when fine-
tuning only the last layer and when fine-tuning all layers simultaneously. For
example, the NF-ResNet-200 model achieves a top-1 accuracy of $79.9\%$ when
fine-tuning only the final layer.

We note that, in Figure 5, fine-tuning only the final classifier layer
consistently performed slightly better than fine-tuning the whole network. We
therefore only report results when fine-tuning the final classifier layer for
our remaining experiments on ImageNet. This also considerably reduces the
computational cost of these experiments, since the features can be pre-
computed only once and stored on disk – assuming that no data augmentation is
used as is the case for our experiments in this section. This enables us to
use a very large batch-size of $B=2^{18}=262144$ for the rest of our ImageNet
experiments.

We provide the performance of NF-ResNet-200 at a range of privacy budgets
$\varepsilon$ in Figure 5, where we compare the effect of pre-training either
on JFT-300M or on the larger JFT-4B. We pre-train on JFT-300M for 10 epochs
and on JFT-4B for 1 epoch. When privately fine-tuning on ImageNet, we set
$T=4000$ (which corresponds to $\sigma=9.1$) when $\varepsilon=8$. We noticed
in Section 3.2 that the optimal number of training iterations scaled
approximately linearly with $\varepsilon$ when keeping other hyper-parameters
constant. We use this scaling rule for these experiments to reduce the cost of
tuning hyper-parameters. For example, at $\varepsilon=4$ we use half the
number of training iterations as our best run with $\varepsilon=8$. The noise
parameter $\sigma$ was scaled accordingly, while using the same learning rate.
From Figure 5, we see that using a larger and more diverse pre-training
dataset consistently improves the accuracy across all values of $\varepsilon$.
After pre-training on the larger JFT-4B dataset and scaling up the batch size,
the NF-ResNet-200 model achieves a top-1 accuracy of $82.2\%$ and top-5
accuracy of $96.1\%$ under $(8,8\cdot 10^{-7})$-DP. Note that these
experiments required a substantial number of training epochs at
$\varepsilon=8$ (roughly 800 epochs at $\sigma=9.1$), significantly larger
than is commonly used when fine-tuning in non-private training.

In all our fine-tuning experiments, we found that a stronger pre-trained
model, achieved either through using a larger model or through using a larger
pre-training dataset, always improved fine-tuning performance. Indeed, shortly
after the first version of our paper was released, Mehta et al. (2022)
published a study on differentially private fine-tuning on ImageNet using a
large ViT model pre-trained on JFT-4B, in which they achieve 81.1% top-1
accuracy under (1, $10^{-6}$)-DP and 81.7$\%$ under (4, $10^{-6}$)-DP. Our
best results when fine-tuning on ImageNet are achieved with an NFNet-F3 (Brock
et al., 2021b) pre-trained for 2 epochs on JFT-4B, which we fine-tune for
$T=1000$ training iterations using a batch size $B=2^{18}$. We show these
NFNet-F3 results in Table 6. We achieve a top-1 accuracy of $86.7\%$ and top-5
accuracy of $98.0\%$ under $(8,8\cdot 10^{-7})$-DP. Most remarkably, this
network also achieves a top-1 accuracy of $83.8\%$ under a much tighter
privacy budget of $(0.5,8\cdot 10^{-7})$-DP.

For comparison, fine-tuning the same pre-trained NFNet-F3 without privacy
using SGD with Momentum, AGC, and strong additional regularization including
Dropout, Weight Decay and Stochastic Depth (Brock et al., 2021b) achieves
$88.5\%$ top-1 accuracy on ImageNet, which is only $1.8\%$ higher than the
accuracy we achieve with DP at $\varepsilon=8$, and $4.7\%$ higher than the
accuracy we obtain at $\varepsilon=0.5$. We conclude that our results
significantly reduce the gap between private and non-private training on
ImageNet when fine-tuning.

Table 6: ImageNet top-1 accuracy when fine-tuning the last layer of an NFNet-F3 pre-trained on JFT-4B. Accuracy (%) | $\varepsilon$ | Non-private
---|---|---
$0.1$ | $0.5$ | $1.0$ | $2.0$ | $4.0$ | $8.0$
Top-1 | 77.6 | 83.8 | 84.4 | 85.6 | 86.0 | 86.7 | 88.5
Top-5 | 93.0 | 96.7 | 96.6 | 97.5 | 97.4 | 98.0 | 98.7
Table 7: Places-365 accuracy when fine-tuning with and without DP an NF-ResNet-50 pre-trained on JFT-300M. $\varepsilon$ | Fine-tuning Method | Accuracy (%)
---|---|---
Top-1 | Top-5
8 | Classifier layer | 54.4 | 84.4
All layers | 55.1 | 84.6
– | Classifier layer | 54.5 | 85.3
All layers | 57.0 | 87.1

###  4.3 Fine-tuning on Places-365

In this section, we provide results for fine-tuning with DP on the Places-365
dataset (Zhou et al., 2017). Places-365 is a dataset for scene recognition
containing 1.8 million training images from 365 scene categories. We use the
NF-ResNet-50 model (Brock et al., 2021a) in this section, which is pre-trained
on JFT-300M as described in Section 4.2. The current non-private SOTA on this
dataset is 60.7$\%$, obtained by a large Vision Transformer (Dosovitskiy et
al., 2020) after pre-training on a labelled dataset of 3.6 billion public
Instagram images (Singh et al., 2022). To the best of our knowledge, there is
no established SOTA for training on Places-365 with DP.

We fine-tune both with and without privacy, in order to illustrate the scale
of the privacy-utility trade-off. For both standard and private fine-tuning,
no data augmentation is applied and the learning rate is kept constant
throughout training. The batch size is set to 1024 for non-private training,
and to 4096 for private training. We do not use any weight decay. We tune the
learning rate independently for each experiment using a single seed. When
using DP-SGD, we set $\delta=5\cdot 10^{-7}$. We also compare the performance
of fine-tuning all layers and fine-tuning only the final classifier layer.
When training all layers with DP-SGD, we consider a single noise parameter
$\sigma=1$, while when fine-tuning the classifier layer we select the best
result from $\sigma\in\\{1,2,3\\}$.

We present our results in Table 7, where we provide the validation accuracy
when training with DP-SGD under $(8,5\cdot 10^{-7})$-DP, and without privacy
using SGD without momentum. Training all layers outperforms training the
classifier layer for both private and non-private training. Our non-private
baseline achieves a top-1 validation accuracy of $57.0\%$, which is within a
few percent of the current non-private SOTA on this dataset, while our best
private model achieves a top-1 accuracy of $55.1\%$, only $1.9\%$ lower than
our non-private baseline.

##  5 The Interplay between Noise, Batch Size, Compute Budget and Learning
Rate

Carefully tuning the DP-SGD hyper-parameters can significantly boost
performance. However intuitions from tuning non-private models do not always
carry over to private training. For example, in Figure 3 we observed an
improvement of ${\sim}8\%$ on the validation accuracy after increasing the
batch size $B$ and re-tuning the learning rate $\eta$ and noise parameter
$\sigma$. By contrast, in non-private training under a constant epoch budget,
the test accuracy is usually either constant or monotonically decreasing as
the batch size rises (Smith et al., 2017, 2020; Shallue et al., 2018; Goyal et
al., 2017). Below we present some of our findings from an empirical analysis
on the impact of different hyper-parameters on the performance of models
trained with DP-SGD.

In this section, we only provide experiments on CIFAR-10, using models trained
from random initialization without extra data, using the training pipeline
described in Section 3.1. On this smaller benchmark, we could perform a
rigorous empirical evaluation with thorough hyper-parameter sweeps. By
contrast, on large datasets (e.g., ImageNet), we were unable to tune hyper-
parameters to maximize performance, and instead tuned the noise parameter
$\sigma$ to ensure the training iteration budget $T$ was feasible within our
computational constraints.

Figure 6: Training WRN-16-4 on CIFAR-10 at batch size 4096 under
$(8,10^{-5})$-DP. (a) There is an optimal noise parameter,
$\sigma_{opt}\approx 3.0$, which maximizes validation accuracy. Note that we
tune the learning rate independently for each $\sigma$. (b) Since the training
iteration budget is a function of the batch size and the noise parameter, the
optimal noise parameter also implies an optimal budget of training iterations,
$T_{opt}\approx 2500$. (c) The optimal noise parameter/training iteration
budget maximizes training accuracy as well as validation accuracy.

#### There is an optimal noise parameter at fixed batch size, implying an
optimal budget of training iterations.

Given a privacy budget $(\varepsilon,\delta)$ and a batch size $B$, the noise
parameter $\sigma$ determines the maximum budget of training iterations $T$.
We now investigate whether it is always better to increase the value of
$\sigma$ and train longer, or if there is an optimal value of the noise
parameter beyond which the performance of DP-SGD degrades. In Figure 6, we
consider a batch size $B=4096$ on the WRN-16-4 under $(8,10^{-5})$-DP, and we
study the validation accuracy achieved across a range of different noise
parameters. We do not use data augmentation (or augmentation multiplicity) in
these experiments, and we re-tune the learning rate on the validation set for
each choice of $\sigma$. We find that there is an optimal value
$\sigma_{opt}\approx 3.0$ that achieves the highest validation accuracy.

The fact there is an optimal value for the noise parameter $\sigma$ implies
that there also exists an optimal value for the number of training iterations
$T$ (since $\sigma$ and $B$ determine $T$, given a privacy budget), which we
illustrate in Figure 6, where the validation accuracy reaches its maximal
value for $T_{opt}\approx 2500$. This observation also holds for the accuracy
on the training set, as illustrated in Figure 6. By contrast, increasing the
number of training iterations almost always achieves higher training accuracy
in non-private training (Smith et al., 2020).

Figure 7: Training a WRN-16-4 on CIFAR-10 under $(8,10^{-5})$-DP. (a) The
optimal training iteration budget falls as the batch size rises for small
batch sizes, but is roughly constant for large batch sizes. (b) The optimal
epoch budget is roughly constant for small batch sizes, but proportional to
batch size for large batch sizes. (c) The optimal learning rate is
proportional to batch size for small batch sizes, but constant for large batch
sizes.

#### The optimal epoch budget increases with the batch size.

In Figure 3, we showed that increasing the batch size improves the accuracy of
DP-SGD (after re-tuning other hyper-parameters). Meanwhile, in Figure 6 we
found that, at a fixed batch size, there is an optimal noise parameter
$\sigma_{opt}$ which implies a corresponding optimal budget of training
iterations $T_{opt}$. In Figure 7, we study the same WRN-16-4 model/training
pipeline described above, and we plot how the optimal iteration budget
$T_{opt}$ (which maximizes validation accuracy after tuning the noise
parameter and learning rate) depends on the batch size. We find that the
optimal iteration budget $T_{opt}$ falls rapidly when the batch size is small,
but is roughly constant when the batch size is large.

However if the number of training iterations is constant, larger batch sizes
will perform more training epochs. Therefore, in Figure 7, we show how the
corresponding optimal epoch budget depends on the batch size. We find that the
optimal epoch budget is roughly constant when the batch size is small, but
increases proportional to the batch size when the batch size is large. We
conclude that, although large batch sizes achieve superior accuracies with DP-
SGD, very large batch sizes also require more compute. We recommended that
practitioners who are compute constrained choose a moderate batch size in
practice (e.g., $B\approx 2048$ for this dataset/architecture). Note we
provide the train/validation accuracies corresponding to Figure 7 in Figure 3.

#### The optimal learning obeys a linear scaling rule for small batch sizes.

In Figure 7, we plot the relationship between the optimal learning rate (after
tuning the noise parameter) and the batch size. When the batch size is small,
the optimal learning rate is proportional to the batch size, while when the
batch size is large, the optimal learning rate is constant. Similar scaling
behaviour has previously been observed in non-private training (McCandlish et
al., 2018; Ma et al., 2018; Smith et al., 2020), while linearly scaling the
learning rate with the batch size was suggested as a heuristic for private
training by Tramèr and Boneh (2021), although as we show this scaling law
holds only up to a certain batch size. Notice that the optimal epoch budget in
Figure 7 begins to rise at the same batch size where the optimal learning rate
ceases to rise in Figure 7.

##  6 Discussion

#### Privacy and fairness considerations.

We have developed a methodology which achieves high accuracy on academic image
classification datasets with DP, however using this methodology for a real-
world application on sensitive data would involve several important additional
considerations. Although in this paper we consider a range of privacy budgets
with $\varepsilon$ ranging from below 1 to 8, choosing the privacy budget for
a real-world application is non-trivial and would depend on the specific
privacy and utility requirements (Abowd et al., 2020). In addition, our
experiments focus on providing record-level privacy at the level of individual
images. In scenarios where a single individual can contribute multiple images
to the training data, user-level privacy may be preferable (McMahan et al.,
2018b). Furthermore, several authors have shown that differentially private
training can potentially reduce the accuracy of models on under-represented
subgroups in the dataset (Carlini et al., 2019; Bagdasaryan et al., 2019;
Suriyakumar et al., 2021; Tran et al., 2021), and therefore deploying any
application with DP would require careful evaluation, particularly when the
dataset is imbalanced. Finally, in Section 4 we demonstrate that pre-training
our networks on large public/non-sensitive datasets significantly enhances the
performance achievable by DP-SGD, however this methodology may introduce
additional privacy risks if the dataset used for pre-training contains
sensitive data (Prabhu and Birhane, 2020).

#### The computational cost of private training.

We consistently found that private training requires significantly more
compute than non-private training to achieve optimal performance. The
computational cost of training with DP-SGD can be broken down into two
components: the cost of performing a single parameter update given a batch
size, and the number of parameter updates that need to be performed for the
model to reach a high accuracy. The cost of a single DP-SGD update is largely
dominated by the cost of computing per-example gradients, which is slower than
computing the averaged gradient and also requires more memory. While recent
deep learning frameworks like JAX (Bradbury et al., 2018) have significantly
reduced these overheads, DP-SGD remains slower than SGD in our experience. For
example, we observe that computing a parameter update at batch size 1024 for a
NF-ResNet-50 on a TPUv3 is about $9\times$ slower with DP-SGD using our
implementation than with standard SGD. Furthermore, we found that DP-SGD also
required more training epochs than non-private training to achieve optimal
performance. This observation is exacerbated by the use of large batch sizes
and/or large augmentation multiplicities, both of which further increase the
computational cost of training. We note that this computational cost is
significantly reduced on tasks where fine-tuning only the last layer is
sufficient to obtain high accuracy. In our experiments however, we typically
found fine-tuning the whole model to be optimal when transferring between
datasets other than from JFT-300M/4B to ImageNet.

#### Correctness.

While differentially private algorithms provide strong formal privacy
certificates, such guarantees are only realized if the algorithm is correctly
implemented, which can be prone to human errors arising from minor variations
in the algorithm (Lyu et al., 2017; Tramer et al., 2022). By releasing our
implementation we aim to make the code used in this paper verifiable by the DP
community. Additionally, we document key implementation details in Section
A.1, and we validate in Section A.2 that the claimed privacy guarantees of our
implementation are not contradicted by DP lower bounds obtained using
membership inference attacks (Tramer et al., 2022). Our implementation
includes unit-tests, and has undergone two independent internal technical
reviews for correctness.

## Conclusion

We have shown how to unlock high-accuracy image classification with
differential privacy by scaling up models, computational budgets, and pre-
training data in order to significantly reduce the utility gap between private
and non-private image classification on academic datasets. For instance, we
achieve $83.8\%$ top-1 accuracy on ImageNet under a $(0.5,8\cdot 10^{-7})$-DP
guarantee. All of our results are obtained using standard architectures
popular in the computer vision community with minimal modification, which
should make our methodology easy to re-use in existing pipelines. We hope that
our insights will significantly reduce the performance penalty that arises
when applying differentially private machine learning in real-world
applications.

## Acknowledgements

We would like to thank: Matthias Bauer and Sven Gowal for insightful comments
that helped improve the paper’s presentation; Robert Stanforth for engineering
support, open-sourcing support and code reviews; Rudy Bunel for code reviews;
John Aslanides for code quality reviews and open-sourcing support; Alison Reid
for support during the open-sourcing process; Andrew Trask and Thomas Steinke
for insightful discussions; Zahra Ahmed and Kitty Stacpoole for project
management support; Andrew Brock, Taylan Cemgil, Raia Hadsell, Koray
Kavukcuoglu, Pushmeet Kohli, Razvan Pascanu and Yee Whye Teh for advice
throughout the project. We would also like to thank Abhradeep Thakurta,
Florian Tramèr and Harsh Mehta for discussions on their related works.

## References

  * Abadi et al. (2016) M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang.  Deep learning with differential privacy.  In _Conference on Computer and Communications Security_ , 2016.
  * Abowd (2018) J. M. Abowd.  The U.S. census bureau adopts differential privacy.  In Y. Guo and F. Farooq, editors, _International Conference on Knowledge Discovery & Data Mining_, 2018.
  * Abowd et al. (2020) J. M. Abowd, G. L. Benedetto, S. L. Garfinkel, S. A. Dahl, A. N. Dajani, M. Graham, M. B. Hawes, V. Karwa, D. Kifer, H. Kim, P. Leclerc, A. Machanavajjhala, J. P. Reiter, R. Rodriguez, I. M. Schmutte, W. N. Sexton, P. E. Singer, and L. Vilhuber.  The modernization of statistical disclosure limitation at the u.s. census bureau.  https://www.census.gov/library/working-papers/2020/adrm/modernization-statistical-disclosure-limitation.html, 2020\.
  * Anil et al. (2021) R. Anil, B. Ghazi, V. Gupta, R. Kumar, and P. Manurangsi.  Large-scale differentially private BERT.  _arXiv_ , 2021.
  * Apple Differential Privacy Team (2017) Apple Differential Privacy Team.  Learning with privacy at scale, 2017.  URL https://docs-assets.developer.apple.com/ml-research/papers/learning-with-privacy-at-scale.pdf.
  * Asi et al. (2021) H. Asi, V. Feldman, T. Koren, and K. Talwar.  Private stochastic convex optimization: Optimal rates in L1 geometry.  In _International Conference on Machine Learning_ , 2021.
  * Babuschkin et al. (2020) I. Babuschkin, K. Baumli, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Budden, T. Cai, A. Clark, I. Danihelka, C. Fantacci, J. Godwin, C. Jones, T. Hennigan, M. Hessel, S. Kapturowski, T. Keck, I. Kemaev, M. King, L. Martens, V. Mikulik, T. Norman, J. Quan, G. Papamakarios, R. Ring, F. Ruiz, A. Sanchez, R. Schneider, E. Sezener, S. Spencer, S. Srinivasan, W. Stokowiec, and F. Viola.  The DeepMind JAX Ecosystem, 2020.  URL http://github.com/deepmind.
  * Bagdasaryan et al. (2019) E. Bagdasaryan, O. Poursaeed, and V. Shmatikov.  Differential privacy has disparate impact on model accuracy.  _Conference on Neural Information Processing Systems_ , 2019.
  * Balle et al. (2022) B. Balle, G. Cherubin, and J. Hayes.  Reconstructing training data with informed adversaries.  _arXiv_ , 2022.
  * Bassily et al. (2014) R. Bassily, A. D. Smith, and A. Thakurta.  Private empirical risk minimization: Efficient algorithms and tight error bounds.  In _Foundations of Computer Science_ , 2014.
  * Bassily et al. (2016) R. Bassily, K. Nissim, A. D. Smith, T. Steinke, U. Stemmer, and J. R. Ullman.  Algorithmic stability for adaptive data analysis.  In _Symposium on Theory of Computing_ , 2016.
  * Bird (2020) S. Bird.  Putting differential privacy into practice to use data responsibly, 2020\.  URL https://blogs.microsoft.com/ai-for-business/differential-privacy/.
  * Bradbury et al. (2018) J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Brock et al. (2021a) A. Brock, S. De, and S. L. Smith.  Characterizing signal propagation to close the performance gap in unnormalized resnets.  In _International Conference on Learning Representations_ , 2021a.
  * Brock et al. (2021b) A. Brock, S. De, S. L. Smith, and K. Simonyan.  High-performance large-scale image recognition without normalization.  _International Conference on Machine Learning_ , 2021b.
  * Bu et al. (2022) Z. Bu, J. Mao, and S. Xu.  Scalable and efficient training of large convolutional neural networks with differential privacy.  _arXiv preprint arXiv:2205.10683_ , 2022.
  * Carlini et al. (2019) N. Carlini, U. Erlingsson, and N. Papernot.  Distribution density, tails, and outliers in machine learning: Metrics and applications.  _arXiv_ , 2019.
  * Carlini et al. (2021a) N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramèr.  Membership inference attacks from first principles.  _arXiv_ , 2021a.
  * Carlini et al. (2021b) N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al.  Extracting training data from large language models.  In _USENIX Security Symposium_ , 2021b.
  * Chaudhuri et al. (2011) K. Chaudhuri, C. Monteleoni, and A. D. Sarwate.  Differentially private empirical risk minimization.  _Journal of Machine Learning Research_ , 2011.
  * Chen et al. (2020) X. Chen, Z. S. Wu, and M. Hong.  Understanding gradient clipping in private SGD: A geometric perspective.  In _Conference on Neural Information Processing Systems_ , 2020.
  * Choquette-Choo et al. (2021) C. A. Choquette-Choo, F. Tramèr, N. Carlini, and N. Papernot.  Label-only membership inference attacks.  In _International Conference on Machine Learning_ , 2021.
  * Chrabaszcz et al. (2017) P. Chrabaszcz, I. Loshchilov, and F. Hutter.  A downsampled variant of imagenet as an alternative to the cifar datasets.  _arXiv_ , 2017.
  * Cubuk et al. (2020) E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Conference on Computer Vision and Pattern Recognition_ , 2020\.
  * De and Smith (2020) S. De and S. L. Smith.  Batch normalization biases residual blocks towards the identity function in deep networks.  In _Conference on Neural Information Processing Systems_ , 2020.
  * Devlin et al. (2018) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv_ , 2018.
  * Dörmann et al. (2021) F. Dörmann, O. Frisk, L. N. Andersen, and C. F. Pedersen.  Not all noise is accounted equally: How differentially private learning benefits from large sampling rates.  _International Workshop on Machine Learning for Signal Processing_ , 2021.
  * Dosovitskiy et al. (2020) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv_ , 2020.
  * Dwork and Roth (2014) C. Dwork and A. Roth.  The algorithmic foundations of differential privacy.  _Foundations and Trends® in Theoretical Computer Science_ , 2014\.
  * Dwork et al. (2006) C. Dwork, F. McSherry, K. Nissim, and A. Smith.  Calibrating noise to sensitivity in private data analysis.  In _Theory of cryptography conference_ , 2006.
  * Dwork et al. (2015) C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth.  The reusable holdout: Preserving validity in adaptive data analysis.  _Science_ , 2015.
  * Erlingsson (2014) Ú. Erlingsson.  Learning statistics with privacy, aided by the flip of a coin, 2014.  URL https://ai.googleblog.com/2014/10/learning-statistics-with-privacy-aided.html.
  * Feldman and Steinke (2017) V. Feldman and T. Steinke.  Generalization for adaptively-chosen estimators via stable median.  In _Conference on Learning Theory_ , 2017.
  * Feldman et al. (2020) V. Feldman, T. Koren, and K. Talwar.  Private stochastic convex optimization: optimal rates in linear time.  In _Symposium on Theory of Computing_ , 2020.
  * Feldman et al. (2021) V. Feldman, A. McMillan, and K. Talwar.  Hiding among the clones: A simple and nearly optimal analysis of privacy amplification by shuffling.  In _Foundations of Computer Science, FOCS_ , 2021.
  * Fort et al. (2021) S. Fort, A. Brock, R. Pascanu, S. De, and S. L. Smith.  Drawing multiple augmentation samples per image during training efficiently decreases test error.  _arXiv_ , 2021.
  * Glorot and Bengio (2010) X. Glorot and Y. Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Conference on Artificial Intelligence and Statistics_ , 2010.
  * Golatkar et al. (2022) A. Golatkar, A. Achille, Y. Wang, A. Roth, M. Kearns, and S. Soatto.  Mixed differential privacy in computer vision.  _Conference on Computer Vision and Pattern Recognition_ , 2022.
  * Google (2018) Google.  Tensorflow privacy.  https://github.com/tensorflow/privacy, 2018.
  * Goyal et al. (2017) P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  _arXiv_ , 2017.
  * Hall et al. (2013) R. Hall, A. Rinaldo, and L. A. Wasserman.  Differential privacy for functions and functional data.  _Journal of Machine Learning Research_ , 2013.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Conference on Computer Vision and Pattern Recognition_ , 2016\.
  * Hoffer et al. (2019) E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry.  Augment your batch: better training with larger batches.  _arXiv_ , 2019.
  * Hoory et al. (2021) S. Hoory, A. Feder, A. Tendler, S. Erell, A. Peled-Cohen, I. Laish, H. Nakhost, U. Stemmer, A. Benjamini, A. Hassidim, et al.  Learning and evaluating a differentially private pre-trained language model.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , 2021.
  * Humphries et al. (2020) T. Humphries, M. Rafuse, L. Tulloch, S. Oya, I. Goldberg, U. Hengartner, and F. Kerschbaum.  Differentially private learning does not bound membership inference.  _arXiv_ , 2020.
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _International Conference on Machine Learning_ , 2015.
  * Jagielski et al. (2020) M. Jagielski, J. Ullman, and A. Oprea.  Auditing differentially private machine learning: How private is private sgd?  _Conference on Neural Information Processing Systems_ , 2020.
  * Jung et al. (2020) C. Jung, K. Ligett, S. Neel, A. Roth, S. Sharifi-Malvajerdi, and M. Shenfeld.  A new analysis of differential privacy’s generalization guarantees.  In _Innovations in Theoretical Computer Science Conference_ , 2020\.
  * Kifer et al. (2012) D. Kifer, A. D. Smith, and A. Thakurta.  Private convex optimization for empirical risk minimization with applications to high-dimensional regression.  In S. Mannor, N. Srebro, and R. C. Williamson, editors, _Conference on Learning Theory_ , 2012.
  * Klause et al. (2022) H. Klause, A. Ziller, D. Rueckert, K. Hammernik, and G. Kaissis.  Differentially private training of residual networks with scale normalisation.  _arXiv_ , 2022.
  * Kolesnikov et al. (2020) A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby.  Big transfer (bit): General visual representation learning.  In _European Conference on Computer Vision_ , 2020.
  * Koskela et al. (2021) A. Koskela, M. A. Heikkilä, and A. Honkela.  Tight accounting in the shuffle model of differential privacy.  _arXiv_ , 2021.
  * Krizhevsky et al. (2009) A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Kurakin et al. (2022) A. Kurakin, S. Chien, S. Song, R. Geambasu, A. Terzis, and A. Thakurta.  Toward training at imagenet scale with differential privacy.  _arXiv_ , 2022.
  * LeCun et al. (1998) Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 1998.
  * Li et al. (2021) X. Li, F. Tramèr, P. Liang, and T. Hashimoto.  Large language models can be strong differentially private learners.  _International Conference on Learning Representations_ , 2021.
  * Liu et al. (2021) Y. Liu, R. Wen, X. He, A. Salem, Z. Zhang, M. Backes, E. D. Cristofaro, M. Fritz, and Y. Zhang.  Ml-doctor: Holistic risk assessment of inference attacks against machine learning models.  _arXiv_ , 2021.
  * Luo et al. (2021) Z. Luo, D. J. Wu, E. Adeli, and L. Fei-Fei.  Scalable differential privacy with sparse network finetuning.  _Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Lyu et al. (2017) M. Lyu, D. Su, and N. Li.  Understanding the sparse vector technique for differential privacy.  _Proceedings of the VLDB Endowment_ , 2017.
  * Ma et al. (2018) S. Ma, R. Bassily, and M. Belkin.  The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning.  In _International Conference on Machine Learning_ , 2018.
  * McCandlish et al. (2018) S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team.  An empirical model of large-batch training.  _arXiv_ , 2018.
  * McMahan and Thakurta (2022) B. McMahan and A. Thakurta.  Federated learning with formal differential privacy guarantees, 2022.  URL https://ai.googleblog.com/2022/02/federated-learning-with-formal.html.
  * McMahan et al. (2018a) H. B. McMahan, G. Andrew, U. Erlingsson, S. Chien, I. Mironov, N. Papernot, and P. Kairouz.  A general approach to adding differential privacy to iterative training procedures.  _arXiv_ , 2018a.
  * McMahan et al. (2018b) H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang.  Learning differentially private recurrent language models.  _International Conference on Learning Representations_ , 2018b.
  * Mehta et al. (2022) H. Mehta, A. Thakurta, A. Kurakin, and A. Cutkosky.  Large scale transfer learning for differentially private image classification.  _arXiv_ , 2022.
  * Mironov et al. (2019) I. Mironov, K. Talwar, and L. Zhang.  Rényi differential privacy of the sampled gaussian mechanism.  _arXiv_ , 2019.
  * Nasr et al. (2021) M. Nasr, S. Songi, A. Thakurta, N. Papemoti, and N. Carlin.  Adversary instantiation: Lower bounds for differentially private machine learning.  In _Symposium on Security and Privacy (SP)_ , 2021.
  * Nayak (2020) C. Nayak.  New privacy-protected Facebook data for independent research on social media’s impact on democracy, 2020.  URL https://research.facebook.com/blog/2020/02/new-privacy-protected-facebook-data-for-independent-research-on-social-medias-impact-on-democracy/.
  * Nielsen and Sun (2018) F. Nielsen and K. Sun.  Guaranteed deterministic bounds on the total variation distance between univariate mixtures.  In _International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2018.
  * Oyallon and Mallat (2015) E. Oyallon and S. Mallat.  Deep roto-translation scattering for object classification.  In _Conference on Computer Vision and Pattern Recognition_ , 2015\.
  * Papernot and Steinke (2021) N. Papernot and T. Steinke.  Hyperparameter tuning with renyi differential privacy.  _arXiv_ , 2021.
  * Papernot et al. (2018) N. Papernot, S. Song, I. Mironov, A. Raghunathan, K. Talwar, and Ú. Erlingsson.  Scalable private learning with PATE.  In _International Conference on Learning Representations_ , 2018.
  * Papernot et al. (2021) N. Papernot, A. Thakurta, S. Song, S. Chien, and Ú. Erlingsson.  Tempered sigmoid activations for deep learning with differential privacy.  _AAAI Conference on Artificial Intelligence_ , 2021.
  * Polyak and Juditsky (1992) B. T. Polyak and A. B. Juditsky.  Acceleration of stochastic approximation by averaging.  _SIAM journal on control and optimization_ , 1992.
  * Prabhu and Birhane (2020) V. U. Prabhu and A. Birhane.  Large image datasets: A pyrrhic win for computer vision?  _arXiv_ , 2020.
  * Qiao et al. (2019) S. Qiao, H. Wang, C. Liu, W. Shen, and A. Yuille.  Micro-batch training with batch-channel normalization and weight standardization.  _arXiv_ , 2019.
  * Rakhlin et al. (2011) A. Rakhlin, O. Shamir, and K. Sridharan.  Making gradient descent optimal for strongly convex stochastic optimization.  _arXiv_ , 2011.
  * Richemond et al. (2020) P. H. Richemond, J.-B. Grill, F. Altché, C. Tallec, F. Strub, A. Brock, S. Smith, S. De, R. Pascanu, B. Piot, et al.  Byol works even without batch statistics.  _arXiv_ , 2020.
  * Russakovsky et al. (2015) O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _Internation Journal of Computer Vision_ , 2015.
  * Shallue et al. (2018) C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl.  Measuring the effects of data parallelism on neural network training.  _arXiv_ , 2018.
  * Shen et al. (2021) Y. Shen, Z. Wang, R. Sun, and X. Shen.  Towards understanding the impact of model size on differential private classification.  _arXiv_ , 2021.
  * Shorten and Khoshgoftaar (2019) C. Shorten and T. M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 2019.
  * Simonyan and Zisserman (2015) K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _International Conference on Learning Representations_ , 2015.
  * Singh et al. (2022) M. Singh, L. Gustafson, A. Adcock, V. d. F. Reis, B. Gedik, R. P. Kosaraju, D. Mahajan, R. Girshick, P. Dollár, and L. van der Maaten.  Revisiting weakly supervised pre-training of visual perception models.  _arXiv_ , 2022.
  * Smith et al. (2020) S. Smith, E. Elsen, and S. De.  On the generalization benefit of noise in stochastic gradient descent.  In _International Conference on Machine Learning_ , 2020.
  * Smith et al. (2017) S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V. Le.  Don’t decay the learning rate, increase the batch size.  _arXiv_ , 2017.
  * Song et al. (2013) S. Song, K. Chaudhuri, and A. D. Sarwate.  Stochastic gradient descent with differentially private updates.  In _Global Conference on Signal and Information Processing_ , 2013\.
  * Song et al. (2021) S. Song, T. Steinke, O. Thakkar, and A. Thakurta.  Evading the curse of dimensionality in unconstrained private glms.  In A. Banerjee and K. Fukumizu, editors, _Conference on Artificial Intelligence and Statistics_ , 2021.
  * Sun et al. (2017) C. Sun, A. Shrivastava, S. Singh, and A. Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In _International Conference on Computer Vision_ , 2017.
  * Suriyakumar et al. (2021) V. M. Suriyakumar, N. Papernot, A. Goldenberg, and M. Ghassemi.  Chasing your long tails: Differentially private prediction in health care settings.  In _Conference on Fairness, Accountability, and Transparency_ , 2021\.
  * Talwar et al. (2015) K. Talwar, A. Thakurta, and L. Zhang.  Nearly optimal private LASSO.  In _Conference on Neural Information Processing Systems_ , 2015.
  * Tan and Le (2019) M. Tan and Q. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International Conference on Machine Learning_ , 2019.
  * Thakkar et al. (2021) O. Thakkar, G. Andrew, and H. B. McMahan.  Differentially private learning with adaptive clipping.  _Conference on Neural Information Processing Systems_ , 2021.
  * Touvron et al. (2021a) H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou.  Training data-efficient image transformers & distillation through attention.  _International Conference on Machine Learning_ , 2021a.
  * Touvron et al. (2021b) H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou.  Going deeper with image transformers.  In _International Conference on Computer Vision_ , 2021b.
  * Tramèr and Boneh (2021) F. Tramèr and D. Boneh.  Differentially private learning needs better features (or much more data).  _International Conference on Learning Representations_ , 2021.
  * Tramer et al. (2022) F. Tramer, A. Terzis, T. Steinke, S. Song, M. Jagielski, and N. Carlini.  Debugging differential privacy: A case study for privacy auditing.  _arXiv_ , 2022.
  * Tran et al. (2021) C. Tran, M. H. Dinh, and F. Fioretto.  Differentially private deep learning under the fairness lens.  _arXiv_ , 2021.
  * van der Maaten and Hannun (2020) L. van der Maaten and A. Y. Hannun.  The trade-offs of private prediction.  _arXiv_ , 2020.
  * Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  _Conference on Neural Information Processing Systems_ , 2017.
  * Welford (1962) B. Welford.  Note on a method for calculating corrected sums of squares and products.  _Technometrics_ , 1962.
  * Wu et al. (2017) X. Wu, F. Li, A. Kumar, K. Chaudhuri, S. Jha, and J. F. Naughton.  Bolt-on differential privacy for scalable stochastic gradient descent-based analytics.  In S. Salihoglu, W. Zhou, R. Chirkova, J. Yang, and D. Suciu, editors, _International Conference on Management of Data_ , 2017.
  * Wu and He (2020) Y. Wu and K. He.  Group normalization.  _Internation Journal of Computer Vision_ , 2020.
  * Yeom et al. (2018) S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha.  Privacy risk in machine learning: Analyzing the connection to overfitting.  In _Computer Security Foundations Symposium_ , 2018.
  * Yousefpour et al. (2021) A. Yousefpour, I. Shilov, A. Sablayrolles, D. Testuggine, K. Prasad, M. Malek, J. Nguyen, S. Ghosh, A. Bharadwaj, J. Zhao, G. Cormode, and I. Mironov.  Opacus: User-friendly differential privacy library in PyTorch.  _arXiv_ , 2021.
  * Yu et al. (2021a) D. Yu, S. Naik, A. Backurs, S. Gopi, H. A. Inan, G. Kamath, J. Kulkarni, Y. T. Lee, A. Manoel, L. Wutschitz, S. Yekhanin, and H. Zhang.  Differentially private fine-tuning of language models.  _International Conference on Learning Representations_ , 2021a.
  * Yu et al. (2021b) D. Yu, H. Zhang, W. Chen, and T. Liu.  Do not let privacy overbill utility: Gradient embedding perturbation for private learning.  _International Conference on Learning Representations_ , 2021b.
  * Yu et al. (2021c) D. Yu, H. Zhang, W. Chen, J. Yin, and T.-Y. Liu.  Large scale private learning via low-rank reparametrization.  _International Conference on Machine Learning_ , 2021c.
  * Yu et al. (2022) J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Zagoruyko and Komodakis (2016) S. Zagoruyko and N. Komodakis.  Wide residual networks.  _The British Machine Vision Conference_ , 2016.
  * Zhai et al. (2021) X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer.  Scaling vision transformers.  _arXiv_ , 2021.
  * Zhang et al. (2018a) H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018a.
  * Zhang et al. (2018b) H. Zhang, Y. N. Dauphin, and T. Ma.  Fixup initialization: Residual learning without normalization.  In _International Conference on Learning Representations_ , 2018b.
  * Zhang et al. (2021) H. Zhang, I. Mironov, and M. Hejazinia.  Wide network learning with differential privacy.  _arXiv_ , 2021.
  * Zhou et al. (2017) B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba.  Places: A 10 million image database for scene recognition.  _Transactions on Pattern Analysis and Machine Intelligence_ , 2017\.
  * Zhou et al. (2021) Y. Zhou, S. Wu, and A. Banerjee.  Bypassing the ambient dimension: Private SGD with gradient subspace identification.  _International Conference on Learning Representations_ , 2021.

##  Appendix A Implementing and Auditing DP-SGD

All the experiments reported in this paper use a JAX (Bradbury et al., 2018)
implementation of DP-SGD based on JAXline (Babuschkin et al., 2020), a re-
usable framework for distributed model training and evaluation. Our full
implementation, including launch scripts for reproducing experiments that do
not depend on models pre-trained on JFT-300M, is available at

https://github.com/deepmind/jax_privacy

Besides enabling reproducibility of our results, another important reason for
open sourcing our code is to allow the differential privacy community to
verify our implementation of DP-SGD. To help navigate our code base, Section
A.1 provides an in-depth description of how our code parallelizes the
computation of privatized gradients across many devices when using virtual
batching and multiple augmentations. Furthermore, Section A.2 describes how we
used the methodology proposed by Tramer et al. (2022) to empirically test our
implementation against membership inference attacks.

Our implementation leverages the default Rényi DP privacy accountant for DP-
SGD provided by the TensorFlow Privacy (Google, 2018) implementation. As a
minor technical note, we remark that this accountant assumes that at each
iteration mini-batches are sampled with replacement from the entire dataset by
including every training example with probability $q$, while in practice we
sample mini-batches using a random shuffling scheme, such that each example is
sampled once per training epoch. This discrepancy commonly arises in empirical
studies of DP-SGD, as well as in popular open-source implementations (Google,
2018; Yousefpour et al., 2021). The main driver behind this discrepancy is the
pervasiveness of shuffling-based mini-batching in deep learning frameworks.
Obtaining tight numerical accounting methods for shuffled DP mechanisms based
on Gaussian perturbations remains an open problem (Feldman et al., 2021;
Koskela et al., 2021).

###  A.1 Implementation Details

Input: Current model parameters $w$, clipping norm $C$, noise multiplier
$\sigma$, device id $d$, per-device per-step batch-size $B_{local}$, number of
gradient accumulation steps $N_{acc}$, number of devices $N_{dev}$, number of
per-example augmentations $K$, training examples
$\\{(x_{d,s,i},y_{d,s,i}):s\in[N_{acc}],i\in[B_{local}]\\}$, shared noise
samples $\xi_{1},\ldots,\xi_{N_{acc}}\stackrel{{\scriptstyle
iid}}{{\sim}}\mathcal{N}(0,I)$

$B\leftarrow B_{local}\cdot N_{dev}\cdot N_{acc}$

$g\leftarrow 0$

for  _$s\in\\{1,\ldots,N_{acc}\\}$_ do

for  _$i\in\\{1,\ldots,B_{local}\\}$_ do

$g_{d,s,i}\leftarrow\frac{1}{C}\texttt{clip}_{C}\left(\frac{1}{K}\sum_{j=1}^{K}\nabla\mathcal{L}(w,\texttt{augment}(x_{d,s,i}),y_{d,s,i})\right)$

$g_{d,s}\leftarrow\frac{1}{B_{local}}\sum_{i}g_{d,s,i}$ // Average over local
mini-batch

$\hat{g}_{d,s}\leftarrow g_{d,s}+\frac{\sigma\sqrt{N_{acc}}}{B}\xi_{s}$ // Add
the same (scaled) noise on each device

$\bar{g}_{s}\leftarrow\frac{1}{N_{dev}}\sum_{d^{\prime}}\hat{g}_{d^{\prime},s}$
// Synchronize average gradient across devices

$g\leftarrow g+\frac{\bar{g}_{s}-g}{s}$ // Numerically stable averaging across
accumulation steps

return _$g$_ // Each device gets the same gradient

Algorithm 1 Private gradient computation across multiple devices with virtual-
batching, multiple augmentations, synchronized noise and gradient
normalization.

Algorithm 1 provides a high-level description of how our DP-SGD implementation
computes the privatized gradient used in each model update step. The structure
of the code is informed by the way model training pipelines are implemented in
JAXline. The implementation is parallelized across $N_{dev}$ devices, where
each device runs a copy of Algorithm 1. To extract the maximum possible
throughput from the implementation, each device processes training examples in
batches of size $B_{local}$, where this parameter is adjusted depending on the
memory available in each device and the size of model gradients for the
present architecture. To accommodate settings where the desired batch size for
a single model update is larger than $B_{local}\cdot N_{dev}$, our
implementation incorporates gradient accumulation (i.e. virtual batching)
where $N_{acc}$ gradient accumulation steps are performed before each model
update, giving a total batch size $B=B_{local}\cdot N_{dev}\cdot N_{acc}$.

As input to the gradient computation step, each device receives the current
model parameters $w$ (which are identical across devices), the desired
clipping norm $C$ and noise standard deviation $\sigma$, and their device
identifier $d\in\\{1,\ldots,N_{dev}\\}$. In addition, each device $d$ has
access to $N_{acc}\cdot B_{local}$ training examples
$\\{(x_{d,s,i},y_{d,s,i}):s\in[N_{acc}],i\in[B_{local}]\\}$, and $N_{acc}$
i.i.d. samples from a standard Gaussian distribution
$\xi_{1},\ldots,\xi_{N_{acc}}$. Crucially, these samples are _shared_ across
devices, meaning that noise is independent across gradient accumulation steps
but not across devices. This is enforced in our implementation by broadcasting
the same pseudo-random number generator key to all devices – this is preferred
over having a different PRNG key per-device because it makes the pipeline more
reproducible across training infrastructures with different numbers of
devices.

In the innermost loop of Algorithm 1, client $d$ computes the individual
contribution $g_{d,s,i}$ of a single example $x_{d,s,i}$ (where $s$ indexes
the accumulation step and $i$ the local batch-size). As discussed in Section
3.1, in our algorithm this contribution is based on the average parameter
gradient with respect to $K$ random augmentations of $x_{d,s,i}$. These
augmentations are returned by _independent_ calls to the augment subroutine.
The result of this average is then clipped to a maximal $\ell_{2}$ norm of $C$
by the $\texttt{clip}_{C}$ function and then normalized by $C$ (cf. Equation
3). This results in a contribution $g_{d,s,i}$ to the model update with norm
bounded by 1, which is then averaged locally over $i$ on device $d$ to produce
$g_{d,s}$. We note that although the loop over $i\in[B_{local}]$ is presented
in Algorithm 1 as a sequential computation for convenience, in reality our
implementation uses hardware parallelism offered by modern accelerators – this
means that as long as the device can process $B_{local}\cdot K$ examples in
parallel, the cost of computing $g_{d,s}$ is constant in these parameters.

After computing the average of clipped gradient contributions $g_{d,s}$, each
device $d$ adds appropriately calibrated Gaussian noise to obtain
$\hat{g}_{d,s}$ – we emphasize again that in step $s$ each device shares the
same random noise sample, so the noise adding process is not independent
across devices. At this point devices synchronize their updates by computing
the average of $\hat{g}_{d,s}$ over $d\in[N_{dev}]$. After this step, each
device has the same averaged noisy gradient $\bar{g}_{s}$. Finally, each
device updates their local copy of the accumulated gradient $g$ by using
Welford’s incremental averaging algorithm for numerical stability (Welford,
1962).

That Algorithm 1 provides a correct implementation of the privatized gradients
required by DP-SGD follows by comparing the following result to Equation 3.

######  Lemma A.1.

Each device participating in Algorithm 1 obtains the same noisy gradient given
by

$\displaystyle
g=\left\\{\frac{1}{B}\sum_{d=1}^{N_{dev}}\sum_{s=1}^{N_{acc}}\sum_{i=1}^{B_{local}}\frac{1}{C}\texttt{clip}_{C}\left(\frac{1}{K}\sum_{j=1}^{K}\nabla\mathcal{L}(w,\texttt{augment}(x_{d,s,i}),y_{d,s,i})\right)\right\\}+\frac{\sigma}{B}\xi\enspace,$
(5)

where $\xi\sim\mathcal{N}(0,I)$.

###### Proof.

By construction it is clear that each device gets the same gradient. Now let
$g^{(s)}$ be the value of $g$ (on an arbitrary device) at the end of iteration
$s$ of the outermost loop. By induction on $s$ we can show that
$g^{(s)}=\frac{1}{s}\sum_{s^{\prime}=1}^{s}\bar{g}_{s^{\prime}}$: it is clear
that $g^{(1)}=\bar{g}_{1}$, and

$\displaystyle
g^{(s+1)}=g^{(s)}+\frac{\bar{g}_{s+1}-g^{(s)}}{s+1}=\frac{sg^{(s)}+\bar{g}_{s+1}}{s+1}=\frac{1}{s+1}\sum_{s^{\prime}=1}^{s+1}\bar{g}_{s^{\prime}}\enspace,$

where the last identity follows by the inductive hypothesis. Thus, at the end
of the algorithm each device gets
$g=g^{(N_{acc})}=\frac{1}{N_{acc}}\sum_{s=1}^{N_{acc}}\bar{g}_{s}$. Unrolling
the computations done at every accumulation step on every device we get:

$\displaystyle g$
$\displaystyle=\frac{1}{N_{acc}}\sum_{s=1}^{N_{acc}}\bar{g}_{s}$
$\displaystyle=\frac{1}{N_{acc}\cdot
N_{dev}}\sum_{s=1}^{N_{acc}}\sum_{d=1}^{N_{dev}}\hat{g}_{d,s}$
$\displaystyle=\frac{1}{N_{acc}\cdot
N_{dev}}\sum_{s=1}^{N_{acc}}\sum_{d=1}^{N_{dev}}g_{d,s}+\frac{\sigma}{\sqrt{N_{acc}}\cdot
N_{dev}\cdot B}\sum_{s=1}^{N_{acc}}\sum_{d=1}^{N_{dev}}\xi_{s}$
$\displaystyle=\frac{1}{N_{acc}\cdot N_{dev}\cdot
B_{local}}\sum_{s=1}^{N_{acc}}\sum_{d=1}^{N_{dev}}\sum_{i=1}^{B_{local}}g_{d,s,i}+\frac{\sigma}{\sqrt{N_{acc}}\cdot
B}\sum_{s=1}^{N_{acc}}\xi_{s}\enspace.$

The result now follows by observing that the first term in the sum above
equals the first term in Equation 5 and
$\frac{1}{\sqrt{N_{acc}}}\sum_{s=1}^{N_{acc}}\xi_{s}\sim\mathcal{N}(0,I)$. ∎

###  A.2 Privacy Lower Bounds via Membership Inference Attacks

Inspired by recent work on auditing DP algorithms with privacy attacks
(Jagielski et al., 2020; Nasr et al., 2021; Tramer et al., 2022), we carry out
membership inference attacks on models trained with Algorithm 1. The extent to
which these attacks are successful can be used to provide (empirical) _lower
bounds_ on the privacy guarantees afforded by our implementation, which can
then be compared with the nominal upper bound. Note that this type of test is
incomplete; a failure to find a violation of the upper bound does not rule out
the possibility that one exists. However, along with independent code reviews
and unit testing, it provides another signal that our implementation of DP-SGD
is correct.

#### DP lower bounds via hypothesis testing.

Given two datasets, $D$ and $D^{\prime}=D\cup\\{z\\}$, that differ by a single
data point $z$, a model trained with a DP algorithm reduces an adversary’s
ability to infer via a simple hypothesis test whether the model was trained on
$D$ or $D^{\prime}$. More formally, if an algorithm is
$(\varepsilon,\delta)$-DP then it must satisfy
$\ln{(\frac{1-\beta-\delta}{\alpha})}\leq\varepsilon$ (Hall et al., 2013),
where $\alpha$ and $\beta$ denote the Type I and Type II errors of the
hypothesis testing procedure. If we can construct datasets $D$ and
$D^{\prime}$ such that this bound is violated, we can be sure the algorithm
does not provide $(\varepsilon,\delta)$-DP; evaluating
$\ln{(\frac{1-\beta-\delta}{\alpha})}$ through a membership inference attack
will give a valid lower bound of $\varepsilon$. In practice, the test relies
on comparing the loss (distribution) of point $z$ under models trained with
$D$ and $D^{\prime}$.

#### Overview of the attack.

Note that DP is a worst-case guarantee, and so we are free to design our
datasets $D$ and $D^{\prime}$ in any way we choose. In addition, our algorithm
provides the same privacy guarantees for many setting of its hyper-parameters.
Our attack operates in two phases: in phase I, we design a learning problem
that we expect will maximize the ability to infer membership, while in phase
II, we use the procedure set out by Tramer et al. (2022) to run a membership
inference attack to find a (statistically valid) lower bound for
$\varepsilon$. In phase I, we sweep over different hyper-parameter
configurations and choices for $z$, to find a choice that is likely to
maximize the ratio $\nicefrac{{1-\beta-\delta}}{{\alpha}}$ when the model is
trained with Algorithm 1. In phase II, we use the best configuration from
phase I and train a large number of models to ensure the lower bound is
statistically valid with enough confidence. For completeness, we repeat the
same procedure with two dataset sizes ($|D|\in\\{100,60K\\}$) and four
settings of the nominal privacy parameters: $\varepsilon\in\\{1,2,4,8\\}$,
where we set $\delta=\nicefrac{{1}}{{|D|}}$.

#### Experimental details.

Throughout, we will use a simple LeNet classifier (LeCun et al., 1998) as the
trained model and take $D$ from the MNIST dataset. The hyper-parameters and
choices of $z$ we evaluated in phase I are given in Table 8. Given a choice of
$(\varepsilon,\delta)$, we find a choice of hyperparameters that maximizes
distinguishability between models trained on $D$ and $D^{\prime}$ as follows:
for each hyperparameter configuration, we train 1K models on $D$ and 1K models
on $D^{\prime}$, where the only randomness originates from the noise added
from DP-SGD. In particular, all models are initialized with the same
parameters, as prior work has shown random initialization weakens privacy
attacks (Jagielski et al., 2020; Balle et al., 2022). We then record the loss
on $z$ for each model trained on $D$ and on $D^{\prime}$, and fit two
Gaussians over these histograms. After this, we record the total variation
distance between these two Gaussians (Nielsen and Sun, 2018), and choose the
hyperparameter configuration that maximizes this distance. The choice of
learning rate, clipping norm, and number of model updates that maximize our
ability to infer if $z$ was included in training varied for each
$\varepsilon\in\\{1,2,4,8\\}$ and $|D|\in\\{100,60K\\}$. However, we found
that selecting $z$ to be a blank image was a consistently better choice than
using uniform noise or mislabeling an MNIST test set example.

In phase II, we repeat the procedure set out by Tramer et al. (2022) for each
choice of privacy parameters, dataset size and best hyper-parameters from
phase I. For both $D$ and $D^{\prime}$ we train 500K models, reserving the
first 10K models on each setting as the set from which we find a threshold
that maximizes the ratio between $1-\beta$ and $\alpha$, and the remaining
models are used to evaluate this chosen threshold and report the $\varepsilon$
lower bound. To gain statistical confidence in our $\varepsilon$ lower bounds
we compute the lower bound of the true positive rate, $1-\beta$, and upper
bound of the false positive rate, $\alpha$, over the remaining 980K models
using Clopper-Pearson confidence intervals. With an appropriate choice of
significance level this gives us a probabilistic $\varepsilon$ lower bound
with 99.9% confidence.

#### Results.

Our results are given in Table 9; we did not find any violation of our
reported $(\varepsilon,\delta)$-DP guarantees. Note that our lower bounds when
using $|D|=100$ are substantially tighter than when training with $|D|=60K$ –
for example at nominal $\varepsilon>1$ training on a small dataset yields
$\varepsilon$ lower bounds which are over $2\times$ stronger. We also provide
membership inference AUC and advantage ($1-\beta-\alpha$) scores for inferring
if $z$ was or was not included in training over the 980K models, and give
upper bounds to the membership advantage that can be derived in closed form
from $\varepsilon$ and $\delta$ (c.f. Humphries et al. (2020)). Throughout our
entire experiment we chose to train models without augmentation multiplicity
to expedite experimentation, however, we ran an additional experiment for
$\varepsilon=1$ and $|D|=100$ where we turned on augmentation multiplicity
during training with 16 augmentations per image, and found the $\varepsilon$
lower bound was almost unchanged.

Table 8: Phase I of the $\varepsilon$ lower bound experiment: Finding the best
choice of hyperparameters that distinguish models trained with and without
$z$.

Hyperparameter | Values
---|---
$z$ | Uniform noise: , Blank: , Label 7 as 8: , Label 6 as 7: , Label 0 as 1:
Learning rate | 0.1, 0.5, 1.0
Clipping norm | 0.1, 1.0, 10.0
$|D|$ | 60K | 100
Batch size | 4096 (accumulated over two steps of batch size 2048) | 64
Number of updates | 500, 1K | 50, 100

Table 9: Phase II of the $\varepsilon$ lower bound experiment: Reporting
probabilistic $\varepsilon$ lower bounds with 99.9% confidence and the
membership inference (inferring if $z$ was or was not used in training) AUC
and advantage ($1-\beta-\alpha$).

$|D|$ | Nominal $\varepsilon$ |  $\varepsilon$ lower bound | Membership AUC | Membership advantage | Membership advantage upper bound
---|---|---|---|---|---
60K | 1 | 0.279 | 0.54 | 0.06 | 0.46
2 | 0.456 | 0.57 | 0.10 | 0.76
4 | 1.139 | 0.62 | 0.17 | 0.96
8 | 2.153 | 0.72 | 0.31 | 1.00
100 | 1 | 0.361 | 0.59 | 0.13 | 0.47
2 | 0.837 | 0.66 | 0.22 | 0.76
4 | 2.461 | 0.79 | 0.43 | 0.96
8 | 4.327 | 0.91 | 0.67 | 1.00

Note, it is unlikely that taking $D$ to be the MNIST dataset is the optimal
choice for maximizing privacy lower bounds; one could imagine taking $D$ to be
a pathological dataset with the same pixel value everywhere, and taking $z$ to
be an image with different pixel values, would improve distinguishability
between models trained on $D$ or $D^{\prime}$. However, our aim is to verify
that the end-to-end implementation of our DP training procedure is correct,
including the data pre-processing part of the pipeline. Data pre-processing
(including augmentations) may have a null effect on such a pathological
dataset, meaning if an error was introduced in this stage of the training
pipeline it would not be captured by our membership attack. Therefore, we
chose to use the MNIST dataset to increase the likelihood that any errors from
data pre-processing pipeline will be reflected and included in our membership
attack. We leave the design of datasets to elicit the best lower bounds for
future work.

##  Appendix B Additional Experimental Results

###  B.1 Choice of the Clipping Norm under the Modified DP-SGD Update

In the DP-SGD update as described in Equation 2, the clipping norm $C$
determines the norm of the parameter update, and therefore affects the optimal
choice of the learning-rate. This results in an undesirable dependency between
the clipping norm and the learning rate, which can make the performance of DP-
SGD appear to be sensitive to the choice of the clipping norm when the
learning rate is not carefully tuned (Li et al., 2021). To decouple the effect
of the learning rate and the clipping norm, we instead use a slightly modified
version of the DP-SGD update as described in Equation 3, where the learning
rate $\eta$ absorbs a factor of $C$. Using this update, performance becomes
less sensitive to the choice of clipping norm, as we demonstrate in Figure 8
where we show the accuracy of the WRN-16-4 over a range of clipping norms and
learning rates at batch size 4096 and augmentation multiplicity 16. More
importantly, the optimal choice of the learning rate does not depend on the
clipping norm under this modified DP-SGD update, as long as the clipping norm
is lower or equal to 1. Therefore, to reduce the cost of hyper-parameter
tuning, for all of our experiments in this paper, we fix $C=1$.

Figure 8: Sensitivity of performance to clipping norm for different learning
rates.

###  B.2 The Optimal Model Depth Depends on the Privacy Budget and Training
Setup

Scaling the depth of neural networks has been crucial in achieving good
performance with non-private training (He et al., 2016; Tan and Le, 2019;
Brock et al., 2021b). In this paper, we were interested in exploring whether
we could leverage the same benefits of deep models when training with DP-SGD.
However, in our initial experiments on WRN models for CIFAR-10 classification
without extra data under $(8,10^{-5})$-DP, we found that both training and
validation set performance degraded for networks deeper than the WRN-16-4.

To understand this, in Figure 9, we train a range of Wide-ResNets at different
depths (from 16 to 100 layers) using a fixed noise parameter $\sigma=2$ and a
batch size of 4096 until exhausting a very large privacy budget of
$(64,10^{-5})$-DP, and track the performance of these models during training.
We emphasize that $\varepsilon=64$ is too large to provide a meaningful notion
of privacy, but our goal here is to gain an understanding for the training
dynamics. In our initial experiment, we train without data augmentation. We
find that the deeper models start converging slower than the shallower models,
likely due to the added noise in DP-SGD. However, when trained for long enough
(i.e., for a sufficiently large privacy budget), the deeper models outperform
the shallower models. This is illustrated in Figure 9(a), where we show that
at low $\varepsilon$ values (which corresponds to early in training for this
experiment), the WRN-16-4 outperforms all deeper models, while at high
$\varepsilon$ values (i.e., late in training), some of the deeper models start
to outperform the shallower models.

Interestingly, we find that using data augmentation with large augmentation
multiplicities significantly speeds up convergence of the deeper models.
Performing the same experiment as before using augmentation multiplicity
$K=16$, we see in Figure 9(c) that even early in training at $\varepsilon=8$,
the WRN-40-4 significantly outperforms the WRN-16-4. Late in training, at
larger values of $\varepsilon$, a much deeper network is optimal compared to
when not using augmentation multiplicity. We note this effect of speeding up
convergence of deeper models is not simply due to using data augmentation (See
Figure 9(b) using augmentation multiplicity 1).

We conclude from these results that the optimal network depth depends on the
privacy budget, as well as other choices made during training. We notice this
in Table 3, where at a small privacy budget of $(1,10^{-5})$-DP, the WRN-16-4
slightly outperforms the WRN-40-4. For larger privacy budgets, we find the
WRN-40-4 performs better. Additionally, we find the optimal network depth to
also depend on whether we train without using any extra data, or if we
privately fine-tune a pre-trained network. In the fine-tuning setting, we find
larger models to consistently perform better than shallower networks, even at
small $\varepsilon$ values, as shown throughout Section 4.

(a) $K=0$ (no augmentation)

(b)  $K=1$

(c)  $K=16$

Figure 9: Validation accuracy at different points in training (corresponding
to different values of $\varepsilon$) for WRNs at different depths and
different augmentation multiplicities $K$, trained with fixed noise parameter
$\sigma=2$.

###  B.3 Private Fine-Tuning on CIFAR-10 Using a WRN-40-4 Model Pre-trained on
ImageNet-32

In Section 4.1, we provided results for privately fine-tuning a WRN-28-10 on
CIFAR-10 and CIFAR-100, which was pre-trained on ImageNet-32. We now provide
additional results for this same task using a pre-trained WRN-40-4 model
instead, and with the same experimental setup as in Section 4.1. We show
results in Table 10. We fine-tuned all layers of the model simultaneously,
which we found to perform better than fine-tuning only the last layer for this
experiment. We show that we achieve 95.6% test accuracy under
$(8,10^{-5})$-DP. As mentioned in Section 4.1, we find that the WRN-28-10 to
outperform the WRN-40-4 at all values of $\varepsilon$ considered, likely due
to the WRN-28-10 model being a better pre-trained model.

Table 10: Test accuracy on fine-tuning on CIFAR-10 using a WRN-40-4 pre-trained on ImageNet-32. Fine-tuning Method | $\varepsilon$ | Test Accuracy (%)
---|---|---
All layers | 1 | 93.6
2 | 94.3
4 | 95.1
8 | 95.6

###  B.4 Private Fine-Tuning on CIFAR-10 Using a WRN-40-4 Model Pre-Trained on
CIFAR-100

In Section 4.1, we provided results for privately fine-tuning WRNs on CIFAR-10
and CIFAR-100 using a model pre-trained on ImageNet-32. We now provide
additional results, when fine-tuning with DP-SGD on CIFAR-10, using a model
pre-trained without privacy on CIFAR-100. We compare our results with Luo et
al. (2021), who provide the current SOTA for this task. We use the WRN-40-4
model, which we pre-train on CIFAR-100 using SGD with momentum for 100K
iterations with a batch size of 128. We use the same fine-tuning procedure on
CIFAR-10 as in Section 4.1 (See Appendix C), and use batch size 4096 and
augmentation multiplicity $K=16$.

We present our results in Table 11. We fine-tuned all layers of the model
simultaneously, which we found to perform better than fine-tuning just the
last layer for this experiment, similar to Section 4.1. As before, we
outperform previous SOTA results for this task at all privacy budgets
considered. In particular we achieve a test accuracy of $80.4\%$ under
$(1,10^{-5})$-DP and a test accuracy of $89.0\%$ under $(8,10^{-5})$-DP. We
note however that the accuracies achieved on CIFAR-10 after pre-training on
CIFAR-100 are significantly lower than those obtained after pre-training on
ImageNet. This emphasizes the importance of using a strong pre-trained model
if we wish to achieve performance competitive with non-private training.

Table 11: Test accuracy on CIFAR-10 when privately fine-tuning from the WRN-40-4 pre-trained on CIFAR-100. We report the median test accuracy and the standard deviation across 5 independent runs. $\varepsilon$ | Method | Test Accuracy (%)
---|---|---
Median | Std. Dev.
0.5 | Luo et al. (2021) | 73.3 | –
Ours | 76.7 | (0.3)
1.0 | Luo et al. (2021) | 76.6 | –
Ours | 80.4 | (0.7)
1.5 | Luo et al. (2021) | 81.6 | –
Ours | 83.8 | (0.3)
2 | Ours | 84.9 | (0.2)
4 | Ours | 87.0 | (0.2)
6 | Ours | 88.4 | (0.2)
8 | Ours | 89.0 | (0.2)

###  B.5 Optimal Noise Scale for a Fixed Batch Size: Experiments on the
WRN-40-4

In Figure 6, we showed that on the WRN-16-4, there is an optimal noise scale,
and consequently an optimal compute budget, for a fixed batch size of 4096. To
provide additional evidence of this, we show a similar plot in Figure 10 on a
WRN-40-4 trained on CIFAR-10 under $(8,10^{-5})$-DP and using a batch size of
16384 (with all other experimental details the same as in Figure 6). We found
this property to hold also at other batch sizes.

Figure 10: Training a WRN-40-4 on CIFAR-10 at batch size 16384 under
$(8,10^{-5})$-DP.

###  B.6 Scaling of the Compute Budget with the Batch Size: Experiments on the
WRN-40-4

In Figure 7, we showed how the optimal budget of training iterations and
epochs and the optimal learning rate varied with the batch size on the
WRN-16-4 under $(8,10^{-5})$-DP. To provide additional evidence of the main
takeaways from Figure 7, we provide additional experiments under the same
experimental setup with the WRN-40-4 model in Figure 11 where we observe
qualitatively the same behaviour as in Figure 7.

Figure 11: Optimal compute budgets and optimal learning rates at a range of
batch sizes on the WRN-40-4 under $(8,10^{-5})$-DP.

##  Appendix C Additional Experimental Details

###  C.1 Model Details

#### Wide-ResNet models.

We use the Wide-ResNet model family from Zagoruyko and Komodakis (2016) for
all of our experiments on CIFAR-10 and CIFAR-100, and replace all the batch
normalization layers with group normalization layers with the number of groups
set to 16. We use Gaussian initialization for all weight parameters with
variance $(1/\text{fan-in})$ (Glorot and Bengio, 2010). We also use Weight
Standardization for each convolutional layer as follows:

$\hat{W}_{i,j}=\dfrac{W_{i,j}-\hat{\mu}_{W_{i,\cdot}}}{\hat{\sigma}_{W_{i,\cdot}}\sqrt{\text{fan-
in}}},$

where $\hat{\mu}_{W_{i,\cdot}}$ and $\hat{\sigma}_{W_{i,\cdot}}$ denote the
empirical mean and the standard deviation calculated over the fan-in extent
for the convolutional layer $W$. This is similar to the implementation in
Brock et al. (2021a). We note for clarity that we employ Weight
Standardization throughout training as a differentiable operation. We remove
explicit regularizers such as dropout from the model.

#### NF-ResNet models.

We use the Normalizer-Free (NF) ResNet family of models from Brock et al.
(2021a) in most of our experiments on ImageNet and Places-365. When training
on ImageNet from random initialization (without using any extra data) as
described in Section 3.3, we initialize the SkipInit parameter in the NF-
ResNets to 1.0, which we found to result in slightly faster convergence when
training NF-ResNets with DP-SGD. We remove all explicit regularizers from the
model such as dropout and stochastic depth.

#### NFNet models.

We use the Normalizer-Free F3 Network (NFNet) from Brock et al. (2021b) to
achieve our best results when fine-tuning on ImageNet. As before, we remove
all explicit regularizers from the model.

###  C.2 Dataset Details

#### Overview.

When using data augmentation, we perform random crops and random horizontal
flips at training time. We never use data augmentation at evaluation time.
When the augmentation multiplicity is set to 0, it means that no data
augmentation is used. Our best hyper-parameters are selected on the validation
set, and we report the corresponding performance on the test set for our best
results, and on the validation set for all other analysis and ablation
experiments. The data splits used for each dataset are reported in Table 12.

Table 12: Number of samples in each split per dataset. Dataset | Train | Valid | Train + Valid | Test
---|---|---|---|---
CIFAR-10 | 45,000 | 5,000 | 50,000 | 10,000
CIFAR-100 | 45,000 | 5,000 | 50,000 | 10,000
ImageNet | 1,271,167 | 10,000 | – | 50,000
ImageNet-32 | 1,271,167 | 10,000 | – | 50,000
Places-365 | 1,803,460 | – | – | 36,500

#### CIFAR.

When results are reported on the validation set, the model is trained on the
$45K$ training images. When results are reported on the test set, the model is
trained on the full $50K$ training $+$ valid images. We ensure that the DP
accountant uses the corresponding number of examples to calculate the
$\varepsilon$ guarantee. Each image is centered and standardized per channel.
When using random crops, every image is padded with 4 pixels on each edge with
mirror padding.

#### ImageNet.

We construct a validation set from the official training set of the ImageNet
dataset, so that we can cross-validate hyper-parameters on it. We report our
best results on the official validation set. Each image is centered and
standardized per channel. For all our experiments training from random
initialization without any extra data (as in Section 3.3), we use an image
size of $224\times 224$ both during training and evaluation. When fine-tuning
on ImageNet from a pre-trained model (as in Section 4.2), we noticed that
using larger image sizes for training and evaluation improved performance, and
we use an image size of $320\times 320$ for all our experiments. When using
random crops, we first resize the image to have $1.2$ times the desired height
and width, and then take a random crop of the desired size within this large
image.

#### ImageNet-32.

We construct a validation set from the official training set similar to the
ImageNet dataset. In the ImageNet-32 dataset, each image from the ImageNet
dataset is down-sampled to $32\times 32$. We follow the same image down-
sampling procedure as described in Chrabaszcz et al. (2017). Each image is
translated and normalized so as to have values between -1 and 1. When using