# Improved AI-generated Solar Farside Magnetograms by STEREO and SDO Data Sets
and Their Release

Hyun-Jin Jeong School of Space Research, Kyung Hee University, Yongin, 17104,
Republic of Korea  Yong-Jae Moon School of Space Research, Kyung Hee
University, Yongin, 17104, Republic of Korea  Department of Astronomy and
Space Science, College of Applied Science, Kyung Hee University, Yongin,
17104, Republic of Korea  Eunsu Park Department of Astronomy and Space
Science, College of Applied Science, Kyung Hee University, Yongin, 17104,
Republic of Korea  Harim Lee Department of Astronomy and Space Science,
College of Applied Science, Kyung Hee University, Yongin, 17104, Republic of
Korea  Ji-Hye Baek Technology Center for Astronomy and Space Science, Korea
Astronomy and Space Science Institute, Daejeon, 34055, Republic of Korea
Space Science Division, Korea Astronomy and Space Science Institute, Korea
Astronomy and Space Science Institute, Daejeon, 34055, Republic of Korea Yong-
Jae Moon<EMAIL_ADDRESS>
###### Abstract

Here we greatly improve Artificial Intelligence (AI)-generated solar farside
magnetograms using data sets of Solar Terrestrial Relations Observatory
(STEREO) and Solar Dynamics Observatory (SDO). We modify our previous deep
learning model and configuration of input data sets to generate more realistic
magnetograms than before. First, our model, which is called Pix2PixCC, uses
updated objective functions which include correlation coefficients (CCs)
between the real and generated data. Second, we construct input data sets of
our model: solar farside STEREO extreme ultraviolet (EUV) observations
together with nearest frontside SDO data pairs of EUV observations and
magnetograms. We expect that the frontside data pairs provide the historic
information of magnetic field polarity distributions. We demonstrate that
magnetic field distributions generated by our model are more consistent with
the real ones than before in view of several metrics. The averaged pixel-to-
pixel CC for full disk, active regions, and quiet regions between real and AI-
generated magnetograms with 8 by 8 binning are 0.88, 0.91, and 0.70,
respectively. Total unsigned magnetic flux and net magnetic flux of the AI-
generated magnetograms are consistent with those of real ones for test data
sets. It is interesting to note that our farside magnetograms produce
consistent polar field strengths and magnetic field polarities with those of
nearby frontside ones for solar cycle 24 and 25. Now we can monitor the
temporal evolution of active regions using solar farside magnetograms by the
model together with the frontside ones. Our AI-generated Solar Farside
Magnetograms (AISFMs) are now publicly available at Korean Data Center (KDC)
for SDO.

Solar magnetic fields (1503), Convolutional neural networks (1938), The Sun
(1693), Astronomy data analysis (1858)

††software: PyTorch (Paszke et al., 2019), NumPy (Harris et al., 2020),
Matplotlib (Hunter, 2007), SciPy (Virtanen et al., 2020), Astropy (Robitaille
et al., 2013; Price-Whelan et al., 2018), SunPy (The SunPy Community et al.,
2020)

##  1 Introduction

Magnetic fields play a fundamental role in producing solar extreme events,
i.e., solar flares and coronal mass ejections (Wiegelmann et al., 2014; Judge
et al., 2021). A series of ground-based and space-borne magnetographs have
provided solar magnetic field data to study the field’s origin and evolution
over the last tens of years (Pietarila et al., 2013). The solar magnetograph
is an instrument producing a map of magnetic field strength and/or direction
on the Sun, and the map is called magnetogram (Babcock, 1953). Helioseismic
and Magnetic Imager (HMI; Scherrer et al., 2012) on board SDO, which is in
geosynchronous orbit, has provided high-resolution magnetograms of the entire
solar disk. Recently, Polarimetric and Helioseismic Imager (PHI; Solanki et
al., 2020) on board Solar Orbiter starts obtaining data of photospheric fields
from outside the Sun-Earth line (Müller et al., 2020).

Before the Solar Orbiter mission, twin STEREO spacecrafts provided the first
stereoscopic view of the Sun drifting ahead and behind the Earth’s orbit
(Kaiser et al., 2008). The STEREO Ahead (A) and Behind (B) were launched in
2006 and offered a complete 360 degree view of the entire Sun with the
frontside observations. The STEREO data, together with the SDO ones, have been
widely used to study the solar atmospheric phenomena in three dimensions
(Sterling et al., 2012; Caplan et al., 2016; Zhou et al., 2021). However,
because they did not have magnetograph, there was a limit to studying magnetic
activities from the frontside to the farside of the Sun during the STEREO era.

Kim, Park, Lee et al. (2019; hereafter KPL19) generated solar farside
magnetograms from the STEREO/Extreme UltraViolet Imager (EUVI; Howard et al.,
2008) 304 Å observations using deep learning. KPL19 applied Pix2Pix model
(Isola et al., 2017) which is a widely used deep learning model in image
translation tasks. They trained and evaluated the model with pairs of solar
frontside SDO/Atmospheric Imaging Assembly (AIA; Lemen et al., 2011) 304 Å
observations and SDO/HMI line-of-sight (LOS) 720s magnetograms. Results of
KPL19 showed that the farside magnetograms could be used to monitor the
temporal evolution of active regions. However they set the upper and lower
saturation limits of the field strength at $\pm 100$ Gauss, because their
model worked well with proper byte scaling (Park et al., 2021). Their model
well generated the distributions and shapes of the active regions, but it was
hard to produce original scale magnetic fluxes (Liu et al., 2021). Jeong et
al. (2020; hereafter Jeong20) improved the AI-generated magnetograms using an
upgraded deep learning approach with $\pm 3000$ Gauss dynamic range based on
the Pix2PixHD model (Wang et al., 2018) and multi-channel SDO/AIA images of
171, 195, and 304 Å for the model input. Jeong20 showed that their results
could reproduce strong magnetic fluxes and distribution of polarities for not
only active regions (ARs), but also quiet regions (QRs). They applied the AI-
generated farside ones to a part of the boundary conditions for the
extrapolation of coronal magnetic fields. Results of the application were much
more consistent with coronal farside EUV observations than those of the
conventional method.

In the present study we generate more accurate solar farside magnetograms than
those of KPL19 and Jeong20. For this we make an upgraded model including a CC-
based objective with additional input data: not only farside STEREO EUV images
but also frontside data pairs of SDO/AIA EUV images and HMI magnetograms as
reference information. In this paper, we call the farside data generated by
KPL19 AISFM 1.0, the data generated by Jeong20 AISFM 2.0, and our data AISFM
3.0, respectively. We describe detailed structure of our model in Section 2,
and our data configurations in Section 3. We show our evaluation results of
the model trained with the frontside evaluation data sets in Section 4.1. Then
we generate AISFMs by the model from the corresponding images of STEREO A (or
B) and the frontside reference data pairs, and show the results in Section
4.2. We release the AISFMs 3.0, and describe the data in Section 5. We
conclude our study in Section 6.

##  2 Deep Learning Model

We use a deep learning model called Pix2PixCC model to generate solar farside
magnetograms from the farside EUV observations and reference data pairs.
Figure 1 shows the main structure of our model. The model consists of three
major components: a generator ($G$), a discriminator ($D$), and an inspector
($I$). The generator is a generative network, and tries to produce target-like
data from input with the help of the discriminator and the inspector. The
discriminator is a discriminative network, which attempts to distinguish
between the more realistic pair between a real pair and a generated pair. The
real pair consists of an input data and a target data. The generated pair
consists of an input data and an output from the generator. The generator gets
updated with objectives from the discriminator, and tries to generate the best
outputs to fool the discriminator. The inspector computes CCs between the
target data and output from the generator to produce realistic values for the
generated ones.

Figure 1:  Flowchart and structures of Pix2PixCC model. $G$, $D$, and $I$ are
generator, discriminator, and inspector, respectively. The generator produces
target-like data from input data. When we train the model, input data are
SDO/AIA EUV 304, 193, and 171 Å images and a reference data pair. The
reference data pair is composed of the three SDO EUV images and the
corresponding SDO/HMI magnetogram which are the nearest available ones. The
discriminator trains for distinguishing between the real pair and generated
pair. The real pair consists of an input data and a target data, and the
generated pair consists of an input data and a generated data. The inspector
computes CCs between the target data and generated data. The generator and
discriminator are updated from the losses calculated from the inspector and
discriminator.

The generator and discriminator are multi-layer networks. The multiple layers
of the generative network are composed of several convolutional and
transposed-convolutional filters of which parameters are updated during the
model training process. Briefly, the convolutional filters try to extract
features automatically from the input data, and the transposed-convolutional
filters attempt to reconstruct outputs from the extracted features. For the
detailed function of the filters in the network, refer to Goodfellow et al.
(2016) and Buduma & Locascio (2017). The discriminative network consists of
several convolution layers. Each convolution layer generates a feature map
based on input. Our model has a Feature Matching (FM) loss ($Loss_{FM}$),
which is an objective function to optimize the parameters of the generator.
The FM loss is to minimize the absolute difference between the feature maps of
the real and generated pair from multiple layers of the discriminator. It is
more effective for a large dynamic range data than the loss function derived
from the absolute difference between the target and generated data directly
(Rana et al., 2019; Marnerides et al., 2021). The objective function
$Loss_{FM}$ is given by

${Loss}_{FM}(G,D)=\sum_{i=1}^{T}\frac{1}{N_{i}}\Big{\|}D^{(i)}(x,y)-D^{(i)}(x,G(x))\Big{\|},$
(1)

where $x$, $y$, $T$, $i$, and $N_{i}$ are input data, target data, the total
number of layers, the serial number of the layers, and the number of elements
in output feature maps of each layer, respectively. $G(x)$ means output data
from the generator. $D^{(i)}$ denotes the $i$th-layer feature extractor of the
discriminator.

Our networks use Least Squares Generative Adversarial Network (LSGAN) losses
(Mao et al., 2017). The LSGAN losses update the generator ($Loss^{G}_{LSGAN}$)
and the discriminator ($Loss^{D}_{LSGAN}$), and are obtained by

$\displaystyle{Loss}_{LSGAN}^{G}(G,D)=\frac{1}{2}\Big{(}D(x,G(x))-1\Big{)}^{2}$
(2)
$\displaystyle{Loss}_{LSGAN}^{D}(G,D)=\frac{1}{2}\Big{(}D(x,y)-1\Big{)}^{2}+\frac{1}{2}\Big{(}D(x,G(x))\Big{)}^{2},$

where $D(x,y)$ and $D(x,G(x))$ are probabilities in the range of 0 (generated)
to 1 (real) at the end of discriminator from real pair and generated pair,
respectively. While the generator tries to minimize the $Loss^{G}_{LSGAN}$,
and discriminator tries to minimize $Loss^{D}_{LSGAN}$. The competition
between the generator and the discriminator contributes to generate realistic
data. The performance of the adversarial objectives have been well
demonstrated in image-to-image translation tasks for solar data(Park et al.,
2019; Shin et al., 2020; Lim et al., 2021; Son et al., 2021). The Pix2PixHD
model of Jeong20 uses both FM and LSGAN losses.

In order for stable training of the generator, we use an additional objective
function called Correlation Coefficient (CC) loss ($Loss_{CC}$). It is known
that the CC-based loss function has resulted in better performance than error-
based loss functions: mean squared error, mean absolute error, etc (Vallejos
et al., 2020; Atmaja & Akagi, 2021). We use Lin’s concordance CC, which takes
bios into Pearson’s CC (Lawrence & Lin, 1989). The concordance CC is commonly
used to assess the reproducibility evaluating the degree to which pairs of
data fall on the $45^{\circ}$ line through the origin. The range of
concordance CC is from -1 (perfect disagreement) to 1 (perfect agreement). The
inspector compute the CC loss with multi-scale target and generated data. The
function of $Loss_{CC}$, which maximizes the agreement between target and
generated data, is defined as

$\displaystyle{Loss}_{CC}(G)=\sum_{i=0}^{T}\frac{1}{T+1}\Big{(}1-CC_{i}(y,G(x))\Big{)},$
(3)

where T and i are the total number of downsampling by a factor of 2 and the
serial number of the downsampling, respectively. $CC_{i}$ means the CC value
between the $2^{i}$ times downsampled target and AI-generated data. The
average of the CC values from multi-scale target and generated data helps the
model to optimize the network parameters. In addition, with the help of
$Loss_{CC}$, we do not impose artificial saturation limits on our model.

Our final objectives are as follows:

$\displaystyle\min_{G}\Big{(}\lambda_{1}{Loss}^{G}_{LSGAN}(G,D)+\lambda_{2}{Loss}_{FM}(G,D)+\lambda_{3}{Loss}_{CC}(G)\Big{)}$
(4) $\displaystyle\min_{D}\Big{(}{Loss}^{D}_{LSGAN}(G,D)\Big{)},$

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyperparameters which
control the importance of $Loss^{G}_{LSGAN}$, $Loss_{FM}$, and $Loss_{CC}$,
respectively. We use 2, 10, 5 for $\lambda_{1}$, $\lambda_{2}$ and
$\lambda_{3}$, respectively. To minimize the objective functions, we use an
adaptive moment estimation (Adam; Kingma & Ba, 2015) optimizer with learning
late 0.0002. We train the model for 1,000,000 iterations, and save the model
and AI-generated data from the evaluation inputs at every 10,000th iteration.
We evaluate all the saved models by the metrics and, use the highest scoring
model to generate farside magnetograms. Our code is available at
https://github.com/JeongHyunJin/Pix2PixCC, and more details of our model are
described in the readme file.

##  3 Data sets

###  3.1 Training data sets

Here we use SDO/AIA EUV 304, 193, and 171 Å images and SDO/HMI LOS
magnetograms to train our deep-learning model. The three EUV passbands
correspond to the chromosphere, corona, and upper transition region of the
Sun, respectively. We use multi-channel inputs to generate target
magnetograms. Channel dimensions of the inputs are composed of three EUV
passband images and a reference data pair. The reference data pair is composed
of three SDO/AIA images and a SDO/HMI magnetogram which were observed one
solar rotation (27.3 days) before. We expect that the differences between the
EUV images and the reference data pair give the model information on how the
intensities or distributions of the magnetic fields have changed.

We use pairs of train data sets with 6 hour cadence (at 01, 07, 13, and 19 UT
each day) from 2011 January 1 to 2021 June 30. We select ten months of data
per year, excluding the data sets for the evaluation of our model. The months
are shifted by four months each year to train various solar inclination
conditions. We take 6437 pairs for the training data sets. We remove data with
poor quality which are flagged by nonzero value of the QUALITY keyword for
both AIA and HMI data sets. Then we align them to have same rotational axis,
pixel size of solar radius ($R_{\odot}$), and location of the disk center. We
downsample them from $4096\times 4096$ to $1024\times 1024$ for computational
capability. The radius of the Sun is fixed at 450 pixels. A mask is applied to
the area outside 0.998 $R_{\odot}$ from the disk center for minimizing the
uncertainty of limb data. All EUV data number are scaled by median values on
the solar disk to calibrate the gradual in-orbit degradation of the AIA
instrument (Ugarte-Urra et al., 2015; Liewer et al., 2017).

###  3.2 Evaluation data sets

We use the remaining SDO data pairs, which are two months of data per year, to
evaluate our model. Among them, we use the pairs of data sets from 2011 to
2017 to compare with the results of KPL19 and Jeong20. We take 1342 pairs for
the evaluation data sets. The pre-processing steps of the evaluation data are
the same as those of training data.

When we compute objective measures between the target magnetograms and AI-
generated data to evaluate our model, we compare the results not only data for
full disk, but also data for ARs and QRs. We select areas of the ARs and the
QRs with a size of $128\times 128$ pixels from the pre-processed target
magnetograms with $1024\times 1024$ pixel resolution. We don’t consider the
areas outside 60 degrees from the disk center to exclude limb data with
uncertainty. We compute the total unsigned magnetic flux (TUMF) for the area,
moving at intervals of 64 pixels up, down, left, and right from the center of
solar disk. When the TUMF of the area is greater than $5\times 10^{21}$ Mx,
the area is classified as an AR (Waldmeier, 1955; van Driel-Gesztelyi & Green,
2015). Otherwise, the area are candidates of QR. The boundaries of all
detected areas do not overlap with one another. In order to balance the number
of AR and QR areas, we select up to 3 AR areas for each magnetogram in the
order of the largest TUMF, and up to 2 QR areas for each magnetogram in the
order of the lowest TUMF.

Figure 2:  Automatic selected areas of AR (red boxes) and QR (blue boxes) in
SDO/HMI magnetogram acquired at 2013 October 13 19:00 UT. Polarities of the
magnetic fields are displayed as white for positive and black for negative.
The detected areas, each having $128\times 128$ pixels ($\sim 273\times 273$
arcsec), are overlaid on AI-generated data corresponding to the real one. The
green grid lines represent 64 pixel ($\sim 136.5$ arcsec) intervals within 60
degrees from the center of solar disk.

Figure 2 shows an example of the selection result on 2013 October 13. There
are three solar active regions near the center of the solar disk. The green
grid lines in Figure 2(a) mean the boundaries of the areas where the TUMF is
calculated. Our method successfully detects the approximate positions of three
AR areas (red boxes in Figure 2(a)), and two QR areas (blue boxes in Figure
2(a)). Each box is placed in the same position on the AI-generated data as
shown in Figure 2(b).

###  3.3 STEREO data sets

We use solar farside STEREO/EUVI EUV observations and pairs of SDO/AIA EUV
images and SDO/HMI magnetograms to generate the farside magnetograms. The EUV
passbands of STEREOs are 304, 195, and 171 Å, which have similar temperatures
response to the passbands of the SDO. We use STEREO data sets with 6 hour
cadence (at 00, 06, 12 and 18 UT each day) from 2011 January 1 to 2021 June
30. Since communications with STEREO B were lost on 2014 October 1, the data
from STEREO B are available until that day. We align, downsample, mask, scale
the STEREO EUV images like the SDO EUV ones. We manually exclude a set of
STEREO data with incorrect header information and noise or missing values
because of solar flares. The SDO pairs are data obtained at the frontside ones
which are selected by considering the separation angle between STEREO A (or B)
and SDO.

Table 1: The average pixel-to-pixel CCs between SDO/HMI magnetograms and AI-
generated ones for full disk, ARs, and QRs.

| pixel-to-pixel CC
---|---
|  $8\times 8$ binning
| Full Disk | AR | QR
| (1342) | (2926) | (2684)
AISFM 3.0 (Ours) | 0.88 | 0.91 | 0.70
AISFM 2.0 (Jeong20) | 0.81 | 0.79 | 0.62
AISFM 1.0 (KPL19) | 0.77 | 0.66 | 0.21

Note. — The results of Jeong20 and KPL19 are shown for comparison.

##  4 Results and Discussions

###  4.1 Evaluation of Our Deep Learning Model

We evaluate our deep learning model using the frontside evaluation data sets
that we did not use when training the model. To compare our results with KPL19
and Jeong20, we use Pearson’s CC as a measure for the evaluation. Table 1
shows the average pixel-to-pixel CCs between SDO/HMI magnetograms and AI-
generated ones with a full dynamic range. Our model shows that the average
pixel-to-pixel CCs after $8\times 8$ binning are 0.88, 0.91, and 0.70 for 1342
full disk, 2926 ARs, and 2684 QRs, respectively. These imply that our model
improves the generation of magnetograms when compared with results of KPL19
and Jeong20. In addition, the pixel-to-pixel CCs between the target and our
AI-generated data after $4\times 4$ binning show better results than the CCs
between the target and KPL19’s ones after $8\times 8$ binning. The latitudinal
or longitudinal heliographic resolution at the center of the solar disk is
approximately 1 degree per pixel after $8\times 8$ binning.

Figure 3:  Two objective measures of comparisons between SDO/HMI magnetograms
and AI-generated ones for 1342 full disk, 2926 ARs, and 2684 QRs. (a), (b),
(c) are scatter plots between TUMFs of the SDO/HMI magnetograms and AI-
generated ones for the full disk, the ARs, and the QRs, respectively. (d),
(e), (f) are scatter plots between NMFs of the SDO/HMI magnetograms and AI-
generated ones for the same data sets, respectively.

Figure 3 shows scatter plots between two objective measures of SDO/HMI
magnetograms and AI-generated ones for the same data sets when calculating the
average pixel-to-pixel CCs. We compare TUMF between the target and AI-
generated data (Figure 3(a) to (c)). The TUMF CCs are 0.99, 0.94, and 0.94,
and slopes are 1.02, 0.82, and 1.05 for full disk, ARs, and QRs, respectively.
We compare net magnetic flux (NMF) between the real and AI-generated data
(Figure 3(d) to (f)). The NMF CCs are 0.90, 0.94, and 0.96, and slopes are
0.90, 0.94, and 0.96 for full disk, ARs, and QRs, respectively. Most values of
TUMF and NMF fall well on the diagonal line (black dotted line in Figure 3)
through the origin. These values support that our model can generate
consistent magnetic fluxes.

Figure 3(b) shows that our model slightly underestimates TUMF of the strong
ARs more than the real ones. We examine why small portion of ARs show
underestimated TUMF. We find that for some ARs the AI-generated data do not
produce strong magnetic fields. In this case, the ARs do not have high
intensities at 304, 193, and 171 Å images but have high intensities at 94 Å
(Fe XVIII) and 131 Å (Fe VIII, XXI) of SDO/AIA. These shorter wavelength
channels are characterized by high-temperature emissions (O’dwyer et al.,
2010; Warren et al., 2011). We think that the 94 and 131 Å observations are
helpful to generate strong magnetic fluxes of the ARs. However, in this study,
we do not use the 94 and 131 Å images to train our model because the
STEREOs/EUVI have only filter bands of 171, 195, 284, and 304 Å.

###  4.2 Generation of Solar Farside Magnetograms

We generate the farside magnetograms from the STEREO EUV images and the
frontside reference data pairs by the model. The dates of AISFMs for STEREO A
(AISFMs A) from 2011 January 1 to 2021 June 30, and the AISFMs for STEREO B
(AISFMs B) from 2011 January 10 to 2014 September 27. In 2011 January, the
position of the STEREO A was about $85^{\circ}$ longitude, and that of the
STEREO B was about $-90^{\circ}$ longitude in Stonyhurst heliographic
coordinates. They drift away at a rate of about $22^{\circ}$ per year from the
Earth.

Figure 4:  (a)-(f) Solar frontside SDO/HMI magnetogram, the farside STEREO
composite EUV images (red: 304 Å, green: 193 Å, and blue: 171 Å), and AISFMs
on 2011 March 5. The purple and brown boxes represent NOAA AR 11166 and 11165,
respectively. (g)-(j) The two ARs are zoomed and converted from full disk data
to heliographic coordinated maps. The color map of zoomed ARs shows the large
dynamic rage values in Gauss.

Figure 4 shows multi-viewpoint data from SDO, and STEREO A and B on 2011 March
5. The position of STEREO A is about $88^{\circ}$ heliographic longitude near
the west limb and the position of STEREO B is about $265^{\circ}$ heliographic
longitude near the east limb of the solar frontside (Figure 4(b)). We select a
NOAA AR 11165 to the west, and a NOAA AR 11166 to the east of the solar disk
from the frontside SDO/HMI magnetogram (Figure 4(e)). The AR 11165 is observed
by the STEREO A (Figure 4(c)) and the AISFM A (Figure 4(f)). TUMF of the AR
11165 from the SDO/HMI magnetogram in Figure 4(i) is about $1.85\times
10^{22}$ Mx and one from AISFM A in Figure 4(j) is about $1.74\times 10^{22}$
Mx. The AR 11166 is observed by the STEREO B (Figure 4(a)) and the AISFM B
(Figure 4(d)). TUMF of the AR11166 from the SDO/HMI magnetogram in Figure 4(h)
is $2.50\times 10^{22}$ Mx, and one from AISFM B in Figure 4(g) is $2.71\times
10^{22}$ Mx. The TUMF of the ARs from our AISFMs are consistent with those of
the real one, and distributions of magnetic fields look like the real one.

Figure 5:  (a) A temporal variation of TUMF in the NOAA AR 11166 from 2011
February 1 to May 7. Green, red, blue dots are TUMFs from SDO/HMI, AISFM A,
and AISFM B, respectively. (b)-(k) A series of magnetograms tracking the AR
over three solar rotations. The magnetograms are converted from full disk data
to heliographic coordinated maps. The color map of magnetic fields is as shown
in Figure 4(g)-(j).

Figure 5 shows the temporal evolution of AR 11166, which is shown in Figure
4(g)-(h). We track the AR for three solar rotations at a Carrington rotation
rate. We calculate the TUMF for each area including the AR when the HMI or
AISFMs are available, and the results are shown in Figure 5(a). We consider
the ARs within 60 degrees from the disk center of the HMI or AISFMs. One
impressive thing is that the TUMFs from the SDO/HMI and those from our AISFMs
between solar frontside and farside are smoothly overlapped, demonstrating
that it is possible to monitor the change in magnetic flux quantitatively
using our method. Figure 5(b)-(k) shows magnetograms of the tracked AR.
Combining SDO/HMI magnetograms and AISFMs makes it possible to continuously
monitor the evolution of magnetic field distribution over the solar surface.

Figure 6:  A Solar farside STEREO A EUV image and AISFMs on 2021 May 20, and
the frontside SDO/HMI magnetogram on 2021 May 24. Yellow dotted circles
represent NOAA AR 12824. A result of Jeong20 (AISFM A 2.0) is shown together
for comparison with our result (AISFM A 3.0).

Figure 6 shows comparisons between a SDO/HMI magnetogram and two AISFMs of
solar cycle 25. When the solar cycle changes, all of solar magnetic field
patterns are reversed (Hale & Nicholson, 1925). On 2021 May 20, the position
of STEREO A is about $309^{\circ}$ heliographic longitude (Figure 6(b)). The
magnetic field polarities of our AISFM A (Figure 6(d)) are consistent with the
ones of a SDO/HMI magnetogram (Figure 6(e)), which is obtained at the
frontside after 4 days. As shown in Figure 6(c), AISFM 2.0 cannot produce the
reasonable polarities which can be identified from HMI magnetograms. It is
noted that the AISFM 2.0 is generated from the STEREO A EUV observations
(Figure 6(a)) without reference information from the solar frontside. The
polarity distributions of AISFM 2.0 are similar to that of solar cycle 24. We
mark NOAA AR 12824 with yellow dotted circles in Figure 6(c)-(e). The AR from
the HMI magnetogram shows leading positive and following negative polarities.
Our AISFM 3.0 well represents the polarity distributions of the AR.

Figure 7:  Comparisons of solar polar field strengths from SDO/HMI and those
from AISFM A and B. Yellow (purple) dots represent the mean radial field
strength above $60^{\circ}$ in the south (north) of SDO/HMI magnetograms; the
lines represent the smoothed average. Brown (green) and red (blue) dots
represent the mean radial field strength above $60^{\circ}$ in the south
(north) of AISFMs A and B, respectively.

Figure 7 shows comparisons between mean polar field strengths from SDO/HMI
magnetograms and those from AISFMs A and B. The results from AISFM A and B are
presented at 5 day intervals. The polar fields computed from our AISFMs 3.0
follow trend of the polar field reversal process shown by the computed results
from the SDO/HMI. We use mean radial fields of the HMI polar field data
series, $\mathrm{hmi.meanpf\\_720s}$, which are provided from the JSOC. The
LOS magnetic fields are converted to radial field, under the assumption that
the actual field vector is radial. The mean polar field strength is calculated
from the values within $\pm 45^{\circ}$ longitude and above $60^{\circ}$
latitude (for details refer to Sun et al. (2015)). We calculate mean polar
fields of the AISFMs according to their study, and our results are consistent
with their results.

##  5 DATA RELEASE

Here we first release the AISFMs at the KDC for SDO (http://sdo.kasi.re.kr).
The names of AISFMs 3.0 A and B recorded in the AI-generated data browse are
aisfm_v3_stereo_a and aisfm_v3_stereo_b, respectively. There are 7913 AISFMs A
from 2011 January 1 to 2021 June 30, and 2890 AISFMs B from 2011 January 10 to
2014 September 27 with 6 hour cadence. When the model inputs from the STEREO
and the SDO have poor quality data, we do not produce the AISFMs. The number
of AISFMs A for 2015 is smaller than the number of those for other years,
because the contact with STEREO A was interrupted as it passed behind the Sun.

The AISFMs 3.0 are saved in the Flexible Image Transport System (FITS) format
(Pence et al., 2010). The data have $1024\times 1024$ pixels and the solar
radius is fixed at 450 pixels. The data outside the solar radius are filled
with Not-a-Number (NaN) values like those of SDO/HMI magnetograms. The data
inside the solar radius represent magnetic fields along the LOS to the STEREO
A or B. The coordinate information of the STEREO is stored into the FITS
header keywords (see more details in Appendix A). We also provide example
codes to understand the AISFMs at https://github.com/JeongHyunJin/AISFM3.0.

##  6 Summary and Conclusion

In this study we have generated the improved solar farside magnetograms by the
STEREO and SDO data sets using a deep learning model. For this work, we have
improved our model including the CC-based objectives and used model inputs the
farside STEREO EUV observations together with the frontside SDO data pairs. We
selected 6437 pairs of input and target data sets from 2011 January 1 to 2021
June 30 for the model training. Targets for the training are SDO/HMI
magnetograms. Inputs for the training consist of the EUV images, and the pairs
of EUV images and magnetograms obtained 27.3 days before. We have evaluated
the model using test data sets not used for training.

The main results of this study are as follows. First, we improve the AI-
generated magnetograms than before. The average pixel-to-pixel CCs between the
SDO/HMI magnetograms and our AI-generated ones after $8\times 8$ binning are
0.88, 0.91, and 0.70 for full disk, ARs, and QRs, respectively, which are
noticeably better than the previous results. We have good agreements between
TUMFs calculated from the SDO/HMI magnetograms and those calculated from the
AI-generated ones, and NMFs calculated from the HMI data and those calculated
from the AI-generated ones. Second, we generate more realistic solar farside
magnetograms using the STEREO EUV images and the frontside data pairs by the
model. We compare magnetic fields of ARs from the AISFMs and HMI when the
STEREOs are about 90 degrees apart from Earth. Together with the AISFMs and
HMI, we can continuously monitor the temporal evolution of TUMF of an AR over
three solar rotations. Third, our model can generate AISFMs of solar cycle 24
and 25, in which data have consistent magnetic field polarities with those of
nearby frontside ones. We show that the temporal variation of mean polar
fields calculated from the AISFMs well represents the Sun’s magnetic field
reversal process.

Our AISFMs show potential for scientific studies. We can track solar ARs and
study their evolution at the solar surface using the AISFMs together with the
frontside magnetograms as shown in Figure 5. We can apply the AISFMs to a part
of the boundary conditions for the solar coronal magnetic field extrapolation
as shown in Jeong20. The extrapolations from the synchronic maps with AISFMs
2.0 were more consistent with EUV observations than those from conventional
data in view of the ARs and coronal holes. Based on their results, we expect
that our AISFMs 3.0 can improve coronal field extrapolations that can be input
data for heliospheric solar wind propagation models such as WSA-ENLIL (Arge &
Pizzo, 2000) and EUHFORIA (Pomoell & Poedts, 2018). Moreover we hope that the
AISFMs can be compared with the state-of-the-art PHI magnetograms of the Solar
Orbiter, and can be used together in studying the global magnetic field
activities.

This study uses a large amount of the STEREO and SDO data. We appreciate
numerous team members who have contributed to the success of the STEREO and
SDO mission. We acknowledge the community efforts devoted to the development
of the following open-source packages that were used for this work. This work
was supported by the Korea Astronomy and Space Science Institute (KASI) under
the R&D program (project No. 2022-1-850-05, 2022-1-850-08) supervised by the
Ministry of Science and ICT, and the Basic Science Research Program through
the NRF funded by the Ministry of Education (NRF-2020R1C1C1003892,
NRF-2021R1I1A1A01049615).

##  Appendix A FITS header Keywords

AISFMs are stored in FITS files, each with a keyword header containing the
information on the data. The keywords follow World Coordinate System (WCS)
conventions for describing the physical coordinate values of the data pixels
(Greisen & Calabretta, 2002), and several ephemeris keywords are provided in
Table 2. Since our data are generated by a deep learning model, not
observational one, we store the keyword name not OBSERVTRY (observatory) but
INPUTDAT (input data). The AISFMs are generated based on the features from
three EUV images of the STEREO A (or B). So we record the mean date and time
of the three EUV observations in the DATE-OBS keyword. More detailed
information on the model inputs is stored in the HISTORY keyword.

Table 2: Ephemeris keywords for the AISFMs.

Keyword | Description | Format or Unit
---|---|---
INPUTDAT | Observer of the model input data | STEREO_A (or _B)
DATE-OBS | Mean date and time of STEREO observations | Universal Time
CTYPE1, CTYPE2 | Helioprojective (Cartesian) system | arcsec
HGLN_OBS, HGLT_OBS | Stonyhurst heliographic longitude & latitude of the STEREO | degree
CRLN_OBS, CRLT_OBS | Carrington heliographic longitude & latitude of the STEREO | degree
RSUN_OBS, RSUN | Observed radius of the Sun in arcsec | arcsec
RSUN_REF | Reference radius of the Sun | meter
R_SUN | Pixel size of the solar radius | pixel
DSUN_OBS | Distance between the center of the Sun and the STEREO | meter
DSUN_REF | Average distance from the Sun to Earth (1 AU) | meter

## References

  * Arge & Pizzo (2000) Arge, C., & Pizzo, V. 2000, Journal of Geophysical Research: Space Physics, 105, 10465
  * Atmaja & Akagi (2021) Atmaja, B. T., & Akagi, M. 2021, Journal of Physics: Conference Series, 1896, 012004
  * Babcock (1953) Babcock, H. W. 1953, The Astrophysical Journal, 118, 387
  * Buduma & Locascio (2017) Buduma, N., & Locascio, N. 2017, Fundamentals of deep learning: Designing next-generation machine intelligence algorithms (“ O’Reilly Media, Inc.”)
  * Caplan et al. (2016) Caplan, R., Downs, C., & Linker, J. 2016, The Astrophysical Journal, 823, 53
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., & Courville, A. 2016, Deep learning (MIT press)
  * Greisen & Calabretta (2002) Greisen, E. W., & Calabretta, M. R. 2002, Astronomy & Astrophysics, 395, 1061
  * Hale & Nicholson (1925) Hale, G. E., & Nicholson, S. B. 1925, The Astrophysical Journal, 62, 270
  * Harris et al. (2020) Harris, C. R., Millman, K. J., Van Der Walt, S. J., et al. 2020, Nature, 585, 357
  * Howard et al. (2008) Howard, R. A., Moses, J., Vourlidas, A., et al. 2008, Space Science Reviews, 136, 67
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90
  * Isola et al. (2017) Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. 2017, in Proceedings of the IEEE conference on computer vision and pattern recognition, 1125–1134
  * Jeong et al. (2020) Jeong, H.-J., Moon, Y.-J., Park, E., & Lee, H. 2020, The Astrophysical Journal Letters, 903, L25
  * Judge et al. (2021) Judge, P., Rempel, M., Ezzeddine, R., et al. 2021, The Astrophysical Journal, 917, 27
  * Kaiser et al. (2008) Kaiser, M. L., Kucera, T., Davila, J., et al. 2008, Space Science Reviews, 136, 5
  * Kim et al. (2019) Kim, T., Park, E., Lee, H., et al. 2019, Nature Astronomy, 3, 397
  * Kingma & Ba (2015) Kingma, D. P., & Ba, J. 2015, CoRR, abs/1412.6980
  * Lawrence & Lin (1989) Lawrence, I., & Lin, K. 1989, Biometrics, 255
  * Lemen et al. (2011) Lemen, J. R., Akin, D. J., Boerner, P. F., et al. 2011, in The solar dynamics observatory (Springer), 17–40
  * Liewer et al. (2017) Liewer, P., Qiu, J., & Lindsey, C. 2017, Solar Physics, 292, 146
  * Lim et al. (2021) Lim, D., Moon, Y.-J., Park, E., & Lee, J.-Y. 2021, The Astrophysical Journal Letters, 915, L31
  * Liu et al. (2021) Liu, J., Wang, Y., Huang, X., et al. 2021, Nature Astronomy, 5, 108
  * Mao et al. (2017) Mao, X., Li, Q., Xie, H., et al. 2017, in Proceedings of the IEEE international conference on computer vision, 2794–2802
  * Marnerides et al. (2021) Marnerides, D., Bashford-Rogers, T., & Debattista, K. 2021, Sensors, 21, 4032
  * Müller et al. (2020) Müller, D., Cyr, O. S., Zouganelis, I., et al. 2020, Astronomy & Astrophysics, 642, A1
  * O’dwyer et al. (2010) O’dwyer, B., Del Zanna, G., Mason, H., Weber, M., & Tripathi, D. 2010, Astronomy & Astrophysics, 521, A21
  * Park et al. (2021) Park, E., Jeong, H.-J., Lee, H., Kim, T., & Moon, Y.-J. 2021, Nature Astronomy, 5, 111
  * Park et al. (2019) Park, E., Moon, Y.-J., Lee, J.-Y., et al. 2019, The Astrophysical Journal Letters, 884, L23
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., et al. 2019, Advances in neural information processing systems, 32
  * Pence et al. (2010) Pence, W. D., Chiappetti, L., Page, C. G., Shaw, R. A., & Stobie, E. 2010, Astronomy & Astrophysics, 524, A42
  * Pietarila et al. (2013) Pietarila, A., Bertello, L., Harvey, J., & Pevtsov, A. 2013, Solar Physics, 282, 91
  * Pomoell & Poedts (2018) Pomoell, J., & Poedts, S. 2018, Journal of Space Weather and Space Climate, 8, A35
  * Price-Whelan et al. (2018) Price-Whelan, A. M., Sipőcz, B., Günther, H., et al. 2018, The Astronomical Journal, 156, 123
  * Rana et al. (2019) Rana, A., Singh, P., Valenzise, G., et al. 2019, IEEE Transactions on Image Processing, 29, 1285
  * Robitaille et al. (2013) Robitaille, T. P., Tollerud, E. J., Greenfield, P., et al. 2013, Astronomy & Astrophysics, 558, A33
  * Scherrer et al. (2012) Scherrer, P. H., Schou, J., Bush, R., et al. 2012, Solar Physics, 275, 207
  * Shin et al. (2020) Shin, G., Moon, Y.-J., Park, E., et al. 2020, The Astrophysical Journal Letters, 895, L16
  * Solanki et al. (2020) Solanki, S. K., del Toro Iniesta, J., Woch, J., et al. 2020, Astronomy & Astrophysics, 642, A11
  * Son et al. (2021) Son, J., Cha, J., Moon, Y.-J., et al. 2021, The Astrophysical Journal, 920, 101
  * Sterling et al. (2012) Sterling, A. C., Moore, R. L., & Hara, H. 2012, The Astrophysical Journal, 761, 69
  * Sun et al. (2015) Sun, X., Hoeksema, J. T., Liu, Y., & Zhao, J. 2015, The Astrophysical Journal, 798, 114
  * The SunPy Community et al. (2020) The SunPy Community, Barnes, W. T., Bobra, M. G., et al. 2020, The Astrophysical Journal, 890, 68
  * Ugarte-Urra et al. (2015) Ugarte-Urra, I., Upton, L., Warren, H. P., & Hathaway, D. H. 2015, The Astrophysical Journal, 815, 90
  * Vallejos et al. (2020) Vallejos, R., Pérez, J., Ellison, A. M., & Richardson, A. D. 2020, Spatial Statistics, 40, 100405
  * van Driel-Gesztelyi & Green (2015) van Driel-Gesztelyi, L., & Green, L. M. 2015, Living Reviews in Solar Physics, 12, 1
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261
  * Waldmeier (1955) Waldmeier, M. 1955, epds
  * Wang et al. (2018) Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., et al. 2018, in Proceedings of the IEEE conference on computer vision and pattern recognition, 8798–8807
  * Warren et al. (2011) Warren, H. P., Brooks, D. H., & Winebarger, A. R. 2011, The Astrophysical Journal, 734, 90
  * Wiegelmann et al. (2014) Wiegelmann, T., Thalmann, J. K., & Solanki, S. K. 2014, The Astronomy and Astrophysics Review, 22, 78
  * Zhou et al. (2021) Zhou, C., Xia, C., & Shen, Y. 2021, Astronomy & Astrophysics, 647, A112


# Bias-Aware Minimisation: Understanding and
Mitigating Estimator Bias in Private SGD

Moritz Knolle1,2, Robert Dorfman3, Alexander Ziller1, Daniel Rueckert 1,2,4
and Georgios Kaissis1,2,4
1 Institute for AI in Medicine, Technical University of Munich
2 Konrad Zuse School for Excelllence in Reliable AI
3 V7 Labs
4 Imperial College London

###### Abstract

Differentially private SGD (DP-SGD) holds the promise of enabling the safe and
responsible application of machine learning to sensitive datasets. However,
DP-SGD only provides a biased, noisy estimate of a mini-batch gradient. This
renders optimisation steps less effective and limits model utility as a
result. With this work, we show a connection between per-sample gradient norms
and the estimation bias of the private gradient oracle used in DP-SGD. Here,
we propose Bias-Aware Minimisation (BAM) that allows for the provable
reduction of private gradient estimator bias. We show how to efficiently
compute quantities needed for BAM to scale to large neural networks and
highlight similarities to closely related methods such as Sharpness-Aware
Minimisation. Finally, we provide empirical evidence that BAM not only reduces
bias but also substantially improves privacy-utility trade-offs on the
CIFAR-10, CIFAR-100, and ImageNet-32 datasets.

##  1 Introduction

The application of machine learning models to datasets containing sensitive
information necessitates meaningful privacy guarantees, as it has been shown
that the public release of unsecured models trained on sensitive data can pose
serious threats to data-contributing individuals [1, 2, 3]. To this end,
differential privacy (DP) [4], the widely accepted gold standard approach to
privacy-preserving data analysis, has been extended to many machine learning
methods. Most significantly, for modern machine learning, differentially
private stochastic gradient descent (DP-SGD) [5, 6] has enabled the
application of powerful deep neural networks to sensitive datasets with
meaningful privacy guarantees.

In practice, however, DP-SGD comes with a substantial utility penalty. As we
demonstrate below, this is because DP-SGD only provides a noisy, biased
gradient estimate, causing optimization to settle in regions of the loss
landscape that yield poorly performing models. We show that (ignoring sampling
noise), the variance of a private gradient estimate is fixed while its bias is
not. In this work, we attempt to improve private gradient estimators by
minimising their bias. The bias of an estimate $\hat{p}$ of the true value $p$
underlying a process generating observations $x$ is defined as:

$\mathrm{Bias}(\hat{p},p)=\mathbb{E}_{x}[\hat{p}]-p.$ (1)

$\boldsymbol{g_{1}}$$\text{clip}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\boldsymbol{g_{1}}})$$\text{clip}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\boldsymbol{g_{2}}})$$\boldsymbol{\hat{g}}$$\boldsymbol{\hat{g}_{\text{clip}}}$
(a)

$\boldsymbol{\text{Bias}(\hat{g}_{\text{priv}},\hat{g},)}$

(b)

Figure 1: (a) Clipping per-sample gradients $g_{1}$ and $g_{2}$ yields a
biased gradient estimate $\boldsymbol{\hat{g}_{\mathrm{clip}}}$ of the true
mini-batch gradient $\boldsymbol{\hat{g}}$ in expectation. Below we show this
bias, represented by the vector
$\boldsymbol{\text{Bias}(\hat{g}_{\text{priv}},\hat{g},)}$ in (b), to depend
on $\lVert g_{1}\rVert_{2}$ and $\lVert g_{2}\rVert_{2}$.

Thus, when an estimator has zero bias, it is called unbiased and its expected
value is equal to the quantity being estimated. When estimating gradients,
unbiasedness is a useful property as it implies that gradient estimates –in
expectation– point towards the correct descent direction.

Bias in DP-SGD results from the per-sample gradient clipping operation, used
to enforce a sensitivity bound on the gradients (illustrated above in Fig. 1).
However, despite its importance, the nature of this bias and how it affects
parameter updates has remained poorly understood. With this work, we show that
private gradient estimator bias is intimately related to the per-sample
gradient norms and can further be decomposed into a magnitude and directional
component (See Appendix C). We develop Bias-Aware Minimisation (BAM) and
provide empirical evidence that our method effectively (I) reduces bias
magnitude and (II) increases classification performance on a range of
challenging vision benchmarks.

##  2 Related Work

The bias-variance tradeoff of the private gradient oracle was first discussed
in [7]. More generally, bias associated with gradient clipping in SGD has been
studied by Zhang et al. [8], Qian et al. [9] and also in the private setting,
where Song et al. [10] demonstrated the importance of the correct choice of
clipping threshold in convex models, while Chen et al. [11] showed that no
bias is incurred when a symmetricity condition on the gradient distribution
holds.

Concurrent work [12, 13] has provided empirical evidence that Sharpness-Aware-
Minisation (SAM) [14] can improve privacy-utility tradeoffs. Park et al. [12]
suggest that the magnitude of the bias vector (referred to as the effect of
clipping in their work) is bounded by the sharpness. With this work, we
provide evidence that sharpness, while closely related, might not be the root
cause determining private gradient bias but rather the per-sample gradient
norms in a mini-batch.

##  3 Preliminaries

Differential privacy [15] (DP), is a stability notion on randomised mechanisms
over sensitive databases. Let $D$ and $D^{\prime}$ be two databases (datasets)
that differ exactly in one individual’s data. We denote this relationship
(i.e. database adjacency) through the symbol $\simeq$ and use the standard
remove/add one relation throughout.

######  Definition 1 (Differential privacy).

A randomised mechanism $M$ executed on the result of a query function $q$
(i.e. $M(q(\cdot))$) preserves $(\varepsilon$, $\delta)$-DP if, for all pairs
of adjacent databases $D$ and $D^{\prime}$, and all subsets $\mathcal{S}$ of
$M(q(\cdot))$’s range:

$p\left[M\left(q(D)\right)\in\mathcal{S}\right]\leq{e^{\varepsilon}}\,p\left[M(q(D^{\prime}))\in\mathcal{S}\right]+\delta,$
(2)

where the relationship between $D$ and $D^{\prime}$ is symmetric. The
guarantee is given over the randomness of $M$.

To achieve $(\varepsilon,\delta)$-DP using Gaussian noise perturbation (i.e.
the Gaussian mechanism $\mathcal{M}$), one needs to calibrate the magnitude of
the noise to the query function’s (global) sensitivity:

$\Delta_{2}(q)=\sup_{D\simeq D^{\prime}}\left\lVert
q(D)-q(D^{\prime})\right\rVert_{2},$

or $\Delta$ for short. Since obtaining a (non-vacuous) sensitivity bound is
generally not possible for deep neural network gradients, it is common
practice to enforce a bound manually by projecting per-sample gradients to the
$L_{2}$-ball [6]. Together, this operation, more commonly known as clipping,
followed by appropriately scaled Gaussian perturbation, then yields the DP-SGD
algorithm, which can be thought of as simply querying a private gradient
oracle at each optimisation step as seen in Algorithm 1:

Algorithm 1 Private gradient oracle $\psi$

Input: (Poisson-sampled) mini-batch $B=\\{(x_{1},y_{1}),...,(x_{l},y_{l})\\}$,
Clipping Bound $C$, noise multiplier $\sigma$, loss function $\mathcal{L}$,
parameters $\theta\in\mathbb{R}^{d}$
Output; private gradient estimate

$(x_{i},y_{i})$in $B$

1:$g_{i}\leftarrow\nabla_{\theta}\mathcal{L}\left(f_{\theta}(x_{i}),y_{i}\right)$

2:$\bar{g}_{i}\leftarrow g_{i}/\max(1,\frac{\left\lVert
g_{i}\right\rVert_{2}}{C})$ $\triangleright$ clip

3:$\hat{g}_{\mathrm{priv}}\leftarrow\frac{1}{l}\left(\sum_{i=1}^{l}\bar{g}_{i}+\mathcal{N}(0,\sigma^{2}C^{2}I_{d})\right)$
$\triangleright$ perturb

4:return $\hat{g}_{\mathrm{priv}}$

Where $l$ is the expected batch size due to the sub-sampling amplification
requirement that $B$ is constructed through a Poisson sample of the Dataset
$D$.

###  3.1 Setting

We focus on supervised learning where, given a dataset
$D=\\{(x_{i},y_{i}),...,(x_{n},y_{n})\\}$ drawn i.i.d from a product
distribution $\mathcal{X}\times\mathcal{Y}$, we wish to find a mapping
$f:\mathcal{X}\rightarrow\mathcal{Y}$, realised through a neural network. This
neural network has parameters $\theta\in\Theta\in\mathbb{R}^{d}$ and is
trained by minimising the empirical loss function
$\mathcal{L}(\theta):=\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}(\theta,x_{i},y_{i})$
using DP-SGD, whereby at each optimisation step the private gradient oracle is
queried to obtain a privatised gradient estimate.

$\displaystyle\theta^{(t+1)}=\theta^{(t)}-\gamma^{(t)}\hat{g}^{(t)}_{\mathrm{priv}}$
(3)
$\displaystyle\hat{g}^{(t)}_{\mathrm{priv}}=\psi(B,C,\sigma,\mathcal{L},\theta^{t})$
(4)

We drop the superscript $t$ for notational simplicity, assuming we are on step
$t$ in the following analyses. We further denote the clipped minibatch
gradient as $\hat{g}_{\mathrm{clip}}=1/l\sum_{i}^{l}\bar{g}_{i}$.

##  4 Clipping dominates private gradient bias

Since we are interested in obtaining an unbiased version of DP-SGD, we first
study the bias introduced by constructing a private estimate for the mini-
batch gradient $\hat{g}$. Our analysis focuses on the quantity:

$\text{Bias}\left(\hat{g}_{\mathrm{priv}},\,\hat{g}\right)=\mathbb{E}[\hat{g}_{\mathrm{priv}}]-\hat{g},$

where we view $\hat{g}$ as fixed, that is, the gradient constructed for an
already observed batch of data to be used in a step of conventional SGD. This
enables us to isolate the bias introduced through private estimation of
$\hat{g}$. Note that when using $\mathbb{E}$ without a subscript, we take the
expectation over all randomness present. In the case of
$\hat{g}_{\mathrm{priv}}$, this randomness is –by assumption– only due to the
Gaussian mechanism. We first observe that this gradient perturbation does not
introduce additional bias:

######  Lemma 1.

The bias of the private gradient estimate,
Bias($\hat{g}_{\mathrm{priv}},\hat{g})$, is unaffected by the noise addition
in the Gaussian mechanism. That is,

$\mathrm{Bias}(\hat{g}_{\mathrm{priv}},\hat{g})=\mathrm{Bias}(\hat{g}_{\mathrm{clip}},\hat{g})$

###### Proof.

$\displaystyle\mathrm{Bias}(\hat{g}_{\mathrm{priv}},$
$\displaystyle\hat{g})=\mathbb{E}\Big{[}\frac{1}{l}\sum_{i=1}^{l}\Big{(}\mathrm{clip}(g_{i})+\mathcal{N}(0,\sigma^{2}C^{2}I_{d})\Big{)}\Big{]}-\hat{g}$
(5) $\displaystyle=$
$\displaystyle\frac{1}{l}\sum_{i=1}^{l}\Big{(}\mathrm{clip}(g_{i})+\mathbb{E}\big{[}\mathcal{N}(0,\sigma^{2}C^{2}I_{d})\big{]}\Big{)}-\hat{g}$
(6) $\displaystyle=$ $\displaystyle\hat{g}_{\mathrm{clip}}-\hat{g}$ (7)
$\displaystyle=$ $\displaystyle\mathrm{Bias}(\hat{g}_{\mathrm{clip}},\hat{g})$
(8)

This holds since $\hat{g}$ and $\hat{g}_{\mathrm{clip}}$ are viewed as
constructed from an observed mini-batch of data and the zero centred Gaussian
random variables are independent of one another. ∎

Thus, private gradient bias is caused by the clipping operation alone. Next,
we develop an objective function that provably minimises the aforementioned
bias.

##  5 A bias-aware objective

We propose $\mathcal{L}_{\mathrm{BAO}}$, an objective that, when minimised,
provably reduces the bias of private gradient estimates
$\hat{g}_{\mathrm{priv}}$ by encouraging small per-sample gradient norms:

$\mathcal{L}_{\mathrm{BAO}}(\theta,x,y)=\underbrace{\mathcal{L}(\theta,x,y)}_{\text{original
loss}}+\underbrace{\lambda\left(\frac{1}{l}\sum_{i=1}^{l}\left\lVert
g_{i}\right\rVert_{2}\right)}_{\text{regularising term}}$ (9)

which can be sub-sampled as follows:

$\mathcal{L}_{\mathrm{BAO}}(\theta,x_{i},y_{i})=\mathcal{L}(\theta,x_{i},y_{i})+\lambda\left\lVert
g_{i}\right\rVert_{2}$ (10)

To motivate this optimisation objective, we will now demonstrate the primary
dependence of Bias($\hat{g}_{\mathrm{priv}},\hat{g})$ on the per-sample
gradient norms for a fixed clipping threshold $C$.

######  Lemma 2.

A smaller per-sample gradient norm $\lVert g_{i}\rVert_{2}$ of the $i$-th
sample in a mini-batch decreases
$\text{Bias}(\hat{g}_{\mathrm{priv}},\hat{g})$.

###### Proof.

First note that

$\mathrm{clip}(g_{i})=\frac{g_{i}}{\mathrm{max}(1,\frac{\lVert
g_{i}\rVert_{2})}{C})}=\begin{cases}g_{i}&\mathrm{if}\,\lVert
g_{i}\rVert_{2}\leq C\\\ \frac{Cg_{i}}{\lVert
g_{i}\rVert_{2}}&\mathrm{if}\,\lVert g_{i}\rVert_{2}>C.\end{cases}$

Thus, if $\lVert g_{i}\rVert_{2}\leq C$ for every $i$ in the mini-batch, then
by Lemma 1:

$\mathrm{Bias}(\hat{g}_{\mathrm{priv}},\hat{g})=\frac{1}{l}\sum_{i=1}^{l}g_{i}-\hat{g}=0.$

That is, if the $L_{2}$-norm of the gradients of the mini-batch are all below
the clipping threshold, the bias of the private gradient estimate reduces to
zero. On the other hand, if $\lVert g_{i}\rVert_{2}>C$:

$\displaystyle\mathrm{Bias}(\hat{g}_{\mathrm{priv}},\hat{g})$
$\displaystyle=\frac{1}{l}\sum_{i=1}^{l}\frac{C}{\lVert
g_{i}\rVert_{2}}g_{i}-\hat{g}.$ (11)

Therefore, the bias is dependent on the ratios $\frac{C}{\lVert
g_{i}\rVert_{2}}\forall i$: as a per-sample gradient norm $\lVert
g_{i}\rVert_{2}$ gets increasingly larger than $C$, the bias grows (note that
here grows means gets further from the origin). Conversely, if we minimise
$\lVert g_{i}\rVert_{2}$ for any $i$, the bias will shrink until all
$\left\lVert g_{i}\right\rVert_{2}$ in the mini-batch are less than $C$, at
which point the estimate becomes unbiased. ∎

Notice that (9) is similar to the gradient norm penalty, used by [16] to
encourage flatness of the loss landscape:

$\mathcal{L}_{Z}(\theta)=\mathcal{L}(\theta)+\left\lVert\nabla_{\theta}\mathcal{L}(\theta)\right\rVert_{2},$
(12)

It can be shown that (9) upper-bounds $\mathcal{L}_{Z}$ (see Appendix B). This
means that a reduction in $\mathcal{L}_{\mathrm{BAO}}$ implies a reduction in
$\mathcal{L}_{Z}$, but the converse is not generally true. Note also that
$\mathcal{L}_{Z}$ does not allow for the required per-sample DP analysis,
precluding the simple application of DP-SGD [12].

Dataset | $\boldsymbol{\varepsilon}$ | $\boldsymbol{\delta}$ | DP-SGD | DP-SAT |  BAM (ours)
---|---|---|---|---|---
CIFAR-10 | $1.0$ | $10^{-5}$ | $60.9\pm 0.49$ | $60.9\pm 0.62$ | $\mathbf{61.4\pm 0.48}$
$2.0$ | $67.1\pm 0.10$ | $67.2\pm 0.30$ | $\mathbf{68.2\pm 0.27}$
$10.0$ | $78.6\pm 0.08$ | $78.1\pm 0.69$ | $\mathbf{79.7\pm 0.13}$
CIFAR-100 | $1.0$ | $10^{-5}$ | $18.1\pm 0.10$ | $18.2\pm 0.13$ | $\mathbf{18.5\pm 0.04}$
$2.0$ | $24.9\pm 0.46$ | $24.9\pm 0.35$ | $\mathbf{25.4\pm 0.40}$
$10.0$ | $40.3\pm 0.21$ | $40.1\pm 0.19$ | $\mathbf{40.8\pm 0.06}$
ImageNet32 | $10.0$ | $8\times 10^{-7}$ | $14.97$ | $14.70$ | $\mathbf{20.67}$

Table 1: Test accuracy (mean$\pm$SD %) for CIFAR-10, CIFAR-100 and ImageNet32
computed over three random seeds at different $(\varepsilon,\delta)$. Due to
computational resource constraints, we report only a single training run for
Imagenet32.

###  5.1 Efficient Computation

Naïvely implementing $\mathcal{L}_{\mathrm{BAO}}$ is problematic, as the
gradient computation now involves computing a Hessian-Vector Product (HVP) for
every sample:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\mathrm{BAO}}(\theta,x,y)$
$\displaystyle=\nabla_{\theta}\mathcal{L}(\theta,x,y)+$
$\displaystyle\lambda\frac{1}{n}\sum_{i=1}^{l}$
$\displaystyle\underbrace{\nabla^{2}_{\theta}\mathcal{L}(\theta,x_{i},y_{i})\frac{\nabla_{\theta}\mathcal{L}(\theta,x_{i},y_{i})}{\left\lVert\nabla_{\theta}\mathcal{L}(\theta,x_{i},y_{i})\right\rVert_{2}}}_{\text{HVP}}.$
(13)

This objective is thus expensive to compute for deep networks using reverse-
mode automatic differentiation (AD) [17]. Fortunately, however, prior work has
shown that HVPs can be computed efficiently, in this case, either exactly
using a combination of forward and reverse mode AD [18, 19], or approximately
using the local gradient ascent step of SAM [16].

##  6 Method

To reduce the bias of the private gradient oracle in DP-SGD, we optimise our
bias-aware objective $\mathcal{L}_{\mathrm{BAO}}$ and approximate the
necessary per-sample gradients. Concretely, we perform the local gradient
ascent step of SAM, at the sample level before computing the gradient:

$\nabla_{\theta}\mathcal{L}_{\mathrm{BAO}}(\theta,x_{i},y_{i})\approx\nabla_{\theta}\mathcal{L}(\theta,x_{i},y_{i})\bigg{|}_{\theta=\theta+\lambda\frac{\nabla_{\theta}\mathcal{L}(\theta,x_{i},y_{i})}{\left\lVert\nabla_{\theta}\mathcal{L}(\theta,x_{i},y_{i})\right\rVert_{2}}}.$
(14)

The entire training procedure is summarised in Algorithm 2.

Algorithm 2 Bias-Aware Minimisation (BAM)

$t\in 1,2,...,T$

1:$B\leftarrow$ Poisson sample of $D$ with probability $q$ $(x_{i},y_{i})\in
B$

2:$\theta^{\prime(t)}\leftarrow\theta^{(t)}+\lambda^{(t)}\frac{\nabla_{\theta}\mathcal{L}(\theta^{(t)},x_{i},y_{i})}{\left\lVert\nabla_{\theta}\mathcal{L}(\theta^{(t)},x_{i},y_{i})\right\rVert_{2}}$
$\triangleright$ SAM step

3:$g^{(t)}_{i}\leftarrow\nabla_{\theta^{\prime}}\mathcal{L}_{\mathrm{BAO}}(\theta^{\prime(t)},x_{i},y_{i})$

4:$\bar{g}_{i}^{(t)}\leftarrow g_{i}^{(t)}/\max\Big{(}1,\,\frac{\left\lVert
g_{i}^{(t)}\right\rVert_{2}}{C}\Big{)}$ $\triangleright$ clip

5:$\hat{g}_{\mathrm{priv}}^{(t)}=\frac{1}{l}\sum_{i}^{l}\left[{\bar{g}^{(t)}_{i}}+\mathcal{N}(0,\,\sigma^{2}C^{2}I_{d})\right]$
$\triangleright$ perturb

6:$\theta^{(t+1)}=\theta^{(t)}-\gamma^{(t)}\,\hat{g}_{\mathrm{priv}}^{(t)}$

Note that, since
$\nabla_{\theta}\mathcal{L}_{\mathrm{BAO}}(\theta,x_{i},y_{i})$ only depends
on sample-level statistics, the sampling process is identical to DP-SGD and
all quantities are privatised as in DP-SGD, every iteration $t$ of Algorithm 2
satisfies $(\varepsilon,\delta)$-DP with identical privacy parameters as DP-
SGD.

##  7 Results

To evaluate our proposed approach and compare its performance to DP-SGD and
DP-SAT [12], we perform a range of experiments on challenging computer vision
datasets. Results for CIFAR-10/100 [20] and ImageNet32[21] are reported above
in Table 1. We employ state-of-the-art (SOTA) training practices for DP-SGD
[22], namely: weight standardisation, group normalisation, large batch sizes
and augmentation multiplicity. Full experimental details and hyperparameter
values can be found in Table 2, Appendix A.

Figure 2: BAM effectively minimises bias: The magnitude of the bias vector
measured on CIFAR-10 at a batch size of $512$ is substantially lower for BAM,
while DP-SAT incurs the same bias as DP-SGD.

We find that BAM effectively minimises private gradient bias in practice (see
Fig. 2 above). On the other hand, DP-SAT largely has no effect on estimator
bias for values of the regularisation parameter $\lambda$ that yield high-
performing models. Further empirical run-time comparisons (data shown in
Appendix A) reveal that while both exact and approximate gradient computations
for BAM do incur higher computational costs than DP-SAT (and DP-SGD), this
burden is manageable for most practically sized networks with less than 200
layers. Finally, our empirical performance comparison reveals that when using
very large batch sizes and the other SOTA practices of De et al. [22],
performance gains on more challenging datasets realised through DP-SAT are
smaller than reported in the original publication [12]. In contrast, BAM
consistently improves performance across different privacy budgets (Table 1).

##  8 Discussion

After deriving the bias of the private gradient oracle from first principles,
we developed a bias-aware regularisation objective
$\mathcal{L}_{\mathrm{BAO}}$ that was empirically confirmed effective at
minimising the bias vector associated with private gradient estimation. We
further demonstrated that using the SAM approximation as suggested by [16],
our objective and its gradient are computable with manageable computational
overhead.

Our method performs the local gradient ascent step (SAM step) at the per-
sample level and is thus closely related to the DP-SAT method of [12], which
performs the ascent step with the previous iteration’s privatised mini-batch
gradient. Our experiments show that our method outperforms DP-SAT at
minimising bias, which is mirrored in the superior accuracy of models trained
with BAM. Notably, our method yields a more than $5\%$ accuracy increase on
the most challenging dataset tested, ImageNet32. Based on our results, we
hypothesise that, with modern private training practices [22] (very large
batch size, large learning rate), the noisy gradient of the previous iteration
($\hat{g}^{(t-1)}_{\mathrm{priv}}$), is a poor approximation for finding the
maximum loss in the local neighbourhood around the current iteration’s
parameter value $\theta^{(t)}$. This is corroborated by a simple experiment
(data shown in Fig. 4, Appendix A) that shows the ascent step vector used in
DP-SAT to point in slightly different directions to the current iteration’s
(un-privatised) mini-batch gradient conventionally used in SAM.

Our results, while not without limitations (BAM incurs slight computational
overhead), have shown that reducing private gradient bias can lead to
effective performance increases on challenging image datasets. More extensive
empirical evaluation, especially on large-scale datasets such as (full-size)
ImageNet, is ongoing work and required to fully assess its benefits. Finally,
future work should investigate the effectiveness of alternative methods that
encourage smoothness [23] and look into connections to and impact on model
fairness with respect to sub-groups [24].

##  9 Acknowledgments

This paper was supported by the DAAD programme Konrad Zuse Schools of
Excellence in Artificial Intelligence, sponsored by the Federal Ministry of
Education and Research.

## References

  * Geiping et al. [2020] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller.  Inverting gradients-how easy is it to break privacy in federated learning?  _Advances in Neural Information Processing Systems_ , 33:16937–16947, 2020.
  * Carlini et al. [2021] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2633–2650, 2021.
  * Carlini et al. [2019] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th USENIX Security Symposium (USENIX Security 19)_ , pages 267–284, 2019.
  * Dwork et al. [2014a] Cynthia Dwork, Aaron Roth, et al.  The algorithmic foundations of differential privacy.  _Foundations and Trends® in Theoretical Computer Science_ , 9(3–4):211–407, 2014a.
  * Shokri and Shmatikov [2015] Reza Shokri and Vitaly Shmatikov.  Privacy-preserving deep learning.  In _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_ , pages 1310–1321, 2015.
  * Abadi et al. [2016] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , pages 308–318, 2016.
  * McMahan et al. [2018] H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang.  Learning differentially private recurrent language models.  In _International Conference on Learning Representations_ , 2018.
  * Zhang et al. [2019] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie.  Why gradient clipping accelerates training: A theoretical justification for adaptivity.  In _International Conference on Learning Representations_ , 2019.
  * Qian et al. [2021] Jiang Qian, Yuren Wu, Bojin Zhuang, Shaojun Wang, and Jing Xiao.  Understanding gradient clipping in incremental gradient methods.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1504–1512. PMLR, 2021.
  * Song et al. [2020] Shuang Song, Om Thakkar, and Abhradeep Thakurta.  Characterizing private clipped gradient descent on convex generalized linear problems.  _arXiv preprint arXiv:2006.06783_ , 2020.
  * Chen et al. [2020] Xiangyi Chen, Steven Z Wu, and Mingyi Hong.  Understanding gradient clipping in private sgd: A geometric perspective.  _Advances in Neural Information Processing Systems_ , 33:13773–13782, 2020.
  * Park et al. [2023] Jinseong Park, Hoki Kim, Yujin Choi, and Jaewook Lee.  Differentially private sharpness-aware training.  In _International Conference on Machine Learning_ , pages XXXX–YYYY. PMLR, 2023.
  * Shi et al. [2023] Yifan Shi, Yingqi Liu, Kang Wei, Li Shen, Xueqian Wang, and Dacheng Tao.  Make landscape flatter in differentially private federated learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 24552–24562, 2023.
  * Foret et al. [2020] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  _arXiv preprint arXiv:2010.01412_ , 2020.
  * Dwork et al. [2014b] Cynthia Dwork, Aaron Roth, et al.  The algorithmic foundations of differential privacy.  2014b.
  * Zhao et al. [2022] Yang Zhao, Hao Zhang, and Xiuyuan Hu.  Penalizing gradient norm for efficiently improving generalization in deep learning.  In _International Conference on Machine Learning_ , pages 26982–26992. PMLR, 2022.
  * Karakida et al. [2022] Ryo Karakida, Tomoumi Takase, Tomohiro Hayase, and Kazuki Osawa.  Understanding gradient regularization in deep learning: Efficient finite-difference computation and implicit bias.  _arXiv preprint arXiv:2210.02720_ , 2022.
  * Wiltschko and Johnson [n.d.] Alex Wiltschko and Matthew Johnson.  Jax- autodiff cookbook, n.d.  Accessed June 20, 2023. https://jax.readthedocs.io/en/latest/notebooks/autodiff˙cookbook.html.
  * Pearlmutter [1994] Barak A Pearlmutter.  Fast exact multiplication by the hessian.  _Neural computation_ , 6(1):147–160, 1994.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Chrabaszcz et al. [2017] Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter.  A downsampled variant of imagenet as an alternative to the cifar datasets.  _arXiv preprint arXiv:1707.08819_ , 2017.
  * De et al. [2022] Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle.  Unlocking high-accuracy differentially private image classification through scale.  _arXiv preprint arXiv:2204.13650_ , 2022.
  * Cha et al. [2021] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  _Advances in Neural Information Processing Systems_ , 34:22405–22418, 2021.
  * Tran et al. [2021] Cuong Tran, My Dinh, and Ferdinando Fioretto.  Differentially private empirical risk minimization under the fairness lens.  _Advances in Neural Information Processing Systems_ , 34:27555–27565, 2021.
  * Dozat [2016] Timothy Dozat.  Incorporating nesterov momentum into adam.  2016\.
  * Bradbury et al. [2018] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Flat minima.  _Neural computation_ , 9(1):1–42, 1997.

Dataset | Method | $\lambda$ | Model | Batch-size | LR | C | Multiplicity
---|---|---|---|---|---|---|---
CIFAR-10 | DP-SGD | $0.0$ | ResNet-9 | $32\times 128$ | 2e-3 | $1$ | 16
DP-SAT | $0.086$
BAM | $0.02$
CIFAR-100 | DP-SGD | $0.0$ | ResNet-9 | $32\times 128$ | 2e-3 | $1$ | 16
DP-SAT | $0.086$
BAM | $0.01$
ImageNet32 | DP-SGD | $0.0$ | WideResNet-16-4 | $8\times 512$ | 1e-3 | $1$ | 4
DP-SAT | $0.07$
BAM | $0.005$
Table 2: Hyperparameter values for experiments on CIFAR-10, CIFAR-100 and
ImageNet32.

##  Appendix A Experimental details & further results

To obtain suitable hyperparameter values for the compared methods (reported
above in Table 2), a random search was first employed using DP-SGD, after
which a separate search for optimal $\lambda$-values was performed for both
DP-SAT and BAM with N=200 random trials each. For Imagenet32, only a very
small-scale hyperparameter search was employed due to the high computational
cost. Augmentation multiplicity, as described in [22], was employed with
probability $p=0.5$ (for each augmentation) across all compared methods with
random pixel shifts (up to $4$ pixels) and random vertical flips. All models
were trained for $75$ epochs with the NAdam[25] optimizer and, besides the
learning rate (LR), otherwise, default hyperparameters. To meet the different
$(\varepsilon,\delta)$ privacy budget requirements the noise multiplier
$\sigma^{2}$ was adjusted accordingly.

###  A.1 Computational aspects

Figure 3: Per-step wall-time (mean $\pm$ SD) for different computational
approaches to bias mitigation in DP-SGD for increasing network depth. Results
were computed over ten trials and five repetitions.

We evaluate the empirical run-time complexity of the two previously mentioned
approaches to compute
$\nabla_{\theta}\mathcal{L}_{\mathrm{BAO}}(\theta,x_{i},y_{i})$. Concretely,
we compare forward-over-reverse mode AD (BAM FWD-REV) and the SAM (BAM SAM)
approximation to compute necessary gradients for BAM and compare to DP-SAT
[12]. DP-SAT uses the previous step’s privatised mini-batch gradient to
approximate (12) at no additional privacy cost and little computational
overhead. Figure 3 showcases the results of an empirical run-time comparison
on a toy dataset, implemented in jax [26] and compiled with XLA to obtain a
fair comparison between methods.

###  A.2 Gradient ascent step effectiveness

To investigate the effectiveness the gradient ascent step in DP-SAT and BAM,
we investigate the cosine similarity between the ascent direction used by the
respective method and the (ground-truth) ascent direction as used in SAM, that
is the un-privatised, current mini-batch gradient. Our findings (below in Fig.
4) indicate that the per-sample gradient ascent step of DP-SAT is
substantially better aligned with the non-private SAM ascent step than the
ascent step with the previous iteration’s privatised gradient of the DP-SAT
method. Concretely, we find DP-SAT ascent steps to point in slightly opposite
directions, as indicated by a small, but negative cosine similarity.

Figure 4: Effectiveness of the ascent step throughout training on CIFAR-10
(three repetitions) with DP-SAT and BAM as measured by the cosine similarity
$\mathrm{cos}(\hat{g}_{\mathrm{priv}}^{(t-1)},\hat{g}^{(t)})$ and
$1/l\sum_{i}^{l}\mathrm{cos}(g_{i}^{(t)},\hat{g}^{(t)})$ respectively.

###  A.3 C. Batch size and its effects on bias and directional error

To investigate the empirical success of recent works using extremely large
batch sizes (or even full-batch training), we investigate the impact of batch
size on private gradient bias and its directional component.

Figure 5: Bigger batches reduce private gradient bias magnitude. The figure
showcases bias magnitude averaged over training iterations when training on
CIFAR-10 with DP-SGD with different batch sizes. Figure 6: Bigger batches
reduce the directional component of private gradient bias as measured by an
increased average cosine similarity between clipped and unclipped mini-batch
gradients during training on CIFAR-10.

We find large batches to reduce both: the magnitude of the bias vector and the
directional components of the bias vector. The latter is indicated by an
increased cosine similarity between clipped and unclipped gradients.

##  Appendix B BAM and effects on loss landscape flatness

Minimising (9) has the additional benefit of smoothing the optimisation
landscape around the minima (flat minima) which, as originally argued by
Hochreiter and Schmidhuber [27], is widely believed to offer better
generalisation performance than sharp minima. To find such flat minima, Zhao
et al. [16] propose the objective:

$\displaystyle\mathcal{L}_{Z}(\theta)$
$\displaystyle=\mathcal{L}(\theta)+\lambda\big{\lVert}\nabla_{\theta}\mathcal{L}(\theta)\big{\rVert}_{2}$
(15)
$\displaystyle=\mathcal{L}(\theta)+\lambda\Big{\lVert}\frac{1}{n}\sum_{i=1}^{n}g_{i}\Big{\rVert}_{2}$
(16)

This objective finds flat minima because
$\lVert\nabla_{\theta}\mathcal{L}(\theta)\rVert_{2}$ approximates the
Lipschitz constant of the loss function $\mathcal{L}$. The Lipschitz constant
is an upper bound on how much the magnitude of a change in the parameter space
changes the magnitude of the loss. This is a way of capturing flatness: the
smaller the Lipschitz constant of the loss function, the flatter the minima
(since the same change in $\theta$ would lead to a smaller bound on the change
in the loss magnitude). Hence, by decreasing the Lipschitz constant of the
loss function, the solution will be driven towards a flatter minimum.

However, objective (15) cannot be used in our case because it does not allow
for the per-sample analysis required for DP. Nonetheless, we can show the
following:

######  Lemma 3.

The bias aware objective $\mathcal{L}_{\mathrm{BAO}}$ (9) upper-bounds the
objective $\mathcal{L}_{Z}$ (15) of [16].

###### Proof.

This follows from the triangle inequality:

$\displaystyle\mathcal{L}(\theta)+\lambda\Big{\lVert}\frac{1}{n}\sum_{i=1}^{n}g_{i}\Big{\rVert}_{2}$
$\displaystyle=\mathcal{L}(\theta)+\lambda\frac{1}{n}\Big{\lVert}\sum_{i=1}^{n}g_{i}\Big{\rVert}_{2}$
(17)
$\displaystyle\leq\mathcal{L}(\theta)+\lambda\frac{1}{n}\sum_{i=1}^{n}\lVert
g_{i}\rVert_{2}.$ (18)

∎

Thus, minimising $\mathcal{L}_{\mathrm{BAO}}$ will additionally drive the
solution to a flat minimum by minimising an upper bound on (15).

##  Appendix C A meaningful decomposition of the private gradient

$\boldsymbol{\text{Bias}(\hat{g}_{\text{priv}},\hat{g},)}$

(a)

$\boldsymbol{(a-1)\hat{g}}$$c$ (b)

Figure 7: Illustration of the proposed bias (a) decomposition into orthogonal
vector components (b): magnitude $\boldsymbol{a}$ and direction
$\boldsymbol{c}$.

With the aim of better capturing the pathological nature of clipping during
private stochastic optimisation, we propose a bias decomposition into
orthogonal components that allow for the isolation of magnitude and
directional estimation error.

In our case, we have an estimator $\hat{g}_{\mathrm{priv}}$ of $\hat{g}$. We
now show a decomposition of $\hat{g}_{\mathrm{priv}}$ in terms of $\hat{g}$,
which will allow us to differentiate harmful from harmless bias.

######  Theorem 1.

The private gradient estimate $\hat{g}_{\mathrm{priv}}$ can be decomposed as:

$\hat{g}_{\mathrm{priv}}=a\cdot\hat{g}+c,$

where $a\in\mathbb{R},c\in\mathbb{R}^{d}$. We call $a$ the magnitude error and
$c$ the directional error that arise through (private) estimation.

###### Proof.

We first relate each per-sample gradient $g_{i}$ to the mini-batch gradient
$\hat{g}$ through a vector decomposition into orthogonal components.

$g_{i}=\mathrm{proj}_{\hat{g}}\,g_{i}+\tau_{i}$

where $\mathrm{proj}_{\hat{g}}\,g_{i}=\frac{\langle
g_{i},\hat{g}\rangle}{\lVert\hat{g}\rVert^{2}}\cdot\hat{g}$ and $\tau_{i}$ is
a suitable vector orthogonal to the projection. Letting
$\eta_{i}=\frac{\langle g_{i},\hat{g}\rangle}{\lVert\hat{g}\rVert^{2}}$, the
decomposition is:

$\mathrm{proj}_{\hat{g}}\,g_{i}=\eta_{i}\cdot\hat{g}+\tau_{i}.$ (19)

Utilising this decomposition inside the $\mathrm{clip}$ function gives us:

$\mathrm{clip}(g_{i})=\frac{g_{i}}{\max(1,\frac{C}{||\hat{g}_{i}||_{2}})}=\frac{\eta_{i}\hat{g}+\tau_{i}}{\max(1,\frac{C}{||g_{i}||_{2}})}.$
(20)

Now, let $M_{i}=\max(1,\frac{C}{||g_{i}||_{2}})$ so that we can simplify the
above as

$\mathrm{clip}(g_{i})=\frac{\eta_{i}}{M_{i}}\hat{g}+\frac{\tau_{i}}{M_{i}}.$
(21)

Averaging the clipped gradients across the mini-batch yields:

$\displaystyle\hat{g}_{\mathrm{clip}}$
$\displaystyle=\frac{1}{l}\sum_{i=1}^{l}\mathrm{clip}(g_{i})$ (22)
$\displaystyle=\frac{1}{l}\sum_{i=1}^{l}\left(\frac{\eta_{i}}{M_{i}}\hat{g}+\frac{\tau_{i}}{M_{i}}\right)$
(23)
$\displaystyle=\left(\frac{1}{l}\sum_{i=1}^{l}\frac{\eta_{i}}{M_{i}}\right)\;\cdot\;\hat{g}+\left(\frac{1}{l}\sum_{i=1}^{l}\frac{\tau_{i}}{M_{i}}\right).$
(24)

Finally, perturbing each $g_{i}$ with a vector
$\beta_{i}\sim\mathcal{N}(0,C\sigma I_{d})$ we obtain the desired
decomposition in terms of magnitude error $a$ and directional error $c$:

$\displaystyle\hat{g}_{\mathrm{priv}}$
$\displaystyle=\frac{1}{l}\sum_{i=1}^{l}\left(\mathrm{clip}(g_{i})+\beta_{i}\right)$
(25)
$\displaystyle=\underbrace{\left(\frac{1}{l}\sum_{i=1}^{l}\frac{\eta_{i}}{M_{i}}\right)}_{a}\;\cdot\;\hat{g}\;+\;\underbrace{\left(\frac{1}{l}\sum_{i=1}^{l}\frac{\tau_{i}}{M_{i}}+\beta_{i}\right)}_{c}$
(26)

∎

This allows us to now effectively isolate directional from magnitude
components of the private gradient bias:

$\displaystyle\mathrm{Bias}(\hat{g}_{\mathrm{priv}},\hat{g})$
$\displaystyle=\mathbb{E}\left[\hat{g}_{\mathrm{priv}}\right]-\hat{g}$ (27)
$\displaystyle=\mathbb{E}\left[a\cdot\hat{g}+c\right]-\hat{g}.$ (28)