# Bye bye, local bias: the statistics of the halo field are poorly determined
by the local mass density

Deaglan J. Bartlett CNRS & Sorbonne Université, Institut d’Astrophysique de
Paris (IAP), UMR 7095, 98 bis bd Arago, F-75014 Paris, France  Matthew Ho CNRS
& Sorbonne Université, Institut d’Astrophysique de Paris (IAP), UMR 7095, 98
bis bd Arago, F-75014 Paris, France  Benjamin D. Wandelt CNRS & Sorbonne
Université, Institut d’Astrophysique de Paris (IAP), UMR 7095, 98 bis bd
Arago, F-75014 Paris, France  Center for Computational Astrophysics, Flatiron
Institute, 162 5th Avenue, New York, NY 10010, USA

###### Abstract

Bias models relating the dark matter field to the spatial distribution of
halos are widely used in current cosmological analyses. Many models predict
halos purely from the local Eulerian matter density, yet bias models in
perturbation theory require the inclusion of other local properties. We assess
the validity of assuming that only the local dark matter density can be used
to predict the number density of halos in a model-independent way and in the
non-perturbative regime. Utilising $N$-body simulations, we study the
properties of the halo counts field after spatial voxels with near-equal dark
matter density have been permuted. If local-in-matter-density biasing were
valid, the statistical properties of the permuted and un-permuted fields would
be indistinguishable since both represent equally fair draws of the stochastic
biasing model. For voxels of side length $\sim 4-30\,h^{-1}{\rm\,Mpc}$ and for
halos less massive than $\sim 10^{15}\,h^{-1}{\rm\,M_{\sun}}$, we find that
the permuted halo field has a scale-dependent bias with greater than 25% more
power on scales relevant for current surveys. These bias models remove small-
scale power by not modelling correlations between neighbouring voxels, which
substantially boosts large-scale power to conserve the field’s total variance.
This conclusion is robust to the choice of initial conditions and cosmology.
Assuming local-in-matter-density halo biasing cannot, therefore, reproduce the
distribution of halos across a large range of scales and halo masses, no
matter how complex the model. One must either allow the biasing to be a
function of other quantities and/or remove the assumption that neighbouring
voxels are statistically independent.

Cosmology (343); Dark marrer distribution (356); Large-scale structure of the
universe (902)

††journal: ApJL††software:  astropy (Astropy Collaboration et al., 2013, 2018,
2022), matplotlib (Hunter, 2007), numpy (Harris et al., 2020), pandas (Wes
McKinney, 2010; pandas development team, 2020), pylians (Villaescusa-Navarro,
2018).

##  1 Introduction

Uncovering the statistical relation between the luminous tracers of large
scale structure and the underlying matter density field – the “bias” – is a
crucial part of cosmological analyses, since we observe the former but wish to
understand the properties of the latter. It has long been known that clusters
form at high density peaks of the density field (Kaiser, 1984), but an exact
relation between these two quantities at the field level remains unknown.

On the largest scales, one expects that the bias is linear, namely the
overdensity of tracers is proportional to the overdensity of matter. This
approximation breaks down as one moves from linear to quasilinear scales, and
more general functional relationships must be used. The simplest assumption
one can make is that the tracer number density depends solely on the local
Eulerian dark matter density - the so-called “local-in-matter-density” (LIMD)
assumption (Desjacques et al., 2018) which, in a perturbative analysis, allows
one to write the tracer number density as an expansion in the overdensity: the
local bias expansion (Fry & Gaztanaga, 1993). More generally, one can express
the bias model as local in gravitational observables, namely not just the
density field but other terms related to the tidal field or higher derivative
terms (Mirbabayi et al., 2015; Desjacques et al., 2018; Lazeyras & Schmidt,
2018, 2019). One can enumerate such terms based on the order of perturbation
theory such that, if one goes to sufficiently high order and is in the large-
scale, perturbative regime, then the result converges to the truth.

Relying on perturbation theory (Bernardeau et al., 2002) or effective field
theory (EFT) (Carrasco et al., 2012; Senatore & Zaldarriaga, 2014; Senatore,
2015; Perko et al., 2016) inherently limits our analysis to these quasi-linear
scales. Although it is possible to study higher order statistics (D’Amico et
al., 2022) or perform field-level analyses in these frameworks (Schmidt et
al., 2019; Elsner et al., 2020; Kostić et al., 2023; Stadler et al., 2023;
Nguyen et al., 2024), to extract information on non-linear scales one may need
to go beyond such local bias expansions, although some studies have shown that
perturbative methods perform well when tested against $N$-body simulations
(Roth & Porciani, 2011; Schmittfull et al., 2019). Instead of relying of power
series, many parametric forms for the biasing relation have been proposed in
the literature based on the LIMD assumption (Szalay, 1988; Matsubara, 1995,
2011; Frusciante & Sheth, 2012; Neyrinck, 2014; Ata et al., 2015). As such,
many reconstruction analyses typically assume that the number density of halos
is solely a function of the smoothed density field (e.g. Schmoldt et al.,
1999; Erdoǧdu et al., 2004, 2006; Kitaura et al., 2009; Jasche et al., 2010;
Jasche & Wandelt, 2012, 2013; Jasche et al., 2015; Lavaux & Jasche, 2016; Modi
et al., 2018; Jasche & Lavaux, 2019; Lavaux et al., 2019; Ramanah et al.,
2019). Alternatively, one can utilise machine learning methods to map the dark
matter density field (using both local and non-local information) to the halo
counts field (Charnock et al. (2020); see Dai & Seljak (2021) for a Lagrangian
approach).

Unlike in the perturbative regime, we do not have theoretical guarantees that
any of these bias models are accurate and so this must be tested empirically.
Mirbabayi et al. (2015) argue that, at any order in perturbation theory, the
bias can be expressed in terms of locally measurable observables, and that
this argument is closed under renormalisation. However, it could be (as is
often assumed (Szalay, 1988; Matsubara, 1995, 2011; Frusciante & Sheth, 2012;
Neyrinck, 2014; Ata et al., 2015)) that the only important observable is the
local dark matter density field. Testing this assumption is the subject of
this letter. There is an important distinction, however, between testing the
assumptions of the halo biasing model (i.e. which variables are needed to
predict the tracer number density) and the particular model chosen (i.e. what
is the functional form of the bias or the stochasticity in the tracer-matter
relation). Previous tests of the latter seem to suggest that information
beyond the local density is required (e.g. Nguyen et al., 2021; Fang et al.,
2024), however this could just be due to insufficiently accurate LIMD biasing
models. Initial studies of the former also seem to support this conclusion
(Balaguera-Antolínez et al., 2019; Pellejero-Ibañez et al., 2020; Balaguera-
Antolínez et al., 2020).

In this letter we study the former issue in the fully non-linear regime.

We present a model-independent test of the assumption of LIMD halo biasing.
Our test is independent of the functional form of the biasing model, but
simply tests whether halo biasing can be a stochastic function of only the
local matter density. We find that LIMD halo biasing cannot reproduce the
distribution of halos less massive than $\sim 10^{15}\,h^{-1}\,{\rm M_{\sun}}$
when applied to gridded dark matter fields with voxels of size less than $\sim
30\,h^{-1}{\rm\,Mpc}$. Our permutation test naturally enforces conservation of
all 1-point moments and count statistics, while demonstrating the
insufficiency of the LIMD hyopthesis to model halo clustering. We demonstrate
that this conclusion is robust across initial conditions and cosmology by
utilising the quijote suite of $N$-body simulations. We conclude that no LIMD
biasing model, however complex, should be used if one wishes to accurately
predict the spatial distribution of halos given a matter density field.

Our test also provides a method for testing the convergence of halo catalogues
from $N$-body simulations; we find that the minimum mass for which the
catalogues are converged for quijote is $\sim
10^{13.1}\,h^{-1}{\rm\,M_{\sun}}$.

In Sec. 2 we explicitly define LIMD biasing, introduce our model-independent
test and describe the simulations used to perform this test. Our results are
presented in Sec. 3, where we demonstrate that our conclusions are highly
robust to the one free parameter in our test, as well as cosmology and initial
conditions, and investigate the adequacy of LIMD biasing as a function of
scale and halo mass. Sec. 4 discusses these results and presents our
conclusions.

##  2 Methods

###  2.1 Assessing the assumption of local biasing

We define local-in-matter-density (LIMD) halo biasing to be the assumption
that the number of halos in a given region of comoving space (voxel) is drawn
from any probability distribution under the assumptions:

  1. 1.

The parameters of this distribution in a given voxel are only functions of the
dark matter density in that voxel.

  2. 2.

Each voxel is statistically independent.

Note that we will not make any assumptions about the type of distribution
(e.g. Poisson) nor the functional form of its parameters (e.g. the mean for a
Poisson distribution) other than we expect that this function should be a
continuous function of the local dark matter density.

Under these assumptions, if we identify several voxels which have the same
dark matter density and then permute the number of halos in these voxels, we
should end up with an equally likely draw from the underlying distribution. If
the statistical properties of the halo-count field are noticeably different
after this permutation, then at least one of the assumptions of LIMD biasing
cannot be true.

Since the dark matter density field takes continuous values, instead of
performing permutations on voxels which have exactly equal dark matter
density, we first sort the voxels into $n_{\rm bin}$ bins of dark matter
density, and consider two voxels equivalent if they fall within the same bin,
namely if their density is approximately equal. In our fiducial analysis, we
choose to use adaptive bin widths so that each bin has approximately the same
number of voxels and use $n_{\rm bin}=1000$. We demonstrate that our results
are highly robust to this choice in Sec. 3.

To assess the level of agreement between the permuted and un-permuted fields,
we compute the power spectrum, $P_{\rm counts}(k)$, of the counts field using
the pylians library (Villaescusa-Navarro, 2018). We perform 100 permutations
and report the mean and standard deviation at each wavenumber, then compare
this to the true $P_{\rm counts}(k)$.

###  2.2 Simulations

To test the assumption of LIMD biasing, we utilise the quijote suite of
$N$-body simulations. These simulations were run in a periodic box of length
$L=1\,h^{-1}{\rm Gpc}$ using the gadget-iii code (Springel, 2005). We focus on
the high-resolution suite which contain $N_{\rm p}^{3}=1024^{3}$ particles, as
opposed to the standard resolution of $N_{\rm p}^{3}=512^{3}$.

For our fiducial analysis, we consider simulation number 0 from the fiducial
cosmology, which matches the Planck 2018 cosmological parameters (Planck
Collaboration, 2020): $\Omega_{\rm m}=0.3175$, $\Omega_{\rm b}=0.049$,
$h=0.6711$, $n_{\rm s}=0.9624$, $\sigma_{8}=0.834$. In Sec. 3 we verify that
our conclusions are robust if we chose different initial conditions (i.e. for
different simulation numbers in the suite). We also study the sensitivity of
our conclusions to the choice of cosmological parameters by performing our
analysis with the (high-resolution) Latin-hypercube suite of quijote
simulations, consisting of 2000 dark-matter-only simulations with cosmological
parameters arranged on a Latin hypercube in the range: $\Omega_{\rm
m}\in[0.1,0.5]$, $\Omega_{\rm b}\in[0.03,0.07]$, $h\in[0.5,0.9]$, $n_{\rm
s}\in[0.8,1.2]$, $\sigma_{8}\in[0.6,1.0]$.

We make all our comparisons at redshift zero, where we use the positions and
masses of halos from the pre-computed Friends-of-Friends (FoF) (Davis et al.,
1985) halo catalogues, which were obtained using a linking length of $b=0.2$.
For brevity, throughout this letter we only report results using FoF halos,
but we have verified that our conclusions are robust to this choice by
rerunning our analysis with the Rockstar halofinder (Behroozi et al., 2013),
and found consistent conclusions.

Since one would expect halos of different masses to be distributed differently
in space, we binned these halos into nine logarithmically spaced mass bins in
the range $10^{13.1}-10^{15.8}\,h^{-1}\,{\rm M}_{\odot}$, and performed our
test separately for each bin. For halos within this mass range, we find that
our results are consistent between the standard and high-resolution quijote
simulations, whereas this is not true for lower masses. We therefore cannot be
confident that the halo catalogues are converged for halos which are less
massive than this, so simply discard them in our analysis. Our test,
therefore, additionally acts as a method of determining the minimum mass for
which the halo catalogues from these simulations are converged.

We divide the simulation volume into $N^{3}$ cubic voxels and compute the
density field in each voxel using a nearest grid point (NGP) estimator. This
is the same estimator used to compute the gridded halo counts field, and thus
we mitigate effects which arise due to different discretisation procedures. We
have tested our method using a range of other density field estimators: cloud
in cell (CIC), triangular-shape cloud (TSC), and piecewise cubic spline (PCS)
kernel, as well as one inspired by smoothed particle hydrodynamics (SPH)
(Monaghan, 1992; Colombi et al., 2007). For our fiducial analysis we used
$N=128$, corresponding to voxels with a side length of $\Delta x=7.8h^{-1}{\rm
Mpc}$, but we show that our results are insensitive to this choice (and thus
to the smoothing scale of the density field) in Sec. 3. At our fiducial grid
resolution, we find that all density estimators give the same qualitative
trends, although the magnitude of the bias induced by LIMD models varies with
this choice. For large voxel sizes, we find the performance is best for the
NGP density estimator, but the degree of bias depends sensitively on the
choice of mass assignment in the definition of the density field and is much
worse for other choices. We thus choose the NGP estimator to ensure our
results are conservative.

##  3 Results

###  3.1 LIMD halo biasing cannot reproduce the halo distribution

Figure 1: Power spectrum of the halo counts field before and after permuting
voxels with approximately equal density, for halos of mass
$10^{13.7}-10^{14.0}\,h^{-1}{\rm\,M}_{\sun}$ . The permuted fields have
significantly greater power on large scales, thus LIMD halo biasing cannot
reproduce the distribution of halos. The mean and standard deviation are
computed over 100 permutations. The vertical coloured lines correspond to the
wavenumbers used to characterise this discrepancy in later plots.

In Fig. 1 we plot $P_{\rm counts}(k)$ for our fiducial setup for halos in the
mass range $10^{13.7}-10^{14.0}\,h^{-1}{\rm\,M}_{\sun}$, both before and after
permuting voxels with equal dark matter density. When compared to the truth,
we see that the permuted fields have very different halo distributions than
the original field on all scales. In particular, we observe a significant
excess of power on scales $k\lesssim 0.3\,h\,{\rm Mpc^{-1}}$ and too little
power on smaller scales. We infer that this is due to removal of small-scale
power, due to LIMD models ignoring local correlations, which then requires an
overestimation of large-scale power, to respect the conservation of the field-
variance under voxel permutations. This represents the main conclusion of this
letter: the assumption of LIMD halo biasing cannot produce the correct
distribution of halos.

The vertical dashed lines in Fig. 1 correspond to four characteristic
wavenumbers which we use for the remainder of the analysis to quantify the
level of discrepancy, corresponding to approximately $0.01$, $0.03$, $0.1$ and
$0.5\,h\,{\rm Mpc^{-1}}$. For our fiducial analysis, we find that $P_{\rm
counts}(k)$ is a factor of $1.6\pm 0.2$, $1.3\pm 0.07$, $1.16\pm 0.02$ and
$0.97\pm 0.01$ times the true $P_{\rm counts}(k)$, respectively, for these
wavenumbers. It is therefore clear that the discrepancy is significant for
this simulation, with the true power spectrum at approximately the $7.6\sigma$
value for $k=0.1\,h\,{\rm Mpc^{-1}}$.

Figure 2: Fractional error on the power spectrum after permuting equal-density
voxels as a function of halo mass. Significant discrepancies are seen for a
wide range of scales for halo masses below $10^{15}\,h^{-1}{\rm\,M}_{\sun}$,
hence LIMD halo biasing is invalid in this regime. The points and error bars
give the mean and standard deviation of the discrepancy across 100
permutations, respectively.

In Fig. 2 we investigate how this conclusion changes as a function of halo
mass. One observes that the fractional difference remains relatively constant
until $\sim 10^{15}\,h^{-1}{\rm\,M}_{\sun}$, indicating that LIMD halo biasing
is invalid for all masses below this point. For the largest halo masses, we
find that the power spectrum of the permuted field is consistent with the
truth. The number density of these objects is low – there are only 381 halos
above $10^{15}\,h^{-1}{\rm\,M}_{\sun}$ in the fiducial simulation – and these
can only exist at the extreme peaks of the density field. Hence, given that
these objects cannot exists in low density environments, it is perhaps
unsurprising that their number density can be well-predicted by just the local
density field. We therefore conclude that LIMD halo biasing is a reasonable
assumption for halos above $\sim 10^{15}\,h^{-1}{\rm\,M}_{\sun}$, but, for the
majority of objects in the simulation, this framework cannot be used.

Figure 3: Fractional error on the power spectrum for halos of mass
$10^{13.7}-10^{14.0}\,h^{-1}{\rm\,M}_{\sun}$ after permuting equal-density
voxels as a function of voxel size. The black dashed line indicates the value
of $N$ where we have as many voxels as density bins. For $N\geq 2^{6}$ we find
significant disagreement on large scales, suggesting that one must smooth the
density field over at least $30\,h^{-1}{\rm\,Mpc}$ for LIMD halo biasing to be
valid. We do not plot values for wavenumbers beyond the Nyquist frequency,
hence some lines are truncated.

Halo biasing is not just a function of halo mass, but also depends on the grid
used to discretise the dark matter density field, or equivalently the
smoothing scale of the field. In Fig. 3 we investigate how the error on
$P_{\rm counts}(k)$ varies as a function of this grid size. One observes that
the level of disagreement is relatively unchanged on the largest scales as one
changes the voxel size from $\sim 4\,h^{-1}{\rm\,Mpc}$ to $\sim
30h^{-1}{\rm\,Mpc}$, indicating that LIMD biasing is inappropriate for density
fields smoothed on these scales. As one further increases the voxel size to
$\sim 60h^{-1}{\rm\,Mpc}$, the bias starts to diminish, with a bias of $\sim
1.4\sigma$ at $k=0.03\,h\,{\rm Mpc^{-1}}$. In this regime the number of voxels
is similar to the number of density bins (as indicated by the black line in
Fig. 3), but we have verified that similar behaviour is seen if we reduce
$n_{\rm bin}$ to 100. However, even on scales $\sim 0.01\,h{\rm\,Mpc}^{-1}$,
we see a mean fractional error of 25% for a voxel size of$\sim
62.5\,h^{-1}{\rm\,Mpc}$. Thus, the LIMD halo biasing is invalid for smoothed
density fields with a smoothing scale smaller than $\sim 30h^{-1}{\rm\,Mpc}$,
although it may be an acceptable approximation on larger scales.

###  3.2 Sensitivity to binning scheme

Figure 4: Fractional error on the power spectrum of halo counts after
permuting voxels within the same density bin as a function of the number of
bins. We consider halos of mass $10^{13.7}-10^{14.0}\,h^{-1}{\rm\,M}_{\sun}$
and choose a binning scheme such that each bin contains approximately an equal
number of voxels. After an initial transient behaviour, we find constant
biases in the power spectrum across many order of magnitude, hence our
conclusion is robust to the choice of $n_{\rm bin}$.

If one did not bin the density sufficiently finely, then one would expect that
this shuffling procedure would not reproduce the correct distribution of halos
even if LIMD halo biasing were correct, since voxels with very different
densities would be considered equivalent and therefore permuting their halos
would not represent any realistic biasing model. To test this, in Fig. 4 we
plot the fractional error on the power spectrum as a function of $n_{\rm
bin}$. When one has only one density bin, the permuted-halo power spectrum is
consistent with random noise. As we increase $n_{\rm bin}$, one sees an
initial transient behaviour, where the fractional error becomes positive on
large scales at approximately $n_{\rm bin}=10$ and then increases to a
constant level by approximately $n_{\rm bin}=30$. Beyond this point, the
fractional error on $P_{\rm counts}(k)$ is independent across orders of
magnitude of $n_{\rm bin}$ suggesting that our conclusions are robust to this
choice.

We have verified that our results are unchanged if we consider equally spaced
bins in both density and the logarithm of density, instead of enforcing each
bin to have approximately an equal number of voxels. The transient behaviour
disappears at $n_{\rm bin}\approx 10$ when using logarithmic bins and at
$n_{\rm bin}\approx 300$ for the linear binning scheme. Above these values, we
again find that the bias is insensitive to the choice of $n_{\rm bin}$.

Of course, if one as many bins as number of voxels ($n_{\rm bin}=N^{3})$, then
(assuming no voxels have identically equal densities) there would be only one
voxel per bin and thus the permuted halo field is equal to the unpermuted
field. For $N=128$ this corresponds to $n_{\rm bin}\approx 10^{6.3}$. Since
our conclusion holds at $n_{\rm bin}=10^{5}$, even with an average of just 21
voxels per bin we cannot reproduce the halo distribution. A LIMD halo biasing
model which varies that rapidly with density seems implausible and highly
fine-tuned. We have verified that using $n_{\rm bin}=100$ for a mock halo
counts field generated from a Poisson distribution with a mean which depends
on density according to a power law (with parameters optimised to fit the
quijote simulations using a Poisson loss function in each voxel) yields
unbiased $P_{\rm counts}(k)$ upon shuffling, again indicating that our
fiducial choice of $n_{\rm bin}=10^{3}$ is sufficient.

###  3.3 Sensitivity to cosmology and initial conditions

To test the robustness of our conclusion to the initial conditions, we
repeated our analysis across the 100 high-resolution quijote simulations at
the fiducial cosmology. We find biases (mean divided by standard deviation
across 100 permutations) of $1.4\pm 0.9$, $4.5\pm 1.0$, $6.6\pm 0.9$, $-4.7\pm
1.0$ at the wavenumbers shown in Fig. 1, indicating that our conclusion is
insensitive to the initial white noise field.

Similarly, to investigate the sensitivity of the result to the cosmological
parameters, we ran our analysis for all 2000 high-resolution simulations in
the Latin hypercube suite. We find that, averaging over cosmological
parameters, the biases for these wavenumbers are $1.7\pm 1.4$, $4.2\pm 1.5$,
$6.4\pm 1.2$, $-4.5\pm 1.1$ demonstrating that LIMD halo biasing does not hold
for any of the cosmologies considered.

##  4 Discussion and conclusion

In this letter we have shown that the distribution of halos cannot be modelled
according to LIMD biasing: the assumption that the number of halos in a given
voxel is drawn from a probability distribution which depends solely on the
local dark matter density and where each voxel is statistically independent.
At least one of these assumptions must be broken to accurately model the
distribution of halos in the Universe when averaging over scales of $\sim
4-30\,h^{-1}{\rm\,Mpc}$.

We have demonstrated this in a model-independent way, without considering a
particular choice of probability distribution nor the dependence of its
parameters on the local dark matter density. By permuting the number of halos
found in voxels of approximately equal density, we showed that the permuted
field has too much power on large scales, whereas this would be statistically
indistinguishable from the un-permuted field if LIMD biasing were true. This
conclusion is robust to the definition of “approximately equal density” as
well as voxel size, cosmology and initial conditions. We find that the most
massive halos can be modelled under these assumptions, but this breaks down
for halos less massive than $\sim 10^{15}\,h^{-1}{\rm\,M_{\sun}}$.

The test described in this letter is similar in approach to the first step of
the Bias Assignment Method (BAM) (Balaguera-Antolínez et al., 2019), which is
used to fit non-parametric bias models. As here, Balaguera-Antolínez et al.
(2019) find that information beyond the local density is required to recover
the halo distribution but perform their analysis with a single grid resolution
of $\Delta x=3\,h^{-1}{\rm\,Mpc}$, which is slightly finer than we consider.
Our work extends this conclusion by systematically exploring the sensitivity
to this smoothing scale and explicitly investigating the mass dependence of
this conclusion in separate mass bins. BAM has recently been used to provide
evidence for non-locality in the bias model by studying the bispectrum
(Pellejero-Ibañez et al., 2020), with up to $4.8\sigma$ significance (Kitaura
et al., 2022). The conclusions presented here are much stronger than this.
Moreover, we demonstrate that the results are robust to cosmological
parameters and provide more thorough tests of the robustness of our results to
the binning scheme. Unlike Balaguera-Antolínez et al. (2019), we have
demonstrated that our conclusions are robust to the resolution of the $N$-body
simulation used in our test.

This conclusion is particularly important as the use of higher-order
statistics and field-level approaches become increasingly popular in
cosmological analyses, since it is no longer sufficient to tune a bias model
to only reproduce the power spectrum or other low-order statistics; one must
be able to reproduce all moments of the field. In field-level studies, the
initial conditions of the Universe are free parameters, so using a LIMD
biasing model implies inferring an incorrect density field to obtain a
predicted galaxy (or whichever tracer is used in the analysis) field which
reasonably matches observations (Nguyen et al., 2021). Although the inferred
galaxy field at the redshift of the observations would be trustworthy,
extrapolating to other redshifts could be problematic and the underlying
density field and its initial conditions would have artefacts. Given that it
is often these initial conditions which one is interested in, it is imperative
that future studies move beyond the LIMD halo biasing assumption to avoid
these problems.

If the issue causing the deficiency of LIMD models was chiefly the dependence
on local density (i.e. independent draws is a reasonable approximation), then
one could apply a similar permutation technique to identify whether other halo
biasing schemes are valid in a model-independent fashion. For example, by
binning in multiple dimensions, one could assess whether the variables used in
EFT based approaches are sufficient in the non-perturbative regime.
Alternatively, one could search for transformations of the density field for
which the permuted and un-permuted halo number counts are indistinguishable
when binned in this variable and then construct bias models with these new
parameters.

As discussed in Sec. 2, the degree to which LIMD biasing fails depends on the
estimator used to obtain the density field. This high sensitivity to the
estimator is problematic if one wishes to predict the halo distribution from
an approximate gravity model. As such, using information beyond the local
density field is essential to produce realistic halo catalogues for analysis
pipelines for current and future surveys.

We conclude that number of halos in a given region of space is a stochastic
function of the local dark matter density field, but it cannot depend solely
on this quantity. Models which rely on this assumption, no matter how complex,
cannot reproduce the distribution of halos and thus must be superseded by more
accurate models. This letter not only disproves the LIMD assumption of halo
biasing for scales $\lesssim 30\,h^{-1}{\rm\,Mpc}$, but provides a framework
to aid the design of future approaches by testing their assumptions in a
model-independent manner.

We thank Andrés Balaguera, Yan-Chuan Cai, Simon Ding, Ludvig Doeser, Rozo
Eduardo, Pedro Ferreira, Joachim Harnois-Deraps, Jens Jasche, Guilhem Lavaux,
Titouan Lazeyras, Stuart McAlpine, Minh Nguyen, Francisco Villaescusa-Navarro
and the Implicit Likelihood Inference, Accelerated Forward Modelling and
Robustness working groups of the Simons Collaboration on “Learning the
Universe” for useful comments and suggestions. We particularly thank Neal
Dalal and Cora Uhlemann for independently checking our analysis after it
appeared as a pre-print, and prompting us to consider the sensitivity of our
results to the mass assignment scheme. DJB and MH are supported by the Simons
Collaboration on “Learning the Universe.” BDW acknowleges the DIM-
ORIGINES-2023, Infinity Next grant. The Flatiron Institute is supported by the
Simons Foundation. This work was done within the Aquila Consortium
(https://www.aquila-consortium.org/). For the purposes of open access, the
authors have applied a Creative Commons Attribution (CC BY) licence to any
Author Accepted Manuscript version arising.

## References

  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Astropy Collaboration et al. (2022) Astropy Collaboration, Price-Whelan, A. M., Lim, P. L., et al. 2022, ApJ, 935, 167, doi: 10.3847/1538-4357/ac7c74
  * Ata et al. (2015) Ata, M., Kitaura, F.-S., & Müller, V. 2015, MNRAS, 446, 4250, doi: 10.1093/mnras/stu2347
  * Balaguera-Antolínez et al. (2019) Balaguera-Antolínez, A., Kitaura, F.-S., Pellejero-Ibáñez, M., Zhao, C., & Abel, T. 2019, MNRAS, 483, L58, doi: 10.1093/mnrasl/sly220
  * Balaguera-Antolínez et al. (2020) Balaguera-Antolínez, A., Kitaura, F.-S., Pellejero-Ibáñez, M., et al. 2020, MNRAS, 491, 2565, doi: 10.1093/mnras/stz3206
  * Behroozi et al. (2013) Behroozi, P. S., Wechsler, R. H., & Wu, H.-Y. 2013, ApJ, 762, 109, doi: 10.1088/0004-637X/762/2/109
  * Bernardeau et al. (2002) Bernardeau, F., Colombi, S., Gaztañaga, E., & Scoccimarro, R. 2002, Phys. Rep., 367, 1, doi: 10.1016/S0370-1573(02)00135-7
  * Carrasco et al. (2012) Carrasco, J. J. M., Hertzberg, M. P., & Senatore, L. 2012, Journal of High Energy Physics, 2012, 82, doi: 10.1007/JHEP09(2012)082
  * Charnock et al. (2020) Charnock, T., Lavaux, G., Wandelt, B. D., et al. 2020, MNRAS, 494, 50, doi: 10.1093/mnras/staa682
  * Colombi et al. (2007) Colombi, S., Chodorowski, M. J., & Teyssier, R. 2007, MNRAS, 375, 348, doi: 10.1111/j.1365-2966.2006.11330.x
  * Dai & Seljak (2021) Dai, B., & Seljak, U. 2021, Proceedings of the National Academy of Sciences, 118, doi: 10.1073/pnas.2020324118
  * D’Amico et al. (2022) D’Amico, G., Donath, Y., Lewandowski, M., Senatore, L., & Zhang, P. 2022, arXiv e-prints, arXiv:2206.08327, doi: 10.48550/arXiv.2206.08327
  * Davis et al. (1985) Davis, M., Efstathiou, G., Frenk, C. S., & White, S. D. M. 1985, ApJ, 292, 371, doi: 10.1086/163168
  * Desjacques et al. (2018) Desjacques, V., Jeong, D., & Schmidt, F. 2018, Phys. Rep., 733, 1, doi: 10.1016/j.physrep.2017.12.002
  * Elsner et al. (2020) Elsner, F., Schmidt, F., Jasche, J., Lavaux, G., & Nguyen, N.-M. 2020, J. Cosmology Astropart. Phys, 2020, 029, doi: 10.1088/1475-7516/2020/01/029
  * Erdoǧdu et al. (2004) Erdoǧdu, P., Lahav, O., Zaroubi, S., et al. 2004, MNRAS, 352, 939, doi: 10.1111/j.1365-2966.2004.07984.x
  * Erdoǧdu et al. (2006) Erdoǧdu, P., Lahav, O., Huchra, J. P., et al. 2006, MNRAS, 373, 45, doi: 10.1111/j.1365-2966.2006.11049.x
  * Fang et al. (2024) Fang, F., Cai, Y.-C., Li, Z., et al. 2024, MNRAS, 530, 2355, doi: 10.1093/mnras/stae858
  * Frusciante & Sheth (2012) Frusciante, N., & Sheth, R. K. 2012, Journal of Cosmology and Astroparticle Physics, 2012, 016, doi: 10.1088/1475-7516/2012/11/016
  * Fry & Gaztanaga (1993) Fry, J. N., & Gaztanaga, E. 1993, ApJ, 413, 447, doi: 10.1086/173015
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Jasche et al. (2010) Jasche, J., Kitaura, F. S., Wandelt, B. D., & Enßlin, T. A. 2010, MNRAS, 406, 60, doi: 10.1111/j.1365-2966.2010.16610.x
  * Jasche & Lavaux (2019) Jasche, J., & Lavaux, G. 2019, Astron. Astrophys., 625, A64, doi: 10.1051/0004-6361/201833710
  * Jasche et al. (2015) Jasche, J., Leclercq, F., & Wandelt, B. D. 2015, J. Cosmology Astropart. Phys, 2015, 036, doi: 10.1088/1475-7516/2015/01/036
  * Jasche & Wandelt (2012) Jasche, J., & Wandelt, B. D. 2012, MNRAS, 425, 1042, doi: 10.1111/j.1365-2966.2012.21423.x
  * Jasche & Wandelt (2013) —. 2013, MNRAS, 432, 894, doi: 10.1093/mnras/stt449
  * Kaiser (1984) Kaiser, N. 1984, ApJ, 284, L9, doi: 10.1086/184341
  * Kitaura et al. (2022) Kitaura, F. S., Balaguera-Antolínez, A., Sinigaglia, F., & Pellejero-Ibáñez, M. 2022, MNRAS, 512, 2245, doi: 10.1093/mnras/stac671
  * Kitaura et al. (2009) Kitaura, F. S., Jasche, J., Li, C., et al. 2009, MNRAS, 400, 183, doi: 10.1111/j.1365-2966.2009.15470.x
  * Kostić et al. (2023) Kostić, A., Nguyen, N.-M., Schmidt, F., & Reinecke, M. 2023, J. Cosmology Astropart. Phys, 2023, 063, doi: 10.1088/1475-7516/2023/07/063
  * Lavaux & Jasche (2016) Lavaux, G., & Jasche, J. 2016, MNRAS, 455, 3169, doi: 10.1093/mnras/stv2499
  * Lavaux et al. (2019) Lavaux, G., Jasche, J., & Leclercq, F. 2019, arXiv e-prints, arXiv:1909.06396
  * Lazeyras & Schmidt (2018) Lazeyras, T., & Schmidt, F. 2018, J. Cosmology Astropart. Phys, 2018, 008, doi: 10.1088/1475-7516/2018/09/008
  * Lazeyras & Schmidt (2019) —. 2019, J. Cosmology Astropart. Phys, 2019, 041, doi: 10.1088/1475-7516/2019/11/041
  * Matsubara (1995) Matsubara, T. 1995, ApJS, 101, 1, doi: 10.1086/192231
  * Matsubara (2011) Matsubara, T. 2011, Phys. Rev. D, 83, 083518, doi: 10.1103/PhysRevD.83.083518
  * Mirbabayi et al. (2015) Mirbabayi, M., Schmidt, F., & Zaldarriaga, M. 2015, J. Cosmology Astropart. Phys, 2015, 030, doi: 10.1088/1475-7516/2015/07/030
  * Modi et al. (2018) Modi, C., Feng, Y., & Seljak, U. 2018, J. Cosmology Astropart. Phys, 2018, 028, doi: 10.1088/1475-7516/2018/10/028
  * Monaghan (1992) Monaghan, J. J. 1992, ARA&A, 30, 543, doi: 10.1146/annurev.aa.30.090192.002551
  * Neyrinck (2014) Neyrinck, M. C. 2014, in Statistical Challenges in 21st Century Cosmology, ed. A. Heavens, J.-L. Starck, & A. Krone-Martins, Vol. 306, 251–254, doi: 10.1017/S1743921314013702
  * Nguyen et al. (2021) Nguyen, N.-M., Schmidt, F., Lavaux, G., & Jasche, J. 2021, J. Cosmology Astropart. Phys, 2021, 058, doi: 10.1088/1475-7516/2021/03/058
  * Nguyen et al. (2024) Nguyen, N.-M., Schmidt, F., Tucci, B., Reinecke, M., & Kostić, A. 2024, arXiv e-prints, arXiv:2403.03220, doi: 10.48550/arXiv.2403.03220
  * pandas development team (2020) pandas development team, T. 2020, pandas-dev/pandas: Pandas, latest, Zenodo, doi: 10.5281/zenodo.3509134
  * Pellejero-Ibañez et al. (2020) Pellejero-Ibañez, M., Balaguera-Antolínez, A., Kitaura, F.-S., et al. 2020, MNRAS, 493, 586, doi: 10.1093/mnras/staa270
  * Perko et al. (2016) Perko, A., Senatore, L., Jennings, E., & Wechsler, R. H. 2016, arXiv e-prints, arXiv:1610.09321, doi: 10.48550/arXiv.1610.09321
  * Planck Collaboration (2020) Planck Collaboration. 2020, A&A, 641, A6, doi: 10.1051/0004-6361/201833910
  * Ramanah et al. (2019) Ramanah, D. K., Lavaux, G., Jasche, J., & Wandelt, B. D. 2019, A&A, 621, A69, doi: 10.1051/0004-6361/201834117
  * Roth & Porciani (2011) Roth, N., & Porciani, C. 2011, MNRAS, 415, 829, doi: 10.1111/j.1365-2966.2011.18768.x
  * Schmidt et al. (2019) Schmidt, F., Elsner, F., Jasche, J., Nguyen, N. M., & Lavaux, G. 2019, J. Cosmology Astropart. Phys, 2019, 042, doi: 10.1088/1475-7516/2019/01/042
  * Schmittfull et al. (2019) Schmittfull, M., Simonović, M., Assassi, V., & Zaldarriaga, M. 2019, Phys. Rev. D, 100, 043514, doi: 10.1103/PhysRevD.100.043514
  * Schmoldt et al. (1999) Schmoldt, I. M., Saar, V., Saha, P., et al. 1999, AJ, 118, 1146, doi: 10.1086/301001
  * Senatore (2015) Senatore, L. 2015, Journal of Cosmology and Astroparticle Physics, 2015, 007, doi: 10.1088/1475-7516/2015/11/007
  * Senatore & Zaldarriaga (2014) Senatore, L., & Zaldarriaga, M. 2014, arXiv e-prints, arXiv:1409.1225, doi: 10.48550/arXiv.1409.1225
  * Springel (2005) Springel, V. 2005, MNRAS, 364, 1105, doi: 10.1111/j.1365-2966.2005.09655.x
  * Stadler et al. (2023) Stadler, J., Schmidt, F., & Reinecke, M. 2023, J. Cosmology Astropart. Phys, 2023, 069, doi: 10.1088/1475-7516/2023/10/069
  * Szalay (1988) Szalay, A. S. 1988, ApJ, 333, 21, doi: 10.1086/166721
  * Villaescusa-Navarro (2018) Villaescusa-Navarro, F. 2018, Pylians: Python libraries for the analysis of numerical simulations, Astrophysics Source Code Library, record ascl:1811.008.  http://ascl.net/1811.008
  * Wes McKinney (2010) Wes McKinney. 2010, in Proceedings of the 9th Python in Science Conference, ed. Stéfan van der Walt & Jarrod Millman, 56 – 61, doi: 10.25080/Majora-92bf1922-00a


# Federated Generative Learning with Foundation Models

Jie Zhang
Zhejiang University
<EMAIL_ADDRESS>
&Xiaohua Qi
University of Science and Technology of China
<EMAIL_ADDRESS>
&Bo Zhao
Beijing Academy of Artificial Intelligence
<EMAIL_ADDRESS>

###### Abstract

Existing federated learning solutions focus on transmitting features,
parameters or gadients between clients and server, which suffer from serious
low-efficiency and privacy-leakage problems. Thanks to the emerging foundation
generative models, we propose a novel federated learning framework, namely
_Federated Generative Learning_ , that transmits prompts associated with
distributed training data between clients and server. The informative training
data can be synthesized remotely based on received prompts containing little
privacy and the foundation generative models. The new framework possesses
multiple advantages, including improved communication efficiency, better
resilience to distribution shift, substantial performance gains, and enhanced
privacy protection, which are verified in extensive experiments on ImageNet
and DomainNet datasets111The implementation is available at
https://github.com/zj-jayzhang/Federated_Generative_Learning..

##  1 Introduction

Recently, significant progresses has been achieved in many learning fields by
scaling up to large models, i.e., BERT Devlin et al. (2018), GPT3 Brown et al.
(2020), ViT Dosovitskiy et al. (2020), CLIP Radford et al. (2021), Stable
Diffusion Rombach et al. (2022a), and web-scale datasets i.e., YFCC100M Thomee
et al. (2016), CC-12M Changpinyo et al. (2021), LAION-5B Schuhmann et al.
(2022). Typically, the large models are first pre-trained with massive low-
quality web data for basic capability, then fine-tuned with a small number of
high-quality data, especially manually-labeled data, for evoking the desired
capability. Though the web data are easy to access, the high-quality training
data are still scarce, since high-quality datasets are usually private or
unsuitable to release. For example, most medical data are expensive to
labelling and sensitive to release due to safety and privacy problems. In
addition, the raw data is usually the key assets of many companies, which is
impossible to be distributed. Therefore, the demand for collaborative machine
learning Gong et al. (2022); Nguyen & Thai (2022); Mothukuri et al. (2021)
with high efficiency and safety has become increasingly urgent.

In recent years, Federated Learning (FL) McMahan et al. (2017); Kairouz et al.
(2021b); Bonawitz et al. (2019); Konečnỳ et al. (2016); Silva et al. (2019);
Zhao et al. (2018) has got much more attention as a promising solution to the
challenges of data privacy and security in collaborative machine learning. The
classic federated learning framework consists of model initialization, local
training, model aggregation, and global updating. In practice, FL systems
encounter several challenges that impede their real-world application:

  1. 1.

_High communication cost_. Existing FL solutions have to transmit the model
parameters or gradients between clients and server. However, in large-model
era, model parameters are usually billions to trillions. The transmission of
model parameters or gradients requires high communication cost which is
impractical even unaffordable. Furthermore, the communication cost will
significantly increase in federated continual learning scenario, since more
commnication rounds are required for learning from continuously received
training data.

  2. 2.

_Data heterogeneity_. In FL, a fundamental challenge arises from the presence
of statistical heterogeneity among local data distributions across distinct
clients. This leads to the problem of objective inconsistency for model-
averaging-based FL methods, as the global model converges to a stationary
point of a mismatched objective Wang et al. (2020); Zhang et al. (2022b), thus
causing remarkable performance degradation.

  3. 3.

_Privacy and security risks_. The most important concern of FL systems is data
privacy and security. The popular solutions transmit model parameters /
gradients McMahan et al. (2017), or synthetic data Zhou et al. (2020) that all
suffer from serious privacy leakage Zhao et al. (2020). How to train models
efficiently and privately is long-standing problem.

Figure 1: Left: The performance comparison between FedAvg and ours on five
ImageNet subsets and a DomainNet subset. Right: Our results consistently
outperform FedAvg in different data distributions on ImageSquawk dataset. Note
that our 5-round results are better than the centralized training. The circles
represent the transmission cost, i.e., bytes.

The recent advancements in foundation generative models, i.e., Stable
diffusion Rombach et al. (2022b), DALL-E2 Ramesh et al. (2022), Imagen Saharia
et al. (2022), and GLIDE Nichol et al. (2021), have provided a high-quality
text conditional image synthesis which can be used to train models.
Especially, these foundation models are off-the-shelf and publicly available.
Besides, the privacy investigation Carlini et al. (2023) shows that Stable
Diffusion synthesized images will contain 109 near-copies of training images
when synthesizing 175 million images, which means extremely low privacy
leakage risk. Thus, we pose the following question:

Can Federated Learning be accomplished by transmitting the compact knowledge
with little privacy of local training data based on the foundation generative
models?

In this paper, we propose a novel framework called Federated Generative
Learning (FGL), which leverages the powerful foundation generative models,
e.g., Stable Diffusion, to synthesize high-quality substitute training data in
the server based on the prompts from clients. Specifically, our method only
requires each client to upload the prompts corresponding to their local
training data. After collecting all prompts from the clients, the server
performs prompt aggregation and synthesize a high-quality substitute training
dataset. Then, a global model can be trained on the synthetic dataset. There
are multiple benefits of the proposed framework:

  1. 1.

_Low Communication Cost_. Compared to previous methods that rely on multi-
round communication of weights or gradients, our method requires only one
communication round between clients and the server. As shown in Figure 1(b),
during the initial communication between any client and the server, the
parameter transmission in FedAvg needs 42.7MB for the small network ResNet18,
while our method only sends 229 Bytes prompts. In addition, this advantage
will be enlarged when training large models with billions of parameters and
implementing federated continua learning in which requires more communication
rounds for learning from new data.

  2. 2.

_Robust to Data Heterogeneity_. Since our method only requires clients to
upload prompts corresponding to their local training data, it allows the
server to collect prompts from all clients and synthesize all training data in
one communication round. As a result, our method exhibits insensitivity to
data distribution. Figure 1(b) compares our method to FedAvg under three
different data distributions. Ours (one-shot), consistently achieves good
performance across all distributions. Furthermore, our method can
significantly improve the performance and even outperform the centralized
training by implementing five-round communication.

  3. 3.

Better privacy-preserving: Traditional FL involves the transmission of model
parameters or gradients between clients and server. However, once these model
parameters are leaked, the attackers can carry out model extraction attacks Li
et al. (2023b), member inference attacks Li et al. (2023a), and other
malicious activities Zhang et al. (2023b); Zhu et al. (2019), posing
significant security threats to the FL system Nguyen & Thai (2022); Liu et al.
(2022). In contrast, foundation generative model based FL can better protect
data privacy by transmitting prompts. Our method can implement FL in few round
communication which significantly reduces the privacy leakage risk.

##  2 Related Work

###  2.1 Federated Learning

The rising concerns on data privacy and security have caused the data silo
problem that prevents the development of artificial general intelligence.
Federated learning (FL) McMahan et al. (2017) has been a promising solution to
this problem which allows multiple clients to collaboratively train a global
model in the server without exposing their private training data. Typically,
each client trains a local model on its private data and periodically shares
its latest model parameters with the server. Then, the server will deliver the
global models to clients. FL has been widely used in real-world applications,
including healthcare Liu et al. (2018); Silva et al. (2019); Cui et al.
(2021); Liu et al. (2021a); Chen et al. (2021), recommendation systems Liang
et al. (2021); Liu et al. (2021b), finance Suzumura et al. (2019); Liu et al.
(2020); Long et al. (2020) and internet of things Chen et al. (2020); Hu et
al. (2020); Savazzi et al. (2020). Nevertheless, federated learning methods
suffer from the problems of high communication cost, data heterogeneity,
privacy and security risks and significant performance decrease.

Many studies Zhou et al. (2020); Dennis et al. (2021); Zhang et al. (2022a)
attempt to reduce communication overhead in federated learning by designing
one-shot communication methods. For example, Zhu et al. Zhou et al. (2020) use
dataset distillation Wang et al. (2018) to implement one-shot federated
learning. But the distilled dataset has potential privacy leakage problem.
Zhang et al. Zhang et al. (2022a) propose a two-stage data-free one-shot
federated learning method. However, compared to few-shot federated learning,
the performance of this method decreases significantly. Different from them,
our method can realize one-shot federated learning with little communication
overhead and achieve high-performance while little privacy leakage by
transmitting prompts.

To address the data heterogeneity problem, data-based methods pre-process
client data before training, such as using data sharing or data augmentation
techniques Zhao et al. (2018); de Luca et al. (2022). Algorithm-based methods
employ learning techniques, such as knowledge distillation, multi-task
learning, and personalized layers, while system-based methods use client-based
clustering methods to better understand the heterogeneous data environment.
However, these methods do not fundamentally solve the problem of imbalanced
data distribution. In contrast, our method can solve the data heterogeneity
problem by synthesizing all training data at once in the server based on the
prompts sent from all clients.

###  2.2 Foundation Models

##### Foundation Generative Models.

Large generative models, such as Stable Diffusion Rombach et al. (2022b),
DALL-E2 Ramesh et al. (2022), Imagen Saharia et al. (2022), and GLIDE Nichol
et al. (2021), have recently emerged as an off-the-shelf tool for high-quality
and real-looking image generation conditioned on text prompts. A few works
have been exploring the usage of synthetic images as training data. For
example, He et al. He et al. (2022) show that synthetic data generated by
diffusion models can improve pre-training, zero-shot, and few-shot image
classification performance. Li et al. Li et al. (2023d) demonstrate that
synthetic data generated by conditional diffusion models can be used for
knowledge distillation without access to original data. Zhou et al.Zhou et al.
(2023) synthesize better images for model training with stable diffusion by
implementing diffusion inversion. Though good performance has been achieved,
inverting every image is expensive and will leak privacy.

##### Foundation Models in FL.

The foundation generative models are still under-explored in federated
learning, though there exist a few related works that study foundation models
in federated learning. The most similar one is Yang et al. (2023) which
leverage diffusion model in the server to synthesize training samples that
complied to the distributions of domain-specific features from clients. Yu et
al. Yu et al. (2023) introduce federated learning into foundation model
training for training foundation models collaboratively with private data.
Like traditional FL methods, they also transmit model parameters between
servers and clients. Based on the shared CLIP model, Guo et al. Guo et al.
(2022) transmit the small number of updated parameters of prompt learner from
clients to server for reducing communication cost. Our study is different from
them in multiple aspects: 1) Their methods transmit features or parameters,
while our federated generative learning framework transmits prompts which
require less communication while are better at privacy-preserving. 2) Our
framework is based on shared foundation generative models, while theirs are
based on the shared foundation classification models.

##  3 Preliminaries

In this section, we present the proposed method, which utilizes off-the-shelf
foundation generative models, especially diffusion models, in the federated
learning scenario, achieving remarkable performance and better privacy-
preserving with minimal communication. We first provide some notations and
background knowledge on diffusion models. Then, we describe the details of the
proposed method and its extension.

###  3.1 Text-to-image Generative Models

Text-to-image generative models, such as Stable Diffusion Rombach et al.
(2022b), DALL-E2 Ramesh et al. (2022), Imagen Saharia et al. (2022), have
demonstrated the remarkable ability to generate stunning images from natural
language descriptions. Given a pre-trained generative model $G$, such as the
stable diffusion model, users can simply provide a text prompt, which enables
the synthesis of high-quality images. Specifically, during the inference
phase, a new image latent $\bm{z}_{0}$ is generated by iteratively denoising a
random noise vector using a conditioning vector, e.g., text prompt embedding
$\bm{p}$. The latent code is subsequently transformed into an image by
employing the pre-trained decoder, resulting in
$x^{\prime}=G(\bm{z}_{0},\bm{p})$.

In this paper, we only use off-the-shelf pre-trained diffusion model for
inference process, i.e., image generation, without conducting any training.

###  3.2 Federated Learning

In the setting, we have $K$ clients with their private datasets
$\mathcal{D}_{k}=\\{(\bm{x}_{i},y_{i})\\}_{i=1}^{N_{k}}$, where $\bm{x}_{i}$
is the training image, $y_{i}$ is its label, $\mathcal{Y}_{k}$ is the label
set, and $N_{k}$ is the number of training samples on $k$-th client. Note that
the label sets of different clients may be different. The objective of the
federated learning framework is to learn a model parameterized with
$\vartheta$ in the server that minimizes the loss on training data of all
clients without access to original data:

$\min_{\vartheta}\frac{1}{K}\sum_{k=1}^{K}\mathbb{E}_{\bm{x}\sim\mathcal{D}_{k}}[l_{k}(\vartheta;\bm{x})]$
(1)

where, $l_{k}$ is the loss function, i.e., cross-entropy loss, for the $k$-th
client.

Figure 2: The proposed Federated Generative Learning. Firstly, the prompts
from clients are uploaded and then aggregated in the server. Then, stable
diffusion is used to generate synthetic data for training the global model.
The updated model weights are distributed to clients.

##  4 Federated Generative Learning

The overall framework of the proposed _Federated Generative Learning_
framework is illustrated in Figure 2. Unlike traditional federated learning
methods that transmit feature, parameters or gradients, our approach transmits
prompts corresponding to the private data in clients to the server, thus is
better privacy-preserving and communication-efficient. Then the training data
are synthesized based on the aggregated prompts in server with the foundation
diffusion model. The synthetic training data are jointly used to train models,
thus can relieve data heterogeneity problem and improve performance. Our
framework can implement one-shot FL, and the clients do not need to train
models locally. We describe the prompt aggregation process in Section 4.1,
training data synthesis in Section 4.2, one-shot and multi-round model
updating in Section 4.3.

###  4.1 Prompt Generation and Aggregation

We investigate two types of prompt generation strategies: class-level prompt
and instance-level prompt. The class-level prompts are generated based on the
class names, providing high-level guidance to the generative model. On the
other hand, the instance-level prompt strategy leverages prompts that are
tailored for individual instances in the private dataset, which are more
informative for training models.

Figure 3: Prompt generation in clients.

Figure 3 illustrates the two types of prompts. For example, we generate prompt
like “A photo of a {class name}” for every class. Each class-level prompt can
used to synthesize many images accompanying different noises. Motivated by Lei
et al. (2023), we use BLIP-v2 to generate captions for each real image
$\bm{x}_{i}$ as the instance-level prompt, denoted as $\bm{p}_{i}$. After
receiving prompts and corresponding labels
$\\{(\bm{p}_{i},y_{i})\\}_{i=1}^{N}$ from all clients, where $N$ is the number
of total prompts, the server aggregates them for data synthesis with
foundation generative models, i.e., Stable Diffusion.

###  4.2 Training Set Synthesis

After receiving all prompts, the server synthesizes every training sample
$\bm{s}_{i}$ by prompting the pre-trained Stable Diffusion with each
$\bm{p}_{i}$ as follows:

$\bm{s}_{i}=G(\bm{z}_{i},\bm{p}_{i})=\sqrt{\beta}\sum_{t=1}^{T}\sqrt{1-\beta^{t}}\cdot\frac{1}{\sqrt{T}}\cdot
G_{\bm{\vartheta}_{t}}(\bm{z}_{i},\bm{p}_{i}),$ (2)

where $\bm{z}_{i}$ is a random noise vector, $\bm{p}_{i}$ is the prompt, and
$G_{\bm{\theta}_{t}}$ is the denoising network parameterized with
$\bm{\theta}_{t}$ at time step $t$. The hyperparameter $\beta$ controls the
trade-off between image quality and diversity, and $T$ is the number of
diffusion steps. The inference process iteratively denoises the image then
outputs the final synthetic training image. Finally, the server generate the
synthetic training set $\mathcal{S}=\\{(\bm{s}_{i},y_{i})\\}_{i=1}^{N}$. Note
that it is easy to synthesize more diverse training samples by combining
multiple random noises with the sample prompt, thus training better models. In
practice, we can adjust the number of synthetic training samples to trade-off
the computational cost and performance.

###  4.3 Model Updating

####  4.3.1 One-shot Updating.

After obtaining the synthetic training set
$\mathcal{S}=\\{(\bm{s}_{i},y_{i})\\}_{i=1}^{N}$, we jointly train the model
in the server, i.e., ResNet18 with cross-entropy loss:

$\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}y_{i}\log(\hat{y_{i}})+(1-y_{i})\log(1-\hat{y_{i}}),$
(3)

where $\hat{y_{i}}$ is the model prediction. Upon completion of global model
training, we send the model weights to each client, thereby completing the
one-shot communication.

⬇

# for first communication round, at server side.

prompts_list = [], global_model = network()

for client in all_clients:

# class-level prompts or instance-level prompts

prompts_client = prompts_generation(client)

prompts_list.append(prompt)

prompts = prompts_aggregation(prompts_list)

synthetic_dataloader = generative_model(prompts, noises)

# train the global model at server side

server_update(synthetic_dataloader, global_model)

# for 5-round communication

for com_round in [2,3,4,5]:

local_weights = []

# randomly seletct m clients

for client in selected_clients:

# SGD update on local training data

weight = local_update(client, global_model)

local_weights.append(weight)

# update global model via aggregating local weights

global_model = model_avegraging(local_weights)

# finetune global model for highly skewed data distribution at server side.

if server_finetune: server_update(synthetic_dataloader, global_model)

Notes: The foundation generative model has never seen any private data from
any client, and all synthetic data is only generated and stored on the server.

Algorithm 1 Federated Generative Learning: PyTorch-like Pseudocode

####  4.3.2 Multi-round Updating

Our method can also implement multi-round communication like traditional FL
methods, which can bring further performance improvement. We consider two
updating algorithms: updating models _without_ and _with_ synthetic data in
the following rounds of communication. Note that model is trained on the
synthetic training set in the first-round communication for both two
algorithms.

##### Without Synthetic Data.

After the first-round communication, each client receives the updated model
from the server and then locally fine-tuning on it on private real training
data, i.e., $\bm{\vartheta}_{k}$. After locally fine-tuning, the server
collect updated models from all clients for model aggregation, which is
formulated as follows:

$\bm{\vartheta}=\frac{1}{K}\sum_{k=1}^{K}\bm{\vartheta}_{k},$ (4)

where $K$ is the total number of clients. In other words, the server only
aggregates models after the first-round communication. This process is
repeated until model converging or reaching the maximum communication rounds
$T$.

##### With Synthetic Data.

Although it takes extra computation cost, we find that fine-tuning the
aggregated model on synthetic training set in every-round communication can
remarkablely improve the performance when the data distribution is extremely
imbalanced in clients. We empirically study it in Section 5.2. The Pytorch-
like pseudocode of our method is presented in Algorithm 1.

##  5 Experimental Results

###  5.1 Experimental Setups

Data Partition: Follow the setting in Kairouz et al. (2021a); Li et al.
(2022), we adopt two classic data partitioning strategies, namely label
distribution skew and feature distribution skew:

  * •

Label Distribution Skew: Following in Zhang et al. (2022b); Li & Zhan (2021);
Zhang et al. (2023b), in which the label distributions varies on different
clients, we employ the Dirichlet distribution $p\sim Dir(\beta)$ to simulate
imbalanced label distributions. The hyper-parameter $\beta$ controls the
degree of label imbalance, where a smaller value of $\beta$ indicates a more
skewed label distribution.

  * •

Feature Distribution Skew: In this setting, clients share the same label space
while different feature distribution, which has been extensively studied in
previous works Li et al. (2021); Zhu et al. (2022); Yao et al. (2022); Gong et
al. (2022). We conduct the classification task on natural images sourced from
DomainNet Peng et al. (2019), which consists of diverse distributions of
natural images from six distinct data sources.

Baselines: In our experiments, we select the popular FedAvg McMahan et al.
(2017) method and centralized training as the baselines. Assume that we have a
total of 5 clients, and every client participates in communication. For both
centralized training and federated learning, the local learning rate is set to
be 0.01, and we utilize the SGD optimizer with a momentum of 0.9. During the
FedAvg training process, each client performs local updates for 5 epochs, and
the communication round is set to be 200. The centralized training consists of
120 rounds of iterations.

Implementation: We employ the class-level prompt by default, which does not
directly utilize local data information, thus providing enhanced privacy
protection. We use Stable Diffusion v2-1-base model to construct synthetic
data. We evaluate the performance of our method in scenarios of one-round
(i.e., Ours (one-shot)) and five-round communications (i.e., Ours (5-round)).
In the one-round communication scenario, we train the model for 120 epochs on
the server. In the five-round communication scenario, we implement two
variations: 1) Ours (5-round): Based on the model trained in the first round,
we perform four additional rounds of communication using the FedAvg algorithm.
2) Ours (5-round-syn) : For extreme data distribution skew scenario, we
further fine-tune the aggregated model using the synthetic dataset generated
on the server during the first round for all five epochs. Please refer to
Section 4.3.2 for more details.

###  5.2 Results on ImageNet Subsets

Table 1: Performance comparison among different methods on subsets of
ImageNet. Hyper-parameter $\beta$ controls the degree of label imbalance. We
synthesize $20k$ images per class. The improvement ↑ is compared to FedAvg IID
results.

Dataset | FedAvg | Ours (one-shot) | Ours (5-round) | Centralized
---|---|---|---|---
| $\beta=0.01$ | $\beta=0.5$ | IID |  | $\beta=0.01$ | $\beta=0.5$ | IID |
ImageNette | 51.6 | 75.0 | 79.2 | $85.2_{{\color[rgb]{0,0,1}\uparrow 6.0}}$ | 82.8 | 94.0 | $\textbf{95.6}_{{\color[rgb]{0,0,1}\uparrow 16.4}}$ | 92.2
ImageFruit | 29.0 | 51.2 | 55.6 | $71.8_{{\color[rgb]{0,0,1}\uparrow 16.2}}$ | 67.2 | 80.2 | $\textbf{83.2}_{{\color[rgb]{0,0,1}\uparrow 27.6}}$ | 78.2
ImageYellow | 50.6 | 70.2 | 74.6 | $82.4_{{\color[rgb]{0,0,1}\uparrow 7.8}}$ | 79.4 | 91.0 | $\textbf{94.8}_{{\color[rgb]{0,0,1}\uparrow 20.2}}$ | 90.8
ImageSquawk | 49.6 | 73.2 | 79.8 | $88.8_{{\color[rgb]{0,0,1}\uparrow 9.0}}$ | 90.0 | 95.0 | $\textbf{95.6}_{{\color[rgb]{0,0,1}\uparrow 15.8}}$ | 92.4
ImageNet100 | 36.3 | 44.6 | 49.4 | $48.4_{{\color[rgb]{0,0,1}\downarrow 1.0}}$ | 70.1 | 74.9 | $\textbf{80.1}_{{\color[rgb]{0,0,1}\uparrow 30.7}}$ | 77.0

(a) Accuracy on Highly Skewed Data ($\beta=0.01$).

(b) Performance with Varying Synthetic Data Volume.

Figure 4: Left: Performance comparison among FedAvg, Ours (5-round), and Ours
(5-round-syn) on four subsets of ImageNet with $\beta=0.01$. Right:
Relationship between the number of synthetic images and testing accuracy on
four datasets.

To evaluate the efficacy of our method on datasets with label distribution
skew, we conduct experiments on five subsets of 224$\times$224 ImageNet
Russakovsky et al. (2015). Firstly, following Howard ; Cazenavette et al.
(2022), we do experiments on four datasets with 10 categories each, namely the
coarse-grained ImageNette and ImageYellow, and the fine-grained ImageFruit and
ImageSquawk. We further conducted experiments on a larger-scale dataset,
namely ImageNet100, involving 100 categories. For each dataset, we generate
20$k$ images per class using the diffusion model. For the 10-category
classification task, we employ the ResNet-18 He et al. (2016) model, while for
the 100-category classification task, we utilize the ResNet-34 model. We
simulate three distinct dataset distributions, i.e., IID, non-IID
($\beta=0.5$) and highly skewed distribution with $\beta=0.01$. As the varying
data distribution does not influence our method during the initial round of
communication for collecting prompts, our results in different settings are
the same. Thus, we report a single value.

##### 10-category Datasets.

The overall experimental results are shown in Table 1. It is evident that our
method with one-round communication outperforms the FedAvg method with 200
rounds of communication, by 6.0%, 16.2%, 7.8% and 9.0% on the four 10-category
datasets in IID setting. Notably, our method is completely insensitive to data
distribution in the first round. Hence, under extreme data distribution, i.e.,
$\beta=0.01$, our method surpasses FedAvg by 33.6%, 42.8%, 31.8% and 39.2% on
the four 10-category datasets respectively. In the case of IID setting, our
method, after five rounds of communication, exhibits significant performance
improvements over FedAvg, namely 16.4%, 27.6%, 20.2% and 15.8% respectively.
It is worth noting that our method with 5 round communication, in IID and
$\beta=0.5$, surpasses the performance of centralized training on all four
datasets.

##### 100-category Datasets.

On the challenging ImageNet100 dataset, our one-round communication method is
slightly (1%) worse than FedAvg with 200 rounds of communication, while
outperforms it in skewed label distribution settings by large margins. When
implementing 5 round communication, our method exceeds FedAvg by around 30% in
different settings.

##### Results on Highly Skewed Data.

Table 1 shows that when the data distribution is extremely skewed
($\beta=0.01$), Ours (5-round) does not outperform Ours (one-shot) on
Imagenette, ImageFruit, and ImageYellow datasets. We attribute this phenomenon
to the fact that in case of highly skewed data distribution, more rounds of
communication are required for the models to converge gradually Zhao et al.
(2018); Wang et al. (2020). In order to achieve better results within five
rounds, we perform fine-tuning on the aggregated model using the synthesized
dataset from the first round on the server for 5 additional epochs before
distributing the updated model, namely Ours (5-round-syn). The results are
presented in Figure 4(a), which clearly demonstrate that fine-tuning on our
synthesized dataset can significantly enhance model performance in scenarios
of extreme data distribution.

Figure 5: Accuracy of class-level prompt and instance-level prompt on four
datasets. We synthesize 1300 images per class. After using instance-level
prompts, the best accuracy improved by 10.2%, 4.6%, 5.8%, and 3.8%
respectively.

##### Class-level Prompts versus Instance-level Prompts.

In main experiments,we synthesize a number of samples based on class-level
prompts, e.g., _A photo of a [class name]_ , and random noise using Stable
Diffusion. Then following Lei et al. (2023), we caption individual images with
foundation models, e.g., BLIP-2 Li et al. (2023c), and synthesize the new
image based on the individual caption and random noise. These prompts provide
more precise guidance to the generative model by considering the specific
characteristics and context of each sample. Note that Zhou et al. (2023) also
produce instance-level prompts by diffusion inversion, while the inverted
samples will leak data privacy and it is very expensive to implement diffusion
inversion on large datasets for clients. In Figure 5, we present the
performances of two kinds of prompts, where we synthesize 1300 images per
class, the same number as the real training set. It is evident that employing
a more precise instance-level prompt leads to higher accuracy, e.g., achieving
78.62% on ImageNette, compared to class-level prompt, which achieves 73.20%.
This result clearly highlights the importance of considering more detailed
instance-level information in the prompt designing. However, the instance-
level prompts involves specific real image information, which may raise
potential privacy concerns, and we leave it as the future work.

##### Number of Synthetic Data.

By comparing the results of 1.3$k$ synthetic images per class in Figure 5 and
20$k$ synthetic images per class Table 1, we find the performance of our
method can significantly improve by synthesizing more training samples, i.e.,
from 73.2% to 85.2% on the Imagenette dataset. We further study the influence
of synthetic image number and model performance in Figure 4(b) on four
10-category datasets. We synthesize more images per class by integrating the
prompts and more random noises. Obviously, as the number of images per class
increases from 2k to 20k, the model’s testing accuracy improves consistently
for all datasets. Hence, our Federated Generative Learning can easily improve
the performance by synthesizing more training samples in the server with
little extra cost.

###  5.3 Results on DomainNet Subset

To simulate the scenario of feature distribution skew, we select 10 categories
from the DomainNet dataset to conduct experiments . Each client is assigned a
specific domain, and we have a total of 6 clients participating in FL. To
ensure an adequate amount of data, we synthesize 3,500 samples for each class
within each domain, resulting in a cumulative dataset of 210$k$ samples. To
train our classification model, we utilize the ResNet-18 architecture as the
backbone. We configure the training hyper-parameters as follows: 120 training
epochs, setting the learning rate to 0.01, employing an SGD optimizer with a
momentum of 0.9, utilizing a batch size of 128, and applying a weight decay of
2e-4.

Table 2 presents the performance of various methods on six domains
respectively and their average accuracy. It shows that in the one-shot
communication scenario, our method outperforms FedAvg by 2% to 19% in five
domains, but exhibits notably poor performance in the Quickdraw domain. To
investigate the underlying reason, we visualize the synthetic and real data in
Figure 6. It becomes apparent that this performance decline is attributed to
the difficulty for diffusion model to synthesize images that align with
Quickdraw domain when using the class-level prompt, i.e., “A black and white
drawing of a {class name}”. We leave this problem as the future work.
Nevertheless, when implementing five-round communication experiment, our
method demonstrates a 2.2% performance improvement over FedAvg specifically on
the Quickdraw domain, and an overall performance improvement of 11.75%.
Remarkably, our method even surpasses the performance of the centralized
training models by 5.95%. We further provide more results regarding the
centralized training and our method on each domain in Appendix.

Table 2: Performance comparison in the scenario of feature distribution skew.
Each client hosts data from a specific domain of DomainNet dataset.

Method | Clipart | Infograph | Painting | Quickdraw | Real | Sketch | Average
---|---|---|---|---|---|---|---
FedAvg | 80.97 | 41.90 | 57.33 | 78.93 | 80.56 | 70.06 | 72.30
Centralized | 81.37 | 50.82 | 60.63 | 92.46 | 82.20 | 73.93 | 78.10
Ours (one-shot) | $83.40_{{\color[rgb]{0,0,1}\uparrow 2.43}}$ | $49.58_{{\color[rgb]{0,0,1}\uparrow 7.68}}$ | $76.88_{{\color[rgb]{0,0,1}\uparrow 19.55}}$ | $51.80_{{\color[rgb]{0,0,1}\downarrow 27.13}}$ | $87.06_{{\color[rgb]{0,0,1}\uparrow 6.5}}$ | $81.10_{{\color[rgb]{0,0,1}\uparrow 11.04}}$ | $71.59_{{\color[rgb]{0,0,1}\downarrow 0.71}}$
Ours (5-round) | $\textbf{90.89}_{{\color[rgb]{0,0,1}\uparrow 9.92}}$ | $\textbf{61.61}_{{\color[rgb]{0,0,1}\uparrow 19.71}}$ | $\textbf{79.52}_{{\color[rgb]{0,0,1}\uparrow 22.19}}$ | $\textbf{81.13}_{{\color[rgb]{0,0,1}\uparrow 2.2}}$ | $\textbf{91.13}_{{\color[rgb]{0,0,1}\uparrow 10.57}}$ | $\textbf{90.20}_{{\color[rgb]{0,0,1}\uparrow 20.14}}$ | $\textbf{84.05}_{{\color[rgb]{0,0,1}\uparrow 11.75}}$

Figure 6: Visualization of original and synthetic data on the Painting and
QuickDraw domains of DomainNet dataset. Obviously, it is easy for diffusion
model to synthesize painting-like images, while challenging to synthesize
images with QuickDraw style.

###  5.4 Privacy Analysis

Figure 7: Retrieving the similar real training images for each synthetic
image. Table 3: The absolute difference between the average values of member
and non-member individuals. The smaller the difference, the better privacy
protection.

Metric | Entropy | Loss | Confidence | Modified Entropy
---|---|---|---|---
Method | FedAvg | Ours | FedAvg | Ours | FedAvg | Ours | FedAvg | Ours
ImageNette | 0.192 | $0.085_{{\color[rgb]{0,0,1}\downarrow 0.107}}$ | 0.111 | $0.083_{{\color[rgb]{0,0,1}\downarrow 0.028}}$ | 0.119 | $0.087_{{\color[rgb]{0,0,1}\downarrow 0.032}}$ | 0.255 | $0.273_{{\color[rgb]{0,0,1}\uparrow 0.018}}$
ImageFruit | 0.123 | $0.012_{{\color[rgb]{0,0,1}\downarrow 0.111}}$ | 0.100 | $0.020_{{\color[rgb]{0,0,1}\downarrow 0.080}}$ | 0.105 | $0.021_{{\color[rgb]{0,0,1}\downarrow 0.084}}$ | 0.288 | $0.074_{{\color[rgb]{0,0,1}\downarrow 0.214}}$
ImageYellow | 0.073 | $0.051_{{\color[rgb]{0,0,1}\downarrow 0.022}}$ | 0.138 | $0.041_{{\color[rgb]{0,0,1}\downarrow 0.097}}$ | 0.141 | $0.043_{{\color[rgb]{0,0,1}\downarrow 0.098}}$ | 0.529 | $0.110_{{\color[rgb]{0,0,1}\downarrow 0.419}}$
ImageSquawk | 0.166 | $0.047_{{\color[rgb]{0,0,1}\downarrow 0.119}}$ | 0.113 | $0.042_{{\color[rgb]{0,0,1}\downarrow 0.071}}$ | 0.119 | $0.044_{{\color[rgb]{0,0,1}\downarrow 0.075}}$ | 0.287 | $0.137_{{\color[rgb]{0,0,1}\downarrow 0.150}}$

##### Detecting Content Replication and Memorization.

Previous research Carlini et al. (2023); Zhang et al. (2023a); van den Burg &
Williams (2021); Somepalli et al. (2023b; a) has indicated that diffusion
models store and reproduce specific images from their training dataset during
the generation process. Although our training process (with class-level
prompts) does not access the private data of clients, we still discuss the
potential privacy risks that may arise. Follow the setting in Somepalli et al.
(2023a), we conduct image retrieval experiments, which allows us to compare
the synthetic images with the original training images and detect any
instances of content duplication. We perform a quantitative analysis on 1000
synthetic images across four datasets. For each synthetic image, we search the
training set by computing the Cosine similarity between its feature and
features of real training images. Figure 7 showcases the top 1 and top 2 most
similar images from each of the four datasets. It is obvious that these images
exhibit no noteworthy similarities in terms of both background and foreground.
This observation verifies that our method does not compromise the privacy of
the client’s private data.

Figure 8: Distribution of the loss metrics between members and non-member
samples. A greater discrepancy in distribution indicates more information
leakage from the model regarding its training set.

##### Membership Inference Attack (MIA).

The objective of MIA is to examine whether a specific data point belongs to
the training set used to train a machine learning model. Due to the fact that
our method does not directly use real training data, intuitively it should be
able to defend against membership inference attacks. To verify this, we employ
popular metrics in MIA, such as confidence Salem et al. (2018), loss
Sablayrolles et al. (2019); Yeom et al. (2018), entropy Salem et al. (2018),
and modified entropy Song & Mittal (2021), to compute the statistical
characteristics of member and non-member data under both FedAvg and our
method. The greater similarity between the distributions of the model’s output
metrics (e.g., loss and entropy) for training members and non-members, the
more membership privacy the model can guarantee. We calculate the average
metric values for training members and non-members, respectively, and then
compare the difference between the two averages. The results are presented in
Table 3, which demonstrates that our method can reduce the discrepancy between
training members and non-members on most metrics compared to FedAvg. In other
words, it indicates that our model’s output distributions of training members
and non-members are more similar. Additionally, we provide the probability
distribution of the metrics for training members and non-members, as depicted
in Figure 8. Obviously, our model exhibits significantly more similar
distributions for training members and non-members compared to FedAvg. The
distributions of the other three metrics can be found in Appendix.

##  6 Conclusion

In this work, we introduce a pioneering framework for Federated Learning,
named _Federated Generative Learning_ , which transmits prompts associated
with distributed training data between clients and the server. By leveraging
foundation generative models, informative training data can be synthesized
remotely using received prompts that contain minimal privacy. The proposed
framework exhibits several noteworthy advantages, including improved
communication efficiency, better resilience to distribution shift, substantial
performance gains, and enhanced privacy protection. We hope this work can
inspire researchers in the field of Federated Learning to shift from
transmitting model parameters/gradients to the new paradigm of transmitting
prompts.

## References

  * Bonawitz et al. (2019) Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečnỳ, Stefano Mazzocchi, Brendan McMahan, et al.  Towards federated learning at scale: System design.  _Proceedings of machine learning and systems_ , 1:374–388, 2019.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Carlini et al. (2023) Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting training data from diffusion models.  _arXiv preprint arXiv:2301.13188_ , 2023.
  * Cazenavette et al. (2022) George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu.  Dataset distillation by matching training trajectories.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4750–4759, 2022.
  * Changpinyo et al. (2021) Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _CVPR_ , 2021.
  * Chen et al. (2020) Mingzhe Chen, H Vincent Poor, Walid Saad, and Shuguang Cui.  Wireless communications for collaborative federated learning.  _IEEE Communications Magazine_ , 58(12):48–54, 2020.
  * Chen et al. (2021) Zhen Chen, Meilu Zhu, Chen Yang, and Yixuan Yuan.  Personalized retrogress-resilient framework for real-world medical federated learning.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24_ , pp. 347–356. Springer, 2021.
  * Cui et al. (2021) Jianfei Cui, He Zhu, Hao Deng, Ziwei Chen, and Dianbo Liu.  Fearh: Federated machine learning with anonymous random hybridization on electronic medical records.  _Journal of Biomedical Informatics_ , 117:103735, 2021.
  * de Luca et al. (2022) Artur Back de Luca, Guojun Zhang, Xi Chen, and Yaoliang Yu.  Mitigating data heterogeneity in federated learning with data augmentation.  _arXiv preprint arXiv:2206.09979_ , 2022.
  * Dennis et al. (2021) Don Kurian Dennis, Tian Li, and Virginia Smith.  Heterogeneity for the win: One-shot federated clustering.  In _International Conference on Machine Learning_ , pp. 2611–2620. PMLR, 2021.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Gong et al. (2022) Xuan Gong, Abhishek Sharma, Srikrishna Karanam, Ziyan Wu, Terrence Chen, David Doermann, and Arun Innanje.  Preserving privacy in federated learning with ensemble cross-domain knowledge distillation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 11891–11899, 2022.
  * Guo et al. (2022) Tao Guo, Song Guo, Junxiao Wang, and Wenchao Xu.  Promptfl: Let federated participants cooperatively learn prompts instead of models–federated learning in age of foundation model.  _arXiv preprint arXiv:2208.11625_ , 2022.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. (2022) Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.  Is synthetic data from generative models ready for image recognition?  _arXiv preprint arXiv:2210.07574_ , 2022.
  * (17) Jeremy Howard.  A smaller subset of 10 easily classified classes from imagenet, and a little more french, 2019.  _URL https://github. com/fastai/imagenette_.
  * Hu et al. (2020) Rui Hu, Yuanxiong Guo, E Paul Ratazzi, and Yanmin Gong.  Differentially private federated learning for resource-constrained internet of things.  _arXiv preprint arXiv:2003.12705_ , 2020.
  * Kairouz et al. (2021a) Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ , 14(1–2):1–210, 2021a.
  * Kairouz et al. (2021b) Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ , 14(1–2):1–210, 2021b.
  * Konečnỳ et al. (2016) Jakub Konečnỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon.  Federated learning: Strategies for improving communication efficiency.  _arXiv preprint arXiv:1610.05492_ , 2016.
  * Lei et al. (2023) Shiye Lei, Hao Chen, Sen Zhang, Bo Zhao, and Dacheng Tao.  Image captions are natural prompts for text-to-image models.  _arXiv preprint_ , 2023.
  * Li et al. (2023a) Jiacheng Li, Ninghui Li, and Bruno Ribeiro.  Effective passive membership inference attacks in federated learning against overparameterized models.  In _The Eleventh International Conference on Learning Representations_ , 2023a.
  * Li et al. (2023b) Jingtao Li, Adnan Siraj Rakin, Xing Chen, Li Yang, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti.  Model extraction attacks on split federated learning.  _arXiv preprint arXiv:2303.08581_ , 2023b.
  * Li et al. (2023c) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023c.
  * Li et al. (2022) Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He.  Federated learning on non-iid data silos: An experimental study.  In _2022 IEEE 38th International Conference on Data Engineering (ICDE)_ , pp. 965–978. IEEE, 2022.
  * Li et al. (2021) Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou.  Fedbn: Federated learning on non-iid features via local batch normalization.  _arXiv preprint arXiv:2102.07623_ , 2021.
  * Li & Zhan (2021) Xin-Chun Li and De-Chuan Zhan.  Fedrs: Federated learning with restricted softmax for label distribution non-iid data.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pp. 995–1005, 2021.
  * Li et al. (2023d) Zheng Li, Yuxuan Li, Penghai Zhao, Renjie Song, Xiang Li, and Jian Yang.  Is synthetic data from diffusion models ready for knowledge distillation?  _arXiv preprint arXiv:2305.12954_ , 2023d.
  * Liang et al. (2021) Feng Liang, Weike Pan, and Zhong Ming.  Fedrec++: Lossless federated recommendation with explicit feedback.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 4224–4231, 2021.
  * Liu et al. (2018) Dianbo Liu, Timothy Miller, Raheel Sayeed, and Kenneth D Mandl.  Fadl: Federated-autonomous deep learning for distributed electronic health record.  _arXiv preprint arXiv:1811.11400_ , 2018.
  * Liu et al. (2022) Pengrui Liu, Xiangrui Xu, and Wei Wang.  Threats, attacks and defenses to federated learning: issues, taxonomy and perspectives.  _Cybersecurity_ , 5(1):1–19, 2022.
  * Liu et al. (2021a) Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng.  Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1013–1023, 2021a.
  * Liu et al. (2021b) Shuchang Liu, Shuyuan Xu, Wenhui Yu, Zuohui Fu, Yongfeng Zhang, and Amelie Marian.  Fedct: Federated collaborative transfer for recommendation.  In _Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval_ , pp. 716–725, 2021b.
  * Liu et al. (2020) Yuan Liu, Zhengpeng Ai, Shuai Sun, Shuangfeng Zhang, Zelei Liu, and Han Yu.  Fedcoin: A peer-to-peer payment system for federated learning.  In _Federated Learning: Privacy and Incentive_ , pp. 125–138. Springer, 2020.
  * Long et al. (2020) Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang.  Federated learning for open banking.  In _Federated Learning: Privacy and Incentive_ , pp. 240–254. Springer, 2020.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pp. 1273–1282. PMLR, 2017.
  * Mothukuri et al. (2021) Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava.  A survey on security and privacy of federated learning.  _Future Generation Computer Systems_ , 115:619–640, 2021\.
  * Nguyen & Thai (2022) Truc Nguyen and My T Thai.  Preserving privacy and security in federated learning.  _arXiv preprint arXiv:2202.03402_ , 2022.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Peng et al. (2019) Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1406–1415, 2019.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Rombach et al. (2022a) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 10684–10695, June 2022a.
  * Rombach et al. (2022b) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10684–10695, 2022b.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Sablayrolles et al. (2019) Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou.  White-box vs black-box: Bayes optimal strategies for membership inference.  In _International Conference on Machine Learning_. PMLR, 2019.
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salem et al. (2018) Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes.  Ml-leaks: model and data independent membership inference attacks and defenses on machine learning models.  _arXiv preprint arXiv:1806.01246_ , 2018.
  * Savazzi et al. (2020) Stefano Savazzi, Monica Nicoli, and Vittorio Rampa.  Federated learning with cooperating devices: A consensus approach for massive iot networks.  _IEEE Internet of Things Journal_ , 7(5):4641–4654, 2020.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _arXiv preprint arXiv:2210.08402_ , 2022.
  * Silva et al. (2019) Santiago Silva, Boris A Gutman, Eduardo Romero, Paul M Thompson, Andre Altmann, and Marco Lorenzi.  Federated learning in distributed medical databases: Meta-analysis of large-scale subcortical brain data.  In _2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019)_ , pp. 270–274. IEEE, 2019.
  * Somepalli et al. (2023a) Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Diffusion art or digital forgery? investigating data replication in diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6048–6058, 2023a.
  * Somepalli et al. (2023b) Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Understanding and mitigating copying in diffusion models.  _arXiv preprint arXiv:2305.20086_ , 2023b.
  * Song & Mittal (2021) Liwei Song and Prateek Mittal.  Systematic evaluation of privacy risks of machine learning models.  In _USENIX Security Symposium_ , 2021.
  * Suzumura et al. (2019) Toyotaro Suzumura, Yi Zhou, Natahalie Baracaldo, Guangnan Ye, Keith Houck, Ryo Kawahara, Ali Anwar, Lucia Larise Stavarache, Yuji Watanabe, Pablo Loyola, et al.  Towards federated graph learning for collaborative financial crimes detection.  _arXiv preprint arXiv:1909.12946_ , 2019.
  * Thomee et al. (2016) Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  _Communications of the ACM_ , 59(2):64–73, 2016\.
  * van den Burg & Williams (2021) Gerrit van den Burg and Chris Williams.  On memorization in probabilistic deep generative models.  _Advances in Neural Information Processing Systems_ , 34:27916–27928, 2021.
  * Wang et al. (2020) Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor.  Tackling the objective inconsistency problem in heterogeneous federated optimization.  _Advances in neural information processing systems_ , 33:7611–7623, 2020.
  * Wang et al. (2018) Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros.  Dataset distillation.  _arXiv preprint arXiv:1811.10959_ , 2018.
  * Yang et al. (2023) Mingzhao Yang, Shangchao Su, Bin Li, and Xiangyang Xue.  Exploring one-shot semi-supervised federated learning with a pre-trained diffusion model.  _arXiv preprint arXiv:2305.04063_ , 2023.
  * Yao et al. (2022) Chun-Han Yao, Boqing Gong, Hang Qi, Yin Cui, Yukun Zhu, and Ming-Hsuan Yang.  Federated multi-target domain adaptation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1424–1433, 2022.
  * Yeom et al. (2018) Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha.  Privacy risk in machine learning: analyzing the connection to overfitting.  In _IEEE computer security foundations symposium_ , 2018.
  * Yu et al. (2023) Sixing Yu, J Pablo Muñoz, and Ali Jannesari.  Federated foundation models: Privacy-preserving and collaborative learning for large models.  _arXiv preprint arXiv:2305.11414_ , 2023.
  * Zhang et al. (2023a) Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi.  Forget-me-not: Learning to forget in text-to-image diffusion models.  _arXiv preprint arXiv:2303.17591_ , 2023a.
  * Zhang et al. (2022a) Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu.  Dense: Data-free one-shot federated learning.  _Advances in Neural Information Processing Systems_ , 35:21414–21428, 2022a.
  * Zhang et al. (2022b) Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu.  Federated learning with label distribution skew via logits calibration.  In _International Conference on Machine Learning_ , pp. 26311–26329. PMLR, 2022b.
  * Zhang et al. (2023b) Jie Zhang, Bo Li, Chen Chen, Lingjuan Lyu, Shuang Wu, Shouhong Ding, and Chao Wu.  Delving into the adversarial robustness of federated learning.  _arXiv preprint arXiv:2302.09479_ , 2023b.
  * Zhao et al. (2020) Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen.  idlg: Improved deep leakage from gradients.  _arXiv preprint arXiv:2001.02610_ , 2020.
  * Zhao et al. (2018) Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra.  Federated learning with non-iid data.  _arXiv preprint arXiv:1806.00582_ , 2018.
  * Zhou et al. (2020) Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu.  Distilled one-shot federated learning.  _arXiv preprint arXiv:2009.07999_ , 2020.
  * Zhou et al. (2023) Yongchao Zhou, Hshmat Sahak, and Jimmy Ba.  Training on thin air: Improve image classification with generated data.  _arXiv preprint arXiv:2305.15316_ , 2023.
  * Zhu et al. (2022) Guogang Zhu, Xuefeng Liu, Shaojie Tang, and Jianwei Niu.  Aligning before aggregating: Enabling cross-domain federated learning via consistent feature extraction.  In _2022 IEEE 42nd International Conference on Distributed Computing Systems (ICDCS)_ , pp. 809–819. IEEE, 2022.
  * Zhu et al. (2019) Ligeng Zhu, Zhijian Liu, and Song Han.  Deep leakage from gradients.  _Advances in neural information processing systems_ , 32, 2019.

##  Appendix A Appendix

##### Testing Accuracy on Each Domain.

In this section, we compare the performance of the ”Centralized” and ”Ours
(One-shot)” methods on the DomainNet dataset across different domains during
the training process. The experimental results, as shown in Figure 11,
demonstrate that, except for the ”Quickdraw” domain, Ours (One-shot)
outperforms the Centralized Training in five domains. Particularly, in the
”Painting” domain, our method achieves a significant performance improvement.
We provide further explanation about the performance gap in ”Quickdraw” domain
through visualization in the following section.

Figure 9: Visualization of synthetic images of four ImageNet subsets. Figure
10: Visualization of synthetic images of six domains from DomainNet dataset.
Figure 11: Performance comparison between Ours (One-shot) and the Centralized
Training with increasing number of training epochs on six domains of DomainNet
dataset. Figure 12: Distribution of the classifier’s confidence between
members and non-member samples. A greater disparity in distribution indicates
a higher degree of information leakage from the model regarding its training
set.  Figure 13: Distribution of the classifier’s entropy between members and
non-member samples. A greater disparity in distribution indicates a higher
degree of information leakage from the model regarding its training set.
Figure 14: Distribution of the classifier’s modified entropy between members
and non-member samples. A greater disparity in distribution indicates a higher
degree of information leakage from the model regarding its training set.