# EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling

Mitch Hill
University of Central Florida
<EMAIL_ADDRESS>
&Jonathan Mitchell11footnotemark: 1
University of Califoria, Los Angeles
<EMAIL_ADDRESS>
&Chu Chen
University of Arizona
<EMAIL_ADDRESS>&Yuan Du
University of Central Florida
<EMAIL_ADDRESS>
&Mubarak Shah
University of Central Florida
<EMAIL_ADDRESS>
&Song-Chun Zhu
University of California, Los Angeles
<EMAIL_ADDRESS>
equal contributionscorresponding author

###### Abstract

This work presents strategies to learn an Energy-Based Model (EBM) according
to the desired length of its MCMC sampling trajectories. MCMC trajectories of
different lengths correspond to models with different purposes. Our
experiments cover three different trajectory magnitudes and learning outcomes:
1) shortrun sampling for image generation; 2) midrun sampling for classifier-
agnostic adversarial defense; and 3) longrun sampling for principled modeling
of image probability densities. To achieve these outcomes, we introduce three
novel methods of MCMC initialization for negative samples used in Maximum
Likelihood (ML) learning. With standard network architectures and an unaltered
ML objective, our MCMC initialization methods alone enable significant
performance gains across the three applications that we investigate. Our
results include state-of-the-art FID scores for unnormalized image densities
on the CIFAR-10 and ImageNet datasets; state-of-the-art adversarial defense on
CIFAR-10 among purification methods and the first EBM defense on ImageNet; and
scalable techniques for learning valid probability densities. Code for this
project can be found at https://github.com/point0bar1/ebm-life-cycle.

##  1 Introduction

Generative modeling of complex signals is one of the fundamental challenges of
computational cognition. Powerful generative models could provide the
foundation of the imaginative capabilities of future machine intelligence. One
approach to generative modeling is to posit the existence of a density $q(x)$
of signals $x$ that generates data samples, and to learn $q(x)$ using a
flexible model $p(x;\theta)$, where $\theta$ is a model parameter. This
approach is taken by Energy-Based Models (EBMs) [1], normalizing flows [2],
score-based models [3, 4], and auto-regressive models [5], as well as by
Variational Auto-encoders (VAEs) [6] using a joint model $p(x,z;\theta)$. In
this work we consider modeling $q(x)$ using an EBM density $p(x;\theta)$ for
image signals $x$. See Appendix A.1 for a brief review of Maximum Likelihood
learning with an EBM.

Figure 1: Illustration of the sampling trajectories that we study in this
work. The shortrun samples are initialized from a generator that is trained in
tandem with the EBM because the goal is self-contained synthesis. Midrun and
longrun samples are initiated from a high-quality starting image obtained from
a pre-trained SNGAN, and we study the ability of the EBMs to preserve the
quality of the input image from defense and density estimation points of view.
The plots show the FID score [7] of 5,000 samples across Langevin steps. The
shortrun samples improve on the generator initialization to achieve high-
quality synthesis around 250 steps. The midrun samples achieve reasonably low
FID in a critical range of about 2k steps where defense is achieved. The
longrun sample maintains reasonable synthesis across the entire trajectory,
and much further. The shortrun and midrun samples eventually produce defective
results outside of their tuned window of stability.

The primary goal of many works that study deep generative models is to
generate realistic images. It is well-known that shortrun sampling with an EBM
is an effective method for image generation, but synthesis results still lag
behind GANs [8], score models, and diffusion models [9]. Other directions of
EBM research include learning valid densities [10], combining discriminative
and generative learning via the Joint Energy Model [11] and relatives, and
using an EBM for defense against adversarial attacks [12]. In this work, we
focus on unconditional learning where EBMs are trained exclusively on
unlabeled images. We explore the tasks of image synthesis, adversarial
defense, and density modeling to examine a breadth of capabilities for the
unconditional EBM. Image synthesis is a shared goal across all generative
models, while defense and density modeling are tasks that are especially
suitable for EBMs. See Appendices A.2 and E.2 for a thorough comparison of EBM
and other generative models from the perspective of synthesis, defense, and
density estimation.

Each task is naturally associated with a certain length of MCMC trajectory.
Image synthesis is most effective with shortrun trajectories (about 20 to 200
steps) that can rapidly generate new images. Adversarial defense requires
midrun trajectories (about 1000 to 3000) that can preserve the class features
while sampling removes adversarial signals. Density modeling requires longrun
trajectories (50K steps or more) to ensure proper calibration of probability
mass for the model steady-state. It is known that EBMs have a near universal
tendency to learn a misaligned steady-state focusing on unrealistic images
[10], and there are very few existing solutions to correct this. We
intuitively refer to the spectrum of trajectories as the life cycle of an MCMC
sample, from youth through middle age to maturity. Young samples have the
highest quality visual appearance, middle age samples are useful for securing
classifiers, and mature samples represent grounded knowledge of the data
density (or lack thereof in the widespread case of a misaligned steady-state).
Figure 1 illustrates the sampling paths that we study in this work.

Ideally, a valid density estimator would also be a good synthesizer and a good
defender. However, synthesis quality and density modeling are goals that tend
to be at odds with each other. High-quality synthesis is easier to achieve
using shortrun sampling with a defective density rather than long-run sampling
with a stable density [10]. Unfortunately, models with high-quality shortrun
synthesis lack the stability needed for defense. The EBM defense from [12]
advocates for the use of a valid density approximation to stabilize
trajectories, but this requirement might be too strict given the difficulty of
density estimation. Learning midrun sampling trajectories sufficient for
defense is much more feasible for complex datasets such as ImageNet than full
density estimation, even if longrun samples from the defensive model are not
realistic. Our work is the first to explore this possibility. The overarching
purpose of this paper is to discuss techniques for building sample paths from
shortrun to longrun, with the hope that these techniques will eventually
enable both high quality synthesis and defense to be accomplished with a model
that is also a valid density. For now, the difficulty of valid density
modeling leads us to restrict our focus to separate time scales of the
sampling regime necessary for each task. Interestingly, EBM training naturally
accommodates learning at different trajectory lengths.

Strategies for improving EBM learning beyond the standard framework (e.g. [13,
14, 10]) can broadly be divided into methods that focus on the initialization
of MCMC samples [13, 15, 16] and methods that focus on the ML learning
objective [17]. Some works explore both [18, 19]. Our novel learning methods
focus on MCMC initialization, and we retain the standard ML objective and use
conventional network architectures. We introduce three new MCMC initialization
strategies which are tailored to the three different trajectories lengths we
explore. During training we exclusively use shortrun MCMC to ensure
computational feasibility. Learning models with midrun and longrun
trajectories is accomplished by simulating longer trajectories via well-chosen
initialization and optimizer annealing. Our initialization strategies are able
to significantly improve the state-of-the-art across the tasks we investigate.
We summarize our contributions below.

  * •

We propose a hybridization of persistent [20] and cooperative [13]
initialization to overcome limitations of each. The proposed method yields
state-of-the-art FID scores for unconditional unnormalized image densities for
the CIFAR-10 and ImageNet datasets. The ImagetNet results surpass GANs trained
with similar resources. See Section 2.

  * •

We show that persistent initialization with appropriately tuned rejuvenation
from in-distribution states can be used to train EBMs with stable trajectories
of several thousand MCMC steps. This allows us to extend the method of [12] to
obtain state-of-the-art purification-based defense for CIFAR-10 and to scale
the EBM defense to ImageNet. See Section 3.

  * •

We propose a method for principled density estimation that allows
incorporation of a rejuvenation step for persistent states. Incorporating
rejuvenation allows us to learn well-formed EBM densities at a greater scale
than previously possible. See Section 4.

Our initialization methods will primarily build upon persistent [20, 14] and
cooperative [13] initialization. All of our methods will use a generator
network as the source of rejuvenation for persistent states. Our shortrun
experiments will learn the generator in tandem with the EBM so the synthesis
process is self-contained, while our midrun and longrun experiments will use
pretrained generators since we will apply sampling paths from in-distribution
initial images rather than synthesizing from scratch.

##  2 Hybrid Persistent Cooperative Learning for Image Synthesis

In this section, we focus on on the conventional task of learning an EBM for
high quality synthesis with shortrun sampling. We restrict our attention to
achieving high quality synthesis without use of pretrained models, in contrast
with works such as [21, 22]. Our method involves hybridizing persistent [20]
and cooperative [13] initialization. We first briefly cover the strengths and
weaknesses of these methods, then discuss our hybridization, and finally
present experimental results.

###  2.1 Motivation: Limitations of Persistent and Cooperative Initialization

A common framework for learning synthesis with an EBM is to use persistent
initialization for MCMC samples with a certain rate of rejuvenation (e.g. 5%
chance) from a noise distribution [14]. This approach can cause instability
because shortrun samples used to update the model include a mix of higher-
energy burn-in samples and lower-energy realistic images, which can
destabilize training by increasing the variance of the gradient of the
negative samples in (4). On the other hand, removing rejuvenation can decrease
the quality of the learned images because persistent images often become stuck
in local modes and develop defects that linger for many updates. Persistent
banks also scale poorly to large datasets such as ImageNet because the bank
cannot efficiently represent the diversity of the data. Initialization from a
cooperative generator network is an appealing alternative because it could
enable efficient in-distribution generation of highly diverse appearances.
Using a generator for rejuvenation allows samples to begin much closer to the
correct energy spectrum, thereby avoiding the instability of noise
rejuvenation. However, we observe a major limitation of cooperative learning
that affects the generator ability to produce diverse initial states. In
particular, a generator trained with the cooperative learning objective has
difficulty breaking the symmetry of its activations. This happens because
shortrun EBM samples are unable to provide novel diversity if the generator
initialization already lacks diversity. This is illustrated in Figure 5.
Learning quickly becomes unstable without auxiliary techniques such as batch
normalization to break generator symmetry. The lack of diversity of shortrun
EBM samples limits the results of cooperative learning even when training
succeeds.

Figure 2: Cooperative-persistent initialization uses paired latent and image
states that a drawn from persistent banks to learn the EBM and generator.

The strengths and limitations of these methods are complementary. Persistent
initialization can reliably provide a diverse set of initial images that
represent prior samples of several different model snapshots. Even if a single
EBM update is biased to a certain image defect (e.g. MCMC samples are blue-
tinged or too bright), samples across previous iterations have the correct
diversity on average. On the other hand, the generator can quickly propose in-
distribution states for rejuvenation instead of relying on out-of-distribution
noise samples or aggressively augmented persistent samples. Furthermore,
training the generator in tandem with the EBM provides an efficient way of
generating new samples from scratch after training that match the samples used
during training. In contrast, models trained with persistent initialization
often cannot produce good samples after training without an in-distribution
initialization [10]. Learning a generator and EBM jointly enables self-
contained and reproducible FID implementation to evaluate the learned model.

###  2.2 Hybrid Persistent Cooperative Initialization

Table 1: Comparison of FID scores among representative generative models. (*=EBM, $\dagger$=conditional) | CIFAR-10 $32\times 32$
---
Model | FID
Ours* | 22.1
Improved CD EBM [19]* | 25.1
VERA [23]* | 27.5
Cooperative EBM[13]* | 33.6
Multigrid EBM [18]* | 37.3
JEM [11]* | 38.4
Persistent EBM [14]* | 40.6
f-EBM (cond.) [24]*$\dagger$ | 30.9
Persistent EBM (cond.)[14]*$\dagger$ | 37.9
DDPM [9] | 3.2
NCSNv2[4] | 10.9
BigGAN [25] | 14.7
SNGAN [26] | 21.7
|  | Celeb-A $64\times 64$
---
Model | FID
Ours* | 19.7
SNGAN [26] | 5.7
NCSNv2[4] | 10.2
Divergence Triangle [28]* | 31.9
| ImageNet $128\times 128$
---
Model | FID
Ours* | 42.3
Self-Supervised GAN [29] | 43.9
InfoMax GAN [30] | 58.9
SNGAN [26] | 65.7
SNGAN (cond.) [26]$\dagger$ | 27.6
Persistent EBM (cond.) [14]*$\dagger$ | 43.7

We visualize our proposed initialization technique in Figure 2. The method
features two banks of persistent states. One bank is for persistent images and
the other bank is for persistent latent vectors. There will always be a one-
to-one pairing between persistent images and persistent latents, meaning that
paired states will be drawn from the banks, return to the banks, and
experience rejuvenation at the same time. All samples are rejuvenated from the
current generator. Other than the pairing and generator rejuvenation source,
sampling from banks and rejuvenation with a fixed probability is conducted in
the same way as persistent initialization. This pairing is necessary for the
cooperative learning loss used to train the generator. Given a batch of paired
samples $\\{Z_{i}\\}_{i=1}^{n}$ and $\\{X_{i,0}^{-}\\}_{i=1}^{n}$ drawn from
the latent and image bank respectively and a generator network $g(z;\phi)$, we
learn $\phi$ using gradient descent on the reconstruction loss

$\displaystyle\mathcal{L}(\phi)\propto\sum_{i=1}^{n}\|g(Z_{i};\phi)-T(X_{i,0}^{-};\theta)\|_{2}^{2}$
(1)

where $T(X_{i,0}^{-};\theta)=X_{i}^{-}$ represents a shortrun Langevin
trajectory with the EBM $p(x;\theta)$. This loss can be derived in the ML
framework as a way to teach the output of $g(z;\phi)$ to match the
distribution of $p(x;\theta)$. A full implementation of cooperative learning
requires an additional sampling process on $Z_{i}$, but we use a straight-
through estimator [27] and approximate latent sampling with the identity
function as originally done in the official MATLAB implementation from [13].
We review further details of the cooperative learning loss in Appendix B.1.

Intuitively, the loss (1) encourages the output of the generator to match the
outcome of the Langevin sampling process (5). Cooperative learning always uses
$X_{i,0}=g(Z_{i};\phi)$, which corresponds to rejuvenating with probability
$1$ in our method. This is the root of the limitation of cooperative learning.
If $g(z;\phi)$ has little diversity, $T(g(z;\phi);\theta)$ will also have
little diversity. This causes extreme oscillation in both the EBM and
generator output as $p(x;\theta)$ attempts to cover the modes of $q(x)$ with
shortrun samples from low diversity initialization. Despite oscillation in
appearance across $\phi$, generator samples tend to remain nearly identical
for any fixed $\phi$, thereby perpetuating the instability and preventing
further learning. By drawing samples from the image bank, we are effectively
choosing $X_{i,0}=T^{\prime}(g(z;\phi^{\prime}))$ where $\phi^{\prime}$
represents a past generator parameter and $T^{\prime}(x)$ represents the
composition of Langevin sampling with past EBMs. This allows us to learn
$\phi$ using appropriately diverse initializations spanning samples of several
past models. We observe that this simple adjustment has a dramatic effect for
improving EBM learning with generator initialization.

###  2.3 Experiments: Image Synthesis with Shortrun MCMC

We present the results of our new learning process applied to the CIFAR-10,
Celeb-A, and ImageNet benchmark datasets, using the image sizes $32\times 32$,
$64\times 64$, and $128\times 128$ respectively. A sketch of the hybrid
learning algorithm is given in Appendix B.2. Besides standard deep learning
techniques for stable optimization, our framework is fully described by the ML
objective and MCMC initialization above. We use the SNGAN [26] architectures
for all models, where EBMs use the SNGAN discriminator with no normalization.
The generator has batch normalization for the CIFAR-10 and Celeb-A experiments
only. Table 1 displays the FID scores achieved by our model in comparison with
prior methods.

When calculating FID scores, samples are first initialized from the generator
then updated with the EBM using a number of Langevin updates tuned to provide
optimal synthesis quality. As mentioned before, an appealing aspect of EBM
learning with generator initialization is the ease of generating new images
from scratch, in contrast with persistent initialization. We use 50K samples
with the official FID code from [7] to calculate all FID scores. In
particular, our scores and framework are consistent with the GAN replication
library from [31]. We publicly release the checkpoints, learning code, and FID
code for each model. The checkpoints are representative of what is achievable
within an ordinary run of the code provided. Our CIFAR-10 results show a
significant improvement over prior EBM synthesis and contribute to closing the
gap between EBMs and other generative models. Surprisingly, our ImageNet
results surpass the results of GANs such as SNGAN [26] and SSGAN [29] on
unconditional synthesis using a similar magnitude of computational resources.
In Figure 5, we demonstrate cooperative and persistent initialization cannot
effectively scale to ImageNet, and that our initialization method provides a
dramatic improvement in synthesis quality.

##  3 Midrun Samplers for Adversarial Defense

This section presents a method for learning EBMs that are capable of
preserving the appearance of an in-distribution initial state across several
thousand MCMC steps. Such models are useful for the purpose of adversarial
defense. Our defense framework is based on the approach in [12], which uses an
EBM to defend an independent naturally trained classifier. Appendix C.1
briefly reviews the EBM defense and compares this approach with other defense
methods. We then present our proposed method for learning a defensive EBM (3),
which is based on persistent initialization using a fixed pretrained generator
as a source of rejuvenation. Finally, we apply our defense to achieve state-
of-the-art performance for purification-based defense on CIFAR-10 and
ImageNet.

Figure 3: Persistent initialization. Positive samples are from data and
negative samples are MCMC samples initialized from a batch from the image
bank. Some states are randomly rejuvenated when returning to the bank.

One limitation of prior EBM defense is the reliance on persistent
initialization with no rejuvenation to learn the defensive model. This is done
to ensure that defensive sampling trajectories remain stable for arbitrary
numbers of steps. However, removing rejuvenation from persistent learning has
drawbacks discussed in Section 2.1. In particular, it becomes very difficult
to learn meaningful EBMs for large and complex datasets such as ImageNet in
this framework because the persistent bank cannot represent the diversity of
the dataset and the quality of persistent images without rejuvenation quickly
degrades. Methods for efficient learning of defensive EBMs at a greater scale
are needed to extend the EBM defense to more realistic situations.

We overcome this obstacle by building on the observation that fully stable
sampling paths are not required for successful defense. While the defense from
[12] uses EBMs with stable sampling for 100K steps or more, the defense
results require less than 2000 steps. A natural question is whether it is
possible to learn stable MCMC trajectories for only a predefined midrun range
to achieve the defensive benefits without fully stabilizing samples over
longrun trajectories. Defining a learning procedure to obtain such models is
the goal of this section. Efficiently learning EBMs with stable midrun
trajectories allows us to scale up the EBM defense to significantly more
challenging domains.

###  3.1 In-Distribution Rejuvenation for Learning a Defensive EBM

The initialization method for our midrun sampler is similar to standard
persistent initialization with the adjustment that persistent states are
rejuvenated from a frozen pretrained generator rather than noise. We use a
generator in our experiments so that the EBM and generator could be used after
training to sample from scratch. This choice is not necessary and rejuvenation
from data samples, or another efficient in-distribution initialization, is
also effective for learning defensive EBMs. We note that MCMC initialization
from a trained generator or from data samples is not explored in recent work
because the goal of most current EBM learning is image synthesis, which
becomes trivial if EBM trajectories are always initialized from high-quality
samples. From the perspective of synthesis our learning process is nearly
invisible but from the perspective of defense its utility becomes concrete.

Learning an EBM for defense involves tuning the length of the sampling
trajectory via the number of shortrun training steps and the rejuvenation
rate, and tuning the annealing schedule of the EBM optimizer. Given a desired
number of MCMC steps $K_{\text{def}}$ for a defensive update and a shortrun
trajectory $K\approx 100$, we simply set the rejuvenation rate to
$p_{\text{rejuv}}=K/K_{\text{def}}$ to ensure that on average samples will
travel $K_{\text{def}}$ steps before rejuvenation. While in practice we use
$K_{\text{def}}=2000$ and $p_{\text{rejuv}}=0.05$, we have found that this
method can yield stable paths for at least $K_{\text{def}}=50$K MCMC steps
when $p_{\text{rejuv}}$ is low.

Initialization alone is insufficient to stabilize MCMC pathways when model
weights are changing quickly. Using a low learning rate late in training is a
key aspect of stabilizing sampling paths [10, 12]. Intuitively, if the EBM
optimizer has a sufficiently low learning rate then MCMC trajectories in the
persistent image bank can function as approximate trajectories from the
current model, since weights change very little as the persistent states are
updated. By annealing in tandem with our initialization, we are effectively
using midrun trajectories of length $K_{\text{def}}$ initialized from the
generator to update the EBM while we are actually using shortrun trajectories
of length $K$ from the persistent bank. Annealing is a crucial component for
stabilizing both midrun and longrun trajectories. Without annealing, sampling
paths are not able to maintain realism for large $K_{\text{def}}$.

###  3.2 Experiments: Defending Natural Classifiers with an EBM

We train our EBMs using the persistent initialization described above in
tandem with a pretrained generator on both CIFAR-10 and ImageNet. We use the
same SNGAN models as before for our CIFAR-10 experiments, with the exception
that the generator is pretrained instead of learned. For our ImageNet
experiments, we use the BigGAN [25] discriminator architecture for our EBM
modified for input size $224\times 224$ and a pretrained BigGAN Generator. Our
naturally trained classifier $f(x)$ is a pretrained WideResNet 28-10 [32] for
CIFAR-10 and a pretrained EfficientNetB-7 architecture [33] for ImageNet.

Table 2: _Left:_ Defense vs. whitebox attacks with $l_{\infty}$ perturbation $\varepsilon=8/255$ for CIFAR-10. _Right:_ Defense vs. $l_{\infty}$ whitebox attacks for ImageNet. Above the line classifier $f(x)$ uses natural images during training, below the line $f(x)$ uses adversarial training. For attack: *=PGD, ${\dagger}$=BPDA, ${\ddagger}$=BPDA+EOT. | Defense | Nat. | Adv.
---|---|---
Ours${\ddagger}$ | 0.79 | 0.567
EBM Defense [12]${\ddagger}$ | 0.8412 | 0.5490
PixelDefend [34]${\dagger}$ | 0.95 | 0.09
MALADE [35]* | – | 0.0048
ME-Net [36]${\ddagger}$ | 0.94 | 0.15
Semi-Supervised AT [37]$*$ | 0.897 | 0.625
TRADES [38]$*$ | 0.849 | 0.5643
Free AT [39]$*$ | 0.859 | 0.4633
AT [40]$*$ | 0.873 | 0.458
| Defense | $\varepsilon$ | Nat. | Adv.
---|---|---|---
Ours | $\frac{2}{255}$ | 0.684 | 0.418
Fast AT [41] | $\frac{2}{255}$ | 0.609 | 0.4339
Free AT [39] | $\frac{2}{255}$ | 0.644 | 0.4339
LLR AT [42] | $\frac{4}{255}$ | 0.822 | 0.427
Smooth AT [43] | $\frac{4}{255}$ | 0.822 | 0.586

We evaluate our models using the attack gradient (8) and Algorithm 2. For
CIFAR-10 we use $K_{\text{def}}=1500$ Langevin steps with $l_{\infty}$
adversarial parameters $\varepsilon=\frac{8}{255}$ and $\alpha=\frac{2}{255}$,
where $\varepsilon$ is the size of the $l_{\infty}$ ball and $\alpha$ is the
gradient step size. For ImageNet we used $K_{\text{def}}=200$ Langevin steps
for defense with $l_{\infty}$ adversarial parameters
$\varepsilon=\frac{2}{255}$ and $\alpha=\frac{1}{255}$. We attack ImageNet for
50 attacks steps across 10K val samples. For CIFAR-10 we perform the same
number of attacks across 5k validation samples. The results are shown in Table
2.

On CIFAR-10, we surpass the robustness of the existing EBM defense using a
much more reliable learning framework. The importance of midrun learning is
clearly demonstrated by our successful application of EBM defense to ImageNet
at the resolution $224\times 224$. The robustness of a naturally trained
classifier secured by our EBM is comparable with adversarial training. While
the ImageNet results for EBM defense are not yet on par with state-of-the-art
adversarially trained models, our experiments are an important proof of
concept that the method can be scaled. See Appendix C.2 for diagnostics and
further discussion.

##  4 Longrun Sampling for Density Estimation

Our final objective is to introduce scalable tools for learning a valid image
density with an EBM. The work [10] revealed that, in the absence of careful
implementation, EBM learning always results in an unexpected outcome where
steady-state samples from the learned density $p(x;\theta)$ have an
oversaturated and unrealistic appearance that differs drastically from
shortrun samples used during learning. This outcome affects all EBMs that are
not specifically trained to overcome this defect, as well as related models
such as normalizing flows, score-based models, and diffusion models (see
Appendix E.2). To our knowledge, across different generative models, the only
way to learn a well-formed potential energy surface approximating a complex
density $q(x)$ that is compatible with efficient MCMC sampling is using an
EBM.

Despite the theoretical formulation of the EBM as a potential surface, the
problem of principled density estimation has received relatively little
attention. Successful image synthesis is sometimes used as misleading evidence
of successful density estimation, but we emphasize these outcomes not
equivalent. In this section, we address the lack scalable methods to learn
valid densities of complex signals by introducing an MCMC initialization can
incorporate rejuvenation while still simulating extremely long trajectories.
Learning a valid density is a fundamental computational problem that is
important in its own right, and we further hope that our learning method leads
to EBM clustering techniques based on the potential energy basins of a well-
formed density [10].

###  4.1 Incorporating Rejuvenation in Density Estimation

Prior work has suggested that persistent learning is the most effective method
for learning a valid EBM density. Furthermore, works that learn a valid
density have avoided rejuvenation because the incorporation of newly
rejuvenated samples into the persistent bank ensures that EBM updates will
always include samples that are not at the steady-state. However, persistent
learning without rejuvenation has shortcomings mentioned in Section 3. We
present hypothesized conditions for learning a valid density that motivate the
design of our MCMC initialization:

  * •

After a certain point in training, all samples used to update the EBM must be
approximate steady-state samples of the current model $p(x;\theta)$.

  * •

Persistent samples that are newly rejuvenated (up to about 50K Langevin steps
since rejuvenation, and possibly many more) cannot be approximate steady-state
samples for any known rejuvenation sources, including data, generators, and
noise.

  * •

Persistent samples that have undergone sufficiently many _lifetime_ Langevin
updates for a model whose weights are changing very slowly can be approximate
steady-state samples.

Figure 4: Visualization of our longrun initialization procedure. Newly
rejuvenated samples must remain the the burnin bank until they have approach
the model steady-state, at which point they move to the update bank to be used
for model gradients.

Both the second and third point are corroborated by prior work [10, 12] as
well as our own observations. The third point means that persistent states
updated with shortrun Langevin can eventually act as longrun Langevin samples
if the optimizer learning rate is small, because the EBM samples in previous
timesteps are essentially samples from the current EBM.

Learning a valid density that includes rejuvenation while satisfying the
conditions above requires separating the newly rejuvenated samples from
samples that are used to update the EBM. This leads us to introduce two
persistent image banks: one for newly rejuvenated samples, and one for samples
that will be used to update the EBM. Samples in the newly rejuvenated bank
that have been updated sufficiently many times will eventually replace samples
from the bank used to update the EBM, at which point newly rejuvenated states
will be added to the burn-in bank. Figure 4 shows a visualization of the MCMC
initialization method. As in Section 3, we will use a pre-trained generator to
efficiently obtain high-quality rejuvenated samples so that the generation
process is fully synthetic, although data samples could be used as well. Our
goal is to preserve the sample quality for an arbitrary number of MCMC steps.
We note that this is a sufficient but not necessary condition for learning a
valid EBM density. Nonetheless, stable sampling is an important step towards
rigorous probabilistic EBMs.

Table 3: FID for 5K samples after 100K Langevin and 1M Langevin steps. FID remains stable over long trajectories. Data | 100K | 1M
---|---|---
CIFAR-10 | 49.2 | 51.7
Celeb-A | 37.4 | 45.9
ImageNet | 82.3 | 77.8

Even with our improved initialization, we find that extremely longrun
trajectories of 1 million or more MCMC steps still tend to oversaturate,
although to a much lesser degree. To further stabilize the appearance of
extremely longrun Langevin samples, we include prior energy terms in the
model. Our longrun EBMs have the form

$U_{\theta_{0},\sigma}(x;\theta)=U^{\prime}(x;\theta)+U^{\prime}(x;\theta_{0})+\frac{1}{2\sigma^{2}}\|x\|_{2}^{2}$
(2)

where $U^{\prime}(x;\theta)$ is the model whose weights are updated,
$U_{0}(x;\theta_{0})$ is a prior EBM with fixed weights $\theta_{0}$ and
$\sigma$ is a parameter controlling the strength of a Gaussian prior. We used
a prior EBM in a shortrun manner. The role of the prior EBM is to provide some
stability but also to provide a tendency to oversaturate at longer
trajectories so that the current EBM learns to correct oversaturation. The
Gaussian prior is meant to discourage unbounded activations outside of the
image hypercube. Further discussion is in Appendix E.1. We find that including
both of these terms significantly improves the ability to learn quality
synthesis over long trajectories.

###  4.2 Experiments: EBM Density Estimation

In these experiments, we learn EBM models that have stable MCMC trajectories
for an arbitrary number of steps. We apply the longrun learning method in
Algorithm 4 to CIFAR-10, Celeb-A, and ImageNet at the resolutions $32\times
32$, $64\times 64$, and $64\times 64$ respectively. Pre-trained SNGANs are
used to rejuvenate images for CIFAR-10 and Celeb-A, while a resized images
from a pre-trained BigGAN are used to rejuvenate ImageNet samples. We require
that samples have been updated for 50K to 75K Langevin steps since the last
rejuvenation before states are moved from the burn-in bank to the update bank.
Our EBMs all use the discriminator architecture of the SNGAN model without
spectral normalization. We use pretrained midrun samples for our prior EBM
terms in the learned energy (2). See Appendix E.4 for an algorithm sketch.
Figure 1 shows the evolution of FID score over Langevin steps starting from
the generator network for our CIFAR-10 model. Table 3 shows the FID scores of
5K samples for each dataset. Visualizations of longrun samples with 100K steps
and extremely longrun samples using 1 million Langevin steps are shown in
Appendix G.

See Appendix E.3 for a discussion of our evaluation in comparison to log
likelihood. In particular, we emphasize that log likelihood of test samples
provides no information about the steady-state distribution of a model [44],
and that models with high log likelihood exhibit the same misalignment as
shortrun and midrun EBMs. We advocate for the use of longrun MCMC samples as a
necessary but not sufficient proof of steady-state alignment, and we encourage
the field to move away from viewing log likelihood as an indicator of accurate
steady-state modeling.

##  5 Related Work

Energy-Based Models. Early forms of EBMs include the exponential family
distribution, the FRAME model [45] and Restricted Boltzmann Machines [46].
Recent work has introduced the EBM with a ConvNet potential [1, 14]. This
dramatically increased the learning capacity of the model which led to many
follow-up works on image synthesis [15, 47, 16], adversarial robustness [12],
and joint learning of discriminative and generative models [11]. Several works
investigate training and EBM in tandem with an auxilary model. [48] train an
EBM and generator and tandem without MCMC by using samples from the generator
as direct approximations of the EBM density and training the generator using a
variational objective. A similar approach is explored by [23]. Cooperative
learning [13] trains the EBM and generator by using the generator to
initialize samples needed to train the EBM and uses reconstruction loss
between generator and EBM samples to learn the generator. [18] learn an EBM
using Noise Contrastive Estimation with an auxiliary flow model. [49] use a
pretrained VAE to facilitate EBM learning. Our work builds on cooperative
learning by identifying and resolving symmetry breaking problems in early
training, leading to state-of-the-art EBM synthesis for unconditional
ImageNet. Despite the formulation of the EBM as an unnormalized density, it
has been shown that most EBMs have strong misaligned steady-state
distributions [10]. Our work introduces new methods to learn a model with
correct steady-state alignment.

Adversarial Robustness. Adversarial Training (AT) [40], which trains a
classifier using PGD-generated adversaries, is the most popular and studied
adversarial defense. Many variations and improvements have been introduced,
including optimizing the training loop by recycling gradients of past
adversaries [39], combining single step FGSM with random initialization to
achieve similar robustness [41], learning with auxiliary unlabeled data [37],
local linearization [42], and the use of smooth activation functions [43]. An
alternative approach to adversarial training involves the use of preprocessing
transformations. Randomized smoothing [50] and related methods [51] add noise
to the input signal to remove adversarial signals. Many other preprocessing
defenses have been proposed [52, 34, 36], but nearly all of these methods can
be broken by adaptive attacks that are aware of the preprocessing method [53].
A notable exception is the EBM defense [12], which uses midrun MCMC
trajectories to purify images. We ease the restriction of learning EBMs with
stability for arbitrary MCMC runs in the EBM defense by introducing a midrun
sampler that enables faster learning of defensive EBMs and allows the EBM
defense to scale to more complex datasets.

##  6 Conclusion and Future Work

We have described three unique MCMC initializations for EBM using different
sampling trajectories: shortrun for synthesis, midrun for defense, and long-
run for density estimation. Furthermore, we have elaborated on different MCMC
initialization strategies used to stabilize these models for different
sampling lengths. We have demonstrated the flexibility of these mechanisms by
using similar architectures, data, and training platforms to create different
EBMs for different applications. We hope that future research incorporates
these new training initialization schemes to improve their generative models
for a wide variety of tasks.

##  7 Acknowledgements

This research was supported with Cloud TPUs from Google’s TPU Research Cloud
(TRC).

## References

  * [1] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu.  A theory of generative convnet.  In Proceedings of the 33rd International Conference on Machine Learning, pages 2635–2644, 2016.
  * [2] Diederik P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  arXiv preprint arXiv:1807.03039, 2018.
  * [3] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  arXiv preprint arXiv:1907.05600, 2019.
  * [4] Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models, 2020.
  * [5] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu.  Conditional image generation with pixelcnn decoders.  arXiv preprint arXiv:1606.05328, 2016.
  * [6] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [7] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems, volume 33, page 6629–6640, 2017.
  * [8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [9] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models, 2020.
  * [10] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu.  On the anatomy of MCMC-based maximum likelihood learning of energy-based models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020.
  * [11] Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky.  Your classifier is secretly an energy based model and you should treat it like one.  In International Conference on Learning Representations, 2020.
  * [12] Mitch Hill, Jonathan Craig Mitchell, and Song-Chun Zhu.  Stochastic security: Adversarial defense using long-run dynamics of energy-based models.  In International Conference on Learning Representations, 2021.
  * [13] Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu.  Cooperative learning of energy-based model and latent variable model via mcmc teaching.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [14] Yilun Du and Igor Mordatch.  Implicit generation and modeling with energy based models.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [15] Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu.  Learning generative convnets via multi-grid modeling and sampling.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9155–9164, 2018.
  * [16] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.  Learning non-convergent non-persistent short-run MCMC toward energy-based model.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [17] Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon.  Training deep energy-based models with f-divergence minimization.  In International Conference on Machine Learning, pages 10957–10967. PMLR, 2020.
  * [18] Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu.  Flow contrastive estimation of energy-based models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7518–7528, 2020.
  * [19] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch.  Improved contrastive divergence training of energy based models.  arXiv preprint arXiv:2012.01316, 2020.
  * [20] Tijmen Tieleman.  Training restricted boltzmann machines using approximations to the likelihood gradient.  In Proceedings of the 25th international conference on Machine learning, pages 1064–1071, 2008.
  * [21] Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio.  Your gan is secretly an energy-based model and you should use discriminator driven latent sampling.  arXiv preprint arXiv:2003.06060, 2020.
  * [22] Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli.  Are labels required for improving adversarial robustness?  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [23] Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud.  No {mcmc} for me: Amortized sampling for fast and stable training of energy-based models.  In International Conference on Learning Representations, 2021.
  * [24] Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon.  Training deep energy-based models with f-divergence minimization, 2020\.
  * [25] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In International Conference on Learning Representations, 2019.
  * [26] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In International Conference on Learning Representations, 2018.
  * [27] Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  arXiv preprint arXiv:1308.3432, 2013.
  * [28] Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.  Divergence triangle for joint training of generator model, energy-based model, and inferential model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8670–8679, 2019.
  * [29] Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby.  Self-supervised gans via auxiliary rotation loss.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12154–12163, 2019.
  * [30] Kwot Sin Lee, Ngoc-Trung Tran, and Ngai-Man Cheung.  Infomax-gan: Improved adversarial image generation via information maximization and contrastive learning.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3942–3952, 2021.
  * [31] Kwot Sin Lee and Christopher Town.  Mimicry: Towards the reproducibility of gan research.  arXiv preprint arXiv:2005.02494, 2020.
  * [32] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  arXiv preprint arXiv:1605.07146, 2016.
  * [33] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
  * [34] Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman.  Pixeldefend: Leveraging generative models to understand and defend against adversarial examples.  In International Conference on Learning Representations, 2018.
  * [35] Vignesh Srinivasan, Arturo Marban, Klaus-Robert Muller, Wojciech Samek, and Shinichi Nakajima.  Defense against adversarial attacks by langevin dynamics.  arxiv preprint arXiv:1805.12017, 2019.
  * [36] Yuzhe Yang, Guo Zhang, Dina Katabi, and Zhi Xu.  Me-net: Towards effective adversarial robustness with matrix estimation.  In Proceedings of the 36th International Conference on Machine Learning, pages 7025–7034, 2019.
  * [37] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang.  Unlabeled data improves adversarial robustness.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [38] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically principled trade-off between robustness and accuracy.  In Proceedings of the 36th International Conference on Machine Learning, pages 7472–7482, 2019.
  * [39] Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein.  Adversarial training for free!, 2019.
  * [40] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In International Conference on Learning Representations, 2018.
  * [41] Eric Wong, Leslie Rice, and J. Zico Kolter.  Fast is better than free: Revisiting adversarial training.  In International Conference on Learning Representations, 2020.
  * [42] Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli.  Adversarial robustness through local linearization, 2019.
  * [43] Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoc V. Le.  Smooth adversarial training, 2021.
  * [44] Lucas Theis, Aäron van den Oord, and Matthias Bethge.  A note on the evaluation of generative models.  CoRR, abs/1511.01844, 2016.
  * [45] Song-Chun Zhu, Ying Nian Wu, and David Mumford.  Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling.  International Journal of Computer Vision, 27(2):107–126, 1998.
  * [46] Geoffrey E. Hinton.  Training products of experts by minimizing contrastive divergence.  Neural Computation, 14(8):1771–1800, 2002.
  * [47] Kwonjoon Lee, Weijian Xu, Fan Fan, and Zhuowen Tu.  Wasserstein introspective neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3702–3711, 2018.
  * [48] Taesup Kim and Yoshua Bengio.  Deep directed generative models with energy-based probability estimation, 2016.
  * [49] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat.  {VAEBM}: A symbiosis between variational autoencoders and energy-based models.  In International Conference on Learning Representations, 2021.
  * [50] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter.  Certified adversarial robustness via randomized smoothing.  In Proceedings of the 36th International Conference on Machine Learning, pages 1310–1320, 2019.
  * [51] Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and J. Zico Kolter.  Denoised smoothing: A provable defense for pretrained classifiers.  In Advances in Neural Information Processing Systems, volume 33, pages 21945–21957, 2020.
  * [52] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten.  Countering adversarial images using input transformations.  In International Conference on Learning Representations, 2018.
  * [53] Anish Athalye, Nicholas Carlini, and David Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  In Proceedings of the 35th International Conference on Machine Learning, pages 274–283, 2018.
  * [54] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry.  On adaptive attacks to adversarial example defenses.  In Advances in Neural Information Processing Systems, volume 33, pages 1633–1645, 2020.
  * [55] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma.  Learning energy-based models by diffusion recovery likelihood.  arXiv preprint arXiv:2012.08125, 2020.
  * [56] Kyungmin Lee and Seyoon Oh.  Efficient randomized smoothing by denoising with learned score function, 2021.
  * [57] Jongmin Yoon, Sung Ju Hwang, and Juho Lee.  Adversarial purification with score-based generative models.  arXiv preprint arXiv:2106.06041, 2021.
  * [58] Aurelien Decelle, Cyril Furtlehner, and Beatriz Seoane.  Equilibrium and non-equilibrium regimes in the learning of restricted boltzmann machines.  In Advances in Neural Information Processing Systems, 2021.

##  Appendix A Background on Energy-Based Models

###  A.1 Review of EBM Learning and MCMC Initialization

We briefly review the main components of EBM learning following the standard
method derived from works such as [46, 45, 1]. Throughout our experiments, we
will use this framework and focus on exploring different possibilities for
MCMC initialization. A deep EBM has the form

$\displaystyle p(x;\theta)=\frac{1}{Z(\theta)}\exp\\{-U(x;\theta)\\}$ (3)

where $U(x;\theta)$ is a ConvNet with weights $\theta$ and $Z(\theta)$ is the
intractable normalizing constant. Maximum Likelihood (ML) learning uses the
objective $\operatorname*{arg\,min}_{\theta}D_{KL}(q(x)||p(x;\theta))$, which
can be minimized using the stochastic gradient

$\displaystyle\nabla\mathcal{L}(\theta)\approx\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}U(X_{i}^{+};\theta)-\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}U(X_{i}^{-};\theta)$
(4)

where the positive samples $\\{X_{i}^{+}\\}_{i=1}^{n}$ are a set of data
samples and the negative samples $\\{X_{i}^{-}\\}_{i=1}^{n}$ are samples from
the current model $p(x;\theta)$. To obtain the negative samples for a deep
EBM, it is common to use MCMC sampling with the $K$ steps of Langevin equation

$\displaystyle
X^{(k+1)}=X^{(k)}-\frac{\eta^{2}}{2}\nabla_{X^{(k)}}U(X^{(k)};\theta)+\eta
Z_{k},$ (5)

where $\eta$ is the step size and $Z_{k}\sim N(0,I)$. The Langevin
trajectories are initialized from a set of states
$\\{X^{-}_{i,0}\\}_{i=1}^{n}$ obtained from a certain initialization strategy.
The method of MCMC initialization determines many of the properties of the
learned model.

There are a number of different methods of MCMC initialization for EBM
learning. Early Restricted Boltzmann Machines use data samples as the initial
states in every iteration [46]. Persistent initialization [45, 20] maintains a
bank of MCMC samples from previous model updates as the source of MCMC states
in the current iteration. Persistent states can be rejuvenated with a certain
probability instead of returning to the image bank. Potential sources for
rejuvenation include noise samples [14], data augmentation applied to
persistent states [19], data samples, or samples from a generator network.
Cooperative learning initializes MCMC states from a generator learned in
tandem with an EBM [13]. The Multi-grid model [15] initializes samples from
lower resolution images. Noise initialization is explored by [16]. The VAEBM
[49] initializes samples from a pretrained VAE generator. EBMs can learn
stable shortrun synthesis from virtually any initialization distribution as
long as the initialization distribution remains fairly consistent across
training updates. EBM sampling can produce realistic images after training
when the testing initialization closely matches the training initialization,
while synthesis can fail for test initializations unseen during training.
[10].

###  A.2 Comparison of EBM and Other Generative Models

Normalizing flows, EBMs, and score models are related because they gradually
relax the degree to which the data density $q(x)$ is modeled. Normalizing
flows model the entire normalized density, EBMs model the unnormalized
density, and score-based models model only the density gradients. The
synthesis quality is typically the highest for score matching models that only
try to learn distribution gradients and typically lower for models like
normalizing flows and auto-regressive objectives that model the full
normalized density. EBM synthesis usually lies somewhere in between. Score-
matching does not require MCMC sampling during learning and as a consequence
learning tends to be more straightforward than learning for EBMs. Yet we
believe that MCMC sampling is not yet fully optimized for EBM learning due to
the subtleties of EBM learning. We hope that EBMs can match or surpass score-
matching models with greater understanding of the learning process. Further,
improving synthesis quality with EBMs is useful for improving the defense and
density modeling outcomes which still require good synthesis to be successful.
Our shortrun initialization method is an effort in this direction.

GANs and diffusion models differ from EBMs in the sense that the former are
explicitly formulated from the perspective of synthesis while the latter are
formulated for density modeling and synthesis is a byproduct. GANs learn to
transform a latent signal to a realistic signal by synthesizing images to fool
a discriminator, and diffusion models define a ladder of intermediate
distributions between a noise distribution and realistic samples. From this
perspective, it is perhaps unsurprising that generative methods such as GANs
and diffusion models surpass EBMs in terms of synthesis, since that the model
formulations are geared directly towards generation.

Nonetheless, we believe that EBMs have unique properties that distinguish them
from other generative models. In particular, EBM learning offers a control of
the MCMC sampling paths from a learned model that $p(x;\theta)$ is, to our
knowledge, is unobtainable under other current frameworks. Unlike GAN models
which map a trivial distribution to a complex signal distribution without
actually describing the signal distribution, EBMs directly model the complex
signal distribution via a potential energy surface. VAE and diffusion models
cannot be efficiently marginalized. Normalizing flows and score-based models
exhibit the same oversaturation behavior as miscalibrated EBMs when sampling
at a constant noise level for many steps (see Appendix E.2). One cannot adjust
MCMC stability during learning as we do for EBMs, because sampling is not used
when training a normalizing flow or score models. Auto-regressive models
depend on a certain sequence of components for generation, and are not
compatible with joint updates of all pixels. On the other hand, MCMC sampling
with an EBM is a natural and relatively efficient process. Uniquely among all
methods, the desired stability and length of MCMC sampling trajectories for an
EBM can easily be controlled by adjusting the sampling phase of Maximum
Likelihood learning. This can yield capabilities not obtainable by other kinds
of unsupervised models.

##  Appendix B Image Synthesis with EBMs

###  B.1 Cooperative Learning for Generator Network

CIFAR-10 |  |  | ImageNet
---|---|---|---
|  |  |
Figure 5: _Left:_ Comparison of cooperative learning [13] and our hybrid
cooperative-persistent learning using appearance of shortrun samples after 500
updates of the EBM using a generator with and without batch norm. The
cooperative models have difficulty achieving diversity in the shortrun samples
because the EBM is unable to significantly change the appearance of initial
generator images. In fact, we find cooperative learning is not possible at all
without generator batch normalizatin. Our hybrid initialization increases the
diversity of shortrun samples by including persistent samples from previous
EBM updates. This accelerates learning and dramatically improves synthesis
results. _Right:_ Cooperative and persistent initialization are unable to
provide high-quality results on unconditional ImageNet. Persistent
initialization also cannot generate diverse samples when initialized from the
noise rejuvenation distribution during test-time generation.

In this section we briefly review the Maximum Likelihood framework for
cooperatively learning the generator in tandem with the EBM [13]. The
generator model assumes a joint distribution $(X,Z)$ of images $X$ and latent
signals $Z$ given by

$Z\sim\text{N}(0,I)\quad\text{and}\quad X|Z\sim\text{N}(g(Z;\phi),\tau^{2}I)$

for a generator network $g$ with weights $\phi$ and a Gaussian parameter
$\tau$ controls the spread of $X$ around $g(Z;\phi)$. Given i.i.d. pairs
$\\{X_{i},Z_{i}\\}_{i=1}^{n}$, the observations have a joint negative log
probability

$-\log
p(\\{X_{i},Z_{i}\\}_{i=1}^{n};\phi)=\sum_{i=1}^{n}\left[\|X_{i}-g(Z_{i};\phi)\|_{2}^{2}/(2\tau^{2})+\frac{1}{2}\|Z_{i}\|_{2}^{2}\right]+C.$

In practice, the $X_{i}$ are shortrun MCMC samples from an EBM and the $Z_{i}$
are unobserved. The full formulation of cooperative learning involves latent
MCMC using the conditional distribution $p(Z|X_{i};\phi)$ to infer each
$Z_{i}$. In practice, since the $X_{i}$ are shortrun samples that are paired
with $Z_{i}$ through the process of rejuvenation, one can roughly approximate
the latent MCMC process with an identity function and obtain a gradient (1).
We find that this works as well or better than including a latent sampling
step on $Z$, and it is consistent with the original implementation from [13].

Figure 5 (_left_) illustrates the importance of including a persistent image
bank in the cooperative learning framework. Shortrun samples from the
generator in standard cooperative learning have low diversity and it is
difficult for the generator to break the symmetry of its activations early in
training. This can lead to catastrophic instability without auxiliary
techniques like batch norm to help break the symmetry. Incorporating
persistent states from prior EBM updates when updating the generator helps the
shortrun distribution achieve a health diversity that enables stable and
effective learning.

Figure 5 (_right_) also illustrates the dramatic improvement in scalability
that is obtained over cooperative and persistent learning when our method is
applied to unconditional ImageNet at resolution $128\times 128$. We find that
cooperative learning is not able to scale well. Persistent learning yields
better results but the quality is still poor. The persistent model also
suffers from the inability to draw diverse samples from the noise rejuvenation
distribution during test time. A significant advantage of using a generator
network for MCMC initialization as opposed to Gaussian noise is that the test-
time and training-time generation processes match.

###  B.2 Cooperative-Persistent Algorithm for Shortrun Learning

Algorithm 1 Cooperative-Persistent Hybrid Learning

d

0: Natural images $\\{x^{+}_{m}\\}_{m=1}^{M}$, EBM $U(x;\theta)$, generator
$g(z;\phi)$ Langevin noise $\eta$, number of shortrun steps $K$, EBM optimizer
$h_{U}$, generator optimizer $h_{g}$, initial weights $\theta_{0}$ and
$\phi_{0}$, rejuvenation probability $p$, max update rounds $w$, number of
training iterations $T$.

0: Learned weights $\theta_{T}$ and $\phi_{T}$.

Initialize bank of random latent states $\\{Z_{i}\\}_{i=1}^{N}$.

Initialize image bank $\\{X^{-}_{i}\\}_{i=1}^{N}$ from generator using
$X_{i}^{-}=g(Z_{i};\phi_{0})$

for $1\leq t\leq T$ do

Select batch $\\{\tilde{X}^{+}_{b}\\}_{b=1}^{B}$ from data samples
$\\{x^{+}_{m}\\}_{m=1}^{M}$.

Get paired batches $\\{\tilde{Z}_{b}\\}_{b=1}^{B}$ and
$\\{\tilde{X}^{-}_{b,0}\\}_{b=1}^{B}$ from $\\{Z_{i}\\}_{i=1}^{N}$ and
$\\{X^{-}_{i}\\}_{i=1}^{N}$.

Update $\\{\tilde{X}^{-}_{b,0}\\}_{b=1}^{B}$ with $K$ Langevin steps (5) to
obtain negative samples $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$.

Get learning gradient $\Delta^{(t)}_{U}$ using (4) with samples
$\\{\tilde{X}^{+}_{b}\\}_{b=1}^{B}$ and $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$.

Update $\theta_{t}$ using gradient $\Delta^{(t)}_{U}$ and optimizer $h_{U}$.

Get learning gradient $\Delta^{(t)}_{g}$ using (1) with samples
$\\{\tilde{Z}_{b}\\}_{b=1}^{B}$ and $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$.

Update $\phi_{t}$ using gradient $\Delta^{(t)}_{g}$ and optimizer $h_{g}$.

Rejuvenate each $\tilde{Z}_{b}$ from latent distribution with probability $p$.

Also rejuvenate states $\tilde{Z}_{b}$ for which $\tilde{X}_{b}$ has been
updated more than $w$ times.

If $\tilde{Z}_{b}$ is rejuvenated, rejuvenate
$\tilde{X}_{b}=g(\tilde{Z}_{b};\phi_{t})$.

Return $\\{\tilde{Z_{b}}\\}_{b=1}^{B}$ to $\\{Z_{i}\\}_{i=1}^{N}$ and
$\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$ to $\\{X^{-}_{i}\\}_{i=1}^{N}$ by
overwriting previous states.

end for

##  Appendix C EBM Defense

###  C.1 Background on EBM Defense

The most popular method for adversarial defense is adversarial training (AT)
[40, 41, 39] which aims to train a classifier to correctly predict adversarial
samples within a small ball around a natural input. Another popular defense
method is randomized smoothing [50], which adds Gaussian noise to images to
remove adversarial signals before classification. Both of these approaches
modify classifier training. Although many methods have been proposed to secure
a naturally trained classifier, most have been broken by stronger attacks. A
recent method that has been shown to secure a natural classifier is Langevin
sampling with an EBM [12].

The EBM defense uses a classifier trained with labeled natural images and an
EBM trained with unlabelled natural images. The two networks are trained
independently, which is a key advantage of EBM defense over adversarial
training and randomized smoothing. Since the EBM is independent of the
classifier, EBM defense has the potential to secure many classifiers across
diverse tasks with a single defensive model, while existing methods are
typically tailored to a single method. Starting with a naturally classifier
trained on natural images $f(x)$, we define its robust counterpart as

$\displaystyle F(x)=E_{T(x)}[f(T(x))]$ (6)

where $T(x)$ is a random variable representing $K$ steps of the Langevin
transformation (5) initialized from a state $x$. We cannot evaluate $F(x)$
directly so we approximate it using

$\displaystyle\hat{F}_{H}(x)=\frac{1}{H}\sum_{h=1}^{H}f(\hat{x}_{h})\quad\text{where}\quad\hat{x}_{h}\sim
T(x)\text{ i.i.d.},$ (7)

where $f(.)$ is a forward pass of our classifier to return logits and where
the accuracy of approximation is driven by the number of replicates $H$.
Meaningful evaluation of adversarial defenses must be based on adaptive attack
methods which are aware of both $f(x)$ and $T(x)$. For our attack we use the
BPDA+EOT formulation from [53, 54] to obtain the attack gradient

$\Delta_{\text{BPDA+EOT}}(x,y)=\frac{1}{H_{\text{adv}}}\sum_{h=1}^{H_{\text{adv}}}\nabla_{\hat{x}_{h}}L\left(\frac{1}{H_{\text{adv}}}\sum_{h=1}^{H_{\text{adv}}}f(\hat{x}_{h}),y\right),\quad\hat{x}_{h}\sim
T(x)\text{ i.i.d.}$ (8)

which we use in the standard PGD framework to generate adversarial examples.
Algorithm 2 gives a sketch of the defense evaluation.

Algorithm 2 EBM Defense Algorithm

d

0: Natural images $\\{x^{+}_{m}\\}_{m=1}^{M}$, EBM $U(x;\theta)$, classifier
$f$, Langevin noise $\eta$, attack replicates $H_{\text{adv}}$, defense
replicates $H_{\text{def}}$, $l_{\infty}$ radius $\varepsilon$, attack step
size, $\alpha$, Langevin steps $K$

0: Defense record $\\{D_{m}\\}_{m=1}^{M}$ for each image initialized as ones.

for $1\leq i\leq M$ do

$\text{select}(X_{i},y_{i})\text{from batch}$

Randomly initialize adversary $\hat{X}_{0}$ inside $L_{\infty}$ ball around
$X_{i}$

for $1\leq j\leq N$ do

$c_{j}=\operatorname*{arg\,max}_{\ell}[\hat{F}_{H_{\text{adv}}}(\hat{X}_{j-1})]_{\ell}$

$\Delta_{j}=\Delta_{\text{BPDA+EOT}}(\hat{X}_{j-1},y_{i})$

if $c_{j}\neq y_{i}$ then

$c_{j}^{\prime}=\operatorname*{arg\,max}_{\ell}[\hat{F}_{H_{\text{def}}}(\hat{X}_{j-1})]_{\ell}$

if $c_{j}^{\prime}\neq y_{i}$ then

$D_{i}=0$

end if

end if

$\hat{X}_{j}=\text{PGD}(\hat{X}_{j-1},\Delta_{j},\varepsilon,\alpha)$

end for

end for

###  C.2 EBM Defense Experiment Details and Diagnostics

Algorithm 3 Training a midrun sampler for EBM defense

0: Natural images $\\{x^{+}_{m}\\}_{m=1}^{M}$, EBM $U(x;\theta)$, frozen pre-
trained generator $g(z)$, Langevin noise $\eta$, Langevin steps $K$, EBM
optimizer $h_{U}$, initial weights $\theta_{0}$, rejuvenation probability $p$,
number of training iterations $T$.

0: Weights $\theta_{T}$ for defensive EBM.

Initialize image bank $\\{X^{-}_{i}\\}_{i=1}^{N}$ from generator using
$X_{i}^{-}=g(Z)$

for $1\leq t\leq T$ do

Select batch $\\{\tilde{X}^{+}_{b}\\}_{b=1}^{B}$ from data samples
$\\{x^{+}_{m}\\}_{m=1}^{M}$.

Get negative sample batch $\\{\tilde{X}^{-}_{b,0}\\}_{b=1}^{B}$ from
$\\{X^{-}_{i}\\}_{i=1}^{N}$.

Update $\\{\tilde{X}^{-}_{b,0}\\}_{b=1}^{B}$ with $K$ Langevin steps (5) to
obtain negative samples $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$.

Get learning gradient $\Delta^{(t)}_{U}$ using (4) with samples
$\\{\tilde{X}^{+}_{b}\\}_{b=1}^{B}$ and $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$.

Update $\theta_{t}$ using gradient $\Delta^{(t)}_{U}$ and optimizer $h_{U}$.

Rejuvenate each $\tilde{X}_{b}$ from a pretrained generator $g$ with
probability $p$.

Return $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$ to $\\{X^{-}_{i}\\}_{i=1}^{N}$ by
overwriting previous states.

end for

Table 4: Defense for $l_{\infty}$ against high-power whitebox attacks on ImageNet. Dataset | Nat | Adv | $H_{\text{adv}}$ | $H_{\text{def}}$ | samples
---|---|---|---|---|---
ImageNet | 0.683 | 0.38 | 32 | 64 | 1600

To verify the integrity of our results we ran an attack with heavily increased
resources for ImageNet compared to our standard evaluation. While using these
resources for all attacks is infeasible in practice, we want to ensure our
defense maintains robustness as attacker resources increase. As shown in Table
4, our benchmark accuracy remains consistent when we increase the number of
attack steps (from $50$ to $200$) and EOT attack replicates
($H_{\text{adv}}$).

We demonstrate results over varying numbers of langevin steps $K$. We can see
in Fig 6 that our sampling trajectory for defending imagenet at $K=200$ is
reasonable to achieve high natural image classification as well as robustness.

Figure 6: Accuracy over varying numbers of langevin steps $K$ for ImageNet
experiments.

##  Appendix D Importance of Learning Rate Annealing

This section demonstrates the importance of learning rate annealing for
learning a robust energy landscape. We repeat the midrun and longrun learning
experiments for CIFAR-10 except that we never anneal the learning rate. We
then sample with the models for 1500 steps for the model trained with the
midrun method and 100K steps for the model trained with the longrun method.
The results in Figure 7 show that learning rate annealing is essential for
stabilizing both midrun and longrun trajectories.

|  |  |
---|---|---|---
Figure 7: Ablation study showing the importance of annealing. _Left:_ Samples
from a non-annealed model trained with the midrun method after 1500 MCMC
steps. _Right:_ Samples from a non-annealed model trained with the longrun
method ater 100K MCMC steps. MCMC samples were initialized from data. This
shows that rejuvenation of the midrun trajectories from data and the
separation of longrun samples into burn-in and update banks alone is not
enough. Annealing ensures that samples from past EBMs function as approximate
samples from the current EBM, since the weights are changing very slowly.

The importance of annealing can be understood as follows. If the EBM is being
updated with a very low learning rate, then samples from recent EBM snapshots
can function as samples from the current EBM. In the case of midrun
trajectory, annealing allows the model to robustify trajectories that are
approximately as long as the lifetime of a persistent sample between
rejuvenation. In the case of longrun learning, annealing allows the burnin
samples to approximately reach the model steady-state before they are included
in the update bank. This allows the persistent samples in the update bank to
function as approximate steady-state samples from the current EBM, leading to
proper modeling of probability mass.

##  Appendix E Longrun Learning and Density Estimation

###  E.1 Prior EBM for Longrun Learning

This section discusses the choice of training longrun EBMs by using a fixed
shortrun EBM as a prior distribution. Intuitively, the prior EBM learns a
reasonable but imperfect approximation of the energy landscape that we know
will have energy basins which leak to low-energy states. Using a fixed
misaligned prior EBM allows the EBM which is being actively updated to focus
on learning landscape features which were not correctly learned by the prior
EBM. In particular, our longrun allows the actively updated EBM to focus on
sealing the leaky energy basins of the prior EBM so that the full model learn
the correct distribution of probability mass. By separating the burn-in and
update banks and longrun learning, we guarantee that the prior EBM alone would
lead to extreme oversaturation by the time a state reaches the update bank
unless the actively updated EBM is counteracting the oversaturation tendency
to preserve realism. This additional effect of directly driving states towards
the oversaturated regions in the absence of a correcting force leads to
improved stability for longrun models learned with a prior EBM compared to
models learned without a prior EBM. The technique has precedent in ResNets. In
both cases, it is easier to learn an approximate direct model first and then
to approximate the residual that remains.

###  E.2 Misaligned Steady-State of Score-Based Models, Normalizing Flows, and
Diffusion Models

To underscore our claims about the difficulty of calibrating the probability
mass of a density model, we investigate longrun Langevin samples from
normalizing flows, score models, and diffusion model. We find that the
normalizing flow from the GLOW model [2], noise-contrastive score matching [4]
and the recovery likelihood diffusion model [55] have misaligned steady-states
as well. This shows that the problem of improper density estimation extends
well beyond the EBM. In particular, the GLOW model, which is learned using
end-to-end backpropagation of a fully normalized likelihood, does not learn a
valid steady-state. Surprisingly, log likelihood provides no meaningful
information about the steady-state of the model, as described in Appendix E.3.
We believe that the calibration of the model steady-state is currently best
diagnosed with longrun MCMC sampling because the distribution of longrun MCMC
samples represents the probability mass of the model. We hope that the
observations in our work can lead to efforts to stabilize the sampling
trajectories of many existing models.

|  |  |
---|---|---|---
Figure 8: _Left:_ MCMC samples after 100K steps using a GLOW model [2] trained
on CIFAR-10. _Right:_ MCMC samples after 100K steps using a conditional
recovery likelihood model [55] trained on CIFAR-10. MCMC samples were
initialized from data samples. Neither model can correctly approximate the
distribution of probability mass for the data density. The problem of steady-
state misalignment extends beyond EBMs to many other generative density
models. We tried several different temperatures close to 1 for the GLOW model
and found equivalent results.

Figure 8 (_left_) displays initial and final states from a GLOW model density
after 100K sampling steps. Despite the fact that the GLOW model has a fully
tractable density, it is unable to learn a valid distribution of probability
mass. Figure 8 (_right_) shows initial and final samples from the Recovery
Likelihood T6 model after 100K steps of the conditional model at the lowest
noise value. We observe the same oversaturation for the conditional density as
for the unconditional density of a standard EBM. Code for the T1K model that
the authors evaluate in their longrun experiments is not released so we have
not yet been able to directly test their results. The T1K model is equivalent
to an EBM version of the score model in Figure 9 which we have shown has a
misaligned steady-state. Further, we note the longrun experiments with the T1K
model are very misleading because the experiments use 100 steps with 1000
distinct conditional models and claim this is a longrun evaluation of 100K
steps. The correct evaluation is to use 100K steps on a single conditional
model. We strongly believe that the Recovery Likelihood model as originally
presented has a misaligned steady-state like many other methods.

The score-based model from [4] and its annealed Langevin dynamics process has
recently been used for purifying adversarial signals [56, 57]. One approach is
add noise and the using the score model to denoise [56]. The robustness of
this method is upper-bounded by standard randomized smoothing [50]. Another
approach is to initiate the Langevin process of the score model from a natural
image as done in the EBM defense. A score model can be used in a langevin
process $T(x)$ that is a direct analogue to the EBM langevin process in (5)
and (8). Given a score model $S_{\theta}(x)$ for a low noise value $\sigma$,
one can define the Langevin equation

$X_{t}={X_{t-1}}\frac{\eta^{2}}{2}S_{\theta}(X_{t-1})+\eta Z_{t}$

where $S_{\theta}(x)\approx\nabla_{x}\log q(x)$ since $\sigma$ is low.

We experimented with this process as a defense mechanism by selecting the
smallest trained noise value $\sigma=0.01$ and using the Langevin process as a
method to purify adversaries. We evaluated this method using our BPDA+EOT
attack framework over different numbers of Langevin steps during purification.
In contrast to reports from [57], we were unable to obtain any significant
defense using a pretrained score model when initializing sampling directly
from adversarial or natural images. In Figure 9, one can see that the score-
based model drives natural images toward saturation quickly, leading to a
sharp decrease in natural classification that undermines the possibility of
robustness from sampling. The misaligned steady-state of the score-based
models prevents it from being an a defensive transformation because natural
accuracy drops before robustness kicks in. Since the score-based model does
not perform sampling during training, and one cannot adjust the stability of
its sampling process as we do in this work. While it is not immediately clear
how to overcome this problem, we believe that defense with a score model is
possible and we hope that our observation lead to efforts to stabilize the
sampling paths of score models as we do for EBMs in this work.

Figure 9: Score-based Langevin experiment on the CIFAR-10 dataset. Left:
Accuracy of natural and adversarial images resulting from a BPDA+EOT defense
using a score-based model with an annealed langevin purification method for
125 samples over varying steps. Right: Samples received from this annealed
langevin diffusion process over the same sampling lengths.

###  E.3 Regarding Density Estimation and Log Likelihood

In this work, we are primarily interested in learning density models that
assign the majority of probability mass in realistic regions of the image
space. This goal is, surprisingly, distinct from the goal of likelihood
maximization. In particular, one can achieve high likelihood with mixture
models where only an infinitesimal portion of mass is assigned to a mixture
component that approximates the true density, while the majority of
probability mass is assigned to a degenerate distribution (see [44], “Great
Log Likelihood but Poor Samples"). The steady-state of MCMC sampling with such
a mixture distribution would concentrate on the degenerate distribution even
though the structures of the true density exist in high-energy regions.
Therefore, log likelihood cannot detect if a model has assigned probability
mass in realistic region of the image space. Recent observations have shown
that this situation is not just hypothetical [10]. Non-convergent EBMs are
practical examples, since these models consist of a mixture of partially
formed high-energy energy basins enabling effective shortrun synthesis in
realistic regions of the image space and much lower-energy basins in
unrealistic image regions that dominate the probability mass. The same
misaligned landscape structure extends beyond EBM to RBMs [58], score models,
normalizing flows, and diffusion models (see Appendix E.2). Log likelihood can
only indicate the presence of energy landscape features that are similar to
the ground truth landscape, but it cannot detect whether these features will
leak into lower-energy basins that represent the true mass distribution. Like
shortrun sample quality, high log likelihood is often misleading false
evidence that a model density concentrates on realistic images.

We believe that calculating FID using longrun samples from the EBM and data
samples is an appropriate, if rough, measure of successful density modeling.
We also believe that the field should move away from viewing log likelihood as
a principled measure of successful density estimation, since the theoretical
shortcomings described by [44] are extremely common but unrecognized in
practice across many probabilistic models, not just EBMs. Realism of longrun
MCMC samples is a necessary but not sufficient condition for valid density
modeling. We argue that investigating the visual quality of longrun samples is
currently the most effective tool to study steady-state distributions, which
are the primary object of interest for a probabilistic model. Though we use
the FID metric as part of our evaluation, our density modeling experiments are
best described as realism preservation experiments rather than image
generation experiments.

###  E.4 Longrun Learning Algorithm

Algorithm 4 Training a longrun sampler for density estimation

0: Natural images $\\{x^{+}_{m}\\}_{m=1}^{M}$, EBM $U(x;\theta)$, frozen pre-
trained generator $g(z)$ Langevin noise $\eta$, Langevin steps $K$, EBM
optimizer $h_{U}$, initial weights $\theta_{0}$, update threshold $D$, number
of training iterations $T$.

0: Weights $\theta_{T}$ for EBM with stable longrun samples.

Initialize burn-in image bank $\\{X^{*}_{i}\\}_{i=1}^{N_{1}}$ and update image
bank $\\{X^{+}_{i}\\}_{i=1}^{N_{2}}$ from generator using $X_{i}^{-}=g(Z)$.

Initialize update counts $\\{d_{i}\\}_{i=1}^{N_{1}}$ using
$d_{i}\sim\text{Unif}(\\{0,\dots,D\\})$.

for $1\leq t\leq T$ do

Select batch $\\{\tilde{X}^{+}_{b}\\}_{b=1}^{B}$ from data samples
$\\{x^{+}_{m}\\}_{m=1}^{M}$.

Get initial sample batch $\\{\tilde{X}^{*}_{b,0}\\}_{b=1}^{B}$ from
$\\{X^{*}_{i}\\}_{i=1}^{N_{1}}$ and $\\{\tilde{X}^{-}_{b,0}\\}_{b=1}^{B}$ from
$\\{X^{-}_{i}\\}_{i=1}^{N_{2}}$.

Get counts $\tilde{d}_{b}$ corresponding to samples
$\\{\tilde{X}^{*}_{b,0}\\}_{b=1}^{B}$.

Update $\\{\tilde{X}^{-}_{b,0}\\}_{b=1}^{B}$ with $K$ Langevin steps (5) to
obtain negative samples $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$.

Update $\\{\tilde{X}^{*}_{b,0}\\}_{b=1}^{B}$ with $K$ Langevin steps (5) to
obtain updated burn-in samples $\\{\tilde{X}^{*}_{b}\\}_{b=1}^{B}$.

Get learning gradient $\Delta^{(t)}_{U}$ using (4) with samples
$\\{\tilde{X}^{+}_{b}\\}_{b=1}^{B}$ and $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$.

Update $\theta_{t}$ using gradient $\Delta^{(t)}_{U}$ and optimizer $h_{U}$.

Update the burn-in count $\tilde{d}_{b}\leftarrow\tilde{d}_{b}+1$.

for $1\leq b\leq B$ do

if $\tilde{d}_{b}\geq D$ then

Randomly overwrite one $\tilde{X}_{b}^{-}$ using $\tilde{X}^{*}_{b}$.

Rejuvenate $\tilde{X}_{b}$ from generator and set $\tilde{d}_{b}$ to 0.

end if

end for

Return $\\{\tilde{X}^{*}_{b,0}\\}_{b=1}^{B}$ from
$\\{X^{*}_{i}\\}_{i=1}^{N_{1}}$ and $\\{\tilde{X}^{-}_{b}\\}_{b=1}^{B}$ to
$\\{X^{-}_{i}\\}_{i=1}^{N_{2}}$ by overwriting previous states.

end for

##  Appendix F Shortrun Synthesis Results

Figure 10: Shortrun samples from CIFAR-10 EBM at resolution $32\times 32$.

Figure 11: Shortrun samples from Celeb-A EBM at resolution $64\times 64$.

Figure 12: Shortrun samples from ImageNet EBM at resolution $128\times 128$.

##  Appendix G Longrun Synthesis Results

Figure 13: Longrun samples at 100,000 steps and extremely longrun samples at 1
million steps for EBMs trained on three datasets. Our initialization is able
to preserve a high degree of realism over the first 100K steps, and a
reasonable degree of realism over very long trajectories. While oversaturation
and distortion is noticeable for some samples using 1M steps, many samples
have reasonable appearance and there is high diversity. Our method makes
significant progress towards aligning longrun samples with high-quality
samples from training to ensure that the model is a valid density.

##  Appendix H Tables of Experiment Parameters

This section gives a full list of experiment parameters. Some notable minor
additions beyond what are discussed in the text are a fixed MCMC temperature
parameter to regulate gradient strength at the beginning of training, a
gradient clipping parameter to restrict the magnitude of EBM and generator
weight updates. We only use gradient clipping for shortrun experiments.

We use the annealing schedule

$\gamma_{\text{anneal}}=[(10^{-4},0),(10^{-5},50000),(10^{-6},75000),(10^{-7},100000),(10^{-8},125000)]$

in all experiments that involve annealing, where the pairs denote a learning
rate and model update step at which that learning rate is first used. We
denote this schedule as $\gamma_{\text{anneal}}$ in the tables.

Shortrun Training
---
Dataset | Celeb-A | CIFAR-10 | ImageNet
Training Steps | 150000 | 100000 | 300000
Data Epsilon | 1e-2 | 1e-2 | 1e-2
EBM LR | 1e-4 | 1e-4 | 1e-4
EBM Optimizer | Adam | Adam | Adam
EBM Gradient Clip | 0 | 0 | 50
Langevin Epsilon | 3e-3 | 5e-3 | 3e-3
MCMC Steps | 75 | 100 | 50
Rejuvenation Probability | 0.5 | 0.5 | 0.5
MCMC Temperature | 1e-6 | 1e-4 | 1e-7
Max Update Rounds | 2 | 2 | 2
Persistent Bank Size | 10000 | 10000 | 10000
Generator LR | 1e-4 | 1e-4 | 5e-5
Generator Optimizer | Adam | Adam | Adam
Generator Gradient Clip | 0 | 0 | 50
Generator Batch Norm | No | Yes | No
Shortrun Evaluation
---
Dataset | Celeb-A | CIFAR-10 | ImageNet
Langevin Epsilon | 3e-3 | 5e-3 | 3e-3
MCMC Steps | 300 | 350 | 320
MCMC Temperature | 1e-6 | 1e-4 | 1e-7

Defense Training
---
Dataset | CIFAR-10 | ImageNet
Training Steps | 150000 | 150000
Data Epsilon | 2e-2 | 3e-2
EBM LR | $\gamma_{\text{anneal}}$ | $\gamma_{\text{anneal}}$
EBM Optimizer | Adam | Adam
Langevin Epsilon | 1e-2 | 2e-2
MCMC Steps | 100 | 100
Rejuvenation Probability | 0.025 | 0.05
MCMC Temperature | 1e-4 | 1e-5
Persistent Bank Size | 20000 | 10000
Defense Evaluation
---
Dataset | CIFAR-10 | ImageNet
Adversarial Steps | 50 | 50
Adversarial Epsilon | $\frac{8}{255}$ | $\frac{2}{255}$
Adversarial Eta | $\frac{2}{255}$ | $\frac{1}{255}$
EOT Attack Reps | 48 | 16
EOT Defense Reps | 128 | 64
Langevin Steps | 2000 | 200
Langevin Epsilon | 1e-2 | 2e-2
MCMC Temperature | 1e-4 | 1e-5
Longrun Training
---
Dataset | Celeb-A | CIFAR-10 | ImageNet
Training Steps | 250000 | 250000 | 250000
Data Epsilon | 2e-2 | 2e-2 | 2e-2
EBM LR | $\gamma_{\text{anneal}}$ | $\gamma_{\text{anneal}}$ | $\gamma_{\text{anneal}}$
EBM Optimizer | Adam | Adam | Adam
Langevin Epsilon | 1e-2 | 1e-2 | 1e-2
MCMC Steps | 100 | 100 | 100
Burn-in Update Rounds | 750 | 750 | 750
MCMC Steps Burn-in | 100 | 100 | 100
MCMC Temperature | 1e-4 | 1e-4 | 1e-5
Tau | 1.5e-1 | 1.5e-1 | 1.5e-1
Persistent Bank Size | 15000 | 10000 | 15000
Burn-in Bank Size | 1000 | 1000 | 1000
Prior EBM
---
Dataset | Celeb-A | CIFAR-10 | ImageNet
Training Steps | 150000 | 150000 | 150000
Data Epsilon | 2e-2 | 2e-2 | 1.5e-2
EBM LR | 1e-4 | 1e-4 | 1e-4
EBM Optimizer | Adam | Adam | Adam
Langevin Epsilon | 1e-2 | 1e-2 | 1e-2
MCMC Steps | 100 | 50 | 100
Rejuvenation Probability | 0.2 | 0.2 | 0.2
MCMC Temperature | 1e-4 | 1e-4 | 1e-5
Tau | 1.5e-1 | 1.5e-1 | 1.5e-1
Persistent Bank Size | 10000 | 10000 | 10000