# One-Line-of-Code Data Mollification Improves Optimization of Likelihood-
based Generative Models

Ba-Hien Tran
Department of Data Science
EURECOM, France
<EMAIL_ADDRESS>
&Giulio Franzese
Department of Data Science
EURECOM, France
<EMAIL_ADDRESS>
Pietro Michiardi
Department of Data Science
EURECOM, France
<EMAIL_ADDRESS>
&Maurizio Filippone
Department of Data Science
EURECOM, France
<EMAIL_ADDRESS>

###### Abstract

Generative Models (gms) have attracted considerable attention due to their
tremendous success in various domains, such as computer vision where they are
capable to generate impressive realistic-looking images. Likelihood-based gms
are attractive due to the possibility to generate new data by a single model
evaluation. However, they typically achieve lower sample quality compared to
state-of-the-art score-based Diffusion Models (dms). This paper provides a
significant step in the direction of addressing this limitation. The idea is
to borrow one of the strengths of score-based dms, which is the ability to
perform accurate density estimation in low-density regions and to address
manifold overfitting by means of data mollification. We connect data
mollification through the addition of Gaussian noise to Gaussian homotopy,
which is a well-known technique to improve optimization. Data mollification
can be implemented by adding one line of code in the optimization loop, and we
demonstrate that this provides a boost in generation quality of likelihood-
based gms, without computational overheads. We report results on image data
sets with popular likelihood-based gms, including variants of variational
autoencoders and normalizing flows, showing large improvements in fid score.

##  1 Introduction

Generative Models (gms) have attracted considerable attention recently due to
their tremendous success in various domains, such as computer vision, graph
generation, physics and reinforcement learning [see e.g., 35, 39, and
references therein]. Given a set of data points, gms attempt to characterize
the distribution of such data so that it is then possible to draw new samples
from this. Popular approaches include Variational Autoencoders (vaes),
Normalizing Flows (nfs), Generative Adversarial Networks (gans), and score-
based Diffusion Models (dms).

In general, the goal of any gms is similar to that of density estimation with
the additional aim to do so by constructing a parametric mapping between an
easy-to-sample-from distribution $p_{\mathrm{s}}$ and the desired data
distribution $p_{\text{data}}$. While different gms approaches greatly differ
in their optimization strategy and formulation, the underlying objectives
share some similarity due to their relation to the optimal transport problem,
defined as
$\arg\min_{\pi}\int\norm{{{\bm{\mathbf{x}}}}-{{\bm{\mathbf{y}}}}}^{2}d\pi({{\bm{\mathbf{x}}}},{{\bm{\mathbf{y}}}})$.
Here $\pi$ is constrained to belong to the set of joint distributions with
marginals $p_{\mathrm{s}},p_{\text{data}}$, respectively [19, 42]. This
unified perspective is explicitly investigated for gans and vaes [19] for
example, whereas other works study nfs [49]. Similarly, dms can be connected
to Schrodinger Bridges [7], which solve the problem of entropy-regularized
optimal transport [53]. Given that extensions of the regularized optimal
transport case are available also for other generative models [63, 56], we
should expect that, in principle, any technique should allow generation of
samples with similar quality, provided it is properly tuned. However, this is
not true in practice. The different formulations lead to a variety of
properties associated with gms, and pros and cons of each formulation can be
understood through the so-called gm tri-lemma [78]. The three desirable
properties of gms are high sample quality, mode coverage, and fast sampling,
and it has been argued that such goals are difficult to be satisfied
simultaneously [78] .

The state-of-the-art is currently dominated by score-based dms, due to their
ability to achieve high sample quality and good mode coverage. However,
generating new samples is computationally expensive due to the need to
simulate stochastic differential equations. Likelihood-based gms are
complementary, in that they achieve lower sample quality, but sampling
requires one model evaluation per sample and it is therefore extremely fast.
While some attempts have been made to bridge the gap by combining gans with
dms [78] or training gans with diffusions [77], these still require careful
engineering of architectures and training schedules. The observation that all
gms share a common underlying objective indicates that we should look at what
makes dms successful at optimizing their objective. Then, the question we
address in this paper is: can we borrow the strengths of score-based dms to
improve likelihood-based gms, without paying the price of costly sample
generation?

One distinctive element of score-based dms is data mollification, which is
typically achieved by adding Gaussian noise [66] or, in the context of image
data sets, by blurring [59]. A large body of evidence points to the manifold
hypothesis [61], which states that the intrinsic dimensionality of image data
sets is much lower than the dimensionality of their input. Density estimation
in this context is particularly difficult because of the degeneracy of the
likelihood for any density concentrated on the manifold where data lies [43].
Under the manifold hypothesis, or even when the target density is multi-modal,
the Lipschitz constant of gms has to be large, but regularization, which is
necessary for robustness, is antagonist to this objective [62, 9]. As we will
study in detail in this paper, the process of data mollification gracefully
guides the optimization mitigating manifold overfitting and enabling a
desirable tail behavior, yielding accurate density estimation in low-density
regions. In likelihood-based gms, data mollification corresponds to smoothing
the loss landscape. In the optimization literature, mollification (a.k.a.
Gaussian homotopy or graduated optimization) has been proven to accelerate
optimization. The literature provides results for stochastic non-convex
problems [22], which is the setting we consider here. Another recent work in
the same setting gives some continuity results on the optimum under small
increments in the level of mollification [18].

Strictly speaking, data mollification in score-based dms and likelihood-based
gms are slightly different. In the latter, the amount of noise injected in the
data is continuously annealed throughout training. At the beginning, the
equivalent loss landscape seen by the optimizer is much smoother, due to the
heavy perturbation of the data, and a continuous reduction of the noise level
allows optimization to be gracefully guided until the point where the level of
noise is zero [22, 18]. dms, instead, are trained at each step of the
optimization process by considering all noise levels simultaneously, where
complex amortization procedures, such as self-attention [67], allow the model
to efficiently share parameters across different perturbation levels. It is
also worth mentioning that score-based dms possess another distinctive feature
in that they perform gradient-based density estimation [66, 29]. It has been
conjectured that this can be helpful to avoid manifold overfitting by allowing
for the modeling of complex densities while keeping the Lipschitz constant of
score networks low [62]. In this work, we attempt to verify the hypothesis
that data mollification is heavily responsible for the success of score-based
dms. We do so by proposing data mollification for likelihood-based gms, and
provide theoretical arguments and experimental evidence that data
mollification consistently improves their optimization. Crucially, this
strategy yields better sample quality and it is extremely easy to implement,
as it requires adding very little code to any existing optimization loop.

We consider a large set of experiments involving vaes and nfs and some popular
image data sets. These provide a challenging test for likelihood-based gms due
to the large dimensionality of the input space and to the fact that density
estimation needs to deal with data lying on manifolds. The results show
systematic, and in some cases dramatic, improvements in sample quality,
indicating that this is a simple and effective strategy to improve
optimization of likelihood-based gms models. The paper is organized as
follows: in § 2 we illustrate the challenges associated with generative
modeling when data points lie on a manifold, particularly with density
estimation in low-density regions and manifold overfitting; in § 3 we propose
data mollification to address these challenges; § 4 reports the experiments
with a discussion of the limitations and the broader impact, while § 5
presents related works, and § 6 concludes the paper.

##  2 Challenges in Training Deep Generative Models

We are interested in unsupervised learning, and in particular on the task of
density estimation. Given a dataset $\mathcal{D}$ consisting of $N$ i.i.d
samples
$\mathcal{D}\stackrel{{\scriptstyle\Delta}}{{=}}\\{{{\bm{\mathbf{x}}}}_{i}\\}_{i=1}^{N}$
with ${{\bm{\mathbf{x}}}}_{i}\in\mathbb{R}^{D}$, we aim to estimate the
unknown continuous generating distribution
$p_{\text{data}}({{\bm{\mathbf{x}}}})$. In order to do so, we introduce a
model $p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ with parameters
${{\bm{\mathbf{\theta}}}}$ and attempt to estimate ${{\bm{\mathbf{\theta}}}}$
based on the dataset $\mathcal{D}$. A common approach to estimate
${{\bm{\mathbf{\theta}}}}$ is to maximize the likelihood of the data, which is
equivalent to minimizing the following objective:

$\displaystyle\mathcal{L}({{\bm{\mathbf{\theta}}}})$
$\displaystyle\stackrel{{\scriptstyle\Delta}}{{=}}-\mathbb{E}_{p_{\text{data}}({{\bm{\mathbf{x}}}})}\left[\log
p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})\right].$ (1)

There are several approaches to parameterize the generative model
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$. In this work, we focus on
two widely used likelihood-based Generative Models (gms), which are
Normalizing Flows (nfs) [51, 39] and Variational Autoencoders (vaes) [34, 57].
Although nfs and vaes are among the most popular deep gms, they are
characterized by a lower sample quality compared to gans and score-based dms.
In this section, we present two major reasons behind this issue by relying on
the manifold hypothesis.

###  2.1 The Manifold Hypothesis and Density Estimation in Low-Density Regions

The manifold hypothesis is a fundamental concept in manifold learning [61, 70,
1] stating that real-world high-dimensional data tend to lie on a manifold
$\mathcal{M}$ characterized by a much lower dimensionality compared to the one
of the input space (ambient dimensionality) [47]. This has been verified
theoretically and empirically for many applications and datasets [50, 47, 55,
69]. For example, [55] report extensive evidence that natural image datasets
have indeed very low intrinsic dimension relative to the high number of pixels
in the images.

Data distribution

Estimated distribution

Figure 1:  Left: Histogram of samples from data distribution
$p_{\text{data}}({{\bm{\mathbf{x}}}})$ and its true scores
$\nabla_{{{\bm{\mathbf{x}}}}}\log p_{\text{data}}({{\bm{\mathbf{x}}}})$;
Right: Histogram of of samples from the estimated distribution
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ and its scores
$\nabla_{{{\bm{\mathbf{x}}}}}\log
p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$. In the low density
regions, the model is unable to capture the true density and scores.

The manifold hypothesis suggests that density estimation in the input space is
challenging and ill-posed. In particular, data points on the manifold should
be associated with high density, while points outside the manifold should be
considered as lying in regions of nearly zero density [45]. This implies that
the target density in the input space should be characterized by high
Lipschitz constants. The fact that data is scarce in regions of low density
makes it difficult to expect that models can yield accurate density estimation
around the tails. These pose significant challenges for the training of deep
gms [9, 45, 66]. Recently, diffusion models [66, 26, 67] have demonstrated the
ability to mitigate this problem by gradually transforming a Gaussian
distribution, whose support spans the full input space, into the data
distribution. This observation induces us to hypothesize that the data
mollification mechanism in score-based dms is responsible for superior density
estimation in low-density regions.

To demonstrate the challenges associated with accurate estimation in low-
density regions, we consider a toy experiment where we use a real-nvp flow
[14] to model a two-dimensional mixture of Gaussians, which is a difficult
test for nfs in general. Details on this experiment are provided in the
Appendix. Fig. 1 depicts the true and estimated densities, and their
corresponding scores, which are the gradient of the log-density function with
respect to the data [29]. Note that the use of “score” here is slightly
different from that from traditional statistics where score usually refers to
the gradient of the log-likelihood with respect to model parameters. As it can
be seen in the figure, in regions of low data density,
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ is completely unable to
model the true density and scores. This problem is due to the lack of data
samples in these regions and may be more problematic under the manifold
hypothesis and for high-dimensional data such as images. In § 4, we will
demonstrate how it is possible to considerably mitigate this issue by means of
data mollification.

###  2.2 Manifold Overfitting

The manifold hypothesis suggests that overfitting on a manifold can occur when
the model $p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ assigns an
arbitrarily large likelihood in the vicinity of the manifold, even if the
distribution does not accurately capture the true distribution
$p_{\text{data}}({{\bm{\mathbf{x}}}})$ [10, 43]. This issue is illustrated in
Fig. 2 of [43] and it will be highlighted in our experiment (§ 4.1), where the
true data distribution $p_{\text{data}}({{\bm{\mathbf{x}}}})$ is supported on
a one-dimensional curve manifold $\mathcal{M}$ in two-dimensional space
$\mathbb{R}^{2}$. Even when the model distribution
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ poorly approximates
$p_{\text{data}}({{\bm{\mathbf{x}}}})$, it may reach a high likelihood value
by concentrating the density around the correct manifold $\mathcal{M}$. If
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ is flexible enough, any
density defined on $\mathcal{M}$ may achieve infinite likelihood and this
might be an obstacle for retrieving $p_{\text{data}}({{\bm{\mathbf{x}}}})$.

A theoretical formalization of the problem of manifold overfitting appears in
[43] and it is based on the concept of Riemannian measure [54]. The Riemannian
measure on manifolds holds an analogous role to that of the Lebesgue measure
on Euclidean spaces. To begin, we establish the concept of smoothness for a
probability measure on a manifold.

######  Definition 1.

Let $\mathcal{M}$ be a finite-dimensional manifold, and $p$ be a probability
measure on $\mathcal{M}$. Let $\mathfrak{g}$ be a Riemannian metric on
$\mathcal{M}$ and $\mu_{\mathcal{M}}^{(\mathfrak{g})}$ the corresponding
Riemannian measure. We say that $p$ is _smooth_ if
$p\ll\mu_{\mathcal{M}}^{(\mathfrak{g})}$ and it admits a continuous density
$p:\mathcal{M}\rightarrow\mathbb{R}_{>0}$ with respect to
$\mu_{\mathcal{M}}^{(\mathfrak{g})}$.

We now report Theorem 1 from [43] followed by a discussion on its implications
for our work.

######  Theorem 1.

(Gabriel Loaiza-Ganem et al. [43]). Let $\mathcal{M}\subset\mathbb{R}^{D}$ be
an analytic $d$-dimensional embedded submanifold of $\mathbb{R}^{d}$ with
$d<D$, $\mu_{D}$ is the Lebesgue measure on $\mathbb{R}^{D}$, and
$p^{\dagger}$ a smooth probability measure on $\mathcal{M}$. Then there exists
a sequence of probability measures
$\big{\\{}p_{{{\bm{\mathbf{\theta}}}}}^{(t)}\big{\\}}_{t=0}^{\infty}$ on
$\mathbb{R}^{D}$ such that:

  1. 1.

$p_{{{\bm{\mathbf{\theta}}}}}^{(t)}\rightarrow p^{\dagger}$ as
$t\rightarrow\infty$.

  2. 2.

$\forall t\geq 0,p_{{{\bm{\mathbf{\theta}}}}}^{(t)}\ll\mu_{D}$ and
$p_{{{\bm{\mathbf{\theta}}}}}^{(t)}$ admits a density
$p_{{{\bm{\mathbf{\theta}}}}}^{(t)}:\mathbb{R}^{D}\rightarrow\mathbb{R}_{>0}$
with respect to $\mu_{D}$ such that:

    1. (a)

$\lim_{t\rightarrow\infty}p_{{{\bm{\mathbf{\theta}}}}}^{(t)}({{\bm{\mathbf{x}}}})=\infty$,
$\forall{{\bm{\mathbf{x}}}}\in\mathcal{M}$.

    2. (b)

$\lim_{t\rightarrow\infty}p_{{{\bm{\mathbf{\theta}}}}}^{(t)}({{\bm{\mathbf{x}}}})=0$,
$\forall{{\bm{\mathbf{x}}}}\notin\text{cl}(\mathcal{M})$, where
$\text{cl}(\cdot)$ denotes closure in $\mathbb{R}^{D}$.

Theorem 1 holds for any smooth probability measure supported in $\mathcal{M}$.
This is an important point because this includes the desired
$p_{\text{data}}$, provided that this is smooth too. The key message in [43]
is that, a-priori, there is no reason to expect that for a likelihood-based
model to converge to $p_{\text{data}}$ out of all the possible $p^{\dagger}$.
Their proof is based on convolving $p^{\dagger}$ with a Gaussian kernel with
variance $\sigma_{t}^{2}$ that decreases to zero as $t\rightarrow\infty$, and
then verify that the stated properties of $p_{{{\bm{\mathbf{\theta}}}}}^{(t)}$
hold. Our analysis, while relying on the same technical tools, is instead
constructive in explaining why the proposed data mollification allows us to
avoid manifold overfitting. The idea is as follows: at time step $t=0$, we
select the desired $p_{\text{data}}$ convolved with a Gaussian kernel with a
large, but finite, variance $\sigma^{2}(0)$ as the target distribution for the
optimization. Optimization is performed and
$p_{{{\bm{\mathbf{\theta}}}}}^{(0)}$ targets this distribution, without any
manifold overfitting issues, since the dimensionality of the corrupted data is
non-degenerate. At the second step, the target distribution is obtained by
convolving $p_{\text{data}}$ with the kernel with variance
$\sigma^{2}(1)<\sigma^{2}(0)$, and again manifold overfitting is avoided. By
iteratively repeating this procedure, we can reach the point where we are
matching a distribution convolved with an arbitrarily small variance
$\sigma^{2}(t)$, without ever experiencing manifold overfitting. We argue that
when removing the last bit of perturbation, the parametric distribution will
match $p_{\text{data}}$, without overfitting; this is equivalent to a good
initialization method. Under a different perspective, the annealing procedure
introduces a memory effect in the optimization process which is important for
the success of mollification. Indeed, as mentioned in [43] and verified by
ourselves in earlier investigations, a small constant amount of noise does not
provide any particular benefits over the original scheme.

##  3 Generative Models with Data Mollification

Motivated by the aforementioned problems with density estimation in low-
density regions and manifold overfitting, we propose a simple yet effective
approach to improve likelihood-based gms. Our method involves mollifying data
using Gaussian noise, gradually reducing its variance, until recovering the
original data distribution $p_{\text{data}}({{\bm{\mathbf{x}}}})$. This
mollification procedure is similar to the reverse process of diffusion models,
where a prior noise distribution is smoothly transformed into the data
distribution [66, 26, 67]. As already mentioned, data mollification alleviates
the problem of manifold overfitting and it induces a memory effect in the
optimization which improves density estimation in regions of low density.

Figure 2: Illustration of Gaussian mollification, where
$\tilde{{{\bm{\mathbf{x}}}}}_{t}$ is the mollified data at iteration $t$.

1 for  _$t\leftarrow 1,2,...,T$_ do

${{\bm{\mathbf{x}}}}\sim p_{\text{data}}({{\bm{\mathbf{x}}}})$ // Sample
training data

$\tilde{{{\bm{\mathbf{x}}}}}_{t}=\alpha_{t}{{\bm{\mathbf{x}}}}+\sigma_{t}\bm{\varepsilon}$
// Mollify data with $\alpha_{t},\sigma^{2}_{t}\leftarrow\gamma(t/T)$ and
$\bm{\varepsilon}\sim\mathcal{N}(\bm{0},{{\bm{\mathbf{I}}}})$

${{\bm{\mathbf{\theta}}}}_{t}\leftarrow\textsc{update}({{\bm{\mathbf{\theta}}}}_{t-1},\tilde{{{\bm{\mathbf{x}}}}}_{t})$
// Train model

2

Algorithm 1 Gaussian mollification

##### Gaussian Mollification.

Given that we train the model
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ for $T$ iterations, we can
create a sequence of progressively less smoothed versions of the original data
${{\bm{\mathbf{x}}}}$, which we refer to as mollified data
$\tilde{{{\bm{\mathbf{x}}}}}_{t}$. Here, $t$ ranges from $t=0$ (the most
mollified) to $t=T$ (the least mollified). For any $t\in[0,T]$, the
distribution of the mollified data $\tilde{{{\bm{\mathbf{x}}}}}_{t}$,
conditioned on ${{\bm{\mathbf{x}}}}$, is given as follows:

$q(\tilde{{{\bm{\mathbf{x}}}}}_{t}\,|\,{{\bm{\mathbf{x}}}})=\mathcal{N}(\tilde{{{\bm{\mathbf{x}}}}}_{t};\alpha_{t}{{\bm{\mathbf{x}}}},\sigma_{t}^{2}{{\bm{\mathbf{I}}}}),$
(2)

where $\alpha_{t}$ and $\sigma^{2}_{t}$ are are positive scalar-valued
functions of $t$. In addition, we define the signal-to-noise ratio
$\text{SNR}(t)={\alpha_{t}^{2}}/{\sigma_{t}^{2}},$ and we assume that it
monotonically increases with $t$, i.e., $\text{SNR}(t)\leq\text{SNR}(t+1)$ for
all $t\in[0,T-1]$. In other words, the mollified data
$\tilde{{{\bm{\mathbf{x}}}}}_{t}$ is progressively less smoothed as $t$
increases. In this work, we adopt the variance-preserving formulation used for
diffusion models [65, 26, 32], where $\alpha_{t}=\sqrt{1-\sigma_{t}^{2}}$ and
$\sigma^{2}_{t}=\gamma(t/T)$. Here, $\gamma(\cdot)$ is a monotonically
decreasing function from $1$ to $0$ that controls the rate of mollification.
Intuitively, this procedure involves gradually transforming an identity-
covariance Gaussian distribution into the distribution of the data. Algorithm
1 summarizes the proposed Gaussian mollification procedure, where the  red
line indicates a simple additional step required to mollify data compared with
vanilla training using the true data distribution.

##### Noise schedule.

The choice of the noise schedule $\gamma(\cdot)$ has an impact on the
performance of the final model. In this work, we follow common practice in
designing the noise schedule based on the literature of score-based dms [48,
27, 8]. In particular, we adopt a sigmoid schedule [31], which has recently
been shown to be more effective in practice compared to other choices such as
linear [26] or cosine schedules [48].

The sigmoid schedule $\gamma(t/T)$ [31] is defined through the sigmoid
function:

$\text{sigmoid}\left(-\frac{t/T}{\tau}\right),$ (3)

where $\tau$ is a temperature hyper-parameter. This function is then scaled
and shifted to ensure that $\gamma(0)=1$ and $\gamma(1)=0$. We encourage the
reader to check the implementation of this schedule, available in Appendix.
Fig. 3 illustrates the sigmoid schedule and the corresponding
$\log(\text{SNR})$ with different values of $\tau$. We use a default
temperature of $0.7$ as it demonstrates consistently good results in our
experiments.

$\mathsf{0}$$\mathsf{0.5}$$\mathsf{1}$$\mathsf{0}$$\mathsf{0.5}$$\mathsf{1}$$\displaystyle
t/T$$\displaystyle\sigma^{2}$$\mathsf{0}$$\mathsf{0.5}$$\mathsf{1}$$\mathsf{0}$$\mathsf{10}$$\displaystyle
t/T$$\displaystyle\log(SNR)$ Figure 3: Illustration of sigmoid schedule and
the corresponding $\log(\text{SNR})$. The temperature values from $0.2$ to
$0.9$ are progressively shaded, with the lighter shade corresponding to lower
temperatures.

##  4 Experiments

In this section, we demonstrate emprically the effectiveness of our proposal
through a wide range of experiments on synthetic data and some popular real-
world image data sets. The Appendix contains a detailed description of each
experiment to guarantee reproducibility.

###  4.1 2D Synthetic Data Sets

We begin our experimental campaign with two 2D synthetic data sets. The two-
dimensional nature of these data sets allows us to demonstrate the
effectiveness of Gaussian mollification in mitigating the challenges
associated with density estimation in low-density regions and manifold
overfitting. Here, we consider
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ to be a real-nvp flow
[14], which comprises five coupling bijections, each consisting of a two-
hidden layer multilayer perceptron (mlp). To assess the capability of
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ to recover the true data
distribution, we use Maximum Mean Discrepancy (mmd) [21] with a radial basis
function (rbf) kernel on a held-out set. In these experiments, we employ the
Gaussian mollification strategy presented in the previous section and compare
the estimated density with the vanilla approach where we use the original
training data without any mollification.

Figure 4: The first column shows the target distribution and the true scores.
The second column depicts the estimated distributions of the gmm , which yield
${\scriptsize\text{MMD}^{2}}$ of ${\scriptsize 15.5}$ and ${\scriptsize 2.5}$
for the vanilla (top) and mollification (bottom) training, respectively. The
remaining columns show histogram of samples from the true (top row) and
mollified data (bottom row), and estimated scores.

$\mathsf{2.3}$$\mathsf{4.61}$MMD2
[log]$\mathsf{0}$$\mathsf{5000}$$\mathsf{10000}$$\mathsf{15000}$$\mathsf{20000}$$\mathsf{1.15}$$\mathsf{1.38}$IterationTraining
loss [log]

| Vanilla |  | Mollification
---|---|---|---

Figure 5: The learning curves of the gmm experiments.

Figure 6: The progression of the estimated densities for the von Mises
distribution from the vanilla (bottom row) and our mollification (top row)
approaches.

##### Mixture of Gaussians.

First, we consider a target distribution that is a mixture of two Gaussians,
as depicted in Fig. 5. As discussed in § 2.1, the vanilla training procedure
fails to accurately estimate the true data distribution and scores,
particularly in the low-density regions. The estimated densities and the
mollified data during the training are depicted in Fig. 5. Initially, the
mollification process considers a simpler coarse-grained version of the target
density, which is easy to model. This is demonstrated by the low training loss
at the beginning of the optimization, as depicted in Fig. 5. Subsequently, the
method gradually reduces the level of noise allowing for a progressive
refinement of the estimated versions of the target density. This process uses
the solution from one level of mollification as a means to guiding
optimization for the next. As a result, Gaussian mollification facilitates the
recovery of the modes and enables effective density estimation in low-density
regions. The vanilla training procedure, instead, produces a poor estimate of
the target density, as evidenced by the trace-plot of the
$\lx@glossaries@gls@link{acronym}{MMD}{\leavevmode\textsc{mmd}}^{2}$ metric in
Fig. 5 and the visualization of the scores in Fig. 5.

##### Von Mises distribution.

We proceed with an investigation of the von Mises distribution on the unit
circle, as depicted in Fig. 6, with the aim of highlighting the issue of
manifold overfitting [43]. In this experiment, the data lies on a one-
dimensional manifold embedded in a two-dimensional space. The vanilla training
procedure fails to approximate the target density effectively, as evidenced by
the qualitative results and the substantially high value of
$\lx@glossaries@gls@link{acronym}{MMD}{\leavevmode\textsc{mmd}}^{2}$ ($\approx
383.44$) shown in Fig. 6. In contrast, Gaussian mollification gradually guides
the estimated density towards the target, as depicted in Fig. 6, leading to a
significantly lower
$\lx@glossaries@gls@link{acronym}{MMD}{\leavevmode\textsc{mmd}}^{2}$ ($\approx
6.13$). Additionally, the mollification approach enables the estimated model
not only to accurately learn the manifold but also to capture the mode of the
density correctly.

###  4.2 Image Experiments

Table 1: FID score on cifar10 and celeba dataset (_lower is better_). The
small colored numbers indicate  improvement or  degration of the mollification
training compared to the vanilla training.

Model | cifar10 | celeba
---|---|---
vanilla |  | gaussian
---
mollifcation
| blurring
---
mollification
vanilla |  | gaussian
---
mollification
| blurring
---
mollificatoin
real-nvp [14] | $131.15$ |  $121.75$ $\downarrow 7.17$% |  $120.88$ $\downarrow 7.83$% | $81.25$ |  $79.68$ $\downarrow 1.93$% |  $85.40$ $\uparrow 5.11$%
glow [36] | $74.62$ |  $64.87$ $\downarrow 13.07$% |  $66.70$ $\downarrow 10.61$% | $97.59$ |  $70.91$ $\downarrow 27.34$% |  $74.74$ $\downarrow 23.41$%
vae[34] | $191.98$ |  $155.13$ $\downarrow 19.19$% |  $175.40$ $\downarrow 8.64$% | $80.19$ |  $72.97$ $\downarrow 9.00$% |  $77.29$ $\downarrow 3.62$%
vae-iaf [37] | $193.58$ |  $156.39$ $\downarrow 19.21$% |  $162.27$ $\downarrow 16.17$% | $80.34$ |  $73.56$ $\downarrow 8.44$% |  $75.67$ $\downarrow 5.81$%
iwae [4] | $183.04$ |  $146.70$ $\downarrow 19.85$% |  $163.79$ $\downarrow 10.52$% | $78.25$ |  $71.38$ $\downarrow 8.78$% |  $76.45$ $\downarrow 2.30$%
$\beta$-vae [25] | $112.42$ |  $93.90$ $\downarrow 16.47$% |  $101.30$ $\downarrow 9.89$% | $67.78$ |  $64.59$ $\downarrow 4.71$% |  $67.08$ $\downarrow 1.03$%
hvae [5] | $172.47$ |  $137.84$ $\downarrow 20.08$% |  $147.15$ $\downarrow 14.68$% | $74.10$ |  $72.28$ $\downarrow 2.46$% |  $77.54$ $\uparrow 4.64$%

##### Setup.

We evaluate our method on image generation tasks on cifar10 [40] and celeba
$64$ [41] datasets, using a diverse set of likelihood-based gms. The evaluated
models include the vanilla vae [34], the ${\beta}$-vae [25], and the vae-iaf
[37] which employs an expressive inverse autoregressive flow for the
approximate posterior. To further obtain flexible approximations of the
posterior of latent variables as well as a tight evidence lower bound (elbo),
we also select the Hamiltonian-vae (hvae) [5] and the importance weighted vae
(iwae) [4]. For flow-based models, we consider the real-nvp [14] and glow [36]
models in our benchmark. We found that that further training the model on the
original data after the mollification procedure leads to better performance.
Hence, in our approach we apply data mollification during the first half of
the optimization phase, and we continue optimize the model using the original
data in the second half. Nevertheless, to ensure a fair comparison, we adopt
identical settings for the vanilla and the proposed approaches, including
random seed, optimizer, and the total number of iterations.

##### Blurring mollification.

Even though Gaussian mollification is motivated by the manifold hypothesis, it
is not the only way to mollify the data. Indeed, Gaussian mollification does
not take into account certain inductive biases that are inherent in natural
images, including their multi-scale nature. Recently, [59, 28, 11] have
proposed methods that incorporate these biases in diffusion-type generative
models. Their approach involves stochastically reversing the heat equation,
which is a partial differential equation (pde) that can be used to erase fine-
scale information when applied locally to the 2D plane of an image. During
training, the model first learns the coarse-scale structure of the data, which
is easier to learn, and then gradually learns the finer details. It is
therefore interesting to assess whether this form of data mollification is
effective in the context of this work compared to the Gaussian homotopy. Note,
however, that under the manifold hypothesis, this type of mollification
produces the opposite effect to Gaussian homotopy in that at time $t=0$
mollified images lie on a 1D manifold and they are gradually transformed to
span the dimension of the data manifold; more details on blurring
mollification can be found in the Appendix.

##### Image generation.

We evaluate the quality of the generated images using the popular Fréchet
Inception Distance (fid) score [23] computed on $50\mathrm{K}$ samples from
the trained model using the pytorch-fid
111https://github.com/mseitzer/pytorch-fid library. The results, reported in
Table 1, indicate that the proposed data mollification consistently improves
model performance compared to vanilla training across all datasets and models.
Additionally, mollification through blurring, which is in line with recent
results from diffusion models [59], is less effective than Gaussian
mollification, although it still enhances the vanilla training in most cases.
We also show intermediate samples in Fig. 8 illustrating the progression of
samples from pure random noise or completely blurred images to high-quality
images. Furthermore, we observe that Gaussian mollification leads to faster
convergence of the FID score for vae-based models, as shown in Fig. 8. We
provide additional results in the Appendix. As a final experiment, we consider
a recent large-scale vae model for the cifar10 data set, which is a deep
hierarchical vae (nvae) [73]. By applying Gaussian mollification without
introducing any additional complexity, e.g., step-size annealing, we improve
the fid score from $53.64$ to $52.26$.

$\mathsf{0}$$\mathsf{50}$$\mathsf{100}$$\mathsf{150}$$\mathsf{200}$$\mathsf{5.07}$$\mathsf{5.53}$$\mathsf{5.99}$VAEEpochvae
\- FID [log]

$\mathsf{0}$$\mathsf{20}$$\mathsf{40}$$\mathsf{60}$$\mathsf{80}$$\mathsf{4.61}$$\mathsf{5.76}$Epochglow
\- FID [log]

| Vanilla |  | Gauss. |  | Blurring
---|---|---|---|---|---

Figure 7: The progression of FID on cifar10 dataset.

Figure 8: Intermediate samples generated from real-nvp flows [14], which are
trained on celeba dataset employed with Gaussian (top row) and blurring
mollification (bottom row).

##### Ablation study on noise schedule.

We ablate on the choice of noise schedule for Gaussian mollification. Along
with the sigmoid schedule, we also consider the linear [26] and cosine [48]
schedules, which are also popular for diffusion models. As shown in Table 2,
our method consistently outperforms the vanilla approach under all noise
schedules. We also observe that the sigmoid schedule consistently produced
good results. Therefore, we chose to use the sigmoid schedule in all our
experiments.

Table 2:  FID score on cifar10 w.r.t. different choices of noise schedule.

Model | vanilla | sigmoid | cosine | linear
---|---|---|---|---
real-nvp | $191.98$ | $121.75$ | $118.71$ | $123.93$
glow | $74.62$ | $64.87$ | $71.90$ | $74.36$
vae | $191.98$ | $155.13$ | $154.71$ | $156.47$
$\beta$-vae | $112.42$ | $93.90$ | $92.86$ | $93.14$
iwae | $183.04$ | $146.70$ | $146.49$ | $149.16$

##### Comparisons with the two-step approach in [43].

Manifold overfitting in likelihood-based gms has been recently analyzed in
[43], which provides a two-step procedure to mitigate the issue. The first
step maps inputs into a low-dimensional space to handle the intrinsic low
dimensionality of the data. This step is then followed by likelihood-based
density estimation on the resulting lower-dimensional representation. This is
achieved by means of a generalized autoencoder, which relies on a certain set
of explicit deep gms, such as vaes. Here, we compare our proposal with this
two-step approach; results are reported in Table 3. To ensure a fair
comparison, we use the same network architecture for our vae and their
generalized autoencoder, and we rely on their official implementation
222https://github.com/layer6ai-labs/two_step_zoo. Following [43], we consider
a variety of density estimators in the low-dimensional space such as nfs,
Autoregressive Models (arms) [72], Adversarial Varitional Bayes (avb) [46] and
Energy-Based Models (ebms) [16]. We observe that Gaussian mollification is
better or comparable with these variants. In addition, our method is extremely
simple and readily applicable to any likelihood-based gms without any extra
auxilary models or the need to modify training procedures.

Table 3: Comparisons of FID scores on cifar10 between mollitication and two-
step methods.

vanilla vae | mollification | two-step
---|---|---
vae+gaussian | vae + blurring | vae+nf | vae+ebm | vae+avb | vae+arm
191.98 | $155.13$ | $175.40$ | $208.80$ | $166.20$ | $153.72$ | $203.32$

##### Limitations.

One limitation is that this work focuses exclusively on likelihood-based gms
and on image data. The improvements in fid score indicate that the performance
boost is generally substantial, but still far from being comparable with
state-of-the-art dms. While this may give an impression of a low impact, we
believe that this work is important in pointing to one of the successful
aspects characterizing dms and show how this can be easily integrated in the
optimization of likelihood-based gms. A second limitation is that, in line
with the literature on gms where models are extremely costly to train and
evaluate, we did not provide error bars on the results reported in the tables
in the experimental section. Having said that, the improvements reported in
the experiments have been shown on a variety of models and on two popular
image data sets. Furthermore, the results are supported theretically and
experimentally by a large literature on Gaussian homotopy.

##### Broader impact.

This work provides an efficient way to improve a class of gms. While we
focused on images, the proposed method can be applied to other types of data.
Like other works in this literature, the proposed method can have both
positive (e.g., synthesizing new data automatically or anomaly detection) and
negative (e.g., deep fakes) impacts on society depending on the application.

##  5 Related work

Our work is positioned within the context of improving gms through the
introduction of noise to the data. One popular approach is the use of
denoising autoencoders [76], which are trained to reconstruct clean data from
noisy samples. Building upon this, [2] proposed a framework for modeling a
Markov chain whose stationary distribution approximates the data distribution.
In addition, [75] showed a connection between denoising autoencoders and score
matching, which is an objective closely related to recent diffusion models
[65, 26]. More recently, [45] introduced a two-step approach to improve
autoregressive generative models, where a smoothed version of the data is
first modeled by adding a fixed level of noise, and then the original data
distribution is recovered through an autoregressive denoising model. In a
similar vein, [44] recently attempted to use Tweedie’s formula [60] as a
denosing step, but surprisingly found that it does not improve the performance
of nfs and vaes. Our work is distinct from these approaches in that Gaussian
mollification guides the estimated distribution towards the true data
distribution in a progressive manner by means of annealing instead of fixing a
noise level. Moreover, our approach does not require any explicit denoising
step, and it can be applied off-the-shelf to the optimization of any
likelihood-based gms without any modifications.

##  6 Conclusion

Inspired by the enormous success of score-based dms, in this work we
hypothesized that data mollification is partially responsible for their
impressive performance in generative modeling tasks. In order to test this
hypothesis, we introduced data mollification within the optimization of
likelihood-based gms, focusing in particular on nfs and vaes. Data
mollification is extremely easy to implement and it has nice theoretical
properties due to its connection with Gaussian homotopy, which is a well-known
technique to improve optimization. We applied this idea to challenging
generative modeling tasks involving imaging data and relatively large-scale
architectures as a means to demonstrate systematic gains in performance in
various conditions and input dimensions. We measured performance in quality of
generated images through the popular fid score. While we are far from closing
the gap with dms in achieving competing fid score, we are confident that this
work will serve as the basis for future works on performance improvements in
state-of-the-art models mixing dms and likelihood-based gms, and in
alternative forms of mollification to improve optimization of state-of-the-art
gms. For example, it would be interesting to study how to apply data
mollification to improve the training of gans; preliminary investigations show
that the strategy proposed here does not offer significant performance
improvements, and we believe this is due to the fact that data mollification
does not help in smoothing the adversarial objective. Also, while our study
shows that the data mollification schedule is not critical, it would be
interesting to study whether it is possible to derive optimal mollification
schedules, taking inspiration, e.g., from [30]. We believe it would also be
interesting to consider mixture of likelihood-based gms to counter problems
due to the union of manifolds hypothesis, whereby the intrinsic dimension
changes over the input [3]. Finally, it would be interesting to investigate
other data, such as 3D point cloud data [79] and extend this work to deal with
other tasks, such as supervised learning.

## Acknowledgments and Disclosure of Funding

MF gratefully acknowledges support from the AXA Research Fund and the Agence
Nationale de la Recherche (grant ANR-18-CE46-0002 and ANR-19-P3IA-0002).

## References

  * [1] Yoshua Bengio, Aaron C. Courville, and Pascal Vincent.  Representation Learning: A Review and New Perspectives.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:1798–1828, 2012.
  * [2] Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski.  Deep Generative Stochastic Networks Trainable by Backprop.  In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 226–234, Bejing, China, 22–24 Jun 2014. PMLR.
  * [3] Bradley CA Brown, Anthony L. Caterini, Brendan Leigh Ross, Jesse C Cresswell, and Gabriel Loaiza-Ganem.  Verifying the Union of Manifolds Hypothesis for Image Data.  In International Conference on Learning Representations, 2023.
  * [4] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov.  Importance Weighted Autoencoders.  In International Conference on Learning Representations, 2015.
  * [5] Anthony L Caterini, Arnaud Doucet, and Dino Sejdinovic.  Hamiltonian Variational Auto-Encoder.  In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [6] Clément Chadebec, Louis Vincent, and Stephanie Allassonniere.  Pythae: Unifying Generative Autoencoders in Python - A Benchmarking Use Case.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 21575–21589. Curran Associates, Inc., 2022.
  * [7] Tianrong Chen, Guan-Horng Liu, and Evangelos Theodorou.  Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory.  In International Conference on Learning Representations, 2022.
  * [8] Ting Chen.  On the Importance of Noise Scheduling for Diffusion Models.  arXiv preprint arXiv:2301.10972, 2023.
  * [9] Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet.  Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows.  In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2133–2143. PMLR, 13–18 Jul 2020.
  * [10] Bin Dai and David Wipf.  Diagnosing and Enhancing VAE Models.  In International Conference on Learning Representations, 2019.
  * [11] Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alex Dimakis, and Peyman Milanfar.  Soft Diffusion: Score Matching with General Corruptions.  Transactions on Machine Learning Research, 2023.
  * [12] Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan.  Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders.  arXiv preprint arXiv:1611.02648, 2016.
  * [13] Laurent Dinh, David Krueger, and Yoshua Bengio.  NICE: Non-linear Independent Components Estimation.  In International Conference on Learning Representations, 2015.
  * [14] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density Estimation Using Real NVP.  In International Conference on Learning Representations, 2017.
  * [15] Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie.  Invertible Generative Modeling using Linear Rational Splines.  In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 4236–4246. PMLR, 26–28 Aug 2020.
  * [16] Yilun Du and Igor Mordatch.  Implicit Generation and Modeling with Energy Based Models.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [17] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios.  Neural Spline Flows.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [18] Matilde Gargiani, Andrea Zanelli, Quoc Tran-Dinh, Moritz Diehl, and Frank Hutter.  Convergence Analysis of Homotopy-SGD for Non-convex Optimization.  arXiv preprint arXiv:2011.10298, 2020.
  * [19] Aude Genevay, Gabriel Peyré, and Marco Cuturi.  GAN and VAE from an Optimal Transport Point of View.  arXiv preprint arXiv:1706.01807, 2017.
  * [20] Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle.  MADE: Masked Autoencoder for Distribution Estimation.  In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 881–889, Lille, France, 07–09 Jul 2015. PMLR.
  * [21] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.  A Kernel Two-Sample Test.  Journal of Machine Learning Research, 13(25):723–773, 2012.
  * [22] Elad Hazan, Kfir Yehuda Levy, and Shai Shalev-Shwartz.  On Graduated Optimization for Stochastic Non-Convex Problems.  In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1833–1841, New York, New York, USA, 20–22 Jun 2016. PMLR.
  * [23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [24] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6626–6637, 2017.
  * [25] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.  In International Conference on Learning Representations, 2017.
  * [26] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising Diffusion Probabilistic Models.  In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  * [27] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans.  simple diffusion: End-to-end Diffusion for High Resolution Images.  arXiv preprint arXiv:2301.11093, 2023.
  * [28] Emiel Hoogeboom and Tim Salimans.  Blurring Diffusion Models.  In International Conference on Learning Representations, 2023.
  * [29] Aapo Hyvärinen.  Estimation of Non-Normalized Statistical Models by Score Matching.  Journal of Machine Learning Research, 6(24):695–709, 2005.
  * [30] Hidenori Iwakiri, Yuhang Wang, Shinji Ito, and Akiko Takeda.  Single Loop Gaussian Homotopy Method for Non-convex Optimization.  In Advances in Neural Information Processing Systems, volume 35, pages 7065–7076. Curran Associates, Inc., 2022.
  * [31] Allan Jabri, David Fleet, and Ting Chen.  Scalable Adaptive Computation for Iterative Generation.  arXiv preprint arXiv:2212.11972, 2022.
  * [32] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational Diffusion Models.  In Advances in Neural Information Processing Systems, volume 34, pages 21696–21707. Curran Associates, Inc., 2021.
  * [33] Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  In International Conference on Learning Representations, 2015.
  * [34] Diederik P. Kingma and Max Welling.  Auto-Encoding Variational Bayes.  In International Conference on Learning Representations, 2014.
  * [35] Diederik P. Kingma and Max Welling.  An Introduction to Variational Autoencoders.  Foundations and Trends in Machine Learning, 12(4):307–392, 2019\.
  * [36] Durk P Kingma and Prafulla Dhariwal.  Glow: Generative Flow with Invertible 1x1 Convolutions.  In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [37] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved Variational Inference with Inverse Autoregressive Flow.  In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [38] Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt.  Learning Hierarchical Priors in VAEs.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [39] Ivan Kobyzev, Simon J. D. Prince, and Marcus A. Brubaker.  Normalizing Flows: An Introduction and Review of Current Methods.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):3964–3979, 2021.
  * [40] A. Krizhevsky and G. Hinton.  Learning Multiple Layers of Features from Tiny Images.  Master’s thesis, Department of Computer Science, University of Toronto, 2009.
  * [41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep Learning Face Attributes in the Wild.  In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 3730–3738. IEEE Computer Society, 2015.
  * [42] Antoine Liutkus, Umut Simsekli, Szymon Majewski, Alain Durmus, and Fabian-Robert Stöter.  Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions.  In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4104–4113. PMLR, 09–15 Jun 2019.
  * [43] Gabriel Loaiza-Ganem, Brendan Leigh Ross, Jesse C Cresswell, and Anthony L. Caterini.  Diagnosing and Fixing Manifold Overfitting in Deep Generative Models.  Transactions on Machine Learning Research, 2022.
  * [44] Gabriel Loaiza-Ganem, Brendan Leigh Ross, Luhuan Wu, John P Cunningham, Jesse C Cresswell, and Anthony L Caterini.  Denoising Deep Generative Models.  In I Can’t Believe It’s Not Better Workshop at NeurIPS 2022, 2022\.
  * [45] Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao, and Stefano Ermon.  Improved Autoregressive Modeling with Distribution Smoothing.  In International Conference on Learning Representations, 2021.
  * [46] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger.  Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks.  In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2391–2400. PMLR, 06–11 Aug 2017.
  * [47] Hariharan Narayanan and Sanjoy Mitter.  Sample Complexity of Testing the Manifold Hypothesis.  In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
  * [48] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved Denoising Diffusion Probabilistic Models.  In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162–8171. PMLR, 18–24 Jul 2021.
  * [49] Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto.  OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9223–9232, 2021.
  * [50] Arkadas Ozakin and Alexander Gray.  Submanifold Density Estimation.  In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
  * [51] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing Flows for Probabilistic Modeling and Inference.  Journal of Machine Learning Research, 22(57):1–64, 2021.
  * [52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [53] Michele Pavon, Giulio Trigila, and Esteban G Tabak.  The Data-Driven Schrödinger Bridge.  Communications on Pure and Applied Mathematics, 74(7):1545–1573, 2021.
  * [54] Xavier Pennec.  Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements.  Journal of Mathematical Imaging and Vision, 25:127–154, 2006.
  * [55] Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein.  The Intrinsic Dimension of Images and Its Impact on Learning.  In International Conference on Learning Representations, 2021.
  * [56] Daria Reshetova, Yikun Bai, Xiugang Wu, and Ayfer Özgür.  Understanding Entropic Regularization in GANs.  In 2021 IEEE International Symposium on Information Theory (ISIT), pages 825–830. IEEE, 2021.
  * [57] Danilo Rezende and Shakir Mohamed.  Variational Inference with Normalizing Flows.  In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1530–1538, Lille, France, 07–09 Jul 2015. PMLR.
  * [58] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pages 1278–1286. JMLR.org, 2014\.
  * [59] Severi Rissanen, Markus Heinonen, and Arno Solin.  Generative Modelling with Inverse Heat Dissipation.  In International Conference on Learning Representations, 2023.
  * [60] Herbert Robbins.  An Empirical Bayes Approach to Statistics.  In Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probability, 1956, volume 1, pages 157–163, 1956.
  * [61] Sam T Roweis and Lawrence K Saul.  Nonlinear Dimensionality Reduction by Locally Linear Embedding.  Science, 290(5500):2323–2326, 2000.
  * [62] Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnes Desolneux.  Can Push-forward Generative Models Fit Multimodal Distributions?  In Advances in Neural Information Processing Systems, volume 35, pages 10766–10779. Curran Associates, Inc., 2022.
  * [63] Maziar Sanjabi, Jimmy Ba, Meisam Razaviyayn, and Jason D Lee.  On the Convergence and Robustness of Training GANs with Regularized Optimal Transport.  In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [64] Casper Kaae Sø nderby, Tapani Raiko, Lars Maalø e, Søren Kaae Sø nderby, and Ole Winther.  Ladder Variational Autoencoders.  In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [65] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics.  In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265, Lille, France, 07–09 Jul 2015. PMLR.
  * [66] Yang Song and Stefano Ermon.  Generative Modeling by Estimating Gradients of the Data Distribution.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [67] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-Based Generative Modeling through Stochastic Differential Equations.  In International Conference on Learning Representations, 2021.
  * [68] Vincent Stimper, David Liu, Andrew Campbell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Hernández-Lobato.  normflows: A PyTorch Package for Normalizing Flows.  arXiv preprint arXiv:2302.12014, 2023.
  * [69] Piotr Tempczyk, Rafał Michaluk, Lukasz Garncarek, Przemysław Spurek, Jacek Tabor, and Adam Golinski.  LIDL: Local Intrinsic Dimension Estimation Using Approximate Likelihood.  In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21205–21231. PMLR, 17–23 Jul 2022.
  * [70] Joshua B Tenenbaum, Vin de Silva, and John C Langford.  A Global Geometric Framework for Nonlinear Dimensionality Reduction.  Science, 290(5500):2319–2323, 2000.
  * [71] Jakub Tomczak and Max Welling.  VAE with a VampPrior.  In Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 1214–1223. PMLR, 09–11 Apr 2018.
  * [72] Benigno Uria, Iain Murray, and Hugo Larochelle.  RNADE: The Real-valued Neural autoregressive Density-estimator.  In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  * [73] Arash Vahdat and Jan Kautz.  NVAE: A Deep Hierarchical Variational Autoencoder.  In Advances in Neural Information Processing Systems, volume 33, pages 19667–19679. Curran Associates, Inc., 2020.
  * [74] Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, and Max Welling.  Sylvester Normalizing Flows for Variational Inference.  In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 393–402. AUAI Press, 2018.
  * [75] Pascal Vincent.  A Connection Between Score Matching and Denoising Autoencoders.  Neural Computation, 23:1661–1674, 2011.
  * [76] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and Composing Robust Features with Denoising Autoencoders.  In Proceedings of the 25th International Conference on Machine Learning, pages 1096–1103, 2008.
  * [77] Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou.  Diffusion-GAN: Training GANs with Diffusion.  In International Conference on Learning Representations, 2023.
  * [78] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat.  Tackling the Generative Learning Trilemma with Denoising Diffusion GANs.  In International Conference on Learning Representations, 2022.
  * [79] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge J. Belongie, and Bharath Hariharan.  PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 4540–4549. IEEE, 2019.

##  Appendix A A Primer on Normalizing Flows and VAEs

Given a dataset $\mathcal{D}$ consisting of $N$ i.i.d samples
$\mathcal{D}\stackrel{{\scriptstyle\Delta}}{{=}}\\{{{\bm{\mathbf{x}}}}_{i}\\}_{i=1}^{N}$
with ${{\bm{\mathbf{x}}}}_{i}\in\mathbb{R}^{D}$, we aim at estimating the
unknown continuous generating distribution
$p_{\text{data}}({{\bm{\mathbf{x}}}})$. In order to do so, we introduce a
model $p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$ with parameters
${{\bm{\mathbf{\theta}}}}$ and attempt to estimate ${{\bm{\mathbf{\theta}}}}$
based on the dataset $\mathcal{D}$. A common approach to estimate
${{\bm{\mathbf{\theta}}}}$ is to maximize the likelihood of the data, which is
equivalent to minimizing the following objective:

$\displaystyle\mathcal{L}({{\bm{\mathbf{\theta}}}})$
$\displaystyle\stackrel{{\scriptstyle\Delta}}{{=}}-\mathbb{E}_{p_{\text{data}}({{\bm{\mathbf{x}}}})}\left[\log
p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})\right].$ (4)

Optimization for this objective can be done through a stochastic gradient
descent algorithm using minibatches of samples from
$p_{\text{data}}({{\bm{\mathbf{x}}}})$.

###  A.1 Normalizing Flows

In flow-based generative models [51, 39], the generative process is defined
as:

$\displaystyle{{\bm{\mathbf{z}}}}\sim
p_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}});\quad{{\bm{\mathbf{x}}}}={{\bm{\mathbf{f}}}}_{{{\bm{\mathbf{\psi}}}}}({{\bm{\mathbf{z}}}}),$
(5)

where ${{\bm{\mathbf{z}}}}\in\mathbb{R}^{D}$ is a latent variable, and
$p_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}})$ is a tractable base
distribution with parameters ${{\bm{\mathbf{\phi}}}}$, such as an isotropic
multivariate Gaussian. The function
${{\bm{\mathbf{f}}}}_{{{\bm{\mathbf{\psi}}}}}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$
is invertible, such that given any input vector ${{\bm{\mathbf{x}}}}$ we have
${{\bm{\mathbf{z}}}}={{\bm{\mathbf{f}}}}^{-1}_{{{\bm{\mathbf{\psi}}}}}({{\bm{\mathbf{x}}}})$.
A Normalizing Flow (nf) [57] defines a sequence of invertible transformations
${{\bm{\mathbf{f}}}}={{\bm{\mathbf{f}}}}_{1}\circ{{\bm{\mathbf{f}}}}_{2}\circ\cdots{{\bm{\mathbf{f}}}}_{K}$,
such that the relationship between ${{\bm{\mathbf{x}}}}$ and
${{\bm{\mathbf{z}}}}$ can be written as:

${{\bm{\mathbf{x}}}}\stackrel{{\scriptstyle{{\bm{\mathbf{f}}}}_{1}}}{{\longleftrightarrow}}{{\bm{\mathbf{h}}}}_{1}\stackrel{{\scriptstyle{{\bm{\mathbf{f}}}}_{2}}}{{\longleftrightarrow}}{{\bm{\mathbf{h}}}}_{2}\cdots\stackrel{{\scriptstyle{{\bm{\mathbf{f}}}}_{K}}}{{\longleftrightarrow}}{{\bm{\mathbf{z}}}},$
(6)

where
${{\bm{\mathbf{h}}}}_{k}={{\bm{\mathbf{f}}}}_{k}^{-1}({{\bm{\mathbf{h}}}}_{k-1};{{\bm{\mathbf{\psi}}}}_{k})$
and ${{\bm{\mathbf{\psi}}}}_{k}$ are the parameters of the transformation
${{\bm{\mathbf{f}}}}_{k}$. For the sake of simplicity, we define
${{\bm{\mathbf{h}}}}_{0}\stackrel{{\scriptstyle\Delta}}{{=}}{{\bm{\mathbf{x}}}}$
and
${{\bm{\mathbf{h}}}}_{K}\stackrel{{\scriptstyle\Delta}}{{=}}{{\bm{\mathbf{z}}}}$.
The likelihood of the model given a datapoint can be computed analytically
using the change of variables as follows:

$\displaystyle\log p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})$
$\displaystyle=\log
p_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}})+\log\left|\det(\partial{{\bm{\mathbf{z}}}}/\partial{{\bm{\mathbf{x}}}})\right|$
(7) $\displaystyle=\log
p_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}})+\sum_{k=1}^{K}\log\left|\det(\partial{{\bm{\mathbf{h}}}}_{k}/\partial{{\bm{\mathbf{h}}}}_{k-1})\right|,$
(8)

where
$\log|\det(\partial{{\bm{\mathbf{h}}}}_{k}/\partial{{\bm{\mathbf{h}}}}_{k-1})|$
is the logarithm of absolute value of the determinant of the Jacobian matrix
$\partial{{\bm{\mathbf{h}}}}_{k}/\partial{{\bm{\mathbf{h}}}}_{k-1}$. This term
accounts for the change of measure when going from ${{\bm{\mathbf{h}}}}_{k-1}$
to ${{\bm{\mathbf{h}}}}_{k}$ using the transformation
${{\bm{\mathbf{f}}}}_{k}$. The resulting nf model is then characterized by the
set of parameters
${{\bm{\mathbf{\theta}}}}=\\{{{\bm{\mathbf{\phi}}}}\\}\cup\\{{{\bm{\mathbf{\psi}}}}_{k}\\}_{k=1}^{K}$,
which can be estimated using the maximum likelihood estimation (mle) objective
Eq. 4.

Though nfs allow for exact likelihood computation, they require
${{\bm{\mathbf{f}}}}_{k}$ to be invertible and to have a tractable inverse and
Jacobian determinant. This restricts the flexibility to certain
transformations that can be used within nfs [see e.g., 51, 39, and references
therein], such as affine coupling [13, 14], invertible convolution [36],
spline [17, 15], or inverse autoregressive transformations [37].

###  A.2 Variational Autoencoders

Variational Autoencoders (vaes) [34, 57] introduce a low-dimensional latent
variable ${{\bm{\mathbf{z}}}}\in\mathbb{R}^{P}$, with $P\ll D$, to the
generative process as follows:

$\displaystyle{{\bm{\mathbf{z}}}}\sim
p({{\bm{\mathbf{z}}}});\quad{{\bm{\mathbf{x}}}}\sim
p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}}\,|\,{{\bm{\mathbf{z}}}}).$
(9)

Here, $p({{\bm{\mathbf{z}}}})$ is a tractable prior distribution over the
latent variables ${{\bm{\mathbf{z}}}}$, and
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}}\,|\,{{\bm{\mathbf{z}}}})$,
which is also known as a decoder, is usually implemented by a flexible neural
network parameterized by ${{\bm{\mathbf{\theta}}}}$. Different from nfs, vaes
employ a stochastic transformation
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}}\,|\,{{\bm{\mathbf{z}}}})$ to
map ${{\bm{\mathbf{z}}}}$ to ${{\bm{\mathbf{x}}}}$. Indeed, nfs can be viewed
as vaes where the decoder and encoder are modelled by Dirac deltas
$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}}\,|\,{{\bm{\mathbf{z}}}})=\delta\big{(}{{\bm{\mathbf{f}}}}_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})\big{)}$
and
$q_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}}\,|\,{{\bm{\mathbf{x}}}})=\delta\big{(}{{\bm{\mathbf{f}}}}^{-1}_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})\big{)}$
respectively, using a restricted family of transformations
${{\bm{\mathbf{f}}}}_{{{\bm{\mathbf{\theta}}}}}$.

The marginal likelihood of vaes is intractable and given by:

$p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})=\int
p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}}\,|\,{{\bm{\mathbf{z}}}})p({{\bm{\mathbf{z}}}})d{{\bm{\mathbf{z}}}}.$
(10)

A variational lower bound on ther marginal likelihood can be obtained by
introducing a variational distribution
$q_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}}\,|\,{{\bm{\mathbf{x}}}})$,
with parameters ${{\bm{\mathbf{\phi}}}}$, which acts as an approximation to
the the unknown posterior $p({{\bm{\mathbf{z}}}}\,|\,{{\bm{\mathbf{x}}}})$:

$\log
p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}})\geq\underbrace{\mathbb{E}_{q_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}}\,|\,{{\bm{\mathbf{x}}}})}[\log
p_{{{\bm{\mathbf{\theta}}}}}({{\bm{\mathbf{x}}}}\,|\,{{\bm{\mathbf{z}}}})]-\textsc{kl}\left[q_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}}\,|\,{{\bm{\mathbf{x}}}})\;\|\;p({{\bm{\mathbf{z}}}})\right]}_{\mathcal{L}_{\text{ELBO}}({{\bm{\mathbf{\theta}}}},{{\bm{\mathbf{\phi}}}})},$
(11)

where,
$\mathcal{L}_{\text{ELBO}}({{\bm{\mathbf{\theta}}}},{{\bm{\mathbf{\phi}}}})$
is known as the elbo, and the expectation can be approximated by using Monte
Carlo samples from
$q_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}}\,|\,{{\bm{\mathbf{x}}}})$.
This objective can be optimized with stochastic optimization w.r.t. parameters
${{\bm{\mathbf{\theta}}}}$ and ${{\bm{\mathbf{\phi}}}}$ in place of Eq. 4.

To tighten the gap between the elbo and the true marginal likelihood, vaes can
be employed with an expressive form of the approximate posterior
$q_{{{\bm{\mathbf{\phi}}}}}({{\bm{\mathbf{z}}}}\,|\,{{\bm{\mathbf{x}}}})$ such
as importance weighted sampling [4] or normalizing flows [37, 74]. In
addition, to avoid the over regularization offect induced by the prior
$p({{\bm{\mathbf{z}}}})$, one can utilize a flexible prior such as multi-modal
distributions [12, 71], hierarchical forms [64, 38], or simply reweighing the
KL divergence term in the elbo [25].

##  Appendix B Details on Blurring Mollification

Recently, [59, 28, 11] have proposed appproaches to destroy information of
images using blurring operations for diffusion-type generative models. Their
approach involves stochastically reversing the heat equation, which is a pde
that can be used to erase fine-scale information when applied locally to the
2D plane of an image. In particular, the Laplace pde for heat diffusions is as
follows:

$\frac{\partial}{\partial
t}\tilde{{{\bm{\mathbf{x}}}}}(i,j,t)=\Delta\tilde{{{\bm{\mathbf{x}}}}}(i,j,t),$
(12)

where we consider the initial state of the system to be ${{\bm{\mathbf{x}}}}$,
the true image data. This pde can be effectively solved by employing a
diagonal matrix within the frequency domain of the cosine transform, provided
that the signal is discretized onto a grid. The solution to this equation at
time $t$ can be effectively computed by:

$\tilde{{{\bm{\mathbf{x}}}}}_{t}={{\bm{\mathbf{A}}}}_{t}{{\bm{\mathbf{x}}}}={{\bm{\mathbf{V}}}}{{\bm{\mathbf{D}}}}_{t}{{\bm{\mathbf{V}}}}^{\top}{{\bm{\mathbf{x}}}},$
(13)

Here, ${{\bm{\mathbf{V}}}}^{\top}$ and ${{\bm{\mathbf{V}}}}$ denote the
discrete cosine transformation (dct) and inverse dct, respectively; the
diagonal matrix ${{\bm{\mathbf{D}}}}_{t}$ is the exponent of a weighting
matrix for frequencies ${{\bm{\mathbf{\Lambda}}}}$ so that
${{\bm{\mathbf{D}}}}_{t}=\exp({{\bm{\mathbf{\Lambda}}}}t)$. We refer the
reader to Appendix A of [59] for the specific definition of
${{\bm{\mathbf{\Lambda}}}}$. We can evaluate Eq. 13 in the Fourier domain,
which is fast to compute, as the dct and inverse dct require
$\mathcal{O}(N\log N)$ operations. The equivalent form of Eq. 13 in the
Fourier domain is as follows:

$\tilde{{{\bm{\mathbf{u}}}}}_{t}=\exp({{\bm{\mathbf{\Lambda}}}}t){{\bm{\mathbf{u}}}},$
(14)

where
${{\bm{\mathbf{u}}}}={{\bm{\mathbf{V}}}}^{\top}{{\bm{\mathbf{x}}}}=\text{DCT}({{\bm{\mathbf{x}}}})$.
As ${{\bm{\mathbf{\Lambda}}}}$ is a diagonal matrix, the above Fourier-space
model is fast to evaluate. A Python implementation of this blurring
mollification is presented in Algorithm 2.

We follow [59] to set the schedule for the blurring mollification. In
particular, we use a logarithmic spacing for the time steps $t_{k}$, where
$t_{0}=\nicefrac{{\sigma^{2}_{B,\text{max}}}}{{2}}$ and
$t_{T}=\nicefrac{{\sigma^{2}_{B,\text{min}}}}{{2}}=\nicefrac{{0.5^{2}}}{{2}}$,
corresponding to sub-pixel-size blurring. Here, $\sigma^{2}_{B,\text{max}}$ is
the effective lengthscale-scale of blurring at the beginning of the
mollification process. Following [59], we set this to half the width of the
image.

⬇

1import numpy as np

2from scipy.fftpack import dct, idct

3

4def blurring_mollify(x, t):

5 # Assuming the image u is an (KxK) numpy array

6 K = x.shape[-1]

7 freqs = np.pi*np.linspace(0,K-1,K)/K

8 frequencies_squared = freqs[:,None]**2 + freqs[None,:]**2

9 x_proj = dct(u, axis=0, norm=’ortho’)

10 x_proj = dct(x_proj, axis=1, norm=’ortho’)

11 x_proj = np.exp(-frequencies_squared * t) * x_proj

12 x_mollified = idct(x_proj, axis=0, norm=’ortho’)

13 x_mollified = idct(x_mollified, axis=1, norm=’ortho’)

14 return x_mollified

Algorithm 2 Python code for blurring mollification

##  Appendix C Implementation of Noise Schedules

Algorithm 3 shows the Python code for the noise schedules used in this work.
For the sigmoid schedule, following [8], we set the default values of start
and end to $0$ and $3$, respectively.

⬇

1import numpy as np

2

3def sigmoid(x):

4 # Sigmoid function.

5 return 1 / (1 + np.exp(-x))

6

7def sigmoid_schedule(t, T, tau=0.7, start=0, end=3, clip_min=1e-9):

8 # A scheduling function based on sigmoid function with a temperature tau.

9 v_start = sigmoid(start / tau)

10 v_end = sigmoid(end / tau)

11 return (v_end - sigmoid((t/T * (end - start) + start) / tau)) / (v_end -
v_start)

12

13def linear_schedule(t, T):

14 # A scheduling function based on linear function.

15 return 1 - t/T

16

17def cosine_schedule(t, T, ns=0.0002, ds=0.00025):

18 # A scheduling function based on cosine function.

19 return np.cos(((t/T + ns) / (1 + ds)) * np.pi / 2)**2

Algorithm 3 Python code for noise schedules

##  Appendix D Experimental Details

###  D.1 Datasets

##### Synthetic datasets.

  * •

_Mixture of Gaussians_ : We consider a mixture of two Gaussians with means
$\mu_{k}=(2\sin(\pi k),2\cos(\pi k))$ and covariance matrices
${{\bm{\mathbf{\Sigma}}}}_{k}=\sigma^{2}{{\bm{\mathbf{I}}}}$, where
$\sigma=\frac{2}{3}\sin(\pi/2)$. We generate 10K samples for training and 10K
samples for testing from this distribution.

  * •

_Von Mises distribution:_ We use a von Mises distribution with parameters
$\kappa=1$, and then transform to Cartesian coordinates to obtain a
distribution on the unit circle in $\mathbb{R}^{2}$. We generate 10K training
samples and 10K testing from this distribution.

##### Image datasets.

We consider two image datasets including cifar10 [40] and celeba [41]. These
datasets are publicly available and widely used in the literature of
generative models. We use the official train/val/test splits for both
datasets. The resolution of cifar10 is $3\times 32\times 32$. For celeba, we
pre-process images by first taking a $148\times 148$ center crop and then
resizing to the $3\times 64\times 64$ resolution.

###  D.2 Software and Computational Resources

We use NVIDIA P100 and A100 GPUs for the experiments, with 16GB and 80GB of
memory respectively. All models are trained on a single GPU except for the
experiments with nvae model [73], where we employ two A100 GPUs. We use
PyTorch [52] for the implementation of the models and the experiments. Our
experiments with vaes and nfs are relied on the pythae [6] and normflows [68]
libraries, respectively.

###  D.3 Training Details

####  D.3.1 Toy examples.

In the experiments on synthetic datasets, we use a real-nvp flow [14] with $5$
affine coupling layers consisting of $2$ hidden layers of $64$ units each. We
train the model for $20000$ itereations using an Adam optimizer [33] with a
learning rate of $5\cdot 10^{-4}$ and a mini-batch size of $256$.

####  D.3.2 Imaging experiments.

#####  real-nvp.

We use the multi-scale architecture with deep convolutional residual networks
in the coupling layers as described in [14]. For the cifar10 dataset, we use
$4$ residual blocks with $32$ hidden feature maps for the first coupling
layers with checkerboard masking. For the celeba dataset, $2$ resdiual blocks
are employed. We use an Adam optimizer [33] with a learning rate of $10^{-3}$
and a mini-batch size of $64$. We train the model for $100$ and $80$ epochs on
the cifar10 and celeba datasets, respectively. For the mollification training,
we perturb the data for $50$ and $40$ epochs for cifar10 and celeba,
respectively.

#####  glow.

We use a multi-scale architecture as described in [36]. The architecture has a
depth level of $K=20$, and a number of levels $L=3$. We use the AdaMax [33]
optimizer with a learning rate of $3\cdot 10^{-4}$ and a mini-batch size of
$64$. We train the model for $80$ and $40$ epochs on the cifar10 and celeba
datasets, respectively. For the mollification training, we perturb the data
for $50$ and $20$ epochs for cifar10 and celeba, respectively.

#####  vaes.

Table 4: Neural network architectures used for vaes in our experiments. Here,
$\textsc{Conv}_{(n,s,p)}$ and $\textsc{ConvT}_{(n,s,p)}$ respectively denotes
convolutional layer and transposed convolutional layers with $n$ filters, a
stride of $s$ and a padding of $p$, whereas $\textsc{FC}_{n}$ represents a
fully-connected layer with $n$ units, and BN denotes a batch-normalization
layer.

|  Cifar10 |  CelebA
---|---|---
Encoder: |  ${{\bm{\mathbf{x}}}}\in\mathbb{R}^{3{\times}32{\times}32}$
$\rightarrow\text{Conv}_{(128,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Conv}_{(256,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Conv}_{(512,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Conv}_{(1024,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Flatten}\rightarrow\text{FC}_{256{\times}2}$ |  ${{\bm{\mathbf{x}}}}\in\mathbb{R}^{3{\times}64{\times}64}$
$\rightarrow\text{Conv}_{(128,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Conv}_{(256,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Conv}_{(512,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Conv}_{(1024,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\quad\rightarrow\text{Flatten}\rightarrow\text{FC}_{256{\times}2}$
Decoder: |  ${{\bm{\mathbf{z}}}}\in\mathbb{R}^{256}\rightarrow\text{FC}_{8{\times}8{\times}1024}$
$\rightarrow\text{ConvT}_{(512,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\rightarrow\text{ConvT}_{(256,4,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\rightarrow\text{ConvT}_{(3,4,1)}$ |  ${{\bm{\mathbf{z}}}}\in\mathbb{R}^{256}\rightarrow\text{FC}_{8{\times}8{\times}1024}$
$\rightarrow\text{ConvT}_{(512,5,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\rightarrow\text{ConvT}_{(256,5,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\rightarrow\text{ConvT}_{(128,5,2)}\rightarrow\text{BN}\rightarrow\text{ReLU}$
$\rightarrow\text{ConvT}_{(3,4,1)}$

We use convolutional networks for both the encoder and decoder of vaes [34,
58]. Table 4 shows the details of the network architectures. We use an Adam
optimizer [33] with a learning rate of $3\cdot 10^{-4}$ and a mini-batch size
of $128$. We train the model for $200$ and $100$ epochs on the cifar10 and
celeba datasets, respectively. For the mollification training, we perturb the
data for $100$ and $50$ epochs for cifar10 and celeba, respectively. The
addtional details of the variants of vaes are as follows:

  * •

vae-iaf [37]: We use a $3$-layer made [20] with $128$ hidden units and relu
activation for each layer and stack $2$ blocks of Masked Autoregressive Flow
to create the flow for approximating the posterior.

  * •

$\beta$-vae [25]: We use a coefficient of $\beta=0.1$ for the Kullback-Leibler
divergence (kl) term in the elbo objective.

  * •

iwae [4]: We use a number of importance samples of $K=5$.

  * •

hvae [5]: We set the number of leapfrog steps to used in the integrator to
$1$. The leapfrog step size is adaptive with an initial value of $0.001$

#####  nvae.

We use the default network architecture as described in [73]. We train the
model on the cifar10 for $300$ epochs with an AdaMax optimizer [33] with a
learning rate of $10^{-3}$ and a mini-batch size of $200$. For the
mollification training, we perturb the data for first $150$ epochs.

###  D.4 Evaluation Metrics

##### Maximum Mean Discrepancy.

The mmd between two distributions $p_{\text{data}}$ and
$p_{{{\bm{\mathbf{\theta}}}}}$ is defined as follows [21]:

$\text{MMD}(p_{\text{data}},p_{{{\bm{\mathbf{\theta}}}}})=\sup_{\|h\|_{\mathcal{H}}\leq
1}\Big{[}\mathbb{E}_{p_{\text{data}}}[h]-\mathbb{E}_{p_{{{\bm{\mathbf{\theta}}}}}}[h]\Big{]},$
(15)

where $\mathcal{H}$ denotes a reproducing kernel Hilbert space (rkhs) induced
by a characteristic kernel $K$. The mmd has the closed form:

$\text{MMD}^{2}(p_{\text{data}},p_{{{\bm{\mathbf{\theta}}}}})=\mathbb{E}_{{{\bm{\mathbf{x}}}},{{\bm{\mathbf{x}}}}^{\prime}\sim
p_{\text{data}}}[K({{\bm{\mathbf{x}}}},{{\bm{\mathbf{x}}}}^{\prime})]+\mathbb{E}_{{{\bm{\mathbf{x}}}},{{\bm{\mathbf{x}}}}^{\prime}\sim
p_{{{\bm{\mathbf{\theta}}}}}}[K({{\bm{\mathbf{x}}}},{{\bm{\mathbf{x}}}}^{\prime})]-2\mathbb{E}_{{{\bm{\mathbf{x}}}}\sim
p_{\text{data}},{{\bm{\mathbf{x}}}}^{\prime}\sim
p_{{{\bm{\mathbf{\theta}}}}}}[K({{\bm{\mathbf{x}}}},{{\bm{\mathbf{x}}}}^{\prime})],$
(16)

which can be estimated by using samples from $p_{\text{data}}$ and
$p_{{{\bm{\mathbf{\theta}}}}}$. In our experiments, we use 10K test samples
from the true data distribution $p_{\text{data}}$ and 10K samples from the
model distribution $p_{{{\bm{\mathbf{\theta}}}}}$ for estimating the mmd
score.

We employ the popular rbf kernel for the mmd, which is defined as follows:

$K({{\bm{\mathbf{x}}}},{{\bm{\mathbf{x}}}}^{\prime})=\sigma^{2}\exp\Big(-\frac{\|{{\bm{\mathbf{x}}}}-{{\bm{\mathbf{x}}}}^{\prime}\|^{2}}{2l}\Big{missing}),$
(17)

with a lengthscale $l=1$ and variance $\sigma^{2}=10^{-4}$.

##### FID score.

To assess the quality of the generated images, we employed the widely used
Fréchet Inception Distance [24]. The FID measures the Fréchet distance between
two multivariate Gaussian distributions, one representing the generated
samples and the other representing the real data samples. By comparing their
distribution statistics, we can assess the similarity between the generated
and real data distributions. The FID score is defined as follows:

$\displaystyle\text{FID}=\|\mu_{\text{real}}-\mu_{\text{gen}}\|^{2}+\text{Tr}(\Sigma_{\text{real}}+\Sigma_{\text{gen}}-2\sqrt{\Sigma_{\text{real}}\Sigma_{\text{gen}}}).$
(18)

The distribution statistics are obtained from the $2048$-dimensional
activations of the pool3 layer of an Inception-v3 network. We use the pytorch-
fid 333https://github.com/mseitzer/pytorch-fid library for calculating the FID
score in our experiments.

##  Appendix E Addtional Results

Table 5 and Table 6 illustrate uncurated samples from the trained models. Fig.
9 and Fig. 10 show the progression of FID scores during training on the
cifar10 and celeba datasets, respectively.

Table 5: Uncurated samples from the models trained on the cifar10 dataset.

|  Vanilla |  Gauss. Mollification |  Blur. Mollification
---|---|---|---
real-nvp |  |  |
glow |  |  |
vae |  |  |
vae-iaf |  |  |
iwae |  |  |
$\beta$-vae |  |  |
hvae |  |  |

Table 6: Uncurated samples from the models trained on the celeba dataset.

|  Vanilla |  Gauss. Mollification |  Blur. Mollification
---|---|---|---
real-nvp |  |  |
glow |  |  |
vae |  |  |
vae-iaf |  |  |
iwae |  |  |
$\beta$-vae |  |  |
hvae |  |  |

$\mathsf{0}$$\mathsf{50}$$\mathsf{100}$$\mathsf{150}$$\mathsf{200}$$\mathsf{5.07}$$\mathsf{5.53}$$\mathsf{5.99}$EpochFID
[log] ($\displaystyle\leftarrow$)VAE

$\mathsf{0}$$\mathsf{50}$$\mathsf{100}$$\mathsf{150}$$\mathsf{200}$$\mathsf{4.61}$$\mathsf{5.07}$$\mathsf{5.53}$$\mathsf{5.99}$EpochFID
[log] ($\displaystyle\leftarrow$)$\displaystyle\beta$-VAE

$\mathsf{0}$$\mathsf{50}$$\mathsf{100}$$\mathsf{150}$$\mathsf{200}$$\mathsf{5.07}$$\mathsf{5.53}$$\mathsf{5.99}$EpochFID
[log] ($\displaystyle\leftarrow$)IWAE

$\mathsf{0}$$\mathsf{50}$$\mathsf{100}$$\mathsf{150}$$\mathsf{200}$$\mathsf{5.07}$$\mathsf{5.53}$$\mathsf{5.99}$EpochFID
[log] ($\displaystyle\leftarrow$)VAE-IAF

$\mathsf{0}$$\mathsf{50}$$\mathsf{100}$$\mathsf{150}$$\mathsf{200}$$\mathsf{5.07}$$\mathsf{5.53}$$\mathsf{5.99}$EpochFID
[log] ($\displaystyle\leftarrow$)HVAE

$\mathsf{0}$$\mathsf{20}$$\mathsf{40}$$\mathsf{60}$$\mathsf{80}$$\mathsf{4.61}$$\mathsf{5.76}$EpochFID
[log] ($\displaystyle\leftarrow$)GLOW

Figure 9: The progression of FID scores during training on the cifar10
dataset.

$\mathsf{20}$$\mathsf{40}$$\mathsf{60}$$\mathsf{80}$$\mathsf{100}$$\mathsf{4.61}$$\mathsf{5.76}$EpochFID
[log] ($\displaystyle\leftarrow$)VAE

$\mathsf{20}$$\mathsf{40}$$\mathsf{60}$$\mathsf{80}$$\mathsf{100}$$\mathsf{4.61}$$\mathsf{5.76}$EpochFID
[log] ($\displaystyle\leftarrow$)$\displaystyle\beta$-VAE

$\mathsf{20}$$\mathsf{40}$$\mathsf{60}$$\mathsf{80}$$\mathsf{100}$$\mathsf{4.61}$$\mathsf{5.76}$EpochFID
[log] ($\displaystyle\leftarrow$)IWAE

$\mathsf{20}$$\mathsf{40}$$\mathsf{60}$$\mathsf{80}$$\mathsf{100}$$\mathsf{4.61}$$\mathsf{5.76}$EpochFID
[log] ($\displaystyle\leftarrow$)VAE-IAF

$\mathsf{20}$$\mathsf{40}$$\mathsf{60}$$\mathsf{80}$$\mathsf{100}$$\mathsf{4.61}$$\mathsf{5.76}$EpochFID
[log] ($\displaystyle\leftarrow$)HVAE

$\mathsf{10}$$\mathsf{20}$$\mathsf{30}$$\mathsf{40}$$\mathsf{4.61}$$\mathsf{5.76}$EpochFID
[log] ($\displaystyle\leftarrow$)GLOW

Figure 10: The progression of FID scores during training on the celeba
dataset.