# Contractive diffusion probabilistic models

Wenpin Tang  Department of Industrial Engineering and Operations Research,
Columbia University<EMAIL_ADDRESS>and  Hanyang Zhao  Department of
Industrial Engineering and Operations Research, Columbia University.
<EMAIL_ADDRESS>
###### Abstract.

Diffusion probabilistic models (DPMs) have emerged as a promising technology
in generative modeling. The success of DPMs relies on two ingredients: time
reversal of Markov diffusion processes and score matching. Most existing work
implicitly assumes that score matching is close to perfect, while this
assumption is questionable. In view of possibly unguaranteed score matching,
we propose a new criterion – the contraction of backward sampling in the
design of DPMs. This leads to a novel class of contractive DPMs (CDPMs),
including contractive Ornstein-Uhlenbeck (OU) processes and contractive sub-
variance preserving (sub-VP) stochastic differential equations (SDEs). The key
insight is that the contraction in the backward process narrows score matching
errors, as well as discretization error. Thus, the proposed CDPMs are robust
to both sources of error. Our proposal is supported by theoretical results,
and is corroborated by experiments. Notably, contractive sub-VP shows the best
performance among all known SDE-based DPMs on the CIFAR-10 dataset.

Key words: Contraction, diffusion probabilistic models, discretization,
generative models, image synthesis, sampling, score matching, stochastic
differential equations.

##  1\. Introduction

Over the past decade, generative models have achieved remarkable success in
creating instances across a wide variety of data modalities, including images
[23, 7, 47], audio [44, 6], video [27], and text [61, 8, 65]. Diffusion
probabilistic models (DPMs) have emerged as a promising generative approach
that is observed to outperform generative adversarial nets on image and audio
synthesis [16, 33], and underpins the major accomplishment in text-to-image
creators such as DALL$\cdot$E 2 [46] and Stable Diffusion [49]. The concept of
DPMs finds its roots in energy-based models [50], and is popularized by [54,
57, 25] in an attempt to produce from noise new samples (e.g. images, audio,
text) that resemble the target data, while maintain diversity. See [64] for a
comprehensive review on DPMs.

DPMs relies on forward-backward Markov processes. The forward process starts
with the target data distribution, and runs for some time until the signal is
destroyed – this gives rise to noise. The backward process is then initiated
with the noise, and reverses the forward process in time to generate samples
whose distribution is close to the target distribution. In [54, 25, 2], DPMs
are discrete time-indexed Markov chains; [57, 53, 12] model DPMs in continuous
time as stochastic differential equations (SDEs). Nevertheless, there is no
conceptual distinction between discrete and continuous DPMs as continuous DPMs
can be viewed as the continuum limits of discrete DPMs, and discrete DPMs are
time discretization of continuous DPMs. In this paper, we adopt a continuous
time perspective, and algorithms are derived by discretization. Being
concrete:

  * •

The forward process $(X_{t},\,0\leq t\leq T)$ is governed by the SDE:

$dX_{t}=b(t,X_{t})dt+\sigma(t)dB_{t},\quad\mbox{with }X_{0}\sim
p_{\tiny\mbox{data}}(\cdot),$

where $(B_{t},\,t\geq 0)$ is Brownian motion in $\mathbb{R}^{d}$,
$b:\mathbb{R}_{+}\times\mathbb{R}^{d}\to\mathbb{R}^{d}$ and
$\sigma:\mathbb{R}_{+}\to\mathbb{R}_{+}$ are model parameters to be designed,
and $p_{\tiny\mbox{data}}(\cdot)$ is the target data distribution.

  * •

The backward process $(\overline{X}_{t},\,0\leq t\leq T)$ is governed by the
SDE:

$d\overline{X}_{t}=\overline{b}(t,\overline{X}_{t})dt+\overline{\sigma}(t)d\overline{B}_{t},\quad\mbox{with
}X_{0}\sim p_{\tiny\mbox{noise}}(\cdot),$

where $(\overline{B}_{t},\,t\geq 0)$ is Brownian motion in $\mathbb{R}^{d}$,
$\overline{b}(t,x)$ and $\overline{\sigma}(\cdot,\cdot)$ are some functions,
and $p_{\tiny\mbox{noise}}(\cdot)$ is the noise distribution that does not
depend on $p_{\tiny\mbox{data}}(\cdot)$.

It is easy to see that the forward process transforms data to noise (with a
suitable choice of $b(\cdot,\cdot)$ and $\sigma(\cdot)$). What is miraculous
is how the backward process recovers the target data distribution
$\overline{X}_{T}\approx p_{\tiny\mbox{data}}(\cdot)$ from noise
$p_{\tiny\mbox{noise}}(\cdot)$. The secret consists of two key ingredients:

  1. (1)

Time reversal of diffusion processes: The backward process has an explicit
form, where $\overline{b}(t,x)$ depends on $b(T-t,x)$, $\sigma(T-t)$ and
Stein’s score functions (the gradients of the log marginal density of the
forward process), and $\overline{\sigma}(t)=\sigma(T-t)$ [24, 9].

  2. (2)

Score matching: The backward process is easily sampled given $b(\cdot,\cdot)$,
$\sigma(\cdot)$ and score functions. The trick is to learn score functions via
forward sampling, referred to as score-based DPMs. Leaning score functions,
also known as score matching, features in a body of active research [26, 62,
56, 32].

In a nutshell, DPMs combine forward (score) learning/matching with backward
sampling.

As is clear, a DPM is specified by the pair
$(b(\cdot,\,\cdot),\sigma(\cdot))$. Popular examples include Ornstein-
Uhlenbeck (OU) processes [15], variance exploding (VE) SDEs, variance
preserving (VP) SDEs, and sub-variance preserving (sub-VP) SDEs [57] (see
Section 2 for definitions). Especially, VE and VP SDEs are the continuum
limits of score matching with Langevin dynamics (SMLD) [54] and denoising
diffusion probabilistic models (DDPMs) [25] respectively. A natural question
is:

$\mbox{\bf How do we design the pair
}\boldsymbol{(b(\cdot,\,\cdot),\sigma(\cdot))}\mbox{ \bf for a DPM?}$

The first rule of thumb, which is satisfied by all the aforementioned
examples, is:

Rule 1. The conditional distributions of $X_{t}\,|\,X_{0}$ are easy to sample
(e.g. Gaussian).

This rule allows efficient sampling in the forward process for score matching
via stochastic optimization. Notably, Rule 1 provides only instructions on the
forward learning step, but no requirement on backward sampling.

The purpose of this paper is to put forward a principle regarding backward
sampling in the design of DPMs. Our proposal is:

Rule 2. The backward process  $\overline{X}$ is contractive.

Roughly speaking, being contractive indicates that the process tends to be
confined, or converge (see Section 3 for explanations). DPMs that comply with
Rule 1 and 2 are called contractive DPMs (CDPMs). Here we abuse the term
contractive DPMs by meaning that their backward processes, rather than the
forward processes, exhibit contractive properties. At a high level, the
contraction of the backward process will prevent score matching errors, which
may be wild, from expanding over the time. The contributions of this work are
summarized as follows.

  * •

Methodology: We propose a new criterion (Rule 2) for designing DPMs. This
naturally leads to a novel class of DPMs, including contractive OU processes
and contractive sub-VP SDEs (see Section 4). The idea of requiring the
backward process to be contractive stems from sampling theory of SDEs, so our
methodology is theory-oriented. To the best of our knowledge, this is the
first paper to integrate contraction into the design of DPMs, with both
theoretical guarantees and good empirical performance.

  * •

Theory: We prove Wasserstein bounds between contractive DPM samplers and the
target data distribution. While most previous work (e.g. [12, 15, 35]) focused
on Kullback–Leibler (KL) or total variation bounds for OU processes, we
consider the Wasserstein metric because it has shown to align with human
judgment on image similarity [5], and the standard evaluation metric – Fréchet
inception distance (FID) is based on Wasserstein distance. Early work [15, 34]
gave Wasserstein bounds for the existing DPMs (OU processes, VE and VP SDEs)
with exponential dependence on $T$. This was improved in recent studies [12,
36, 22] under various assumptions of $p_{\tiny\mbox{data}}(\cdot)$, where the
bounds are typically of form:

$\underbrace{(\small\mbox{noise inaccuracy})\cdot
e^{-T}}_{\tiny\mbox{initialization error}}+\underbrace{(\small\mbox{score
mismatch})\cdot\texttt{Poly}(T)}_{\tiny\mbox{score
error}}+\underbrace{\texttt{Poly}(\small\mbox{step
size})\cdot\texttt{Poly}(T)}_{\tiny\mbox{discretization error}},$

with $\texttt{Poly}(\cdot)$ referring to polynomial in the variable. Our
result gives a Wasserstein bound for CDPMs:

$\underbrace{(\small\mbox{noise inaccuracy})\cdot
e^{-T}}_{\tiny\mbox{initialization error}}+\underbrace{(\small\mbox{score
mismatch})\cdot(1-e^{-T})}_{\tiny\mbox{score
error}}+\underbrace{\texttt{Poly}(\small\mbox{step
size})}_{\tiny\mbox{discretization error}}.$

Score matching is often trained using blackbox function approximations, and
the errors incurred in this step may be large. So CDPMs are designed to be
robust to score mismatch and discretization, at the cost of possible
initialization bias.

  * •

Experiments: We apply the proposed CDPMs to both synthetic and real data. In
dimension one, we compare contractive OU with OU, VE, VP and sub-VP. Our
result shows that contractive OU consistently outperforms OU, and is robust to
different target distributions. On the CIFAR-10 data, we obtain an FID score
of 2.47 and an inception score of 10.18 for contractive sub-VP, which is
comparable to the performance of VE with an FID score of 2.47 and an inception
score of 9.68, and outperforms other known SDE-based diffusion models.

Literature review. To conclude the introduction, we give a review on the
relevant literature. In the context of generative modeling, DPMs were
initiated by [54] (SMLD) and [25] (DDPM) using forward-backward Markov chains.
The work [57, 53] unified the previous models via a score-based SDE framework,
which also led to deterministic ordinary differential equation (ODE) samplers.
Since then the field has exploded, and lots of work has been built upon DPMs
and their variants. Examples include DPMs in constrained domains [19, 39, 48,
17], DPMs on manifolds [43, 14, 10], DPMs in graphic models [40], variational
DPMs [30, 59] and consistency models [52], just to name a few. Early theory
[15, 14, 34] established the convergence of DPMs with exponential dependence
on time horizon $T$ and dimension $d$. Recently, polynomial convergence of
various DPMs has been proved for stochastic samplers [12, 3, 38, 22, 35, 36]
and deterministic samplers [4, 11, 13, 38], under suitable conditions on the
target data distribution.

The remainder of the paper is organized as follows. In Section 2, we provide
background on DPMs and score matching techniques. Theoretical results for
CDPMs are presented in Section 3, and experiments are reported in Section 4.
We conclude with Section 5.

##  2\. Background

###  2.1. Diffusion models

Let’s explain DPMs in the context of SDEs. Consider the forward process:

$dX_{t}=b(t,X_{t})dt+\sigma(t)dB_{t},\quad\mbox{with }X_{0}\sim
p_{\tiny\mbox{data}}(\cdot),$ (2.1)

where $b:\mathbb{R}_{+}\times\mathbb{R}^{d}\to\mathbb{R}^{d}$ and
$\sigma:\mathbb{R}_{+}\to\mathbb{R}_{+}$ are model parameters. Some conditions
on $b(\cdot,\cdot)$ and $\sigma(\cdot)$ are required so that the SDE (2.1) is
at least well-defined (see [28, Chapter 5], [58, Section 3.1]). Let
$p(t,\cdot)$ be the probability density of $X_{t}$.

Set $T>0$ to be fixed, and run the SDE (2.1) until time $T$ to get $X_{T}\sim
p(T,\cdot)$. Now if we start with $p(T,\cdot)$ and run the process $X$
backward, then we can generate a copy of
$p(0,\,\cdot)=p_{\tiny\mbox{data}}(\cdot)$. Being precise, consider the time
reversal $\overline{X}_{t}:=X_{T-t}$ for $0\leq t\leq T$. Assuming that
$\overline{X}$ also satisfies an SDE, we can implement the backward procedure
by

$d\overline{X}_{t}=\bar{b}(t,\overline{X}_{t})dt+\bar{\sigma}(t)d\overline{B}_{t},\quad\mbox{with
}\overline{X}_{0}\sim p(T,\cdot),$

which samples the desired $\overline{X}_{T}\sim p_{\tiny\mbox{data}}(\cdot)$
at time $T$. Note that the distribution $p(T,\cdot)$ depends on the target
distribution $p_{\tiny\mbox{data}}(\cdot)$. The idea of DPMs is, however, to
generate samples from noise. So we need to replace $p(T,\cdot)$ by a proxy
$p_{\tiny\mbox{noise}}(\cdot)$ that is independent of
$p_{\tiny\mbox{data}}(\cdot)$. This yields the backward process:

$d\overline{X}_{t}=\bar{b}(t,\overline{X}_{t})dt+\bar{\sigma}(t)d\overline{B}_{t},\quad\mbox{with
}\overline{X}_{0}\sim p_{\tiny\mbox{noise}}(\cdot).$ (2.2)

There are two questions:

  1. (1)

How can we choose $p_{\tiny\mbox{noise}}(\cdot)$?

  2. (2)

What are the parameters $\bar{b}(\cdot,\cdot)$ and $\bar{\sigma}(\cdot)$?

For (1), the noise $p_{\tiny\mbox{noise}}(\cdot)$ is often derived by
decoupling the conditional distribution of $X_{t}\,|\,X_{0}$ from $X_{0}$, as
we will illustrate with examples later. It is expected that the closer the
distributions $p(T,\cdot)$ and $p_{\tiny\mbox{noise}}(\cdot)$ are, the closer
the distribution of $\overline{X}_{T}$ sampled from (2.2) is to
$p_{\tiny\mbox{data}}(\cdot)$. For (2), it relies on the following result on
the time reversal of SDEs [1, 24].

######  Theorem 2.1.

Under suitable conditions on $b(\cdot,\cdot)$, $\sigma(\cdot)$ and
$\\{p(t,\cdot)\\}_{0\leq t\leq T}$, we have

$\overline{\sigma}(t)=\sigma(T-t),\quad\overline{b}(t,x)=-b(T-t,x)+\sigma^{2}(T-t)\nabla\log
p(T-t,x),$ (2.3)

where the term $\nabla\log p(\cdot,\cdot)$ is called Stein’s score function.

We give a derivation of Theorem 2.1 with further references in Appendix A. As
a consequence, the backward process is:

$d\overline{X}_{t}=\left(-b(T-t,\overline{X}_{t})+\sigma^{2}(T-t)\nabla\log
p(T-t,\overline{X}_{t})\right)dt\\\ +\sigma(T-t)d\overline{B}_{t}.$ (2.4)

Since $b(\cdot,\cdot)$ and $\sigma(\cdot)$ are chosen in advance, all but the
term $\nabla\log p(T-t,\overline{X}_{t})$ in (2.4) are available.

Now we present a few examples of DPMs.

  * •

OU processes: $b(t,x)=\theta(\mu-x)$ with $\theta>0$, $\mu\in\mathbb{R}^{d}$;
$\sigma(t)=\sigma>0$. The distribution of $(X_{t}\,|\,X_{0}=x)$ is
$\mathcal{N}(\mu+(x-\mu)e^{-\theta t},\frac{\sigma^{2}}{2\theta}(1-e^{-2\theta
t})I)$, which is approximately $\mathcal{N}(\mu,\frac{\sigma^{2}}{2\theta}I)$
as $t$ is large. The backward process specializes to

$d\overline{X}_{t}=(\theta(\overline{X}_{t}-\mu)+\sigma^{2}\nabla\log
p(T-t,\overline{X}_{t}))dt+\sigma
dB_{t},\,\,\overline{X}_{0}\sim\mathcal{N}\left(\mu,\frac{\sigma^{2}}{2\theta}I\right).$
(2.5)

  * •

VE SDEs: $b(t,x)=0$ and
$\sigma(t)=\sigma_{\min}\left(\frac{\sigma_{\max}}{\sigma_{\min}}\right)^{\frac{t}{T}}\sqrt{\frac{2}{T}\log\frac{\sigma_{\max}}{\sigma_{\min}}}$
with $\sigma_{\min}\ll\sigma_{\max}$. The distribution of
$(X_{t}\,|\,X_{0}=x)$ is
$\mathcal{N}\left(x,\sigma^{2}_{\min}\left(\left(\frac{\sigma_{\max}}{\sigma_{\min}}\right)^{\frac{2t}{T}}-1\right)I\right)$,
which can be approximated by $\mathcal{N}(0,\sigma^{2}_{\max}I)$ at $t=T$. The
backward process is:

$d\overline{X}_{t}=\sigma^{2}(T-t))\nabla\log
p(T-t,\overline{X}_{t})+\sigma(T-t)dB_{t},\,\,\overline{X}_{0}\sim\mathcal{N}(0,\sigma^{2}_{\max}I).$
(2.6)

  * •

VP SDEs: $b(t,x)=-\frac{1}{2}\beta(t)x$ and $\sigma(t)=\sqrt{\beta(t)}$, where
$\beta(t):=\beta_{\min}+\frac{t}{T}(\beta_{\max}-\beta_{\min})$ with
$\beta_{\min}\ll\beta_{\max}$. The distribution of $(X_{t}\,|\,X_{0}=x)$ is

$\mathcal{N}(e^{-\frac{t^{2}}{4T}(\beta_{\max}-\beta_{\min})-\frac{t}{2}\beta_{\min}}x,(1-e^{-\frac{t^{2}}{2T}(\beta_{\max}-\beta_{\min})-t\beta_{\min}})I),$

which can be approximated by $\mathcal{N}(0,I)$ at $t=T$. The backward process
is:

$\displaystyle d\overline{X}_{t}$
$\displaystyle=\bigg{(}\frac{1}{2}\beta(T-t)\overline{X}_{t}+\beta(T-t)\nabla\log
p(T-t,\overline{X}_{t})\bigg{)}dt$ (2.7)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\quad+\sqrt{\beta(T-t))}dB_{t},\,\,\overline{X}_{0}\sim\mathcal{N}(0,I).$

  * •

Sub-VP SDEs: $b(t,x)=-\frac{1}{2}\beta(t)x$ and
$\sigma(t)=\sqrt{\beta(t)(1-e^{-2\int_{0}^{t}\beta(s)ds})}$. The distribution
of $(X_{t}\,|\,X_{0}=x)$ is
$\mathcal{N}(e^{-\frac{t^{2}}{4T}(\beta_{\max}-\beta_{\min})-\frac{t}{2}\beta_{\min}}x,(1-e^{-\frac{t^{2}}{2T}(\beta_{\max}-\beta_{\min})-t\beta_{\min}})^{2}I)$,
which can be approximated by $\mathcal{N}(0,I)$ at $t=T$. The backward process
is:

$\displaystyle d\overline{X}_{t}$
$\displaystyle=\bigg{(}\frac{1}{2}\beta(T-t)\overline{X}_{t}+\beta(T-t)(1-\gamma(T-t))\nabla\log
p(T-t,\overline{X}_{t})\bigg{)}dt$ (2.8)
$\displaystyle\qquad\qquad\qquad\qquad+\sqrt{\beta(T-t)(1-\gamma(T-t))}dB_{t},\,\,\overline{X}_{0}\sim\mathcal{N}(0,I),$

where
$\gamma(t):=e^{-2\int_{0}^{t}\beta(s)ds}=e^{-\frac{t^{2}}{T}(\beta_{\max}-\beta_{\min})-2t\beta_{\min}}$.

###  2.2. Score matching

As previously mentioned, we need to compute the score function $\nabla\log
p(t,x)$ for backward sampling. The idea from recently developed score-based
generative modeling [25, 54, 57] is to estimate $\nabla\log p(t,x)$ by
function approximations. More precisely, denote by
$\\{s_{\theta}(t,x)\\}_{\theta}$ a family of functions on
$\mathbb{R}_{+}\times\mathbb{R}^{d}$ parametrized by $\theta$. Fixing $t$, the
goal is to solve the problem:

$\min_{\theta}\mathcal{J}_{\text{ESM}}(\theta):=\mathbb{E}_{p(t,\cdot)}|s_{\theta}(t,X)-\nabla\log
p(t,X)|^{2},$ (2.9)

which is known as the explicit score matching (ESM) objective. Again the
stochastic optimization (2.9) is far-fetched since the scores $\nabla\log
p(t,x)$ are not available. Interestingly, this problem has been studied in the
context of estimating statistical models with unknown normalizing constant.
The following result [26] shows that the score matching problem (2.9) can be
recast into a feasible stochastic optimization with no $\nabla\log
p(t,X)$-term, known as implicit score matching (ISM) objective.

######  Theorem 2.2.

Let
$\mathcal{J}_{\text{ISM}}(\theta):=\mathbb{E}_{p(t,\cdot)}\left[|s_{\theta}(t,X)|^{2}+2\,\nabla\cdot
s_{\theta}(t,X)\right]$. Under suitable conditions on $s_{\theta}$, we have
$\mathcal{J}_{\text{ISM}}(\theta)=\mathcal{J}_{\text{ESM}}(\theta)+C$ for some
$C$ independent of $\theta$. Consequently, the minimum point of
$\mathcal{J}_{\text{ISM}}$ and that of $\mathcal{J}_{\text{ESM}}$ coincide.

We give a proof of this theorem in Appendix B.1 for completeness. In practice,
the score matching problem with a continuous weighted combination is
considered:

$\min_{\theta}\tilde{\mathcal{J}}_{\text{ESM}}(\theta)=\mathbb{E}_{t\in\mathcal{U}(0,T)}\mathbb{E}_{p(t,\cdot)}\left[\lambda(t)|s_{\theta}(t,X)-\nabla\log
p(t,X)|^{2}\right].$ (2.10)

where $\mathcal{U}(0,T)$ denotes a uniform distribution on $[0,T]$, and
$\lambda:\mathbb{R}\rightarrow\mathbb{R}_{+}$ is a positive weighting
function. We can alternate the inside part by ISM to solve the problem:

$\min_{\theta}\widetilde{\mathcal{J}}_{\text{ISM}}(\theta)=\mathbb{E}_{t\in\mathcal{U}(0,T)}\mathbb{E}_{p(t,\cdot)}\left[\lambda(t)\left(|s_{\theta}(t,X)|^{2}+2\,\nabla\cdot
s_{\theta}(t,X)\right)\right].$ (2.11)

However, the problem (2.11) can still be computationally costly when the
dimension $d$ is large. Using a neural network for $s_{\theta}(t,x)$, we need
to conduct $d$ times of backward propagation of all parameters to compute
$\nabla\cdot s_{\theta}(t,x)$. This means that the computation of the gradient
scales linearly with the dimension, thus making the gradient descent methods
not efficient for solving the problem (2.11) with respect to examples such as
image data in high dimension. Denoising score matching (DSM) [62] serves as a
scalable alternative:

$\tilde{\mathcal{J}}_{\text{DSM}}(\theta)=\mathbb{E}_{t\sim\mathcal{U}(0,T)}\left\\{\lambda(t)\mathbb{E}_{X_{0}\sim
p_{data}(\cdot)}\mathbb{E}_{X_{t}\mid
X_{0}}\left[\left|s_{\theta}(t,X(t))-\nabla_{X_{t}}\log p(t,X_{t}\mid
X_{0})\right|^{2}\right]\right\\}.$

Its equivalent form and other methods such as sliced score matching [56] are
discussed in Appendix B.2. Now we replace $\nabla p(t,x)$ with the matched
score $s_{\theta}(t,x)$ in (2.2) to get the backward process:

$d\overline{X}_{t}=\left(-b(T-t,\overline{X}_{t})+\sigma^{2}(T-t)\,s_{\theta}(T-t,\overline{X}_{t})\right)dt+\sigma(T-t)d\overline{B}_{t},\,\,\overline{X}_{0}\sim
p_{\tiny\mbox{noise}}(\cdot).$ (2.12)

##  3\. Theory for contractive DPMs

In this section, we introduce the idea of CDPMs, and present supportive
theoretical results. For a DPM specified by (2.1)-(2.12), we also consider the
Euler-Maruyama discretization of its backward process $\overline{X}$. Fix
$\delta>0$ as the step size, and set $t_{k}:=k\delta$ for
$k=0,\ldots,N:=T/\delta$. Let $\widehat{X}_{0}=\overline{X}_{0}$, and

$\displaystyle\widehat{X}_{k}:=\widehat{X}_{k-1}+(-b(T-t_{k},\widehat{X}_{k-1})+$
$\displaystyle a(T-t_{k-1})s_{\theta}(T-t_{k-1},\widehat{X}_{k-1}))\delta$
(3.1) $\displaystyle+\sigma(T-t_{k-1})(B_{t_{k}}-B_{t_{k-1}}),\quad\mbox{for
}k=1,\ldots,N.$

Our goal is to bound the Wasserstein-2 distance
$W_{2}(p_{\tiny\mbox{data}}(\cdot),\widehat{X}_{N})$. Clearly,

$W_{2}(p_{\tiny\mbox{data}}(\cdot),\widehat{X}_{N})\leq
W_{2}(p_{\tiny\mbox{data}}(\cdot),\overline{X}_{T})+\left(\mathbb{E}|\overline{X}_{T}-\widehat{X}_{N}|^{2}\right)^{\frac{1}{2}},$
(3.2)

where the first term on the right side of (3.2) is the sampling error at the
continuous level, and the second term is the discretization error. We will
study these two terms in the next two subsections, where CDPMs are justifed.

###  3.1. Sampling error in continuous time

We are concerned with the term
$W_{2}(p_{\tiny\mbox{data}}(\cdot),\overline{X}_{T})$. Existing work [12, 36,
15] established $W_{2}$ bounds for OU processes under a bounded support
assumption. Closer to our result (and proof) is the concurrent work [22],
where a $W_{2}$ bound is derived for a class of DPMs with $b(t,x)$ separable
in $t$ and $x$, under a strongly log-concavity assumption.

######  Assumption 3.1.

The following conditions hold:

  1. (1)

There exists $r_{b}:[0,T]\to\mathbb{R}$ such that
$(x-x^{\prime})\cdot(b(t,x)-b(t,x^{\prime}))\geq r_{b}(t)|x-x^{\prime}|^{2}$
for all $t$ and $x,x^{\prime}$.

  2. (2)

There exists $L>0$ such that $|\nabla\log p(t,x)-\nabla\log
p(t,x^{\prime})|\leq L|x-x^{\prime}|$ for all $t$ and $x,x^{\prime}$.

  3. (3)

There exists $\varepsilon>0$ such that
$\mathbb{E}_{p(t,\cdot)}|s_{\theta}(t,x)-\nabla\log
p(t,x)|^{2}\leq\varepsilon^{2}$ for all $t$.

The condition (1) assumes the monotonicity of $b(t,\cdot)$ and (2) assumes the
Lipschitz property of the score functions. In the previous examples, $b(t,x)$
is linear in $x$ so the density $p(t,\cdot)$ is Gaussian-like, and its score
is almost affine. Thus, it is reasonable to assume (2). Conditions (1) and (2)
are used to quantify how a perturbation of the model parameters in an SDE
affects its distribution. The condition (3) specifies how accurate Stein’s
score is estimated by a blackbox estimation. There has been work (e.g. [10,
32, 42]) on the efficiency of score approximations. So it is possible to
replace the condition (3) with those score approximation bounds.

######  Theorem 3.2.

Let Assumption 3.1 hold, and $h>0$. Define
$\eta:=W_{2}(p(T,\cdot),p_{\tiny\mbox{noise}}(\cdot))$, and

$u(t):=\int_{T-t}^{T}\left(-2r_{b}(s)+(2L+2h)\sigma^{2}(s)\right)ds.$ (3.3)

Then we have

$W_{2}(p_{\tiny\mbox{data}}(\cdot),\overline{X}_{T})\leq\sqrt{\eta^{2}e^{u(T)}+\frac{\varepsilon^{2}}{2h}\int_{0}^{T}\sigma^{2}(t)e^{u(T)-u(T-t)}dt}.$
(3.4)

The proof of Theorem 3.2 is deferred to Appendix C. A similar result was given
in [34] under an unconventional assumption that the score matching functions
$s_{\theta}(t,x)$, rather than the score functions $\nabla p(t,x)$, are
Lipschitz. In fact, the (impractical) assumption that the score matching
functions are Lipschitz is not needed at the continuous level, and can be
replaced with the Lipschitz condition on the score functions. On the other
hand, the Lipschitz property of the score matching functions are required, for
technical purposes, to bound the discretization error in Section 3.2.

It is possible to establish sharper bounds under extra (structural) conditions
on $(b(\cdot,\cdot),\sigma(\cdot))$, and also specify the dependence in
dimension $d$ (e.g. [12, 22]). For instance, if we assume $b(t,x)$ is
separable in $t$ and $x$ and linear in $x$, and $p_{\tiny\mbox{data}}(\cdot)$
is strongly log-concave, then the term $L\sigma^{2}(s)$ in (3.3) will become
$-L^{\prime}\sigma^{2}(s)$ for some $L^{\prime}>0$. Since the purpose of this
paper is to introduce the methodology of CDPMs, we leave the full
investigation of its theory to the future work.

Now let’s explain contractive DPMs. Looking at the bound (3.4), the sampling
error $W_{2}(p_{\tiny\mbox{data}}(\cdot),\overline{X}_{T})$ is linear in the
score matching error $\varepsilon$ and the initialization error $\eta$, and
these errors may be amplified in time $T$ – in most aforementioned DPMs,
$r_{b}(t)\leq 0$ so $u(t)$ is positive and at least linear. This implies that
the sampling error explodes if $T$ is set to be large even at the continuous
level. As mentioned earlier, it is especially problematic if we don’t know how
good a blackbox score matching $s_{\theta}(t,x)$ is. Our idea is simply to
make $u(t)$ be negative, that is to set $r_{b}(t)>0$ sufficiently large, in
order to prevent the score matching error from propagating in backward
sampling. This yields the class of CDPMs, which is inherently different from
existing DPMs in the sense that these DPMs often have contractive forward
processes, while our proposal requires contractive backward processes.
Quantitatively, we can set for some $\alpha>0$,

$\inf_{0\leq t\leq T}\left(r_{b}(t)-(L+h)\sigma^{2}(t)\right)\geq\alpha.$
(3.5)

In practice, it suffices to design $(b(\cdot,\cdot),\sigma(\cdot))$ with a
positive $r_{b}(t)$. We present two examples, contractive OU processes and
contractive sub-VP SDEs, for experiments in Section 4.

###  3.2. Discretization error

We study the discretization error
$\left(\mathbb{E}|\overline{X}_{T}-\widehat{X}_{N}|^{2}\right)^{\frac{1}{2}}$
for CDPMs. Classical SDE theory [31] indicates that
$\left(\mathbb{E}|\overline{X}_{T}-\widehat{X}_{N}|^{2}\right)^{\frac{1}{2}}\leq
C(T)\delta$, with the constant $C(T)$ exponential in $T$. Here we show that by
a proper choice of the pair $(b(\cdot,\cdot),\sigma(\cdot))$ leading to CDPMs,
the constant $C(T)$ can be made independent of $T$. In other words, the
discretization error will not expand over the time.

We need some technical assumptions.

######  Assumption 3.3.

The following conditions hold:

  1. (1)

There exists $L_{\sigma}>0$ such that $|\sigma(t)-\sigma(t^{\prime})|\leq
L_{\sigma}|t-t^{\prime}|$ for all $t,t^{\prime}$.

  2. (2)

There exists $R_{\sigma}>0$ such that $\sigma(t)\leq R_{\sigma}$ for all $t$.

  3. (3)

There exists $L_{b}>0$ such that $|b(t,x)-b(t^{\prime},x^{\prime})|\leq
L_{b}(|t-t^{\prime}|+|x-x^{\prime}|)$ for all $t,t^{\prime}$ and
$x,x^{\prime}$.

  4. (4)

There exists $L_{s}>0$ such that
$|s_{\theta}(t,x)-s_{\theta}(t^{\prime},x^{\prime})|\leq
L_{s}(|t-t^{\prime}|+|x-x^{\prime}|)$ for all $t,t^{\prime}$ and
$x,x^{\prime}$.

  5. (5)

There exists $R_{s}>0$ such that $|s_{\theta}(T,x)|\leq R_{s}(1+|x|)$ for all
$x$.

Next we introduce a contractive assumption that is consistent with (3.5)

######  Assumption 3.4.

There exists $\beta>0$ such that

$\int_{T-t}^{T}(r_{b}(s)-L_{s}\sigma^{2}(s))\,ds\geq\beta t,\quad\mbox{for all
}t,$ (3.6)

or simply

$\beta:=\inf_{0\leq t\leq T}\left(r_{b}(t)-L_{s}\sigma^{2}(t)\right)>0.$ (3.7)

######  Theorem 3.5.

Let Assumptions 3.1, 3.3 and 3.4 hold. Then there exists $C>0$ (independent of
$\delta,T$) such that for $\delta>0$ sufficiently small,

$\left(\mathbb{E}|\overline{X}_{T}-\widehat{X}_{N}|^{2}\right)^{\frac{1}{2}}\leq
C\sqrt{\delta}.$ (3.8)

The proof of Theorem 3.5 is given in Appendix D. By (3.2), Theorem 3.2 and
Theorem 3.5, we get a bound
$W_{2}(p_{\tiny\mbox{data}}(\cdot),\widehat{X}_{N})$ of order $\eta
e^{-T}+\varepsilon(1-e^{-T})+\sqrt{\delta}$ for CDPMs, where the
discretization error is independent of $T$.

##  4\. Experiments

In this section, we consider some examples of CDPMs. As mentioned in Section
3.1, the ideal candidates should satisfy the contractive assumptions such as
(3.5) or (3.7). However, we generally don’t know the Lipschitz constants $L$
and $L_{s}$. The practical way is only to require
$\inf_{t\in\left[0,T\right]}r_{b}(t)>0$, leading to the following examples.

  * •

Contractive Orstein-Ulenback (COU) process: $b(t,x)=\theta(x-\mu)$ with
$\theta>0$, $\mu\in\mathbb{R}^{n}$ and $\sigma(t)=\sigma$. The backward
process is:

$d\overline{X}_{t}=\bigg{(}-\theta(\overline{X}_{t}-\mu)+\sigma^{2}\nabla\log
p(T-t,\overline{X}_{t})\bigg{)}dt+\sigma dB_{t},$ (4.1)

with
$\overline{X}_{0}\sim\mathcal{N}\left(0,\frac{\sigma^{2}}{2\theta}(e^{2\theta
T}-1)I\right)$.

  * •

Contractive sub-variance preserving (C sub-VP) SDEs:
$b(t,x)=\frac{1}{2}\beta(t)x$ and
$\sigma(t)=\sqrt{\beta(t)(e^{2\int_{0}^{t}\beta(s)ds}-1)}$, where
$\beta(t)=\beta_{\min}+\frac{t}{T}(\beta_{\max}-\beta_{\min})$. By setting
$\gamma(t)=e^{2\int_{0}^{t}\beta(s)ds}$, the backward process is:

$\displaystyle d\overline{X}_{t}$
$\displaystyle=\bigg{(}-\frac{1}{2}\beta(T-t)\overline{X}_{t}+\beta(T-t)(\gamma(T-t)-1)\nabla\log
p(T-t,\overline{X}_{t})\bigg{)}dt$ (4.2)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\sqrt{\beta(T-t)(\gamma(T-t)-1)}dB_{t},$

with
$\overline{X}_{0}\sim\mathcal{N}\left(0,(e^{\frac{T}{2}(\beta_{\max}+\beta_{\min})}-1)^{2}I\right)$.

###  4.1. 1-dimensional examples

We conduct experiments on two synthetic datasets: (i) a single point mass at
$x_{0}=-1$; (ii)) the mixture of two equally-weighted Gaussian distributions
$\mathcal{N}(\mu_{1},\sigma_{1}^{2})$ and
$\mathcal{N}(\mu_{2},\sigma_{2}^{2})$, with parameters $\mu_{1}=-1.5$,
$\sigma_{1}=1$ and $\mu_{2}=1.5$, $\sigma_{2}=0.5$ respectively. To illustrate
the effect of CDPMs in stablizing score matching and discretization errors, we
initialize the backward process with the exact distribution $p(T,\cdot)$, i.e.
a sample of $X_{T}$. We use a three layer neural network (with residual
connection) to learn the score functions. Implementation details are provided
in Appendix E.1.

(a) single point

(b) Gaussian mixture

Figure 1. Evolution of $W_{2}$ distance in time.

Figure 1 plots the evolution of the Wasserstein-2 distance between the
generated sample distribution and the true distribution over $T\leq T_{\max}$.
Here $T_{\max}$ is the time horizon on which the score functions are learnt,
and the diffusion coefficients are scaled accordingly. Our result shows that
C-OU processes have a similar performance as VE, while outperform other models
(OU, VP and sub-VP) in both settings. We also plot the learned score and the
true score difference in Appendix E.1.

###  4.2. Swiss Roll and MNIST datasets

We apply contractive sub-VP to Swiss Roll and MNIST datasets. Implementation
details are reported in Appendix E.2. Figure 2(a) plots the original Swiss
Roll; Figure 2(b) shows the evolution process of contractive sub-VP sampling
on the Swiss Roll data. Figure 3 illustrates image synthesis by contractive
sub-VP on the MNIST data.

(a) Original Swiss Roll

(b) Swiss Roll generation with 200, 400, 600, 800, 10000 iterations.

Figure 2. Swiss Roll generation by contractive sub-VP. Figure 3. MNIST
synthesis by contractive sub-VP.

###  4.3. CIFAR-10 dataset

We demonstrate the performance of our proposed contractive sub-VP SDEs on the
task of unconditional synthesis of the CIFAR-10 dataset. We compute and
compare the FID scores of the images generated by previous (diffusion) models
and our proposed model. Implementation details are given in Appendix E.3.

Figure 4. CIFAR-10 Synthesis

Model |  Inception $\uparrow$ |  FID $\downarrow$ |
---|---|---|---
PixelCNN [60] | 4.60 | 65.9 |
IGEBM [18] | 6.02 | 40.6 |
ViTGAN [37] | $9.30$ | $6.66$ |
StyleGAN2-ADA [29] | $9.83$ | $2.92$ |
NCSN [54] | ${8.87}$ | $25.32$ |
NCSNv2 [55] | ${8.40}$ | $10.87$ |
DDPM [25] | $9.46$ | $3.17$ |
DDIM, $T=50$ [51] | - | $4.67$ |
DDIM, $T=100$ [51] | - | $4.16$ |
NCSN$++$ |  |  |
VP SDE [57] | $9.58$ | $2.55$ |
sub-VP SDE [57] | $9.56$ | $2.61$ |
VE SDE [57] | $9.68^{*}$ | $2.47^{*}$ |
CDPM (ours) | 10.18 | 2.47 |

Table 1. Inception & FID scores

Figure 4 provides image synthesis on the CIFAR-10 data. From Table 1, our
proposed model (CDPM) shows the best performance among all known classes of
SDE-based diffusion models. In particular, it outperforms VE SDEs (non-deep
version in [57]) in the sense of achieving a better inception score when
having the same FID ($*$ the model evaluation is conducted on our own machine
(4 3080Ti GPUs) given the checkpoints provided by [57]).

##  5\. Conclusion

In this paper, we propose a new criterion – the contraction of backward
sampling in the design of SDE-based DPMs. This naturally leads to a novel
class of contractive DPMs. The main takeaway is that the contraction of the
backward process limits score matching errors from propagating, and controls
discretization error as well. Our proposal is supported by theoretical
considerations, and is corroborated by experiments. Notably, our proposed
contractive sub-VP outperforms other SDE-based DPMs on CIFAR 10 dataset.
Though the intention of this paper is not to beat SOTA diffusion models, CDPMs
show promising results that we hope to trigger further research.

There are a few directions to extend this work. First, we assume that score
matching errors are bounded in $L^{2}$ as in [12, 22, 36]. It is interesting
to see whether this assumption can be relaxed to more realistic conditions
given the application domain. Second, it is desirable to establish sharp
theory for CDPMs, with dimension dependence. Finally, our formulation is based
on SDEs, and hence stochastic samplers. We don’t look into ODE samplers as in
[57, 63, 38, 11]. This leaves open the problems such as whether the proposed
CDPMs perform well by ODE samplers, and why the ODE samplers derived from SDEs
outperform those directly learnt, as observed in previous studies.

Acknowledgement: We thank Jason Altschuler, Yuxin Chen, Sinho Chewi, Xuefeng
Gao, Dan Lacker, Yuting Wei and David Yao for helpful discussions. We
gratefully acknowledges financial support through NSF grants DMS-2113779 and
DMS-2206038, and by a start-up grant at Columbia University.

## Appendix

### A. Proof of Theorem 2.1

###### Proof.

Here we give a heuristic derivation of the time reversal formula (2.3). First,
the infinitesimal generator of $X$ is
$\mathcal{L}:=\frac{1}{2}\sigma^{2}(t)\Delta+b\cdot\nabla$. It is known that
the density $p(t,x)$ satisfies the the Fokker–Planck equation:

$\frac{\partial}{\partial t}p(t,x)=\mathcal{L}^{*}p(t,x),$ (5.1)

where $\mathcal{L}^{*}:=\frac{1}{2}\sigma^{2}(t)\Delta-\nabla\cdot b$ is the
adjoint of $\mathcal{L}$. Let $\overline{p}(t,x):=p(T-t,x)$ be the probability
density of the time reversal $\overline{X}$. By (5.1), we get

$\frac{\partial}{\partial
t}\overline{p}(t,x)=-\frac{1}{2}\sigma^{2}(t)\Delta\overline{p}(t,x)+\nabla\cdot\left(b(T-t,x)\,\overline{p}(t,x)\right).$
(5.2)

On the other hand, we expect the generator of $\overline{X}$ to be
$\overline{\mathcal{L}}:=\frac{1}{2}\overline{\sigma}^{2}(t)\Delta+\overline{b}\cdot\nabla$.
The Fokker-Planck equation for $\overline{p}(t,x)$ is

$\frac{\partial}{\partial
t}\overline{p}(t,x)=\frac{1}{2}\overline{\sigma}^{2}(t)\Delta\overline{p}(t,x)-\nabla\cdot\left(\overline{b}(t,x)\,\overline{p}(t,x)\right).$
(5.3)

Comparing (5.2) and (5.3), we set $\overline{\sigma}(t)=\sigma(T-t)$ and then
get

$\left(b(T-t,x)+\overline{b}(t,x)\right)\overline{p}(t,x)=\sigma^{2}(T-t)\,\nabla\overline{p}(t,x).$

This yields the desired result. ∎

Let’s comment on Theorem 2.1. [24] proved the result by assuming that
$b(\cdot,\cdot)$ and $\sigma(\cdot,\cdot)$ are globally Lipschitz, and the
density $p(t,x)$ satisfies an a priori $H^{1}$ bound. The implicit condition
on $p(t,x)$ is guaranteed if $\partial_{t}+\mathcal{L}$ is hypoelliptic. These
conditions were relaxed in [45]. In another direction, [20, 21] used an
entropy argument to prove the time reversal formula in the case
$\sigma(t)=\sigma$. This approach was further developed in [9] which made
connections to optimal transport.

### B. Score matching

#### B.1. Proof of Theorem 2.2

###### Proof.

We have

$\displaystyle\nabla_{\theta}\mathcal{J}_{\text{ISM}}(\theta)$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{p(t,\cdot)}\left[|s_{\theta}(t,X)|^{2}\right]-2\mathbb{E}_{p(t,\cdot)}\left[\nabla_{\theta}s_{\theta}(t,X)\cdot\nabla\log
p(t,X)\right]$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{p(t,\cdot)}\left[|s_{\theta}(t,X)|^{2}\right]-2\int\nabla_{\theta}s_{\theta}(t,x)\cdot\nabla
p(t,x)dx$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{p(t,\cdot)}\left[|s_{\theta}(t,X)|^{2}\right]-2\,\nabla_{\theta}\int
s_{\theta}(t,x)\cdot\nabla p(t,x)dx$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{p(t,\cdot)}\left[|s_{\theta}(t,X)|^{2}\right]+2\nabla_{\theta}\int\nabla\cdot
s_{\theta}(t,x)\,p(t,x)dx$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{p(t,\cdot)}\left[|s_{\theta}(t,X)|^{2}+2\,\nabla\cdot
s_{\theta}(t,X)\right]=\nabla_{\theta}\widetilde{\mathcal{J}}(\theta),$

where we use the divergence theorem in the fourth equation. ∎

#### B.2. Scalable score matching methods

(a) Sliced score matching. One way is that we further address the term
$\nabla_{x}\cdot s_{\theta}(t,x)$ by random projections. The method proposed
in [56] is called sliced score matching. Considering the Jacobian matrix
$\nabla s_{\theta}(t,x)\in\mathbb{R}^{d\times d}$, we have

$\nabla\cdot s_{\theta}(t,x)=\operatorname{Tr}(\nabla
s_{\theta}(t,x))=\mathbb{E}_{v\sim\mathcal{N}(0,I)}\left[v^{\top}\nabla
s_{\theta}(t,x)v\right].$

We can then rewrite the training objective as:

$\min_{\theta}\widetilde{\mathcal{J}}_{\text{SSM}}(\theta)=\mathbb{E}_{t\in\mathcal{U}(0,T)}\mathbb{E}_{v_{t}\sim\mathcal{N}(0,I)}\mathbb{E}_{p(t,\cdot)}\left[\lambda(t)\left(\|s_{\theta}(t,X)\|^{2}+2\,v^{\top}\nabla(v^{\top}s_{\theta}(t,x))\right)\right].$
(5.4)

which can be computed easily. It requires only two times of back propagation,
as $v^{\top}s_{\theta}(t,x)$ can be seen as adding a layer of the inner
product between $v$ and $s_{\theta}$.

(b) Denoising score matching. The second way is that we go back to the
objective (2.10), and use a nonparametric estimation. The idea stems from [26,
62], in which it was shown that $\mathcal{J}_{ESM}$ is equivalent to the
following denoising score matching (DSM) objective:

$\tilde{\mathcal{J}}_{\text{DSM}}(\theta)=\mathbb{E}_{t\sim\mathcal{U}(0,T)}\left\\{\lambda(t)\mathbb{E}_{X_{0}\sim
p_{data}(\cdot)}\mathbb{E}_{X_{t}\mid
X_{0}}\left[\left|s_{\theta}(t,X(t))-\nabla_{X_{t}}\log p(t,X_{t}\mid
X_{0})\right|^{2}\right]\right\\}$

######  Theorem 5.1.

Let $\mathcal{J}_{\text{DSM}}(\theta):=\mathbb{E}_{X_{0}\sim
p_{data}(\cdot)}\mathbb{E}_{X_{t}\mid
X_{0}}\left[\left|s_{\theta}(t,X(t))-\nabla_{X_{t}}\log p(t,X_{t}\mid
X_{0})\right|^{2}\right]$. Under suitable conditions on $s_{\theta}$, we have
$\mathcal{J}_{\text{DSM}}(\theta)=\mathcal{J}_{\text{ESM}}(\theta)+C$ for some
$C$ independent of $\theta$. Consequently, the minimum point of
$\mathcal{J}_{\text{DSM}}$ and that of $\mathcal{J}_{\text{ESM}}$ coincide.

###### Proof.

We have

$\displaystyle\mathcal{J}_{\text{ESM}}(\theta)$
$\displaystyle=\mathbb{E}_{p(t,\cdot)}|s_{\theta}(t,X)-\nabla\log p(t,X)|^{2}$
$\displaystyle=\mathbb{E}_{p(t,\cdot)}\left[|s_{\theta}(t,X)|^{2}-2s_{\theta}(t,X)^{\top}\nabla\log
p(t,X)+|\nabla\log p(t,X)|^{2}\right].$

Consider the inner product term, rewriting it as:

$\displaystyle\mathbb{E}_{p(t,\cdot)}\left[s_{\theta}(t,X)^{\top}\nabla\log
p(t,X)\right]$ $\displaystyle=\int_{x}p(t,x)s_{\theta}(t,x)^{\top}\nabla\log
p(t,x)\mathrm{d}x$ $\displaystyle=\int_{x}s_{\theta}(t,x)^{\top}\nabla
p(t,x)\mathrm{d}x$
$\displaystyle=\int_{x}s_{\theta}(t,x)^{\top}\nabla\int_{x_{0}}p(0,x_{0})p(t,x|x_{0})\mathrm{d}x_{0}\mathrm{d}x$
$\displaystyle=\int_{x_{0}}\int_{x}s_{\theta}(t,x)^{\top}p(0,x_{0})\nabla
p(t,x|x_{0})\mathrm{d}x\mathrm{d}x_{0}$
$\displaystyle=\int_{x_{0}}p(0,x_{0})\int_{x}s_{\theta}(t,x)^{\top}p(t,x|x_{0})\nabla\log
p(t,x|x_{0})\mathrm{d}x\mathrm{d}x_{0}$ $\displaystyle=\mathbb{E}_{X_{0}\sim
p(0,\cdot)}\mathbb{E}_{X_{t}\mid
X_{0}}\left[s_{\theta}(t,X(t))^{\top}\nabla_{X_{t}}\log p(t,X_{t}\mid
X_{0})\right],$

combining
$\mathbb{E}_{X}|s_{\theta}(t,X)|^{2}=\mathbb{E}_{X_{0}}\mathbb{E}_{X\mid
X_{0}}|s_{\theta}(t,X)|^{2}$ concludes our proof. ∎

The intuition of DSM is that following the gradient $s_{\theta}$ of the log
density at some corrupted point $\tilde{x}$ should ideally move us towards the
clean sample $x$. The reason that the objective $\mathcal{J}_{DSM}$ is
comparatively easy to solve is that conditional distribution usually satisfies
a good distribution, like Gaussian kernel, i.e. $p(X_{t}\mid X_{0})\sim
N\left(X_{t};\mu_{t}(X_{0}),\sigma_{t}^{2}I\right)$, which is satisfied in
many cases of DPM, we can explicitly compute that:

$\nabla_{X_{t}}\log p(t,X_{t}\mid
X_{0})=\frac{1}{\sigma_{t}^{2}}(\mu_{t}(X_{0})-X_{t}).$

Direction $\frac{1}{\sigma_{t}^{2}}(X_{0}-\mu_{t}(X_{0}))$ clearly corresponds
to moving from $\tilde{x}$ back towards clean sample $x$, and we want
$s_{\theta}$ to match that as best it can. Moreover, empirically validated by
e.g. [57], a good candidate of $\lambda(t)$ is chosen as:

$\lambda(t)\propto 1/\mathbb{E}\left[\left|\nabla_{X_{t}}\log p(t,X_{t}\mid
X_{0})\right|^{2}\right]=\sigma_{t}^{2}$

Thus, our final optimization objective is:

$\displaystyle\tilde{\mathcal{J}}_{\text{DSM}}(\theta)$
$\displaystyle=\mathbb{E}_{t\sim\mathcal{U}(0,T)}\left\\{\sigma_{t}^{2}\mathbb{E}_{X_{0}\sim
p_{data}(\cdot)}\mathbb{E}_{X_{t}\mid
X_{0}}\left[\left|s_{\theta}(t,X(t))-\nabla_{X_{t}}\log p(t,X_{t}\mid
X_{0})\right|^{2}\right]\right\\}$
$\displaystyle=\mathbb{E}_{t\sim\mathcal{U}(0,T)}\left\\{\mathbb{E}_{X_{0}\sim
p_{data}(\cdot)}\mathbb{E}_{X_{t}\mid
X_{0}}\left[\left|\sigma_{t}s_{\theta}(t,X(t))+\frac{X_{t}-\mu_{t}(X_{0})}{\sigma_{t}}\right|^{2}\right]\right\\}$
$\displaystyle=\mathbb{E}_{t\sim\mathcal{U}(0,T)}\left\\{\mathbb{E}_{X_{0}\sim
p_{data}(\cdot)}\mathbb{E}_{\epsilon_{t}\sim\mathcal{N}(0,I)}\left[\left|\sigma_{t}s_{\theta}(t,\mu_{t}(X_{0})+\sigma_{t}\epsilon_{t})+\epsilon_{t}\right|^{2}\right]\right\\}$

where the second equality holds when $X_{t}\mid X_{0}$ follows a conditionally
normal and the third equality follows from a reparameterization/change of
variables.

### C. Proof of Theorem 3.2

###### Proof.

The idea relies on coupling, which is similar to [66, Lemma 4]. Consider the
coupled SDEs:

$\left\\{\begin{array}[]{lcl}dY_{t}=\left(-b(T-t,Y_{t})+\sigma^{2}(T-t)\nabla\log
p(T-t,Y_{t})\right)dt+\sigma(T-t)dB_{t},\\\
dZ_{t}=\left(-b(T-t,Z_{t})+\sigma^{2}(T-t)s_{\theta}(T-t,Z_{t})\right)dt+\sigma(T-t)dB_{t},\end{array}\right.$

where $(Y_{0},Z_{0})$ are coupled to achieve
$W_{2}(p(T,\cdot),p_{\tiny\mbox{noise}}(\cdot))$, i.e.
$\mathbb{E}|Y_{0}-Z_{0}|^{2}=W_{2}(p(T,\cdot),p_{\tiny\mbox{noise}}(\cdot))$.
It is easy to see that

$W_{2}^{2}(p_{\tiny\mbox{data}}(\cdot),\overline{X}_{T})\leq\mathbb{E}|Y_{T}-Z_{T}|^{2}.$
(5.5)

So the goal is to bound $\mathbb{E}|Y_{T}-Z_{T}|^{2}$. By Itô’s formula, we
get

$d|Y_{t}-Z_{t}|^{2}=2(Y_{t}-Z_{t})\cdot(-b(T-t,Y_{t})+\sigma^{2}(T-t)\nabla\log
p(T-t,Y_{t})\\\ +b(T-t,Z_{t})-\sigma^{2}(T-t)s_{\theta}(T-t,Z_{t}))dt$

which implies that

$\frac{d\,\mathbb{E}|Y_{t}-Z_{t}|^{2}}{dt}=-2\underbrace{\mathbb{E}((Y_{t}-Z_{t})\cdot(b(T-t,Y_{t})-b(T-t,Z_{t}))}_{(a)})\\\
+2\underbrace{\mathbb{E}((Y_{t}-Z_{t})\cdot\sigma^{2}(T-t)(\nabla\log
p(T-t,Y_{t})-s_{\theta}(T-t,Z_{t})))}_{(b)}.$ (5.6)

By Assumption 3.1 (1), we get

$(a)\geq r_{b}(T-t)\,\mathbb{E}|Y_{t}-Z_{t}|^{2}.$ (5.7)

Moreover,

$\displaystyle(b)$
$\displaystyle=\sigma^{2}(T-t)\bigg{(}\mathbb{E}((Y_{t}-Z_{t})\cdot(\nabla\log
p(T-t,Y_{t})-\nabla\log p(T-t,Z_{t})))$ (5.8)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\mathbb{E}((Y_{t}-Z_{t})\cdot(\nabla\log
p(T-t,Z_{t})-s_{\theta}(T-t,Z_{t})))\bigg{)}$
$\displaystyle\leq\sigma^{2}(T-t)\left(L\,\mathbb{E}|Y_{t}-Z_{t}|^{2}+h\mathbb{E}|Y_{t}-Z_{t}|^{2}+\frac{1}{4h}\varepsilon^{2}\right),$

where we use Assumption 3.1 (2)(3) in the last inequality. Combining (5.6),
(5.7) and (5.8), we have

$\frac{d\,\mathbb{E}|Y_{t}-Z_{t}|^{2}}{dt}\leq\left(-2r_{b}(T-t)+(2h+2L)\sigma^{2}(T-t)\right)\mathbb{E}|Y_{t}-Z_{t}|^{2}+\frac{\varepsilon^{2}}{2h}\sigma^{2}(T-t).$
(5.9)

Applying Grönwall’s inequality, we have:

$\mathbb{E}|Y_{T}-Z_{T}|^{2}\leq
e^{u(T)}\mathbb{E}|Y_{0}-Z_{0}|^{2}+\frac{\varepsilon^{2}}{2h}\int_{0}^{T}\sigma^{2}(T-t)e^{u(T)-u(t)}dt,$

which combined with (5.5) yields (3.4). ∎

### D. Proof of Theorem 3.5

The analysis of the error
$\left(\mathbb{E}|\overline{X}_{T}-\widehat{X}_{N}|^{2}\right)^{\frac{1}{2}}$
relies on the following lemmas. The lemma below proves the contraction of the
backward SDE $\overline{X}$.

######  Lemma 5.2.

Let $(\overline{X}^{x}_{t},\,0\leq t\leq T)$ be defined by (2.12) with
$\overline{X}^{x}_{0}=x$. Let Assumptions 3.1, 3.3 and 3.4 hold. Then

$\left(\mathbb{E}|\overline{X}^{x}_{t}-\overline{X}^{y}_{t}|^{2}\right)^{\frac{1}{2}}\leq\left(\mathbb{E}|x-y|^{2}\right)^{\frac{1}{2}}\exp(-2\beta
t),\quad\mbox{for all }t,$ (5.10)

where $\overline{X}^{x}$ and $\overline{X}^{y}$ be coupled, i.e. they are
driven by the same Brownian motion with (different) initial values $x$ and $y$
respectively ($x$ and $y$ represent two random variables).

###### Proof.

Note that

$d|\overline{X}^{x}_{s}-\overline{X}^{y}_{s}|^{2}=2\left(\overline{X}^{x}_{s}-\overline{X}^{y}_{s}\right)\cdot\bigg{(}-b(T-s,\overline{X}^{x}_{s})+\sigma^{2}(T-s)s_{\theta}(T-s,\overline{X}^{x}_{s})\\\
+b(T-s,\overline{X}^{y}_{s})-\sigma^{2}(T-s)s_{\theta}(\overline{X}^{y}_{s})\bigg{)}ds.$

Thus,

$\displaystyle\frac{d}{ds}\mathbb{E}|\overline{X}^{x}_{s}-\overline{X}^{y}_{s}|^{2}=-2$
$\displaystyle\underbrace{\mathbb{E}\left[(\overline{X}^{x}_{s}-\overline{X}^{y}_{s})\cdot(b(T-s,\overline{X}^{x}_{s})-b(T-s,\overline{X}^{y}_{s}))\right]}_{(a)}$
(5.11)
$\displaystyle+2\underbrace{\mathbb{E}\left[(\overline{X}^{x}_{s}-\overline{X}^{y}_{s})\sigma^{2}(T-s)(s_{\theta}(T-s,\overline{X}^{x}_{s})-s_{\theta}(T-s,\overline{X}^{y}_{s}))\right]}_{(b)}.$

By Assumption 3.1 (1), we get

$(a)\geq r_{b}(T-s)\mathbb{E}|\overline{X}^{x}_{s}-\overline{X}^{y}_{s}|^{2}.$
(5.12)

By Assumption 3.3 (4), we obtain

$(b)\leq
L_{s}\sigma^{2}(T-s)\mathbb{E}|\overline{X}^{x}_{s}-\overline{X}^{y}_{s}|^{2}.$
(5.13)

Combining (5.11), (5.12) and (5.13) yields

$\frac{d}{ds}\mathbb{E}|\overline{X}^{x}_{s}-\overline{X}^{y}_{s}|^{2}\leq-2(r_{b}(T-s)-L_{s}\sigma^{2}(T-s))\mathbb{E}|\overline{X}^{x}_{s}-\overline{X}^{y}_{s}|^{2}.$

By Grönwall’s inequality, we have

$\mathbb{E}|\overline{X}^{x}_{s}-\overline{X}^{y}_{s}|^{2}\leq\mathbb{E}|x-y|^{2}\exp\left(-2\int_{0}^{t}(r_{b}(T-s)-L_{s}\sigma^{2}(T-s))ds\right),$

which, by the condition (3.6), yields (5.10) ∎

Next we deal with the local (one-step) discretization error of the process
$\overline{X}$. Fixing $t_{\star}\leq T-\delta$, the (one-step) discretization
of $\overline{X}$ starting at $\overline{X}_{t_{\star}}=x$ is:

$\displaystyle\widehat{X}^{t_{\star},x}_{1}=x+(-b(T-t_{\star},x)+\sigma^{2}(T-t_{\star})s_{\theta}(T-t_{\star},x))\delta+\sigma(T-t_{\star})(B_{t_{\star}+\delta}-B_{t_{\star}}).$
(5.14)

The following lemma provides an estimate of the local discretization error.

######  Lemma 5.3.

Let $(\overline{X}_{t}^{t_{\star},x},\,t_{\star}\leq t\leq T)$ be defined by
(2.12) with $\overline{X}^{t_{\star},x}_{t_{\star}}=x$, and
$\widehat{X}^{t_{\star},x}_{1}$ be given by (5.14). Let Assumption 3.3 hold.
Then for $\delta$ sufficiently small (i.e. $\delta\leq\overline{\delta}$ for
some $\overline{\delta}<1$), there exists $C_{1},C_{2}>0$ independent of
$\delta$ and $x$ such that

$\left(\mathbb{E}|\overline{X}^{t_{\star},x}_{t_{\star}+\delta}-\widehat{X}^{t_{\star},x}_{1}|^{2}\right)^{\frac{1}{2}}\leq(C_{1}+C_{2}\sqrt{\mathbb{E}|x|^{2}})^{\frac{1}{2}}\delta^{\frac{3}{2}},$
(5.15)
$|\mathbb{E}(\overline{X}^{t_{\star},x}_{t_{\star}+\delta}-\widehat{X}^{t_{\star},x}_{1})|\leq(C_{1}+C_{2}\sqrt{\mathbb{E}|x|^{2}})^{\frac{1}{2}}\delta^{\frac{3}{2}}.$
(5.16)

###### Proof.

For ease of presentation, we write $\overline{X}_{t}$ (resp.
$\widehat{X}_{1}$) for $\overline{X}^{t_{\star},x}_{t}$ (resp.
$\widehat{X}^{t_{\star},x}_{1}$). Without loss of generality, set
$t_{\star}=0$. We have

$\displaystyle\overline{X}_{\delta}=x+\int_{0}^{\delta}-b(T-t,\overline{X}_{t})+\sigma^{2}(T-t)s_{\theta}(T-t,\overline{X}_{t})dt+\int_{0}^{\delta}\sigma(T-t)dB_{t},$
$\displaystyle\widehat{X}_{1}=x+\int_{0}^{\delta}-b(T,x)+\sigma^{2}(T)s_{\theta}(T,x)dt+\int_{0}^{\delta}\sigma(T)dB_{t}.$

So

$\displaystyle\mathbb{E}|\overline{X}_{\delta}-\widehat{X}_{1}|^{2}$ (5.17)
$\displaystyle=\mathbb{E}\bigg{|}\int_{0}^{\delta}b(T,x)-b(T-t,\overline{X}_{t})dt+\int_{0}^{\delta}\sigma^{2}(T-t)s_{\theta}(T-t,\overline{X}_{t})-\sigma^{2}(T)s_{\theta}(T,x)dt$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\int_{0}^{\delta}\sigma(T-t)-\sigma(T)dB_{t}\bigg{|}^{2}$
$\displaystyle\leq
3\mathbb{E}\bigg{(}\left|\int_{0}^{\delta}b(T,x)-b(T-t,\overline{X}_{t})dt\right|^{2}+\left|\int_{0}^{\delta}\sigma^{2}(T-t)s_{\theta}(T-t,\overline{X}_{t})-\sigma^{2}(T)s_{\theta}(T,x)dt\right|^{2}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\left|\int_{0}^{\delta}\sigma(T-t)-\sigma(T)dB_{t}\right|^{2}\bigg{)}$
$\displaystyle\leq
3\bigg{(}\delta\underbrace{\int_{0}^{\delta}\mathbb{E}|b(T,x)-b(T-t,\overline{X}_{t})|^{2}dt}_{(a)}+\delta\underbrace{\int_{0}^{\delta}\mathbb{E}|\sigma^{2}(T-t)s_{\theta}(T-t,\overline{X}_{t})-\sigma^{2}(T)s_{\theta}(T,x)|^{2}dt}_{(b)}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\underbrace{\int_{0}^{\delta}|\sigma(T-t)-\sigma(T)|^{2}dt}_{(c)}\bigg{)},$

where we use the Cauchy–Schwarz inequality and Itô’s isometry in the last
inequality. By Assumption 3.3 (1), we get

$(c)\leq\int_{0}^{\delta}L_{\sigma}^{2}t^{2}dt=\frac{L_{\sigma}^{2}}{3}\delta^{3}.$
(5.18)

By Assumption 3.3 (3), we have

$(a)\leq\int_{0}^{\delta}2L_{b}^{2}(t^{2}+\mathbb{E}|\overline{X}_{t}-x|^{2})dt=2L_{b}^{2}\left(\frac{\delta^{3}}{3}+\int_{0}^{\delta}\mathbb{E}|\overline{X}_{t}-x|^{2}dt\right).$

According to [31, Theorem 4.5.4], we have
$\mathbb{E}|\overline{X}_{t}-x|^{2}\leq C(1+\mathbb{E}|x|^{2})te^{Ct}$ for
some $C>0$ (independent of $x$). Consequently, for $t\leq\delta$ sufficiently
small (bounded by $\overline{\delta}<1$),

$\mathbb{E}|\overline{X}_{t}-x|^{2}\leq
C^{\prime}(1+\mathbb{E}|x|^{2})t,\quad\mbox{for some }C^{\prime}>0\mbox{
(independent of }\delta,x).$

We then get

$(a)\leq
2L_{b}^{2}\left(\frac{\delta^{3}}{3}+\frac{C^{\prime}(1+\mathbb{E}|x|^{2})}{2}\delta^{2}\right)\leq
2L_{b}^{2}\left(\frac{1}{3}+\frac{C^{\prime}}{2}+\frac{C^{\prime}}{2}\mathbb{E}|x|^{2}\right)\delta^{2}.$
(5.19)

Similarly, we obtain by Assumption 3.3 (1)(2)(4)(5):

$(b)\leq C^{\prime\prime}(1+\mathbb{E}|x|^{2})\delta^{2},\quad\mbox{for some
}C^{\prime\prime}>0\mbox{ (independent of }\delta,x).$ (5.20)

Combining (5.17), (5.18), (5.19) and (5.20) yields the estimate (5.15).

Next we have

$\displaystyle|\mathbb{E}(\overline{X}_{\delta}-\widehat{X}_{1})|$
$\displaystyle=\left|\mathbb{E}\int_{0}^{\delta}b(T,x)-b(T-t,\overline{X}_{t})dt+\mathbb{E}\int_{0}^{\delta}\sigma^{2}(T-t)s_{\theta}(T-t,\overline{X}_{t})-\sigma^{2}(T)s_{\theta}(T,x)dt\right|$
$\displaystyle\leq\int_{0}^{\delta}\mathbb{E}|b(T,x)-b(T-t,\overline{X}_{t})|dt+\int_{0}^{\delta}\mathbb{E}|\sigma^{2}(T-t)s_{\theta}(T-t,\overline{X}_{t})-\sigma^{2}(T)s_{\theta}(T,x)|dt$
$\displaystyle\leq
C^{\prime\prime\prime}\int_{0}^{\delta}\left(t(1+\mathbb{E}|x|)+\mathbb{E}|\overline{X}_{t}-x|\right)dt$
$\displaystyle\leq
C^{\prime\prime\prime\prime}(1+\sqrt{\mathbb{E}|x|^{2}})\delta^{\frac{3}{2}},\quad\mbox{for
some }C^{\prime\prime\prime\prime}>0\mbox{ (independent of }\delta,x).$

where the third inequality follows from Assumption 3.3, and the last
inequality is due to the fact that
$\mathbb{E}|\overline{X}_{t}-x|\leq\left(\mathbb{E}|\overline{X}_{t}-x|^{2}\right)^{\frac{1}{2}}\leq\sqrt{C^{\prime}(1+\mathbb{E}|x|^{2})t}$.
This yields the estimate (5.16). ∎

###### Proof of Theorem 3.5.

The proof is split into four steps.

Step 1. Recall that $t_{k}=k\delta$ for $k=0,\ldots,N$. Denote
$\overline{X}_{k}:=\overline{X}_{t_{k}}$, and let

$e_{k}:=\left(\mathbb{E}|\overline{X}_{k}-\widehat{X}_{k}|^{2}\right)^{\frac{1}{2}}.$

The idea is to build a recursion for the sequence $(e_{k})_{k=0,\ldots,N}$.
Also write $(\overline{X}^{t_{\star},x}_{t},\,t_{\star}\leq t\leq T)$ to
emphasize that the reversed SDE (2.12) starts at
$\overline{X}^{t_{\star},x}_{t_{\star}}=x$, so
$\overline{X}_{k+1}=\overline{X}^{t_{k},\overline{X}_{k}}_{t_{k+1}}$. We have

$\displaystyle e_{k+1}^{2}$
$\displaystyle=\mathbb{E}\left|\overline{X}_{k+1}-\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}+\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1}\right|^{2}$
(5.21)
$\displaystyle=\underbrace{\mathbb{E}|\overline{X}_{k+1}-\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}|^{2}}_{(a)}+\underbrace{\mathbb{E}|\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1}|^{2}}_{(b)}+2\underbrace{\mathbb{E}\left[(\overline{X}_{k+1}-\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}})(\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1})\right]}_{(c)}.$

Step 2. We analyze the term (a) and (b). By Lemma 5.2 (the contraction
property), we get

$(a)=\mathbb{E}|\overline{X}_{t_{k+1}}^{t_{k},\overline{X}_{k}}-\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}|^{2}\leq
e_{k}^{2}\exp(-2\beta\delta).$ (5.22)

By (5.15) (in Lemma 5.3), we have

$(b)\leq\left(C_{1}+C_{2}\mathbb{E}|\widehat{X}_{k}|^{2}\right)\delta^{3}.$
(5.23)

Step 3. We analyze the cross-product (c). By splitting

$\overline{X}_{k+1}-\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}=(\overline{X}_{k}-\widehat{X}_{k})+\underbrace{\left[(\overline{X}_{k+1}-\overline{X}_{k})-(\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k})\right]}_{:=d_{\delta}(\overline{X}_{k},\widehat{X}_{k})},$

we obtain

$(c)=\underbrace{\mathbb{E}\left[(\overline{X}_{k}-\widehat{X}_{k})(\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1})\right]}_{(d)}+\underbrace{\mathbb{E}\left[d_{\delta}(\overline{X}_{k},\widehat{X}_{k})(\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1})\right]}_{(e)}.$
(5.24)

For the term (d), we have

$\displaystyle(d)$
$\displaystyle=\mathbb{E}\left[(\overline{X}_{k}-\widehat{X}_{k})\,\mathbb{E}(\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1}|\mathcal{F}_{k})\right]$
(5.25) $\displaystyle\leq
e_{k}\left(\mathbb{E}|\mathbb{E}(\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1}|\mathcal{F}_{k})|^{2}\right)^{\frac{1}{2}}$
$\displaystyle\leq
e_{k}\left(C_{1}+C_{2}\sqrt{\mathbb{E}|\widehat{X}_{k}|^{2}}\right)\delta^{\frac{3}{2}},$

where we use the tower property (of the conditional expectation) in the first
equation, the Cauchy-Schwarz inequality in the second inequality, and (5.16)
in the final inequality. According to [41, Lemma 1.3], there exists $C_{0}>0$
(independent of $\delta,\widehat{X}_{k}$) such that

$\left(\mathbb{E}d^{2}_{\delta}(\overline{X}_{k},\widehat{X}_{k})\right)^{\frac{1}{2}}\leq
C_{0}e_{k}\sqrt{\delta}.$ (5.26)

Thus,

$\displaystyle(e)$
$\displaystyle\leq\left(\mathbb{E}d^{2}_{\delta}(\overline{X}_{k},\widehat{X}_{k}\right)^{\frac{1}{2}}\left(\mathbb{E}|\overline{X}_{t_{k+1}}^{t_{k},\widehat{X}_{k}}-\widehat{X}_{k+1}|^{2}\right)^{\frac{1}{2}}$
(5.27) $\displaystyle\leq
C_{0}e_{k}\left(C_{1}+C_{2}\sqrt{\mathbb{E}|\widehat{X}_{k}|^{2}}\right)\delta^{2}.$

where we use (5.15) and (5.26) in the last inequality. Combining (5.24),
(5.25) and (5.27) yields for $\delta$ sufficiently small,

$(c)\leq
e_{k}\left(C^{\prime}_{1}+C^{\prime}_{2}\sqrt{\mathbb{E}|\widehat{X}_{k}|^{2}}\right)\delta^{\frac{3}{2}},\quad\mbox{for
some }C^{\prime}_{1},C^{\prime}_{2}>0\mbox{ (independent of
}\delta,\widehat{X}_{k}).$ (5.28)

Step 4. Combining (5.21) with (5.22), (5.23) and (5.28) yields

$e_{k+1}^{2}\leq
e_{k}^{2}\exp(-2\beta\delta)+\left(C_{1}+C_{2}\mathbb{E}|\widehat{X}_{k}|^{2}\right)\delta^{3}+e_{k}\left(C^{\prime}_{1}+C^{\prime}_{2}\sqrt{\mathbb{E}|\widehat{X}_{k}|^{2}}\right)\delta^{\frac{3}{2}}.$

A standard argument shows that Lemma 5.2 (the contraction property) implies
$\mathbb{E}|\overline{X}_{t}|^{2}\leq C$ for some $C>0$. Thus,
$\mathbb{E}|\widehat{X}_{k}|^{2}\leq 2(C+e_{k}^{2})$. As a result, for
$\delta$ sufficiently small,

$e_{k+1}^{2}\leq
e_{k}^{2}\left(1-\frac{3}{4}\beta\delta\right)+D_{1}\delta^{3}+D_{2}e_{k}^{2}\left(\delta^{3}+\delta^{\frac{3}{2}}\right)+D_{3}e_{k}\delta^{\frac{3}{2}},$
(5.29)

for some $D_{1},D_{2},D_{3}>0$ (independent of $\delta$). Note that

$D_{2}e_{k}^{2}\left(\delta^{3}+\delta^{\frac{3}{2}}\right)\leq\frac{1}{4}e_{k}^{2}\beta\delta,\quad\mbox{
for }\delta\mbox{ sufficiently small},$

and

$D_{3}e_{k}\delta^{\frac{3}{2}}\leq\frac{1}{4}e_{k}^{2}\beta\delta+\frac{2D^{2}_{3}}{\beta}\delta^{2}.$

Thus, the estimate (5.29) leads to

$e_{k+1}^{2}\leq
e_{k}^{2}\left(1-\frac{1}{4}\beta\delta\right)+D\delta^{2},\quad\mbox{for some
}D>0\mbox{ (independent of }\delta).$ (5.30)

Unfolding the inequality (5.30) yields the estimate (3.8). ∎

As a remark, if we can improve the estimate in (5.16) to

$|\mathbb{E}(\overline{X}^{t_{\star},x}_{t_{\star}+\delta}-\widehat{X}^{t_{\star},x}_{1})|\leq(C_{1}+C_{2}\sqrt{\mathbb{E}|x|^{2}})^{\frac{1}{2}}\delta^{2},$
(5.31)

(i.e. $\delta^{2}$ local error instead of $\delta^{\frac{3}{2}}$), then the
discretization error is $C\delta$.

### E. Experimental Details

We provide implementation details of the experiments in Section 4. All codes
will soon be available at github.

### E.1. 1-dimensional data experiments

For both experiments, starting from either a single point or Gaussian mixture,
we use a three-layer neural network for score matching, and use $J_{ISM}$ with
$\lambda(t)\equiv 1$ as the loss optimization objective. The parameters of the
DPMs are chosen as below. (Detailed implementation can be found at here)

Parameter | Value | Used in Model
---|---|---
$\mu$ | 0 | OU, COU
$\theta$ | $0.2$ | OU, COU
$\sigma$ | $0.5$ | OU, COU
$\beta_{min}$ | $0.02$ | VP, subVP, CsubVP
$\beta_{max}$ | $0.2$ | VP, subVP, CsubVP
$\sigma_{min}$ | $0.05$ | VE
$\sigma_{max}$ | $0.5$ | VE
$T_{max}$ | $10$ | all
Table 2. Parameters of the DPMs in one dimension.

Below we plot the score mismatch of the learned score functions at different
times for the single point case.

Figure 5. OU score mismatch. Figure 6. VE score mismatch. Figure 7. VP score
mismatch. Figure 8. sub-VP score mismatch. Figure 9. COU score mismatch.

In all cases, score matching errors explode when $t$ is small, and become
smaller when $t$ is larger. Notably, OU and COU have similar score matching
errors (by using a simple yet general 3-layer neural network), but COU
ourperforms OU in data generation. This supports our proposal of CDPMs.

### E.2. Swiss Roll and MNIST experiments

In both experiments, we use contractive sub-VP with predictor-corrector
sampler. We recommend to use $\beta_{min}=0.01$ and $\beta_{max}=8$ for a good
result. The signal-noise-ratio is set to be $0.2$ for Swiss Roll, and $0.1$
for MNIST. The Jupyter Notebook can be found at Swiss Roll, and MNIST.

### E.3. CIFAR10 experiments

We use contractive sub-VP with predictor-corrector sampler. We recommend to
use $\beta_{min}=0.01$ and $\beta_{max}=8$ for a good result. The signal-
noise-ratio is set to be 0.11 for the best result. The experiments based on
NCSN++ [57] are as below.

Parameter | Value
---|---
$\beta_{min}$ | $0.01$
$\beta_{max}$ | $8$
$snr$ | $0.11$
$\operatorname{grad}_{clip}$ | $10$
$\operatorname{\alpha_{lr}}$ | $5e^{-4}$
Table 3. Parameters of the CDPM for CIFAR-10.

## References

  * [1] B. D. O. Anderson.  Reverse-time diffusion equation models.  Stochastic Process. Appl., 12(3):313–326, 1982.
  * [2] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg.  Structured denoising diffusion models in discrete state-spaces.  In Neurips, volume 34, pages 17981–17993, 2021.
  * [3] J. Benton, V. De Bortoli, A. Doucet, and G. Deligiannidis.  Linear convergence bounds for diffusion models via stochastic localization.  2023\.  arXiv:2308.03686.
  * [4] J. Benton, G. Deligiannidis, and A. Doucet.  Error bounds for flow matching methods.  2023\.  arXiv:2305.16860.
  * [5] A. Borji.  Pros and cons of GAN evaluation measures.  Comput. Vis. Image Underst., 179:41–65, 2019.
  * [6] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, and M. Tagliasacchi.  AudioLM: a language modeling approach to audio generation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [7] A. Brock, J. Donahue, and K. Simonyan.  Large scale GAN training for high fidelity natural image synthesis.  In ICLR, 2018.
  * [8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, and A. Askell.  Language models are few-shot learners.  In Neurips, volume 33, pages 1877–1901, 2020.
  * [9] P. Cattiaux, G. Conforti, I. Gentil, and C. Léonard.  Time reversal of diffusion processes under a finite entropy condition.  Ann. Inst. Henri Poincaré Probab. Stat., 59(4):1844–1881, 2023.
  * [10] M. Chen, K. Huang, T. Zhao, and M. Wang.  Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data.  In ICML, volume 40, pages 4672–4712, 2023.
  * [11] S. Chen, S. Chewi, H. Lee, Y. Li, J. Lu, and A. Salim.  The probability flow ode is provably fast.  2023\.  arXiv:2305.11798.
  * [12] S. Chen, S. Chewi, J. Li, Y. Li, A. Salim, and A. R. Zhang.  Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.  2023\.
  * [13] S. Chen, G. Daras, and A. Dimakis.  Restoration-degradation beyond linear diffusions: A non-asymptotic analysis for ddim-type samplers.  In ICML, volume 40, pages 4462–4484, 2023.
  * [14] V. De Bortoli.  Convergence of denoising diffusion models under the manifold hypothesis.  2022\.  arXiv:2208.05314.
  * [15] V. De Bortoli, J. Thornton, J. Heng, and A. Doucet.  Diffusion Schrödinger bridge with applications to score-based generative modeling.  In Neurips, volume 34, pages 17695–17709, 2021.
  * [16] P. Dhariwal and A. Nichol.  Diffusion models beat GANs on image synthesis.  In Neurips, volume 34, pages 8780–8794, 2021.
  * [17] S. Dieleman, L. Sartran, A. Roshannai, N. Savinov, Y. Ganin, P. H. Richemond, A. Doucet, R. Strudel, C. Dyer, and C. Durkan.  Continuous diffusion for categorical data.  2022\.  arXiv:2211.15089.
  * [18] Y. Du, S. Li, J. Tenenbaum, and I. Mordatch.  Improved contrastive divergence training of energy based models.  2020\.  arXiv:2012.01316.
  * [19] N. Fishman, L. Klarner, V. D. Bortoli, E. Mathieu, and M. J. Hutchinson.  Diffusion models for constrained domains.  Transactions on Machine Learning Research, 2023.
  * [20] H. Föllmer.  An entropy approach to the time reversal of diffusion processes.  In Stochastic differential systems (Marseille-Luminy, 1984), volume 69 of Lect. Notes Control Inf. Sci., pages 156–163. Springer, Berlin, 1985.
  * [21] H. Föllmer.  Time reversal on Wiener space.  In Stochastic processes—mathematics and physics (Bielefeld, 1984), volume 1158 of Lecture Notes in Math., pages 119–129. Springer, Berlin, 1986.
  * [22] X. Gao, H. M. Nguyen, and L. Zhu.  Wasserstein convergence guarantees for a general class of score-based generative models.  2023\.  arXiv:2311.11003.
  * [23] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In NIPS, volume 27, pages 2672–2680, 2014.
  * [24] U. G. Haussmann and E. Pardoux.  Time reversal of diffusions.  Ann. Probab., 14(4):1188–1205, 1986.
  * [25] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  In Neurips, volume 33, pages 6840–6851, 2020.
  * [26] A. Hyvärinen.  Estimation of non-normalized statistical models by score matching.  J. Mach. Learn. Res., 6:695–709, 2005.
  * [27] N. Kalchbrenner, A. Oord, K. Simonyan, I. Danihelka, O. Vinyals, A. Graves, and K. Kavukcuoglu.  Video pixel networks.  In ICML, volume 34, pages 1771–1779, 2017.
  * [28] I. Karatzas and S. E. Shreve.  Brownian motion and stochastic calculus, volume 113 of Graduate Texts in Mathematics.  Springer-Verlag, New York, second edition, 1991.
  * [29] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila.  Training generative adversarial networks with limited data.  In Neurips, volume 33, pages 12104–12114, 2020.
  * [30] D. Kingma, T. Salimans, B. Poole, and J. Ho.  Variational diffusion models.  In Neurips, volume 34, pages 21696–21707, 2021.
  * [31] P. E. Kloeden and E. Platen.  Numerical solution of stochastic differential equations, volume 23 of Applications of Mathematics (New York).  Springer-Verlag, Berlin, 1992.
  * [32] F. Koehler, A. Heckett, and A. Risteski.  Statistical efficiency of score matching: The view from isoperimetry.  In ICLR, 2023.
  * [33] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  In ICLR, 2021.
  * [34] D. Kwon, Y. Fan, and K. Lee.  Score-based generative modeling secretly minimizes the Wasserstein distance.  In Neurips, volume 35, pages 20205–20217, 2022.
  * [35] H. Lee, J. Lu, and Y. Tan.  Convergence for score-based generative modeling with polynomial complexity.  In Neurips, volume 35, pages 22870–22882, 2022.
  * [36] H. Lee, J. Lu, and Y. Tan.  Convergence of score-based generative modeling for general data distributions.  In ALT, pages 946–985. PMLR, 2023.
  * [37] K. Lee, H. Chang, L. Jiang, H. Zhang, Z. Tu, and C. Liu.  ViTGAN: Training GANs with vision transformers.  In ICLR, 2022.
  * [38] G. Li, Y. Wei, Y. Chen, and Y. Chi.  Towards faster non-asymptotic convergence for diffusion-based generative models.  2023\.  arXiv:2306.09251.
  * [39] A. Lou and S. Ermon.  Reflected diffusion models.  2023\.  arXiv:2304.04740.
  * [40] S. Mei and Y. Wu.  Deep networks as denoising algorithms: Sample-efficient learning of diffusion models in high-dimensional graphical models.  2023\.  arXiv:2309.11420.
  * [41] G. N. Milstein and M. V. Tretyakov.  Stochastic numerics for mathematical physics.  Scientific Computation. Springer-Verlag, Berlin, 2004.
  * [42] K. Oko, S. Akiyama, and T. Suzuki.  Diffusion models are minimax optimal distribution estimators.  In ICML, volume 40, page 26517–26582, 2023.
  * [43] J. Pidstrigach.  Score-based generative models detect manifolds.  In Neurips, volume 35, pages 35852–35865, 2022.
  * [44] R. Prenger, R. Valle, and B. Catanzaro.  Waveglow: A flow-based generative network for speech synthesis.  In ICASSP, pages 3617–3621, 2019.
  * [45] J. Quastel.  Time reversal of degenerate diffusions.  In In and out of equilibrium (Mambucaba, 2000), volume 51 of Progr. Probab., pages 249–257. Birkhäuser Boston, Boston, MA, 2002.
  * [46] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  2022\.  arXiv:2204.06125.
  * [47] A. Razavi, A. Van den Oord, and O. Vinyals.  Generating diverse high-fidelity images with VQ-VAE-2.  In Neurips, volume 32, pages 14866–14876, 2019.
  * [48] P. H. Richemond, S. Dieleman, and A. Doucet.  Categorical SDEs with simplex diffusion.  2022\.  arXiv:2210.14784.
  * [49] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, pages 10684–10695, 2022.
  * [50] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In ICML, volume 32, pages 2256–2265, 2015.
  * [51] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [52] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever.  Consistency models.  In ICML, volume 40, page 32211–32252, 2023.
  * [53] Y. Song, C. Durkan, I. Murray, and S. Ermon.  Maximum likelihood training of score-based diffusion models.  In Neurips, volume 34, pages 1415–1428, 2021.
  * [54] Y. Song and S. Ermon.  Generative modeling by estimating gradients of the data distribution.  In Neurips, volume 32, page 11918–11930, 2019.
  * [55] Y. Song and S. Ermon.  Improved techniques for training score-based generative models.  In Neurips, volume 33, pages 12438–12448, 2020.
  * [56] Y. Song, S. Garg, J. Shi, and S. Ermon.  Sliced score matching: A scalable approach to density and score estimation.  In UAI, volume 35, pages 574–584, 2020.
  * [57] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole.  Score-based generative modeling through stochastic differential equations.  In ICLR, 2021.
  * [58] W. Tang and X. Y. Zhou.  Tail probability estimates of continuous-time simulated annealing processes.  Numer. Algebra Control Optim., 13(3-4):473–485, 2023.
  * [59] A. Vahdat, K. Kreis, and J. Kautz.  Score-based generative modeling in latent space.  In Neurips, volume 34, pages 11287–11302, 2021.
  * [60] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, and A. Graves.  Conditional image generation with pixelcnn decoders.  In NIPS, volume 29, pages 4797–4805, 2016.
  * [61] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In NIPS, volume 30, pages 6000–6010, 2017.
  * [62] P. Vincent.  A connection between score matching and denoising autoencoders.  Neural Comput., 23(7):1661–1674, 2011.
  * [63] Y. Xu, Z. Liu, M. Tegmark, and T. Jaakkola.  Poisson flow generative models.  In Neurips, volume 35, pages 16782–16795, 2022.
  * [64] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, W. Zhang, B. Cui, and M.-H. Yang.  Diffusion models: A comprehensive survey of methods and applications.  ACM Comput. Surv., 56(4):1–39, 2023.
  * [65] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le.  XLNet: Generalized autoregressive pretraining for language understanding.  In Neurips, volume 32, pages 5753–5763, 2019.
  * [66] H. Zhao, W. Tang, and D. D. Yao.  Policy optimization for continuous reinforcement learning.  2023\.  arXiv:2305.18901.