subsecref subsecname = thmref thmname = theorem lemref lemname = lemma
secname=section ,names=sections ,Name=Section ,Names=Sections
condname=condition ,names=conditions ,Name=Condition ,Names=Conditions
appdxname=appendix ,names=appendices ,Name=Appendix ,Names=Appendices
thmname=theorem ,names=theorems ,Name=Theorem ,Names=Theorems

# Controlling Moments with Kernel Stein Discrepancies

Heishiro Kanagawa Correspondence<EMAIL_ADDRESS>Gatsby
Computational Neuroscience Unit, UCL  Arthur Gretton  Gatsby Computational
Neuroscience Unit, UCL  Lester Mackey  Microsoft Research

###### Abstract

Quantifying the deviation of a probability distribution is challenging when
the target distribution is defined by a density with an intractable
normalizing constant. The kernel Stein discrepancy (KSD) was proposed to
address this problem and has been applied to various tasks including
diagnosing approximate MCMC samplers and goodness-of-fit testing for
unnormalized statistical models. This article investigates a convergence
control property of the diffusion kernel Stein discrepancy (DKSD), an instance
of the KSD proposed by Barp et al. (2019). We extend the result of Gorham and
Mackey (2017), which showed that the KSD controls the bounded-Lipschitz
metric, to functions of polynomial growth. Specifically, we prove that the
DKSD controls the integral probability metric defined by a class of pseudo-
Lipschitz functions, a polynomial generalization of Lipschitz functions. We
also provide practical sufficient conditions on the reproducing kernel for the
stated property to hold. In particular, we show that the DKSD detects non-
convergence in moments with an appropriate kernel.

##  1 Introduction

Consider two probability distributions $P$ and $Q$ over $\mathbb{R}^{D},$
where $P$ is defined by a density function $p$ and $Q$ is arbitrary. We assume
that the density $p$ may contain an unknown normalizing constant. Suppose we
are interested in comparing the expectations of a test function $f_{0}$ under
these two distributions. As a test function is usually only known up to
certain properties (e.g., differentiability or growth conditions), a natural
measure to consider is the worst case error, or an integral probability metric
(Müller, 1997),

$\sup_{f\in{\cal F}}\big{|}\mathbb{E}_{X\sim P}[f(X)]-\mathbb{E}_{Y\sim
Q}[f(Y)]\bigr{|}$

over a function class ${\cal F}$ containing $f_{0}.$ This setting may be
interpreted as either evaluating the goodness of fit of a statistical model
$P$ against the data distribution $Q,$ or assessing an approximation $Q$ to a
posterior distribution $P$ in Bayesian inference. Our particular focus in this
article is on continuous functions of polynomial growth. In this case, the IPM
above implies disagreement in terms of moments, since they are defined by
monomials of coordinates.

As the expectations are rarely analytically tractable, one might compare
empirical estimates using samples from both distributions. In some scenarios,
however, assuming access to samples from $P$ may be problematic; e.g.,
_correct_ samples from $P$ are unavailable if the goal is to assess the
quality of a Markov chain Monte Carlo sampler $Q$ targeting intractable $P.$
One possible way to sidestep the intractable integral is using a Stein
discrepancy (Gorham and Mackey, 2015)

$\sup_{g\in{\cal{\cal G}}}\bigl{|}\mathbb{E}_{Y\sim Q}[{\cal
T}_{P}g(Y)\big{|},$

where ${\cal T}_{P}$ and ${\cal G}$ are a Stein operator and a Stein set (a
function class in the domain of ${\cal T}_{P}$) inducing functions ${\cal
T}_{P}g$ whose expectations under $P$ are zero (see Anastasiou et al., 2021,
for a review). Different choices of the operator and the function class yield
distinct Stein discrepancies. Two major classes are the graph Stein
discrepancies (Gorham and Mackey, 2015; Gorham et al., 2019) and the kernel
Stein discrepancy (KSD) (Chwialkowski et al., 2016; Liu et al., 2016; Oates et
al., 2017; Gorham and Mackey, 2017). Notably, it is possible to compute these
discrepancies: The KSD circumvents the supremum in the definition and admits a
closed-form expression involving kernel evaluations on samples, whereas the
graph Stein discrepancy involves solving a linear program.

While successfully avoiding the intractable expectation, a drawback of Stein
discrepancies is that they lack interpretability: the test functions ${\cal
T}_{P}g$ in a Stein discrepancy are not immediately interpretable in terms of
a given class ${\cal F}$ of interest. Following the spirit of Stein’s method,
this problem has been addressed by lower bounding a Stein discrepancy by a
known IPM – convergence in the Stein discrepancy then implies the IPM
convergence. Gorham and Mackey (2015) showed that the Langevin graph Stein
discrepancy controls the $L^{1}$-Wasserstein distance (the IPM defined by
$1$-Lipschitz functions) for distantly dissipative target distributions;
Gorham et al. (2019) later generalized this result to heavy-tailed targets
with diffusion graph Stein discrepancies. Gorham and Mackey (2017) proved that
the Langevin KSD with the inverse multi-quadratic kernel (IMQ) controls the
bounded-Lipschitz metric; Chen et al. (2018) offer other kernel choices.
Random feature Stein discrepancies by Huggins and Mackey (2018) controls the
bounded-Lipschitz metric with appropriate features while computable in linear
time.

Despite its computational appeal, the KSD is limited in that it need not
control convergence in unbounded functions, particularly functions of
polynomial growth. This issue has been in part addressed by the aforementioned
works on the graph Stein discrepancies. However, their analyses are limited to
linearly growing functions – it is unclear how these results extend to
functions of faster growth. Polynomially growing functions are of practical
interest since they are related to fundamental statistical quantities, such as
mean and variance (the latter corresponds to a quadratically growing
function). Our objective is thus to extend the reach of the KSD to functions
of arbitrary polynomial growth.

In this article, we investigate conditions under which the KSD controls the
convergence of expectations of polynomially growing functions. Specifically,
we prove that the KSD controls the IPM defined by a class of pseudo-Lipschitz
functions, a polynomial generalization of Lipschitz functions. Our KSD bound
builds on the finite Stein factor results of Erdogdu et al. (2018). Our
specific contributions are twofold. First, our analysis considers the
diffusion kernel Stein discrepancy (DKSD) (Barp et al., 2019), a
generalization of the Langevin KSD (Chwialkowski et al., 2016; Liu et al.,
2016; Oates et al., 2017; Gorham and Mackey, 2017); this extension allows us
to consider heavy-tailed targets. Second, we, for the first time, identify
specific practical reproducing kernels that provide polynomial convergence
control.

##  2 Background

We begin with background material required to present our main results.

###  2.1 Notation and definitions

##### Pseudo-Lipschitz functions.

A function $h:\mathbb{R}^{D}\to\mathbb{R}$ is called pseudo-Lipschitz
continuous (or simply $C$-pseudo-Lipschitz) of order $q$ if it satisfies, for
some constant $C>0,$

$\lvert h(x)-h(y)\rvert\leq C(1+\lVert x\rVert_{2}^{q}+\lVert
y\rVert_{2}^{q})\lVert x-y\rVert_{2}\ \text{for all }x,y\in\mathbb{R}^{D},$
(1)

where $\lVert\cdot\rVert_{2}$ denotes the Euclidean norm. We denote the
smallest constant $C$ satisfying (1) by
$\tilde{\mu}_{\mathrm{pLip}}(h)_{\mathrm{1},q}.$ We denote by
$\mathrm{pLip}_{1,q}$ the set of pseudo-Lipschitz functions of order $q$ with
$\tilde{\mu}_{\mathrm{pLip}}(h)_{\mathrm{1},q}\leq 1.$ The pseudo-Lipschitz
continuity generalizes the Lipschitz continuity (corresponding to the case
$q=0$) and allows us to describe functions of polynomial growth.

##### Vectors, matrices, and tensors.

For a real vector $v,$ $\lVert v\rVert_{\mathrm{op}}=\lVert v\rVert_{2}.$ We
identify an order $L$ tensor $T\in\mathbb{R}^{D_{1}\times\cdots\times D_{L}}$
as a multilinear map from
$\mathbb{R}^{D_{1}}\times\cdots\times\mathbb{R}^{D_{L}}$ to $\mathbb{R}$ via
the natural inner product

$T:(u_{1},\dots,u_{L})\in\mathbb{R}^{D_{1}\times\cdots\times
D_{L}}\mapsto\mathbb{R},\left\langle T,v^{(1)}\otimes\cdots\otimes
v^{(L)}\right\rangle=\sum_{i_{1},\dots,i_{L}}T_{i_{1},\dots,i_{L}}v_{i_{1}}^{(1)}\cdots
v_{i_{L}}^{(L)},$

and define the operator norm $\lVert T\rVert_{\mathrm{op}}$ by

$\lVert
T\rVert_{\mathrm{op}}=\sup_{\left\|u^{(l)}\right\|_{2}=1}\left|\left\langle
T,u^{(1)}\otimes\cdots\otimes u^{(l)}\otimes\cdots\otimes
u^{(L)}\right\rangle\right|.$

##### Derivatives.

The symbol $\nabla=(\partial_{1},\dots,\partial_{D})^{\top}$ denotes the
gradient operator with $\partial_{d}$ denoting the partial derivative with
respect to the $d$-th coordinate. The symbol $\nabla^{m}$ denotes the operator
that outputs all the $m$-th order partial derivatives, defined as
$(\nabla^{m}g(x))_{i_{1},\dots
i_{m}}=\partial_{i_{1}}\cdots\partial_{i_{m}}g(x).$ For a vector-valued
function $g:\mathbb{R}^{D}\to\mathbb{R}^{D^{\prime}},$ we define
$\nabla^{i}g:\mathbb{R}^{D}\to\underbrace{\mathbb{R}^{D}\otimes\cdots\otimes\mathbb{R}^{D}}_{i\
\text{times}}\otimes\mathbb{R}^{D^{\prime}}$ by
$(\nabla^{i}g(x))_{k_{1},\dots,k_{i-1},d}=(\nabla^{i}g_{d}(x))_{k_{1},\dots,k_{i-1}};$
i.e., $\nabla^{i}$ is applied to $g$ element-wise. For a matrix valued
function $f,$ we define its column-wise divergence by
$\langle\nabla,f(x)\rangle;$ i.e.,
$\langle\nabla,f(x)\rangle_{i}=\sum_{j}\partial_{j}f_{ji}(x).$

##### Generalized Fourier transform.

Our main result (Proposition 3.4) requires an analogue of the Fourier
transform of a function that is not an element of $L^{1}$ or $L^{2},$ where
$L^{r},$ $r\in\\{1,2\\},$ denotes the Banach space of $r$-integrable functions
with respect to the Lebesgue measure. For our purpose, we use the following
generalized Fourier transform.

######  Definition 2.1 (Generalized Fourier transform (Wendland, 2004,
Defintion 8.9)).

Let $\Phi$ be a continuous complex-valued function on $\mathbb{R}^{D}$ such
that for some a constant $q\geq 0,$ $\Phi(x)=O(\lVert x\rVert_{2}^{q})$ in the
limit of $\lVert x\rVert_{2}\to\infty.$ A measurable function $\hat{\Phi}$ is
called the generalized Fourier transform of $\Phi$ if it satisfies the
following conditions: (a) the restriction of $\hat{\Phi}$ on every compact set
$K\subset\mathbb{R}^{D}\setminus\\{0\\}$ is square-integrable, and (b) there
exists an nonnegative integer $m$ such that

$\int\Phi(x)\hat{\gamma}(x)\differential
x=\int\hat{\Phi}(\omega)\gamma(\omega)\differential\omega$

is true for all Schwartz functions $\gamma$ satisfying
$\gamma(\omega)=O(\lVert\omega\rVert_{2}^{2m})$ for $\lVert\omega\rVert_{2}\to
0.$ The integer $m$ is called the order of $\hat{\Phi}.$

The Fourier transform of $\Phi$ coincides with the generalized Fourier
transform, if it exists. In the following, we limit ourselves to generalized
Fourier transforms of order zero.

###  2.2 Primer of Stein’s method

Stein’s method is a technique to compare distributions, introduced in the
seminal paper by Stein (1972). Stein’s method serves two purposes: it provides
a characterization of probability distributions and enables us to upper bound
an integral probability metric (IPM) (Müller, 1997). This section provides a
brief introduction to the subject. We refer the reader to the expository
papers by Ross (2011) and Anastasiou et al. (2021) for more detailed
descriptions of the technique; the latter reference also provides an overview
of applications in computational statistics.

A staring point of Stein’s method is identifying an operator characterizing a
probability distribution. Formally, for a distribution $P$ on a set ${\cal
X},$ let ${\cal T}$ be a linear operator that acts on a set ${\cal G}({\cal
T})$ of functions on ${\cal X}$ such that

$\mathbb{E}_{X\sim P}\bigl{[}{\cal T}g(X)\bigr{]}=0\ \text{\text{for each}}\
g\in{\cal G}({\cal T}),$ (2)

where we assume that each ${\cal T}g$ is a real-valued function. Such an
operator ${\cal T}$ and a set ${\cal G}({\cal T})$ are respectively called a
_Stein operator_ and a _Stein set_ ; the identity of the form (2) is known as
Stein’s identity.

One can measure the dissimilarity between $Q$ and $P$ by examining the
magnitude of the expectation $\mathbb{E}_{X\sim Q}\bigl{[}{\cal
T}g(X)\bigr{]}$ for some $g\in{\cal G}({\cal T}),$ since a non-zero
expectation indicates $Q\neq P.$ Following this idea, for any subset ${\cal
G}\subset{\cal G}({\cal T}),$ one can construct a discrepancy summary

${\cal S}(Q,{\cal T},{\cal G})=\sup_{g\in{\cal G}}\left|\mathbb{E}_{X\sim
Q}\bigl{[}{\cal T}g(X)\bigr{]}\right|.$ (3)

This worst-case discrepancy measure is called a _Stein discrepancy_ ,
introduced by Gorham and Mackey (2015). Remarkably, by choosing an appropriate
Stein operator and a Stein set, one can construct a computable Stein
discrepancy (Gorham and Mackey, 2015; Chwialkowski et al., 2016; Liu et al.,
2016; Oates et al., 2017; Gorham et al., 2019; Gorham and Mackey, 2017).

A Stein operator is linked to an IPM via a Stein equation:

${\cal T}g_{f}=f-\mathbb{E}_{X\sim P}[f(X)],$ (4)

where $f$ is a function of interest, and $g_{f}\in{\cal G}({\cal T})$ is a
solution to the Stein equation (4). The existence of a solution depends on the
properties of the test function $f,$ the target distribution $P,$ and the
operator ${\cal T}.$ Assuming that we can take expectations, we obtain

$\mathbb{E}_{Y\sim Q}[f(Y)]-\mathbb{E}_{X\sim P}[f(X)]=\mathbb{E}_{Y\sim
Q}\bigl{[}{\cal T}g_{f}(Y)\bigr{]}.$

For a function class ${\cal F},$ this relation yields

$\displaystyle\begin{aligned} d_{{\cal F}}(P,Q)&\coloneqq\sup_{f\in{\cal
F}}\left|\mathbb{E}_{Y\sim Q}[f(Y)]-\mathbb{E}_{X\sim P}[f(X)]\right|\\\
&=\sup_{g\in{\cal G}_{{\cal F}}}\left|\mathbb{E}_{Y\sim Q}\bigl{[}{\cal
T}g(Y)\bigr{]}\right|,\end{aligned}$ (5)

where ${\cal G}_{{\cal F}}=\\{g_{f}:f\in{\cal F}\\}\subset{\cal G}({\cal T})$
is the set of solutions to the Stein equation. The core of Stein’s method is
that the study of the IPM $d_{{\cal F}}(P,Q)$ can be reduced to the evaluation
of the Stein discrepancy in (5). The evaluation is typically performed by
upper bounding the Stein discrepancy, as a solution $g_{f}$ to the Stein
equation is often not explicit and only known up to some regularity
properties. In Section 3.1, we obtain an upper bound on the IPM defined by a
class of pseudo-Lipschitz functions; the upper bound is expressed by a kernel
Stein discrepancy, introduced in Section 2.4.

###  2.3 Generators of Itô diffusions and their Stein equations

In the previous section, the Stein operator was unspecified. Here, we consider
the generator of an Itô diffusion as a Stein operator and introduce the result
of Erdogdu et al. (2018) characterizing a solution to the Stein equation. This
object is related to the diffusion Stein operator considered in the next
Section; the diffusion Stein operator is a first-order differential operator,
whereas generators considered in this section are of the second order.

For a function $f$ pseudo-Lipschitz of order $q,$ consider the Stein equation

$\mathcal{A}_{P}u_{f}=f-\mathbb{E}_{X\sim P}[f(X)],$ (6)

defined by the generator ${\cal A}_{P}$ of an Itô diffusion with invariant
distribution $P,$ where $u_{h}:\mathbb{R}^{D}\to\mathbb{R}$ is a solution to
the equation. The diffusion is defined by the following stochastic
differential equation:

$\differential Z_{t}^{x}=b(Z_{t}^{x})\differential
t+\sigma(Z_{t}^{x})\differential B_{t}\ \text{with }Z_{0}^{x}=x.$ (7)

Here, $(B_{t})_{t\geq 0}$ is an $D^{\prime}$-dimensional Wiener process;
$b:\mathbb{R}^{D}\to\mathbb{R}^{D}$ and
$\sigma:\mathbb{R}^{D}\to\mathbb{R}^{D\times D^{\prime}}$ represent the drift
and the diffusion coefficients. In the following, we assume that $b$ and
$\sigma$ are locally Lipschitz. The drift coefficient $b$ is chosen so that
the diffusion has $P$ as an invariant measure:

$b(x)=\frac{1}{2p(x)}\left\langle\nabla,p(x)\\{a(x)+c(x)\\}\right\rangle,$

where $a(x)=\sigma(x)\sigma(x)^{\top}$ is the covariance coefficient,
$c(x)=-c(x)^{\top}\in\mathbb{R}^{D\times D}$ is the skew-symmetric stream
coefficient. Then, the generator ${\cal A}_{P}$ is an operator defined by

${\cal A}_{P}u_{f}(x)\coloneqq\left\langle b(x),\nabla
u_{f}(x)\right\rangle+\frac{1}{2}\left\langle\sigma(x)\sigma(x)^{\top},\nabla^{2}u_{f}(x)\right\rangle.$

Characterizing the regularity of a solution $u_{f}$ to (6) requires additional
assumptions on the diffusion. Erdogdu et al. (2018) revealed that a solution
is a pseudo-Lipschitz function for a fast-converging diffusion. To introduce
their result, we first detail required assumptions.

######  Condition 2.2 (Polynomial growth of coefficients).

For some $q_{a}\in\\{0,1\\}$ and any $x\in\mathbb{R}^{D},$ the drift and the
diffusion coefficients of (7) satisfy the growth condition

$\lVert b(x)\rVert_{2}\leq\frac{\lambda_{b}}{4}(1+\lVert
x\rVert_{2}),\lVert\sigma(x)\rVert_{\mathrm{F}}\leq\frac{\lambda_{\sigma}}{4}(1+\lVert
x\rVert_{2}),\text{and
}\lVert\sigma(x)\sigma(x)^{\top}\rVert_{\mathrm{op}}\leq\frac{\lambda_{a}}{4}(1+\lVert
x\rVert_{2}^{q_{a}+1}),$

with $\lambda_{b},\lambda_{\sigma},\lambda_{a}>0.$

######  Condition 2.3 (Dissipativity).

For $\alpha,\beta>0,$ the diffusion (7) satisfies the dissipativity condition

${\cal A}_{P}\lVert x\rVert_{2}^{2}\leq-\alpha\lVert x\rVert_{2}^{2}+\beta.$

The operator ${\cal A}_{P}$ is the generator of an Itô diffusion with
coefficients $b$ and $\sigma,$ and ${\cal{\cal A}}_{P}\lVert
x\rVert_{2}^{2}=2\left\langle
b(x),x\right\rangle+\lVert\sigma(x)\rVert_{F}^{2}.$

######  Condition 2.4 (Wasserstein rate).

For $q\geq 1,$ the diffusion $Z_{t}^{x}$ has $L^{q}$-Wasserstein rate
$\rho_{q}:[0,\infty)\to\mathbb{R}$ if

$\inf_{\mathrm{couplings}(Z_{t}^{x},Z_{t}^{y})}\mathbb{E}[\lVert
Z_{t}^{x}-Z_{t}^{y}\rVert_{2}^{q}]^{1/q}\leq\rho_{q}(t)\lVert x-y\rVert_{2}\
\text{for }x,y\in\mathbb{R}^{D}\ \text{and }t\geq 0,$

where the infimum is taken over all couplings between $Z_{t}^{x}$ and
$Z_{t}^{y}.$ We further define the relative rates

$\tilde{\rho}_{1}(t)=\log(\rho_{2}(t)/\rho_{1}(t))\ \text{and
}\tilde{\rho}_{2}(t)=\log[\rho_{1}(t)/\\{\rho_{2}(t)\rho_{1}(0)\\}]/\log[\rho_{1}(t)/\rho_{1}(0)].$

Erdogdu et al. (2018) shows that the solution $u_{f}$ to the Stein equation
(6) satisfies the following property:

######  Theorem 2.5 (Finite Stein factors from Wasserstein decay, Erdogdu et
al. 2018, Theorem 3.2).

Assume that Conditions 2.2, 2.3, and 2.4 for the $L^{1}$-Wasserstein distance
hold and that $f$ is pseudo-Lipschitz of order $q$ with at most degree-$q$
polynomial growth of its $i$-th derivatives for $i=2,3,4$. Then, the solution
$u_{f}$ to the equation (6) is pseudo-Lipschitz of order $q$ with constant
$\zeta_{1},$ and has $i$-th order derivative with degree-$q$ polynomial growth
for $i=2,3,4:$

$\lVert\nabla^{i}u_{f}(x)\rVert_{\mathrm{op}}\leq\zeta_{i}(1+\lVert
x\rVert_{2}^{q})\ \text{for }i\in\\{2,3,4\\},\ \text{and }x\in\mathbb{R}^{D}.$

The constants $\zeta_{i}$ (called Stein factors) are given as follows:

$\displaystyle\zeta_{i}$
$\displaystyle=\tau_{i}+\xi_{i}\int_{0}^{\infty}\rho_{1}(t)\omega_{q_{a}+1}(t+i-2)\differential
t\ \text{for }i=1,2,3,4,$

where

$\omega_{q_{a}+1}(t)=1+4\rho_{1}(t)^{1-1/(q_{a}+1)}\rho_{1}(0)^{\frac{1}{2}}\left[1+\frac{1}{\tilde{\alpha}_{q_{a}+1}^{q}}\\{(1\vee\tilde{\rho}_{q_{a}+1}(t))2\lambda_{a}q+3(q_{a}+1)\beta\\}^{q}\right],$

with $\tilde{\alpha}_{1}=\alpha,$ $\tilde{\alpha}_{2}=\inf_{t\geq
0}[\alpha-q\lambda_{a}(1\vee\tilde{\rho}_{2}(t)]_{+},$

$\displaystyle\tau_{1}=0,\ \tau_{i}$
$\displaystyle=\tilde{\mu}_{\mathrm{pLip}}(f)_{\mathrm{1},q}\tilde{\pi}(f)_{2:i,q}\tilde{\nu}_{1:q}(\sigma)\kappa_{q_{a}}(6q)\
\text{for }i=2,3,4,$
$\displaystyle\xi_{1}=\tilde{\mu}_{\mathrm{pLip}}(f)_{\mathrm{1},q},$
$\displaystyle\xi_{i}=\tilde{\mu}_{\mathrm{pLip}}(f)_{\mathrm{1},q}\tilde{\nu}_{1:i}(b)\tilde{\nu}_{0:i-2}(\sigma^{-1})\rho_{1}(0)\omega_{q_{a}+1}(1)\kappa_{q_{a}+1}(6q)^{i-1}\
\text{for }i=2,3,4,$

where
$\tilde{\pi}(f)_{i,q}=\sup_{x\in\mathbb{R}^{D}}\lVert\nabla^{i}f(x)\rVert_{\mathrm{op}}/(1+\lVert
x\rVert^{q}),$
$\tilde{\pi}(f)_{a:b,q}\coloneqq\max_{i=a,\dots,b}\tilde{\pi}(f)_{i,q},$
$\tilde{\nu}_{a:b}(g)$ is a constant whose precise form is given in the proof
of Erdogdu et al. (2018, Theorem 3.2), and

$\kappa_{q_{a}+1}(q)=2+\frac{2\beta}{\alpha}+\frac{q\lambda_{a}}{4\alpha}+\frac{\tilde{\alpha}_{q_{a}+1}}{\alpha}\left(\frac{q\lambda_{a}+6(q_{a}+1)\beta}{2r\tilde{\alpha}_{q_{a}+1}}\right).$

There are two known sufficient conditions for establishing exponential
Wasserstein decay (Condition 2.4). The first is uniform dissipativity, which
is a simple (but more restrictive) condition leading to exponential $L^{1}$\-
or $L^{2}$\- exponential decay rates.

######  Proposition 2.6 (Wasserstein decay from uniform dissipativity, Wang
2020, Theorem 2.5 ).

A diffusion with drift and diffusion coefficients $b$ and $\sigma$ has
$L^{q}$-Wasserstein rate $\rho_{q}(t)=e^{-rt/2},$ if for all
$x,y\in\mathbb{R}^{D},$

$2\langle
b(x)-b(y),x-y\rangle+\lVert\sigma(x)-\sigma(y)\rVert_{\mathrm{F}}^{2}-(q-2)\lVert\sigma(x)-\sigma(y)\rVert_{\mathrm{op}}^{2}\leq-r\lVert
x-y\rVert_{2}^{2}.$

The second and more general condition is distant dissipativity. Explicit
$L^{1}$-Wasserstein decay rates from distant dissipativity are obtained by the
following result of Gorham et al. 2019, which builds upon the analyses of
Eberle (2015) and Wang (2020).

######  Proposition 2.7 (Wasserstein decay from distant dissipativity, Gorham
et al. 2019, Corollary 12).

A diffusion with drift and diffusion coefficients $b$ and $\sigma$ is called
distantly dissipative if for the truncated diffusion coefficient

$\tilde{\sigma}(x)\coloneqq(\sigma(x)\sigma(x)^{\top}-s^{2}\mathrm{Id})^{1/2}\
\text{with}\ s\in[0,1/M_{0}(\sigma^{-1}))$

with
$M_{0}(\sigma^{-1})=\sup_{x\in\mathbb{R}^{D}}\lVert\sigma^{-1}(x)\rVert_{\mathrm{op}},$
it satisfies

$\displaystyle 2\frac{\langle b(x)-b(y),x-y\rangle}{s\lVert
x-y\rVert_{2}^{2}}+\frac{\lVert\tilde{\sigma}(x)-\tilde{\sigma}(y)\rVert_{\mathrm{F}}^{2}}{s^{2}\lVert
x-y\rVert_{2}^{2}}-\frac{\lVert\bigl{(}\tilde{\sigma}(x)-\tilde{\sigma}(y)\bigr{)}^{\top}(x-y)\rVert_{\mathrm{F}}^{2}}{s^{2}\lVert
x-y\rVert_{2}^{4}}$ $\displaystyle\leq\begin{cases}-K&\lVert
x-y\rVert_{2}>R\\\ L&\lVert x-y\rVert_{2}\leq R\end{cases}$

for some $K>0$ and $R,L\geq 0.$ If the distant dissipativity holds, then the
diffusion has Wasserstein rate $\rho_{1}(t)=2e^{LR^{2}/8}e^{-rt/2}$ for

$s^{2}r^{-1}\leq\begin{cases}\frac{e-1}{2}R^{2}+e\sqrt{8K^{-1}}R+4K^{-1}&\text{if}\
LR^{2}\leq 8,\\\
8\sqrt{2\pi}R^{-1}L^{-1/2}(L^{-1}+K^{-1})\exp\Bigl(\frac{LR^{2}}{8}\Bigr{missing})+32R^{-2}K^{-2}&\text{otherwise.}\end{cases}$

These two conditions also conveniently lead to the dissipativity condition
(Condition 2.3) defined above. Therefore, we assume that the diffusion
satisfies either of these conditions in the following.

###  2.4 Diffusion Stein operator and the diffusion kernel Stein discrepancy

We next recall the diffusion Stein operator of Gorham et al. (2019), which
will be the basis of the Stein discrepancy considered in this paper. For a
diffusion process defined in (7), the diffusion Stein operator is defined as
an operator that takes as input a vector-valued differentiable function
$g:\mathbb{R}^{D}\to\mathbb{R}^{D}$ and outputs a real-valued function as
follows:

$\displaystyle{\cal T}_{P}g(x)$
$\displaystyle=\frac{1}{p(x)}\langle\nabla,p(x)m(x)g(x)\rangle$
$\displaystyle=2\langle b(x),g(x)\rangle+\left\langle m(x),\nabla
g(x)\right\rangle$

where $b(x)=\langle\nabla,p(x)m(x)\rangle/(2p(x))$ and $m(x)=a(x)+c(x).$ Note
that we can recover the Langevin Stein operator (Gorham and Mackey, 2015) by
taking $a\equiv I$ and $c\equiv 0.$

Recall the Stein equation for the second-order Stein operator
$\mathcal{A}_{P}u_{f}=f-\mathbb{E}_{X\sim P}[f(X)].$ By relating the
definition of ${\cal A}_{P}$ to ${\cal T}_{P},$ we have that the function
$g_{f}=\nabla u_{f}/2$ solves the Stein equation ${\cal
T}_{P}g=f-\mathbb{E}_{X\sim P}[f(X)]$ (Gorham et al., 2019, Section 2). As a
result of Theorem 2.5, we have the following corollary:

######  Corollary 2.8.

Let $q\geq 0.$ Let $f\in C^{3}$ be a pseudo-Lipschitz function of order $q$
with derivatives satisfying the polynomial decay condition in Theorem 2.5. The
solution $g_{f}=\nabla u_{f}/2$ to the Stein equation ${\cal
T}_{P}g=f-\mathbb{E}_{X\sim P}[f(X)]$ belongs to the set

$\displaystyle{\cal G}$
$\displaystyle=\\{g:\mathbb{R}^{D}\to\mathbb{R}^{D}:\lVert
g(x)\rVert_{2}\leq\sqrt{D}\zeta_{1}(1+\lVert x\rVert_{2}^{q}),$
$\displaystyle\hphantom{=}\quad\lVert\nabla^{i}g(x)\rVert_{\mathrm{op}}\leq\zeta_{i+1}(1+\lVert
x\rVert_{2}^{q})\ \text{for }i\in\\{1,2\\},x\in\mathbb{R}^{D}\\},$

where $\zeta_{1},$ $\zeta_{2},$ and $\zeta_{3}$ are the Stein factors from
Theorem 2.5.

###### Proof.

The derivative norm bounds follow directly from Theorem 2.5. Note that the
bound on $\lVert g(x)\rVert_{2}$ follows from the pseudo-Lipschitzness of
$u_{f}$ as

$\displaystyle\lvert\partial_{j}u_{f}(x)\rvert$ $\displaystyle=\lim_{h\to
0}\left|\frac{u_{f}(x+he_{j})-u_{f}(x)}{h}\right|$
$\displaystyle\leq\lim_{h\to 0}\zeta_{1}(1+\lVert
x+he_{j}\rVert_{2}^{q-1}+\lVert x\rVert_{2}^{q-1})\frac{\lVert
he_{j}\rVert_{2}}{\lvert h\rvert}$ $\displaystyle=\zeta_{1}(1+2\lVert
x\rVert_{2}^{q-1})\leq 2\zeta_{1}(1+\lVert x\rVert_{2}^{q-1}),$

where $\\{e_{1},\dots,e_{D}\\}$ is the standard basis of $\mathbb{R}^{D}.$ ∎

The following proposition shows that the diffusion Stein operator induces
zero-mean functions (see Appendix 6.3 for a proof).

######  Proposition 2.9 (The diffusion Stein operator generates zero-mean
functions).

Let $q_{a}\in\\{0,1\\}$ be the additional growth exponent of $\lVert
a(x)\rVert_{\mathrm{op}}$ from Condition 2.2. If $q_{a}=0,$ assume $P$ has a
finite $q$-th moment; if $q_{a}=1,$ a finite $(q+1)$-th moment. Let $g\in
C^{1}$ be a function with the following growth conditions:

$\displaystyle\lVert g(x)\rVert_{\mathrm{2}}\leq C_{0}(1+\lVert
x\rVert_{2}^{q-1}),$ $\displaystyle\lVert\nabla g(x)\rVert_{\mathrm{op}}\leq
C_{1}(1+\lVert x\rVert_{2}^{q-1}),$

for each $x\in\mathbb{R}^{D},$ and some positive constants $C_{0}$ and
$C_{1}.$ Then, we have $\mathbb{E}_{X\sim P}[{\cal T}_{P}g(X)]=0.$

We can construct a computable Stein discrepancy by combining the diffusion
Stein operator with a reproducing kernel Hilbert space (RKHS) (Aronszajn,
1950). Here, we recall the definition of the diffusion kernel Stein
discrepancy (DKSD) proposed by Barp et al. (2019). For a distribution $Q$ on
$\mathbb{R}^{D},$ the DKSD is defined by

${\cal S}\bigl{(}Q,{\cal T}_{P},\mathcal{B}_{1}({\cal
G}_{\kappa})\bigr{)}=\sup_{g\in{\cal B}_{1}({\cal
G}_{\kappa})}\left|\mathbb{E}_{Y\sim Q}\left[{\cal T}_{P}g(Y)\right]\right|,$

where ${\cal B}_{1}({\cal G}_{\kappa})$ is the unit ball of a vector-valued
RKHS ${\cal G}_{\kappa}$ determined by a matrix-valued kernel
$\kappa:\mathbb{R}^{D}\times\mathbb{R}^{D}\to\mathbb{R}^{D\times D}$ (Carmeli
et al., 2006). In the following, by abuse of notation, we use ${\cal
S}\bigl{(}Q,{\cal T}_{P},{\cal G}_{\kappa}\bigr{)}$ to denote the DKSD ${\cal
S}\bigl{(}Q,{\cal T}_{P},\mathcal{B}_{1}({\cal G}_{\kappa})\bigr{)}.$ Note
that the Langevin KSD (Chwialkowski et al., 2016; Liu et al., 2016; Gorham and
Mackey, 2017) is obtained as a specific instance of the DKSD by choosing
$a(x)\equiv\mathrm{Id},$ $c\equiv 0,$ and $\kappa=k\mathrm{Id}$ for a scalar-
valued kernel $k$ and the $D$-dimensional identity matrix $\mathrm{Id}.$ Note
that in this case, the vector-valued RKHS ${\cal G}_{\kappa}$ corresponds to
the Cartesian product $\prod_{d=1}^{D}{\cal H}_{k}$ of $D$-copies of a scalar-
valued RKHS ${\cal H}_{k}$ with inner product $\langle
g,g^{\prime}\rangle=\sum_{d=1}^{D}\langle
g_{d},g_{d}^{{}^{\prime}}\rangle_{{\cal H}_{k}}.$ Also, we obtain the Langevin
KSD with a reweighted kernel $w(x)k(x,y)w(y)$ by choosing
$a(x)=w(x)\mathrm{Id}$ with $w$ a scalar-valued positive function. The use of
an RKHS leads to a closed-form expression for the DKSD,

${\cal S}(Q,{\cal T}_{P},{\cal G}_{\kappa})^{2}=\mathbb{E}_{X,X^{\prime}\sim
Q\otimes Q}[h_{p}(X,X^{\prime})],$ (8)

where

$h_{p}(x,y)=\frac{1}{p(x)p(y)}\left\langle\nabla_{y},\langle\nabla_{x},(p(x)m(x)\kappa(x,y)m(y)^{\top}p(y)\rangle\right\rangle,$

provided that $x\mapsto\lVert{\cal T}_{p}\kappa(x,\cdot)\rVert_{{\cal
G_{\kappa}}}$ is integrable with respect to $Q$ (Barp et al., 2019, Theorem
1). In particular, we can compute the DKSD exactly if the distribution $Q$ is
given by a finitely-supported distribution $Q_{N}=\sum_{i\leq
N}w_{i}\delta_{x_{i}}$ with $\\{x_{1},\dots,x_{N}\\}\subset\mathbb{R}^{D},$
$\delta_{x_{i}}$ being the Dirac measure having all mass at $x_{i},$ and
nonnegative weights $w_{i}$ such that $\sum_{i\leq N}w_{i}=1;$ in this case,
the expression (8) yields

${\cal S}(Q_{N},{\cal T}_{P},{\cal
G}_{\kappa})^{2}=\sum_{i,j=1}^{N}w_{i}w_{j}h_{p}(x_{i},x_{j}).$ (9)

Although the majority of the following analyses focus on the simple RKHS
$\kappa=k\mathrm{Id},$ we present results using a general vector-valued RKHS
where possible.

An appropriate choice of the RKHS enables the DKSD to distinguish
distributions. Proposition 1 of Barp et al. (2019) states that ${\cal
S}(Q,{\cal T}_{P},{\cal G}_{\kappa})^{2}=0$ if and only if $P=Q$ under the
following conditions: (a) $Q$ admits a differentiable density $q$ such that
$\nabla\log q-\nabla\log p$ is integrable with respect to $Q;$ (b) $\kappa$ is
integrally strictly positive definite, i.e., $\int\mu(\differential
x)^{\top}\kappa(x,y)\mu(\differential y)>0$ for any vector Borel measure
$\mu;$ and (c) the RKHS of $\kappa$ is contained in the Stein class with
respect to ${\cal T}_{Q}.$ In the next section, we show that the DKSD
separates arbitrary Borel measures from $P,$ lifting the density requirement
for $Q.$ Our result is proved by lower bounding the DKSD with a separating
IPM.

##  3 Main results

We formalize our objective in this article, as outlined in the introduction.
Let $P$ be a distribution over $\mathbb{R}^{D}$ defined by a continuously
differentiable density function $p.$ For a distribution $Q,$ we are interested
in the maximum discrepancy over a function class ${\cal F}:$ $\sup_{f\in{\cal
F}}\lvert\mathbb{E}_{X\sim P}[f(X)]-\mathbb{E}_{Y\sim Q}[f(Y)]\rvert.$ Our
focus is on functions with polynomial growth order of $q\geq 1.$ Therefore, we
specify the function class ${\cal F}$ to be a subset of pseudo-Lipschitz
functions of order $q-1$

$\displaystyle{\cal F}_{q}\coloneqq\\{$ $\displaystyle\qquad
f:\mathbb{R}^{D}\to\mathbb{R}:f\in C^{3}\ \text{with}\
\tilde{\mu}_{\mathrm{pLip}}(f)_{\mathrm{1},q-1}\leq 1$
$\displaystyle\qquad\text{and}\
\sup_{x}\lVert\nabla^{i}f(x)\rVert_{\mathrm{op}}/\bigl{(}1+\lVert
x\rVert_{2}^{q-1}\bigr{)}\leq 1\ \text{for }i\in\\{2,3\\}$ $\displaystyle\\}.$

We define an IPM corresponding to the class ${\cal F}_{q}$ by

$d_{{\cal F}_{q}}(P,Q)=\sup_{f\in{\cal{\cal F}}_{q}}\bigl{|}\mathbb{E}_{X\sim
P}[f(X)]-\mathbb{E}_{Y\sim Q}[f(Y)]\bigr{|}.$ (10)

Appendix 7.5 shows that ${\cal F}_{q}$ contains degree-$q$ polynomial
functions of $(x_{1},\dots,x_{D})\in\mathbb{R}^{D},$ if scaled appropriately.
Note that for this class, we can take the Stein factors
$\\{\zeta_{1},\zeta_{2},\zeta_{3}\\}$ that are independent of test functions
and only depend on the diffusion through $b(x)$ and $\sigma(x).$
Unfortunately, the IPM $d_{\mathcal{F}_{q}}$ is not computable as it involves
an intractable integral. Thus, we aim to relate the IPM $d_{{\cal F}_{q}}$ to
the DKSD, a computable discrepancy measure.

Before presenting results concerning the DKSD, we first show a convergence
property of $d_{{\cal F}_{q}}.$ In the following, we denote the set of
probability measures with finite $q$-th moments by ${\cal
P}_{q}\coloneqq\\{\text{\text{probability measure}\ }\mu:\int\lVert
x\rVert_{2}^{q}\differential\mu(x)<\infty\\}.$

######  Proposition 3.1.

Let $P\in{\cal P}_{q}$ be a probability measure on $\mathbb{R}^{D}$ with a
finite $q$-th moment with $q\geq 1.$ For a sequence of probability measures
$\\{Q_{1},Q_{2},\dots,\\}\subset\mathcal{P}_{q},$ the following conditions are
equivalent: (a) $d_{{\cal F}_{q}}(Q_{n},P)\to 0$ as $n\to\infty,$ and (b) as
$n\to\infty,$ the sequence $Q_{n}$ converges weakly to $P,$ and
$\mathbb{E}_{X\sim Q_{n}}\bigl{[}\lVert
X\rVert_{2}^{q}\bigr{]}\to\mathbb{E}_{X\sim P}\bigl{[}\lVert
X\rVert_{2}^{q}\bigr{]}.$

###### Proof.

We relate the metric $d_{{\cal F}_{q}}$ to the bounded-Lipschitz metric (see,
e.g., Dudley, 2002, Section 11.2) and make use of uniform integrability. See
Appendix LABEL:Subsec:Characterization-of-pseudo-Lipsc for a complete proof. ∎

The above result shows that the distance $d_{{\cal F}_{q}}$ characterizes both
weak convergence (convergence in distribution) and convergence with respect to
the $q$-th moment. For sequences in ${\cal P}_{q},$ another example of a
metric characterizing this topology is $L^{q}$-Wasserstein distance defined by
the Euclidean distance (see, e.g., Villani, 2009, Theorem 6.9).

In the following, we make an additional assumption on the growth of the stream
coefficient in the DKSD.

######  Condition 3.2 (Polynomial growth of the stream coefficient).

For any $x\in\mathbb{R}^{D},$ the stream coefficient of (7) satisfies the
growth condition

$\lVert c(x)\rVert_{\mathrm{op}}\leq\frac{\lambda_{c}}{4}(1+\lVert
x\rVert_{2})^{q_{a}+1}$

with $\lambda_{c}>0$ and $q_{a}$ as in Condition 2.2.

Note that under Conditions 2.2, 3.2, we have

$\lVert m(x)\rVert_{\mathrm{F}}\leq\sqrt{D}\lVert
m(x)\rVert_{\mathrm{op}}\leq\sqrt{D}\lambda_{m}(1+\lVert
x\rVert_{2})^{q_{a}+1}\ \text{with
}\lambda_{m}=\frac{\lambda_{a}\vee\lambda_{c}}{4}.$

###  3.1 Uniform integrability and DKSD bounds

To prove our main result, we make use of a standard notion of uniformly
integrable $q$-th moments, defined as follows:

######  Definition 3.3 (Uniform integrability with respect to $q$-th moments
(Ambrosio et al., 2005, Eq. 5.1.19)).

Let $q>0.$ A sequence of probability measures ${\cal
Q}=\\{Q_{1},Q_{2}\dots,\\}\subset{\cal P}_{q}$ is said to have uniformly
integrable $q$-th moments if

$\lim_{r\to\infty}\limsup_{n\to\infty}\int_{\\{\lVert x\rVert_{2}>r\\}}\lVert
x\rVert_{2}^{q}\differential Q_{n}(x)=0.$

It is known that for weakly converging sequences, convergence of a moment is
equivalent to the uniform integrability of the moment (Ambrosio et al., 2005,
Lemma 5.1.7) (see also Lemma 7.3 in Appendix 7.2). Intuitively, uniform
integrability is a condition enforcing that the probability mass does not
diverge too fast along the sequence. If $q=0,$ the above definition is reduced
to uniform tightness (see, e.g., Dudley, 2002, Chapter 9). The uniform
integrability is a stricter condition in that it requires the decay rate of
the tail probability to stay faster than a degree-$q$ polynomial.

Our first result shows that for sequences with a uniformly integrable moment,
the DKSD implies convergence in $d_{{\cal F}_{q}}.$ In fact, the complete
proof of this result in Appendix 6.2 provides an explicit upper bound on
$d_{{\cal F}_{q}}$ in terms of the DKSD.

######  Proposition 3.4.

Let $\Phi\in C^{2}$ be a positive definite function with non-vanishing
generalized Fourier transform $\hat{\Phi}.$ Let $w(x)=(v^{2}+\lVert
x\rVert_{2}^{2})^{q_{w}}$ with real numbers $v\geq 1$ and $q_{w}\geq 0.$ Let
${\cal G}_{k\mathrm{Id}}$ be the RKHS defined by kernel $k\mathrm{Id}$ with
$k(x,y)=\Phi_{w}(x,x^{\prime})+\ell(x,x^{\prime}),$ where
$\Phi_{w}(x,x^{\prime})=w(x)w(x^{\prime})\Phi(x-x^{\prime})$ and $\ell$ is an
optional positive definite kernel. Then, for a sequence of measures ${\cal
Q}=\\{Q_{1},Q_{2},\dots,\\}\subset{\cal P}_{q+q_{a}}$ with uniformly
integrable $(q+q_{a})$-th moments, we have ${\cal S}(Q_{n},{\cal T}_{P},{\cal
G}_{k\mathrm{Id}})\to 0$ only if $d_{{\cal F}_{q}}(P,Q_{n})\to 0.$

###### Proof.

We only provide a proof sketch here and refer the reader to Appendix 6.2 (see
Proposition 6.6 for the precise statement) for the full proof. Our strategy is
to bound the IPM $d_{{\cal F}_{q}}$ explicitly by the DKSD. Consider the Stein
equation ${\cal T}_{P}g_{f}=f-\mathbb{E}_{X\sim P}[f(X)]$ for $f\in{\cal
F}_{q}.$ We approximate ${\cal T}_{p}g_{f}$ by mollification. Specifically, we
decompose the function ${\cal T}_{P}g_{f}$ into three parts:

${\cal T}_{P}g_{f}=\underbrace{{\cal T}_{P}g_{f}-{\cal
T}_{P}g_{\mathrm{trunc}}}_{\text{Step 1: truncation}}+\underbrace{{\cal
T}_{P}g_{\mathrm{trunc}}-{\cal T}_{P}g_{\mathrm{RKHS}}}_{\text{Step 2:
approximation}}+\underbrace{\lVert g_{\mathrm{RKHS}}\rVert}_{\text{Step 3:
measuring RKHS norm}}\frac{{\cal T}_{P}g_{\mathrm{RKHS}}}{\lVert
g_{\mathrm{RKHS}}\rVert}.$

In the first step, we truncate ${\cal T}_{p}g_{f}$ to make it bounded and
ignore the tail expectation. In the second step, we approximate the truncated
function with a smooth function. In the last step, we show that the smooth
function is an RKHS function if the kernel $\Phi$ is chosen appropriately. The
expectations of truncation and approximation errors are evaluated using the
characterization of $g_{f}$ derived in Corollary 2.8; the expectation of the
third term can be bounded by the DKSD. Thus, we obtain a bound on $d_{{\cal
F}_{q}}$ in terms of the DKSD. If the DKSD term vanishes, the rest of the
error terms can be made arbitrarily small. ∎

When $q_{a}=1,$ i.e., we have a quadratic growth of the operator norm of the
covariance coefficient, we need approximating distributions
$\\{Q_{1},Q_{2},\dots\\}$ to have an extra moment. This requirement is placed
in order to make the Stein discrepancy upper bound well-defined; otherwise,
the upper bound is vacuous. This being said, if one is willing to make the
assumption $Q\neq P,$ it is appropriate to assume a higher-order moment. For
example, any numerical quadrature method is represented by a discrete measure
with a finite support, thereby guaranteeing a finite moment of any order. Note
that in proving the uniform integrability of a sequence of measures ${\cal
Q}=\\{Q_{1},Q_{2}\dots\\},$ it is not sufficient to assume that each
$Q\in{\cal Q}$ has a finite moment of a higher order (a sufficient condition
is having their moments of that order uniformly bounded).

###  3.2 The DKSD detects non-uniform integrability

In the previous section, we assumed that the approximating distributions
$\\{Q_{1},Q_{2},\dots\\}$ had uniformly integrable $q$-th moments if $\lVert
a(x)\rVert_{\mathrm{op}}$ grows linearly, or $(q+1)$-th moments if
quadratically. Proposition 3.4 indicates that we can use any kernel formed by
a positive definite function for diagnosing non-convergence. However, such a
kernel alone is not enough when the uniform integrability is violated. Indeed,
in the case of weak convergence, Gorham and Mackey (2017) demonstrated that an
inadequate choice of the kernel yields a discrepancy that converges to zero
even when the sequence is not uniformly tight and hence non-convergent; the
IMQ kernel suggested by Gorham and Mackey (2017) ensures that vanishing KSD
implies the tightness of the sequence. Analogously, we explore conditions that
allow us to check the uniform integrability using the DKSD.

The following two lemmas characterize uniform integrability using the DKSD:

######  Lemma 3.5.

Let ${\cal Q}=\\{Q_{1},Q_{2},\dots\\}\subset{\cal P}_{q}$ be a sequence of
probability measures for $q>0.$ Let ${\cal G}_{\kappa}$ be the RKHS of
$\mathbb{R}^{D}$-valued functions defined by a matrix-valued kernel
$\kappa:\mathbb{R}^{D}\times\mathbb{R}^{D}\to\mathbb{R}^{D\times D}.$ Suppose
that for any $\varepsilon>0,$ there exists $r_{\varepsilon}>0$ and a function
$g\in{\cal G}_{\kappa}$ such that ${\cal T}_{P}g(x)\geq\lVert
x\rVert_{2}^{q}1\\{\lVert x\rVert_{2}>r_{\varepsilon}\\}-\varepsilon$ for any
$x\in\mathbb{R}^{D}.$ Then, ${\cal Q}$ has uniformly integrable $q$-th moments
if ${\cal S}(Q_{n},{\cal T}_{P},{\cal G}_{\kappa})\to 0$ as $n\to\infty.$

###### Proof.

For any $\varepsilon>0,$ we have

$\int_{\\{\lVert x\rVert_{2}>r_{\varepsilon}\\}}\lVert
x\rVert_{2}^{q}\differential Q_{n}(x)\leq\int{\cal T}_{p}g(x)\differential
Q_{n}(x)\leq\lVert g\rVert_{{\cal G}_{\kappa}}{\cal S}(Q_{n},{\cal
T}_{P},{\cal G}_{\kappa})+\varepsilon.$

Letting $n\to\infty$ concludes the proof. ∎

######  Lemma 3.6 (KSD upper-bounds the integrability rate).

Let ${\cal G}_{\kappa}$ be the RKHS of $\mathbb{R}^{D}$-valued functions
defined by a matrix-valued kernel
$\kappa:\mathbb{R}^{D}\times\mathbb{R}^{D}\to\mathbb{R}^{D\times D}.$ Suppose
there exists a function $g\in{\cal G}_{\kappa}$ such that ${\cal
T}_{P}g(x)\geq\nu$ for any $x\in\mathbb{R}^{D}$ with some constant
$\nu\in\mathbb{R},$ and $\liminf\lVert x\rVert_{2}^{-(q+\theta)}{\cal
T}_{P}g(x)\geq\eta$ for some $q\geq 0,$ $\eta>0,$ and $\theta>0$ as $\lVert
x\rVert_{2}\to\infty.$ Then, for sufficiently small $\varepsilon>0,$ we have

$\displaystyle R_{q}(Q,\varepsilon)$ $\displaystyle\coloneqq\inf\left\\{r\geq
1:\int_{\\{\lVert x\rVert_{2}>r\\}}\lVert x\rVert_{2}^{q}\differential
Q(x)\leq\varepsilon\right\\}$
$\displaystyle\leq\left\\{2\left(1+\frac{q}{\theta}\right)\left(\frac{{\cal
S}(Q,{\cal T}_{P},{\cal
G}_{\kappa})-\nu}{\eta\varepsilon}\right)\right\\}^{\frac{1}{\theta}\lor\frac{q}{\theta}},$

where ${\cal S}(Q,{\cal T}_{P},{\cal G}_{\kappa})=\infty$ if $Q\notin{\cal
P}_{q+\theta}.$ Thus, for a sequence of measures
$\\{Q_{1},Q_{2}\dots\\}\subset{\cal P}_{q}$ we have

$\limsup_{n\to\infty}{\cal S}(Q_{n},{\cal T}_{P},{\cal
G}_{\kappa})<\infty\Rightarrow\limsup_{n\to\infty}R_{q}(Q_{n},\varepsilon)<\infty.$

In particular, if the sequence $\\{Q_{1},Q_{2}\dots\\}$ does not have
uniformly integrable $q$-th moments, then Stein discrepancy ${\cal
S}(Q_{n},{\cal T}_{P},{\cal G}_{\kappa})$ diverges.

###### Proof.

The proof is in Appendix 7.4.1. ∎

The quantity $R_{q}(Q,\varepsilon)$ (termed an integrability rate) in Lemma
3.6 represents the radius of a ball, outside of which the tail moment integral
becomes negligible. Note that $Q\in{\cal P}_{q}$ is equivalent to having
$R_{q}(Q,\varepsilon)<\infty$ for each $\varepsilon>0.$ In particular, if a
sequence $\\{Q_{n}\\}_{n\geq 1}$ does not have uniformly integrable $q$-th
moments, the integrability rate $R_{q}(Q_{n},\varepsilon)$ diverges.

The above two lemmas require the Stein-modified RKHS to have a function
growing at a certain rate. The first lemma requires a stronger condition in
that it requires a function that approximates the power function $\lVert
x\rVert_{2}^{q}$ arbitrarily well. The existence proof for the first lemma is
left as future work, and we focus on the second lemma. Note that the second
lemma in contrast relies on the existence of a function that behaves as
$\lVert x\rVert_{2}^{q}$ outside a ball; we can create an RKHS that satisfy
this requirement using a linear kernel and the diffusion Stein operator,
provided that the diffusion satisfies the dissipativity condition (Condition
2.3).

######  Lemma 3.7 (Tilted linear kernels have the lower bound properties).

Suppose the diffusion targeting $P$ satisfies the dissipativity condition
(Condition 2.3) with $\alpha,\beta>0$ and the coefficient condition (Condition
2.2) with $\lambda_{a}>0$ and $q_{a}\in\\{0,1\\}.$ Let $w(x)=(v^{2}+\lVert
x\rVert_{2}^{2})^{q_{w}-u}$ with $q_{w}\geq 0,\ u\geq 0,\ \text{and}\ v>0.$
Assume $(q_{w}-u)<2\alpha/\lambda_{a}$ if $q_{a}=1.$ Let

$k(x,x^{\prime})=w(x)w(x^{\prime})\langle x,x^{\prime}\rangle.$

There exists a function $g\in{\cal G}_{k\mathrm{Id}}$ such that $\lVert
g\rVert_{{\cal G}_{k\mathrm{Id}}}=\sqrt{D}$ and the corresponding diffusion
Stein operator ${\cal T}_{P}$ satisfies

${\cal T}_{P}g(x)\geq\nu\ \text{for any }x\in\mathbb{R}^{D},\ \text{and
}\liminf_{\lVert x\rVert_{2}\to\infty}\lVert x\rVert_{2}^{-2(q_{w}-u+1)}{\cal
T}_{P}g(x)\geq\eta$

for some $\nu\in\mathbb{R}$ and $\eta>0.$

###### Proof.

The proof can be found in Appendix 7.4.2. $2(\theta+1/2)$ ∎

The next result is an immediate consequence of Lemma 3.7.

######  Corollary 3.8.

Define symbols as in Lemma 3.7. For the RKHS ${\cal G}_{k\mathrm{Id}}$ of a
kernel $k=k_{1}+k_{2}$ with $k_{1}$ an arbitrary positive definite kernel and
$k_{2}$ the kernel from Lemma 3.7, there exists a function $g\in{\cal
G}_{k\mathrm{Id}}$ such that the corresponding diffusion Stein operator ${\cal
T}_{P}$ satisfies

${\cal T}_{P}g(x)\geq\nu\ \text{for any }x\in\mathbb{R}^{D},\ \text{and
}\liminf_{\lVert x\rVert_{2}\to\infty}\lVert x\rVert_{2}^{-2(q_{w}-u+1)}{\cal
T}_{P}g(x)\geq\eta$

for some $\nu\in\mathbb{R}$ and $\eta>0.$ In particular, if the RKHSs of
kernels $k_{1}$ and $k_{2}$ do not overlap, $\lVert g\rVert_{{\cal
G}_{k\mathrm{Id}}}=\sqrt{D}.$

###  3.3 Recommended kernel choice

We have established the conditions required for the DKSD to control the
pseudo-Lipschitz metric $d_{{\cal F}_{q}}.$ In the following, we present our
recommended settings.

##### Linear growth case

When $q_{a}=0,$ we need the uniform integrability with respect to the $q$-th
moment. We recommend the following kernel function

$k_{\Phi,q,\theta}(x,x^{\prime})=w_{q,\theta}(x;v)w_{q,\theta}(x^{\prime};v)\Bigl{(}\Phi(x-x^{\prime})+\tilde{k}_{\mathrm{lin}}(x,x^{\prime};v)\Bigr{)},$
(11)

where the weight $w_{q,\theta}$ is given by

$w_{q,\theta}(x;v)=(v^{2}+\lVert x\rVert_{2}^{2})^{\frac{q+\theta-1}{2}}$

with $v\geq 1,$ and $\tilde{k}_{\mathrm{lin}}$ denotes the normalized linear
kernel:

$\tilde{k}_{\mathrm{lin}}(x,x^{\prime})=\frac{v^{2}+\langle
x,x^{\prime}\rangle}{\sqrt{v^{2}+\lVert x\rVert_{2}^{2}}\sqrt{v^{2}+\lVert
x^{\prime}\rVert_{2}^{2}}}.$

This choice ensures that the two kernels in the sum (11) have the same growth
rate. The kernel $\Phi(x-x^{\prime})$ can be any function with a non-vanishing
(generalized) Fourier transform. Examples are the exponentiated quadratic (EQ)
kernel, the IMQ kernel, and the Matérn class kernels (Matérn, 1986; Stein,
1999). Note that the normalized linear kernel enables us to use light-tailed
kernels (e.g., the EQ or the Matérn kernels) that are recommended against by
Gorham and Mackey (2017). In particular, one benefit of the Matérn class is
that we can use rougher functions: the RKHS of an IMQ kernel consists of
infinitely differentiable functions, whereas a Matérn kernel can specify an
RKHS of finitely differentiable functions. The increased complexity would
render the DKSD more sensitive to the difference between distributions (see
Section 4.3).

##### Quadratic growth case

When $q_{a}=1,$ we need the uniform integrability with respect to the
$(q+1)$-th moment. Therefore, we recommend to use the same form of the kernel
as in (11) except that $w_{q,\theta}$ is replaced with

$w_{q+1,\theta}(x)=(v^{2}+\lVert x\rVert_{2}^{2})^{\frac{q+\theta}{2}}.$ (12)

###  3.4 The DKSD detects convergence

We have shown that vanishing DKSD implies convergence in $d_{{\cal F}_{q}}.$
Here, we clarify conditions on which the DKSD converges to zero.

######  Proposition 3.9.

Let ${\cal G}_{\kappa}$ be the RKHS defined by a matrix-valued kernel
$\kappa:\mathbb{R}^{D}\times\mathbb{R}^{D}\to\mathbb{R}^{D\times D}.$ Let
$q\geq 1.$ Assume that any function $g$ in the unit ball ${\cal
B}_{1}({\cal{\cal G}}_{\kappa})$ satisfies the following: there exist some
constants $C_{0},$ $C_{1},$ and $C_{2}$ such that

$\displaystyle\lVert\nabla^{i}g(x)\rVert_{\mathrm{op}}\leq C_{i}(1+\lVert
x\rVert_{2}^{q-1})\ \text{for }\ \text{any}\ x\in\mathbb{R}^{D}\ \text{and}\
i\in\\{0,1,2\\}.$

Assume a linear growth condition on $m$ (Conditions 2.2, 3.2 with $q_{a}=0).$
Assume that $b$ is $\phi_{1}(b)$-Lipschitz in the Euclidean norm, and $m$ is
$\phi_{1}(m)$-Lipschitz in the Frobenius norm. Suppose $P\in{\cal P}_{q}.$
Then,

${\cal S}(Q,{\cal T}_{P},{\cal G}_{\kappa})\leq
C_{b,m}d_{\mathrm{pLip}_{1,q}}(Q,P),$

where

$C_{b,m}=\frac{\lambda_{b}C_{1}(5+2^{q-1})}{4}+4C_{0}\phi_{1}(b)+\frac{\lambda_{m}C_{2}D(5+2^{q-1})}{2}+2\sqrt{D}C_{1}\phi_{1}(m)$

In particular, we have ${\cal S}(Q_{n},{\cal T}_{P},{\cal G}_{\kappa})\to 0$
if $d_{{\cal F}_{q}}(P,Q_{n})\to 0.$

###### Proof.

The claim follows from showing that a Stein-modified RKHS function ${\cal
T}_{P}g$ is pseudo-Lipschitz of order $q.$ The full proof is available in
Appendix 6.4. ∎

######  Proposition 3.10.

Let ${\cal G}_{\kappa}$ be the RKHS of $\mathbb{R}^{D}$-valued functions
defined by a matrix-valued kernel
$\kappa:\mathbb{R}^{D}\times\mathbb{R}^{D}\to\mathbb{R}^{D\times D}.$ Let
$q\geq 1.$ Assume that any function $g$ in the unit ball ${\cal
B}_{1}({\cal{\cal G}}_{\kappa})$ satisfies the following: there exist some
constants $C_{0},$ $C_{1},$ and $C_{2}$ such that

$\displaystyle\lVert\nabla^{i}g(x)\rVert_{\mathrm{op}}\leq C_{i}(1+\lVert
x\rVert_{2}^{q-1})\ \text{for }\ \text{any}\ x\in\mathbb{R}^{D}\ \text{and}\
i\in\\{0,1,2\\}.$

Assume a quadratic growth condition on $m$ (Conditions 2.2, 3.2 with
$q_{a}=1).$ Assume that $b$ is $\phi_{1}(b)$-Lipschitz, and $m$ is pseudo-
Lipschitz of order $1$ in the operator norm with constant
$\tilde{\mu}_{\mathrm{pLip}}(m)_{\mathrm{1},1}.$ Suppose $P\in{\cal P}_{q+1}.$
Then,

${\cal S}(Q,{\cal T}_{P},{\cal G}_{\kappa})\leq
C_{b,m}d_{\mathrm{pLip}_{1,q+1}}(Q,P),$

where

$C_{b,m}=(5+2^{q})\left(\frac{\lambda_{b}C_{1}}{4}+\frac{\lambda_{m}C_{2}D}{2}+C_{1}D\tilde{\mu}_{\mathrm{pLip}}(m)_{\mathrm{1},1}\right)+4\phi_{1}(b)C_{0}.$

In particular, we have ${\cal S}(Q_{n},{\cal T}_{P},{\cal G}_{\kappa})\to 0$
if $d_{{\cal F}_{q+1}}(P,Q_{n})\to 0.$

###### Proof.

The proof proceeds as in the previous proposition and can be found in Appendix
6.4. ∎

Note that the two propositions above require stronger convergence requirements
than that of $d_{{\cal F}_{q}}.$

##  4 Experiments

We conduct numerical experiments to examine the theory developed above. In the
first two experiments, we investigate how a kernel choice affects the KSD’s
ability to detect the non-convergence in $d_{{\cal F}_{q}}$ using simple
light-tailed and heavy-tailed target distributions. Then, we present a
cautionary case study, in which, even though the KSD will asymptotically
detect moment discrepancies by Proposition 3.4, a large sample size is needed
for the KSD to detect discrepancies arising from isolated modes.

###  4.1 The Langevin KSD

Our first problem studies the behavior of Langevin KSD corresponding to the
choice $a\equiv\mathrm{Id}$, $c\equiv 0$. This setting allows us to contrast
with the IMQ kernel previously recommended by Gorham and Mackey (2017), which
is known to control the bounded-Lipschitz metric.

####  4.1.1 Non-convergence in mean

We first consider a problem where a sequence does not converge in mean to
their target and therefore not in $d_{{\cal F}_{q}}$ $(q=1).$ In this case,
the function class $\mathcal{F}_{q}$ includes linearly growing functions and
therefore characterizes mean non-convergence. We choose a target $P$ and an
approximating sequence $\\{Q_{1},Q_{2},\dots,\\}$ as follows:

$P={\cal N}(-\bar{\mathbf{1}},\mathrm{Id}),\
Q_{n}=\left(1-\frac{1}{n+1}\right)P+\frac{1}{n+1}{\cal{\cal
N}}\bigl{(}(n+1)\bar{\mathbf{1}},\mathrm{Id}\bigr{)},\ n\geq 1,$

where $\bar{\mathbf{1}}=\mathbf{1}/\sqrt{D}$ and ${\cal N}(\mu,\Sigma)$
denotes the multivariate Gaussian distribution over $\mathbb{R}^{D}$ with mean
$\mu$ and covariance $\Sigma.$ We take $D=5.$ While the sequence
$\\{Q_{1},Q_{2},\dots,\\}$ converges weakly to $P,$ it does not converge in
mean. Indeed, by construction, the approximating sequence has the following
biased limit:

$\displaystyle\lim_{n\to\infty}\mathbb{E}_{Y\sim Q_{n}}[Y]$
$\displaystyle=\lim_{n\to\infty}\left(1-\frac{1}{n+1}\right)\mathbb{E}_{X\sim
P}[X]+\frac{1}{n+1}\mathbb{E}_{Z\sim{\cal{\cal
N}}\bigl{(}(n+1)\bar{\mathbf{1}},\mathrm{Id}\bigr{)}}[Z]$
$\displaystyle=\mathbb{E}_{X\sim P}[X]+\bar{\mathbf{1}}.$

We examine the kernel choice (11) recommended in Section 3.3. For the weight
function $w_{1,\theta}$, we take $\theta=0.5.$ We use $v=1$ for the linear
kernel and the weight function. For the translation-invariant kernel $\Phi,$
we use the IMQ kernel $k_{\mathrm{IMQ}}(x,x^{\prime})=(1+\lVert
x-x^{\prime}\rVert_{2}^{2})^{-1/2}.$ We also consider the case $\theta=0.$We
compare these two choices against using $k_{\mathrm{IMQ}}$ alone.

For the above set of kernels, we investigate how the KSD between $P$ and
$Q_{n}$ changes along the sequence. The KSD requires estimation as the
expectation with respect to $Q_{n}$ is not tractable. As $Q_{n}$ is a mixture
of two distributions, we generate two i.i.d. samples from both mixture
components, $\\{X_{1},\dots,X_{N}\\}\overset{\mathrm{i.i.d.}}{\sim}P,$
$\\{\tilde{X}_{1},\dots,\tilde{X}_{N}\\}\overset{\mathrm{i.i.d.}}{\sim}{\cal{\cal
N}}\bigl{(}(n+1)\bar{\mathbf{1}},\mathrm{Id}\bigr{)},$ and define a sample
approximation to $Q_{n}:$

$Q_{n,N}=\left(1-\frac{1}{n+1}\right)\frac{1}{N}\sum_{i=1}^{N}\delta_{X_{i}}+\frac{1}{n+1}\frac{1}{N}\sum_{i=1}^{N}\delta_{\tilde{X}_{i}}.$

As an estimate of the KSD, we compute the KSD between $P$ and $Q_{n,N};$ this
estimate can be exactly computed using (9). In the following, we set $N=500.$
One might instead generate an i.i.d. sample from $Q_{n}$ directly. This
approach might result in underestimating the KSD, since the mixture weight of
the perturbation distribution ${\cal{\cal
N}}\bigl{(}(n+1)\bar{\mathbf{1}},\mathrm{Id}\bigr{)}$ decreases as $n$ grows
and thus we may not observe samples from this distribution if the sample size
is relatively small.

(a) Off-target sequence with non-converging means with $N$ fixed at $500.$

(b) Effect of sample size $N.$ Sequence index $n$ fixed at $\tilde{n}=10^{6}.$

(c) On-target sequence formed by i.i.d samples from the target.

Figure 1: Comparisons of KSD with different kernels. Settings: (a) IMQ; the
IMQ kernel $k_{\mathrm{IMQ}}$ (solid line), (b) IMQ sum (lin.) $\theta=0$; the
sum of the IMQ kernel and the normalized linear kernel (dashed line), (c) IMQ
sum (lin.) $\theta=0.1$; the sum of the IMQ kernel and the normalized linear
kernel with tilting $\theta=0.1$ (dash-dotted line).

Figure 1a shows the change of the KSD along the sequence for the three
kernels. It can be seen that both the IMQ kernel and the sum kernel with
$\theta=0$ decay to non-zero constant values. To investigate the effect of
bias due to the finite-sample approximation, we also vary the sample size $N$
for a large value of $n;$ Figure 1b shows the result with $\tilde{n}=10^{6}.$
We can see that the KSD increases for all kernels for off-target samples,
indicating that their population KSDs are underestimated in Figure 1a. For
comparison, we also plot the KSD between $P$ and $P_{N}=\sum_{i\leq
N}\delta_{X_{i}}$ with $X_{i}\overset{\mathrm{i.i.d.}}{\sim}P$ in Figure 1c,
where all the KSDs decrease to zero. These observations imply that the IMQ
kernel alone might be sufficient for detecting mean non-convergence. In
contrast, the kernel with the additional tilting $\theta=0.1$ diverges as $n$
increases. By design, this kernel choice induces a function growing super-
linearly, and its KSD can therefore capture the non-convergence of the mean.
Remarkably, although the case $\theta=0$ is not guaranteed by our theory, the
KSD does not decay to zero, implying possibility to relax the growth condition
in Lemma 3.6.

####  4.1.2 Non-convergence in variance

As in the previous section, we consider a case where a sequence does not
converge in variance to their target and therefore not in $d_{{\cal F}_{q}}$
$(q=2).$ We use the following target and approximating sequence:

$P={\cal N}(\mathbf{0},\mathrm{Id}),\
Q_{n}=\left(1-\frac{1}{n+1}\right)P+\frac{1}{n+1}{\cal{\cal
N}}\bigl{(}\mathbf{0},2(n+1)\mathrm{Id}\bigr{)},\ n\geq 1.$

As with the mean shift problem above, this example is constructed so that the
sequence converges in distribution but not in variance, since the Gaussian in
the second term always adds diagonal covariance $2\mathrm{Id}.$

(a) Off-target sequence with non-converging variances with $N$ fixed at $500.$

(b) Effect of sample size $N.$ Sequence index $n$ fixed at $\tilde{n}=10^{6}.$
(c) On-target sequence formed by i.i.d samples from the target.

Figure 2: Comparisons of KSD with different kernels in the variance
perturbation problem. Settings: (a) IMQ; the IMQ kernel $k_{\mathrm{IMQ}}$
(solid line), (b) IMQ sum (lin.) $\theta=0$; the sum of the IMQ kernel and the
normalized linear kernel $q=1$ (dashed line), (c) IMQ sum (quad.) $\theta=0$;
the sum of the IMQ kernel and the normalized linear kernel with quadratic
tilting $q=2$ (dotted line), (d) IMQ sum (quad.) $\theta=0.1$; The sum of the
IMQ kernel and the normalized linear kernel with quadratic tilting $q=2$ and
additional reweighting $\theta=0.1$ (dash-dotted line).

Figure 2a shows the KSD’s transition along the approximating sequence for our
four kernel choices. Again, the IMQ and the IMQ sum (lin.) kernels have a
similar trend. In this case, however, their KSDs decay to zero as the sample
size $N$ increases (Figure 2b), confirming that having a function of linear
growth is not sufficient to detect the non-convergence. For the two kernels
yielding quadratically growing Stein-RKHS functions, we see that their KSDs
indeed detect the non-convergence; in particular, the behavior of the KSD with
$\theta=0$ closely follows that of $\theta=0.1.$

###  4.2 The DKSD and heavy-tailed distributions

In this section, we turn to heavy-tailed distribution to investigate the
performance of the DKSD. Heavy-tailed distributions such as Student’s
t-distributions have bounded score functions, and it is known that the
Langevin KSD fail to detect non-convergence for this target class (Gorham and
Mackey, 2017, Theorem 10).

As our target $P,$ we use the standard multivariate t-distribution with the
degrees of freedom $\nu>1$ defined by the density

$p(x)=\frac{\Gamma\left(\frac{\nu+D}{2}\right)}{\Gamma\left(\frac{\nu}{2}\right)\nu^{\frac{D}{2}}\pi^{\frac{D}{2}}}\left(1+\frac{\lVert
x\rVert_{2}^{2}}{\nu}\right)^{-\frac{\nu+D}{2}}.$

The t-distribution is uniformly dissipative with
$\sigma(x)=\sqrt{1+\nu^{-1}\lVert x\rVert_{2}^{2}}\mathrm{Id}$ and
$\lambda_{a}=4.$ If $\nu>2,$ the diffusion is dissipative (Condition 2.3) with
$\alpha=1-2\nu^{-1}$ (see Lemma 7.9). With this choice, the DKSD is reduced to
the Langevin KSD with the kernel tilted by $1+\nu^{-1}\lVert x\rVert_{2}^{2}.$
According to Lemma 3.7, the allowed power of the weight is
$q_{w}<(\alpha+1)/2<1.$ As Proposition 3.4 requires uniform integrability of
the higher moment, this bound on $q_{w}$ implies that the DKSD can only be
used for examining mean convergence. In the following, we take $\nu=4$ and
consider the same perturbation as in the previous section:

$Q_{n}=\left(1-\frac{1}{n+1}\right)P+\frac{1}{n+1}{\cal{\cal
N}}\bigl{(}(n+1)\bar{\mathbf{1}},\mathrm{Id}\bigr{)},\ n\geq 1.$

The DKSD is estimated as in the previous section.

We consider the quadratic growth case in Section 3.3 (the tilting function is
defined by Eq. 12). We compare the following two kernels: (a) $q=0$ (uniform
integrability of the first moment), (b) $q=1$ (the second moment). The case
$q=1$ conforms to the requirement of Proposition 3.4, whereas the case $q=0$
violates it. We also include the IMQ kernel as a baseline.

(a) Off-target sequence with non-converging means with $N$ fixed at $500.$

(b) Effect of sample size $N.$ Sequence index $n$ fixed at $\tilde{n}=10^{3}.$

(c) On-target sequence formed by i.i.d samples from the target.

Figure 3: Comparisons of DKSD with different kernels with the target the
standard t-distribution. Settings: (a) IMQ sum (lin.) $\theta=0.1$; the sum of
the IMQ kernel and the normalized linear kernel $q=0$ (dashed line), (b) IMQ
sum (quad.) $\theta=0.1$; The sum of the IMQ kernel and the normalized linear
kernel with quadratic tilting $q=1$ and additional reweighting $\theta=0.1$
(dash-dotted line).

Figure 3 demonstrates the result. While our theory requires the uniform
integrability of the second moment, the linear-growth DKSD detects non-
convergence; the same trend is observed for the quadratic-growth counterpart
as expected. Recall that as the linear-growth DKSD characterizes the uniform
integrability of the first moment, weak convergence control is sufficient for
determining convergence in mean. The success of the linear-growth DKSD may
therefore be attributed to its ability to control weak convergence; this
feature should be proved without the extra uniform integrability condition,
and we leave this task as future work. Finally, the stricter requirement may
be considered as an artifact of the strong claim of Proposition 3.4, which
deals with the uniform convergence under a class of pseudo-Lipschitz
functions.

We also examine a sequence formed by samples from the target, as in Section
4.1.1 (Figure 3c). Unlike the Gaussian case, three kernels show different
rates of DKSD convergence. The DKSD with growing functions are more sensitive
to discrepancies in the tail of the distribution and therefore are slower to
converge; this observation is more relevant to heavy-tailed distributions than
lighter-tailed ones.

###  4.3 Cautionary tale: distribution mixtures with isolated components

Our final experiment concerns the following distributions:

$P=\frac{1}{2}{\cal N}(\mu_{1},\mathrm{Id})+\frac{1}{2}{\cal
N}(\mu_{2},\mathrm{Id}),\ \tilde{Q}_{\pi}=\pi{\cal
N}(\mu_{1},\mathrm{Id})+(1-\pi){\cal N}(\mu_{2},\mathrm{Id}),$

where $0\leq\pi\leq 1.$ The target $P$ is supported by our theory, since
Gaussian mixtures are known to be distantly dissipative (Gorham et al., 2019,
Example 3). However, when the distance $\lVert\mu_{1}-\mu_{2}\rVert_{2}$
between the two modes is large, the KSD is unable to capture the discrepancy
of the mixture ratio $\pi$ unless a very large sample size is observed.
Indeed, the Wasserstein rate $\rho_{1}(t)=2e^{LR^{2}/8}e^{-rt/2}$ in
Proposition 2.7 has an exponent depending on
$R=\lVert\mu_{1}-\mu_{2}\rVert_{2}$, and the diffusion therefore suffers from
the slow convergence when $R$ is large, rendering the KSD insensitive to this
difference for smaller sample sizes; Figure 4 illustrates this point. This
issue has been noted by Gorham et al. (2019, Section 5.1) and Wenliang and
Kanagawa (2021).

Figure 4: KSDs for Gaussian mixture target $P,$ where mode separation
parameter $\lVert\mu_{1}-\mu_{2}\rVert_{2}$ is chosen from $\\{3,5\\}$ with
$D=5.$ The IMQ sum kernel with $\theta=0$ is used. The dashed line indicates
the KSD between $P$ and empirical measures $Q_{\pi,N}$ formed by i.i.d.
samples of size $N$ from $\tilde{Q}_{\pi}$ with $\pi=0.1.$ The dash-dotted
line shows the KSD of empirical measures $P_{N}$ given by i.i.d. samples of
size $N$ from the target $P.$ The larger the mode separation
$\lVert\mu_{1}-\mu_{2}\rVert_{2},$ the large number of sample points is
required to distinguish from the target.

We detail our experimental procedure. In this setting, we use an i.i.d. sample
$\\{X_{i}\\}_{i=1}^{N}$from $\tilde{Q}_{\pi}$ to form a sequence of empirical
distributions $Q_{\pi,N}=N^{-1}\sum_{i\leq N}\delta_{X_{i}}$ and again exactly
compute the KSD between $P$ and $Q_{\pi,N}.$ In the following, we set
$\mu_{1}=-30\cdot\mathbf{1},$ $\mu_{2}=-10\cdot\mathbf{1}$ and $D=5.$ With $N$
fixed at $500,$ we vary the mixture ratio $\pi$ from $0$ to $1/2$ using a
regular grid of size $30.$ We noticed that the KSD’s trajectory has different
trends depending on the drawn samples. We therefore repeat this procedure 100
times and provide a summary.

Our main question is whether the KSD can detect mean discrepancies due to
isolated components based on a sample of size $N.$ It is clear that the mean
of $Q_{\pi,N}$ changes significantly as $\pi$ increases to $1/2$. To this end,
we consider the sum kernel in (11) with $q=1,$ ensuring that the Stein-
modified RKHS has a function of linear growth. As the sequence has uniformly
integrable first moments, we simply do not apply the additional tilting (i.e.,
$\theta=0$). We examine two choices of the translation invariant kernel $\Phi$
: the IMQ kernel $k_{\mathrm{IMQ}}(x,x^{\prime})=\bigl{(}1+\lVert
x-x^{\prime}\rVert_{2}^{2}/\sigma^{2}\bigr{)}^{-1/2}$ and the following Matérn
kernel

$k_{\mathrm{Mat}}(x,x^{\prime})=\left(1+\frac{\sqrt{3}\lVert
x-x^{\prime}\rVert_{2}}{\sigma}\right)\exp\left(-\frac{\sqrt{3}}{\sigma}\lVert
x-x^{\prime}\rVert_{2}\right).$

First, we fix the bandwidth $\sigma$ to $1$ for both kernels. Figure 5 plots
the KSD value against the mixture ratio, along with the KSD’s density estimate
computed from different sample draws. We observe that for both kernels, their
KSDs do not change along the sequence.

Figure 5: Comparison between the IMQ and the Matérn kernels in the mixture
problem. Without bandwidth optimization. A dot represents the average of KSDs
computed with different sample draws.

Figure 6: Comparison between the IMQ and the Matérn kernels in the mixture
problem. With optimized bandwidth for each kernel. A dot represents the
average of KSDs computed with different sample draws.

This observation implies that the bandwidth choice may be suboptimal, making
the KSD too weak to detect the change in the mixing proportion. Therefore, we
next consider optimizing the bandwidth $\sigma$ for each $Q_{\pi,N}$ to
improve the sensitivity. Following the approaches in nonparametric hypothesis
testing (Gretton et al., 2012; Sutherland et al., 2016; Jitkrittum et al.,
2016, 2017), we choose a bandwidth by optimizing the power of a test using the
objective ${\cal S}(Q_{\pi,N},{\cal T}_{P},{\cal
G}_{k\mathrm{Id}})^{2}/\sqrt{v_{H_{1}}},$ where $v_{H_{1}}=\mathrm{Var}_{X\sim
Q_{\pi,N}}\bigl{[}\mathbb{E}_{X^{\prime}\sim
Q_{\pi,N}}[h_{p}(X,X^{\prime})]\bigr{]}.$ This objective is a proxy of the
power of the KSD goodness-of-fit tests (Chwialkowski et al., 2016; Liu et al.,
2016) and can be computed exactly. The optimization procedure may be
interpreted as seeking a bandwidth that allows us to conclude $P\neq
Q_{\pi,N}$ using as few sample points as possible, without using the whole of
$Q_{\pi,N}.$ We choose a bandwidth that maximizes the objective from a regular
grid between $10^{-3}$ and $10^{3}$ (in the logarithmic scale with base $10$);
the grid size $20.$

Figure 6 shows the result. Both kernels manifest decreasing trends. The IMQ
kernel tends to take larger values near $\pi=0,$ but stays at the same value
past the point $\pi=0.3.$ In contrast, the Matérn kernel keeps the decreasing
trend and captures the discrepancy. The Matérn kernel may therefore be thought
of as more stable and inducing a stringent discrepancy measure.

We can attribute the KSD’s weak dependence on the mixture ratio to the score
function (Wenliang and Kanagawa, 2021). To see this, consider a density

$r_{\pi}(x)=\pi p_{1}(x)+(1-\pi)p_{2}(x)$, 0$<\pi<$1.

The score function of $r_{\pi}$ is given by

$\mathbf{s}_{r,\pi}(x)=\tilde{\pi}(x)\mathbf{s}_{p_{1}}(x)+\bigl{(}1-\tilde{\pi}(x)\bigr{)}\mathbf{s}_{p_{2}}(x),\
\text{where}\ \tilde{\pi}(x)=\frac{\pi p_{1}(x)}{\pi
p_{1}(x)+(1-\pi)p_{2}(x)},$

and $\mathbf{s}_{p}(x)$ denotes $\nabla\log p(x)$ for a density $p.$ The
function $\tilde{\pi}(x)$ represents the posterior probability of an
observation $x$ arising from the mixture component $p_{1}.$ We claim that for
any two $\pi\neq\pi^{\prime},$ the difference between $\mathbf{s}_{r,\pi}$ and
$\mathbf{s}_{r,\pi^{\prime}}$ is virtually absent. Our reasoning is as
follows: $\mathbf{s}_{r,\pi}(x)$ becomes $\mathbf{s}_{p_{1}}$ (or
$\mathbf{s}_{p_{2}}$ ) in the high-probability region of $p_{1}$ (or $p_{2})$;
thus, the difference between two score functions is negligible under
$r_{\pi^{\prime}}$ for any other configuration $\pi^{\prime}$ so long as two
components are concentrated in separate regions as in the Gaussian mixture
example. This pathology is due to the following behavior of the posterior
density $\tilde{\pi}(x):$ it becomes effectively a binary-valued function that
outputs $1$ in the high-density region of $p_{1}$ and $0$ in the counterpart
of $p_{2}(x).$ Indeed, if $p_{1}(x)\gg p_{2}(x),$ the posterior
$\tilde{\pi}(x)\approx 1$ and is effectively independent of $\pi.$ Figure 7
illustrates this situation for a Gaussian mixture in $D=1.$ Hence, in this
case, varying the mixing proportion $\pi$ does not modify the score function
$\mathbf{s}_{r,\pi}$ significantly. Thus, as the KSD between two densities
depends on the difference of their score functions (Liu et al., 2016,
Definition 3.2), the KSD can be insensitive to mismatches of the mixture
ratios.

Figure 7: A two component Gaussian mixture and its posterior probability
$\tilde{\pi}(x)$ of component assignment.

##  5 Conclusion

In this article, we have shown that the diffusion kernel Stein discrepancy
controls the pseudo-Lipschitz metric $d_{{\cal F}_{q}},$ the worst-case
expectation error over a class of polynomially growing function. A particular
consequence of this result is that we may interpret the KSD in terms of
convergence of moments. Our experiments confirm that the kernel Stein
discrepancy with our proposed kernel choices indeed detects non-convergence in
$d_{{\cal F}_{q}}.$

A theoretical shortcoming of our result is that we need a coercive function to
characterize the uniform integrability (Lemma 3.6). Lemma 3.5 shows that we
only need a function dominating a function of the form $x\mapsto\lVert
x\rVert^{q}1_{\\{\lVert x\rVert>r\\}}.$ A related question to address is
whether we can remove the uniform integrability of a higher order moment in
Proposition 3.4. As heavy-tailed distributions have finitely many moments,
assuming a higher-order moment can be restrictive.

## References

  * Abramowitz and Stegun [1965] Milton Abramowitz and I.A. Stegun.  _Handbook of Mathematical Functions: With Formulas, Graphs, and Mathematical Tables_.  DOVER PUBN INC, June 1965.  ISBN 0486612724.  URL https://www.ebook.de/de/product/1675514/handbook_of_mathematical_functions_with_formulas_graphs_and_mathematical_tables.html.
  * Ambrosio et al. [2005] Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré.  _Gradient flows in metric spaces and in the space of probability measures_.  Lectures in Mathematics ETH Zürich. Birkhäuser Verlag, Basel, 2005\.  ISBN 978-3-7643-2428-5; 3-7643-2428-7.
  * Anastasiou et al. [2021] Andreas Anastasiou, Alessandro Barp, FranÃ§ois-Xavier Briol, Bruno Ebner, Robert E. Gaunt, Fatemeh Ghaderinezhad, Jackson Gorham, Arthur Gretton, Christophe Ley, Qiang Liu, Lester Mackey, Chris. J. Oates, Gesine Reinert, and Yvik Swan.  Stein’s method meets computational statistics: A review of some recent developments.  To appear in Statistical Science, May 2021.
  * Aronszajn [1950] N. Aronszajn.  Theory of reproducing kernels.  _Transactions of the American Mathematical Society_ , 68(3):337–337, March 1950.
  * Barp et al. [2019] Alessandro Barp, Francois-Xavier Briol, Andrew Duncan, Mark Girolami, and Lester Mackey.  Minimum Stein discrepancy estimators.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * Carmeli et al. [2006] Claudio Carmeli, Ernesto De Vito, and Alessandro Toigo.  Vector valued reproducing kernel Hilbert spaces of integrable functions and Mercer theorem.  _Analysis and Applications_ , 4(4):377–408, 2006\.  ISSN 0219-5305.  doi: 10.1142/S0219530506000838.
  * Chen et al. [2018] Wilson Ye Chen, Lester Mackey, Jackson Gorham, Francois-Xavier Briol, and Chris Oates.  Stein points.  In _Proceedings of the 35th International Conference on Machine Learning_ , pages 844–853, 2018.  URL https://proceedings.mlr.press/v80/chen18f.html.
  * Chwialkowski et al. [2016] Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton.  A kernel test of goodness of fit.  In _Proceedings of The 33rd International Conference on Machine Learning_ , pages 2606–2615, 2016.
  * Dudley [2002] R. M. Dudley.  _Real Analysis and Probability_.  Cambridge University Press, oct 2002.  doi: 10.1017/cbo9780511755347.
  * Eberle [2015] Andreas Eberle.  Reflection couplings and contraction rates for diffusions.  _Probability Theory and Related Fields_ , 166(3-4):851–886, oct 2015.  doi: 10.1007/s00440-015-0673-1.
  * Erdogdu et al. [2018] Murat A. Erdogdu, Lester Mackey, and Ohad Shamir.  Global non-convex optimization with discretized diffusions.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , pages 9694–9703, 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/3ffebb08d23c609875d7177ee769a3e9-Abstract.html.
  * Gorham and Mackey [2015] Jackson Gorham and Lester Mackey.  Measuring sample quality with Stein’s method.  In _Advances in Neural Information Processing Systems_ , volume 28, pages 226–234, 2015.
  * Gorham and Mackey [2017] Jackson Gorham and Lester Mackey.  Measuring sample quality with kernels.  In _Proceedings of The 34th International Conference on Machine Learning_ , pages 1292–1301, 2017.
  * Gorham et al. [2019] Jackson Gorham, Andrew B Duncan, Sebastian J Vollmer, and Lester Mackey.  Measuring sample quality with diffusions.  _Annals of Applied Probability_ , 2019.
  * Gretton et al. [2012] Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur.  Optimal kernel choice for large-scale two-sample tests.  In _Advances in Neural Information Processing Systems_ , volume 25, pages 1205–1213, 2012.
  * Huggins and Mackey [2018] Jonathan Huggins and Lester Mackey.  Random feature Stein discrepancies.  In _Advances in Neural Information Processing Systems_ , volume 31, pages 1899–1909, 2018.
  * Jitkrittum et al. [2016] Wittawat Jitkrittum, Zoltán Szabó, Kacper P Chwialkowski, and Arthur Gretton.  Interpretable distribution features with maximum testing power.  In _Advances in Neural Information Processing Systems_ , volume 29, pages 181–189. 2016.
  * Jitkrittum et al. [2017] Wittawat Jitkrittum, Wenkai Xu, Zoltan Szabo, Kenji Fukumizu, and Arthur Gretton.  A linear-time kernel goodness-of-fit test.  In _Advances in Neural Information Processing Systems_ , volume 30, 2017.
  * Kallenberg [2021] Olav Kallenberg.  _Foundations of Modern Probability_ , volume 99 of _Probability Theory and Stochastic Modelling_.  Springer International Publishing, third edition, 2021.  ISBN 978-3-030-61871-1; 978-3-030-61870-4.  doi: 10.1007/978-3-030-61871-1.
  * Liu et al. [2016] Qiang Liu, Jason Lee, and Michael Jordan.  A kernelized Stein discrepancy for goodness-of-fit tests.  In _Proceedings of The 33rd International Conference on Machine Learning_ , pages 276–284, 2016.
  * Matérn [1986] Bertil Matérn.  _Spatial variation_ , volume 36 of _Lecture Notes in Statistics_.  Springer-Verlag, Berlin, second edition, 1986.  ISBN 3-540-96365-0.  doi: 10.1007/978-1-4615-7892-5.  With a Swedish summary.
  * Müller [1997] Alfred Müller.  Integral probability metrics and their generating classes of functions.  _Advances in Applied Probability_ , 29(2):429–443, 1997.  ISSN 0001-8678.  doi: 10.2307/1428011.
  * Oates et al. [2017] Chris J Oates, Mark Girolami, and Nicolas Chopin.  Control functionals for Monte Carlo integration.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 79(3):695–718, 2017.
  * Ross [2011] Nathan Ross.  Fundamentals of Stein’s method.  _Probability Surveys_ , 8:210–293, 2011.
  * Stein [1972] Charles Stein.  A bound for the error in the normal approximation to the distribution of a sum of dependent random variables.  In _Proceedings of the Sixth Berkeley Symposium on Mathematical Statistics and Probability (Univ. California, Berkeley, Calif., 1970/1971), Vol. II: Probability theory_ , pages 583–602, 1972.
  * Stein [1999] Michael L. Stein.  _Interpolation of spatial data_.  Springer Series in Statistics. Springer-Verlag, New York, 1999.  ISBN 0-387-98629-4.  doi: 10.1007/978-1-4612-1494-6.  Some theory for Kriging.
  * Sutherland et al. [2016] Danica J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton.  Generative models and model criticism via optimized maximum mean discrepancy.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_. 2016.
  * Villani [2009] Cédric Villani.  _Optimal transport: old and new_.  Springer Berlin Heidelberg, 2009.  doi: 10.1007/978-3-540-71050-9.
  * Wang [2020] Feng-Yu Wang.  Exponential contraction in Wasserstein distances for diffusion semigroups with negative curvature.  _Potential Analysis_ , 53(3):1123–1144, feb 2020\.  doi: 10.1007/s11118-019-09800-z.
  * Wendland [2004] Holger Wendland.  _Scattered Data Approximation_.  Cambridge University Press, dec 2004.  doi: 10.1017/cbo9780511617539.
  * Wenliang and Kanagawa [2021] Li K. Wenliang and Heishiro Kanagawa.  Blindness of score-based methods to isolated components and mixing proportions.  In _NeurIPS Workshop "Your Model is Wrong: Robustness and misspecification in probabilistic modeling"_ , December 2021.

##  6 Proofs of main results

###  6.1 Characterization of pseudo-Lipschitz metrics

See 3.1

###### Proof.

$\text{(a)}\ \Leftarrow\ \text{(b)}$ Our goal is to show that the following
quantity can be made arbitrarily small by taking sufficiently large $n:$

$\displaystyle\sup_{f\in{\cal F}_{q}}\left|\int f\differential Q_{n}-\int
f\differential P\right|$ $\displaystyle=\sup_{f\in{\cal
F}_{q}}\left|\int\bigl{(}f-f(0)\bigr{)}\differential(Q_{n}-P)+\underbrace{\int
f(0)\differential(Q_{n}-P)}_{=0}\right|$ $\displaystyle=\sup_{f\in{\cal
F}_{q}}\left|\int\bar{f}\differential(Q_{n}-P)\right|,$

where $\bar{f}$ denotes $f-f(0).$ To this end, we smoothly truncate a function
$\bar{f}$ using the bump function $1_{R,1}$ of Lemma 7.8 with $r=R$ and
$\delta=1$ for some $R\geq 1$ (the function $1_{R,1}(x)$ vanishes if $\lVert
x\rVert_{2}>R+1).$ Specifically, we break up the integral on the RHS as

$\sup_{f\in{\cal
F}_{q}}\left|\int\bar{f}\differential(Q_{n}-P)\right|=\sup_{f\in{\cal
F}_{q}}\left|\int\bar{f}1_{R,1}\differential(Q_{n}-P)+\int\bar{f}(1-1_{R,1})\differential(Q_{n}-P)\right|$

and evaluate each term below.

As a preparatory step, we clarify some properties of $\bar{f}.$ Note that for
any $f\in{\cal F}_{q},$ we have $\lvert\bar{f}(x)\rvert=\lvert
f(x)-f(0)\rvert\leq(1+\lVert x\rVert_{2}^{q-1})\lVert x\rVert_{2}$ for
$x\in\mathbb{R}^{D}.$ This implies that we have

$\lVert\bar{f}1_{R,1}\rVert_{\infty}=\sup_{x\in\mathbb{R}}\lvert\bar{f}(x)1_{R,1}(x)\rvert<(R+1)+(R+1)^{q}\eqqcolon
C_{1,R}$

and as $\bar{f}1_{R,1}(x)$ is everywhere differentiable,

$\displaystyle\lVert\bar{f}1_{R,1}\rVert_{\mathrm{L}}=$
$\displaystyle\sup_{x,y\in\mathbb{R}^{D},x\neq
y}\frac{\lvert\bar{f}1_{R,1}(x)-\bar{f}1_{R,1}(y)\rvert}{\lVert
x-y\rVert_{2}}$
$\displaystyle\leq\sup_{x\in\mathbb{R}^{D}}1_{R,1}(x)\lVert\nabla\bar{f}(x)\rVert_{2}+\bar{f}(x)\lVert\nabla
1_{R,1}(x)\rVert_{2}$
$\displaystyle\leq\sqrt{D}\bigl{\\{}1+2(R+1)^{q-1}\bigr{\\}}+8e^{-1}\\{1+R+(R+1)^{q}\\}\eqqcolon
C_{2,R}$

Let $C_{R}=2(C_{1,R}\lor C_{2,R})$ Then, $\bar{f}1_{R,1}/C_{R}$ belongs to the
set of bounded Lipschitz functions
$\mathrm{BL}_{1}=\\{f:\mathbb{R}^{D}\to\mathbb{R}:\lVert
f\rVert_{\infty}+\lVert f\rVert_{\mathrm{L}}\leq 1\\}.$

We are ready to bound the quantify of interest. For $\varepsilon>0,$ using the
weak convergence assumption, take $n$ large enough so that

$d_{\mathrm{BL}_{1}}(P,Q_{n})\coloneqq\sup_{f\in\mathrm{BL}_{1}}\left|\int
f\differential Q_{n}-\int f\differential
P)\right|<\frac{\varepsilon}{2C_{R}},$

which is possible as the bounded Lipschitz metric $d_{\mathrm{BL}_{1}}$
metrizes weak convergence [Dudley, 2002, Section 11.3]. The definition of $R$
has been left unspecified; here, we take $R$ such that

$\int_{\\{\lVert x\rVert_{2}>R\\}}\lVert x\rVert_{2}^{q}\differential
P(x)\lor\sup_{n\geq 1}\int_{\\{\lVert x\rVert_{2}>R\\}}\lVert
x\rVert_{2}^{q}\differential Q_{n}(x)<\frac{\varepsilon}{8}.$

The existence of $R$ is guaranteed by Lemma 7.3: for a weakly converging
sequence of probability measures $\\{Q_{1},Q_{2,}\dots\\},$ the convergence in
the $q$-th moment is equivalent to the $q$-th moment uniform integrability.
Then,

$\displaystyle\sup_{f\in{\cal
F}_{q}}\left|\int\bar{f}\differential(Q_{n}-P)\right|$
$\displaystyle\leq\sup_{f\in{\cal F}_{q}}C_{R}\left|\int
C_{R}^{-1}\cdot\bar{f}1_{R,1}\differential(Q_{n}-P)\right|+\sup_{f\in{\cal
F}_{q}}\left|\int\bar{f}(1-1_{R,1})\differential(Q_{n}-P)\right|$
$\displaystyle\leq C_{R}\sup_{f\in\mathrm{BL}_{1}}\left|\int
f\differential(Q_{n}-P)\right|+\sup_{f\in{\cal
F}_{q}}\left|\int\bar{f}(1-1_{R,1})\differential Q_{n}\right|+\sup_{f\in{\cal
F}_{q}}\left|\int\bar{f}(1-1_{R,1})\differential P\right|$ $\displaystyle\leq
C_{R}\sup_{f\in\mathrm{BL}_{1}}\left|\int
f\differential(Q_{n}-P)\right|+\sup_{f\in{\cal F}_{q}}\int_{\\{\lVert
x\rVert_{2}>R\\}}\lvert\bar{f}\rvert\differential Q_{n}+\sup_{f\in{\cal
F}_{q}}\int_{\\{\lVert x\rVert_{2}>R\\}}\lvert\bar{f}\rvert\differential P$
$\displaystyle\leq
C_{R}\cdot\frac{\varepsilon}{2C_{R}}+\frac{\varepsilon}{4}+\frac{\varepsilon}{4}=\varepsilon.$

The third line follows from the bounded Lipschitzness of
$\bar{f}1_{R,1}/C_{R}$ and the triangle inequality; the fourth line is another
application of the triangle inequality and $1-1_{R,1}(x)\leq 1\\{\lVert
x\rVert_{2}>R\\};$ the fifth line is true because
$\lvert\bar{f}(x)\rvert=\lvert f(x)-f(0)\rvert\leq(1+\lVert
x\rVert_{2}^{q-1})\lVert x\rVert_{2}$ for any $x\in\mathbb{R}^{D},$ and
because of the definition of $R.$

$\text{(a)}\ \Rightarrow\ \text{(b)}$ We first prove that convergence in
$d_{{\cal F}_{q}}$ implies weak convergence. For any $\varepsilon>0$ and
$f\in\mathrm{BL}_{1},$ define the Gaussian convolution
$\tilde{f}_{\varepsilon}(x)=\mathbb{E}_{G}[f(x-\varepsilon G)]$ with $G$ is a
standard Gaussian random vector. The function is infinitely differentiable and
satisfies, by the Lipschitzness of $f,$

$\sup_{x\in\mathbb{R}^{D}}\lvert
f(x)-\tilde{f}_{\varepsilon}(x)\rvert\leq\varepsilon\mathbb{E}\bigl{[}\lVert
G\rVert_{2}\bigr{]}.$

By Lemma 7.1, we have constant bounds on the operator norms of the derivatives
up to the third order; the bounding constants depend on $\varepsilon.$ With
$C_{\varepsilon}$ the maximum of the bounding constants, we
$C_{\varepsilon}^{-1}\tilde{f}_{\varepsilon}\in{\cal F}_{q}.$ Then,

$\displaystyle
d_{\mathrm{BL}_{1}}(Q_{n},P)=\sup_{f\in\mathrm{BL}_{1}}\left|\int
f\differential Q_{n}-\int f\differential P)\right|$
$\displaystyle\leq\sup_{f\in\mathrm{BL}_{1}}\int\lvert
f-\tilde{f}_{\varepsilon}\rvert\differential Q_{n}+\int\lvert
f-\tilde{f}_{\varepsilon}\rvert\differential P+C_{\varepsilon}\left|\int
C_{\varepsilon}^{-1}\tilde{f}_{\varepsilon}\differential(Q_{n}-P)\right|$
$\displaystyle\leq 2\varepsilon\mathbb{E}\bigl{[}\lVert
G\rVert_{2}\bigr{]}+C_{\varepsilon}d_{{\cal F}_{q}}(P,Q_{n}).$

Taking the successive limits of $n,\varepsilon$ shows that
$d_{\mathrm{BL}_{1}}(Q_{n},P)\to 0,$ implying the weak convergence of the
sequence.

Next, we show that convergence in $d_{{\cal F}_{q}}$ implies
$\mathbb{E}_{X\sim Q_{n}}\lVert X\rVert_{2}^{q}\to\mathbb{E}_{X\sim P}\lVert
X\rVert_{2}^{q}.$ As the weak convergence has been established above, we only
need to show that the sequence $\\{Q_{1},Q_{2},\dots,\\}$ has uniformly
integrable $q$-th moments. As $P\in{\cal{\cal P}}_{q},$ we can take $R$
satisfying

$\int_{\lVert x\rVert_{2}>R}\lVert x\rVert_{2}^{q}\differential
P(x)\leq\varepsilon.$

Consider the function $f_{q,R}(x)=\lVert
x\rVert_{2}^{q}\bigl{(}1-1_{R,1}(x)\bigr{)}$ with the smooth bump function
$1_{R,1}$ of Lemma 7.8. This function has derivatives, up to the third order,
growing in the order of $\lVert x\rVert_{2}^{q-1},$ and therefore with a
proper scaling $C_{R}>0$ (depending on $R$), we have $\tilde{f}_{q,R}\coloneqq
C_{R}f_{q,R}\in{\cal F}_{q}.$ By the convergence in $d_{{\cal F}_{q}},$ for
any $\varepsilon>0$ we can take $N$ such that for any $n>N,$

$\displaystyle\int\tilde{f}_{q,R}\differential Q_{n}$
$\displaystyle\leq\int\tilde{f}_{q,R}\differential P+C_{R}\varepsilon.$

These choices yield,

$\displaystyle\int_{\lVert x\rVert_{2}>R+1}\lVert x\rVert_{2}^{q}\differential
Q_{n}(x)$ $\displaystyle\leq\int f_{q,R}\differential Q_{n}$
$\displaystyle\leq C_{R}^{-1}\left(\int\tilde{f}_{q,R}\differential
P+C_{R}\varepsilon\right)$ $\displaystyle\leq\int_{\lVert
x\rVert_{2}>R+1}\lVert x\rVert_{2}^{q}\differential P+\varepsilon\leq
2\varepsilon.$

Thus, we have arrived at the desired conclusion
$\lim_{r\to\infty}\limsup_{n\to\infty}\int_{\lVert x\rVert_{2}>r}\lVert
x\rVert_{2}^{q}\differential Q_{n}(x)=0.$ ∎

######  Corollary 6.1.

For $q\geq 1,$ let $d_{\mathrm{pLip}_{1,q-1}}(P,Q)$ be the IPM defined by
$\mathrm{pLip}_{1,q-1},$ the set of functions that are pseudo-Lipschitz of
order $q-1$ with its pseudo-Lipschitz constant bounded by $1.$ For a sequence
of probability measures $\\{Q_{1},Q_{2},\dots,\\}\subset\mathcal{P}_{q},$ the
following conditions are equivalent: (a)
$d_{\mathrm{pLip}_{1,q-1}}(Q_{n},P)\to 0$ as $n\to\infty,$ and (b) as
$n\to\infty,$ the sequence $Q_{n}$ converges weakly to $P,$ and
$\mathbb{E}_{X\sim Q_{n}}\lVert X\rVert_{2}^{q}\to\mathbb{E}_{X\sim P}\lVert
X\rVert_{2}^{q}.$

###### Proof.

The direction$\text{$\mathrm{(a)}$}\Rightarrow\mathrm{(b)}$ results from
Proposition 3.1, as $d_{{\cal F}_{q}}(P,Q)\leq
d_{\mathrm{pLip}_{1,q-1}}(P,Q).$ The other direction
$\text{$\mathrm{(b)}$}\Rightarrow\mathrm{(a)}$ can be shown as in the proof of
Proposition 3.1. ∎

###  6.2 Uniform integrability and a DKSD lower bound

Our goal is to show a KSD bound on the IPM $d_{{\cal F}_{q}}(P,Q).$

Let $g$ be a solution to the Stein equation ${\cal T}_{P}g=f-\mathbb{E}_{X\sim
P}[f(X)]$ for $f\in{\cal F}_{q}.$ We approximate ${\cal T}_{p}g$ by
mollification. Specifically, we decompose the function ${\cal T}_{P}g$ into
three parts:

${\cal T}_{P}g=\underbrace{{\cal T}_{P}g-{\cal
T}_{P}g_{\mathrm{trunc}}}_{\text{Step 1: truncation}}+\underbrace{{\cal
T}_{P}g_{\mathrm{trunc}}-{\cal T}_{P}g_{\mathrm{RKHS}}}_{\text{Step 2:
approximation}}+\underbrace{\lVert g_{\mathrm{RKHS}}\rVert}_{\text{Step 3:
measuring RKHS norm}}\frac{{\cal T}_{P}g_{\mathrm{RKHS}}}{\lVert
g_{\mathrm{RKHS}}\rVert}.$

We elaborate on each step as follows.

##### Step 1: Smoothly truncating $g.$

Let us start with the following notion.

######  Definition 6.2 (Integrability rate of order $q$).

For any probability measure $Q$ and $\epsilon>0,$ the integrability rate
$R_{q}(Q,\varepsilon)$ of order $q$ is defined as

$R_{q}(Q,\varepsilon)\coloneqq\inf\left\\{r\geq 1:\int_{\\{\lVert
x\rVert_{2}>r\\}}\lVert x\rVert_{2}^{q}\differential
Q(x)\leq\varepsilon\right\\},$

where we use the convention $\inf\emptyset=\infty.$

Note that $Q\in{\cal P}_{q}$ is equivalent to having
$R_{q}(Q,\varepsilon)<\infty$ for each $\varepsilon>0.$ In the following, for
each $\varepsilon>0$ and a probability measure $Q$, we consider the
integrability rate $R=R_{q+q_{a}}(Q,\varepsilon).$ Let where $1_{R,1}$ be a
smooth bump function from Lemma 7.8, which vanishes outside the centered
Euclidean ball of radius $R+1.$ For a function $g\in{\cal G},$ we consider a
truncated version $g_{R,1}\coloneqq 1_{R,1}\cdot g.$ The truncation results
yields the following error estimate:

######  Lemma 6.3.

Let $Q\in{\cal P}_{q+q_{a}}$ a probability measure and
$R=R_{q+q_{a}}(Q,\varepsilon).$ For each $\varepsilon>0,$ we have

$\left|\int{\cal T}_{P}g\differential Q-\int{\cal T}_{P}g_{R,1}\differential
Q\right|\leq c_{P,D}\cdot\varepsilon$

with
$c_{P,D}=2\sqrt{D}\zeta_{1}\left(\lambda_{b}+8e^{-1}\right)+D\lambda_{m}\zeta_{2}.$

###### Proof.

Observe that for each $x\in\mathbb{R}^{D},$

$\displaystyle\lvert{\cal T}_{P}g(x)-{\cal T}_{P}g_{R,1}(x)\rvert$
$\displaystyle\leq\lvert\langle
2b(x),g(x)(1-1_{R,1})(x)\rangle\rvert+\lvert\langle m(x),\nabla
g(1-1_{R,1})(x)\rangle\rvert$ $\displaystyle\leq 2\lVert b(x)\rVert_{2}\lVert
g(x)\rVert_{2}\lvert(1-1_{R,1})(x)\rvert+D\lvert 1-1_{R,1}(x)\rvert\lVert
m(x)\rVert_{\mathrm{\mathrm{op}}}\lVert\nabla g(x)\rVert_{\mathrm{op}}$
$\displaystyle\hphantom{}\quad+\lVert\nabla 1_{R,1}(x)\rVert_{2}\lVert
g(x)\rVert_{\mathrm{2}}$ $\displaystyle\leq 1\bigl{\\{}\lVert
x\rVert_{2}>R\bigr{\\}}\left[\Bigl{\\{}\frac{\lambda_{b}}{2}(1+\lVert
x\rVert_{2})+8e^{-1}\Bigr{\\}}\cdot
D\zeta_{1}+D\zeta_{2}\lambda_{m}\bigl{(}1+\lVert
x\rVert_{2}^{q_{a}+1}\bigr{)}\right](1+\lVert x\rVert_{2}^{q-1}).$

By the definition of $R,$ we have

$\int_{\\{\lVert x\rVert_{2}>R\\}}\lVert x\rVert_{2}^{q^{\prime}}\differential
Q\leq\int_{\\{\lVert x\rVert_{2}>R\\}}\lVert
x\rVert_{2}^{q+q_{a}}\differential Q\leq\varepsilon.$

for any $0\leq q^{\prime}<q+q_{a}.$ Thus,

$\displaystyle\left|\int{\cal T}_{P}g\differential Q-\int{\cal
T}_{P}g_{R,1}\differential Q\right|$ $\displaystyle\leq\int_{\lVert
x\rVert_{2}>R}\left[\Bigl{\\{}\frac{\lambda_{b}}{2}(1+\lVert
x\rVert_{2})+8e^{-1}\Bigr{\\}}\cdot\sqrt{D}\zeta_{1}(1+\lVert
x\rVert_{2}^{q-1})\right.$ $\displaystyle\hphantom{\leq\int_{\lVert
x\rVert_{2}>R}}\ \left.+D\zeta_{2}\lambda_{m}\bigl{(}1+\lVert
x\rVert_{2}^{q_{a}+1}\bigr{)}\bigl{(}1+\lVert
x\rVert_{2}^{q-1}\bigr{)}\right]\differential Q(x)$
$\displaystyle\leq\left\\{2\sqrt{D}\zeta_{1}\left(\lambda_{b}+8e^{-1}\right)+D\lambda_{m}\zeta_{2}\right\\}\varepsilon.$

∎

##### Step 2: Constructing a smooth approximation to $g_{R,1}$.

We consider approximating the function $g_{R,1}$ with an RKHS function. For
later use, we consider factorizing $g_{R,1}$ using a differentiable positive
function $w(x):\mathbb{R}^{D}\to[1,\infty);$ i.e.,

$g_{R,1}(x)=w(x)g_{R,1}^{w},$

where $g_{R,1}^{w}\coloneqq g_{R,1}/w.$ We assume that
$\sup_{x\in\mathbb{R}^{D}}\left\|\nabla\log w(x)\right\|_{2}\eqqcolon
M_{w}<\infty.$ We also define a helper function

$B_{w}(z)\coloneqq\sup_{x\in\mathbb{R}^{D},u\in[0,1]}\frac{w(x)}{w(x-uz)}.$

The form of $w$ will be specified below, which will be of the form
$w(x)=\bigl{(}v^{2}+\lVert x\rVert_{2}^{2}\bigr{)}^{q_{w}}$; for this $w,$ we
have $B_{w}(z)=O(\lVert z\rVert_{2}^{q_{w}}).$ Note that $0<\rho\leq 1,$
$B_{w}(\rho z)\leq B_{w}(z).$

For fixed $\rho>0,$ we define a smooth approximation $g_{\rho}^{w}$ to by
convolution,

$g_{\rho}^{w}(x)\coloneqq\mathbb{E}_{Z}\left[g_{R,1}^{w}(x-\rho Z)\right],$

where $Z$ is a $\mathbb{R}^{D}$-valued random variable with
$\mathbb{E}\bigl{[}\lVert Z\rVert_{2}B_{w}(Z)\bigr{]}<\infty$ (its law will be
specified in the sequel). The following result quantifies the approximation
error and mirrors the proof of Lemma 12 of Gorham and Mackey [2017]; here, we
do not assume the Lipschitzness of the drift $b$ (assuming the Lipschitzness
will improve the $D$-dependency of the bound).

######  Lemma 6.4.

Let $g_{R,1}^{w}\coloneqq g_{R,1}/w,$and
$g_{\rho}^{w}(x)\coloneqq\mathbb{E}_{Z}\left[g_{R,1}^{w}(x-\rho Z)\right].$
For each fixed $\rho\in(0,1],$ the approximation $g_{\rho}^{w}$ satisfies the
following:

$\displaystyle\left|\int{\cal T}_{P}(wg_{\rho}^{w})\differential Q-\int{\cal
T}_{P}g_{R,1}\differential Q\right|$ $\displaystyle\leq\rho\cdot
U_{P,D,w}\cdot(1+R+2\varepsilon)\cdot\\{1+(R+1)^{q-1}\\}$

with

$U_{P,D,w}=4\left\\{\left(2\lambda_{b}+M_{w}D\lambda_{m}\right)\cdot\tilde{u}_{P,D,w}^{(1)}+D\lambda_{m}\tilde{u}_{P,D,w}^{(2)}\right\\}$

where $\tilde{u}_{P,D,w}^{(1)}$ and $\tilde{u}_{P,D,w}^{(2)}$ are constants
given respectively in Lemmas 7.10, 7.11 with $\delta=1.$

###### Proof.

By Lemmas 7.10, 7.11, for each $x\in\mathbb{R}^{D},$ we have

$\displaystyle\lvert{\cal T}_{P}wg_{\rho}^{w}(x)-{\cal
T}_{P}wg_{R,1}^{w}(x)\rvert$ $\displaystyle\leq\lvert 2w(x)\langle
b(x),g_{\rho}^{w}(x)-g_{R,1}^{w}(x)\rangle\rvert+\lvert w(x)\langle
m(x),\nabla g_{\rho}^{w}(x)-\nabla g_{R,1}^{w}(x)\rangle\rvert$
$\displaystyle\hphantom{\leq}\quad+w(x)\lvert\langle m(x),\nabla\log
w(x)\otimes(g_{\rho}^{w}(x)-g_{K,\delta}^{w}(x))\rangle\rvert$
$\displaystyle\leq w(x)\\{2\lVert b(x)\rVert_{2}+\lVert
m(x)\rVert_{\mathrm{op}}\lVert\nabla\log w(x)\rVert_{2}\\}\lVert
g_{\rho}^{w}(x)-g_{K,\delta}^{w}(x)\rVert_{2}$
$\displaystyle\hphantom{}\quad+Dw(x)\lVert
m(x)\rVert_{\mathrm{\mathrm{op}}}\lVert\nabla g_{\rho}^{w}(x)-\nabla
g_{K,\delta}^{w}(x)\rVert_{\mathrm{op}}$
$\displaystyle\leq\rho\cdot\\{1+(R+1)^{q-1}\\}\left[\left\\{\frac{\lambda_{b}}{2}(1+\lVert
x\rVert_{2})+M_{w}\lambda_{m}(1+\lVert
x\rVert_{2}^{q_{a}+1})\right\\}\tilde{u}_{P,D,w}^{(1)}\right.$
$\displaystyle\hphantom{\leq\rho\cdot\\{1+(R+1)^{q-1}\\}}\quad\left.+D\lambda_{m}\cdot(1+\lVert
x\rVert_{2}^{q_{a}+1})\tilde{u}_{P,D,w}^{(2)}\right]$

Note that

$\displaystyle\int(1+\lVert x\rVert_{2}^{q_{a}+1})\differential Q(x)$
$\displaystyle\leq 1+R^{q_{a}+1}+\int_{\\{\lVert x\rVert_{2}>R\\}}(1+\lVert
x\rVert_{2}^{q_{a}+1})\differential Q(x)$ $\displaystyle\leq
1+R^{q_{a}+1}+2\varepsilon.$

As a consequence,

$\displaystyle\left|\int{\cal T}_{P}wg_{\rho}^{w}\differential Q-\int{\cal
T}_{P}g_{R,1}^{w}\differential Q\right|$
$\displaystyle\leq\rho(1+R^{q_{a}+1}+2\varepsilon)\left[\left\\{\frac{\lambda_{b}}{2}+M_{w}\lambda_{m}\right\\}\tilde{u}_{P,D,w}^{(1)}+D\lambda_{m}\tilde{u}_{P,D,w}^{(2)}\right]\\{1+(R+1)^{q-1}\\}.$

∎

##### Step 3: Measuring the RKHS norm of $wg_{\rho}^{w}$.

The RKHS norm of $wg_{\rho}^{w}$ is derived once we specify the convolution
distribution and the RKHS kernel function. The former will be specified below;
for the latter, we assume that the scalar kernel is given by a weighted kernel

$k(x,x^{\prime})=w(x)w(x^{\prime})\tilde{k}(x,x^{\prime}),$

where $\tilde{k}(x,x^{\prime})$ is another positive definite kernel. This
choice yields

$\lVert wg_{\rho}^{w}\rVert_{{\cal
G}_{k\mathrm{Id}}}=\sqrt{\sum_{d=1}^{D}\left\|w(g_{\rho}^{w})_{d}\right\|_{{\cal
G}_{k}}^{2}}=\sqrt{\sum_{d=1}^{D}\left\|(g_{\rho}^{w})_{d}\right\|_{{\cal
G}_{\tilde{k}}}^{2}},$

where $(g_{\rho}^{w})_{d}$ is the $d$th component of $g_{\rho}^{w}.$ Thus, we
only need to check the ${\cal G}_{\tilde{k}}$ norm of each coordinate function
of $g_{\rho}^{w}.$

We consider the tilting function $w(x)=(v^{2}+\lVert x\rVert_{2}^{2})^{q_{w}}$
with $v\geq 1$ and $q_{w}\in[0,\infty).$ We specify the law of the convolution
variable $Z$ in 6.2 (i.e., the convolution kernel) with the sinc density from
Lemma 7.5

$s_{\tilde{q}_{w}}(x)=\prod_{d=1}^{D}\frac{\mathrm{sinc}^{4\tilde{q}_{w}}(x_{d})}{S_{\tilde{q}_{w}}},$

where
$S_{\tilde{q}_{w}}=\int_{-\infty}^{\infty}\mathrm{sinc}^{4\tilde{q}_{w}}(x)\differential
x,$ $\tilde{q}_{w}=\lceil q_{w}\rceil+1$ with the symbol$\lceil x\rceil$
denoting the smallest integer greater than or equal to $x.$ Lemma 7.5
guarantees that $s_{\tilde{q}_{w}}$ is well-defined and has a finite
$\tilde{q}_{w}$-th moment. Note that by the convolution theorem and $\lVert
s_{\tilde{q}_{w}}\rVert_{L^{1}}=1$, the Fourier transform
$\hat{s}_{\tilde{q}_{w}}$ of $s_{\tilde{q}_{w}}$ satisfies

$\lvert\hat{s}_{\tilde{q}_{w}}(\omega)\rvert^{2}\leq(2\pi)^{-D}1\\{\lVert\omega\rVert_{\infty}\leq
4\tilde{q}_{w}\\}\ \text{for}\ \omega\in\mathbb{R}^{D},$

where $\lVert\omega\rVert_{\infty}=\max_{d=1,\dots,D}\lvert\omega_{i}\rvert.$

######  Lemma 6.5.

Let $g_{R,1}=g\cdot 1_{R,1}$ with $1_{R,1}$ a smooth bump function from Lemma
7.8. Let $w(x)=(v^{2}+\lVert x\rVert_{2}^{2})^{q_{w}}$ with $v\geq 1$ and
$q_{w}>0.$ Let $\Phi_{w}(x,x^{\prime})=w(x)w(x^{\prime})\Phi(x-x^{\prime})$
where $\Phi\in C^{2}$ is a positive definite function with non-vanishing
generalized Fourier transform $\hat{\Phi}.$ Define the convolution
$g_{\rho}^{w}$ of $g_{R,1}^{w}=g_{R,1}/w$ by

$g_{\rho}^{w}(x)\coloneqq\mathbb{E}_{Z}\left[g_{R,1}^{w}(x-\rho Z)\right],$

where $Z$ has the law with density $s_{\tilde{q}_{w}}$ from Lemma 7.5 with
$\tilde{q}_{w}=\lceil q_{w}\rceil+1.$ Let $k=\Phi_{w}.$ Then, the RKHS norm
$\lVert wg_{\rho}^{w}\rVert_{{\cal G}_{k\mathrm{Id}}}$ is evaluated as

$\lVert wg_{\rho}^{w}\rVert_{{\cal
G}_{k\mathrm{Id}}}\leq(2\pi)^{-D/4}D\zeta_{1}\sqrt{\sup_{\lVert\omega\rVert_{\infty}\leq
4\tilde{q}_{w}/\rho}\hat{\Phi}(\omega)^{-1}}\cdot\sqrt{\mathrm{Vol}({\cal
B}_{R+1})}\cdot(1+(R+1)^{q-2q_{w}-1}),$

where $\mathrm{Vol}(\mathcal{B}_{R+1})$ is the volume of the Euclidean ball of
radius $R+1.$ In particular, the same conclusion holds if the kernel $k$ is
given by $k=\Phi_{w}+\ell,$ where $\ell$ is another positive definite kernel.

###### Proof.

We first address the case $k(x,x^{\prime})=\Phi_{w}(x,x^{\prime}).$ As
$\bigl{(}g_{\rho}^{w}\bigr{)}_{d}$ is given by the convolution with
$s(x)=\rho^{-D}s_{\tilde{q}_{w}}(x/\rho),$ by the convolution theorem
[Wendland, 2004, Theorem 5.16], its Fourier transform of is expressed by the
product

$(2\pi)^{D/2}\widehat{(g_{R,1}^{w})_{d}}(\omega)\hat{s}(\omega)=(2\pi){}^{D/2}\widehat{(g_{R,1}^{w})_{d}}(\omega)\hat{s}_{\tilde{q}_{w}}(\rho\omega),$

(note that we are using the definition of Fourier transform in [Wendland,
2004, Definition 5.15]). Thus, by [Wendland, 2004, Theorem 10.21], the RKHS
norm of $(g_{\rho}^{w})_{d}$ is given by

$\displaystyle\left\|(g_{\rho}^{w})_{d}\right\|_{{\cal G}_{\Phi_{w}}}^{2}$
$\displaystyle=(2\pi)^{-D/2}\int\frac{\lvert\widehat{(g_{\rho}^{w})_{d}}(\omega)\rvert^{2}}{\hat{\Phi}(\omega)}\differential\omega=(2\pi)^{D/2}\int\frac{\lvert\widehat{(g_{R,1}^{w})_{d}}(\omega)\hat{s}_{\tilde{q}_{w}}(\rho\omega)\rvert^{2}}{\hat{\Phi}(\omega)}\differential\omega$
$\displaystyle\leq(2\pi)^{-D/2}\sup_{\lVert\omega\rVert_{\infty}\leq
4\tilde{q}_{w}/\rho}\hat{\Phi}(\omega)^{-1}\cdot\lVert\widehat{(g_{R,1}^{w})_{d}}\rVert_{L^{2}}^{2}$
$\displaystyle=(2\pi)^{-D/2}\sup_{\lVert\omega\rVert_{\infty}\leq
4\tilde{q}_{w}/\rho}\hat{\Phi}(\omega)^{-1}\cdot\lVert(g_{R,1}^{w})_{d}\rVert_{L^{2}}^{2},$

where the last equality is a result of the Plancherel theorem, since
$(g_{R,1}^{w})_{d}\in L^{1}\cap L^{2}.$ One can check
$\lVert(g_{R,1}^{w})_{d}\rVert_{L^{2}}\leq\sqrt{\mathrm{Vol}({\cal
B}_{R+1})}\cdot\sup_{x\in\mathbb{R}^{D}}\lvert(g_{R,1}^{w})_{d}\rvert$ and

$\displaystyle\sup_{x\in\mathbb{R}^{D}}\lvert(g_{R,1}^{w})_{d}\rvert$
$\displaystyle\leq\sqrt{D}\zeta_{1}\sup_{x\in\mathbb{R}^{D}}1_{R,1}(x)\left\\{1+\frac{\lVert
x\rVert_{2}^{q-1}}{(v^{2}+\lVert x\rVert_{2}^{2})^{q_{w}}}\right\\}$
$\displaystyle\leq\sqrt{D}\zeta_{1}(1+(R+1)^{q-2q_{w}-1}),$

Therefore, we obtain

$\displaystyle\lVert wg_{\rho}^{w}\rVert_{{\cal G}_{k\mathrm{Id}}}$
$\displaystyle=\sqrt{\sum_{d=1}^{D}\left\|w(g_{\rho}^{w})_{d}\right\|_{{\cal
G}_{k}}^{2}}=\sqrt{\sum_{d=1}^{D}\left\|(g_{\rho}^{w})_{d}\right\|_{{\cal
G}_{\Phi_{w}}}^{2}}$
$\displaystyle\leq(2\pi)^{-D/4}D\zeta_{1}\sqrt{\sup_{\lVert\omega\rVert_{\infty}\leq
4\tilde{q}_{w}/\rho}\hat{\Phi}(\omega)^{-1}}\cdot\sqrt{\mathrm{Vol}({\cal
B}_{R+1})}\cdot(1+(R+1)^{q-2q_{w}-1}).$

We next deal with the case $k=\Phi_{w}+\ell.$ According to Aronszajn [1950],
because $k$ is the sum of two kernels, the RKHS norm of ${\cal G}_{k}$ of a
(scalar-valued) function $g\in{\cal G}_{k}$ is given by

$\lVert g\rVert_{{\cal G}_{k}}=\inf\\{\lVert g_{1}\rVert_{{\cal
G}_{\Phi_{w}}}+\lVert g_{2}\rVert_{{\cal
G}_{\ell}}:g=g_{1}+g_{2},g_{1}\in{\cal G}_{\Phi_{w}},g_{2}\in{\cal
G}_{\ell}\\}.$

As we have shown that each component of $wg_{\rho}^{w}$ is an element of
${\cal G}_{\Phi_{w}},$ we have $\lVert(wg_{\rho}^{w})_{d}\rVert_{{\cal
G}_{k}}\leq\lVert(wg_{\rho}^{w})_{d}\rVert_{{\cal G}_{\Phi_{w}}}.$ The rest of
the proof follows from the definition of the vector-valued RKHS ${\cal
G}_{k\mathrm{Id}}.$ ∎

The following proposition summarizes the above three steps.

######  Proposition 6.6.

Let $\varepsilon>0$ be an arbitrary fixed positive real number. Let
$\rho\in(0,1]$ be a fixed number. Let $\Phi\in C^{2}$ be a positive definite
function with non-vanishing generalized Fourier transform $\hat{\Phi}.$ Let
$w(x)=(v^{2}+\lVert x\rVert_{2}^{2})^{q_{w}}$ with real numbers $v\geq 1$ and
$q_{w}\geq 0.$ Let ${\cal G}_{k\mathrm{Id}}$ be the RKHS defined by kernel
$k\mathrm{Id}$ with $k(x,y)=\Phi_{w}(x,x^{\prime})+\tilde{\ell}(x,x^{\prime})$
where $\Phi_{w}(x,x^{\prime})=w(x)w(x^{\prime})\Phi(x-x^{\prime}).$ Let $Q$ be
a probability measure in ${\cal P}_{q+q_{a}}$ and $R\coloneqq
R_{q+q_{a}}(Q,\varepsilon/c_{P,D}).$ Then, we have

$\displaystyle d_{{\cal F}_{q}}(P,Q)$ $\displaystyle\leq\varepsilon+\rho
U_{P,D,w}\cdot(1+R^{q_{a}+1}+2\varepsilon)\cdot\\{1+(R+1)^{q-1}\\}$
$\displaystyle\hphantom{\leq}+(2\pi)^{-\frac{D}{4}}D\zeta_{1}\sqrt{\mathrm{Vol}\bigl{(}{\cal
B}_{1}(\mathbb{R}^{D})\bigr{)}(R+1)^{D}}\cdot(1+(R+1)^{q-2q_{w}-1})F_{\Phi}(4\tilde{q}_{w}\rho^{-1}){\cal
S}(Q,{\cal T}_{P},{\cal G}_{k\mathrm{Id}}),$

where $\mathrm{\mathrm{Vol}}\bigl{(}{\cal B}_{1}(\mathbb{R}^{D})\bigr{)}$
denotes the volume of the unit ball in the Euclidean space $\mathbb{R}^{D},$
$F_{\Phi}(t)=\sup_{\lVert\omega\rVert_{\infty}\leq t}\hat{\Phi}(\omega)^{-1},$
and $\tilde{q}_{w}=\lceil q_{w}\rceil+1.$ Therefore, for a sequence of
measures $\\{Q_{1},Q_{2},\dots,\\}$ in ${\cal P}_{q+q_{a}}$ with uniformly
integrable $(q+q_{a})$-th moments, we have ${\cal S}(Q_{n},{\cal T}_{P},{\cal
G}_{k\mathrm{Id}})\to 0$ only if $d_{{\cal F}_{q}}(P,Q_{n})\to 0.$

###### Proof.

Using the results obtained so far, we have for a function $f\in{\cal F}_{q},$

$\displaystyle\lvert\mathbb{E}_{Y\sim Q}f(Y)-\mathbb{E}_{X\sim
P}[f(X)]\rvert=\lvert\mathbb{E}_{Y\sim Q}{\cal T}_{P}g(Y)\rvert$
$\displaystyle\leq\lvert\mathbb{E}_{Y\sim Q}[{\cal T}_{P}g(Y)-{\cal
T}_{P}g_{R,1}(Y)]\rvert$
$\displaystyle\hphantom{\leq}\quad+\lvert\mathbb{E}_{Y\sim Q}[{\cal
T}_{P}g_{R,1}(Y)-{\cal T}_{P}wg_{\rho}^{w}(Y)]\rvert+\lvert\mathbb{E}_{Y\sim
Q}{\cal T}_{P}wg_{\rho}^{w}(Y)\rvert$
$\displaystyle\leq\lvert\mathbb{E}_{Y\sim Q}[{\cal T}_{P}g(Y)-{\cal
T}_{P}g_{R,1}(Y)]\rvert$
$\displaystyle\hphantom{}\quad+\lvert\mathbb{E}_{Y\sim Q}[{\cal
T}_{P}g_{R,1}(Y)-{\cal
T}_{P}wg_{\rho}^{w}(Y)]\rvert+\left\|wg_{\rho}^{w}\right\|_{{\cal
G}_{kI}}{\cal S}(Q,{\cal T}_{P},{\cal G}_{kI})$
$\displaystyle\leq\varepsilon+\rho
U_{P,D,w}(1+R^{2}+2\varepsilon)\cdot\\{1+(R+1)^{q-1}\\}$
$\displaystyle\hphantom{\leq}\quad+(2\pi)^{-D/4}D\zeta_{1}F_{\Phi}(4\tilde{q}_{w}\rho^{-1})\sqrt{\mathrm{Vol}\bigl{(}\mathcal{B}_{1}(\mathbb{R}^{D})\bigr{)}(R+1)^{D}}{\cal
S}(Q,{\cal T}_{P},{\cal G}_{k}).$

Taking the supremum over ${\cal F}_{q}$ completes the proof of the first
claim.

For the second claim, if the sequence has uniformly integrable $q$-th moments,
we can take finite $r\geq 1$ such that $R_{q+q_{a}}(Q_{n},\varepsilon)\leq r$
for all $n\geq 1.$ Thus, if ${\cal S}(Q_{n},{\cal T}_{P},{\cal
G}_{k\mathrm{Id}})\to 0,$ taking successive limits of $n,\rho,\epsilon$ shows
$d_{{\cal F}_{q}}(P,Q_{n})\to 0.$ ∎

###  6.3 The diffusion Stein operator and zero-mean functions

See 2.9

###### Proof.

The proof is essentially that of Gorham et al. [2019, Proposition 3]. Note
that by the moment assumption on $P,$ we have $\mathbb{E}_{X\sim
P}\left[(1+\lVert X\rVert_{2}^{q_{a}+1})(1+\lVert
X\rVert_{2}^{q-1})\right]<\infty$ and thus

$\displaystyle\mathbb{E}_{X\sim P}\lvert{\cal T}_{P}g(X)\rvert\leq
2\mathbb{E}_{X\sim P}\lVert b(X)\rVert_{2}\lVert
g(X)\rVert_{2}+D\mathbb{E}_{X\sim P}\lVert
m(x)\rVert_{\mathrm{op}}\lVert\nabla g(X)\rVert_{\mathrm{op}}<\infty.$

Thus, we may apply the dominated convergence theorem and then the divergence
theorem to obtain

$\displaystyle\mathbb{E}_{X\sim P}[{\cal T}_{P}g(X)]$
$\displaystyle=\lim_{r\to\infty}\int_{{\cal
B}_{r}}\langle\nabla,p(x)\\{a(x)+c(x)\\}g(x)\rangle\differential x$
$\displaystyle=\lim_{r\to\infty}\int_{\partial{\cal B}_{r}}\langle
n_{r}(z),\\{a(x)+c(x)\\}g(z)\rangle p(z)\differential z,$

where $\differential z$ denotes the $(D-1)$-dimensional Hausdorff measure,
${\cal B}_{r}=\\{x\in\mathbb{R}^{D}:\lVert x\rVert_{2}\leq r\\},$ and
$\partial{\cal B}_{r}=\\{x\in\mathbb{R}^{D}:\lVert x\rVert_{2}=r\\}.$ Let

$\displaystyle f(r)$ $\displaystyle=\int_{\partial{\cal B}_{r}}\lVert
a(z)+c(z)\rVert_{\mathrm{op}}\lVert g(z)\rVert_{2}p(z)\differential z$

Then we have

$\int_{\partial{\cal B}_{r}}\langle n_{r}(z),g(z)\rangle p(z)\differential
z\leq f(r).$

By the coarea formula (and integration with polar coordinates), we have

$\displaystyle\int_{0}^{\infty}f(r)\differential r$
$\displaystyle=\int_{0}^{\infty}\left\\{\int_{\partial{\cal B}_{r}}\lVert
a(x)+c(x)\rVert_{\mathrm{op}}\lVert g(z)\rVert_{2}p(z)\differential
z\right\\}\differential r$ $\displaystyle\leq
2\lambda_{m}C_{0}\int_{0}^{\infty}\left\\{\int_{\partial{\cal B}_{r}}(1+\lVert
z\rVert_{2}^{q_{a}+1})(1+\lVert z\rVert_{2}^{q-1})p(z)\differential
z\right\\}\differential r$ $\displaystyle=2\lambda_{m}C_{0}\int(1+\lVert
x\rVert_{2}^{q_{a}+1})(1+\lVert x\rVert_{2}^{q-1})p(x)\differential x<\infty.$

Thus, we have $\liminf_{r\to\infty}f(r)=0$ and therefore $\mathbb{E}_{X\sim
P}[{\cal T}_{P}g(X)]=0.$ ∎

###  6.4 DKSD upper bounds

See 3.9

###### Proof.

For any $g\in{\cal B}_{1}({\cal{\cal G}}_{\kappa}),$ we show that ${\cal
T}_{P}g$ is a pseudo-Lipschitz function of order $q.$ By the derivative
assumptions, we have

$\lVert g(x)-g(y)\rVert_{2}\leq\frac{C_{1}}{2}\bigl{(}1+\lVert
x\rVert_{2}^{q-1}+\lVert y\rVert_{2}^{q-1}\bigr{)}\lVert x-y\rVert_{2},$

and

$\lVert\nabla g(x)-\nabla g(y)\rVert_{2}\leq\frac{C_{2}}{2}\bigl{(}1+\lVert
x\rVert_{2}^{q-1}+\lVert y\rVert_{2}^{q-1}\bigr{)}\lVert x-y\rVert_{2}.$

Also,

$\frac{(1+\lVert x\rVert_{2})\bigl{(}1+\lVert x\rVert_{2}^{q-1}+\lVert
y\rVert_{2}^{q-1}\bigr{)}}{\bigl{(}1+\lVert x\rVert_{2}^{q}+\lVert
y\rVert_{2}^{q}\bigr{)}}\bigl{(}1+\lVert x\rVert_{2}^{q}+\lVert
y\rVert_{2}^{q}\bigr{)}\leq 5+2^{q-1}$

Using these estimates, we obtain

$\displaystyle\lvert{\cal T}_{P}g(x)-{\cal T}_{P}g(y)\rvert$
$\displaystyle\leq 2\lVert b(x)\rVert_{2}\lVert g(x)-g(y)\rVert_{2}+2\lVert
b(x)-b(y)\rVert_{2}\lVert g(y)\rVert_{2}$ $\displaystyle\hphantom{=}+D\lVert
m(x)\rVert_{\mathrm{op}}\lVert\nabla g(x)-\nabla
g(y)\rVert_{\mathrm{op}}+\sqrt{D}\lVert
m(x)-m(y)\rVert_{\mathrm{F}}\lVert\nabla g(y)\rVert_{\mathrm{op}}$
$\displaystyle\leq C_{b,m}(1+\lVert x\rVert_{2}^{q}+\lVert
y\rVert_{2}^{q})\lVert x-y\rVert_{2},$

where

$C_{b,m}=\frac{\lambda_{b}C_{1}(5+2^{q-1})}{4}+4C_{0}\phi_{1}(b)+\frac{\lambda_{m}C_{2}D(5+2^{q-1})}{2}+2\sqrt{D}C_{1}\phi_{1}(m).$

As a result, for any $g\in{\cal B}_{1}({\cal{\cal G}}_{\kappa}),$

$\displaystyle\left|\mathbb{E}_{X\sim Q}{\cal T}_{P}g(X)\right|$
$\displaystyle=C_{b,m}C_{b,m}^{-1}\left|\mathbb{E}_{X\sim Q}{\cal
T}_{P}g(X)-\mathbb{E}_{Y\sim P}{\cal T}_{P}g(Y)\right|$ $\displaystyle\leq
C_{b,m}d_{{\cal F}_{q}}(Q_{n},P),$

where the equality holds since $\mathbb{E}_{Y\sim P}{\cal T}_{P}g(Y)=0$ by
Proposition 2.9. Taking the supremum over ${\cal B}_{1}({\cal G}_{\kappa})$
provides the required relation ${\cal S}(Q,{\cal T}_{P},{\cal G}_{\kappa})\leq
C_{b,m}d_{{\cal F}_{q}}(Q_{n},P).$ ∎

See 3.10

###### Proof.

The proof proceeds as in Proposition 3.9. For any $g\in{\cal B}_{1}({\cal
G}_{\kappa}),$ we have

$\displaystyle\lvert{\cal T}_{P}g(x)-{\cal T}_{P}g(y)\rvert$
$\displaystyle\leq 2\lVert b(x)\rVert_{2}\lVert g(x)-g(y)\rVert_{2}+2\lVert
b(x)-b(y)\rVert_{2}\lVert g(y)\rVert_{2}$ $\displaystyle\hphantom{=}+D\lVert
m(x)\rVert_{\mathrm{op}}\lVert\nabla g(x)-\nabla
g(y)\rVert_{\mathrm{op}}+\sqrt{D}\lVert
m(x)-m(y)\rVert_{\mathrm{F}}\lVert\nabla g(y)\rVert_{\mathrm{op}}$
$\displaystyle\leq\frac{\lambda_{b}C_{1}}{4}(1+\lVert
x\rVert_{2})\bigl{(}1+\lVert x\rVert_{2}^{q-1}+\lVert
y\rVert_{2}^{q-1}\bigr{)}\lVert x-y\rVert_{2}+2\phi_{1}(b)C_{0}(1+\lVert
x\rVert_{2}^{q-1})\lVert x-y\rVert_{2}$
$\displaystyle\hphantom{=}+\frac{\lambda_{m}C_{2}D}{8}(1+\lVert
x\rVert_{2}^{2})(1+\lVert x\rVert_{2}^{q-1}+\lVert y\rVert_{2}^{q-1})\lVert
x-y\rVert_{2}$
$\displaystyle\hphantom{=}+C_{1}D\tilde{\mu}_{\mathrm{pLip}}(m)_{\mathrm{1},1}(1+\lVert
x\rVert_{2}^{q-1})(1+\lVert x\rVert_{2}+\lVert y\rVert_{2})\lVert
x-y\rVert_{2}$ $\displaystyle\leq C_{b,m}(1+\lVert x\rVert_{2}^{q+1}+\lVert
y\rVert^{q+1})\lVert x-y\rVert_{2},$

where

$C_{b,m}=(5+2^{q})\left(\frac{\lambda_{b}C_{1}}{4}+\frac{\lambda_{m}C_{2}D}{2}+C_{1}D\tilde{\mu}_{\mathrm{pLip}}(m)_{\mathrm{1},1}\right)+4\phi_{1}(b)C_{0}.$

∎

##  7 Auxiliary results

###  7.1 Results from previous work

######  Lemma 7.1 (An extended version of Gorham et al. [2019, Lemma 17]).

Let $G$ be a $D$-dimensional standard normal random vector, and fix $s>0.$ If
$f:\mathbb{R}^{D}\to\mathbb{R}$ bounded and measurable, and
$f_{s}(x)=\mathbb{E}[f(x+sG)]$, then

$\displaystyle M_{0}(f_{s})$ $\displaystyle\leq M_{0}(f),\
M_{1}(f_{s})\leq\sqrt{\frac{2}{\pi}}\frac{M_{0}(f)}{s},$ $\displaystyle
M_{2}(f_{s})$ $\displaystyle\leq\sqrt{2}\frac{M_{0}(f)}{s^{2}},\ \text{and}\
M_{3}(f_{s})\leq\frac{3M_{0}(f)}{s^{3}},$

where $M_{0}(f_{s})=\sup_{x\in\mathbb{R}^{D}}\lvert f_{s}(x)\rvert,$ and
$M_{i}(f_{s})=\sup_{x\in\mathbb{R}^{D}}\lVert\nabla^{i}f_{s}(x)\rVert_{\mathrm{op}}$
for $i\in\\{1,2,3\\}.$

###### Proof.

We prove the bound on $M_{3}(f_{s}),$ as the other bounds are given in Gorham
et al. [2019, Lemma 17]. Let $\phi_{s}\in C^{\infty}$ be the density of $sG$
and $*$ be the convolution operator. By Leibniz’s rule,

$\left\langle\nabla^{3}f_{s},u_{1}\otimes u_{2}\otimes
u_{3}\right\rangle=\left\langle\bigl{(}f*\nabla^{3}\phi_{s}\bigr{)}(x),u_{1}\otimes
u_{2}\otimes u_{3}\right\rangle.$

The RHS can be evaluated as

$\displaystyle\left|\left\langle\bigl{(}f*\nabla^{3}\phi_{s}\bigr{)}(x),u_{1}\otimes
u_{2}\otimes u_{3}\right\rangle\right|$ $\displaystyle=\left|\int
f(x-y)\left\langle\nabla^{3}\phi_{s}(y),u_{1}\otimes u_{2}\otimes
u_{3}\right\rangle\differential y\right|$
$\displaystyle\leq\frac{M_{0}(f)}{s^{6}}\int\left|\left\\{\prod_{i=1}^{3}\langle
y,u_{i}\rangle-s^{2}\sum_{ijk}\langle u_{i},y\rangle\langle
u_{j},u_{k}\rangle\right\\}\right|\phi_{s}(y)\differential y$
$\displaystyle\leq\frac{M_{0}(f)}{s^{6}}\sqrt{\int\left\\{\prod_{i=1}^{3}\langle
y,u_{i}\rangle-s^{2}\sum_{ijk}\langle u_{i},y\rangle\langle
u_{j},u_{k}\rangle\right\\}^{2}\phi_{s}(y)\differential y}$
$\displaystyle=\frac{M_{0}(f)}{s^{6}}\sqrt{s^{6}\mathbb{E}\left[\Bigl{(}\sum_{ijk}\langle
u_{i},G\rangle\langle u_{j},u_{k}\rangle\Bigr{)}^{2}\right]}$
$\displaystyle\leq\frac{3M_{0}(f)}{s^{3}}\cdot\lVert u_{1}\rVert_{2}\lVert
u_{2}\rVert_{2}\lVert u_{3}\rVert_{2}.$

where $\sum_{ijk}$ denotes the sum over the choices of $(i,j,k)$ from
$\\{(1,2,3),(2,1,3),(3,1,2)\\},$ the equality holds by Isserlis’ theorem, and
the last inequality follows from the Cauchy-Schwarz inequality after expanding
the square. ∎

###  7.2 Miscellaneous results

######  Definition 7.2 ($f$-uniform integrability).

Let ${\cal Q}$ be a set of probability measures on $\mathbb{R}^{D},$ and
$f:\mathbb{R}^{D}\to[0,\infty)$ be a nonnegative function that is integrable
for each $Q\in{\cal Q}.$ The function $f$ is called uniformly integrable with
respect to ${\cal Q}$, or the set ${\cal Q}$ is called $f$-uniformly
integrable if

$\lim_{r\to\infty}\sup_{Q\in{\cal Q}}\int_{\\{f(x)>r\\}}f\differential Q=0.$

Note that for a sequence${\cal Q}=\\{Q_{1},Q_{2}\dots\\},$ the $f$-uniform
integrability is equivalent to having

$\lim_{r\to\infty}\limsup_{n\to\infty}\int_{\\{f(x)>r\\}}f\differential
Q_{n}=0.$

The following lemma is an analogue of Kallenberg [2021, Lemma 5.11]:

######  Lemma 7.3.

Let $Q_{1},Q_{2},\dots$ be a sequence of probability measures on a separable
metric space $\mathcal{X}$ weakly converging to $Q.$ Then, for a nonnegative
continuous function $f,$ we have

  1. 1.

$\int f\differential Q\leq\liminf_{n\to\infty}\int f\differential Q_{n},$

  2. 2.

$\lim_{n\to\infty}\int f\differential Q_{n}\to\int f\differential Q<\infty$
$\Leftrightarrow$${\cal Q}=\\{Q_{1},Q_{2},\dots\\}$ is $f$-uniformly
integrable.

###### Proof.

The proof follows Kallenberg [2021, Lemma 5.11]. Below, we use the notation
$\mu(f)=\int f\differential\mu$ for a measure $\mu.$

For any $r>0,$ $x\mapsto f(x)\wedge r$ is a bounded continuous function. Thus,
by the weak convergence assumption,

$\displaystyle\liminf_{n\to\infty}Q_{n}(f)$
$\displaystyle\geq\liminf_{n\to\infty}Q_{n}(f\wedge r)$
$\displaystyle=Q(f\wedge r).$

The first claim follows as we let $r\to\infty.$

For the second claim, let us assume that ${\cal Q}$ is $f$-uniformly
integrable. For any $r>0,$ we have

$\displaystyle\lvert Q_{n}(f)-Q(f)\rvert$ $\displaystyle\leq\lvert
Q_{n}(f)-Q_{n}(f\wedge r)\rvert+\lvert Q_{n}(f\wedge r)-Q_{n}(f\wedge
r)\rvert+\lvert Q(f\wedge r)-Q(f)\rvert.$

The first term on the RHS satisfies

$\displaystyle\lvert Q_{n}(f)-Q_{n}(f\wedge r)\rvert$
$\displaystyle=Q_{n}\bigl{(}(f-r)1_{\\{f>r\\}}\bigr{)}$ $\displaystyle\leq
2Q_{n}\bigl{(}f1_{\\{f>r\\}}\bigr{)}$ $\displaystyle\leq
2\sup_{n}Q_{n}\bigl{(}f1_{\\{f>r\\}}\bigr{)}.$

Note that $Q(f)\leq\liminf Q_{n}(f)<\infty.$ Thus, letting $n\to\infty$ and
then $r\to\infty$ proves the claim. For the other direction, assume
$Q_{n}(f)\to Q(f)<\infty.$ With a fixed $r>0,$ as $n\to\infty,$

$\displaystyle Q_{n}(f1_{\\{f>r\\}})$ $\displaystyle\leq
Q_{n}\bigl{(}f-f\wedge(r-f)_{+}\bigr{)}$ $\displaystyle\to
Q\bigl{(}f-f\wedge(r-f)_{+}\bigr{)},$

where we denote $(a)_{+}=\max(a,0).$ Since $x\mapsto f(x)\wedge(r-f(x))_{+}$
converges to $f$ point-wise as $r\to\infty,$ by the dominated convergence
theorem we have $Q\bigl{(}f-f\wedge(r-f)_{+}\bigr{)}\to 0.$ ∎

######  Lemma 7.4.

Let $w(x)=\Bigl{(}a+b\lVert x\rVert_{2}^{2}\Bigr{)}^{q}$ with $a\geq 1,$
$b>0,$ and $q>0.$ Then,

$\displaystyle B_{w}(z)$
$\displaystyle\coloneqq\sup_{x\in\mathbb{R}^{D},u\in[0,1]}\frac{w(x)}{w(x-uz)}$
$\displaystyle\leq\left(1+2\left(1+\frac{b\lVert
z\rVert_{2}^{2}}{a}\right)\right)^{q}$

and

$M_{w}\coloneqq\sup_{x\in\mathbb{R}^{D}}\lVert\nabla\log
w(x)\rVert_{2}=\sup_{x\in\mathbb{R}^{D}}\frac{2bq\lVert x\rVert_{2}}{a+b\lVert
x\rVert_{2}^{2}}\leq q\sqrt{\frac{b}{a}}.$

###### Proof.

We have

$\displaystyle\frac{w(x+uz)}{w(x)}$ $\displaystyle=\left(\frac{a+b\lVert
x+uz\rVert_{2}^{2}}{a+b\lVert
x\rVert_{2}^{2}}\right)^{q}\leq\left(1+\frac{\lVert
x+uz\rVert_{2}^{2}}{(a/b)+\lVert x\rVert_{2}^{2}}\right)^{q}$
$\displaystyle\leq\left(1+2\frac{\lVert x\rVert_{2}^{2}+\lVert
z\rVert_{2}^{2}}{(a/b)+\lVert x\rVert_{2}^{q}}\right)^{q}$
$\displaystyle\leq\left(1+2\left(1+\frac{b\lVert
z\rVert_{2}^{2}}{a}\right)\right)^{q}.$

The first claim follows by observing

$\sup_{x\in\mathbb{R}^{D},u\in[0,1]}\frac{w(x)}{w(x-uz)}\leq\sup_{u\in[0,1]}\sup_{x\in\mathbb{R}^{D}}\frac{w(x+uz)}{w(x)}.$

The second claim can be checked easily. ∎

######  Lemma 7.5 (A sinc function density and its moments).

For an integer $q\geq 1,$ let $s_{q}:\mathbb{R}^{D}\to(0,\infty)$ be a
probability density function defined by

$s_{q}(x)=\left(\frac{1}{I_{4q,\infty}}\right)^{D}\prod_{d=1}^{D}(\mathrm{sinc}(x_{d}))^{4q},$

where

$\mathrm{sinc}:\mathbb{R}\to\mathbb{R},\
\mathrm{sinc}(x)=\begin{cases}\frac{\sin(x)}{x}&x\neq 0\\\ 1&x=0\end{cases},$

and $I_{4q,t}=2\int_{0}^{t}\mathrm{sinc}^{4q}(x)\differential x$ for
$t\in(0,\infty].$ Let $Z$ be a random variable with the law specified by
$s_{4q}.$ Then, the $q$-th moment $\mathbb{E}[\lVert Z\rVert_{2}^{q}]$ is
finite. Specifically, it has the following upper bound:

$\mathbb{E}[\lVert Z\rVert_{2}^{q^{\prime}}]\leq\begin{cases}\frac{3D\log
2}{\pi}&q=1,\\\
D^{q^{\prime}}\left(\frac{I_{4q,1}}{I_{4q,\infty}}+\frac{1}{2q-q^{\prime}}\frac{1}{I_{4q,\infty}}\right)^{D},&q>1,\end{cases}$

where $q^{\prime}$ is an integer such that $1\leq q^{\prime}\leq q.$

###### Proof.

The density $s_{q}$ is nonnegative due to the even power. We have

$I_{4q,\infty}=\int_{0}^{\infty}\mathrm{sinc}^{4q}(x)\differential
x\leq\int_{0}^{\infty}\mathrm{sinc}^{2}(x)\differential x=\frac{\pi}{2},$

where inequality holds as $\lvert\mathrm{sinc}^{2}(x)\rvert\leq 1$ everywhere,
and the upper integral value is obtained by integration by parts and using
$\int_{0}^{\infty}\mathrm{sinc}(x)=\pi/2$ [see, e.g., Abramowitz and Stegun,
1965, Chapter 5]. Thus, $I_{4q,\infty}<\infty,$ and the existence of the
density $s_{q}$ is guaranteed.

Assume $q>1,$ the moment estimates are derived as follows:

$\displaystyle I_{4q,\infty}^{D}\mathbb{E}[\lVert Z\rVert_{2}^{q^{\prime}}]$
$\displaystyle=\int\lVert z\rVert_{2}^{q^{\prime}}s_{q}(z)\differential
z\leq\int\lVert z\rVert_{1}^{q^{\prime}}s_{q}(z)\differential z$
$\displaystyle=\sum_{i_{1}+\cdots+i_{D}=q^{\prime}}\left(\frac{q^{\prime}!}{i_{1}!i_{2}!\cdots
i_{D}!}\right)\prod_{d=1}^{D}2\int_{0}^{\infty}\frac{\sin^{2q}(z_{d})}{\lvert
z_{d}\rvert^{2q-i_{d}}}\left(\frac{\sin(z_{d})}{z_{d}}\right)^{2q}\differential
z_{d}.$

The inequality holds because $\lVert\cdot\rVert_{1}$ upper bounds
$\lVert\cdot\rVert_{2}.$ By noting that $\mathrm{sinc}(x)\leq 1$ for all
$x\in\mathbb{R},$ we can further evaluate the integral:

$\displaystyle\int_{0}^{\infty}\frac{\sin^{2q}(z_{d})}{\lvert
z_{d}\rvert^{2q-i_{d}}}\left(\frac{\sin(z_{d})}{z_{d}}\right)^{2q}\differential
z_{d}$
$\displaystyle\leq\int_{0}^{1}\left(\frac{\sin(z_{d})}{z_{d}}\right)^{2q}\differential
z_{d}+\int_{1}^{\infty}\frac{1}{\lvert z_{d}\rvert^{2q-i_{d}}}\differential
z_{d}$
$\displaystyle=\frac{I_{4q,1}}{2}+\frac{1}{(2q-i_{d})-1}\leq\frac{I_{4q,1}}{2}+\frac{1}{(2q-q^{\prime})-1}.$

Thus, we have

$\displaystyle\mathbb{E}[\lVert Z\rVert_{2}^{q^{\prime}}]$ $\displaystyle\leq
D^{q^{\prime}}\left(\frac{I_{4q,1}}{I_{4q,\infty}}+\frac{2}{(2q-q^{\prime})-1}\frac{1}{I_{4q,\infty}}\right)^{D}.$

When $q=1,$ a similar computation shows

$\displaystyle\mathbb{E}[\lVert Z\rVert_{2}]$
$\displaystyle\leq\frac{2D}{I_{4,\infty}^{D}}\int_{0}^{\infty}\frac{\sin^{4}(z_{d})}{z_{d}^{3}}\differential
z_{d}\cdot\prod_{d^{\prime}\neq
d}^{D}\int\frac{\sin^{4}(z_{d})}{z_{d}^{4}}\differential z_{d^{\prime}}$
$\displaystyle=\frac{2D\log 2}{I_{4,\infty}}=\frac{3D\log 2}{\pi}\cdot$

∎

######  Remark 7.6.

The tedious expectation estimate is derived to ensure that the inside of the
power function is small (typically less than 1).

######  Lemma 7.7.

Let $f(t)=e^{-1/t}1_{(0,\infty)}(t).$ Let

$\text{$h(t)$}=\frac{f\bigl{(}r+\delta-t\bigr{)}}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}.$

Then, $h(t)=0$ for $t\geq r+\delta,$ $h(t)=1$ for $t\leq r,$ and $0<h(t)<1$
otherwise. Moreover, the first two derivatives of $h$ satisfies the following:
for any $t\geq 0,$

$\displaystyle\left|\derivative{h}{t}\right|(t)\leq\begin{cases}\frac{2}{\delta^{2}}&\delta\leq
1/2\\\ 8e^{1/\delta-2}&\delta>1/2\end{cases},$
$\displaystyle\left|\derivative[2]{h}{t}\right|(t)\leq\begin{cases}2\left(\frac{1-2\delta}{\delta^{4}}\right)+\frac{2}{\delta^{4}}\left(1+4e^{-1/\delta}\right)&0<\delta\leq
1/2-\sqrt{1/12},\\\
\frac{96\sqrt{3}}{(\sqrt{3}-1)^{4}}e^{-\frac{2\sqrt{3}}{\sqrt{3}-1}}e^{1/\delta}+\frac{2}{\delta^{4}}\left(1+4e^{-1/\delta}\right)&1/2-\sqrt{1/12}<\delta\leq
1/2,\\\
\left(\frac{96\sqrt{3}}{(\sqrt{3}-1)^{4}}e^{-\frac{2\sqrt{3}}{\sqrt{3}-1}}+32e^{-4}(e^{1/\delta}+4)\right)e^{1/\delta}.&\delta>1/2.\end{cases}$

###### Proof.

The cutoff property of $h$ is well known, and therefore we focus only on the
bounds on the derivatives.

As a preparatory step, let us write down the first three derivatives of $f,$
which are given, for $t>0,$ as follows:

$\displaystyle f^{\prime}(t)=\frac{1}{t^{2}}e^{-1/t},$ $\displaystyle
f^{\prime\prime}(t)=\frac{1}{t^{4}}e^{-1/t}(-2t+1),\ \text{and}$
$\displaystyle
f^{(3)}(t)=\frac{1}{t^{6}}e^{-1/t}(6t^{2}-6t+1)=\frac{6}{t^{6}}e^{-1/t}\left\\{t-\left(\frac{1}{2}+\sqrt{\frac{1}{12}}\right)\right\\}\left\\{t-\left(\frac{1}{2}-\sqrt{\frac{1}{12}}\right)\right\\}.$

The first derivative $f^{\prime}$ increases from $0$ to $1/2$ and then
decreases. Thus, $f^{\prime}(t)\leq f^{\prime}(\delta\land 1/2)$ for
$\delta>0$ and $0<t\leq\delta.$ The second derivative $f^{\prime\prime}$
increases from zero to its maximum at $t=\left(1/2-\sqrt{1/12}\right),$ and we
have $f^{\prime\prime}(t)\leq
f^{\prime\prime}\bigl{\\{}\delta\land(1/2-\sqrt{1/12})\bigr{\\}}$ for
$\delta>0$ and $0<t\leq\delta.$ Also, note that we have for $0\leq
t-r\leq\delta,$

$\frac{1}{f\bigl{(}\delta-(t-r)\bigr{)}+f(t-r)}\leq\frac{1}{f(\delta)}.$

Now we evaluate the first derivative of $h.$ We consider the region
$r<t<r+\delta,$ as otherwise $h$ is constant. Using $h\leq 1,$ we obtain

$\displaystyle\left|\derivative{h}{t}\right|$
$\displaystyle=\left|-\frac{f^{\prime}\bigl{(}r+\delta-t\bigr{)}f\bigl{(}t-r\bigr{)}}{\left(f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}\right)^{2}}+\frac{f\bigl{(}r+\delta-t\bigr{)}f^{\prime}\bigl{(}t-r\bigr{)}}{\left(f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}\right)^{2}}\right|$
$\displaystyle\leq\left|\frac{f^{\prime}\bigl{(}r+\delta-t\bigr{)}}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}\right|+\left|\frac{f^{\prime}\bigl{(}t-r\bigr{)}}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}\right|$
$\displaystyle\leq 2\frac{f^{\prime}(\delta\land 1/2)}{f(\delta)}$
$\displaystyle=\begin{cases}\frac{2}{\delta^{2}}&\delta\leq 1/2\\\
8e^{1/\delta-2}&\delta>1/2\end{cases}.$

The second derivative can be similarly evaluated as follows:

$\displaystyle\left|\derivative[2]{h}{t}\right|$
$\displaystyle=\left|-\frac{f^{{}^{\prime\prime}}(r+\delta-t)f(t-r)-2f^{\prime}(r+\delta-t)f^{\prime}(r-t)+f^{\prime\prime}(t-r)f(r+\delta-t)}{\left(f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}\right)^{2}}\right.$
$\displaystyle\hphantom{=}\left.-2\frac{\bigl{(}-f^{\prime}(r+\delta-t)f(r-t)+f^{\prime}(t-r)f(r+\delta-t)\bigr{)}^{2}}{\left(f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}\right)^{3}}\right|$
$\displaystyle\leq\left|\frac{f^{{}^{\prime\prime}}(r+\delta-t)}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}\right|+2\left|\frac{f^{\prime}(r+\delta-t)f^{\prime}(r-t)}{\left(f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}\right)^{2}}\right|$
$\displaystyle\hphantom{\leq}\
+\left|\frac{f^{\prime\prime}(t-r)}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}\right|$
$\displaystyle\hphantom{\leq}+2\left|\frac{-f^{\prime}(r+\delta-t)f(r-t)+f^{\prime}(t-r)f(r+\delta-t)}{\left(f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}\right)}\right|^{2}\frac{1}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}$
$\displaystyle\leq\left|\frac{f^{{}^{\prime\prime}}(r+\delta-t)}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}\right|+2\left|\frac{f^{\prime}(r+\delta-t)f^{\prime}(t-r)}{\left(f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}\right)^{2}}\right|$
$\displaystyle\hphantom{\leq}\
+\left|\frac{f^{\prime\prime}(t-r)}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}\right|+2\frac{\left(\left|f^{\prime}(r+\delta-t)\right|+\left|f^{\prime}(t-r)\right|\right)^{2}}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}.$

Evaluating each term yields

$\displaystyle\left|\derivative[2]{h}{t}\right|(t)$ $\displaystyle\leq
2\left|\frac{f^{\prime\prime}\bigl{\\{}\delta\land(1/2-\sqrt{1/12})\bigr{\\}}}{f(\delta)}\right|+2\left|\frac{f^{\prime}(\delta\land
1/2)}{f(\delta)}\right|^{2}+8\frac{\left|f^{\prime}(\delta\land
1/2)\right|^{2}}{f\bigl{(}\delta)}.$
$\displaystyle\leq\begin{cases}2\left(\frac{1-2\delta}{\delta^{4}}\right)+\frac{2}{\delta^{4}}\left(1+4e^{-1/\delta}\right)&0<\delta\leq\frac{1}{2}-\frac{1}{\sqrt{12}},\\\
\frac{96\sqrt{3}}{(\sqrt{3}-1)^{4}}e^{-\frac{2\sqrt{3}}{\sqrt{3}-1}}e^{1/\delta}+\frac{2}{\delta^{4}}\left(1+4e^{-1/\delta}\right)&\frac{1}{2}-\frac{1}{\sqrt{12}}<\delta\leq
1/2,\\\
\left(\frac{96\sqrt{3}}{(\sqrt{3}-1)^{4}}e^{-\frac{2\sqrt{3}}{\sqrt{3}-1}}+32e^{-4}(e^{1/\delta}+4)\right)e^{1/\delta}.&\delta>\frac{1}{2}.\end{cases}$

∎

######  Lemma 7.8.

Let $f(t)=e^{-1/t}1_{(0,\infty)}(t).$ Let $1_{r,\delta}(x)=h\bigl{(}\lVert
x\rVert_{2}\bigr{)}$ be a smooth bump function defined by

$\text{$h(t)$}=\frac{f\bigl{(}r+\delta-t\bigr{)}}{f\bigl{(}r+\delta-t\bigr{)}+f\bigl{(}t-r\bigr{)}}.$

Then, we have $1_{r,\delta}(x)=1$ for $\lVert x\rVert_{2}\leq r,$
$0<1_{r,\delta}(x)<1$ for $r<\lVert x\rVert_{2}<r+\delta,$ and
$1_{r,\delta}(x)=0$ for $\lVert x\rVert_{2}\geq r+\delta;$ the gradient
$\nabla 1_{r,\delta}(x)$ vanishes outside $\\{x\in\mathbb{R}^{D}:r<\lVert
x\rVert_{2}<r+\delta\\}.$ Furthermore, we can uniformly bound the derivative
norms $\lVert\nabla_{x}1_{r,\delta}(x)\rVert_{2}$ and
$\lVert\nabla_{x}1_{r,\delta}(x)\rVert_{2}$ by constants depending only on
$\delta.$ Specifically,

$\displaystyle\lVert\nabla_{x}1_{r,\delta}(x)\rVert_{2}$
$\displaystyle\leq\frac{2}{\delta^{2}}1_{(0,1/2]}(\delta)+8e^{1/\delta-2}1_{(1/2,\infty)}(\delta)$

and

$\displaystyle\lVert\nabla_{x}^{2}1_{r,\delta}(x)\rVert_{\mathrm{op}}$
$\displaystyle<\left|\derivative[2]{h}{r}\bigl{(}\lVert
x\rVert_{2}\bigr{)}\right|+\frac{2}{r}\left|\derivative{h}{r}\bigl{(}\lVert
x\rVert_{2}\bigr{)}\right|$
$\displaystyle\leq\begin{cases}\frac{2}{r\delta^{2}}+2\left(\frac{1-2\delta}{\delta^{4}}\right)+\frac{2}{\delta^{4}}\left(1+4e^{-1/\delta}\right)&0<\delta\leq\frac{1}{2}-\frac{1}{\sqrt{12}},\\\
\frac{2}{r\delta^{2}}+\frac{96\sqrt{3}}{(\sqrt{3}-1)^{4}}e^{-\frac{2\sqrt{3}}{\sqrt{3}-1}}e^{1/\delta}+\frac{2}{\delta^{4}}\left(1+4e^{-1/\delta}\right)&\frac{1}{2}-\frac{1}{\sqrt{12}}<\delta\leq\frac{1}{2},\\\
\frac{8}{r}e^{1/\delta-2}+\left(\frac{96\sqrt{3}}{(\sqrt{3}-1)^{4}}e^{-\frac{2\sqrt{3}}{\sqrt{3}-1}}+32e^{-4}(e^{1/\delta}+4)\right)e^{1/\delta}.&\delta>\frac{1}{2}.\end{cases}$

###### Proof.

We restrict the evaluation of the derivatives to the region $r<\lVert
x\rVert_{2}\leq r+\delta,$ as otherwise $h$ is constant and has zero
derivatives. According to Lemma 7.7, the first derivative satisfies

$\displaystyle\lVert\nabla_{x}h(\lVert x\rVert_{2})\rVert_{2}$
$\displaystyle=\left\|\frac{x}{\lVert
x\rVert_{2}}\left.\derivative{h}{t}\right|_{t=\lVert x\rVert_{2}}\right\|_{2}$
$\displaystyle=\left|\derivative{h}{t}\bigl{(}\lVert x\rVert\bigr{)}\right|$
$\displaystyle\leq\frac{2}{\delta^{2}}1_{(0,1/2]}(\delta)+8e^{1/\delta-2}1_{(1/2,\infty)}(\delta).$

The second derivative can also be evaluated as

$\displaystyle\nabla_{x}^{2}h(\lVert x\rVert_{2})=$
$\displaystyle\derivative[2]{h}{t}(\lVert x\rVert_{2})\frac{x\otimes x}{\lVert
x\rVert^{2}}+\derivative{h}{t}(\lVert x\rVert_{2})\frac{1}{\lVert
x\rVert}\left(I-\frac{x\otimes x}{\lVert x\rVert^{2}}\right).$

Consequently, we have

$\displaystyle\lVert\nabla_{x}^{2}h(\lVert x\rVert_{2})\rVert_{\mathrm{op}}$
$\displaystyle=\sup_{\lVert u\rVert_{2}=\lVert v\rVert_{2}=1,}\langle
u,\nabla_{x}^{2}h(\lVert x\rVert_{2})v\rangle$
$\displaystyle<\left|\derivative[2]{h}{r}\bigl{(}\lVert
x\rVert_{2}\bigr{)}\right|+\frac{2}{r}\left|\derivative{h}{r}\bigl{(}\lVert
x\rVert_{2}\bigr{)}\right|.$

The rest of the proof follows from the estimates of Lemma 7.7. ∎

######  Lemma 7.9.

For $\nu>2,$ let the standard multivariate t-distribution be

$p(x)=\frac{\Gamma\left(\frac{\nu+D}{2}\right)}{\Gamma\left(\frac{\nu}{2}\right)\nu^{\frac{D}{2}}\pi^{\frac{D}{2}}}\left(1+\frac{\lVert
x\rVert_{2}^{2}}{\nu}\right)^{-\frac{\nu+D}{2}}.$

Then, the t-distribution satisfies the dissipativity condition in Proposition
2.6 with $q=1$ and $\sigma(x)=\sqrt{1+\nu^{-1}\lVert
x\rVert_{2}^{2}}\mathrm{Id}.$ For this choice, we have $\lambda_{a}=4,$ and
the diffusion is dissipative (Condition 2.3) with $\alpha=1-2\nu^{-1}.$

###### Proof.

For this density, we have

$\displaystyle\nabla\log p(x)=$
$\displaystyle-\frac{\nu+D}{\nu}\frac{x}{1+\nu^{-1}\lVert x\rVert_{2}^{2}},\
a(x)=1+\nu^{-1}\lVert x\rVert_{2}^{2},\ \text{and}$ $\displaystyle 2b(x)$
$\displaystyle=a(x)\nabla\log
p(x)+\langle\nabla,a(x)\rangle=-\frac{\nu+D-2}{\nu}x.$

Thus,

$\displaystyle 2\langle
b(x)-b(y),x-y\rangle+\lVert\sigma(x)-\sigma(y)\rVert_{\mathrm{F}}^{2}-\lVert\sigma(x)-\sigma(y)\rVert_{\mathrm{op}}^{2}$
$\displaystyle=-\frac{\nu+D-2}{\nu}\lVert
x-y\rVert_{2}^{2}+(D-1)\left(\sqrt{1+\nu^{-1}\lVert
x\rVert_{2}^{2}}-\sqrt{1+\nu^{-1}\lVert y\rVert_{2}^{2}}\right)^{2}$
$\displaystyle\leq-\frac{\nu+D-2}{\nu}\lVert
x-y\rVert_{2}^{2}+\frac{D-1}{\nu}\lVert x-y\rVert_{2}^{2}$
$\displaystyle=-\left(1-\frac{1}{\nu}\right)\lVert x-y\rVert_{2}^{2}.$

In particular,

$\displaystyle{\cal A}_{P}\lVert x\rVert_{2}^{2}$
$\displaystyle{\cal=}2\langle
b(x),x\rangle+\lVert\sigma(x)\rVert_{\mathrm{F}}^{2}$
$\displaystyle=-\frac{\nu+D-2}{\nu}\lVert x\rVert^{2}+D(1+\nu^{-1}\lVert