#  Learning heavy-tailed distributions with Wasserstein-proximal-regularized
$\alpha$-divergences

Ziyu Chen
Department of Mathematics and Statistics
University of Massachusetts Amherst
Amherst, MA 01003, USA
<EMAIL_ADDRESS>
&Hyemin Gu
Department of Mathematics and Statistics
University of Massachusetts Amherst
Amherst, MA 01003, USA
<EMAIL_ADDRESS>
&Markos A. Katsoulakis
Department of Mathematics and Statistics
University of Massachusetts Amherst
Amherst, MA 01003, USA
<EMAIL_ADDRESS>
&Luc Rey-Bellet
Department of Mathematics and Statistics
University of Massachusetts Amherst
Amherst, MA 01003, USA
<EMAIL_ADDRESS>&Wei Zhu
Department of Mathematics and Statistics
University of Massachusetts Amherst
Amherst, MA 01003, USA
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose Wasserstein proximals of $\alpha$-divergences as
suitable objective functionals for learning heavy-tailed distributions in a
stable manner. First, we provide sufficient, and in some cases necessary,
relations among data dimension, $\alpha$, and the decay rate of data
distributions for the Wasserstein-proximal-regularized divergence to be
finite. Finite-sample convergence rates for the estimation in the case of the
Wasserstein-1 proximal divergences are then provided under certain tail
conditions. Numerical experiments demonstrate stable learning of heavy-tailed
distributions—even those without first or second moment—without any explicit
knowledge of the tail behavior, using suitable generative models such as GANs
and flow-based models related to our proposed Wasserstein-proximal-regularized
$\alpha$-divergences. Heuristically, $\alpha$-divergences handle the heavy
tails and Wasserstein proximals allow non-absolute continuity between
distributions and control the velocities of flow-based algorithms as they
learn the target distribution deep into the tails.

##  1 Introduction

Heavy tails are ubiquitous, emerging in various fields such as extreme events
in ocean waves [9], floods [21], social sciences [27, 16], human activities
[17, 35], biology [18] and computer sciences [29]. Learning to generate heavy-
tailed target distributions has been explored using GANs through tail
estimation [10, 15, 1]. While estimating the tail behavior of a heavy-tailed
distribution is important, selecting objectives that measure discrepancies
between these distributions and facilitate stable learning is equally crucial.
In generative modeling, the goal is to generate samples that mimic those from
an underlying data distribution, typically by designing algorithms that
minimize a probability divergence between the generated and target
distributions. Thus, it is crucial to choose a divergence that flexibly and
accurately respects the behavior of the data distribution.

In this work, we propose the Wasserstein-proximal-regularized
$\alpha$-divergence ($W_{p}$-$\alpha$-divergence) as the objective functional
for learning heavy-tailed distributions. It is defined as

$D_{\alpha,p}^{\lambda}(P\|Q)\mathrel{\mathop{\mathchar
58\relax}}=\inf_{\eta\in\mathcal{P}(\mathbb{R}^{d})}\\{D_{\alpha}(\eta\|Q)+\lambda\cdot
W_{p}^{p}(P,\eta)\\}.$ (1)

In this work, we primarily focus on the cases when $p=1$ and $p=2$. This
proposed divergence combines the advantages of both $\alpha$-divergences and
Wasserstein distances: $\alpha$-divergences are effective for comparing heavy-
tailed distributions, while Wasserstein distances can accommodate non-absolute
continuity and measure transport distances, ensuring stable learning of tail
behavior. As illustrated in Figure 1, a GAN based on
$W_{1}$-$\alpha$-divergence learns a heavy-tailed distribution stably, whereas
a GAN without proximal regularization produces “run-away” points (details in
Section 5).

Figure 1: Learning heavy-tailed distrubtion using GANs based on
$\alpha$-divergence. Left: without $W_{1}$ proximal. Right: with $W_{1}$
proximal. See Section 5 for details.

This phenomenon is explained by our theoretical results in Section 4 and
numerical experiments in Section 5.

In this work, we theoretically show that the $W_{p}$-$\alpha$-divergences are
suitable for learning heavy-tailed distributions by analyzing the (sufficient
and necessary) conditions under which they remain finite for $p=1,2$; we also
provide a finite-sample convergence rate for the empirical estimations of the
proposed divergence for $p=1$. Our results show that algorithms using
$W_{p}$-$\alpha$-divergences can stably learn heavy tails without extensive
prior knowledge of the tail behavior, while those without a $W_{p}$ proximal
generally fail.

The rest of the paper is organized as follows. We review and discuss some
related work in Section 2. Section 3 provides background and motivation for
the proposed divergences. Theoretical results for $W_{p}$-$\alpha$-divergences
with $p=1,2$ are presented in Section 4. Numerical experiments are detailed in
Section 5, and Section 6 concludes with discussions.

##  2 Related work

Generative models for heavy-tailed distributions. Although heavy-tail
distributions are common, there are few results to date in their generative
modeling, primarily using GANs. For example, [33] generates heavy-tailed
financial time series data by logarithmically transforming the data and then
exponentiating the output, which does not produce truly heavy-tailed
distributions. In a different approach, GANs are used for cosmological
analysis [10], sharing a similarity with Pareto GANs [15] in their use of a
heavy-tailed latent variable. However, both papers require accurate estimation
of the tail decay rate for each marginal distribution. EV-GANs [1] use neural
network approximations of the quantile function that to encode the tail decay
rate in an asymptotic sense, which is essentially also a tail estimation
approach. We note that the focus of our work is to devise appropriate
divergences as objective functionals for comparing and learning heavy-tailed
distributions stably, without prior knowledge of the tail behavior.

Empirical estimations of divergences. [28, 23, 31] estimate $f$-divergences
using various assumptions and estimators, and [8] consider in particular the
$\alpha$-divergences. However, these studies either make additional structural
assumptions or consider light tails or without establishing a convergence rate
of the estimation. Recently, [20, 19] studied the convergence rate of entropic
optimal transport and optimal transport with smooth costs. While our proof of
the convergence rate in the estimations of the $W_{p}$-$\alpha$-divergences is
inspired by these works, the structure inherited from the $\alpha$-divergences
in our study requires different, non-trivial treatment, particularly as we
consider even heavier tails.

##  3 Wasserstein-proximal-regularized $\alpha$-divergences

Let $\mathcal{X}\subset\mathbb{R}^{d}$ be a measurable space, and
$\mathcal{P}(\mathcal{X})$ be the set of probability measures on
$\mathcal{X}$. A map $D\mathrel{\mathop{\mathchar
58\relax}}\mathcal{P}(\mathcal{X})\times\mathcal{P}(\mathcal{X})\to[0,\infty]$
is called a divergence on $\mathcal{P}(\mathcal{X})$ if

$D(P,Q)=0\iff P=Q\in\mathcal{P}(\mathcal{X}),$

hence providing a notion of “distance” between probability measures. The class
of $\alpha$-divergences [2, 14], a subset of $f$-divergences [6], is capable
of handling heavy-tailed distributions and is defined as

$\displaystyle
D_{\alpha}(P\|Q)=\int_{\mathbb{R}^{d}}f_{\alpha}\left(\frac{\mathop{}\\!\mathrm{d}{P}}{\mathop{}\\!\mathrm{d}{Q}}\right)\mathop{}\\!\mathrm{d}{Q},\quad\text{if}~{}P\ll
Q,$ (2)

where $f_{\alpha}(x)=\frac{x^{\alpha}-1}{\alpha(\alpha-1)}$, with $\alpha>0$
and $\alpha\neq 1$, and $P\ll Q$ means $P$ is absolutely continuous with
respect to $Q$. In the limiting case as $\alpha\to 1$, one recovers the
Kullback–Leibler (KL) divergence with $f_{\text{KL}}(x)=x\ln x$. The
$\alpha$-divergence can be equivalently formulated in its dual form [24, 3] as

$\displaystyle
D_{\alpha}(P\|Q)=\sup_{\gamma\in\mathcal{M}_{b}(\mathbb{R}^{d})}\left\\{\mathbb{E}_{P}[\gamma]-\mathbb{E}_{Q}[f_{\alpha}^{*}(\gamma)]\right\\},$
(3)

where $\mathcal{M}_{b}(\mathbb{R}^{d})$ is the set of bounded measurable
functions and $f_{\alpha}^{*}(y)$ is the convex conjugate (Legendre transform)
of $f_{\alpha}$,

$\displaystyle
f_{\alpha}^{*}(y)=\begin{cases}\alpha^{-1}(\alpha-1)^{\frac{\alpha}{\alpha-1}}y^{\frac{\alpha}{\alpha-1}}\mathbf{1}_{y>0}+\frac{1}{\alpha(\alpha-1)},&\alpha>1,\\\
\infty\mathbf{1}_{y\geq
0}+\left(\alpha^{-1}(1-\alpha)^{-\frac{\alpha}{1-\alpha}}\mathinner{\\!\left\lvert
y\right\rvert}^{-\frac{\alpha}{1-\alpha}}-\frac{1}{\alpha(1-\alpha)}\right)\mathbf{1}_{y<0},&\alpha\in(0,1).\end{cases}$
(4)

In practice, such as in generative modeling, we estimate the divergence from
finite samples of $P$ and $Q$, where the absolute continuity assumption
typically no longer holds. To accommodate cases where $P\centernot{\ll}Q$, we
add Wasserstein-proximal regularization to the $\alpha$-divergences.
Specifically, we consider Wasserstein-1 ($W_{1}$) and Wasserstein-2 ($W_{2}$)
proximals, defined as

$\displaystyle D_{\alpha,1}^{L}(P\|Q)\mathrel{\mathop{\mathchar
58\relax}}=\inf_{\eta\in\mathcal{P}(\mathbb{R}^{d})}\\{D_{\alpha}(\eta\|Q)+L\cdot
W_{1}(P,\eta)\\},$ (5)

and

$\displaystyle D_{\alpha,2}^{\lambda}(P\|Q)\mathrel{\mathop{\mathchar
58\relax}}=\inf_{\eta\in\mathcal{P}(\mathbb{R}^{d})}\\{D_{\alpha}(\eta\|Q)+\lambda\cdot
W_{2}^{2}(P,\eta)\\},$ (6)

where $W_{1}$ and $W_{2}$ are the Wasserstein-1 and Wasserstein-2 distances,
respectively. One can easily verify that both $D_{\alpha,1}^{\lambda}$ and
$D_{\alpha,2}^{\lambda}$ satisfy the conditions for being divergences.

The reason for using $L$ as the superscript in Eq. 5 is that the
$W_{1}$-$\alpha$-divergences coincide with the Lipschitz-regularized
$\alpha$-divergence proposed by [3] for $\alpha\geq 1$. According to [3,
Theorem 15 (1)], the following dual equality holds for $\alpha\geq 1$:

$\displaystyle
D_{\alpha,1}^{L}(P\|Q)=\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\left\\{\mathbb{E}_{P}[\gamma]-\mathbb{E}_{Q}[f_{\alpha}^{*}(\gamma)]\right\\},$
(7)

where $\text{Lip}_{L}(\mathbb{R}^{d})$ is the class of bounded $L$-Lipschitz
functions.

To illustrate the benefit of combining $\alpha$-divergences and Wasserstein
proximals, we provide an example where neither the $\alpha$-divergence nor the
$W_{1}$ distance is finite, but the $W_{1}$-$\alpha$-divergence, i.e.,
$D_{\alpha,1}^{L}$ in Eq. 5, is finite.

######  Example 1.

Let $P$ and $Q$ be distributions on $\mathbb{R}$ such that

$p(x)=(1+\delta)x^{-(2+\delta)}\mathbf{1}_{x\geq 1},\quad
q(x)=\frac{1}{2}\mathbf{1}_{0\leq x<1}+\frac{1}{x^{2}}\mathbf{1}_{x\geq 2}.$

It can be shown that neither $D_{\alpha}(P\|Q)$ nor $W_{1}(P,Q)$ is finite for
any $\alpha>1,\delta>0$, but $D_{\alpha,1}^{L}(P\|Q)$ is nonetheless finite.
See the calculations in Appendix B.

##  4 Theoretical analysis

In generative modeling, a specific divergence between the target and generated
distributions is often chosen as the loss function. To optimize and minimize
this loss, it is essential to ensure that (1) the loss function or divergence
is finite, and (2) it can be accurately estimated from finite minibatches of
samples. Therefore, we examine how the tail behaviors of $P$ and $Q$ determine
the finiteness of $W_{p}$-$\alpha$-divergences and their finite-sample
estimation rate. We make the following assumption about distributions $P$ and
$Q$.

######  Assumption 1.

Let $P$ and $Q$ be distributions on $\mathbb{R}^{d}$ whose densities $p(x)$
and $q(x)$ are absolutely continuous with respect to the Lebesgue measure.

######  Definition 1.

For a pair of distributions $(P,Q)$ on $\mathbb{R}^{d}$, we say they are of
heavy-tail $(\beta_{1},\beta_{2})$, $\beta_{1},\beta_{2}>d$, if there exists
some $R>0$, such that

$p(x)\asymp\mathinner{\\!\left\lVert x\right\rVert}^{-\beta_{1}},\quad
q(x)\asymp\mathinner{\\!\left\lVert x\right\rVert}^{-\beta_{2}},$

for $\mathinner{\\!\left\lVert x\right\rVert}\geq R$. That is, there exist
constants $0<c_{p,1}\leq c_{p,2}$ and $0<c_{q,1}\leq c_{q,2}$ such that

$c_{p,1}\mathinner{\\!\left\lVert x\right\rVert}^{-\beta_{1}}\leq p(x)\leq
c_{p,2}\mathinner{\\!\left\lVert x\right\rVert}^{-\beta_{1}},\quad
c_{q,1}\mathinner{\\!\left\lVert x\right\rVert}^{-\beta_{2}}\leq q(x)\leq
c_{q,2}\mathinner{\\!\left\lVert x\right\rVert}^{-\beta_{2}},$

for $\mathinner{\\!\left\lVert x\right\rVert}\geq R$.

######  Remark 1.

Definition 1 implies that $P\centernot{\ll}Q$ at most on a bounded subset of
$\mathbb{R}^{d}$.

###  4.1 Finiteness of $D_{\alpha,1}^{L}$ and $D_{\alpha,2}^{\lambda}$

We focus on the case where $\alpha>1$. We first study the finiteness of
$D_{\alpha,1}^{L}$, making extensive use of its dual formula Eq. 7. We prove
the following sufficient and necessary conditions on the tail behaviors of
$(P,Q)$ for the $W_{1}$-$\alpha$-divergence to be finite. The proof is
provided in Appendix C.

######  Theorem 1 (Necessary and sufficient condition for
$D_{\alpha,1}^{L}<\infty$).

Suppose $\alpha>1$, and $(P,Q)$ are distributions on $\mathbb{R}^{d}$ of
heavy-tail $(\beta_{1},\beta_{2})$. Then $D_{\alpha,1}^{L}(P\|Q)<\infty$ if
and only if one of the following two conditions holds:
(i) $d<\beta_{1}\leq d+1$ and
$\beta_{2}-\beta_{1}<\frac{\beta_{1}-d}{\alpha-1}$;
(ii) $\beta_{1}>d+1$.

######  Remark 2.

If $P$ has finite first moment (mean), then $D_{\alpha,1}^{L}(P\|Q)$ is finite
for any $\beta_{2}>d$ and $\alpha>1$. This indicates that we can learn a
heavy-tailed distribution, with or without a finite first moment, starting
from an easy-to-sample light-tailed distribution such as the Gaussian.

######  Remark 3.

We can prove a similar finiteness result for $W_{1}$-KL-divergences, i.e., the
special case where $\alpha=1$. More specifically,
$D_{\text{KL},1}^{L}(P\|Q)<\infty$ is finite for any $\beta_{1},\beta_{2}>d$.
The precise statement and proof can be found in Lemma 3 and Proposition 1 in
the appendix.

In cases where the data distribution lies on a low-dimensional submanifold,
which is typically the case for real-world datasets, 1 can be relaxed as shown
in the following corollary. The proof can be found in Appendix C.

######  Corollary 1.

Let $\mathcal{M}$ be a $d^{*}$-dimensional smooth embedded submanifold of
$\mathbb{R}^{d}$ via an $L^{*}$-Lipschitz embedding
$\varphi\mathrel{\mathop{\mathchar
58\relax}}\mathbb{R}^{d^{*}}\to\mathbb{R}^{d}$ with
$\mathcal{M}=\varphi(\mathbb{R}^{d^{*}})$. Suppose $(P,Q)$ are of heavy-tail
$(\beta_{1},\beta_{2})$ on $\mathbb{R}^{d^{*}}$, and let $p_{\mathcal{M}}$ and
$q_{\mathcal{M}}$ be their push-forward distributions on $\mathcal{M}$, i.e.,
$p_{\mathcal{M}}=p\circ\varphi^{-1}$ and $q_{\mathcal{M}}=q\circ\varphi^{-1}$.
Then the $W_{1}$-$\alpha$-divergence between $p_{\mathcal{M}}$ and
$q_{\mathcal{M}}$, defined as

$D_{\alpha,1}^{L}(p_{\mathcal{M}}\|q_{\mathcal{M}})=\sup_{\gamma\in\text{Lip}_{L}(\mathcal{M})}\left\\{\mathbb{E}_{p_{\mathcal{M}}}[\gamma]-\mathbb{E}_{q_{\mathcal{M}}}[f_{\alpha}^{*}(\gamma)]\right\\},$

is finite if and only if one of the following two conditions holds:
(i) $d^{*}<\beta_{1}\leq d^{*}+1$ and
$\beta_{2}-\beta_{1}<\frac{\beta_{1}-d^{*}}{\alpha-1}$;
(ii) $\beta_{1}>d^{*}+1$.

######  Remark 4.

The Lipschitz condition on the embedding $\varphi$ is necessary to guarantee
that the tails of $p_{\mathcal{M}}$ and $q_{\mathcal{M}}$ do not become
heavier than those of $p$ and $q$.

Lastly, we provide a sufficient condition for the $W_{2}$-$\alpha$-divergences
to be finite. The proof can be found in Appendix C.

######  Theorem 2 (Sufficient condition for $D_{\alpha,2}^{\lambda}<\infty$).

Suppose $\alpha>1$ and $(P,Q)$ are distributions on $\mathbb{R}^{d}$ of heavy-
tail $(\beta_{1},\beta_{2})$. Then $D_{\alpha,2}^{\lambda}(P\|Q)<\infty$ if
one of the following two conditions holds:
(i) $d<\beta_{1}\leq d+2$ and
$\beta_{2}-\beta_{1}<\frac{\beta_{1}-d}{\alpha-1}$;
(ii) $\beta_{1}>d+2$.

###  4.2 Finite-sample estimations of $D_{\alpha,1}^{L}$

The finiteness result in Section 4.1 is crucial for comparing distributions
with heavy tails, making it feasible to learn heavy-tailed distributions by
minimizing these divergences. In this section, we address two additional
questions: providing an easily computable a posteriori upper bound on
$D_{\alpha,1}^{L}(P\|Q)$ and deriving its finite-sample estimation rate,
allowing the loss function based on $W_{1}$-$\alpha$-divergences to be
efficiently estimated on minibatches. Let $X=\\{x_{1},\dots,x_{m}\\}$ and
$Y=\\{y_{1},\dots,y_{n}\\}$ be i.i.d. samples from $P$ and $Q$, with empirical
distributions $P_{m}=\frac{1}{m}\sum_{i=1}^{m}\delta_{x_{i}}$ and
$Q_{n}=\frac{1}{n}\sum_{j=1}^{n}\delta_{y_{j}}$, respectively.

Using formula (15) by [3], the $W_{1}$-$\alpha$-divergences can be
equivalently formulated as
$D_{\alpha,1}^{L}(P\|Q)=\sup_{\gamma\in\Gamma}\\{\mathbb{E}_{P}[\gamma]-\Lambda_{f_{\alpha}}^{Q}[\gamma]\\}$,
where $\Lambda_{f_{\alpha}}^{Q}[\gamma]\mathrel{\mathop{\mathchar
58\relax}}=\inf_{\nu\in\mathbb{R}}\\{\nu+\mathbb{E}_{Q}[f_{\alpha}^{*}(\gamma-\nu)]\\}$
by adding an additional shift parameter. The following lemma provides a
pseudo-triangle inequality for $D_{\alpha,1}^{L}(P\|Q)$, with the proof in
Appendix D.

######  Lemma 1.

For any distribution $\eta\in\mathcal{P}(\mathbb{R}^{d})$, we have

$D_{\alpha,1}^{L}(P\|Q)\leq L\cdot W_{1}(P,\eta)+D_{\alpha,1}^{L}(\eta\|Q).$
(8)

By setting $P=P_{m}$ and $\eta=Q_{n}$ in Eq. 8, we obtain the following
corollary.

######  Corollary 2.

$D_{\alpha,1}^{L}(P_{m}\|Q)\leq L\cdot
W_{1}(P_{m},Q_{n})+D_{\alpha,1}^{L}(Q_{n}\|Q).$ (9)

If we set $\eta=Q$ in Eq. 8, we get

######  Corollary 3.

For any $P,Q\in\mathcal{P}(\mathbb{R}^{d})$, we have

$D_{\alpha,1}^{L}(P\|Q)\leq L\cdot W_{1}(P,Q).$ (10)

Corollary 3 indicates that the convergence rate of the expectation of
$D_{\alpha,1}^{L}(P_{m}\|P)$ can be bounded by the rate of Wasserstein-1
distance $W_{1}(P_{m},P)$. Moreover, combining Lemma 1, Corollary 2 and
Corollary 3, we have

$\displaystyle D_{\alpha,1}^{L}(P\|Q)$ $\displaystyle\leq L\cdot
W_{1}(P,P_{m})+D_{\alpha,1}^{L}(P_{m}\|Q)$ $\displaystyle\leq
L\left(W_{1}(P,P_{m})+W_{1}(P_{m},Q_{n})+W_{1}(Q_{n},Q)\right),$ (11)

where each term on the right hand side can be bounded using the convergence
result of $W_{1}$ in [11].

######  Remark 5.

Eq. 11 provides a practical perspective for generative modeling through
divergence $D_{\alpha,1}^{L}$. Suppose our goal is to learn a heavy-tailed
target distribution $Q$ from its finite samples $Q_{n}$, and we want to
estimate how close the generated distribution $P$ is to $Q$. If $P_{m}$ and
$Q_{n}$ accurately represent $P$ and $Q$ in terms of $W_{1}$ distance (which
holds true when $m$ and $n$ are large), and the generated samples $P_{m}$ are
close to true samples $Q_{n}$ such that $W_{1}(P_{m},Q_{n})$ is small, then we
can conclude $P$ is close to the target $Q$ in terms of $D_{\alpha,1}^{L}$.
Eq. 11 essentially provides an easily computable a posteriori upper bound on
$D_{\alpha,1}^{L}$, where the dorminating term $W_{1}(P_{m},Q_{n})$ can be
computed via linear programming.

Finally, we derive the convergence bound for the empirical estimations of
$D_{\alpha,1}^{L}(P\|Q)$ under certain tail conditions. The result for $d=3$
is stated below, with its proof deferred to Appendix D. The results for
$d=1,2$ can be found as Proposition 2 and Proposition 3 in Appendix D.

######  Theorem 3 (Finite sample estimation of $D_{\alpha,1}^{L}$ for heavy-
tailed distributions).

Assume $d\geq 3$. For $\alpha>1$, let $(P,Q)$ be distributions on
$\mathbb{R}^{d}$ of heavy-tail $(\beta_{1},\beta_{2})$, where $\beta_{1}>3d$
and $\beta_{2}>5d$. Suppose $\alpha$ satisfies
$\frac{2d\alpha}{\alpha-1}<\beta_{1}-d$ and
$\frac{2\alpha}{\alpha-1}<\frac{\beta_{2}}{d}-3$. If $m$ and $n$ are
sufficiently large, we have

$\mathbb{E}_{X,Y}\mathinner{\\!\left\lvert
D_{\alpha,1}^{L}(P_{m}\|Q_{n})-D_{\alpha,1}^{L}(P\|Q)\right\rvert}\leq\frac{C_{1}}{m^{1/d}}+\frac{C_{2}}{n^{1/d}},$
(12)

where $C_{1}$ depends on $M_{\frac{d}{d-1}}(p)$ and $C_{2}$ depends on
$M_{\frac{2d\alpha}{\alpha-1}}(p)$, $M_{\frac{2d\alpha}{\alpha-1}}(q)$, and
$M_{dr_{2}}(q)$ for any
$2+\frac{2\alpha}{\alpha-1}<r_{2}<\frac{\beta_{2}}{d}-1$. Here, we use
$M_{r}(p)$ to denote the $r$-th moment of $p(x)$. Both $C_{1}$ and $C_{2}$ are
independent of $m,n$.

##  5 Numerical experiments

In this section, we demonstrate how using our proposed Wasserstein-proximal-
regularized $\alpha$-divergences as objective functionals enables stable
learning of heavy-tailed distributions in various generative models. In
Section 5.1, we introduce the generative models used and explain how their
learning objectives relate to $W_{p}$-$\alpha$-divergences. In Section 5.2, we
illustrate our points with two examples. First, we compare the effects of
incorporating Wasserstein proximals in the learning objectives by training on
a 2D Student-t distribution. Then, we provide an example of learning a real-
world heavy-tailed dataset, comparing the efficacy of $W_{1}$ and $W_{2}$
proximals in the objective functions. We use Gaussian priors for all our
experiments.

###  5.1 Generative models with Wasserstein-proximal-regularized learning
objectives

$W_{1}$ and $W_{2}$ proximals can be found, sometimes implicitly, in the
learning objectives of several existing generative models. Below, we list
various models based on $\alpha$-divergences used in our experiments and
explain why some of them are (either implicitly or explicitly) regularized by
Wasserstein proximals.

$\bullet$ Generative models without proximal regularization:

  * •

$\alpha$-GAN: Generative adversarial networks (GAN) [12, 25] based on the
variational representation of the $\alpha$-divergence Eq. 3;

  * •

$\alpha$-GPA: Generative particle algorithm (GPA) based on the
$\alpha$-divergence [13];

  * •

CNF: Continuous normalizing flow (CNF) by [4], where the loss function is
based on the KL divergence, a special case of the $\alpha$-divergence when
$\alpha=1$.

$\bullet$ Generative models with $W_{1}$-proximal regularization:

  * •

$W_{1}$-$\alpha$-GAN [3]: GAN using the $W_{1}$-$\alpha$-divergence 7 as the
objective function, with the Lipschitz constant set to $L=1$ in our
experiment;

  * •

$W_{1}$-$\alpha$-GPA [13]: GPA using the $W_{1}$-$\alpha$-divergence 7 as the
objective function, with the Lipschitz constant set to $L=1$ in our
experiment.

$\bullet$ Generative models with $W_{2}$-proximal regularization: We consider
the following class of flow-based models, which minimize
$W_{2}$-$\alpha$-divergences Eq. 6 written as Eq. 13 via the Benamou-Brenier
formula,

$\inf_{v,\rho}\mathcal{F}(\rho(\cdot,T))+C\int_{0}^{T}\frac{1}{2}|v(x,t)|^{2}\rho(x,t)\mathop{}\\!\mathrm{d}{x}\mathop{}\\!\mathrm{d}{t}.$
(13)

Here, $\rho\mathrel{\mathop{\mathchar
58\relax}}\mathbb{R}^{d}\times[0,T]\to\mathbb{R}$ is the evolution of the
probability measure via the (trainable) velocity field
$v\mathrel{\mathop{\mathchar
58\relax}}\mathbb{R}^{d}\times[0,T]\to\mathbb{R}^{d}$, satisfying the
Fokker–Planck equation:

$\displaystyle\rho_{t}+\nabla\cdot(\rho
v)=\frac{\sigma^{2}}{2}\Delta\rho,\quad\rho(\cdot,0)=\rho_{0}~{}\text{is a
tractable prior distribution, e.g., Gaussian.}$ (14)

  * •

OT flow [26]: Optimal transport (OT) normalizing flow, which is equivalent to
the $W_{2}$-proximal of CNF, with
$\mathcal{F}(\rho(\cdot,T))=D_{\text{KL}}(Q,\rho(\cdot,T))$ and $\sigma=0$ in
Eq. 13;

  * •

VE-SGM [30]: Score-based generative model (SGM) with variance-exploding (VE)
forward SDE [30]. According to the mean-field game formulation by [34], it is
equivalent to Eq. 13 with stochastic dynamics ($\sigma>0$) and a cross-entropy
terminal cost $\mathcal{F}(\rho(\cdot,T))=-\mathbb{E}_{\rho(\cdot,T))}[\log
Q]$, essentially also a $W_{2}$-proximal of CNF.

Refer to Fig. 2 for a visual illustration of the relationships among the
models being compared.

Figure 2: Generative models in the experiment and their relationship with the
$W_{p}$-$\alpha$-divergences. See Section 5.1 for detailed explanations of the
models and notations.

###  5.2 Learning heavy-tailed distributions

(a)  $\alpha$-GAN (left) and its counterpart with $W_{1}$-proximal
regularization, $W_{1}$-$\alpha$-GAN (right)

(b)  $\alpha$-GPA (left) and its counterpart with $W_{1}$-proximal
regularization, $W_{1}$-$\alpha$-GPA (right)

Figure 3: Learning a 2D isotropic Student-t with degree of freedom $\nu=1$
(tail index $\beta=3.0$) using generative models based on
$W_{1}$-$\alpha$-divergences with $\alpha=2$. Models with $W_{1}$-proximal
regularizations (right) learn the heavy-tailed distribution significantly
better than those without (left). See Section 5.1 for detailed explanations of
the models.

2D Student-t example. We first consider various generative models for learning
the heavy-tailed 2D isotropic Student-t distribution with $\nu$ degrees of
freedom, $q(x)\propto(1+\frac{|x|^{2}}{\nu})^{\frac{\nu+2}{2}}$. This
synthetic example allows us to adjust the tail decay rate $\beta=\nu+2$ by
selecting different degrees of freedom $\nu$. In the main text, we present an
extremely heavy-tailed example with $\beta=3$, while the relatively easier
case of $\beta=5$ is deferred to Appendix F. We used 10,000 samples to train
the models.

Figure 3 and Figure 4 present the performance of various generative models.
Each model is evaluated in two plots. First, a 2D scatter plot displays the
generated samples (orange) and the true samples (blue), providing a visual
assessment of the sample quality. Next, the tail behavior is assessed by
plotting the ground truth Radial Complementary Cumulative Distribution
Function (rCCDF) (red curve) and the histogram of the radii of generated
samples (gray). The rCCDF is defined as:

$\displaystyle\text{rCCDF}(r)=1-\text{CDF}(r),$ (15)

where CDF($r$) is the cumulative distribution function of the radius. We then
calculate the $L_{1}$ error between the ground truth rCCDF and the generated
sample histogram. Generative models with $W_{1}$\- or $W_{2}$-proximal
regularizations significantly outperform those without in learning heavy-
tailed distributions, corroborating our theoretical results in Section 4.

Additionally, without proximal regularizations, models often exhibit a
“runaway” behavior, chasing far-away samples deep into the heavy tails and
creating a “star-like” unstable pattern in the generated samples. This issue
is resolved with $W_{1}$\- or $W_{2}$\- proximal regularizations, as the
$W_{p}$-proximal term controls the “velocity” of the generative particles. The
$W_{2}$-proximal controls velocities through the kinetic energy of the
particles (see Eq. 13), while the $W_{1}$-proximal provides pointwise control
of velocities [13]. These proximal regularizations enable stable learning of
the tail behavior in a “tactically controlled” manner.

(a) CNF

(b) OT flow

(c) VE-SGM

Figure 4: Learning a 2D isotropic Student-t with degree of freedom $\nu=1$
(tail index $\beta=3.0$) using generative models based on
$W_{2}$-$\alpha$-divergences with $\alpha=1$. Models with $W_{2}$-proximal
regularizations, (b) and (c), learn the heavy-tailed distribution
significantly better than that without, (a). See Section 5.1 for detailed
explanations of the models.

Keystroke example. Finally, we provide a real-world heavy-tailed example of
learning the inter-arrival time between keystrokes from multiple users typing
sentences [7]. The target dataset consists of 7,160 scalar samples, and we
generated 10,000 samples using generative models with $W_{1}$\- or
$W_{2}$-proximal regularization.

We display the tail behavior by plotting the ground truth CCDF (red curve) and
the corresponding histogram of the generated samples (gray). Unlike the
previous synthetic example, the ground truth CCDF here is obtained by
interpolating the heights of the histogram bins of the true samples. In Figure
5, generative models with $W_{1}$-proximal regularization
($W_{1}$-$\alpha$-GPA and $W_{1}$-$\alpha$-GAN) outperform those regularized
with $W_{2}$-proximals (OT flow and VE-SGM) in capturing the tails. This
observation suggests that $W_{1}$-proximal algorithms can potentially handle
heavier tails more effectively than $W_{2}$-proximal methods.

(a) $W_{1}$-$\alpha$ GPA (left), $W_{1}$-$\alpha$ GAN (right)

(b) $W_{2}$ flow (left), VE SGM (right)

Figure 5: Sample generation of inter-arrival time between keystrokes.
Generative models with $W_{1}$-proximal regularization, panel (a), outperform
those with $W_{2}$-proximal regularization, panel (b), in capturing the tails.
This observation suggests that $W_{1}$-proximal algorithms can potentially
handle heavier tails more effectively than $W_{2}$-proximal methods.

##  6 Conclusions and discussions

We propose Wasserstein-proximal-regularized $\alpha$-divergences as objective
functionals for learning heavy-tailed distributions and provide rigorous
analysis to demonstrate their feasibility. This includes analyzing the
sufficient and necessary conditions for the divergence to be finite and
quantifying its finite-sample estimation rate. Numerical simulations show that
adding Wasserstein proximals to $\alpha$-divergences significantly improves
the learning process of heavy-tailed distributions.

Limitations and future work: There are several future directions that can be
explored:

  * •

Choosing optimal $\alpha$ values: Theorem 1 does not impose any restriction on
the selection of $\alpha$ when starting with a light-tailed distribution.
However, Theorem 3 suggests using larger $\alpha$, as simulations in [3] show
that larger $\alpha$ can lead to faster convergence.

  * •

Closing the gap in the conditions: Theorem 3 considers $\beta_{1}>3d$ and
$\beta_{2}>5d$. More techniques are needed to close the gap $d+1<\beta_{1}\leq
3d$ and $d+1<\beta_{2}\leq 5d$. Additionally, we lack finite sample estimation
for target data distributions without a finite first moment, as the law of
large numbers does not apply.

  * •

Guarantees for learning heavy-tailed distributions: Our paper focuses on
devising divergences that are efficiently computable from finite samples and
guaranteed to effectively compare heavy-tailed distributions, which is a
necessary condition for learning such distributions. Future work will focus on
ensuring learning of heavy-tailed distributions with theoretical guarantees
for specific generative algorithms.

  * •

Completing the analysis of $W_{2}$-proximals: The story of $W_{2}$-proximals
is not complete. We only provided sufficient conditions for its finiteness in
Theorem 2, and further study on their finite-sample estimation rate is needed.

  * •

Comparison of $W_{1}$ and $W_{2}$ proximals: Numerical experiments suggest
that both $W_{1}$ and $W_{2}$ proximal can make learning heavy-tailed
distributions stable. However, when the target distribution lacks a finite
first moment, $W_{1}$-proximal-based algorithms learn deeper into the tails
compared to their $W_{2}$ counterparts.

  * •

More testing on benchmark datasets: There are limited large-scale and/or high-
dimensional benchmark heavy-tailed datasets, such as MNIST or CIFAR-10 for
images. Future applications can be explored once such datasets become broadly
available.

##  7 Acknowledgement

Z. Chen, H. Gu, M. Katsoulakis, L. Rey-Bellet are partially funded by AFOSR
grant FA9550-21-1-0354. M.K. and L. R.-B. are partially funded by NSF
DMS-2307115. M.K. is partially funded by NSF TRIPODS CISE-1934846. Z. Chen and
W. Zhu are partially supported by NSF under DMS-2052525, DMS-2140982, and
DMS2244976.

## References

  * [1] M. Allouche, S. Girard, and E. Gobet.  Ev-gan: Simulation of extreme events with relu neural networks.  Journal of Machine Learning Research, 23(150):1–39, 2022.
  * [2] S.-i. Amari and H. Nagaoka.  Methods of information geometry, volume 191.  American Mathematical Soc., 2000.
  * [3] J. Birrell, P. Dupuis, M. A. Katsoulakis, Y. Pantazis, and L. Rey-Bellet.  $(f,{\Gamma})$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics.  Journal of Machine Learning Research, 23(39):1–70, 2022.
  * [4] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud.  Neural ordinary differential equations.  Advances in neural information processing systems, 31, 2018.
  * [5] Z. Chen, M. Katsoulakis, L. Rey-Bellet, and W. Zhu.  Sample complexity of probability divergences under group symmetry.  In International Conference on Machine Learning, pages 4713–4734. PMLR, 2023.
  * [6] I. Csiszar.  Eine information’s theoretische ungleichung und ihre anwendung auf den beweis der ergodizitat von markoschen ketten, magyar tud.  Akad. Mat, 1963.
  * [7] V. Dhakal, A. M. Feit, P. O. Kristensson, and A. Oulasvirta.  Observations on typing from 136 million keystrokes.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, page 1–12, New York, NY, USA, 2018. Association for Computing Machinery.
  * [8] R. Ding and A. Mullhaupt.  Empirical squared hellinger distance estimator and generalizations to a family of $\alpha$-divergence estimators.  Entropy, 25(4):612, 2023.
  * [9] M. Farazmand and T. P. Sapsis.  Closed-loop adaptive control of extreme events in a turbulent flow.  Phys. Rev. E, 100:033110, Sep 2019.
  * [10] R. M. Feder, P. Berger, and G. Stein.  Nonlinear 3d cosmic web simulation with heavy-tailed generative adversarial networks.  Physical Review D, 102(10):103504, 2020.
  * [11] N. Fournier and A. Guillin.  On the rate of convergence in Wasserstein distance of the empirical measure.  Probability theory and related fields, 162(3-4):707–738, 2015.
  * [12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [13] H. Gu, P. Birmpa, Y. Pantazis, L. Rey-Bellet, and M. A. Katsoulakis.  Lipschitz regularized gradient flows and latent generative particles.  arXiv preprint arXiv:2210.17230, 2022.
  * [14] J. Havrda and F. Charvát.  Quantification method of classification processes. concept of structural $a$-entropy.  Kybernetika, 3(1):30–35, 1967.
  * [15] T. Huster, J. Cohen, Z. Lin, K. Chan, C. Kamhoua, N. O. Leslie, C.-Y. J. Chiang, and V. Sekar.  Pareto gan: Extending the representational power of gans to heavy-tailed distributions.  In International Conference on Machine Learning, pages 4523–4532. PMLR, 2021.
  * [16] L. B. Klebanov and Y. V. Kuvaeva.  Heavy-tailed probability distributions in social sciences.  arXiv preprint arXiv:2301.09393, 2023.
  * [17] A. J. Lotka.  The frequency distribution of scientific productivity.  Journal of the Washington academy of sciences, 16(12):317–323, 1926\.
  * [18] C. W. Lynn, C. M. Holmes, and S. E. Palmer.  Heavy-tailed neuronal connectivity arises from hebbian self-organization.  Nature Physics, pages 1–8, 2024.
  * [19] T. Manole and J. Niles-Weed.  Sharp convergence rates for empirical optimal transport with smooth costs.  The Annals of Applied Probability, 34(1B):1108–1135, 2024.
  * [20] G. Mena and J. Niles-Weed.  Statistical bounds for entropic optimal transport: sample complexity and the central limit theorem.  Advances in neural information processing systems, 32, 2019.
  * [21] B. Merz, S. Basso, S. Fischer, D. Lun, G. Blöschl, R. Merz, B. Guse, A. Viglione, S. Vorogushyn, E. Macdonald, et al.  Understanding heavy tails of flood peak distributions.  Water Resources Research, 58(6):e2021WR030506, 2022.
  * [22] M. Mohri, A. Rostamizadeh, and A. Talwalkar.  Foundations of machine learning.  MIT press, 2018.
  * [23] K. Moon and A. Hero.  Multivariate f-divergence estimation with confidence.  Advances in neural information processing systems, 27, 2014.
  * [24] X. Nguyen, M. J. Wainwright, and M. I. Jordan.  Estimating divergence functionals and the likelihood ratio by convex risk minimization.  IEEE Transactions on Information Theory, 56(11):5847–5861, 2010\.
  * [25] S. Nowozin, B. Cseke, and R. Tomioka.  f-gan: Training generative neural samplers using variational divergence minimization.  Advances in neural information processing systems, 29, 2016.
  * [26] D. Onken, S. W. Fung, X. Li, and L. Ruthotto.  Ot-flow: Fast and accurate continuous normalizing flows via optimal transport, 2021.
  * [27] V. Pareto.  Cours d’économie politique, volume 1.  Librairie Droz, 1964.
  * [28] P. Rubenstein, O. Bousquet, J. Djolonga, C. Riquelme, and I. O. Tolstikhin.  Practical and consistent estimation of f-divergences.  Advances in Neural Information Processing Systems, 32, 2019.
  * [29] H. Sasaki, F.-H. Su, T. Tanimoto, and S. Sethumadhavan.  Why do programs have heavy tails?  In 2017 IEEE International Symposium on Workload Characterization (IISWC), pages 135–145. IEEE, 2017.
  * [30] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole.  Score-based generative modeling through stochastic differential equations, 2021.
  * [31] S. Sreekumar, Z. Zhang, and Z. Goldfeld.  Non-asymptotic performance guarantees for neural estimation of f-divergences.  In International Conference on Artificial Intelligence and Statistics, pages 3322–3330. PMLR, 2021.
  * [32] A. W. van der Vaart and J. A. Wellner.  Weak convergence and empirical processes: with applications to statistics.  Springer-Verlag, New York, 1996.
  * [33] M. Wiese, R. Knobloch, R. Korn, and P. Kretschmer.  Quant gans: deep generation of financial time series.  Quantitative Finance, 20(9):1419–1440, 2020.
  * [34] B. J. Zhang and M. A. Katsoulakis.  A mean-field games laboratory for generative modeling, 2023.
  * [35] G. K. Zipf.  Human behavior and the principle of least effort: An introduction to human ecology.  Ravenio Books, 2016.

##  Appendix A Notation

We denote by $A\lesssim B$ if there are some $c,d>0$, such that $A\leq cB+d$;
and $A\asymp B$ if both $A\lesssim B$ and $B\lesssim A$ hold. For a bounded
set $\Omega\subset\mathbb{R}^{d}$,
$\text{diam}(\Omega)=\sup_{x,y\in\Omega}\mathinner{\\!\left\lVert
x-y\right\rVert}_{2}$, where $\mathinner{\\!\left\lVert\cdot\right\rVert}_{2}$
is the Euclidean norm on $\mathbb{R}^{d}$. Moreover, given a probability
density $p(x)$, we use $M_{r}(p)$ to denote the $r$-th moment of $p(x)$. For
convenience, we will abuse notation and use symbols $p,q,P,Q$, to represent
probability distributions as well as the density functions associated with
them. Whether a character refers to a probability distribution or a density
should be clear from the context.

##  Appendix B Proof of Example 1

Since $P$ is not absolutely continuous with respect to $Q$, we have
$D_{\alpha}(P\|Q)=\infty$; applying the cumulative distribution functions
formula for 1-dimensional Wasserstein-1 distance, it is straightforward to see
$W_{1}(P,Q)=\infty$. Consider the formula (5), and in particular, we design
the intermediate probability measure as

$d\eta=(1+\delta)2^{1+\delta}x^{-(2+\delta)}\mathbf{1}_{x\geq 2}.$

Then we have

$\displaystyle
D_{\alpha}(\eta\|Q)=\int_{2}^{\infty}\frac{(1+\delta)^{\alpha}2^{\alpha(1+\delta)}x^{-\alpha\delta}-1}{\alpha(\alpha-1)}\cdot\frac{1}{x^{2}}\mathop{}\\!\mathrm{d}{x}<\infty,$

and

$\displaystyle W_{1}(P,\eta)$
$\displaystyle=\int_{1}^{2}\int_{1}^{y}(1+\delta)x^{-(2+\delta)}\mathop{}\\!\mathrm{d}{x}dy$
$\displaystyle\quad+\int_{2}^{\infty}\mathinner{\\!\left\lvert\int_{1}^{y}(1+\delta)x^{-(2+\delta)}\mathop{}\\!\mathrm{d}{x}-\int_{2}^{y}(1+\delta)2^{1+\delta}x^{-(2+\delta)}\mathop{}\\!\mathrm{d}{x}\right\rvert}dy$
$\displaystyle=\int_{1}^{2}1-y^{-(1+\delta)}dy+\int_{2}^{\infty}\mathinner{\\!\left\lvert(1-y^{-(1+\delta)})-(1-2^{1+\delta}y^{-(1+\delta)})\right\rvert}dy$
$\displaystyle=\int_{1}^{2}1-y^{-(1+\delta)}dy+\int_{2}^{\infty}(2^{1+\delta}-1)y^{-(1+\delta)}dy<\infty.$

Therefore, $D_{\alpha,1}^{L}(P\|Q)\leq D_{\alpha}(\eta\|Q)+L\cdot
W_{1}(P,\eta)<\infty.$

##  Appendix C Proof of Theorem 1

We first provide a lemma that generalizes Lemma A.12 in [5].

######  Lemma 2.

For $\alpha>1$ and any non-negative integrable functions $p(x)$ and $q(x)$
defined on some bounded $\Omega\subset\mathbb{R}^{d}$ with non-zero integrals,
$\Gamma=\text{Lip}_{L}(\Omega)$, we have

$\sup_{\gamma\in\Gamma}\left\\{\int_{\Omega}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\}=\sup_{\gamma\in\mathcal{F}}\left\\{\int_{\Omega}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\},$
(16)

where

$\mathcal{F}=\left\\{\gamma\in\text{Lip}_{L}(\Omega)\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert\gamma\right\rVert}_{\infty}\leq(\alpha-1)^{-1}\left(\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}\right)^{\alpha-1}+L\cdot\text{diam}(\Omega)\right\\}.$

###### Proof of Lemma 2.

For any fixed $\gamma\in\Gamma$, define

$h(\nu)=\int_{\Omega}\left(\gamma(x)+\nu\right)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\alpha}^{*}[\gamma(x)+\nu]q(x)\mathop{}\\!\mathrm{d}{x}.$

Since $\sup_{x\in\Omega}\gamma(x)-\inf_{x\in\Omega}\gamma(x)\leq
L\cdot\text{diam}(\Omega)$, interchanging the integration with differentiation
is allowed by the dominated convergence theorem:

$h^{\prime}(\nu)=\int_{\Omega}p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\alpha}^{*\prime}(\gamma+\nu)q(x)\mathop{}\\!\mathrm{d}{x},$

where

$\displaystyle
f_{\alpha}^{*\prime}(y)=(\alpha-1)^{\frac{1}{\alpha-1}}y^{\frac{1}{\alpha-1}}\mathbf{1}_{y>0}.$
(17)

If $\inf_{x\in\Omega}\gamma(x)>(\alpha-1)^{-1}\left(\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}\right)^{\alpha-1}$, then $h^{\prime}(0)<0$. So
there exists some $\nu_{0}<0$ such that $h(\nu_{0})>h(0)$. This indicates the
supremum on the left side of (16) is attained only if
$\sup_{x\in\Omega}\gamma(x)\leq(\alpha-1)^{-1}\left(\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}\right)^{\alpha-1}+L\cdot\text{diam}(\Omega)$.
On the other hand, if $\sup_{x\in\Omega}\gamma(x)<0$, then there exists
$\nu_{0}>0$ that satisfies $\sup_{x\in\Omega}\gamma(x)+\nu_{0}<0$ such that

$\displaystyle\int_{\Omega}\left(\gamma(x)+\nu_{0}\right)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\alpha}^{*}[\gamma(x)+\nu_{0}]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{\Omega}\left(\gamma(x)+\nu_{0}\right)p(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle>\int_{\Omega}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{\Omega}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}.$

This indicates that the supremum on the left side of (16) is attained only if
$\inf_{x\in\Omega}\gamma(x)\geq-L\cdot\text{diam}(\Omega)$. Therefore, we have
that the supremum on the left side of (16) is attained only if
$\mathinner{\\!\left\lVert\gamma\right\rVert}_{\infty}\leq(\alpha-1)^{-1}\left(\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}\right)^{\alpha-1}+L\cdot\text{diam}(\Omega)$. ∎

###### Proof of Theorem 1.

1\. Sufficiency.
Let $\Gamma=\text{Lip}_{L}(\mathbb{R}^{d})$, and we have

$\displaystyle D_{\alpha,1}^{L}(P\|Q)$
$\displaystyle=\sup_{\gamma\in\Gamma}\left\\{\int\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int
f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\}$
$\displaystyle\leq\sup_{\gamma\in\text{Lip}_{L}(\mathinner{\\!\left\lVert
x\right\rVert}<R)}\left\\{\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\}$
$\displaystyle\quad+\sup_{\gamma\in\text{Lip}_{L}(\mathinner{\\!\left\lVert
x\right\rVert}\geq R)}\left\\{\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\}$
$\displaystyle\coloneqq I_{1}+I_{2}.$

For $I_{1}$, by Lemma 2, we have

$\displaystyle I_{1}$ $\displaystyle\leq C\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}p(x)\mathop{}\\!\mathrm{d}{x}+\left(\alpha^{-1}(\alpha-1)^{\frac{\alpha}{\alpha-1}}C^{\frac{\alpha}{\alpha-1}}+\alpha^{-1}(\alpha-1)^{-1}\right)\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}q(x)\mathop{}\\!\mathrm{d}{x}<\infty,$

where $C=(\alpha-1)^{-1}\left(\frac{\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}p(x)\mathop{}\\!\mathrm{d}{x}}{\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}q(x)\mathop{}\\!\mathrm{d}{x}}\right)^{\alpha-1}+2LR$.

For $I_{2}$, we have

$\displaystyle\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}p(x)\left(\gamma(x)-f_{\alpha}^{*}[\gamma(x)]\frac{q(x)}{p(x)}\right)\mathop{}\\!\mathrm{d}{x}.$

(i) If $d<\beta_{1}\leq d+1$ and
$\beta_{2}-\beta_{1}<\frac{\beta_{1}-d}{\alpha-1}$:
Note that the set of bounded $L$-Lipschitz functions on
$\\{x\mathrel{\mathop{\mathchar 58\relax}}\mathinner{\\!\left\lVert
x\right\rVert}\geq R\\}$ is a subset of
$\mathcal{M}_{b}(x\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}\geq R)$. Therefore, we have

$\displaystyle\sup_{\gamma\in\text{Lip}_{L}(x\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}\geq
R)}\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}p(x)\left(\gamma(x)-f_{\alpha}^{*}[\gamma(x)]\frac{q(x)}{p(x)}\right)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\leq\sup_{\gamma\in\mathcal{M}_{b}(x\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}\geq
R)}\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}p(x)\left(\gamma(x)-f_{\alpha}^{*}[\gamma(x)]\frac{q(x)}{p(x)}\right)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\frac{1}{\alpha(\alpha-1)}\left(\left[\frac{p(x)}{q(x)}\right]^{\alpha}-1\right)q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\asymp\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\mathinner{\\!\left\lVert
x\right\rVert}^{\alpha(\beta_{2}-\beta_{1})-\beta_{2}}\mathop{}\\!\mathrm{d}{x}<\infty,$

since
$\alpha(\beta_{2}-\beta_{1})-\beta_{2}=(\alpha-1)(\beta_{2}-\beta_{1})-\beta_{1}<-d$,
and the equality is due to the dual formula Eq. 3.
(ii) If $\beta_{1}>d+1$:
In this case, we still have $I_{1}<\infty$. Now we prove that $I_{2}<+\infty$.
Let $M(\gamma)=\sup_{\mathinner{\\!\left\lVert
x\right\rVert}=R}\mathinner{\\!\left\lvert\gamma(x)\right\rvert}$, where
$\gamma\in\text{Lip}_{L}(\mathinner{\\!\left\lVert x\right\rVert}\geq R)$. We
show that there exists some $\overline{M}>0$ such that

$I_{2}=\sup_{\gamma\in\mathcal{G}}\left\\{\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\},$ (18)

where

$\mathcal{G}=\left\\{\gamma\in\text{Lip}_{L}(\mathinner{\\!\left\lVert
x\right\rVert}\geq R)\mathrel{\mathop{\mathchar
58\relax}}M(\gamma)\leq\overline{M}\right\\}.$

Indeed, we have for any $\gamma\in\text{Lip}_{L}(\mathinner{\\!\left\lVert
x\right\rVert}\geq R)$,

$\displaystyle\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{R\leq\mathinner{\\!\left\lVert
x\right\rVert}<2R}\gamma(x)p(x)-f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}+\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
2R}\gamma(x)p(x)-f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\leq\int_{R\leq\mathinner{\\!\left\lVert
x\right\rVert}<2R}\gamma(x)p(x)-f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}+\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq 2R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\leq(M(\gamma)+LR)\int_{R\leq\mathinner{\\!\left\lVert
x\right\rVert}<2R}p(x)\mathop{}\\!\mathrm{d}{x}-\int_{R\leq\mathinner{\\!\left\lVert
x\right\rVert}<2R}f_{\alpha}^{*}(M(\gamma)-3LR)q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\quad+\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
2R}p(x)\left(M(\gamma)+LR+L\mathinner{\\!\left\lVert
x\right\rVert}\right)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=LR\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}p(x)\mathop{}\\!\mathrm{d}{x}+L\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq 2R}p(x)\mathinner{\\!\left\lVert
x\right\rVert}\mathop{}\\!\mathrm{d}{x}+M(\gamma)\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq R}p(x)\mathop{}\\!\mathrm{d}{x}$ $\displaystyle\quad-
f_{\alpha}^{*}(M(\gamma)-3LR)\int_{R\leq\mathinner{\\!\left\lVert
x\right\rVert}<2R}q(x)\mathop{}\\!\mathrm{d}{x},$

where the last inequality is due to the fact that$\gamma(x)$ is $L$-Lipschitz
and that for any $x\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}\geq R$, we have
$\mathinner{\\!\left\lvert\gamma(x)-M(\gamma)\right\rvert}\leq
L(R+\mathinner{\\!\left\lVert x\right\rVert})$. The first two terms are finite
and are independent of $\gamma$ since $\beta_{1}>d+1$. For the difference
between the last two terms, we have

$\lim_{M(\gamma)\to+\infty}M(\gamma)\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}p(x)\mathop{}\\!\mathrm{d}{x}-f_{\alpha}^{*}(M(\gamma)-3LR)\int_{R\leq\mathinner{\\!\left\lVert
x\right\rVert}<2R}q(x)\mathop{}\\!\mathrm{d}{x}=-\infty,$

since the exponent of $x$ in $f_{\alpha}^{*}(x)$ is
$\frac{\alpha}{\alpha-1}>1$. This indicates that the supremum in $I_{2}$
should be taken over $\gamma$ such that $M(\gamma)\leq\overline{M}$ for some
$\overline{M}>0$. Therefore,

$\displaystyle I_{2}$
$\displaystyle=\sup_{\gamma\in\mathcal{G}}\left\\{\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\}$
$\displaystyle\leq\sup_{\gamma\in\mathcal{G}}\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\leq\sup_{\gamma\in\mathcal{G}}\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq R}p(x)\left(LR+L\mathinner{\\!\left\lVert
x\right\rVert}+M(\gamma)\right)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\leq\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}p(x)\left(LR+L\mathinner{\\!\left\lVert
x\right\rVert}+\overline{M}\right)\mathop{}\\!\mathrm{d}{x}<\infty.$

2\. Necessity.
Suppose $\beta_{1}\leq d+1$ and
$\beta_{2}-\beta_{1}\geq\frac{\beta_{1}-d}{\alpha-1}$. We split
$\beta_{2}-\beta_{1}\geq\frac{\beta_{1}-d}{\alpha-1}$ into two cases.
(i) If $\beta_{2}-\beta_{1}\geq\frac{1}{\alpha-1}$:
Let $\widehat{\gamma}(x)=\tau\mathinner{\\!\left\lVert x\right\rVert}$, where
$\tau\in(0,L]$ is to be determined. Then we have
$\widehat{\gamma}\in\text{Lip}_{L}(\mathbb{R}^{d})$. Using this
$\widehat{\gamma}$, we have

$\displaystyle D_{\alpha,1}^{L}(P\|Q)$
$\displaystyle\geq\int\widehat{\gamma}(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int
f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}\widehat{\gamma}(x)p(x)-f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}+\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}\widehat{\gamma}(x)p(x)-f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}.$

It is straightforward that the first integral over $\mathinner{\\!\left\lVert
x\right\rVert}<R$ is finite. For the latter one, we have

$\displaystyle\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\widehat{\gamma}(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\gtrsim\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\left(\tau\mathinner{\\!\left\lVert
x\right\rVert}^{1-\beta_{1}}-\tau^{\frac{\alpha}{\alpha-1}}\mathinner{\\!\left\lVert
x\right\rVert}^{\frac{\alpha}{\alpha-1}-\beta_{2}}\right)\mathop{}\\!\mathrm{d}{x}.$

We need to show the right-hand side is infinite. First, since
$\frac{\alpha}{\alpha-1}>1$, we can choose $\tau$ sufficiently small such that
$\tau>\tau^{\frac{\alpha}{\alpha-1}}$. Moreover, by the assumption, we have
$1-\beta_{1}\geq-d$ and $\frac{\alpha}{\alpha-1}-\beta_{2}\leq 1-\beta_{1}$,
so that we have

$\displaystyle\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\left(\tau\mathinner{\\!\left\lVert
x\right\rVert}^{1-\beta_{1}}-\tau^{\frac{\alpha}{\alpha-1}}\mathinner{\\!\left\lVert
x\right\rVert}^{\frac{\alpha}{\alpha-1}-\beta_{2}}\right)\mathop{}\\!\mathrm{d}{x}=\infty,$

and thus $D_{\alpha,1}^{L}(P\|Q)=\infty$.
(ii) If
$\frac{\beta_{1}-d}{\alpha-1}\leq\beta_{2}-\beta_{1}<\frac{1}{\alpha-1}$:
Define

$\widehat{\gamma}(x)=\begin{cases}\tau
R^{(\alpha-1)(\beta_{2}-\beta_{1})},&\text{if }\mathinner{\\!\left\lVert
x\right\rVert}<R;\\\ \tau\mathinner{\\!\left\lVert
x\right\rVert}^{(\alpha-1)(\beta_{2}-\beta_{1})},&\text{if
}\mathinner{\\!\left\lVert x\right\rVert}\geq R,\end{cases}$

where $\tau\in(0,L]$ is to be determined. Since in this case we have
$(\beta_{2}-\beta_{1})(\alpha-1)<1$, we have
$\widehat{\gamma}(x)\in\text{Lip}_{L}(\mathbb{R}^{d})$ if we pick $R$
sufficiently large which is independent of $\tau\leq L$. Using this
$\widehat{\gamma}(x)$, we have

$\displaystyle D_{\alpha,1}^{L}(P\|Q)$
$\displaystyle\geq\int\widehat{\gamma}(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int
f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}\widehat{\gamma}(x)p(x)-f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}+\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}\widehat{\gamma}(x)p(x)-f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}.$

By the definition of $\widehat{\gamma}$, we know that the first integral over
$\mathinner{\\!\left\lVert x\right\rVert}<R$ is finite. For the latter one, we
have in this case

$\displaystyle\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\widehat{\gamma}(x)p(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle-\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}f_{\alpha}^{*}[\widehat{\gamma}(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\gtrsim\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\left(\tau\mathinner{\\!\left\lVert
x\right\rVert}^{(\alpha-1)(\beta_{2}-\beta_{1})-\beta_{1}}-\tau^{\frac{\alpha}{\alpha-1}}\mathinner{\\!\left\lVert
x\right\rVert}^{(\alpha-1)(\beta_{2}-\beta_{1})-\beta_{1}}\right)\mathop{}\\!\mathrm{d}{x}.$

We show the right-hand side is infinite. Again, we can choose $\tau$
sufficiently small such that $\tau>\tau^{\frac{\alpha}{\alpha-1}}$. On the
other hand, by the assumption in this case, we have
$(\alpha-1)(\beta_{2}-\beta_{1})-\beta_{1}\geq-d$, so that we have

$\displaystyle\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\left(\tau\mathinner{\\!\left\lVert
x\right\rVert}^{(\alpha-1)(\beta_{2}-\beta_{1})-\beta_{1}}-\tau^{\frac{\alpha}{\alpha-1}}\mathinner{\\!\left\lVert
x\right\rVert}^{(\alpha-1)(\beta_{2}-\beta_{1})-\beta_{1}}\right)\mathop{}\\!\mathrm{d}{x}=\infty,$

hence $D_{\alpha,1}^{L}(P\|Q)=\infty$. ∎

We prove the finiteness result for the $W_{1}$ proximal-regularized KL
divergence ($\alpha=1$).

######  Lemma 3.

For the KL case, i.e., $f_{\text{KL}}^{*}(y)=e^{y-1}$ and any non-negative
integrable functions $p(x)$ and $q(x)$ defined on some bounded
$\Omega\subset\mathbb{R}^{d}$ with non-zero integrals,
$\Gamma=\text{Lip}_{L}(\Omega)$, we have

$\sup_{\gamma\in\Gamma}\left\\{\int_{\Omega}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\text{KL}}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\}=\sup_{\gamma\in\mathcal{F}}\left\\{\int_{\Omega}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\text{KL}}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}\right\\},$
(19)

where

$\mathcal{F}=\left\\{\gamma\in\text{Lip}_{L}(\Omega)\mathrel{\mathop{\mathchar
58\relax}}\ln\frac{\int p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}+1-L\cdot\text{diam}(\Omega)\leq\gamma\leq\ln\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}+1+L\cdot\text{diam}(\Omega)\right\\}.$

###### Proof.

For any fixed $\gamma\in\Gamma$, define

$h(\nu)=\int_{\Omega}\left(\gamma(x)+\nu\right)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\text{KL}}^{*}[\gamma(x)+\nu]q(x)\mathop{}\\!\mathrm{d}{x}.$

Since $\sup_{x\in\Omega}\gamma(x)-\inf_{x\in\Omega}\gamma(x)\leq
L\cdot\text{diam}(\Omega)$, interchanging the integration with differentiation
is allowed by the dominated convergence theorem:

$h^{\prime}(\nu)=\int_{\Omega}p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\Omega}f_{\text{KL}}^{*\prime}(\gamma+\nu)q(x)\mathop{}\\!\mathrm{d}{x}.$

If $\inf_{x\in\Omega}\gamma(x)>\ln\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int q(x)\mathop{}\\!\mathrm{d}{x}}+1$, then
$h^{\prime}(0)<0$. So there exists some $\nu_{0}<0$ such that
$h(\nu_{0})>h(0)$. This indicates the supremum on the left side of (19) is
attained only if $\sup_{x\in\Omega}\gamma(x)\leq\ln\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}+1+L\cdot\text{diam}(\Omega)$. On the other
hand, if $\sup_{x\in\Omega}\gamma(x)<\ln\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int q(x)\mathop{}\\!\mathrm{d}{x}}+1$, then
$h^{\prime}(0)>0$. So there exists some $\nu_{0}>0$ such that
$h(\nu_{0})>h(0)$. This indicates that the supremum on the left side of (19)
is attained only if $\inf_{x\in\Omega}\gamma(x)\geq\ln\frac{\int
p(x)\mathop{}\\!\mathrm{d}{x}}{\int
q(x)\mathop{}\\!\mathrm{d}{x}}+1-L\cdot\text{diam}(\Omega)$. ∎

######  Proposition 1.

Suppose $\alpha=1$ (the KL case), and $(P,Q)$ are distributions on
$\mathbb{R}^{d}$ of heavy-tail $(\beta_{1},\beta_{2})$, then
$D_{\text{KL},1}^{L}(P\|Q)<\infty$ for any $\beta_{1},\beta_{2}>d$.

###### Proof.

Same as in the beginning of the proof of Theorem 1, we can split
$D_{\text{KL},1}^{L}(P\|Q)$ into $I_{1}$ and $I_{2}$, where $I_{1}$ is bounded
by Lemma 3 with appropriate $R$.

For $I_{2}$, we have

$\displaystyle\sup_{\gamma\in\text{Lip}_{L}(x\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}\geq
R)}\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}f_{\text{KL}}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\leq\sup_{\gamma\in\mathcal{M}_{b}(x\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}\geq
R)}\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}f_{\text{KL}}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\ln\frac{p(x)}{q(x)}p(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\asymp\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\mathinner{\\!\left\lVert
x\right\rVert}^{-\beta_{1}}\ln\mathinner{\\!\left\lVert
x\right\rVert}\mathop{}\\!\mathrm{d}{x}<\infty,$

since $\beta_{1}>d$ and the equality is due to the dual formula of KL
divergence. ∎

###### Proof of Corollary 1.

Note the change-of-variable formula

$\displaystyle\int_{\mathbb{R}^{d}}\gamma(y)\mathop{}\\!\mathrm{d}{p_{\mathcal{M}}}(y)=\int_{\mathbb{R}^{d^{*}}}(\gamma\circ\varphi)(x)\cdot
p(x)\mathop{}\\!\mathrm{d}{x},\,\text{(similarly for $q_{\mathcal{M}}$ and
$q$)}$

and $\gamma\circ\varphi$ is an $LL^{*}$-Lipschitz function on
$\mathbb{R}^{d^{*}}$ for any $\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})$. Then
the proof of Theorem 1 can be followed. ∎

###### Proof of Theorem 2.

Suppose $d<\beta_{1}\leq d+2$ and
$\beta_{2}-\beta_{1}<\frac{\beta_{1}-d}{\alpha-1}$. Let $\Omega$ be the ball
$B_{R}(0)$ as in Definition 1 or even with larger $R$ if necessary. We design
the intermediate measure $\eta^{*}$ such that

$\displaystyle\eta^{*}(x)=\begin{cases}0,&x\in\Omega\backslash\text{supp}(q)\\\
p(x),&\mathinner{\\!\left\lVert x\right\rVert}>R.\end{cases}$

Basically $\eta^{*}$ has the same tail as $P$ outside $\Omega$, and we
redistribute the density of $p(x)$ inside $\Omega$ to form
$\eta^{*}|_{\Omega}$ so that
$\text{supp}(\eta^{*}|_{\Omega})\subset\text{supp}(q)$. Hence the
Wasserstein-2 distance $W_{2}(P,\eta^{*})$ is bounded since we only need to
transport inside $\Omega$. Moreover,

$\displaystyle D_{\alpha}(\eta^{*}\|Q)$
$\displaystyle=\int_{\Omega}\frac{(\frac{\eta^{*}(x)}{q(x)})^{\alpha}-1}{\alpha(\alpha-1)}q(x)\mathop{}\\!\mathrm{d}{x}+\int_{\mathbb{R}^{d}\backslash\Omega}\frac{(\frac{\eta^{*}(x)}{q(x)})^{\alpha}-1}{\alpha(\alpha-1)}q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\leq
C+\int_{\mathbb{R}^{d}\backslash\Omega}\frac{(\frac{p(x)}{q(x)})^{\alpha}-1}{\alpha(\alpha-1)}q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle\asymp
C+\int_{\mathbb{R}^{d}\backslash\Omega}\mathinner{\\!\left\lVert
x\right\rVert}^{(\beta_{2}-\beta_{1})\alpha}\mathinner{\\!\left\lVert
x\right\rVert}^{-\beta_{2}}\mathop{}\\!\mathrm{d}{x},$

and to ensure finiteness, we are only to require
$(\beta_{2}-\beta_{1})\alpha-\beta_{2}<-d$, which is
$\beta_{2}-\beta_{1}<\frac{\beta_{1}-d}{\alpha-1}$. Therefore, we have

$\displaystyle D_{\alpha,2}^{\lambda}(P\|Q)$
$\displaystyle=\inf_{\eta\in\mathcal{P}(\mathbb{R})}\\{D_{\alpha}(\eta\|Q)+\lambda
W_{2}(P,\eta)\\}$ $\displaystyle\leq D_{\alpha}(\eta^{*}\|Q)+\lambda
W_{2}(P,\eta^{*})<\infty.$

For the case when $\beta_{1}>d+2$, we can design $\eta^{*}$ so that
$\text{supp}(\eta^{*})\subset\text{supp}(q)\cap\Omega$, so that both
$D_{\alpha}(\eta^{*}\|Q)$ and $W_{2}(P,\eta^{*})$ are finite. ∎

##  Appendix D Proof of Lemma 1 and Theorem 3

###### Proof of Lemma 1.

From Theorem 70 in [3], we have

$\displaystyle\Lambda_{f_{\alpha}}^{Q}[\gamma]=\sup_{\mu}\\{\mathbb{E}_{\mu}[\gamma]-D_{\alpha,1}^{L}(\mu\|Q)\\},$

such that we have

$\displaystyle D_{\alpha,1}^{L}(P\|Q)$
$\displaystyle=\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\\{\mathbb{E}_{P}[\gamma]-\Lambda_{f_{\alpha}}^{Q}[\gamma]\\}$
$\displaystyle=\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\\{\mathbb{E}_{P}[\gamma]-\sup_{\mu}\\{\mathbb{E}_{\mu}[\gamma]-D_{\alpha,1}^{L}(\mu\|Q)\\}\\}$
$\displaystyle\leq\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\\{\mathbb{E}_{P}[\gamma]-\\{\mathbb{E}_{\eta}[\gamma]-D_{\alpha,1}^{L}(\eta\|Q)\\}\\}$
$\displaystyle=\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\\{\mathbb{E}_{P}[\gamma]-\mathbb{E}_{\eta}[\gamma]\\}+D_{\alpha,1}^{L}(\eta\|Q)$
$\displaystyle=L\cdot W_{1}(P,\eta)+D_{\alpha,1}^{L}(\eta\|Q).$

∎

To prove Theorem 3, we need a few lemmas. Let
$x_{1},x_{2},\dots,x_{m}\in\mathbb{R}^{d}$ be i.i.d. samples of distribution
$P$, and $P_{m}$ be the corresponding empirical distributions. We define
$L_{2}(P_{m})$ the metric between any functions $f,g$ as
$L_{2}(P_{m})(f,g)=\sqrt{\frac{1}{m}\sum_{i=1}^{m}\mathinner{\\!\left\lvert
f(x_{i})-g(x_{i})\right\rvert}^{2}}$.

######  Lemma 4 (Metric entropy with empirical measures).

Let $\mathcal{F}$ be a class of real-valued functions on $\mathbb{R}^{d}$ and
$0\in\mathcal{F}$. Let $\xi=\\{\xi_{1},\xi_{2},\dots,\xi_{m}\\}$ be a set of
independent random variables that take values on $\\{-1,1\\}$ with equal
probabilities (also known as Rademacher variables). Suppose
$X=\\{x_{1},x_{2},\dots,x_{m}\\}\subset\mathbb{R}^{d}$ are i.i.d. samples of
distribution $P$, then we have

$\displaystyle\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}f(x_{i})\right\rvert}\leq\inf_{0<\theta<M_{X}}\left(4\theta+\frac{12}{\sqrt{m}}\int_{\theta}^{M_{X}}\sqrt{\ln\mathcal{N}(\mathcal{F},\delta,L_{2}(P_{m}))}\mathop{}\\!\mathrm{d}{\delta}\right),$

where
$M_{X}=\sup_{f\in\mathcal{F}}\sqrt{\frac{1}{m}\sum_{i=1}^{m}\mathinner{\\!\left\lvert
f(x_{i})\right\rvert}^{2}}$.

###### Proof.

Let $N\in\mathbb{N}$ be an arbitrary positive integer and
$\delta_{k}=M_{X}\cdot 2^{-(k-1)}$, $k=1,\dots,N$, with
$M_{X}=\sup_{f\in\mathcal{F}}\sqrt{\frac{1}{m}\sum_{i=1}^{m}\mathinner{\\!\left\lvert
f(x_{i})\right\rvert}^{2}}$. Let $V_{k}$ be the cover achieving
$\mathcal{N}(\mathcal{F},\delta_{k},L_{2}(P_{m}))$, and denote
$\mathinner{\\!\left\lvert
V_{k}\right\rvert}=\mathcal{N}(\mathcal{F},\delta_{k},L_{2}(P_{m}))$. For any
$f\in\mathcal{F}$, let $\pi_{k}(f)\in V_{k}$, such that
$\sqrt{\frac{1}{m}\sum_{i=1}^{m}\mathinner{\\!\left\lvert
f(x_{i})-\pi_{k}(f)(x_{i})\right\rvert}^{2}}\leq\delta_{k}$. We have

$\displaystyle\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}f(x_{i})\right\rvert}$
$\displaystyle\leq\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}\left(f(x_{i})-\pi_{N}(f)(x_{i})\right)\right\rvert}+\sum_{j=1}^{N-1}\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}\left(\pi_{j+1}(f)(x_{i})-\pi_{j}(f)(x_{i})\right)\right\rvert}$
$\displaystyle\quad+\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}\pi_{1}(f)(x_{i})\right\rvert}.$

For the third term, observe that it suffices to take $V_{1}=\\{0\\}$ so that
$\pi_{1}(f)$ is the zero function and the third term vanishes. The first term
can be bounded using Cauchy-Schwartz inequality as

$\displaystyle\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}\left(f(x_{i})-\pi_{N}(f)(x_{i})\right)\right\rvert}$
$\displaystyle\leq\frac{1}{m}\sqrt{\sum_{i=1}^{m}\mathbb{E}_{\xi}(\xi_{i})^{2}}\sqrt{\sup_{f\in\mathcal{F}}\sum_{i=1}^{m}\left(f(x_{i})-\pi_{N}(f)(x_{i})\right)^{2}}$
$\displaystyle\leq\delta_{N}.$

To handle the middle term, for each $j$, let
$W_{j}=\\{\pi_{j+1}(f)-\pi_{j}(f)\mathrel{\mathop{\mathchar
58\relax}}f\in\mathcal{F}\\}$. We have $\mathinner{\\!\left\lvert
W_{j}\right\rvert}\leq\mathinner{\\!\left\lvert
V_{j+1}\right\rvert}\mathinner{\\!\left\lvert
V_{j}\right\rvert}\leq\mathinner{\\!\left\lvert V_{j+1}\right\rvert}^{2}$,
then

$\displaystyle\sum_{j=1}^{N-1}\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}\left(\pi_{j+1}(f)(x_{i})-\pi_{j}(f)(x_{i})\right)\right\rvert}=\sum_{j=1}^{N-1}\mathbb{E}_{\xi}\sup_{w\in
W_{j}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}w(x_{i})\right\rvert}.$

Moreover, we have

$\displaystyle\sup_{w\in W_{j}}\sqrt{\sum_{i=1}^{m}w(x_{i})^{2}}$
$\displaystyle=\sup_{f\in\mathcal{F}}\sqrt{\sum_{i=1}^{m}\left(\pi_{j+1}(f)(x_{i})-\pi_{j}(f)(x_{i})\right)^{2}}$
$\displaystyle\leq\sup_{f\in\mathcal{F}}\sqrt{\sum_{i=1}^{m}\left(\pi_{j+1}(f)(x_{i})-f(x_{i})\right)^{2}}+\sup_{f\in\mathcal{F}}\sqrt{\sum_{i=1}^{m}\left(f(x_{i})-\pi_{j}(f)(x_{i})\right)^{2}}$
$\displaystyle\leq\sqrt{m}\delta_{j+1}+\sqrt{m}\cdot\delta_{j}$
$\displaystyle=3\sqrt{m}\delta_{j+1}.$

By the Massart finite class lemma (see, e.g. [22]), we have

$\displaystyle\mathbb{E}_{\xi}\sup_{w\in
W_{j}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}w(x_{i})\right\rvert}\leq\frac{3\sqrt{m}\delta_{j+1}\sqrt{2\ln\mathinner{\\!\left\lvert
W_{j}\right\rvert}}}{m}\leq\frac{6\delta_{j+1}\sqrt{\ln\mathinner{\\!\left\lvert
V_{j+1}\right\rvert}}}{\sqrt{m}}.$

Therefore,

$\displaystyle\mathbb{E}_{\xi}\sup_{f\in\mathcal{F}}\mathinner{\\!\left\lvert\frac{1}{m}\sum_{i=1}^{m}\xi_{i}f(x_{i})\right\rvert}$
$\displaystyle\leq\delta_{N}+\frac{6}{\sqrt{m}}\sum_{j=1}^{N-1}\delta_{j+1}\sqrt{\ln\mathcal{N}(\mathcal{F},\delta_{j+1},L_{2}(P_{m}))}$
$\displaystyle\leq\delta_{N}+\frac{12}{\sqrt{m}}\sum_{j=1}^{N}(\delta_{j}-\delta_{j+1})\sqrt{\ln\mathcal{N}(\mathcal{F},\delta_{j},L_{2}(P_{m}))}$
$\displaystyle\leq\delta_{N}+\frac{12}{\sqrt{m}}\int_{\delta_{N+1}}^{M_{X}}\sqrt{\ln\mathcal{N}(\mathcal{F},\delta,L_{2}(P_{m}))}\mathop{}\\!\mathrm{d}{\delta}.$

Finally, select any $\theta\in(0,M_{X})$ and let $N$ be the largest integer
with $\delta_{N+1}>\theta$, (implying $\delta_{N+2}\leq\theta$ and
$\delta_{N}=4\delta_{N+2}\leq 4\theta$), so that

$\delta_{N}+\frac{12}{\sqrt{m}}\int_{\delta_{N+1}}^{M_{X}}\sqrt{\ln\mathcal{N}(\mathcal{F},\delta,L_{2}(P_{m}))}\mathop{}\\!\mathrm{d}{\delta}\leq
4\theta+\frac{12}{\sqrt{m}}\int_{\theta}^{M_{X}}\sqrt{\ln\mathcal{N}(\mathcal{F},\delta,L_{2}(P_{m}))}\mathop{}\\!\mathrm{d}{\delta}.$

∎

######  Lemma 5.

Suppose $P_{m}$ is the empirical distribution of $P$ of heavy-tail
$\beta>d+1$, and $\Lambda=\frac{1}{m}\sum_{i=1}^{m}\mathinner{\\!\left\lVert
x\right\rVert}^{\hat{\beta}}$ with $1\leq\hat{\beta}<\beta-d$, then for $1\leq
z\leq\hat{\beta}$, we have

$\displaystyle\mathbb{E}_{P_{m}}\mathinner{\\!\left\lVert
x\right\rVert}^{z}\leq\Lambda+1.$

###### Proof.

Note that $\mathinner{\\!\left\lVert
x\right\rVert}^{z}\leq\max\\{1,\mathinner{\\!\left\lVert
x\right\rVert}^{\hat{\beta}}\\}\leq 1+\mathinner{\\!\left\lVert
x\right\rVert}^{\hat{\beta}}$, so we have the bound. ∎

We provide the following lemma that sets up a landmark for the magnitude of
the Lipschitz functions under the supremum.

######  Lemma 6.

Suppose $\alpha>1$, and $(P,Q)$ are distributions on $\mathbb{R}^{d}$ of
heavy-tail $(\beta_{1},\beta_{2})$ with $\beta_{1}>d+1$. Let
$M(\gamma)=\sup_{\mathinner{\\!\left\lVert
x\right\rVert}=R}\mathinner{\\!\left\lvert\gamma(x)\right\rvert}$, then there
exists $\overline{M}$ that depends on $P,Q$ and $R$, such that

$D_{\alpha,1}^{L}(P\|Q)=\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq\overline{M}\end{subarray}}\left\\{\mathbb{E}_{P}[\gamma]-\mathbb{E}_{Q}[f_{\alpha}^{*}(\gamma)]\right\\}.$

###### Proof.

For any $\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})$, let

$J_{1}\mathrel{\mathop{\mathchar 58\relax}}=\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x},$
$\displaystyle J_{2}\mathrel{\mathop{\mathchar
58\relax}}=\int_{\mathinner{\\!\left\lVert x\right\rVert}\geq
R}\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq R}f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x},$

then

$\displaystyle\int\gamma(x)p(x)\mathop{}\\!\mathrm{d}{x}-\int
f_{\alpha}^{*}[\gamma(x)]q(x)\mathop{}\\!\mathrm{d}{x}=J_{1}+J_{2}.$

We have for any $\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})$,

$\displaystyle J_{1}$ $\displaystyle\leq\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}(M(\gamma)+LR)p(x)\mathop{}\\!\mathrm{d}{x}-\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}f_{\alpha}^{*}(M(\gamma)-3LR)q(x)\mathop{}\\!\mathrm{d}{x}$
$\displaystyle=(M(\gamma)+LR)\cdot\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}p(x)\mathop{}\\!\mathrm{d}{x}-f_{\alpha}^{*}(M(\gamma)-3LR)\cdot\int_{\mathinner{\\!\left\lVert
x\right\rVert}<R}q(x)\mathop{}\\!\mathrm{d}{x}.$

On the other hand, by the same argument in the proof of part (ii) of the
sufficiency of Theorem 1, we have

$\displaystyle J_{2}$ $\displaystyle\leq LR\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq
R}p(x)\mathop{}\\!\mathrm{d}{x}+L\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq 2R}p(x)\mathinner{\\!\left\lVert
x\right\rVert}\mathop{}\\!\mathrm{d}{x}+M(\gamma)\int_{\mathinner{\\!\left\lVert
x\right\rVert}\geq R}p(x)\mathop{}\\!\mathrm{d}{x}$ $\displaystyle\quad-
f_{\alpha}^{*}(M(\gamma)-3LR)\int_{R\leq\mathinner{\\!\left\lVert
x\right\rVert}<2R}q(x)\mathop{}\\!\mathrm{d}{x},$

Both the upper bounds for $J_{1}$ and $J_{2}$ tend to $-\infty$ as
$M(\gamma)\to\infty$. Thus, there exists such $\overline{M}$ as claimed.

$\displaystyle J_{1}+J_{2}$ $\displaystyle\leq(M(\gamma)+LR)\int
p(x)\mathop{}\\!\mathrm{d}{x}+L\int p(x)\mathinner{\\!\left\lVert
x\right\rVert}\mathop{}\\!\mathrm{d}{x}$ $\displaystyle\quad-
f_{\alpha}^{*}(M(\gamma)-3LR)\int_{\mathinner{\\!\left\lVert
x\right\rVert}<2R}q(x)\mathop{}\\!\mathrm{d}{x}$

Therefore, we can pick $\overline{M}>0$ as

$\inf\left\\{\hat{M}\mathrel{\mathop{\mathchar 58\relax}}(M(\gamma)+LR)\int
p(x)\mathop{}\\!\mathrm{d}{x}+L\int p(x)\mathinner{\\!\left\lVert
x\right\rVert}\mathop{}\\!\mathrm{d}{x}-f_{\alpha}^{*}(M(\gamma)-3LR)\int_{\mathinner{\\!\left\lVert
x\right\rVert}<2R}q(x)\mathop{}\\!\mathrm{d}{x}<0,\forall
M(\gamma)>\hat{M}\right\\},$

Such $\overline{M}>0$ only depends on $P,Q$ and $R$. ∎

Let $\overline{M}_{m,n}$ be the quantity in Lemma 6 where $(P,Q)$ are replaced
by their empirical counterparts $(P_{m},Q_{n})$, then $\overline{M}_{m,n}$ is
a random variable. We have the following lemma to estimate the expectation of
the $r$-th moment ($r\geq 1$) of $\overline{M}_{m,n}$. The proof is different
from that for Lemma 6.

######  Lemma 7.

Suppose $\alpha>1$, and $(P,Q)$ are distributions on $\mathbb{R}^{d}$ of
heavy-tail $(\beta_{1},\beta_{2})$ with $\beta_{1},\beta_{2}>d+r$ for some
$r\geq 1$. Let $M(\gamma)=\sup_{\mathinner{\\!\left\lVert
x\right\rVert}=R}\mathinner{\\!\left\lvert\gamma(x)\right\rvert}$, then there
exists $\overline{M}_{m,n}$ that depends on $P_{m},Q_{n}$ and $R$, such that

$D_{\alpha,1}^{L}(P_{m}\|Q_{n})=\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq\overline{M}_{m,n}\end{subarray}}\left\\{\mathbb{E}_{P_{m}}[\gamma]-\mathbb{E}_{Q_{n}}[f_{\alpha}^{*}(\gamma)]\right\\},$

Moreover, we have

$\displaystyle\mathbb{E}_{X,Y}\left[\overline{M}_{m,n}^{r}\right]\leq
M_{p,q,r},$

where $M_{p,q,r}$ depends on $\alpha,L,R,M_{r}(p)$ and $M_{r}(q)$, and is
independent of $m,n$.

###### Proof.

We have

$\displaystyle\mathbb{E}_{P_{m}}[\gamma]-\mathbb{E}_{Q_{n}}[f_{\alpha}^{*}(\gamma)]$
$\displaystyle\leq\sum_{i=1}^{m}\frac{M(\gamma)+L\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
x_{i}\right\rVert}-R\right\rvert}}{m}-\sum_{j=1}^{n}\frac{f_{\alpha}^{*}\left(M(\gamma)-2LR-L\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
y_{j}\right\rVert}-R\right\rvert}\right)}{n}.$

Hence $\overline{M}_{m,n}$ can be taken as

$\displaystyle\overline{M}_{m,n}=\inf\left\\{z\mathrel{\mathop{\mathchar
58\relax}}\sum_{i=1}^{m}\frac{s+L\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
x_{i}\right\rVert}-R\right\rvert}}{m}<\sum_{j=1}^{n}\frac{f_{\alpha}^{*}\left(s-2LR-L\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
y_{j}\right\rVert}-R\right\rvert}\right)}{n},\forall s>z\right\\}.$

Moreover, by Jensen’s inequality, we have

$\displaystyle\sum_{j=1}^{n}\frac{f_{\alpha}^{*}\left(s-2LR-L\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
y_{j}\right\rVert}-R\right\rvert}\right)}{n}\geq
f_{\alpha}^{*}\left(s-2LR-L\sum_{j=1}^{n}\frac{\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
y_{j}\right\rVert}-R\right\rvert}}{n}\right),$

since the convex conjugate $f_{\alpha}^{*}$ is convex, and so that

$\displaystyle\overline{M}_{m,n}$
$\displaystyle\leq\inf\left\\{z\mathrel{\mathop{\mathchar
58\relax}}\sum_{i=1}^{m}\frac{s+L\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
x_{i}\right\rVert}-R\right\rvert}}{m}<f_{\alpha}^{*}\left(s-2LR-L\sum_{j=1}^{n}\frac{\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
y_{j}\right\rVert}-R\right\rvert}}{n}\right),\forall s>z\right\\}$
$\displaystyle\coloneqq\widetilde{M}_{m,n}.$

It is obvious that $\widetilde{M}_{m,n}$ solves the following equation in
variable $z$:

$f_{\alpha}^{*}(z-c_{1})=z+c_{2},$ (20)

where

$\displaystyle c_{1}$
$\displaystyle=2LR+L\sum_{j=1}^{n}\frac{\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
y_{j}\right\rVert}-R\right\rvert}}{n},$ $\displaystyle c_{2}$
$\displaystyle=\sum_{i=1}^{m}\frac{L\mathinner{\\!\left\lvert\mathinner{\\!\left\lVert
x_{i}\right\rVert}-R\right\rvert}}{m}.$

Equation (20) can be reformulated as to find $y^{*}$ that solves:

$f_{\alpha}^{*}(y)-y=c_{1}+c_{2},$ (21)

where $z-c_{1}=y$. We derive an upper bound for $y^{*}$ as follows. Let
$g(y)=f_{\alpha}^{*}(y)-y$, then

$\displaystyle
g^{\prime}(y)=(\alpha-1)^{\frac{1}{\alpha-1}}y^{\frac{1}{\alpha-1}}\mathbf{1}_{y>0}-1,$

such that $g^{\prime}(y)\geq 1$ for $y>2^{\alpha-1}(\alpha-1)^{-1}$. Given
that
$g\left(2^{\alpha-1}(\alpha-1)^{-1}\right)=\frac{2^{\alpha}}{\alpha}+\frac{1}{\alpha(\alpha-1)}-\frac{2^{\alpha-1}}{\alpha-1}$,
we can take $y^{*}\leq
2^{\alpha-1}(\alpha-1)^{-1}+c_{1}+c_{2}+\mathinner{\\!\left\lvert\frac{2^{\alpha}}{\alpha}+\frac{1}{\alpha(\alpha-1)}-\frac{2^{\alpha-1}}{\alpha-1}\right\rvert}$.
Therefore, we have

$\displaystyle\overline{M}_{m,n}\leq\widetilde{M}_{m,n}=y^{*}+c_{1}\leq
2^{\alpha-1}(\alpha-1)^{-1}+2c_{1}+c_{2}+\mathinner{\\!\left\lvert\frac{2^{\alpha}}{\alpha}+\frac{1}{\alpha(\alpha-1)}-\frac{2^{\alpha-1}}{\alpha-1}\right\rvert}.$

The claim follows since by Jensen’s inequality,
$\mathbb{E}_{X}\left[\left(\sum_{i=1}^{m}\frac{\mathinner{\\!\left\lVert
x_{i}\right\rVert}}{m}\right)^{r}\right]\leq\mathbb{E}_{X}\left[\sum_{i=1}^{m}\frac{\mathinner{\\!\left\lVert
x_{i}\right\rVert}^{r}}{m}\right]=M_{r}(p)$. (Similarly for
$\mathbb{E}_{Y}\left[\left(\sum_{j=1}^{n}\frac{\mathinner{\\!\left\lVert
y_{j}\right\rVert}}{n}\right)^{r}\right]$.) ∎

###### Proof of Theorem 3.

Without loss of generality, we assume that both

$\displaystyle\int_{\mathinner{\\!\left\lVert x\right\rVert}\leq
1}p(x)\mathop{}\\!\mathrm{d}{x}>0,\quad\int_{\mathinner{\\!\left\lVert
x\right\rVert}\leq 1}q(x)\mathop{}\\!\mathrm{d}{x}>0.$

Let $\Omega_{0}=\\{x\in\mathbb{R}^{d}\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}\leq 1\\}$ and
$\Omega_{k}=\\{x\in\mathbb{R}^{d}\mathrel{\mathop{\mathchar
58\relax}}2^{k-1}<\mathinner{\\!\left\lVert x\right\rVert}\leq 2^{k}\\}$ for
$k\geq 1$. For each $k\in\mathbb{N}$, the Lebesgue measure of
$\\{x\mathrel{\mathop{\mathchar
58\relax}}\mathop{}\\!\mathrm{d}(x,\Omega_{k})\leq 1\\}$ is bounded by
$C_{d}2^{kd}$ for some $C_{d}>0$. Let
$\Lambda_{2}=\frac{1}{n}\sum_{j=1}^{n}\mathinner{\\!\left\lVert
y_{j}\right\rVert}^{\hat{\beta}_{2}}$, where
$2+\frac{2\alpha}{\alpha-1}<\hat{\beta}_{2}<\frac{\beta_{2}}{d}-1$. By
Markov’s inequality, the mass or proportion of $Q_{n}$ that lies in
$\Omega_{k}$ is bounded by

$\displaystyle\Pr(x\sim Q_{n}\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert x\right\rVert}>2^{k-1})$
$\displaystyle=\Pr(x\sim Q_{n}\mathrel{\mathop{\mathchar
58\relax}}\mathinner{\\!\left\lVert
x\right\rVert}^{\hat{\beta}_{2}}>2^{(k-1)\hat{\beta}_{2}})$
$\displaystyle\leq\frac{\mathbb{E}_{Q_{n}}\mathinner{\\!\left\lVert
x\right\rVert}^{\hat{\beta}_{2}}}{2^{(k-1)\hat{\beta}_{2}}}=\Lambda_{2}2^{-(k-1)\hat{\beta}_{2}}.$

Let $M=\max(\overline{M},\overline{M}_{m,n})$, where $\overline{M}$ is the
quantity in Lemma 6 with $R=1$, and $\overline{M}_{m,n}$ is the random
counterpart for $(P_{m},Q_{n})$ as defined in Lemma 7. $M$ is a random
variable since $\overline{M}_{m,n}$ is random. Let $\mathcal{F}_{M}$ be the
following class of functions

$\displaystyle\mathcal{F}_{\alpha,M}=\left\\{f_{\alpha}^{*}(\gamma)\mathrel{\mathop{\mathchar
58\relax}}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d}),\sup_{\mathinner{\\!\left\lVert
x\right\rVert}=1}\mathinner{\\!\left\lvert\gamma(x)\right\rvert}\leq
M\right\\}.$

By formulas (4) and (17) , functions in $\mathcal{F}_{\alpha,M}$ have Hölder
norm on $\Omega_{k}$ bounded by
$C_{\alpha}(M^{\frac{\alpha}{\alpha-1}}+L^{\frac{\alpha}{\alpha-1}}2^{\frac{\alpha
k}{\alpha-1}})$ for some $C_{\alpha}>0$ that only depends on $\alpha$. By
Corollary 2.7.4 in [32] with $V=d$ and $r=2$, we have

$\displaystyle\ln(\mathcal{F}_{\alpha,M},\delta,L_{2}(Q_{n}))$
$\displaystyle\leq
K\delta^{-d}\left(\sum_{k=0}^{\infty}(C_{d}2^{kd})^{\frac{2}{d+2}}\left(C_{\alpha}(M^{\frac{\alpha}{\alpha-1}}+L^{\frac{\alpha}{\alpha-1}}2^{\frac{\alpha
k}{\alpha-1}})\right)^{\frac{2d}{d+2}}(\Lambda_{2}2^{-(k-1)\hat{\beta}_{2}})^{\frac{d}{d+2}}\right)^{d+2}$
$\displaystyle\leq
K\delta^{-d}(M+L)^{\frac{2d\alpha}{\alpha-1}}\Lambda_{2}^{d}\left(\sum_{k=0}^{\infty}2^{\frac{2kd}{d+2}+\frac{2\alpha
kd}{(\alpha-1)(d+2)}-\frac{\hat{\beta}_{2}d(k-1)}{d+2}}\right)^{d+2}$
$\displaystyle\leq
K\delta^{-d}(M+L)^{\frac{2d\alpha}{\alpha-1}}\Lambda_{2}^{d}.$

where the constant $K$ can vary from line to line and does not depend on $n$,
and the last step follows as the choice of $\hat{\beta}_{2}$ such that the
series is summable over $k$ independent of $Q_{n}$. Then we have

$\displaystyle\mathbb{E}_{X,Y}\mathinner{\\!\left\lvert
D_{\alpha,1}^{L}(P_{m}\|Q_{n})-D_{\alpha,1}^{L}(P\|Q)\right\rvert}$
$\displaystyle=\mathbb{E}_{X,Y}\mathinner{\\!\left\lvert\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq\overline{M}_{m,n}\end{subarray}}\left\\{\mathbb{E}_{P_{m}}[\gamma]-\mathbb{E}_{Q_{n}}[f_{\alpha}^{*}(\gamma)]\right\\}-\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq\overline{M}\end{subarray}}\left\\{\mathbb{E}_{P}[\gamma]-\mathbb{E}_{Q}[f_{\alpha}^{*}(\gamma)]\right\\}\right\rvert}$
$\displaystyle\leq\mathbb{E}_{X,Y}\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq
M\end{subarray}}\mathinner{\\!\left\lvert\mathbb{E}_{P_{m}}[\gamma]-\mathbb{E}_{Q_{n}}[f_{\alpha}^{*}(\gamma)]-\left(\mathbb{E}_{P}[\gamma]-\mathbb{E}_{Q}[f_{\alpha}^{*}(\gamma)]\right)\right\rvert}$
$\displaystyle\leq\mathbb{E}_{X}\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\mathinner{\\!\left\lvert\mathbb{E}_{P}[\gamma]-\mathbb{E}_{P_{m}}[\gamma]\right\rvert}+\mathbb{E}_{X,Y}\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq
M\end{subarray}}\mathinner{\\!\left\lvert\mathbb{E}_{Q}[f_{\alpha}^{*}(\gamma)]-\mathbb{E}_{Q_{n}}[f_{\alpha}^{*}(\gamma)]\right\rvert}$
$\displaystyle\leq\mathbb{E}_{X}\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\mathinner{\\!\left\lvert\mathbb{E}_{P}[\gamma]-\mathbb{E}_{P_{m}}[\gamma]\right\rvert}+\mathbb{E}_{X}\mathbb{E}_{Y}\mathbb{E}_{Y^{\prime}}\mathbb{E}_{\xi}\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq
M\end{subarray}}\mathinner{\\!\left\lvert\frac{1}{n}\sum_{j=1}^{n}\xi_{i}\left(f_{\alpha}^{*}[\gamma(y_{j})]-f_{\alpha}^{*}[\gamma(y_{j}^{\prime})]\right)\right\rvert}$
$\displaystyle\leq\mathbb{E}_{X}\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\mathinner{\\!\left\lvert\mathbb{E}_{P}[\gamma]-\mathbb{E}_{P_{m}}[\gamma]\right\rvert}+2\mathbb{E}_{X}\mathbb{E}_{Y}\mathbb{E}_{\xi}\sup_{\begin{subarray}{c}\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})\\\
M(\gamma)\leq
M\end{subarray}}\mathinner{\\!\left\lvert\frac{1}{n}\sum_{j=1}^{n}\xi_{i}f_{\alpha}^{*}[\gamma(y_{j})]\right\rvert}$
$\displaystyle\leq\mathbb{E}_{X}\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\mathinner{\\!\left\lvert\mathbb{E}_{P}[\gamma]-\mathbb{E}_{P_{m}}[\gamma]\right\rvert}+2\mathbb{E}_{X,Y}\inf_{\theta>0}\left(4\theta+\frac{12}{\sqrt{n}}\int_{\theta}^{\infty}\sqrt{\ln\mathcal{N}(\mathcal{F}_{\alpha,M},\delta,L_{2}(Q_{n}))}\mathop{}\\!\mathrm{d}{\delta}\right),$

where $\xi_{i}$’s are the Rademacher variables.

First note that the first term
$\mathbb{E}_{X}\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\mathinner{\\!\left\lvert\mathbb{E}_{P}[\gamma]-\mathbb{E}_{P_{m}}[\gamma]\right\rvert}$
is the convergence rate of the Wasserstein-1 distance and the bound follows
the result of Theorem 1 in [11]:

$\displaystyle\mathbb{E}_{X}\sup_{\gamma\in\text{Lip}_{L}(\mathbb{R}^{d})}\mathinner{\\!\left\lvert\mathbb{E}_{P}[\gamma]-\mathbb{E}_{P_{m}}[\gamma]\right\rvert}\leq\frac{CM_{r}^{1/r}(p)}{m},$

with $r=\frac{d}{d-1}$. For the second term, we have

$\displaystyle\mathbb{E}_{X,Y}\inf_{\theta>0}\left(4\theta+\frac{12}{\sqrt{n}}\int_{\theta}^{\infty}\sqrt{\ln\mathcal{N}(\mathcal{F}_{\alpha,M},\delta,L_{2}(Q_{n}))}\mathop{}\\!\mathrm{d}{\delta}\right)$
$\displaystyle\leq\mathbb{E}_{X,Y}\inf_{\theta>0}\left(4\theta+\frac{12}{\sqrt{n}}K(M+L)^{\frac{d\alpha}{(\alpha-1)}}\Lambda_{2}^{d/2}\int_{\theta}^{\infty}\delta^{-\frac{d}{2}}\mathop{}\\!\mathrm{d}{\delta}\right)$
$\displaystyle\leq\mathbb{E}_{X,Y}\inf_{\theta>0}\left(4\theta+\frac{12}{\sqrt{n}}K(M+L)^{\frac{d\alpha}{(\alpha-1)}}\Lambda_{2}^{d/2}\cdot\frac{2}{2-d}\theta^{1-d/2}\right)$
$\displaystyle\leq\mathbb{E}_{X,Y}\left(4n^{-\frac{1}{d}}+12K(M+L)^{\frac{d\alpha}{(\alpha-1)}}\Lambda_{2}^{d/2}\cdot\frac{2}{2-d}n^{-\frac{1}{d}}\right)$
$\displaystyle=4n^{-\frac{1}{d}}+\frac{24K}{2-d}n^{-\frac{1}{d}}\cdot\mathbb{E}_{X,Y}\left[(M+L)^{\frac{d\alpha}{(\alpha-1)}}\Lambda_{2}^{d/2}\right]$

where we pick $\theta=n^{-\frac{1}{d}}$. By the Cauchy-Schwartz inequality, we
have

$\displaystyle\mathbb{E}_{X,Y}\left[(M+L)^{\frac{d\alpha}{(\alpha-1)}}\Lambda_{2}^{d/2}\right]$
$\displaystyle\leq\sqrt{\mathbb{E}_{X,Y}(M+L)^{\frac{2d\alpha}{(\alpha-1)}}}\sqrt{\mathbb{E}_{Y}\Lambda_{2}^{d}}.$

Notice that $\mathbb{E}_{X,Y}(M+L)^{\frac{2d\alpha}{(\alpha-1)}}$ is bounded
by Lemma 7 and the bound depends on $M_{\frac{2d\alpha}{\alpha-1}}(p)$ and
$M_{\frac{2d\alpha}{\alpha-1}}(q)$. And we have

$\displaystyle\mathbb{E}_{Y}\Lambda_{2}^{d}=\mathbb{E}_{Y}\left(\frac{1}{n}\sum_{j=1}^{n}\mathinner{\\!\left\lVert
y_{j}\right\rVert}^{\hat{\beta}_{2}}\right)^{d}\leq\mathbb{E}_{Y}\left(\frac{1}{n}\sum_{j=1}^{n}\mathinner{\\!\left\lVert
y_{j}\right\rVert}^{\hat{\beta}_{2}d}\right)=M_{\hat{\beta}_{2}d}(q),$

where the inequality follows Jensen’s inequality. Combining all these bounds,
we obtain the result as in the statement of the theorem. ∎

######  Proposition 2.

For $d=2$. Assume $(P,Q)$ are distributions on $\mathbb{R}^{d}$ of heavy-tail
$(\beta_{1},\beta_{2})$, where $\beta_{1}>10$ and $\beta_{2}>18$. Suppose
$\alpha$ satisfies $\frac{4\alpha}{\alpha-1}+4<\beta_{1}-2$ and
$\frac{8\alpha}{\alpha-1}<\beta_{2}-10$, then if $m$ and $n$ are sufficiently
large, we have

$\mathbb{E}_{X,Y}\mathinner{\\!\left\lvert
D_{\alpha,1}^{L}(P_{m}\|Q_{n})-D_{\alpha,1}^{L}(P\|Q)\right\rvert}\leq\frac{C_{1}\ln
m}{m^{1/2}}+\frac{C_{2}\ln n}{n^{1/2}},$ (22)

where $C_{1}$ depends on $M_{r_{1}}(p)$ for any $r_{1}>2$ and $C_{2}$ depends
on $M_{\frac{4\alpha}{\alpha-1}+4}(p)$, $M_{\frac{4\alpha}{\alpha-1}+4}(q)$
and $M_{dr_{2}}(q)$ for any
$2+\frac{2\alpha}{\alpha-1}<r_{2}<\frac{\beta_{2}-2}{4}$; both $C_{1}$ and
$C_{2}$ are independent of $m,n$.

######  Proposition 3.

For $d=1$. Assume $(P,Q)$ are distributions on $\mathbb{R}^{d}$ of heavy-tail
$(\beta_{1},\beta_{2})$, where $\beta_{1}>7$ and $\beta_{2}>13$. Suppose
$\alpha$ satisfies $\frac{2\alpha}{\alpha-1}+4<\beta_{1}-1$ and
$\frac{6\alpha}{\alpha-1}<\beta_{2}-7$, then if $m$ and $n$ are sufficiently
large, we have

$\mathbb{E}_{X,Y}\mathinner{\\!\left\lvert
D_{\alpha,1}^{L}(P_{m}\|Q_{n})-D_{\alpha,1}^{L}(P\|Q)\right\rvert}\leq\frac{C_{1}}{m^{1/2}}+\frac{C_{2}}{n^{1/2}},$
(23)

where $C_{1}$ depends on $M_{2}(p)$ and $C_{2}$ depends on
$M_{\frac{2\alpha}{\alpha-1}+4}(p)$, $M_{\frac{2\alpha}{\alpha-1}+4}(q)$ and
$M_{dr_{2}}(q)$ for any
$2+\frac{2\alpha}{\alpha-1}<r_{2}<\frac{\beta_{2}-1}{3}$; both $C_{1}$ and
$C_{2}$ are independent of $m,n$.

###### Proof.

The only difference from the proof of Theorem 3 is that we need to bound the
random metric entropy differently since
$\sqrt{\ln\mathcal{N}(\mathcal{F}_{\alpha,M},\delta,L_{2}(Q_{n}))}$ is no
longer integrable at infinity, and the upper limit of the integral in Lemma 4
cannot be relaxed to $\infty$. Instead, we have

$\displaystyle\mathbb{E}_{X,Y}\inf_{0<\theta<M_{Y}}\left(4\theta+\frac{12}{\sqrt{n}}\int_{\theta}^{M_{Y}}\sqrt{\ln\mathcal{N}(\mathcal{F}_{\alpha,M},\delta,L_{2}(Q_{n}))}\mathop{}\\!\mathrm{d}{\delta}\right)$
$\displaystyle\leq\mathbb{E}_{X,Y}\inf_{0<\theta<M_{Y}}\left(4\theta+\frac{12}{\sqrt{n}}K(M+L)^{\frac{d\alpha}{(\alpha-1)}}\Lambda_{2}^{d/2}\int_{\theta}^{M_{Y}}\delta^{-\frac{d}{2}}\mathop{}\\!\mathrm{d}{\delta}\right),$

where
$M_{Y}=\sup_{\gamma\in\mathcal{F}_{\alpha,M}}\sqrt{\frac{1}{n}\sum_{j=1}^{n}\mathinner{\\!\left\lvert\gamma(y_{j})\right\rvert}^{2}}\leq\sqrt{\frac{1}{n}\sum_{j=1}^{n}(M+L+L\mathinner{\\!\left\lVert
y_{j}\right\rVert})^{2}}$.

For $d=2$, we have
$\int_{\theta}^{M_{Y}}\delta^{-\frac{d}{2}}\mathop{}\\!\mathrm{d}{\delta}=\ln
M_{y}-\ln\theta$, and we can pick $\theta=\frac{\ln n}{\sqrt{n}}$, and use the
inequality $\ln M_{y}\leq M_{y}-1$ and combine it with Lemma 5 and Lemma 7 as
in the proof of Theorem 3.

For $d=1$, we have
$\int_{\theta}^{M_{Y}}\delta^{-\frac{d}{2}}\mathop{}\\!\mathrm{d}{\delta}=\frac{\sqrt{M_{y}}-\sqrt{\theta}}{2}$,
and we can pick $\theta=\frac{1}{\sqrt{n}}$ to balance the two terms. ∎

##  Appendix E Compute resources

Our experiment is computed using personal computer in the environment: Apple
M2 8 cores and Apple M2 24 GB - Metal 3.

##  Appendix F Additional experiments

(a) $\alpha$-GAN (left) and its counterpart with $W_{1}$-proximal
regularization, $W_{1}$-$\alpha$-GAN (right)

(b) $\alpha$-GPA (left) and its counterpart with $W_{1}$-proximal
regularization, $W_{1}$-$\alpha$-GPA (right)

Figure 6:  Learning a 2D isotropic Student-t with degree of freedom $\nu=3$
(tail index $\beta=5.0$) using generative models based on
$W_{1}$-$\alpha$-divergences with $\alpha=2$. Models with $W_{1}$-proximal
regularizations (right) learn the heavy-tailed distribution significantly
better than those without (left). See Section 5.1 for detailed explanations of
the models.

(a) CNF

(b) OT flow

(c) VE-SGM

Figure 7: Learning a 2D isotropic Student-t with degree of freedom $\nu=3$
(tail index $\beta=5.0$) using generative models based on
$W_{2}$-$\alpha$-divergences with $\alpha=1$. Models with $W_{2}$-proximal
regularizations, (b) and (c), learn the heavy-tailed distribution
significantly better than that without, (a). See Section 5.1 for detailed
explanations of the models.