# Learning Latent Space Hierarchical EBM Diffusion Models

Jiali Cui  Tian Han

###### Abstract

This work studies the learning problem of the energy-based prior model and the
multi-layer generator model. The multi-layer generator model, which contains
multiple layers of latent variables organized in a top-down hierarchical
structure, typically assumes the Gaussian prior model. Such a prior model can
be limited in modelling expressivity, which results in a gap between the
generator posterior and the prior model, known as the prior hole problem.
Recent works have explored learning the energy-based (EBM) prior model as a
second-stage, complementary model to bridge the gap. However, the EBM defined
on a multi-layer latent space can be highly multi-modal, which makes sampling
from such marginal EBM prior challenging in practice, resulting in
ineffectively learned EBM. To tackle the challenge, we propose to leverage the
diffusion probabilistic scheme to mitigate the burden of EBM sampling and thus
facilitate EBM learning. Our extensive experiments demonstrate a superior
performance of our diffusion-learned EBM prior on various challenging tasks.

Machine Learning, ICML

##  1 Introduction

The hierarchical generative model with multiple layers of latent variables
(a.k.a., multi-layer generator model) has made significant progress in
learning complex data distribution (Sø nderby et al., 2016; Vahdat & Kautz,
2020) and has garnered particular interest for its top-down hierarchical
structure, where multiple layers of latent variables that are organized from
the top to the bottom layers tend to capture levels of (hierarchical) data
representations, with high-level semantic representations captured by the
latent variables at the top layers and low-level detail representations by
those at the bottom layers (Maaløe et al., 2019; Child, 2020). Learning such
hierarchical representation can be essential and crucial to various downstream
applications (Havtorn et al., 2021; Nijkamp et al., 2020b). However, such
multi-layer generator models typically assume the Gaussian prior model, which
can be limited in statistical expressivity by primarily focusing on the inter-
layer relation between layers of latent variables while largely ignoring the
intra-layer relation between latent units within each layer (Cui et al.,
2023a, b). This may result in the prior hole problem (Rosca et al., 2018;
Hoffman & Johnson, 2016; Takahashi et al., 2019; Bauer & Mnih, 2019) where the
non-expressive Gaussian prior fails to match the aggregated generator
posterior.

Recent studies (Aneja et al., 2021; Cui et al., 2023a) have investigated the
utilization of the energy-based (EBM) prior model as a complementary model to
address this limitation. The EBM prior is typically trained with a fixed
generator model (referred to as the Two-Stage learning scheme) to tilt the
non-expressive Gaussian prior to match the generator posterior. However,
learning a single (marginal) EBM is challenging because the generator
posterior is often multi-modal, and more importantly, the Markov Chain Monte
Carlo (MCMC) sampling required to maximize the marginal EBM likelihood can be
difficult, as multiple layers of latent variables are interwoven and require
exploration at different latent scales. In addition, MCMC sampling, such as
Langevin dynamics, usually starts from a noise-initialized point, which is
hard to explore the energy landscape and mix between different local modes.
Therefore, for multi-layer latent variables, EBM prior sampling may serve as
the bottleneck for effective EBM learning, which still poses a challenge.

Inspired by recent diffusion probabilistic frameworks (Ho et al., 2020; Gao et
al., 2020; Zhu et al., 2023; Du et al., 2023), we propose learning the EBM
prior of multi-layer latent variables in a diffusion learning scheme. We
construct a series of conditional EBMs prior to gradually matching the highly
multi-modal generator posterior, with each conditional EBM prior only matching
perturbed samples at each step. Compared to marginal EBM prior, such a
conditional EBM prior can be less multi-modal, leading to more tractable
conditional likelihood learning. For EBM sampling, the proposed conditional
EBM prior can render a smoother energy landscape, which mitigates the burden
of MCMC sampling and thus further facilitates effective EBM learning. However,
for multi-layer latent variables, MCMC sampling needs to account for their
different latent scales at different layers (i.e., the scales of latent
variables at the top and bottom layers can be very different); moreover,
directly perturbating the latent samples may destroy their inter-layer
relation (i.e., conditional dependency formulated in the hierarchical
structure). Therefore, we further employ a uni-scale
${\mathbf{\tilde{u}}}$-space (see definition in Sec. 3.1) converted from the
multi-scale latent space, which allows us to preserve the hierarchical
dependency along the forward process while at the same time, further reducing
the burden of MCMC sampling by traversing a uni-scale latent space. Our
experiments demonstrate the effectiveness of the proposed method in various
challenging tasks and show that our model is capable of generating high-
quality samples and capturing hierarchical representations at different
layers.

Contribution: 1) We develop a learning framework that incorporates the
diffusion probabilistic scheme for learning the joint EBM prior for the multi-
layer generator model; 2) To preserve hierarchical structures and enable more
effective EBM sampling, we adopt a uni-scale space to further mitigate the
burden of MCMC sampling; 3) We conduct various experiments to examine our
model in generating high-quality samples and learning effective hierarchical
representations.

##  2 Preliminary

###  2.1 Multi-layer Latent Variable Model

Let ${\mathbf{x}}\in R^{D}$ be the high-dimensional observed example and
${\mathbf{z}}\in R^{d}$ be the low-dimensional latent variable. The multi-
layer generator model contains multiple latent variables (i.e.,
${\mathbf{z}}_{1},{\mathbf{z}}_{2},\dots,{\mathbf{z}}_{L}$) organized in a
top-down hierarchical structure and can be specified as a joint distribution.
We denote
${\mathbf{\tilde{z}}}=({\mathbf{z}}_{1},{\mathbf{z}}_{2},\dots,{\mathbf{z}}_{L})$,
then

$\displaystyle p_{\beta}({\mathbf{x}},{\mathbf{\tilde{z}}})=$ $\displaystyle
p_{\beta_{0}}({\mathbf{x}}|{\mathbf{\tilde{z}}})p_{\beta_{>0}}p({\mathbf{\tilde{z}}})\;\;\;\;\text{where}$
(1) $\displaystyle p_{\beta_{>0}}({\mathbf{\tilde{z}}})=$
$\displaystyle\prod_{i=1}^{L-1}p_{\beta_{i}}({\mathbf{z}}_{i}|{\mathbf{z}}_{i+1})p({\mathbf{z}}_{L})$

in which $p_{\beta_{0}}({\mathbf{x}}|{\mathbf{\tilde{z}}})$ is the generation
model that maps from the latent space to the data space, and
$p_{\beta_{>0}}({\mathbf{\tilde{z}}})$ is the prior model that factories
consecutive layers of latent variables with conditional Gaussian distribution
(i.e.,
$p_{\beta_{i}}({\mathbf{z}}_{i}|{\mathbf{z}}_{i+1})\sim{\mathcal{N}}(\mu_{\beta_{i}}({\mathbf{z}}_{i+1}),\sigma^{2}_{\beta_{i}}({\mathbf{z}}_{i+1}))$)
parameterized by learnable parameter $\beta_{i}$. The
$p({\mathbf{z}}_{L})\sim{\mathcal{N}}(0,{\mathbf{I}}_{d})$ is assumed to be
unit Gaussian at the top layer.

Learning such a hierarchical generative model can be achieved using maximum
likelihood estimation (MLE) with the gradient estimated as
$\nabla_{\beta}{\mathbb{E}}_{p_{\beta}({\mathbf{\tilde{z}}}|{\mathbf{x}})}[\log
p_{\beta}({\mathbf{x}},{\mathbf{\tilde{z}}})]$. For the generator posterior
$p_{\beta}({\mathbf{\tilde{z}}}|{\mathbf{x}})$, prior works utilize MCMC
sampling to obtain approximated posterior samples (Nijkamp et al., 2020b), and
(Sø nderby et al., 2016; Child, 2020; Maaløe et al., 2019) propose the
variational learning that introduces a parameterized inference network (e.g.,
$q_{\phi}({\mathbf{\tilde{z}}}|{\mathbf{x}})$) learned to approximate the
generator posterior distribution.

However, such multi-layer generator models often fall short in generating
high-quality image synthesis, as the Gaussian prior typically only focuses on
the inter-layer relation modelling while largely ignoring the intra-layer
relation modelling (Cui et al., 2023a), resulting in the prior hole problem
with mismatch regions between the prior and aggregate posterior distribution
(Dai & Wipf, 2019; Ghosh et al., 2019).

###  2.2 Energy-based Prior Model.

Another generative model, the energy-based model (EBM), is shown to be
expressive in capturing the intra-layer relation and representing data
uncertainty. In general, on data space ${\mathbf{x}}$, the EBM can be defined
as

$p_{\omega}({\mathbf{x}})=\frac{1}{\mathrm{Z}_{\omega}}\exp{[f_{\omega}({\mathbf{x}})]}$
(2)

where $\mathrm{Z}_{\omega}$ is the normalizing constant or partition function,
$f_{\omega}({\mathbf{x}})$ is the energy function parameterized with $\omega$.

Learning the EBM via MLE estimates the gradient as ${\mathbb{E}}_{p_{\rm
data}({\mathbf{x}})}[\nabla_{\omega}f_{\omega}({\mathbf{x}})-{\mathbb{E}}_{p_{\omega}({\mathbf{x}})}[f_{\omega}({\mathbf{x}})]]$.
For EBM samples from $p_{\omega}({\mathbf{x}})$, (Du & Mordatch, 2019; Du et
al., 2020; Nijkamp et al., 2019) adopt MCMC sampling such as Langevin dynamics
(LD). In particular, it is applied as

$\displaystyle{\mathbf{x}}_{\tau+1}={\mathbf{x}}_{\tau}+s\nabla_{{\mathbf{x}}_{\tau}}\log
p({\mathbf{x}}_{\tau})+\sqrt{2s}U_{\tau}$ (3)

where $\tau$ indexes the time step, $s$ is the step size and
$U_{\tau}\sim{\mathcal{N}}(0,{\mathbf{I}}_{D})$, and ${\mathbf{x}}_{\tau=0}$
is usually initialized from the Gaussian noise. However, in practice, it may
take a long time to explore the energy landscape and mix between different
local modes. To mitigate the burden of EBM sampling, recent advances have
explored EBMs on low-dimensional latent space $p_{\omega}({\mathbf{z}})$ (Pang
et al., 2020a; Xiao & Han, 2022; Yu et al., 2022), but a single-layer
$p_{\omega}({\mathbf{z}})$ prior model can still be limited in modelling
compacity of the whole model.

Two-stage complementary EBM prior. Learning the EBM prior for multi-layer of
latent variables
${\mathbf{\tilde{z}}}=({\mathbf{z}}_{1},{\mathbf{z}}_{2},\dots,{\mathbf{z}}_{L})$
can be more expressive than single-layer latent variables, but jointly
learning both the multi-layer generator model and EBM prior can be extremely
inefficient, especially with a deep hierarchical structure involved (Vahdat &
Kautz, 2020; Child, 2020). This motivates a Two-Stage learning scheme (Xiao et
al., 2020; Aneja et al., 2021; Cui et al., 2023a) that learns the Gaussian
prior generator model at the first stage (see Sec. 2.1) and then learns the
EBM, as a complementary model at the second stage with the fixed generator
backbone. In our work, we adopt such a learning scheme for its efficiency.

With multi-layer of latent variables, the NCP-VAE (Aneja et al., 2021)
factories a conditional EBM prior

$p_{\omega_{i},\beta_{i}}({\mathbf{z}}_{i}|{\mathbf{z}}_{i+1})=\frac{1}{\mathrm{Z}_{\omega_{i},\beta_{i}}({\mathbf{z}}_{i+1})}\exp{[f_{\omega_{i}}({\mathbf{z}}_{i})]}p_{\beta_{i}}({\mathbf{z}}_{i}|{\mathbf{z}}_{i+1})$

which aims to tilt the Gaussian prior toward the generator posterior
distribution. The noise-contrastive estimation (NCE) is used for learning,
which treats EBM as a classifier and thus does not need MCMC approximation.
However, the NCE scheme can render suboptimal learning with a large gap
between the two distributions (Xiao & Han, 2022), which exists between the
generator posterior and the Gaussian prior (Arjovsky et al., 2017; Dai & Wipf,
2019).

The recent work (Cui et al., 2023a) considers learning EBM prior via MLE by
jointly modelling all layers of latent variables

$p_{\omega,\beta_{>0}}({\mathbf{\tilde{z}}})=\frac{1}{\mathrm{Z}_{\omega,\beta_{>0}}}\exp\left[F_{\omega}({\mathbf{\tilde{z}}})\right]p_{\beta_{>0}}({\mathbf{\tilde{z}}})$
(4)

where the energy function
$F_{\omega}({\mathbf{\tilde{z}}})=\sum_{i=1}^{L}f_{\omega_{i}}({\mathbf{z}}_{i})$.
However, MCMC sampling for Eqn. 4 can be practically challenging as layers of
${\mathbf{z}}_{i}$ can have different scales (i.e.,
${\mathbf{z}}_{L}\sim{\mathcal{N}}(0,{\mathbf{I}}_{d})$ and
${\mathbf{z}}_{1}\sim p_{\beta_{>0}}({\mathbf{z}}_{1})=\int
p_{\beta_{>0}}({\mathbf{z}}_{1},\dots,{\mathbf{z}}_{L})d{\mathbf{z}}_{2},\dots,d{\mathbf{z}}_{L}$),
which requires special designs for MCMC sampling to account for such
variation.

Learning both the multi-layer EBM prior can be viewed to minimize the
Kullback-Leibler (KL) divergence between the generator posterior distribution
and the EBM prior, i.e.,
${\mathrm{KL}}(p_{\theta}({\mathbf{\tilde{z}}}|{\mathbf{x}})||p_{\omega,\beta_{>0}}({\mathbf{\tilde{z}}}))$,
which is difficult due to the highly multi-modal generator posterior and the
multi-scale latent space, resulting in ineffective MCMC sampling for EBM
learning.

##  3 Methodology

Inspired by recent diffusion probabilistic methods that focus on learning a
sequence of parameterized models to gradually match target data distribution,
we study a probabilistic framework that can leverage such diffusion scheme
with a sequence of conditional EBMs prior for the multi-layer generator
models.

###  3.1 Diffusion with Multi-layer Latent Variables

Attempt on ${\mathbf{\tilde{z}}}$-space. The diffusion probabilistic scheme
assumes a sequence of perturbed samples
${\mathbf{z}}_{0:T}=({\mathbf{z}}_{0},{\mathbf{z}}_{1},\dots,{\mathbf{z}}_{T})$
for each diffusion step $t=0,1,\dots,T$. In particular, the noisy sample
${\mathbf{\tilde{z}}}_{t}$ is generated by a pre-defined Gaussian perturbation
kernel as

$q({\mathbf{\tilde{z}}}_{t+1}|{\mathbf{\tilde{z}}}_{t})\sim{\mathcal{N}}(\alpha_{t+1}{\mathbf{\tilde{z}}}_{t},\sigma^{2}_{t+1}\mathbf{I}_{|d|})$
(5)

where $\alpha_{t+1}$ is typically set to be $\sqrt{1-\sigma^{2}_{t+1}}$ to
ensure a spherical interpolation between samples and noise.

However, directly employing Eqn. 5 does not suit for multi-layer latent
variables ${\mathbf{\tilde{z}}}$, since it does not take into account the
hierarchical structure between layers of latent variables. Their inter-layer
relation is consequently destroyed during the progress, i.e., each
${\mathbf{z}}_{i}$ becomes independently distributed as standard Gaussian
noise at the final diffusion step. Our goal is to reach the Gaussian prior
model $p_{\beta_{>0}}({\mathbf{\tilde{z}}})$ (Eqn. 1) at the final step, such
that the reverse process can start from the Gaussian prior model to
approximate the generator posterior distribution gradually.

Toward ${\mathbf{\tilde{u}}}$-space. Instead of latent space, we formulate our
diffusion model on ${\mathbf{\tilde{u}}}$-space. In particular, for multi-
layer generator models, the Gaussian prior model
$p_{\beta_{>0}}({\mathbf{\tilde{z}}})$ is factorized to be the multiplication
of consecutive layers of conditional Gaussian distribution
$p_{\beta_{i}}({\mathbf{z}}_{i}|{\mathbf{z}}_{i+1})\sim{\mathcal{N}}(\mu_{\beta_{i}}({\mathbf{z}}_{i+1}),\sigma^{2}_{\beta_{i}}({\mathbf{z}}_{i+1}))$,
which features the re-parametrization sampling, i.e.,
${\mathbf{z}}_{i}=\mu_{\beta_{i}}({\mathbf{z}}_{i+1})+\sigma_{\beta_{i}}({\mathbf{z}}_{i+1})\cdot{\mathbf{u}}_{i+1}$
where ${\mathbf{u}}_{i+1}\sim{\mathcal{N}}(0,{\mathbf{I}}_{d})$. We denote
such an invertible deterministic transformation function to be
$T_{\beta_{>0}}$, i.e.,
${\mathbf{\tilde{z}}}=T_{\beta_{>0}}({\mathbf{\tilde{u}}})$ and
${\mathbf{\tilde{u}}}=T_{\beta_{>0}}^{-1}({\mathbf{\tilde{z}}})$ (see App.
A.1), which allows us to adapt our diffusion model on
${\mathbf{\tilde{u}}}$-space. The ${\mathbf{\tilde{u}}}$-space enables the
hierarchical structure to be maintained during the forward progress. When
$q({\mathbf{\tilde{u}}}_{T})\sim{\mathcal{N}}(0,{\mathbf{I}}_{|d|})$ becomes
standard Gaussian noise after the forward diffusion process, the corresponding
latent variables reach the desired Gaussian prior model at the final step.

Forward on ${\mathbf{\tilde{u}}}$-space. The perturbation kernel is defined as

$q({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})\sim{\mathcal{N}}(\alpha_{t+1}{\mathbf{\tilde{u}}}_{t},\sigma^{2}_{t+1}\mathbf{I}_{d})$
(6)

With a designed diffusion $\sigma$-schedule (e.g., signal-to-noise ratio,
SNR), ${\mathbf{\tilde{u}}}_{T}$ at final step $t=T$ becomes Gaussian noise
and $q({\mathbf{\tilde{u}}}_{T})={\mathcal{N}}(0,{\mathbf{I}}_{|d|})$ is then
the stationary distribution.

For the case of $t=0$, we obtain ${\mathbf{\tilde{u}}}_{0}$ via transformation
function
${\mathbf{\tilde{u}}}_{0}=T_{\beta_{>0}}^{-1}({\mathbf{\tilde{z}}}_{0})$ with
${\mathbf{\tilde{z}}}_{0}$ sampled from the generator posterior. For
variational-based generator models (see Sec. 2.1), ${\mathbf{\tilde{z}}}_{0}$
can be inferred from the inference model, while we can also perform MCMC
posterior sampling for ${\mathbf{\tilde{z}}}_{0}$. For deep hierarchical
structures, the inference model is preferred for efficiency. The forward
trajectory then becomes

$q_{\beta}({\mathbf{\tilde{u}}}_{0:T}|{\mathbf{x}})=\prod_{t=0}^{T-1}q({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})q_{\beta}({\mathbf{\tilde{u}}}_{0}|{\mathbf{x}})$
(7)

where $q_{\beta}({\mathbf{\tilde{u}}}_{0}|{\mathbf{x}})$ is the unknown target
distribution. The goal is now to reverse from ${\mathbf{\tilde{u}}}_{T}$ to
${\mathbf{\tilde{u}}}_{0}$, which, in view of distributions, reverses from the
Gaussian prior model $p_{\beta_{>0}}({\mathbf{\tilde{z}}})$ to the generator
posterior $p_{\beta}({\mathbf{\tilde{z}}}|{\mathbf{x}})$. This strategy
circumvents the problem of destroying hierarchical patterns during the forward
process and thus satisfies our goal to bridge the gap between Gaussian prior
model and the generator posterior.

###  3.2 Reverse with Multi-layer Latent Variables

${\mathbf{\tilde{u}}}$-space for marginal EBM prior. The uni-scale
${\mathbf{\tilde{u}}}$-space is adopted in works (Xiao et al., 2020; Cui et
al., 2023a) for EBM sampling. Specifically, sampling from the marginal EBM
(Eqn. 4) is equivalent to sampling from

$\displaystyle
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}})=\frac{1}{\mathrm{Z}_{\omega,\beta_{>0}}}\exp\left[F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}))\right]p_{0}({\mathbf{\tilde{u}}})$
(8)

where $p_{0}({\mathbf{\tilde{u}}})$ is standard Gaussian. The uni-scale
${\mathbf{\tilde{u}}}$-space can make EBM sampling easier than the multi-scale
latent space ${\mathbf{\tilde{z}}}$. However, this is still challenging as it
aims to match the highly multi-modal generator posterior and Gaussian prior
with a single (marginal) EBM.

To tackle this challenge, prior works (Gao et al., 2020; Yu et al., 2022)
leverage diffusion scheme and learn sequential conditional EBMs, which has
seen some success in modelling the high-dimensional ${\mathbf{x}}$-space and
single-layer ${\mathbf{z}}$-space. Inspired by their work, we propose to learn
a sequence of conditional EBMs prior but focus on hierarchical generative
models with ${\mathbf{\tilde{u}}}$-space to further alleviate the burden of
EBM sampling and learning.

${\mathbf{\tilde{u}}}$-space for conditional EBM prior. For our diffusion
model, we formulate the marginal EBM prior (Eqn. 8) to a sequence of
conditional EBMs prior, where each marginal EBM prior at each diffusion step
(i.e., $p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t})$) is constrained by
the forward generated ${\mathbf{\tilde{u}}}_{t+1}$, i.e.,

$\displaystyle
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})\propto
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t})p({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})=$
(9)
$\frac{1}{\mathrm{Z}_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t+1})}\exp\left[F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)\right]p_{0}({\mathbf{\tilde{u}}}_{t})\cdot
p({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})$

where we slightly abuse the notation and use
$p({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})$ for the perturbation
kernel as in Eqn. 6. The energy function essentially couples all layers of
latent variables (i.e.,
$F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)=\sum_{i=1}^{L}f_{\omega_{i}}({\mathbf{z}}_{i},t)$),
and thus the inter-layer and intra-layer relation of each ${\mathbf{z}}_{i}$
can be effectively modelled. Each $f_{\omega_{i}}({\mathbf{z}}_{i},t)$
corresponding to ${\mathbf{z}}_{i}$ at each layer can also capture the
representation of different layers.

Compared to Eqn. 8 that directly models the complex ${\mathbf{\tilde{u}}}_{0}$
(generator posterior) with a single marginal EBM, the conditional EBM only
models ${\mathbf{\tilde{u}}}_{t}$, reversing step by step until
${\mathbf{\tilde{u}}}_{0}$.
$p({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})$ from Gaussian noise
perturbation kernel can serve to localize ${\mathbf{\tilde{u}}}_{t}$ to
${\mathbf{\tilde{u}}}_{t+1}$, making our conditional EBM less multi-modal and
easier to be sampled than the marginal EBM. The reverse trajectory constitutes

$\displaystyle
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T})=\prod_{t=0}^{T-1}p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})p({\mathbf{\tilde{u}}}_{T})$
(10)

where $p({\mathbf{\tilde{u}}}_{T})\sim N(0,{\mathbf{I}}_{|d|})$ is standard
Gaussian as $q({\mathbf{\tilde{u}}}_{T})$ in forward trajectory.

EBM learning. The proposed method now contains a sequence of parameterized
EBMs prior. With forward trajectory Eqn. 7, we minimize
${\mathrm{KL}}(q_{\beta}({\mathbf{\tilde{u}}}_{0:T}|{\mathbf{x}})||p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T}))$
for EBM learning. The gradient is estimated as

$\displaystyle\nabla_{\omega}{\mathbb{E}}_{q_{\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T}|{\mathbf{x}})}[\sum_{t=0}^{T-1}F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)-$
(11)
$\displaystyle{\mathbb{E}}_{p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T})}[\sum_{t=0}^{T-1}F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)]]$

which involves sampling from the whole forward and reverse trajectories. To
provide more efficient sampling and learning, we follow the strategy used in
(Ho et al., 2020) and utilize random diffusion steps $t$ at each iteration of
optimization. Thus, the learning gradient is

$\displaystyle{\mathbb{E}}_{q_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t},{\mathbf{\tilde{u}}}_{t+1}|{\mathbf{x}})}[\nabla_{\omega}F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)-$
(12)
$\displaystyle{\mathbb{E}}_{p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})}[\nabla_{\omega}F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)]]$

where we only need perturbed sample
${\mathbf{\tilde{u}}}_{t},{\mathbf{\tilde{u}}}_{t+1}$ from forward trajectory
(see Eqn. 7) and prior samples ${\mathbf{\tilde{u}}}_{t}$ from the conditional
EBM
$p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})$.
To obtain prior samples, we perform Langevin dynamics (Eqn. 3) with the
gradient

$\displaystyle\nabla_{{\mathbf{\tilde{u}}}_{t}}\log
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})=$
(13)
$\displaystyle\nabla_{{\mathbf{\tilde{u}}}_{t}}[F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)-\frac{||{\mathbf{\tilde{u}}}_{t}||^{2}}{2}-\frac{||\alpha_{t+1}{\mathbf{\tilde{u}}}_{t}-{\mathbf{\tilde{u}}}_{t+1}||^{2}}{2\sigma^{2}_{t+1}}]$

Compared to MCMC sampling of marginal EBM (see Eqn. 8) that can be hard to mix
between different local modes with noise initial points, MCMC sampling of
conditional EBM can start from the given ${\mathbf{\tilde{u}}}_{t+1}$ and only
needs to search for the local modes around ${\mathbf{\tilde{u}}}_{t+1}$.
Specifically, the quadratic term is from the noise-aware term in Eqn. 9 and
constrains the exploration of energy landscape to be localized around
${\mathbf{\tilde{u}}}_{t+1}$, which is much easier to obtain EBM samples than
marginal EBMs. This conditional sampling can be more effective and efficient,
which in turn benefits the learning of the proposed EBM prior.

The overall training algorithm and sampling process are shown in Alg. 1 and
Alg. 2.

###  3.3 Coupling with symbol vector

In this section, we present the applicability of the proposed EBM prior. For
multi-layer generator models, they are typically learned in an unsupervised
scheme and thus are only feasible to generate random samples. However, the
controllable and compositional ability to generate desired synthesis nowadays
becomes a key requirement for many downstream tasks, yet it would be
computationally expensive to re-train these models to fit the job.

The proposed method is flexible in coupling labels or attributes, which in
turn empowers controllability and composition ability for the learned multi-
layer generator models. Specifically, we can adapt our model with given labels
${\mathbf{y}}$ as

$\displaystyle
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t},{\mathbf{y}}|{\mathbf{\tilde{u}}}_{t+1})=\frac{1}{\mathrm{Z}_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t+1})}\cdot$
(14) $\displaystyle\exp\left[\langle
F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t),{\mathbf{y}}\rangle\right]p_{0}({\mathbf{\tilde{u}}}_{t})p({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})$

where the energy function couples both ${\mathbf{y}}$ and
${\mathbf{\tilde{u}}}_{t}$, forming an associative memory that allows sampling
${\mathbf{\tilde{u}}}_{t}$ with a given ${\mathbf{y}}$. Generating images by
such sampled ${\mathbf{\tilde{u}}}_{t}$ is known as the controllable
generation. (Pang et al., 2020b; Yu et al., 2022) adopt similar formulations
but only focus on single-layer latent space, while we focus on a multi-layer
latent space with a hierarchical structure, allowing coupling ${\mathbf{y}}$
at different layers for specific tasks. We formulate such EBM prior at
diffusion step $t=0$ as the signal of ${\mathbf{y}}$ correlates strongly with
clean ${\mathbf{\tilde{u}}}_{0}$ samples, while noisy
${\mathbf{\tilde{u}}}_{t>0}$ can be less correlated with ${\mathbf{y}}$.

The proposed EBM prior is capable of capturing inter-layer and intra-layer
relations for different layers of latent variables, thus rendering better
controllability and compositionality with effectively learned latent
representations. We refer to learning derivation and details in App. A.2.

0: Training images ${\mathbf{x}}$; Number of learning iterations $M$;
Hierarchical generator model $\beta$; Diffusion steps $T$; Langevin steps
$k$;.

1: Let $m\leftarrow 0$, initialize EBM parameters $\omega$.

2: repeat

3: Sample ${\mathbf{\tilde{u}}}_{0}$: obtain ${\mathbf{\tilde{z}}}\sim
p_{\beta}({\mathbf{\tilde{z}}}|{\mathbf{x}})$ and
${\mathbf{\tilde{u}}}_{0}=T_{\beta_{>0}}({\mathbf{\tilde{z}}})$

4: Diffusion step $t$: $t\sim U(0,T-1)$

5: Noise sample: sample ${\mathbf{\tilde{u}}}_{t}^{*}$,
${\mathbf{\tilde{u}}}_{t+1}^{*}$ from Eqn. 7.

6: Prior sample: sample ${\mathbf{\tilde{u}}}_{t}^{k}$ by Eqn. 3 and Eqn. 13
with ${\mathbf{\tilde{u}}}_{t+1}^{*}$ being the initialization

7: Learn $\omega$: Update $\omega$ with ${\mathbf{\tilde{u}}}_{t}^{*}$ and
${\mathbf{\tilde{u}}}_{t}^{k}$ using Eqn. 12

8: Let $m\leftarrow m+1$.

9: until $m=M$

Algorithm 1 Learning EBM parameter $\omega$

0: Diffusion steps $T$; Hierarchical generator model $\beta$; EBM prior
$\omega$.

1: Let $t\leftarrow T-1$ and ${\mathbf{\tilde{u}}}_{T}\sim
N(0,{\mathbf{I}}_{|d|})$.

2: repeat

3: Sample ${\mathbf{\tilde{u}}}_{t}$: sample ${\mathbf{\tilde{u}}}_{t}\sim
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})$
using Eqn. 3 and Eqn. 13

4: Let $t\leftarrow t-1$.

5: until $t=-1$

6: Sample ${\mathbf{\tilde{z}}}$: obtain
${\mathbf{\tilde{z}}}=T^{-1}_{\beta_{>0}}({\mathbf{\tilde{u}}}_{0})$.

7: Generate ${\mathbf{x}}$: generate ${\mathbf{x}}\sim
p_{\beta_{0}}({\mathbf{x}}|{\mathbf{\tilde{z}}})$ with obtained
${\mathbf{\tilde{z}}}$.

Algorithm 2 Sampling and Image Synthesis

##  4 Related Work

Energy-based model. The EBM is expressive in representing the data
uncertainty. Most existing works learn EBM on data space by maximizing EBM
likelihood, which involves challenging MCMC sampling for the EBM. To tackle
the challenge, (Du & Mordatch, 2019; Du et al., 2020) propose the use of a
replay buffer, while (Cui & Han, 2023; Han et al., 2020b; Xie et al., 2021,
2018) consider learning a complementary generator model to jump-start MCMC
sampling. (Xiao & Han, 2022; Pang et al., 2020a, b) focus on learning EBM
prior on low-dimensional latent space to mitigate the burden of EBM sampling,
but these works only deal with a single-layer latent space. We consider
learning EBM prior for multi-layer latent variables, which enables
hierarchical representation learning and better sample generation.

Hierarchical generative model. For hierarchical generative models (Vahdat &
Kautz, 2020; Maaløe et al., 2019; Sø nderby et al., 2016), they typically
assume a Gaussian prior model, which can be less informative, resulting in the
prior hole problem and poor generation quality. Our work aims to learn
effective EBM prior for hierarchical generative models, which is related to
joint EBM prior (see Sec. 2.2). These works intend to learn a single
(marginal) EBM on ${\mathbf{\tilde{z}}}$-space, while we study learning a
sequence of conditional EBMs on ${\mathbf{\tilde{u}}}$-space. NCP-VAE (Aneja
et al., 2021) discuss a autoregressive-style model, which can be intractable
for MLE learning, as the normalization constant includes the top layer
${\mathbf{z}}_{i+1}$ and needs an additional inner loop for sampling. Our
model leverages diffusion probabilistic models such that we can directly draw
${\mathbf{\tilde{u}}}_{t+1}$ through the forward trajectory. Compared to (Cui
et al., 2023a) (Eqn. 4), our EBM prior only models ${\mathbf{\tilde{u}}}_{t}$
conditioned on the perturbed sample ${\mathbf{\tilde{u}}}_{t+1}$ generated by
${\mathbf{\tilde{u}}}_{t+1}=\alpha_{t+1}{\mathbf{\tilde{u}}}_{t}+\epsilon_{t+1}\sigma_{t+1}$,
which can be much easier than directly recovering from
${\mathbf{\tilde{u}}}_{T}$ to ${\mathbf{\tilde{u}}}_{0}$ (if consider
equivalent form Eqn. 8).

EBM diffusion model. Other diffusion EBMs (Gao et al., 2020; Yu et al., 2022,
2023) motivate EBM learning with a diffusion scheme. They build energy-based
recovery model on data space and single-layer latent space, respectively. In
this work, we focus on multi-layer generator model with a top-down
hierarchical structure, which is shown to be capable of learning meaningful
hierarchical representations (Child, 2020; Zhao et al., 2017; Havtorn et al.,
2021). It is typically challenging for the diffusion scheme to maintain such
hierarchical structures as the goal of the forward process is to destroy the
data pattern, which includes the inter-layer relation (conditional dependency)
among multi-layer latent variables. To preserve the hierarchical structure of
latent variables, we conduct the forward and reverse processes on
${\mathbf{\tilde{u}}}$-space, which also leads to more effective learning and
sampling than multi-scale ${\mathbf{x}}$-space (Gao et al., 2020) and
${\mathbf{z}}$-space (Yu et al., 2022).

Figure 1: Image synthesis on CelebA-HQ-256 (left), LSUN-Church-64 (center) and
CIFAR-10 (right).

Table 1: IS$(\uparrow)$ and FID$(\downarrow)$ on CIFAR-10. Model∗ indicates
our backbone model.

Method | IS | FID
---|---|---
Ours ($T=3$) | 9.03 | 8.93
Joint-EBM (Cui et al., 2023a) | 8.99 | 11.34
DRL-EBM ($T=6$) (Gao et al., 2020) | 8.40 | 9.58
NCP-VAE (Aneja et al., 2021) | - | 24.08
Hierarchical Generative Models w Gaussian Prior |  |
NVAE∗ (Vahdat & Kautz, 2020) | 5.30 | 37.73
NVAE∗-Recon | - | 0.68
HVAE (Sø nderby et al., 2016) | - | 81.44
BIVA (Maaløe et al., 2019) | - | 66.37
Energy-based Models |  |
Architectural-EBM (Cui et al., 2023b) | - | 63.42
Dual-MCMC (Cui & Han, 2023) | 8.55 | 9.26
Adaptive-CE (Xiao & Han, 2022) | - | 65.01
VAEBM (Xiao et al., 2020) | 8.43 | 12.19
Hat EBM (Hill et al., 2022a) | - | 19.15
ImprovedCD (Du et al., 2020) | 7.85 | 25.1
Divergence Triangle (Han et al., 2020a) | - | 30.10
Adv-EBM (Yin et al., 2020) | 9.10 | 13.21
GANs+Score+Diffusion Models |  |
StyleGANv2 w/o ADA (Karras et al., 2020) | 8.99 | 9.9
Diffusion-Amortized (Yu et al., 2023) | - | 57.72
NCSN (Song & Ermon, 2019) | 8.87 | 25.32
LSGM (Vahdat et al., 2021) | - | 2.10
DDPM ($T=1000$) (Ho et al., 2020) | 9.46 | 3.17

Table 2: FID on CelebA-HQ-256 and LSUN-Church-64.

Method | CelebA-HQ-256 | LSUN-Church-64
---|---|---
Ours ($T=3$) | 8.78 | 7.34
Joint-EBM (Cui et al., 2023a) | 9.89 | 8.38
DRL-EBM ($T=6$) (Gao et al., 2020) | - | 7.04
NCP-VAE (Aneja et al., 2021) | 24.79 | -
NVAE∗ (Vahdat & Kautz, 2020) | 30.25 | 38.13
NVAE∗-Recon | 1.64 | 2.45
Adv-EBM (Yin et al., 2020) | 17.31 | 10.84
GLOW (Kingma & Dhariwal, 2018) | 68.93 | 59.35
PGGAN (Karras et al., 2017) | 8.03 | 6.42

##  5 Experiment

###  5.1 Image Synthesis

First, we examine the sample quality of our model. The proposed model is
learned as a reverse approximation model in a diffusion probabilistic scheme,
which allows sampling reverse steps ${\mathbf{\tilde{u}}}_{t}$ to reach
${\mathbf{\tilde{u}}}_{0}$. With sampled ${\mathbf{\tilde{u}}}_{0}$, we obtain
${\mathbf{\tilde{z}}}_{0}$ through the transformation function and then
generate images ${\mathbf{x}}$ through the generator model (see Alg. 2). ††Our
project page is available at https://jcui1224.github.io/diffusion-
hierarchical-ebm-proj/.

We assess our model on the standard benchmark CIFAR-10 and the challenging
high-resolution CelebA-HQ-256 and large-scale LSUN-Church-64. We compare with
our direct baseline model Joint-EBM (Cui et al., 2023a), NCP-VAE (Aneja et
al., 2021) that learn signal (marginal) EBM prior for hierarchical generative
models, and Diffusion Recovery (DRL) EBM (Gao et al., 2020) which learn EBM
with diffusion probabilistic scheme on data space, as well as hierarchical
generative model with the Gaussian prior and other powerful advanced
generative models.

We recruit Fr$\acute{e}$Chet Inception Distance (FID) and Inception Score (IS)
metrics to evaluate the quality of image synthesis. We report our results in
Tab. 1 and Tab. 2 as well as the FID score of the reconstructed images. It can
be observed that our hierarchical EBM prior shows superior performance
compared to our baseline models and can even be competitive with those
powerful GANs and diffusion-based methods. For a fair comparison, we adopt the
NVAE model (Vahdat & Kautz, 2020) as the backbone generator model as our
direct baseline Joint-EBM and NCP-VAE. More quantitative and qualitative
results can be found in the ablation studies and App. B.

###  5.2 Hierarchical Representation

Different from other generator models, the hierarchical generator model has an
appealing structure, in which the latent variables at the top layers tend to
learn high-level semantic representations, while low-level details
representation can be learned by latent variables at lower layers. We examine
our model in learning such hierarchical representations.

Hierarchical sampling. First, we demonstrate our model by performing
hierarchical sampling, which generates variations of image synthesis that can
visualize the different levels of data representation learned by different
layers of latent variables. In particular, for multi-layer latent variables,
we only generate random samples for some layers of latent variables while
fixing other layers; hence, the corresponding variation of features is
captured by the latent variables randomly sampled. We show visualization
results on CelebA-HQ-256 in Fig. 2. It can be observed that by randomly
sampling for the top layers of latent variables, the general structure (e.g.,
genders and face identities) would be changed, while for lower layers of
latent variables, low-level features (e.g., hair color, skin color) can vary
correspondingly. We note that this is a challenging task where some minority
of features are entangled across layers, but the majority of features can be
successfully captured. This showcases the capability of our model in learning
hierarchical representations.

Figure 2: Hierarchical sampling. Visualization of representations learned by
latent variables from the top to bottom layers, arranged as top-left, top-
right, bottom-left and bottom-right.

Out-of-distribution detection. Then, we evaluate our model in out-of-
distribution (OOD) detection task to further demonstrate learned hierarchical
representations. Typically, EBMs can be applied to the OOD task by computing
the energy score as the decision function (Hill et al., 2022b; Cui & Han,
2023). For latent space EBMs, the inferred latent sample (i.e.,
${\mathbf{\tilde{u}}}_{0}\sim
q_{\beta}({\mathbf{\tilde{u}}}_{0}|{\mathbf{x}})$ for our case) from in-
distribution (ID) data is usually assigned with a lower energy than from the
OOD data. In this work, we compute the energy score of inferred latent samples
at the top layers as the decision function for using high-level semantic
representations learned at the top layers of latent variables to distinguish
the OOD and the ID data (Havtorn et al., 2021).

To better leverage the diffusion probabilistic scheme, we propose conducting
EBM sampling based on perturbed inference samples at the top layers, together
with prior samples at the bottom layers. Specifically, with testing images
${\mathbf{x}}$ and layer index $k$, we first obtain
${\mathbf{\tilde{u}}}_{1}^{>k}\sim
q_{\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T}|{\mathbf{x}})$ (see Eqn. 7) and
${\mathbf{\tilde{u}}}_{1}^{\leq k}\sim
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T})$ (see Eqn. 10) where we
choose diffusion step $t=1$ to ensure that only minor noise is added. Then, we
perform reverse sampling conditioned on ${\mathbf{\tilde{u}}}_{1}^{>k}$ and
${\mathbf{\tilde{u}}}_{1}^{\leq k}$ for jointly sampling final
${\mathbf{\tilde{u}}}_{0}$, i.e., ${\mathbf{\tilde{u}}}_{0}\sim
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0}|[{\mathbf{\tilde{u}}}_{1}^{>k},{\mathbf{\tilde{u}}}_{1}^{\leq
k}])$ where $[,]$ is the operation of concatenation. If
${\mathbf{\tilde{u}}}_{1}^{>k}$ is from ID data, then
${\mathbf{\tilde{u}}}_{0}^{>k}$ should render lower energy scores as both the
${\mathbf{\tilde{u}}}_{1}^{>k}$ and ${\mathbf{\tilde{u}}}_{1}^{\leq k}$ are
from similar local modes of the learned energy landscape. If
${\mathbf{\tilde{u}}}_{1}^{>k}$ is from OOD data, then
${\mathbf{\tilde{u}}}_{1}^{>k}$ and ${\mathbf{\tilde{u}}}_{1}^{\leq k}$ can be
in different modes, making the final reverse sampling difficult to traverse
the energy landscape and thus rendering higher energy score of sampled
${\mathbf{\tilde{u}}}_{0}^{>k}$. We follow the standard protocol and evaluate
by AUROC score for our EBM prior trained on CIFAR-10 with SVHN dataset being
the OOD dataset. The result is reported in Fig. 3 where the performance of our
model indeed improves as the layer of $k$ increases, which agrees with the
observation in (Havtorn et al., 2021) that our model can capture hierarchical
representations at different layers. Compared to using inferred latent samples
(inference scheme in Fig. 3), the proposed diffusion-based method (diffusion
scheme in Fig. 3) can render better performance by conducting additional MCMC
sampling of the learned EBM.

Figure 3: AUROC results for energy scores of different layers (denoted as
$L>k$ for using top layers above $k$-th layer). Top figure visualizes the
comparison between the diffusion scheme (${\mathbf{\tilde{u}}}_{0}$ sampled
from EBM) and the inference scheme (${\mathbf{\tilde{u}}}_{0}$ inferred from
inference model) in different layers. Bottom figure is the histogram of energy
scores using all layers $L>0$ and top layers $L>27$. Total number of layers is
30.

Figure 4: Fine-tuned image synthesis with multiple attributes on CelebA-64.

###  5.3 Controllable Synthesis

For hierarchical generator models that are typically learned without labels
(unsupervised learning), they can only generate random synthesis. Our
diffusion EBM prior, as a flexible complementary model, can be coupled with
labels to make hierarchical generative models more applicable in downstream
tasks, such as controllable and compositional generation. In practice, we fix
the hierarchical generative model and train our adapted model (Eqn. 14) with
given ${\mathbf{y}}$ (supervised learning), and then we assess if our model
can render controllability and compositionality with the captured hierarchical
representations. We refer to App. A.2 for details of training and sampling.

Categorical labels. First, we evaluate our model with categorical label
information, e.g., ${\mathbf{y}}$ represents labels of objects in CIFAR-10. To
better examine our model, we only consider coupling ${\mathbf{y}}$ at the top
layers by which the semantic representations are captured. With the learned
model, we conduct controllable generation by specifying target labels
${\mathbf{y}}$ and sampling corresponding ${\mathbf{\tilde{u}}}$ for
generating the images. We show the results in Fig. 5, and it can be seen that
our model can capture data representations and thus can generate synthesis
with specific categories.

Figure 5: Controllable synthesis on CIFAR-10.

Multiple attributes. Then, we showcase our model by utilizing multiple data
attributes for the challenging fine-tuned image synthesis. In particular, for
example of CelebA-64 dataset, we can have multiple attributes
${\mathbf{y}}_{1:N}$ for different levels of data features. To better suit the
hierarchical structure, we choose to couple high-level attributes, such as
gender information, at the top layers and gradually couple lower-level
attributes, such as face and hair features, at the lower layers. In Fig. 4, we
first specify gender attributes (e.g., ”Male”) and sample
${\mathbf{\tilde{u}}}$ at the corresponding top layers for generating gender-
specified images. Then, we fix these ${\mathbf{\tilde{u}}}$ of top layers and
specify lower-level attributes (e.g., ”Smile” and ”Blond Hair”) for sampling
${\mathbf{\tilde{u}}}$ at lower layers. We observe our EBM prior successfully
renders the desired compositional synthesis by gradually adding specific
features to the sampled images without changing the majority of previously
fixed features. This suggests learned hierarchical representation of our EBM
prior.

###  5.4 Langevin Transition for Energy Landscape

Figure 6: Long-run Langevin transition on CelebA-HQ-256. Visualization for
each 30 steps.

We examine the energy landscape of our learned EBM prior by visualizing the
Langevin transition. Our conditional EBM prior should render a smooth energy
landscape such that Langevin dynamics can effectively explore with a smooth
Langevin transition. We visualize the corresponding image synthesis of the
Langevin trajectory of each diffusion step, i.e., $t=T-1,\dots,1,0$. We show
in Fig. 7 for each step of the short-run Langevin dynamics (e.g., 30 Langevin
steps) and in Fig. 6 for a challenging long-run setting (e.g., 300 Langevin
steps for each diffusion step). We observe in Fig. 7 that the quality of image
synthesis becomes better as the Langevin progresses, with large improvement at
diffusion step $t=T-1$ and minor improvement at diffusion step $t=0$; while,
in Fig. 6, we do not see an oversaturated problem of EBM learning as observed
in (Nijkamp et al., 2020a). We conduct such experiments to demonstrate a
smooth energy landscape learned for our EBMs prior.

Figure 7: Short-run Langevin transition on CelebA-HQ-256. The top rows of
figures show the transition for the diffusion step at $t=T-1$, and the bottom
rows show the transition for the diffusion step at $t=0$.

###  5.5 Ablation Studies

Diffusion step $T$. First, we train our diffusion-based EBM prior with more
diffusion steps, e.g., $T=6$. By doing so, our model should render better
performance with easier EBM sampling by matching less perturbed samples at
each step. We report the FID score and sampling time in Tab.3 where the
synthesis quality indeed improves with more diffusion steps but also requires
more sampling time. We thus report $T=3$ as our result.

Table 3: Langevin steps $K$ and diffusion steps $T$.

| $K=30$ | $K=100$ |  $K=50$, $T=3$ | $T=6$
---|---|---|---|---
FID | 9.98 | 8.13 | 8.93 | 8.13
Time (seconds) | 75.17 | 166.34 | 94.23 | 193.45

Langevin step $K$. By using more Langevin steps, we should explore the energy
landscape better and obtain more effective EBM samples for learning. The
learned EBMs can thus generate high-quality samples for image synthesis. We
show our results in Tab. 3 where using 50 steps (denoted as $K=50$) delivers
better synthesis than using 30 steps, while using 100 steps only shows a minor
improvement but costs much more training and sampling time.

Figure 8: Images synthesis on CIFAR-10 with different backbones. Top-row
images are generated with the NVAE (30 layers) backbone, and bottom-row images
are generated with the HVAE (3 layers) backbone.

Other hierarchical generator models. In addition to the NVAE backbone model,
we also train a simple backbone hierarchical VAE with 3-layer latent
variables. We visualize images generated by the Gaussian prior and our model
in Fig. 8, where our model still improves the quality of the generation in a
large way (from FID 81.44 to 35.13), suggesting the effectiveness of the
proposed method.

##  6 Limitation

In this work, the proposed method still renders inferior performance compared
to state-of-the-art models (e.g., modern diffusion probabilistic models
(Vahdat et al., 2021)). In addition, sampling from our EBM prior requires an
iterative Langevin dynamics sampler, which can be further improved or even
bypassed; we will consider it in our future studies.

##  7 Conclusion

We propose learning EBM prior for hierarchical generative model with a
diffusion probabilistic scheme, which features more tractable conditional
likelihood learning and more effective EBM sampling. We employ a uni-scale
${\mathbf{\tilde{u}}}$-space to maintain the hierarchical structure and
further mitigate the burden of MCMC sampling. Such learned EBM prior can
generate high-quality samples for image synthesis and can capture hierarchical
representations for downstream tasks.

## Impact Statement

This paper presents a generative probabilistic framework whose goal is to
advance the field of Machine Learning and may share the limitations and
negative impact as other advanced generative models. There are many potential
societal consequences of our work, none of which we feel must be specifically
highlighted here.

## References

  * Aneja et al. (2021) Aneja, J., Schwing, A., Kautz, J., and Vahdat, A.  A contrastive learning approach for training variational autoencoder priors.  _Advances in neural information processing systems_ , 34:480–493, 2021.
  * Arjovsky et al. (2017) Arjovsky, M., Chintala, S., and Bottou, L.  Wasserstein generative adversarial networks.  In _International conference on machine learning_ , pp. 214–223. PMLR, 2017.
  * Bauer & Mnih (2019) Bauer, M. and Mnih, A.  Resampled priors for variational autoencoders.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pp. 66–75. PMLR, 2019.
  * Child (2020) Child, R.  Very deep vaes generalize autoregressive models and can outperform them on images.  _arXiv preprint arXiv:2011.10650_ , 2020.
  * Cui & Han (2023) Cui, J. and Han, T.  Learning energy-based model via dual-mcmc teaching.  _Advances in Neural Information Processing Systems_ , 36:28861–28872, 2023.
  * Cui et al. (2023a) Cui, J., Wu, Y. N., and Han, T.  Learning joint latent space ebm prior model for multi-layer generator.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3603–3612, June 2023a.
  * Cui et al. (2023b) Cui, J., Wu, Y. N., and Han, T.  Learning hierarchical features with joint latent space energy-based prior.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 2218–2227, 2023b.
  * Dai & Wipf (2019) Dai, B. and Wipf, D.  Diagnosing and enhancing vae models.  _arXiv preprint arXiv:1903.05789_ , 2019.
  * Du & Mordatch (2019) Du, Y. and Mordatch, I.  Implicit generation and generalization in energy-based models.  _arXiv preprint arXiv:1903.08689_ , 2019.
  * Du et al. (2020) Du, Y., Li, S., Tenenbaum, J., and Mordatch, I.  Improved contrastive divergence training of energy based models.  _arXiv preprint arXiv:2012.01316_ , 2020.
  * Du et al. (2023) Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. S.  Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc.  In _International conference on machine learning_ , pp. 8489–8510. PMLR, 2023.
  * Gao et al. (2020) Gao, R., Song, Y., Poole, B., Wu, Y. N., and Kingma, D. P.  Learning energy-based models by diffusion recovery likelihood.  _arXiv preprint arXiv:2012.08125_ , 2020.
  * Ghosh et al. (2019) Ghosh, P., Sajjadi, M. S., Vergari, A., Black, M., and Schölkopf, B.  From variational to deterministic autoencoders.  _arXiv preprint arXiv:1903.12436_ , 2019.
  * Grathwohl et al. (2019) Grathwohl, W., Wang, K.-C., Jacobsen, J.-H., Duvenaud, D., Norouzi, M., and Swersky, K.  Your classifier is secretly an energy based model and you should treat it like one.  _arXiv preprint arXiv:1912.03263_ , 2019.
  * Han et al. (2020a) Han, T., Nijkamp, E., Zhou, L., Pang, B., Zhu, S.-C., and Wu, Y. N.  Joint training of variational auto-encoder and latent energy-based model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020a.
  * Han et al. (2020b) Han, T., Nijkamp, E., Zhou, L., Pang, B., Zhu, S.-C., and Wu, Y. N.  Joint training of variational auto-encoder and latent energy-based model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7978–7987, 2020b.
  * Havtorn et al. (2021) Havtorn, J. D. D., Frellsen, J., Hauberg, S., and Maaløe, L.  Hierarchical vaes know what they don’t know.  In _International Conference on Machine Learning_ , pp. 4117–4128. PMLR, 2021.
  * Hill et al. (2022a) Hill, M., Nijkamp, E., Mitchell, J., Pang, B., and Zhu, S.-C.  Learning probabilistic models from generator latent spaces with hat ebm.  _Advances in Neural Information Processing Systems_ , 35:928–940, 2022a.
  * Hill et al. (2022b) Hill, M., Nijkamp, E., Mitchell, J., Pang, B., and Zhu, S.-C.  Learning probabilistic models from generator latent spaces with hat ebm.  _Advances in Neural Information Processing Systems_ , 35:928–940, 2022b.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Hoffman & Johnson (2016) Hoffman, M. D. and Johnson, M. J.  Elbo surgery: yet another way to carve up the variational evidence lower bound.  In _Workshop in Advances in Approximate Bayesian Inference, NIPS_ , volume 1, 2016.
  * Karras et al. (2017) Karras, T., Aila, T., Laine, S., and Lehtinen, J.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Karras et al. (2020) Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T.  Training generative adversarial networks with limited data.  _Advances in Neural Information Processing Systems_ , 33:12104–12114, 2020.
  * Kingma & Dhariwal (2018) Kingma, D. P. and Dhariwal, P.  Glow: Generative flow with invertible 1x1 convolutions.  _Advances in neural information processing systems_ , 31, 2018.
  * Langley (2000) Langley, P.  Crafting papers on machine learning.  In Langley, P. (ed.), _Proceedings of the 17th International Conference on Machine Learning (ICML 2000)_ , pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  * Maaløe et al. (2019) Maaløe, L., Fraccaro, M., Liévin, V., and Winther, O.  Biva: A very deep hierarchy of latent variables for generative modeling.  _Advances in neural information processing systems_ , 32, 2019.
  * Nie et al. (2021) Nie, W., Vahdat, A., and Anandkumar, A.  Controllable and compositional generation with latent-space energy-based models.  _Advances in Neural Information Processing Systems_ , 34:13497–13510, 2021.
  * Nijkamp et al. (2019) Nijkamp, E., Hill, M., Zhu, S.-C., and Wu, Y. N.  Learning non-convergent non-persistent short-run mcmc toward energy-based model.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Nijkamp et al. (2020a) Nijkamp, E., Hill, M., Han, T., Zhu, S.-C., and Wu, Y. N.  On the anatomy of mcmc-based maximum likelihood learning of energy-based models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 5272–5280, 2020a.
  * Nijkamp et al. (2020b) Nijkamp, E., Pang, B., Han, T., Zhou, L., Zhu, S.-C., and Wu, Y. N.  Learning multi-layer latent variable model via variational optimization of short run mcmc for approximate inference.  In _European Conference on Computer Vision_ , pp. 361–378. Springer, 2020b.
  * Pang et al. (2020a) Pang, B., Han, T., Nijkamp, E., Zhu, S.-C., and Wu, Y. N.  Learning latent space energy-based prior model.  _Advances in Neural Information Processing Systems_ , 33:21994–22008, 2020a.
  * Pang et al. (2020b) Pang, B., Nijkamp, E., Cui, J., Han, T., and Wu, Y. N.  Semi-supervised learning by latent space energy-based model of symbol-vector coupling.  _arXiv preprint arXiv:2010.09359_ , 2020b.
  * Rosca et al. (2018) Rosca, M., Lakshminarayanan, B., and Mohamed, S.  Distribution matching in variational inference.  _arXiv preprint arXiv:1802.06847_ , 2018.
  * Sø nderby et al. (2016) Sø nderby, C. K., Raiko, T., Maalø e, L., Sø nderby, S. r. K., and Winther, O.  Ladder variational autoencoders.  In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.  URL https://proceedings.neurips.cc/paper/2016/file/6ae07dcb33ec3b7c814df797cbda0f87-Paper.pdf.
  * Song & Ermon (2019) Song, Y. and Ermon, S.  Generative modeling by estimating gradients of the data distribution.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Takahashi et al. (2019) Takahashi, H., Iwata, T., Yamanaka, Y., Yamada, M., and Yagi, S.  Variational autoencoder with implicit optimal priors.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 5066–5073, 2019.
  * Vahdat & Kautz (2020) Vahdat, A. and Kautz, J.  Nvae: A deep hierarchical variational autoencoder.  _Advances in Neural Information Processing Systems_ , 33:19667–19679, 2020.
  * Vahdat et al. (2021) Vahdat, A., Kreis, K., and Kautz, J.  Score-based generative modeling in latent space.  _Advances in Neural Information Processing Systems_ , 34:11287–11302, 2021.
  * Xiao & Han (2022) Xiao, Z. and Han, T.  Adaptive multi-stage density ratio estimation for learning latent space energy-based model.  _arXiv preprint arXiv:2209.08739_ , 2022.
  * Xiao et al. (2020) Xiao, Z., Kreis, K., Kautz, J., and Vahdat, A.  Vaebm: A symbiosis between variational autoencoders and energy-based models.  _arXiv preprint arXiv:2010.00654_ , 2020.
  * Xie et al. (2018) Xie, J., Lu, Y., Gao, R., Zhu, S.-C., and Wu, Y. N.  Cooperative training of descriptor and generator networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(1):27–45, 2018.
  * Xie et al. (2021) Xie, J., Zheng, Z., and Li, P.  Learning energy-based model with variational auto-encoder as amortized sampler.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 10441–10451, 2021.
  * Yin et al. (2020) Yin, X., Li, S., and Rohde, G. K.  Analyzing and improving generative adversarial training for generative modeling and out-of-distribution detection.  _arXiv preprint arXiv:2012.06568_ , 2020.
  * Yu et al. (2022) Yu, P., Xie, S., Ma, X., Jia, B., Pang, B., Gao, R., Zhu, Y., Zhu, S.-C., and Wu, Y.  Latent diffusion energy-based model for interpretable text modeling.  In _International Conference on Machine Learning (ICML 2022)._ , 2022.
  * Yu et al. (2023) Yu, P., Zhu, Y., Xie, S., Ma, X., Gao, R., Zhu, S.-C., and Wu, Y. N.  Learning energy-based prior model with diffusion-amortized mcmc.  _arXiv preprint arXiv:2310.03218_ , 2023.
  * Zhao et al. (2017) Zhao, S., Song, J., and Ermon, S.  Learning hierarchical features from generative models.  _arXiv preprint arXiv:1702.08396_ , 2017.
  * Zhu et al. (2023) Zhu, Y., Xie, J., Wu, Y., and Gao, R.  Learning energy-based models by cooperative diffusion recovery likelihood.  _arXiv preprint arXiv:2309.05153_ , 2023.

##  Appendix A Theorectical Derivation

###  A.1 Between ${\mathbf{\tilde{u}}}$-space and ${\mathbf{\tilde{z}}}$-space

For conditional Gaussian distribution
$p_{\beta_{i}}({\mathbf{z}}_{i}|{\mathbf{z}}_{i+1})$ (see Eqn. 1), we can have
an invertible transformation function $T_{\beta_{>0}}$. For example of 2-layer
latent variables, it is defined specifically as

$\displaystyle{\mathbf{z}}_{2}=T_{\beta_{>0}}^{{\mathbf{z}}_{2}}({\mathbf{u}}_{2})={\mathbf{u}}_{2}\;\;\text{and}\;\;{\mathbf{z}}_{1}=T_{\beta_{>0}}^{{\mathbf{z}}_{1}}({\mathbf{u}}_{1},{\mathbf{u}}_{2})=\mu_{\beta_{1}}({\mathbf{z}}_{2})+\sigma_{\beta_{1}}({\mathbf{z}}_{2})\cdot{\mathbf{u}}_{1}$
(15)

where ${\mathbf{u}}_{1}$ and ${\mathbf{u}}_{2}$ are distributed as independent
Gaussian noise, i.e., $({\mathbf{u}}_{1},{\mathbf{u}}_{2})\sim
p_{0}({\mathbf{u}}_{1},{\mathbf{u}}_{2})$ and
$p_{0}({\mathbf{u}}_{1},{\mathbf{u}}_{2})=p_{0}({\mathbf{u}}_{1})p_{0}({\mathbf{u}}_{2})$
with each $p_{0}({\mathbf{u}}_{i})\sim{\mathcal{N}}(0,I_{d})$. By change-of-
variable rule, we can have

$\displaystyle
p_{\beta_{>0}}({\mathbf{z}}_{1},{\mathbf{z}}_{2})=p_{0}({\mathbf{u}}_{1},{\mathbf{u}}_{2})|\mathrm{det}(J_{T_{\beta_{>0}}^{-1}})|\;\;\text{and}\;\;p_{0}({\mathbf{u}}_{1},{\mathbf{u}}_{2})=p_{\beta_{>0}}({\mathbf{z}}_{1},{\mathbf{z}}_{2})|\mathrm{det}(J_{T_{\beta_{>0}}})|$
(16)

where $J_{T_{\beta_{>0}}}$ is the Jacobian of $T_{\beta_{>0}}$.

For joint EBM prior on ${\mathbf{\tilde{z}}}$-space
$p_{\omega,\beta_{>0}}({\mathbf{\tilde{z}}})$ (see Eqn. 4), we can apply the
change-of-variable rule and Eqn. 16 as

$\displaystyle
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}})=p_{\omega,\beta_{>0}}({\mathbf{\tilde{z}}})|\mathrm{det}(J_{T_{\beta_{>0}}})|$
(17)
$\displaystyle=\frac{1}{\mathrm{Z}_{\omega,\beta_{>0}}}\exp[F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}))]p_{\beta_{>0}}({\mathbf{\tilde{z}}})|\mathrm{det}(J_{T_{\beta_{>0}}})|$
$\displaystyle=\frac{1}{\mathrm{Z}_{\omega,\beta_{>0}}}\exp[F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}))]p_{0}({\mathbf{\tilde{u}}})$

which is the Eqn. 8. With such marginal EBM prior on
${\mathbf{\tilde{u}}}$-space, we construct our conditional EBM prior as shown
in Eqn. 9.

###  A.2 Coupling with symbol vector

Energy-based model can be flexible to couple with symbol vector (Pang et al.,
2020b; Yu et al., 2022; Nie et al., 2021; Grathwohl et al., 2019). For Eqn.
14, we adapt our model to couple with symbol vector ${\mathbf{y}}$. The
marginal version of Eqn. 14 is given as

$\displaystyle\hat{p}_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})=\frac{1}{\mathrm{Z}_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t+1})}\exp\left[\hat{F}_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)\right]p_{0}({\mathbf{\tilde{u}}}_{t})p({\mathbf{\tilde{u}}}_{t+1}|{\mathbf{\tilde{u}}}_{t})$
(18)

where
$\hat{F}_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)=\log\sum_{\mathbf{y}}\exp(\langle
F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t),{\mathbf{y}}\rangle)$.
This forms a softmax classifier, i.e.,

$\displaystyle
p_{\omega,\beta_{>0}}({\mathbf{y}}|{\mathbf{\tilde{u}}}_{t},{\mathbf{\tilde{u}}}_{t+1})=\frac{p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t},{\mathbf{y}}|{\mathbf{\tilde{u}}}_{t+1})}{\hat{p}_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})}=\frac{\exp(\langle
F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t),{\mathbf{y}}\rangle)}{\sum_{\mathbf{y}}\exp(\langle
F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t),{\mathbf{y}}\rangle)}$
(19)

where energy function $F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)$
outputs the logit score of categories. Recall that
$F_{\omega}(T_{\beta_{>0}}({\mathbf{\tilde{u}}}_{t}),t)=\sum_{i=1}^{L}f_{\omega_{i}}({\mathbf{z}}_{i},t)$,
we therefore can couple ${\mathbf{y}}$ at different layers and let the energy
score $f_{\omega_{i}}({\mathbf{z}}_{i},t)$ at $i$-th layer serve as the
softmax classifier for ${\mathbf{y}}$.

Recall that we only couple ${\mathbf{y}}$ at $t=0$, learning such model can be
achieved by maximizing the likelihood of $\log
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T},{\mathbf{y}})$. Specifically,
we have

$\displaystyle\log
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0:T},{\mathbf{y}})$
$\displaystyle=$
$\displaystyle\underbrace{\log\hat{p}_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0}|{\mathbf{\tilde{u}}}_{1})+\log
p_{\omega,\beta_{>0}}({\mathbf{y}}|{\mathbf{\tilde{u}}}_{0},{\mathbf{\tilde{u}}}_{1})}_{\text{with
symbol vector}}$ (20) $\displaystyle+$
$\displaystyle\underbrace{\sum_{t=1}^{T-1}\log
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{t}|{\mathbf{\tilde{u}}}_{t+1})}_{\text{without
symbol vector}}$ (21)

where Eqn. 21 computes the gradient similar as Eqn. 12, while for Eqn. 20, it
is learned with an extra term computing the gradient for the softmax
classifier
$p_{\omega,\beta_{>0}}({\mathbf{y}}|{\mathbf{\tilde{u}}}_{0},{\mathbf{\tilde{u}}}_{1})$,
i.e., optimizing using standard cross-entropy.

For sampling ${\mathbf{\tilde{u}}}_{0}$ with specified ${\mathbf{y}}$, we
first obtain ${\mathbf{\tilde{u}}}_{1}$ by reversing step by step from
${\mathbf{\tilde{u}}}_{T}$ via Langevin dynamics (see Alg. 2). Then, we
perform Langevin dynamics (see Eqn. 3) to sample from
$p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0},{\mathbf{y}}|{\mathbf{\tilde{u}}}_{1})$.
The gradient is then computed as $\nabla_{{\mathbf{\tilde{u}}}_{0}}\log
p_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0},{\mathbf{y}}|{\mathbf{\tilde{u}}}_{1})=\nabla_{{\mathbf{\tilde{u}}}_{0}}\left[\log\hat{p}_{\omega,\beta_{>0}}({\mathbf{\tilde{u}}}_{0}|{\mathbf{\tilde{u}}}_{1})+\log\hat{p}_{\omega,\beta_{>0}}({\mathbf{y}}|{\mathbf{\tilde{u}}}_{0},{\mathbf{\tilde{u}}}_{1})\right]$

##  Appendix B Additional Result

###  B.1 Additional Result

Figure 9: Images synthesis on CIFAR-10 and LSUN-Chruch-64.

Figure 10: Images synthesis on CelebA-HQ-256. Left figure shows
temperature=1.0. Right figure shows temperature=0.7.


A critical response to Moshe Chaichian and colleagues on mass dimension one
fields, and related matters

Dharam Vir Ahluwaliaa,1, Cheng-Yang Leeb,2

> a Center for the Studies of the Glass Bead Game, Notting Hill,
> Victoria 3168, Australia
> b Center for Theoretical Physics, College of Physics, Sichuan University,
> Chengdu, 610064, China
>
> 1 Email<EMAIL_ADDRESS>
> ORCID: https://orcid.org/0000-0002-6331-6243, Corresponding author
> 2 E-mail<EMAIL_ADDRESS>
> ORCID: https://orcid.org/0000-0002-6926-2717

> Abstract. In a recent article published in EPJC, Moshe Chaichian and
> colleagues have presented an overarching critique of mass dimension one
> fields and related matters. In this Letter we show without ambiguity where
> these authors go wrong.
>
> Keywords.  Elko, Mass dimension one fermions, spin-half bosons, Wigner
> degeneracy, dark matter, dark energy

Authors of reference [1] claim that the the quantum field theoretic formalism
of mass dimension one fields [2, 3, 4] is not physically viable on the grounds
that the formalism cannot satisfy the rotational constraint. They make the
same claim in the context of mass dimension three half bosons, and assert that
duals and adjoints of expansion coefficients and the associated fields do not
enjoy the freedom we invoke. We here give briefest of arguments to show that
these authors are completely mistaken in all their claims.

To see the necessity of a new spinorial dual let
$\lambda(\boldsymbol{\mbox{\boldmath$\displaystyle\boldsymbol{p}$}})$ be a
Majorana spinor associated with mass $m\neq 0$. Its right-transforming
component has opposite spin projection when compared with its left-
transforming component. So it is genuinely different from a Dirac spinor.
Furthermore, if we define its dual as

$\overline{\lambda}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\lambda^{\dagger}(\boldsymbol{\mbox{\boldmath$\displaystyle\boldsymbol{p}$}})\gamma_{0}$
(1)

as is often done in the literature then
$\overline{\lambda}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\lambda(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$
identically vanishes [5]. Unless one deviates from the Dirac formalism and
promotes the Majorana spinors to Grassmann variables, the mass term
identically vanishes and for that reason one cannot construct a Lagrangian
density [5]. In the just cited reference, this is used as a justification to
promote Majorana spinors to Grassmann variables. A well prepared reader knows
that avoiding this step led in 2004 to the “unexpected theoretical discovery”
of mass dimension one fields. The line of arguments leading to this conclusion
is well documented [2].

To proceed further, consider a
$\lambda(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$ with its left
transforming component with spin projection $+1/2$. Call this spinor as
$\lambda_{+}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$. Next consider a
$\lambda(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$ with its left
transforming component with spin projection $-1/2$. Call this spinor as
$\lambda_{-}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$. Then, with
$\mathcal{D}\stackrel{{\scriptstyle\mathrm{def}}}{{=}}m^{-1}\gamma_{\mu}p^{\mu}$
one readily finds that Majorana spinors can not be eigenspinors of
$\mathcal{D}$, but instead, modulo factors of $\pm i$, $\mathcal{D}$ takes
$\lambda_{+}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$ into
$\lambda_{-}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$ and vice versa

$\mathcal{D}\lambda_{+}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})=i\lambda_{-}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$}),\quad\mathcal{D}\lambda_{-}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})=-i\lambda_{+}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$}),$
(2)

To the best of our knowledge, result (2) was first obtained by Valeri
Dvoeglazov [6], and only later refined in [3].

This readies the stage to define a new dual defined as

$\overset{\>{}^{{}^{\boldsymbol{\neg}}}}{\smash[t]{\lambda}}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\left[\mathcal{D}\lambda(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\right]^{\dagger}\gamma_{0}.$
(3)

Under this dual,
$\overset{\>{}^{{}^{\boldsymbol{\neg}}}}{\smash[t]{\lambda}}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\lambda(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$
yields a non-zero norm. All this is well known to the community of people
working on the subject. Our only reason to repeat this in this briefest of
reviews is to show how duals, and the associated adjoints, come to be, and to
point out how claims of reference [1] are totally in error.

We now make a transition from Majorana spinors to Elko. The charge conjugation
operator

$\mathcal{C}=\left(\begin{array}[]{cc}\mbox{\boldmath$\displaystyle\boldsymbol{0}$}&i\Theta\\\
-i\Theta&\mbox{\boldmath$\displaystyle\boldsymbol{0}$}\end{array}\right)K.$
(4)

where $K$ complex conjugates to the right, has two doubly degenerate
eigenvalues.111Because of the nonlinearity of $\mathcal{C}$ we have a choice
of choosing these to be real, or complex. We choose these to be real. Two of
these account for Majorana spinors, and we take it to be $+1$. The other two,
which we choose to be $-1$, are associated with the anti self conjugate
spinors. Together, these form what in literature are called Elko and are
represented by self conjugate spinors
$\lambda_{\pm}^{S}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$, and anti
self conjugate spinors,
$\lambda_{\pm}^{A}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$.

Because the charge conjugation operator is _an antilinear_ operator, the
product,
$i\times\lambda_{\pm}^{A}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$ is
self conjugate under $\mathcal{C}$. Similarly, the product
$i\times\lambda_{\pm}^{S}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})$
yields an anti self conjugate spinor. We thus have four self conjugate Elko,
and four anti self conjugate Elko. This doubling in the degrees of freedom is
over and above that which arises from the two fold degeneracy under
$\mathcal{C}$. Such a possibility was first envisioned by Wigner [7].

These can be used as expansion coefficients of a quantum field that carries a
two fold Wigner degeneracy. Such a spin one half field has mass dimension one
and can satisfy fermionic, and surprisingly also the bosonic statistics.

Our remaining task is two fold. First, we must argue as to how the rotational
symmetry is preserved in our formalism. Second, we must show how spin one half
bosons arise in the mass dimension three half, as well mass dimension one
case.

Generally, it is assumed that the expansion coefficients of a quantum field at
rest are completely unconstrained as long as they form a complete set. A
detailed study of Weinberg’s quantum theory of fields [8] indeed finds this
widespread misconception needs to be corrected. The authors of reference [1]
indeed pay a superficial attention to his analysis. In fact, their “analysis”
verges on plagiarism of Weinberg’s profound work. Furthermore, they apply
their analysis to the wrong degrees of freedom! In this instance, the correct
field to apply the rotational constraints must have a two fold Wigner
degeneracy. When this is done, one indeed finds that the said rotational
constraints are fully satisfied.

Here is the brief outline of how the calculation on rotational symmetry
proceeds. The rotational constraints read

$\sum_{{\sigma}^{\prime}}\lambda^{S}_{{\ell^{\prime}}}(\mbox{\boldmath$\displaystyle\boldsymbol{0}$},\sigma^{\prime})\,\mbox{\boldmath$\displaystyle\boldsymbol{J}$}_{{\sigma}^{\prime}\sigma}=\sum_{\ell}\boldsymbol{\mathcal{J}}_{{\ell}^{\prime}\ell}\,\lambda^{S}_{\ell}(\mbox{\boldmath$\displaystyle\boldsymbol{0}$},\sigma),$
(5)

and

$\sum_{{\sigma^{\prime}}}\lambda^{A}_{{\ell^{\prime}}}(\mbox{\boldmath$\displaystyle\boldsymbol{0}$},\sigma^{\prime})\,\mbox{\boldmath$\displaystyle\boldsymbol{J}$}^{\ast}_{{\sigma^{\prime}}\sigma}=-\sum_{\ell}\boldsymbol{\mathcal{J}}_{{\ell^{\prime}}\ell}\,\lambda^{A}_{\ell}(\mbox{\boldmath$\displaystyle\boldsymbol{0}$},\sigma).$
(6)

The $\lambda^{S}(\mbox{\boldmath$\displaystyle\boldsymbol{0}$},\sigma)$ and
$\lambda^{A}(\mbox{\boldmath$\displaystyle\boldsymbol{0}$},\sigma)$ are the
expansion coefficients at rest of the quantum field with a two fold Wigner
degeneracy, and thus $\sigma=1,2,3,4$. The $\boldsymbol{\mathcal{J}}$ are

$\displaystyle\mathcal{J}_{x}=\frac{1}{2}\left(\begin{array}[]{cc}\sigma_{x}&\mathbb{o}\\\
\mathbb{o}&\sigma_{x}\end{array}\right),\quad\mathcal{J}_{y}=\frac{1}{2}\left(\begin{array}[]{cc}\sigma_{y}&\mathbb{o}\\\
\mathbb{o}&\sigma_{y}\end{array}\right),$ (11)
$\displaystyle\mathcal{J}_{z}=\frac{1}{2}\left(\begin{array}[]{cc}\sigma_{z}&\mathbb{o}\\\
\mathbb{o}&\sigma_{z}\end{array}\right).$ (14)

where $\displaystyle\boldsymbol{\sigma}$ are the usual Pauli matrices with
$\sigma_{z}$ diagonal. If the formalism is to satisfy (5) and (6), there must
exist an $\mathfrak{su}(2)$ satisfying $4\times 4$ matrices
$\displaystyle\boldsymbol{J}$ with _doubly_ degenerate eigenvalues $\pm 1/2$.
Such a $\displaystyle\boldsymbol{J}$ exists with the consequence that the
formalism of mass dimension one fields satisfies the rotational constraints.
The calculational difference with the Weinberg’s formalism _without_ Wigner
degeneracy for spin one half is that Weinberg takes $2\times 2$ matrices
$\displaystyle\boldsymbol{J}$ as input and obtains the particle and
antiparticle expansion coefficients as an output. Here, apart from matters
arising from Wigner degeneracy, the calculations feed into the rotational
constraints the particle and antiparticle expansion coefficients at rest and
solve for the $4\times 4$ $\displaystyle\boldsymbol{J}$ matrices.

Finally, to attend to the last concern of Moshe Chaichian and colleagues, we
note that generations of physicists have learned that the quantisation of the
Dirac field necessarily requires the use of anti-commutators for the field-
field (and also for the conjugate momentum) at space-like separation for the
Hamiltonian to be positive definite. A nuanced exposition of this is not
required as anyone reading this critique knows this as an irrefutable wisdom.
There is however a caveat, an implicit assumption, that duals of the Dirac
spinors must necessarily be
$\psi^{\dagger}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\gamma_{0}$, and
not be in parallel with (3):
$\overset{\>{}^{{}^{\boldsymbol{\neg}}}}{\smash[t]{\psi}}(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\left[\mathcal{D}\psi(\mbox{\boldmath$\displaystyle\boldsymbol{p}$})\right]^{\dagger}\gamma_{0}$.
The use of such a dual changes the sign of the norm of the antiparticle
spinors. When this sign is propagated through quantum field theoretic
calculations, one finds that the Dirac fields have positive definite
hamiltonians, and are local. The statistics: bosonic [4].222Similar remarks
apply to the formalism of mass dimension one bosons.

For the reasons outlined here we categorically assert that the paper of Moshe
Chaichian and colleagues is based on deep rooted misconceptions, and
application of a profound formalism of Weinberg to a wrong quantum field. More
details shall be published elsewhere.

Acknowledgements

DVA is supported by the Center for the Studies of the Glass Bead Game, CYL is
supported by the Sichuan University Postdoctoral Research Fund No.2022SCU12119

## References

  * [1] A. R. Aguirre, M. M. Chaichian, B. A. Couto e. Silva, and B. L. Sánchez-Vega, Spin-1/2 “bosons” with mass dimension 3/2 and fermions with mass dimension 1 cannot represent physical particle states, Eur. Phys. J. C 82 (2022), no. 10 958, [arXiv:2112.14797].
  * [2] D. V. Ahluwalia, J. M. Hoff da Silva, C.-Y. Lee, Y.-X. Liu, S. H. Pereira, and M. M. Sorkhi, Mass dimension one fermions: Constructing darkness, Physics Reports 967 (2022) 1–43. Mass dimension one fermions: Constructing darkness.
  * [3] D. V. Ahluwalia, Mass Dimension One Fermions.  Cambridge Monographs on Mathematical Physics. Cambridge University Press, 2019.
  * [4] D. V. Ahluwalia and C.-Y. Lee, Spin-half bosons with mass dimension three-half: Evading the spin-statistics theorem, Europhysics Letters 140 (oct, 2022) 24001.
  * [5] I. J. R. Aitchison and A. J. G. Hey, Gauge theories in particle physics: A practical introduction. Vol. 2: Non-Abelian gauge theories: QCD and the electroweak theory.  Bristol, UK: IOP (2004) 454 p, 2004.  The citations in the monograph refer to this edition.
  * [6] V. V. Dvoeglazov, Lagrangian for the Majorana-Ahluwalia construct, Nuovo Cim. A 108 (1995) 1467–1476.
  * [7] E. P. Wigner, Unitary representations of the inhomogeneous Lorentz group including reflection, in Group theoretical concepts and methods in elementary particle Physics: Lectures of the Istanbul Summer School of theoretical physics, 1962, edited by F. Gursey, pp. 564–607, Gordon and Breach, 1964.
  * [8] S. Weinberg, The Quantum theory of fields. Vol. 1: Foundations.  Cambridge University Press, 6, 2005.