# Learning Distributions on Manifolds with Free-form Flows

Peter Sorrenson*, Felix Draxler*, Armand Rousselot*, Sander Hummerich, Ullrich
Köthe
Computer Vision and Learning Lab, Heidelberg University
*equal contribution<EMAIL_ADDRESS>
###### Abstract

Many real world data, particularly in the natural sciences and computer
vision, lie on known Riemannian manifolds such as spheres, tori or the group
of rotation matrices. The predominant approaches to learning a distribution on
such a manifold require solving a differential equation in order to sample
from the model and evaluate densities. The resulting sampling times are slowed
down by a high number of function evaluations. In this work, we propose an
alternative approach which only requires a single function evaluation followed
by a projection to the manifold. Training is achieved by an adaptation of the
recently proposed free-form flow framework to Riemannian manifolds. The
central idea is to estimate the gradient of the negative log-likelihood via a
trace evaluated in the tangent space. We evaluate our method on various
manifolds, and find significantly faster inference at competitive performance
compared to previous work. We make our code public at
https://github.com/vislearn/FFF.††Preprint. Under review.

##  1 Introduction

Figure 1: Manifold free-form flows (M-FFF) learn a distribution directly on
the manifold via a generator network in the embedding space
$\tilde{g}_{\phi}:\mathbb{R}^{m}\to\mathbb{R}^{m}$ (top left) whose outputs
are projected to the manifold (top right). In this example, a uniform
distribution (bottom left) is mapped to a bimodal distribution on the circle
(bottom right), which we show as a histogram over model samples and the
learned density as a function of the angle. The embedding conserves the
topology so that the resulting distribution is periodic and continuous
everywhere.

In recent years, generative models have achieved remarkable success in various
domains like image synthesis [44], natural language processing [6], science
[39] and more. However, the approaches are not naively applicable when dealing
with data inherently structured in non-Euclidean spaces, particularly
prevalent in fields such as the natural sciences and computer vision. Examples
of such data are earth science data supported on a sphere, angles between
functional groups in proteins, which are naturally represented on a torus, as
well as the orientation of real world objects given as a rotation matrix in
$SO(3)$. Naively, data on such an $n$-dimensional manifold $\mathcal{M}$ can
be represented using internal coordinates such as angles. However, this may
not respect the topology, causing discontinuities or artifacts.

Luckily, many generative models can be generalized to arbitrary manifolds.
Most existing methods involve solving differential equations, either
stochastic (SDEs) or ordinary (ODEs), for sampling and density estimation such
as [45, 30, 20, 10, 8]. While theoretically sound, these approaches are
computationally intensive due to the need for numerous function evaluations
during integration, slowing down sampling.

In this paper, we introduce a novel approach for modeling distributions on
arbitrary Riemannian manifolds that circumvents the computational burden of
previous methods. We achieve this by using a single feed-forward neural
network on an embedding space as a generator whose outputs are projected to
the manifold Fig. 1. We learn this network as a normalizing flow, made
possible through adapting the free-form flow framework [11], recently proposed
for Euclidean spaces, to Riemannian manifolds. The core change is estimating
the gradient of the negative log-likelihood within the tangent space of the
manifold.

In particular, we make the following contributions:

  * •

We adapt free-form flows to Riemannian manifolds, yielding manifold free-form
flows (M-FFF) in Sec. 4. The model is trained end-to-end and involves only a
single function evaluation in training and sampling.

  * •

We demonstrate the model’s quality and speed on several manifold benchmarks,
such as rotation matrices, earth data, and molecular torsion angles in Sec. 5.
We find competitive quality to recent work at typically two orders of
magnitude faster sampling.

Together, manifold free-form flows are a novel approach for learning
distributions on manifolds. They are applicable to any Riemannian manifold
with a known embedding and projection.

##  2 Related work

Existing work on learning distributions on manifolds can be broadly
categorized as follows: (i) leveraging Euclidean generative models; (ii)
building specialized architectures that respect one particular kind of
geometry; and (iii) learning a continuous time process on the manifold. We
compare our method to these approaches in Tab. 1 and give additional detail
below.

Euclidean generative models. One approach maps the $n$-dimensional manifold to
$\mathbb{R}^{n}$ and learns the resulting distribution [17]. Another approach
generalizes the reparameterization trick to Lie groups by sampling on the Lie
algebra which can be parameterized in Euclidean space [16]. These approaches
come with the downside that a Euclidean representation may not respect the
geometry of the manifold sufficiently, e.g. mapping the earth to a plane
causes discontinuities at the boundaries. This can be overcome by learning
distributions on overlapping charts that together span the full manifold [23].
An orthogonal solution is to embed the data and add noise to it in the off-
manifold directions, so that the distribution can be learnt directly in an
embedding space $\mathbb{R}^{m}$ [5]; this only gives access to an ELBO
instead of the exact density. Our method also works in the embedding space so
that it respects the geometry of the manifold, but directly optimizes the
likelihood on the manifold.

Specialized architectures take advantage of the specific geometry of a certain
kind of manifold to come up with special coupling blocks for building
normalizing flows such as $SO(3)$ [27], $SU(n),U(n)$ [3, 24]; hyperbolic space
[2]; and tori and spheres [43]. Manifold free-form flows are not restricted to
one particular manifold, but can be easily applied to any manifold for which
an embedding and a projection to the manifold is known. As such, our model is
an alternative to all of the above specialized architectures.

Continuous time models build a generative model based on parameterizing a ODE
or SDE on any Riemannian manifold, meaning that they specify the (stochastic)
differential equation in the tangent space [45, 14, 15, 20, 30, 10, 8, 28, 1].
These methods come with the disadvantage that sampling and density evaluation
integrates the ODE or SDE, requiring many function evaluations. Our manifold
free-form flows do not require repeatedly evaluating the model, a single
function call followed by a projection is sufficient.

At its core, our method adapts the recently introduced free-form flow (FFF)
framework [11] based on an estimator for the gradient of the change of
variables formula [46]. We give more details in Sec. 3.1.

|  Respect topology |  Single step sampling |  Arbitrary manifolds
---|---|---|---
Euclidean | ✗ | ✓ | ✓
Specialized | ✓ | ✓ | ✗
Continuous time | ✓ | ✗ | ✓
M-FFF (ours) | ✓ | ✓ | ✓

Table 1: Comparison of generative models on manifolds. We give a “✓” if any
method in a category meets this requirement.

##  3 Background

In this section, we give some background for our method: We present an
introduction to free-form flows, followed by Riemannian manifolds and
distributions thereon.

###  3.1 Free-form flows

Free-form flows are a generative model that modify normalizing flows to work
with arbitrary feed-forward neural network architectures [11]. Traditionally,
normalizing flows are based on invertible neural networks (INNs) that learn an
invertible transformation $z=f_{\theta}(x)$ from data $x\in\mathbb{R}^{n}$ to
latent codes $z\in\mathbb{R}^{n}$. This gives an explicit parameterized
probability density $p_{\theta}(x)$ via the change-of-variables:

$\log p_{\theta}(x)=\log p_{Z}(f_{\theta}(x))+\log|f_{\theta}^{\prime}(x)|,$
(1)

where $f_{\theta}^{\prime}(x)\in\mathbb{R}^{n\times n}$ is the Jacobian matrix
of $f_{\theta}(x)$ with respect to $x$, evaluated at $x$;
$|f_{\theta}^{\prime}(x)|$ is its absolute determinant. The distribution of
latent codes $p_{Z}(z)$ is chosen such that the log-density is easy to
evaluate and it is easy to sample from. Normalizing flows can be trained by
minimizing the negative log-likeliriemannianhood over the training data:

$\min_{\theta}\mathcal{L}_{\text{NLL}}=\min_{\theta}\mathbb{E}_{x\sim
p_{\text{data}}}[\log p_{\theta}(x)].$ (2)

This is equivalent to minimizing the Kullback-Leibler-divergence between the
true data distribution and the parameterized distribution
$\operatorname{KL}(p_{\text{data}}\|p_{\theta})$. Sampling from the model is
achieved by pushing samples from the latent distribution $z\sim p_{Z}$ through
the inverse of the learned function: $x=f_{\theta}^{-1}(z)\sim p_{\theta}(x)$.

Free-form flows (FFF, [11]) soften the restriction that the learned model be
invertible. Instead, they learn two separate networks, an encoder $f_{\theta}$
and a decoder $g_{\phi}$ coupled by a reconstruction loss:

$\mathcal{L}_{\text{R}}=\mathbb{E}_{q(x)}\left[\|g_{\phi}(f_{\theta}(x))-x\|^{2}\right].$
(3)

Encoder and decoder replace $f_{\theta}$ respectively its inverse
$f_{\theta}^{-1}$ of an invertible neural network.

Typical normalizing flow architectures are optimized to allow efficient
computation of the change of variables term $\log|f_{\theta}^{\prime}(x)|$.
This usually goes hand in hand with a constrained architecture that is
analytically and practically invertible. Free-form flows are based on
arbitrary neural networks and the assumption of a tractable change of
variables does not hold in general. FFFs therefore estimate the gradient of
the change of variables via the gradient trick proposed by [46]:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\text{NLL}}$
$\displaystyle=\nabla_{\theta}\mathbb{E}_{q(x)}\left[-\log
p_{Z}(z)-\operatorname{tr}(f_{\theta}^{\prime}(x)\texttt{SG}[g_{\phi}^{\prime}(z)])\right]$
(4) $\displaystyle=\nabla_{\theta}\mathbb{E}_{x,v}\left[-\log
p_{Z}(z)-v^{T}f_{\theta}^{\prime}(x)\texttt{SG}[g_{\phi}^{\prime}(z)v]\right]$
(5)

Here, $z=f_{\theta}(x)$, $v\in\mathbb{R}^{n}$ is a random vector with unit
covariance, and SG (stop-grad) ensures that gradients only flow to the encoder
parameters $\theta$. Instead of $\mathcal{O}(n)$ automatic differentiation
steps to compute the full Jacobian $f_{\theta}^{\prime}(x)$, this estimator
only requires a single vector-Jacobian and a Jacobian-vector product, each of
which require one automatic differentiation operation. Note that while the
gradient in Eq. 5 is meaningful, the value of $\mathcal{L}_{\text{NLL}}$ is
not.

Together, this gives the loss of free-form flows with $\beta$, the
reconstruction weight as a hyperparameter:

$\mathcal{L}_{\text{FFF}}=\mathcal{L}_{\text{NLL}}+\beta\mathcal{L}_{\text{R}}.$
(6)

In Sec. 4, we demonstrate how this loss can be used to estimate the gradient
of the change of variables formula on manifolds, which we now come to.

###  3.2 Riemannian manifolds

A manifold is a fundamental concept in mathematics, providing a framework for
describing and analyzing spaces that locally resemble Euclidean space, but may
have different global structure. For example, a small region on a sphere is
similar to Euclidean space, but walking in a straight line on the sphere in
any direction will return back to the starting point, unlike in Euclidean
space.

Mathematically, an $n$-dimensional manifold, denoted as $\mathcal{M}$, is a
space where every point has a neighborhood that is topologically equivalent to
$\mathbb{R}^{n}$. A Riemannian manifold $(\mathcal{M},G)$ extends the concept
of a manifold by adding a Riemannian metric $G$ which introduces a notion of
distances and angles. At each point $x$ on the manifold, there is an
associated tangent space $\mathcal{T}_{x}\mathcal{M}$ which is an
$n$-dimensional Euclidean space, characterizing the directions in which you
can travel and still stay on the manifold. The metric $G$ acts in this space,
defining an inner product between vectors. From this inner product, we can
compute the length of paths along the manifold and hence distances between two
points.

In this paper, we consider Riemannian manifolds globally embedded into an
$m$-dimensional Euclidean space $\mathbb{R}^{m}$, with $n\leq m$. Embedding
means that we represent a point on the manifold $x\in\mathcal{M}$ as a vector
in $\mathbb{R}^{m}$ confined to an $n$-dimensional subspace. A global
embedding is a smooth, injective mapping of the entire manifold into a
$\mathbb{R}^{m}$, its smoothness preserving the topology.

We only work with isometrically embedded manifolds, meaning that the metric is
inherited from the ambient Euclidean space. Intuitively, this means that the
length of a path on the manifold is just the Euclidean length of that path. We
note that due to the Nash embedding theorem [36], every Riemannian manifold
has a smooth isometric embedding into Euclidean space of some finite
dimension, so in this sense using isometric embeddings is not a limitation.
Nevertheless, for some manifolds (especially with negative curvature, e.g.
hyperbolic space) there may not be a sensible isometric embedding. We provide
a generalization of our method to such manifolds in Sec. 7.2 but do not pursue
it experimentally. In Tab. 2, we list the manifolds considered in this paper,
their dimension, embedding and how to project from the embedding space to the
manifold.

Manifold | Dimension $n$ | Embedding | Projection
---|---|---|---
Rotations $SO(d)$ | $(d-1)d/2$ | $\\{Q\in\mathbb{R}^{d\times d}:QQ^{T}=I,\det Q=1\\}$ |  $R\mapsto\operatorname*{arg\,min}_{Q\in SO(d)}\|Q-R\|_{F}$; see Eq. 20
Sphere $\mathbb{S}^{n}$ | $n$ | $\\{x\in\mathbb{R}^{n+1}:\|x\|=1\\}$ | $x\mapsto x/\|x\|$
Torus $\mathbb{T}^{n}=(\mathbb{S}^{1})^{n}$ | $n$ | $\\{X\in\mathbb{R}^{n\times 2}:\|X_{i}\|=1\text{ for }i=1...n\\}$ | $X_{i}\mapsto X_{i}/\|X_{i}\|\text{ for }i=1...n$

Table 2: Manifolds, a global embedding and corresponding projections
considered in this paper.

###  3.3 Change of variables on manifolds

Figure 2: Changing variables between manifolds requires calculations in the
tangent spaces of $\mathcal{M}_{X}$ and $\mathcal{M}_{Z}$. Specifically, when
changing variables from $x$ to $f(x)$, we need to account for the volume
change of the transformation between $\mathcal{T}_{x}\mathcal{M}_{X}$ and
$\mathcal{T}_{f(x)}\mathcal{M}_{Z}$. Consider a basis $\\{e_{i}\\}$ for the
tangent space at $x$. This basis is transformed under $f$ into
$\\{e_{i}^{\prime}\\}$ where $e_{i}^{\prime}=f^{\prime}(x)e_{i}$. The volume
change is the ratio between the areas of the parallelograms (purple regions)
formed by $e_{i}$ and $e_{i}^{\prime}$, equal to the determinant of
$f^{\prime}(x)$, with an additional factor if the manifold is not
isometrically embedded, see Sec. 7.1 for details.

Let $(\mathcal{M}_{X},G_{X})$ and $(\mathcal{M}_{Z},G_{Z})$ be $n$-dimensional
Riemannian manifolds. Let $f:\mathcal{M}_{X}\rightarrow\mathcal{M}_{Z}$ . Then
the probability density induced by changing variables with $f$ is

$\log p_{X}(x)=\log
p_{Z}(f(x))+\log|f^{\prime}(x)|+\tfrac{1}{2}\log\frac{|G_{Z}(f(x))|}{|G_{X}(x)|}$
(7)

with $f^{\prime}(x)$ the Jacobian of $f$ at $x$ and $|\cdot|$ the absolute
determinant. Some care has to be taken with the determinant. Strictly
speaking, $f^{\prime}$ is a linear map between
$\mathcal{T}_{x}\mathcal{M}_{X}$ and $\mathcal{T}_{f(x)}\mathcal{M}_{Z}$ and
the determinant should be evaluated in those spaces, as visualized in Fig. 2.
Naively evaluating in $\mathbb{R}^{m}$ will give zero if $n<m$: see Sec. 7.1
for a more rigorous treatment and a derivation of Eq. 7.

For the remainder of this paper, we will drop the index on the manifolds
$\mathcal{M}_{X}$ and $\mathcal{M}_{Z}$ and simply write $\mathcal{M}$. We do
this to simplify notation, but also we use $\mathcal{M}_{X}=\mathcal{M}_{Z}$
in all cases. Note that our method easily generalizes to using different
manifolds in data and latent space, as long as they are topologically
equivalent.

Along with the simplification that the manifolds are isometrically embedded,
meaning $|G_{X}|=|G_{Z}|=1$, we have

$\log p_{X}(x)=\log p_{Z}(f(x))+\log|f^{\prime}(x)|,$ (8)

which is identical to the familiar change of variables formula used in
normalizing flows, with the determinant being evaluated in the tangent spaces.

##  4 Manifold free-form flows

The free-form flow (FFF) framework allows training any pair of parameterized
encoder $f_{\theta}(x)$ and decoder $g_{\phi}(z)$ as a generative model [11],
see Sec. 3.1. In this section, we first show how we set up our models to learn
encoders and decoders that are restricted to the manifold. Then, we proceed to
show how we adapt the FFF training to manifold free-form flows (M-FFF).

###  4.1 Learning a function on the manifold

For both encoder $f_{\theta}$ and decoder $g_{\phi}$, we employ feed-forward
neural networks
$\tilde{f}_{\theta},\tilde{g}_{\phi}:\mathbb{R}^{m}\to\mathbb{R}^{m}$ that act
on the embedding space $\mathbb{R}^{m}$ of the manifold $\mathcal{M}$. Since
the data and latent codes both lie on the manifold, we project the outputs of
the networks to the closest point on the manifold:

$\displaystyle f_{\theta}(x)=\operatorname{proj}(\tilde{f}_{\theta}(x)),$ (9)
$\displaystyle g_{\phi}(z)=\operatorname{proj}(\tilde{g}_{\phi}(z)).$ (10)

By restricting the input of both networks to the manifold, we obtain functions
acting on the manifold:

$\displaystyle f_{\theta}(x):\mathcal{M}\to\mathcal{M},$ (11) $\displaystyle
g_{\phi}(z):\mathcal{M}\to\mathcal{M}.$ (12)

Figure 1 shows an example that maps from a latent uniform distribution to a
bimodal distribution on the circle. In the next section, we explain how to
estimate the gradient of the negative log-likelihood.

###  4.2 Manifold gradient estimator

The adaptation of free-form flows to isometrically embedded manifolds is
remarkably simple. Due to the similarity of the change of variables formula
(Eq. 8) to the Euclidean case, we can simply adapt the FFF estimator:

$\nabla_{\theta}\mathcal{L}_{\text{NLL}}=\nabla_{\theta}\mathbb{E}_{x,v}\left[-\log
p_{Z}(z)-v^{T}f_{\theta}^{\prime}(x)\texttt{SG}[g_{\phi}^{\prime}(z)v]\right]$
(13)

where the only change is that $v$ must be sampled from the tangent space at
$f_{\theta}(x)$, and must have unit covariance in that space. We achieve this
by sampling standard normal vectors $\tilde{v}\in\mathbb{R}^{m}$ and then
projecting them into the tangent space using the Jacobian of the projection
function:

$v=\operatorname{proj}^{\prime}(f_{\theta}(x))\tilde{v}.$ (14)

We derive this estimator in detail in Sec. 7.2.

An equivalent formulation is to prepend $\tilde{f}$ and $\tilde{g}$ by a
projection in addition to appending the projection:

$f_{\theta}=\operatorname{proj}\circ\tilde{f}_{\theta}\circ\operatorname{proj}$
(15)

and

$g_{\phi}=\operatorname{proj}\circ\tilde{g}_{\phi}\circ\operatorname{proj}.$
(16)

In this case $v$ can be directly sampled in $\mathbb{R}^{m}$, so that the
Euclidean gradient estimator from Eq. 5 applies without modification.
Nevertheless, we choose to use the first formulation ($v$ projected to the
tangent space) as it allows us to reduce the variance of the estimate by
normalizing $v$ after projection. A discussion of this point can found in Sec.
7.3.

###  4.3 Reconstruction loss and other regularization

The reconstruction loss (Eq. 3) can be modified to use an on-manifold distance
(e.g. great circle distance for the sphere) but we find that ambient Euclidean
distance works well in practice, since it is almost identical for small
distances. Hence we use the same Euclidean reconstruction loss as in free-form
flows.

We find that adding the following two regularizations to the loss improve the
stability and performance of our models. Firstly, the reconstruction loss on
points sampled uniformly from the data manifold:

$\mathcal{L}_{\text{U}}=\mathbb{E}_{x\sim\mathcal{U}(\mathcal{M})}[\|g_{\phi}(f_{\theta}(x))-x\|^{2}]$
(17)

which helps ensure that we have a globally consistent mapping between the data
and latent manifolds in low data regions. Secondly, the squared distance
between the output of $\tilde{f}_{\theta}$ and its projection to the manifold:

$\mathcal{L}_{\text{P}}=\mathbb{E}_{q(x)}[\|\tilde{f}_{\theta}(x)-f_{\theta}(x)\|^{2}]$
(18)

which discourages the output of $\tilde{f}_{\theta}$ from entering
unprojectable regions, for example the origin when the manifold is
$\mathbb{S}^{n}$. The same regularizations can be applied starting from the
latent space.

The full loss is:

$\mathcal{L}=\mathcal{L}_{\text{NLL}}+\beta_{\text{R}}\mathcal{L}_{\text{R}}+\beta_{\text{U}}\mathcal{L}_{\text{U}}+\beta_{\text{P}}\mathcal{L}_{\text{P}}$
(19)

where $\beta_{\text{R}}$, $\beta_{\text{U}}$ and $\beta_{\text{P}}$ are
hyperparameters. We give our choices in Sec. 8.

##  5 Experiments

In this section, we demonstrate the practical performance of manifold free-
form flows on a variety of manifolds. We choose established experiments to
ensure comparability with previous methods. Generally, we find that we obtain
competitive performance with the advantage that our method samples orders of
magnitudes faster because it requires only a single function evaluation. In
contrast, previous general purpose work integrates a differential equation on
the manifold, which requires multiple model evaluations.

We give all details on training in the appendix.

###  5.1 Synthetic distribution over rotation matrices

The group of 3D rotations $SO(3)$ can be represented by rotations matrices
with positive determinant, i.e. all $Q\in\mathbb{R}^{3\times 3}$ with
$Q^{T}Q=I$ and $\det Q=1$. To apply manifold free-form flows, we project an
output matrix $R\in\mathbb{R}^{3\times 3}$ from the encoder/decoder to the
subspace of special orthogonal matrices by finding the matrix $Q\in SO(3)$
minimizing the Frobenius norm $\|Q-R\|_{F}$. This is known as the constrained
Procrustes problem and the solution $Q$ can be determined via the SVD
$R=U\Sigma V^{T}$ [26]:

$Q=USV^{T},$ (20)

where the diagonal entries of $\Sigma$ were sorted from largest to smallest
and $S=\operatorname{Diag}(1,\dots,1,\det(UV^{T}))$.

We learn a M-FFF on synthetic mixture distributions proposed by [10] with $M$
mixture components for $M=16,32$ and $64$. Samples from one of the
distributions and samples from our model are depicted in Fig. 3.

M-FFF achieves negative log-likelihoods on par or better than previous
methods, see Tab. 3. At the same time, it speeds up sampling by about two
orders of magnitude since it requires only a single function evaluation to
sample.

Figure 3: Manifold free-form flows on a synthetic $SO(3)$ mixture distribution with $M=64$ mixture components proposed by [10]. (Top) 10,000 samples each from the ground truth distribution and (bottom) our model. This visualization computes three Euler angles, which fully describe a rotation matrix, and then plot the first two angles on the projection of a sphere and the last by color [34]. We find that our model nicely samples from the distribution with few outliers between the modes. | $M=16$ | $M=32$ | $M=64$ | Sampling steps
---|---|---|---|---
Moser flow [45] | -0.85  $\pm$ 0.03 | -0.17  $\pm$ 0.03 | 0.49  $\pm$ 0.02 |  $\sim$ 1000
Exp-wrapped SGM [10] | -0.87  $\pm$ 0.04 | -0.16  $\pm$ 0.03 | 0.58  $\pm$ 0.04 |  $\sim$ 500
Riemannian SGM [10] | -0.89  $\pm$ 0.03 | -0.20  $\pm$ 0.03 | 0.49  $\pm$ 0.02 |  $\sim$ 100
M-FFF (ours) | -0.87  $\pm$ 0.02 | -0.21  $\pm$ 0.02 | 0.45  $\pm$ 0.02 | 1
Table 3: Results on synthetic $SO(3)$ mixtures. We compare the final model’s
negative log-likelihood (NLL, $\downarrow$) and the number of function
evaluations necessary to obtain a single sample averaged over five runs $\pm$
standard deviations. Our manifold free-form flows (M-FFF) achieve comparable
likelihoods at greatly improved sampling speed. Baselines values due to [10].

###  5.2 Earth data on the sphere

Figure 4: Density estimates of our model on the earth datasets. Blue points show the training dataset, red points the test dataset. | Volcano | Earthquake | Flood | Fire | Sampling steps
---|---|---|---|---|---
Mixture of Kent [41] | -0.80  $\pm$ 0.47 | 0.33  $\pm$ 0.05 | 0.73  $\pm$ 0.07 | -1.18  $\pm$ 0.06 | N/A
Riemannian CNF [30] | -6.05  $\pm$ 0.61 | 0.14  $\pm$ 0.23 | 1.11  $\pm$ 0.19 | -0.80  $\pm$ 0.54 |  $\sim$ 100
Moser flow [45] | -4.21  $\pm$ 0.17 | -0.16  $\pm$ 0.06 | 0.57  $\pm$ 0.10 | -1.28  $\pm$ 0.05 | $\sim 100$
Stereographic score-based [10] | -3.80  $\pm$ 0.27 | -0.19  $\pm$ 0.05 | 0.59  $\pm$ 0.07 | -1.28  $\pm$ 0.12 |  $\sim$ 100
Riemannian score-based [10] | -4.92  $\pm$ 0.25 | -0.19  $\pm$ 0.07 | 0.45  $\pm$ 0.17 | -1.33  $\pm$ 0.06 |  $\sim$ 100
Riemannian diffusion [20] | -6.61  $\pm$ 0.97 | -0.40  $\pm$ 0.05 | 0.43  $\pm$ 0.07 | -1.38  $\pm$ 0.05 |  $>$ 100
Riemannian flow matching [8] | -7.93  $\pm$1.67 | -0.28  $\pm$ 0.08 | 0.42  $\pm$ 0.05 | -1.86  $\pm$ 0.11 | not reported
M-FFF (ours) | -2.25  $\pm$ 0.02 | -0.23  $\pm$ 0.01 | 0.51  $\pm$ 0.01 | -1.19  $\pm$ 0.03 | 1
Table 4: Negative log-likelihoods for the earth datasets (lower is better).
Results for the baselines taken from [10], [20] and [8]. While our results
show mixed performance compared to the baselines, we uniquely provide a
competitive single-step sampling method.

We evaluate manifold free-form flows on spheres with datasets from the domain
of earth sciences. We use four established datasets compiled by [30] for
density estimation on $\mathbb{S}^{2}$: Volcanic eruptions [38], earthquakes
[37], floods [4] and wildfires [12].

We compare M-FFF to Riemannian score-based generative models [10], Riemannian
diffusion models [20], Riemannian flow matching [8] and the other baseline
models used in [10]. We provide a full comparison of negative log-likelihoods
in Table 4 and visualize the learned densities in Figure 4. Our method has a
mixed performance, but provides the only competitive solution that allows for
single-step sampling. While it is outperformed by Riemannian flow matching in
all datasets, the computational cost of training and evaluation is
significantly smaller due to the single step sampling and greatly reduced
model size (around 100 times smaller).

We notice that, since there are many regions without training data in the
different datasets, the predicted latent codes and thus likelihood estimates
of the validation set are often inaccurate. To ensure an accurate estimation
of the data likelihood under the decoder, during evaluation we sample multiple
latent points $z\sim\mathcal{N}(f_{\theta}(x),10^{-4})$ from a small region
around the predicted latent point and project them back onto the manifold. To
estimate the likelihood of $x$ we pick the point
$z^{\star}=\operatorname*{arg\,min}_{z}d(x,g_{\phi}(z))$. This results in a
reduction of the reconstruction error from $80$km to less than $10$km on
average. We find that the likelihoods computed in this way now agree with the
decoder distribution. Note that this does not affect sampling speed.

###  5.3 Torsion angles of molecules on tori

| General | Glycine | Proline | Pre-Pro | RNA | Sampling steps
---|---|---|---|---|---|---
Mixture of power spherical [20] | 1.15  $\pm$ 0.002 | 2.08  $\pm$ 0.009 | 0.27  $\pm$ 0.008 | 1.34  $\pm$ 0.019 | 4.08  $\pm$ 0.368 | N/A
Riemannian diffusion models [20] | 1.04  $\pm$ 0.012 | 1.97  $\pm$ 0.012 |  0.12 $\pm$ 0.011 | 1.24  $\pm$ 0.004 | -3.70  $\pm$ 0.592 |  $\sim$ 1000
Riemannian flow matching [8] |  1.01 $\pm$ 0.025 |  1.90 $\pm$ 0.055 | 0.15  $\pm$ 0.027 |  1.18 $\pm$ 0.055 |  -5.20 $\pm$ 0.067 | not reported
M-FFF (ours) | 1.03  $\pm$ 0.0002 | 1.92  $\pm$ 0.0013 | 0.21  $\pm$ 0.0019 | 1.27  $\pm$ 0.004 | -4.27  $\pm$ 0.09 | 1
Table 5: Negative log-likelihoods (NLL, $\downarrow$) on torus datasets.
Experimental data for a mixture of 4096 power spherical distributions [7] and
Riemannian diffusion models are taken from [20]. Furthermore, we compare
against Riemannian flow matching [8]. Mean and standard deviation are
evaluated over five independent models for each dataset. Notably, our method
is able to achieve on par negative log-likelihoods with a single sampling step
on most of the datasets and even outperforms Riemannian diffusion models on
some of them.

Figure 5: Log density of M-FFF models in the $(\Phi,\Psi)$-plane of protein
backbone dihedral angles. The test data for each dataset is shown as black
dots. This representation was introduced by [42] and is known as Ramachandran
plot. Note how the learned distribution adheres to the periodic boundary
conditions.

To benchmark manifold free-form flows on tori $\mathbb{T}^{n}$, we follow [20]
and evaluate our model on two datasets from structural biology. We consider
the torsion (dihedral) angles of the backbone of protein and RNA substructures
respectively.

We represent a tuple of angles $(\phi_{1},\dots,\phi_{n})\in[0,2\pi]^{n}$ by
mapping each angle to a position on a circle:
$X_{i}=(\cos\phi_{i},\sin\phi_{i})\in\mathbb{S}^{1}$. Then we stack all
$X_{i}$ into a matrix $X\in\mathbb{R}^{n\times 2}$, compare Tab. 2.

The first dataset is comprised of 500 proteins assembled by [29] and is
located on $\mathbb{T}^{2}$. The three dimensional arrangement of a protein
backbone can be described by the so called Ramachandran angles [42] $\Phi$ and
$\Psi$, which represent the torsion of the backbone around the $N-C_{\alpha}$
and $C_{\alpha}-C$ bonds. The data is split into four distinct subsets
General, Glycine, Proline and Pre-Proline, depending on the residue of each
substructure.

The second dataset is extracted from a subset of RNA structures introduced by
[35]. As the RNA backbone structure can be characterized by seven torsion
angles, in this case we are dealing with data on $\mathbb{T}^{7}$.

We record the negative log-likelihoods of our method in Tab. 5 and compare it
to previous approaches. Notably, our approach demonstrates comparable
performance across all datasets at fast sampling speed. In addition to the
quantitative results, in Fig. 5 we show the log densities of the M-FFF models
for the four protein datasets.

##  6 Conclusion

In this paper, we present manifold free-form flows (M-FFF) to learn a
probability distribution on a manifold. They are based on a neural network
construction that learns a function in an embedding space whose outputs are
projected to the manifold (Fig. 1). This naturally makes the learned
distributions adhere to the topology of the manifold at hand.

The distribution on the manifold is then learned by adapting the framework of
free-form flows (FFF) to manifolds. We extend it to estimate the gradient of
the change of variables formula for manifold-to-manifold functions.

The main advantage of M-FFF is that they offer competitive performance similar
to previous ODE- or SDE-based methods at greatly reduced sampling costs,
typically between two and three orders of magnitude faster. In addition, they
are very easy to set up, since only a projection from the embedding space to
the manifold needs to be known.

## Acknowledgements

This work is supported by Deutsche Forschungsgemeinschaft (DFG, German
Research Foundation) under Germany’s Excellence Strategy EXC-2181/1 -
390900948 (the Heidelberg STRUCTURES Cluster of Excellence). It is also
supported by the Vector Stiftung in the project TRINN (P2019-0092), and by
Informatics for Life funded by the Klaus Tschira Foundation. AR acknowledges
funding from the Carl-Zeiss-Stiftung. The authors acknowledge support by the
state of Baden-Württemberg through bwHPC and the German Research Foundation
(DFG) through grant INST 35/1597-1 FUGG.

## References

  * Ben-Hamu et al. [2022] Heli Ben-Hamu, Samuel Cohen, Joey Bose, Brandon Amos, Maximillian Nickel, Aditya Grover, Ricky TQ Chen, and Yaron Lipman.  Matching normalizing flows and probability paths on manifolds.  In _International Conference on Machine Learning_ , pages 1749–1763. PMLR, 2022.
  * Bose et al. [2020] Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton.  Latent variable modelling with hyperbolic normalizing flows.  In _International Conference on Machine Learning_ , pages 1045–1055. PMLR, 2020.
  * Boyda et al. [2021] Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez Rezende, Michael S. Albergo, Kyle Cranmer, Daniel C. Hackett, and Phiala E. Shanahan.  Sampling using $\mathrm{SU}(n)$ gauge equivariant flows.  _Phys. Rev. D_ , 103:074504, 2021.
  * Brakenridge [2017] G Brakenridge.  Global active archive of large flood events.  http://floodobservatory.colorado.edu/Archives/index.html, 2017.
  * Brofos et al. [2021] James A Brofos, Marcus A Brubaker, and Roy R Lederman.  Manifold density estimation via generalized dequantization.  _arXiv preprint arXiv:2102.07143_ , 2021.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cao and Aziz [2020] Nicola De Cao and Wilker Aziz.  The power spherical distribution, 2020.
  * Chen and Lipman [2023] Ricky TQ Chen and Yaron Lipman.  Riemannian flow matching on general geometries.  _arXiv preprint arXiv:2302.03660_ , 2023.
  * Davidson et al. [2018] Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak.  Hyperspherical variational auto-encoders.  _34th Conference on Uncertainty in Artificial Intelligence (UAI-18)_ , 2018.
  * De Bortoli et al. [2022] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet.  Riemannian score-based generative modelling.  _Advances in Neural Information Processing Systems_ , 35:2406–2422, 2022.
  * Draxler et al. [2023] Felix Draxler, Peter Sorrenson, Lea Zimmermann, Armand Rousselot, and Ullrich Köthe.  Free-form flows: Make any architecture a normalizing flow.  _arXiv preprint arXiv:2310.16624_ , 2023.
  * EOSDIS [2020] EOSDIS.  Land, atmosphere near real-time capability for eos (lance) system operated by nasa’s earth science data and information system (esdis).  https://earthdata.nasa.gov/earth-observation-data/near-real-time/firms/active-fire-data, 2020.
  * Falcon and The PyTorch Lightning team [2019] William Falcon and The PyTorch Lightning team.  PyTorch lightning, 2019.
  * Falorsi [2021] Luca Falorsi.  Continuous normalizing flows on manifolds.  _arXiv preprint arXiv:2104.14959_ , 2021.
  * Falorsi and Forré [2020] Luca Falorsi and Patrick Forré.  Neural ordinary differential equations on manifolds.  _arXiv preprint arXiv:2006.06663_ , 2020.
  * Falorsi et al. [2019] Luca Falorsi, Pim de Haan, Tim R Davidson, and Patrick Forré.  Reparameterizing distributions on lie groups.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pages 3244–3253. PMLR, 2019.
  * Gemici et al. [2016] Mevlana C Gemici, Danilo Rezende, and Shakir Mohamed.  Normalizing flows on riemannian manifolds.  _arXiv preprint arXiv:1611.02304_ , 2016.
  * Girard [1989] A Girard.  A fast ‘Monte-Carlo cross-validation’ procedure for large least squares problems with noisy data.  _Numerische Mathematik_ , 56:1–23, 1989.
  * Harris et al. [2020] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.  Array programming with NumPy.  _Nature_ , 585(7825):357–362, 2020.
  * Huang et al. [2022] Chin-Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron C Courville.  Riemannian diffusion models.  _Advances in Neural Information Processing Systems_ , 35:2750–2761, 2022.
  * Hunter [2007] J. D. Hunter.  Matplotlib: A 2D graphics environment.  _Computing in Science & Engineering_, 9(3):90–95, 2007.
  * Jost [2008] Jürgen Jost.  _Riemannian geometry and geometric analysis_.  Springer, 2008.
  * Kalatzis et al. [2021] Dimitris Kalatzis, Johan Ziruo Ye, Alison Pouplin, Jesper Wohlert, and Søren Hauberg.  Density estimation on smooth manifolds with normalizing flows.  _arXiv preprint arXiv:2106.03500_ , 2021.
  * Kanwar et al. [2020] Gurtej Kanwar, Michael S. Albergo, Denis Boyda, Kyle Cranmer, Daniel C. Hackett, Sé bastien Racanière, Danilo Jimenez Rezende, and Phiala E. Shanahan.  Equivariant flow-based sampling for lattice gauge theory.  _Physical Review Letters_ , 125(12), 2020.
  * Kühmichel and Draxler [2023] Lars Erik Kühmichel and Felix Draxler.  Lightning trainable, 2023.
  * Lawrence et al. [2019] Jim Lawrence, Javier Bernal, and Christoph Witzgall.  A purely algebraic justification of the kabsch-umeyama algorithm.  _Journal of research of the National Institute of Standards and Technology_ , 124:1, 2019.
  * Liu et al. [2023] Yulin Liu, Haoran Liu, Yingda Yin, Yang Wang, Baoquan Chen, and He Wang.  Delving into discrete normalizing flows on so (3) manifold for probabilistic rotation modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21264–21273, 2023.
  * Lou et al. [2020] Aaron Lou, Derek Lim, Isay Katsman, Leo Huang, Qingxuan Jiang, Ser Nam Lim, and Christopher M De Sa.  Neural manifold ordinary differential equations.  _Advances in Neural Information Processing Systems_ , 33:17548–17558, 2020.
  * Lovell et al. [2003] Simon C. Lovell, Ian W. Davis, W. Bryan Arendall III, Paul I. W. de Bakker, J. Michael Word, Michael G. Prisant, Jane S. Richardson, and David C. Richardson.  Structure validation by c$\alpha$ geometry: $\phi$,$\psi$ and c$\beta$ deviation.  _Proteins: Structure, Function, and Bioinformatics_ , 50(3):437–450, 2003.
  * Mathieu and Nickel [2020] Emile Mathieu and Maximilian Nickel.  Riemannian continuous normalizing flows.  _Advances in Neural Information Processing Systems_ , 33:2503–2515, 2020.
  * McKinney [2010] Wes McKinney.  Data Structures for Statistical Computing in Python.  In _9th Python in Science Conference_ , 2010.
  * Miolane et al. [2020] Nina Miolane, Nicolas Guigui, Alice Le Brigant, Johan Mathe, Benjamin Hou, Yann Thanwerdas, Stefan Heyder, Olivier Peltre, Niklas Koep, Hadi Zaatiti, Hatem Hajri, Yann Cabanes, Thomas Gerald, Paul Chauchat, Christian Shewmake, Daniel Brooks, Bernhard Kainz, Claire Donnat, Susan Holmes, and Xavier Pennec.  Geomstats: A python package for riemannian geometry in machine learning.  _Journal of Machine Learning Research_ , 21(223):1–9, 2020.
  * Miolane et al. [2023] Nina Miolane, Saiteja Utpala, Nicolas Guigui, Luís F. Pereira, Alice Le Brigant, Hzaatiti, Yann Cabanes, Johan Mathe, Niklas Koep, elodiemaignant, ythanwerdas, xpennec, tgeral68, Christian, Tra My Nguyen, Olivier Peltre, pchauchat, Jules-Deschamps, John Harvey, mortenapedersen, Maya95assal, Quentin Barthélemy, Abdellaoui-Souhail, Adele Myers, Felix Ambellan, Florent-Michel, Shubham Talbar, Stefan Heyder, Yann de Mont-Marin, and Marius.  geomstats/geomstats: Geomstats v2.7.0, 2023.
  * Murphy et al. [2021] Kieran A Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia.  Implicit-pdf: Non-parametric representation of probability distributions on the rotation manifold.  In _International Conference on Machine Learning_ , pages 7882–7893. PMLR, 2021.
  * Murray et al. [2003] Laura Murray, W Arendall, David Richardson, and Jane Richardson.  Rna backbone is rotameric.  _Proceedings of the National Academy of Sciences of the United States of America_ , 100:13904–9, 2003.
  * Nash [1956] John Nash.  The imbedding problem for riemannian manifolds.  _Annals of mathematics_ , 63(1):20–63, 1956.
  * [37] National Geophysical Data Center / World Data Service (NGDC/WDS).  Ncei/wds global significant earthquake database.  https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ngdc.mgg.hazards:G012153, 2022a.
  * [38] National Geophysical Data Center / World Data Service (NGDC/WDS).  Ncei/wds global significant volcanic eruptions database.  https://www.ncei.noaa.gov/access/metadata/landing-page/bin/iso?id=gov.noaa.ngdc.mgg.hazards:G10147, 2022b.
  * Noé et al. [2019] Frank Noé, Simon Olsson, Jonas Köhler, and Hao Wu.  Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning.  _Science_ , 365(6457):eaaw1147, 2019.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Peel et al. [2001] David Peel, William J Whiten, and Geoffrey J McLachlan.  Fitting mixtures of kent distributions to aid in joint set identification.  _Journal of the American Statistical Association_ , 96(453):56–63, 2001.
  * Ramachandran et al. [1963] G. N. Ramachandran, C. Ramakrishnan, and V. Sasisekharan.  Stereochemistry of polypeptide chain configurations.  _Journal of Molecular Biology_ , 7(1):95–99, 1963.
  * Rezende et al. [2020] Danilo Jimenez Rezende, George Papamakarios, Sébastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer.  Normalizing flows on tori and spheres.  In _International Conference on Machine Learning_ , pages 8083–8092. PMLR, 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Rozen et al. [2021] Noam Rozen, Aditya Grover, Maximilian Nickel, and Yaron Lipman.  Moser flow: Divergence-based generative modeling on manifolds.  _Advances in Neural Information Processing Systems_ , 34:17669–17680, 2021.
  * Sorrenson et al. [2023] Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, Lea Zimmermann, and Ullrich Köthe.  Lifting architectural constraints of injective flows.  _arXiv:2306.01843_ , 2023.
  * The pandas development team [2020] The pandas development team.  Pandas-dev/pandas: Pandas, 2020.

Supplementary Material

##  7 Free-form flows on Riemannian manifolds

In this appendix, we will focus on intuitive definitions of concepts from
topology and differential geometry. For a more rigorous treatment of these
concepts, see [22].

An $n$-dimensional manifold $\mathcal{M}$ is a space where every point $x$ has
a neighborhood which is homeomorphic to an open subset of $\mathbb{R}^{n}$.
Intuitively, this means that there is a small region of $\mathcal{M}$
containing $x$ which can be bent and stretched in a continuous way to map onto
a small region in $\mathbb{R}^{n}$. This is what is meant when we say that the
manifold locally resembles $\mathbb{R}^{n}$. If all these maps from
$\mathcal{M}$ to $\mathbb{R}^{n}$ are also differentiable then the manifold
itself is differentiable, as long as there is a way to connect up the local
neighborhoods in a differentiable and consistent way.

The tangent space of the manifold at $x$, denoted
$\mathcal{T}_{x}\mathcal{M}$, is an $n$-dimensional Euclidean space, which is
a linearization of the manifold at $x$: if we zoom in to a very small region
around $x$ the manifold looks flat, and this flat Euclidean space is aligned
with the tangent space. Because the tangent space is a linearization of the
manifold, this is where derivatives on the manifold live, e.g. if
$f:\mathcal{M}_{X}\rightarrow\mathcal{M}_{Z}$ is a map between two manifolds,
then the Jacobian $f^{\prime}(x)$ is a linear map from
$\mathcal{T}_{x}\mathcal{M}_{X}$ to $\mathcal{T}_{f(x)}\mathcal{M}_{Z}$.

A Riemannian manifold $(\mathcal{M},g)$ is a differentiable manifold which is
equipped with a Riemannian metric
$g:\mathcal{T}_{x}\mathcal{M}\times\mathcal{T}_{x}\mathcal{M}\rightarrow\mathbb{R}$
which defines an inner product on the tangent space, which allows us to
calculate lengths and angles in this space. The length of a smooth curve
$\gamma:[0,1]\rightarrow\mathcal{M}$ is given by the integral of the length of
its velocity vector $\gamma^{\prime}(t)\in\mathcal{T}_{\gamma(t)}\mathcal{M}$.
This ultimately allows us to define a notion of distance on the manifold, as
the curve of minimal length connecting two points.

In the remainder of the appendix we only consider Riemannian manifolds.

###  7.1 Manifold change of variables

#### Embedded manifolds

We define an $n$-dimensional manifold embedded in $\mathbb{R}^{m}$ via a
projection function

$\operatorname{proj}:P\rightarrow\mathbb{R}^{m}$ (21)

where $P\subseteq\mathbb{R}^{m}$ is the projectable set. We require the
projection to have the following properties (the first is true of all
projections, the others are additional requirements):

  1. 1.

$\operatorname{proj}\circ\operatorname{proj}=\operatorname{proj}$

  2. 2.

$\operatorname{proj}$ is smooth on $P$

  3. 3.

$\operatorname{rank}(\operatorname{proj}^{\prime}(\operatorname{proj}(x)))=n$
for all $x\in P$

Given such a projection, we define a manifold by

$\mathcal{M}=\\{x\in\mathbb{R}^{m}:\operatorname{proj}(x)=x\\}$ (22)

with the tangent space

$\mathcal{T}_{x}\mathcal{M}=\operatorname{col}(\operatorname{proj}^{\prime}(x))$
(23)

where $\operatorname{col}$ denotes the column space. Since the rank of
$\operatorname{proj}^{\prime}(x)$ with $x\in\mathcal{M}$ is $n$, the tangent
space is $n$-dimensional and $\mathcal{M}$ is an $n$-dimensional manifold. The
Riemannian metric
$g_{x}:\mathcal{T}_{x}\mathcal{M}\times\mathcal{T}_{x}\mathcal{M}\rightarrow\mathbb{R}$
defined on this manifold is realized as an $m\times m$ matrix $G(x)$. If
$\mathcal{M}$ is isometrically embedded then $G(x)$ is just the identity
matrix.

The Jacobian of the projection is a projection matrix, meaning
$\operatorname{proj}^{\prime}(x)\operatorname{proj}^{\prime}(x)=\operatorname{proj}^{\prime}(x)$
for $x\in\mathcal{M}$. For any $v$ in the column space of
$\operatorname{proj}^{\prime}(x)$, there is a $u$ such that
$v=\operatorname{proj}^{\prime}(x)u$ and due to the projection property,
$\operatorname{proj}^{\prime}(x)v=\operatorname{proj}^{\prime}(x)u=v$.
Similarly, for any $w$ in the row space of $\operatorname{proj}^{\prime}(x)$,
$w\operatorname{proj}^{\prime}(x)=w$. If $\operatorname{proj}$ is an
orthogonal projection, $\operatorname{proj}^{\prime}$ is symmetric by
definition and hence the row and column spaces are identical.

#### Integration on embedded manifolds

In order to perform integration on the manifold, we cannot work directly in
the $m$-dimensional coordinates of the embedding space, instead we have to
introduce some local $n$-dimensional coordinates. This means that the domain
of integration has to be diffeomorphic to an open set in $\mathbb{R}^{n}$.
Since this might not be the case for the whole region of integration, we might
need to partition it into such regions and perform integration on each
individually (each such region, together with its map to $\mathbb{R}^{n}$, is
known as a chart and a collection of charts is an atlas). For example, if we
want to integrate a function on the sphere, we could split the sphere into two
hemispheres and integrate each separately. A hemisphere can be continuously
stretched and flattened into a 2-dimensional region, whereas the whole sphere
cannot without creating discontinuities.

Given an open set $U$ in $\mathbb{R}^{n}$, and a diffeomorphic local embedding
function $\phi:U\rightarrow\mathcal{M}$, the integral of a scalar function
$p:\mathcal{M}\rightarrow\mathbb{R}$ on $\phi(U)\subseteq\mathcal{M}$ is

$\int_{\phi(U)}pdV=\int_{U}(p\circ\phi)\sqrt{|\phi^{\prime}(u)^{T}G(\phi(u))\phi^{\prime}(u)|}du^{1}\cdots
du^{n}$ (24)

where $G(\phi(u))$ is the matrix representation of $g_{\phi(u))}$. The
integral on the right is an ordinary integral in $\mathbb{R}^{n}$. The
quantity inside the determinant is known as the pullback metric.

######  Theorem 1 (Manifold change of variables).

Let $(\mathcal{M}_{X},g_{X})$ and $(\mathcal{M}_{Z},g_{Z})$ be $n$-dimensional
Riemannian manifolds embedded in $\mathbb{R}^{m}$ with the same global
topological structure. Let $p_{X}$ be a probability distribution on
$\mathcal{M}_{X}$ and let $f:\mathcal{M}_{X}\rightarrow\mathcal{M}_{Z}$ be a
diffeomorphism. Let $p_{Z}$ be the pushforward of $p_{X}$ under $f$ (i.e. if
$p_{X}$ is the probability density of $X$, $p_{Z}$ is the probability density
of $f(X)$).

Let $x\in\mathcal{M}_{X}$. Let $Q\in\mathbb{R}^{m\times n}$ be an orthonormal
basis for $\mathcal{T}_{x}\mathcal{M}_{X}$ and $R\in\mathbb{R}^{m\times n}$ an
orthonormal basis for $\mathcal{T}_{f(x)}\mathcal{M}_{Z}$.

Then $p_{X}$ and $p_{Z}$ are related under the change of variables $x\mapsto
f(x)$ by:

$\log p_{X}(x)=\log p_{Z}(f(x))+\log|R^{T}f^{\prime}(x)Q|\\\
+\tfrac{1}{2}\log\frac{|R^{T}G_{Z}(f(x))R|}{|Q^{T}G_{X}(x)Q|}.$ (25)

Note that $Q$ and $R$ depend on $x$ and $f(x)$ respectively, but this is
omitted for brevity.

In the main text we state a simplified form of the equation:

$\log p_{X}(x)=\log
p_{Z}(f(x))+\log|f^{\prime}(x)|+\tfrac{1}{2}\log\frac{|G_{Z}(f(x))|}{|G_{X}(x)|}$
(26)

with the statement that the determinants have to be evaluated in the tangent
spaces, not in the ambient $m$-dimensional space. This is equivalent to the
statement of Theorem 1.

Below, we provide two versions of the proof, the second being a less rigorous
and more geometric variant of the first.

###### Proof.

Let $\phi:\mathbb{R}^{n}\rightarrow\mathcal{M}_{X}$ be defined by
$\phi(u)=\operatorname{proj}_{X}(x+Qu)$. Let $U$ be an open subset of
$\mathbb{R}^{n}$ containing the origin which is small enough so that $\phi$ is
bijective. Let $\psi:\mathbb{R}^{n}\rightarrow\mathcal{M}_{Z}$ be defined by
$\psi(w)=\operatorname{proj}_{Z}(f(x)+Rw)$. Define $\varphi=\psi^{-1}\circ
f\circ\phi$ and let $W=\varphi(U)$.

Note that $\phi^{\prime}(u)=\operatorname{proj}_{X}^{\prime}(x+Qu)\cdot Q$ and
hence $\phi^{\prime}(0)=\operatorname{proj}_{X}^{\prime}(x)Q=Q$ (since each
column of $Q$ is in
$\mathcal{T}_{x}\mathcal{M}_{X}=\operatorname{col}(\operatorname{proj}_{X}^{\prime}(x))$).

Similarly, $\psi^{\prime}(0)=R$. Since $\psi$ is a map from $n$ to $m$
dimensions, there is not a unique function from $\mathbb{R}^{m}$ to
$\mathbb{R}^{n}$ which is $\psi^{-1}$ on the manifold and there are remaining
degrees of freedom which can result in different Jacobians. For our purposes,
we define the inverse $\psi^{-1}$ such that $\psi\circ\psi^{-1}$ is an
orthogonal projection onto $\mathcal{M}_{Z}$. This means
$\psi^{\prime}(\psi^{-1}(f(x)))(\psi^{-1})^{\prime}(f(x))=RR^{T}$ and hence
$(\psi^{-1})^{\prime}(f(x))=R^{T}$.

Since $p_{Z}$ is the pushforward of of $p_{X}$ under $f$, the amount of
probability mass contained in $\phi(U)$ is the same as that contained in
$f(\phi(U))=\psi(W)$:

$\int_{\phi(U)}p_{X}(x)dV_{X}=\int_{\psi(W)}p_{Z}(z)dV_{Z}$ (27)

and therefore:

$\int_{U}p_{X}(\phi(u))\sqrt{|Q^{T}G_{X}(\phi(u))Q|}du^{1}\cdots du^{n}\\\
=\int_{W}p_{Z}(\psi(w))\sqrt{|R^{T}G_{Z}(\psi(w))R|}dw^{1}\cdots dw^{n}.$ (28)

Note that $Q$ and $R$ also depend on $u$. Changing variables of the RHS with
$w=\varphi(u)$ gives us

$\int_{U}p_{X}(\phi(u))\sqrt{|Q^{T}G_{X}(\phi(u))Q|}du^{1}\cdots du^{n}\\\
=\int_{U}p_{Z}(f(\phi(u)))\sqrt{|R^{T}G_{Z}(f(\phi(u)))R|}\cdot\left|\frac{\partial
w}{\partial u}\right|du^{1}\cdots du^{n}.$ (29)

Since $U$ was arbitrary, we can make it arbitrarily small, demonstrating that
the integrands must be equal for $u=0$:

$p_{X}(x)\sqrt{|Q^{T}G_{X}(x)Q|}\\\
=p_{Z}(f(x))\sqrt{|R^{T}G_{Z}(f(x))R|}\cdot\left|\frac{\partial w}{\partial
u}\right|.$ (30)

where now $Q$ is evaluated at $u=0$ and $R$ at $w=\varphi(0)$. Since
$w=\psi^{-1}(f(\phi(u)))$, the Jacobian has the following form when evaluated
at the origin (note $\phi(0)=x$):

$\displaystyle\frac{\partial w}{\partial u}$
$\displaystyle=(\psi^{-1})^{\prime}(f(x))\cdot
f^{\prime}(x)\cdot\phi^{\prime}(0)$ (31) $\displaystyle=R^{T}f^{\prime}(x)Q.$
(32)

Substituting this into the equality, rearranging and taking the logarithm
gives the result:

$\log p_{X}(x)=\log p_{Z}(f(x))+\log|R^{T}f^{\prime}(x)Q|\\\
+\tfrac{1}{2}\log\frac{|R^{T}G_{Z}(f(x))R|}{|Q^{T}G_{X}(x)Q|}.$ (33)

∎

#### Alternative proof

Here is a less rigorous and more geometric proof, which may be more intuitive
for some readers.

###### Proof.

Let $x$ be a point on $\mathcal{M}_{X}$. Consider a small square region
$U\subseteq\mathcal{M}$ around $x$ (hypercubic region in higher dimensions).
If the sides of the square are small enough, the square is approximately
tangent to the manifold since the manifold looks very flat if we zoom in.
Suppose $Q$ is a basis for the tangent space at $x$ and $q^{1},\dots,q^{n}$
are the columns of $Q$. Suppose that the sides of the square (or hypercube)
are spanned by $u^{i}=\epsilon q^{i}$ for a small $\epsilon$. The volume
spanned by a parallelotope (higher-dimensional analog of a parallelogram) is
the square root of the determinant of the Gram matrix of inner products:

$\operatorname{vol}(u^{1},\dots,u^{n})=\sqrt{|\langle u^{i},u^{j}\rangle|}.$
(34)

The inner product is given by $G$, namely $\langle u,v\rangle=u^{T}Gv$. We can
therefore write the volume of $U$ as

$\operatorname{vol}(U)\approx\epsilon^{n}\sqrt{|Q^{T}GQ|}.$ (35)

Now consider how $U$ is transformed under $f$. It will be mapped to a region
$f(U)$ on $\mathcal{M}_{z}$ with approximately straight edges, forming an
approximate parallelotope in the tangent space at $z=f(x)$. This region will
be spanned by the columns of $f^{\prime}(x)\epsilon Q$ (since
$f(x+u^{i})\approx f(x)+f^{\prime}(x)u^{i}$) and hence will have a volume of

$\displaystyle\operatorname{vol}(f(U))$
$\displaystyle\approx\epsilon^{n}\sqrt{|Q^{T}f^{\prime}(x)^{T}G_{Z}(z)f^{\prime}(x)Q|}$
(36)
$\displaystyle=\epsilon^{n}\sqrt{|Q^{T}f^{\prime}(x)^{T}RR^{T}G_{Z}(z)RR^{T}f^{\prime}(x)Q|}$
(37) $\displaystyle=\epsilon^{n}|R^{T}f^{\prime}(x)Q|\sqrt{|R^{T}G_{Z}(z)R|}$
(38)

where $R$ is a basis for the tangent space at $f(x)$. We can introduce
$RR^{T}$ into the expression since it is a projection in the tangent space at
$f(x)$ and is essentially the identity within that space. Since the RHS of
$G_{Z}(z)$ and the LHS of $f^{\prime}(x)$ both live in this tangent space, we
can introduce $RR^{T}$ between them without changing the expression. Then in
the last step we use that $|AB|=|A||B|$ for square matrices.

Since the probability mass in both regions should be the same we can write

$p_{X}(x)\operatorname{vol}(U)\approx p_{Z}(z)\operatorname{vol}(f(U))$ (39)

and therefore

$p_{X}(x)=p_{Z}(f(x))|R^{T}f^{\prime}(x)Q|\frac{\sqrt{|R^{T}G_{Z}(f(x))R|}}{\sqrt{|Q^{T}G_{X}(x)Q|}}$
(40)

where the approximation becomes exact by taking the limit of infinitesimally
small $\epsilon$. Taking the logarithm, we arrive at the result of Theorem 1.
∎

###  7.2 Loss function

In this section, we consider $f_{\theta}$ parameterized by $\theta$, but drop
the subscript for ease of presentation.

######  Theorem 2.

Let the assumptions of Theorem 1 hold and let
$g:\mathcal{M}_{Z}\rightarrow\mathcal{M}_{X}$ be the inverse of $f$. Then, the
derivative of the change of variables term has the following trace expression:

$\nabla_{\theta}\log|R^{T}f^{\prime}(x)Q|=\operatorname{tr}(R^{T}(\nabla_{\theta}f^{\prime}(x))g^{\prime}(f(x))R).$
(41)

###### Proof.

First, a reminder that $\varphi^{\prime}(u)=R^{T}f^{\prime}(x)Q$ with
$\varphi=\psi^{-1}\circ f\circ\phi$. Let $\chi=\varphi^{-1}$, i.e.
$\chi=\phi^{-1}\circ g\circ\psi$. Jacobi’s formula tells us that

$\frac{d}{dt}\log|A(t)|=\operatorname{tr}\left(\frac{dA(t)}{dt}A(t)^{-1}\right).$
(42)

Note also that since $\chi(\varphi(u))=u$, therefore
$\chi^{\prime}(\varphi(u))\varphi^{\prime}(u)=I$ and
$\chi^{\prime}(\varphi(u))=\varphi^{\prime}(u)^{-1}$. Applying Jacobi’s
formula to $\varphi^{\prime}(u)$:

$\displaystyle\nabla_{\theta}\log|\varphi^{\prime}(u)|$
$\displaystyle=\operatorname{tr}((\nabla_{\theta}\varphi^{\prime}(u))\varphi^{\prime}(u)^{-1})$
(43)
$\displaystyle=\operatorname{tr}((\nabla_{\theta}\varphi^{\prime}(u))\chi^{\prime}(\varphi(u)))$
(44)

and substituting in $f$ and $g$:

$\nabla_{\theta}\log|R^{T}f^{\prime}(x)Q|=\operatorname{tr}(\nabla_{\theta}(R^{T}f^{\prime}(x)Q)Q^{T}g^{\prime}(f(x))R).$
(45)

$Q$ does not depend on $\theta$, but $R$ depends on $f(x)$ and hence $\theta$,
so it must be considered in the derivative. However,

$\nabla_{\theta}\operatorname{tr}(RR^{T})=\operatorname{tr}((\nabla_{\theta}R)R^{T}+R\nabla_{\theta}R^{T})=2\operatorname{tr}(R\nabla_{\theta}R^{T})$
(46)

and since $\operatorname{tr}(RR^{T})=\operatorname{tr}(I)$ is a constant,
$\operatorname{tr}(R\nabla_{\theta}R^{T})=0$. Expanding Eq. 45:

$\nabla_{\theta}\log|R^{T}f^{\prime}(x)Q|=\operatorname{tr}(\nabla_{\theta}(R^{T})f^{\prime}(x)QQ^{T}g^{\prime}(f(x))R)\\\
+\operatorname{tr}(R^{T}\nabla_{\theta}(f^{\prime}(x))QQ^{T}g^{\prime}(f(x))R).$
(47)

Since $Q$ is an orthonormal basis for $\mathcal{T}_{x}\mathcal{M}_{X}$,
$QQ^{T}$ is a projection matrix onto $\mathcal{T}_{x}\mathcal{M}_{X}$. This is
because $(QQ^{T})^{2}=QQ^{T}QQ^{T}=QQ^{T}$, using $Q^{T}Q=I$. As a result,
$QQ^{T}\operatorname{proj}^{\prime}(x)=\operatorname{proj}^{\prime}(x)$. Since
$g$ is wrapped inside a projection:
$g(z)=\operatorname{proj}_{Z}(\tilde{g}(z))$, therefore
$g^{\prime}(z)=\operatorname{proj}_{Z}^{\prime}(\tilde{g}(z))\tilde{g}^{\prime}(z)$,
so $QQ^{T}g^{\prime}(z)=g^{\prime}(z)$. Note also that
$f^{\prime}(x)g^{\prime}(f(x))=I$ since $g$ is the inverse of $f$. This
simplifies the equation:

$\nabla_{\theta}\log|R^{T}f^{\prime}(x)Q|=\operatorname{tr}(\nabla_{\theta}(R^{T})R)\\\
+\operatorname{tr}(R^{T}\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))R)$ (48)

and finally

$\nabla_{\theta}\log|R^{T}f^{\prime}(x)Q|=\operatorname{tr}(R^{T}\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))R).$
(49)

∎

In the above proof we used the fact that $QQ^{T}g^{\prime}(z)=g^{\prime}(z)$.
Can we use $RR^{T}f^{\prime}(x)=f^{\prime}(x)$ to simplify the equation
further? No, we cannot, since the expression involving $f^{\prime}$ is
actually its derivative with respect to parameters, which may not have the
same matrix structure as $f^{\prime}$. Is it instead possible to use
$g^{\prime}(z)RR^{T}=g^{\prime}(z)$ for simplification? If $g$ is implemented
as $\operatorname{proj}_{Z}(\tilde{g}(z))$, this is not necessarily true, as
$g^{\prime}(z)$ might not be a map from the tangent space at $z$ to the
tangent space at $g(z)$. For example, if we add a small deviation $v$ to $z$,
where $v$ is orthogonal to the tangent space at $z$, then $g(z+v)$ might not
equal $g(z)$. However, this would mean that derivatives in the off-manifold
direction can be non-zero, meaning that $g^{\prime}(z)v\neq
g^{\prime}(z)RR^{T}v=0$ (since $RR^{T}$ will project $v$ to 0). We can change
this by prepending $g$ by a projection:

$g=\operatorname{proj}_{X}\circ\tilde{g}\circ\operatorname{proj}_{Z}.$ (50)

If $\operatorname{proj}_{Z}$ is an orthogonal projection, meaning that
$\operatorname{proj}_{Z}^{\prime}$ is symmetric, the column space and row
space of $\operatorname{proj}_{Z}$ will both be the same as those of $RR^{T}$,
meaning
$\operatorname{proj}_{Z}^{\prime}(z)RR^{T}=\operatorname{proj}_{Z}^{\prime}$
and hence $g^{\prime}(z)RR^{T}=g^{\prime}(z)$. This leads to the following
corollary:

######  Corollary 3.

Suppose the assumptions of Theorem 1 hold, and $g$ is the inverse of $f$ with
the following implementation:

$g=\operatorname{proj}_{X}\circ\tilde{g}\circ\operatorname{proj}_{Z}$ (51)

where $\operatorname{proj}_{Z}$ is an orthogonal projection. Then the
derivative of the change of variables term has the following trace expression:

$\nabla_{\theta}\log|R^{T}f^{\prime}(x)Q|=\operatorname{tr}((\nabla_{\theta}f^{\prime}(x))g^{\prime}(f(x))).$
(52)

###### Proof.

Take the result of Theorem 2 and use the cyclic property of the trace and the
properties of $g^{\prime}$ discussed above:

$\displaystyle\operatorname{tr}(R^{T}\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))R)$
$\displaystyle=\operatorname{tr}(\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))RR^{T})$
(53)
$\displaystyle=\operatorname{tr}(\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))).$
(54)

∎

We use Hutchinson-style trace estimators to approximate the traces given
above. This uses the property that, for a matrix $A\in\mathbb{R}^{n\times n}$
and a distribution $p(v)$ in $\mathbb{R}^{n}$ with unit second moment (meaning
$\mathbb{E}[vv^{T}]=I$),

$\displaystyle\mathbb{E}_{p(v)}[v^{T}Av]$
$\displaystyle=\operatorname{tr}(\mathbb{E}_{p(v)}[v^{T}Av])$ (55)
$\displaystyle=\operatorname{tr}(\mathbb{E}_{p(v)}[vv^{T}]A)$ (56)
$\displaystyle=\operatorname{tr}(A)$ (57)

meaning that $v^{T}Av\approx\operatorname{tr}(A)$ is an unbiased estimate of
the trace of $A$.

We have two variants of the trace estimate derived above, one evaluated in
$\mathbb{R}^{n}$, the other in $\mathbb{R}^{m}$. The first can be estimated
using the following equality:

$\displaystyle\operatorname{tr}(R^{T}$
$\displaystyle\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))R)$
$\displaystyle=\mathbb{E}_{p(u)}[u^{T}R^{T}\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))Ru]$
(58)
$\displaystyle=\mathbb{E}_{p(v)}[v^{T}\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x))v]$
(59)
$\displaystyle=\nabla_{\theta}\mathbb{E}_{p(v)}[v^{T}\nabla_{\theta}(f^{\prime}(x))\texttt{SG}[g^{\prime}(f(x))]v]$
(60)

where $p(u)$ has unit second moment in $\mathbb{R}^{n}$ and $p(v)$ is the
distribution of $Ru$, which lies in the tangent space at $x$ and has unit
second moment in that space by which we mean $\mathbb{E}[vv^{T}]=RR^{T}$. An
example of such a distribution is the standard normal projected to the tangent
space, i.e. $v=RR^{T}\tilde{v}$ where $\tilde{v}$ is standard normal.

In the second case, we can just sample from a distribution with unit second
moment in the embedding space $\mathbb{R}^{m}$:

$\operatorname{tr}(\nabla_{\theta}(f^{\prime}(x))g^{\prime}(f(x)))\\\
=\nabla_{\theta}\mathbb{E}_{p(v)}[v^{T}\nabla_{\theta}(f^{\prime}(x))\texttt{SG}[g^{\prime}(f(x))]v].$
(61)

###  7.3 Variance reduction

When using a Hutchinson trace estimator with standard normal $v\in R^{n}$, we
can reduce the variance of the estimate by scaling $v$ to have length
$\sqrt{n}$ (see [18]). The scaled variable will still have zero mean and unit
covariance so the estimate remains unbiased, but the variance is reduced, with
the effect especially pronounced in low dimensions.

While we can take advantage of this effect in both our options for trace
estimator, the effect is more pronounced in lower dimensions, so we reduce the
variance more by estimating the trace in an $n$-dimensional space rather than
an $m$-dimensional space. Hence the first version of the trace estimator,
where $v$ is sampled from a distribution in $\mathcal{T}_{x}\mathcal{M}_{X}$
is preferable in this regard.

Let’s provide some intuition with an example. Suppose $n=1$, $m=2$ and
$R=(1,0)^{T}$. We want to estimate the trace of $A=\operatorname{diag}(1,0)$.
Using the first estimator, we first sample $v=RR^{T}\tilde{v}$ with
$\tilde{v}$ standard normal which results in $v=(u,0)^{T}$ where
$u\in\mathbb{R}$ is standard normal. Then we scale $v$ so it has length
$\sqrt{n}=1$. This results in $v=(r,0)^{T}$ where $r$ is a Rademacher variable
(taking the value $-1$ and $1$ with equal probability). The trace estimate is
therefore $r^{2}=1$, meaning we always get the correct answer, so the variance
is zero. The second estimator samples $v$ directly from a 2d standard normal,
then scales it to have length $\sqrt{m}=\sqrt{2}$. Hence $v$ is sampled
uniformly from the circle with radius $\sqrt{2}$. We can write
$v=\sqrt{2}(\cos\theta,\sin\theta)^{T}$ with $\theta$ sampled uniformly in
$[0,2\pi]$. The estimate $v^{T}Av=2\cos^{2}\theta$. This is a random variable
whose mean is indeed 1 as required but has a nonzero variance, showing that
the variance is higher when estimating in the $m$-dimensional space.

For this reason, we choose the first estimator, sampling $v$ in the tangent
space at $x$. This also simplifies the definition of $g$, meaning that we
don’t have to prepend it with a projection.

##  8 Experimental details

Hyperparameter | Value
---|---
Layer type | ResNet
Residual blocks | 2
Inner depth | 5
Inner width | $512$
Activation | $\operatorname{ReLU}$
$\beta_{\text{R}}^{x}$ | $500$
$\beta_{\text{R}}^{z}$ | $0$
$\beta_{\text{U}}$ | $10$
$\beta_{\text{P}}$ | $10$
Latent distribution | uniform
Optimizer | Adam
Learning rate | $5\times 10^{-3}$
Scheduler |  Exponential w/ $\gamma=1-10^{-5}$
Gradient clipping | 1.0
Weight decay | $3\times 10^{-5}$
Batch size | 1,024
Step count | 585,600
Table 6: Hyperparameter choices for the rotation experiments. $\beta_{\text{U}}$ and $\beta_{\text{P}}$ are the same for both the sample and latent space. Dataset | Number of instances | Noise strength
---|---|---
Volcano | 827 | 0.008
Earthquake | 6120 | 0.0015
Flood | 4875 | 0.0015
Fire | 12809 | 0.0015
Table 7: Dataset overview for the earth data experiments. Each dataset is split into 80% for training, 10% for validation and 10% for testing. Hyperparameter | Value
---|---
Layer type | ResNet
residual blocks | 4
Inner depth | 2
Inner width | 256
Activation | $\sin$
$\beta^{x}_{\text{R}}$ | $10^{5}$
$\beta^{z}_{\text{R}}$ | 0
$\beta_{\text{U}}$ | $2\times 10^{2}$
$\beta_{\text{P}}$ | 0
Latent distribution |  VMF-Mixture ($n_{\text{comp}}=5$)
Optimizer | Adam
Learning rate | $2\times 10^{-4}$
Scheduler | onecyclelr
Gradient clipping | 10.0
Weight decay | $5\times 10^{-5}$
Batch size | 32
Step count |  $\sim$ 1.2M
Table 8: Hyperparameter choices for the earth data experiments. $\beta_{\text{U}}$ and $\beta_{\text{P}}$ are the same for both the sample and latent space. Dataset | Number of instances | Noise strength
---|---|---
General | $138208$ | $0$
Glycine | $13283$ | $0$
Proline | $7634$ | $0$
Pre-Proline | $6910$ | $0$
RNA | $9478$ | $1\times 10^{-2}$
Table 9: Details on the torus datasets. Each dataset is randomly split into a train dataset ($80\%$), validation dataset ($10\%$) and test dataset ($10\%$). During training, we add Gaussian noise with mean zero and standard deviation given by ‘noise strength’ to the data, to counteract overfitting. Hyperparameter |  Value ($\mathbb{T}^{2}$) |  Value ($\mathbb{T}^{7}$)
---|---|---
Layer type | ResNet | ResNet
residual blocks | 6 | 2
Inner depth | 2 | 2
Inner width | $256$ | $256$
Activation | SiLU | SiLU
$\beta_{\text{R}}^{x}$ | $100$ | $1000$
$\beta_{\text{R}}^{z}$ | 100 | $100$
$\beta_{\text{U}}^{x}$ | $100$ | $100$
$\beta_{\text{U}}^{z}$ | $0$ | $1000$
$\beta_{\text{P}}$ | 0 | 0
Latent distribution | uniform | uniform
Optimizer | Adam | Adam
Learning rate | $1\times 10^{-3}$ | $1\times 10^{-3}$
Scheduler | oneclyclelr | onecyclelr
Gradient clipping | - | -
Weight decay | $1\times 10^{-3}$ | $1\times 10^{-3}$
Batch size | 512 | 512
Step count |  $\sim$ 60k |  $\sim$ 120k
Table 10: Details on the model architecture, loss weights and optimizer
parameters for the torus datasets. We use the same configuration for all
protein datasets on $\mathbb{T}^{2}$.

In accordance with the details provided in Sec. 4.3, our approach incorporates
multiple regularization loss components in addition to the negative log-
likelihood objective. This results in the final loss expression:

$\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{\text{NLL}}+\beta_{\text{R}}^{x/z}\mathcal{L}_{\text{R}}^{x/z}+\beta_{\text{U}}^{x/z}\mathcal{L}_{\text{U}}^{x/z}+\beta_{\text{P}}^{x/z}\mathcal{L}_{\text{P}}^{x/z}.$
(62)

For each of the terms, there is a variant in $x$\- and in $z$-space, as
indicated by the superscript. In detail:

The first loss $\mathcal{L}_{\text{R}}$ represents the reconstruction loss:

$\displaystyle\mathcal{L}_{\text{R}}^{x}$ $\displaystyle=\mathbb{E}_{x\sim
p_{\text{data}}}[d(x,g_{\phi}(f_{\theta}(x))],$ (63)
$\displaystyle\mathcal{L}_{\text{R}}^{z}$ $\displaystyle=\mathbb{E}_{x\sim
p_{\text{data}}}[d(f_{\theta}(x),f_{\theta}(g_{\phi}(f_{\theta}(x)))].$ (64)

Here $d(x,y)=\|x-y\|^{2}$ is the standard reconstruction loss in the embedding
space. This could be replaced with a distance on the manifold. However, this
would be more expensive to compute and since we initialize networks close to
identity, the distance in the embedding space is almost equal to shortest path
length on the manifold.

In order to have accurate reconstructions outside of training data, we also
add reconstruction losses for data uniformly sampled on the manifold, both for
$x$ and $z$:

$\displaystyle\mathcal{L}_{\text{U}}^{x}$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{U}(\mathcal{M})}[d(x,g_{\phi}(f_{\theta}(x))],$
(65) $\displaystyle\mathcal{L}_{\text{U}}^{z}$
$\displaystyle=\mathbb{E}_{x\sim\mathcal{U}(\mathcal{M})}[d(z,f_{\theta}(g_{\phi}(z))].$
(66)

Finally, we make sure that the function learned by the neural networks is easy
to project by regularizing the distance between the raw outputs by the neural
networks in the embedding space and the subsequent projection to the manifold
(compare Eq. 9):

$\displaystyle\mathcal{L}_{\text{P}}^{x}$ $\displaystyle=\mathbb{E}_{x\sim
p_{\text{data}}}[\|g_{\phi}(f_{\theta}(x))-\tilde{g}_{\phi}(f_{\theta}(x))\|^{2}],$
(67) $\displaystyle\mathcal{L}_{\text{P}}^{z}$
$\displaystyle=\mathbb{E}_{x\sim
p_{\text{data}}}[\|f_{\theta}(x)-\tilde{f}_{\theta}(x)\|^{2}].$ (68)

If these superscripts are not specified explicitly in the following summary of
experimental details, we mean $\beta^{x}=\beta^{z}$.

###  8.1 Special orthogonal group

The special orthogonal group data set is synthetically generated. We refer to
[10] for a description of the data generation process. They use an infinite
stream of samples. To emulate this, we generate a data set of $10^{7}$ samples
from their code, of which we reserve 1,000 for validation during training and
5,000 for testing. We vectorize the $3\times 3$ matrices before passing them
into the fully-connected networks. All training details are given in Tab. 6.

###  8.2 Earth data

We follow previous works and use a dataset split of 80% for training, 10% for
validation and 10% for testing. For the earth datasets we use a mixture of 5
learnable Von-Mises-Fisher distributions for the target latent distribution.
We base our implementation on the hyperspherical_vae library [9]. In order to
stabilize training we apply a small amount of Gaussian noise to every batch
(see table 7) and project the resulting data point back onto the sphere. Other
training hyperparameters can be found in table 8.

###  8.3 Tori

The torus datasets are randomly split into a train dataset (80%), validation
dataset (10%) and test dataset (10%). To counteract overfitting, we augment
the RNA dataset with random Gaussian noise. The noise strength and total
number of instances is reported in Tab. 9. We use a uniform latent
distribution. While we train for $\sim$ 60k steps on the protein datasets
(General, Glycine, Proline and Pre-Proline), we train for $\sim$ 120k steps on
the RNA datasets, both with a batch size of 512. Further hyperparameters used
in training can be found in Tab. 10.

###  8.4 Libraries

We base our code on PyTorch [40], PyTorch Lightning [13], Lightning Trainable
[25], Numpy [19], Matplotlib [21] for plotting and Pandas [31, 47] for data
evaluation. We use the geomstats [32, 33] package for embeddings and
projections.


# Bridging Bayesian, frequentist and fiducial (BFF) inferences using
confidence distribution

Suzanne Thornton and Min-ge Xie

1212footnotetext:  Email addresses<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>The research is supported in part by NSF research
grants: DMS1737857, DMS1812048, DMS2015373 and DMS2027855


# Minimality conditions equivalent to the finitude of Fermat and Mersenne
primes

Menachem Shlossberg  School of Computer Science Reichman University, 4610101
Herzliya Israel<EMAIL_ADDRESS>
###### Abstract.

It is still open whether there exist infinitely many Fermat primes or
infinitely many composite Fermat numbers. The same question concerning the
Mersenne numbers is also unsolved. Extending some results from [9], we
characterize the Fermat primes and the Mersenne primes in terms of topological
minimality of some matrix groups. This is done by showing, among other things,
that if ${\mathbb{F}}$ is a subfield of a local field of characteristic $\neq
2$, then the special upper triangular group
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal precisely when the special
linear group $\operatorname{SL}(n,{\mathbb{F}})$ is. We provide criteria for
the minimality (and total minimality) of $\operatorname{SL}(n,{\mathbb{F}})$
and $\operatorname{ST^{+}}(n,{\mathbb{F}}),$ where ${\mathbb{F}}$ is a
subfield of ${\mathbb{C}}.$

Let $\mathcal{F}_{\pi}$ and $\mathcal{F}_{c}$ be the set of Fermat primes and
the set of composite Fermat numbers, respectively. As our main result, we
prove that the following conditions are equivalent for
$\mathcal{A}\in\\{\mathcal{F}_{\pi},\mathcal{F}_{c}\\}:$

  * •

$\mathcal{A}$ is finite;

  * •

$\prod_{F_{n}\in\mathcal{A}}\operatorname{SL}(F_{n}-1,{\mathbb{Q}}(i))$ is
minimal, where ${\mathbb{Q}}(i)$ is the Gaussian rational field;

  * •

$\prod_{F_{n}\in\mathcal{A}}\operatorname{ST^{+}}(F_{n}-1,{\mathbb{Q}}(i))$ is
minimal.

Similarly, denote by $\mathcal{M}_{\pi}$ and $\mathcal{M}_{c}$ the set of
Mersenne primes and the set of composite Mersenne numbers, respectively, and
let $\mathcal{B}\in\\{\mathcal{M}_{\pi},\mathcal{M}_{c}\\}.$ Then the
following conditions are equivalent:

  * •

$\mathcal{B}$ is finite;

  * •

$\prod_{M_{p}\in\mathcal{B}}\operatorname{SL}(M_{p}+1,{\mathbb{Q}}(i))$ is
minimal;

  * •

$\prod_{M_{p}\in\mathcal{B}}\operatorname{ST^{+}}(M_{p}+1,{\mathbb{Q}}(i))$ is
minimal.

###### Key words and phrases:

Fermat primes, Fermat numbers, Mersenne primes, minimal group, special linear
group, Gaussian rational field.

###### 2020 Mathematics Subject Classification:

11A41, 11Sxx, 20H20, 54H11, 54H13

##  1\. Introduction

A Fermat number has the form $F_{n}=2^{2^{n}}+1,$ where $n$ is a non-negative
integer while a Mersenne number has the form $M_{p}=2^{p}-1$ for some prime
$p.$ Note that $2^{n}-1$ is composite when $n$ is composite. In other words, a
Mersenne prime is a prime number that is one less than a power of two. There
are several open problems concerning these numbers (e.g., see [6]). For
example, it is still unknown whether the Fermat primes, the composite Fermat
numbers, the Mersenne primes or the composite Mersenne numbers are infinitely
many.

All topological groups in this paper are Hausdorff. Let ${\mathbb{F}}$ be a
topological subfield of a local field. Recall that a _local field_ is a non-
discrete locally compact topological field. Denote by
$\operatorname{SL}(n,{\mathbb{F}})$ the special linear group over
${\mathbb{F}}$ of degree $n$ equipped with the pointwise topology inherited
from ${\mathbb{F}}^{n^{2}},$ and by $\operatorname{ST^{+}}(n,{\mathbb{F}})$
its topological subgroup consisting of upper triangular matrices.
Megrelishvili and the author characterized the Fermat primes in terms of the
topological minimality of some special linear groups. Recall that a
topological group $G$ is _minimal_ [5, 12] if every continuous isomorphism
$f\colon G\to H$, with $H$ a topological group, is a topological isomorphism
(equivalently, if $G$ does not admit a strictly coarser group topology).

######  Theorem 1.1.

[9, Theorem 5.5] For an odd prime $p$ the following conditions are equivalent:

  1. (1)

$p$ is a Fermat prime;

  2. (2)

$\operatorname{SL}(p-1,({\mathbb{Q}},\tau_{p}))$ is minimal, where
$({\mathbb{Q}},\tau_{p})$ is the field of rationals equipped with the $p$-adic
topology;

  3. (3)

$\operatorname{SL}(p-1,{\mathbb{Q}}(i))$ is minimal, where
${\mathbb{Q}}(i)\subset{\mathbb{C}}$ is the Gaussian rational field.

A similar characterization for the Mersenne primes is provided in Theorem 3.1.
Note that it follows from Gauss–Wantzel Theorem that an odd prime $p$ is a
Fermat prime if and only if a $p$-sided regular polygon can be constructed
with compass and straightedge.

We prove in Theorem 2.7 that if ${\mathbb{F}}$ is a subfield of a local field
of characteristic distinct than $2$, then the special upper triangular group
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal if and only if the special
linear group $\operatorname{SL}(n,{\mathbb{F}})$ is minimal. This result with
some other tools yield criteria for the minimality (and total minimality) of
$\operatorname{SL}(n,{\mathbb{F}})$ and
$\operatorname{ST^{+}}(n,{\mathbb{F}}),$ where ${\mathbb{F}}$ is a subfield of
${\mathbb{C}}$ (see Proposition 2.12, Remark 2.13 and Corollary 2.14).

As a main result, we prove in Theorem 3.6 that the finitude of Fermat and
Mersenne primes as well as the finitude of composite Fermat and Mersenne
numbers is equivalent to the minimality of some topological products of some
matrix groups.

##  2\. Minimality of $\operatorname{ST^{+}}(n,{\mathbb{F}})$ and
$\operatorname{SL}(n,{\mathbb{F}})$

Let $\mathsf{N}:=\operatorname{UT}(n,{\mathbb{F}})$ and $\mathsf{A}$ be the
subgroups of $\operatorname{ST^{+}}(n,{\mathbb{F}})$ consisting of upper
unitriangular matrices and diagonal matrices, respectively. Note that
$\mathsf{N}$ is normal in $\operatorname{ST^{+}}(n,{\mathbb{F}})$ and
$\operatorname{ST^{+}}(n,{\mathbb{F}})\cong\mathsf{N}\rtimes_{\alpha}\mathsf{A}$,
where $\alpha$ is the action by conjugations. It is known that $\mathsf{N}$ is
the derived subgroup of $\operatorname{ST^{+}}(n,{\mathbb{F}}).$ Recall also
that $\operatorname{SL}(n,{\mathbb{F}})$ has finite center (e.g., see [11,
3.2.6])

$Z(\operatorname{SL}(n,{\mathbb{F}}))=\\{\lambda I_{n}:\lambda\in\mu_{n}\\},$

where $\mu_{n}$ is a finite group consisting of the $n$-th roots of unity in
${\mathbb{F}}$ and $I_{n}$ is the identity matrix of size $n.$

######  Lemma 2.1.

Let ${\mathbb{F}}$ be a field and $n\in{\mathbb{N}}.$ Then
$Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))=Z(\operatorname{SL}(n,{\mathbb{F}}))$
and
$\operatorname{ST^{+}}(n,{\mathbb{F}})/Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))$
is center-free.

###### Proof.

Let $(C,D)\in Z(\mathsf{N}\rtimes\mathsf{A})$ and $E\in\mathsf{A}$ such that
$e_{ii}\neq e_{jj}$ whenever $i\neq j.$ For every $i>j$ it holds that

$(CE)_{ij}=\sum_{t=1}^{n}c_{it}e_{tj}=c_{ij}e_{jj}$

and

$(EC)_{ij}=\sum_{t=1}^{n}e_{it}c_{tj}=c_{ij}e_{ii}$

Then, the equality $(C,D)(I,E)=(I,E)(C,D)$ implies that $CE=EC$ and
$c_{ij}e_{jj}=c_{ij}e_{ii}$. As $e_{ii}\neq e_{jj}$ we deduce that $c_{ij}=0.$
Since $C$ is an upper unitriangular matrix it follows that $C=I.$ To prove
that
$Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))=Z(\operatorname{SL}(n,{\mathbb{F}}))$
it suffices to show that the diagonal matrix $D$ is scalar. To this aim, pick
distinct indices $i,j$ and a matrix $F\in\mathsf{N}$ such that $f_{ij}\neq 0.$
As $(I,D)\in Z(\mathsf{N}\rtimes\mathsf{A}),$ it follows that
$(I,D)(F,I)=(F,I)(I,D).$ This implies that $DF=FD$ and, in particular,
$(DF)_{ij}=(FD)_{ij}.$ This yields the equality $d_{ii}f_{ij}=d_{jj}f_{ij}$
since $D$ is diagonal. We conclude that $d_{ii}=d_{jj}$ in view of the
inequality $f_{ij}\neq 0.$ This proves that
$Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))=Z(\operatorname{SL}(n,{\mathbb{F}})).$
Now, let $(B,D)Z(\mathsf{N}\rtimes\mathsf{A})\in
Z(\mathsf{N}\rtimes\mathsf{A}/Z(\mathsf{N}\rtimes\mathsf{A}))$ and
$(C,E)\in\mathsf{N}\rtimes\mathsf{A}.$ By what we proved, there exists a
scalar $\lambda\in{\mathbb{F}}$ such that $(B,D)(C,E)=(C,E)(B,D)(I,\lambda
I).$ Therefore, $DE=\lambda DE$ and $\lambda=1.$ This means that
$(B,D)(C,E)=(C,E)(B,D)$ for every $(C,E)\in\mathsf{N}\rtimes\mathsf{A}.$
Therefore, $(\mathsf{N}\rtimes\mathsf{A})/Z(\mathsf{N}\rtimes\mathsf{A})$ and
its isomorphic copy
$\operatorname{ST^{+}}(n,{\mathbb{F}})/Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))$
are center-free. ∎

The following lemma will be useful in proving Theorem 2.7.

######  Lemma 2.2.

Let ${\mathbb{F}}$ be a subfield of a field $H$ and let $n\geq 3$ be a natural
number. If $L$ is a normal subgroup of $\operatorname{ST^{+}}(n,H)$ that
intersects $\operatorname{UT}(n,H)$ non-trivially, then it intersects
$\operatorname{UT}(n,{\mathbb{F}})$ non-trivially.

###### Proof.

Since $L\cap\operatorname{UT}(n,H)$ is a non-trivial normal subgroup of the
nilpotent group $\operatorname{UT}(n,H)$ it must non-trivially intersects the
center $Z(\operatorname{UT}(n,H)).$ Then there exists

$I\neq B=\left(\begin{array}[]{ccccc}1&0&\ldots&0&b\\\ 0&1&\ddots&\vdots&0\\\
\vdots&\ddots&\ddots&\ddots&\vdots\\\ \vdots&\ldots&0&1&0\\\
0&\ldots&\ldots&0&1\end{array}\right)\in L\cap Z(\operatorname{UT}(n,H))$

for some $b\in H$ (see [14, p. 94] for example). Since $n\geq 3$ there exists
a diagonal matrix $D\in\operatorname{ST^{+}}(n,H)$ such that $d_{11}=b^{-1}$
and $d_{nn}=1$. This implies that

$I\neq DBD^{-1}\in Z(\operatorname{UT}(n,{\mathbb{F}})).\qed$

######  Definition 2.3.

Let $H$ be a subgroup of a topological group $G$. Then $H$ is essential in $G$
if $H\cap L\neq\\{e\\}$ for every non-trivial closed normal subgroup $L$ of
$G$.

The following minimality criterion of dense subgroups is well-known (for
compact $G$ see also [10, 12]).

######  Fact 2.4.

[2, Minimality Criterion] Let $H$ be a dense subgroup of a topological group
$G.$ Then $H$ is minimal if and only if $G$ is minimal and $H$ is essential in
$G$.

######  Remark 2.5.

If ${\mathbb{F}}$ is a subfield of a local field $P,$ then its completion
$\widehat{\mathbb{F}}$ is a topological field that can be identified with the
closure of ${\mathbb{F}}$ in $P$. In case ${\mathbb{F}}$ is infinite then
$\widehat{\mathbb{F}}$ is also a local field, as the local field $P$ contains
no infinite discrete subfields (see [7, p. 27]).

######  Proposition 2.6.

[9, Proposition 5.1] Let ${\mathbb{F}}$ be a subfield of a local field. Then
the following conditions are equivalent:

  1. (1)

$\operatorname{SL}(n,{\mathbb{F}})$ is minimal;

  2. (2)

any non-trivial central subgroup of
$\operatorname{SL}(n,\widehat{{\mathbb{F}}})$ intersects
$\operatorname{SL}(n,{\mathbb{F}})$ non-trivially (i.e., if
$1\neq\lambda\in\mu_{n}(\widehat{{\mathbb{F}}})$, then there exists
$k\in{\mathbb{Z}}$ such that $1\neq\lambda^{k}\in{\mathbb{F}}$).

######  Theorem 2.7.

Let ${\mathbb{F}}$ be a subfield of a local field of characteristic distinct
than $2.$ Then, $\operatorname{SL}(n,{\mathbb{F}})$ is minimal if and only if
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal.

###### Proof.

Without loss of generality we may assume that ${\mathbb{F}}$ is infinite.
Suppose first that $\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal. By
Lemma 2.1,
$Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))=Z(\operatorname{SL}(n,{\mathbb{F}}))$.
Since this center is finite it follows from the Minimality Criterion that any
non-trivial central subgroup of
$\operatorname{ST^{+}}(n,\widehat{{\mathbb{F}}})$ intersects
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ non-trivially. This implies that any
non-trivial central subgroup of $\operatorname{SL}(n,\widehat{{\mathbb{F}}})$
intersects $\operatorname{SL}(n,{\mathbb{F}})$ non-trivially. By Proposition
2.6, $\operatorname{SL}(n,{\mathbb{F}})$ is minimal.

Conversely, let us assume that $\operatorname{SL}(n,{\mathbb{F}})$ is minimal.
In case $n=2$ then $\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal by [9,
Theorem 3.4] as an infinite subfield of a local field is locally retrobounded
and non-discrete. So, we may assume that $n\geq 3.$ By [9, Theorem 3.19],
$\operatorname{ST^{+}}(n,\widehat{{\mathbb{F}}})$ is minimal as
$\widehat{{\mathbb{F}}}$ is a local field (see Remark 2.5). In view of the
Minimality Criterion, it suffices to show that
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ is essential in
$\operatorname{ST^{+}}(n,\widehat{{\mathbb{F}}})$. Let $L$ be closed normal
non-trivial subgroup of $\operatorname{ST^{+}}(n,\widehat{{\mathbb{F}}})$. If

$L\subseteq
Z(\operatorname{ST^{+}}(n,\widehat{{\mathbb{F}}}))=Z(\operatorname{SL}(n,\widehat{{\mathbb{F}}})),$

then $L$ intersects $\operatorname{SL}(n,{\mathbb{F}})$ non-trivially by
Proposition 2.6. Clearly, this implies that $L$ intersects
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ non-trivially. If $L$ is not central,
then it must non-trivially intersect
$\operatorname{UT}(n,\widehat{{\mathbb{F}}}),$ the derived subgroup of
$\operatorname{ST^{+}}(n,\widehat{{\mathbb{F}}})$, in view of [15, Lemma 2.3].
Now, Lemma 2.2 implies that $L$ intersects
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ non-trivially and we deduce that
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ is essential in
$\operatorname{ST^{+}}(n,\widehat{{\mathbb{F}}})$. ∎

In view of Theorem 1.1 and Theorem 2.7, the following characterization of the
the Fermat primes is obtained.

######  Theorem 2.8.

For an odd prime $p$ the following conditions are equivalent:

  1. (1)

$p$ is a Fermat prime;

  2. (2)

$\operatorname{ST^{+}}(p-1,({\mathbb{Q}},\tau_{p}))$ is minimal;

  3. (3)

$\operatorname{ST^{+}}(p-1,{\mathbb{Q}}(i))$ is minimal.

The following concept has a key role in the Total Minimality Criterion.

######  Definition 2.9.

A subgroup $H$ of a topological group $G$ is totally dense if for every closed
normal subgroup $L$ of $G$ the intersection $L\cap H$ is dense in $L.$

######  Fact 2.10.

[3, Total Minimality Criterion] Let $H$ be a dense subgroup of a topological
group $G$. Then $H$ is totally minimal if and only if $G$ is totally minimal
and $H$ is totally dense in $G$.

######  Theorem 2.11.

[9, Theorem 4.7] Let ${\mathbb{F}}$ be a subfield of a local field. Then
$\operatorname{SL}(n,{\mathbb{F}})$ is totally minimal if and only if
$Z(\operatorname{SL}(n,{\mathbb{F}}))=Z(\operatorname{SL}(n,\widehat{{\mathbb{F}}}))$
(i.e., $\mu_{n}({\mathbb{F}})=\mu_{n}(\widehat{{\mathbb{F}}})$).

Let $\rho_{m}=e^{\frac{2\pi i}{m}}$ be the $m$-th primitive root of unity. The
next result extends [9, Corollary 5.3], where $\rho_{4}=i$ is considered.

######  Proposition 2.12.

Let ${\mathbb{F}}$ be a dense subfield of ${\mathbb{C}}.$ Then,

  1. (1)

$\operatorname{SL}(n,{\mathbb{F}})$ is totally minimal if and only if
$\rho_{n}\in{\mathbb{F}};$

  2. (2)

$\operatorname{SL}(n,{\mathbb{F}})$ is minimal if and only if
$\langle\rho_{m}\rangle\cap{\mathbb{F}}$ is non-trivial whenever $m$ divides
$n.$

###### Proof.

(1) Necessity: follows from Theorem 2.11. Indeed,
$\lambda=\rho_{n}\in{\mathbb{C}}$ is an $n$-th root of unity.

Sufficiency: if $\lambda\in{\mathbb{C}}$ and $\lambda^{n}=1$, then
$\lambda\in\langle\rho_{n}\rangle\subseteq{\mathbb{F}}.$ So, we may use
Theorem 2.11 again.

(2) Necessity: Let $1\neq\lambda\in{\mathbb{C}}$ be an $n$-th root of unity.
Then $\lambda$ is an $m$-th primitive root of unity where $m$ divides $n.$
Since $\operatorname{SL}(n,{\mathbb{F}})$ is minimal it follows that there
exists $k$ such that $1\neq\lambda^{k}\in{\mathbb{F}}.$ Clearly,
$\lambda^{k}\in\langle\rho_{m}\rangle\cap{\mathbb{F}}.$ So,
$\langle\rho_{m}\rangle\cap{\mathbb{F}}$ is non-trivial. Now use Proposition
2.6.

Sufficiency: Let $1\neq\lambda\in{\mathbb{C}}$ be an $n$-th root of unity.
Then $\lambda$ is an $m$-th primitive root of unity where $m$ divides $n.$
This means that $\lambda=e^{\frac{2\pi ik}{m}}=(\rho_{m})^{k}$, where $1\leq
k\leq m$ with $\gcd(k,m)=1.$ By our assumption,
$\langle\rho_{m}\rangle\cap{\mathbb{F}}$ is non-trivial. Hence, there exists
$l$ such that $1\neq(\rho_{m})^{l}\in\langle\rho_{m}\rangle\cap{\mathbb{F}}.$
Since $\gcd(k,m)=1$ and $\lambda=(\rho_{m})^{k}$, it follows that there exists
$t\in{\mathbb{Z}}$ such that $(\rho_{m})^{l}=\lambda^{t}.$ This proves the
minimality of $\operatorname{SL}(n,{\mathbb{F}}),$ in view of Proposition 2.6.
∎

######  Remark 2.13.

It is known that a subfield ${\mathbb{F}}$ is dense in ${\mathbb{C}}$ if and
only if it is not contained in ${\mathbb{R}}.$ By [9, Corollary 4.8], if
${\mathbb{F}}\subseteq{\mathbb{R}},$ then $\operatorname{SL}(n,{\mathbb{F}})$
is totally minimal for every $n\in{\mathbb{N}}$. So, together with Proposition
2.12 we obtain criteria for the minimality and total minimality of
$\operatorname{SL}(n,{\mathbb{F}}),$ where ${\mathbb{F}}$ is any subfield of
${\mathbb{C}}$ and $n\in{\mathbb{N}}.$

Since ${\mathbb{C}}$ has zero characteristic, Theorem 2.7, Proposition 2.12
and Remark 2.13 imply:

######  Corollary 2.14.

Let ${\mathbb{F}}$ be a topological subfield of ${\mathbb{C}}.$

  1. (1)

If ${\mathbb{F}}$ is dense in ${\mathbb{C}}$, then
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal if and only if
$\langle\rho_{m}\rangle\cap{\mathbb{F}}$ is non-trivial whenever $m$ divides
$n.$

  2. (2)

If ${\mathbb{F}}\subseteq{\mathbb{R}}$, then
$\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal for every
$n\in{\mathbb{N}}$.

##  3\. Proof of the main result

By [9, Corollary 5.3], $\operatorname{SL}(n,{\mathbb{Q}}(i))$ is minimal if
and only if $n=2^{k},$ where $k$ is a non-negative integer. This immediately
implies the following theorem concerning the Mersenne primes (compare with
Theorem 1.1 and Theorem 2.8).

######  Theorem 3.1.

For a prime $p$ the following conditions are equivalent:

  1. (1)

$p$ is a Mersenne prime;

  2. (2)

$\operatorname{SL}(p+1,{\mathbb{Q}}(i))$ is minimal;

  3. (3)

$\operatorname{ST^{+}}(p+1,{\mathbb{Q}}(i))$ is minimal.

######  Theorem 3.2.

  1. (1)

If ${\mathbb{F}}$ is a local field, then
$\prod_{n\in{\mathbb{N}}}\operatorname{SL}(n,{\mathbb{F}})$ is minimal.

  2. (2)

If, in addition, $\mathrm{char}({\mathbb{F}})\neq 2,$ then
$\prod_{n\in{\mathbb{N}}}\operatorname{ST^{+}}(n,{\mathbb{F}})$ is minimal.

###### Proof.

(1) Since a compact group is minimal we may assume without loss of generality
that ${\mathbb{F}}$ is infinite. By [1] (see also [9, Theorem 4.3]), the
projective special linear group
$\operatorname{PSL}(n,{\mathbb{F}})=\operatorname{SL}(n,{\mathbb{F}})/Z(\operatorname{SL}(n,{\mathbb{F}}))$
(equipped with the quotient topology) is minimal for every $n\in{\mathbb{N}}.$
Being algebraically simple (see [11, 3.2.9]),
$\operatorname{PSL}(n,{\mathbb{F}})$ has trivial center. Therefore, the
topological product
$\prod_{n\in{\mathbb{N}}}\operatorname{PSL}(n,{\mathbb{F}})$ is minimal by [8,
Theorem 1.15]. As

$\prod_{n\in{\mathbb{N}}}\operatorname{PSL}(n,{\mathbb{F}})\cong\prod_{n\in{\mathbb{N}}}\operatorname{SL}(n,{\mathbb{F}})/Z\Big{(}\prod_{n\in{\mathbb{N}}}\operatorname{SL}(n,{\mathbb{F}})\Big{)},$

where
$Z\Big{(}\prod_{n\in{\mathbb{N}}}\operatorname{SL}(n,{\mathbb{F}})\Big{)}$ is
compact, it follows from [4, Theorem 7.3.1] that

$\prod_{n\in{\mathbb{N}}}\operatorname{SL}(n,{\mathbb{F}})$ is minimal.
(2) By Lemma 2.1, the center of
$\operatorname{ST^{+}}(n,{\mathbb{F}})/Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))$
is trivial for every $n\in{\mathbb{N}}$, where
$Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))=Z(\operatorname{SL}(n,{\mathbb{F}})).$
By [9, Theorem 3.17],
$\operatorname{ST^{+}}(n,{\mathbb{F}})/Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))$
is minimal. We complete the proof using the topological isomorphism

$\prod_{n\in{\mathbb{N}}}\operatorname{ST^{+}}(n,{\mathbb{F}})/\prod_{n\in{\mathbb{N}}}Z(\operatorname{ST^{+}}(n,{\mathbb{F}}))\cong\prod_{n\in{\mathbb{N}}}(\operatorname{ST^{+}}(n,{\mathbb{F}})/Z(\operatorname{ST^{+}}(n,{\mathbb{F}})))$

and similar arguments to those appearing in the proof of (1). ∎

######  Definition 3.3.

[13] A minimal group $G$ is perfectly minimal if $G\times H$ is minimal for
every minimal group $H.$

######  Proposition 3.4.

Let ${\mathbb{F}}$ be a subfield of a local field. Then
$\operatorname{SL}(2^{k},{\mathbb{F}})$ is perfectly minimal for every
$k\in{\mathbb{N}}.$ If $\mathrm{char}({\mathbb{F}})\neq 2,$ then
$\operatorname{ST^{+}}(2^{k},{\mathbb{F}})$ is perfectly minimal for every
$k\in{\mathbb{N}}.$

###### Proof.

Let ${\mathbb{F}}$ be a subfield of a local field and $k\in{\mathbb{N}}.$ By
[9, Corollary 5.2], $\operatorname{SL}(2^{k},{\mathbb{F}})$ is minimal. Being
finite, the center $Z(\operatorname{SL}(2^{k},{\mathbb{F}}))$ is perfectly
minimal (see [5]). Having perfectly minimal center, the minimal group
$\operatorname{SL}(2^{k},{\mathbb{F}})$ is perfectly minimal in view of [8,
Theorem 1.4]. The last assertion is proved similarly, taking into account that
$Z(\operatorname{SL}(2^{k},{\mathbb{F}}))=Z(\operatorname{ST^{+}}(2^{k},{\mathbb{F}}))$
and the fact that $\operatorname{ST^{+}}(2^{k},{\mathbb{F}})$ is minimal by
Theorem 2.7. ∎

######  Theorem 3.5.

Let $(n_{k})_{k\in{\mathbb{N}}}$ be an increasing sequence of natural numbers.
Then, neither
$\prod_{k\in{\mathbb{N}}}\operatorname{SL}(2^{n_{k}},{\mathbb{Q}}(i))$ nor
$\prod_{k\in{\mathbb{N}}}\operatorname{ST^{+}}(2^{n_{k}},{\mathbb{Q}}(i))$ are
minimal.

###### Proof.

We first prove that
$G=\prod_{k\in{\mathbb{N}}}\operatorname{SL}(2^{n_{k}},{\mathbb{Q}}(i))$ is
not minimal. In view of the Minimality Criterion, it suffices to show that $G$
is not essential in
$\widehat{G}=\prod_{k\in{\mathbb{N}}}\operatorname{SL}(2^{n_{k}},{\mathbb{C}}).$
To this aim, let

$N=\\{(\lambda_{k}I_{2^{n_{k}}})_{k\in{\mathbb{N}}}\in\widehat{G}|\
(\lambda_{k+1})^{2}=\lambda_{k}\ \forall k\in{\mathbb{N}}\\}.$

The equality $\lambda_{k+1}^{2}=\lambda_{k}$ implies that $N$ is a closed
central subgroup of $\widehat{G}.$ Moreover, $N$ is non-trivial as the
sequence $(n_{k})_{k\in{\mathbb{N}}}$ is increasing. Let us see that $N$
trivially intersects $G.$ Otherwise, there exists a sequence
$(\lambda_{k})_{k\in{\mathbb{N}}}$ of roots of unity in ${\mathbb{Q}}(i)$ such
that $(\lambda_{k+1})^{2}=\lambda_{k}$ for every $k\in{\mathbb{N}}$ and
$\lambda_{k_{0}}\neq 1$ for some $k_{0}\in{\mathbb{N}}.$ It follows that
$\lambda_{k_{0}},\lambda_{k_{0}+1},\lambda_{k_{0}+2},\lambda_{k_{0}+3}$ are
different non-trivial roots of unity in ${\mathbb{Q}}(i),$ contradicting the
fact that $\pm 1,\pm i$ are the only roots of unity in ${\mathbb{Q}}(i).$

Now consider the group
$H=\prod_{k\in{\mathbb{N}}}\operatorname{ST^{+}}(2^{n_{k}},{\mathbb{Q}}(i))$.
In view of Lemma 2.1 and what we have just proved, $N$ is also a closed non-
trivial central subgroup of $\widehat{H}$ that trivially intersects $H.$ This
means that $H$ is not essential in $\widehat{H}$. By the Minimality Criterion,
$H$ is not minimal. ∎

######  Theorem 3.6.

  1. (1)

Let $\mathcal{F}_{\pi}$ and $\mathcal{F}_{c}$ be the set of Fermat primes and
the set of composite Fermat numbers, respectively, and let
$\mathcal{A}\in\\{\mathcal{F}_{\pi},\mathcal{F}_{c}\\}.$ Then the following
conditions are equivalent:

    1. (a)

$\mathcal{A}$ is finite;

    2. (b)

$\prod_{F_{n}\in\mathcal{A}}\operatorname{SL}(F_{n}-1,{\mathbb{Q}}(i))$ is
minimal;

    3. (c)

$\prod_{F_{n}\in\mathcal{A}}\operatorname{ST^{+}}(F_{n}-1,{\mathbb{Q}}(i))$ is
minimal.

  2. (2)

Let $\mathcal{M}_{\pi}$ and $\mathcal{M}_{c}$ be the set of Mersenne primes
and the set of composite Mersenne numbers, respectively, and let
$\mathcal{B}\in\\{\mathcal{M}_{\pi},\mathcal{M}_{c}\\}.$ Then the following
conditions are equivalent:

    1. (a)

$\mathcal{B}$ is finite;

    2. (b)

$\prod_{M_{p}\in\mathcal{B}}\operatorname{SL}(M_{p}+1,{\mathbb{Q}}(i))$ is
minimal;

    3. (c)

$\prod_{M_{p}\in\mathcal{B}}\operatorname{ST^{+}}(M_{p}+1,{\mathbb{Q}}(i))$ is
minimal.

###### Proof.

(1) Assume first that $\mathcal{A}$ is finite. Note that $F_{n}-1$ is a power
of two for every Fermat number $F_{n}\in\mathcal{A}.$ It is easy to see that a
product of finitely many perfectly minimal groups is minimal. Therefore, both
topological groups
$\prod_{F_{n}\in\mathcal{A}}\operatorname{SL}(F_{n}-1,{\mathbb{Q}}(i))$ and
$\prod_{F_{n}\in\mathcal{A}}\operatorname{ST^{+}}(F_{n}-1,{\mathbb{Q}}(i))$
are minimal, in view of Proposition 3.4. If $\mathcal{A}$ is infinite, then
$\prod_{F_{n}\in\mathcal{A}}\operatorname{SL}(F_{n}-1,{\mathbb{Q}}(i))$ and
$\prod_{F_{n}\in\mathcal{A}}\operatorname{ST^{+}}(F_{n}-1,{\mathbb{Q}}(i))$
are not minimal by Theorem 3.5.
(2) The proof is similar to (1). The only difference is that for every prime
$p$ it holds that $M_{p}+1$ is a power of two. ∎

## References

  * [1] U. Bader, T. Gelander, _Equicontinuous actions of semisimple groups_ , Groups, Geometry and Dynamics 11 (2017), 1003–1039.
  * [2] B. Banaschewski, Minimal topological algebras, Math. Ann. 211 (1974), 107–114.
  * [3] D. Dikranjan, Iv. Prodanov, Totally minimal topological groups, Annuaire Univ. Sofia Fat. Math. Méc. 69 (1974/75), 5–11.
  * [4] D. Dikranjan, Iv. Prodanov, L. Stoyanov, _Topological groups: characters, dualities and minimal group topologies,_ Pure and Appl. Math. 130, Marcel Dekker, New York-Basel, 1989.
  * [5] D. Doïtchinov, Produits de groupes topologiques minimaux, Bull. Sci. Math. 97 (2) (1972), 59–64.
  * [6] R.K Guy, Unsolved Problems in Number Theory, Problem Books in Math, 3rd ed. New York: Springer-Verlag, 2004.
  * [7] G.A. Margulis, _Discrete Subgroups of Semisimple Lie Groups_ , Springer-Verlag, 1990.
  * [8] M. Megrelishvili, _Group representations and construction of minimal topological groups_ , Topology Appl. 62 (1995), 1–19.
  * [9] M. Megrelishvili, M. Shlossberg, Minimality of topological matrix groups and Fermat primes, Topol. Appl. 322 (2022), doi:10.1016/j.topol.2022.108272.
  * [10] Iv. Prodanov, Precompact minimal group topologies and $p$-adic numbers, Annuaire Univ. Sofia Fac. Math. Méc. 66 (1971/72), 249–266.
  * [11] D.J.S. Robinson, _A Course in the Theory of Groups_ , 2nd edn., Springer-Verlag, New York, 1996.
  * [12] R.M. Stephenson, Jr., Minimal topological groups, Math. Ann. 192 (1971), 193–195.
  * [13] L. Stoyanov, On products of minimal and totally minimal groups, Proc. 11th Conf. of the Union of Bulg. Math., Sunny Beach 1982, 287–294.
  * [14] M. Suzuki, Group theory I, Springer, Berlin-Heidelberg-New York, 1982.
  * [15] W. Xi, D. Dikranjan, M. Shlossberg, D. Toller, _Hereditarily minimal topological groups_ , Forum Math. 31:3 (2019), 619–646.