# Medial Spectral Coordinates for 3D Shape Analysis

###### Abstract

In recent years there has been a resurgence of interest in our community in
the shape analysis of 3D objects represented by surface meshes, their
voxelized interiors, or surface point clouds. In part, this interest has been
stimulated by the increased availability of RGBD cameras, and by applications
of computer vision to autonomous driving, medical imaging, and robotics. In
these settings, spectral coordinates have shown promise for shape
representation due to their ability to incorporate both local and global shape
properties in a manner that is qualitatively invariant to isometric
transformations. Yet, surprisingly, such coordinates have thus far typically
considered only local surface positional or derivative information. In the
present article, we propose to equip spectral coordinates with medial (object
width) information, so as to enrich them. The key idea is to couple surface
points that share a medial ball, via the weights of the adjacency matrix. We
develop a spectral feature using this idea, and the algorithms to compute it.
The incorporation of object width and medial coupling has direct benefits, as
illustrated by our experiments on object classification, object part
segmentation, and surface point correspondence.

Morteza Rezanejad123, Mohammad Khodadad4, Hamidreza Mahyar5, Herve Lombaert6,
Michael Gruninger3, Dirk Walther1, Kaleem Siddiqi7
1Department of Psychology, University of Toronto, Toronto, Canada
2Department of Computer Science, University of Toronto, Toronto, Canada
3Department of Mechanical & Industrial Engineering, University of Toronto,
Toronto, Canada
4Department of Computer Engineering, Sharif University of Technology, Tehran,
Iran
5W Booth School of Engineering Practice and Technology, Hamilton, Canada
6Département de génie logiciel et des TI, École de technologie supérieure ÉTS,
Montréal, Canada
7School of Computer Science and Centre for Intelligent Machines, McGill
University, Montréal, Canada

_K_ eywords Medial Spectral Coordinates, Medial Surface, 3D Shape Analysis

##  1 Introduction

Advances in manufacturing sensors that can perceive 3D information, including
RGBD cameras, are beginning to revolutionize 3D perception using computer
vision methods. In this context, 3D object shape analysis has received much
attention due to its relevance to a variety of applications in robotics,
medical imaging, autonomous driving, surveillance, and other domains. The
technologies that are being developed for these applications must grapple with
fundamental problems including 3D shape classification, part segmentation, and
shape matching.

The problem of finding correspondences between 3D objects has a long history
[1, 2, 3, 4, 5, 6, 7]. Moving beyond object matching, the segmentation of a 3D
object into its constituent parts is another classic computer vision problem
for which research is very much still active [8, 9, 10, 11, 12, 13]. In
addition to part segmentation, 3D shape classification remains a topic of
active research. On this front, deep learning methods, including PointNet
[14], PointNet++ [15], DeepSet [16], ShapeContextNet [17], PointGrid [18],
DynamicGCN [19], and SampleNet [20], have shown great promise. As such models
grow and undergo further development, they also become more data-hungry and
require more resources in terms of computing time. Deep neural network models
are typically limited by their design and they cannot easily learn features
that are not describable by the parts from which they are made. In the context
of 3D shape analysis, it is also important for a neural network based model to
be able to learn appropriate latent features when the point cloud is sparse or
has been geometrically transformed [21].

|  |  |  |
---|---|---|---|---
(a) | (b) | (c) | (d) | (e)

|  |
---|---|---
(f) | (g) | (h)

Figure 1: An overview of our main results. First row: (a) We begin with a 3D
object and compute its medial surface. (b) The radius associated with the
medial surface is mapped to its boundary. (c) Boundary points that are
associated with the same medial surface point are coupled and shown with the
same color. (d) Entries of the adjacency matrix are computed using the
intersection volume of the associated medial spheres to drive adjacency
weights, for subsequent spectral analysis. (e) The first non-trivial
eigenvector is mapped onto the original shape. Second row: The use of our
medially driven spectral coordinates for segmentation (f), object
classification (g), and shape correspondence (h).

To facilitate these tasks, a descriptive and versatile representation for 3D
object shape is key. To this end, in the present article we propose a novel 3D
object shape representation method that computes spectral coordinates using
the associated 3D medial surface. Medial representations have been used for 2D
shape analysis in the literature for decades [22, 23, 24, 25, 26, 27, 28].
Applications of medial representations also exist in 3D [29, 30, 31], but
challenges exist around computing them reliably and robustly. In addition, the
computer vision community has shifted away from the use of hand-crafted
features to embrace the use of deep learning based descriptors, learned from
geometric data. Taken together, these methods have allowed for better
performance in general, in most tasks that the models have been curated for.

In the present article, we aim to equip present models with an enriched
representation of 3D object shape, one that leverages the local width of the
object to construct a spectral feature at each of its boundary points. We make
three main contributions.

  1. 1.

We develop a robust, and reliable implementation of an algorithm for computing
3D medial surfaces based on an average outward flux measure.

  2. 2.

We introduce a novel approach to computing spectral coordinates, exploiting
the duality between the medial surface and the boundary it represents. This
approach brings inferred local and global geometrical information to the
boundary and allows for an enriched spectral representation of that boundary.
In particular, surface points that are bi-tangent to medial spheres that have
a common volume are coupled to capture local object part symmetry.

  3. 3.

We demonstrate that by using such spectral coordinates that are associated
with the medial surface, there are performance benefits for popular 3D shape
analysis tasks, including finding correspondences between surface points, 3D
object part segmentation, and 3D object shape classification.

Our article is organized as follows. In Section 2, we review average outward
medial surfaces and their computation. In Section 3, we introduce our novel
medial spectral coordinates. In Section 4 we develop a number of applications
that take advantage of our these spectral coordinates. Finally, we conclude
with a summary of contributions and a discussion of possible future research
directions, in Section 5.

##  2 Medial Representation of 3D Shapes

Figure 2: A schematic of a portion of a medial surface together with a
selected maximal inscribed sphere, and the associated bi-tangent points
($\mathbf{b}_{1}$, $\mathbf{b}_{2}$) where the sphere touches the boundary.
The medial surface patch is shown in light green and the associated local
boundary patches are in light purple. Each point $\mathbf{p}$ on the medial
manifold is associated with two distinct bi-tangent points on the object’s
surface, to which it is closest in the sense of Euclidean distance (the figure
is adapted from [32]).

Introduced by Harry Blum [33], medial axes or skeletons have been used in the
vision and computational geometry literature for years [29, 34, 26, 35, 36,
37, 38, 39]. Formally, the medial axis is defined as the locus of centers of
maximal inscribed spheres within a watertight object. In 3D this locus can
include both 3D curves and 3D medial manifolds, that together comprise the 3D
medial surface of an object. A promise of this representation is that it
captures the duality between both boundary and interior object properties, the
latter by considering the radius associated with the inscribed sphere to
capture local object width. In addition, it is possible to reconstruct the
original object from its medial surface, so in this sense the representation
is complete. Unfortunately, in practice, computing a medial surface robustly
turns out to be a challenging computational problem. We focus here on the
medial surface computation approach of [40], which is based on a notion of
average outward flux (AOF), and which was later used for hippocampal shape
analysis [32] as well as other applications [29].

###  2.1 Average Outward Flux Medial Surfaces

To provide the intuition behind this algorithm consider the object shown in
Figure 2, together with a local portion of its medial surface (light green)
and the associated boundary surface patches (purple). The medial surface is
homotopic to the original object and thus reflects its topology while making
local reflective symmetries explicit. Let $\mathcal{S}$ represent the closed
surface of a 3D object and $D_{E}(\mathbf{p})$ be the Euclidean distance to
the boundary of this object at each point $\mathbf{p}$ in its interior. Now
consider the gradient vector field of $D_{E}(\mathbf{p})$ as $\dot{q}$. Where
there is just one closest point $\mathbf{b}=(b_{x},b_{y},b_{z})$ on the
boundary, to a particular interior-point $\mathbf{a}=(a_{x},a_{y},a_{z})$, the
gradient vector at $\mathbf{a}$ is computed as:

$\dot{q}(\mathbf{a})=\frac{\mathbf{a}-\mathbf{b}}{||\mathbf{a}-\mathbf{b}||}$
(1)

However, the gradient vector field is multi-valued at locations on the medial
surface. Exploiting this property, an “average outward flux” measure was
proposed in [40], which computes the outward flux of $\dot{q}$ through a
shrinking sphere averaged over the surface area of that sphere:

$AOF(\mathbf{p})=\frac{\int_{\partial
R}\langle\dot{q},\mathcal{N}_{0}\rangle\partial\mathcal{S}}{Area(\partial
R)}.$ (2)

Here $\partial R$ represents the surface area of the shrinking sphere and
$\partial\mathcal{S}$ is the surface area element. $\mathcal{N}_{0}$ is the
outward normal at each point on the sphere. Earlier work on medial axis
computations has shown that as the size of this sphere shrinks to zero, this
measure gives non-zero values on the medial surface, but zero-values off the
medial locus, and hence provides an effective means for localizing medial
surface points [41, 40]. In the present article, we use a robust
implementation of this algorithm. Figure 3 illustrates the sequence of steps
for 3D medial surface generation.

|  |  |
---|---|---|---
(a) 3D Shape |  (b) $D_{E}(\mathbf{p})$ |  (c) $\frac{\int_{\partial R}\langle\dot{q},\mathcal{N}_{0}\rangle\partial\mathcal{S}}{Area(\partial R)}$ |  (d) MAT
Figure 3: Using the divergence theorem, medial axis voxels can be identified
by considering the behavior of the average outward flux (AOF) of the gradient
of the Euclidean distance function to the boundary of a 3D object, through a
shrinking sphere [29]. In particular, the limiting AOF value of all points not
located on the skeleton is equal to zero. Starting with the boundary of a 3D
object (a), we first compute the distance map to the boundary (b), and then
compute the 3D AOF 3D map (c) from this distance. Finally, by retaining the
non-zero AOF voxels, we obtain the 3D medial axis transform (d). Since at each
medial voxel the maximal inscribed sphere (Fig. 2) touches the boundary at two
distinct points, it is possible to reconstruct the boundary purely from the
medial locus.

###  2.2 Preserving object topology

Whereas in a continuous setting one can associate medial surface points with
locations where the AOF is non-zero, in a discrete setting care has to be
taken to ensure that the resultant set of voxels is homotopic to the original
object. We follow past work [32], which rests on a definition of endpoints and
simple points on a digital 3D lattice:

  1. 1.

A simple point is a point that cannot be removed without changing the topology
of the object. Its removal will either disconnect the object or create a hole
or a cavity.

  2. 2.

Consider a plane that passes through a point $\mathbf{p}$ such that its
intersection with the 3D object results in an open curve. If this curve ends
at $\mathbf{p}$, then $\mathbf{p}$ is an endpoint of that 3D curve. Additional
examples of endpoints include when $\mathbf{p}$ is on the rim or corner of a
3D surface.

On a 3D digital lattice we can classify both simple points and endpoints by
considering the 26 neighborhoods of a particular voxel (the cubic lattice of
$3\times 3\times 3$). We implement these two notions of simple points and
endpoints according to the characterization of [42], which is based upon
different sets of neighbors that share either a point, an edge, or a face with
the considered voxel.

Algorithm 1 Medial Surface Generation Procedure

procedure Average Outward Flux Computation

Compute the Euclidean Distance Function $D_{E}(\mathbf{p})$

Compute the gradient vector field $\nabla D_{E}$

Compute the average outward flux of $\nabla D_{E}$ using 2

procedure AOF Topology Preserving Thinning

$\text{Heap}\rightarrow H$, Threshold $\rightarrow\tau$

for each point $\mathbf{p}$ on the boundary of the object do

if $\mathbf{p}$ is simple then

insert point $\mathbf{p}$ into $H$ with key value $AOF(\mathbf{p})$

while $H$ is not empty do

$\mathbf{p}$ = HeapExtractMax($H$)

if $\mathbf{p}$ is simple then

if $\mathbf{p}$ is end point and $AOF(\mathbf{p})>\tau$ then

Label $\mathbf{p}$ as a medial surface point

else

Remove $\mathbf{p}$

for all neighbors $\mathbf{q}$ of $\mathbf{p}$ do

if if $\mathbf{q}$ is simple then inset point $\mathbf{q}$ into $H$

Algorithm 1 describes the resultant medial surface generation method, which
combines the AOF measure with a topology preserving thinning approach, using
the above characterization of simple points and endpoints. Figure 3 provides
several examples for a variety of 3D objects. At the end of this process the
surface of each 3D object $\mathcal{S}$ can be represented by a set of
skeletal points where:

$\mathcal{S}=\bigcup_{i=1}^{n}\mathbf{p}_{i}^{sk}=\bigcup_{i=1}^{n}(x_{i},y_{i},z_{i},r_{i},\lambda_{i}).$
(3)

Here the triplet $(x_{i},y_{i},z_{i})$ represents the location of the surface
point $\mathbf{p}_{i}$, $r_{i}$ is the radius of the maximal inscribed sphere
touching the object at $\mathbf{p}_{i}$, and $\lambda_{i}$ is the average
outward flux value at the medial surface point at the center of that sphere.
As shown in [29] the AOF value also reveals the object angle, and thus
provides additional useful object and symmetry information. In the next
subsection, we will demonstrate that this representation can be used for
object reconstruction.

|  |  |  |
---|---|---|---|---
(a) | (b) | (c)
Figure 4: (a) The original object. (b) Its skeleton and subsequent
reconstruction using the method of [43]. (c) Its AOF based medial surface and
subsequent reconstruction.

###  2.3 Medial Spectral Coordinates for 3D Shape Analysis

Perhaps the most obvious criterion for the suitability of an object shape
representation method is whether it can be used to reconstruct the original
object. This criterion, alongside other criteria such as completeness,
hierarchy, invariance, stability, and similarity has been used to judge shape
representation choices. Whereas medial representations satisfy several of
these criteria [29], algorithms implemented on a discrete lattice may not
allow for faithful object reconstruction. Here, for purposes of
reconstruction, we compare the AOF skeletonization process against a commonly
used skeletonization method that extracts 3D skeletons by thinning to produce
medial curves [43]. Since the medial surface consists of the locus of
maximally inscribed spheres, the original object can be reconstructed by
considering the envelope of the maximally inscribed spheres at all the medial
surface points. Figure 4 compares reconstructions obtained by these two
approaches. Here we see that for the chair examples, the 3D object
reconstructed from the method in [43] (Figure 4 bottom) is far from
satisfactory and important object details are lost, in contrast to the
reconstruction obtained from AOF medial surfaces (Figure 4 middle). A
quantitative comparison via a mean intersection over union (mIoU) measure with
original objects is shown in Table 1, over the entire datasets of ModelNet10
[44] and COSEG [45].

COSEG [45] | ModelNet10 [44]
---|---
Model | Ours | Curve | Model | Ours | Curve
Candelabra | 96.99 | 72.82 | Table | 99.68 | 88.61
Chairs | 96.38 | 81.40 | Bathtub | 93.25 | 86.50
Fourleg | 90.33 | 72.44 | Bed | 95.31 | 73.13
Goblets | 90.68 | 81.69 | Chair | 96.10 | 85.04
Guitars | 93.19 | 75.42 | Desk | 97.78 | 80.73
Lamps | 95.30 | 66.71 | Dresser | 94.23 | 91.31
Vases | 96.54 | 69.51 | Monitor | 90.90 | 73.02
Irons | 94.07 | 80.43 | Night stand | 92.66 | 82.24
Tele-aliens | 98.19 | 75.46 | Sofa | 91.53 | 81.48
Chairs (L) | 97.18 | 79.05 | Toilet | 91.05 | 82.21
Mean | 94.89 | 75.49 | Mean | 94.25 | 82.43
Table 1: The mean Intersection-Over-Union (mIoU) reconstruction measures, for
the two datasets of COSEG and ModelNet10, comparing the use of medial curves
[43] and AOF medial surfaces [40].

##  3 Spectral Coordinates using Medial Manifolds

We now propose to use the medial manifold to equip the object’s surface with a
new spectral signature, one that explicitly considers local object width.
Whereas spectral approaches have been popular for shape correspondence [46, 6,
47, 5] and spectral methods have also been successfully used in combination
with graph convolution to learn features from point sets and meshes [48, 49,
50], the explicit consideration of object width and the associated symmetry
properties and coupling of surface points which share a medial ball is new.

Figure 5: Qualitative results showing correspondences obtained using our
medial spectral coordinates across different poses of the same models in the
TOSCA [51] dataset.

Let the object’s surface $\mathcal{S}$ be represented by its set of boundary
points $\\{\mathbf{b}_{1},\mathbf{b}_{2},\dots,\mathbf{b}_{n}\\}$. We voxelize
the object and compute the medial surface using Algorithm 1 in Section 2. Let
the set of resulting medial surface voxels be
$\\{\mathbf{p}_{1}^{sk},\mathbf{p}_{2}^{sk},\dots,\mathbf{p}_{m}^{sk}\\}$.
Using the reconstruction algorithm discussed in Section 2.3, we reconstruct a
boundary surface $\mathcal{S}^{r}$ that is close to $\mathcal{S}$, the surface
of the original object. Using a $kd$-tree partitioning algorithm, we then find
a correspondence map between points from $\mathcal{S}$ to $\mathcal{S}^{r}$.
Using this map, we associate each point $\mathbf{b}_{i}$ on the boundary with
its corresponding medial locus point $\mathbf{p}_{j}^{sk}$ on the medial
surface. Now, assuming that the boundary is tessellated so that the boundary
points are the vertices (or set of nodes) in a connected graph, eigenmaps of
these nodes are computed as follows. We first compute an adjacency matrix (as
shown in Figure 3 (d)) where a particular metric is used to compute the
adjacency weights in the matrix $W$, as follows. Consider two boundary points
(nodes) $\mathbf{b}_{i}$ and $\mathbf{b}_{j}$. Let these boundary points be
associated with skeletal points $\mathbf{p}_{i}$ and $\mathbf{p}_{j}$,
respectively. Now, let the maximal inscribed sphere at skeletal point
$\mathbf{p}_{i}$ be given by $\rho(\mathbf{p}_{i}^{sk})$. We consider the
degree of overlap between the inscribed spheres touching the object’s surface
at $\mathbf{b}_{i},\mathbf{b}_{j}$ to construct the adjacency weight between
the associated nodes

$w_{ij}=\frac{C(i,j)}{\rho(\mathbf{p}_{i}^{sk})},$ (4)

where $C(i,j)$ is the volume of the intersection between the spheres
$\rho(\mathbf{p}_{i}^{sk})$ and $\rho(\mathbf{p}_{j}^{sk})$. We then construct
the general Laplacian operator as $L=D-W$, where $W$ is the weighted adjacency
matrix of the graph with affinity weights (see [52]), and the degree matrix,
$D$, is a diagonal matrix, where $D_{ii}=\Sigma_{j}w_{ij}$. We then compute
the eigenvectors of the Laplacian matrix:

$LE_{\lambda_{i}}=\lambda_{i}E_{\lambda_{i}},i=1,\dots,n.$ (5)

Here, since matrix $W$ is not symmetric (leading to a possibility of imaginary
numbers numerically), we create two symmetric matrices $W^{sym}$ and $D^{sym}$
as follows. $W^{sym}$ has entries given by

$w_{ij}^{sym}=C(i,j),$ (6)

and $D^{sym}$ which is a diagonal matrix, with entries
$d_{ii}^{sym}=\rho(\mathbf{p}_{i}^{sk})$. Now, by using the Arnoldi method of
ARPACK [53], we can obtain eigenvectors of the following equation:

$W^{sym}E_{\lambda_{i}}=\lambda_{i}D^{sym}E_{\lambda_{i}},i=1,\dots,n$ (7)

where the obtain eigenvalues are real and non-negative. Now that we have our
eigenvectors, we can obtain spectral coordinates by mapping the coordinates
from the boundary of the 3D shape. For each boundary point $\mathbb{b}_{i}$
element (e.g. $x$, $y$, $z$ or its corresponding sphere radius value $r$ from
the medial surface), we can compute the mapped information on the $i^{th}$
basis eigenvectors as:

$\mathbf{m}_{i}=\mathbf{b}_{i}D^{sym}E_{\lambda_{i}}.$ (8)

Referring back to the Dirichlet energy equation introduced by Bronstein et al.
[47] that measures change over the boundary $\mathcal{S}$, we can introduce
our spectral coordinates as follows:

$S_{C}(\mathcal{S})=\Big{(}\bigcup_{i=1}^{n}\lambda_{i}\sum_{j=1}^{k}(\mathbf{m}_{ij}^{2})\Big{)}.$
(9)

We are careful to always remove the trivial eigenvalue of 0 (the smallest
eigenvalue) from our set of spectral coordinates.

##  4 Applications

We now evaluate the use of our medial spectral coordinates for three popular
applications in 3D shape analysis: a) finding correspondence between surface
points, b) 3D object part segmentation of 3D objects and c) 3D object
classification. For each case, we demonstrate that the explicit use of local
object width, as reflected in the medial radius, and the coupling of surface
points that share a common medial sphere, provides direct benefits and
improved performance.

###  4.1 Shape Correspondence

Figure 6: Correspondence accuracy on the TOSCA dataset [51] for different
methods. The methods we have compared include PMF [54], SGMDS [55], FM [56],
BIM [57], Mobius Voting [7], Conformal [57], and the use of our new
MedialSpectral signature.

To obtain correspondences between surface points on two distinct objects, one
needs a matching algorithm that matches one against the other. In the present
setting, we have spectral coordinates alongside eigenvalues that are obtained
from a Laplacian operator. When spectra are computed, two situations are
possible that make the direct comparison of spectral coordinates challenging.
First, eigenvector computation may generate a sign ambiguity. Second, it is
possible that when eigenvectors are being computed for the same data but for
two different shapes, they might be computed in opposite orders due to the
fact that the ordering of the lowest eigenvector may change. [6] suggests a
method for mitigating the effects of this flipping problem by favoring three
factors: 1) pairs of eigenvectors that are most likely to match based on the
similarity between their eigenvalues 2) histograms 3) the spatial
distributions of their spectral coordinate value. We use these techniques to
reorder and align spectra during the correspondence finding process. The
process of reordering is sped up by downsampling all eigenvectors. After
reordering and aligning the spectra, two points that are closest in the
embedded representations can be treated as corresponding points across the two
objects. This is achieved by using the Coherent Point Drift (CPD) method [58].
We test our correspondence finding algorithm on the TOSCA dataset [51], which
includes a total of 80 objects, including four limb animals such as cats and
dogs, female and male figures. We compare our methods against several other
existing methods and report the results in Figure 6. The results show the
advantages of using medially driven spectral coordinates when compared to the
best conformal model that we could find. We also achieve results that are
comparable to those produced by the state of the art deep neural network based
models, such as PMF [54]. We show qualitative results of correspondences
obtained on four different models from the TOSCA dataset in Figure 5.

|  |
---|---|---
Glasses | Airplane | Teddy
|  |
Octopus | Chair | Bird
Figure 7: Qualitative part segmentation results obtained using medially driven spectral signatures on the Princeton segmentation benchmark (see text for details). Model | SpectralMedial | Spectral | MV-RNN [59] | Shu [60] | WcSeg | RandCuts | ShapeDiam | NormCuts | CoreExtra | RandWalks | FitPrim | KMeans
---|---|---|---|---|---|---|---|---|---|---|---|---
Human | 0.0557 | 0.1240 | 0.106 | 0.116 | 0.128 | 0.131 | 0.179 | 0.152 | 0.225 | 0.219 | 0.153 | 0.163
Cup | 0.1413 | 0.1262 | 0.100 | 0.096 | 0.171 | 0.219 | 0.358 | 0.244 | 0.307 | 0.358 | 0.413 | 0.459
Glasses | 0.0802 | 0.1304 | 0.066 | 0.173 | 0.173 | 0.101 | 0.204 | 0.141 | 0.301 | 0.311 | 0.235 | 0.188
Airplane | 0.0728 | 0.0891 | 0.085 | 0.150 | 0.089 | 0.122 | 0.092 | 0.186 | 0.256 | 0.248 | 0.166 | 0.211
Ant | 0.0279 | 0.0900 | 0.021 | 0.001 | 0.021 | 0.025 | 0.022 | 0.047 | 0.065 | 0.068 | 0.086 | 0.131
Chair | 0.0483 | 0.0518 | 0.051 | 0.040 | 0.103 | 0.184 | 0.111 | 0.088 | 0.187 | 0.156 | 0.212 | 0.213
Octopus | 0.0255 | 0.0835 | 0.022 | 0.036 | 0.029 | 0.063 | 0.045 | 0.061 | 0.051 | 0.067 | 0.101 | 0.101
Table | 0.0630 | 0.1158 | 0.072 | 0.040 | 0.091 | 0.383 | 0.184 | 0.093 | 0.244 | 0.131 | 0.181 | 0.369
Teddy | 0.0521 | 0.1105 | 0.035 | 0.024 | 0.056 | 0.045 | 0.057 | 0.121 | 0.114 | 0.128 | 0.132 | 0.182
Hand | 0.0670 | 0.1443 | 0.076 | 0.135 | 0.116 | 0.090 | 0.202 | 0.155 | 0.155 | 0.189 | 0.202 | 0.154
Plier | 0.0970 | 0.1433 | 0.054 | 0.151 | 0.087 | 0.109 | 0.375 | 0.183 | 0.093 | 0.230 | 0.169 | 0.263
Fish | 0.1030 | 0.2339 | 0.146 | 0.288 | 0.203 | 0.297 | 0.248 | 0.394 | 0.273 | 0.388 | 0.424 | 0.413
Bird | 0.0049 | 0.0850 | 0.059 | 0.171 | 0.101 | 0.107 | 0.115 | 0.184 | 0.124 | 0.250 | 0.196 | 0.190
Armadillo | 0.0178 | 0.1265 | 0.060 | 0.073 | 0.081 | 0.092 | 0.090 | 0.116 | 0.141 | 0.115 | 0.091 | 0.117
Bust | 0.1081 | 0.1729 | 0.162 | 0.275 | 0.266 | 0.232 | 0.298 | 0.316 | 0.315 | 0.298 | 0.300 | 0.334
Mech | 0.1018 | 0.1133 | 0.121 | 0.073 | 0.182 | 0.277 | 0.238 | 0.159 | 0.387 | 0.211 | 0.306 | 0.425
Bearing | 0.0660 | 0.1516 | 0.080 | 0.056 | 0.122 | 0.124 | 0.119 | 0.183 | 0.398 | 0.246 | 0.188 | 0.280
Vase | 0.0772 | 0.1504 | 0.106 | 0.212 | 0.161 | 0.133 | 0.239 | 0.236 | 0.226 | 0.246 | 0.257 | 0.387
FourLeg | 0.0921 | 0.1772 | 0.135 | 0.140 | 0.152 | 0.174 | 0.161 | 0.208 | 0.191 | 0.218 | 0.185 | 0.193
Average | 0.0696 | 0.1273 | 0.082 | 0.118 | 0.123 | 0.153 | 0.176 | 0.172 | 0.211 | 0.215 | 0.210 | 0.251
|  |  |  |  |  |  |  |  |  |  |  |
Table 2: The Rand Index (error) segmentation scores, for object categories in
the Princeton segmentation benchmark, comparing across different methods. In
this table, a lower number indicates better performance. The use of medially
drive spectral coordinates (column two) consistently outperforms the use of
spectral coordinates without the medial (radius) component (column three). The
approach is also competitive against many other approaches for several of the
object categories.

###  4.2 Part Segmentation

We now examine whether the addition of medially driven spectral coordinates
can be beneficial to the task of part segmentation. To achieve this aim, we
propose the following task to carry out unsupervised 3D shape decomposition.
We collect all the spectral coordinates computed and add them as additional
features to the boundary representation
$\mathbf{b}_{i}=(x_{i},y_{i},z_{i},r_{i})$. We then use the high-dimensional
data spectral clustering of [61] to cluster all object models within a
category together. The algorithm of [61] provides a spectral clustering
approach based on subspace randomization and graph fusion for high-dimensional
data, which enables us to carry out both segmentation and co-segmentation
tasks for a single shape or family of shape models (co-segmentation). To
examine whether these added features to the original shape help with the
segmentation task, we tested our off-the-shelf method on the Princeton
Segmentation Benchmark [62]. We report our results in Table 2 and show typical
qualitative part segmentation results in Figure 7.

###  4.3 Object Classification

The last set of experiments we present in this paper relate to the problem of
object classification. For this task, we examine whether the inclusion of
medial spectral coordinates leads to an improvement in the performance of a
vanilla neural network model. To achieve this goal, we carried out the
following classification task. Inspired by the idea presented in [21], we
created a new feature vector for each point in a 3d point cloud as a Geometry
Similarity Connection, capturing the local behavior of each point in the
eigenvector space. First, we find the $k$ nearest neighbors to each point in
the eigenvector space. Then, we add the mean and standard deviation of those
$k$ points to the $x$,$y$,$z$ coordinates of the considered point. Finally, we
add these additional features to the point’s features, creating a vector of
$9$ features for each point in the point cloud. We tested the added features
on the popular PointNet model [14]. The added features result in an
improvement in the overall accuracy of the model from 88.76% to 90.41% on
ModelNet-10 and from 86.23% to 88.26% on ModelNet-40. We present a detailed
account of the results for each category for this experiment in Table 3.

MNet10 |  | bathtub | bed | chair | desk | dresser | monitor | night stand | sofa | table | toilet
---|---|---|---|---|---|---|---|---|---|---|---
Vanila | 83.87 | 92.85 | 94.73 | 77.77 | 83.87 | 96.48 | 84.02 | 91.70 | 80.78 | 95.87
Ours | 85.71 | 94.28 | 92.85 | 82.80 | 85.54 | 96.51 | 84.33 | 96.44 | 83.40 | 97.02
MNet40 |  | airplane | bathtub | bed | bench | bookshelf | bottle | bowl | car | chair | cone
Vanila | 100.00 | 79.86 | 93.88 | 74.93 | 92.81 | 93.98 | 100.00 | 97.77 | 95.83 | 100.00
Ours | 100.00 | 83.50 | 98.19 | 76.38 | 96.57 | 94.67 | 100.00 | 98.58 | 98.88 | 100.00
| cup | curtain | desk | door | dresser | flower pot | glass box | guitar | keyboard | lamp
Vanila | 69.85 | 89.83 | 78.94 | 94.89 | 64.90 | 29.89 | 93.93 | 100.00 | 100.00 | 89.85
Ours | 73.95 | 92.46 | 81.39 | 99.29 | 66.83 | 32.20 | 98.71 | 100.00 | 100.00 | 90.93
| laptop | mantel | monitor | night stand | person | piano | plant | radio | range hood | sink
Vanila | 100.00 | 95.90 | 94.86 | 82.41 | 84.93 | 88.63 | 72.85 | 69.81 | 90.99 | 79.93
Ours | 100.00 | 98.87 | 98.30 | 84.37 | 88.03 | 92.77 | 73.09 | 70.41 | 95.77 | 83.19
|  | sofa | stairs | stool | table | tent | toilet | tv stand | vase | wardrobe | xbox
| Vanila | 95.87 | 84.94 | 89.95 | 87.86 | 94.88 | 98.88 | 86.87 | 78.79 | 59.93 | 69.91
| Ours | 97.13 | 86.97 | 90.60 | 89.31 | 96.26 | 100.00 | 87.74 | 80.50 | 60.59 | 74.32
|  |  |  |  |  |  |  |  |  |  |
Table 3: The difference in object classification accuracy when using vanilla
features and our medial spectral features with the PointNet model[14], using
the ModelNet10 and ModelNet40 datasets [44].

Our results show that in several of these object categories, there is a boost
in performance when we use spectral coordinates that use medial surface
information over the use of raw data (with the spectral clustering algorithm).
These results are reflected in the second (SpectralMedial) and third
(Spectral) columns of Table 2. We also observe that the use of spectral
coordinates outperforms several of the other existing approaches including the
deep model-based ([59]). Finally, the mean performance and the rand index
measure of our approach is the best overall on the entire Princeton dataset
[62].

##  5 Conclusion

We have proposed a novel spectral coordinate for 3D shape analysis
applications, one that includes the local object width associated with the
medial surface radius function. To our knowledge, this is the first attempt to
include medial width in a spectral feature, and our experiments demonstrate
the considerable benefits of doing this. Our model ties surface points on a 3D
object to their associated medial surface points and uses the duality between
the medial surface of an object and its boundary to extract spectral
coordinates with explicit consideration of local object width and object part
symmetry. We have introduced a novel way to compute the adjacency weights
between boundary nodes, based on the volume of intersection of their
associated inscribed spheres. The use of this medially driven spectral
coordinate leads to improved correspondences when compared against the use of
conformal methods. When applied to 3D part segmentation the use of the medial
surface to group together points on the boundary that are not necessarily
close to one another in geodesic terms, but are tied to each other
geometrically in terms of local object symmetry, has advantages. Finally, our
object classification experiments using the medial spectral coordinates in
addition to the raw data demonstrate that a vanilla deep neural network model
can benefit from the incorporation of these additional spectral features.

In future work, we aim to examine whether it is possible to compute medial
spectral coordinates implicitly, via a deep neural network model. We also hope
to examine the potential of medial spectral coordinates for graph neural
network models.

## References

  * [1] Silvia Biasotti, Simone Marini, Michela Spagnuolo, and Bianca Falcidieno.  Sub-part correspondence by structural descriptors of 3d shapes.  Computer-Aided Design, 38(9):1002–1019, 2006.
  * [2] Rune Brincker, Anders Skafte, Manuel López-Aenlle, Aldo Sestieri, Walter D’Ambrogio, and Alfonso Canteli.  A local correspondence principle for mode shapes in structural dynamics.  Mechanical Systems and Signal Processing, 45(1):91–104, 2014.
  * [3] Dinggang Shen, Wai-him Wong, and Horace HS Ip.  Affine-invariant image retrieval by correspondence matching of shapes.  Image and Vision Computing, 17(7):489–499, 1999.
  * [4] Masaki Hilaga, Yoshihisa Shinagawa, Taku Kohmura, and Tosiyasu L Kunii.  Topology matching for fully automatic similarity estimation of 3d shapes.  In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 203–212, 2001.
  * [5] Varun Jain and Hao Zhang.  Robust 3d shape correspondence in the spectral domain.  In IEEE International Conference on Shape Modeling and Applications 2006 (SMI’06), pages 19–19. IEEE, 2006.
  * [6] Herve Lombaert, Leo Grady, Jonathan R Polimeni, and Farida Cheriet.  Focusr: feature oriented correspondence using spectral regularization–a method for precise surface matching.  IEEE transactions on pattern analysis and machine intelligence, 35(9):2143–2160, 2012.
  * [7] Yaron Lipman and Thomas Funkhouser.  Möbius voting for surface correspondence.  ACM Transactions on Graphics (TOG), 28(3):1–12, 2009.
  * [8] Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri.  3d shape segmentation with projective convolutional networks.  In proceedings of the IEEE conference on computer vision and pattern recognition, pages 3779–3788, 2017.
  * [9] Yecheng Lyu, Xinming Huang, and Ziming Zhang.  Learning to segment 3d point clouds in 2d image space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12255–12264, 2020.
  * [10] Mutian Xu, Junhao Zhang, Zhipeng Zhou, Mingye Xu, Xiaojuan Qi, and Yu Qiao.  Learning geometry-disentangled representation for complementary understanding of 3d object point cloud.  arXiv preprint arXiv:2012.10921, 2020.
  * [11] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411–6420, 2019.
  * [12] Sindhu Hegde and Shankar Gangisetty.  Pig-net: Inception based deep learning architecture for 3d point cloud segmentation.  Computers & Graphics, 95:13–22, 2021.
  * [13] Morteza Rezanejad and Kaleem Siddiqi.  View sphere partitioning via flux graphs boosts recognition from sparse views.  Frontiers in ICT, 2:24, 2015.
  * [14] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [15] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  arXiv preprint arXiv:1706.02413, 2017.
  * [16] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola.  Deep sets.  arXiv preprint arXiv:1703.06114, 2017.
  * [17] Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu.  Attentional shapecontextnet for point cloud recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4606–4615, 2018.
  * [18] Truc Le and Ye Duan.  Pointgrid: A deep network for 3d shape understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9204–9214, 2018.
  * [19] Fanfan Ye, Shiliang Pu, Qiaoyong Zhong, Chao Li, Di Xie, and Huiming Tang.  Dynamic gcn: Context-enriched topology learning for skeleton-based action recognition.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 55–63, 2020.
  * [20] Itai Lang, Asaf Manor, and Shai Avidan.  Samplenet: Differentiable point cloud sampling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7578–7588, 2020.
  * [21] Mingye Xu, Zhipeng Zhou, and Yu Qiao.  Geometry sharing network for 3d point cloud classification and segmentation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12500–12507, 2020.
  * [22] Jun Xie, Pheng-Ann Heng, and Mubarak Shah.  Shape matching and modeling using skeletal context.  Pattern Recognition, 41(5):1756–1767, 2008.
  * [23] Roger Tam and Wolfgang Heidrich.  Shape simplification based on the medial axis transform.  In IEEE Visualization, 2003. VIS 2003., pages 481–488. IEEE, 2003\.
  * [24] Xiang Bai, Xingwei Yang, Deguang Yu, and Longin Jan Latecki.  Skeleton-based shape classification using path similarity.  International Journal of Pattern Recognition and Artificial Intelligence, 22(04):733–746, 2008.
  * [25] Diego Macrini, Sven Dickinson, David Fleet, and Kaleem Siddiqi.  Bone graphs: Medial shape parsing and abstraction.  Computer Vision and Image Understanding, 115(7):1044–1061, 2011\.
  * [26] Kaleem Siddiqi, Ali Shokoufandeh, Sven J Dickinson, and Steven W Zucker.  Shock graphs and shape matching.  International Journal of Computer Vision, 35(1):13–32, 1999.
  * [27] Morteza Rezanejad.  Medial measures for recognition, mapping and categorization.  McGill University, 2020.
  * [28] Morteza Rezanejad, Babak Samari, I Rekleitis, Kaleem Siddiqi, and Gregory Dudek.  Robust environment mapping using flux skeletons.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5700–5705. IEEE, 2015.
  * [29] Kaleem Siddiqi and Stephen Pizer.  Medial representations: mathematics, algorithms and applications, volume 37.  Springer Science & Business Media, 2008.
  * [30] Andrea Tagliasacchi, Thomas Delame, Michela Spagnuolo, Nina Amenta, and Alexandru Telea.  3d skeletons: A state-of-the-art report.  Computer Graphics Forum, 35(2):573–597, 2016.
  * [31] Sungmin Hong, James Fishbaugh, Morteza Rezanejad, Kaleem Siddiqi, Hans Johnson, Jane Paulsen, Eun Young Kim, and Guido Gerig.  Subject-specific longitudinal shape analysis by coupling spatiotemporal shape modeling with medial analysis.  In Medical Imaging 2017: Image Processing, volume 10133, page 101331A. International Society for Optics and Photonics, 2017.
  * [32] Sylvain Bouix, Jens C Pruessner, D Louis Collins, and Kaleem Siddiqi.  Hippocampal shape analysis using medial surfaces.  Neuroimage, 25(4):1077–1089, 2005.
  * [33] Harry Blum et al.  A transformation for extracting new descriptors of shape, volume 43.  MIT press Cambridge, MA, 1967.
  * [34] Jacob Feldman and Manish Singh.  Bayesian estimation of the shape skeleton.  Proceedings of the National Academy of Sciences, 103(47):18014–18019, 2006.
  * [35] Der-Tsai Lee.  Medial axis transformation of a planar shape.  IEEE Transactions on pattern analysis and machine intelligence, (4):363–369, 1982.
  * [36] Nina Amenta, Sunghee Choi, and Ravi Krishna Kolluri.  The power crust, unions of balls, and the medial axis transform.  Computational Geometry, 19(2-3):127–153, 2001.
  * [37] Stavros Tsogkas and Sven Dickinson.  Amat: Medial axis transform for natural images.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2708–2717, 2017.
  * [38] Charles-Olivier Dufresne Camaro, Morteza Rezanejad, Stavros Tsogkas, Kaleem Siddiqi, and Sven Dickinson.  Appearance shock grammar for fast medial axis extraction from real images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14382–14391, 2020.
  * [39] Morteza Rezanejad, Gabriel Downs, John Wilder, Dirk B Walther, Allan Jepson, Sven Dickinson, and Kaleem Siddiqi.  Gestalt-based contour weights improve scene categorization by cnns.  In Conference on Cognitive Computational Neuroscience (CCN 2019), 2019.
  * [40] Kaleem Siddiqi, Sylvain Bouix, Allen Tannenbaum, and Steven W Zucker.  Hamilton-jacobi skeletons.  International Journal of Computer Vision, 48(3):215–231, 2002.
  * [41] Pavel Dimitrov, James N Damon, and Kaleem Siddiqi.  Flux invariants for shape.  In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 1, pages I–I. IEEE, 2003.
  * [42] Chris Pudney.  Distance-ordered homotopic thinning: a skeletonization algorithm for 3d digital images.  Computer vision and image understanding, 72(3):404–413, 1998.
  * [43] Ta-Chih Lee, Rangasami L Kashyap, and Chong-Nam Chu.  Building skeleton models via 3-d medial surface axis thinning algorithms.  CVGIP: Graphical Models and Image Processing, 56(6):462–478, 1994\.
  * [44] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  * [45] Yunhai Wang, Shmulik Asafi, Oliver Van Kaick, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen.  Active co-analysis of a set of shapes.  ACM Transactions on Graphics (TOG), 31(6):1–10, 2012.
  * [46] Michael M Bronstein and Alexander M Bronstein.  Shape recognition with spectral distances.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):1065–1071, 2010.
  * [47] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst.  Geometric deep learning: going beyond euclidean data.  IEEE Signal Processing Magazine, 34(4):18–42, 2017.
  * [48] Chu Wang, Babak Samari, and Kaleem Siddiqi.  Local spectral graph convolution for point set feature learning.  In Proceedings of the European conference on computer vision (ECCV), pages 52–66, 2018.
  * [49] Li Yi, Hao Su, Xingwen Guo, and Leonidas J Guibas.  Syncspeccnn: Synchronized spectral cnn for 3d shape segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2282–2290, 2017.
  * [50] Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov.  Deep geometric functional maps: Robust feature learning for shape correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8592–8601, 2020.
  * [51] Alexander M Bronstein, Michael M Bronstein, and Ron Kimmel.  Numerical geometry of non-rigid shapes.  Springer Science & Business Media, 2008.
  * [52] Leo J Grady and Jonathan R Polimeni.  Discrete calculus: Applied analysis on graphs for computational science, volume 3.  Springer, 2010.
  * [53] Richard B Lehoucq, Danny C Sorensen, and Chao Yang.  ARPACK users’ guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods.  SIAM, 1998.
  * [54] Zorah Lähner, Matthias Vestner, Amit Boyarski, Or Litany, Ron Slossberg, Tal Remez, Emanuele Rodola, Alex Bronstein, Michael Bronstein, Ron Kimmel, et al.  Efficient deformable shape correspondence via kernel matching.  arXiv preprint arXiv:1707.08991, 2017.
  * [55] Yonathan Aflalo, Anastasia Dubrovina, and Ron Kimmel.  Spectral generalized multi-dimensional scaling.  International Journal of Computer Vision, 118(3):380–392, 2016\.
  * [56] Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas.  Functional maps: a flexible representation of maps between shapes.  ACM Transactions on Graphics (TOG), 31(4):1–11, 2012.
  * [57] Vladimir G Kim, Yaron Lipman, and Thomas Funkhouser.  Blended intrinsic maps.  ACM transactions on graphics (TOG), 30(4):1–12, 2011.
  * [58] Andriy Myronenko and Xubo Song.  Point set registration: Coherent point drift.  IEEE transactions on pattern analysis and machine intelligence, 32(12):2262–2275, 2010.
  * [59] Truc Le, Giang Bui, and Ye Duan.  A multi-view recurrent neural network for 3d mesh segmentation.  Computers and Graphics, 66:103–112, 2017.  Shape Modeling International 2017.
  * [60] Zhenyu Shu, Chengwu Qi, Shiqing Xin, Chao Hu, Li Wang, Yu Zhang, and Ligang Liu.  Unsupervised 3d shape segmentation and co-segmentation via deep learning.  Computer Aided Geometric Design, 43:39–52, 2016.  Geometric Modeling and Processing 2016.
  * [61] Xiaosha Cai, Dong Huang, Chang-Dong Wang, and Chee-Keong Kwoh.  Spectral clustering by subspace randomization and graph fusion for high-dimensional data.  Advances in Knowledge Discovery and Data Mining, 12084:330, 2020\.
  * [62] Xiaobai Chen, Aleksey Golovinskiy, and Thomas Funkhouser.  A benchmark for 3D mesh segmentation.  ACM Transactions on Graphics (Proc. SIGGRAPH), 28(3), August 2009\.


# Lorentz violating inflation and the Swampland

Oem Trivedi  School of Arts and Sciences, Ahmedabad University,
Ahmedabad 380009, Gujarat, India

###### Abstract

The swampland conjectures from String theory have had very interesting
implications for cosmology and particularly for Inflation. It has been shown
that the single field inflationary models in a GR based cosmology are in
unavoidable tensions with these conjectures and so it becomes interesting to
see whether there is a way to overcome the issues of the swampland and single
field inflation in an essentially GR based cosmology. We show that this can be
the case if one considers a certain type of Lorentz violating inflationary
scenario. We work out the requirements for these inflationary models to be
swampland consistent and then show how this scenario allows some popular
inflationary potentials to be swampland consistent as well.

##  1 Introduction

The concept of cosmic inflation has been extremely effective in explaining
different properties of the early universe [1, 2, 3]. It has also been shown
that a wide range of inflationary models are consistent with recent
observational data. But even though inflation has been investigated in a range
of exotic cosmological settings, single field inflationary models still hold a
very important place in the literature. It is then very surprising that these
models, which have been repeatedly shown to be observatinally consistent, are
at odds with the swampland conjectures [4]. The swampland conjectures are a
set of possible quantum gravity consistency criterion having their roots in
string theory, which are aimed for low energy EFT’s in order to check which of
the vast arrays of EFT’s out there today cqn eventually be consistent with
quantum gravity. In particular, the dS and the distannce conjectures are in
direct loggerheads with the observational requirements for inflation.

##  2 The swampland issues of inflation and their Lorentz violating cure

The dS conjecture is based on the proposition that no meta-stable dS spaces
can be found in any consistent theory of quantum gravity 111This bold claim
comes from the fact that it has been very hard to find such dS spaces in the
landscape of string theory  and from the point of view of scalar field
cosmology, it sets a constraint on the potential (this treatment is in Planck
units $m_{p}=1$ ) 222There is another ”refined ” form of this conjecture but
we are not interested in that here

$\frac{|V^{\prime}|}{V}\geq\mathcal{O}(1)$ (1)

The distance conjecture on the other hand is based on the proposition that
there is a limit of applicability of any low energy EFT wich is consistent
with the underlying theory of quantum gravity. From the point of view of
scalar field cosmology, it translates to a limit of traversable distance for
the scalar field

$\Delta\phi\leq\mathcal{O}(1)$ (2)

The conjectures (1-2) set some severe implications for single field inflation.
In order to have sufficient inflation, one requires that the slow roll
parameters during inflation take small values (there are constant roll models
as well where this is not a necessary requirement but we are not considering
them here) but this requirement comes into direct issues with the dS
conjecture. In particular, the $\epsilon$slow roll parameter is given by

$\epsilon=\frac{{V^{\prime}}^{2}}{2V^{2}}$ (3)

During inflation one requires for $\epsilon<<1$ but it is immediately evident
that this cannot be the case if one considers (1). Similarly, another
observational requirement for inflation is that one requires at least 60-70
e-folds of inflation to occur for it to consistently solve the issues of big
bang cosmology. It hence becomes clear that the e-folding number

$N=\frac{\Delta\phi}{\frac{V^{\prime}}{V}}$ (4)

will not even be greater than unity if one considers both (1) and (2). These
are the primary swampland issues of inflation and to eradicate them, we
considered a form of Lorentz violating theory given by the Lagrangian density
[5]

$\mathcal{L}=\Bigg{(}\frac{R}{2\zeta^{2}}-\frac{1}{2}\left(g{{{}_{\mu}}{{}_{\nu}}}+\xi_{1}k^{1}{{{}_{\mu}}{{}_{\nu}}}\right)\partial^{\mu}\phi\partial^{\nu}\phi-V(\phi)\Bigg{)}\sqrt{-g}$
(5)

Using this, we arrived at a modified Klein Gordon equation

$\ddot{\phi}+3H\dot{\phi}+\frac{V^{\prime}(\phi)}{1+\kappa}=0$ (6)

where $\kappa=\xi_{1}\beta_{1}$ and we call this the Lorentz violating
parameter. It can then be showed that if $\kappa>>1$, then inflationary models
in this regime can evade both the primary swampland issues of single field
inflation this essentially GR based cosmology. It can also be showed that this
is true for 3 well known inflatonary models . For the Higgs potential in this
regime with no free parameters, $\simeq 154.495$ and hence it is swampland
consistent. For radion gauge inflation with a free parameter $\alpha$, it can
be shown that for $\alpha>0.0058$, $\kappa$ remains very much greater than
unity. Finally, for a spontaneous symmetry breaking potential with 2 free
parameters $\alpha$ and $\gamma$, the requirement for $\kappa$ is fulfilled
for $\alpha\leq-1,\gamma<0$. The interesting thing is that these models would
be inconsistent with the swampland conjectures in the usual cosmology with
standard lorentz abiding General relativity and hence this work has two main
takeaways.Hence the swampland criterion, and possibly by extension quantum
gravity, might be pointing towards a greater role of Lorentz violations in the
very early universe.

## References

  * [1] AA Starobinskii.  Spectrum of relict gravitational radiation and the early state of the universe.  JETP Letters, 30(11):682–685, 1979.
  * [2] Katsuhiko Sato.  First-order phase transition of a vacuum and the expansion of the universe.  Monthly Notices of the Royal Astronomical Society, 195(3):467–479, 1981.
  * [3] Alan H Guth.  Inflationary universe: A possible solution to the horizon and flatness problems.  Physical Review D, 23(2):347, 1981.
  * [4] Cumrun Vafa.  The string landscape and the swampland.  arXiv preprint hep-th/0509212, 2005.
  * [5] Oem Trivedi.  Lorentz violating inflation and the swampland.  The European Physical Journal Plus, 137(4):1–11, 2022.


# A Spatio-Temporal Aligned SUNet Model FOR LOW-LIGHT VIDEO ENHANCEMENT

###### Abstract

Distortions caused by low-light conditions are not only visually unpleasant
but also degrade the performance of computer vision tasks. The restoration and
enhancement have proven to be highly beneficial. However, there are only a
limited number of enhancement methods explicitly designed for videos acquired
in low-light conditions. We propose a Spatio-Temporal Aligned SUNet (STA-
SUNet) model using a Swin Transformer as a backbone to capture low light video
features and exploit their spatio-temporal correlations. The STA-SUNet model
is trained on a novel, fully registered dataset (BVI), which comprises dynamic
scenes captured under varying light conditions. It is further analysed
comparatively against various other models over three test datasets. The model
demonstrates superior adaptivity across all datasets, obtaining the highest
PSNR and SSIM values. It is particularly effective in extreme low-light
conditions, yielding fairly good visualisation results.

Index Terms—  Low-light, Video enhancement, Swin transformer, Restoration

##  1 Introduction

Images and videos serve as powerful mediums for capturing moments and
conveying information. However, the quality of these visual representations
can often be compromised by distortions and noise, often introduced during
acquisition. For example, incorrect exposure settings are often the result of
an inaccurate balance among the components of the exposure triangle: shutter
speed, aperture, and ISO value. This imbalance can lead to a low signal-to-
noise ratio (SNR), undesirable noise due to high ISO sensitivity, blurring
effects caused by moving objects (low shutter speed), and out-of-focus objects
with a shallow depth of field (large aperture). These distortions inevitably
lead to poor subjective (perceptual) quality but also degrade the performance
of various machine vision tasks such as detection, classification and tracking
as used in surveillance, autonomous driving, medical imaging and natural
history filmmaking [1]. Addressing the challenge of how to reliably and
robustly enhance low-light visual content is thus a key component in future
video production and analytics pipelines.

Fig. 1: Distortions in cropped images of the ‘Faces2’ sequence. (Left) Normal
light. (Middle) Enhanced low light (10% brightness) using histogram matching
to the normal light to visualise distortions under low-light conditions.
(Right) Normal light plus Gaussian noise. Please zoom-in to see a clear effect
of noise.

Fig. 2: Enhancement visualisation in cropped images of the ‘Faces2’ sequence.
(Left) 10% light level input. (Middle) Enhanced result of our method. (Right)
Normal light (100%).

Deep learning techniques have evolved as the most effective tools for image
and video processing, in particular the enhancement of low-light content [2,
3, 4]. However, the common approach of sequentially applying image-based
techniques to each frame in a video often creates temporal inconsistencies and
flickering artefacts. This can, in part, be addressed by enhancing quality on
a frame-by-frame basis through temporal smoothing [5]. A further issue that
impacts low-light video enhancement is the amount of training data available
and it’s quality and diversity. This is compounded by a lack of accurate
ground truth information for imagery acquired in low-light conditions, since
gathering clean ground truth with accurate brightness, colour and motion is
practically impossible. Without spatio-tempoaral alignment between low-light
inputs and their ground truth, it is challenging to accurately evaluate the
performance of trained models.

Some methods overcome this limitation by creating synthetic datasets or
adopting self-supervised learning methods [6]. However, the generalisation on
real-world data is often compromised when training on synthetic data since
augmenting the dataset with over-simplified synthetic noise, often modelled as
zero-mean Gaussian noise, fails to achieve effective results on real world
data captured in the wild. This is particularly evident in low-light scenarios
where noise characteristics are much more complex than Gaussian noise, as
shown in Figure 2. This figure shows a cropped book cover with a human eye,
taken from a sequence named ‘Faces2’ with fast linear motion in the BVI
dataset [7]. When histogram matching is employed on the low light (10% for
noise visualisation), it is noticeable that the shape of the eyeball is
slightly shifted and distorted compared to the normal light ground truth.
Please zoom-in to better visualise the effect. This is a typical scenario
caused by motion blur. The human eye also experiences significant colour
flattening and distortion under low-light conditions. Low-light imagery
typically loses information during acquisition, including edge distortions,
texture loss, motion blur and color changes; these often lead to bias and
clipping problems when used to train machine learning methods. Self-supervised
learning techniques, although independent of paired input-ground truth
datasets and capable of enhancing contrast and brightness by learning subtle
content priors, often demand excessive computational complexity. Challenges
also arise in the context of color restoration and correction in low-light
images where the limited availability of colour information adds to the
complexity.

Given that relatively few low-light enhancement methods have been designed
specifically for low-light video processing, coupled with the ubiquity and
importance of low light content, there is an urgent need for effective low-
light video enhancement methods that exploit spatio-temporal correlations.
Potential for achieving this is offered by recent advances in vision
transformer based mechanisms. Transformers have gained increasing popularity
due to their self-attention mechanism, which accelerates pixel-wise
optimisation and effectively addresses long-range dependencies in the data
[8]. They have underpinned successes in Natural Language Processing (NLP) and
computer vision, outperforming many Convolutional Neural Network (CNN)-based
models.

In this paper, we introduce a lightweight STA-SUNet model based on the Swin
Transformer [9] backbone. Traditional convolution, being content-dependent,
can raise challenges with larger patches and long-range dependencies, failing
to capture global features. SUNet, merging Transformer and UNet, addresses
these issues by substituting the convolution layer with the Swin Transformer
block, surpassing CNN-based methods in image denoising across well-known
benchmark datasets. By realising the importance of frame alignment in video
processing, we integrate an additional feature alignment module as proposed in
EDVR [10]. This module helps to align features spatio-temporally across
multiple input frames at the feature level. These aligned features are then
passed to SUNet for further feature extraction and reconstruction. An example
of our low-light enhancement result is shown in Figure 2.

We train and verify the model with a new high-quality and large input-ground
truth aligned dataset. The model achieves effective enhancement of low-light
video, addressing many of the previously mentioned challenges. We discuss the
implementation, evaluation and analysis of the proposed method, with a
comprehensive performance assessment evaluating the effectiveness of the
model.

Our main contributions can be summarised as follows:

  * •

We propose a lightweight Spatio-Temporal Aligned Swin Transformer-based SUNet
model (STA-SUNet), specifically designed for low-light video enhancement. Our
model provides high adaptivity to natural low-light videos.

  * •

We train and test the STA-SUNet model on a novel, high-quality, fully
registered dataset [7] captured under different light levels, featuring
dynamic scenes. This directly contributes to the increased effectiveness of
the enhancement process.

  * •

We conduct thorough quantitative and qualitative analyses to evaluate the
effectiveness of the proposed method using three different test datasets.

##  2 Related Work

Low-light video enhancement is challenging due to the complexity and
inconsistency of real low-light conditions, which involve multiple
combinations of distortion types. This section discusses prior work on low-
light enhancement, including both single-image and video processing. It also
describes the datasets utilised in these methods, followed by a brief overview
of general video restoration tasks.

###  2.1 Low-light image enhancement (LLIE)

Before the recent advances in deep learning, traditional digital image
processing approaches such as Histogram-equalisation (HE) [11], Retinex theory
models [12], unsharp masking algorithms [13], BM3D (Block-Matching and 3D
filtering) along with its extension to 4D (BM4D)[14], represented the state of
the art. Retinex decomposes an image into reflection and illumination parts
using well-designed priors, treating the estimated reflection as the
restoration result. Although this contributed to Retinex’s popularity, it
lacks adaptivity and is associated with lengthy computational time for complex
optimisation processes. BM3D and BM4D effectively reduce noise through
multidimensional filtering but are less preferred due to the noticeable
artefacts they produce. Histogram-equalisation and unsharp masking, while
effective, can potentially cause over-enhancement in specific regions, leading
to noise amplification. All these methods can result in unrealistic
enhancements and undesired artefacts which, in turn, lead to the loss of
unique information in areas with distinct details.

Recent approaches predominantly leverage deep-learning techniques for
enhancement. LLNet [15] initially employed an auto-encoder for low-light image
denoising and brightening in 2017. More recently, methods such as addressing
low SNR on a spatial basis through an SNR-guided transformer [2], learning
noise through an unfolding variation regularisation model in sRGB space [3],
have also proven to be effective, yielding state-of-the-art results.

In the context of frame-to-frame processing, single-image processing struggles
to aggregate temporal information between frames in video restoration tasks
effectively and may face flickering problems if applied to videos directly.
This challenge makes it more difficult to adapt single-image enhancement
models to video, especially when high levels of motion or dynamic textures are
present.

###  2.2 Low-light video enhancement (LLVE)

Despite the rapid advancements in deep learning for low-light image
enhancement, there remains a noticeable gap when it comes to extending these
models to enhance low-light video [16].

Most approaches rely on supervised learning methods, such as ResNet [17] and
UNet [18], for feature extraction. In 2018, Chen et al. introduced the real
low-light image SID dataset [19], initially trained using UNet [18]. A Siamese
network based on ResNet was proposed in [20] along with the dataset DRV,
assuming that modelling with static scenes could generalise well to dynamic
scenes. SMOID [21] modified UNet to enable multiple convolutional layers to
handle the displacement between frames caused by moving objects. However, its
effectiveness is compromised with fast-moving objects exhibiting substantial
temporal displacement. SDSD [22] introduces two branches of the network for
noise reduction and illumination enhancement based on Retinex theory. Issues
such as spatial non-alignment [22], limited motion variation [20], and data
scarcity [21], further complicate the scenario. Some research methods assume
that inputs are in the raw Bayer space [20, 21]. However, relying on raw Bayer
format in datasets constraints the model’s accessibility to the wider research
community, as consumer cameras typically do not support raw video formats.

###  2.3 Video restoration

As the methods for low-light video enhancement are limited, it is worth
considering other video restoration approaches. One such example is EDVR [10],
which is implemented with DCN (Deformable Convolution) [23] and attention
mechanisms to align and fuse features both spatially and temporally. It
demonstrates superior performance on video super-resolution and deblurring.
Our previous work, integrating feature-level alignment with the traditional
UNet (PCDUNet) [7], reveals effectiveness in enhancing low light videos with
motion. Transformer models have found success in natural language processing
(NLP) and have been widely applied to vision tasks. The Swin Transformer [24]
is a state-of-the-art method that addresses issues in pixel-wise vision tasks.
It effectively solves problems posed by non-linear, high computational
complexity relative to image size, outperforming many convolutional neural
networks (CNN)-based methods by constructing hierarchical feature maps with
shifted windows. This gave rise to SUNet [9], which adopts the Swin
Transformer as a backbone and introduces a dual up-sample layer to alleviate
the checker-board effect in traditional UNet, achieving excellent results in
image denoising.

##  3 Proposed Method

Fig. 3: Proposed STA-SUNet framework

The proposed STA-SUNet method, shown in Fig. 3, is specifically designed for
low-light video enhancement. The proposed method starts with aligning multiple
input frames at the spatio-temporal feature level. Subsequently, the
enhancement is performed using a U-Net-like architecture with Swin Transformer
[24]. The end-to-end framework is trained with L1 loss. Detailed explanations
of the architecture are provided below.

###  3.1 Feature alignment

Multiple input images, denoted by $I_{t+i},i\in\\{-N,-N+1,\ldots,N-1,N\\}$,
initially proceed to the feature space via convolutional layers before passing
through the alignment module. $N$ represents the number of neighbouring frames
at time $t$. This module aligns features of $N$ neighbouring frames with the
target frame $I_{t}$ using deformable convolution. This aims to utilise
information from adjacent frames in a video while minimising disparities.
Pyramid processing and cascading refinement gradually reduce differences
between adjacent frames and the target frame. At a lower level $L$, features
of two neighbouring frames $I_{t+i}$ and $I_{t}$ concatenate, obtaining the
offset $\delta^{L}_{t+i}$ after consecutive convolutional layers, denoted as
$L_{a}$ and $L_{b}$. This offset, combined with adjacent frame features
$I^{L}_{t+i}$, inputs into deformable convolution, denoted as DC, at that
level. They are then upsampled together with a factor of 2 ($C^{L+1}$, where
$C$ represents $\delta$ and $I$) and input into the upper layer ($L$+1). For
subsequent layers, the offset for each layer concatenates with the upsampled
feature from the previous layer. The offset-feature prediction process can be
represented as follows:

$\begin{gathered}\delta^{L}_{t+i}=L_{a}([I_{t+i},I_{t}],\delta^{L+1}_{t+i}),\\\
(\hat{I}_{t+i})^{L}=L_{b}(DC(I^{L}_{t+i},\delta^{L}_{t+i}),(\hat{I}_{t+i})^{L+1})\end{gathered}$
(1)

This iterative 3-layered pyramid process continues until aligned features
cascade. A final deformable convolution produces the output of this alignment
module - i.e. aligned features. These features are subsequently upsampled in
the first up-sampling layer, as depicted in Figure 3, using the Pixel Shuffle
technique [25]. This enables patch partitioning in the feature extraction
process.

###  3.2 Enhancement module

Following Swin Transformer UNet (SUNet) [9], the enhancement module extracts
and concatenates the features of neighbouring patches through a $3\times 3$
convolution layer and patch merging, which can be viewed as a downsampling
process. SUNet replaces traditional UNet convolution layers with Swin
Transformer blocks, with 8 layers in each block. These Swin Transformer layers
consist of multiple groups of window multi-head self-attention (W-MSA) and
shifted-window multi-head self-attention (SW-MSA). To address the lack of
connections between windows in the window-based self-attention module, a
shifted-window approach is used for cross-window connections among multiple
layers, creating a hierarchical representation. An efficient patch computation
method known as cycle-shifting is applied to self-attention in the shifted
window partitioning, reducing computational time. This approach proves
effective when dealing with situations where extra padding would increase
computation, especially in the presence of small windows during shifting. The
computation of consecutive Swin Transformer blocks in this process follows:

$\begin{gathered}\hat{o}^{L}=\text{A}(LN(o^{L-1}))+o^{L-1},\\\
o^{L}=MLP(LN(\hat{o}^{L}))+\hat{o}^{L}\\\
\hat{o}^{L+1}=\text{B}(LN(o^{L}))+o^{L},\\\
o^{L+1}=MLP(LN(\hat{o}^{L+1}))+\hat{o}^{L+1}\end{gathered}$ (2)

Here, A and B represent window-based modules using the normal (W-MSA) and
shifted-window (SW-MSA) methods, respectively. The output $o$ is the output
feature for each layer $L$. A Layer-Norm layer, denoted as $LN$ is implemented
before each MSA module and each MLP (multi-layer perception), with a residual
connection applied after each module. In the second upsampling stage shown in
Figure 3, the traditional transposed convolution is substituted with a
combination of Bi-linear and Pixel Shuffle techniques. This blend, referred to
as dual upsampling, is implemented to reduce the checker-board effect.
Following the original paper, SUNet architecture has 5 layers, utilising skip
connections to transfer feature maps from the patch merging stage to the
second upsampling stage. Finally, a last $3\times 3$ convolution layer is
employed to reconstruct the restored frame with enhanced lightning.

##  4 Experiments

###  4.1 Dataset

We employed the BVI low-light dataset [7] to evaluate our proposed method.
This dataset offers high-quality low-light videos, along with their
corresponding normal light ground truths, ensuring full registration in both
spatial and temporal dimensions. This supports the training of supervised
learning models and facilitates full-reference quality assessment. These
videos were captured using a Sony Alpha 7SII camera, with a Kessler CineDrive
shuttle dolly system. There are a total of 40 scenes with various contents,
textures and motions. As shown in Figure 4, each scene provides two low-light
levels of 10% and 20%, as well as normal lighting (100% light level). The
videos are captured in full HD resolution (1920x1080 pixels) in a standard RGB
format. Following [7], the training and testing sets have been pre-defined,
with 32 and 8 scenes used for training and testing, respectively.

###  4.2 Experiment settings

The STA-SUNet network is trained on the BVI dataset without relying on any
pre-trained networks. We use 5 RGB frames as inputs, with an image size of
512$\times$512, a patch size of 4, a window size of 8, and a batch size of 1.
The training data is augmented with random flipping and cropping as a pre-
processing step. Adam optimiser is used with the initial learning rate of
$1\times 10^{-6}$. The loss function used for optimisation is the L1 loss. We
implement our model on PyTorch with a single NVIDIA GeForce RTX 3090 GPU. The
metrics used for evaluation are PSNR (Peak Signal-to-Noise Ratio) and SSIM
(Structural Similarity Index Measure), where higher values indicate better
performance.

Fig. 4: Low light data example: from top to bottom, light levels of 10%, 20%,
and 100% (normal light). from left to right, soft toys and books with faces.

###  4.3 Impact of different light levels

The impact of light levels on the model’s performance is analysed and
presented below through quantitative analysis and illustrative figures. The
STA-SUNet model is firstly trained on a dataset at light levels of 10%, and
20%, respectively, and then on the collective dataset of light levels at both
10% and 20%. Table 1 summarises the results. It is observed that the model,
when trained and tested using data at the same light level, achieves better
performance than when using data with different light levels. However, the
model trained under extremely low lighting conditions, such as 10% light level
only, tends to perform poorly when tested with data in brighter light
conditions, such as 20% light level. This is indicated by PSNR and SSIM values
of 11.30 and 0.633, respectively, in the first row on Table 1. Conversely, the
model trained with data under 20% light levels only tends to perform worse
when tested with data taken from 10% light conditions, as shown by PSNR and
SSIM values in the second row of Table 1. The model demonstrates better
performance when trained across a diverse dataset at different light levels,
as illustrated in the third column in Table 1.

Table 1: Performance of the STA-SUNet model when trained and tested with different light levels test | 10% | 20% | 10%+20%
---|---|---|---
train | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM
10% | 27.32 / 0.847 | 11.30 / 0.633 | 19.32 / 0.740
20% | 12.04 / 0.630 | 31.41 / 0.930 | 21.74 / 0.781
10%+20% | 24.44 / 0.822 | 27.84 / 0.876 | 26.14 / 0.849
Fig. 5: Visualisation results when using 5-frame inputs comparison for low-
light enhancement on cropped images of frame 102 in the ‘Figures2’ sequence
from the BVI dataset. (Top) From left to right, 10% light level input,
enhanced results, and 100% normal light groundtruth. (Bottom) From left to
right, 20% light level input, enhanced results, and 100% normal light
groundtruth.

We further investigated how our proposed method deals with low-light
distortion. First, we mitigated the influence of varying brightness and color
balance as follows. Before feeding the test data into the model, histogram
matching is conducted to align the histogram of the test data with that of the
training data, leaving noise in the results. The PSNR and SSIM values
demonstrate a significant improvement upon such post-processing, as depicted
in Table 2. This suggests that restoring brightness and colour changes is a
challenging task. Fine-tuning state-of-the-art denoisers may not yield
immediate results.

Table 2: Performance of the STA-SUNet model when trained with light levels of
10% and 20% and tested with histogram matching With histogram matching
---
test | 10% | 20% | 10%+20%
train | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM
10% | 27.32 / 0.847 | 27.28 / 0.860 | 27.30 / 0.853
20% | 27.15 / 0.811 | 31.41 / 0.930 | 29.28 / 0.871

###  4.4 Impact of number of input frames

Table 3 shows the test results for different numbers of input frames trained
on the mixed light samples and tested on the 10% light level. Through our
experiments we observe that, in general, an increasing number of input frames
from 1 to 5 improves the resulting enhanced quality as temporal consistency
improves. While using only three input frames proves insufficient for
gathering enough temporal information. The advantage of using multiple frames
as input is particularly relevant in processing video sequences, where spatio-
temporal correlations often exist among adjacent frames. This obviously trades
off against memory requirements.

Table 3: Performance of the STA-SUNet model when trained on different numbers of input frames Input frame | 1 | 3 | 5
---|---|---|---
PSNR/SSIM | 18.41 / 0.677 | 18.45 / 0.680 | 24.44 / 0.822

###  4.5 Performance comparison

Table 4: Performance comparison of the models tested on different datasets (trained on the BVI dataset). Dataset |  DRV [20] |  SDSD [22] |  BVI [7]
---|---|---|---
Methods | PSNR/SSIM | PSNR/SSIM | PSNR/SSIM
UNet [18] | 11.39 / 0.282 | 16.89 / 0.667 | 21.82 / 0.777
PCDUNet [7] | 18.52 / 0.696 |  19.36 / 0.729 | 24.51 / 0.845
SDSD-net [22] | 16.18 / 0.595 | 18.83 / 0.688 | 10.30 / 0.476
SUNet [9] | 17.70 / 0.599 | 14.89 / 0.635 | 18.95 / 0.645
STA-SUNet |  18.74 / 0.699 |  19.97 / 0.714 |  26.14 / 0.873
Table 5: Average performance comparison Methods | Average PSNR/SSIM
---|---
UNet [18] | 16.70 / 0.575
PCDUNet [7] | 20.79 / 0.757
SDSD-net [22] | 15.10 / 0.586
SUNet [9] | 17.18 / 0.626
STA-SUNet |  21.62 / 0.762

To further explore the effectiveness of the STA-SUNet model, we conduct a
comparative performance evaluation by testing the model on three datasets,
namely, DRV [20], SDSD [22], and BVI [7]. The DRV data undergoes an adjustment
process, with its green channel reduced by half to standard RGB, ensuring a
fair comparison. The objective results are illustrated in Table 4. The last
column highlights the importance of training on a fully-registered dataset,
with the STA-SUNet model achieving the highest PSNR values. Table 5 presents
the average objective results across all three test datasets. In general, the
STA-SUNet model demonstrates superior adaptivity across all datasets,
achieving the highest average PSNR and SSIM values. The visualisation results
are shown in Figure 5 with cropped images of frame 102 in the ‘Figures2’
sequence. It is evident from the results that the STA-SUNet model demonstrates
outperforms, especially under extreme low-light conditions (10% light level),
producing good visual results. While 20% light levels are generally easier for
enhancement models, both PCDUNet and UNet exhibit over-exposure at this level,
even though PCDUNet has similar objective performance to STA-SUNet. Undesired
speckle effect in the output of SDSD-net, as shown in Figure 6, contributes to
its low objective results.

Fig. 6: Visualisation comparison between STA-SUNet and SDSD-net: from left to
right, SDSD-net result, STA-SUNet result and normal light 100%.

##  5 Conclusion

We propose a Spatio-Temporal Aligned SUNet (STA-SUNet) model, leveraging the
Swin Transformer backbone, to enhance low-light videos. Trained and validated
on a novel, fully-registered dataset with diverse motions, our model’s
effectiveness is evaluated on two other datasets as well. Through extensive
experiments, we analyze the impact of various light levels and the temporal
consistency. Quantitative results show the STA-SUNet outperforming other
models in terms of PSNR and/or SSIM across all datasets. Comparative analyses
against UNet, PCDUNet, SDSD-net, and SUNet are conducted. By achieving the
highest average PSNR and SSIM values across all datasets, our model
demonstrates superior adaptivity, especially in extreme low-light conditions,
yielding fairly good visualisation results.

## References

  * [1] Anqi Yi and Nantheera Anantrasirichai,  “A Comprehensive Study of Object Tracking in Low-Light Environments,”  arXiv:2312.16250, 2023.
  * [2] Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia,  “SNR-Aware Low-Light Image Enhancement,”  IEEE/CVF CVPR, pp. 17714–17724, 2022.
  * [3] Chuanjun Zheng, Daming Shi, and Wentian Shi,  “Adaptive Unfolding Total Variation Network for Low-Light Image Enhancement,”  IEEE/CVF ICCV, pp. 4419–4428, 2021.
  * [4] Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo,  “Toward Fast, Flexible, and Robust Low-Light Image Enhancement,”  arXiv:2204.10137, 2022.
  * [5] Nantheera Anantrasirichai and David Bull,  “Contextual Colorization and Denoising for Low-Light Ultra High Resolution Sequences,”  IEEE/CVF ICIP proc., pp. 1614–1618, 2021.
  * [6] Danai Triantafyllidou, Sean Moran, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh,  “Low Light Video Enhancement using Synthetic Data Produced with an Intermediate Domain Mapping,”  arXiv:2007.09187, 2020.
  * [7] Nantheera Anantrasirichai, Ruirui Lin, Alexandra Malyugina, and David Bull,  “BVI-Lowlight: Fully Registered Datasets for Low-light Image and Video Enhancement,”  arXiv:2402.01970, 2024.
  * [8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention Is All You Need,”  arXiv:1706.03762, 2023.
  * [9] Chi-Mao Fan, Tsung-Jung Liu, and Kuan-Hsien Liu,  “SUNet: Swin Transformer UNet for Image Denoising,”  arXiv:2202.14009, 2022.
  * [10] Xintao Wang, Kelvin C. K. Chan, Ke Yu, Chao Dong, and Chen Change Loy,  “EDVR: Video Restoration with Enhanced Deformable Convolutional Networks,”  arXiv:1905.02716, 2019.
  * [11] Haidi Ibrahim and Nicholas Sia Pik Kong,  “Brightness Preserving Dynamic Histogram Equalization for Image Contrast Enhancement,”  IEEE/CVF TCE, vol. 53, no. 4, pp. 1752–1758, 2007.
  * [12] Xiang Fu, Ying Liao, Di Zeng, Yixin Huang, Xiaopeng Zhang, and Xinghao Ding,  “A Probabilistic Method for Image Enhancement With Simultaneous Illumination and Reflectance Estimation,”  IEEE/CVF TIP, vol. 24, no. 12, pp. 4965–4977, 2015.
  * [13] Guang Deng,  “A Generalized Unsharp Masking Algorithm,”  IEEE/CVF TIP, vol. 20, no. 5, pp. 1249–1261, 2011.
  * [14] Matteo Maggioni, Vladimir Katkovnik, Karen Egiazarian, and Alessandro Foi,  “Nonlocal transform-domain filter for volumetric data denoising and reconstruction,”  IEEE TIP, vol. 22, no. 1, pp. 119–133, 2013.
  * [15] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar,  “LLNet: A Deep Autoencoder Approach to Natural Low-light Image Enhancement,”  Pattern Recognition, vol. 61, pp. 650–662, 2017.
  * [16] Chongyi Li, Chunle Guo, Linghao Han, Jun Jiang, Ming-Ming Cheng, Jinwei Gu, and Chen Change Loy,  “Low-Light Image and Video Enhancement Using Deep Learning: A Survey,”  arXiv:2104.10729, 2021.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep Residual Learning for Image Recognition,”  arXiv:1512.03385, 2015.
  * [18] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-Net: Convolutional Networks for Biomedical Image Segmentation,”  MICCAI, pp. 234–241, 2015.
  * [19] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun,  “Learning to See in the Dark,”  arXiv:1805.01934, 2018.
  * [20] Chen Chen, Qifeng Chen, Minh Do, and Vladlen Koltun,  “Seeing Motion in the Dark,”  IEEE/CVF ICCV, pp. 3184–3193, 2019.
  * [21] Haiyang Jiang and Yinqiang Zheng,  “Learning to See Moving Objects in the Dark,”  IEEE/CVF ICCV, pp. 7323–7332, 2019.
  * [22] Ruixing Wang, Xiaogang Xu, Chi-Wing Fu, Jiangbo Lu, Bei Yu, and Jiaya Jia,  “Seeing Dynamic Scene in the Dark: A High-Quality Video Dataset with Mechatronic Alignment,”  IEEE/CVF ICCV, pp. 9680–9689, 2021.
  * [23] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei,  “Deformable convolutional networks,”  arXiv:1703.06211, 2017.
  * [24] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows,”  arXiv:2103.14030, 2021.
  * [25] Wenzhe Shi, et al.,  “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,”  arXiv:1609.05158, 2016.