#  Geometry of Degeneracy in Potential and Density Space

Markus Penz  Basic Research Community for Physics, Innsbruck, Austria
<EMAIL_ADDRESS>Robert van Leeuwen  Department of Physics, Nanoscience Center,
University of Jyväskylä, Finland

###### Abstract

In a previous work [J. Chem. Phys. 155, 244111 (2021)], we found
counterexamples to the fundamental Hohenberg–Kohn theorem from density-
functional theory in finite-lattice systems represented by graphs. Here, we
demonstrate that this only occurs at very peculiar and rare densities, those
where density sets arising from degenerate ground states, called degeneracy
regions, touch each other or the boundary of the whole density domain.
Degeneracy regions are shown to generally be in the shape of the convex hull
of an algebraic variety, even in the continuum setting. The geometry arising
between density regions and the potentials that create them is analyzed and
explained with examples that, among other shapes, feature the Roman surface.

##  I Introduction

The mapping from potentials to the corresponding one-particle ground-state
density in a many-particle quantum system and the possibility of its inversion
forms the basis of density-functional theory (DFT) [1; 2; 3; 4] and many of
its variants [5; 6; 7; 8; 9]. This theory is widely employed in electronic-
structure calculations, allowing for a good balance between accuracy and
computational cost. The problem can also be viewed as a control problem, where
a potential that produces a given target density is sought. Despite its
prominent role, the topological structure of the density-potential mapping is
poorly studied, even within ground-state DFT. Notable exceptions are a work by
Ullrich and Kohn [10] and a recent paper by Garrigue [11]. The former
investigated the dimensionality of sets in potential space preserving a
certain degeneracy in a finite-lattice system. This indicated that while
degeneracy is quite common in the density domain, it remains rare in potential
space (yet, the latter statement does not follow in their proof that fails to
consider possible linear dependencies of conditions, see Section V). The
authors of the paper at hand also started their inquiries in this area by
focusing on finite-lattice systems, generalized by graphs, and a number of
surprising results were already found, including the nullity of the celebrated
Hohenberg–Kohn theorem for systems with special symmetry [12]. The studied
examples suggested a far-reaching conjecture on the special geometry of the
mapping, about the connection between densities that arise from degenerate
ground states, later called _degeneracy regions_ , and those for which the
Hohenberg–Kohn theorem fails, as well as their corresponding potentials. This
conjecture will be proven here (Section VI), but first the shape of such
degeneracy regions will be clarified, with the surprising discovery of highly
intricate objects from algebraic geometry, e.g., the Roman surface, for which
we give a basic classification (Sections II-III). With the same techniques we
give precise conditions where non-pure-state $v$-representable densities, a
concept from DFT, arise (Section IV). To establish a link to the full geometry
of the mapping, the study of Ullrich and Kohn [10] is rendered more precise
and a new proof based on Rellich’s theorem is provided (Section V). The
examples are all for small lattice systems and spinless fermions, but it must
be stressed that the results from Sections II-IV also apply to continuum
systems and everything can also be extended to incorporate spin by simply
including additional internal degrees of freedom. Since this work can be
considered a sequel to our previous paper [12], we recommend looking there for
a deepened understanding.

The vector space that includes one-particle densities shall generally be
denoted $\mathsf{Dens}$ and similarly $\mathsf{Pot}$ is the vector space of
one-body potentials. In a lattice system with $M$ vertices this means that
$\mathsf{Dens}=\mathsf{Pot}=\mathbb{R}^{M}$, while in a continuum setting one
can choose $\mathsf{Dens}=L^{1}(\mathbb{R}^{3})\cap L^{3}(\mathbb{R}^{3})$ and
$\mathsf{Pot}=L^{\infty}(\mathbb{R}^{3})+L^{3/2}(\mathbb{R}^{3})$, its
topological dual, like in Lieb [13]. We define the density map
$\rho:\mathcal{H}\to\mathsf{Dens},\Psi\mapsto\langle\Psi,\hat{\rho}\Psi\rangle/\|\Psi\|^{2}$
(also we use
$\rho:\mathcal{D}\to\mathsf{Dens},\Gamma\mapsto\operatorname{Tr}(\hat{\rho}\Gamma)$
on the set of all density matrices for ensemble states $\mathcal{D}$) that
takes fermionic many-particle states in Hilbert space to their respective one-
particle density via the density operator $\hat{\rho}$ [4]. We study a class
of Hamiltonians $H_{v}=H_{0}+V$ that only differ with respect to their real,
scalar one-body potential $v$. Here, $H_{0}$ is the fixed (internal) part that
is always assumed to be real (i.e., it does not include a vector potential,
or, more generally, allows for time-reversal symmetry), while $V$ is the
operator acting on $\mathcal{H}$ that corresponds to the one-body potential
$v$. The following lemma will be important for the choice of eigenvectors of
such Hamiltonians.

######  Lemma 1.

If $H_{v}$ is real symmetric and has a $g$-dimensional eigenspace
$\mathcal{U}$ then this space is spanned by $g$ _real_ orthonormal vectors
$\\{\Phi_{k}\\}_{k=1}^{g}$ with _complex_ coefficients,
$\mathcal{U}=\operatorname{span}_{\mathbb{C}}\\{\Phi_{1},\ldots,\Phi_{g}\\}$.

###### Proof.

For the $g$-dimensional eigenspace $\mathcal{U}$ with eigenvalue $E$ choose a
general basis $\\{\Psi_{k}\\}_{k=1}^{g}$. Then observe that since $H_{v}$ and
$E$ are real, the real and imaginary part of those vectors are eigenvectors as
well,
$H_{v}\mathop{\mathsf{missing}}{Re}\Psi_{k}=E\mathop{\mathsf{missing}}{Re}\Psi_{k}$
and
$H_{v}\mathop{\mathsf{missing}}{Im}\Psi_{k}=E\mathop{\mathsf{missing}}{Im}\Psi_{k}$.
This gives a total of $2g$ real eigenvectors that span the whole $\mathcal{U}$
with complex coefficients, from which one can get $g$ real and orthonormal
eigenvectors $\\{\Phi_{k}\\}_{k=1}^{g}$ by the Gram–Schmidt process that span
the same space. ∎

In particular, we can always already choose an orthonormal basis of real
eigenstates to span the ground-state eigenspace, a property that we will use
extensively below. The following theorem allows us in principle to switch any
discussion about ground states over to just densities. This implies an
enormous reduction of complexity.

######  Theorem 2.

Assume that two Hamiltonians $H_{v}=H_{0}+V,H_{v^{\prime}}=H_{0}+V^{\prime}$
that differ only in their scalar one-body potentials $v,v^{\prime}$ share a
common ground-state density $\rho$. Then an (ensemble) ground state $\Psi$
$(\Gamma)$ of $H_{v}$ with density $\rho$ is also an (ensemble) ground state
of $H_{v^{\prime}}$ and vice versa.

This result means that to every ground-state density $\rho$ we can assign a
ground state $\Psi(\rho)$ (or, more generally, a class of ground states or
ensembles with the same density), _irrespective_ of the present potential. As
such, this is well-known as a part of the Hohenberg–Kohn theorem. We refer to
our previous work [12] for two distinct proofs of the above theorem. Already
Theorem 2 allows to define a universal energy functional
$F_{\mathrm{HK}}(\rho)=\langle\Psi(\rho),H_{0}\Psi(\rho)\rangle$ for every
ground-state density $\rho$ that gives the lowest possible internal (kinetic
plus interactions) energy of a pure state with the prescribed density. It is
customary to instead use the much more well-behaved constrained-search
functional over ensemble states
$F(\rho)=\inf_{\Gamma\mapsto\rho}\operatorname{Tr}(H_{0}\Gamma)$ [13] that is
convex and equal to $F_{\mathrm{HK}}$ on its domain. Any ground-state density
$\rho$ to a potential $v$ will then minimize the functional $F(\rho)+\langle
v,\rho\rangle$ since this gives the total energy $E(v)$. An excellent
mathematical review on the universal functionals in DFT is Lewin  _et al._
[14].

The second part of the Hohenberg–Kohn theorem, which then maps ground states
to unique potentials (modulo a constant) and in conjunction establishes the
mapping from ground-state densities to potentials, is omitted here. While it
holds in the continuum case for a large class of potentials [15], it generally
fails for finite lattice systems [12] that will serve as the prime examples in
this work. Where it fails, we find densities that can be represented by
multiple potentials that differ in more than a constant. Such densities we
call _non-uniquely $v$-representable_ (“ _non-uv_ ”) and one observes that
they always arise when degeneracy regions touch, a feature that is the main
element of our geometry theorem in Section VI. Thus we start by investigating
the shape of such density sets.

##  II State subspaces under the density map

Let $\mathcal{U}$ be a $g$-dimensional subspace of a given Hilbert space
$\mathcal{H}$ spanned by _real_ vectors $\\{\Phi_{k}\\}_{k=1}^{g}$ with
_complex_ coefficients. The Hilbert space can be $\mathcal{H}=\mathbb{C}^{L}$,
$L=\binom{M}{N}$, in the case of $N$ spinless fermionic particles on an
$M$-vertex lattice [12, Sec. II.B], or the usual anti-symmetric many-particle
Hilbert space for continuum systems
$\mathcal{H}=\Lambda^{N}L^{2}(\mathbb{R}^{3})$. It is possible to extend to
particles with spin by including the internal degrees of freedom into the
Hilbert space of the individual particles. Later, the subspace $\mathcal{U}$
will be the space of $g$-fold degenerate ground states of a Hamiltonian
$H_{v}$ and so we call $g$ the _degree_. We are interested in the image of
$\mathcal{U}$ under $\rho$. Let $x=(x_{1},\ldots,x_{g})\in\mathbb{C}^{g}$,
$\|x\|=1$, be the coordinates for a normalized state $\Psi\in\mathcal{U}$ with
respect to the real, orthonormal subspace basis $\\{\Phi_{k}\\}_{k=1}^{g}$ of
$\mathcal{U}$. The density is then evaluated as

$\displaystyle\rho(\Psi)=\sum_{k=1}^{g}|x_{k}|^{2}\rho(\Phi_{k})+\sum_{\begin{subarray}{c}k,l=1\\\
k<l\end{subarray}}^{g}2\mathop{\mathsf{missing}}{Re}(x_{k}^{*}x_{l})\langle\Phi_{k},\hat{\rho}\Phi_{l}\rangle$
(1)

and we will later also use the notation $\rho(x)$ for a fixed basis
$\\{\Phi_{k}\\}_{k=1}^{g}$. We can also define the density map for ensembles
$\Gamma\in\mathcal{D}$ in $\mathcal{U}$, i.e.,
$\Gamma\mathcal{H}\subseteq\mathcal{U}$, that we call $\rho$ as well. If
$P_{j}$ are the projections on orthonormal pure states $\Psi_{j}$ from the
subspace $\mathcal{U}$ and $\Gamma=\sum_{j}w_{j}P_{j}$ is a density matrix
with coefficients $w_{j}\geq 0$, $\sum_{j}w_{j}=1$, then we define accordingly

$\rho(\Gamma)=\sum_{j}w_{j}\rho(\Psi_{j}).$ (2)

For the further discussion we define the concept of a _density region_ as the
set of all densities belonging to states in $\mathcal{U}$. Later this will be
a _degeneracy region_ , the set of all densities belonging to states in the
ground-state eigenspace $\mathcal{U}$ for a given potential $v$. Therein, we
differentiate three levels, by first mapping only
$\mathcal{U}_{\mathbb{R}}=\operatorname{span}_{\mathbb{R}}\\{\Phi_{1},\ldots,\Phi_{g}\\}$,
i.e., the subspace spanned by the basis vectors with only real expansion
coefficients, then the linear span
$\mathcal{U}=\operatorname{span}_{\mathbb{C}}\\{\Phi_{1},\ldots,\Phi_{g}\\}$
for general, complex expansion coefficients (which just means the whole
subspace), and lastly by forming the full density region as all densities from
that subspace including mixed states.

$\displaystyle D_{\mathbb{R}}=\rho(\mathcal{U}_{\mathbb{R}})$ (3a)
$\displaystyle D_{\mathbb{C}}=\rho(\mathcal{U})$ (3b) $\displaystyle
D=\rho(\\{\Gamma\in\mathcal{D}\mid\Gamma\mathcal{H}\subseteq\mathcal{U}\\})$
(3c)

From the definition we see that the density region does not depend on the
choice of basis for $\mathcal{U}$ and that $D_{\mathbb{C}}$ is limited to
pure-state densities, while $D$ also includes non-pure-state densities (see
Section IV where it is shown that in general $D_{\mathbb{C}}\neq D$). That
$D_{\mathbb{R}}\subset D_{\mathbb{C}}$ follows by definition and $D$ includes
the previous two sets, since $D$ is the set of all convex combinations of
pure-state densities as expressed in (2). We thus find the following sequence
of inclusions,

$D_{\mathbb{R}}\subset D_{\mathbb{C}}\subseteq D\subset\mathsf{Dens}.$ (4)

As the continuous image of a compact and connected set those sets are all
compact and connected. In order to get $D_{\mathbb{C}}$ from $D_{\mathbb{R}}$
we define the _segment set_ for an arbitrary set $X$ in a vector space as

$\operatorname{seg}X=\\{\lambda x+(1-\lambda)y\mid x,y\in X,0\leq\lambda\leq
1\\}.$ (5)

Hence, instead of taking an arbitrary, finite number of points and form their
convex combination, like in the convex hull, we only take two points in the
segment set. The segment set is equivalent to the geometric join, as
introduced in Bárány and Karasev [16], of a set with itself. The construction
of $D_{\mathbb{C}}$ from $D_{\mathbb{R}}$ is then furnished by the following
lemma.

######  Lemma 3.

$D_{\mathbb{C}}=\operatorname{seg}D_{\mathbb{R}}$.

###### Proof.

Take any density in $D_{\mathbb{C}}$, then it must be the density of a
normalized $\Psi\in\mathcal{U}$. Such a vector can always be split up into its
real and imaginary part as
$\Psi=\sqrt{\lambda}\Psi_{1}+\mathrm{i}\sqrt{1-\lambda}\Psi_{2}$ with
$0\leq\lambda\leq 1$ and
$\Psi_{1},\Psi_{2}\in\operatorname{span}_{\mathbb{R}}\\{\Phi_{1},\ldots,\Phi_{g}\\}$,
both normalized as well. Now,

$\displaystyle\rho(\Psi)$
$\displaystyle=\langle\Psi,\hat{\rho}\Psi\rangle=\lambda\langle\Psi_{1},\hat{\rho}\Psi_{1}\rangle+(1-\lambda)\langle\Psi_{2},\hat{\rho}\Psi_{2}\rangle$
(6) $\displaystyle=\lambda\rho(\Psi_{1})+(1-\lambda)\rho(\Psi_{2}),$

where the mixed term cancels since $\Psi_{1}$ and $\Psi_{2}$ are real. Since
$\rho(\Psi_{1}),\rho(\Psi_{2})\in D_{\mathbb{R}}$ this proves the assertion. ∎

By (2) the $\rho(\Gamma)$ is a convex combination of pure-state densities and
it holds $D=\operatorname{ch}D_{\mathbb{C}}$, the convex hull, so the
following result follows directly from Lemma 3.

######  Corollary 4.

$D=\operatorname{ch}D_{\mathbb{R}}$.

This greatly simplifies the analysis of a density region $D$, since we can
limit ourselves to the study of $D_{\mathbb{R}}$ and then just form the convex
hull. In order to get $D_{\mathbb{R}}$, we choose $x\in\mathbb{R}^{g}$,
$\|x\|=1$, and split the map $x\mapsto\rho(x)$ from coordinates to densities
in (1) into two stages. First, form the $(g+1)g/2$ second-order monomials
$x_{k}^{2}$ and $x_{k}x_{l}$ that then live in a larger space,

$\displaystyle\nu:\mathbb{R}^{g}$
$\displaystyle\longrightarrow\mathbb{R}^{(g+1)g/2}$ (7)
$\displaystyle(x_{1},\ldots,x_{g})$
$\displaystyle\longmapsto(x_{1}^{2},\ldots,x_{g}^{2},x_{1}x_{2},\ldots,x_{g-1}x_{g}).$

This map is known as the Veronese embedding in algebraic geometry [17; 18] and
its image of the unit sphere is the Veronese variety $\mathbb{V}_{g}$. We
immediately see that antipodal points $\pm x$ on the unit sphere are mapped to
the same vector, so the map is usually studied within projective space. Great
circles on the unit sphere are mapped to a ellipses [19, Prop. 3.3]. The
second stage consists of _linearly_ mapping to densities with a map $P$ formed
by the factors $\rho_{\overline{k}}=\rho(\Phi_{k})$ and
$\rho_{\overline{kl}}=2\langle\Phi_{k},\hat{\rho}\Phi_{l}\rangle$ as given in
(1),

$\displaystyle\rho=P\circ\nu:\mathbb{R}^{g}$
$\displaystyle\longrightarrow\mathsf{Dens}$ (8)
$\displaystyle(x_{1},\ldots,x_{g})$
$\displaystyle\longmapsto\sum_{k=1}^{g}x_{k}^{2}\rho_{\overline{k}}+\sum_{\begin{subarray}{c}k,l=1\\\
k<l\end{subarray}}^{g}x_{k}x_{l}\rho_{\overline{kl}}.$

The notation with the overlined index has been introduced to avoid confusion
with the density $\rho_{i}$ at a certain lattice point. This already shows
that $D_{\mathbb{R}}$ is the linear map $P$ applied to $\mathbb{V}_{g}$ and
thus forms a parametrized algebraic variety, while the whole density region
$D$ is then just its convex hull. In order to arrive at a possible
classification for $D$, we determine its dimension within density space. This
is given by the number of linearly independent factors $\rho_{\overline{k}}$
and $\rho_{\overline{kl}}$, that is a total of $(g+1)g/2$ minus the dimension
of the kernel (nullity) of $P$ given by $\kappa=\dim\ker P$, and then finally
minus 1 from the normalization constraint for densities,

$\dim D=\frac{1}{2}(g+1)g-\kappa-1.$ (9)

Hence, the basic classification of density regions will be in $(g,\kappa)$,
the _degree_ $g\in\\{2,3,\ldots\\}$ and the _nullity_
$\kappa\in\\{0,\ldots,(g+1)g/2-1\\}$. That $\kappa=(g+1)g/2-1$ is possible in
principle, but this implies that every state in $\mathcal{U}$ is mapped to the
exactly same density, which seems unlikely in the absence of internal degrees
of freedom. This also implies that all $\rho_{\overline{kl}}$ must be zero,
since they sum up to zero and cannot be equal to the density. The same two
assertions also appear in Garrigue [11, Cor. 1.7] for the conservation of
degeneracy in certain directions of potential variation. The density region is
then just a single point, a _singleton set_ , and no special geometry arises.
A larger $\kappa$ cannot occur because then $\rho$ would be the zero map. We
can now imagine a mapping $\deg:\rho\mapsto g$ that assigns to every element
of $\mathsf{Dens}$ that is the ground-state density for some potential $v$
(“$v$-representable”) the degree of the degeneracy region it belongs to. The
“rich structure” [20] of this map will become apparent in the following
examples.

##  III Further classification and examples for density regions

In this section the setting is the same as before, with $\mathcal{U}$ a
$g$-dimensional subspace of a Hilbert space $\mathcal{H}$ that corresponds to
a lattice or a continuum system. The aim is to study and classify the density
regions originating from $\mathcal{U}$. We begin with the lowest degree $g=2$,
so the possible nontrivial values for $\kappa$ are either 0 or 1. Since
$x_{1}^{2}+x_{2}^{2}=1$ on the unit circle, we choose
$x_{1}=\cos\varphi,x_{2}=\sin\varphi$ and (8) transforms into

$\displaystyle\rho(x)$
$\displaystyle=(\cos\varphi)^{2}\rho_{\overline{1}}+(\sin\varphi)^{2}\rho_{\overline{2}}+\sin\varphi\,\cos\varphi\,\rho_{\overline{12}}$
(10)
$\displaystyle=\frac{1+\cos(2\varphi)}{2}\rho_{\overline{1}}+\frac{1-\cos(2\varphi)}{2}\rho_{\overline{2}}+\frac{\sin(2\varphi)}{2}\rho_{\overline{12}}$
$\displaystyle=\frac{\rho_{\overline{1}}+\rho_{\overline{2}}}{2}+\frac{\rho_{\overline{1}}-\rho_{\overline{2}}}{2}\cos(2\varphi)+\frac{\rho_{\overline{12}}}{2}\sin(2\varphi).$

Here, the doubling of the angle $\varphi$ is a result of the identification of
antipodal points. The formula shows that all densities in $D_{\mathbb{R}}$ lie
on an ellipse with center
$\overline{\rho}=\frac{1}{2}(\rho_{\overline{1}}+\rho_{\overline{2}})$ and
axes $\frac{1}{2}(\rho_{\overline{1}}-\rho_{\overline{2}})$ and
$\frac{1}{2}\rho_{\overline{12}}$. Consequently, $D$ will be the filled
ellipse. If $\kappa=1$ then all three factors
$\rho_{\overline{1}},\rho_{\overline{2}},\rho_{\overline{12}}$ span the image
of $P$ as a 2-dimensional plane in density space, so $\overline{\rho}$,
$\frac{1}{2}(\rho_{\overline{1}}-\rho_{\overline{2}})$,
$\frac{1}{2}\rho_{\overline{12}}$ and with them the whole elliptical $D$ lie
in a plane through the origin. But the normalization constraint for densities
subtracts one further dimension, so $D$ actually collapses into a line
segment, as required by (9). If $\kappa=0$ this restriction does not occur and
$D$ will really be a filled ellipse. A simple example for this case is the
ground-state eigenspace for two non-interacting spinless particles on a
lattice with three vertices in the shape of a triangle that is extensively
discussed in Sec. VI.C of Penz and van Leeuwen [12]. There $D$ forms the
incircle of the triangular density domain that itself is the convex hull of
the three extremal densities $(1,1,0)$, $(1,0,1)$ and $(0,1,1)$.

If $g=3$ and $\kappa\geq 2$, the maximal dimension of the density region given
by (9) is 3, so we still have a possibility to easily visualize it. The set
$D_{\mathbb{R}}$ as the image of $\mathbb{V}_{3}$ under $P$ with $\kappa=2$ is
generally called a _Steiner surface_ [21]. Here we will limit the discussion
to one example which yields the most famous such surfaces, the _Roman surface_
, while it remains open if other types even appear as density regions. For
this example we utilize again a finite-lattice system with two non-interacting
spinless particles, now in the form of a tetrahedron graph with equal hopping
between all vertex pairs. Since the density is defined on just four vertices,
the density space is $\mathbb{R}^{4}$ while the Pauli exclusion principle and
the normalization of the density additionally yield the restrictions
$0\leq\rho_{i}\leq 1$ and $\sum_{i}\rho_{i}=2$. This gives a density domain in
the shape of an octahedron within the three-dimensional affine hyperplane
defined by the normalization constraint. The density region $D_{\mathbb{R}}$
to be determined here will then be a set within this octahedron. The one-
particle Hamiltonian for the given system when derived from a graph Laplacian
[12, Sec. II.D] is

$h_{0}=\begin{pmatrix}[r]3&-1&-1&-1\\\ -1&3&-1&-1\\\ -1&-1&3&-1\\\
-1&-1&-1&3\end{pmatrix}$ (11)

and for $v=0$ has the ground-state orbital

$\phi_{0}=\tfrac{1}{2}(1,1,1,1)$ (12)

as well as the 3-fold degenerate first-excited states

$\displaystyle\phi_{1}$ $\displaystyle=\tfrac{1}{2}(-1,-1,1,1)$ (13)
$\displaystyle\phi_{2}$ $\displaystyle=\tfrac{1}{2}(-1,1,-1,1)$ (14)
$\displaystyle\phi_{3}$ $\displaystyle=\tfrac{1}{2}(1,-1,-1,1)$ (15)

that all give rise to a uniform density. These orbitals yield the two-particle
Slater determinants $\Phi_{k}=\phi_{0}\wedge\phi_{k}$, $k\in\\{1,2,3\\}$, that
span the three-dimensional ground-state subspace $\mathcal{U}$. We note here
that for Hamiltonians derived from such graph Laplacians it generally follows
from the Perron–Frobenius theorem that the ground-state orbital is non-
degenerate [12, Sec. V.A], so all our examples will be for $N=2$ non-
interacting spinless particles, where orbitals from the first-excited state
also get filled. Using the orbital expressions we get

$\rho_{\overline{k},i}=\phi_{0,i}^{2}+\phi_{k,i}^{2},\quad\rho_{\overline{kl},i}=2\phi_{k,i}\phi_{l,i}$
(16)

and putting in the numbers

$\displaystyle\rho_{\overline{k}}$
$\displaystyle=\overline{\rho}=\tfrac{1}{2}(1,1,1,1)$ (17)
$\displaystyle\rho_{\overline{12}}$ $\displaystyle=\tfrac{1}{2}(1,-1,-1,1)$
(18) $\displaystyle\rho_{\overline{13}}$
$\displaystyle=\tfrac{1}{2}(-1,1,-1,1)$ (19)
$\displaystyle\rho_{\overline{23}}$ $\displaystyle=\tfrac{1}{2}(-1,-1,1,1).$
(20)

These vectors now make up the six columns in $P$ and we can check that indeed
$\kappa=\dim\ker P=2$. For $D_{\mathbb{R}}$ as the image of all
$x\in\mathbb{R}^{3}$, $\|x\|=1$, under $\rho$ we get

$\rho(x)=\overline{\rho}\underbrace{\sum_{k=1}^{3}x_{k}^{2}}_{1}+\frac{1}{2}\begin{pmatrix}[r]1&-1&-1&1\\\
-1&1&-1&1\\\ -1&-1&1&1\\\ 1&1&1&1\end{pmatrix}\begin{pmatrix}x_{1}x_{2}\\\
x_{1}x_{3}\\\ x_{2}x_{3}\\\ 0\end{pmatrix}.$ (21)

The surface defined by the last vector and parametrized by the unit sphere
$\|x\|=1$ is an amazing structure of tetrahedronal symmetry that is known as
_Roman surface_ , while the matrix in front is orthogonal and just describes a
mirroring at a plane with normal vector $(1,1,1,-1)$. The situation is
displayed in Figure 1. The full density region $D$ is then the convex hull of
$D_{\mathbb{R}}$ and in this case is equal to the segment set, so we have
$D=D_{\mathbb{C}}=\operatorname{seg}D_{\mathbb{R}}=\operatorname{ch}D_{\mathbb{R}}$.

Figure 1: The three-fold degeneracy region $D_{\mathbb{R}}$ of the tetrahedron
graph inside the octahedronal density domain. The corners of the octahedron
correspond to the extreme density $(1,1,0,0)$ and its permutations,
generalized barycentric coordinates are used to display densities.

Since $P$ maps $\mathbb{R}^{6}\to\mathbb{R}^{4}$ and $\kappa=\dim\ker P=2$, an
embedding of $\mathbb{R}^{4}$ into $\mathbb{R}^{6}$ orthogonal to $\ker P$
allows to see $P$ as a projection with projection center $\ker P$ [22, 1.2.7].
Using the following general lemma, that the projection center does not
intersect the Veronese variety, helps to reduce the number of possible classes
of Steiner surfaces appearing as density regions next to the Roman surface.

######  Lemma 5.

$\mathbb{V}_{g}\cap\ker P=\emptyset$.

###### Proof.

This follows directly from the fact that for all $x\in\mathbb{R}^{g}$ with
$\|x\|=1$, as the parameters of the Veronese variety, $\rho(x)=P\circ\nu(x)$
is a normalized density, so it cannot be zero. ∎

A subclassification of Steiner surfaces with $(g,\kappa)=(3,2)$ conducted by
Degen [19] considers various positions of the projection center and basically
looks at the the shadows cast by the ellipses that compose the Veronese
variety. By Theorem 4.1 from the reference and our Lemma 5 their class (B) is
ruled out. If an ellipse is projected such that only a line remains, this
appears as a singular point in projective space, and counting those points
gives the five possible subclasses (Aa)-(Ae). In the case (Aa), three ellipses
from the Veronese surface are projected exactly such that they appear as
lines, the three lines of self-intersection of the Roman surface. The shapes
of class (C) are quadrics, such as ellipsoids, that did not show up in the
examples studied here. If Lemma 5 could be strengthened to state that the
complex extension of the Veronese variety has empty intersection with the
projection center then also class (C) can be excluded. Since the possible
surfaces that one can construct like this correspond to triangular Bézier-
surface patches, this classification also has a relevance in computer graphics
and finite element methods [23]. A similar type of classification directly on
the basis of the matrix $P$ that is less geometrical was given by Coffman  _et
al._ [24] and Apéry [21, Ch. 1.3]. For some classical literature that presents
more interesting properties in particular of the Roman surface, see Cayley
[25] and Hilbert and Cohn-Vossen [26, §46]. Very recently, the possibility of
physically realizing the Roman surface by a spin-induced polarization vector
in a particular cubic crystal was reported in Liu  _et al._ [27]. The example
of the Roman surface as the degeneracy region for $v=0$ in a tetrahedron graph
clearly shows that while an individual ground-state density does not need to
reflect the symmetry of the system, the whole degeneracy region does.

After this discovery, we expect even more intricate geometric objects for
$g=3,\kappa<2$ or $g>3$. However, it is clear from the fact that a higher-
dimensional $\mathcal{U}$ is composed of lower-dimensional subspaces that all
$D_{\mathbb{R}}$ are parametrized families of other $D_{\mathbb{R}}$ of lower
degree, just like the Roman surface can be constructed from a family of
ellipses (see Figure 2). Note that in the case $\kappa=0$ the full Veronese
variety is retained as a density region since all factors in (8) are linearly
independent. This actually happens in the cuboctrahedron-graph example
presented in in Sec. VI.E of Penz and van Leeuwen [12] that is also discussed
in Section IV here. For general $(g,\kappa)$ a further subclassification, like
it was conducted by Degen [19] for $(g,\kappa)=(3,2)$ and which was briefly
discussed above, seems entirely possible.

Figure 2: Three different parametrizations for the Roman surface that all show
how it can be constructed from ellipses. The first corresponds to introducing
polar coordinates for $x\in\mathbb{R}^{3}$ in (8), the second to doing the
same in (21), while the third is from Apéry [21, Sec. 2.4].

##  IV Non-pure-state $v$-representable densities

As already exemplified for the cases $g=2$ and $g=3$ in the previous section,
the $g$-dimensional $\mathcal{U}$ can be thought of as the $g$-fold degenerate
eigenspace for a given Hamiltonian with potential $v$. Since the Hamiltonian
is assumed real symmetric, Lemma 1 shows that such an eigenspace can always be
spanned by real eigenvectors. Then we write $D(v)$ for the corresponding
density region and it is called a _degeneracy region_ , the set of all
possible densities coming from (ensemble) ground states to the potential $v$.
As demonstrated in Section II, the degeneracy regions can be classified by
$(g,\kappa)$. We define the _subdifferential_ of a convex functional at a
certain point as the set of all tangent lines in that point that lie fully
below the functional [28, Def. 2.30]. Likewise the _superdifferential_ of a
concave functional collects all tangent lines above. Then, interestingly, it
holds that $D(v)=\overline{\partial}E(v)$, the degeneracy region is the
superdifferential of the concave energy functional
$E(v)=\inf_{\Psi}\\{\langle\Psi,H_{v}\Psi\rangle\\}$ that itself is the convex
conjugate of the universal functional $F(\rho)$. On the other hand,
$v\in-\underline{\partial}F(\rho)$, the potential lies in the subdifferential
of the universal functional if $\rho$ is the ground-state density for
potential $v$. [29; 30]

From the definition of $D_{\mathbb{C}}(v)$ it follows that this set includes
all densities coming from pure ground states (pure-state $v$-representable).
It is then known from arguments in Levy [31] and Lieb [13] that if $g\geq 3$
there can occur densities that are _not_ pure-state $v$-representable, i.e.,
they are in $D(v)$ but not in $D_{\mathbb{C}}(v)$, but the provided proofs
remained vague. The following theorem helps to establish a precise condition
when and for which density non-pure-state $v$-representability will occur.
More precisely, the condition $\kappa=0$ exactly prevents “certain linear
dependencies” that are mentioned in Levy [31] as a situation where pure-state
$v$-representability is still possible.

######  Lemma 6.

If $g\geq 3$ and $\kappa=0$ then the central density
$\overline{\rho}=(\rho_{\overline{1}}+\ldots+\rho_{\overline{g}})/g$ is _not_
in $D_{\mathbb{C}}$.

###### Proof.

Because of Lemma 3 in order for $\overline{\rho}$ to be in $D_{\mathbb{C}}$ it
must hold that there is a $\lambda\in[0,1]$ and $x,y\in\mathbb{R}^{g}$ with
$\|x\|=\|y\|=1$ such that

$(\rho_{\overline{1}}+\ldots+\rho_{\overline{g}})/g=\lambda\rho(x)+(1-\lambda)\rho(y).$
(22)

Since $\kappa=0$ it holds $\ker P=0$ and we can apply $P^{-1}$ and formulate
the equation above on the codomain of the Veronese embedding as

$\displaystyle\frac{1}{g}$ $\displaystyle=\lambda
x_{k}^{2}+(1-\lambda)y_{k}^{2}$ (23) $\displaystyle 0$ $\displaystyle=\lambda
x_{k}x_{l}+(1-\lambda)y_{k}y_{l}\quad\quad(k\neq l)$ (24)

with $k,l\in\\{1,\ldots,g\\}$. Suppose that some $x_{k}=0$ then $\lambda\neq
1$ and $y_{k}\neq 0$ follows from (23) and for all $l\neq k$ it holds
$y_{l}=0$ from (24). Then (23) again says that all $x_{l},l\neq k,$ have the
same non-zero value and that $\lambda\neq 0$. Now since $g\geq 3$ we can
choose $l\neq m$ both different from $k$ and then (24) yields a contradiction.
The same holds if some $y_{k}=0$. Therefore, it will be assumed from here on
that all coefficients $x_{k},y_{k}\neq 0$. This implies
$\lambda\notin\\{0,1\\}$ because else (24) cannot be fulfilled. We can thus
write

$x_{k}x_{l}=-\frac{1-\lambda}{\lambda}y_{k}y_{l}=-\mu\,y_{k}y_{l}\quad\quad(k\neq
l),$ (25)

where $\mu=(1-\lambda)/\lambda>0$. This gives for $k,l,m$ all different,

$\displaystyle(x_{k}x_{l})(x_{k}x_{m})=\mu^{2}(y_{k}y_{l})(y_{k}y_{m})$ (26)
$\displaystyle\Rightarrow$ $\displaystyle
x_{k}^{2}(x_{l}x_{m})=\mu^{2}y_{k}^{2}(y_{l}y_{m})$ $\displaystyle\Rightarrow$
$\displaystyle-\mu\,x_{k}^{2}(y_{l}y_{m})=\mu^{2}y_{k}^{2}(y_{l}y_{m})$
$\displaystyle\Rightarrow$ $\displaystyle x_{k}^{2}=-\mu\,y_{k}^{2}.$

However, this is a contradiction since $\mu>0$ and $x_{k},y_{k}$ were assumed
to be non-zero and it follows that $\overline{\rho}\notin D_{\mathbb{C}}$. ∎

######  Corollary 7.

If there exists a potential $v$ that gives a degeneracy region $D(v)$ with
$g\geq 3$ and $\kappa=0$ then the central density $\overline{\rho}$ plus a
neighbourhood is non-pure-state $v$-representable.

###### Proof.

This is a direct consequence of Theorem 2. If $\overline{\rho}\in D(v)$ would
be the density of a pure ground state $\Psi$ of any $v^{\prime}$ then this
$\Psi$ must also be the (pure) ground state for $v$ which is in contradiction
to the statement of Lemma 6 that $\overline{\rho}\notin D_{\mathbb{C}}(v)$.
Since $D_{\mathbb{C}}(v)$ was noted to be closed, we can find an open set
including $\overline{\rho}$ that has an empty intersection with
$D_{\mathbb{C}}(v)$. ∎

An explicit example for non-pure-state $v$-representability was given by the
authors in Sec. VI.E of Penz and van Leeuwen [12] with two non-interacting
spinless particles on a cuboctahedron-graph system that exhibits three-fold
degeneracy in the ground state. We will briefly show that this example fits to
the results from above, i.e., it yields a $P$ with kernel zero. In order to
determine the $12\times 6$ matrix $P$ (12 because of the number of vertices in
the cuboctahedron and 6 from the dimension of the codomain of the Veronese
embedding for $g=3$) we have to calculate the factors
$\rho_{\overline{k}},\rho_{\overline{kl}}$ from the one-particle orbitals
$\phi_{k}$ where the orbital energies are
$\epsilon_{0}<\epsilon_{1}=\epsilon_{2}=\epsilon_{3}$, all written out in the
reference. Like before in the tetrahedron-graph example, these orbitals yield
the two-particle Slater determinants $\Phi_{k}=\phi_{0}\wedge\phi_{k}$,
$k\in\\{1,2,3\\}$, that span the three-dimensional ground-state subspace.
Equation (16) then again determines the vectors that form the columns of $P$
which can be checked to give a matrix with $\ker P=0$. Similar investigations
show that the icosahedron and dodecahedron graphs for two particles described
by graph Laplacians have $(g,\kappa)=(3,0)$ at $v=0$ too.

##  V Degeneracy-preserving potential manifolds

When degeneracy occurs for a fixed $H_{0}$ and a certain potential $v$,
variation of this potential can still preserve the symmetry and thus the
degree of degeneracy. For finite-lattice systems, the work of Ullrich and Kohn
[10] gives an upper limit to the dimensional size of the manifold in which
such potentials with $g$-fold degeneracy are embedded. The vertices of the
lattice will be indexed by $i\in\\{1,\ldots,M\\}$ and thus a density as well
as a scalar potential are given as vectors $\rho,v\in\mathbb{R}^{M}$. Since
the density $\rho_{i}$ at each vertex $i$ must be in $[0,1]$ for fermionic
particles and since the whole density is normalized to the number of
particles, $\sum_{i}\rho_{i}=N$, the density domain has the shape of an
$(M,N)$-hypersimplex $\mathcal{P}_{M,N}$. The dual pairing (inner product)
between the potential and the density that yields the potential energy is
$\langle v,\rho\rangle=\sum_{i}v_{i}\rho_{i}$. The whole expectation value of
the Hamiltonian for a normalized state $\Psi\in\mathcal{H}$ with density
$\rho$ is thus
$\langle\Psi,(H_{0}+V)\Psi\rangle=\langle\Psi,H_{0}\Psi\rangle+\langle
v,\rho\rangle$. We will extend the main result of Ullrich and Kohn [10] with a
proof resting solely on the classification for degeneracy regions introduced
in Section II and an extremely useful result of Rellich [32, 33] (see also
Kato [34] for a mode modern treatment) on the analyticity of eigenvalues and
eigenvectors under perturbations.

######  Theorem 8.

Let $H_{0}$ be a real symmetric matrix and $v\in\mathbb{R}^{M}$ a potential
such that $H_{v}=H_{0}+V$ has a degeneracy region of class $(g,\kappa)$. Then
$v$ (modulo constants) is embedded in a potential manifold of at most
dimension $M-(g+1)g/2+\kappa$ in which the degree of degeneracy $g$ is
preserved.

###### Proof.

Let $H_{v}$ have a $g$-fold degenerate ground state. Then in the ground-state
eigenspace we can choose the real orthonormal eigenstates
$\Phi_{1},\ldots,\Phi_{g}$ by Lemma 1. Let $H_{v}(\lambda)=H_{v}+\lambda U$ be
a perturbed Hamiltonian such that the degeneracy is preserved by $U$.
According to Rellich’s theorem we can choose $g$ orthonormal ground states
$\Psi_{1}(\lambda),\ldots,\Psi_{g}(\lambda)$ with common eigenvalue
$E(\lambda)$ which are analytic in $\lambda$ for small enough $|\lambda|$.
These states are generally not analytically connected to our chosen basis
$\\{\Phi_{k}\\}_{k=1}^{g}$ at $\lambda=0$. However, the states

$\tilde{\Phi}_{k}(\lambda)=\sum_{j=1}^{g}\Psi_{j}(\lambda)\langle\Psi_{j}(\lambda),\Phi_{k}\rangle$
(27)

are analytic in $\lambda$ as well and do have the property that
$\tilde{\Phi}_{k}(0)=\Phi_{k}$, which follows by noting that the states
$\Psi_{j}(0)$ form a complete basis set in the ground-state eigenspace of the
unperturbed system. Using the eigenstate property
$H_{v}(\lambda)\tilde{\Phi}_{k}(\lambda)=E(\lambda)\tilde{\Phi}_{k}(\lambda)$
we differentiate with respect to $\lambda$ and then set $\lambda=0$ to get

$\tilde{H}_{v}\tilde{\Phi}_{k}^{\prime}(0)+U\Phi_{k}=E(0)\tilde{\Phi}_{k}^{\prime}(0)+E^{\prime}(0)\Phi_{k}.$
(28)

We project this onto the subspace spanned by $\Phi_{l}$ and use
$H_{v}\Phi_{l}=E(0)\Phi_{l}$ to get

$\langle\Phi_{l},U\Phi_{k}\rangle=E^{\prime}(0)\delta_{kl}.$ (29)

Since we only consider potentials defined up to a constant, we can use this
gauge freedom to set $E^{\prime}(0)=0$ and thus arrive at the conditions
$\langle\Phi_{1},U\Phi_{1}\rangle=\ldots=\langle\Phi_{g},U\Phi_{g}\rangle=0$
in the diagonal and $\langle\Phi_{l},U\Phi_{k}\rangle=0$ off-diagonal, a total
of $g+g(g-1)/2=(g+1)g/2$ constraints. However, by remembering the definition
of $\rho_{\overline{k}},\rho_{\overline{kl}}$ from Section II, we see that
those constraints are actually equivalent to $\langle
u,\rho_{\overline{k}}\rangle=0$ and $\langle u,\rho_{\overline{kl}}\rangle=0$
for $k,l\in\\{1,\ldots,g\\}$, $k<l$, which is nothing but the matrix equation
${P}^{\mathsf{T}}u=0$. Thus, if $u$ is in the kernel of the transpose of the
$(M\times(g+1)g/2)$-matrix $P$, the constraints are fulfilled and this assures
that the added potential does not change the degree of degeneracy to first
order in $\lambda$ (since it still can change the degree to second order, we
only get an upper bound for the dimension of the invariant manifold). To get
the dimension of the kernel, we simply use the rank-nullity theorem for $P$
and ${P}^{\mathsf{T}}$ and the fact that the rank of a matrix is equal to the
rank of its transpose.

$\displaystyle\dim\ker P=\kappa=\frac{1}{2}(g+1)g-\operatorname{rank}P$ (30)
$\displaystyle\dim\ker{P}^{\mathsf{T}}=M-\operatorname{rank}P=M-\frac{1}{2}(g+1)g+\kappa$
(31)

These considerations restrict the dimension of the potential manifold (modulo
constants) that preserves the $g$-fold degeneracy to at most
$M-(g+1)g/2+\kappa$. ∎

The previous result of Ullrich and Kohn [10] did not contain a reference to
the nullity and implicitly assumed $\kappa=0$, so their dimension is
$M-(g+1)g/2$, which is less then the possible maximal dimension of Theorem 8
if $\kappa>0$. Taking the maximal nullity, $\kappa=(g+1)g/2-1$, the maximal
dimension for the potential manifold that preserves degeneracy evaluates as
$M-1$, which is just the full dimensionality of the potential space (modulo
constants) and thus invalidates the argument of Ullrich and Kohn [10] that
degeneracy is rare in potential space. Nevertheless, the potentials that lead
to degeneracy are still expected to be of measure zero in the absence of
internal degrees of freedom, since they either have to obey certain symmetries
or trigger accidental degeneracy, so Theorem 8 is simply not useful for the
case of maximal nullity. The result also does not fit to the maximal dimension
derived in Penz and van Leeuwen [12, Sec. IV.B] from the _original_ result of
Ullrich and Kohn [10] as $M-3$, which lead to the premature conclusion that
the set of non-uv potentials has measure zero.

The relevance of Theorem 8 in the present context is that whenever $v$ leads
to a degeneracy region $D(v)$ of class $(g,\kappa)$, it can actually be the
member of a whole family of degeneracy regions of the same degree with an
$(M-(g+1)g/2+\kappa)$-dimensional parametrization. Each degeneracy region
within this family can be reached by moving the potential inside the manifold
that preserves the degree of degeneracy. Such a family will be called a
_degeneracy bundle_. Note that the separate degeneracy regions in such a
bundle can lie arbitrarily close together but never touch, a notion that will
become important in Section VI. Since the dimension of a degeneracy region is
given by (9), the largest possible dimension of a whole degeneracy bundle is

$M-\frac{1}{2}(g+1)g+\kappa+\frac{1}{2}(g+1)g-\kappa-1=M-1,$ (32)

which is precisely the full dimensionality of the density domain when
normalization is taken into account. This led Ullrich and Kohn [10] to the
statement that degeneracy can be considered common in the density domain.
Precise numbers for the ratio of degeneracy in the density domain are
calculated for a few examples in the concluding Section VII.

To give an example, we will resort again to the tetrahedron graph with two
particles from Section III. There we studied the degeneracy set $D(0)$ of
class $(g,\kappa)=(3,2)$, the convex hull of the Roman surface
$D_{\mathbb{R}}(0)$, that appears for $v=0$, the potential with maximal
symmetry. Here Theorem 8 for $M=4$, $g=3$, $\kappa=2$ leaves no possible
dimension for the manifold of invariant degeneracy. Indeed, by changing the
potential, we will necessarily diminish the system of symmetry and thus reduce
the degree of degeneracy. However, for a potential of type $v=(s,0,0,0)$ (and
all permutations), $s>0$, still a degeneracy of class $(g,\kappa)=(2,0)$
remains, since the vertices apart from the first one still retain a triangular
symmetry. (For $s<0$ the two-fold degeneracy is in the first excited state.)
The application of Theorem 8 for $M=4$, $g=2$, $\kappa=0$ now gives a one-
dimensional manifold, parametrized by $s>0$, in which the degree of degeneracy
is preserved. In the limit $s\to\infty$ the first vertex is effectively
removed from the system and the case of the triangle graph, with the incircle
of the triangular density domain $\mathcal{P}_{3,2}$ as degeneracy region,
remains. In summary, four degeneracy bundles in the shape of cylinders
parametrized by $s>0$ will extend from the flat sides of the convexified Roman
surface until they reach the faces of the octahedron $\mathcal{P}_{4,2}$. The
whole situation is displayed in Figure 3. The collection of all degeneracy
regions and bundles will be called the _degeneracy structure_. In the next
section we investigate how this geometrically relates to non-uv densities and
the corresponding potentials.

Figure 3: The union of all degeneracy regions (_degeneracy structure_) of the
tetrahedron graph inside the octahedral set of densities $\mathcal{P}_{4,2}$.
The four degeneracy bundles in the shape of cylinders reaching out from the
flat sides of the convexified Roman surface have to be imagined as filled.

##  VI Geometrization of the potential-density mapping

Contrary to the continuum case, where the Hohenberg–Kohn theorem implies that
every ground-state density comes from a unique potential [15], the lattice
case introduced in Section V allows for non-uniquely $v$-representable (“non-
uv”) densities. We proved their existence together with some examples and
basic properties in our previous work [12], where we also put forward the idea
of a full geometrization of the mapping from potentials to densities with
degeneracy regions and non-uv densities and their corresponding potentials as
the prime elements. We wrongly conjectured that degeneracy regions have a
spherical shape, while actually their shape is much more intriguing, as was
demonstrated in Section III. With these first results in hand, we will here
finalize and prove the geometry conjecture.

######  Theorem 9 (Density-Potential Geometry).

Non-uv densities _only_ occur when

  * •

two different degeneracy regions $D(v_{\mathrm{I}}),D(v_{\mathrm{II}})$
intersect (in a single density or forming a degeneracy region of strictly
lower degree) or when

  * •

a degeneracy region $D(v_{\mathrm{A}})$ touches the boundary of the density
domain $\mathcal{P}_{M,N}$.

In the first case, all potentials on the line segment that connects
$v_{\mathrm{I}}$ and $v_{\mathrm{II}}$ yield the same ground-state density. In
the second case, all potentials from a ray with apex $v_{\mathrm{A}}$ yield
the same ground-state density. Furthermore, the densities from the second case
are the _only_ $v$-representable densities on the boundary of
$\mathcal{P}_{M,N}$.

###### Proof.

If a density $\rho$ is non-uv and thus arises from two different potentials
$v,v^{\prime}$ then by Theorem 2 both Hamiltonians $H_{v},H_{v^{\prime}}$
share a ground-state density matrix $\Gamma\mapsto\rho$. It is then clear that
this $\Gamma$ is also an eigenstate of $H_{v_{\lambda}}$ with
$v_{\lambda}=\lambda v+(1-\lambda)v^{\prime}$ for any $\lambda\in\mathbb{R}$.
But since $\rho$ is the ground-state density for both potentials
$v,v^{\prime}$, it must also minimize both convex functionals $F(\rho)+\langle
v,\rho\rangle$, $F(\rho)+\langle v^{\prime},\rho\rangle$ by the definition of
a ground state. It is then also a minimizer for $\lambda(F(\rho)+\langle
v,\rho\rangle)$, $(1-\lambda)(F(\rho)+\langle v^{\prime},\rho\rangle)$
($\lambda\in[0,1]$) and their sum $F(\rho)+\langle\lambda
v+(1-\lambda)v^{\prime},\rho\rangle$. This shows that $\rho$ is the _ground-
state_ density for _all_ potentials on the line segment between $v$ and
$v^{\prime}$. Since the eigenvalues of $H_{v_{\lambda}}$ are all continuous in
$\lambda$ by Rellich’s theorem, there are two possibilities if $\lambda$ is
varied into both directions outside the interval $[0,1]$: Either level
crossings occur and another ground state takes over in both directions (Fig.
4(a)), or $\Gamma$ remains the ground state while $|\lambda|\to\infty$ into
one direction and a level crossing occurs in the other direction (Fig. 4(b)).

$\lambda$$E$$\lambda_{1}$$\lambda_{2}$ (a)

$\lambda$$E$$v_{\mathrm{A}}$ (b)

Figure 4: The two situations of level crossings in the proof of Theorem 9.

The case that it remains the ground state in both directions is impossible, as
the following will show.
Like above we take $\rho\in D(v_{\lambda})$, where in this special case
$D(v_{\lambda})$ also denotes a set containing the single ground-state density
for a potential $v_{\lambda}$ if no degeneracy arises. If level crossings
occur in both directions, say at $\lambda_{1}$ and $\lambda_{2}$ with
$\lambda_{1}<\lambda_{2}$, this implies additional degeneracy at
$v_{\mathrm{I}}=v_{\lambda_{1}}$ and $v_{\mathrm{II}}=v_{\lambda_{2}}$. Any
non-uv density $\rho\in D(v_{\lambda})$, $\lambda_{1}<\lambda<\lambda_{2}$,
then belongs to two degeneracy regions $D(v_{\mathrm{I}})$,
$D(v_{\mathrm{II}})$ of strictly higher degree than $D(v_{\lambda})$ because
the crossings add at least one to the degree. This means
$D(v_{\lambda})=D(v_{\mathrm{I}})\cap D(v_{\mathrm{II}})$, so $\rho$ really
lies in the intersection of two degeneracy regions.
Now consider the case where $|\lambda|\to\infty$ is possible in one direction
without a level crossing. This means at least one component $v_{\lambda,i}$ of
$v_{\lambda}$ diverges. If $v_{\lambda,i}\to+\infty$ then it must hold that
$\rho_{i}=0$ for this density to still belong to the ground state. But then
performing $v_{\lambda,i}\to-\infty$ means that at one point another
eigenstate (and there must always be at least one vertex $i$ with
$\rho_{i}>0$) will take over the role of the ground state. But this must
happen at a level crossing, say at $v_{\mathrm{A}}$, where the energies match
(because by Rellich’s theorem the variation of $v$ leads to a continuously
changing energy) and such we arrive at a degeneracy region
$D(v_{\mathrm{A}})\ni\rho$. If $v_{\lambda,i}\to-\infty$ then since any
ground-state density minimizes $F(\rho)+\langle v_{\lambda},\rho\rangle$ and
$F$ is bounded on $\mathcal{P}_{M,N}$ (as a finite convex function on a
compact domain), only a density with $\rho_{i}=1$ qualifies. Going into the
opposite direction $v_{\lambda,i}\to+\infty$, the energy must increase and
thus at one point another eigenstate will again take over. Consequently, if
$|\lambda|\to\infty$ is possible in one direction without a level crossing for
a non-uv density, this density simultaneously belongs to the boundary of
$\mathcal{P}_{M,N}$ (because it has either $\rho_{i}=0$ or $\rho_{i}=1$) and
to a degeneracy region $D(v_{\mathrm{A}})$.
Conversely, assume a $v$-representable $\rho$ is on the boundary of
$\mathcal{P}_{M,N}$, which means it has at least one component $\rho_{i}=0$ or
$\rho_{i}=1$. In the case $\rho_{i}=1$, $v_{i}$ can always decrease from its
previous value and will only make the energy smaller, also compared to the
energy of other eigenstates that have $\rho_{i}\leq 1$, thus not influencing
the ground state. This gives a ray of potentials that all lead to the same
density $\rho$, thus making it non-uv. Yet, by continuously increasing $v_{i}$
one will find the end point, call it $v_{\mathrm{A}}$ as before, of such a ray
when the potential is large enough such that another eigenstate takes over and
assumes the role of ground state, exactly at a point of degeneracy. In the
case $\rho_{i}=0$, the ray just extends into the other direction, since
changing $v_{i}$ arbitrarily will not influence the energy of the original
ground state, but continuously decreasing $v_{i}$ will eventually allow
another eigenstate to take over the role of the ground state, again at a point
of degeneracy. This proves that a density on the boundary of
$\mathcal{P}_{M,N}$ is $v$-representable if and only if it also belongs to a
degeneracy region $D(v_{\mathrm{A}})$. ∎

It must be noted that the last part of the proof shows that if for a
$v$-representable boundary density multiple components $\rho_{i}$ are either 0
or 1, which is possible at least in principle, the rays combine into a convex
set of non-uv potentials. But such a situation was never observed in any of
the examples studied in the course of this work, just as the possibility of
degeneracy regions touching each other or the boundary in more than a single
point has not been verified. While singleton degeneracy regions are not
principally ruled out in the theorem, degeneracy regions that just arise from
internal degrees of freedom do not play any role in the context of the
theorem, because the external potential does not couple to the internal
coordinates.

The following corollary, put forward as a conjecture in Penz and van Leeuwen
[12], is imminent from the fact that degeneracy regions are always closed and
that non-uv densities only arise as their intersection or when they touch the
boundary. We also include our previous results that the non-uv densities have
measure zero in the density domain [12, Sec. III.D], which is visible from the
density-potential geometry theorem as well. That the corresponding non-uv
potentials have measure zero too is the content of the main theorem in a
recent work by Xu  _et al._ [35] that applies DFT to a more general setting
but relies on further assumption.

######  Corollary 10.

The set of non-uv densities is closed and has measure zero in the
$(M-1)$-dimensional density domain $\mathcal{P}_{M,N}$.

Conversely, this shows that almost all densities are uniquely
$v$-representable, which means the failure of the Hohenberg–Kohn theorem for
lattice systems can be considered less drastic. If a density is non-uv after
all then by Theorem 9 it must lie at the touching point of two degeneracy
regions or where a degeneracy region touches the border of the density domain,
a quite peculiar property. If, on the other hand, we can guarantee the
validity of the Hohenberg–Kohn theorem in some setting, then this in turn
means that degeneracy regions can never touch each other or the boundary.
Another previous result [12, Cor. 20] is also a direct consequence of the
theorem on density-potential geometry: The set of potentials leading to a non-
uv density that is not on the boundary of the density domain is always
bounded.

We would like to give examples for the geometrical situations present in
Theorem 9, two where a degeneracy region touches the boundary of the density
domain $\mathcal{P}_{M,N}$ and another where degeneracy regions touch each
other. The first situation was already present in the triangle graph example
discussed in Sec. VI.C of Penz and van Leeuwen [12] where the degeneracy
region for $v=0$ touches the boundary of $\mathcal{P}_{3,2}$ at the three
points where the incircle touches an equilateral triangle and correspondingly
three rays of non-uv potentials shoot off from $v=0$ to infinity in the
potential space. Another example can be found in the tetrahedron graph that
features the Roman surface as a degeneracy region for $v=0$ and that has
already been discussed twice here. The Roman surface touches the boundary of
the octahedron $\mathcal{P}_{4,2}$ exactly at the four points
$(1,\tfrac{1}{3},\tfrac{1}{3},\tfrac{1}{3})$ (and permutations) corresponding
to the vertices of its tetrahedral symmetry. Those are opposite to the flat
face where the cylinder-shaped degeneracy bundles are attached, as can be seen
from Figure 3. Since those degeneracy bundles correspond to the potential
$v=(s,0,0,0)$ (and all permutations), $s>0$, one can intuitively state that
the non-uv potential rays corresponding to the touching points are given by
$v=(s,0,0,0)$ (and all permutations), $s<0$. On the other hand, the
tetrahedron graph example, although it contains a large amount of degeneracy,
does not feature any degeneracy regions touching each other. For this we
switch over to two non-interacting spinless particles on a simple square
graph, a setting studied before in Sec. III.C and IV.B of Penz and van Leeuwen
[12]. There it was already noted that non-uv densities occur at the diagonals
of the middle plane of the octahedron $\mathcal{P}_{4,2}$ that correspond to
line-segments between potentials leading to degeneracy, but no picture of the
full degeneracy structure was provided. We show this in Figure 5 that exhibits
a pillow-like structure composed from families of ellipses (degeneracy
bundles) that is symmetric around the middle plane of the octahedron
$\mathcal{P}_{4,2}$. Those ellipses get flatter as the potential strength
increases and they approach the edge of the density domain. For $v=0$, on the
other hand, we have
$\rho_{\overline{1}}=\rho_{\overline{2}}=\frac{1}{2}(1,1,1,1)$ and
$\rho_{\overline{12}}=\frac{1}{2}(1,-1,1,-1)$ as the columns of $P$ which
means $(g,\kappa)=(2,1)$ and thus, as it was described in Section III, the
degeneracy region collapses into a vertical line segment in the center of
$\mathcal{P}_{4,2}$.

(a)

(b)

Figure 5: Left, the union of all degeneracy regions of the square graph inside
the octahedral set of densities $\mathcal{P}_{4,2}$. Right, four elliptical
degeneracy regions from it are displayed that mutually touch at the diagonals
drawn in red.

It is interesting to note that the geometrical features discussed here also
appear in the universal functional $F(\rho)$. Since all densities $\rho\in
D(v)=\overline{\partial}E(v)$ in a degeneracy region arise from a single
potential and $v\in-\underline{\partial}F(\rho)$ as discussed in Section IV,
this means that $F(\rho)$ is always flat on $D(v)$. Conversely, $F(\rho)$ is
locally strictly convex outside of degeneracy regions. When two degeneracy
regions touch at a non-uv $\rho$ then $F(\rho)$ displays a kink between the
flat regions at that point which is responsible for non-differentiability of
$F(\rho)$ and a subdifferential $-\underline{\partial}F(\rho)$ containing more
than a single potential. Such a situation is displayed in Figure 6, where the
example of the square graph is repeated limited to densities on the middle
plane of the octahedral density domain $\mathcal{P}_{4,2}$ with touching
degeneracy regions along the diagonals.

Figure 6: The universal functional $-F(\rho)$ displayed on the middle plane of
the density octahedron from Figure 5. Where degeneracy regions touch, the
functional becomes non-differentiable, while it is constant along the lines
that are inside an elliptical degeneracy region. Note that it is
differentiable again on the center point.

##  VII Summary and open questions

In summary, we analyzed the geometrical structures arising from degeneracy in
the density domain and in potential space. While a state eigenspace is mapped
to intricate shapes given by the convex hulls of algebraic varieties (Sections
II and III) and their geometric properties tell when a density is not pure-
state $v$-representable (Section IV, those shapes can even form bundles of a
certain dimensionality when the potential is varied (Section V). Finally, by
just observing where those objects, the degeneracy regions, touch each other
or the boundary of the density domain, one finds densities that are
represented by multiple potentials (non-uv densities), a situation where the
Hohenberg–Kohn theorem from density-functional theory fails. Overall, a
beautiful geometrical picture of the relation between potentials and their
ground-state densities emerges.

Looking at the different examples that have been discussed here reveals that a
substantial part of the whole density domain is filled by the degeneracy
structures. We might thus wonder about the precise _degeneracy ratio_ , i.e.,
the volume of the complete degeneracy structure divided by the volume of the
hypersimplex $\mathcal{P}_{M,N}$. For the triangle graph and the square graph
this can be calculated by elementary methods and we get $0.605$ and $0.589$,
respectively. To calculate the volume of the convexified Roman surface is much
harder, but we can give a lower and an upper estimate by considering a
tetrahedron inside the Roman surface and one enclosing it. The inner
tetrahedron shares the four points where the Roman surface touches the
boundary of $\mathcal{P}_{3,2}$ as vertices. The outer tetrahedron is taken
such that it shares the flat sides of the convexified Roman surface as faces,
which means that the cylindrical degeneracy bundles lead exactly to the
incircles of the tetrahedron faces. This gives a degeneracy ratio for the
tetrahedron-graph system between $0.528$ and $0.703$. Hence, in all those
examples that are highly symmetric and thus can give rise to degeneracy more
easily, the degeneracy ratio is about 60%, still an unexpectedly large number.
Such a large amount of degeneracy was also observed in the discretization of
continuum systems [20], so it is not exclusively an effect in small lattice
systems.

We finally list a number of open challenges and ideas for further
investigations linked to the theory developed in this work:

  * •

One obvious open task would be to completely classify all possible shapes of
degeneracy regions for general $(g,\kappa)$ following the analysis of Degen
[19] and Coffman  _et al._ [24] that settle the case $(g,\kappa)=(3,2)$. Some
could already be ruled out by Lemma 5, but possibly other classes also never
appear as degeneracy regions, which could be demonstrated by further
investigations related to the nature of $\ker P$. After all, in our examples
of all Steiner surfaces only the Roman surface appeared. The case of a
singleton degeneracy region with $\kappa=(g+1)g/2-1$ was also never observed
in an example and it is open if it can even occur without internal degrees of
freedom. If it can be proven that they do not exist then this would save the
original argument of Ullrich and Kohn [10] that degeneracy in potential space
is rare.

  * •

In order to find more classes of degeneracy regions or whole degeneracy
structures, one can naturally study other interesting finite-lattice system.
For the preparation of this work graph systems apart from the triangle,
square, tetrahedron and cuboctahedron with two particles have been studied:
the diamond graph shows only accidental degeneracy for a certain symmetrical
potential with $(g,\kappa)=(2,1)$, the claw graph just one cylindrical
degeneracy bundle, the octahedron graph has a degeneracy region with
$(g,\kappa)=(3,3)$ that has the shape of a triangle, and the cube graph leads
to bundles of $(g,\kappa)=(3,0)$ regions (this yields another example for non-
pure-state $v$-representability by Corollary 7), further cylindrical bundles
and a Roman surface in the center. But this list is far from exhaustive.

  * •

It was already noted in Section VI that cases of whole degeneracy regions as
the intersection between two larger degeneracy regions or at the boundary have
not yet been observed. The question remains if they are even possible. It also
remains open if touching the boundary of $\mathcal{P}_{M,N}$ only occurs at
the highest-dimensional faces of the hypersimplex, where all $\rho_{i}>0$, or
also at lower-dimensional edges. In case of the former and since by Theorem 9
densities on the border are only $v$-representable if they are contained in a
degeneracy region, this means that densities with $\rho_{i}=0$ can never occur
for any potential $v$ in a finite-lattice system or, in other words, all
ground-state densities are strictly positive. Some corresponding results for
non-interacting or weakly-interacting lattice systems can be found in Penz and
van Leeuwen [12, Sec. V.A].

  * •

Non-uv densities and especially densities from degeneracy regions, since they
are more frequent, can pose a problem to the convergence of the Kohn–Sham
iteration by introducing ambiguities [20]. Imagine the target density of an
interacting system that lies within a degeneracy region of the non-interacting
reference system (this can indeed happen, since for example a Hamiltonian with
a central potential commutes with the Laplace–Runge–Lenz vector, leading to
additional symmetries, while the Coulomb interaction breaks this symmetry),
then even if we find an almost correct effective potential we would lie
outside of the degeneracy region and thus potentially stay far away from the
target density.

  * •

In the end of Section VI we noted that $F(\rho)$ is flat on degeneracy
regions, a feature that will not be respected by the usual approximations to
the universal functional. More insight into its shape, arising from the
discussed geometrical structure, can thus improve such approximations.

  * •

Moreau–Yosida regularization is applied to the universal functional $F(\rho)$
in order to remedy non-differentiability of the functional [29; 30] and to
help proving convergence of the Kohn–Sham iteration in the finite lattice case
[36; 37]. In the geometrical picture, this leads to degeneracy regions that
shrink slightly, thus moving apart from each other, and a boundary of the
density domain that is entirely removed. This then avoids any non-uv densities
by Theorem 9.

  * •

Further dual settings apart from “density-potential” with even more
complicated geometries can be considered as well. For example current DFT maps
the tuple $(\rho,\mathbf{j})$ including the current density to
$(v,\mathbf{A})$, where the potential is complemented by a vector potential
[38]. Since Theorem 2 also holds for this case, a corresponding geometrical
analysis can be conducted. Yet, through the appearance of Landau quantization
in uniform magnetic fields that have infinite degeneracy in the continuum
case, the situation can be much more intricate.

  * •

Theorem 9 exhibits an interesting dual structure between the geometric
elements in the density and potential spaces introduced here. That the spaces
themselves are topological duals of each other was already noted, but we also
see that each degeneracy region (as an extended object) is connected to a
single density point, while the straight lines connecting those points
correspond to the density where the degeneracy regions touch. This structure
is very similar to the _nerve complex_ of a set family.

###### Acknowledgements.

The foundation of this work was laid at a joint research visit in November
2021 at the Oberwolfach Research Institute for Mathematics as a part of their
“Research in Pairs” program. Hence, our special gratitude goes to the
institute for inviting us, to Michael Ruggenthaler who joined us there, and to
Kevin Piterman who was a visiting fellow at that time and helped us with some
questions in algebraic geometry. Coincidentally, the Roman surface was only
recognized at first because of a photo showing a sculpture of it in the
immense library at Oberwolfach. R. v. L. further acknowledges the Academy of
Finland for support under project no. 317139.

## References

  * von Barth [2004] U. von Barth,  _Basic density-functional theory—an overview,_Phys. Scr. 2004, 9 (2004).
  * Burke and friends [2007] K. Burke and friends, The ABC of DFT,  (2007).
  * Dreizler and Gross [2012] R. M. Dreizler and E. K. Gross,  _Density functional theory: An approach to the quantum many-body problem_ (Springer, 2012).
  * Eschrig [2003] H. Eschrig,  _The fundamentals of density functional theory_ , 2nd ed. (Springer, 2003).
  * Ullrich [2011] C. A. Ullrich,  _Time-dependent density-functional theory: Concepts and applications_ (OUP Oxford, 2011).
  * Ullrich and Yang [2014] C. A. Ullrich and Z. Yang,  _A brief compendium of time-dependent density functional theory,_Braz. J. Phys. 44, 154 (2014).
  * Vignale and Rasolt [1987] G. Vignale and M. Rasolt,  _Density-functional theory in strong magnetic fields,_Phys. Rev. Lett. 59, 2360 (1987).
  * Vignale [2004] G. Vignale,  _Mapping from current densities to vector potentials in time-dependent current density functional theory,_Phys. Rev. B 70, 201102 (2004).
  * Ruggenthaler  _et al._ [2014] M. Ruggenthaler, J. Flick, C. Pellegrini, H. Appel, I. V. Tokatly, and A. Rubio,  _Quantum-electrodynamical density-functional theory: Bridging quantum optics and electronic-structure theory,_Phys. Rev. A 90, 012508 (2014).
  * Ullrich and Kohn [2002] C. A. Ullrich and W. Kohn,  _Degeneracy in density functional theory: Topology in the v and n spaces,_Phys. Rev. Lett. 89, 156401 (2002).
  * Garrigue [2021] L. Garrigue,  _Some properties of the potential-to-ground state map in quantum mechanics,_Commun. Math. Phys. 386, 1803 (2021).
  * Penz and van Leeuwen [2021] M. Penz and R. van Leeuwen,  _Density-functional theory on graphs,_J. Chem. Phys. 155, 244111 (2021).
  * Lieb [1983] E. H. Lieb,  _Density functionals for Coulomb-systems,_Int. J. Quantum Chem. 24, 243 (1983).
  * Lewin  _et al._ [2019] M. Lewin, E. H. Lieb, and R. Seiringer,  _Universal functionals in density functional theory,_ arXiv preprint (2019), arXiv:1912.10424 [math-ph] .
  * Garrigue [2018] L. Garrigue,  _Unique continuation for many-body Schrödinger operators and the Hohenberg–Kohn theorem,_Math. Phys. Anal. Geom. 21, 27 (2018).
  * Bárány and Karasev [2012] I. Bárány and R. Karasev,  _Notes about the Carathéodory number,_Discrete Comput. Geom. 48, 783 (2012).
  * Beltrametti  _et al._ [2009] M. C. Beltrametti, E. Carletti, D. Gallarati, and G. Monti Bragadin,  _Lectures on curves, surfaces and projective varieties_ (European Mathematical Society, 2009).
  * Harris [1992] J. Harris,  _Algebraic geometry: A first course_ (Springer, 1992).
  * Degen [1994] W. L. F. Degen,  _The types of triangular Bézier surfaces,_Proceedings of the 6th IMA Conference on the Mathematics of Surfaces , 153 (1994).
  * Garrigue [2022] L. Garrigue,  _Building Kohn-Sham potentials for ground and excited states,_Arch. Rational Mech. Anal. (2022), 10.1007/s00205-022-01804-1.
  * Apéry [1987] F. Apéry,  _Models of the Real Projective Plane_ (Vieweg, 1987).
  * Fortuna  _et al._ [2016] E. Fortuna, R. Frigerio, and R. Pardini,  _Projective Geometry: Solved Problems and Theory Review_ , Vol. 104 (Springer, 2016).
  * Sederberg and Anderson [1985] T. Sederberg and D. Anderson,  _Steiner surface patches,_IEEE Comput. Graph. Appl. 5, 23 (1985).
  * Coffman  _et al._ [1996] A. Coffman, A. Schwartz, and C. Stanton,  _The algebra and geometry of Steiner and other quadratically parametrizable surfaces,_Comput. Aided Geom. Des. 13, 257 (1996).
  * Cayley [1873] C. Cayley,  _On Steiner’s surface,_Proc. Lond. Math. Soc. 1, 14 (1873).
  * Hilbert and Cohn-Vossen [2021] D. Hilbert and S. Cohn-Vossen,  _Geometry and the Imagination_ , Vol. 87 (American Mathematical Society, 2021).
  * Liu  _et al._ [2022] G. Liu, M. Pi, L. Zhou, Z. Liu, X. Shen, X. Ye, S. Qin, X. Mi, X. Chen, L. Zhao, _et al._ ,  _Physical realization of topological Roman surface by spin-induced ferroelectric polarization in cubic lattice,_Nature Comm. 13, 2373 (2022).
  * Barbu and Precupanu [2012] V. Barbu and T. Precupanu,  _Convexity and Optimization in Banach Spaces_ , 4th ed. (Springer, 2012).
  * Kvaal  _et al._ [2014] S. Kvaal, U. Ekström, A. M. Teale, and T. Helgaker,  _Differentiable but exact formulation of density-functional theory,_J. Chem. Phys. 140, 18A518 (2014).
  * Laestadius  _et al._ [2018] A. Laestadius, M. Penz, E. I. Tellgren, M. Ruggenthaler, S. Kvaal, and T. Helgaker,  _Generalized Kohn–Sham iteration on Banach spaces,_J. Chem. Phys. 149, 164103 (2018).
  * Levy [1982] M. Levy,  _Electron densities in search of Hamiltonians,_Phys. Rev. A 26, 1200 (1982).
  * Rellich [1937] F. Rellich,  _Störungstheorie der Spektralzerlegung, I. Mitteilung,_Mathematische Annalen 113, 600 (1937).
  * Rellich [1969] F. Rellich,  _Perturbation theory of eigenvalue problems_ (Gordon and Breach Science Publishers, 1969).
  * Kato [1995] T. Kato,  _Perturbation theory for linear operators_ (Springer, 1995).
  * Xu  _et al._ [2022] L. Xu, J. Mao, X. Gao, and Z. Liu,  _Does Hohenberg-Kohn theorem apply to a general quantum system?_ arXiv preprint (2022), arXiv:2201.09177 [quant-ph] .
  * Penz  _et al._ [2019] M. Penz, A. Laestadius, E. I. Tellgren, and M. Ruggenthaler,  _Guaranteed convergence of a regularized Kohn–Sham iteration in finite dimensions,_Phys. Rev. Lett. 123, 037401 (2019).
  * Penz  _et al._ [2020] M. Penz, A. Laestadius, E. I. Tellgren, M. Ruggenthaler, and P. E. Lammert,  _Erratum: Guaranteed convergence of a regularized Kohn–Sham iteration in finite dimensions,_Phys. Rev. Lett. 125, 249902 (2020).
  * Laestadius  _et al._ [2019] A. Laestadius, E. I. Tellgren, M. Penz, M. Ruggenthaler, S. Kvaal, and T. Helgaker,  _Kohn–Sham theory with paramagnetic currents: Compatibility and functional differentiability,_J. Chem. Theory Comput. 15, 4003 (2019).


# Understanding Training-Data Leakage from Gradients in Neural Networks for
Image Classification

Cangxiong Chen
Department of Computer Science
University of Bath
<EMAIL_ADDRESS>
&Neill D. F. Campbell
Department of Computer Science
University of Bath
<EMAIL_ADDRESS>
##  1 Introduction

In federated learning [6] of deep learning models for supervised tasks such as
image classification and segmentation, gradients from each participant are
shared either with another participant or are aggregated at a central server.
In many applications of federated learning, the privacy of the training data
will need to be protected and we want to obtain guarantees that a malicious
participant will not be able to recover fully the training data from other
participants, with shared gradients and knowledge of the model architecture.
The guarantee will be indispensable in removing the barriers for applying
federated learning in tasks such as image segmentations in film post-
production where the training data are usually under strict IP protections. In
this scenario, it is the training data that needs to be protected, rather than
the information we can infer about them.

In order to develop protection mechanisms, an appropriate understanding of the
source of leakage of the training data is needed. For this work, we are
concerned with the following question: for a deep learning model performing
image classifications, what determines the success of reconstructing the
training data given its label, its gradients from training, and the model
architecture? We will focus on the case when we aim to reconstruct a single
target image with an untrained model. Although our work was inspired by R-GAP
[10], our method COPA (combined optimisation attack) provides a more general
theoretical framework to training-data reconstructions, particularly for
convolutional layers. Compared with DLG [11], COPA provides more insight to
the mechanism of training-data leakage through a more informative formulation
of the objective function, making it clearer the source of constraints.

##  2 The COPA method

### Overall assumptions

We only consider the problem of reconstructing a single training image in this
work. The target image-classification model is assumed to consist of
consecutive convolutional layers with the last layer being fully-connected. We
assume the ground-truth label of the target image is given. This does not lose
generality when the activation is nonnegative because otherwise we can
reconstruct the true label from the last layer via observing the signs of the
gradients of the weights according to [9]. The sign of the gradients of the
weights corresponding to the true class will be different from those
corresponding to other classes. We assume that the activation function for
each layer is piecewise invertible and piecewise differentiable. For
simplicity we do not include pooling in the design of the network and
dimensionality reduction in the feature space is mainly achieved by taking
strides to be greater than one. Notice that CNN without pooling was also
considered in [7].

### Notation

We introduce notation to be used throughout the paper. Superscript with
parenthesis indicates the index of a layer in the network.

  1. $W^{(i)}$:

weight from layer $i$ of size $m$ by $n$, where $0\leq i\leq d$ and $d$ is the
total number of layers. For simplicity of notation, we omit $i$ in $m$ and $n$
when possible, but readers should be aware that the size of the weight need
not be the same for different layers. For a convolutional layer, it denotes
the circulant representation of the kernel following [4, chapter 4.8.2].

  2. $J$:

composition of softmax function with negative log likelihood loss function.

  3. $\nabla_{W^{(i)}}J$:

gradient of the cost function $J$ w.r.t. $W^{(i)}$.

  4. $Z^{(i)}$:

the linear output of the layer $i$ before activation given by
$W^{(i)}X^{(i)}+B^{(i)}$ with input $X^{(i)}$ and bias $B^{(i)}$. This also
expresses the convolutional operation following the circulant form of
$W^{(i)}$.

  5. $A^{(i)}(\cdot)$:

activation function after linearity in vector form. We use $a^{(i)}$ to denote
its component.

  6. $|.|$:

when applied to a matrix, the absolute value sign $|.|$ denotes the number of
elements.

  7. $w,x,z,b$:

we use small letters with subscripts to denote the component at specified
indices of the corresponding matrix in capital letters.

Our method COPA consists of reconstructions of input for a fully connected
layer and a convolutional layer. First we formulate a linear system using
weights and gradients.

### Weight and gradient constraints:

At a given layer $i$, the forward and backward propagations give rise to the
following equations:

$\displaystyle W^{(i)}X^{(i)}+B^{(i)}=Z^{(i)},$ (1a)
$\displaystyle\nabla_{Z^{(i)}}J\cdot X^{(i)}=\nabla_{W^{(i)}}J.$ (1b)

We note that this represents both the fully-connected and the convolutional
cases, using the circulant representation for the weight $W^{(i)}$ and the
gradient $\nabla_{Z^{(i)}}J$. Both $Z^{(i)}$ and $\nabla_{W^{(i)}}J$ are
written as vectors. From the reconstruction point of view, we treat $X^{(i)}$
as the unknown and regard the above equations as weight and gradient
constraints imposed on the unknown. The term $Z^{(i)}$ is computed from
inverting the reconstruction from the subsequent layer $X^{(i+1)}$:

$Z^{(i)}=(A^{(i)})^{-1}(X^{(i+1)}).$ (2)

The term $\nabla_{Z^{(i)}}J$ can be computed by using the following relations
deduced from backpropagation:

$\displaystyle\nabla_{Z^{(i)}}J=\nabla_{X^{(i+1)}}J\cdot\nabla_{Z^{(i)}}A^{(i)},$
(3a)
$\displaystyle\nabla_{X^{(i)}}J=\nabla_{X^{(i+1)}}J\cdot\nabla_{Z^{(i)}}A^{(i)}\cdot
W^{(i)}.$ (3b)

It is worth noticing that the circulant representation of the gradient
$\nabla_{Z^{(i)}}J$ is determined by the circulant form of the weight
$W^{(i)}$ from backpropagating through the weight constraint (1a).

### Fully connected layer

For a fully connected layer, the input can be solved uniquely in closed form.
This is because there is no weight sharing in a fully-connected layer. So the
circulant form of $\nabla_{Z^{(i)}}J$ consists of blocks of diagonal matrices
with the same element along the diagonal in each block, which allows (1b) to
be solved exactly. The solution and its proof can be found in Appendix A.

### Convolutional layer

For a convolutional layer, we can no longer uniquely determine the input in
general because of weight sharing. We proceed by first combining the weight
and gradient constraints (1a) (1b) into a quadratic function. More precisely,
write $U^{(i)},V^{(i)}$ to denote the following block matrices

$U^{(i)}:=\begin{bmatrix}W^{(i)}\\\
\nabla_{Z^{(i)}}J\end{bmatrix},V^{(i)}:=\begin{bmatrix}(A^{(i)})^{-1}(X^{(i+1)})\\\
\nabla_{W^{(i)}}J\end{bmatrix}.$ (4)

Here both the term $(A^{(i)})^{-1}(X^{(i+1)})$ and the term
$\nabla_{W^{(i)}}J$ are written as vectors. The term $\nabla_{W^{(i)}}J$ has
the same dimension as $|W^{(i)}|$, i.e. the number of elements of the weight
in its non-circulant form as a 4-dimensional array. Notice that we can absorb
the bias term into the product $W^{(i)}X^{(i)}$ by replacing the the weight
matrix with its augmentation by $B^{(i)}$ and $X^{(i)}$ by its augmentation by
one. In order to make sure the solution can be inverted to the preceding layer
by the activation function $A^{(i-1)}$ from that layer, we consider the
following reparameterisation and define:

$f(X):=||U^{(i)}A^{(i-1)}(X)-V^{(i)}||^{2}.$ (5)

By minimising this function with respect to $X$, we can obtain an approximate
solution to both constraints (1a) and (1b) which is guaranteed to be
invertible by $A^{(i-1)}$. If the layer in question is the first one in the
network, then $f(X)$ will be defined as the following quadratic function
without the reparameterisation term:

$f(X):=||U^{(i)}X-V^{(i)}||^{2}.$ (6)

### Pull-back constraint

In addition to the constraints defined by (1a) and (1b), the weight constraint
from the preceding layer might also impose constraint on the reconstruction in
the current layer. This happens when the weight of the preceding layer has
full-rank but the number of rows is greater than the number of columns. This
will reduce the degree of freedom for the input to the current layer via the
activation function. We can formulate this non-linear constraint
mathematically and refer to it as the pull-back constraint. The details of the
formulation can be found in Appendix B.

### Optimisation problem in the convolutional case

Based on the above considerations, the reconstruction problem for the
convolutional layer can be formulated as solving the following optimisation
problem:

$X^{*}=\arg\min\left[f(X)+||g^{(i)}A^{(i)}(X)||^{2}\right],$ (7)

where $g^{(i)}$ is the pull-back constraint. When the layer in question is the
first one in the network, then the function to be optimised in (7) will reduce
to $f(X)$ given in (6).

### COPA

We present the Combined Optimisation Attack (COPA) algorithm. It starts from
the bottom of the network and iteratively solves for the input of a layer
following (10) and (7).

Input: Label $Y_{0}$ for the target image; Gradients $\nabla_{W^{(i)}}J,1\leq
i\leq d$ from the backward pass using the target image $X_{0}$; Initialise
$W^{(i)}$ randomly for each $i$. Set $X^{(d)}=Y_{0}$.

for  _$i$ in $(d-1,...,0)$_ do

Compute $\nabla_{X^{(i+1)}}J$ using (3b);

Compute $\nabla_{Z^{(i)}}J$ from $\nabla_{X^{(i+1)}}J$ using (3a);

if the current layer is fully connected, solve for $X^{(i)}$ using (10);

else if the current layer is convolutional:

Define $U^{(i)},V^{(i)}$ following (4);

if $i=0$, solve for $X^{(i)}$ using (6);

else if $\operatorname{rank}(W^{(i-1)})<|X^{(i)}|$, solve for $X^{(i)}$ using
(5);

else solve for $X^{(i)}$ using (7) ;

end for

Result: Input $X^{(0)}$ to the network.

Algorithm 1 Constraint Optimisation Attack (COPA).

### A security measure

For a fully connected layer, we have shown in Lemma A.1 that we can always
reconstruct the input in full. This can be regarded as no level of security
and we omit it from our definition of the metric. For a convolutional layer,
we consider $\operatorname{rank}(U)-|X|$ as an index to measure the efficacy
of COPA. The larger this number is, the less rank-deficient the linear system
(5) is and so more likely for COPA to have a full reconstruction for this
layer. We also notice that the position where the rank-deficiency happens also
matters. The closer it is to the first layer, the bigger impact it has on the
reconstruction. This is consistent with our intuition that if the
representation of the input data loses information at the first layer, it will
be unlikely to compensate for that loss in later layers. To accommodate for
this effect, we discount the index $\operatorname{rank}(U)-|X|$ by the
position of the layer in the network. Here is the definition of the proposed
metric:

######  Definition 2.1.

Suppose the model $M$ has $d$ convolutional layers indexed by $1,...,d$,
followed by a fully-connected layer. We define the following metric:

$c(M):=\sum_{i=1}^{d}\frac{d-(i-1)}{d}\cdot\big{(}\operatorname{rank}(U^{(i)})-n_{i}\big{)},$
(8)

where $U^{(i)}$ is defined in (4) and $n_{i}$ is the dimension of the input
for the $i$-th layer.

Because $\operatorname{rank}(U^{(i)})\leq n_{i}$ for each convolutional layer,
$c(M)$ will be non-positive. The larger the value of the metric is, the less
secure the model tends to be and the more likely for COPA to create better
reconstructions. The metric is better interpreted as an estimate of the
security of the model against COPA. Our experiments have shown that it is
possible to fully reconstruct the input using COPA or R-GAP when $c(M)=0$.

### Recommendations on architectural design

Based on the proposed metric and analysis, we can see that a network tends to
be less secure against COPA if it has wider convolutional layers that greatly
increase the dimensions of the feature spaces at the beginning of the network
and only shrinks the feature spaces towards the final layer. Overall, we would
recommend designing a model with a small value of $c(M)$ if defendability
against COPA-type attack is the main concern.

##  3 Experiments

We demonstrate the performance of COPA and how our proposed index $c(M)$ can
be an indicator of model security in practice. We consider a three-layer
network CNN3 consisting of two convolutional layers and a fully connected
layer, and performing classification on CIFAR-10 [5] [1]. For simplicity, we
assume the convolutional layer to be bias-free and the fully-connected layer
to have non-zero bias. However, these assumptions on bias are not necessary
for the COPA to work. We have considered CNN3 because we need at least two
convolutional layers to demonstrate the effect of the pull-back constraints.
In order to illustrate typical architecture designs for the network, we
consider four variations of the network, each representing a different case of
changes in dimensions of intermediate feature spaces. These variants covers
all possibilities of non-trivial changes of feature spaces in a three-layer
CNN. For each network, weights are initialised randomly from a uniform
distribution. The target image will go through one forward and backward pass
to generate the gradients. As a post-processing step, we apply Total Variation
denoising implemented by the scikit-image library [8] with the weight set to
0.15. More results where the activations are chosen differently can be found
in Appendix C.

| Layer 1 | Layer 2 | Fully Connected | $c(M)$
---|---|---|---|---
CNN3 Variant 1 | 3,6,1 | 4,3,2 |  0588 | -2267
CNN3 Variant 2 | 4,6,2 | 3,3,2 |  0147 | -1995
CNN3 Variant 3 | 3,6,1 | 3,9,1 | 7056 |  -0000
CNN3 Variant 4 | 3,1,1 | 3,6,1 | 4704 | -2146

Table 1: Model architecture for all four variants of the CNN3 model, with
their $c(M)$. The numbers in columns ‘layer 1’ and ‘layer 2’ refer to kernel
width, channels, strides accordingly. All of the variants have no padding,
because padding does not play a decisive role in the efficacy of COPA. The
numbers in the column ‘fully connected’ refer to the input dimension of that
fully-connected layer, whereas the output dimension is always 10.

(a) Variant 1

(b) Variant 2

(c) Variant 3

(d) Variant 4

Figure 1: Sample outputs from COPA applied to all four variants, when the
activations are Tanh for all layers. Variant 1 and 3 have better
reconstructions overall compared to Variant 2 and 4. This has been captured by
our metric $c(M)$: the first layer of Variant 1 and 3 have contribution 0
(i.e. $\operatorname{rank}(U)=|X|$) to the metric; on top of that, Variant 3
has 0 contribution from the second layer as well, so it has the best
reconstruction quality. There is little difference between COPA and R-GAP
between Variant 1 and 3, showing limited impact from the pull-back constraint.

##  4 Conclusion and future work

In this paper, we advance our understanding of training-data leakage from
gradients by developing a novel method COPA to reconstruct training data given
gradients and the architecture of the target model. The key insight from COPA
is formulating the reconstruction problem as solving linear systems at each
layer formed by weights and gradients. This allows us to attribute the source
of training-data leakage to the architecture of the model and we propose a
metric to capture the level of security of the model against COPA. The metric
can provide us a guideline in designing a deep network more securely. For
future work, we will extend COPA to tackle batch-image reconstructions,
against ResNet architecture, and against pre-trained models.

## References

## References

  * [1] URL: https://peltarion.com/knowledge-center/documentation/terms/dataset-licenses/cifar-10
  * [2] Yoshinori Aono, Takuya Hayashi, Lihua Wang and Shiho Moriai  “Privacy-preserving deep learning via additively homomorphic encryption”  In _IEEE Transactions on Information Forensics and Security_ 13.5, 2017, pp. 1333–1345
  * [3] Lixin Fan et al.  “Rethinking Privacy Preserving Deep Learning: How to Evaluate and Thwart Privacy Attacks”  In _Federated Learning: Privacy and Incentive_ Cham: Springer International Publishing, 2020, pp. 32–50  DOI: 10.1007/978-3-030-63076-8_3
  * [4] G.H. Golub and C.F. Van Loan  “Matrix Computations”, Johns Hopkins Studies in the Mathematical Sciences  Johns Hopkins University Press, 2013  URL: https://books.google.co.uk/books?id=5U-l8U3P-VUC
  * [5] Alex Krizhevsky  “Learning multiple layers of features from tiny images”, 2009
  * [6] Brendan McMahan et al.  “Communication-Efficient Learning of Deep Networks from Decentralized Data”  In _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ 54, Proceedings of Machine Learning Research  Fort Lauderdale, FL, USA: PMLR, 2017, pp. 1273–1282  URL: http://proceedings.mlr.press/v54/mcmahan17a.html
  * [7] J.T. Springenberg, A. Dosovitskiy, T. Brox and M. Riedmiller  “Striving for Simplicity: The All Convolutional Net”  In _ICLR (workshop track)_ , 2015  URL: http://lmb.informatik.uni-freiburg.de/Publications/2015/DB15a
  * [8] Stéfan Walt et al.  “scikit-image: image processing in Python”  In _PeerJ_ 2, 2014, pp. e453  DOI: 10.7717/peerj.453
  * [9] Bo Zhao, Konda Reddy Mopuri and Hakan Bilen  “iDLG: Improved Deep Leakage from Gradients”  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019
  * [10] Junyi Zhu and Matthew Blaschko  “R-GAP: Recursive Gradient Attack on Privacy”  In _International Conference on Learning Representations - (ICLR)_ , 2021
  * [11] Ligeng Zhu, Zhijian Liu and Song Han  “Deep Leakage from Gradients”  In _Advances in Neural Information Processing Systems_ 32 Curran Associates, Inc., 2019, pp. 14774–14784  URL: https://proceedings.neurips.cc/paper/2019/file/60a6c4002cc7b29142def8871531281a-Paper.pdf

## Understanding Training-Data Leakage from Gradients in Neural Networks for
Image Classification: Supplementary Material

##  Appendix A Statement of solution and proof for the fully-connected layer

######  Lemma A.1.

If a fully connected layer has non-zero bias
$B^{(i)}=(b_{1}^{(i)},...,b_{m}^{(i)})\in\mathbb{R}^{m}$, then the input
$X^{(i)}=(x_{1}^{(i)},...,x_{n}^{(i)})\in\mathbb{R}^{n}$ is uniquely
determined from the gradient constraint (1b). Suppose $\exists k,1\leq k\leq
m$, such that $b_{k}^{(i)}\neq 0$ and $\frac{\partial J}{\partial
b_{k}^{(i)}}\neq 0$. Then $X^{(i)}$ is given by:

$x_{l}^{(i)}=\frac{\partial J}{\partial w_{kl}^{(i)}}\left(\frac{\partial
J}{\partial b_{k}^{(i)}}\right)^{-1},\ \ 1\leq l\leq n.$ (9)

More generally, assuming both $\frac{\partial J}{\partial x_{k}^{(i+1)}}$ and
$\frac{\partial a_{k}^{(i)}}{\partial z_{k}^{(i)}}$ are nonzero, we have

$x_{l}^{(i)}=\frac{\partial J}{\partial w_{kl}^{(i)}}\left(\frac{\partial
J}{\partial x_{k}^{(i+1)}}\right)^{-1}\left(\frac{\partial
a_{k}^{(i)}}{\partial z_{k}^{(i)}}\right)^{-1}.$ (10)

The case of non-zero bias is due to [2] and we show that it can be extended to
the general case.

###### Proof.

We will prove the general case first. By the construction of a fully-connected
layer, we can take the $j$-th column of the gradient constraint (1b) which
gives:

$(\frac{\partial J}{\partial w_{1j}^{(i)}},...,\frac{\partial J}{\partial
w_{nj}^{(i)}})^{T}=x_{j}^{(i)}(\frac{\partial J}{\partial
z_{1}^{(i)}},...,\frac{\partial J}{\partial z_{n}^{(i)}})^{T}.$ (11)

This implies that if $\frac{\partial J}{\partial z_{k}^{(i)}}\neq 0$ for some
$k,1\leq k\leq n$, then $X^{(i)}$ can be uniquely determined:

$x_{j}^{(i)}=\frac{\partial J}{\partial w_{kj}^{(i)}}(\frac{\partial
J}{\partial z_{k}^{(i)}})^{-1}.$ (12)

In the special case when $b_{k}^{(i)}\neq 0$, we can see from the weight
constraint (1a) that:

$\frac{\partial J}{\partial z_{k}^{(i)}}=\frac{\partial J}{\partial
b_{k}^{(i)}},$ (13)

which was observed in [2] and subsequently also in [3]. In general, since the
activation functions are assumed to be piecewise invertible and piecewise
differentiable, and since $\frac{\partial J}{\partial x_{k}^{(i+1)}}$ is
assumed to be nonzero, we can compute $x_{j}^{(i)}$ using (3), which gives
(10). ∎

##  Appendix B Formulation of the pull-back constraint

Consider the weight constraint from the preceding layer:

$W^{(i-1)}X^{(i-1)}=Z^{(i-1)}.$ (14)

Let $m,n$ denote the number of rows and columns for $W^{(i-1)}$. For the
moment, we view both $X^{(i-1)}$ and $Z^{(i-1)}$ as unknown. If $n<m$ and
$W^{(i-1)}$ has full-rank, then
$n=\operatorname{rank}(W^{(i-1)})<m=|Z^{(i-1)}|$, which implies that the
degree of freedom for $Z^{(i-1)}$ will be smaller than $m$. This reduction of
degree of freedom will impose a prior on the input to the current layer
through the activation $A^{(i-1)}$ because $X^{(i)}=A^{(i-1)}(Z^{(i-1)})$. In
the following discussion, we will express this prior as a non-linear
constraint on $X^{(i)}$ which we call the pull-back constraint (via the
activation function).

We proceed by applying Singular Value Decomposition to $W^{(i-1)}$:

$W^{(i-1)}=R\begin{bmatrix}\Sigma_{n}\\\ 0\\\ \end{bmatrix}S^{T},$ (15)

where $R$ and $S^{T}$ are unitary with sizes $m$ and $n$ respectively,
$\Sigma_{n}$ is diagonal of size $n$ and the zero block has size $m-n$ by $n$.
Notice that we do not require the singular values to be ordered, as long as
they are all contained in the first $n$ rows in the block matrix.

We can now express the weight constraint (14) as a single equation for
$X^{(i)}$. To do that, first we define the following projection operator:

$P^{(i-1)}:=\begin{bmatrix}\operatorname{I}_{n}&0\\\ \end{bmatrix},$ (16)

where $\operatorname{I}_{n}$ is the identity matrix of size $n$ and the zero
block has size $n$ by $m$. Now we rewrite (14) w.r.t. $Z^{(i-1)}$ using (15):

$W^{(i-1)}(\Sigma_{n}S^{T})^{-1}P^{(i-1)}R^{-1}Z^{(i-1)}-Z^{(i-1)}=0.$ (17)

This can be further simplified into:

$R\begin{bmatrix}0&\\\ &\operatorname{I}_{m-n}\\\
\end{bmatrix}R^{T}Z^{(i-1)}=0,$ (18)

where the block matrix has size $m$ by $m$ and we have used $R^{T}=R^{-1}$
since $R$ is unitary. We define an operator $g^{(i)}$ as follows:

$g^{(i)}:=R\begin{bmatrix}0&\\\ &\operatorname{I}_{m-n}\\\
\end{bmatrix}R^{T}(A^{(i)})^{-1},$ (19)

which will be non-linear if $A^{(i)}$ is non-linear. With this definition, the
weight constraint (14) can now be written as a constraint on $X^{(i)}$:

$g^{(i)}(X^{(i)})=0.$ (20)

In two special cases, the pull-back constraint will not be defined. The first
case is when the layer in question is the first one in the network. The second
one is when the weight $W^{(i-1)}$ is invertible or satisfies
$\operatorname{rank}(W^{(i-1)})=m<n$.

######  Remark.

If $W^{(i-1)}$ is rank-deficient and $\operatorname{rank}(W^{(i-1)})<m$, there
will still be a prior for $X^{(i)}$ for the same reason. However, in this case
we cannot obtain the constraint as a function of $X^{(i)}$ because it also
depends on $X^{(i-1)}$ which cannot be eliminated in the deduction of (17).

##  Appendix C Further experimental results

Here we provide results when the activation functions are LeakyReLU for layer
1 and 2, and Sigmoid for the last layer.

(a) Variant 1

(b) Variant 2

(c) Variant 3

(d) Variant 4

Figure 2: Sample outputs from COPA applied to all four variants, when the
activations are LeakyReLU for layer 1 and 2, and Sigmoid for the last layer.
We can observe a consistent pattern compared to Figure 1.

Furthermore, we have computed the MSE and PSNR for the outputs in Figure 1 and
2, to show that they are consistent with the visual observations (the numbers
are MSE,PSNR respectively):

Figure 1, variant 1 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.0512, 61.0389 | 0.0426, 61.8361 | 0.0483, 61.2932 |
Animal | 0.0766, 59.2867 | 0.0596, 60.3813 | 0.0578, 60.5147 |

Figure 1, variant 2 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.1921, 55.2966 | 0.0738, 59.4519 | 0.0502, 61.1218 |
Animal | 0.1913, 55.3148 | 0.0707, 59.6379 | 0.0533, 60.8664 |

Figure 1, variant 3 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.0001, 87.9242 | 0.0088, 68.6745 | 0.0000, 181.5003 |
Animal | 0.0001, 90.5780 | 0.0031, 73.2162 | 0.0000, 180.6153 |

Figure 1, variant 4 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.1007, 58.1021 | 0.0524, 60.9338 | 0.0472, 61.3900 |
Animal | 0.0861, 58.7818 | 0.0373, 62.4189 | 0.0385, 62.2730 |

Figure 2, variant 1 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.0546, 60.7559 | 0.0516, 61.0035 | 25.2923, 34.1009 |
Animal | 0.1046, 57.9361 | 0.0950, 58.3548 | 31.3128, 33.1736 |

Figure 2, variant 2 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.1751, 55.6978 | 0.1016, 58.0623 | 12.3527, 37.2132 |
Animal | 0.2571, 54.0298 | 0.1822, 55.5247 | 3.6857, 42.4656 |

Figure 2, variant 3 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.0355, 62.6284 | 0.0386, 62.2610 | 0.0000, 137.5824 |
Animal | 0.0565, 60.6117 | 0.0563, 60.6291 | 0.0000, 143.4470 |

Figure 2, variant 4 | COPA original | COPA denoised | R-GAP |
---|---|---|---|---
Truck | 0.0868, 58.7466 | 0.0630, 60.1398 | 0.0483, 61.2918 |
Animal | 0.1061, 57.8722 | 0.0908, 58.5509 | 0.0382, 62.3075 |

Table 2: MSE and PSNR for the outputs in Figure 1 and 2 for all variants. In
each cell, the numbers are MSE and PSNR correspondingly.