#  INR-TH-2023-021 Shower formation constraints on cubic Lorentz Invariance
Violation parameters in quantum electrodynamics

Petr Satunin1,2, Andrey Sharofeev2,1
1 Institute for Nuclear Research of the Russian Academy of Sciences,
60th October Anniversary Prospect, 7a, 117312 Moscow, Russia
2 Department of Particle Physics and Cosmology, Physics Faculty,
Moscow State University,
Leninskiye Gory, 119991 Moscow, Russia e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present shower formation constraints on the Lorentz Invariance Violation
(LIV) energy scale for photons with cubic dispersion relation from recent
gamma ray observations in $100$ TeV – PeV energy range by LHAASO observatory.
We assume Myers-Pospelov effective field theory framework, and calculate the
suppression for the Bethe-Heitler process which is mainly responsible for the
formation of photon-initiated atmosphere showers. Comparing the high-energy
photon events with the suppressed flux predictions we obtain $95\%$ CL
constraints on the LIV energy scale. The obtained shower constraint $E_{\rm
LIV}\gtrsim\mathcal{O}\left(10^{20}\mbox{ GeV}\right)$ is significantly weaker
than existing birefringence constraints but is independent.

##  1 Introduction

The formalism of quantum field theory based on some fundamental symmetries
including Lorentz Invariance (LI) as a cornerstone. However, possible
deviations from LI may appear in several scenarios for quantum gravity (see
[1] for a review, [2, 3] and reference therein). Traditionally, in the
effective field theory (EFT) approach one can gather all possible LIV
operators for the Standard Model fields into so-called Standard Model
Extension (SME) [4]. While low-order LIV terms are usually tested with precise
laboratory experiments [5], high-order terms seem irrelevant at laboratory
energies but may be significant at higher energies.

High-energy cosmic rays are the best tool for these tests since its energy in
the laboratory frame is larger than those achievable at ground-based
accelerators. Hypothetical LIV can influence different processes relevant to
production, propagation and detection of these cosmic rays. A pure separate
channel of the SME is the quantum electrodynamics (QED) sector. The number of
parameters in the QED sector reduces significantly, so it can be deeply
studied. The key but not only characteristic of LIV of this type is the
modification of dispersion relations,

$E^{2}=p^{2}\pm\frac{p^{3}}{E_{\rm{LIV},1}}\pm\frac{p^{4}}{E_{\rm{LIV},2}^{2}}+...$
(1)

Thus, extra terms proportional to high powers of energy suppressed by large
energy scale appear.

The most consistent consideration of the case leading to LIV can be
investigated through the construction of EFTs, as it was noted above. From
such theories, for example, a non-standard form of dispersion relations can
immediately follow. The advantage of considering such theories allows one to
use the whole formalism of quantum field theory, thus obtaining most
consistently, for example, cross sections of processes. The only EFT leading
to the cubic momentum violation of LI in the QED is the Myers-Pospelov model
[6] which is considered in the current article.

LIV induces some processes forbidden in LI and modifies thresholds of existing
processes [7, 8, 9]. For example, some LIV allows channels of photon decay
which are forbidden in LI, such as $\gamma\to e^{+}e^{-}$ and $\gamma\to
3\gamma$. Being allowed, these processes eliminate photons of given energy
from the flux. Additionally, LIV can influence air shower formation.
Particularly, the cross-section of the most probable first reaction in air
showers – the Bethe-Heitler process — modified in case of LIV which leads to
deeper air showers [10]. Detection of normal air showers sets constraints on
LIV parameters.

The paper is organized as follows. In Sec. 2, we discuss the Myers-Pospelov
EFT and its basis properties.In Sec. 3, we discuss the air shower formation by
very-high-energy photons and the following constraints to the observed flux.
In Sec. 4, we estimate observational bounds at the LHAASO experiment for three
astrophysical sources. The Sec. 5 is devoted to conclusions.

##  2 Myers-Pospelov model – an EFT for cubic LIV

The full EFT Lagrangian which leads to cubic LIV in particle dispersion
relations has been written by Myers and Pospelov [6]. The part of the Myers-
Pospelov action related separately to photons reads,

$\mathcal{L}_{\gamma}=-\frac{1}{4}F_{\mu\nu}F_{\mu\nu}+\frac{\xi}{M_{\rm{Pl}}}n^{\mu}F_{\mu\nu}n\cdot\partial\left(n_{\sigma}\tilde{F}^{\sigma\nu}\right).$
(2)

The parameter $\xi$ is a dimensionless constant, $M_{\rm{Pl}}$ is the Planck
mass. The four-vector $n^{\mu}$ is a purely time-like vector,
$n^{\mu}=(1,0,0,0)$. Note that CPT parity is violated as well.

The equations of motion in the model (2) reads,

$\Box{A^{\tau}}=\frac{\xi}{M_{\rm{Pl}}}n_{\sigma}\epsilon^{\sigma\tau\rho\nu}\left(n\cdot\partial\right)^{2}F_{\rho\nu},$
(3)

The two transverse polarizations of the photon, determined by the equations of
motion (3), have the following dispersion relations:

$\omega^{2}=k^{2}\pm\frac{k^{3}}{E_{\rm LIV}},$ (4)

where the sign plus (minus) related to left (right) polarized photon, and we
introduce the typical energy scale for LIV,

$E_{\rm LIV}\equiv\frac{M_{\rm Pl}}{2\xi}.$ (5)

The main effect of modified dispersion relation (4) is that the group velocity
of traveling photons depends on its energy and polarization, $v=1\pm k/E_{\rm
LIV}$, where we recall left polarization related to the sign “+” as
“superluminal” polarization, right polarization related to the sign “-” — as
“subluminal” one.

The clue difference with the LIV QED, quartic on spatial momentum [11] is that
in quartic theory both polarizations have the same dispersion relation with
specific sign, “+” or “-”. So, the “super-” and “subluminal” photons are
related to independent sectors of model parameters in that model, and the
constraints on these sectors are set independently. Contrary, in our cubic
model for fixed LIV mass scale one has both super- and subluminal photons
simultaneously. Two phenomena based on modified velocities and important in
bounding $E_{\rm LIV}$ are time delays and vacuum birefringence.

#### Time delays.

Due to the energy dependence for the photon velocity, photons traveling from
cosmologically distant astrophysical sources would acquire additional
dispersion. The absence of such phenomena constrain $E_{\rm LIV}$. The actual
constraint came from GRB090510, $E_{\rm LIV}>9.3\cdot 10^{19}$ GeV [12].

#### Vacuum birefringence.

This phenomenon manifests itself as the rotation of the polarization plane for
a photon beam. Even small rotation, being accumulated over vast cosmological
distances, may be potentially detectable, see Sec. 5.2.1 of [1] for details.
The most stringent constraint on the mass scale came from polarized GRB061122
[13], $E_{\rm LIV}\sim\mathcal{O}\left(10^{34}\mbox{ GeV}\right)$.

This constraint is sufficiently trans-Planckian and may close the Quantum
Gravity motivation for these models. Nevertheless, independent consideration
of cubic LIV from threshold processes may also be interesting. Any
contradictions, if they appear, may lead to a breakdown of the EFT approach.

#### Modification of perturbative processes

The cross-sections of several perturbative processes including photons in the
model (2) get modified compared to the standard case. Moreover, super- and
subluminal photons may lead to different outcomes. Superluminal photons have
an excess of energy compared to the momentum and can release it in any allowed
decay process. Subluminal photons, in turn, have a lack of energy any
interaction channel of the subluminal photon gets suppressed.

#### Photon decay

The simplest perturbative process is the photon decay. Superluminal photon
polarization undergoes photon decay to an electron-positron pair for a photon
with energy larger than the threshold one, $E^{\rm
th}_{\gamma}=\sqrt[3]{4m^{2}E_{\rm LIV}}$, $m$ is the mass of an electron.
Once kinematically allowed, the decay occurs almost immediately. Since, the
detection of even one superluminal photon with energy $E_{\gamma}$ sets the
bound,

$E_{\rm LIV}>\frac{E_{\gamma}^{3}}{4m^{2}}.$ (6)

The corresponding bound for superluminal photons are well-studied in the
literature [7, 8, 14, 9]. Another possible decay channel for the superluminal
polarization is the photon splitting [15, 16].

Let us turn to what does mean the constraint on super(or sub)luminal photons
in the context of the Myers-Pospelov model. At high energies of the photon the
photon polarization cannot be easily measured in the process of the detection.
Thus, if we detect a photon of a given energy, one cannot immediately write
the bound (6) but first study the outcomes if it was a photon of a subluminal
polarization.

Two processes whose modifications are important in case of subluminal photon
dispersion relation are pair production by a high energy photon in a Coulomb
field (Bethe-Heitler process), and pair production on soft background photons.

##  3 Pair production on a nuclei and Air shower formation

The first process suppressed for subluminal photon is the Bethe-Heitler
process — photon decay to an electron-positron pair in the Coulomb field of a
nuclei. This process is responsible for the first interaction of astrophysical
photons in the atmosphere. The suppression of the Bethe-Heitler process in a
subluminal LIV scenario was firstly estimated by [10].

Consequently, in such scenarios, atmospheric showers initiated by photons
would penetrate more deeply into the atmosphere compared to the conventional
scenario. These exceptionally deep showers may remain undetected during
experiments. Therefore, the forecast for subluminal LIV resembles that of the
superluminal case, both resulting in the suppression of photon flux,
particularly for the highest-energy photons. Hence, in the subluminal
scenario, photon-induced air showers would commence at greater atmospheric
depths compared to the standard case.

In the event that the initial photon interaction’s depth, represented as
$X_{0}$, within the atmosphere surpasses the overall atmospheric depth,
denoted as $X_{\rm{atm}}$, the development of the shower will be impeded,
resulting in the event remaining undetected.

The probability for a photon to produce pair in the atmosphere reads,

$P=\int\limits_{0}^{X_{\rm{atm}}}\differential{X_{0}}\frac{e^{-X_{0}/\expectationvalue{X_{0}}_{\rm{LIV}}}}{\expectationvalue{X_{0}}_{\rm{LIV}}}=1-e^{-X_{\rm{atm}}/\expectationvalue{X_{0}}_{\rm{LIV}}},$
(7)

where the mean depth of the interaction $\expectationvalue{X_{0}}_{LIV}$ in
the LIV case can be expressed via the mean depth
$\expectationvalue{X_{0}}_{LI}=57\mbox{ g cm}^{-2}$ in the LI case and the
ratio of the cross sections $\sigma^{\rm{LIV}}$ and $\sigma^{\rm{LI}}$ as
follows,

$\expectationvalue{X_{0}}_{\rm{LIV}}=\frac{\sigma^{\rm{LI}}}{\sigma^{\rm{LIV}}}\expectationvalue{X_{0}}_{\rm{LI}}.$
(8)

The ratio of cross sections will be used as follows,

$\frac{\sigma^{\rm{LIV}}}{\sigma^{\rm{LI}}}\simeq
1.7\cdot\frac{m^{2}E_{\rm{LIV}}}{E_{\gamma}^{3}}\log\frac{E_{\gamma}^{3}}{2m^{2}E_{\rm{LIV}}}.$
(9)

We see that the cross section in LIV case decreases with energy as
$E_{\gamma}^{-3}\log E_{\gamma}$ under the condition that $E_{\rm{LIV}}$.

Using (9) we can transform (7) in the following way,

$P(E_{\gamma},E_{\rm LIV})=1-\exp(-1.7\cdot\frac{X_{\rm{atm}}}{57\mbox{ g
cm}^{-2}}\frac{m^{2}E_{\rm{LIV}}}{E_{\gamma}^{3}}\log\frac{E_{\gamma}^{3}}{2m^{2}E_{\rm{LIV}}}).$
(10)

The prediction for the observed flux can be formulated as follows:

$\left(\derivative{\Phi}{E}\right)_{\rm{LIV}}=P(E_{\gamma},E_{\rm
LIV})\times\evaluated{\derivative{\Phi}{E}}_{\rm{source}}.$ (11)

Here, we use the suppression factor denoted as $P(E_{\gamma},E_{\rm LIV})$,
which is defined in equation (10), to assess the potential for particle shower
formation. The hypothesis regarding a specific value of $P(E_{\gamma},E_{\rm
LIV})$ can be tested with experimental data. It is important to note that
conducting these tests requires selecting a model that describes the photon
flux from the source. To obtain the most reliable constraints on the LIV
parameter $E_{\rm LIV}$, when considering multiple source models, we choose
the model that predicts the highest photon flux.

###  3.1 Pair production on EBL and CMB

High energy photons, traveling in soft photon backgrounds, undergo the pair
production process, $\gamma\gamma_{b}\to e^{+}e^{-}$, which results in the
attenuation of the photon flux $e^{-\tau}$ . This process goes if photon
energy exceed the threshold [17],

$E_{\rm{th}}=\frac{m^{2}}{\omega_{b}}\mp\frac{1}{4}\frac{k^{2}}{E_{\rm{LIV}}},$
(12)

where the sign ”$+$” goes from the subluminal case, the sign ”$-$” goes from
the superluminal case.

The cross-section of the process above the threshold remains relatively
insensitive to LIV [18]. A significant consequence of this process is the
reduction in the flux of high-energy photons, typically in the TeV energy
range, originating from extragalactic sources as they interact with
extragalactic background light (EBL) [19]. In the case of subluminal
dispersion relation, the pair production threshold shifts to higher energies
of background photons, which is associated with lower EBL density, leading to
less attenuation in the extragalactic photon spectrum [20, 21]. The absence of
this phenomenon places constraints on the viability of the subluminal
dispersion relation[22]. When considering galactic gamma rays within the
energy range of 1 to 100 TeV, one can safely disregard the pair production
effect due to the relatively short distances within the galaxy compared to the
mean free path of pair production in the presence of the infrared background.

The detection of galactic photons with energies below a PeV by Tibet-
AS$\gamma$ and LHAASO has brought about changes in the threshold for pair
production, shifting it into the range of cosmic microwave background (CMB)
energy levels. Notably, the density of CMB surpasses that of extragalactic
background light (EBL) by over two orders of magnitude. This implies that the
mean free path for a photon with an energy of approximately $\sim\mbox{ PeV}$,
involved in CMB-induced pair production, is around $\sim 10\mbox{ kpc}$, a
length scale comparable to galactic scales [23, 24].

The attenuation of galactic gamma-ray flux can be expressed through the
attenuation factor denoted as $P_{\rm CMB}$, given by the exponential function

$P_{\rm CMB}(L_{\rm source},E_{\gamma},E_{\rm LIV})=\exp(-L_{\rm
source}/L_{\rm MFP}(E_{\gamma},E_{\rm LIV})),$ (13)

where $L_{\rm MFP}$ represents the mean free path of photons. In the scenario
of subluminal LIV, the threshold for pair production is elevated above the
energy corresponding to the peak of the CMB spectrum, causing the attenuation
effect to vanish. Consequently, in the subluminal LIV framework, the
modification of pair production results in an increase in the observed gamma-
ray flux, while the shower suppression effect leads to a reduction.

Nevertheless, it is worth noting that for the spectra observed in references
[25, 26], the factor $P_{\rm CMB}$ cannot fall below $0.5$. Therefore,
although the impact of modified pair production on CMB is secondary in
comparison to the shower suppression effect it should be definitely taken into
account by dividing eq. (11) on (13),

$\left(\derivative{\Phi}{E}\right)_{\rm{LIV}}=\frac{P(E_{\gamma},E_{\rm
LIV})}{P_{\rm CMB}(L_{\rm source},E_{\gamma},E_{\rm
LIV})}\times\evaluated{\derivative{\Phi}{E}}_{\rm{source}}.$ (14)

This formula is to test the hypothesis of a certain $E_{\rm LIV}$

##  4 Observational bounds – LHAASO

The Large High Altitude Air Shower Observatory (LHAASO), located at $4,410$
meters above sea level in Daocheng, China, began its first-stage operations in
2019. It will have sensitivity to gamma rays up to $\mbox{TeV}-\mbox{PeV}$
range and cosmic rays in the TeV to $10^{3}\mbox{ PeV}$ range. LHAASO has
recently reported spectrum of 12 sources in our galaxy with energy up to
$1.4\mbox{ PeV}$ [25], and Crab later based on the observation of the highest-
energy photons ever detected, up to $1.4\mbox{ PeV}$, and later the photons
from Crab [27] with implications for the LIV purposes [28, 25, 29, 30, 31].
Particularly, the 3 gamma-ray sources with energy, which were provided by
LHAASO, especially, J2226+6057 and J1908+0621 [25], Crab Nebula [27] were
considered in this work to constrain $E_{\rm LIV}$ with shower formation.

The maximum zenith angle at which LHAASO events occur is 50 degrees [25]. So,
one can compute the maximum atmospheric depth for inclined showers at the
LHAASO altitude, $X_{\rm atm}=890\mbox{ g }\mbox{cm}^{-2}$.

We introduce the following criterion for the selection of bins in the case of
J2226+6057, J1908+0621, and Crab Nebula events. On one hand, we consider only
those energy bins that exhibit a significance greater than $2\sigma$,
therefore, we omit the last bin for J1908+0621 and the two last bins for
J2226+6057. On the other hand, we do not use energy bins in cases where the
variation between the LI and LIV scenarios is negligible.

Figure 1: The galactic source spectra measured by LHAASO (data points), the
power-law fit for the spectra (green solid curve), the predicted flux under
subluminal LIV (yellow dash-dotted curve). Left panel: J2226+6057, Right
panel: J1908+0621

In Fig.1, we illustrate the differential energy spectra corresponding to the
sources J2226+6057 and J1908+0621. The data points were obtained
experimentally [25], and the graphical representation includes the optimal
power-law fit alongside the predicted flux under the assumption of subluminal
LIV. By following these guidelines, we conduct a $\chi^{2}$ test and constrain
the mass scale $E_{\rm LIV}$ at a $95\%$ confidence level (CL). The results of
the analysis are presented in the Table 1.

Source | $L,\mbox{ kpc}$ | Bound $E_{\rm LIV}$, $10^{20}\mbox{ GeV}$
---|---|---
Crab Nebula | 2 | $0.5$
J2226+6057 | 0.8 | $1.5$
J1908+0621 | 2.37 | $2.1$
Table 1: The $95\%$ CL constraints on LIV mass scale from 3 sub-PeV sources
observed by LHAASO.

The simplest bound for the superluminal polarization came from the photon
decay. Substituting LHAASO maximal energy $1.4$ PeV into eq. (6), one obtains

$E_{\rm LIV}>2.6\cdot 10^{24}\,\mbox{GeV}.$ (15)

The combined bound on $E_{\rm LIV}$ from super- and subluminal polarization is
as follows. Supposing superluminal dispersion for these detected very-high-
energy photons, one obtains the bound (15). Supposing subluminal dispersion,
we should take the best bound from Table 1, $E_{\rm LIV}>2.1\cdot
10^{20}\,\mbox{GeV}$. Not knowing the real initial polarization, we should
take the worst bound from two aforementioned ones, so we take finally the
value

$E_{\rm LIV}>2.1\cdot 10^{20}\,\mbox{GeV}.$ (16)

for the constraint. No super/subluminal polarization here is assumed.

##  5 Discussion

In this article we studied the suppression for the air shower formation for
the general EFT with cubic dispersion relation for photons (Myers-Pospelov
model). We test the hypothesis of LIV with a certain mass scale $E_{\rm LIV}$
with the photon spectra of the three LHAASO sources. Taking into account that
these photons have not been completely vanished by the photon decay, and the
spectra haven’t been significantly depleted due to the suppression of air
shower formation, we set the bound on LIV mass scale,

$E_{\rm LIV}>2.1\cdot 10^{20}\,\mbox{GeV}.$

Compared to the QED with quartic photon dispersion in which a model with
certain LIV mass scale divided additionally to “super” and “subluminal”
scenario [32], we want to stress out that in cubic Myers-Pospelov model there
are no additional scenarios. Super and subluminality is a property of a given
photon polarization, not a whole model.

Note that the obtained $E_{\rm LIV}$ bound is sufficiently trans-Planckian.
Although it is hard to imagine any reasonable quantum gravity theory at these
energy scales, trans-Plankian coupling constant still may appear due to fine
tuning or feebly interaction between QED sector and gravity.

#### Acknowledgements

The authors thank Jose Manuel Carmona, Tomislav Tersic, Gleb Suzdalov,
Nickolay Martynenko, Pavel Demidov and Sergey Troitsky for helpful
discussions. This work is supported by RSF foundation under contract
22-12-00253.

## References

  * [1] A. Addazi et al., “Quantum gravity phenomenology at the dawn of the multi-messenger era—A review,” Prog. Part. Nucl. Phys., vol. 125, p. 103948, 2022.
  * [2] S. Liberati, “Tests of lorentz invariance: a 2013 update,” Classical and Quantum Gravity, vol. 30, p. 133001, jun 2013.
  * [3] D. Mattingly, “Modern tests of lorentz invariance,” Living Reviews in relativity, vol. 8, no. 1, pp. 1–84, 2005.
  * [4] D. Colladay and V. A. Kostelecky, “Lorentz violating extension of the standard model,” Phys. Rev. D, vol. 58, p. 116002, 1998.
  * [5] V. A. Kostelecky and N. Russell, “Data Tables for Lorentz and CPT Violation,” Rev. Mod. Phys., vol. 83, pp. 11–31, 2011.
  * [6] R. C. Myers and M. Pospelov, “Ultraviolet modifications of dispersion relations in effective field theory,” Phys. Rev. Lett., vol. 90, p. 211601, May 2003.
  * [7] S. Coleman and S. L. Glashow, “Cosmic ray and neutrino tests of special relativity,” Physics Letters B, vol. 405, pp. 249–252, jul 1997.
  * [8] S. Coleman and S. L. Glashow, “High-energy tests of lorentz invariance,” Physical Review D, vol. 59, apr 1999.
  * [9] T. Jacobson, S. Liberati, and D. Mattingly, “Lorentz violation at high energy: Concepts, phenomena, and astrophysical constraints,” Annals of Physics, vol. 321, pp. 150–196, jan 2006.
  * [10] H. Vankov and T. Stanev, “Lorentz invariance violation and the QED formation length,” Phys. Lett. B, vol. 538, pp. 251–256, 2002.
  * [11] G. Rubtsov, P. Satunin, and S. Sibiryakov, “On calculation of cross sections in Lorentz violating theories,” Phys. Rev. D, vol. 86, p. 085012, 2012\.
  * [12] V. Vasileiou, A. Jacholkowska, F. Piron, J. Bolmont, C. Couturier, J. Granot, F. W. Stecker, J. Cohen-Tanugi, and F. Longo, “Constraints on Lorentz Invariance Violation from Fermi-Large Area Telescope Observations of Gamma-Ray Bursts,” Phys. Rev. D, vol. 87, no. 12, p. 122001, 2013.
  * [13] D. Gotz, S. Covino, A. Fernandez-Soto, P. Laurent, and Z. . Bosnjak, “The polarized Gamma-Ray Burst GRB 061122,” Mon. Not. Roy. Astron. Soc., vol. 431, p. 3550, 2013.
  * [14] T. Jacobson, S. Liberati, and D. Mattingly, “Threshold effects and Planck scale Lorentz violation: Combined constraints from high-energy astrophysics,” Phys. Rev. D, vol. 67, p. 124011, 2003.
  * [15] G. Gelmini, S. Nussinov, and C. E. Yaguna, “On photon splitting in theories with Lorentz invariance violation,” JCAP, vol. 06, p. 012, 2005.
  * [16] K. Astapov, D. Kirpichnikov, and P. Satunin, “Photon splitting constraint on Lorentz Invariance Violation from Crab Nebula spectrum,” JCAP, vol. 04, p. 054, 2019.
  * [17] M. Galaverni and G. Sigl, “Lorentz Violation in the Photon Sector and Ultra-High Energy Cosmic Rays,” Phys. Rev. Lett., vol. 100, p. 021102, 2008.
  * [18] G. Rubtsov, P. Satunin, and S. Sibiryakov, “Calculation of cross sections in lorentz-violating theories,” Phys. Rev. D, vol. 86, p. 085012, Oct 2012\.
  * [19] J. Biteau and D. A. Williams, “The extragalactic background light, the hubble constant, and anomalies: Conclusions from 20 years of tev gamma-ray observations,” The Astrophysical Journal, vol. 812, p. 60, oct 2015.
  * [20] T. Kifune, “Invariance violation extends the cosmic-ray horizon?,” The Astrophysical Journal, vol. 518, p. L21, may 1999.
  * [21] F. W. Stecker and S. L. Glashow, “New tests of Lorentz invariance following from observations of the highest energy cosmic gamma-rays,” Astropart. Phys., vol. 16, pp. 97–99, 2001.
  * [22] H. Martínez-Huerta, R. G. Lang, and V. de Souza, “Lorentz invariance violation tests in astroparticle physics,” Symmetry, vol. 12, no. 8, 2020\.
  * [23] R. J. Gould and G. P. Schréder, “Opacity of the universe to high-energy photons,” Phys. Rev., vol. 155, pp. 1408–1411, Mar 1967.
  * [24] S. Vernetto and P. Lipari, “Absorption of very high energy gamma rays in the Milky Way,” Phys. Rev. D, vol. 94, no. 6, p. 063009, 2016.
  * [25] Z. Cao, F. Aharonian, Q. An, Axikegu, L. Bai, Y. Bai, Y. Bao, D. Bastieri, X. Bi, Y. Bi, et al., “Ultrahigh-energy photons up to 1.4 petaelectronvolts from 12 $\gamma$-ray galactic sources,” Nature, vol. 594, no. 7861, pp. 33–36, 2021.
  * [26] M. Amenomori et al., “First Detection of Photons with Energy Beyond 100 TeV from an Astrophysical Source,” Phys. Rev. Lett., vol. 123, no. 5, p. 051101, 2019.
  * [27] Z. Cao et al., “Peta–electron volt gamma-ray emission from the Crab Nebula,” Science, vol. 373, no. 6553, pp. 425–430, 2021.
  * [28] D. Cyranoski, “China’s mountain observatory begins hunt for origins of cosmic rays.,” Nature, 2019.
  * [29] C. Li and B.-Q. Ma, “Ultrahigh-energy photons from LHAASO as probes of Lorentz symmetry violations,” Phys. Rev. D, vol. 104, no. 6, p. 063012, 2021.
  * [30] L. Chen, Z. Xiong, C. Li, S. Chen, and H. He, “Strong constraints on Lorentz violation using new $\gamma$-ray observations around PeV,” Chin. Phys. C, vol. 45, no. 10, p. 105105, 2021.
  * [31] P. Satunin, “Two-sided constraints on Lorentz invariance violation from Tibet-AS$\gamma$ and LHAASO very-high-energy photon observations,” Eur. Phys. J. C, vol. 81, no. 8, p. 750, 2021.
  * [32] G. Rubtsov, P. Satunin, and S. Sibiryakov, “Constraints on violation of Lorentz invariance from atmospheric showers initiated by multi-TeV photons,” JCAP, vol. 05, p. 049, 2017.


i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h\mathbb{E}\Biggr{[}1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{i_{1},i_{2}}\right)\right\\}}\chi\left(\frac{X_{t_{l}}^{i_{1},i_{2}}}{\varepsilon}\right)\\\
&\qquad\qquad\times\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},i_{2}}\right)\theta\left(X_{t_{l}}^{i_{1},i_{2}},t_{l}\right)\Biggr{]},\end{split}$
(6.7)

for $k=0,1,2,\ldots$, where

$\varTheta_{D,\delta}(y,x_{1})=-\left.\frac{\partial}{\partial
x_{2}}\right|_{x_{2}=0+}K_{D,\delta}^{1}(y,x).$

We compute the derivative explicitly as follows

$\displaystyle\varTheta_{D,\delta}(y,x_{1})=1_{\\{y_{2}>0\\}}(y)\frac{1}{\pi}\left(\left(1-\textrm{e}^{-\frac{|y_{1}-x_{1}|^{2}+|y_{2}|^{2}}{\delta^{2}}}\right)\frac{|y_{1}-x_{1}|^{2}-|y_{2}|^{2}}{\left(|y_{1}-x_{1}|^{2}+|y_{2}|^{2}\right)^{2}}\right.$
$\displaystyle\left.\frac{1}{\delta^{2}}\textrm{e}^{-\frac{|y_{1}-x_{1}|^{2}+|y_{2}|^{2}}{\delta^{2}}}\frac{|y_{2}|^{2}}{|y_{1}-x_{1}|^{2}+|y_{2}|^{2}}\right).$

In the second step, one has to handle the mathematical expectations in
representations in (6.4) and (6.7), which is the core of the Monte-Carlo
schemes. We propose two slightly different approaches which lead to the
following schemes.

###  6.1 Numerical scheme 1

We are now in a position to formulate our first Monte-Carlo scheme in which we
drop the expectation by running independent (two-dimensional) Brownian
motions. More precisely, we run the following stochastic differential
equations:

$X_{0}^{i_{1},i_{2}}=x^{i_{1},i_{2}},$
$X_{t_{k+1}}^{i_{1},i_{2}}=X_{t_{k}}^{i_{1},i_{2}}+hu(X_{t_{k}}^{i_{1},i_{2}},t_{k})+\sqrt{2\nu}\sqrt{t_{k+1}-t_{k}}\Phi_{k}$

where

$\Phi_{k}=\left(\Phi_{k}^{1},\Phi_{k}^{2}\right)\equiv\frac{B_{t_{k+1}}-B_{t_{k}}}{\sqrt{t_{k+1}-t_{k}}},$

with $\Phi_{k}^{1}$, $\Phi_{l}^{2}$ (for $k,l=0,1,\ldots$) being independent
standard Gaussian random variables. $u(x,t)$ is updated according to the
following:

$\displaystyle u(x,t_{k+1})$
$\displaystyle=\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\left(K_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},i_{2}}\right)-K_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},-i_{2}}\right)\right)\omega_{i_{1},i_{2}}$
$\displaystyle+\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{i_{1},i_{2}}\right)\right\\}}K_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},i_{2}}\right)G\left(X_{t_{l}}^{i_{1},i_{2}},t_{l}\right)$
$\displaystyle+\frac{\nu}{\varepsilon^{2}}\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{i_{1},i_{2}}\right)\right\\}}\chi\left(\frac{X_{t_{l}}^{i_{1},i_{2}}}{\varepsilon}\right)$
$\displaystyle\qquad\qquad\times
K_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},i_{2}}\right)\theta\left(X_{t_{l}}^{i_{1},i_{2}},t_{l}\right),$

and

$\displaystyle\theta(x_{1},t_{k+1})$
$\displaystyle=\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\left(\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},i_{2}}\right)-\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},-i_{2}}\right)\right)\omega_{i_{1},i_{2}}$
$\displaystyle+\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{i_{1},i_{2}}\right)\right\\}}\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},i_{2}}\right)G\left(X_{t_{l}}^{i_{1},i_{2}},t_{l}\right)$
$\displaystyle+\frac{\nu}{\varepsilon^{2}}\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{i_{1},i_{2}}\right)\right\\}}\chi\left(\frac{X_{t_{l}}^{i_{1},i_{2}}}{\varepsilon}\right)$
$\displaystyle\qquad\qquad\times\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{i_{1},i_{2}}\right)\theta\left(X_{t_{l}}^{i_{1},i_{2}},t_{l}\right),$

for $x=(x_{1},x_{2})$ with $x_{2}>0$, and

$(u^{1}(x,t),u^{2}(x,t))=(u^{1}(\overline{x},t),-u^{2}(\bar{x},t))\quad\textrm{
when }x_{2}<0.$

###  6.2 Numerical scheme 2

In this scheme we appeal to the strong law of large numbers, that is we
replace the expectations by averages. Therefore we run the following
stochastic differential equations:

$X_{0}^{m;i_{1},i_{2}}=x^{i_{1},i_{2}},$
$X_{t_{k+1}}^{m;i_{1},i_{2}}=X_{t_{k}}^{m;i_{1},i_{2}}+hu(X_{t_{k}}^{m;i_{1},i_{2}},t_{k})+\sqrt{2\nu}\sqrt{t_{k+1}-t_{k}}\Phi_{k}^{m}$

where

$\Phi_{k}^{m}=\left(\Phi_{k}^{m,1},\Phi_{k}^{m,2}\right)\equiv\frac{B_{t_{k+1}}^{m}-B_{t_{k}}^{m}}{\sqrt{t_{k+1}-t_{k}}},$

and $\Phi_{k}^{m,1}$, $\Phi_{l}^{n,2}$ (for $m,n=1,2,\ldots$ and
$k,l=0,1,\ldots$) are independent standard Gaussian random variables. Here
$u(x,t)$ is updated according to the following:

$\displaystyle u(x,t_{k+1})$
$\displaystyle=\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\frac{1}{N}\sum_{m=1}^{N}\left(K_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},i_{2}}\right)-K_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},-i_{2}}\right)\right)\omega_{i_{1},i_{2}}$
$\displaystyle+\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h\frac{1}{N}\sum_{m=1}^{N}1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{m;i_{1},i_{2}}\right)\right\\}}K_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},i_{2}}\right)G\left(X_{t_{l}}^{m;i_{1},i_{2}},t_{l}\right)$
$\displaystyle+\frac{\nu}{\varepsilon^{2}}\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h\frac{1}{N}\sum_{m=1}^{N}1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{m;i_{1},i_{2}}\right)\right\\}}\chi\left(\frac{X_{t_{l}}^{m;i_{1},i_{2}}}{\varepsilon}\right)$
$\displaystyle\qquad\qquad\times
K_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},i_{2}}\right)\theta\left(X_{t_{l}}^{m;i_{1},i_{2}},t_{l}\right),$

and

$\displaystyle\theta(x_{1},t_{k+1})$
$\displaystyle=\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\frac{1}{N}\sum_{m=1}^{N}\left(\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},i_{2}}\right)-\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},-i_{2}}\right)\right)\omega_{i_{1},i_{2}}$
$\displaystyle+\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h\frac{1}{N}\sum_{m=1}^{N}1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{m;i_{1},i_{2}}\right)\right\\}}\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},i_{2}}\right)G\left(X_{t_{l}}^{m;i_{1},i_{2}},t_{l}\right)$
$\displaystyle+\frac{\nu}{\varepsilon^{2}}\sum_{\begin{subarray}{c}i_{1},i_{2}\\\
i_{2}>0\end{subarray}}A_{i_{1},i_{2}}\sum_{l=0}^{k}h\frac{1}{N}\sum_{m=1}^{N}1_{\left\\{t_{l}>\gamma_{t_{k}}\left(X^{m;i_{1},i_{2}}\right)\right\\}}\chi\left(\frac{X_{t_{l}}^{m;i_{1},i_{2}}}{\varepsilon}\right)$
$\displaystyle\qquad\qquad\times\varTheta_{D,\delta}\left(x,X_{t_{k+1}}^{m;i_{1},i_{2}}\right)\theta\left(X_{t_{l}}^{m;i_{1},i_{2}},t_{l}\right),$

for $x=(x_{1},x_{2})$ with $x_{2}>0$, and

$(u^{1}(x,t),u^{2}(x,t))=(u^{1}(\overline{x},t),-u^{2}(\bar{x},t))\quad\textrm{
when }x_{2}<0.$

###  6.3 Numerical experiment

Recall the Reynolds number is given by $\textrm{Re}=\frac{U_{0}L}{\nu}$ with
the main stream velocity $U_{0}$ determined by the initial velocity in our
experiments, and $L$ is the typical size. In our experiment, we set

$\nu=0.01$ (6.8)

for the viscosity constant, and the typical length scale

$L=6.$ (6.9)

We also set the initial main stream velocity

$U_{0}=1,$ (6.10)

which altogether imply that $\textrm{Re}=600$ in our setup.

In the experiment, we choose the initial velocity to be
$u_{0}(x_{1},x_{2})=U_{0}1_{x_{2}}$, i.e. the uniform flow that is set to
satisfy the no-slip condition at the wall. Some non-trivial motion is then
incited by the external force term $G(x,t)\equiv G_{0}$ which we choose as
$G_{0}=-0.2$. The kernel regularisation parameter $\sigma=0.01$, and the
cutoff parameter $\varepsilon=0.05$.

As described above, we choose an outer lattice to be a rectangular lattice in
the domain $\\{-6\leq x_{1}\leq 6,0\leq x_{2}\leq 6\\}$. The mesh size for the
lattice is $h_{0}=0.15$ and the number $N_{0}=40$. For the lattice in the
boundary layer, we the mesh size has to satisfy

$h_{2}\ll\delta=L\sqrt{\frac{1}{\textrm{Re}}}\sim 0.2449.$

In the simulation the chosen parameters are $h_{1}=0.1$, $h_{2}=0.00125$ with
numbers $N_{1}=60$, $N_{2}=80$. This choice gives $26,042$ points in total.

We implement the numerical scheme 1 from Subsection 6.1. The time mesh is
taken as $h=0.01$ and the iteration is conducted for $300$ steps. The
simulation results for the boundary flow are presented in Figure 6.1. In the
figures, the streamlines are coloured by the magnitude of the velocity and the
background colour gives the value of the vorticity. For the outer outer flow,
we present the results in Figure 6.2.

(a) $t=0.01$

(b) $t=0.25$

(c) $t=0.5$

(d) $t=1.0$

(e) $t=1.5$

(f) $t=2.0$

(g) $t=2.5$

(h) $t=3.0$

Figure 6.1: The boundary layer flow at different times $t$.

(a) $t=0.01$

(b) $t=1.0$

(c) $t=2.0$

(d) $t=3.0$

Figure 6.2: The outer layer flow at different times $t$.

The outside boundary layer flow close to the rigid boundary can be seen in
Figure 2.

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## Acknowledgement

The authors would like to thank Oxford Suzhou Centre for Advanced Research for
providing the excellent computing facility VC and ZQ are supported (fully and
partially, respectively) by the EPSRC Centre for Doctoral Training in
Mathematics of Random Systems: Analysis, Modelling and Simulation
(EP/S023925/1). SWE is funded by Deutsche Forschungsgemeinschaft (DFG) -
Project-ID 410208580 - IRTG2544 (’Stochastic Analysis in Interaction’).

## References

  * [1] Anderson, C. and Greengard, C. 1985 On vortex methods _. SIAM J. Numer. Anal_. $\mathbf{22}$ (3), 413-440.
  * [2] Chorin, A. J. 1973 Numerical study of slightly viscous flow. _J. Fluid Mech._ $\mathbf{57}$, 785-796.
  * [3] Cottet, G. -H., and Koumoutsakos, P. D. 2000 _Vortex methods: theory and practice_. Cambridge University Press.
  * [4] Freidlin, M. 1985 _Functional integration and partial differential equations_. Ann. of Math. Studies 109, Princeton University Press.
  * [5] Goodman, J. 1987 Convergence of the random vortex method. _Comm. Pure Appl. Math._ $\mathbf{40}$(2), 189-220.
  * [6] Long, D. G. 1988 Convergence of the random vortex method in two dimensions. _J. of Amer. Math. Soc_. $\mathbf{1}$(4 ), 779-804.
  * [7] Majda, A. J. and Bertozzi A. L. 2002 _Vorticity and incompressible flow_. Cambridge University Press.
  * [8] Moin, P. and Mahesh, K. 1998 Direct numerical simulation: a tool in turbulence research. _Annu. Rev. Fluid Mech._ $\mathbf{30}$, 539–78
  * [9] Orszag S.A. and Patterson G.S. 1972 Numerical simulation of three-dimensional homogeneous isotropic turbulence. _Phys. Rev. Lett_. $\mathbf{28}$, 76–79.
  * [10] Prandtl, L. 1904 Über Flüssigkeitsbewegung bei sehr kleiner Reibung. _Proc. Third Intern. Math. Congress_ , Heidelberg, 848-491.
  * [11] Qian Z, Süli E, Zhang Y. 2022 Random vortex dynamics via functional stochastic differential equations. _Proc. R. Soc_. $\mathbf{A}$ 478: 20220030. https://doi.org/10.1098/rspa.2022.0030 _._
  * [12] Qian Z., and Yao Y. 2022 _McKean–Vlasov type stochastic differential equations arising from the random vortex method._ Partial Differential Equations and Applications. (2022) 3:7 https://doi.org/10.1007/s42985-021-00146-z
  * [13] Spalart, P. R. 1988 Direct simulation of a turbulent boundary layer up to $R_{\theta}=1410$. _J. Fluid Mech_. $\mathbf{187}$, 61–98.
  * [14] Stroock, D. W. and Varadhan, S.R.S. 1979 _Multidimensional diffusion processes_. Springer.
  * [15] Veretennikov, A. J. 1981 _On strong solutions and explicit formulas for solutions of stochastic integral equations._ Math. USSR-Sbornik 39(3), 387.


# Learning Signal-Agnostic Manifolds of Neural Fields

Yilun Du1
<EMAIL_ADDRESS>
&Katherine Collins1,2
<EMAIL_ADDRESS>
&Joshua B. Tenenbaum1,2,3
<EMAIL_ADDRESS>
&Vincent Sitzmann1
<EMAIL_ADDRESS>

###### Abstract

Deep neural networks have been used widely to learn the latent structure of
datasets, across modalities such as images, shapes, and audio signals.
However, existing models are generally modality-dependent, requiring custom
architectures and objectives to process different classes of signals. We
leverage neural fields to capture the underlying structure in image, shape,
audio and cross-modal audiovisual domains in a modality-independent manner. We
cast our task as one of learning a manifold, where we aim to infer a low-
dimensional, locally linear subspace in which our data resides. By enforcing
coverage of the manifold, local linearity, and local isometry, our model —
dubbed GEM — learns to capture the underlying structure of datasets across
modalities. We can then travel along linear regions of our manifold to obtain
perceptually consistent interpolations between samples, and can further use
GEM to recover points on our manifold and glean not only diverse completions
of input images, but cross-modal hallucinations of audio or image signals.
Finally, we show that by walking across the underlying manifold of GEM, we may
generate new samples in our signal domains111Code and additional results are
available at https://yilundu.github.io/gem/..

11footnotemark: 1

MIT CSAIL 22footnotemark: 2MIT BCS 33footnotemark: 3 MIT CBMM

https://yilundu.github.io/gem/

##  1 Introduction

Every moment, we receive and perceive high-dimensional signals from the world
around us. These signals are in constant flux; yet remarkably, our perception
system is largely invariant to these changes, allowing us to efficiently infer
the presence of coherent objects and entities across time. One hypothesis for
how we achieve this invariance is that we infer the underlying manifold in
which perceptual inputs lie [1], naturally enabling us to link high-
dimensional perceptual changes with local movements along such a manifold. In
this paper, we study how we may learn and discover a low-dimensional manifold
in a signal-agnostic manner, over arbitrary perceptual inputs.

Manifolds are characterized by three core properties [2, 3]. First, a manifold
should exhibit data coverage, i.e., all instances and variations of a signal
are explained in the underlying low-dimensional space. Second, a manifold
should be locally metric, enabling perceptual manipulation of a signal by
moving around the surrounding low-dimensional space. Finally, the underlying
structure of a manifold should be globally-consistent; e.g. similar signals
should be embedded close to one another.

Existing approaches to learning generative models, such as GANs [4], can be
viewed as instances of manifold learning. However, such approaches have two
key limitations. First, low-dimensional latent codes learned by generative
models do not satisfy all desired properties for a manifold; while the
underlying latent space of a GAN enables us to perceptually manipulate a
signal, GANs suffer from mode collapse, and the underlying latent space does
not cover the entire data distribution. Second, existing generative
architectures are biased towards particular signal modalities, requiring
custom architectures and losses depending on the domain upon which they are
applied – thereby preventing us from discovering a manifold across arbitrary
perceptual inputs in a signal-agnostic manner.

As an example, while existing generative models of images are regularly based
on convolutional neural networks, the same architecture in 1D does not readily
afford high-quality modeling of audio signals. Rather, generative models for
different domains require significant architecture adaptations and tuning.
Existing generative models are further constrained by the common assumption
that training data lie on a regular grid, such as grids of pixels for images,
or grids of amplitudes for audio signals. As a result, they require uniformly
sampled data, precluding them from adequately modeling irregularly sampled
data, like point clouds. Such a challenge is especially prominent in cross-
modal generative modeling, where a system must jointly learn a generative
model over multiple signal sources (e.g., over images and associated audio
snippets). While this task is trivial for humans – we can readily recall not
only an image of an instrument, but also a notion of its timbre and volume –
existing machine learning models struggle to jointly fit such signals without
customization.

Figure 1: GEM learns a low-dimensional latent manifold over signals. Given a
cross-modal signal, latents in GEM are mapped, using a hypernetwork $\psi$,
into neural networks $\phi_{1}$ and $\phi_{2}$. $\phi_{1}$ represents a image
by mapping each pixel position $(x,y)$ to its associated color $c(x,y)$.
$\phi_{2}$ represents an audio spectrogram by mapping each pixel position
$(u,v)$ to its intensity $w(u,v)$. This enables GEM to be applied in a domain
agnostic manner across separate (multi-modal) signals, by utilizing a separate
function $\phi$ for each mode of a signal.

To obtain signal-agnostic learning of manifolds over signals, we propose to
model distributions of signals in the function space of neural fields, which
are capable of parameterizing image, audio, shape, and audiovisual signals in
a modality-independent manner. We then utilize hypernetworks [5], to regress
individual neural fields from an underlying latent space to represent a signal
distribution. To further ensure that our distribution over signals corresponds
to a manifold, we formulate our learning objective with explicit losses to
encourage the three desired properties for a manifold: data coverage, local
linearity, and global consistency. The resulting model, which we dub
GEM222short for GEnerative Manifold learning, enables us to capture the
manifold of a variety of signals – ranging from audio to images and 3D shapes
– with almost no architectural modification, as illustrated in Figure 1. We
further demonstrate that our approach reliably recovers distributions over
cross-modal signals, such as images with a correlated audio snippet, while
also eliciting sample diversity.

We contribute the following: first, we present GEM, which we show can learn
manifolds over images, 3D shapes, audio, and cross-modal audiovisual signals
in a signal-agnostic manner. Second, we demonstrate that our model recovers
the global structure of each signal domain, permitting easy interpolation
between nearby signals, as well as completion of partial inputs. Finally, we
show that walking along our learned manifold enables us to generate new
samples for each modality.

##  2 Related Work

Manifold Learning. Manifold learning is a large and well-studied topic [6, 2,
3, 7, 8, 9, 10] which seeks to obtain the underlying non-linear, low-
dimensional subspace in which naturally high-dimensional input signals lie.
Many early works in manifold learning utilize a nearest neighbor graph to
obtain such a low-dimensional subspace. For instance, Tenenbaum et al. [2]
employs the geodesic distance between nearby points, while Roweis et al. [3]
use locally linear subregions around each subspace. Subsequent work has
explored additional objectives [7, 8, 9, 10] to obtain underlying non-linear,
low-dimensional subspaces. Recently, [11, 12, 13] propose to combine
traditional manifold learning algorithms with deep networks by training
autoencoders to explicitly encourage latents to match embeddings obtained from
classical manifold learning methods. Further, [14] propose to utilize a heat
kernel diffusion process to uncover the underlying manifold of data. In this
work, we show how we may combine insights from earlier works in manifold
learning with modern deep learning techniques to learn continuous manifolds of
data in high-dimensional signal spaces. Drawing on these two traditions
enables our model to smoothly interpolate between samples, restore partial
input signals, and further generate new data samples in high-dimensional
signal space.

Learning Distributions of Neural Fields. Recent work has demonstrated the
potential of treating fully connected networks as continuous, memory-efficient
implicit (or coordinate-based) representations for shape parts [15, 16],
objects [17, 18, 19, 20], or scenes [21, 22, 23, 24]. Sitzmann et al. [25]
showed that these coordinate-based representations may be leveraged for
modeling a wide array of signals, such as audio, video, images, and solutions
to partial differential equations. These representations may be conditioned on
auxiliary input via conditioning by concatenation [17, 26], hypernetworks
[21], gradient-based meta-learning [27], or activation modulation [28].
Subsequently, they may be embedded in a generative adversarial [29, 28] or
auto-encoder-based framework [21]. However, both of these approaches require
modality-dependent architecture choices. Specifically, modification is
necessary in the encoder of auto-encoding frameworks, or the discriminator in
adversarial frameworks. In contrast, we propose to represent the problem of
learning a distribution as one of learning the underlying manifold in which
the signals lie, and show that this enables signal-agnostic modeling.
Concurrent to our work, Dupont et al. [30] propose a signal-agnostic
discriminator based on pointwise convolutions, which enables discrimination of
signals that do not lie on regular grids. In contrast, we do not take an
adversarial approach, thereby avoiding the need for a convolution-based
discriminator. We further show that our learned manifold better captures the
underlying data manifold and demonstrate that our approach is able to capture
manifolds across cross-modal distributions, which we find destabilizes
training of [30].

Generative Modeling. Our work is also related to existing work in generative
modeling. GANs [4] are a popular framework used to model modalities such as
images [31] or audio [32], but often suffer from training instability and mode
collapse. VAEs [33], another class of generative models, utilize an encoder
and decoder to learn a shared latent space for data samples. Recently
autoregressive models have also been used for image [34], audio [35], and
multi-modal text and image generation [36]. In contrast to previous generative
approaches, which require custom architectures for encoding and decoding, our
approach is a modality-independent method for generating samples distribution
and presents a new view of generative modeling: one of learning an underlying
manifold of data.

##  3 Parameterizing Spaces of Signals with Neural Fields

An arbitrary signal $\mathcal{I}$ in $\mathbb{R}^{D_{1}\times\cdots\times
D_{k}}$ can be represented efficiently as a function $\Phi$ which maps each
coordinate ${\bm{x}}\in\mathbb{R}^{k}$ to the value ${\bm{v}}$ of the feature
coordinate at that dimension.

$\Phi:\mathbb{R}^{k}\rightarrow\mathbb{R},\quad{\bm{x}}\rightarrow\Phi({\bm{x}})={\bm{v}}$
(1)

Such a formulation enables us to represent any signal, such as an image,
shape, or waveform – or a set of signals, like images with waveforms – as a
continuous function $\Phi$. We use a three-layer multilayer perceptron (MLP)
with hidden dimension 512 as our $\Phi$. Only the input dimension of $\Phi$ is
varied, depending on the dimensionality of the underlying signal – all other
architectural choices are the same across signal types. We refer to $\Phi$ as
a neural field.

Parameterizing Spaces over Signals. A set of signals may be parameterized as a
subspace $\mathcal{F}$ of functions $\Phi_{i}$. To efficiently parameterize
functions $\Phi$, we represent each $\Phi$ using a low-dimensional latent
space in $\mathbb{R}^{n}$ and map latents to $\mathcal{F}$ via a hypernetwork
$\Psi$:

$\Psi:\mathbb{R}^{n}\rightarrow\mathcal{F},\quad{\bm{z}}\rightarrow\Psi({\bm{z}})=\Phi,$
(2)

To regress $\Phi$ with our hypernetwork, we predict individual weights
$\bm{W}^{\mathcal{L}}$ and biases $\bm{b}^{\mathcal{L}}$ for each layer
$\mathcal{L}$ in $\Phi$. Across all experiments, our hypernetwork is
parameterized as a three-layer MLP (similar to our $\Phi$), where all weights
and biases of the signal representation $\Phi$ are predicted in the final
layer. Directly predicting weights $\bm{W}^{\mathcal{L}}$ is a high-
dimensional regression problem, so following [37], we parameterize $\bm{W}$
via a low-rank decomposition as
$\bm{W}^{\mathcal{L}}=\bm{W}^{\mathcal{L}}_{s}\odot\sigma(\bm{W}^{\mathcal{L}}_{h})$,
where $\sigma$ is the sigmoid function. $\bm{W}^{\mathcal{L}}_{s}$ is shared
across all predicted $\Phi$, while
$\bm{W}^{\mathcal{L}}_{h}=A^{\mathcal{L}}\times B^{\mathcal{L}}$ is
represented as two low-rank matrices $A^{\mathcal{L}}$ and $B^{\mathcal{L}}$ –
and is regressed separately for each individual latent ${\bm{z}}$.

##  4 Learning Structured Manifold of Signals

Figure 2: GEM learns to embed an arbitrary signal $\mathcal{I}$ in a latent
manifold via three losses: (a) a reconstruction loss which encourages neural
network $\phi$, decoded from ${\bm{z}}_{1}$ through a hypernetwork, to match
training signal $\mathcal{I}_{1}$ at each coordinate ${\bm{x}}$, (b) a local
isometry loss to enforce perceptual consistency, which encourages distance
between nearby latents ${\bm{z}}_{1}$ and ${\bm{z}}_{2}$ to be porportional to
their distance in signal space $\mathcal{I}_{1}$, $\mathcal{I}_{2}$, (c) a LLE
loss which encourages a latent ${\bm{z}}_{1}$ to be represented as a convex
combination $\alpha_{i}$ of its neighbors.

Given a training set $\mathcal{C}=\\{\mathcal{I}_{i}\\}_{i=1}^{N}$, of $N$
distinct arbitrary signals
$\mathcal{I}_{i}\in\mathbb{R}^{D_{1}\times\cdots\times D_{k}}$, our goal is to
learn – in a signal-agnostic manner – the underlying low-dimensional manifold
$\mathcal{M}\subset\mathbb{R}^{D_{1}\times\cdots\times D_{k}}$. In particular,
a manifold $\mathcal{M}$ [3, 2], over signals $\mathcal{I}$, is a locally low-
dimensional subspace consisting and describing all possible variations of
$\mathcal{I}$. A manifold has three crucial properties. First, it must embed
all possible instances of $\mathcal{I}$. Second, $\mathcal{M}$ should be
locally metric around each signal $\mathcal{I}$, enabling us to effectively
navigate and interpolate between nearby signals. Finally, it must be locally
perceptually consistent; nearby points on $\mathcal{M}$ should be similar in
identity. To learn a manifold $\mathcal{M}$ with these properties, we
introduce a loss for each property: $\mathcal{L}_{\text{Rec}}$,
$\mathcal{L}_{\text{LLE}}$, and $\mathcal{L}_{\text{Iso}}$, respectively.
$\mathcal{L}_{\text{Rec}}$ ensures that all signals $\mathcal{I}$ are embedded
inside $\mathcal{M}$ (Section 4.1), $\mathcal{L}_{\text{LLE}}$ forces the
space near each $\mathcal{I}$ to be locally metric (Section 4.2), and
$\mathcal{L}_{\text{Iso}}$ encourages the underlying manifold to be
perceptually consistent (Section 4.3). As such, our training loss takes the
form:

$\mathcal{L}_{\text{Total}}=\mathcal{L}_{\text{Rec}}+\mathcal{L}_{\text{LLE}}+\mathcal{L}_{\text{Iso}}$
(3)

Please see Fig. 2 for an overview of the proposed manifold learning approach.
We utilize equal weighting across each introduced loss term. We discuss each
component of the loss in more detail below.

###  4.1 Data Coverage via Auto-Decoding

Given a parameterization of a subspace $\mathcal{F}$ (as introduced in Section
3) which represents our manifold $\mathcal{M}$, we must ensure that $\Psi$,
our hypernetwork, can effectively cover the entire subspace in which our
signal $\mathcal{I}$ lies. A common approach to learn subspaces over signals
is to employ a GAN [4], whereby a discriminator trains a hypernetwork
$\Psi({\bm{z}})$ to generate signals that are indistinguishable from real
examples. However, an issue with such an approach is that many $\mathcal{I}$
may be missing in the resultant mapping.

Here, to ensure that our manifold covers all instances, we utilize the auto-
decoder framework [17, 21]. We learn a separate latent ${\bm{z}}_{i}$ for each
training signal $\mathcal{I}_{i}$ and train models utilizing the loss

$\mathcal{L}_{\text{Rec}}=\|\Psi({\bm{z}}_{i})({\bm{x}})-\mathcal{I}_{i}({\bm{x}})\|^{2}$
(4)

over each possible input coordinate ${\bm{x}}$. By explicitly learning to
reconstruct each training point in our latent space, we enforce that our
latent space, and thus our manifold $\mathcal{M}$, covers all training signals
$\mathcal{I}$.

###  4.2 Local Metric Consistency via Linear Decomposition

A manifold $\mathcal{M}$ is locally metric around a signal $\mathcal{I}$ if
there exist local linear directions of variation on the manifold along which
the underlying signals vary in a perceptually coherent way. Inspired by [3],
we enforce this constraint by encouraging our manifold to consist of a set of
local convex linear regions $\mathcal{R}_{i}$ in our learned latent space.

We construct these convex linear regions $\mathcal{R}_{i}$ using autodecoded
latents ${\bm{z}}_{i}$ (Section 4.1). Each latent ${\bm{z}}_{i}$ defines a
convex region, $\mathcal{R}_{i}$ – obtained by combining nearest latent
neighbors ${\bm{z}}_{i}^{1},\ldots,{\bm{z}}_{i}^{j}$, as illustrated in Figure
2.

To ensure that our manifold $\mathcal{M}$ consists of these convex local
regions, during training, we enforce that each latent ${\bm{z}}_{i}$ can be
represented in a local convex region, $\mathcal{R}_{i}$, e.g. that it can be
expressed as a set of weights ${\bm{w}}_{i}$, such that
${\bm{z}}_{i}=\sum_{j}w_{i}^{j}{\bm{z}}_{i}^{j}$ and $\sum_{j}w_{i}^{j}=1$.
Given a ${\bm{z}}_{i}$ we may solve for a given ${\bm{w}}_{i}$ by minimizing
the objective
$\|{\bm{z}}_{i}-\sum_{j}w_{i}^{j}{\bm{z}}_{i}^{j}\|^{2}={\bm{w}}_{i}^{T}\bm{G}_{i}{\bm{w}}_{i}$
with respect to ${\bm{w}}_{i}$. In the above expression, $\bm{G}_{i}$ is the
Gram matrix, and is defined as $\bm{G}_{i}={\bm{k}}{\bm{k}}^{T}$, where
${\bm{k}}$ is a block matrix, with row $j$ corresponding to
${\bm{k}}_{j}={\bm{z}}_{i}-{\bm{z}}_{i}^{j}$. Adding the constraint that all
weights add up to one via a Legrange multiplier, leads to the following
optimization objective:

$\mathcal{L}({\bm{w}}_{i},\lambda)={\bm{w}}_{i}^{T}\bm{G}_{i}{\bm{w}}_{i}+\lambda(\bm{1}^{T}{\bm{w}}_{i}-1)$
(5)

where $\bm{1}$ corresponds to a vector of all ones. Finding the optimal
${\bm{w}}_{i}$ in the above expression then corresponds to the best projection
of ${\bm{z}}_{i}$ on the linear region $\mathcal{R}_{i}$ it defines. Taking
gradients of the above expression, we find that
${\bm{w}}_{i}=\frac{\lambda}{2}\bm{G}_{i}^{-1}\bm{1}$, where $\lambda$ is set
to ensure that elements of ${\bm{w}}_{i}$ sum up to 1. The mapped latent
${\bm{z}}_{i}^{\prime}=\sum_{j}{\bm{w}}_{i}^{j}{\bm{z}}_{i}^{j}$ corresponds
to the best projection ${\bm{z}}_{i}$ on $\mathcal{R}_{i}$. To enforce that
${\bm{z}}_{i}$ lies in $\mathcal{R}_{i}$, we enforce that this projection

$\mathcal{L}_{\text{LLE}}=\|\Psi({\bm{z}}_{i}^{\prime})({\bm{x}})-\mathcal{I}_{i}({\bm{x}})\|^{2},$
(6)

also decodes to the training signal $\mathcal{I}_{i}$, where we differentiate
through intermediate matrix operations. To encourage ${\bm{w}}_{i}$ to be
positive, we incorporate an additional L1 penalty to negative weights.

###  4.3 Perceptual Consistency through Local Isometry

Finally, a manifold $\mathcal{M}$ should be perceptually consistent. In other
words, two signals that are perceptually similar should also be closer in the
underlying manifold, according to a distance metric on the respective latents,
than signals that are perceptually distinct. While in general the MSE distance
between two signals is not informative, when such signals are relatively
similar to one another, their MSE distance does contain useful information.

Therefore, we enforce that the distance between latents ${\bm{z}}_{i}$ and
${\bm{z}}_{j}$ in our manifold $\mathcal{M}$ are locally isometric to the MSE
distance of the corresponding samples $\mathcal{I}_{i}$ and $\mathcal{I}_{j}$,
when relative pairwise distance of samples is small. On the lowest 25% of
pairwise distances, we utilize the following loss:

$\mathcal{L}_{\text{Iso}}=\|\alpha*\|({\bm{z}}_{i}-{\bm{z}}_{j})\|-\|\mathcal{I}_{i}-\mathcal{I}_{j}\|\|.$
(7)

Such local isometry may alternatively be enforced by regularizing the
underlying Jacobian map.

###  4.4 Applications

By learning a modality-independent manifold $\mathcal{M}$, GEM permits the
following signal-agnostic downstream applications, each of which we explore in
this work.

Interpolation. By directly interpolating between the learned latents
${\bm{z}}_{i}$ and ${\bm{z}}_{j}$ representing each signal, we can interpolate
in a perceptually consistent manner between neighboring signals in
$\mathcal{M}$. Further, latents for novel signals may be obtained by
optimizing $\mathcal{L}_{\text{Rec}}$ on the signal.

Conditional Completion. Given a partial signal $\hat{\mathcal{I}}$, we can
obtain a point on the manifold $\mathcal{M}$ by optimizing a latent ${\bm{z}}$
using $\mathcal{L}_{\text{Rec}}+\mathcal{L}_{\text{LLE}}$. Decoding the
corresponding latent ${\bm{z}}$ then allows us to complete the missing
portions of the image.

Sample Generation. Our manifold $\mathcal{M}$ consists of a set of locally
linear regions $\mathcal{R}_{i}$ in a underlying latent space ${\bm{z}}$. As
such, new samples in $\mathcal{M}$ can be generated by walking along linear
regions $\mathcal{R}_{i}$, and sampling points each such region. To sample a
random point within $\mathcal{R}_{i}$, we generate $\alpha$ between 0 and 1,
and compute a latent
$\hat{{\bm{z}}_{i}}=\alpha{\bm{z}}_{i}+(1-\alpha)*{\bm{z}}_{i}^{j}+\beta*\mathcal{N}(0,1)$
(corresponding to a random sample in the neighborhood of ${\bm{z}}_{i}$) and
project it onto the underlying region $\mathcal{R}_{i}$.

##  5 Learning Manifolds of Data

Modality | Model |  MSE $\downarrow$ |  PSNR $\uparrow$
---|---|---|---
Images | Heat Kernel | 0.0365 | 14.99
VAE | 0.0331 | 15.12
FDN | 0.0060 | 22.59
StyleGAN2 | 0.0044 | 24.03
| GEM | 0.0025 | 26.53
Audio | VAE | 0.0147 | 18.87
FDN | 0.0050 | 23.68
StyleGAN2 | 0.0015 | 28.52
GEM | 0.0011 | 29.98
ShapeNet | FDN | 0.0296 | 16.52
GEM | 0.0153 | 21.32
Image and Audio | VAE | 0.0193 | 17.23
FDN | 0.0663 | 11.78
StyleGAN2 | 0.0063 | 22.36
GEM | 0.0034 | 24.38

Figure 3: Test-time reconstruction. Comparison of GEM against baselines on the
task of fitting test signals over a diverse set of signal modalities: images,
3D shapes, and audio-visual, respectively. GEM (center) achieves significantly
better reconstruction performance – qualitatively and quantitatively – on test
signals than StyleGAN2 or FDN, indicating better coverage of the data manifold
across all modalities. Results are run across one seed, but we report results
across three seeds in the appendix, and find limited overall variance.

We validate the generality of GEM by first showing that our model is capable
of fitting diverse signal modalities. Next, we demonstrate that our approach
captures the underlying structure across these signals; we are not only able
to cluster and perceptually interpolate between signals, but inpaint to
complete partial ones. Finally, we show that we can draw samples from the
learned manifold of each signal type, illustrating the power of GEM to be used
a signal-agnostic generative model. We re-iterate that nearly identical
architectures and training losses are used across separate modalities.

Datasets. We evaluate GEM on four signal modalities: image, audio, 3D shape,
and cross-modal image and audio signals, respectively. For the image modality,
we investigate performance on the CelebA-HQ dataset [38] fit on 29000
$64\times 64$ training celebrity images, and test on 1000 $64\times 64$ test
images. To study GEM behavior on audio signals, we use the NSynth dataset
[39], and fit on a training set of 10000 one-second 16kHz sounds clips of
different instruments playing, and test of 5000 one-second 16kHz sound clips.
We process sound clips into spectrograms following [32]. For the 3D shape
domain, we work with the ShapeNet dataset from [40]. We train on 35019
training shapes at $64\times 64\times 64$ resolution and test on 8762 shapes
at $64\times 64\times 64$ resolution. Finally, for the cross-modal image and
audio modality, we utilize the cello image and audio recordings from the Sub-
URMP dataset [41]. We train on 9800 images at $128\times 128$ resolution and
0.5 second 16kHz audio clips, which are also processed into spectrograms
following [32] and test on 1080 images and associated audio clips. We provide
additional cross-modal results in the supplement.

Setup and Baselines. We benchmark GEM against existing manifold learning
approaches. Namely, we compare our approach with that of StyleGAN2 [31], VAE
[42], and concurrent work FDN of [30], as well as heat kernel manifold
learning [14]. We utilize the authors’ provided codebase for StyleGAN2, the
PytorchVAE library 333https://github.com/AntixK/PyTorch-VAE for the VAE, and
the original codebase of FDN [30], which the authors graciously provided. We
train all approaches with a latent dimension of 1024, and re-scale the size of
the VAE to ensure parameter counts are similar. We report model architecture
details in the appendix. We need to change the architecture of both the VAE
and StyleGAN2 baselines, respectively, in order to preserve proper output
dimensions, depending on signal modality. However, we note that identical
architectures readily can and are used for GEM. For fitting cross-modal
audiovisual signals, all models utilize two separate decoders, with each
decoder in GEM identical to each other. Architectural details are provided in
the appendix.

Figure 4: Test-time audio reconstruction. GEM (center) achieves better
spectrogram recovery than FDN (left), even recovering fine details in the
original signals.

Modality | $\mathcal{L}_{\text{LLE}}$ | $\mathcal{L}_{\text{Iso}}$ |  MSE $\downarrow$ |  PSNR $\uparrow$
---|---|---|---|---
Image | No | No | 0.0028 | 26.11
Yes | No | 0.0025 | 26.53
Yes | Yes | 0.0024 | 26.69
Audio | No | No | 0.0020 | 27.63
No | Yes | 0.0014 | 29.02
Yes | Yes | 0.0011 | 29.98
Shapes | No | No | 0.0422 | 16.23
Yes | No | 0.0323 | 17.43
Yes | Yes | 0.0153 | 21.32
Image and | No | No | 0.0070 | 21.72
Audio | Yes | No | 0.0058 | 22.57
Yes | Yes | 0.0034 | 24.38

Table 1: Ablation. Impact of $\mathcal{L}_{\text{Iso}}$ and
$\mathcal{L}_{\text{LLE}}$ on test signal reconstruction performance. Ablating
each loss demonstrates that both components enable better fitting of the
underlying test distribution.

###  5.1 Covering the Data Manifold

We first address whether our model is capable of covering the underlying data
distribution of each modality, compared against our baselines, and study the
impact of our losses on this ability.

Modality Fitting. We quantitatively measure reconstruction performance on test
signals in Figure 3 and find the GEM outperforms each of our baselines. In the
cross-modal domain in particular, we find that FDN training destabilizes
quickly and struggles to adequately fit examples. We illustrate test signal
reconstructions in Figure 3 and Figure 4. Across each modality, we find that
GEM obtains the sharpest samples. In Figure 3 we find that while StyleGAN2 can
reconstruct certain CelebA-HQ images well, it fails badly on others,
indicating a lack of data coverage. Our results indicate that our approach
best captures the underlying data distribution.

Figure 5: Nearest Neighbor Interpolation. Interpolation between nearest
neighbors in GEM, in the images and 3D shape domains. Our model is able to
make perceptually continuous interpolations.

Figure 6: t-SNE Structure. t-SNE plot of NSynth audio clip manifold embeddings
colored by corresponding pitch. Pitch is seperated by t-SNE.

Ablations. In Table 1, we assess the impact of two of our proposed losses –
$\mathcal{L}_{\text{LLE}}$ and $\mathcal{L}_{\text{Iso}}$ – at enabling test-
signal reconstruction, and find to both help improve the resultant test
construction. We posit that both losses serve to regularize the underlying
learned data manifold, enabling GEM to more effectively cover the overall
signal manifold.

Applications to Other Models. While we apply $\mathcal{L}_{\text{Rec}}$ ,
$\mathcal{L}_{\text{LLE}}$, $\mathcal{L}_{\text{Iso}}$ to GEM, our overall
losses may be applied more generally across different models, provided the
model maps an underlying latent space to an output domain. We further assess
whether these losses improve the manifold captured by StyleGAN2 via measuring
reconstruction performance on test CelebA-HQ images. We find that while a base
StyleGAN2 model obtains a reconstruction error of MSE 0.0044 and PSNR 24.03,
the addition of $\mathcal{L}_{\text{Rec}}$ improves test reconstruction to MSE
0.0041 and PSNR 24.29 with $\mathcal{L}_{\text{LLE}}$ losses further improving
test reconstruction to MSE 0.0038 and PSNR 24.61. In contrast, we found that
$\mathcal{L}_{\text{Iso}}$ leads to a slight regression in performance. We
posit that both $\mathcal{L}_{\text{Rec}}$ and $\mathcal{L}_{\text{LLE}}$
serve to structure the underlying StyleGAN2 latent space, while the underlying
Gaussian nature of the StyleGAN2 latent space precludes the need for
$\mathcal{L}_{\text{Iso}}$. Our results indicate the generality of our
proposed losses towards improving the recovery of data manifolds.

###  5.2 Learning the Structure of Signals

Next, we explore the extent to which the manifold learned by GEM captures the
underlying global and local structure inherent to each signal modality.
Additionally, we probe our model’s understanding of individual signals by
studying GEM’s ability to reconstruct partial signals.

Global Structure. We address whether GEM captures the global structure of a
signal distribution by visualizing the latents learned by GEM on the NSynth
dataset with t-SNE [43]. We find in Figure 6 that the underlying pitch of a
signal maps onto the resulting t-SNE. We further visualize the inferred
connectivity structure of our manifold in Figure 7 (left) and visualize the
nearest neighbors in latent space of the autodecoded latents (right), finding
that nearby latents correspond to semantically similar faces – further
supporting that GEM learns global structure. Note, we find that without either
$\mathcal{L}_{\text{LLE}}$ or $\mathcal{L}_{\text{Iso}}$, the underlying
connectivity of our manifold is significantly poorer.

Figure 7: Diverse Image Inpainting (Left). GEM generates multiple possible
completions of a partial image. Across completions, skin and lip hues are
completed correctly and eyeglasses are reconstructed consistently. Nearest
Neighbors (Right). Nearest neighbors in the manifold of GEM with and without
regularization. With regularization, neighbors in latent space correspond to
perceptually similar images. Figure 8: Audio Hallucinations (Left). GEM
generates multiple possible audio spectrogram completions of an input image.
Generations are cross-modal. For instance, conditioned on an image of a lifted
bow, empty spectrograms are generated; likewise, higher frequency spectrograms
are generated when the bow is on high strings, and lower frequency ones on
lower strings. Image Hallucinations (Right). Given an input spectrogram, GEM
generates multiple possible corresponding image completions, again
highlighting the cross-modality of the manifold. When the input spectrogram is
empty, GEM generates images having the bow off the cello, and depending on the
frequency composition of the spectrogram, the bow position is sampled along
different strings.

Local Structure. We next probe whether GEM learns a densely connected
manifold, e.g. one which allows us to interpolate between separate signals,
suggestive of capturing the local manifold structure. In Figure 6, we
visualize nearest neighbor interpolations in our manifold and observe that the
combination of $\mathcal{L}_{\text{LLE}}$ and $\mathcal{L}_{\text{Iso}}$
enables us to obtain perceptually smooth interpolations over both image and
shape samples. We provide additional interpolations and failure cases in the
supplement.

Signal Completion. To investigate the ability of GEM to understand individual
signals, we assess the recovery of a full signal when only a subset of such a
signal is given. In Figure 7, we consider the case of inpainting CelebA-HQ
faces, where a subset of the face is missing. We observe that GEM is able to
obtain several different perceptually consistent inpaintings of a face, with
individual completions exhibiting consistent skin and lip colors, and
restoring structural features such as eyeglasses. We provide additional
examples and failure cases in the supplement.

Additionally, we consider signal completion in the cross-modal audiovisual
domain. Here, we provide only an input image and ask GEM to generate possible
audio spectograms. As seen in Figure 8, we observe that GEM is able to
generate empty spectograms when the bow is not on the cello, as well as
different spectrograms dependent on the position of the bow. Alternatively, in
Figure 8, we provide only an input audio spectrogram and ask GEM to generate
possible images. In this setting, we find that generated images have the bow
off the cello when the input spectrogram is empty (with the bow missing due to
GEM blurring the uncertainty of all possible locations of the bow), and at
different positions depending on the input audio spectrogram. In contrast, we
found that all baselines were unable to fit the audiovisual setting well, as
discussed in Section 5.1 and shown in Figure 11.

###  5.3 Generating Data Samples

Model |  FID $\downarrow$ |  Precision $\uparrow$ |  Recall $\uparrow$
---|---|---|---
CelebA-HQ 64x64 |  |  |
VAE | 175.33 | 0.799 | 0.001
StyleGAN V2 | 5.90 | 0.618 | 0.481
FDN [30] | 13.46 | 0.577 | 0.397
GEM | 30.42 | 0.642 | 0.502
ShapeNet |  Coverage $\uparrow$ |  MMD $\downarrow$ |
Latent GAN [40] | 0.389 | 0.0017 |
FDN [30] | 0.341 | 0.0021 |
GEM (Ours) | 0.409 | 0.0014 |

Table 2: Generation Performance. Performance of GEM and baselines on signal
generation.

Figure 9: Audio Generations. GEM (right) generates sharp audio samples
compared to FDN and StyleGAN2.

Finally, we investigate the ability of GEM to generate new data samples.

Figure 10: Image Generations (Top). GEM (right) generates comparable images to
FDN. Nearest Neighbors (Bottom). Illustration of nearest neighbors in latent
space of generations.

Figure 11: Generations. GEM (right) produces reasonable samples across the
shape and audiovisual modalities. In contrast, audiovisual generations from
StyleGAN2 exhibit noise, while FDN generates poor samples in both modalities.

Qualitative Generations. We show random samples drawn from GEM and compare
these to our baselines. We consider images in Figure 11, shapes in Figure 11,
audio snippets in Figure 9 and audiovisual inputs in Figure 11. While we find
that GEM performs comparably to FDN in the image regime, our model
significantly outperforms all baselines on domains of audio, shape, and
audiovisual modalities. We further display nearest latent space neighbors of
generations on CelebA-HQ in Figure 11 and find that our generations are
distinct from those in the training dataset.

Quantitative Comparisons. Next, we provide quantitiative evaluations of
generations in Table 2 on image and shape modalities. We report the FID [44],
precision, and recall [45] metrics on CelebA-HQ $64\times 64$. We find that
GEM performs better than StyleGAN2 and FDN on precision and recall, but worse
in terms of FID. We note that our qualitative generations in Figure 11 are
comparable to those of FDN; however, we find that our approach obtains high
FID scores due to the inherent sensitivity of the FID metric to blur. Such
sensitivity to bluriness has also been noted in [46], and we find that even
our autodecoded training distribution obtains an FID of 23.25, despite images
appearing near perceptually perfect (Figure 3). On ShapeNet, we report
coverage and MMD metrics from [47] using Chamfer Distance. To evaluate
generations, we sample 8762 shapes (the size of test Shapenet dataset) and
generate 2048 points following [40]. We compare generations from GEM with
those of FDN and latent GAN trained on IMNET (using the provided code in
[40]). We find that GEM outperforms both approaches on the task of 3D shape
generation.

##  6 Conclusion

We have presented GEM, an approach to learn a modality-independent manifold.
We demonstrate how our model enables us to interpolate between signals,
complete partial signals, and further generate new signals. A limitation of
our approach is that while underlying manifolds are recovered, they are not
captured with high-fidelity. We believe that a promising direction of future
work involves the pursuit of further engineered inductive biases, and general
structure, which enable the recovery of high-fidelity manifolds of the
plethora of signals in the world around us. We emphasize, however, that as our
manifold is constructed over a dataset; GEM may learn to incorporate and
propogate existing prejudices and biases present in such data, posing risks
for mass deployment of our model. Additionally, while our work offers exciting
avenues for cross-modal modeling and generation, we note that GEM has the
potential to be used to create enhanced "deep fakes" and other forms of
synthetic media.

#### Acknowledgements

This project is supported by DARPA under CW3031624 (Transfer, Augmentation and
Automatic Learning with Less Labels), the Singapore DSTA under
DST00OECI20300823 (New Representations for Vision), as well as ONR MURI under
N00014-18-1-2846. We would like to thank Bill Freeman and Fredo Durand for
giving helpful comments on the manuscript. Yilun Du is supported by an NSF
graduate research fellowship.

## References

  * [1] Sebastian H. Seung and Daniel Lee.  The manifold ways of perception.  Science, 290:2268–2269, 2000.
  * [2] Joshua B Tenenbaum, Vin De Silva, and John C Langford.  A global geometric framework for nonlinear dimensionality reduction.  Sci., 290(5500):2319–2323, 2000.
  * [3] Sam T Roweis and Lawrence K Saul.  Nonlinear dimensionality reduction by locally linear embedding.  Sci., 290(5500):2323–2326, 2000.
  * [4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [5] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  In Proc. ICLR, 2017.
  * [6] Bernhard Schölkopf, Alexander J. Smola, and Klaus-Robert Müller.  Nonlinear component analysis as a kernel eigenvalue problem.  Neural Comput., 10(5):1299–1319, 1998.
  * [7] Mikhail Belkin and Partha Niyogi.  Laplacian eigenmaps for dimensionality reduction and data representation.  Neural Comput., 15(6):1373–1396, June 2003.
  * [8] Ronald R. Coifman and Stéphane Lafon.  Diffusion maps.  Applied and Computational Harmonic Analysis, 21(1):5–30, 2006.  Special Issue: Diffusion Maps and Wavelets.
  * [9] David L. Donoho and Carrie Grimes.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data.  Proceedings of the National Academy of Sciences, 100(10):5591–5596, 2003.
  * [10] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of Machine Learning Research, 9:2579–2605, 2008.
  * [11] Kui Jia, Lin Sun, Shenghua Gao, Zhan Song, and Bertram E. Shi.  Laplacian auto-encoders: An explicit learning of nonlinear data manifold.  Neurocomputing, 160:250–260, 2015.
  * [12] Gal Mishne, Uri Shaham, Alexander Cloninger, and Israel Cohen.  Diffusion nets.  Applied and Computational Harmonic Analysis, 47(2):259–285, 2019\.
  * [13] Gautam Pai, Ronen Talmon, Alex Bronstein, and Ron Kimmel.  Dimal: Deep isometric manifold learning using sparse geodesic sampling, 2018.
  * [14] Yufan Zhou, Changyou Chen, and Jinhui Xu.  Learning manifold implicitly via explicit heat-kernel learning.  arXiv preprint arXiv:2010.01761, 2020.
  * [15] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser.  Learning shape templates with structured implicit functions.  In Proc. ICCV, pages 7154–7164, 2019.
  * [16] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Deep structured implicit functions.  arXiv preprint arXiv:1912.06126, 2019.
  * [17] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proc. CVPR, 2019.
  * [18] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Implicit surface representations as layers in neural networks.  In Proc. ICCV, pages 4743–4752, 2019.
  * [19] Matan Atzmon and Yaron Lipman.  Sal: Sign agnostic learning of shapes from raw data.  In Proc. CVPR, 2020.
  * [20] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  In Proc. ICML, 2020.
  * [21] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  In Proc. NeurIPS 2019, 2019.
  * [22] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser.  Local implicit grid representations for 3d scenes.  In Proc. CVPR, pages 6001–6010, 2020.
  * [23] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In Proc. ECCV, 2020.
  * [24] Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe.  Deep local shapes: Learning local sdf priors for detailed 3d reconstruction.  arXiv preprint arXiv:2003.10983, 2020.
  * [25] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions, 2020\.
  * [26] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proc. CVPR, 2019.
  * [27] Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein.  Metasdf: Meta-learning signed distance functions.  Proc. NeurIPS, 2020.
  * [28] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  Proc. CVPR, 2020.
  * [29] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  Proc. NeurIPS, 2020.
  * [30] Emilien Dupont, Yee Whye Teh, and Arnaud Doucet.  Generative models as distributions of functions.  arXiv preprint arXiv:2102.04776, 2021.
  * [31] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [32] Chris Donahue, Julian McAuley, and Miller Puckette.  Adversarial audio synthesis, 2019.
  * [33] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling.  Semi-supervised learning with deep generative models.  In NeurIPS, 2014.
  * [34] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  Proc. ICML, 2016.
  * [35] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio, 2016.
  * [36] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation, 2021.
  * [37] Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny.  Adversarial generation of continuous images.  arXiv preprint arXiv:2011.12026, 2020.
  * [38] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In ICLR, 2017.
  * [39] Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi.  Neural audio synthesis of musical notes with wavenet autoencoders, 2017\.
  * [40] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proc. CVPR, pages 5939–5948, 2019.
  * [41] Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu.  Deep cross-modal audio-visual generation, 2017.
  * [42] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [43] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  JMLR, 9(11):2579–2605, 2008.
  * [44] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Advances in Neural Information Processing Systems, pages 6626–6637, 2017.
  * [45] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Improved precision and recall metric for assessing generative models.  CoRR, abs/1904.06991, 2019.
  * [46] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2, 2019.
  * [47] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas J Guibas.  Learning representations and generative models for 3d point clouds.  arXiv preprint arXiv:1707.02392, 2017.
  * [48] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [49] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proc. ECCV, 2020.

##  Appendix A.1 Appendix for Learning Signal-Agnostic Manifolds of Neural
Fields

Please visit our project website at https://yilundu.github.io/gem/ for
additional qualitative visualizations of test-time reconstruction of audio and
audiovisual samples, traversals along the underlying manifold of GEM on
CelebA-HQ as well as interpolations between audio samples. We further
illustrate additional image in-painting results, as well as audio completion
results. Finally, we visualize several audio and audiovisual generations.

In Section A.1.1 below, we provide details on training settings, as well as
the underlying baseline model architectures utilized for each modality. We
conclude with details on reproducing our work in Section A.1.2.

###  A.1.1 Experimental Details

Training Details For each separate training modality, all models and baselines
are trained for one day, using one 32GB Volta machine. GEM is trained with the
Adam optimizer [48], using a training batch size of 128 and a learning rate of
1e-4. Each individual datapoint is fit by fitting the value of 1024 sampled
points in the sample (1024 for each modality in the multi-modal setting). We
normalize the values of a signals to be between -1 and 1. When computing
$\mathcal{L}_{\text{Iso}}$, a scalar constant of $\alpha=100$ is employed to
scale distances in the underlying manifold to that of distances of signals in
sample space. When enforcing $\mathcal{L}_{\text{LLE}}$, a total of 10
neighbors are considered to compute the loss across modalities. We utilize
equal loss weight across $\mathcal{L}_{\text{Rec}}$,
$\mathcal{L}_{\text{Iso}}$, $\mathcal{L}_{\text{LLE}}$, and found that the
relative magnitudes of each loss had little impact on the overall performance.

Model Details We provide the architectures of the hypernetwork $\psi$ and
implicit function $\phi$ utilized by GEM across separate modalities in Table 4
and Table 5, respectively. Additionally, we provide the architectures used in
each domain for our baselines: StyleGAN2 in Table 15 each domain in Table 15
and VAE in Table 10. Note that for the VAE, the ReLU nonlinearity is used,
with each separate convolution having stride 2.

We obtained the hyperparameters for implicit functions and hypernetworks based
off of [37]. Early in the implementation of the project, we explored a variety
of additional architectural choices; however, we ultimately found that neither
changing the number of layers in the hypernetworks, nor changing the number of
underlying hidden units in networks, significantly impacted the performance of
GEM. We will add these details to the appendix of the paper.

###  A.1.2 Reproducibility

We next describe details necessary to reproduce each of other underlying
empirical results.

Hyperparameter Settings for Baselines We employ the default hyperparameters,
as used in the original papers for StyleGAN2 [31] and FDN [30], to obtain
state-of-the-art performance on their respective tasks. Due to computational
constraints, we were unfortunately unable to do a complete hyperparameter
search for each method over all tasks considered. Despite this, we were able
to run the models on toy datasets and found that these default hyperparameters
performed the best. We utilized the author’s original codebases for
experiments.

Variance Across Seeds Results in the main tables of the paper are run across a
single evaluated seed. Below in Table 3, we rerun test reconstruction results
on CelebA-HQ across different models utilizing a total of 3 separate seeds. We
find minimal variance across separate runs, and still find the GEM performs
significantly outperforms baselines.

Modality | Model | MSE $\downarrow$ | PSNR $\uparrow$
---|---|---|---
Images | VAE | 0.0327 $\pm$ 0.0035 | 15.16 $\pm$ 0.06
FDN | 0.0062 $\pm$ 0.0003 | 22.57 $\pm$ 0.02
StyleGANv2 | 0.0044 $\pm$ 0.0001 | 24.03 $\pm$ 0.01
GEM |  0.0025 $\pm$ 0.0001 |  26.53 $\pm$ 0.01

Table 3: Test CelebA-HQ reconstruction results of different methods evaluated
across 3 different seeds. We further report standard deviation between
different runs.

Datasets We provide source locations to download each of the datasets we used
in the paper. The CelebA-HQ dataset can be downloaded at
https://github.com/tkarras/progressive_growing_of_gans/blob/master/dataset_tool.py
and is released under the Creative Commons license. The NSynth dataset may be
downloaded at https://magenta.tensorflow.org/datasets/nsynth and is released
under the Creative Commons license. The ShapeNet dataset can be downloaded at
https://github.com/czq142857/IM-NET and is released under the MIT License, and
finally the Sub-URMP dataset we used may be downloaded at
https://www.cs.rochester.edu/~cxu22/d/vagan/.

Dense $\rightarrow$ 512
---
Dense $\rightarrow$ 512
Dense $\rightarrow$ 512
Dense $\rightarrow$ $\phi$ Parameters

Table 4: The architecture of the hypernetwork utilized by GEM.

Pos Embed (512)
---
Dense $\rightarrow$ 512
Dense $\rightarrow$ 512
Dense $\rightarrow$ 512
Dense $\rightarrow$ Output Dim

Table 5: The architecture of the implicit function $\phi$ used to agnostically
encode each modality. We utilize the Fourier embedding from [49] to embed
coordinates.

3x3 Conv2d, 64
---
3x3 Conv2d, 128
3x3 Conv2d, 256
3x3 Conv2d, 512
3x3 Conv2d, 512
z $\leftarrow$ Encode
Reshape(2, 2)
3x3 Conv2d Transpose, 512
3x3 Conv2d Transpose, 512
3x3 Conv2d Transpose, 256
3x3 Conv2d Transpose, 128
3x3 Conv2d Transpose, 3

Table 6: The encoder and decoder of the VAE utilized for CelebA-HQ.

3x3 Conv2d, 32
---
3x3 Conv2d, 64
3x3 Conv2d, 128
3x3 Conv2d, 256
3x3 Conv2d, 512
z $\leftarrow$ Encode
Reshape(4, 2)
3x3 Conv2d Transpose, 512
3x3 Conv2d Transpose, 256
3x3 Conv2d Transpose, 128
3x3 Conv2d Transpose, 64
3x3 Conv2d Transpose, 32
3x3 Conv2d Transpose, 1
Crop

Table 7: The encoder and decoder of the VAE utilized for NSynth

3x3 Conv2d, 32
---
3x3 Conv2d, 64
3x3 Conv2d, 128
3x3 Conv2d, 256
3x3 Conv2d, 512
z $\leftarrow$ Encode
Reshape(2, 2)
3x3 Conv2d Transpose, 512
3x3 Conv2d Transpose, 256
3x3 Conv2d Transpose, 128
3x3 Conv2d Transpose, 64
3x3 Conv2d Transpose, 32
3x3 Conv2d Transpose, 3

Table 8: The architecture of encoder and decoder of the VAE utilized for
audiovisual dataset on images. Latent encodings from image and audio
modalities are added together.

3x3 Conv2d, 32
---
3x3 Conv2d, 64
3x3 Conv2d, 128
3x3 Conv2d, 256
3x3 Conv2d, 512
z $\leftarrow$ Encode
Reshape(4, 1)
3x3 Conv2d Transpose, 512
3x3 Conv2d Transpose, 256
3x3 Conv2d Transpose, 128
3x3 Conv2d Transpose, 64
3x3 Conv2d Transpose, 32
3x3 Conv2d Transpose, 1
Crop

Table 9: The architecture of encoder and decoder of the VAE utilized for
audiovisual dataset on audio. Latent encodings from image and audio modalities
are added together.

Table 10: The architecture of the VAE utilized across datasets.

Constant Input (512, 4, 4)
---
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 256
3x3 Conv2d, 3

Table 11: The generator architecture of StyleGAN2 for CelebA-HQ.

Constant Input (512, 8, 4)
---
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 256
3x3 Conv2d, 1
Crop

Table 12: The generator architecture of StyleGAN2 for NSynth

Constant Input (512, 4, 4)
---
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 256
3x3 Conv2d, 3

Table 13: The generator architecture of StyleGAN2 for audiovisual domain for
images.

Constant Input (512, 8, 2)
---
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 512
StyleConv 256
3x3 Conv2d, 1
Crop

Table 14: The generator architecture of StyleGAN2 for audiovisual domain for
audio.

Table 15: The architecture of the StyleGAN generator utilized across datasets.