# Unbalanced clustering and solitary states in coupled excitable systems

Igor Franović<EMAIL_ADDRESS>Scientific Computing Laboratory, Center for
the Study of Complex Systems, Institute of Physics Belgrade, University of
Belgrade, Pregrevica 118, 11080 Belgrade, Serbia  Sebastian Eydam
<EMAIL_ADDRESS>Neural Circuits and Computations Unit, RIKEN Center
for Brain Science, 2-1 Hirosawa, 351-0106 Wako, Japan  Nadezhda Semenova
<EMAIL_ADDRESS>Saratov State University, Astrakhanskaya str. 83,
Saratov, 410012 Russia
Départment d’Optique P. M. Duffieux, Institut FEMTO-ST, Université Bourgogne-
Franche-Comte CNRS UMR 6174, Besançon, France  Anna Zakharova
<EMAIL_ADDRESS>Institut für Theoretische Physik, Technische
Universität Berlin, Hardenbergstr. 36, 10623 Berlin, Germany

###### Abstract

We demonstrate the mechanisms of emergence and the link between two types of
symmetry-broken states, the unbalanced periodic two-cluster states and
solitary states, in coupled _excitable_ systems with prevalent repulsive
interactions. Solitary states in non-locally coupled arrays inherit their
dynamical features from unbalanced cluster states in globally coupled
networks. Apart from self-organization based on phase-locked synchrony,
interplay of excitability and local multiscale dynamics also gives rise to
_leap-frog_ states involving alternating order of spiking. Noise suppresses
multistability of cluster states and induces pattern homogenization,
transforming solitary states into patterns of patched synchrony.

The remarkable discovery of chimera states [1, 2] has led to a profound change
of paradigm in understanding of self-organization in assemblies of coupled
oscillators. Instead of synchronization transition and onset of collective
mode [3], attention has shifted to states emerging via symmetry breaking of
synchrony [4], where assemblies of indistinguishable oscillators with
symmetric couplings split into groups with different dynamics. A common
ingredient to many systems displaying clustering [8, 6, 5, 7], chimeras [11,
9, 10, 12] or solitary states [13, 15, 18, 14, 16, 17] are the long-range
interactions. In contrast to low-dimensional dynamics of cluster states, where
units within each group behave identically, chimeras are patterns of
coexisting coherence and incoherence [11]. Similar coexistence underlies
solitary states, where a single or a small subset of solitary units split from
the synchronized cluster, but unlike chimeras, the minority units spread
randomly instead of forming localized domains. Another difference is that
solitary states involve spatial chaos [19], reflecting sensitive dependence of
the dynamics on spatial coordinates, which gives rise to extensive
multistability where the number of states grows exponentially with system
size. Still, both chimeras and solitary states conform to the class of weak
chimeras [20].

For coupled oscillators, much progress has been made in resolving two main
problems, namely the mechanisms of the onset and potential links between
symmetry-broken states along the path from complete coherence to incoherence.
In particular, emergence of clusters from complete synchrony has been
explained by unfolding of cluster singularity, revealing cascade transitions
from synchronous state to balanced two-cluster partition via unbalanced
cluster states [5]. Also, clustering has been identified as a prerequisite for
the onset of chimeras [21]. Self-organization of strong chimeras has been
shown to involve stabilization of coherent cluster by the incoherent one [22],
while solitary states have been found to mediate transition from complete
coherence to chimeras [15].

However, in a myriad of examples, from neural and cardiac tissue to chemical
reactions and lasers, system components are not intrinsic oscillators, but are
rather _excitable_ units [23, 24], nonlinear threshold elements which in the
absence of input lie at rest, but may be triggered to oscillate by
sufficiently strong perturbations. There is no reason to expect _a priori_
that the results for coupled oscillators translate to realm of excitable
systems, where even the onset of collective oscillations requires repulsive
rather than attractive interactions. Thus, the fundamental questions on the
emergence and the relation between periodic cluster states, chimeras and
solitary states in coupled excitable systems remain open.

In this Letter, we address these issues by revealing the mechanisms of onset
and the link between different types of unbalanced periodic two-cluster states
and solitary states in coupled excitable systems with multiscale local
dynamics, typical for neuroscience [24], and varying attractive/repulsive [25]
character of interactions. We show that solitary states in non-locally coupled
arrays inherit dynamical features from unbalanced periodic two-cluster states
in a globally coupled network. Apart from symmetry-broken states with phase-
locked spiking, we discover peculiar cluster and solitary states where self-
organization is based on _leap-frog_ dynamics [26, 27, 28, 29], which involves
alternating order of spiking (leader-switching) between units. Leap-frogging
cannot exist for phase oscillators, and has recently been shown to emerge from
slow-fast dynamics in vicinity of the canard transition [26]. Excitable
systems are highly sensitive to noise [23], and it has been known to play a
facilitatory role for spontaneous clustering [30] and emergence of chimeras
[32, 31, 33]. Here we demonstrate the effect of noise-induced preference of
attractors [34, 35, 36], where the noise suppresses system’s multistability by
promoting certain types of cluster states as well as patched patterns at the
expense of solitary states.

Our system is an array of $N$ identical FitzHugh-Nagumo units whose dynamics
is given by

$\displaystyle\varepsilon\frac{du_{k}}{dt}$
$\displaystyle=u_{k}-\frac{u_{k}^{3}}{3}-v_{k}+\frac{\kappa}{2R}\sum\limits_{l=k-R}^{k+R}[g_{uu}(u_{l}-u_{k})$
$\displaystyle+g_{uv}(v_{l}-v_{k})]+\sqrt{\varepsilon}\sigma\xi_{k}(t)$
$\displaystyle\frac{dv_{k}}{dt}$
$\displaystyle=u_{k}+a+\frac{\kappa}{2R}\sum\limits_{l=k-R}^{k+R}[g_{vu}(u_{l}-u_{k})$
$\displaystyle+g_{vv}(v_{l}-v_{k})],$ (1)

where the local slow-fast dynamics is governed by the activator variables
$u_{k}$ and the recovery variables $v_{k}$, and the scale separation is
introduced by a small parameter $\varepsilon=0.05$. All indices are modulo
$N$. For neuronal systems, fast and slow variables describe the dynamics of
membrane potential and the coarse-grained action of gating variables,
respectively. Local bifurcation parameter $a$, here fixed to $a=1.001$,
mediates the transition from excitable ($|a|>1$) to oscillatory behaviour
($|a|<1$). Due to the singular character of Hopf bifurcation at $a=1$, the
onset of oscillations is followed by a _canard transition_ ($a\approx
1-\varepsilon/8$) from small-amplitude (subthreshold) to large-amplitude
relaxation oscillations [37]. Non-local interactions are characterized by the
coupling strength $\kappa$, here fixed to $\kappa=0.4$, and the radius
$r=R/N$, whereby the case $r=1/2$ conforms to a globally connected network.
Apart from terms involving only $u_{k}$ or $v_{k}$, there are also cross-
coupling terms, which is compactly written via rotational coupling matrix [41]
$G=\left(\begin{matrix}g_{uu}&g_{uv}\\\
g_{vu}&g_{vv}\end{matrix}\right)=\left(\begin{matrix}\cos\phi&\sin\phi\\\
-\sin\phi&\cos\phi\end{matrix}\right)$. Parameter $\phi$ affects the
prevalence of attractive and repulsive interactions. The latter are necessary
to induce oscillations as attractive interactions cannot destabilize the
collective rest state. Spiking can also emerge due to noise, acting in the
fast variables in analogy to synaptic noise [42]. Each unit is influenced by
independent Gaussian white noise $\xi_{k}(t)$ of intensity $\sigma$. We first
focus on how the stability of unbalanced two-cluster states in globally
connected networks changes with $\phi$ and then analyze the onset of solitary
states in non-locally coupled arrays ($r<1/2$) of excitable elements.
Previously, solitary states were studied for FitzHugh-Nagumo oscillators in
one- [16, 38] and two-layer [39, 40] networks.

## Two-cluster states in a globally coupled network

To gain insight into the structure of unbalanced periodic two-cluster states,
their stability domains and bifurcations at their boundaries, we implement a
twofold approach, combining the semi-analytical method of _evaporation
exponents_ and the numerical path-following method based on introducing _probe
oscillators_. Since our interest is in solutions where both clusters emit
spikes, the splitting scenario by which clusters emerge from the collective
rest state is beyond the scope of this Letter. We just note that for stable
local dynamics ($|a|>1$), interaction-induced destabilization of the
stationary state at $\phi^{*}=\arccos(\frac{1-a^{2}}{2\kappa})$ is a highly
degenerate point where $2(N-1)$ Jacobian eigenvalues simultaneously become
critical, giving rise to a large number of different cluster partitions
featuring subthreshold oscillations, which in an exponentially small $\phi$
region start to display spikes via secondary canard transitions. Stability of
the stationary state is regained at $\bar{\phi}=\phi^{*}+\pi$. Onsets of
cluster instability and subsequent periodic cluster states for type I
excitable units are addressed in [43, 44].

Unlike Lyapunov exponents, evaporation exponents [45, 47, 46] can describe
perturbations that destroy cluster partitions. They characterize stability of
clusters to emanation of elements, induced by perturbations transversal to
invariant subspace of a particular partition. Clusters with negative
evaporation exponents are attractors of an assembly, while their positive
values indicate instability. Let us consider a two-cluster state with
$N_{A}=pN$ units in cluster A and $N_{B}=(1-p)N$ units in cluster B. The
dynamics of this state is independent on $N$, and is governed by the reduced
system

$\displaystyle\varepsilon\dot{u_{i}}$
$\displaystyle=u_{i}-\frac{1}{3}u_{i}^{3}-v_{i}+\kappa
w_{i}(g_{uu}(u_{j}-u_{i})+g_{uv}(v_{j}-v_{i}))$ $\displaystyle\dot{v_{i}}$
$\displaystyle=u_{i}+a+\kappa w_{i}(g_{vu}(u_{j}-u_{i})+g_{vv}(v_{j}-v_{i})),$
(2)

with $i,j\in\\{A,B\\},i\neq j$, and $(w_{A},w_{B})=(1-p,p)$ being additional
coupling weights derived from particular cluster partition. For $p\neq 1/2$,
system (2) is equivalent to a pair of nonidentical excitable units with
attractive and repulsive interactions. Each value of partition parameter $p$
specifies an invariant subspace in the complete phase space, so that subspaces
for different $p$ intersect only in the full synchrony plane
$u_{A}=u_{B},v_{A}=v_{B}$. To introduce evaporation exponents, we consider
symmetric small perturbations to two units, 1 and 2, in each of the clusters:
$u_{i,1/2}=u_{i}\pm\delta u_{i},v_{i,1/2}=v_{i}\pm\delta v_{i},i\in\\{A,B\\}$.
Due to permutation symmetry, they can be applied to an arbitrary pair of
elements, leaving the cluster mean-fields unchanged. Linearized equations for
deviations $(\delta u_{i}(t),\delta v_{i}(t))$ transversal to clusters
$i\in\\{A,B\\}$ read

$\displaystyle\varepsilon\dot{\delta u_{i}}$
$\displaystyle=(1-u_{i}^{2}-\kappa g_{uu})\delta u_{i}-(1+\kappa g_{uv})\delta
v_{i}$ $\displaystyle\dot{\delta v_{i}}$ $\displaystyle=(1-\kappa
g_{vu})\delta u_{i}-\kappa g_{vv}\delta v_{i}.$ (3)

Evaporation exponents
$\lambda_{ev,i}=\underset{T\rightarrow\infty}{lim}\frac{1}{2}\ln\frac{\delta
u_{i}^{2}(T)+\delta v_{i}^{2}(T)}{\delta u_{i}^{2}(0)+\delta v_{i}^{2}(0)}$
for $i\in\left\\{A,B\right\\}$ are obtained by integrating system (2)-(3).

Figure 1: Unbalanced periodic two-cluster states in globally coupled
network.(a) Stability diagram in the $(\phi,p)$ parameter plane. Darker
(lighter) shading: stable (unstable) solutions. Black solid lines: period-
doubling bifurcations, green lines: curves of branching points. Black dashed
lines: destabilization and reappearance of collective rest state at
$\phi=\phi^{*}\approx 1.573$ and $\phi=\bar{\phi}\approx 4.715$. (b)-(e) Time
traces $u_{i}(t),i\in\\{A,B\\}$ and phase portraits corresponding to
characteristic domains from (a). Associated $(\phi,p)$ values are shown by
blue squares in (a). Remaining parameters are
$\kappa=0.4,a=1.001,\varepsilon=0.05$.

Bifurcations at stability boundaries of particular cluster states are
determined by numerical continuation using probe oscillators, designed to test
whether a unit added to the cluster asymptotically remains in or emanates from
it. Probes are introduced to each of the clusters by coupling to the
corresponding mean-fields, but without affecting the mean-fields themselves,
such that their dynamics $(\tilde{u}_{i}(t),\tilde{v}_{i}(t))$ obeys

$\displaystyle\varepsilon\dot{\tilde{u}}_{i}$
$\displaystyle=\tilde{u}_{i}-\tilde{u}_{i}^{3}-\tilde{v}_{i}+\kappa[w_{i}(g_{uu}(u_{i}-\tilde{u}_{i})+g_{uv}(v_{i}-\tilde{v}_{i}))$
$\displaystyle+w_{j}(g_{uu}(u_{j}-\tilde{u}_{i})+g_{uv}(v_{j}-\tilde{v}_{i}))]$
$\displaystyle\dot{\tilde{v}}_{i}$
$\displaystyle=\tilde{u}_{i}+a+\kappa[w_{i}(g_{vu}(u_{i}-\tilde{u}_{i})+g_{vv}(v_{i}-\tilde{v}_{i}))$
$\displaystyle+w_{j}(g_{vu}(u_{j}-\tilde{u}_{i})+g_{vu}(v_{j}-\tilde{v}_{i}))],$
(4)

for $i,j\in\\{A,B\\},i\neq j$. Continuation of system (2) together with (4)
was performed by AUTO [48].

Combined results of the two approaches shown in Fig. 1(a) summarize the
stability diagram for the system (2)-(3) in the $(\phi,p)$ plane, with darker
(lighter) shading indicating the stable (unstable) solutions. Black curves at
the boundaries of stable regions indicate period-doubling bifurcations,
whereas the green lines correspond to curves of branching points. The latter
are typically pitchfork bifurcations in the reduced system, but correspond to
unfolding of highly degenerate bifurcation points [5] of system (1), where $p$
becomes a solution parameter. System (2) supports six characteristic regimes
with 1:1 (regions IV, V and VI), 1:2 (I, II) and 2:3 (region III) frequency
locking, all conforming to mixed-mode oscillations [49, 50] with interspersed
large- and small-amplitude oscillations, cf. Fig. 1(b)-(f). Apart from the
classical phase-locked solutions I,II,III and V, one also observes unbalanced
cluster states with leap-frog patterns [26, 27, 28, 29] between the clusters.
Switching of leadership occurs via subthreshold oscillations, such that the
current leader performs an extra small oscillation allowing it to be overtaken
by the lagging cluster. Stable two-cluster solutions at the line $p=1/2$ may
acquire an additional antiphase space-time symmetry
$u_{A}(t)=u_{B}(t+P/2),v_{A}(t)=v_{B}(t+P/2)$, where $P$ is the oscillation
period. Different types of leap-frog patterns and the underlying mechanisms
have recently been analysed in [26], showing that they may qualitatively be
understood within the framework of phase-sensitive excitability of a periodic
orbit [51], a new concept manifested as a strong and non-uniform sensitivity
to perturbations of both the small- and large-amplitude oscillations in the
FitzHugh-Nagumo system.

Figure 2: Persistence of unbalanced periodic two-cluster states under noise.
(a) Quantity $d(\phi,p)$ distinguishes between four cases: cluster states
reorganize to a partition with smaller (blue, $\lambda_{ev,A}>\lambda_{ev,B}$)
or larger $p$ values (red, $\lambda_{ev,B}>\lambda_{ev,A}$); two-cluster
states are unstable (gray, $d=4$); only the synchronous stationary state is
stable (white, $d=1$). (b)-(e) Examples of evolution of partition parameter
$p(t)$ under noise. From left to right, interaction parameters and noise
levels are $\phi=2,4.4,2.3,5.0$, $\sigma=5,0.6,0.6,5\times 10^{-3}$,
respectively.

Evaporation exponents can also be used to approximate the impact of
sufficiently small noise to stability of two-cluster partitions. In
particular, for (1) with $r=1/2,\sigma>0$, we find that noise may cause a
transition to another type of two-cluster state, or may reorganize the state’s
structure by inducing migration of units between the clusters, without
qualitatively changing the dynamics of cluster mean-fields. Reorganization
process eventually settles to a partition where the net transport between the
clusters reaches a dynamical balance. Of course, both splitting of a unit from
a cluster and migration to another one may involve nonlinear effects. Still,
at the linear level, the ”potential barrier” that has to be overcome when unit
leaves the cluster is proportional to $\lambda_{ev,i}$. This is used to
characterize susceptibility of two-cluster states to noise in Fig. 2(a). We
distinguish between the cases where the noise is more likely to shift a two-
cluster partition toward smaller ($0>\lambda_{ev,A}>\lambda_{ev,B}$; blue
regions) or larger $p$ value ($0>\lambda_{ev,B}>\lambda_{ev,A}$; red regions),
depending on which of the two stable exponents dominates. There are also
domains where the unbalanced two-cluster states are unstable
($\lambda_{ev,A/B}>0$, shown gray) and where only the synchronous stationary
state is stable (white). For convenience, each case is assigned with a
discrete variable $d\in\\{1,2,3,4\\}$. Evolution of cluster partition under
noise, described by variation of partition parameter $p(t)$, is illustrated in
Fig. 2(b)-(e) for typical solutions from several regions I-VI. While typical
states from regions I and V eventually stabilize at a certain $p$ level close
to initial one, displaying persistence under noise, a representative state
from region II migrates to region III. Interestingly, the asymmetric leap-frog
solution from region IV evolves toward balanced partition $p=1/2$, gaining
antiphase space-time symmetry.

Figure 3: Solitary state SS1 in a non-locally coupled array
($N=100,\phi=1.85,r=0.2,\kappa=0.4,a=1.001,\varepsilon=0.05$). (a) Spatial
profile of the average local spiking frequencies $\omega_{k}$; (b) Red and
blue: snapshots of local variables $(u_{k},v_{k})$ at two time moments
$t_{1}<t_{2}$, black: nullclines of an isolated unit; (c) Spatiotemporal
evolution of activator variables $u_{k}(t)$; (d) Phase portraits
$(u_{k}(t),v_{k}(t))$ for units representing solitary/minority ($k=84$) and
synchronized/majority ($k=60$) cluster; (e) Cross-correlation matrix $C_{kl}$;
(f) Time traces $u_{k}(t)$ for two representative units from (d).

## Solitary states in non-locally coupled arrays

We show that the prevalent solitary states in non-locally coupled arrays, SS1
and SS2, inherit their dynamical features from the corresponding unbalanced
cluster states from Fig. 1(a). In particular, state SS1, illustrated in Fig.
3, is a dynamical counterpart of the two-cluster state from region I, whereas
SS2 (not shown) derives from the cluster state from region V. In both cases,
solitary states occur within the same $\phi$ interval and preserve frequency
locking of the cluster states, but due to nonlocal interactions and the
associated fluctuations of local mean-fields, the clusters of solitary and
typical units are fuzzy [15] rather than exact. Spatial profile of average
local frequencies $\omega_{k}=2\pi M_{k}/\Delta$, where $M_{k}$ is the number
of complete rotations around the origin within the time interval $\Delta$,
indeed shows a 2:1 frequency ratio between solitary and typical units. Analogy
to two-cluster state in terms of local phase portraits and time traces
$u_{k}(t)$ is illustrated in Fig. 3(d) and (f). Intrinsic dynamics of SS1 is
characterized by the cross-correlation matrix
$C_{kl}=\frac{\langle\hat{u}_{k}(t)\hat{u}_{l}(t)\rangle_{T}}{\sqrt{\langle\hat{u}_{k}(t)^{2}\rangle_{T}\langle\hat{u}_{l}(t)^{2}\rangle_{T}}}$,
where $\langle\cdot\rangle_{T}$ refers to time averaging, while
$\hat{u}_{k}(t)=u_{k}(t)-\langle u_{k}(t)\rangle_{T}$ are the deviations of
local activator variables from their means, cf. Fig. 3(e).

Figure 4: Chaotic solitary state SS3
($N=100,\phi=1.788,r=0.2,\kappa=0.4,a=1.001,\varepsilon=0.05$) (a) Spatial
profile of $\omega_{k}$; (b) First return map of interspike intervals $\Delta
t_{n+1}(\Delta t_{n})$ for a solitary unit $k=76$; (c) Space-time evolution of
$u_{k}(t)$; (d) Time traces $u_{k}(t)$ for a solitary unit ($k=76$) and a unit
from synchronized cluster ($k=20$); (e) Cross-correlation matrix $C_{kl}$; (f)
Time traces $u_{k}(t)$ show leap-frog dynamics within synchronized cluster
($k=20$ and $k=40$).

We also find less likely solitary states that do not have two-cluster state
counterparts. A typical example is a state SS3, see Fig. 4, which unlike SS1
and SS2, features leap-frog dynamics, both for solitary-typical pairs of
units, and pairs of only solitary or typical units, cf. Fig. 4(d) and (f). SS3
admits chaotic rather than periodic dynamics, see Fig. 4(b), and emerges due
to nonlocal interactions, which induce self-localized excitations [52] at
interfaces separating domains with distinct dynamics. Frequency profile
$\omega_{k}$ shows two clusters, but with a frequency ratio distinct from SS1,
cf. Fig. 3(a). The difference in average frequencies here derives from random
events where solitary units emit two successive spikes instead of a single
spike followed by a subthreshold oscillation, see the arrow in Fig. 4(d). Also
note the more complex correlation structure of SS3 compared to SS1, cf. Fig.
4(e) and Fig. 3(e).

Figure 5: Transformation of solitary states into patched states under noise.
(a) Typical spatiotemporal dynamics before introduction of noise (SS1 state);
(b) Patched pattern developed from SS1 under noise $\sigma=0.0011$. Time axes
in (a) and (b) are shifted. Parameters are
$N=200,\phi=2.0,r=0.2,\kappa=0.4,a=1.001,\varepsilon=0.05$.

While in _locally_ coupled excitable systems noise may strongly influence
pattern formation by inducing, enhancing or controlling wave propagation,
spiral dynamics and pacemaking [23], here the deterministic dynamics of a
nonlocally coupled array features an additional landmark, namely extensive
multistability, such that the main impact of noise is qualitatively different.
We find that the noise reduces the system’s multistability, suppressing the
solitary states. This reflects the effect called noise-induced preference of
attractors [34], which may be understood as follows: in a highly multistable
system, stability boundaries of attractors become smeared by noise, and only
those with sufficiently large basins of attraction remain visible. For small
noise, unbalanced splitting into frequency clusters is still preserved, but
the preferred spatial distribution of minority units is _localized_ rather
than random. This gives rise to _patched_ patterns with 1:2 subharmonic
frequency locking. A typical example is provided in Fig. 5, which shows how an
initial configuration corresponding to SS1 transforms under noise into a state
of patched synchrony [41].

## Disussion

We have demonstrated the mechanism of onset and the link between the
unbalanced periodic two-cluster states and solitary states, as a form of weak
chimeras, in coupled excitable systems where repulsion dominates over
attractive interactions. The fact that the prevalent solitary states in non-
locally coupled arrays inherit their dynamical features from unbalanced
cluster states in globally coupled networks is to a certain extent
qualitatively similar to the finding for globally coupled Stuart-Landau
oscillators, where clustering is identified as a symmetry-breaking step
required for emergence of chimeras [21]. The peculiar property of symmetry-
broken states we discovered is the leap-frog dynamics. Such a behavior derives
from multiscale character of the system, and in particular the phase-sensitive
excitability of relaxation oscillations, underlying their strong sensitivity
to perturbations in vicinity of the canard transition [51, 26]. Current
results, together with [53, 26], indicate the importance of this concept to
pattern formation in multiscale systems, universal to coupled excitable units
and coupled oscillators. Multistability of the system dynamics is suppressed
by noise, as a hallmark of noise-induced preference of attractors, an effect
previously corroborated in coupled oscillators [35], the Hénon map [54] and
multistable fiber lasers [36]. Small noise is found to influence pattern
formation by promoting homogeneous patched patterns at the expense of solitary
states. Given that solitary states in coupled oscillators may mediate the
transition from complete synchrony to chimeras [15], an interesting open
question is whether they may play a similar role in coupled excitable systems.

###### Acknowledgements.

I.F. acknowledges funding from Institute of Physics Belgrade through grant by
Ministry of Education, Science and Technological Development of Republic of
Serbia. AZ acknowledges support from the Deutsche Forschungsgemeinschaft (DFG)
within the framework of the SFB 910, Projektnummer 163436311.

## References

  * [1] Y. Kuramoto, and D. Battogtokh, Nonlinear Phenom. Complex Syst. 5,380 (2002).
  * [2] D. M. Abrams, and S. H. Strogatz, Phys. Rev. Lett. 93, 174102 (2004).
  * [3] A. Pikovsky, M. Rosenblum, and J. Kurths, _Synchronization: A Universal Concept in Nonlinear Sciences_(Cambridge University Press, Cambridge, 2003).
  * [4] A. E. Motter, Nat. Phys. 6(3), 164 (2010).
  * [5] F. P. Kemeth, S. W. Haugland, K. Krischer, Chaos 29, 023107 (2019).
  * [6] W. L. Ku, M. Girvan, and E. Ott, Chaos 25, 123122 (2015).
  * [7] A. Ismail and P. Ashwin, Dyn. Syst. 30, 122 (2015).
  * [8] D. Hansel, G. Mato, and C. Meunier, Phys. Rev. E 48, 3470 (1993).
  * [9] M. J. Panaggio, and D. M. Abrams, Nonlinearity 28, R67 (2015).
  * [10] O. E. Omel’chenko, Nonlinearity 31, R121 (2018).
  * [11] A. Zakharova, _Chimera Patterns in Networks: Interplay Between Dynamics, Structure, Noise, and Delay - Understanding Complex Systems_ , (Springer Nature, Switzerland, 2020).
  * [12] F. Parastesh, S. Jafari, H. Azarnoush, Z. Shahriari, Z. Wang, S. Boccaletti, and M. Perc, Phys. Rep. 898, 1 (2021).
  * [13] Y. Maistrenko, B. Penkovsky, and M. Rosenblum, Phys. Rev. E 89, 060901(R) (2014).
  * [14] V. Maistrenko, O. Sudakov, and O. Osiv, Chaos 30, 063113 (2020).
  * [15] P. Jaros, Y. Maistrenko, and T. Kapitaniak, Phys. Rev. E 91, 022907 (2015).
  * [16] E. Rybalova, V.S. Anishchenko, G.I. Strelkova, and A. Zakharova, Chaos 29, 071106 (2019).
  * [17] R. Berner, A. Polanska, E. Schöll, and S. Yanchuk, Eur. Phys. J. Spec. Top. 229, 2183 (2020).
  * [18] P. Jaros, S. Brezetsky, R. Levchenko, D. Dudkowski, T. Kapitaniak, and Y. Maistrenko, Chaos 28, 011103 (2018).
  * [19] I. Omelchenko, Y. Maistrenko, P. Hövel, and E. Schöll, Phys. Rev. Lett. 106, 234102 (2011).
  * [20] P. Ashwin, and O. Burylko, Chaos 25, 013106 (2015).
  * [21] L. Schmidt, and K. Krischer, Phys. Rev. Lett. 114, 034101 (2015).
  * [22] Y. Zhang, and A. Motter, Phys. Rev. Lett. 126, 094101 (2021).
  * [23] B. Lindner, J. García-Ojalvo, A. Neiman, and L. Schimansky-Geier, Phys. Rep. 392 321 (2004).
  * [24] E. M. Izhikevich, _Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting_ (MIT Press, Cambridge, MA, 2007).
  * [25] I. Belykh, R. Reimbayev, and K. Zhao, Phys. Rev. E 91, 062919 (2015).
  * [26] S. R. Eydam, I. Franović, and M. Wolfrum, Phys. Rev. E 99, 042207 (2019).
  * [27] P. Goel, and G. B. Ermentrout, Physica D (Amsterdam) 163, 191 (2002).
  * [28] C. D. Acker, N. Kopell, and J. A. White, J. Comp. Neurosci. 15, 71 (2003).
  * [29] M. Oh, and V. Matveev,J. Comput. Neurosci. 26, 303 (2009).
  * [30] I. Franović, K. Todorović, N. Vasović, and N. Burić, Phys. Rev. Lett. 108, 094101 (2012).
  * [31] A. Zakharova, N. Semenova, V. Anishchenko, and E. Schöll, Chaos 27, 114320 (2017).
  * [32] N. Semenova, A. Zakharova, V. Anishchenko, and E. Schöll, Phys. Rev. Lett. 117, 014102 (2016).
  * [33] N.Semenova, Eur. Phys. J. Special Topics 229, 2295 (2020)
  * [34] A. N. Pisarchik, and U. Feudel, Phys. Rep. 540, 167 (2014).
  * [35] K. Kaneko, Phys. Rev. Lett. 78, 2736 (1997).
  * [36] A. N. Pisarchik, R. Jaimes-Reátegui, R. Sevilla-Escoboza, G. Huerta-Cuillar, and M. Taki, Phys. Rev. Lett. 107, 274101 (2011).
  * [37] S. M. Baer and T. Erneux, SIAM J. Appl. Math. 46, 721 (1986).
  * [38] M. Mikhaylenko, L. Ramlow, S. Jalan, and A. Zakharova, Chaos 29, 023122 (2019).
  * [39] L. Schülen, D. A. Janzen, E. S. Medeiros, and A. Zakharova, Chaos Soliton. Fract. 145, 110670 (2021).
  * [40] E. V. Rybalova, A. Zakharova, and G.I. Strelkova, Chaos Soliton. Fract. 148, 111011 (2021).
  * [41] I. Omelchenko, O. E. Omel’chenko, P. Hövel, and E. Schöll, Phys. Rev. Lett. 110, 224101 (2013).
  * [42] A. Destexhe, and M. Rudolph-Lilith, _Neuronal noise_ (Springer, New York, 2012).
  * [43] R. Ronge, and M. A. Zaks, Phys. Rev. E 103, 012206 (2021).
  * [44] M. A. Zaks, and P. Tomov, Phys. Rev. E 93, 020201(R) (2016).
  * [45] A. Pikovsky, and A. Politi, _Lyapunov Exponents: A Tool to Explore Complex Dynamics_ (Cambridge University Press, Cambridge, 2016).
  * [46] A. Pikovsky, O. Popovych, and Y. Maistrenko, Phys. Rev. Lett. 87, 044102 (2001).
  * [47] M. Zaks, and A. Pikovsky, Sci. Rep. 7, 4648 (2017).
  * [48] E. J. Doedel, and B. E. Oldeman, AUTO-07P: Continuation and bifurcation software for ordinary differential equations, (Concordia University, Montreal, 2012).
  * [49] M. Desroches, J. Guckenheimer, B. Krauskopf, C. Kuehn, H. M. Osinga, and M. Wechselberger, SIAM Rev. 54, 211 (2012).
  * [50] C. Kuehn, _Multiple Time Scale Dynamics_ (Springer International Publishing, Switzerland, 2015).
  * [51] I. Franović, O. E. Omel’chenko, and M. Wolfrum, Chaos 28, 071105 (2018).
  * [52] M. Wolfrum, O. E. Omel’chenko, and J. Sieber, Chaos 25, 053113 (2015).
  * [53] H. G. Rotstein and H. Wu, Phys. Rev. E 86, 066207 (2012).
  * [54] B. E. Martínez-Zérega, and A. N. Pisarchik, Commun. Nonlinear Sci. Numer. Simul. 17, 4023 (2012).


# DeepTreeGANv2: Iterative Pooling of Point Clouds

Moritz A.W. Scham
Deutsches Elektronen-Synchrotron DESY,
Germany
<EMAIL_ADDRESS>
Dirk Krücker
Deutsches Elektronen-Synchrotron DESY,
Germany
<EMAIL_ADDRESS>
Kerstin Borras
Deutsches Elektronen-Synchrotron DESY,
Germany
<EMAIL_ADDRESS>
Also at Jülich Supercomputing Centre, Institute for Advanced Simulation,
Germany and RWTH Aachen University, III. Physikalisches Institut A,
GermanyAlso at RWTH Aachen University, III. Physikalisches Institut A, Germany
0000-0001-9494-2151 0000-0003-1610-8844 0000-0003-1111-249X

###### Abstract

In High Energy Physics, detailed and time-consuming simulations are used for
particle interactions with detectors. To bypass these simulations with a
generative model, the generation of large point clouds in a short time is
required, while the complex dependencies between the particles must be
correctly modelled. Particle showers are inherently tree-based processes, as
each particle is produced by the decay or detector interaction of a particle
of the previous generation. In this work, we present a significant extension
to DeepTreeGAN [1], featuring a critic, that is able to aggregate such point
clouds iteratively in a tree-based manner. We show that this model can
reproduce complex distributions, and we evaluate its performance on the public
JetNet 150 dataset.

##  1 Introduction

In particle physics, detailed, near-perfect simulations of the underlying
physical processes, the measurements, and the details of the experimental
apparatus are state-of-the-art. However, accurate simulations of large and
complex detectors, especially calorimeters, using current Monte Carlo-based
tools such as those implemented in the Geant4 [2] toolkit are computationally
intensive. Currently, more than half of the worldwide Large Hadron Collider
(LHC) grid resources are used for the generation and processing of simulated
data [3]. For the future High-Luminosity upgrade of the LHC (HL-LHC [4]), the
computational requirements will exceed the computational resources without
significant speedups, e.g., for the CMS experiment by 2028 [5] projecting the
current technologies. This will become even more demanding for future high-
granularity calorimeters, e.g., the CMS HGCAL [6] with its complex geometry
and extremely large number of channels. To address these challenges, fast
simulation approaches based on Deep Learning, including Generative Adversarial
Networks (GAN) [7] and Variational Autoencoders (VAE) [8], have been explored
early [9, 10, 11, 12, 13, 14] and deployed recently [15].

The majority of these approaches consider the data as voxels living on three-
dimensional grids. Especially for high-granularity calorimeters like the
HGCAL, the calorimeter cells are highly irregular, and the data is often
sparse. In such a situation, it is beneficial to consider the data as Point
Clouds (PC), e.g., tuples of cell energies and three-dimensional coordinates,
i.e., four-dimensional point clouds [16, 17, 18]. For modelling calorimeter
showers, which are cascades of particles, it seems natural to model the
creation of such point clouds as a tree-like process. Such an approach can be
implemented by Graph Neural Networks (GNN) [19] and Graph Convolutions [20].
For modelling PC data representing three-dimensional objects, a similar
approach [21] called tree-GAN [21] exists. For our use case, especially for
modelling the large dynamic range of the energy dimension, this approach is
not sufficient. Here we report on the development of a largely extended Graph
GAN approach, which we named DeepTreeGAN that can be applied to multiple areas
of PC generation. To demonstrate the abilities of our model, we present a
first application to the JetNet dataset introduced in the next section.

###  1.1 JetNet

In this study, the JetNet [22, 23] datasets are used. Each dataset contains
PYTHIA [24] jets with an energy of about 1 TeV, with each jet containing up to
30 or 150 constituents (here: 150). The datasets differ in the jet-initiating
parton. Here, the dataset for top quark jets is studied. Each dataset contains
about 170k individual jets split as 110k/10k/50k for training/test/validation,
where the validation dataset is used for our results. The jet constituents,
the particles, are clustered with a cone radius of $R=0.8$. These particles
are considered to be massless and can therefore be described by their
3-momenta or equivalently by transverse momentum $p_{T}$, pseudorapidity
$\eta$, and azimuthal angle $\phi$. In the JetNet datasets, these variables
are given relative to the jet momentum:
$\eta^{\mathrm{rel}}_{i}\coloneqq\eta_{i}-\eta^{\mathrm{jet}}$,
$\phi^{\mathrm{rel}}_{i}\coloneqq\phi_{i}-(\phi^{\mathrm{jet}}\mod 2\pi)$, and
$p_{T,i}^{\mathrm{rel}}\coloneqq p_{T,i}/{p}_{\mathrm{T,jet}}$, where $i$ runs
over the particles in a jet. Calculating the invariant mass from these
relative quantities, for example, for the jet mass, implies
$m^{\mathrm{rel}}=m_{jet}/{p}_{\mathrm{T,jet}}$. This dataset has gained
popularity in the particle physics community as a benchmark for PC-based
generative models [25, 26, 18, 16, 17, 27, 28, 29, 30, 31, 32, 33, 34].

##  2 Architecture

PC-based GANs [17, 27, 30, 33] frequently choose a refinement approach:
Starting with a PC of the desired size with features sampled from noise, the
PC is updated multiple times in the generator and the critic. In the final
step of the critic, the points are aggregated and mapped to a single value,
usually with a feedforward neural network (FFN). In this way, the
dimensionality is greatly reduced in a single step. Commonly, the PC is
updated by first transforming the points individually. Then they are
aggregated into a global vector, which is used in-turn to transform the points
again. In an image-based GAN approach like DCGAN [35], the generator would
apply convolutions in sequence to iteratively upscale a random vector to an
image and the critic would apply convolutions in sequence to iteratively
downscale an image to a scalar. These convolutions cannot be directly
translated to PCs, which is why the mentioned refinement approach is prevalent
for PC-based GANs. Our approach instead is to translate this principle and to
construct a critic that reduces the dimensionality iteratively. In this way,
we complement the DeepTree generator [1], that iteratively upscales PCs.

###  2.1 Generator

The generator part of this model largely matches the one published in [1],
with a few key differences: Instead of splitting the leaf vector into one part
per branch and mapping the latter separately to the desired number of
branches, the full leaf vector is now mapped to the number of branches times
number of features. The generator uses 2, 3, 5, and 5 branches (with a product
of 150) and goes from 64 to 33 to 20 to 10 to 3 features. For each event, the
number of constituents is sampled from the dataset and the output of the
generator is cut to this size.

###  2.2 Critic

The critic is shown in Figure 1, its _components_ are described in the
following section. It features three _subcritics_ , that are applied to the
different stages of aggregation: The first is applied directly on the input PC
(up to 150 points), the other two are applied after each _bipartite pool_
(pooling to 30 points and 6 points). The expectation is that the first
_subcritic_ provides feedback on the more low-level features, while the two
later _subcritics_ operate on aggregated points and thus provide feedback on
more high-level features. Before the _bipartite pool_ , the _embedding_ layer
is applied, which maps the number of features to 10 and transforms the input.
The critic is conditioned on ${p}_{\mathrm{T}},\eta$ and the mass of the jet.

Figure 1: The critic, as described in 2.2

#### Bipartite Pool

For the iterative reduction of the number of points a pooling operation should
fulfill the following conditions: First, it must be differentiable to enable
backpropagation. Then it should be invariant under permutation because the
input is permutation invariant. As the number of input points varies, it needs
to accept an arbitrary number of points, down to a single one. The run-time
pooling operation should scale well to large PCs, ideally linear with the
number of points. Finally, it should map to a fixed number of points to
construct a regular shaped output tensor, that allows faster and more
convenient implementations. To achieve these requirements, we propose
constructing a bipartite graph, densely connecting the input PC to a fixed
number of trainable nodes and applying a Message Passing Layer (MPL) to this
graph. This will return a PC with a fixed number of points. The dense
connection yields a number of edges that equals the number of input points
times the number of trainable nodes. Because each edge yields one message, the
run-time of the MPL should be proportional to the size of the input PC. As an
MPL, Gatv2Conv [36] is used with 16 attention heads, as implemented in PyTorch
Geometric [37].
A similar approach has been presented in [38], but implemented as an attention
mechanism instead of an MPL. An advantage of our choice is that the bipartite
pool can handle variable sized PCs without masking.

#### FFNs

The _FFNs_ used in this critic consist of 3 layers with 100 hidden nodes
without bias. The first two hidden layers are followed by a 50% dropout layer
and LeakyReLU activation with a negative slope of 0.1. Spectral normalization
[39] is applied to the _FFNs_ , except for the _FFN_ in the _embedding_ layer,
where batch normalization [40] is applied before the activation.

#### Multi-Aggregation

At different stages, the points need to be aggregated in a single vector. This
is commonly done by summing over the points, e.g., [33]. To provide additional
information about the distribution, we compute this vector by concatenating
the sum, the maximum, the number of points and the width. The width of the
distribution is estimated by computing the mean absolute deviation:
$\frac{1}{n}\sum_{i}|x_{i}-\bar{x}|$.

#### Central Node Update

For the following description of the _embedding_ layer and the _subcritics_ ,
we define a _Central Node Update_ Layer (_CNU_): First, the input is
transformed with an _FFN_. Then the transformed PC is aggregated with the
_multi-aggregation_. Lastly, each of the transformed nodes is concatenated
with the aggregated vector and passed through a second _FFN_ that maps the
nodes back to their original dimension.

#### Embedding

The _embedding_ layer first maps the points independently to 10 features with
an _FFN_ and passes the PC though a _CNU_ layer with a residual connection.

#### Subcritics

Each subcritic is constructed using two _CNUs_ with a residual connection. The
points are then aggregated in the _multi-aggregation_ scheme. This aggregated
vector is then concatenated with the condition (${p}_{\mathrm{T}},\eta$ and
mass of the jet) and passed to an _FFN_ , mapping the vector to a single
output. This yields three output values for each event which are summed such
that the application of the loss and the backpropagation takes place for all
the outputs in parallel.

###  2.3 Training

Generator and critic are trained using the Hinge [41] loss, with a ratio of 1
to 2 gradients steps. The optimizer for the generator (critic) is Adam [42]
with $\beta_{1}=0.9,\beta_{2}=0.999$, a weight decay of $10^{-4}$ and a
learning rate of $10^{-5}$ ($3*10^{-5}$). Additional to the Hinge loss, the
generator is also trained to minimize the feature matching loss [43] with a
factor of 0.1. As an “intermediate layer” the concatenation of the _multi-
aggregation_ of the input PC, together with the output of the _embedding_ and
_bipartite pool_ layers are used.

###  2.4 Preprocessing & Postprocessing

For the training, the $\eta^{\mathrm{rel}}$ and $\phi^{\mathrm{rel}}$
distributions are scaled separately to a normal distribution. The
${p}_{\mathrm{T}}^{\mathrm{rel}}$ distribution is scaled using a Box-Cox
transformation with standardizing as implemented in [44]. To produce samples,
the inverse scaling is applied to the output of the generator. The original
dataset is scaled so that $\sum_{i}{p}_{\mathrm{T}}^{i}=1$. This feature is
difficult to model for the generator, while it is easy to recognize for the
critic. To avoid limiting the performance of the model, the output of the
generator is scaled in the same way.

##  3 Results

Figure 2: The distributions of $\eta^{\mathrm{rel}},\phi^{\mathrm{rel}},{p}_{\mathrm{T}}^{\mathrm{rel}}$ for constituents of the jets and the mass of the jets for the top-quarks JetNet 150 dataset (Simulation), as described in Section 1.1 and DeepTreeGAN (Model). Table 1: Comparison of the proposed DeepTreeGAN to EPiC-GAN [33] and MDMA [27]. The metrics were computed on a 25k event hold-out sample of the JetNet 150 top quark dataset. The ‘Limit’ row gives the measured in-sample distance by bootstrapping the hold-out dataset. Lower is better for all metrics. Values taken from [27, Table 1]. The metrics are provided by the JetNet library [45]. The best performance is printed in bold. Model | $W_{1}^{M}(\times 10^{3})$ | $W_{1}^{P}(\times 10^{3})$ | $W_{1}^{EFP}(\times 10^{5})$ | $FPD(\times 10^{4})$
---|---|---|---|---
Limit | $0.42\pm 0.09$ | $0.12\pm 0.04$ | $1.22\pm 0.32$ | $1.2\pm 0.6$
EPiC-GAN | $0.69\pm 0.08$ | $0.65\pm 0.03$ | $2.67\pm 0.39$ | $22\pm 1$
MDMA | $\mathbf{0.57\pm 0.09}$ | $0.10\pm 0.02$ | $\mathbf{2.12\pm 0.64}$ | $5.3\pm 0.9$
DeepTreeGAN | $1.49\pm 0.04$ | $0.13\pm 0.02$ | $5.01\pm 0.08$ | $\mathbf{3.4\pm 0.7}$

The variables of the constituents and the mass of the top quark dataset are
shown in Figure 2. The generated distributions of the constituents closely
match the distribution of the data. While the double peak structure of the
mass distribution is clearly visible, the model is not able to produce the
peak quite as sharp as in the dataset. In Table 1, DeepTreeGAN is compared to
other state-of-the-art GANs and found to be competitive in this context.
Diffusion and flow-matching based models [34, 29] have recently provided
leading results for the JetNet dataset, but sample production is generally
slower compared to GAN approaches. The DeepTreeGAN code and weights are
available on GitHub.111https://github.com/DeGeSim/nips23DeepTreeGANv2

##  4 Conclusion

In this work, the DeepTreeGAN model, first introduced in [1], was
significantly extended to model up to 150 jet constituents of the JetNet
dataset. To this end, a PC-based critic has been developed, featuring a
pooling operation (Section 2.2), that allows the iterative downscaling of PCs.
Through its significant advantages, this novel pooling implementation may
prove useful not only for generative tasks, but also any PC regression or
classification task. This extension represents a significant milestone in
scaling this model to handle even larger point clouds, such as particle
showers in high-granularity calorimeters.

## Acknowledgement

Moritz Scham is funded by Helmholtz Association’s Initiative and Networking
Fund through Helmholtz AI (grant number: ZT-I-PF-5-3). This research was
supported in part through the Maxwell computational resources operated at
Deutsches Elektronen-Synchrotron DESY (Hamburg, Germany). The authors
acknowledge support from Deutsches Elektronen-Synchrotron DESY (Hamburg,
Germany), a member of the Helmholtz Association HGF.

## References

  * [1] Moritz A.W. Scham, Dirk Krücker, Benno Käch and Kerstin Borras  “DeepTreeGAN: Fast Generation of High Dimensional Point Clouds” Submitted to EPJ Web of Conferences CHEP, https://arxiv.org/abs/2311.12616 In _EPJ Web Conf._ , 2023
  * [2] S. Agostinelli  “GEANT4–a simulation toolkit”  In _Nucl. Instrum. Meth. A_ 506, 2003, pp. 250  DOI: 10.1016/S0168-9002(03)01368-8
  * [3] Johannes Albrecht  “A Roadmap for HEP Software and Computing R&D for the 2020s”  In _Computing and Software for Big Science_ 3.1 Springer ScienceBusiness Media LLC, 2019, pp. 7  DOI: 10.1007/s41781-018-0018-8
  * [4] G. Apollinari, O. Brüning, T. Nakamoto and Lucio Rossi  “High Luminosity Large Hadron Collider HL-LHC. High Luminosity Large Hadron Collider HL-LHC” Chapter 1 in High-Luminosity Large Hadron Collider (HL-LHC) : Preliminary Design Report  In _CERN Yellow Report_ , 2015, pp. 1–19  DOI: 10.5170/CERN-2015-005.1
  * [5] CMS Offline Software and Computing  “CMS Phase-2 Computing Model: Update Document”, 2022  URL: https://cds.cern.ch/record/2815292
  * [6] The CMS collaboration  “The CMS HGCAL detector for HL-LHC upgrade”  In _5th Large Hadron Collider Physics Conference_ , 2017  DOI: 10.48550/arXiv.1708.08234
  * [7] Ian Goodfellow et al.  “Generative Adversarial Networks”  In _Advances in Neural Information Processing Systems_ 63.11 New York, NY, USA: Association for Computing Machinery, 2020, pp. 139  DOI: 10.1145/3422622
  * [8] Diederik P Kingma and Max Welling  “Auto-Encoding Variational Bayes”  arXiv, 2013  DOI: 10.48550/arxiv.1312.6114
  * [9] Luke Oliveira, Michela Paganini and Benjamin Nachman  In _Comput. Software Big Sci._ 4.1, 2017  DOI: 10.1007/s41781-017-0004-6
  * [10] Michela Paganini, Luke Oliveira and Benjamin Nachman  “Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multilayer Calorimeters”  In _Phys. Rev. Lett._ 120.4, 2018, pp. 042003  DOI: 10.1103/PhysRevLett.120.042003
  * [11] Michela Paganini, Luke Oliveira and Benjamin Nachman  “CaloGAN: Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks”  In _Phys. Rev. D_ 97.1, 2018, pp. 014021  DOI: 10.1103/PhysRevD.97.014021
  * [12] Martin Erdmann, Lukas Geiger, Jonas Glombitza and David Schmidt  “Generating and refining particle detector simulations using the Wasserstein distance in adversarial networks”  In _Comput. Softw. Big Sci._ 2.1, 2018, pp. 4  DOI: 10.1007/s41781-018-0008-x
  * [13] Martin Erdmann, Jonas Glombitza and Thorben Quast  “Precise simulation of electromagnetic calorimeter showers using a Wasserstein Generative Adversarial Network”  In _Comput. Softw. Big Sci._ 3.1, 2019, pp. 4  DOI: 10.1007/s41781-018-0019-7
  * [14] F. Carminati et al.  “Three dimensional Generative Adversarial Networks for fast simulation”  In _J. Phys. Conf. Ser._ 1085.3, 2018, pp. 032016  DOI: 10.1088/1742-6596/1085/3/032016
  * [15] Georges Aad  “AtlFast3: The Next Generation of Fast Simulation in ATLAS”  In _Comput Softw Big Sci_ 6, 2022, pp. 7  DOI: https://doi.org/10.1007/s41781-021-00079-7
  * [16] Benno Käch et al.  “JetFlow: Generating Jets with Conditioned and Mass Constrained Normalising Flows”, 2022  arXiv:2211.13630 [hep-ex]
  * [17] Benno Käch, Dirk Krücker and Isabell Melzer-Pellmann  “Point Cloud Generation using Transformer Encoders and Normalising Flows”, 2022  arXiv:2211.13623 [hep-ex]
  * [18] Simon Schnake, Dirk Krücker and Kerstin Borras  “Generating Calorimeter Showers as Point Clouds”  In _Machine Learning and the Physical Sciences, Workshop at the 36th conference on Neural Information Processing Systems (NeurIPS)_ , 2022  URL: https://ml4physicalsciences.github.io/2022/files/NeurIPS_ML4PS_2022_77.pdf
  * [19] Franco Scarselli et al.  “The Graph Neural Network Model”  In _IEEE Transactions on Neural Networks_ 20, 2009, pp. 61–80  URL: https://api.semanticscholar.org/CorpusID:206756462
  * [20] Thomas N Kipf and Max Welling  “Semi-supervised classification with graph convolutional networks”  In _IEEE Transactions on Neural Networks._ 5.1, 2016, pp. 61–80  DOI: 10.1109/TNN.2008.2005605
  * [21] Dong Wook Shu, Sung Woo Park and Junseok Kwon  “3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions”, 2019  arXiv:1905.06292 [cs.CV]
  * [22] Raghav Kansal et al.  “JetNet”  Zenodo, 2022  DOI: 10.5281/zenodo.6975118
  * [23] Raghav Kansal  “JetNet150”  Zenodo, 2022  DOI: 10.5281/zenodo.6975117
  * [24] Torbjorn Sjostrand, Stephen Mrenna and Peter Z. Skands  “A Brief Introduction to PYTHIA 8.1”  In _Comput. Phys. Commun._ 178, 2008, pp. 852  DOI: 10.1016/j.cpc.2008.01.036
  * [25] Erik Buhmann  “Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed”, 2020
  * [26] Erik Buhmann et al.  “Decoding Photons: Physics in the Latent Space of a BIB-AE Generative Network”  In _EPJ Web Conf._ 251, 2021, pp. 03003  DOI: 10.1051/epjconf/202125103003
  * [27] Benno Käch and Isabell Melzer-Pellmann  “Attention to Mean-Fields for Particle Cloud Generation”, 2023  arXiv:2305.15254 [hep-ex]
  * [28] Vinicius Mikuni and Benjamin Nachman  “Score-based generative models for calorimeter shower simulation”  In _Phys. Rev. D_ 106.9, 2022, pp. 092009  DOI: 10.1103/PhysRevD.106.092009
  * [29] Vinicius Mikuni, Benjamin Nachman and Mariel Pettee  “Fast Point Cloud Generation with Diffusion Models in High Energy Physics”, 2023  arXiv:2304.01266 [hep-ph]
  * [30] Raghav Kansal et al.  “Particle Cloud Generation with Message Passing Generative Adversarial Networks”, 2022  arXiv:2106.11535 [cs.LG]
  * [31] Raghav Kansal et al.  “Evaluating generative models in high energy physics”  In _Physical Review D_ 107.7 American Physical Society (APS), 2023  DOI: 10.1103/physrevd.107.076017
  * [32] Matthew Leigh et al.  “PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physics”, 2023  arXiv:2303.05376 [hep-ph]
  * [33] Erik Buhmann, Gregor Kasieczka and Jesse Thaler  “EPiC-GAN: Equivariant Point Cloud Generation for Particle Jets”, 2023  arXiv:2301.08128 [hep-ph]
  * [34] Erik Buhmann et al.  “EPiC-ly Fast Particle Cloud Generation with Flow-Matching and Diffusion”, 2023  arXiv:2310.00049 [hep-ph]
  * [35] Alec Radford, Luke Metz and Soumith Chintala  “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, 2016  arXiv:1511.06434 [cs.LG]
  * [36] Shaked Brody, Uri Alon and Eran Yahav  “How Attentive are Graph Attention Networks?”  In _International Conference on Learning Representations_ , 2022  URL: https://openreview.net/forum?id=F72ximsx7C1
  * [37] Matthias Fey and Jan E. Lenssen  “Fast Graph Representation Learning with PyTorch Geometric”  In _ICLR Workshop on Representation Learning on Graphs and Manifolds_ , 2019
  * [38] Juho Lee et al.  “Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks” arXiv:1810.00825 [cs, stat] type: article, 2019  DOI: 10.48550/arXiv.1810.00825
  * [39] Takeru Miyato, Toshiki Kataoka, Masanori Koyama and Yuichi Yoshida  “Spectral Normalization for Generative Adversarial Networks”  In _6th International Conference on Learning Representations_ , 2018  arXiv: https://openreview.net/forum?id=B1QRgziT-
  * [40] Sergey Ioffe and Christian Szegedy  “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, 2015  arXiv:1502.03167 [cs.LG]
  * [41] Jae Hyun Lim and Jong Chul Ye  “Geometric GAN”, 2017  arXiv:1705.02894 [stat.ML]
  * [42] Diederik P. Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015  DOI: 10.48550/arXiv.1412.6980
  * [43] Tim Salimans et al.  “Improved Techniques for Training GANs”, 2016  arXiv:1606.03498 [cs.LG]
  * [44] F. Pedregosa et al.  “Scikit-learn: Machine Learning in Python”  In _Journal of Machine Learning Research_ 12, 2011, pp. 2825–2830
  * [45] Raghav Kansal, Carlos Pareja, Zichun Hao and Javier Duarte  “JetNet: A Python package for accessing open datasets and benchmarking machine learning methods in high energy physics”  In _Journal of Open Source Software_ 8.90, 2023, pp. 5789  DOI: 10.21105/joss.05789