# E(n) Equivariant Topological Neural Networks

Claudio Battiloro 1
George Dasoulas∗,1
Ege Karaismailoğlu∗,1,2
Michelle Audirac1
Mauricio Tec∗,1
Francesca Dominici1
Equal contribution, co-first authors. Corresponding author:
<EMAIL_ADDRESS>
1 Harvard University, 2 ETH Zurich.

###### Abstract

Graph neural networks excel at modeling pairwise interactions, but they cannot
flexibly accommodate higher-order interactions and features. Topological deep
learning (TDL) has emerged recently as a promising tool for addressing this
issue. TDL enables the principled modeling of arbitrary multi-way,
hierarchical higher-order interactions by operating on combinatorial
topological spaces, such as simplicial or cell complexes, instead of graphs.
However, little is known about how to leverage geometric features such as
positions and velocities for TDL. This paper introduces E(n)-Equivariant
Topological Neural Networks (ETNNs), which are E(n)-equivariant message-
passing networks operating on combinatorial complexes, formal objects unifying
graphs, hypergraphs, simplicial, path, and cell complexes. ETNNs incorporate
geometric node features while respecting rotation and translation
equivariance. Moreover, ETNNs are natively ready for settings with
heterogeneous interactions. We provide a theoretical analysis to show the
improved expressiveness of ETNNs over architectures for geometric graphs. We
also show how several E(n) equivariant variants of TDL models can be directly
derived from our framework. The broad applicability of ETNNs is demonstrated
through two tasks of vastly different nature: i) molecular property prediction
on the QM9 benchmark and ii) land-use regression for hyper-local estimation of
air pollution with multi-resolution irregular geospatial data. The experiment
results indicate that ETNNs are an effective tool for learning from diverse
types of richly structured data, highlighting the benefits of principled
geometric inductive bias.

##  1 Introduction

Graph Neural Networks (GNNs) are employed across various fields, including
computational chemistry [1], physics simulations [2], and social networks [3],
to highlight a few. Their effectiveness in tasks with graph-structured data
stems from merging neural network adaptability with insights about data
relationships through the graph topology. Research on GNNs covers a broad
range of approaches, mainly divided into spectral [4] and non-spectral [5]
categories. In both cases, GNNs fundamentally aim to learn representations for
node (and/or) edge attributes through local aggregation driven by the graph
topology, essentially defining message passing networks (MPN) [6]. Utilizing
this capability, GNNs have delivered significant achievements in tasks like
node and graph classification [7], link prediction [8], and specific
challenges such as protein folding [9].

Figure 1: Overview of the $E(n)$ Equivariant Topological Neural Networks
framework. An input geometric graph/point cloud with (possibly) non-geometric
features is provided. A combinatorial complex, whose elements are called
cells, is constructed from the input geometric graph/point cloud to encode
higher order hierarchical interactions, based on topological or domain-
specific considerations. If available, higher order cell features can be
injected. Geometric invariants for the cells (e.g. pairwise distances,
Hausdorff distances, volumes, …) are computed. Finally, $E(n)$ Equivariant
Topological Neural Networks update both geometric and non-geometric features
to improve a downstream task while respecting rotation and translation
equivariance.

However, GNNs usually struggle to model higher-order and multi-way
interactions that the intrinsically pairwise structure of graphs cannot
capture. Consequently, the field of Topological Deep Learning (TDL) [10, 11,
12] started to gain significant interest as it seeks to address these
limitations. Although the general scope of TDL is designing models for data
defined over a variety of topological spaces, the main focus is usually on
combinatorial topological spaces (CTS), i.e. topological spaces that can be
described combinatorially, such as cell complexes [13, 14, 15]. Leveraging
this combinatorial structure, it is possible to design more sophisticated
adjacency schemes among cells, i.e. single nodes or groups of nodes, than the
usual adjacency in graphs. Message passing networks over CTS have been proven
to be more expressive [16, 17, 18, 19, 20] (using the Weisfeiler-Lehman test
criterion [21]), more suitable to handle long-range interactions [22, 23], and
more effective to handle heterophilic data settings [24, 25] than classical
GNNs. Cell complexes are powerful objects to model certain classes of higher-
order interactions along with a hierarchical structure among them, while
general combinatorial objects like hypergraphs are useful to model arbitrary
higher-order interactions, but without any hierarchy. In [11], the notion of
Combinatorial Complex (CC) has been introduced, providing a formal but simple
and flexible object that (pseudo-)generalizes simplicial/path/cell complexes
and hypergraphs, thus enabling the design of arbitrary higher-order
interactions along with a hierarchical structure. Message passing networks
over combinatorial complexes (CCMPNs) have been introduced in the same work
[11].

Little is known about how to integrate and leverage geometric information of
the data in TDL models. In this context, nodes are embedded in a manifold or,
in general, in a metric space, and have geometric features, such as positions
or velocities. In the case of geometric graphs, a common practice is designing
message passing networks able to respect some suitable symmetry, i.e. being
equivariant w.r.t. the action of some symmetry group. Here we focus on $E(n)$
equivariant neural networks, i.e. networks being equivariant w.r.t. rotations
and translations. The $E(n)$ Equivariant Graph Neural Network (EGNN) from [26]
is one of the simplest yet powerful equivariant models for graph-structured
data. The work in [27] generalized the approach from [26] to simplicial
complexes (a particular instance of cell complexes), introducing $E(n)$
Equivariant Message Passing Simplicial Networks (EMPSNs), able to handle graph
or simplicial-structured data. However, both graphs and simplicial complexes
are combinatorial objects that are often not flexible enough to model
arbitrary higher-order interactions, as we will show in the paper. Appendix B
presents a discussion and a detailed review of related works.

Contribution. We generalize the approaches from [26] and [27] to design $E(n)$
Equivariant Topological Neural Networks (ETNNs), i.e. $E(n)$ Equivariant
Message Passing Network over Combinatorial COmplexes. CCs represent a pseudo-
generalization of graphs, simplicial complexes, path complexes, cell
complexes, and hypergraphs, therefore ETNNs represent a legitimate framework
for designing arbitrary $E(n)$ Equivariant Message Passing Networks over
combinatorial topological spaces or, more in general, over combinatorial
objects used to model interactions among entities, such as hierarchical
graphs. An ETNN layer is made of a message passing round over the CC
comprising scalar invariants of the geometric features and a learnable update
of the geometric features themselves added up in the current directions. As
such, ETNNs are a scalarization method [28]. As an additional contribution, we
show, for the first time, how the flexibility of combinatorial complexes can
be leveraged to model molecular and irregular multi-resolution geospatial data
by injecting domain knowledge that cannot be readily exploited via graphs,
simplicial/path/cell complexes, or hypergraphs. We study the improved
expressiveness of ETNNs by evaluating their ability to distinguish
$k$-distinct geometric graphs, showing that they are at least as expressive as
other well-known scalarization-based $E(n)$ equivariant graph neural networks.
Generalizing [11] to the geometric setting, we show how several $E(n)$
equivariant variants of TDL models can be directly derived from our framework.
Finally, we show the efficacy of ETNNs on i) molecular properties prediction
on the QM9 benchmark, and on ii) land-use regression for hyper-local
estimation of air pollution with multi-resolution irregular geospatial data,
representing also a completely new challenging benchmark for TDL models. The
obtained numerical results jointly validate ETNNs, our CC-based approach, and,
in general, the principled injection of geometric information in TDL models.

##  2 Combinatorial Complexes and Equivariance

We start with the fundamentals of combinatorial complexes (CCs), a fairly
general object that provides an effective way to represent complex interaction
systems of various orders in a flexible way. Such structures pseudo-generalize
graphs, simplicial complexes, path complexes, cell complexes, and hypergraphs.
We first define the domain of interest -CCs-, the relations among its elements
-neighborhood functions-, the data defined on it -topological signals-, and a
general class of deep networks operating on it -CC message passing networks
(CCMPNs)-[11]. We finally introduce the formal notion of equivariance, showing
some properties of CCMPNs.

Definition 1. A _combinatorial complex_ (CC) is a triple
$(\mathcal{S},\mathcal{X},\mathrm{rk})$ consisting of a set $\mathcal{S}$, a
subset $\mathcal{X}$ of $\mathcal{P}(S)\backslash\\{\emptyset\\}$, and a
function $\mathrm{rk}:\mathcal{X}\rightarrow\mathbb{Z}_{\geq 0}$ with the
following properties:

  1. 1.

for all $s\in\mathcal{S},\\{s\\}\in\mathcal{X}$;

  2. 2.

the function $\mathrm{rk}$ is order-preserving, i.e. if $x,y\in\mathcal{X}$
satisfy $x\subseteq y$, then $\mathrm{rk}(x)\leq$ $\mathrm{rk}(y)$.

We refer to the elements of $\mathcal{S}$ as nodes, to the elements of
$\mathcal{X}$ as cells, and to $\mathrm{rk}(\cdot)$ as rank function.
$\mathcal{X}$ simplifies notation for $(\mathcal{S},\mathcal{X},\mathrm{rk})$.
The rank function in a CC organizes its relations hierarchically. Therefore,
CCs can model both hierarchical (as in simplicial and cell complexes) and set-
type (as in hypergraphs) relations. For each singleton cell $\\{s\\}$ in a CC,
we set $\mathrm{rk}(\\{s\\})=0$, aligning CCs with simplicial and cell
complexes. The rank of a cell $x\in\mathcal{X}$ is $k:=\mathrm{rk}(x)$, and we
call it a $k$-cell. The dimension $\mathrm{dim}(\mathcal{X})$ of a CC is the
maximal rank among its cells. Finally, we denote the set of $k$-cells in a CC
$\mathcal{X}$ as $\mathcal{D}_{k}=\\{x\in\mathcal{X}:\mathrm{rk}(x)=k\\}$, and
$|\mathcal{D}_{k}|=N_{k}$.

Neighbhorhood functions. Combinatorial Complexes can be equipped with a notion
of neighborhood among cells. In particular, we can define a “neighborhood"
function $\mathcal{N}:\mathcal{X}\rightarrow\mathcal{P}(\mathcal{X})$ on
$\mathcal{X}$ as a function that assigns to each cell $x$ in $\mathcal{X}$ a
nonempty collection of “neighbor cells" $\mathcal{N}(x)\subset\mathcal{X}$.
Usually, two main types of neighborhood functions are considered [11]:
_adjacencies_ , in which all of the elements of $\mathcal{N}(x)$ have the same
rank as $x$, and _incidences_ , in which its elements are from a higher or
lower rank. A generally applicable choice for these functions is as follows.
First, up/down incidences $\mathcal{N}_{I,\uparrow}$ and
$\mathcal{N}_{I,\downarrow}$ are defined by the _containment_ criterion

$\mathcal{N}_{I,\uparrow}(x)=\\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x)+1,x\subset
y\\},\quad\mathcal{N}_{I,\downarrow}(x)=\\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x)-1,y\subset
x\\}.$ (1)

Therefore, a $k+1$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to
$\mathcal{N}_{I,\uparrow}$ if $x$ is contained in $y$; analogously, a
$k-1$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to
$\mathcal{N}_{I,\downarrow}$ if $y$ is contained in $x$. These incidences
induce adjacency functions $\mathcal{N}_{A,\uparrow}$ and
$\mathcal{N}_{A,\uparrow}$ by the _common neighbor_ criterion

$\displaystyle\mathcal{N}_{A,\uparrow}(x)=\\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x),\exists
z\in\mathcal{X}:\mathrm{rk}(z)<\mathrm{rk}(x),z\subset y,\text{ and }z\subset
x\\},$
$\displaystyle\mathcal{N}_{A,\downarrow}(x)=\\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x),\exists
z\in\mathcal{X}:\mathrm{rk}(z)>\mathrm{rk}(x),y\subset z,\text{ and }x\subset
z\\}.$ (2)

Therefore, a $k$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to
$\mathcal{N}_{A,\uparrow}$ if they are both contained in a $k+1$-cell $z$;
analogously, a $k$-cell $y$ is a neighbor of a $k$-cell $x$ w.r.t. to
$\mathcal{N}_{A,\downarrow}$ if they both contain a $k-1$-cell $z$.

While this choice of neighborhood functions applies to any combinatorial
complex, other neighborhood functions are more natural in specific
applications. This scenario will be illustrated in Section 4.

Topological signals A topological signal over $\mathcal{X}$ is defined as a
mapping $f:\mathcal{X}\rightarrow\mathbb{R}$ from the set of cells
$\mathcal{X}$ to real numbers. Therefore, the feature vectors
$\mathbf{h}_{x}\in\mathbb{R}^{F}$ and $\mathbf{h}_{y}\in\mathbb{R}^{F}$ of
cells $x$ and $y$ are a collection of $F$ CC signals, i.e.
$\mathbf{h}_{x}=[f_{1}(x),\dots,f_{F}(x)]$ and
$\mathbf{h}_{y}=[f_{1}(y),\dots,f_{F}(y)]$.

Message passing networks over CCs. Let $\mathcal{CN}$ be a collection of
neighborhood functions. The $l$-th layer of a CC Message Passing Network
(CCMPN) updates the embedding $\mathbf{h}^{l}_{x}$ of cell $x$ as

$\mathbf{h}_{x}^{l+1}=\beta\left(\mathbf{h}^{l}_{x},\bigotimes_{\mathcal{N}\in\mathcal{CN}}\bigoplus_{y\in\mathcal{N}(x)}\psi_{\mathcal{N},\mathrm{rk}(x)}\left(\mathbf{h}^{l}_{x},\mathbf{h}^{l}_{y}\right)\right),$
(3)

where $\mathbf{h}_{x}^{0}:=\mathbf{h}_{x}$ are the initial features,
$\bigoplus$ is an intra-neighborhood permutation invariant aggregator,
$\bigotimes$ is an inter-neighborhood (possibly) permutation invariant
aggregator, and the rank- and neighborhood-dependent message functions
$\psi_{\mathcal{N},\mathrm{rk}(x)}$ and the update function $\beta$ are
learnable functions. In other words, the embedding of a cell is updated in a
learnable fashion through aggregated messages with its neighboring cells over
a set of neighborhoods, generalizing what has been done with Graph MPN [6].
Interestingly. the message functions can be customized to incorporate
additional information, such as the orientation of the cells [10, 29] (if
available), or the embeddings of "common" cells (if present) like $x\cup y$ or
$x\cap y$. Finally, please notice that CCMPN as in (3) are inherently able to
model heterogeneous settings, as two cells can be neighbors in multiple
neighborhoods, but messages among them will be dependent on the neighborhood
itself. Thus, this property allows us to model different relation types
through different (overlapping) neighborhoods. In Appendix C, we show how to
model graphs, simplicial complexes, cell complexes, and hypergraphs through
the CC framework, deriving the corresponding message passing architectures
from the CCMPN in (3).

Equivariances of CCMPNs. Symmetries are formally framed as groups [30]. Let
$G$ be a group with an action $a:G\times\mathcal{Y}\rightarrow\mathcal{Y}$ on
a set $\mathcal{Y}$. Given a function $e:\mathcal{Y}\rightarrow\mathcal{Y}$,
it is said to be equivariant w.r.t. the action of $G$ if, for all
$x\in\mathcal{X}$ and $g\in G$, it holds

$e(a(g,x))=a(g,e(x)).$ (4)

If $e(a(g,x))=e(x)$, $e$ is said to be invariant w.r.t. the action of $G$. As
Graph MPNs [6], CCMPNs are permutation equivariant too, i.e. they are
equivariant w.r.t. the action of the symmetric group
$\mathrm{Sym}(\mathcal{X})$ on $\mathcal{X}$. In other words, CCMPNs are
equivariant to the relabeling of cells in the complex.

##  3 E(n) Equivariant Topological Neural Networks

Let us now introduce the setting in which nodes (0-cells) are embedded in some
Euclidean space, i.e. they come with both non-geometric and geometric
features, such as positions or velocities. In the following, we assume to have
only positions for the sake of exposition, but velocities can be directly
injected in our model and derivations following the same approach from [26,
27]. We denote the non-geometric features with
$\mathbf{h}_{x}\in\mathbb{R}^{F}$ and the positions with
$\mathbf{x}_{x}\in\mathbb{R}^{n}$ for cell $x\in\mathcal{S}$. Since many
problems exhibit $n$-dimensional rotation and translation symmetries, it is
desirable [26, 27, 28] to design models that are equivariant w.r.t. the action
of the $E(n)$ group. An element of the $E(n)$ group is a tuple
$(\mathbf{O},\mathbf{b})$ consisting of an orthogonal matrix
$\mathbf{O}\in\mathbb{R}^{n\times n}$ (the rotation) and a vector
$\mathbf{b}\in\mathbb{R}^{n}$ (the translation). Therefore, the action $a$ of
$E(n)$ on a vector $\mathbf{x}\in\mathbb{R}^{n}$ is given by

$a((\mathbf{O},\mathbf{b}),\mathbf{x})=\mathbf{O}\mathbf{x}+\mathbf{b}.$ (5)

As has been done for graphs [26] and simplicial complexes [27], we represent
equivariance by scalarization. Therefore, geometric vectors undergo an initial
transformation into invariant scalars. Subsequently, they are processed in a
learnable way (e.g., MLPs) before being combined along the original directions
to achieve equivariance.

$E(n)$ Equivariant Topological Neural Networks. Let $\mathcal{X}$ be a CC and
$\mathcal{CN}$ be a collection of neighborhood functions. The $l$-th layer of
an $E(n)$ Equivariant Topological Neural Networks (ETNN) updates the
embeddings $\mathbf{h}^{l}_{x}$ of every cell $x\in\mathcal{X}$ and the
positions $\mathbf{x}_{z}^{l}$ of every node $z\in\mathcal{S}$ as

$\displaystyle\mathbf{h}_{x}^{l+1}=\beta\left(\mathbf{h}^{l}_{x},\bigotimes_{\mathcal{N}\in\mathcal{CN}}\bigoplus_{y\in\mathcal{N}(x)}\underbrace{\psi_{\mathcal{N},\mathrm{rk}(x)}\left(\mathbf{h}^{l}_{x},\mathbf{h}^{l}_{y},\mathrm{Inv}\left(\\{\mathbf{x}_{z}^{l}\\}_{z\in
x},\\{\mathbf{x}_{z}^{l}\\}_{z\in
y}\right)\right)}_{\mathbf{m}_{x,y}^{\mathcal{N}}}\right),\text{ for all
}x\in\mathcal{X}$ (6)
$\displaystyle\mathbf{x}_{z}^{l+1}=\mathbf{x}_{z}^{l}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in
S:\\{t\\}\in\mathcal{N}(z)}\left(\mathbf{x}_{z}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right),\text{
for all }z\in\mathcal{S},$ (7)

where $\mathbf{h}_{x}^{0}=\mathbf{h}_{x}$,
$\mathbf{x}_{z}^{0}=\mathbf{x}_{z}$, and $\xi$ is a scalar learnable function.
$\mathrm{Inv}$ takes as input the positions of the nodes forming the involved
cells, and it is an invariant function w.r.t. the action of $E(n)$, i.e.
$\mathrm{Inv}(\\{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\\}_{z\in
x},\\{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\\}_{z\in
y})=\mathrm{Inv}(\\{\mathbf{x}_{z}^{l}\\}_{z\in
x},\\{\mathbf{x}_{z}^{l}\\}_{z\in y})$. We refer to a specific $\mathrm{Inv}$
as a geometric invariant. The update in (7) naturally allows for neighborhood
heterogeneities: two nodes can be neighbors via more than a single
neighborhood function, carrying different messages. This is common in many
applications, as illustrated in Section 4.

The following theorem demonstrates the equivariance property of ETNNs.

###### Theorem 1 .

An ETNN layer as in (6)-(7), synthetically denoted as
$\\{\mathbf{h}_{x}^{l+1}\\}_{x\in\mathcal{X}},\\{\mathbf{x}_{z}^{l+1}\\}_{z\in\mathcal{S}}=\mathrm{ETNN}\left(\\{\mathbf{h}_{x}^{l}\\},\\{\mathbf{x}_{z}^{l}\\}_{z\in
x}\right)$, is $E(n)$ equivariant, i.e. it holds

$\\{\mathbf{h}_{x}^{l+1}\\}_{x\in\mathcal{X}},\\{\mathbf{O}\mathbf{x}_{z}^{l+1}+\mathbf{b}\\}_{z\in
x}=\mathrm{ETNN}\left(\\{\mathbf{h}_{x}^{l}\\}_{x\in\mathcal{X}},\\{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\\}_{z\in
x}\right),$ (8)

for all $(\mathbf{O},\mathbf{b})\in E(n)$.

###### Proof.

See Appendix D. ∎

When only (6) is applied but the update step in (7) is not performed, then
ETNNs are $E(n)$-invariant. At this point, it is important to discuss which
geometric invariants should be employed. As has been shown in [26, 27],
geometric invariants should make use of the underlying topological structure.
For instance, the pairwise distance of connected nodes has been used in EGNN
[26], while the simplex volume has been used in EMPSN [27]. However, both EGNN
and EMPSN operate on rigid domains, such as simplicial complexes, while CCs
can have a much more arbitrary set of relations. In the following, we present
a series of geometric invariants that can be used in ETNNs, without assuming
any specific structure for the CC. We consider two cells $x$ and $y$, along
their corresponding node positions $\\{\mathbf{x}_{z}\\}_{z\in x}$ and
$\\{\mathbf{x}_{z}\\}_{z\in y}$.

Permutation invariant functions of pairwise distances. If $x$ and $y$ have the
same cardinality, any function of the form

$\bigoplus_{z\in x,t\in y}\tau\left(\|\mathbf{x}_{z}-\mathbf{x}_{t}\|\right)$
(9)

is a geometric invariant, with $\bigoplus$ being a permutation invariant
aggregator, and $\tau$ a (possibly) learnable function. A basic instance is
$\bigoplus=\sum$ and $\tau=\mathrm{Id}$, i.e. the sum of pairwise distances.

Distances of permutation invariant functions. Regardless of the cardinalities
of $x$ and $y$, any function of the form

$\tau\left(\Bigg{\|}\bigoplus_{z\in x}\mathbf{x}_{z}-\bigoplus_{t\in
y}\mathbf{x}_{t}\Bigg{\|}\right)$ (10)

is a geometric invariant, with $\bigoplus$ being a linear permutation
invariant aggregator, and $\tau$ a (possibly) learnable function. A basic
instance is $\bigoplus=\frac{1}{|\cdot|}\sum$ and $\tau=\mathrm{Id}$, i.e. the
distance between centroids.

Hausdorff distance. Regardless of the cardinalities of $x$ and $y$, the
Hausdorff distance, defined as

$\max\left\\{\max_{z\in x}\min_{t\in
y}\|\mathbf{x}_{z}-\mathbf{x}_{t}\|,\max_{t\in y}\min_{z\in
x}\|\mathbf{x}_{t}-\mathbf{x}_{z}\|\right\\}$ (11)

is a geometric invariant. The Hausdorff distance measures the mutual proximity
of two sets whose elements live in a metric space. It does so by computing the
maximal distance between any point of one set to the other set. Moreover, the
two $\max\min$ terms inside the outer $\max$, usually referred to as directed
Hausdorff distances, can be used as geometric invariants themselves. This
geometric invariant has been used, so far, in various machine learning
applications, especially in the area of computer vision [31, 32].

Volume of the convex hull. Regardless of the cardinalities of $x$ and $y$, any
function of the volumes of the convex hulls of $\\{\mathbf{x}_{z}\\}_{z\in x}$
and $\\{\mathbf{x}_{z}\\}_{z\in y}$ is a geometric invariant. Computing the
volume of a convex hull is quite an expensive procedure, thus it is not always
suitable to be performed at each training step. However, it can still be used
if the cells have small cardinality, or if ETNNs are used in an invariant
fashion, i.e. without the update in (7), because the volume needs to be
computed only once before training. Finally, the volume of convex hulls
reduces to the volume of simplices in case the considered CC is a (geometric)
simplicial complex, generalizing what has been done in [27].

###### Remark 1 .

It is noteworthy that if higher rank cells, not only nodes, come with attached
geometric features, they can be seamlessly integrated into ETNNs. In that
case, there would be geometric invariants of node geometric features,
geometric invariants of higher-rank geometric features, and, possibly,
geometric invariants of both nodes and higher rank cells. The equivariant
update in (7) would be straightforwardly modified to include all the cells
with geometric features in the sum.

Expressiveness of ETNN. The expressive power of neural networks over CTS is
proportional to their ability to distinguish non-isomorphic CTSs. As the
renowned Weisfeiler-Leman (WL) test for GNNs [21], several generalized WL
tests have been developed for CTSs (e.g. simplicial [16], cell [18], or path
[19] complexes) and have been shown to be upper bounds on the expressiveness
of the corresponding TDL models. These tools also demonstrated that TDL models
are usually more expressive than vanilla GNNs when it comes to distinguishing
isomorphic graphs. For geometric graphs, the Geometric-WL (GWL) test [33] has
been recently introduced as an upper bound on the expressiveness of
invariant/equivariant geometric GNNs. In Appendix E, we introduce the notion
of Geometric Augmented Hasse Graph and we combine it with the GWL to show that
ETNNs are, in general, more expressive than scalarization-based graph methods.
We also perform synthetic experiments to validate our claim. Next, we present
an informal statement of our result, while in Appendix E we provide the formal
statement and the proof.

Proposition 2 . (Informal) An ETNN is at least as powerful as an EGNN [26] in
distinguishing $k$-hop distinct graphs. In most of the cases, an ETNN is
strictly more powerful than an EGNN. The quantitative gap between ETNNs and
EGNNs is dependent on how the original graphs are lifted into a CC, i.e. how
the set of cells $\mathcal{X}$ and the collection of neighborhood functions
$\mathcal{CN}$ are defined.

Formal Statement and Proof . See Appendix E.

Computational complexity of ETNN. In Appendix F, we present a brief analysis
of the computational complexity of ETNNs.

Figure 2: Combinatorial complexes at various resolutions. _(Left)_ : A
molecular CC where 0-cells are atoms, 1-cells are bonds, and 2-cell repersent
rings and functional groups. _(Right)_ : A geospatial CC where 0-cells are
grid points, 1-cells are road polylines, and 2-cells are census tract
polygons. For visibility, the geospatial CC only shows one road (rank 1) and
its incident lower and higher rank cells.

##  4 Real-World Combinatorial Complexes

Designing equivariant models for CCs tailored for specific applications is an
exciting direction since CCs can accommodate hierarchical higher-order
features that cannot be readily incorporated in the literature’s most used
CTSs, including graphs, simplicial complexes, path complexes, cell complexes,
and hypergraphs111Appendix C provides overview of the aforementioned CTSs.,
This section presents two vastly different applications. The first application
considers molecular data. While this data has been investigated under the lens
of CTS, our approach allows us to naturally integrate their higher-order
structures such as rings and functional groups. Next, we present a novel TDL
framework for geospatial data. We show that the interactions between
geographical objects in geospatial data such as polylines and polygons can be
modeled via CCs.

Molecular combinatorial complexes. We introduce the notion of Molecular
Combinatorial Complexes, showing that they can jointly retain all the
properties of graphs, cell complexes, and hypergraphs, summarized in Appendix
G, thus overcoming their mutual limitations. In a molecular CC, the set of
vertices $\mathcal{S}$ is the set of atoms, while the set of cells
$\mathcal{X}$ and the rank function $\mathrm{rk}$ are such that: atoms are
$0$-cells, bonds and functional groups made of two atoms are $1$-cells, and
rings and functional groups made of more than two atoms are $2$-cells (see
Figure 2). In the next section, we will use molecular CCs to test our ETNN on
a molecular properties prediction task.

Geospatial combinatorial complexes. We now introduce the notion of Geospatial
Combinatorial Complexes. Points, polylines, and polygons constitute some of
the most basic geometric objects in geospatial data and are widely used
throughout geographic information systems. These objects can be modeled as
cells of a CC and their rank correspond to their intrinsic dimensionality.
Fig. 2. illustrates this scenario, where 0-cells are points of a city with air
quality monitoring units, 1-cells are road polylines, and 2-cells are census
tract polygons, a type of geographic area with homogeneous demographic
characteristics. A CC like this is an example of a _Geospatial CC_ ,
characterized by a spatial representation and a neighborhood system determined
by spatial proximity. In general, and formally, let
$\mathcal{T}\subset\mathbb{R}^{n}$ represent a geospatial domain represented
as a compact subset of Euclidean space. Then, there exists a function
$s:\mathcal{X}\to\mathcal{T}$ mapping each cell to its geographic space
$s(x)\subset\mathcal{T}$. Cells of a geospatial CC have a dual view. From the
CC perspective, a cell $x$ such as a road (polyline) or census tract (polygon)
is represented by the points it contains. On the other hand, neighborhood
functions can be constructed from the geographic space $s(x)$ corresponding to
$x$, e.g.:

$\displaystyle\mathcal{N}_{A}(x)=\\{y\in\mathcal{X}\mid\mathrm{rk}(y)=\mathrm{rk}(x),s(x)\cap
s(y)\neq\emptyset\\}$
$\displaystyle\mathcal{N}_{I,\uparrow}(x)=\\{y\in\mathcal{X}\mid\mathrm{rk}(y)=\mathrm{rk}(x)+1,s(x)\cap
s(y)\neq\emptyset\\}$
$\displaystyle\mathcal{N}_{I,\downarrow}(x)=\\{y\in\mathcal{X}\mid\mathrm{rk}(y)=\mathrm{rk}(x)-1,s(x)\cap
s(y)\neq\emptyset\\}.$ (12)

For instance, $\mathcal{N}_{A}(x)$ can represent adjacencies between
geographic spaces that share a portion or a boundary, while the up and down
incidences $\mathcal{N}_{I,\uparrow}(x)$ and $\mathcal{N}_{I,\downarrow}(x)$
can capture the notion of road crossing a county, or two geographic areas of
different rank overlapping. Evidently, more than one neighborhood function can
be used or combined with the standard choices in (1) -(2) according to the
application. It is not always necessary to use points and polylines. One can
also consider geographic divisions (polygons) with multiple resolution levels,
each mapping to a different rank. For example, in the United States, census
tracts are smaller than zip codes, which are smaller than counties, and so on.
A census tract can intersect multiple zip codes, and zip codes can belong to
multiple counties, partitioning space irregularly. Similarly, Fig. 2 portrays
an example where a road polyline traverses multiple census tract polygons.

Unlike classical multi-resolution spatial modeling [34, 35], we don’t require
regular structures such as grids or nested partitions. Importantly, we allow
each resolution to have unique features. This point illustrates a crucial
difference and advantage over graph modeling, which requires all data to be
pre-aggregated at the same rank (typically the 0-cells), see Appendix G. We
are the first to explore combinatorial topological modeling of multi-
resolution irregular spatial data. With this formulation, the dataset, and the
task that we will present in the next section, we not only propose a novel
technique for processing and learning from irregular multi-resolution
geospatial data, but we also set a novel benchmark for TDL-based hyper-local
air pollution downscaling.

##  5 Experiments

Molecular property prediction. In this study, we evaluate our model on
molecular property prediction using the QM9 dataset [36]. The QM9 dataset is a
comprehensive collection of quantum chemical calculations for small organic
molecules. It includes geometric, energetic, electronic, and thermodynamic
properties for 134k stable molecules with up to nine heavy atoms (C, O, N, F).
See Appendix H.1.1 for a summary.

We consider several different variants of ETNNs (see Appendix I), but the most
general molecular combinatorial complexes (CCs) that we design is the one as
in Fig. 2. It employs three ranks to capture different levels of molecular
interactions: i) atoms as 0-cells, ii) bonds, i.e. pairs of atoms, as 1-cells,
and iii) rings and functional groups of size greater than two, i.e. at least
triplets of atoms, as 2-cells. Similar to methodologies for global connections
(e.g. EGNN [26] and VN-EGNN [37]), we generalize the notion of virtual node by
incorporating a single cell, that we refer to as virtual cell, containing the
whole set of 0-cells (the atoms), and having maximum rank, e.g. in the most
general case it would have rank 3. As neighborhood functions, we employed the
up/down adjacencies and incidences from (1)-(2), and a "max-rank" adjacency
defined as

$\mathcal{N}_{A,\textrm{max}}(x)=\\{y\in\mathcal{X}|\mathrm{rk}(y)=\mathrm{rk}(x),\exists
z\in\mathcal{X}:\mathrm{rk}(z)=\underset{z\in\mathcal{X}}{\textrm{max
}\mathrm{rk}(z)},y\subset z,\text{ and }y\subset z\\}$ (13)

The virtual cell together with the max-rank adjacency in (13) acts as a hub
across all cells of the same rank, connecting each pair. This setup ensures
that the complex interactions within molecules are effectively captured. In
this setting, ETNN naturally handles heterogeneity, e.g., the same pair of
bonds could be connected because part of the same functional group (up
adjacency) and the virtual cell (max-rank adjacency), but the two messages
will be different across the neighborhoods. Similarly, ETNN allows us to equip
each rank with unique feature spaces without the need for ad-hoc aggregations
at the atom or bond levels. Here, we consider four distinct types of non-
geometric feature vectors: atom features, bond features, ring features, and
functional group features. We provide a full list of the non-geometric feature
vectors we used in Appendix H.1.2. As geometric features, we use the atoms’
positions. The complete list of the model parameters is in Appendix H.1.3. A
description of all the configurations and a detailed reproducibility
discussion is presented in Appendix I. In all the experiments, following [26,
27], we use the invariant version of ETNN, but we keep the name ETNN for the
sake of exposition.

Table 1: Mean Absolute Error for the molecular property prediction benchmark
in QM9 dataset (Part 1). ETNN-* corresponds to the configuration, among the
ones we tested, that produced the best results for each property. ETNN-graph-W
is EGNN [26] derived from the ETNN framework with our codebase. All the models
except ETNN-w/o VC use the virtual cell. We report the improvement of the best
ETNN over the original EGNN. Details and full results for all $12$ properties
are described in Appendix I. Baseline results are taken from [26].

Task | $\alpha$ | $\Delta\varepsilon$ | $\varepsilon_{\mathrm{HOMO}}$ | $\varepsilon_{\mathrm{LUMO}}$ | $\mu$ | $C_{\nu}$
---|---|---|---|---|---|---
Units ($\downarrow$) | bohr3 | meV | meV | meV | D | cal/mol K
NMP | .092 | 69 | 43 | 38 | .030 | .040
Schnet | .235 | 63 | 41 | 34 | .033 | .033
Cormorant | .085 | 61 | 34 | 38 | .038 | .026
L1Net | .088 | 68 | 46 | 35 | .043 | .031
LieConv | .084 | 49 | 30 | 25 | .032 | .038
DimeNet++* | .044 | 33 | 25 | 20 | .030 | .023
TFN | .223 | 58 | 40 | 38 | .064 | .101
SE(3)-Tr. | .142 | 53 | 35 | 33 | .051 | .054
EGNN | .071 | 48 | 29 | 25 | .029 | .031
ETNN |  |  |  |  |  |
(ETNN-*) | .062 | 48 | 30 | 24 | .023 | .033
(ETNN-graph-W) | .067 | 46 | 27 | 25 | .030 | .036
(ETNN-w/o VC) | .161 | 73 | 49 | 48 | .306 | .051
Improvement | -13% | 0 % | +3% | -4% | -21% | +5%

In Table 1, we present ETNN’s best results for 6 of the 12 molecular
properties. ETNN improves the prediction performance of EGNN for 3 out of
these 6 targets, with notable improvements on $\alpha$ and $\mu$, achieving
state-of-the-art performance for $\mu$. Additionally, the table shows the
results of a 1.5M EGNN obtained through our framework and codebase (EGNN-
graph-W, as explained in Appendix I. Comparing this with the original EGNN, we
observe performance improvements, even though the original EGNN was trained
for 1000 epochs, while our configurations were run for only 350 epochs.
Finally, the low prediction performance of ETNN-w/o VC (ETNN without the
virtual cell) highlights the significant impact of the virtual cell, as was
found in [26, 37, 27].

Hyperlocal air pollution downscaling. We introduce a novel benchmark for TDL
based on the geospatial framework introduced in Section 4. We provide an
overview of the data and tasks for completeness. Our benchmark dataset
includes additional documentation and it is publicly available in our
repository. The task consists of predicting $\textrm{PM}_{2.5}$ air pollution
at a high resolution of $0.0002^{\circ}\approx 22m$. The prediction targets of
$\textrm{PM}_{2.5}$ measurements are obtained from a publicly available
dataset [38] consisting of measurements by mobile air sensors installed on
cars. The data was collected during the last quarter of 2021 in the Bronx, NY,
USA. The targets are aggregated at the desired resolution of $0.0002^{\circ}$.
We map each point of the roads and census tracts, and create a geospatial CC
using the neighborhood functions described by Section 4. An overview of the
features for each rank is given in Fig. 3. We again employ a virtual cell as
defined in the previous paragraph. See Section G.2 for additional details.

Table 2: Root Mean Square Error for _air pollution downscaling_ benchmark.

Baseline | RMSE
---|---
Linear | 1.040
MLP | 0.954
GNN | 0.972
EGNN | 0.979
ETNN | 0.939

(a) Baseline model comparison

Ablation | Change
---|---
No virtual cell | $-1.06\%$
No position update (invariant ETNN) | $-0.43\%$
No geometric features (CCMPN) | $-0.21\%$

(b) Ablation study results

The experiment results are shown in Table 2 (a). Since only ETNN can support
multi-level features, we create point-level features for MLP, GNN, and EGNN by
concatenating the features of each point with the features of the 1-cell and
2-cell that the point belongs to. The results are averaged over 3 seeds. Refer
to Section H.2 for additional details on the training procedure.

The table indicates that ETNNs outperform EGNN, GNN, and MLP, emphasizing the
benefits of multi-level modeling combined with $E(n)$ equivariance. We
additionally conduct ablations studies on ETNN, presented in Table 2 (b). The
results show that, as expected, not including geometric features decreases
performance. Perhaps surprisingly, the ablation of ETNN using only the
geospatial CC structure without geometric features (invariants) performed
slightly better than using the features but not using the position update in
(7), i.e. using the invariant version of ETNN. Nonetheless, the largest
decrease came from not using the virtual cell.

Figure 3: Overview of features used in the _air pollution downscaling_
benchmark.The task is to predict the local $\mathrm{PM}_{2.5}$ measured at the
0-cells, which have an approximate resolution of $22\times 22$ meters.

##  6 Conclusion

We introduced E(n) Equivariant Topological Neural Networks (ETNNs), a
framework for designing scalarization-based equivariant message passing
networks on combinatorial complexes. Combinatorial complexes flexibly
represent arbitrary hierarchical higher-order interactions, thus ETNNs are a
legitimate proxy for designing equivariant models on a wide class of
combinatorial topological spaces, such as simplicial or cell complexes, or,
more in general, over combinatorial objects used to model interactions among
entities, such as hierarchical graphs. We showed that ETNNs are at least as
expressive as existing scalarization-based equivariant architectures for
geometric graphs. Experiments on the QM9 molecular benchmark demonstrated
ETNNs’ effectiveness for richly structured molecular data. Moreover, we
introduced a novel geographic regression task, highlighting ETNNs’ ability and
showing for the first time how combinatorial complexes can be used to model
irregular multi-resolution geospatial data. We discuss the limitations and
future directions in Appendix A. ETNNs represent, overall, a unifying
framework showcasing the benefits of a principled synergy between
combinatorial and geometric equivariances over beyond-graph domains.

## References

  * [1] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl,  “Neural message passing for quantum chemistry,”  in International Conference on Machine learning, 2017.
  * [2] Jonathan Shlomi, Peter Battaglia, and Jean-Roch Vlimant,  “Graph neural networks in particle physics,”  Machine Learning: Science and Technology, vol. 2, no. 2, pp. 021001, dec 2020.
  * [3] Wenwen Xia, Yuchen Li, Jun Wu, and Shenghong Li,  “Deepis: Susceptibility estimation on social networks,”  in Proceedings of the 14th ACM International Conference on Web Search and Data Mining, New York, NY, USA, 2021, WSDM ’21, p. 761–769, Association for Computing Machinery.
  * [4] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun,  “Spectral networks and locally connected networks on graphs,”  in International Conference on Learning Representations, Banff, Canada, 2014.
  * [5] M. Gori, G. Monfardini, and F. Scarselli,  “A new model for learning in graph domains,”  in IEEE International Joint Conference on Neural Networks, 2005.
  * [6] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl,  “Neural message passing for quantum chemistry,”  in Proceedings of the 34th International Conference on Machine Learning - Volume 70. 2017, ICML’17, p. 1263–1272, JMLR.org.
  * [7] T. N. Kipf and M. Welling,  “Semi-Supervised Classification with Graph Convolutional Networks,”  in International Conference on Learning Representations, 2017.
  * [8] M. Zhang and Y. Chen,  “Link prediction based on graph neural networks,”  Advances in Neural Information Processing Systems, 2018.
  * [9] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al.,  “Highly accurate protein structure prediction with alphafold,”  Nature, 2021.
  * [10] S. Barbarossa and S. Sardellitti,  “Topological signal processing over simplicial complexes,”  IEEE Transactions on Signal Processing, 2020.
  * [11] Mustafa Hajij, Ghada Zamzmi, Theodore Papamarkou, Nina Miolane, Aldo Guzmán-Sáenz, Karthikeyan Natesan Ramamurthy, Tolga Birdal, Tamal K. Dey, Soham Mukherjee, Shreyas N. Samaga, Neal Livesay, Robin Walters, Paul Rosen, and Michael T. Schaub,  “Topological deep learning: Going beyond graph data,” 2023.
  * [12] Theodore Papamarkou, Tolga Birdal, Michael Bronstein, Gunnar Carlsson, Justin Curry, Yue Gao, Mustafa Hajij, Roland Kwitt, Pietro Liò, Paolo Di Lorenzo, et al.,  “Position paper: Challenges and opportunities in topological deep learning,”  arXiv preprint arXiv:2402.08871, 2024.
  * [13] L. J. Grady and J. R. Polimeni,  Discrete calculus: Applied analysis on graphs for computational science,  Springer, 2010.
  * [14] Cristian Bodnar,  Topological Deep Learning: Graphs, Complexes, Sheaves,  Ph.D. thesis, Cambridge University, 2023.
  * [15] Claudio Battiloro,  Signal Processing and Learning over Topological Spaces,  Ph.D. thesis, Sapienza Unviersity of Rome, 2024.
  * [16] C. Bodnar, F. Frasca, Y. Wang, N. Otter, G.F. Montufar, P. Liò, and M. Bronstein,  “Weisfeiler and lehman go topological: Message passing simplicial networks,”  in International Conference on Machine Learning, 2021.
  * [17] Claudio Battiloro, Lucia Testa, Lorenzo Giusti, Stefania Sardellitti, Paolo Di Lorenzo, and Sergio Barbarossa,  “Generalized simplicial attention neural networks,”  arXiv preprint arXiv:2309.02138, 2023.
  * [18] C. Bodnar, F. Frasca, N. Otter, Y. Wang, P. Liò, G. F. Montufar, and M. Bronstein,  “Weisfeiler and lehman go cellular: Cw networks,”  in Advances in Neural Information Processing Systems, 2021.
  * [19] Quang Truong and Peter Chin,  “Weisfeiler and lehman go paths: Learning topological features via path complexes,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, vol. 38, pp. 15382–15391.
  * [20] Fabian Jogl, Maximilian Thiessen, and Thomas Gärtner,  “Expressivity-preserving GNN simulation,”  in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [21] K. Xu, W. Hu, J. Leskovec, and S. Jegelka,  “How powerful are graph neural networks?,”  in International Conference on Learning Representations, 2019.
  * [22] Lorenzo Giusti, Teodora Reu, Francesco Ceccarelli, Cristian Bodnar, and Pietro Liò,  “Cin++: Enhancing topological message passing,”  arXiv preprint arXiv:2306.03561, 2023.
  * [23] Lorenzo Giusti, Claudio Battiloro, Lucia Testa, Paolo Di Lorenzo, Stefania Sardellitti, and Sergio Barbarossa,  “Cell attention networks,”  in 2023 International Joint Conference on Neural Networks (IJCNN). IEEE, 2023, pp. 1–8.
  * [24] Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Lio, and Michael M. Bronstein,  “Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs,”  in Advances in Neural Information Processing Systems, 2022.
  * [25] Jakob Hansen and Thomas Gebhart,  “Sheaf neural networks,” 2020.
  * [26] Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling,  “E (n) equivariant graph neural networks,”  in International conference on machine learning. PMLR, 2021, pp. 9323–9332.
  * [27] Floor Eijkelboom, Rob Hesselink, and Erik J Bekkers,  “E $(n)$ equivariant message passing simplicial networks,”  in International Conference on Machine Learning. PMLR, 2023, pp. 9071–9081.
  * [28] Jiaqi Han, Yu Rong, Tingyang Xu, and Wenbing Huang,  “Geometrically equivariant graph neural networks: A survey,”  arXiv preprint arXiv:2202.07230, 2022.
  * [29] S. Sardellitti, S. Barbarossa, and L. Testa,  “Topological signal processing over cell complexes,”  in Asilomar Conference on Signals, Systems, and Computers, 2021.
  * [30] M.M. Bronstein, J. Bruna, T. Cohen, and P. Veličković,  “Geometric deep learning: Grids, groups, graphs, geodesics, and gauges,”  arXiv preprint arXiv:2104.13478, 2021.
  * [31] Orhun Utku Aydin, Abdel Aziz Taha, Adam Hilbert, Ahmed A. Khalil, Ivana Galinovic, Jochen B. Fiebach, Dietmar Frey, and Vince Istvan Madai,  “On the usage of average hausdorff distance for segmentation performance assessment: hidden error when used for ranking,”  European Radiology Experimental, vol. 5, no. 1, Jan. 2021.
  * [32] Marc van Kreveld, Tillmann Miltzow, Tim Ophelders, Willem Sonke, and Jordi L. Vermeulen,  “Between shapes, using the hausdorff distance,”  Computational Geometry, vol. 100, pp. 101817, 2022.
  * [33] Chaitanya K Joshi, Cristian Bodnar, Simon V Mathis, Taco Cohen, and Pietro Lio,  “On the expressive power of geometric graph neural networks,”  in International Conference on Machine Learning. PMLR, 2023, pp. 15330–15355.
  * [34] Eric D. Kolaczyk and Haiying Huang,  “Multiscale statistical models for hierarchical spatial aggregation,”  Geographical Analysis, vol. 33, no. 2, pp. 95–118, 2001.
  * [35] Tomislav Hengl, Matthew A. E. Miller, Josip Križan, Keith D. Shepherd, Andrew Sila, Milan Kilibarda, Ognjen Antonijević, Luka Glušica, Achim Dobermann, Stephan M. Haefele, Steve P. McGrath, Gifty E. Acquah, Jamie Collinson, Leandro Parente, Mohammadreza Sheykhmousa, Kazuki Saito, Jean-Martial Johnson, Jordan Chamberlin, Francis B. T. Silatsa, Martin Yemefack, John Wendt, Robert A. MacMillan, Ichsani Wheeler, and Jonathan Crouch,  “African soil properties and nutrients mapped at 30 m spatial resolution using two-scale ensemble machine learning,”  Scientific Reports, vol. 11, no. 1, Mar. 2021.
  * [36] Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld,  “Quantum chemistry structures and properties of 134 kilo molecules,”  Scientific Data, vol. 1, no. 1, Aug. 2014.
  * [37] Florian Sestak, Lisa Schneckenreiter, Johannes Brandstetter, Sepp Hochreiter, Andreas Mayr, and Günter Klambauer,  “Vn-egnn: E (3)-equivariant graph neural networks with virtual nodes enhance protein binding site identification,”  arXiv preprint arXiv:2404.07194, 2024.
  * [38] An Wang, Simone Mora, Yuki Machida, Priyanka deSouza, Sanjana Paul, Oluwatobi Oyinlola, Fábio Duarte, and Carlo Ratti,  “Hyperlocal environmental data with a mobile platform in urban environments,”  Scientific data, vol. 10, no. 1, pp. 524, 2023.
  * [39] Cong Liu, David Ruhe, Floor Eijkelboom, and Patrick Forré,  “Clifford group equivariant simplicial message passing networks,”  in The Twelfth International Conference on Learning Representations, 2024.
  * [40] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla,  “Heterogeneous graph neural network,”  in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 793–803.
  * [41] Kelly Maggs, Celia Hacker, and Bastian Rieck,  “Simplicial representation learning with neural $k$-forms,”  in The Twelfth International Conference on Learning Representations, 2024.
  * [42] M. T. Schaub, Y. Zhu, J.B. Seby, T. M. Roddenberry, and S. Segarra,  “Signal processing on higher-order networks: Livin’on the edge… and beyond,”  Signal Processing, 2021.
  * [43] T. Mitchell Roddenberry, Michael T. Schaub, and Mustafa Hajij,  “Signal processing on cell complexes,”  in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 8852–8856.
  * [44] S. Ebli, M. Defferrard, and G. Spreemann,  “Simplicial neural networks,”  in Advances in Neural Information Processing Systems Workshop on Topological Data Analysis and Beyond, 2020.
  * [45] Ruochen Yang, Frederic Sala, and Paul Bogdan,  “Efficient representation learning for higher-order data with simplicial complexes,”  in Proceedings of the First Learning on Graphs Conference, Bastian Rieck and Razvan Pascanu, Eds. 09–12 Dec 2022, vol. 198 of Proceedings of Machine Learning Research, pp. 13:1–13:21, PMLR.
  * [46] Mustafa Hajij, Kyle Istvan, and Ghada Zamzmi,  “Cell complex neural networks,”  in Advances in Neural Information Processing Systems Workshop on TDA & Beyond, 2020.
  * [47] Maosheng Yang and Elvin Isufi,  “Convolutional learning on simplicial complexes,” 2023.
  * [48] T. M. Roddenberry, N. Glaze, and S. Segarra,  “Principled simplicial neural networks for trajectory prediction,”  in International Conference on Machine Learning, 2021.
  * [49] M. Hajij, G. Zamzmi, T. Papamarkou, N. Miolane, A. Guzmán-Sáenz, and K. N. Ramamurthy,  “Higher-order attention networks,” 2022.
  * [50] L. Giusti, C. Battiloro, P. Di Lorenzo, S. Sardellitti, and S. Barbarossa,  “Simplicial attention neural networks,” 2022.
  * [51] C. Wei Jin Goh, C. Bodnar, and P. Lio,  “Simplicial attention networks,”  in International Conference on Learning Representations Workshop on Geometrical and Topological Representation Learning, 2022.
  * [52] J. Hansen and R. Ghrist,  “Toward a spectral theory of cellular sheaves,”  Journal of Applied and Computational Topology, 2019.
  * [53] C. Battiloro, Z. Wang, H. Riess, P. Di Lorenzo, and A. Ribeiro,  “Tangent bundle filters and neural networks: From manifolds to cellular sheaves and back,”  in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5.
  * [54] Claudio Battiloro, Zhiyang Wang, Hans Riess, Paolo Di Lorenzo, and Alejandro Ribeiro,  “Tangent bundle convolutional learning: from manifolds to cellular sheaves and back,”  IEEE Transactions on Signal Processing, 2024.
  * [55] Federico Barbero, Cristian Bodnar, Haitz Sáez de Ocáriz Borde, Michael Bronstein, Petar Veličković, and Pietro Liò,  “Sheaf neural networks with connection laplacians,” 2022.
  * [56] Sravanthi Gurugubelli and Sundeep Prabhakar Chepuri,  “SaNN: Simple yet powerful simplicial-aware neural networks,”  in The Twelfth International Conference on Learning Representations, 2024.
  * [57] Bohan Tang, Siheng Chen, and Xiaowen Dong,  “Hypergraph-mlp: learning on hypergraphs without message passing,”  in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 13476–13480.
  * [58] Claudio Battiloro, Indro Spinelli, Lev Telyatnikov, Michael M. Bronstein, Simone Scardapane, and Paolo Di Lorenzo,  “From latent graph to latent topology inference: Differentiable cell complex module,”  in The Twelfth International Conference on Learning Representations, 2024.
  * [59] Mathilde Papillon, Sophia Sanborn, Mustafa Hajij, and Nina Miolane,  “Architectures of topological deep learning: A survey on topological neural networks,” 2023.
  * [60] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in IEEE conference on computer vision and pattern recognition, 2016.
  * [61] Taco Cohen and Max Welling,  “Group equivariant convolutional networks,”  in International conference on machine learning. PMLR, 2016, pp. 2990–2999.
  * [62] Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling,  Equivariant and Coordinate Independent Convolutional Networks,  2023\.
  * [63] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley,  “Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds,”  arXiv preprint arXiv:1802.08219, 2018.
  * [64] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling,  “Se (3)-transformers: 3d roto-translation equivariant attention networks,”  Advances in neural information processing systems, vol. 33, pp. 1970–1981, 2020.
  * [65] Taco S Cohen, Mario Geiger, and Maurice Weiler,  “A general theory of equivariant cnns on homogeneous spaces,”  Advances in neural information processing systems, vol. 32, 2019.
  * [66] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson,  “Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data,”  in International Conference on Machine Learning. PMLR, 2020, pp. 3165–3176.
  * [67] Michael J Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim,  “Lietransformer: Equivariant self-attention for lie groups,”  in International Conference on Machine Learning. PMLR, 2021, pp. 4533–4543.
  * [68] Jonas Köhler, Leon Klein, and Frank Noé,  “Equivariant flows: sampling configurations for multi-body systems with symmetric energies,”  arXiv preprint arXiv:1910.00753, 2019.
  * [69] Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller,  “Schnet: A continuous-filter convolutional neural network for modeling quantum interactions,”  Advances in neural information processing systems, vol. 30, 2017.
  * [70] Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik J Bekkers, and Max Welling,  “Geometric and physical quantities improve e (3) equivariant message passing,”  arXiv preprint arXiv:2110.02905, 2021.
  * [71] Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kornbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky,  “E (3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials,”  Nature communications, vol. 13, no. 1, pp. 2453, 2022.
  * [72] Marco Pacini, Xiaowen Dong, Bruno Lepri, and Gabriele Santin,  “A characterization theorem for equivariant networks with point-wise activations,”  in The Twelfth International Conference on Learning Representations, 2024.
  * [73] David Ruhe, Johannes Brandstetter, and Patrick Forré,  “Clifford group equivariant neural networks,”  Advances in Neural Information Processing Systems, vol. 36, 2024.
  * [74] Johann Brehmer, Pim De Haan, Sönke Behrends, and Taco S Cohen,  “Geometric algebra transformer,”  Advances in Neural Information Processing Systems, vol. 36, 2024.
  * [75] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl,  “Neural message passing for quantum chemistry,”  in Proceedings of the 34th International Conference on Machine Learning - Volume 70. 2017, ICML’17, p. 1263–1272, JMLR.org.
  * [76] C. Giusti, R. Ghrist, and D. S. Bassett,  “Two’s company, three (or more) is a simplex,”  Journal of computational neuroscience, 2016.
  * [77] L. Kanari, P. Dłotko, M. Scolamiero, R. Levi, J. Shillcock, K. Hess, and H. Markram,  “A topological representation of branching neuronal morphologies,”  Neuroinfor., vol. 16, no. 1, pp. 3–13, 2018.
  * [78] A. Patania, G. Petri, and F. Vaccarino,  “The shape of collaborations,”  EPJ Data Sci., vol. 6, no. 1, pp. 18, 2017.
  * [79] L. Lim,  “Hodge Laplacians on graphs,”  Siam Review, 2020.
  * [80] D. Mulder and G. Bianconi,  “Network geometry and complexity,”  Journal of Statistical Physics, vol. 173, no. 3, pp. 783–805, 2018.
  * [81] A. N. Hirani, K. Kalyanaraman, and S. Watts,  “Least squares ranking on graphs, Hodge Laplacians, time optimality, and iterative methods,”  arXiv preprint arXiv:1011.1716, 2010.
  * [82] Phillip Griffiths and Wilfried Schmid,  “Recent developments in hodge theory: a discussion of techniques and results,”  1975\.
  * [83] Claudio Battiloro, Stefania Sardellitti, Sergio Barbarossa, and Paolo Di Lorenzo,  “Topological signal processing over weighted simplicial complexes,”  in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023, pp. 1–5.
  * [84] Emily Ribando-Gros, Rui Wang, Jiahui Chen, Yiying Tong, and Guo-Wei Wei,  “Graph and hodge laplacians: Similarity and difference,”  arXiv preprint arXiv:2204.12218, 2022.
  * [85] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao,  “Hypergraph neural networks,”  in Proceedings of the AAAI conference on artificial intelligence, 2019, vol. 33, pp. 3558–3565.
  * [86] Sajjad Heydari and Lorenzo Livi,  “Message passing neural networks for hypergraphs,”  in International Conference on Artificial Neural Networks. Springer, 2022, pp. 583–592.
  * [87] B. Weisfeiler and A. Leman,  “The reduction of a graph to canonical form and the algebra which appears therein,”  NTI, Series, 1968.
  * [88] Lorenz Demey and Hans Smessaert,  “The relationship between aristotelian and hasse diagrams,”  in International Conference on Theory and Application of Diagrams. Springer, 2014, pp. 213–227.
  * [89] Francesco Di Giovanni, Lorenzo Giusti, Federico Barbero, Giulia Luise, Pietro Lio, and Michael M Bronstein,  “On over-squashing in message passing neural networks: The impact of width, depth, and topology,”  in International Conference on Machine Learning. PMLR, 2023, pp. 7865–7885.
  * [90] Johannes Gasteiger, Janek Groß, and Stephan Günnemann,  “Directional message passing for molecular graphs,”  in International Conference on Learning Representations, 2020.
  * [91] Dejun Jiang, Zhenxing Wu, Chang-Yu Hsieh, Guangyong Chen, Ben Liao, Zhe Wang, Chao Shen, Dongsheng Cao, Jian Wu, and Tingjun Hou,  “Could graph neural networks learn better molecular representation for drug discovery? a comparison study of descriptor-based and graph-based models,”  Journal of Cheminformatics, vol. 13, no. 1, Feb. 2021.
  * [92] Hehuan Ma, Yatao Bian, Yu Rong, Wenbing Huang, Tingyang Xu, Weiyang Xie, Geyan Ye, and Junzhou Huang,  “Multi-view graph neural networks for molecular property prediction,” 2020.
  * [93] Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang,  “Protein representation learning by geometric structure pretraining,”  in The Eleventh International Conference on Learning Representations, 2023.
  * [94] Xuan Li, Zhanke Zhou, Jiangchao Yao, Yu Rong, Lu Zhang, and Bo Han,  “Neural atoms: Propagating long-range interaction in molecular graphs through efficient communication channel,”  in The Twelfth International Conference on Learning Representations, 2024.
  * [95] Junwu Chen and Philippe Schwaller,  “Molecular hypergraph neural networks,”  arXiv preprint arXiv:2312.13136, 2023.
  * [96] Junyoung Park, Fangying Chen, and Jinkyoo Park,  “A molecular hypergraph convolutional network with functional group information,”  2021\.
  * [97] Zhixuan Jia, Yushun Fan, Chunyu Wei, and Ruyu Yan,  “A spatial–temporal hypergraph based method for service recommendation in the mobile internet of things-enabled service platform,”  Advanced Engineering Informatics, vol. 57, pp. 102038, 2023.
  * [98] Paolo Bajardi, Matteo Delfino, André Panisson, Giovanni Petri, and Michele Tizzoni,  “Unveiling patterns of international communities in a global city using mobile phone data,”  EPJ Data Science, vol. 4, no. 1, Apr. 2015.
  * [99] Derek Lo and Briton Park,  “Modeling the spread of the zika virus using topological data analysis,”  PLOS ONE, vol. 13, no. 2, pp. e0192120, Feb. 2018.
  * [100] Bernadette Stolz, Heather Harrington, and Mason Alexander Porter,  “The topological 'shape' of brexit,”  SSRN Electronic Journal, 2016.
  * [101] Michelle Feng and Mason A. Porter,  “Persistent homology of geospatial data: A case study with voting,”  SIAM Review, vol. 63, no. 1, pp. 67–99, 2021.
  * [102] Abigail Hickok, Deanna Needell, and Mason A. Porter,  “Analysis of spatial and spatiotemporal anomalies using persistent homology: Case studies with covid-19 data,”  SIAM Journal on Mathematics of Data Science, vol. 4, no. 3, pp. 1116–1144, 2022.
  * [103] Brandon Anderson, Truong-Son Hy, and Risi Kondor,  Cormorant: covariant molecular neural networks,  Curran Associates Inc., Red Hook, NY, USA, 2019.
  * [104] Jing Wei, Jun Wang, Zhanqing Li, Shobha Kondragunta, Susan Anenberg, Yi Wang, Huanxin Zhang, David Diner, Jenny Hand, Alexei Lyapustin, et al.,  “Long-term mortality burden trends attributed to black carbon and pm2· 5 from wildfire emissions across the continental usa from 2000 to 2020: a deep learning modelling study,”  The Lancet Planetary Health, vol. 7, no. 12, pp. e963–e975, 2023.
  * [105] New York City Department of City Planning,  “Primary land use tax lot output (pluto) data,” https://www1.nyc.gov/site/planning/data-maps/open-data/dwn-pluto-mappluto.page, 2024,  Accessed: 2024-05-22.
  * [106] New York City Department of Transportation,  “Average daily traffic (aadt) data,” https://data.cityofnewyork.us/Transportation/Average-Daily-Traffic-AADT-/fp86-8v8r, 2024,  Accessed: 2024-05-22.
  * [107] Geoff Boeing,  “Osmnx: A python package to work with openstreetmap data,” https://github.com/gboeing/osmnx, 2017,  Version 1.1.1, Accessed: 2024-05-22.
  * [108] Peter J. Huber,  “Robust estimation of a location parameter,”  Annals of Mathematical Statistics, vol. 35, no. 1, pp. 73–101, 1964.

## Reproducibility Statement

We include all the details about our experimental setting, including the
choice of hyperparameters and the specifications of our machine, in Appendix
H. We will soon provide all the code, data splits, and virtual environment
needed to replicate the experiments. An in-depth discussion about the
reproducibility of the molecular property prediction task is presented in
Appendix I.

##  Appendix A Limitations & Future Directions

To our knowledge, ETNN is the first unifying framework for scalarization-based
$E(n)$ equivariant networks over combinatorial topological spaces or, more in
general, over combinatorial objects used to model interactions among entities.
In this work, we introduced ETNNs as a more general, expressive, and flexible
framework compared to prior geometric/topological deep learning models
operating on geometric graphs [26] or simplicial complexes [27]. The promising
results open several avenues for future work.

Methodological. First, while we introduced several useful invariants like
pairwise distances, centroids, Hausdorff distances, and convex hull volumes,
there are likely many other meaningful geometric quantities that could be
leveraged, especially for specific application domains. Therefore, developing
a more comprehensive set of geometric invariants, as well as learning methods
to discover them from data automatically, is an important future direction. In
this sense, generalizing the approach from [39] of integrating Clifford group-
equivariant layers with message passing layers. i.e. going beyond
scalarization, is a possible solution, although requiring to solve possible
scalability issues. Second, extending our framework to make it scale invariant
could be useful in geospatial tasks. Third, at this stage ETNNs cannot
directly handle time-varying scenarios, thus we aim to go beyond the static
settings and develop dynamic or temporal variants of ETNNs. This could enable
new applications in areas like spatio-temporal forecasting, and physical
simulations. Fourth, and finally, ETNNs assume that (potentially) each rank
should satisfy the same $E(n)$ equivariance, but another interesting direction
is studying the setting in which each rank is required to satisfy different
symmetries, pointing towards a principled use of products of groups.

Computational. While our experimental validation of ETNNs covered both
supervised inductive and semisupervised transductive tasks, future works could
tailor ETNNs to fit challenging specific applications at scale. In this sense,
unsupervised or self-supervised training methods for ETNNs could enable the
learning of meaningful representations in a data-driven manner without
requiring labeled data for every task. This could allow ETNNs to better
leverage large, unlabeled geometric datasets. Moreover, though computationally
tractable, ETNNs may benefit from further improvements to effectively scale on
very large datasets. The main bottlenecks are MP operations and geometric
invariants computation. A possible solution could be neighbor samplers [40],
or developing $E(n)$ equivariant message passing-free ETNNs, moving towards
the direction of works like [41].

##  Appendix B Related Works

Topological Deep Learning. TDL is informed by foundational work in Topological
Signal Processing (TSP) [10, 42, 43, 29], highlighting the value of analyzing
multi-way relationships in data. The works in [16, 18] extended the
Weisfeiler-Lehman graph isomorphism test to simplicial and regular cell
complexes [18], respectively, showing that message passing over these spaces
are more expressive than message passing over graphs. Convolutional [44, 45,
46, 47, 48, 49] and attentional architectures [17, 50, 51, 23] over simplicial
and cell complexes have been introduced as well. Additionally, the application
of message passing or diffusion on cellular sheaves [52] over graphs [25, 24,
53, 54, 55] has proven effective in heterophilic scenarios. Models without
message passing have been introduced for simplicial complexes [56, 41] and
hypergraphs [57]. An architecture for inferring a latent regular cell complex
to improve a downstream task has been introduced in [58]. An extensive review
of TDL is available in [59]. Finally, a comprehensive framework for TDL was
outlined in [11], proposing combinatorial complexes and message passing
networks over them.

Equivariant Neural Networks The effectiveness of Convolutional Neural Networks
[60], and their first group extension to the SO(2) group [61] initially
demonstrated the advantages of equivariant models, and paved the way for a
series of works exploring how to design models respecting some symmetries
[62]. Loosely speaking, some of these works employ natively equivariant
function spaces [63, 64], or port the spatial space into high-dimensional
spaces [65, 66, 67]. An alternative strategy is designing neural networks [68,
69, 70, 71] that perform equivariant operations directly within the original
space, including the (VN-)EGNN from [26, 37]. A characterization theorem for
equivariant networks has been presented in [72]. Recent works showed how to
achieve equivariance by leveraging neural network architectures operating on
the Clifford algebra [73, 74]. Each of these approaches has limitations and
advantages, and the effectiveness of a specific design choice is mainly
related to the considered application and computational constraints [28].

Our paper builds on both classes of works. To the best of our knowledge, the
only two works merging TDL models and $E(n)$ equivariant neural networks are
[27, 39]. The latter adopts the approach from [73] by combining Clifford group
equivariant layers with simplicial message passing, while the former
generalizes the graph-based approach from [26] to the simplicial domain.
However, no other combinatorial topological spaces are considered, and,
overall, a general framework for designing equivariant neural networks over
combinatorial topological spaces is missing. Here we address this gap by
introducing our ETNNs.

##  Appendix C Combinatorial Topological Spaces and Combinatorial Complexes

In this appendix, we revisit a subclass of mathematical objects (not always
formal topological spaces) that can be described combinatorially, message
passing networks operating on them, and how they can be cast in the ETNN
frameworks. As we did for CCs, we will often overload the notation
$\mathcal{X}$ in the following subsections, incurring some notation abuses for
the sake of exposition and consistency.

###  C.1 Graphs

The first combinatorial objects we describe are obviously graphs. They are
remarkably useful in several fields, across several tasks, thanks to their
ability to encode prior data knowledge, i.e. their networked structure.

Graphs. A graph $\mathcal{X}$ is a pair $(\mathcal{S},\mathcal{E})$, where
$\mathcal{S}$ is the set of nodes, and $\mathcal{E}$ is the set of edges
(i.e., pairs of nodes).

Neighbhorhood Functions in a Graph. Given a graph $\mathcal{X}$, for all
$x\in\mathcal{S}$, the up adjacency $\mathcal{N}_{A,\uparrow}$ (i.e. the usual
node adjacency) is defined as

$\mathcal{N}_{A,\uparrow}(x)=\\{y\in\mathcal{S}|\exists
z\in\mathcal{E}:x\subset z\textrm{ and }y\subset z\\}.$ (14)

Graphs as Combinatorial Complexes. Graphs can be easily cast as combinatorial
complexes. Let us denote the set of singletons
$\widetilde{\mathcal{S}}=\\{\\{s\\}\\}_{s\in\mathcal{S}}$, a graph
$\mathcal{X}$ is a combinatorial complex
$(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of
nodes, the set of cells $\mathcal{X}=\widetilde{\mathcal{S}}$ is the set of
singletons, and the rank function $\mathrm{rk}$ assigns rank 0 to each
singleton. Please notice that there is no need to include edges as cells in
the complex, differently from, e.g., simplicial complexes, as detailed in the
next subsection, because a CCMPN as in (3) with
$\mathcal{CN}=\\{\mathcal{N}_{A,\uparrow}\\}$ recovers the standard Graph
Message Passing Network from [75]. An ETNN as in (6) with the same collection
of neighborhoods $\mathcal{CN}$ recovers the $E(n)$ Equivariant Graph Neural
Network (EGNN) from [26].

###  C.2 Simplicial Complexes

Graphs are capable of explicitly managing only pairwise interactions among
nodes. However, in numerous applications [76, 77, 78, 29], including the ones
presented in this work, higher order multiway interactions need to be taken
into account. Simplicial complexes are one of the most intuitive objects to
start incorporating them.

Simplicial Complexes. Given a finite set of nodes $\mathcal{S}$, a $k$-simplex
$x$ is a subset of $\mathcal{S}$ with cardinality $k+1$. A face of $x$ is a
$k-1$-simplex being a subset of it, thus a $k$-simplex has $k+1$ faces. A
coface of $x$ is a $(k+1)$-simplex being a superset of it [10, 79]. A
simplicial complex (SC) $\mathcal{X}$ of order $K$ is a collection of
$k$-simplices $x$, $k=0,\ldots,K$ such that, if a simplex $x$ belongs to
$\mathcal{X}$, then all its subsets $y\subset x$ also belong to the complex
(inclusivity property). In most of the cases, the focus is on complexes of
order up to two, thus having a set of vertices, a set of edges, and a set of
triangles. To give a simple example, in Fig. 4 we sketch a simplicial complex
of order $2$.

Figure 4: A simplicial complex of order 2, comprising of nodes (cells of rank
0), edges (cells of rank 1), and triangles (cells of rank 2).

Neighbhorhood Functions in a Simplicial Complex. Given a SC $\mathcal{X}$, the
coboundary and boundary neighborhood functions $\mathcal{N}_{C}$ and
$\mathcal{N}_{B}$, respectively, are defined, as

$\mathcal{N}_{C}(x)=\\{y\in\mathcal{X}|y\textrm{ is a coface of
}x\\},\quad\mathcal{N}_{B}(x)=\\{y\in\mathcal{X}|y\textrm{ is a face of
}x\\}.$ (15)

The up and down adjacencies $\mathcal{N}_{A,\uparrow}$ and
$\mathcal{N}_{A,\downarrow}$, respectively, are defined, as

$\displaystyle\mathcal{N}_{A,\uparrow}(x)=\\{y\in\mathcal{X}|y\textrm{ shares
a coface with
}x\\},\;\mathcal{N}_{A,\downarrow}(x)=\\{y\in\mathcal{X}|y\textrm{ shares a
face with }x\\}.$ (16)

Simplicial Complexes as Combinatorial Complexes. It is straightforward to
frame SCs in the CC framework. In particular, a simplicial complex
$\mathcal{X}$ is a combinatorial complex
$(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of
nodes, the set of cells $\mathcal{X}$ is the set of simplices, and the rank
function $\mathrm{rk}$ assigns to each cell its cardinality minus one. A CCMPN
as in (3) with
$\mathcal{CN}=\\{\mathcal{N}_{C},\mathcal{N}_{B},\mathcal{N}_{A,\uparrow},\mathcal{N}_{A,\downarrow}\\}$
recovers the Message Passing Simplicial Network from [16]. An ETNN as in (6)
with the same collection of neighborhoods $\mathcal{CN}$ recovers the $E(n)$
Equivariant Message Passing Simplicial Network (EMPSN) from [27].

###  C.3 Cell Complexes

Simplicial complexes are capable of managing relationships of varying degrees,
yet the inclusion property often binds them. This property mandates that if a
set is part of the space, then all its respective subsets must also be
included in that space. In numerous applications, this restriction is not only
unnecessary but also lacks a justified basis. Consequently, cell complexes are
frequently employed instead [80, 81]. These structures are versatile and
support a hierarchical organization.

Regular Cell Complexes. A regular cell complex (CW) is a topological space
$\mathcal{X}$ together with a partition
$\\{\mathcal{X}_{x}\\}_{x\in\mathcal{H}_{\mathcal{X}}}$ of subspaces
$\mathcal{X}_{x}$ of $\mathcal{X}$ called cells, where
$\mathcal{H}_{\mathcal{X}}$ is the indexing set of $\mathcal{X}$, s.t.

  1. 1.

For each $x$ $\in$ $\mathcal{X}$, every sufficient small neighborhood of $x$
intersects finitely many $\mathcal{X}_{x}$;

  2. 2.

For all $x$,$y$ we have that $\mathcal{X}_{x}$ $\cap$
$\overline{\mathcal{X}}_{y}$ $\neq$ $\varnothing$ iff $\mathcal{X}_{x}$
$\subseteq$ $\overline{\mathcal{X}}_{y}$, where $\overline{\mathcal{X}}_{y}$
is the closure of the cell;

  3. 3.

Every $\mathcal{X}_{x}$ is homeomorphic to $\mathbb{R}^{k}$ for some $k$;

  4. 4.

For every $x$ $\in$ $\mathcal{H}_{\mathcal{X}}$ there is a homeomorphism $g$
of a closed ball in $\mathbb{R}^{k}$ to $\overline{\mathcal{X}}_{x}$ such that
the restriction of $g$ to the interior of the ball is a homeomorphism onto
$\mathcal{X}_{x}$.

From 2, $\mathcal{H}_{\mathcal{X}}$ has a poset structure, given by $y$ $\leq$
$x$ iff $\mathcal{X}_{y}$ $\subseteq$ $\overline{\mathcal{X}_{x}}$, and we say
that $y$ bounds $x$. This is known as the face poset of $\mathcal{X}$. From 4,
all of the topological information about $\mathcal{X}$ is encoded in the poset
structure of $\mathcal{H}_{\mathcal{X}}$. Then, a regular cell complex can be
identified with its face poset. From now on we will indicate the cell
$\mathcal{X}_{x}$ with its corresponding face poset element $x$. The dimension
or order $\textrm{dim}(\sigma)$ of a cell $\sigma$ is $k$, and we call it a
$k-$cell. The dimension or order of a cell complex is the largest dimension of
any of its cells. Regular cell complexes can be described via an incidence
relation (boundary relation) with a reflexive and transitive closure that is
consistent with the partial order introduced in the above definition. In
particular, we have the boundary relation $y$ $\prec$ $x$ iff $\dim({y})$
$\leq$ $\dim({x})$ and there is no cell $z$ such that $y$ $\leq$ $z$ $\leq$
$x$. In other words, the boundary of a cell $x$ of dimension $k$ is the set of
all cells of dimension less than $k$ bounding $x$. The k-skeleton of a regular
cell complex $\mathcal{X}$ is the subcomplex of $\mathcal{X}$ consisting of
cells of dimension at most $k$. Therefore, although cell complexes are a
broadly general object, we can interpret the 0-skeleton of a cell complex as a
set of 0-cells (vertices) and the 1-skeleton as a set of vertices together
with a set of 1-cells (edges), thus a graph. For this reason, given a graph
$\mathcal{G}$, i.e. a realization of an order-1 cell complex, it is possible
to lift it to an order-2 cell complex $\mathcal{X}$ whose 1-skeleton is
isomorphic to $\mathcal{G}$. A way to do it is by attaching order 2 cells to
the/a set/subset of simple/induced cycles of $\mathcal{G}$. We refer to cell
complexes of this type as polygonal cell complexes [29, 13]. In Fig. 5, we
sketch an order 2 polygonal cell complex.

Figure 5: A polygonal cell complex of order 2, comprising of nodes (cells of
rank 0), edges (cells of rank 1), and cycles (cells of rank 2).

###### Remark 2 .

It is easy to notice that a polygonal CW considering cycles of length up to 3
is equivalent to a simplicial complex of order 2 having the same set
$\mathcal{S}$ of 0-simplices (vertices), the same set of 1-simplices (edges),
and the same set of 2-simplices (triangles).

###### Remark 3 .

The reader may have noticed that we gave no explicit characterization of a
simplicial complex as a topological space, although it is pretty easy to
particularize cell to simplicial complexes. This is because our definition of
SC in Sec. C.2 is, to be completely rigorous, the definition of an abstract
simplicial complex, that itself is not a topological space. However, when we
refer to a simplicial complex as a topological space, we are implicitly
talking about the geometric realization of an abstract simplicial complex.
Therefore, we are implicitly stating that the SC has a topology coherent with
its simplices, which are "topologized" as homeomorphisms of the standard
simplices living in Euclidean space. Please refer to [10] for a detailed but
accessible description. We will not incur any ambiguity by referring to
abstract simplicial complexes as just simplicial complexes.

Neighbhorhood Functions in a Cell Complex. Given a CW $\mathcal{X}$, the
coboundary and boundary neighborhood functions $\mathcal{N}_{C}$ and
$\mathcal{N}_{B}$, respectively, are defined, as

$\mathcal{N}_{C}(x)=\\{y\in\mathcal{X}|y\prec
x\\},\quad\mathcal{N}_{B}(x)=\\{y\in\mathcal{X}|x\prec y\\}.$ (17)

The up and down adjacencies $\mathcal{N}_{A,\uparrow}$ and
$\mathcal{N}_{A,\downarrow}$, respectively, are defined, as

$\displaystyle\mathcal{N}_{A,\uparrow}(x)=\\{y\in\mathcal{X}|\exists z\textrm{
such that }x\prec z,y\prec
z\\},\;\mathcal{N}_{A,\downarrow}(x)=\\{y\in\mathcal{X}|\exists z\textrm{ such
that }z\prec x,z\prec y\\}.$ (18)

Cell Complexes as Combinatorial Complexes. It is again straightforward to
frame polygonal CWs in the CC framework. In particular, a polygonal CW is a
combinatorial complex $(\mathcal{S},\mathcal{X},\mathrm{rk})$, where
$\mathcal{S}$ is the set of nodes, $\mathcal{X}$ is the set of cells, and the
rank function $\mathrm{rk}$ assigns to each cell its dimension $\mathrm{dim}$.
A CCMPN as in (3) with
$\mathcal{CN}=\\{\mathcal{N}_{B},\mathcal{N}_{A,\uparrow}\\}$ on a polygonal
CW recovers the Molecular Message Passing Network from [18]. ETNNs as in (6)
with the same collection of neighborhoods $\mathcal{CN}$ give rise to $E(n)$
Equivariant Molecular Message Passing Networks, not present in literature.

###### Remark 4 .

Simplicial Complexes and Cell Complexes are rich topological objects.
Neighborhoods in a simplicial or a cell complex are strongly rooted in
arguments from algebraic topology. In particular, Hodge theory [82] is used to
derive formal notions of boundary, coboundary, and Hodge Laplacians [10]. More
detailed descriptions can be found in [13, 10, 83, 84]. In this work, we are
interested in the definition of cells (in the CC sense) and neighborhood
functions that these objects (SCs and CWs) induce. For this reason, a reader
familiar with algebraic topology could find our definitions and treatment not
totally comprehensive. However, we aim to provide a unifying framework in the
context of (equivariant) TDL, thus our presentation is sufficiently consistent
and strikes a good tradeoff between generality, rigor, and applicability.

###  C.4 Hypergraphs

Cell complexes have their formal hierarchical and topology-grounded
characterization as their main strength. However, their structure is still
constrained by the homeomorphism requirements, preventing them from modeling
arbitrary higher order hierarchical interactions. On the other hand,
hypergraphs are combinatorial objects that have total flexibility, but without
any hierarchical and (in the general case) topological structure.

Hypergraphs. A hypergraph (HG) on a nonempty set of nodes $\mathcal{S}$ is a
pair $(\mathcal{S},\mathcal{E})$, where $\mathcal{X}$ is a subset of the power
set of $\mathcal{P}(\mathcal{S})\backslash\\{\emptyset\\}$ of $\mathcal{S}$.
Elements of $\mathcal{S}$ are called hyperedges. In Fig. 6, we sketch a
hypergraph with 3 vertices and 3 hyperedges.

Neighbhorhood Functions in a Hypergraph. Let us again denote the set of
singletons $\widetilde{\mathcal{S}}=\\{\\{s\\}\\}_{s\in\mathcal{S}}$. Given a
HG $\mathcal{X}$, for all $x\in\widetilde{\mathcal{S}}\cup\mathcal{E}$, the up
and down incidences $\mathcal{N}_{I,\uparrow}$ and
$\mathcal{N}_{I,\downarrow}$, respectively, are defined as

$\mathcal{N}_{I,\uparrow}(x)=\\{y\in\widetilde{\mathcal{S}}\cup\mathcal{E}|x\subset
y\\},\quad\;\mathcal{N}_{I,\downarrow}(x)=\\{y\in\widetilde{\mathcal{S}}\cup\mathcal{E}|y\subset
x\\}.$ (19)

Therefore, if $x$ is a node, the down and up incidences evaluated at $x$ are
the empty set and the hyperedges containing $x$, respectively. If $x$ is a
hyperedge, then the down and up neighborhood functions evaluated at $x$ are
the nodes contained in $x$ and the empty set, respectively.

Hypergraphs as Combinatorial Complexes. Even in this case, framing HGs in the
CC framework is straightforward. In particular, a hypergraph $\mathcal{X}$ is
a combinatorial complex $(\mathcal{S},\mathcal{X},\mathrm{rk})$, where
$\mathcal{S}$ is the set of nodes, the set of cells
$\mathcal{X}=\widetilde{\mathcal{S}}\cup\mathcal{E}$, and the rank function
$\mathrm{rk}$ assigns rank 0 to the nodes and rank 1 to the hyperedges. A
CCMPN as in (3) with
$\mathcal{CN}=\\{\mathcal{N}_{I,\uparrow},\mathcal{N}_{I,\downarrow}\\}$
recovers the Hypergraph Message Passing Networks from [85] (in the
convolutional realization) and [86]. ETNNs as in (6) with the same collection
of neighborhoods $\mathcal{CN}$ give rise to $E(n)$ Equivariant Hypergraph
Message Passing Networks, not present in literature.

Figure 6: A hypergraph, comprising of nodes (cells of rank 0), hyperedges
(cells of rank 1).

##  Appendix D Proof of Theorem 1

The proof of Theorem 1 is a straightforward generalization of the proof
provided for Equivariant Graph Neural Networks (EGNN) in [26]. In particular,
we can directly state that the update of the features in (6) is $E(n)$
invariant because the positions $\\{\mathbf{x}_{z}\\}_{z\in x}$ are used only
as input to the geometric invariant(s) $\mathrm{Inv}$ and the features
$\mathbf{h}_{x}$ contain no geometric information, i.e.

$\mathbf{h}_{x}^{l+1}=\beta\left(\mathbf{h}^{l}_{x},\bigotimes_{\mathcal{N}\in\mathcal{CN}}\bigoplus_{y\in\mathcal{N}(x)}\underbrace{\psi_{\mathcal{N},\mathrm{rk}(x)}\left(\mathbf{h}^{l}_{x},\mathbf{h}^{l}_{y},\mathrm{Inv}\left(\\{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\\}_{z\in
x},\\{\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\\}_{z\in
y}\right)\right)}_{\mathbf{m}_{x,y}^{\mathcal{N}}}\right),$ (20)

for all $x\in\mathcal{X}$ and $(\mathbf{O},\mathbf{b})\in E(n)$. Therefore, we
just need to show that the update of the positions in (7) is $E(n)$
equivariant, i.e. we want to prove that

$\mathbf{O}\mathbf{x}_{z}^{l+1}+\mathbf{b}=\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in
S:\\{t\\}\in\mathcal{N}(z)}\left(\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}-(\mathbf{O}\mathbf{x}_{t}^{l}+\mathbf{b})\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right),$
(21)

for all $z\in\mathcal{S}$ and $(\mathbf{O},\mathbf{b})\in E(n)$. By direct
computation, as showed in [26], we can write

$\displaystyle\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in
S:\\{t\\}\in\mathcal{N}(z)}\left(\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}-\mathbf{O}\mathbf{x}_{t}^{l}+\mathbf{b}\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right)$
$\displaystyle=\mathbf{O}\mathbf{x}_{z}^{l}+\mathbf{b}+\mathbf{O}C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in
S:\\{t\\}\in\mathcal{N}(z)}\left(\mathbf{x}_{z}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right)$
$\displaystyle=\mathbf{O}\left(\mathbf{x}_{z}^{l}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in
S:\\{t\\}\in\mathcal{N}(z)}\left(\mathbf{x}_{z}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{z,t}^{\mathcal{N}}\right)\right)+\mathbf{b}=\mathbf{O}\mathbf{x}_{z}^{l+1}+\mathbf{b}$
(22)

Being the update in (6) $E(n)$ invariant and the update in (7) $E(n)$
equivariant, the proof is concluded.

##  Appendix E Expressiveness of ETNNs

The ability of Graph Neural Networks (GNNs) to differentiate between non-
isomorphic graphs is typically employed as an expressivity metric. In
particular, it is well known that vanilla GNNs are at maximum as powerful as
the Weisfeiler-Leman (WL) test [87] in discriminating isomorphic graphs. As
such, the WL test sets a limit on the expressiveness of GNNs. To broaden the
use of this method to include geometric graphs, the Geometric Weisfeiler-Leman
test (GWL) has been introduced [33]. This test evaluates whether two
(undirected) graphs are geometrically isomorphic, as defined next.

Geometrically Isomorphic Graphs.[33, 37] Two geometric graphs
$\mathcal{G}_{1}=(\mathcal{S}_{1},\mathcal{E}_{1})$ and
$\mathcal{G}_{2}=(\mathcal{S}_{2},\mathcal{E}_{2})$ with
$|\mathcal{S}_{1}|=|\mathcal{S}_{2}|$, non-geometric and geometric $i$-th node
features $\mathbf{h}_{x_{i}}^{\mathcal{G}_{j}}$ and
$\mathbf{x}_{x_{i}}^{\mathcal{G}_{j}}$ ($j\in\\{1,2\\}$), are called
geometrically isomorphic if there exists an edge-preserving bijection
$b:\mathcal{I}_{1}\rightarrow\mathcal{I}_{2}$ between their corresponding node
indices set $\mathcal{I}_{1}$ and $\mathcal{I}_{2}$, such that their positions
are equivalent up to $E(n)$ group actions, i.e.:

$\mathbf{h}_{x_{b(i)}}^{\mathcal{G}_{2}},\mathbf{x}_{x_{b(i)}}^{\mathcal{G}_{2}}=\mathbf{h}_{x_{i}}^{\mathcal{G}_{1}},\mathbf{O}\mathbf{x}_{x_{i}}^{\mathcal{G}_{1}}+\mathbf{b}$
(23)

for all $i\in\mathcal{I}_{1},(\mathbf{O},\mathbf{b})\in E(n)$.

$\mathbf{k}$-hop Distinct Graphs.[33, 37] Two geometric graphs
$\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ are said to be $k$-hop distinct if for
all graph isomorphisms $b$, there is some node
$i\in\mathcal{I}_{1},b(i)\in\mathcal{I}_{2}$ such that the corresponding
$k$-hop neighborhood subgraphs are distinct. Otherwise, if they are identical
up to $E(n)$ actions for all $i\in\mathcal{I}_{1}$, we say $\mathcal{G}_{1}$
and $\mathcal{G}_{2}$ are said to be $k$-hop identical.

To study Combinatorial Complexes (CCs) and the expressiveness of Equivariant
CCMPNs (ETNNs), we introduce the notion of Geometric Augmented Hasse Graph
representation of CCs, generalizing [11] and the notion of Hasse diagram for
finite partially ordered sets [88] to the geometric setting.

Geometric Augmented Hasse Graph of a Combinatorial Complex.[11] Let
$(\mathcal{S},\mathcal{X},\mathrm{rk})$ be a CC where each node
$z\in\mathcal{S}$ comes with positions $\mathbf{x}_{z}$, and $\mathcal{CN}$ a
collection of neighborhoods function. The Geometric Augmented Hasse graph
$\mathcal{G}_{\mathcal{X}}$ of $(\mathcal{S},\mathcal{X},\mathrm{rk})$ is a
(possibly) directed geometric graph $\mathcal{G}_{\mathcal{X}}=$
$\left(\mathcal{X},\mathcal{E}\right)$ with cells as nodes, edges given by

$\mathcal{E}=\\{(x,y)|x\in\mathcal{X},y\in\mathcal{X},\exists\mathcal{N}\in\mathcal{CN}:x\in\mathcal{N}(y)\textrm{
or }y\in\mathcal{N}(x)\\},$ (24)

and positions of cell $x$ being a linear permutation invariant function
$\bigoplus_{z\in x}\mathbf{x}_{z}$ of its corresponding node
positions$\\{\mathbf{x}_{z}\\}_{z\in x}$, $z\in\mathcal{S}$. Therefore, the
geometric augmented Hasse graph of a CC is a graph representation of it
obtained by considering the cells as vertices, by inserting edges among them
if the cells are neighbors in the CC, and by assigning positions to higher-
order cells as functions of the nodes (of the CC) they contain. Fig. 7 shows
an example of a CC and the corresponding geometric augmented Hasse graph.
Given a CC and a collection of neighborhood functions, the corresponding
geometric augmented Hasse graph is equivalent to the computational graph of an
ETNN [11]. The following also holds.

(a)

(b)

Figure 7: (a) A combinatorial complex with nodes of the underlying graph as
cells of  rank 0, and with three arbitrary cells of  rank 1. (b) The
corresponding geometric augmented Hasse graph if
$\mathcal{CN}=\\{{\color[rgb]{0.68359375,0.25390625,0.1953125}\mathcal{N}_{A,\uparrow}},{\color[rgb]{0.421875,0.55859375,0.75}\mathcal{N}_{I,\uparrow}},{\color[rgb]{0.421875,0.55859375,0.75}\mathcal{N}_{I,\downarrow}}\\}$
with ${\color[rgb]{0.68359375,0.25390625,0.1953125}\mathcal{N}_{A,\uparrow}}$
as in (14), and $\bigoplus$ is the mean. This is also an example of a
skeleton-preserving lift.

###### Proposition 1 .

Let $(\mathcal{S},\mathcal{X},\mathrm{rk})$ be a CC where each node
$z\in\mathcal{S}$ comes with positions $\mathbf{x}_{z}$, $\mathcal{CN}$ a
collection of neighborhood functions, and $\mathcal{G}_{\mathcal{X}}$ the
corresponding geometric augmented Hasse graph. Assuming:

  1. 1.

to have a single message function for all neighborhoods and ranks, i.e.
$\psi_{\mathcal{N},\mathrm{rk}(x)}=\psi$ in (6),

  2. 2.

that inter- and intra-neighborhood aggregation functions are the same, i.e.
$\bigotimes=\bigoplus$ in (6),

  3. 3.

that the only employed geometric invariants are sum of distances of linear
permutation invariant functions as in (10), e.g. sum or mean;

then ETNNs over $(\mathcal{S},\mathcal{X},\mathrm{rk})$ are equivalent to
EGNNs [26] over $\mathcal{G}_{\mathcal{X}}$ with the following modified update
for positions of node $x$ (of the Hasse graph):

$\mathbf{x}_{x}^{l+1}=\begin{cases}\mathbf{x}_{x}^{l}+C\sum_{\mathcal{N}\in\mathcal{CN}}\sum_{t\in
S:\\{t\\}\in\mathcal{N}(z)}\left(\mathbf{x}_{x}^{l}-\mathbf{x}_{t}^{l}\right)\xi\left(\mathbf{m}_{x,t}\right),\text{
if }x\in\mathcal{S},\\\ \mathbf{x}_{x}^{l+1}=\bigoplus_{z\in
x}\mathbf{x}_{z}^{l+1},x\in\mathcal{X}\setminus\mathcal{S}\end{cases}$ (25)

###### Proof.

The proof is trivial and is obtained by simple direct substitution in (6)-(7).
∎

Intuitively, the modified update is required because higher order cells do not
come with positions, thus they cannot be updated as the positions of the nodes
in $\mathcal{S}$.

Lifting of a Graph into a CC. Given a graph, many ways exist to lift it into a
combinatorial complex. As shown in Appendix C.1, denoting the set of
singletons $\widetilde{\mathcal{S}}=\\{\\{s\\}\\}_{s\in\mathcal{S}}$, a graph
$\mathcal{G}(\mathcal{S},\mathcal{E})$ is a combinatorial complex
$(\mathcal{S},\mathcal{X},\mathrm{rk})$, where $\mathcal{S}$ is the set of
nodes, the set of cells $\mathcal{X}=\widetilde{\mathcal{S}}$ is the set of
singletons, the rank function $\mathrm{rk}$ assigns rank 0 to each singleton,
and the up adjacency from (14) is employed. However, we can enrich the
representation by including the edges $\mathcal{E}$ as cells, i.e.
$\mathcal{X}=\widetilde{\mathcal{S}}\bigcup\mathcal{E}$, and assign rank 1 to
them. In this setting, it is possible to make nodes communicate with edges and
vice-versa, rather than just making nodes communicate through edges. In
general, a natural question is how to define higher order cells, i.e. cells of
rank greater than 0. We have already shown in Section 4 that domain knowledge
can be injected by properly modeling higher order cells. However, we are now
interested in understanding which conditions should higher order cells respect
to result in an increased expressive power of ETNN. To perform a fair
analysis, we consider only skeleton-preserving lifts [18], i.e. lifts in which
the original connectivity among the nodes in $\mathcal{S}$ is preserved. In
other words, we don’t directly rewire the underlying graph $\mathcal{G}$. As
shown in Fig. 7, a straightforward example of skeleton-preserving lifts is
choosing (1)-(14) as neighborhood functions without necessarily adding the
edges as cells. Another example is choosing (1)-(2) as neighborhood functions
including the edges as cells.

Expressivity of ETNN. Due to Proposition 1, given two geometric graphs
$\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ and two combinatorial complexes
$\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$ obtained
by respectively lifting them, we can study how expressive ETNNs are by
analyzing how the GWL performs on the geometric augmented Hasse graphs
$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ and
$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ of
$\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$,
respectively, w.r.t. how it would perform directly on the underlying graphs
$\mathcal{G}_{1}$ and $\mathcal{G}_{2}$. Since the GWL operates on undirected
graphs, we need to assume that the produced Hasse graph is undirected. The two
examples of skeleton-preserving lifts clearly result in undirected Hasse
graphs. We would obtain a directed Hasse graph if, e.g., only one of the
up/down incidences in (1) is used as a neighborhood function. Similar to the
standard graph WL test, the GWL method updates node colors iteratively based
on the features of nodes in the local neighborhood. Additionally, GWL
maintains $E(n)$-equivariant hash values that capture each node’s local
geometry. Therefore, any $k$-hop distinct, $(k-1)$-hop identical geometric
graphs can be distinguished by $k$ iterations of GWL, i.e. when the updated
hash values that capture each node’s local geometry differ across the two
graphs for the first time, or by $k$ layers of an EGNN, i.e. when the
geometric invariants differ across the two graphs for the first time [37, 33].

###### Proposition 2 .

Assume to have two $k$-hop distinct geometric graphs
$\mathcal{G}_{1}=(\mathcal{S}_{1},\mathcal{E}_{1})$ and
$\mathcal{G}_{2}=(\mathcal{S}_{2},\mathcal{E}_{2})$ where the underlying
graphs are isomorphic in the standard sense, two combinatorial complexes
$\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$ and a
collection $\mathcal{CN}$ of neighborhood functions obtained via a skeleton-
preserving lift and leading to undirected geometric Hasse graphs
$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ and
$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{2}}}$, respectively. An ETNN operating
over $\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$
respecting Assumptions 1-3 can distinguish $\mathcal{G}_{1}$ and
$\mathcal{G}_{2}$ in $M$ layers, where $M$ is the number of layers required to
have at least one cell in
$\mathcal{X}_{\mathcal{G}_{1}}$/$\mathcal{X}_{\mathcal{G}_{2}}$ whose
receptive field is the whole set of nodes in
$\mathcal{G}_{1}$/$\mathcal{G}_{2}$.

###### Proof.

To prove the result, we study the behavior of the GWL on the geometric
augmented Hasse graphs $\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$ and
$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{2}}}$ of
$\mathcal{X}_{\mathcal{G}_{1}}$ and $\mathcal{X}_{\mathcal{G}_{2}}$,
respectively. For 1-hop distinct graphs, one iteration of GWL suffices to
distinguish them even without any lifting into a CC and, thus, the proposition
holds.

Now, let us assume that $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$ are $k$-hop
distinct and $(k-1)$-hop identical for any $k>1,k\in\mathbb{N}$. As a direct
consequence of Proposition 1, $M$ GWL iterations are required for (at least
one) node of the Hasse graphs
$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{1}}}$/$\mathcal{G}_{\mathcal{X}_{\mathcal{G}_{2}}}$
to have the entire set of nodes $\mathcal{S}_{1}$/$\mathcal{S}_{2}$ in its
receptive field. Denote that node with $x_{i}\in\mathcal{G}_{1}$ and
$x_{b(i)}\in\mathcal{G}_{2}$, where $b$ is any isomorphism between
$\mathcal{G}_{1}$ and $\mathcal{G}_{2}$. Due to the k-hop distinctness of the
graphs, the hash values corresponding to node $x_{i}\in\mathcal{G}_{1}$ and
$x_{b(i)}\in\mathcal{G}_{2}$ will differ in the $M$-th iteration, thus $M$
iterations of GWL can distinguish the graphs. Therefore. Combining the fact
that EGNNs are as powerful as the GWL in distinguishing $k$-hop distinct
graphs [37, 33] with the result from Proposition 1, the proof is completed. ∎

###### Remark 5 .

The reader may have noticed that, in proving Proposition 2, we used the result
from Proposition 1 ignoring the modified position update rule in (25), i.e.
implicitly assuming that also the positions of higher order cells, when seen
as nodes of the corresponding augmented Hasse graph, are updated as in (7),
following the standard EGNN. This is not relevant, as the modified update rule
in (25) is actually more powerful than the standard rule in (7), because the
positions of the higher order cell $x$ at layer $l$ is updated using the
already updated node positions $\bigoplus_{z\in x}\mathbf{x}_{z}^{l+1}$ and
not the $\mathbf{x}_{z}^{l}$s. For this reason, Proposition 2 is somehow a
pessimistic scenario for the practical use of ETNNs, and there could be corner
cases in which ETNNs can distinguish $k$-hop distinct geometric graphs with a
lower number of layers than prescribed. Finally, in practice, we observed it
is not important to stick to the homogeneous setting (Assumption 1), to a
unique aggregation function (Assumption 2), or to a certain geometric
invariant (Assumption 3). These assumption are useful to simplify the
tractation of ETNNs and have a perfect match with the geometric augmented
Hasse graph formalism.

Discussion. Proposition 2 provides formal hints about how different lifts of
graphs into combinatorial complexes can impact the expressive power of ETNNs.
Our result can be seen as a proper generalization of Proposition 2 in [37]
about the expressivity of VN-EGNNs. Indeed, adding a virtual node connected to
all the other nodes can be interpreted as a combinatorial complex in which the
cells are the set of singletons (the nodes) plus a cell containing all the
nodes (the virtual node). A (simplified) variant of a VG-EGNN is then obtained
by using the adjacency induced by the edges of the original graph and the
up/down incidences as neighborhood functions. In this case, as prescribed from
Proposition 2, the receptive field of the virtual node is the whole graph by
definition, making VN-EGNNs able to distinguish $k$-hop distinct graphs in one
layer. Our results also go in the direction of [20], in which the expressivity
of standard graph message passing networks on transformed/lifted graphs is
systematically studied. This work provides a first, preliminary intuition on
how the framework from [20] could be extended to the geometric graph setting.
Moreover, it is worth noting that our expressivity analysis could be enriched
through the same GWL framework from [33], eventually leading to a
topological/higher order generalization of it. An interesting direction is
studying the expressivity of ETNNs when the lifts result in directed geometric
augmented Hasse graphs. Finally, our result could shed light on which lifts
should be preferred. In the current literature [18, 11, 17, 27] except for
[58], the lifts are always decided based on simple criteria (e.g., if the
domain is a simplicial complex, all the triangles of the underlying graphs are
considered as 2-simplices). However, lifts could strike desirable tradeoffs
between expressivity and learning performance (e.g. if the domain is again a
simplicial complex, only some triangles could be chosen without expressivity
drop).

Figure 8: $k$-chain geometric graphs, i.e. $k$ aligned nodes and two end nodes
with opposite orientations.

Experimental Validation. To validate our theoretical claims, we follow the
approach from [33, 37]. In particular, we employ pairs of $k$-chain geometric
graphs, where each graph pair consists of $k$ nodes linearly aligned and two
endpoints with different orientations, as shown in Fig. 8. These graphs are
clearly $\left(\left\lfloor\frac{k}{2}\right\rfloor+1\right)$-hop distinct,
and thus should be distinguishable by
$\left(\left\lfloor\frac{k}{2}\right\rfloor+1\right)$ EGNN layers or the same
number of GWL iterations [33, 37]. To show the correctness of our result, we
consider $k=4$, and we lift each pair of graphs into a CC in different ways,
listed below. In all the cases, we use the mean to assign positions to cells
in the Hasse graph (Assumption 3 of Proposition 1). Without loss of
generality, we use $\mathcal{N}_{A,\uparrow}$ as in (14) (the usual node
adjacency) and the up/down incidences
$\mathcal{N}_{I,\uparrow}$/$\mathcal{N}_{I,\downarrow}$ in (1) as neighborhood
functions, i.e. we keep the connectivity among nodes as in the original pair
of graphs without including edges as cells and we allow higher order cells to
communicate only with nodes but not among them (thus resulting in a
hypergraph-like structure). The readout classifier takes as input the
geometric invariants computed on the final updated positions.

  * •

Lift 1: we consider a set of cells composed of the nodes of the original
graphs plus one 1-cell containing all the nodes, as shown in Fig. 9(a) (a). In
Figure 9(b), we show the corresponding geometric Hasse graphs. It is clear
that all the nodes will have the whole graph as their receptive field before
the first iteration, thus we expect the GWL on the Hasse graph, hence ETNN, to
distinguish the graphs using one layer. Clearly, this case would be solved
even by just feeding the initial geometric invariants to the readout
classifier (thus, with "0 layers").

  * •

Lift 2: we consider a set of cells composed of the nodes of the original
graphs plus two 1-cells of the same cardinality being overlapping paths of
length 4, as shown in Fig. 10(a). In Figure 10(b), we show the corresponding
geometric Hasse graphs. It is clear that nodes $\\{p_{1}\\}$,$\\{p_{2}\\}$
will have the whole graph as their receptive field in one iteration, thus we
expect the GWL on the Hasse graph, hence ETNN, to distinguish the graph using
one layer.

  * •

Lift 3: we consider a set of cells composed of the nodes of the original
graphs plus two 1-cells of different cardinality and not overlapping, as shown
in Fig. 11(a) (a). In Figure 11(b), we show the corresponding geometric Hasse
graphs. It is clear that nodes
$\\{p_{1}\\}$,$\\{p_{2}\\}$,$\\{o_{1}\\}$,$\\{p_{2}\\}$ will have the whole
graph as their receptive field in two iterations, thus we expect the GWL on
the Hasse graph, hence ETNN, to distinguish the graph using two layers.

Per each lift, we trained ETNNs with an increasing number of layers to
classify the pairs of graphs. Due to possible oversquashing effects [33], we
also trained ETNNs with 5 different hidden dimensions and averaged over 10
different seeds. In Table 3, we report the results. We also report the results
for EGNN (taken from [37]) and the GWL on the original pair of graphs without
any lifting into a CC. As the reader can see, empirical results confirm the
validity of our analysis, and the clear advantage, in terms of expressivity,
of using CCs and ETNNs over standard EGNNs. Further empirical advantage is
reported, pointing at higher order interactions as a practical solution to
oversquashing [22, 89].

Table 3: Classification Accuracy on $4$-chain geometric graphs. |  | Number of Layers
---|---|---
$k=4$ | Hidden Dim. | 1 | 2 | 3 | 4
GWL |  | $50\%$ | $50\%$ | $100\%$ | $100\%$
EGNN [37] | 32 | $50\%$ | $50\%$ | $56.5\%$ | $50\%$
64 | $50\%$ | $50\%$ | $100\%$ | $99\%$
128 | $50\%$ | $50\%$ | $96.5\%$ | $98.5\%$
ETNN Lift 1 | 32 | $100\%$ | $100\%$ | $100\%$ | $100\%$
64 | $100\%$ | $100\%$ | $100\%$ | $100\%$
128 | $100\%$ | $100\%$ | $100\%$ | $100\%$
ETNN Lift 2 | 32 | $100\%$ | $100\%$ | $100\%$ | $100\%$
64 | $90\%$ | $100\%$ | $100\%$ | $100\%$
128 | $90\%$ | $100\%$ | $100\%$ | $100\%$
ETNN Lift 3 | 32 | $50\%$ | $85\%$ | $100\%$ | $100\%$
64 | $50\%$ | $80\%$ | $95\%$ | $85\%$
128 | $50\%$ | $80\%$ | $95\%$ | $95\%$

###### Remark 6 .

The lifts we used in the experiments above are handcrafted lifts, employed to
validate Proposition 2 in a easy and consistent way. However, our result
obviously holds in all the cases, even with more ranks and more sophisticated
collections of neighborhood functions (e.g. induced by skeleton-preserving
lift to simplicial, cell, or path complexes[18, 23, 17, 19, 16]). Moreover, it
is worth it to notice that, with the experiments above, we investigated the
role of the choice of the cells given the collection of neighborhood
functions. However, it is interesting to study even the opposite case, in
which the cells are fixed and the neighborhood functions are chosen. For
example, in Lift 2, adding the lower adjacency from (2) would have led to
solving the task with just one layer because the cells $c_{1}$/$c_{2}$ and
$d_{1}$/$d_{2}$ would have had the whole graph as their receptive field before
the first iteration. On the other hand, in Lift 3, if we used the up adjacency
from (2) without including the edges as cells, we would have not been able to
solve the task, because we would have obtained a disconnected graph, since
$p_{1}$/$p_{2}$ and $o_{1}$/$o_{2}$ would have not been linked.

(a)

(b)

Figure 9: (a) Lift 1. (b) The corresponding geometric augmented Hasse graphs.

(a)

(b)

Figure 10: (a) Lift 2. (b) The corresponding geometric augmented Hasse graphs.

(a)

(b)

Figure 11: (a) Lift 3. (b) The corresponding geometric augmented Hasse graphs.

##  Appendix F Complexity Analysis

Given the flexibility in the definition of cells and neighborhood functions,
the complexity of an ETNN layer can vary. To provide a reference complexity,
let us consider a CC with $|\mathcal{X}|$ cells, and the up/down incidences
and adjacencies from (1)-(2) as neighborhood functions. A cell
$x\in\mathcal{X}$ has then
$|\mathcal{N}_{I,\uparrow}(x)|+|\mathcal{N}_{I,\downarrow}(x)|$ incident cells
and there are
$|\mathcal{N}_{A,\uparrow}(x)|+|\mathcal{N}_{A,\downarrow}(x)|=\binom{|\mathcal{N}_{I,\uparrow}(x)|+|\mathcal{N}_{I,\downarrow}(x)|}{2}$
up/down adjacencies between them. Therefore, each node $z\in\mathcal{S}$ has
$|\mathcal{N}_{A,\downarrow}(z)|=\binom{|\mathcal{N}_{I,\uparrow}(z)|}{2}$
adjacent nodes. Then, an ETNN layer as in (6)-(7) has a computational
complexity given by

$\displaystyle\Theta\left(\underbrace{\sum_{x\in\mathcal{X}}\left(\left(|\mathcal{N}_{I,\uparrow}(x)|+\mathcal{N}_{I,\downarrow}(x)|\right)+\binom{|\mathcal{N}_{I,\uparrow}(x)|+|\mathcal{N}_{I,\downarrow}(x)|}{2}\right)}_{\textrm{$\mathbf{h}_{x}$
update from
\eqref{eq:eq_message_passing}}}+\underbrace{\sum_{z\in\mathcal{S}}\binom{|\mathcal{N}_{I,\uparrow}(z)|}{2}}_{\textrm{$\mathbf{x}_{z}$
update from \eqref{eq:eq_update_pos}}}\right)$
$\displaystyle=\Theta\left(\sum_{x\in\mathcal{X}}\left(\left(|\mathcal{N}_{I,\uparrow}(x)|+\mathcal{N}_{I,\downarrow}(x)|\right)+\left(|\mathcal{N}_{A,\uparrow}(x)|+|\mathcal{N}_{A,\downarrow}(x)|\right)\right)+\sum_{z\in\mathcal{S}}|\mathcal{N}_{A,\uparrow}(z)|\right)$
$\displaystyle=\Theta\left(\sum_{x\in\mathcal{X}}\left(|\mathcal{N}_{A,\uparrow}(x)|+|\mathcal{N}_{A,\downarrow}(x)|\right)\right).$
(26)

If we consider the connectivity of the CC to be sufficiently sparse, which is
usually the case, then the neighborhoods’ cardinalities can be absorbed in the
bound. Therefore, the overall complexity will be linear in the number of the
cells $\Theta\left(|\mathcal{X}|\right)$. Clearly, in the worst case, each
cell could be connected to all the other cells, resulting in a quadratic cost
in the number of cells.

##  Appendix G Real-World Combinatorial Topological Spaces

###  G.1 Molecular Data

Molecular graphs. One of the most renowned representations of molecules is
graphs. In this case, atoms are the nodes, and bonds among them are the edges
[75, 90, 91]. Message passing networks on molecular graphs have been proven to
be effective on several tasks. However, graphs cannot explicitly model higher
order, multiway interactions among the atoms. For instance, graphs cannot
explicitly consider the information coming from the rings, i.e. the induced
cycles of the molecular graphs. To represent different atomic relations,
efforts focused on learning representations for small molecules and larger
biomolecules (i.e. proteins) via multi-view or relational graphs [92, 93, 37,
94].

Molecular cell complexes. To overcome the above limitation, cell complexes
(CWs), being a generalization of simplicial complexes [27, 16], have been
employed [18]. Molecular CWs can be seen as "hierarchical augmented graphs" in
which atoms are $0$-cells (nodes), bonds are $1$-cells (edges), and rings are
$2$-cells (induced cycles). Message passing networks on molecular CWs have
shown superior performance w.r.t. models on molecular graphs [22, 18].
However, CWs cannot model higher order interactions among the atoms and bonds
that are not described by a cycle in the underlying molecular graph. For
instance, molecular CWs cannot explicitly consider the information coming from
the functional groups of the molecule, i.e. specific motifs (not necessarily
cycles) of the underlying molecular graphs.

Molecular hypergraphs. Hypergraphs (HGs) can model arbitrary higher order
interactions due to the flexibility of hyperedges. HGs and message passing
networks on HGs have been employed for modeling and learning from molecules
[95, 96]. In molecular HGs, atoms are nodes, and bonds, rings, functional
groups, etc, are hyperedges. However, HGs cannot explicitly consider any
hierarchy in the hyperedges, thus putting pairwise and higher-order
interactions of any kind on the same level.

###  G.2 Geospatial CCs

Spatial graphs and CTSs. The single resolution version of the spatial setting
in 4 has been traditionally handled using proximity graphs, i.e. geometric
graphs whose connectivity is dictated by some notion of spatial closeness In
the multiresolution case, the most common approach is to aggregate all the
data at a common denominator of spatial resolution. However, substantial
information is clearly lost in this process. Some works used hypergraphs [97],
whose limitation have been explained in the previous subsection. Some
Topological Data Analysis (TDA) work leveraged simplicial complexes [98, 99,
100, 101, 102]. However, TDL should not be confused with TDA, which is a data-
driven approach to describe the mathematical properties of a topological
space. Instead, TDL focuses on modeling properties of data defined over such
topological spaces [10].

##  Appendix H Experimental Setup

###  H.1 Molecular Property Prediction

In this section, we present all dataset information, data featurization, and
model parameter configuration for the molecular property prediction task.

####  H.1.1 Dataset Statistics

QM9 dataset consists of over $130,000$ molecules with $12$ property prediction
targets. See Table 4 for a detailed list of the $12$ molecular properties, as
well as statistics over the number of molecules, atoms, bonds, rings, and
functional groups.

Table 4: Statistics of the QM9 Dataset

Property | Description | Units
---|---|---
$\alpha$ | Electronic spatial extent | bohr3
$\Delta\epsilon$ | Gap between $\epsilon_{HOMO}$ and $\epsilon_{LUMO}$ | meV
$\epsilon_{HOMO}$ | Energy of the highest occupied molecular orbital | meV
$\epsilon_{LUMO}$ | Energy of the lowest unoccupied molecular orbital | meV
$\mu$ | Dipole moment | D
$C_{v}$ | Heat capacity at 298.15 K | cal/mol K
$G$ | Free energy at 298.15 K | meV
$H$ | Enthalpy at 298.15 K | meV
$R^{2}$ | Radius of gyration | bohr2
$U$ | Internal energy at 298.15 K | meV
$U_{0}$ | Internal energy at 0 K | meV
$ZPVE$ | Zero-point vibrational energy | meV
Number of molecules |  | 133,885
Average number of atoms |  | 18 nodes
Average number of bonds |  | 19
Average number of rings |  | 1.6
Average number of functional groups |  | 1.2

####  H.1.2 Cell Features

For the molecular prediction task, we considered different peculiar feature
vectors for each rank.

  1. 1.

0-cells features (atom features):

     * •

Atom Types: QM9 molecular samples consist of $5$ atoms: hydrogen (H), carbon
(C), nitrogen (N), oxygen (O), and fluorine (F). We used one-hot encoding for
the atom types.

     * •

Functions over atom types: Following Cormorant’s methodology [103], we
computed the atomic fraction (current atomic number divided by the maximum
atomic number), multiplied the atomic number with the atomic fraction, and
computed the squared quantities.

  2. 2.

1-cells features (bond features):

     * •

Bond Types: We can meet $4$ different bond types in QM9 molecules: single,
double, triple, aromatic. We used one-hot encoding for the bond types.

     * •

Conjugation: A binary feature telling whether the bond is part of a conjugated
system, affecting electronic and optical properties.

     * •

Ring Membership: binary feature telling whether the bond is part of a ring,
affecting electronic and structural characteristics of the molecule.

  3. 3.

2-cells features (ring and functional group features):

Ring features

     * •

Ring Size: Number of atoms that form the ring.

     * •

Aromaticity: A binary feature telling whether the ring is aromatic.

     * •

Has Heteroatom: A binary feature telling whether the ring contains a
heteroatom, which is an atom other than carbon, such as nitrogen, oxygen, or
sulfur. Heteroatoms can introduce different electronic effects and reactivity
patterns within the ring structure.

     * •

Saturation: A binary feature telling whether the ring is saturated, meaning
all the bonds between carbon atoms are single bonds.

Functional Group features

     * •

Conjugation: A binary feature telling whether the functional group is part of
a conjugated system, which involves alternating double and single bonds.

     * •

Acidity: The ability of the functional group to donate protons ($H+$ ions) in
a solution. Possible values: ["neutral", "high", "basic", "weakly acidic"].

     * •

Hydrophobicity: Tendency of the functional group to repel water. $3$ Possible
values: [hydrophilic, "moderate", "hydrophobic"]

     * •

Electrophilicity: Tendency of the functional group to accept electrons during
a chemical reaction. High electrophilicity means the functional group readily
accepts electrons, influencing its reactivity with nucleophiles. $3$ Possible
values: ["low", "moderate", "high"].

     * •

Nucleophilicity: Tendency of the functional group to donate electrons during a
chemical reaction. High nucleophilicity means the functional group readily
donates electrons, affecting its reactivity with electrophiles. $3$ possible
values: ["low", "moderate", "high"].

     * •

Polarity: Distribution of electron density within the functional group. High
polarity can significantly affect solubility, reactivity, and interactions
with other molecules. $3$ possible values: ["low", "moderate", "high"].

####  H.1.3 Parameter Configuration

Next, we present the ETNN’s hyperparameters we used for the molecular property
prediction tasks. Table 5 shows the list of the hyperparameters of the
optimizer, as well of our model. Specifically:

  * •

Following EGNN’s experimentation setup, we use $1e$-$3$ as the initial
learning rate for the properties
$\Delta\epsilon,\epsilon_{\text{HOMO}},\epsilon_{\text{LUMO}}$, and $5e$-$4$
for the rest of the properties.

  * •

We calibrate the number of hidden units (i.e. layer width) among
$\\{70,104,128,182\\}$ for each configuration, so that we maintain roughly the
same size of parameter space ($\approx$ 1.5M). The motivation for this choice
is to provide a fair comparison, and more specifically to show the
contribution of imposing structure into the parameter space.

  * •

All the reported configurations were run for $350$ epochs. This deviates from
the original EGNN [26] model (that uses $1000$ epochs), see Appendix I for
further motivation and details.

  * •

We used the same train/validation/test splits as introduced in EGNN [26]. See
Appendix I for further motivation and details.

  * •

For all the reported configurations we normalize the geometric invariants via
batch normalization. That includes the normalization of pairwise distances,
Hausdorff distances and volumes. Moreover, we perform gradient clipping to a
maximum norm of gradients equal to $1.0$. See Appendix I for further
motivation and details.

###  H.2 Hyperlocal Air Pollution Downscaling

####  H.2.1 Details on the Features used in the Geospatial Benchmark

The features for the census tracts contain use coarse-resolution estimates of
mean $\textrm{PM}_{2.5}$ values for the last quarter of 2020 [104]. We project
them to the census tract level using bilinear interpolation. We also generate
land-use features from the Primary Land Use Tax Lot Output (PLUTO) Data by the
NYC Department of City Planning [105]. For rank-1 features, we use the 2020
annual average daily traffic data (AADT) data from the NYC Department of
Transportation [106]. Lastly, for the point level, we first include the median
time of measurement to account for the fact that sensor measurements are not
uniform throughout the day. We consider only points that correspond to
weekdays between 9 and 5 pm. Lastly, we generate additional point-level
features using the osmnx Python package [107], calculating metrics such as
density and distance to points of interest within a buffer radius. We remove
roads and tracts that do not contain any measurements. The resulting dataset
contains 3,946 point measurements, 550 roads, and 151 census tracts,
corresponding to 0-, 1- and 2-cells, respectively. We featurize the cells
similarly to [38]. Code for reproducibility is provided in the package
repository.

####  H.2.2 Training and Evaluation

The training and evaluation scheme is similar to the molecular case. For
training, we use the Huber loss which provides additional robustness to
outliers [108]. The evaluation metric is obtained similarly to molecular task.
Since the data consists of a single graph, we use a standard procedure for
spatial data and generate training, validation, and test masks. To do so while
being mindful of spatial leaks and correlations, we randomly select 70% of
census tracts for training and split the remaining 30% equally in test and
validation tracts. Since points in the dataset only belong to a single census
tract, we can use these splits to partition points in their corresponding
sets, ensuring that neighboring points belong, on average, to the same split.
While all the nodes (training/validation/test) are given as inputs to the
model, the loss function for backpropagation is only computed from the error
in the training nodes, as in any standard semisupervised setting [7].

The evaluation metric is the root-mean-squared error (RMSE) in the test split
from the best model of each baseline. The base model is determined by the
lowest RMSE in the validation RMSE at any point during training from a sweep
over the hyperparameters described in Table 6. The test split only reports the
final metric to ensure validity. The EGNN baseline used the same architecture
but no lower/upper adjacencies. It also has higher-order features concatenated
with point-level features. The multi-layer perception and linear baselines are
evaluated similarly.

Table 5: Hyperparameters for ETNN Model in the Molecular Task Hyperparameter | Value
---|---
Optimizer | Adam
Initial Learning Rate | [$5*10^{-4}$, $10^{-3}$]
Learning Rate Scheduler | Cosine Annealing
Weight Decay | 1e-5
Batch Size | 96
Epochs | $[100,200,350,1000]$
Model |
Number of Message Passing Layers | [4, 7, 10]
Hidden Units per Layer | $[70,104,128,182]$
Activation Function | SiLU
Invariant Normalization | [True, False]
Gradient Clipping | [True, False]

##### Computational Resources

All experiments that produced the reported configurations in both tasks were
conducted using Nvidia A100-SXM4-40GB GPUs. Each system has a capacity of 80GB
RAM, 40GB GPU memory, and $6$ CPUs per unit.

Table 6: Hyperparameters for ETNN Model in the Air Pollution Downscaling Task Hyperparameter | Value
---|---
Optimizer | Adam
Initial Learning Rate | [$10^{-3}$, $10^{-2}$]
Learning Rate Scheduler | Cosine Annealing
Weight Decay | 1e-4
Batch Size | 1
Epochs | $500$
Model |
Number of Message Passing Layers | 4
Hidden Units per Layer | $[4,32]$
Activation Function | SiLU
Invariant Normalization | False
Gradient Clipping | True
Dropout | [0.025, 0.25]

##  Appendix I Ablation Study and Reproducibility for Molecular Property
Prediction

In this section, we present the whole set of experiments we carried out to
obtain our results in Tables 7 and 8, i.e. all the different configurations of
ETNN we tested. Moreover, we want to further specify some details that are
useful to fairly evaluate our results and ensure reproducibility.

Reproducibility. When we started to set up the experiments, we found a poor
reproducibility level for the molecular prediction task on QM9 in the
literature. In particular, practically every model of the ones listed in Table
1 uses different data splits. The two closest models to ETNN are EGNN [26] and
EMPSN [27], which are also particular cases of our framework. We were not able
to reproduce the results of EMPSN (and this is also the reason we did not
include it in Table 1), while we were able to reproduce the EGNN results using
the original codebase. For all of the reasons above, we decided to employ the
same data splits as EGNN [26]. As explained in Appendix C.1, EGNN can be
directly derived from ETNN. Before exploring different ETNN configurations, we
aligned our training and architecture parameters with EGNN. We ensured that
our ETNN code, configured as an EGNN (referred to as ETNN-graph), could
reproduce all 12 properties of the QM9 dataset reported in the original EGNN
paper [26]. However, training for 1000 epochs as required by the original EGNN
was beyond our computational and time constraints. Consequently, we could not
test all the ETNN configurations presented in the next section by training
them for 1000 epochs. To address this limitation, we employed two training
techniques not present in the original EGNN: geometric invariant normalization
and gradient clipping. These techniques allowed our ETNN-graph configuration
to reproduce as many properties as possible within 350 epochs. We observed
that our ETNN-graph could perfectly reproduce the original EGNN results for
the 6 properties presented in Table 1 within 350 epochs. Consequently, we
included these properties in the main body. In the next paragraph, we describe
in detail all the configurations of ETNN we tested (again, on 350 epochs with
normalization and gradient clipping), and we also report the original EGNN
results, the results of ETNN-graph on 1000 epochs, the results of ETNN-graph
on 350 epochs, and the results of ETNN-graph-W, a variant of ETNN-graph with
more hidden units.

Results. In Tables 7 and 8, we present the full list of ETNN configurations we
tested, representing also an exhaustive ablation study on the main components
of our architectures. We use the following coding for names:
ETNN-$\\{\textrm{cells}\\}$-$\\{\textrm{neighborhoods}\\}$.$\\{\textrm{cells}\\}$
is always a subset of $\\{\textrm{atoms, bonds, rings, functional groups,
virtual cell}\\}$ and indicates which cells we are using in the Molecular CC;
the rank is assigned in the same order, e.g. $\\{\textrm{bonds, functional
groups}\\}$ means that atoms are 0-cells and functional groups are 1-cells.
$\\{\textrm{neighborhoods}\\}$ is always a subset of
$\\{\mathcal{N}_{I,\uparrow},\mathcal{N}_{I,\downarrow},\mathcal{N}_{A,\uparrow},\mathcal{N}_{A,\downarrow},\mathcal{N}_{A,\textrm{max}}\\}$,
being the up/down incidences and adjacencies from (1)-(2), and the max-rank
adjacency from (13), respectively. For instance, EGNN as used in the original
paper [26], can be obtained as ETNN-$\\{\textrm{atoms, virtual
cell}\\}$-$\\{\mathcal{N}_{A,\textrm{max}}\\}$ (and this is obviously our
ETNN-graph). As stated above, ETNN-graph is EGNN derived from the ETNN
framework with our codebase (thus it has the original number of parameters of
748K), while EGNN-W is a 1.5M parameters version of ETNN-graph, which we
inserted for fairness because all of the variants of ETNN we present have
exactly 1.5M parameters. ETNN-graph (1000 epochs) is ETNN-graph trained for
1000 epochs, without gradient clipping and geometric invariant normalization,
thus the exact reproduction of EGNN through the ETNN framework with our
codebases, as explained in the previous paragraph. ETNN-graph-only_graph_adj
is ETNN-graph but without the virtual cell, thus the actual molecular graph is
used instead of a fully connected one. Interestingly, the last configuration
is also an invariant variant of [16, 18]. Finally, no_feat means that no
features are used except for atom features. As the reader can notice, several
higher order ETNN configurations reach significant improvements over the
reference baselines, ETNN-graph and ETNN-graph-W. These results validate our
TDL approach. Moreover, they show that the flexibility of CCs is necessary, as
not a single architecture, fixed the underlying combinatorial domain, is able
to outperform all the others across all the properties.

Table 7: Mean Absolute Error for the first six molecular property predictions
in the QM9 dataset.

Task | $\alpha$ | $\Delta\varepsilon$ | $\varepsilon_{\mathrm{HOMO}}$ | $\varepsilon_{\mathrm{LUMO}}$ | $\mu$ | $C_{\nu}$
---|---|---|---|---|---|---
Units | bohr3 | meV | meV | meV | D | cal/mol K
NMP | .092 | 69 | 43 | 38 | .030 | .040
Schnet | .235 | 63 | 41 | 34 | .033 | .033
Cormorant | .085 | 61 | 34 | 38 | .038 | .026
L1Net | .088 | 68 | 46 | 35 | .043 | .031
LieConv | .084 | 49 | 30 | 25 | .032 | .038
DimeNet++* | .044 | 33 | 25 | 20 | .030 | .023
TFN | .223 | 58 | 40 | 38 | .064 | .101
SE(3)-Tr. | .142 | 53 | 35 | 33 | .051 | .054
EGNN | .071 | 48 | 29 | 25 | .029 | .031
EMPSN | .066 | 37 | 25 | 20 | .023 | .024
ETNN |  |  |  |  |  |
ETNN-graph (1000 epochs) | .071 | 48 | 29 | 25 | .029 | .031
ETNN-graph | .071 | 49 | 30 | 25 | .029 | .036
ETNN-graph-W | .067 | 46 | 27 | 25 | .030 | .036
ETNN-graph-only_graph_adj | .161 | 73 | 49 | 49 | .306 | .052
ETNN-$\\{\textrm{atoms, bonds, virtual cell}\\}$-$\\{\mathcal{N}_{A,\textrm{max}}\\}$ | .075 | 53 | 30 | 25 | .030 | .040
ETNN-$\\{\textrm{atoms, bonds, functional groups, virtual cell}\\}$-$\\{\mathcal{N}_{A,\textrm{max}}\\}$ | .078 | 53 | 32 | 28 | .030 | .034
ETNN-$\\{\textrm{atoms, bonds, rings, functional groups, virtual cell}\\}$-$\\{\mathcal{N}_{A,\textrm{max}}\\}$ | .078 | 54 | 33 | 27 | .032 | .035
ETNN-$\\{\textrm{atoms, bonds, virtual cell}\\}$-$\\{\mathcal{N}_{I,\uparrow},\mathcal{N}_{I,\downarrow},\mathcal{N}_{A,\uparrow},\mathcal{N}_{A,\downarrow},\mathcal{N}_{A,\textrm{max}}\\}$ | .071 | 54 | 30 | 26 | .023 | .039
ETNN-$\\{\textrm{atoms, bonds, functional groups, virtual cell}\\}$-$\\{\mathcal{N}_{I,\uparrow},\mathcal{N}_{I,\downarrow},\mathcal{N}_{A,\uparrow},\mathcal{N}_{A,\downarrow},\mathcal{N}_{A,\textrm{max}}\\}$ | .085 | 56 | 30 | 27 | .024 | .042
ETNN-$\\{\textrm{atoms, bonds, virtual cell}\\}$-$\\{\mathcal{N}_{A,\textrm{max}}\\}$-no_feat | .062 | 49 | 30 | 24 | .032 | .033