Architectures of Topological Deep Learning: A Survey of Message-Passing
  Topological Neural Networks
The natural world is full of complex systems characterized by intricate
relations between their components: from social interactions between
individuals in a social network to electrostatic interactions between atoms in
a protein. Topological Deep Learning (TDL) provides a comprehensive framework
to process and extract knowledge from data associated with these systems, such
as predicting the social community to which an individual belongs or predicting
whether a protein can be a reasonable target for drug development. TDL has
demonstrated theoretical and practical advantages that hold the promise of
breaking ground in the applied sciences and beyond. However, the rapid growth
of the TDL literature for relational systems has also led to a lack of
unification in notation and language across message-passing Topological Neural
Network (TNN) architectures. This presents a real obstacle for building upon
existing works and for deploying message-passing TNNs to new real-world
problems. To address this issue, we provide an accessible introduction to TDL
for relational systems, and compare the recently published message-passing TNNs
using a unified mathematical and graphical notation. Through an intuitive and
critical review of the emerging field of TDL, we extract valuable insights into
current challenges and exciting opportunities for future development.
Mathilde Papillon, Sophia Sanborn, Mustafa Hajij, Nina Miolane
# Architectures of Topological Deep Learning:

###### Abstract

The natural world is full of complex systems characterized by intricate relations between their components: from social interactions between individuals in a social network to electrostatic interactions between atoms in a protein. Topological Deep Learning (TDL) provides a comprehensive framework to process and extract knowledge from data associated with these systems, such as predicting the social community to which an individual belongs or predicting whether a protein can be a reasonable target for drug development. TDL has demonstrated theoretical and practical advantages that hold the promise of breaking ground in the applied sciences and beyond. However, the rapid growth of the TDL literature has also led to a lack of unification in notation and language across Topological Neural Network (TNN) architectures. This presents a real obstacle for building upon existing works and for deploying TNNs to new real-world problems. To address this issue, we provide an accessible introduction to TDL, and compare the recently published TNNs using a unified mathematical and graphical notation. Through an intuitive and critical review of the emerging field of TDL, we extract valuable insights into current challenges and exciting opportunities for future development.

**Keywords**: deep learning, topology, message passing, graph, hypergraph, simplicial complex, cellular complex, combinatorial complex

## 1 Introduction

Many natural systems as diverse as social networks (Knoke and Yang, 2019) and proteins (Jha et al., 2022) are characterized by _relational structure_. This is the structure of interactions between components in the system, such as social interactions between individuals or electrostatic interactions between atoms. In Geometric Deep Learning (Bronstein et al., 2021), Graph Neural Networks (GNNs) (Zhou et al., 2020) have demonstrated remarkable achievements in processing relational data using graphs--mathematical objects commonly used to encode _pairwise relations_.

However, the pairwise structure of graphs is limiting. Social interactions can involve more than two individuals, and electrostatic interactions more than two atoms. Topological Deep Learning (TDL) (Hajij et al., 2023; Bodnar, 2023) leverages more general abstractions to process data with higher-order relational structure. The theoretical guarantees (Bodnar et al., 2021, 2021, 2022) of its models, Topological Neural Networks (TNNs), lead to state-of-the-art performance on many machine learning tasks (Long et al., 2020; Hajij et al., 2022; Barbarossa and Sardellitti, 2020; Chen et al., 2022)--and reveal high potential for the applied sciences and beyond.

However, the abstraction and fragmentation of mathematical notation across the TDL literature significantly limits the field's accessibility, while complicating model comparison and obscuring opportunities for innovation. To address this, we present an intuitive and systematic comparison of published TNN architectures. Compared to reviews that focus on data representation (Torres et al.,
2021), physics-inspired models (Battiston et al., 2021), and topological data analysis for machine learning (Hensel et al., 2021), we contribute:

* **A pedagogical resource** accessible to newcomers interested in applying TNNs to real-world problems.
* **A unified notational and graphical taxonomy** to intuitively summarize the landscape of TNN architectures.
* **A comprehensive and critical review** of TNNs, their implementations and practical applications, with equations rewritten in our notation available at github.com/awesome-tnns.
* **A summary of open research questions**, challenges, and opportunities for innovation.

By establishing a common and accessible language in the field, we hope to provide newcomers and experienced practitioners alike with a solid foundation for cutting-edge research in TDL.

## 2 Topological Neural Networks

Topological Neural Networks (TNNs) are deep learning architectures that extract knowledge from data associated with topologically rich systems such as protein structures, city traffic maps, or citation networks. A TNN, like a GNN, is comprised of stacked layers that transform data into a series of features (Figure 1). Each layer leverages the fundamental concepts of _data and computational domains_, _neighborhoods_, and _message passing_--presented in this section.

### Domains

In Topological Deep Learning (TDL), data are features defined on discrete domains. Traditional examples of discrete domains include sets and graphs (Figure 2, Left). A **set** is a collection of points called _nodes_ without any additional structure. A **graph** is a set with _edges_ that encode _pairwise relations_ between nodes, representing either geometric proximity or more abstract relationships. For example, a graph may represent a protein, with nodes encoding its atoms and edges encoding the pairwise bonds between them. Alternatively, a graph may represent a social network, where nodes represent individuals and edges denote social relationships. The domains of TDL go _beyond graphs_, generalizing pairwise relations to _part-whole_ and _set-types_ relations that permit the representation of more complex relational structure (Figure 2, Right). Here, we describe the key attributes of each domain and highlight their suitability for different data types. We refer the reader to Torres et al. (2021) and Hajij et al. (2023) for more extensive discussions.

Figure 1: **Topological Neural Network**: Data associated with a complex system are features defined on a _data domain_, which is preprocessed into a _computational domain_ that encodes interactions between the system’s components with _neighborhoods_. The TNN’s layers use _message passing_ to successively update features and yield an output, e.g. a categorical label in classification or a quantitative value in regression. The output represents new knowledge extracted from the input data.
**Beyond Graphs: The Domains of Topological Deep Learning**

Set + Pairwise Relations

**Graph**: A set of points (_nodes_) connected with _edges_ that denote pairwise relationships.

Set + Part-Whole Relations

**Simplicial Complex** (SC): A generalization of a graph in which three edges can form a _triangular face_, four triangles can form a _tetrahedral volume_, and so on. Edges only connect pairs of nodes.

**Cellular Complex** (CC): A generalization of an SC in which _faces, volumes, etc_ are not restricted to be triangles or tetrahedrons but may instead take any shape. Still, edges only connect pairs of nodes.

Set + Set-Type Relations

**Hypergraph** (HG): A generalization of a graph, in which higher-order edges called _hyperedges_ can connect arbitrary _sets_ of two or more nodes.

Set + Part-Whole and Set-Type Relations

**Combinatorial Complex** (CCC): A structure that combines features of HGs and CCs. Like an HG, edges may connect any number of nodes. Like a CC, cells can be combined to form higher-ranked structures.

**Simplicial complexes (SCs)** generalize graphs to incorporate hierarchical _part-whole relations_ through the multi-scale construction of _cells_. Nodes are _rank 0_ cells that can be combined to form edges (_rank 1_ cells). Edges are, in turn, combined to form faces (_rank 2_ cells), which are combined to form volumes (_rank 3_ cells), and so on. As such, an SC's faces must be triangles, volumes must be tetrahedrons, and so forth. SCs are commonly used to encode discrete representations of 3D geometric surfaces represented with triangular meshes (Figure 3). They may also be used to represent more abstract relations; however, there is a risk of introducing spurious connections if the strict geometric constraints of an SC are not respected by the data--a point we elaborate on in Section 2.1.2.

**Cellular complexes (CCs)** generalize SCs such that cells are not limited to simplexes: faces can involve more than three nodes, volumes more than four faces, and so on. This flexibility endows CCs with greater expressivity than SCs (Bodnar et al., 2021b). A practitioner should consider employing this domain when studying a system that features part-whole interactions between more than three nodes, such as a molecule with benzene rings (Figure 3).

**Hypergraphs (HGs)** extend graphs in that their edges, called _hyperedges_, can connect more than

Figure 2: Domains: Nodes in blue, (hyper)edges in pink, and faces in dark red. Inspired by Hajij et al. (2022a).
two nodes. Connections in HGs represent _set-type relationships_, in which participation in an interaction is not implied by any other relation in the system. This makes HGs an ideal choice for data with abstract and arbitrarily large interactions of equal importance, such as semantic text and citation networks. Protein interaction networks (Figure 3) also exhibit this property: an interaction between proteins requires a precise set of molecules--no more and no less. The interaction of Proteins A, B, and C does not imply an interaction between A and B on their own.

**Combinatorial complexes (CCCs)** generalize CCs and HGs to incorporate both _part-whole_ and _set-type_ relationships. The benefit of this can be observed in the example of molecular representation. The strict geometric constraints of simplicial and cellular complexes are too rigid for capturing much of hierarchical structure observed in molecules. By contrast, the flexible but hierarchically ranked hyperedges of a combinatorial complex can capture the full richness of molecular structure, as depicted in Figure 3.

#### 2.1.1 Terminology

Across discrete domains, we use the term _cell_ to denote any node or relation between nodes such as (hyper)edges, faces, or volumes. Cells possess two attributes: _size_--the number of cells it contains--and _rank_--where nodes are said to have rank 0, edges and hyperedges rank 1, faces rank 2, and so on. The part-whole relationships of simplicial and cellular complexes impose a relationship between the _rank_ of a cell and its _size_: cells of rank \(r\) contains exactly (resp. at least) \(r+1\) cells of rank \(r-1\): faces (\(r=2\)) contain exactly (resp. at least) three edges (\(r-1=1\)). By contrast, hypergraph cells do not encode part-whole relations and hyperedges may have any size. However, hypergraph cells are limited to ranks 0 and 1. A combinatorial complex is unrestricted in both rank and size: nodes have rank 0 and cells of any size \(>1\) can have any rank.

There is an important distinction between the inherent domain of the data (the _data domain_) and the domain in which the data will be processed within a TNN: the _computational domain_. Data defined on a graph, for example, may be "lifted" (Figure 4) to an alternative domain through a pre-processing stage (Figure 1). For instance, a protein originally given as the graph of its atoms (nodes) and covalent bounds (edges) may be lifted into a CC computational domain that explicitly represents its rings (faces). In this review, _domain_ refers to the computational domain. Additionally, the computational domain may be _dynamic_, changing from layer to layer in a TNN.

Figure 3: **Examples of Data on Topological Domains.** (a) Higher-order interactions in protein networks. (b) Limited molecular representation: rings can only contain three atoms. (c) Triangular mesh of a protein surface. (d) More flexible molecular representation, permitting the representation of any ring-shaped functional group. (e) Flexible mesh which includes arbitrarily shaped faces. (f) Fully flexible molecular representation, permitting the representation of the complex nested hierarchical structure characteristic of molecules and other natural systems. (g) Hierarchical higher-order interactions in protein networks.
#### 2.1.2 Limitations

An important limitation of both SCs and CCs is that faces (and analogous higher-order structures) can only form rings; the nodes on the boundary of the face must be connected in pairs. In many cases, this requirement is too stringent and can introduce artificial connections to the domain (Yang et al., 2022). For instance, lifting a citation network into an SC necessarily requires that any set of three co-authors having written a paper together (A, B, C) are also pairwise connected (A and B, A and C, B and C), even if no paper was ever exclusively authored by authors A and B, authors A and C, or authors B and C. Yang et al. (2022) propose a "relaxed" definition of the SC that remedies this. They show how training a TNN on such a modified domain increases performance. We note that even with artificial connections, SCs and CCs allow TNNs to leverage richer topological structure and avoid computational problems faced by GNNs (Rusch et al., 2023). We further note that any topological domain is mathematically equivalent to a (possibly larger) graph (Velickovic, 2022). We choose to express domains in their form above in order to provide better intuition to newcomers and reflect the widely adopted approaches in the literature.

#### 2.1.3 Features on a Domain

Consider a domain, denoted \(\mathcal{X}\), encoding relationships between components of a system. Data on the domain are represented as features supported on the domain's cells. Typically, features are vectors in \(\mathbb{R}^{d}\) that encode attributes of each cell. For example, features may encode the atom (node), bond (edge), and functional group (face) types in a molecule. A feature associated with the interaction between a set of drugs (hyperedge) could indicate the probability of adverse reaction.

We denote with \(\mathbf{h}_{x}^{t_{(r)}}\) a feature supported on the cell \(x\in\mathcal{X}\) at layer \(t\) of the TNN, with \(r\) indicating the rank of \(x\) (Figure 5). The domain is decomposed into ranks, with \(X^{(r)}\), or \(r\)_-skeleton_, referring to all cells of rank \(r\). Features can be categorical or quantitative. If the feature dimension varies across skeletons, the domain is _heterogeneous_.

Figure 4: **Lifting Topological Domains.** (a) A graph is “lifted” to a hypergraph by adding hyperedges that connect groups of nodes. (b) In the process of lifting a graph to a simplicial complex, a pairwise edge must be added in order to form triangular faces. (c) A graph can be converted to a cellular complex by adding faces of any shape. (d) Hyperedges can be added to a cellular complex to lift the structure to a combinatorial complex. Figure adapted from Hajij et al. (2023).
**Heterogeneous Domains**

**Homogeneity vs. Heterogeneity**: In a _heterogeneous_ domain, the dimension \(d_{r}\) of a feature \(\mathbf{h}_{x}^{(r)}\) depends on the rank \(r\) of the cell \(x\) supporting it. A _homogeneous_ domain uses the same dimensionality \(d\) for all ranks.

The features assigned to each cell may come directly from the data or be hand-designed by the practitioner. Alternatively, features can be assigned in a pre-processing stage using _embedding_ methods, which compute cell feature vectors that encode the local structure of the space. For graphs, methods such as DeepWalk (Perozzi et al., 2014) and Node2Vec (Grover and Leskovec, 2016) are commonly used to embed nodes. Recent works have generalized these approaches to topological domains: Hyperedge2Vec (Sharma et al., 2018) and Deep Hyperedge (Payne, 2019) for hypergraphs, Simplex2Vec (Billings et al., 2019) and k-Simplex2Vec (Hacker, 2020) for simplicial complexes, and Cell2Vec (Hajji et al., 2020) for cellular complexes.

### Neighborhood Structure

A TNN successively updates cell features throughout its layers by using a notion of _nearness_ between cells: the _neighborhood structure_ (Figure 1). Neighborhood structures are defined by **boundary relations**, which describe how cells of different ranks relate to each other. A cell \(y\) of rank \(r\) is said to be on the **boundary** of cell \(x\) of rank \(R\) if it is connected to \(x\) and rank \(r<R\). This relation is expressed as \(y\prec x\). For example, a node connected to an edge is said to be on the boundary of that edge.

Figure 5: **Features on a Domain. Left: Features onto three cells—\(x\), \(y\), and \(z\). Right: Skeletons for the entire complex: \(X^{(0)}\) contains node features, \(X^{(1)}\) contains edge features, and so on.**

Figure 6: **Incidence Matrices. Examples of an SC, a CC, an HG, and a CCC with their corresponding boundary matrices \(B_{1}\) which map from 1-cells to 0-cells. The SC and CC maps are signed to encode edge orientation: the node appearing first in the arbitrary ordering (a,b,c,d) is always assigned -1.**
Boundary relations are encoded in **incidence matrices**. Specifically, we denote with \(B_{r}\) the matrix that records which (regular) cells of rank \(r-1\) bound which cells of rank \(r\) (Figure 6). Formally, \(B_{r}\) is a matrix of size \(n_{r-1}\cross n_{r}\), with \(n_{r}\) denoting the number of cells of rank \(r\geq 1\), defined:

\[(B_{r})_{i,j}=\begin{cases}\pm 1&x_{i}^{(r-1)}\prec x_{j}^{(r)}\\ 0&\text{otherwise},\end{cases} \tag{1}\]

where \(x_{i}^{(r-1)}\), \(x_{j}^{(r)}\) are two cells of ranks \(r-1\) and \(r\) respectively. The \(\pm 1\) sign encodes a notion of orientation required for SCs and CCs (Aschbacher, 1996; Klette, 2000), and is always \(+1\) for HGs and CCCs.

Incidence matrices can be used to encode the four most common neighborhood structures used in the literature, which we define in the text box below. Here, \(L_{\uparrow,0}\) denotes the typical graph Laplacian. Its higher order generalization, the \(r\)-Hodge Laplacian, is \(H_{r}=L_{\downarrow,r}+L_{\uparrow,r}\)(Barbarossa and Sardellitti, 2020; Schaub et al., 2021). \(D_{r}\in\mathbb{N}^{n_{r}\times n_{r}}\) denotes the degree matrix, a diagonal matrix representing the number of connections of \(r\)-cells with \((r+1)\)-cells.

**Neighborhood Structures**

**Boundary Adjacent Neighborhood**\(\mathcal{B}(y)=\{x\mid x\prec y\}\):

The set of \(y\)-connected \(x\) cells of next lower rank. The neighborhood is specified with the _boundary matrix_\(B_{r}\). _Example: The set of nodes \(x\) connected to edge \(y\)_.

**Co-Boundary Adjacent Neighborhood**\(\mathcal{C}(y)=\{x\mid y\prec x\}\):

The set of \(y\)-connected \(x\) cells of next higher rank. The neighborhood is specified with the _co-boundary matrix_\(B_{r}^{T}\). _Example: The set of edges \(x\) connected to node \(y\)_.

**Lower Adjacent Neighborhood**\(\mathcal{L}_{1}(y)=\{x\mid\exists x\text{ s.t. }z\prec y\text{ and }z\prec x\}\):

The set of \(x\) cells that share a boundary \(z\) with \(y\). The neighborhood is specified with either the _lower Laplacian matrix_\(L_{\downarrow,r}=B_{r}B_{r}^{T}\) or the _lower adjacency matrix_\(A_{\downarrow,r}=D_{r}-L_{\downarrow,r}\). _Example: the set of edges \(x\) that connect to any of the nodes \(z\) that touch edge \(y\)_.

**Upper Adjacent Neighbors**\(\mathcal{L}_{\uparrow}(y)=\{x\mid\exists z\text{ s.t. }y\prec z\text{ and }x\prec z\}\):

The set of \(x\) cells that share a co-boundary \(z\) with \(y\). The neighborhood is specified with either the _upper Laplacian matrix_\(L_{\uparrow,r}=B_{r+1}^{T}B_{r+1}\) or the _upper adjacency matrix_\(A_{\uparrow,r}=D_{r}-L_{\uparrow,r}\).

_Example: The set of nodes \(x\) that touch any of the edges \(z\) that touch node \(y\)_.
Figure 7: Neighborhood Structures: their neighborhood matrices and illustrations for a cell \(x\) in the neighborhood of a cell \(y\).
### Message Passing

_Message passing_ defines the computation performed by a single layer \(t\) of the TNN. During message passing, each cell's feature \(\mathbf{h}_{\mathbf{x}}^{t,(\mathbf{r})}\) is updated to incorporate: (1) the features associated with cells in its neighborhood and (2) the layer's learnable parameters denoted \(\Theta^{t}\). The term "message passing" reflects that a signal is "traveling" through the network, passing between cells on paths laid out by the neighborhood structure. The output \(\mathbf{h}^{t+1}\) of layer \(t\) becomes the input to layer \(t+1\). In this way, deeper layers incorporate information from more distant cells, as information diffuses through the network.

#### 2.3.1 The Steps of Message Passing

We decompose message passing into four steps, adapted from the framework of Hajij et al. (2022). Each step is represented with a different color--_red_, _orange_, _green_, or _blue_--illustrated in Figure 8.

**The Steps of Message Passing**

**1. Message:** First, a message \(m_{y\to x}^{(r^{\prime}\to r)}\) travels from a \(r^{\prime}\)-cell \(y\) to a \(r\)-cell \(x\) through a neighborhood \(k\) of \(x\) denoted \(\mathcal{N}_{k}(x)\):

\[m_{y\to x}^{(r^{\prime}\to r)}=M_{\mathcal{N}_{k}}\left(\mathbf{h}_{x}^{t,(r) },\mathbf{h}_{y}^{t,(r^{\prime})},\Theta^{t}\right). \tag{2}\]

via the function \(M_{\mathcal{N}_{k}}\) depicted in red in Figure 8. Here, \(\mathbf{h}_{x}^{t,(r)}\) and \(\mathbf{h}_{y}^{t,(r^{\prime})}\) are features of dimension \(d_{r}\) and \(d_{r^{\prime}}\) on cells \(y\) and \(x\) respectively, and \(\Theta^{t}\) are learnable parameters. In the simplest case, this step looks like a neighborhood matrix \(M\) propagating a feature \(\mathbf{h}_{y}^{t,(r^{\prime})}\) on \(r^{\prime}\)-cell \(y\) to \(r\)-cell \(x\) as:

\[m_{y\to x}^{(r^{\prime}\to r)}=M_{xy}\cdot\mathbf{h}_{y}^{t,(r^{\prime})} \cdot\Theta^{t}, \tag{3}\]

where \(M_{xy}\) is the scalar entry of matrix \(M\) at the row corresponding to cell \(x\) and column corresponding to cell \(y\) and \(m_{y\to x}^{(r^{\prime}\to r)}\) and \(\Theta\) is a \(d_{r^{\prime}}\times d_{r}\) matrix. If \(y\) is not in the neighborhood structure of \(x\), then \(M_{xy}\) will be 0, and \(x\) cannot receive any message from \(y\).

2. Within-Neighborhood Aggregation: Next, messages are aggregated across all cells \(y\) belonging to the neighborhood \(\mathcal{N}_{k}(x)\): \[m_{x}^{(r^{\prime}\to r)}=AGG_{y\in\mathcal{N}_{k}(x)}m_{y\to x}^{(r^{\prime} \to r)},\] (4) resulting in the _within-neighborhood aggregated message_\(m_{x}^{(r^{\prime}\to r)}\). Here, \(AGG\) is an aggregation function, depicted in orange in Figure 8, analogous to pooling in standard convolutional networks.

**3. Between-Neighborhood Aggregation:** Then, messages are aggregated across neighborhoods in a neighborhood set \(\mathcal{N}\):

\[m_{x}^{(r)}=AGG_{\mathcal{N}_{k}\in\mathcal{N}}m_{x}^{(r^{\prime}\to r)}, \tag{5}\]

where \(\mathrm{AGG}\) is a (potentially different) aggregation function depicted in green in Figure 8, and \(m_{x}^{(r)}\) is the message received by cell \(x\) that triggers the update of its feature.

4. Update: Finally, the feature on cell \(x\) is updated via a function \(U\) depicted in blue in Figure 8, which may depend on the previous feature \(\mathbf{h}_{x}^{t,(r)}\) on cell \(x\):

\[\mathbf{h}_{x}^{t+1,(r)}=U\left(\mathbf{h}_{x}^{t,(r)},m_{x}^{(r)}\right), \tag{6}\]

The result \(\mathbf{h}_{x}^{t+1,(r)}\) is the updated feature on cell \(x\) that is input to layer \(t+1\).

In this review, we decompose the structure of TNN architectures proposed in the literature into these four message passing steps--a unified notational framework that allows us to contrast existing
approaches. Many architectures repeat steps and/or modify their order. We note that this conceptualization of message passing as a local, cell-specific operation is called the _spatial approach_(Gilmer et al., 2017). In GNNs and TNNs alike, message passing can alternatively be expressed in its dual _spectral_ form, using global Fourier analysis over the domain. For this review, we choose to write all equations in spatial form for intuitiveness and generality (Bodnar et al., 2021; Hajji et al., 2022; Heydari and Livi, 2022).

#### 2.3.2 Tensor Diagrams

We visually represent message passing schemes with an adapted version of the _tensor diagram_ from Hajji et al. (2022), which provides a graphical representation of a TNN architecture. Figure 8 explains the recipe for constructing a tensor diagram from message passing steps.

Figure 8: **Message passing steps: 1: Message (red), 2: Within-neighborhood aggregation (orange), 3: Between-neighborhood aggregation (green), 4: Update (blue). The scheme updates a feature \(\mathbf{h}_{x}^{t(r)}\) on a \(r\)-cell \(x\) at layer \(t\) (left column) into a new feature \(\mathbf{h}_{x}^{t+1,(r)}\) on that same cell at the next layer \(t+1\) (right column). Here, the scheme uses four neighborhood structures \(\mathcal{N}_{k}\) for \(k\in\{1,2,3,4\}\) (middle column). Inspired by (Hajji et al., 2023).**

Figure 9: **Tensor Diagrams: a graphical notation for the four steps of a message passing scheme. A diagram depicts how a feature on cell \(y\) at layer \(t\), \(\mathbf{h}_{y}^{t(t)}\), becomes a feature on cell \(x\) at layer \(t+1\), \(\mathbf{h}_{x}^{t(t+1)}\).**
#### 2.3.3 Types of Message Passing Functions

The message passing function \(M_{\mathcal{N}_{k}}\) employed in Step 1 is defined by the practitioner. There are three kinds of functions commonly used in the literature, as outlined in Figure 10(Bronstein, 2022). The variety used determines how layer parameters weight each incoming message from cell \(y\) to cell \(x\). The _standard convolutional_ case multiplies each message by some learned scalar. The _attentional convolutional_ case weights this multiplication depending on the features of the cells involved. The _general_ case implements a potentially non-linear function that may or may not incorporate attention. Some schemes also make use of fixed, non-learned weights to assign different levels of importance to higher-order cells. Figure 10 illustrates each type with tensor diagrams.

## 3 Literature Review

We now review the literature on topological neural networks (TNNs) over hypergraphs, simplicial complexes, cellular complexes, and combinatorial complexes, using the conceptual framework of Section 2. We summarize and compare the TNNs in terms of their architectures (Section 3.1), the machine learning tasks to which they have been applied (Section 3.2), and their geometric properties (Section 3.3).

### Architectures

Figure 11 summarizes TNN architectures according to the fundamental concepts introduced in Section 2, with the _domain_ on the vertical axis, the _message passing type_ on the horizontal axis, _neighborhood structures_ and _message passing equations_ visually represented with tensor diagrams. We share complete message passing equations for each architecture--decomposed according to the four steps introduced in Section 2.3.1 and rewritten in unifying notations --at github.com/awesome-tnns.

#### 3.1.1 Hypergraphs

Of the domains considered here, hypergraph neural networks have been most extensively researched, and have been surveyed previously (Ling et al., 2021; Gao et al., 2022; Hu et al., 2021; Wang et al.,

Figure 10: **Types of Message Passing Functions.** In each case, a cell \(x_{i}\) (an edge) receives information from its various neighbors, cells \(y_{j}\) (two nodes, an edge, and a face). The message received by cell \(x_{i}\) from cell \(y_{j}\) is determined by a specific function \(c(x_{i},y_{j})\), \(a(x_{i},y_{j})\), or \(g(x_{i},y_{j})\). Top: Each neighborhood cell \(y_{j}\) sends a message to cell \(x_{i}\). (Inspired by P. Velickovic and (Bronstein, 2022)). Bottom: Illustration of the message-passing scheme above using tensor diagrams.
Figure 11: **Topological Neural Networks (TNNs): A Graphical Literature Review. We organize TNNs according to the _domain_ (rows), the _message passing type_ (columns).**
2021a; Fischer et al., 2021). Many papers in the early literature do not use hypergraphs as the computational domain. Rather, algorithms like clique-expansion (Zien et al., 1999; Agarwal et al., 2005; Zhou et al., 2006) are used to reduce hypergraphs to graphs, which are then processed by the model. This reduction adversely affects performance, as structural information is lost (Hein et al., 2013; Li et al., 2013; Chien et al., 2019). Many such graph-based models--including HGNN (Feng et al., 2019), HyperConv(Bai et al., 2021), HyperGCN (Yadati et al., 2019), and HNHN (Dong et al., 2020)--are used as benchmarks for more recent models that _do_ computationally operate on hypergraphs. Here, we focus on models that preserve hypergraph structure during learning.

Many hypergraph models use a message passing scheme comprised of two phases, with information flowing from nodes to their hyperedges and then back to the nodes. We call this the _two-phase scheme_. The scheme appears in many tensor diagrams of Figure 11 where information flows from blue to pink (phase 1) and then from pink to blue (phase 2). The scheme is used in models with both standard and attentional message passing.

**Standard**

Of those using a standard message passing, the models from Arya et al. (2020), Yi and Park (2020), Wei et al. (2021), and Huang and Yang (2021) use the two-phase scheme. Yi and Park (2020) is unique in using a learnable weight matrix in the first phase of message passing. On the second phase, Wei et al. (2021) and the UniGCN model from Huang and Yang (2021) are unique in using a fixed weight matrix on top of learnable weights. In Arya et al. (2019), Yi and Park (2020), and the UniGNN, UniSAGE, and UniGCNII models from Huang and Yang (2021), the initial feature on each node is recurrently used to update each incoming message--denoted with a looped black arrow in Figure 11. We note that Huang and Yang (2021) systematically generalizes some of the most popular GNN architectures to hypergraphs with its unifying framework: UniGNN.

In Dong et al. (2020), fixed weights are used on both the node to hyperedge and hyperedge to node phases. The paper AllSet (Chien et al., 2022) uses a similar structure while incorporating fully learnable multi-set functions for neighborhood aggregation, which imbues its TNNs with high expressivity and generality. EHNN (Kim et al., 2022) (excluded from Figure 11 for its complexity; see written equations) proposes a maximally expressive model using sparse symmetric tensors to process data on hypergraphs with uniformly sized hyperedges.

**Attentional / General**

The models from Jiang et al. (2019), Ding et al. (2020), Yi and Park (2020), Wang et al. (2021a), and the UniGAT model from Huang and Yang (2021) employ the two-phase scheme in concert with attentional message passing. The architectures of Li et al. (2022) and Li et al. (2022) apply multi-head attention. Heydari and Livi (2022) adapts the two-phase scheme in order to update node features through two parallel paths. Chien et al. (2022) and Kim et al. (2022) offer transformer-based variants of their standard architectures, concurrently with Li et al. (2022).

#### 3.1.2 Simplicial Complexes

Simplicial complexes were first explored from a signal processing perspective (Battiston et al., 2020), with initial focus on edge flows (Jiang et al., 2011; Schaub and Segarra, 2018), Hodge Laplacians (Barbarossa and Sardellitti, 2020; Schaub et al., 2020), and convolution (Yang et al., 2021, 2022b; Isufi and Yang, 2022). As a precursor to deep learning, Roddenberry and Segarra (2019) introduced \(\mathcal{L}_{\downarrow,1}\) in HodgeNet to learn convolutions on edge features on graphs. This contrasts with former GNN approaches processing node features.

**Standard**

Ebli et al. (2020) (SNN) and Bunch et al. (2020) (SCCONV) first generalized the convolutional approach of Roddenberry and Segarra (2019) to features supported on faces and cells of higher ranks. Unlike HodgeNet, SNN and SCCONV use both \(\mathcal{L}_{\downarrow,1}\) and \(\mathcal{L}_{\uparrow,1}\). In SNN (Ebli et al., 2020) messages are not passed between adjacent ranks. By contrast, SCCONV uses independent boundary and co-boundary neighborhoods, hence incorporating features from adjacent ranks. Yang et al. (2022c) also makes use of this multi-neighborhood scheme for updating and classifying edge features. They also propose a single neighborhood scheme with an update including the initial cell's feature. Roddenberry
et al. (2021) and Yang et al. (2022d) devise schemes where messages coming from \(\mathcal{L}_{\downarrow,1}\) and \(\mathcal{L}_{\uparrow,1}\) are weighted separately, providing greater learning flexibility. Yang et al. (2022d) allows features to travel multiple hops through the domain by using a polynomial form of the neighborhood structures, leveraging the simplicial convolutional filter fromYang et al. (2022b). Yang and Isufi (2023) extend this multiple-hop model with additional neighborhood structures. Keros et al. (2022) used a modified version of \(\mathcal{L}_{\downarrow}\) to find signals coiled around holes in the complex. BSCNet (Chen et al., 2022) combines node and edge-level shifting to predict links between nodes. This was the first model to pass messages between arbitrary ranks, leveraging a pseudo Hodge Laplacian.

MPSN (Bodnar et al., 2021a) explicitly details their message-passing scheme in the spatial domain, subsuming previous models described from a spectral approach. Hajji et al. (2022b) introduces High Skip Networks (HSNs), in which each layer updates features through multiple sequential convolutional steps, "skipping" it through higher ranks as a generalization of skip-connections in conventional neural networks.

**Attentional / General**

SAN (Giusti et al., 2022a), SAT (Goh et al., 2022), and SGAT (Lee et al., 2022) concurrently introduced attentional message passing networks on the simplicial domain. Each model makes use of a unique set of neighborhood structures and attention coefficients. SGAT is the only model as of yet developed for heterogeneous simplicial complexes of general rank. Hajji et al. (2022c) introduces a variety of general message passing schemes with two neighborhood structures. Bodnar et al. (2021a) uses all four neighborhood structures, endowing each with a separate learnable matrix and general aggregation function.

#### 3.1.3 Cellular Complexes

Just as for simplicial complexes, cellular complex networks have been significantly influenced by work in signal processing (Barbarossa and Sardellitti, 2020; Sardellitti et al., 2021; Roddenberry et al., 2022). These works demonstrated that representing data in the CC domain yields substantially better results than the more rigid SC domain.

**Standard**

Roddenberry et al. (2022) proposes theoretically possible message passing schemes for CCs inspired by works in the SC domain. As of yet, these models have not been implemented.

**Attentional / General**

Hajji et al. (2020) introduces the first TNNs to be theoretically defined on the CC domain. Bodnar et al. (2021b) was the first to implement and evaluate such a model, and demonstrated that TNNs on CCs outperform state-of-the-art graph-based models in expressivity and classification tests. The CAN model from (Giusti et al., 2022b) adapts a modified version of the message passing scheme from Giusti et al. (2022a) onto the CC domain.

#### 3.1.4 Combinatorial Complexes

The combinatorial complex domain was only recently mathematically defined by Hajji et al. (2022a). This work introduces four attentional message passing schemes for CCCs tailored to mesh and graph classification. A more extensive analysis is needed to quantify the advantages of this domain over other topological domains.

### Tasks

Table 3.2 reviews the tasks studied by each paper proposing TNNs. Tasks are first categorized into: _node-level_ tasks assigning labels to nodes, as in node classification, regression or clustering; _edge-level tasks_ assigning labels to edges, as in edge classification or link prediction; and _complex-level_ tasks assigning labels to each complex as a whole, as in hypergraph classification. Tasks are additionally labeled according to their purpose (e.g. classification, regression, prediction). We also indicate the extent of benchmarking performed on each model and code availability.
\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Domain** & **Model** & **Task Level** & **Task Purpose** & **Comparisons** \\ \hline \hline
**HG** & HyperSage Arya et al. (2020) & ✓ & & Classification (Inductive + Transductive) & GNN SOTA \\ \cline{2-5}  & AllSet Chien et al. (2022) & ✓ & & Classification & TNN SOTA \\ \cline{2-5}  & HyperGat Ding et al. (2020) & ✓ & & Classification & GNN SOTA \\ \cline{2-5}  & HNHN Dong et al. (2020) & ✓ & ✓ & Classification, Dimensionality Reduction & GNN SOTA \\ \cline{2-5}  & HMPNN* Heydari and Livi (2022) & ✓ & & Classification & TNN SOTA \\ \cline{2-5}  & UniGNN Huang and Yang (2021) & ✓ & & Classification (Inductive + Transductive) & TNN SOTA \\ \cline{2-5}  & DHGNN Jiang et al. (2019) & ✓ & & Classification (Multimodal) & GNN SOTA \\ \cline{2-5}  & EHNN Kim et al. (2022) & ✓ & & Classification, Keypoint Matching & TNN SOTA \\ \cline{2-5}  & HHNN Li et al. (2022a) & ✓ & & Link prediction & TNN SOTA \\ \cline{2-5}  & HTNN Li et al. (2022b) & ✓ & & Classification & TNN SOTA \\ \cline{2-5}  & SHARE* Wang et al. (2021a) & ✓ & & Prediction & GNN SOTA \\ \cline{2-5}  & DHGCN* Wei et al. (2021) & & ✓ & Classification & GNN SOTA \\ \cline{2-5}  & HGC-RNN* Yi and Park (2020) & ✓ & & Prediction & GNN SOTA \\ \hline
**SC** & MPSN Bodnar et al. (2021a) & & ✓ & ✓ & Classification, Trajectory Classification & GNN SOTA \\ \cline{2-5}  & SCCONV Bunch et al. (2020) & & ✓ & Classification & Graph \\ \cline{2-5}  & BScNet Chen et al. (2022) & ✓ & & Link prediction & GNN SOTA \\ \cline{2-5}  & SNN Ebli et al. (2020) & & ✓ & Imputation & None \\ \cline{2-5}  & SAN Giusti et al. (2022a) & ✓ & & Classification, Trajectory Classification & TNN SOTA \\ \cline{2-5}  & SAT Goh et al. (2022) & & ✓ & ✓ & Classification, Trajectory Classification & TNN SOTA \\ \cline{2-5}  & HSN* Hajji et al. (2022b) & ✓ & ✓ & ✓ & Classification, Link prediction, Vector embedding & Graph \\ \cline{2-5}  & SCA* Hajji et al. (2022c) & & ✓ & Clustering & Graph \\ \cline{2-5}  & Dist2Cycle Keros et al. (2022) & & ✓ & & Homology Localization & GNN SOTA \\ \cline{2-5}  & SGAT Lee et al. (2022) & ✓ & & Classification & GNN SOTA \\ \cline{2-5}  & SCoNe Roddenberry et al. (2021) & ✓ & & Trajectory Classification & TNN SOTA \\ \cline{2-5}  & SCNN* Yang et al. (2022b) & ✓ & & Imputation & TNN SOTA \\ \cline{2-5}  & SCCNN Yang and Isufi (2023) & ✓ & & Link prediction, Trajectory Classification & TNN SOTA \\ \cline{2-5}  & SCN Yang et al. (2022c) & ✓ & & Classification & TNN SOTA \\ \hline
**CC** & CWN Bodnar et al. (2021b) & ✓ & ✓ & Classification, prediction, regression & GNN SOTA \\ \cline{2-5}  & CAN Giusti et al. (2022b) & & ✓ & Classification & GNN SOTA \\ \hline
**CCC** & HOAN* Hajji et al. (2022a) & ✓ & ✓ & Classification & GNN SOTA \\ \hline \end{tabular}
\end{table}
Table 1: **Applications of Topological Neural Networks (TNNs).** We organize papers according to domain and task level, task purpose, and extent of benchmark testing (Graph: compared to graph-based models, GNN SOTA: compared to GNN state-of-the-art, TNN SOTA: compared to state-of-the-art on topological domain). We exclude papers without implementation, and use * to indicate that an implementation has not been shared.
### Symmetries and Geometric Properties

Topological domains possess symmetries and other geometric properties that should be respected to ensure the quality of the features learned by a TNN (Bronstein et al., 2017). Here, we outline such properties harnessed by models in the literature.

Hypergraphs.On hypergraphs, the following symmetries are desirable:

1. _Permutation Invariance:_ Relabeling the nodes and applying the TNN yields an output that is identical to the original output obtained without relabeling. This requires the aggregation functions to be permutation invariant, such as a mean or a sum (Arya et al., 2020; Kim et al., 2022; Chien et al., 2022; Dong et al., 2020; Keros et al., 2022). This is also called _hypergraph isomorphism invariance_.
2. _Global Neighborhood Invariance:_ The network's representation of a node is invariant to hyperedge cardinality: a hyperedge connecting many nodes is weighted the same as a hyperedge connecting less nodes (Arya et al., 2020).

Simplicial Complex.For simplicial complexes, the following symmetries have been considered:

1. _Permutation Invariance:_ Invariance to node relabeling; the same as for HGs. (Schaub et al., 2021; Roddenberry et al., 2021; Bodnar et al., 2021)
2. _Orientation Equivariance:_ Changing the orientation of the simplicial complex (i.e. flipping the signs in the incidence matrix) re-orients the output of that network accordingly (Schaub et al., 2021; Roddenberry et al., 2021; Bodnar et al., 2021).
3. _Simplicial Locality (geometric property):_ In each layer, messages are only passed between \(r\)-cells and \((r\pm 1)\)-cells (Schaub et al., 2021). If that property is not verified, and messages can pass between any \(r\)- and \(r^{\prime}\)-cells, then the network has _extended simplicial locality_.

In addition, _simplicial awareness_ can be imposed, such that message passing on a simplicial complex with maximum cell rank \(r\) depends on every rank \(r^{\prime}\leq r\)(Roddenberry et al., 2021).

Cellular Complex and Combinatorial Complex.Permutation invariance is defined for CCs (Bodnar et al., 2021) and CCCs (Hajij et al., 2022) just as for SCs and HGs. Beyond generalizing global neighborhood invariance to CCC, more research is required to understand the symmetries that can equip this general topological domain.

## 4 Discussion

Our literature review has revealed the diversity of TNN architectures as well as their main axes of comparison. Looking to the future, we highlight four salient opportunities for development.

Within-Domain and Between-Domain Benchmarking.Table 3.2 shows that the domain choice strongly correlates with a TNN's task level. This necessarily makes within-domain comparisons difficult, regardless of code sharing. We also emphasize that many TNNs are only benchmarked against graph-based models or early models in their respective domain, which makes between-domain comparisons equally difficult. As the field grows, improving within and between-domain benchmarking mechanisms will be critical to better informing model selection and quantifying progress.

TNN Architectures on General Domains.The diversity of implementations on HGs and SCs point to a strong potential for similar development in the cellular and combinatorial domains. For instance, only one attentional CC model has been proposed (Giusti et al., 2022). Moreover, any previously developed HG/SC/CC model can be reproduced in the CCC domain and, if desirable, improved with greater flexibility. Evaluating the impact of this added flexibility will directly characterize utility of richer topological structure in deep learning.
Connecting to the Graph Literature.The HG field's ties to the graph community has led to GNN-based advancements not yet propagated to other domains. A first example are dynamic domains, successful with HGs for tasks like pose estimation (Liu et al., 2020), rail transit modeling (Wang et al., 2021), and co-authorship prediction (Jiang et al., 2019). No work in other discrete domains has explored dynamism. In addition, outside of the HG domain, TNNs are largely implemented as homogeneous networks. This leaves room for heterogeneous and non-Euclidean generalizations.

Going Deeper.Over-smoothing occurs when a network is too effective at aggregating signal over multiple layers. This leads to very similar features across cells and poor performance on the downstream learning task. While this issue draws attention in the graph community (Chen et al., 2020; Oono and Suzuki, 2020; Rusch et al., 2023), little of this work has been generalized to TNNs, causing them to remain mostly shallow. UniGCNII (Huang and Yang, 2021) achieves a 64-layer deep TNN by generalizing over-smoothing solutions from GNNs (Chen et al., 2020) to the HG domain. HSNs (Hajji et al., 2022) generalize skip connections to allow signal to propagate further, but are still implemented as shallow networks.

## 5 Conclusion

In this work, we have provided a comprehensive, intuitive and critical view of the advances in TNNs through unifying notations and graphical illustrations. We have characterized each neural network by its choice of data domain and its model, which we further specify through choice of neighboring structure(s) and message-passing scheme. We hope that this review will make this rich body of work more accessible to practitioners whose fields would benefit from topology-sensitive deep learning.
