Generalized Simplicial Attention Neural Networks
The aim of this work is to introduce Generalized Simplicial Attention Neural
Networks (GSANs), i.e., novel neural architectures designed to process data
defined on simplicial complexes using masked self-attentional layers. Hinging
on topological signal processing principles, we devise a series of
self-attention schemes capable of processing data components defined at
different simplicial orders, such as nodes, edges, triangles, and beyond. These
schemes learn how to weight the neighborhoods of the given topological domain
in a task-oriented fashion, leveraging the interplay among simplices of
different orders through the Dirac operator and its Dirac decomposition. We
also theoretically establish that GSANs are permutation equivariant and
simplicial-aware. Finally, we illustrate how our approach compares favorably
with other methods when applied to several (inductive and transductive) tasks
such as trajectory prediction, missing data imputation, graph classification,
and simplex prediction.
Claudio Battiloro, Lucia Testa, Lorenzo Giusti, Stefania Sardellitti, Paolo Di Lorenzo, Sergio Barbarossa
# Generalized Simplicial Attention Neural Networks

###### Abstract

The aim of this work is to introduce Generalized Simplicial Attention Neural Networks (GSANs), i.e., novel neural architectures designed to process data defined on simplicial complexes using masked self-attentional layers. Hinging on topological signal processing principles, we devise a series of self-attention schemes capable of processing data components defined at different simplicial orders, such as nodes, edges, triangles, and beyond. These schemes learn how to weight the neighborhoods of the given topological domain in a task-oriented fashion, leveraging the interplay among simplices of different orders through the Dirac operator and its Dirac decomposition. We also theoretically establish that GSANs are permutation equivariant and simplicial-aware. Finally, we illustrate how our approach compares favorably with other methods when applied to several (inductive and transductive) tasks such as trajectory prediction, missing data imputation, graph classification, and simplex prediction.

 Topological signal processing, attention networks, topological deep learning, neural networks, simplicial complexes.

## I Introduction

Over the past few years, the rapid and expansive evolution of deep learning techniques has significantly enhanced the state-of-the-art in numerous learning tasks. From Feed-Forward [2] to Transformer [3], via Convolutional [4] and Recurrent [5] Neural Networks, increasingly sophisticated architectures have driven substantial advancements from both theoretical and practical standpoints. In today's world, data defined on irregular domains (e.g., graphs) are ubiquitous, with applications spanning social networks, recommender systems, cybersecurity, sensor networks, and natural language processing. Since their introduction [6, 7], Graph Neural Networks (GNNs) have demonstrated remarkable results in learning tasks involving data defined over a graph domain. Here, the versatility of neural networks is combined with prior knowledge about data relationships, expressed in terms of graph topology. The literature on GNNs is extensive, with various approaches explored, primarily grouped into spectral [8, 9] and non-spectral methods [10, 11, 12]. In a nutshell, the idea is to learn from data defined over graphs by computing a principled representation of node features through local aggregation with the information gathered from neighbors, defined by the underlying graph topology. This simple yet powerful concept has led to exceptional performance in many tasks such as node or graph classification [9, 13, 10] and link prediction [14], to name a few. At the same time, the introduction of attention mechanisms has significantly enhanced the performance of deep learning techniques. Initially introduced to handle sequence-based tasks [15, 16], these mechanisms allow for variable-sized inputs and focus on the most relevant parts of them. Attention-based models (including Transformers) have a wide range of applications, from learning sentence representations [17] to machine translation [15], from machine reading [18] to multi-label image classification [19], achieving state-of-the-art results in many of these tasks.

Pioneering works have generalized attention mechanisms to data defined over graphs [13, 20, 21]. However, despite their widespread use, graph-based representations can only account for pairwise interactions. As a result, graphs may not fully capture all the information present in complex interconnected systems, where interactions cannot be reduced to simple pairwise relationships. This is particularly evident in biological networks, where multi-way links among complex substances, such as genes, proteins, or metabolites exist [22]. Recent works on Topological Signal Processing (TSP) [23, 24, 25, 26, 27] have shown the advantages of learning from data defined on higher-order complexes, such as simplicial or cell complexes. These topological structures possess a rich algebraic description and can readily encode multi-way relationships hidden within the data. This has sparked interest in developing (deep) neural network architectures capable of handling data defined on such complexes, leading to the emergence of the field of Topological Deep Learning [28, 29]. In the sequel, we review the main topological neural network architectures, with emphasis to those built to process data on simplicial complexes (SCs).

**Related Works.** Recently, several neural architectures for simplicial data processing have been proposed. In [30], the authors introduced the concept of simplicial convolution, which was then exploited to build a principled simplicial neural network architecture that generalizes GNNs by leveraging on higher-order Laplacians. However, this approach does not enable separate processing for the lower and upper neighborhoods of a simplicial complex. Then, in [31], message passing neural networks (MPNNs) were adapted to simplicial complexes [32], and a Simplicial Weisfeiler-Lehman (SWL) coloring procedure was introduced to differentiate non-isomorphic SCs. The aggregation and updating functions in this model are capable of processing data exploiting lower and upper neighborhoods, and the interaction of simplices of different orders. The architecture in [31] can also be viewed as a generalization of the architectures in [33] and [34], with a specific aggregation function provided by simplicial filters [35]. In [36], recurrent MPNNs architectures were considered for flow interpolation and graph classification tasks. The works in [37, 38] introduced simplicial convolutional neural networks architectures that explicitly enable multi-hop processing based on upper and lower neighborhoods. These architectures also