Torsion Graph Neural Networks
Geometric deep learning (GDL) models have demonstrated a great potential for
the analysis of non-Euclidian data. They are developed to incorporate the
geometric and topological information of non-Euclidian data into the end-to-end
deep learning architectures. Motivated by the recent success of discrete Ricci
curvature in graph neural network (GNNs), we propose TorGNN, an analytic
Torsion enhanced Graph Neural Network model. The essential idea is to
characterize graph local structures with an analytic torsion based weight
formula. Mathematically, analytic torsion is a topological invariant that can
distinguish spaces which are homotopy equivalent but not homeomorphic. In our
TorGNN, for each edge, a corresponding local simplicial complex is identified,
then the analytic torsion (for this local simplicial complex) is calculated,
and further used as a weight (for this edge) in message-passing process. Our
TorGNN model is validated on link prediction tasks from sixteen different types
of networks and node classification tasks from three types of networks. It has
been found that our TorGNN can achieve superior performance on both tasks, and
outperform various state-of-the-art models. This demonstrates that analytic
torsion is a highly efficient topological invariant in the characterization of
graph structures and can significantly boost the performance of GNNs.
Cong Shen, Xiang Liu, Jiawei Luo, Kelin Xia
# Torsion Graph Neural Networks

###### Abstract

Geometric deep learning (GDL) models have demonstrated a great potential for the analysis of non-Euclidean data. They are developed to incorporate the geometric and topological information of non-Euclidian data into the end-to-end deep learning architectures. Motivated by the recent success of discrete Ricci curvature in graph neural network (GNNs), we propose TorGNN, an analytic Torsion enhanced Graph Neural Network model. The essential idea is to characterize graph local structures with an analytic torsion based weight formula. Mathematically, analytic torsion is a topological invariant that can distinguish spaces which are homotopy equivalent but not homeomorphic. In our TorGNN, for each edge, a corresponding local simplicial complex is identified, then the analytic torsion (for this local simplicial complex) is calculated, and further used as a weight (for this edge) in message-passing process. Our TorGNN model is validated on link prediction tasks from sixteen different types of networks and node classification tasks from three types of networks. It has been found that our TorGNN can achieve superior performance on both tasks, and outperform various state-of-the-art models. This demonstrates that analytic torsion is a highly efficient topological invariant in the characterization of graph structures and can significantly boost the performance of GNNs.

 Geometric deep learning, Graph neural networks, Analytic torsion, Message Passing.

## 1 Introduction

With the accumulation of non-Euclidean data, the development of deep learning models, which have revolutionized sequence and image data analysis [1], has given rise to geometric deep learning (GDL). Among all the non-Euclidean data are graphs and networks, which are arguable the most powerful topological representation of real-world data and systems. As a special type of GDL, graph neural networks (GNNs) have demonstrated remarkable learning capability and have become prevalent models for various graph tasks, such as node classification, link prediction, and graph classification [2, 3, 4, 5]. GNNs have achieved great success in applications, such as molecule property prediction [6], recommender systems [7], natural language processing [8], critical data classification [9], computer vision [10], particle physics [11], and resource allocation in computer networks [12]. Recent years have seen a rapid increase of research in the field of GNNs. Great efforts have been devoted to algorithm efficiency improvement (especially for large graphs), special architecture design, and various applications [13].

In general, GNN models can be divided into several types, including recurrent-based GNNs, convolution-based GNNs, graph autoencoders, graph reinforcement learning, and graph adversarial networks [13, 14]. The recurrent-based GNNs refer to the initial GNN models, which employ recurrent units as its combination functions. Typical examples are CommNet [15] and GG-NN [16]. The convolution-based GNNs expand the idea of convolution in the graph space to graph spectral space, based on spectral graph theory. These models are computationally much more affordable, flexible, and scalable. Examples of these models include GCN [2], CurvGN [17], FastGCN [18], Cluster-GCN [19], LGCL [20], ST-GCN [21], AGCN [22] and SGGs [23]. Graph Autoencoder (GAE) is a different type of GNNs, which converts the graph structure into a latent representation (i.e., encoding), that can be later expanded to a graph structure as close as possible to the original one (i.e., decoding). NetRA [24] is a classical GAE with a good performance. Finally, graph reinforcement learning and graph adversarial networks are newly-proposed GNNs that combine reinforcement learning and generative confrontation networks with graph neural architecture. Typical models include MolGAN [25] and MINERVA [26]. In terms of application, recurrent-based GNNs are mainly used in sequence data; Convolution-based GNNs are mainly used in various graphs, networks or knowledge graphs; Graph autoencoders are mainly used in the field of unsupervised models, which are suitable for feature selection or dimensionality reduction; Graph reinforcement learning and graph adversarial networks are applied to generative models, such as molecular generation.

Even with the great development and progress in GNNs, the incorporation of geometric and topological information into graph neural architecture is still a key issue for all GNN and GDL models. Recently, discrete Ricci curvatures have been used in the characterization of "over-squashing" phenomenon [27], which happens at the bottleneck region of a network where the messages in GNN models propagated from distant nodes distort significantly. Curvature-based GNNs have been developed by the incorporation of Ollivier Ricci curvature, a discrete Ricci curvature model, into GNN models and have achieved great success in various synthetic and real-world graphs, from social networks, coauthor networks, citation networks, and Amazon co-purchase graphs [17, 28]. The model can significantly outperform state-of
the-art models when the underlying graphs are of large-sized and dense [17, 28].

Analytic torsion, which is an alternating product of determinants of Hodge Laplacians, is a topological invariant which can distinguish spaces that are homotopy equivalent but not homeomorphic [29, 30]. Analytic torsion is introduced as the analytic version of Reidemeister torsion (or R-torsion for short), which is an algebraic topology invariant that takes values in the multiplicative group of the units of a commutative ring [31]. R-torsion was developed for the classification of three dimensional (3D) lens spaces in 1935 by Reidemeister [32]. It is found that a complete classification of 3D lens spaces can be achieved in terms of R-torsion and fundamental group [29]. To describe the R-torsion in analytic terms, Ray and Singer defined the analytic torsion for any compact oriented manifold and orthogonal representation of the fundamental group [31]. Their definition uses the spectrum of the Hodge Laplacian on twisted forms. When the orthogonal representation is acyclic and orthogonal, Cheeger and Muller proves that the R-torsion is equivalent to analytic torsion [33, 34]. With the unique characterization capability, analytic torsion provides a powerful representation of the topological and geometric information within the data. As demonstrated in Figure 1, both Klein bottle and torus surface can be obtained by "gluing" together the upper edge with lower one and left edge with right one. However, a twist exists in Klein bottle when its upper-side is glued with low-side as indicated by red colors, i.e., regions in upper edge and low edge are glued together when they are of the same color. This can be seen more clearly if the manifolds are discretized into two simplicial complexes with vertices of same numbers gluing together. Interestingly, the topological difference between Klein bottle and torus surface can be well characterized by their analytic torsion. The unique characterization capability of analytic torsion makes it an efficient topological invariant for GNNs, and more general GDLs.

Here we propose a new graph neural network model called analytic Torsion enhanced Graph Neural Network (TorGNN). An analytic torsion based message passing process is developed in our TorGNN. Mathematically, a local simplicial complex is constructed for each edge, and its analytic torsion is used as the weight of the special edge in node feature aggregation. Our TorGNN model shows the best performance and outperforms all state-of-the-art methods, on link prediction tasks from 16 different datasets and node classification tasks from 3 different datasets. This demonstrate that our TorGNN can better capture the complexity of the local structure of graph data.

## 2 Related work

### _Geometry-aware graph neural networks_

Geometry-aware GNNs have been proposed to incorporate algebraic and geometric information of graph data, though the refined message passing and aggregation mechanisms. Among them is TFN, which is a SE(3) equivariant GNN model based on the group set of 3D translation and rotation transformations [35]. LieConv is based on Lie group set of differential transformations beyond 3D translations and rotations [36]. EGNN makes use of all \(n\)-dimension Euclidean transformations including translations, rotations and reflections [37].

In addition, other GNN models take into account the curvature information, such as CurvGN [17], SELFGMNN [38] and CurvGAN [28]. CurvGN is the first graph convolutional network built on advanced graph curvature information. SELFGMNN is the first attempt to study the self-supervised graph representation learning in the mixed-curvature spaces. CurvGAN is the first GAN-based graph representation method in the Riemannian geometric manifold. Curvature has also been combined with hyperbolic graph neural networks, such as, HGCN [39], HAT [40], HGNN [41], ACE-HGNN [42] and HRGCN+ [43].

### _Analytic Torsion_

The Reidemeister torsion, or R-torsion, was proposed by Kurt Reidemeister in 1935 [32]. It is the first topological invariant that could distinguish spaces with the same homotopy type. With fundamental group, it has been used in the complete classification of 3D lens spaces. In 1970s, Ray and Singer introduced the analytic torsion as the analytic version of Reidemeister torsion for any compact oriented manifold and orthogonal representation of the fundamental group [31]. When the orthogonal representation is acyclic and orthogonal, Cheeger and Muller proves that the R-torsion is equivalent to analytic torsion [33, 34]. Computationally, analytic torsion can be calculated from the determinants of Hodge Laplacians [30]. Note that for Hodge Laplacians with zero eigenvalues, the determinant is replaced by the multiplication of all non-zero eigenvalues. With the unique characterization capability, analytic torsion provides a powerful representation of the topological and geometric information within the data. As far as we know, this is the first attempt to combine analytic torsion and GNNs.

## 3 TorGNN

### _Notations and Problem Formulation_

**Notations** Let \(G=(V,E)\) represents a graph with nodes \(v_{i}\in V\) (\(x\) and \(y\) are also used to denote nodes) and

Fig. 1: **The illustration of characterization capability of analytic torsion for Klein bottle and torus surface. Both Klein bottle and torus surface can be obtained by “gluing” together their edges. A twist exists in Klein bottle when its upper-side is “glued” with low-side as indicated by red colors, i.e., regions in upper edge and low edge are glued together when they are of the same color. More specifically, the two manifolds can be discretized into two simplicial complexes with vertices of same numbers gluing together. Using 1D Hodge Laplacians, analytic torsion for Klein bottle and torus are 1.061 and 0.408, respectively.**
edges \((v_{i},v_{j})\in E\). Node features are denoted as \(H=\{h_{1},\cdots,h_{N}\}\in\mathbb{R}^{N\times m}\). A total number of \(N\) nodes in vertex set \(V\) and each node is encoded with a predefined \(m\)-dimension attribute vector (e.g., a generated graph embedding or a one-hot coding). We use \(\mathcal{N}(x)\) to denote the neighbors of node \(x\), \(d(x)\) represent node degree of nodes \(x\).

We use \(K\) to represent a simplicial complex, \(K_{x,y}\) the local simplicial complex from nodes \(x\) and \(y\), and \(T(K_{x,y})\) the analytic torsion for local simplicial complex \(K_{x,y}\). The \(p\)-th Hodge Laplacian is denoted as \(L_{p}\) and its determinant is denoted as \(|L_{p}|\). If the matrix has zero eigenvalues, the determinant \(|L_{p}|\) equals to the multiplication of all non-zero eigenvalues. The \(p\)-th boundary matrix is denoted as \(B_{p}\). Zeta function can be defined based on the eigenvalues of \(L_{p}\) and is denoted as \(\zeta_{p}(s)\).

**Problem formulation** We consider two types of tasks, one is link prediction and the other is node classification. The link prediction task is to learn a mapping function \(\Phi:E\rightarrow[0,1]\) from edges to scores, such that we can obtain the probability of two arbitrary nodes interacting with each other. Similarity, node classification is to learn a mapping function \(\Psi:V\rightarrow\mathbb{R}_{c}^{N}\) from nodes to vectors and \(N_{c}\) is dimension of node's labels.

### _Analytic torsion_

#### 3.2.1 Simplicial complex

As a generalization of graph, a simplicial complex \(K\) is composed of simplices. A simplex with \(p+1\) vertices from a vertex set \(V\) is called a \(p\)-simplex, denoted by \(\sigma^{p}=\{v_{0},v_{1},\cdots,v_{p}\}\). For a \(p\)-simplex \(\sigma^{p}\), any nonempty subset is called its face. A face of \(\sigma^{p}\) with \(p-1\) dimension is called a boundary of \(\sigma^{p}\). Geometrically, a \(p\)-simplex can be seen as the convex hull formed by \(p+1\) affinely independent points. In this way, \(0\)-simplex is a vertex, \(1\)-simplex is an edge, \(2\)-simplex is a triangle, and \(3\)-simplex is a tetrahedron.

#### 3.2.2 Combinatorial Laplacian of simplicial complex

Given an oriented simplicial complex \(K\), that is, a simplicial complex with an order on the vertex set, we use \(Z/2\) coefficient and denote a \(p\)-simplex as \(\sigma^{p}=[v_{0},v_{1},\cdots,v_{p}]\). An \(p\)-chain is defined as a finite sum of \(p\)-simplices in \(K\), denoted as \(c=\sum\limits_{i}\sigma_{i}^{p}\). Let \(C_{p}\) be the set of \(p\)-chains of \(K\), which is spanned by \(p\)-simplices of \(K\) over \(Z/2\), called the \(p\)-chain group of \(K\). The boundary operator \(\partial_{p}:C_{p}\to C_{p-1}\) for a \(p\)-simplex \(\sigma^{p}\) is defined as follows,

\[\partial_{p}\sigma^{p}=\sum\limits_{i=0}^{p}\left(-1\right)^{i}\left[v_{0}, \cdots,\hat{v}_{i},\cdots,v_{p}\right]\]

where \(\hat{v}_{i}\) means that \(v_{i}\) has been removed. The boundary operator satisfies \(\partial_{p-1}\partial_{p}=0\). These chain groups together with boundary operators form a chain complex,

\[0\xleftarrow{\partial_{0}}C_{0}\xleftarrow{\partial_{1}}\cdots\xleftarrow{ \partial_{p}}C_{p}\xleftarrow{\partial_{1}}\cdots\]

Considering the canonical inner product on chain groups, that is, let all the simplices orthogonal. Denote the adjoint of \(\partial_{p}\) by \(\delta^{p}\), then the \(p\)-th Hodge (combinatorial) Laplacian operator of \(K\) is defined as,

\[\Delta_{p}=\delta^{p}\partial_{p}+\partial_{p+1}\delta^{p+1}.\]

Let \(B_{p}\) be the matrix form of \(\partial_{p}\), then, the \(p\)-th Hodge Laplacian matrix of \(K\) is,

\[L_{p}=B_{p}^{T}B_{p}+B_{p+1}B_{p+1}^{T}. \tag{1}\]

For \(1\)-dimension simplicial complex \(K\), i.e., a graph, we have \(L_{0}=B_{1}B_{1}^{T}\), which is the graph Laplacian. Further, for an \(n\)-dimension simplicial complex \(K\), its highest order Hodge Laplacian is \(L_{n}=B_{n}^{T}B_{n}\).

The Hodge Laplacian matrices in Eq.(1) can be explicitly described in terms of simplex relations. More specifically, \(L_{0}\) can be expressed as,

\[L_{0}\left(i,j\right)=\begin{cases}d\left(\sigma_{i}^{0}\right),&\text{if }i=j\\ -1,&\text{if }i\neq j\text{ and }\sigma_{i}^{0}\cap\sigma_{j}^{0}\\ 0,&\text{if }i\neq j\text{ and }\sigma_{i}^{0}\not\cap\sigma_{j}^{0}\end{cases}\]

where \(d\left(\sigma_{i}^{0}\right)\) is the degree of vertex \(\sigma_{i}^{0}\). Furthermore, when \(p>0\), \(L_{p}\) can be expressed as

\[L_{p}\left(i,j\right)=\begin{cases}d\left(\sigma_{i}^{p}\right)+p+1,&\text{if }i=j \\ 1,\text{if }i\neq j,&\sigma_{i}^{p}\not\cap\sigma_{j}^{p},\,\sigma_{i}^{p}\cup \sigma_{j}^{p}\text{ and }\sigma_{i}^{p}\sim\sigma_{j}^{p}\\ -1,\text{if }i\neq j,&\sigma_{i}^{p}\not\cap\sigma_{j}^{p},\,\sigma_{i}^{p}\cup \sigma_{j}^{p}\text{ and }\sigma_{i}^{p}\sim\sigma_{j}^{p}\\ 0,&\text{if }i\neq j,&\sigma_{i}^{p}\cap\sigma_{j}^{p}\text{ or }\sigma_{i}^{p}\not\cap\sigma_{j}^{p}\end{cases}\]

here \(d\left(\sigma_{i}^{p}\right)\) is (upper) degree of \(p\)-simplex \(\sigma_{i}^{p}\). It is the number of \(\left(p+1\right)\)-simplexes, of which \(\sigma_{i}^{p}\) is a face. Notation \(\sigma_{i}^{p}\cap\sigma_{j}^{p}\) represents that two simplexes are upper adjacent, i.e. they are faces of a common \(\left(p+1\right)\)-simplex, and \(\sigma_{i}^{p}\not\cap\sigma_{j}^{p}\) represents the opposite. Notation \(\sigma_{i}^{p}\cup\sigma_{j}^{p}\) represents that two simplexes are lower adjacent, i.e. they share a common \(\left(p-1\right)\)-simplex as their face, and \(\sigma_{i}^{p}\not\cup\sigma_{j}^{p}\) represents the opposite. Notation \(\sigma_{i}^{p}\sim\sigma_{j}^{p}\) represents that two simplexes have the same orientation, i.e. oriented similarly, and \(\sigma_{i}^{p}\approx\sigma_{j}^{p}\) represents the opposite. The eigenvalues of combinatorial Laplacian matrices are independent of the choice of the orientation.

The Laplacian matrix has various important properties. First, it is always positive semi-definite, thus all its eigenvalues are non-negative. Second, the multiplicity of zero eigenvalues, i.e., the total number of zero eigenvalues, of \(L_{p}\) is equal to the \(p\)-th Betti number \(\beta_{p}\). In particular, If \(K\) is a graph, the number (multiplicity) of zero eigenvalues is equal to the topological invariant \(\beta_{0}\), which counts the number of connected components in the graph. Third, the second smallest eigenvalue, i.e., the first non-zero eigenvalue, is called Fiedler value or algebraic connectivity, which describes the general connectivity. The corresponding eigenvector can be used in classification and clustering.

#### 3.2.3 Analytic torsion for simplicial complex

For an \(n\)-dimension simplicial complex \(K\), its \(p\)-th Laplacian matrix is \(L_{p}=B_{p}^{T}B_{p}+B_{p+1}B_{p+1}^{T}\) where \(B_{p}\) is the boundary matrix. Since \(L_{p}\) is a positive semi-definite, all of its eigenvalues, denoted as \(\left\{\lambda_{i}\right\}\), are non-negative. A special Zeta function can be defined based on the eigenvalues of \(L_{p}\) as follows,

\[\zeta_{p}(s)=\sum\limits_{\lambda_{i}>0}\frac{1}{\lambda_{i}^{s}}.\]

Note that the Zeta function is composed of all the positive eigenvalues \(L_{p}\).
The logarithm of analytic torsion \(T(K)\) of \(K\) can be defined as,

\[\log T(K)=\frac{1}{2}\sum_{p=0}^{n}{(-1)^{p}p\zeta_{p}^{\prime}(0)} \tag{2}\]

where \(\zeta_{p}^{\prime}(0)=\zeta_{p}^{\prime}(s=0)\) is the derivative of the Zeta function at \(s=0\). Mathematically, if we let \(|L_{p}|=\Pi_{\lambda_{i}>0}\lambda_{i}\) be the product of all non-zero eigenvalues of \(L_{p}\), we have \(\zeta_{p}^{\prime}(0)=-\log|L_{p}|\). Further, the logarithm of analytic torsion \(T(K)\) can be rewritten as,

\[\log T(K)=\frac{1}{2}\sum_{p=0}^{n}(-1)^{p+1}p\log|L_{p}|.\]

Note that the sum is from \(0\) to \(n\), the highest order of simplicial complex \(K\).

For an 1-dimension simplicial complex \(K\), i.e., a graph, its analytic torsion \(T(K)\) can be expressed as,

\[T(K)=|L_{1}|^{\frac{1}{2}}.\]

Note that here \(L_{1}=B_{1}^{T}B_{1}\) from Eq.(1). Further, for a \(2\)-dimension simplicial complex \(K\), its analytic torsion \(T(K)\) is,

\[T(K)=\frac{|L_{1}|^{\frac{1}{2}}}{|L_{2}|}.\]

The analytic torsion is highly related to the dimension of the simplical complex. Figure 2 illustrates the process to calculate the logarithm of analytic torsion for a 2-dimension simplicial complex.

### _Analytic torsion based graph neural networks_

In our TorGNN model, the analytic torsion is incorporated into graph neural network architecture by adding analytic torsion into message-passing process.

**Analytic torsion based message-passing process:** An essential idea of our TorGNN is to aggregate node features by a analytic torsion-based edge weight as follows,

\[h_{x}^{l}=\sigma\left(\sum_{y\in\mathcal{N}(x)\cup\{x\}}\frac{1}{\sqrt{d(x)} \sqrt{d(y)}}\left|\log T(K_{x,y})\right|W_{\mathrm{GNN}}h_{y}^{l-1}\right),\]

where \(\mathcal{N}(x)\) is the neighbors of node \(x\), \(d(x)\) and \(d(y)\) represent node degree of nodes \(x\) and \(y\) respectively, \(h_{x}^{l}\) and \(h_{y}^{l-1}\) are the node features of \(x\) and \(y\) after \(l\) and \(l-1\) (message-passing) iterations respectively, and \(W_{\mathrm{GNN}}\) is the weight matrix to be learned.

More importantly, \(K_{x,y}\) is the local simplicial complex constructed based on edge \(x,y\). There are various ways to define this local simplicial complex. The most straightforward approach is to build a subgraph using the first-order neighbors. This subgraph contains vertices \(x\), \(y\), and all their neighbors together with all the edges among these vertices. If we allow each triangle in the subgraph to form a 2-simplex, a 2-dimension \(K_{x,y}\) can be obtained. Further, we can consider not only the direct neighbors of vertices \(x\) and \(y\), but also the indirect neighbors that within \(l_{\mathrm{sub}}\) steps. An \(p\)-simplex (\(p\leq n\)) is formed among \(p+1\) vertices if each two of them form an edge. In general, our local simplicial complex \(K_{x,y}\) depends on two parameters, i.e., step \(l_{\mathrm{sub}}\) for neighbors and simplicial complex order \(n\). The corresponding TorGNN model is denoted as \(\mathrm{TorGNN}(l_{\mathrm{sub}},n)\).

Our analytic torsion based message-passing process is illustrated in Figure 3. We consider \(\mathrm{TorGNN}(1,1)\) model, that is the local simplicial complex is constructed using \(l_{\mathrm{sub}}=1\) and \(n=1\). The logarithm of the analytic torsion for the local simplicial complex is used as a weight parameter in the message-passing process.

**TorGNN architectures:** Once we get the representation vector for each node, we can use the node representation for downstream tasks. In this section, we focus on link prediction tasks and node classification tasks.

The link prediction tasks are to calculate whether or not there is an edge between two nodes or the possibility of an edge. After the message-passing process, node representations \(h_{x}\) and \(h_{y}\) are obtained for nodes \(x\) and \(y\). The representation vector of the edge is then used as input for multilayer perceptron (\(MLP\)) to predict the possibility of the existence of the edge as follows,

\[\hat{p}_{(x,y)}=MLP_{link}\left(\parallel(h_{x}+h_{y},h_{x}\odot h_{y},h_{x}, \ h_{y})\right),\]

where \(\parallel()\) is to concatenate the vectors within the bracket into a long vector, and \(\odot\) is element-wise product. It is worth noting that various operations are used in the above equation to model the relationship between two nodes.

The node classification tasks are to predict the label of a node. In these tasks, we use the node representations learned from GNN part, and train them with a \(MLP\) model with

Fig. 2: **An example of calculating the logarithm of analytic torsion for a two dimension simplicial complex.** Based on the simplicial complex \(K\), we can construct boundary matrices, namely \(B_{1}\) and \(B_{2}\). The Hodge Laplacian matrices, including \(L_{0}\), \(L_{1}\) and \(L_{2}\), can be calculated according to Eq. (1). Then, the determinant of Hodge Laplacian matrices \(|L_{p}|\) can be evaluated. Finally, the logarithm of the analytic torsion \(\log T(K)\) can be calculated.
output the predicted label of the node. The process can be expressed as follows,

\[\hat{p}_{x}=MLP_{node}\left(h_{x}\right)\]

Note that in both tasks, we use the cross-entropy as loss function. A detailed illustration of our TorGNN architectures are illustrated in Figure 3. For more detailed parameter introduction of the model, please refer to the source code1.

Footnote 1: A reference implementation of TorGNN may be found at [https://github.com/CS-BIO/TorGNN](https://github.com/CS-BIO/TorGNN)

## 4 Experiments

This section covers four tests, including a link prediction test on 16 networks, a node classification test on 3 networks, a representation learning test and a parameter analysis test. Our TorGNN model shows good performance on all tests.

### _Datasets_

In this test, we utilize six types of datasets with a total of 19 networks to verify the performance of TorGNN. These six types of datasets include: 4 biomedical networks (HuRI-PPI [44], ChG-Miner [45], DisGeNET [46], Drugbank_DTI [47]), 4 social networks (lastfm_asia [48], twitch_EN [49], facebook_comp [50], facebook_TV [50]), 2 collaboration networks (CA-HepTh [51], CA-GrQc [51]), 3 internet peer-to-peer networks (p2p-Gnutella04 [51], p2p-Gnutella05 [51], p2p-Gnutella06 [51]), 3 autonomous systems networks (as20000102 [52], eregon1_010407 [52]), and 3 citation networks (Citeseer [53], Cora [53], Pubmed [53]). Note that biomedical networks, social networks, collaboration networks, internet peer-to-peer networks and autonomous systems networks are mainly used for link prediction tasks. See Table I for details of these dataset. The citation networks are mainly used for node classification tasks. See Table II for details of the dataset.

### _Baselines_

In link prediction tasks, we compare TorGNN against state-of-the-art GNN methods and network embedding methods. We adopt a total of 5 GNN models, including LightGCN [54], SkipGNN [55], KGIN [56], GCN [2] and GAT [5]. The network embedding method is a kind of representation learning methods, and it's main goal is to reduce high-dimensional representation vectors of nodes, edges, or subgraphs into low-dimensional vectors. We select three classic network embedding methods, including DeepWalk [57], LINE [58] and SDNE [59].

In node classification tasks, we select 9 models as the comparison methods, namely ManiReg [60], SemiEmb [61], DeepWalk [57], Planetoid [53], GCN [2], DCNN [62], SAGE [3], N-GCN [9] and N-SAGE [9]. These 9 methods cover various model types that can be applied to node classification

\begin{table}
\begin{tabular}{l l l l l} \hline Categories & Networks & Nodes & Edges & Density \\ \hline \multirow{3}{*}{Biomedical networks} & HuRI-PPI & 5604 & 23322 & 0.15\% \\  & ChG-Miner & 7341 & 15138 & 0.06\% \\  & DisGeNET & 19783 & 81746 & 0.04\% \\  & Drugbank\_DTI & 12566 & 18866 & 0.02\% \\ \hline \multirow{3}{*}{Social networks} & lastfm\_asia & 7624 & 27806 & 0.10\% \\  & twitch\_EN & 7126 & 35324 & 0.14\% \\  & facebook\_comp & 14113 & 52310 & 0.05\% \\  & facebook\_TV & 3892 & 17262 & 0.23\% \\ \hline Collaboration & CA-HepTh & 9877 & 25998 & 0.05\% \\ networks & CA-GrQc & 5242 & 14496 & 0.11\% \\ \hline Internet & p2p-Gnutella04 & 10876 & 39994 & 0.07\% \\ peer-to-peer & p2p-Gnutella05 & 8846 & 31839 & 0.08\% \\ networks & p2p-Gnutella06 & 8717 & 31525 & 0.08\% \\ \hline Autonomous & as2000102 & 6474 & 13895 & 0.07\% \\ systems & oregon1\_010331 & 10670 & 22002 & 0.04\% \\ networks & oregon1\_010407 & 10729 & 21999 & 0.04\% \\ \hline \end{tabular}
\end{table} TABLE I: Details of 16 network data on link prediction tasks.

\begin{table}
\begin{tabular}{l l l l l} \hline Networks & Nodes & Edges & Density & Classes & Feature \\ \hline Citeseer & 3327 & 4732 & 0.09\% & 6 & 3703 \\ Cora & 2708 & 5429 & 0.15\% & 7 & 1433 \\ Pubmed & 19717 & 44338 & 0.02\% & 3 & 500 \\ \hline \end{tabular}
\end{table} TABLE II: Details of 3 network data on node classification tasks.

Fig. 3: **The overview of TorGNN architecture.****A.** A large network. **B.** Local simplicial complex construction. **C.** Analytic torsion-based message-passing process. **D.** TorGNN architecture for link prediction task and node classification.
tasks, including graph neural network, network embedding model, traditional machine model, etc. In this way, the performance of our TorGNN model can be evaluated more comprehensively.

### _Performance on link prediction tasks_

Five types of datasets, including a total of 16 network data, are used for the evaluation of the performance of our TorGNN. For each network, we randomly sample the same number of negative and positive samples, and then divide them into training set, validation set and test set with a ratio of 7:1:2. The area under the receiver operating characteristic curve (AUC) and the area under the precision-recall curve (AUPR) are used for evaluating the performance of each model. This process is repeated 10 times, and the average value is taken as the final result. We use batch size 128 with Adam optimizer and run TorGNN model in PyTorch. For the learning rate, after adjustment, it is found that the learning rate of 5e-4 is the most suitable for all 16 networks.

Table III presents the prediction results on the link prediction tasks. Overall, our TorGNN model outperforms other models on most datasets, and the average AUC and AUPR on 16 networks are 2.49% and 3.45% higher than the second-ranked model, respectively. It is worth noting that the performance of the TorGNN model on all data sets is better than that of the 3 network embedding models (DeepWalk, LINE and SDNE), which shows that the TorGNN model is not only a good graph neural network model, but also an excellent network embedding models. Although the AUC value of the TorGNN model on the "as20000102" network and the AUPR value on the "facebook_comp" network did not reach the highest value, the AUPR value of the TorGNN model on the "as20000102" network and the AUC value on the "facebook_comp" network are still the best. Note that our tasks are from various types of networks, including biomedical networks, collaboration networks and internet peer-to-peer networks, the great performance of our TorGNN shows that it has a strong generalization capability for network/graph data.

### _Performance on node classification tasks._

Three networks, including Citeseer, Cora and Pubmed, are used in the evaluation of the performance of TorGNN model. Following the general way to set up training, verification and test sets of these three datasets [63], we select 500 nodes in each data set as the verification set, 1000 nodes as the test set, and the remaining nodes as the training set. The accuracy is used to evaluate the performance of TorGNN. Each step is run 10 times, and the average is taken as the final result. We use Adam optimizer of learning rate 0.02 and run TorGNN model in PyTorch.

Figure 4 shows the results of the TorGNN model and the comparison methods for node classification tasks on three datasets. Our TorGNN model has achieved the best results, and it is significantly better than the second-ranked model N-GCN. It is worth noting that N-GCN and N-SAGE are improved models based on GCN and SAGE. Specifically, they use a data perturbation method to prevent overfitting of the traditional model. As can be seen in Figure 4, the performance of N-GCN and N-SAGE models is superior to the traditional GCN and SAGE models. Nevertheless, these models are still slightly inferior to our TorGNN model, which shows that the advantages of our TorGNN model over the traditional GNNs are not only reflected in the overall performance, but also in preventing overfitting. The reason may be that the analytic torsion of the local structure plays a restrictive role in the adjustment of the parameters in GNNs and reduces overfitting.

### _Performance on representation learning_

To further illustrate the representation learning ability of our TorGNN, we set up a visualization test. First, we get the representation vectors for node pairs in the test sets. Then, we use t-SNE [64] to project the high-dimensional representation vectors to 2D space, where the representation vectors are mainly divided into two categories, positive samples (node pairs with link relationship) and negative samples (node pairs without link relationship). Finally, these representation vectors in 2D space are illustrated in Figure 5 (red and blue points represent negative and positive samples, respectively). Three datasets of HuRI-PPI, ChG-Miner and Drugbank_DTI are used in our test. At the same time, we choose DeepWalk, LINE and SDNE three network embedding methods for comparison. It can be seen from Figure 5 that our TorGNN model has obvious advantages in distinguishing node pairs

Fig. 4: **The comparison of different models on node classification performance based on three citation datasets.** **). **A**, **B**, and **C** illustrate the comparison of accuracy for various models on Citeseer, Cora and Pubmed datasets, respectively.
with and without link relationship, which shows that our TorGNN model has a better ability in representation learning than traditional network embedding model.

### _Influence of local simplicial complex_

In order to analyze the impact of the number of neighbor layers and the highest order of the local simplicial complex on \(\mathrm{TorGNN}(l_{\mathrm{sub}},n)\), we set up the following parameter analysis model:

* TorGNN(1,1): \(l_{sub}\)=1, \(n\)=1. It means that TorGNN only considers first-order neighbors when constructed local simplicial complex, the highest order of the simplicial complex is only up to 1, that is, only edges are considered.
* TorGNN(1,2): \(l_{sub}\)=1, \(n\)=2. It means that TorGNN only considers first-order neighbors when constructed local simplicial complex, the highest order of the simplicial complex is up to 2, that is, triangles (2-simplicplices) are considered.
* TorGNN(2,1): \(l_{sub}\)=2, \(n\)=1. It means that TorGNN considers second-order neighbors when constructed local simplicial complex, the highest order of the simplicial complex is only up to 1, that is, only edges are considered.

Table IV and Table V show the results of these three models on link prediction tasks and node classification tasks. It can be seen that the performance of the three models are relatively stable. The differences between the three models for both AUC and AUPR are less than 0.02 for all the test cases, except "drugbank_DTI", "facebook_comp" and "p2p-Gnutella06". Comparably speaking, TorGNN(2,1) model has the best overall performance. This may due to the reason that the corresponding local simplicial complex of TorGNN(2,1) has a better representability of the local structures. Note that with \(l_{sub}=2\), the local simplicial complex contains all second-order neighbors and is a much larger structure. Computationally, more efficient algorithms will be needed

Fig. 5: **The comparison of the representation learning capability of different models on HuH-PP, ChG-Miner and Drugbank_DTI network datasets. The representation vectors of each nodes in the test datasets are projected into 2D spaces by t-SNE. The red and blue points represent node pairs without and with link relationships, respectively. Four network embedding methods are considered in our comparison.**

\begin{table}
\begin{tabular}{l l l l l l l l l l l} \hline \hline  & Networks & TorGNN & LightGCN & SkipGNN & KGIN & GCN & GAT & DeepWalk & LINE & SDNE \\ \hline AUC & HuRI-TPPI & **0.9369** & 0.8170 & 0.9119 & 0.9007 & 0.9164 & 0.8994 & 0.7294 & 0.8223 & 0.9243 \\  & ChG-Miner & **0.9583** & 0.7457 & 0.9526 & 0.9493 & 0.9352 & 0.9514 & 0.8093 & 0.7036 & 0.6108 \\  & DisGeNET & **0.9868** & 0.8597 & 0.9145 & 0.9154 & 0.9723 & 0.9829 & 0.8492 & 0.9586 \\  & Drugbank\_DTI & **0.9651** & 0.6821 & 0.8946 & 0.9453 & 0.9234 & 0.9476 & 0.8572 & 0.6312 & 0.8522 \\  & lastfm\_asia & **0.9204** & 0.8787 & 0.8117 & 0.9125 & 0.8693 & 0.8619 & 0.7312 & 0.8049 & 0.8952 \\  & twitch\_EN & **0.9159** & 0.8071 & 0.8640 & 0.8807 & 0.8965 & 0.8863 & 0.6823 & 0.7836 & 0.8627 \\  & facebook\_comp & **0.8974** & 0.8793 & 0.7823 & 0.9146 & 0.8614 & 0.8598 & 0.7525 & 0.6756 & 0.7733 \\  & facebook\_TV & **0.9500** & 0.9164 & 0.7941 & 0.8951 & 0.9146 & 0.9165 & 0.8637 & 0.6841 & 0.8145 \\  & CA-HepTh & **0.9476** & 0.6004 & 0.7859 & 0.8117 & 0.9206 & 0.9185 & 0.7454 & 0.6285 & 0.8514 \\  & CA-GrQc & **0.9599** & 0.8304 & 0.8062 & 0.8084 & 0.9430 & 0.9420 & 0.8224 & 0.7307 & 0.8785 \\  & p2p-Gnutella0 & **0.9060** & 0.6232 & 0.8034 & 0.8898 & 0.8901 & 0.8816 & 0.6175 & 0.5980 & 0.7883 \\  & p2p-Gnutella05 & **0.8956** & 0.6037 & 0.8074 & 0.8871 & 0.8847 & 0.8785 & 0.6353 & 0.5945 & 0.8051 \\  & p2p-Gnutella06 & **0.9051** & 0.6240 & 0.8112 & 0.8864 & 0.8872 & 0.8828 & 0.6490 & 0.5687 & 0.8003 \\  & as20000102 & 0.9292 & 0.7834 & 0.9125 & **0.9334** & 0.9120 & 0.8926 & 0.8276 & 0.8340 & 0.8902 \\  & oregon1\_010331 & **0.9602** & 0.7913 & 0.9544 & 0.9450 & 0.9516 & 0.9338 & 0.8165 & 0.8812 & 0.9219 \\  & oregon1\_010407 & **0.9603** & 0.8060 & 0.9492 & 0.9416 & 0.9518 & 0.9594 & 0.8142 & 0.8657 & 0.9172 \\ \hline Average & **0.9372** & 0.7655 & 0.8597 & 0.9011 & 0.9144 & 0.9107 & 0.7529 & 0.7285 & 0.8465 \\ \hline AUPR & HuRI-TPPI & **0.9424** & 0.8589 & 0.9182 & 0.9006 & 0.9189 & 0.8965 & 0.6958 & 0.8520 & 0.9324 \\  & ChG-Miner & **0.9606** & 0.8006 & 0.9524 & 0.9493 & 0.9409 & 0.9499 & 0.8267 & 0.7514 & 0.6114 \\  & DisGeNET & **0.9882** & 0.8915 & 0.9271 & 0.9154 & 0.9785 & 0.9849 & 0.8646 & 0.8477 & 0.9554 \\  & Drugbank\_DTI & **0.9664** & 0.7593 & 0.6764 & 0.9452 & 0.9371 & 0.9533 & 0.8729 & 0.7044 & 0.8672 \\  & lastfm\_asia & **0.9311** & 0.9028 & 0.8269 & 0.9124 & 0.8810 & 0.8810 & 0.7365 & 0.8395 & 0.9127 \\  & twitch\_EN & **0.9265** & 0.8573 & 0.8750 & 0.8807 & 0.9051 & 0.8899 & 0.6729 & 0.8134 & 0.8642 \\  & facebook\_comp & 0.9112 & 0.9045 & 0.7993 & **0.9146** & 0.8761 & 0.871 & 0.7517 & 0.6956 & 0.8092 \\  & facebook\_TV & **0.9569** & 0.9359 & 0.8116 & 0.8951 & 0.9256 & 0.9265 & 0.8672 & 0.7461 & 0.8517 \\  & CA-HepTh & **0.9487** & 0.5991 & 0.7889 & 0.7794 & 0.9169 & 0.9077 & 0.7313 & 0.6778 & 0.8614 \\  & CA-GrQc & **0.9626** & 0.8275 & 0.8228 & 0.7769 & 0.9427 & 0.9419 & 0.8469 & 0.7863 & 0.8845 \\  & p2p-Gnutella04 & **0.9027** & 0.6884 & 0.7660 & 0.8898 & 0.8901 & 0.8843 & 0.6166 & 0.5967 & 0.7735 \\  & p2p-Gnutella05 & **0.8892** & 0.6764 & 0.7748 & 0.8870 & 0.8804 & 0.8751 & 0.6514 & 0.6177 & 0.7838 \\  & p2p-Gnutella06 & **0.8996** & 0.6918 & 0.7763 & 0.8862 & 0.8830 & 0.8814 & 0.6457 & 0.5967 & 0.7718 \\  & as20000102 & **0.9383** & 0.8403 & 0.9333 & 0.9334 & 0.9290 & 0.9090 & 0.8157 & 0.8742 & 0.9159 \\  & oregon1\_010331 &
for the fast evaluation of the analytical torsion if large-sized simplicial complex is used.

## 5 Conclusion

The incorporation of geometric and topological information into graph neural architecture remains to be a key issue for geometric deep learning models, in particular graph neural network models.

In this paper, we propose an analytic torsion enhanced graph neural network (TorGNN) model, which mainly uses analytic torsion to capture the complexity of the structures, and then incorporate it into message-passing process to improve the performance of the traditional graph neural network models. From the tasks of link prediction and node classification, our TorGNN is found to be not only better than traditional GNNs models, but also better than other graph deep learning models.

## Acknowledgements

This work was supported in part by National Natural Science Foundation of China (NSFC grant no. 61873089, 62032007), Nanyang Technological University Startup Grant (grant no. M4081842), Singapore Ministry of Education Academic Research fund (grant no. Tier 1 RG109/19, MOE-T2EP20120-0013, MOE-T2EP2020-0010) and China Scholarship Council (CSC grant no.202006130147).
