FMGNN: Fused Manifold Graph Neural Network
Graph representation learning has been widely studied and demonstrated
effectiveness in various graph tasks. Most existing works embed graph data in
the Euclidean space, while recent works extend the embedding models to
hyperbolic or spherical spaces to achieve better performance on graphs with
complex structures, such as hierarchical or ring structures. Fusing the
embedding from different manifolds can further take advantage of the embedding
capabilities over different graph structures. However, existing embedding
fusion methods mostly focus on concatenating or summing up the output
embeddings, without considering interacting and aligning the embeddings of the
same vertices on different manifolds, which can lead to distortion and
impression in the final fusion results. Besides, it is also challenging to fuse
the embeddings of the same vertices from different coordinate systems. In face
of these challenges, we propose the Fused Manifold Graph Neural Network
(FMGNN), a novel GNN architecture that embeds graphs into different Riemannian
manifolds with interaction and alignment among these manifolds during training
and fuses the vertex embeddings through the distances on different manifolds
between vertices and selected landmarks, geometric coresets. Our experiments
demonstrate that FMGNN yields superior performance over strong baselines on the
benchmarks of node classification and link prediction tasks.
Cheng Deng, Fan Xu, Jiaxing Ding, Luoyi Fu, Weinan Zhang, Xinbing Wang
# FMGNN: Fused Manifold Graph Neural Network

###### Abstract.

Graph representation learning has been widely studied and demonstrated effectiveness in various graph tasks. Most existing works embed graph data in Euclidean space, while recent works extend the embedding models to hyperbolic or spherical spaces to achieve better performance on graphs with complex structures, such as hierarchical or ring structures. Fusing the embedding from different manifolds can take advantage of the embedding capabilities over different graph structures. However, existing embedding fusion methods mainly focus on concatenating or summing up the output embeddings without considering interacting and aligning the embeddings of the same vertices on different manifolds, which can lead to distortion and imprecision in the final fusion results. Besides, it is also challenging to fuse the embeddings of the same vertices from different coordinate systems. In the face of these challenges, we propose the **F**used **M**anifold **G**raph **N**eural **N**etwork (**FMGNN**). This novel GNN architecture embeds graphs into different Riemannian manifolds with interaction and alignment among these manifolds during training and fuses the vertex embeddings through the distances on different manifolds between vertices and selected landmarks, geometric coresets. Our experiments demonstrate that FMGNN yields superior performance over strong baselines on the benchmarks of node classification and link prediction tasks.

Graph Representation Learning, Geometric Deep Learning, Manifold Fusion, Geometric Coreset +
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Journal: Information Systems

+
Footnote †: journal: Journal: Journal: Journal: Information Systems

+
Footnote †:
spaces. This kind of interaction and aggregation can be operated by the geometry interaction over manifolds. Nevertheless, without proper interaction and alignment, the embeddings of same vertices obtained from different spaces can be of random shift and distortion due to randomness during model training, even though the mutual relative relations between vertices in each respective space are preserved at best. Thus, concatenating or mixing such embeddings results in imprecision and fluctuation, which we will demonstrate in this work. Moreover, it would be difficult to reasonably fuse the embeddings and design further operations, such as regression and softmax [17], since such operations on the coordinates of different spaces are different.

To this end, in this work, we propose Fused Manifold Graph Neural Network (**FMGNN**), which properly aligns embeddings on different manifolds with interations via tangent mapping before neighborhood aggregation and fuses information on each manifold by distances to landmark geometry coresets. With **FMGNN**, embeddings from different manifolds are appropriately aligned during training and efficiently fused in the final results. The overview flowwork of FMGNN is illustrated in Figure 1. While elaborating the details in Section 5, for ease of understanding, here we would like to briefly unfold the cores components of FMGNN. We first utilize the exponential and logarithmic mappings to bridge the spherical, Euclidean, hyperbolic manifold embeddings and propose the tangent mapping for the interaction among the vertex embeddings on different manifolds. In this way, the embedding of the same vertices are properly aligned and the information of the neighborhood on different manifolds are well communicated. When it comes to fuse the final embedding obtained from different manifolds, we turn vertex embedding into the features measured by the distance to the centroid coreset. Hence, FMGNN still guarantees near vertices can have similar distances, a figuration of coordinates.

Our contributions are summarized as follows:

* We propose a novel graph representation learning framework, fused manifold graph neural network (FMGNN), which, to the best of our knowledge, first studies graph embedding on all the Euclidean, hyperbolic, and spherical manifolds, with novel fusion method based on tangent mapping to align embeddings from different spaces and integrate neighborhood information throughout each embedding layer during training.
* We put forward a novel geometric coreset-based centroid mapping method, which can find the coreset of nodes from randomly sampled nodes in manifolds, and unify the graph embedding of different manifolds into pairwise distance between the vertices and the geometric coresets on the corresponding manifolds.

Figure 1: Overview of the Fused Manifold Graph Neural Network (FMGNN). In FMGNN, a graph will be embedded via GCNs on three Riemannian manifolds, manifold fusion via tangent mapping before neighborhood aggregation, information aggregation on each manifold by distances to landmark geometry coresets, and adjust to the final representation with a self-attention mechanism.
* Extensive experiment results show that FMGNN has achieved SOTA performance on several benchmarks, especially including graphs with sparsity and high hyperbolicity. Our model absorbs the advantages of hyperbolic models on tree structure and maintains the strength of Euclidean space in some free scale topology.

The rest of the paper is organized as follows: In Section 2, we list the related works of FMGNN. In Section 3, the background, metrics, operational rules of Riemannian manifolds are introduced. Particularly, we present the observation on GNN training over different manifolds in Section 4. Then, we share the methodology of FMGNN in Section 5. Finally, the details of experiments on FMGNN will be discussed in Section 6.

## 2. Related Work

In this section, we briefly review the related works about GNNs on both Euclidean and non-Euclidean manifolds, along with the geometric coreset.

**GNN on the Euclidean manifold.** GNNs has received rising attention in the field of machine learning since it was first put forward [12, 26], especially after the GCN [16] was proposed, which combines the information of neighbor nodes with the message passing mechanism. Further, the GAT [31] introduces the attention mechanism into GCN and the SGC [35] convert the nonlinear GCN into a simple linear model. For large-scale graphs, with the high computation cost of unified message passing, GraphSAGE [14] is proposed on node sampling and hierarchical sampling. In complex network analysis, topology on edges have different effects of information dissemination, CurvGN [38] incorporates Ricci curvature to measure the topology difference into the message passing as an important parameter in the attention mechanism of GCN, while M\({}_{2}\)GNN [32] set curvature as a learnable parameter to train knowledge graph embedding.

**GNN on the non-Euclidean manifold.** GNN on the non-Euclidean manifold has attracted more and more attention recently. Texts are embedded into the hyperbolic or spherical manifold for better performance with word2vec [10, 20, 36], and the multiple relations in the knowledge graph are also successfully embedded in the hyperbolic space [7]. The HNN [11] trains neural network in the hyperbolic space, in the tasks of the parametric classification and regression methods. HGCN [8] and HGNN [17] embed graphs into hyperbolic space and apply convolution computation on hyperbolic vector mapping and transition. Recently, Geometric Interaction Learning (GIL) [40] put forward a schema to fuse features at the last layer of the GCN on different manifolds, while \(\kappa\)-GCN [4] investigate the constant curvature GNN finding that different datasets suit different manifolds. In addition, H2H-GCN [9] proposes the direct convolution mechanism without the transformation via the tangent space in a hyperbolic manifold. What's more, a theoretical analysis on the suitable curvature space for a real world graph is conducted [33]. And GIE [6] deploys a similar mechanism as GIL [40] on knowledge graph embedding scenario.

**Geometric Coreset.** In our work, we make use of the geometric coreset [3, 27], which is widely used to approximate the geometry properties with a constant size of points, in the field of computational geometry, statistics and computer vision. In geometry deep learning, geometric coreset is used to draw the outline of the geometry elements efficiently with the constant number of points [30] and analyzing the coresets or sub-level sets of the distance to a compact manifold is a common method in topology data analysis to understand its topology [5]. Moreover, the paradigm of coresets has emerged as an appropriate means for efficiently approximating various extent measures of a point sets [2]. This inspires us to adopt coreset-related means to measure the features of the nodes on graph so that the nodes can be represented with a relatively low-dimension embedding. As mentioned and discussed in HGNN [17], regression and classification methods can not suit well on non-Euclidean manifolds. HGNN uses a set of coordinate-trainable nodes as reference frame to transform the nodes coordinates into nodes and these features represents the node-wise distance with the reference nodes.
FMGNN, proposed in this paper, tries to avoid downgrading and unstable performances by deploy tangent mapping before neighborhood aggregation to align the embedding (coordinates) from Euclidean and non-Euclidean manifolds, and take coreset as a heuristic "landmark" to extract and fuse information on each manifold by distances.

## 3. Preliminaries

In this section, we will recall the metrics and operational rules on Riemannian manifolds (S
mapping over surfaces, at each point \(x\in\mathcal{M}^{d}\), the tangent map \(F^{*}\) is a linear transformation from the flat tangent space \(\mathcal{T}_{x}\mathcal{M}^{d}\) to the tangent space \(\mathcal{T}_{F(x)}\mathcal{N}^{d}\). [(23)]

### Graph Neural Network

Graph neural networks can be viewed as an appropriate means to perform message passing between nodes [(26)]. FMGNN uses a graph convolutional network proposed in [(16)], where node embeddings are updated by aggregating adjacent node information.

Based on general GCN message passing rule, and referring to the constant curvature graph convolution neural network, HGNN and HGCN, via exponential and logarithmic mapping, we can have the general GCN message passing rules at layer \(\ell\) for node \(v_{i}\) on manifold \(\hat{\mathcal{M}}\) (\(\mathbb{B},\mathbb{H}\) and \(\mathbb{S}\)) are shown as follows:

Feature transform function is :

\[\mathrm{h}_{i}^{\ell,\hat{\mathcal{M}}}=W^{\ell}\exp_{x^{\prime}}^{\hat{ \mathcal{M}}}(\mathrm{v}_{i}^{\ell-1,\hat{\mathcal{M}}})+\lambda^{\ell}. \tag{2}\]

Neighborhood aggregation function is :

\[\mathrm{v}_{i}^{\ell,\hat{\mathcal{M}}}=\sigma(\log_{x^{\prime}}^{\hat{ \mathcal{M}}}(\mathrm{h}_{i}^{\ell,\hat{\mathcal{M}}})+\sum_{v_{j}\in\mathcal{ N}(v_{i})}w(v_{i},v_{j})\log_{x^{\prime}}^{\hat{\mathcal{M}}}(\mathrm{h}_{j}^{ \ell,\hat{\mathcal{M}}})), \tag{3}\]

where \(h\) represents the input and output of the middle layer of a network, \(i\) for vertice \(i\), \(l\) for \(l\)-th layer, \(M\) for different manifolds. Besides, \(w(v_{i},v_{j})\) is an aggregation weight that can be computed following the dot product and other operations of the respective Riemannian manifolds, \(\lambda^{\ell}\) is a bias parameters for layer \(\ell\), and \(\sigma\) is denoted as a non-linear activation function.

## 4. Observations of Training GNN on Manifolds

In this section, we present our observations that the inherent randomness in GNN training can result in embedding centroid shifts and task performance fluctuations. When training GNN, randomness is introduced from initialization and stochastic gradient descent. Due to such randomness, the final output embeddings of GNNs can be overall shifted or distorted, though the relative mutual relations are still preserved. Such a natural phenomenon can become a crucial issue when combining graph embeddings from different spaces. An example can be found in Figure 3.

Figure 2. The illustration of exponential (Mapping from tangent spaces to manifolds), logarithmic (Mapping from manifolds to tangent spaces), and tangent mappings (\(F^{*}\)) on non-Euclidean manifolds (e.g. Mapping from hyperbolic space to spherical space, \(F:\mathcal{M}^{d}\rightarrow\mathcal{N}^{d}\)).
We investigate the centroid offsets and performance fluctuations over the final embeddings obtained by repeatedly running the same training process with different randomly generated seeds for GCNs on the three-manifolds, GIL, Hybrid-GCN, and our proposed FMGNN. We design Hybrid-GCN to demonstrate the performance over directly combining the embedding from three different manifolds without interaction, which extends the HGCN framework to directly superimpose the vertex embeddings from the three-manifolds, inspired by the constant curvature graph neural network (Beng et al., 2019). FMGNN is our proposed model, listed for comparison, whose details we will describe in the following sections. We train GCN, HGCN (GCN on Hyperbolic manifold), SGCN (GCN on spherical manifold), GIL, Hybrid-GCN, and FMGNN on the datasets of Cora, PubMed and CiteSeer, for \(10\) times and compare the performance.

### Centroid Offsets

For centroid offsets, we analyze the random coordinate shifts and distortions of embeddings. Given graph \(G(V,E)\), on all the models, the graph embeddings \(X=\{x_{i}|o_{i}\in V\}\) are obtained after each training, and we use the average offsets among vertex embedding centroids \(c=\frac{\sum_{i\in V}x_{i}}{|V|}\) of different training times for the same model to measure the shifts. Thus, the centroid offset is defined as

\[offset=\frac{2}{T(T-1)}\sum_{i\neq j}d(c_{i},c_{j}), \tag{4}\]

where \(c_{i}\) is the centroid on the \(i\)th run, \(T=10\) is the number of training times, and \(d(\cdot)\) is the distance function on respective spaces. Considering that the training results can have different scales, to make a fair comparison and

Figure 3. An example of combining embeddings from different spaces with random shifts. We fix the embedding on Euclidean space, obtain the hyperbolic embedding from two trails with different random seeds, and compare the results when combining the embeddings. Take the docker in orange as an example, from which we can see that nodes 2 and 3 with connection on the original graph has a greater distance than nodes 2 and 7 without direct connection. Such a combination can result in errors in downstream tasks.
reflect topology difference, we measure the scales of the graph embeddings by calculating the average distance between vertices and their centroids:

\[scale=\frac{1}{|V|}\Sigma_{u_{i}\in V}d(x_{i},c). \tag{5}\]

Thereafter, the centroid shift can be normalized by scale as

\[Norm.\ Offset=\frac{offset}{scale}. \tag{6}\]

The results can be found in Table 2. We can see that GCN, HGCN, and SGCN have the same order of magnitude on normalized centroid offsets. If we combine the embedding results of different spaces as Hybrid-GCN or interact between manifolds without proper alignment, the normalized shifts can be accumulated from each space and become even greater. Such offset can result in variance and fluctuation in the performance, as we will show in the following subsection. Our proposed model has the most minor offsets.

### Performance Fluctuation

We proceed to describe our observations on the performance fluctuation of models combining graph embeddings from different manifolds, namely Hybrid-GCN and GIL, to show that mixing such different manifold embeddings without proper alignment, can also result in performance variances, indicated by the large centroid offsets.

For GIL and Hybrid-GCN, we choose to change embedding training only on one manifold, each at a time and remain the embeddings of the other manifolds unchanged. We observe the performance fluctuation introduced by the random shift of manifold embedding training with changed randomly selected seeds. The performance measured by the accuracy of node classification over the three datasets is demonstrated in Figure 4. For results in each dataset, the three rightmost boxplots are the performance variances of GCN, HGCN, and SGCN, which are

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline
**Benchmark** & **Model** & **Scale** & **Offset** & **Norm. Offset** \\ \hline \multirow{6}{*}{**Cora**} & GCN & 0.58 & 83.16 & 144.48 \\  & HGCN & 0.48 & 31.20 & 65.49 \\  & SGCN & 0.51 & 32.21 & 63.44 \\  & Hybrid-GCN & 1.30 & 393.05 & 301.60 \\  & GIL & 1.32 & 215.22 & 162.27 \\  & **FMGNN** & **1.60** & **11.14** & **6.97** \\ \hline \multirow{6}{*}{**PubMed**} & GCN & 0.79 & 112.38 & 141.59 \\  & HGCN & 0.69 & 51.37 & 74.71 \\ \cline{1-1}  & SGCN & 0.66 & 47.93 & 72.99 \\ \cline{1-1}  & Hybrid-GCN & 1.49 & 531.12 & 356.45 \\ \cline{1-1}  & GIL & 0.34 & 161.18 & 475.32 \\ \cline{1-1}  & **FMGNN** & **1.89** & **14.93** & **7.91** \\ \hline \multirow{6}{*}{**CiteSeer**} & GCN & 0.43 & 74.78 & 173.54 \\ \cline{1-1}  & HGCN & 0.25 & 22.80 & 90.56 \\ \cline{1-1}  & SGCN & 0.28 & 25.96 & 92.96 \\ \cline{1-1}  & Hybrid-GCN & 2.58 & 414.93 & 160.82 \\ \cline{1-1}  & GIL & 0.25 & 207.75 & 848.54 \\ \cline{1-1}  & **FMGNN** & **3.20** & **11.75** & **3.67** \\ \hline \hline \end{tabular}
\end{table}
Table 2. Comparison of Models’ Centroid Offsets.
used as benchmarks for the variances of GCN on the three different manifolds, while the results for Hybrid-GCN are shown on the leftmost and those for GIL are in the middle. We can see that for both Hybrid-GCN and GIL, the performance variances are more significant than the variances of GCN on any manifold. This indicates that simplifying combining embeddings from different manifolds without considering the proper alignment, the random shift and distortion of graph embedding on each manifold can be enlarged when combining the different embedding and result in greater performance fluctuation.

Therefore, it is crucial to consider the random shifts of embedding on different manifolds on the embedding fusion and provide interaction and alignment between embeddings during training on all the manifolds if we choose to combine embeddings on different manifolds to approximate the best-fit space for each vertex and enhance the performance. Now we present our proposed work FMGNN, which fusing different manifold embeddings with alignment during training and extract the mutual relation information on different manifolds by distances to the geometric coresets.

## 5. The Fused Manifold Graph Neural Network

Based on the observation above, FMGNN incorporates different manifolds information using a feature fusion mechanism, and converts coordinates on three manifolds into low-dimensional representations through geometric coreset based centroid mapping. We present our method in details as follows.

### Fused Manifold Aggregation

FMGNN fuse the embedding from each manifold by interaction and alignment of the embeddings. First, according to Eq. (2), we execute the feature transform function to obtain the node representation of each manifold, which is denoted as \(\mathrm{h}_{i}^{\ell,\mathbb{E}},\mathrm{h}_{i}^{\ell,\mathbb{H}}\) and \(\mathrm{h}_{i}^{\ell,\mathbb{S}}\). According to the tangent mapping, if there is a smooth mapping between general manifolds, there is a linear mapping between the tangent spaces of the points before and after the mapping. Therefore, the representation vector of a point on one manifold can be mapped to another one via a selected linear mapping on the tangent spaces, and FMGNN takes advantage of this property. The linear mapping in FMGNN is trained from six trainable parameters denoted as \(\lambda_{\mathbb{E}\rightarrow\mathbb{H}},\lambda_{\mathbb{E}\rightarrow \mathbb{S}},\lambda_{\mathbb{H}\rightarrow\mathbb{E}},\lambda_{\mathbb{E} \rightarrow\mathbb{S}}\), \(\lambda_{\mathbb{S}\rightarrow\mathbb{E}}\) and \(\lambda_{\mathbb{S}\rightarrow\mathbb{E}}\), representing the weight of mapping between two manifolds.

Taking hyperbolic manifold as an example, at first, the vectors on Euclidean manifold (\(\mathrm{h}_{i}^{\ell,\mathbb{E}}\)) are linearly mapped (\(\lambda_{\mathbb{E}\rightarrow\mathbb{H}}\)) to the tangent space of hyperbolic manifold, and further projected to the surface of hyperbolic manifold

Figure 4. The performance fluctuations of the models on node classification task, where Hybrid-\(X\) denotes that we change the random seed of embedding training on \(X\) manifold of the Hybrid-GCN, and the same goes for GIL-\(X\).
through exponential mapping, while the vectors on spherical space \(\mathrm{h}_{i}^{\ell,\mathbb{S}}\) requires logarithmic functions to project the vectors on the surface to the tangent space of the sphere, through the linear mapping weight (\(\lambda_{\mathbb{S}\rightarrow\mathbb{H}}\)) to the tangent space of the hyperbolic manifold, and further projected into the surface of hyperbolic manifold via the exponential mapping. Finally, we use plain-vanilla _Mobius_ addition Eq. (1) to adjust the node embeddings and conduct neighborhood aggregation via Riemannian GCN aggregation function Eq. (3). Hence the feature fusion functions in a FMGNN layer is as follows:

\[\mathrm{h}_{i}^{\ell,\mathbb{B}}=\mathrm{h}_{i}^{\ell-1,\mathbb{B}}+\lambda_{ \mathbb{H}\rightarrow\mathbb{B}}\log^{\mathbb{H}}\mathrm{h}_{i}^{\ell-1, \mathbb{H}}+\lambda_{\mathbb{S}\rightarrow\mathbb{B}}\log^{\mathbb{S}}\mathrm{ h}_{i}^{\ell-1,\mathbb{S}}, \tag{7}\]

(8)

(9)

Such operations can be viewed as aligning embeddings of the same vertex into their weighted center on each manifold through interaction.

Then on all the manifolds, we conduct feature transformation and neighborhood aggregation via (2) and (3). In this way, FMGNN executes GCN on three different Manifolds sharing the same weight matrix, fuses the information from other manifolds, and finally gets three groups of node representations, denoted as \(A^{\mathbb{B}}=\{\alpha_{i}^{r},0<i\leq|\mathcal{V}|\}\) for the Euclidean manifold, \(A^{\mathbb{H}}=\{\beta_{i}^{r},0<i\leq|\mathcal{V}|\}\) for the hyperbolic manifold, and \(A^{\mathbb{S}}=\{Y_{i}^{r},0<i\leq|\mathcal{V}|\}\) for the spherical manifolds, among which \(\alpha\), \(\beta\) and \(\gamma\) denotes the final embeddings generated by three-manifolds.

### Geometric Coreset Based Centroid Mapping

After obtaining embeddings from three manifolds, a challenge is that the parameter learning methods for regression and prediction are only applicable in the Euclidean manifold and the arithmetic rules in the three manifolds are different. In this work, we use distance to extract the mutual relation information between vertices on the three manifolds. We select a vertex set, called geometric coreset, as "landmarks" on each manifold, for each vertex calculate distances to all the vertices in the geometric coreset, and use such distances as the new representation vector of the vertex on the manifold. Such new coordinates are the mutual relations extracted from each manifold and can be applied to commonly used algebraic operations, such as addition and multiplication. We first define geometric coreset in FMGNN as follows:

**Definition 5.1**.: **Geometric Coresets in Manifolds**: Given a node set in a manifold \(\mathcal{Z}=\{\zeta_{1},...,\zeta_{n},\zeta_{i}\in\mathcal{M}\}\) and a sensitive quality function \(Q(\mathcal{Z})\), a \(m\)-nodes geometric coreset \(C=\{c_{1},c_{2},...,c_{m}\},c_{i}\in\mathcal{M}\) is a subset of \(\mathcal{Z}\) whose \(Cost(\mathcal{C})\) can be used to approximate \(Q(\mathcal{Z})\) for any set \(\mathcal{Z}\) of \(x\) points.

The process of finding the geometric coreset is as follows.

1. First, we find out a coreset \(C=\{c_{1},c_{2},...,c_{|C|}\},c_{i}\in\mathcal{M}\), noting that the coresets are generated based on parameter-free methods and independent of node embeddings (\(\mathcal{E}=e_{i},i\in|\mathcal{V}|\)).
2. Then, after project the coreset to the manifolds by exponential mapping, the pairwise distance between \(c_{i}\) and \(e_{j}\) can be calculated based on the distance measurement of each manifold (1), \(\alpha_{i,j}=d(c_{i},e_{j})\).
3. Finally, we sum up all the distances \((\alpha_{0,j},...,\alpha_{|C|,j})\in\mathbb{R}^{|C|}\) to represent position of \(e_{j}\) mapping to the coreset, and \(e_{j}\) is the node representation which can be learnt and updated in the GCN processes deployed on different Riemannian manifolds.

The dimensions of vertex representations are only decided by the number of vertices in the geometric coreset, which is constant in theory (Kolmogorov, 1955) and small in empirical experiments. Therefore, the embedding of manifolds are further reduced to lower dimensions by the coreset.
In geometric coreset based centroid mapping, on different manifolds, we take advantage of the KMeans coreset [15]. First, we generate a large number of nodes, \(\mathcal{Z}=\{\zeta_{1},...,\zeta_{n},\zeta_{i}\in\mathcal{E}\}\), represented by vectors with the same dimension with the embedding size in fused manifold aggregation process. Next, referring to [3], we perform KMeans clustering: randomly initialize \(k\) cluster centroid points, and calculate the distance from each point in \(\mathcal{Z}\) to these centroid points, then assign each data point to the cluster with the closest cluster centroid point and recalculate the centroid point for each cluster; we repeat the above steps until the algorithm converges. Finally, we have \(k\) centroid point:

\[\mathcal{P}=\{\rho_{1},...,\rho_{k},\rho_{i}\in\mathcal{M}\}. \tag{10}\]

After we obtain the geometric coreset and its corresponding vertex embeddings on the manifolds, we calculate distance between each vertex in \(\mathcal{P}\) and each vertex in \(\mathcal{G}\) with the embeddings from the output of fused manifold aggregation:

\[E^{\mathcal{M}}=d_{\mathcal{M}}(A^{\mathcal{M}},\mathcal{C}_{\mathcal{M}}), \tag{11}\]

where \(\mathcal{M}\) denotes manifold \(\mathbb{R}\), \(\mathbb{H}\) and \(\mathbb{S}\), and \(E^{\mathcal{M}}\) is the centroid mapping's output from each manifold. Each of \(E^{\mathcal{M}}\) is a \(|C|\)-dimensional vector. The embedding obtained by geometric coreset based centroid mapping is a new feature of a node after training and passing related nodes' messages in the corresponding manifold. In this way, the embedding obtained by graph convolution of each manifold are transformed to new representations with the dimension decided by the geometric coresets, the mutual relations on different manifolds are extracted through the distances between vertices and such geometric coreset landmarks, and such extracted information can be applied to common algebraic operations.

### Attention based Graph Embedding Fusion

Furthermore, we fuse these three embedding vectors via attention mechanism. In this module, FMGNN first to use the representation vector through geometric coreset of each point as the _query_ vector and such embeddings on all three manifolds (_keys_) to calculate the similarity to obtain the weights, then use function \(softmax\) to

Figure 5: The self-attention mechanism in FMGNN. We take hyperbolic manifold as an example, we first _query_ the similarity with embeddings on other manifolds including hyperbolic manifold itself, and then adopt _softmax_ to gain the final weights.
normalize these weights, and finally combine the weights with the corresponding embeddings on the manifold to perform weighted addition so as to generate the find graph embedding. The process is shown in Fig 5 and the equation of the attention is shown as follows.

\[E^{mm}=Attention(Q,K,V)=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V, \tag{12}\]

where \(d_{k}\) is the dimension of the embedding, equal to \(|\mathcal{C}|\), the size of the coreset and \(K\) is equal to \(Q\).

Finally, with the output, FMGNN is trained on node classification and link prediction tasks. For node classification, we map \(E^{mm}=\{e_{i}^{mm},0<i<|\mathcal{V}|\}\) to perform \(softmax\) in order to find the most possible one from \(\tau\) classes. The function is

\[p(e_{j}^{mm})=softmax(\mathrm{w}(e_{1,j}^{mm},...,e_{|\mathcal{C}|,j}^{mm})), 0<j<|\mathcal{V}|, \tag{13}\]

where \(\mathrm{w}\in\mathbb{R}^{r\times|\mathcal{C}|}\). For link prediction, we refer to the Fermi-Dirac method, a generalization of sigmoid, to compute probability for potential edges:

\[p((i,j)\in\mathcal{V}\mid E_{i}^{mm},E_{j}^{mm})=[e^{(d_{k}(E_{i}^{mm},E_{j}^{ mm})^{2}-r)/t}+1]^{-1}, \tag{14}\]

where \(r\) and \(t\) are hyperparameters. We train on both node classification and link prediction tasks by minimizing the cross-entropy loss using negative sampling.

To sum up, FMGNN fuses embedding on different manifolds with interaction and alignment, aggregates information in the same space after the manifold fusion, and converts the embeddings on each manifold through geometric coreset based centroid mapping to exact mutual information.

## 6. Experiments

In this section, we present and analyze our experiment results in detail. First, we describe real-life datasets and the baselines for comparison. After that, we compare these baselines with **FMGNN** on node classification (NC) and link prediction (LP) tasks. Moreover, we discuss the centroid offset of the FMGNN. Finally, we perform ablation experiments to show the necessity of geometric coresets and the contribution of different manifolds.

### Experiments Setup

**Benchmarks.** We use various open-sourced real-life graph datasets described in Table 3.

1. **Citation networks**. CORA, PubMed, and CiteSeer (Huang et al., 2017) are standard node classification and link prediction benchmarks, and the nodes are divided by research (sub)fields.
2. **Disease propagation tree**(Kumar et al., 2017). Disease, taken from HGCN, its labels of nodes is infected or not infected.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **\#N.** & **\#Edge** & **\#Classes** & **\#Features** & **\#Dia.** & **\#Triangle** \\ \hline Disease & 1,044 & 1,043 & 2 & 1,000 & 10 & 0 \\ Airport & 3,188 & 18,631 & 4 & 4 & 12 & 98,669 \\ PubMed & 19,717 & 44,327 & 3 & 500 & 18 & 12,520 \\ CORA & 2,708 & 5,429 & 7 & 1,433 & 19 & 1,630 \\ CiteSeer & 3,327 & 4,732 & 6 & 3,703 & 28 & 1,167 \\ AmazonPhoto & 13,381 & 119,043 & 8 & 745 & 11 & 717,400 \\ Coauthor Phy. & 34,493 & 245,778 & 5 & 8,415 & 17 & 468,550 \\ \hline \hline \end{tabular}
\end{table}
Table 3. Statistics of the selected benchmarks.
3. **Co-author networks**. Coauthor Physics is a co-authorship graph based on the Microsoft Academic Graph from the KDD Cup 2016 challenge. Its node labels are each author's most active fields of study.
4. **Flight network**. Airport, origin from [39], is a transductive dataset updating by HGCN, whose nodes represent airports and edges represent the airline routes. The labels of nodes are the countries to which the airport belongs.
5. **Goods co-consume network**. AmazonPhoto [19, 28] is a segment of the Amazon co-purchase graph, where the product categories give node labels.

**Baselines.** We compare the performance of FMGNN with baselines, including shallow and neural network-based methods, GNNs in the Euclidean space, and GNNs in the hyperbolic space. For neural network-based methods, we choose MLP and HNN [11]. For GNNs in the Euclidean space, GCN [16], GAT [31], GraphSAGE [14] and SGC [35] are taken into comparison. When it comes to GNNs in hyperbolic space, HGCN [8], HGNN [17], and H2H-GCN [9] are the most commonly adopted baselines. Besides, we also take GIL [40] into comparison as a baseline equipped with the embedding interaction only on the final outputs of Euclidean and hyperbolic manifolds.

**Settings.** Referring to the HGCN, in the LP task, we use the random split point 85%/5%/10% as the training sets, validation sets, and test sets. Moreover, in the NC task, we divide the disease with 30%/10%/60% and 70%/15%/15% for the airport, as for CORA, PubMed, and CiteSeer, we follow the division ratio in [16], and for amazon photo and coauthor, we follow the division on the GNN benchmark [28]. Following the HGCN and H2H-GCN, we evaluate link prediction by measuring the area under the ROC curve on the test set and evaluate node classification by measuring the F1 score, apart from CORA, CiteSeer, PubMed, where the performance is measured by accuracy.

### Results

Table 4 summarizes the performance of FMGNN and the baseline methods on seven benchmarks, where the best results are shown in bold and the second best are shown with an underline. We ran five times and reported the average results and standard deviations. From the Table 3 and Table 4 we can see that:

1. [leftmargin=*]
2. FMGNN outperforms the baselines or achieves similar results in classifying different types of nodes and link prediction between nodes.
3. Compared to GIL, FMGNN has better performance on node classification tasks on six benchmarks and has more significant advantages on large-scale datasets such as AmazonPhoto and Coauthor Network on Physics due to the feature fusion and alignment before aggregation on each GCN layer.
4. Compared to H2H-GCN, although our model achieves lower accuracy on the high hyperbolicity benchmark, the overall performance is close. On datasets such as Cora, Pubmed, and CiteSeer, the performance is significantly

\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c c} \hline \hline
**Benchmarks** & **DISEASE** & \multicolumn{2}{c}{**AIRPORT**} & \multicolumn{2}{c}{**PUBMED**} & \multicolumn{2}{c}{**CORA**} & \multicolumn{2}{c}{**CITESER**} & \multicolumn{2}{c}{**AmazonPhoto**} & \multicolumn{2}{c}{**Coauthor-Physics**} \\ \hline
**Hyperbolicity \(\delta\)** & **0** & **1** & **2.5** & **3.0** & **4.5** & **2.5** & **2.5** & **3.0** & **4.5** & **2.5** & **2.5** \\ \hline
**Task** & **LP** & **NC** & **LP** & **NC** & **LP** & **NC** & **LP** & **NC** & **LP** & **NC** & **LP** & **NC** & **LP** & **NC** \\ \hline
**MLP** & 63.6 \(\pm\) 0.6 & 26.8 \(\pm\) 2.5 & 89.8 \(\pm\) 0.5 & 68.6 \(\pm\) 0.6 & 84.1 \(\pm\) 0.9 & 72.4 \(\pm\) 0.2 & 83.1 \(\pm\) 0.5 & 51.5 \(\pm\) 1.0 & 86.3 \(\pm\) 0.0 & 59.7 \(\pm\) 0.0 & 69.1 \(\pm\) 0.7 & 72.3 \(\pm\) 1.8 & 68.8 \(\pm\) 1.2 & 78.1 \(\pm\) 3.1 \\
**HNN** & 75.1 \(\pm\) 0.3 & 41.0 \(\pm\) 1.8 & 98.2 \(\pm\) 0.2 & 90.5 \(\pm\) 0.5 & 94.9 \(\pm\) 0.1 & 69.8 \(\pm\) 0.4 & 80.9 \(\pm\) 0.1 & 54.6 \(\pm\) 0.3 & 87.8 \(\pm\) 0.7 & 72.7 \(\pm\) 0.0 & 79.4 \(\pm\) 8.7 \(\pm\) 0.1 & 81.2 \(\pm\) 0.8 & 87.7 \(\pm\) 2.2 \\
**GCN** & 64.7 \(\pm\) 0.5 & 63.7 \(\pm\) 0.5 & 63.7 \(\pm\) 0.4 & 89.3 \(\pm\) 0.4 & 81.4 \(\pm\) 0.6 & 91.2 \(\pm\) 0.5 & 79.0 \(\pm\) 0.2 & 90.4 \(\pm\) 0.2 & 81.5 \(\pm\) 0.3 & 91.1 \(\pm\) 0.0 & 70.3 \(\pm\) 0.0 & 88.6 \(\pm\) 0.6 & 91.2 \(\pm\) 1.2 & 89.3 \(\pm\) 3.1 & 92.8 \(\pm\) 0.5 \\
**GAT** & 69.8 \(\pm\) 0.3 & 70.4 \(\pm\) 0.4 & 90.5 \(\pm\) 0.3 & 81.5 \(\pm\) 0.2 & 81.2 \(\pm\) 0.7 & 79.0 \(\pm\) 0.3 & 93.7 \(\pm\) 0.1 & 83.0 \(\pm\) 0.7 & 91.2 \(\pm\) 0.1 & 72.5 \(\pm\) 0.0 & 90.0 \(\pm\) 5.2 & 85.7 \(\pm\) 0.3 & 92.0 \(\pm\) 4.1 & 92.5 \(\pm\) 0.5 \\
**SAGE** & 65.9 \(\pm\) 0.3 & 69.1 \(\pm\) 0.6 & 94.0 \(\pm\) 0.5 & 82.1 \(\pm\) 0.5 & 86.2 \(\pm\) 1.7 & 74.2 \(\pm\) 2.2 & 85.5 \(\pm\) 0.6 & 77.9 \(\pm\) 2.4 & 87.9 \(\pm\) 0.3 & 65.1 \(\pm\) 0.1 & 89.9 \(\pm\) 0.9 & 90.4 \(\pm\) 1.3 & 92.1 \(\pm\) 0.1 & 90.3 \(\pm\) 0.8 \\
**SGC** & 61.2 \(\pm\) 0.2 & 69.5 \(\pm\) 0.2 & 89.8 \(\pm\) 0.3 & 80.6 \(\pm\) 0.1 & 94.1 \(\pm\) 0.7 & 78.9 \(\pm\) 0.0 & 91.5 \(\pm\) 0.1 & 81.0 \(\pm\) 0.1 & 91.3 \(\pm\) 0.1 & 71.4 \(\pm\) 0.5 & 88.3 \(\pm\) 0.2 & 83.7 \(\pm\) 0.1 & 91.4 \(\pm\) 0.3 & 92.0 \(\pm\) 0.2 \\ \hline
**HGCN** & 90.8 \(\pm\) 0.3 & 74.5 \(\pm\) 0.9 & 96.4 \(\pm\) 0.1 & **91.6** & 92.6 \(\pm\) 0.0 & 80.3 \(\pm\) 0.3 & 92.9 \(\pm\) 0.1 & 79.9 \(\pm\) 0.2 & 93.9 \(\pm\) 0.0 & 77.6 \(\pm\) 0.0 & **95.6 \(\pm\) 0.2** & 90.0 \(\pm\) 0.5 & 95.7 \(\pm\) 0.6 & 92.5 \(\pm\) 0.2 \\
**HGNN** & 90.6 \(\pm\) 0.2 & 85.7 \(\pm\) 0.8 & 95.8 \(\pm\) 0.1 & 85.1 \(\pm\) 0.3 & 94.1 \(\pm\) 0.2 & 75.9 \(\pm\) 0.1 & 56.1 \(\pm\) 0.0 & 78.3 \(\pm\) 0.2 & 97.4 \(\pm\) 0.2 & 71.3 \(\pm\) 0.1 & - & - & - & - & - \\
**H2H-GCN** & 92.6 \(\pm\) 0.8 & 85.6 \(\pm\) 1.7 & 94.0 \(\pm\) 0.3 & 89.3 \(\pm\) 0.5 & 95.9 \(\pm\) 0.0 & 79.9 \(\pm\) 0.5 & 95.0 \(\pm\) 0.0 & 82.8 \(\pm\) 0.4 & 94.3 \(\pm\) 0.2 & 65.7 \(\pm\) 0.1 & 92.4 \(\pm\) 0.2 & 91.3 \(\pm\) 0.2 & 92.1 \(\pm\) 0.4 \\ \hline
**GIL** & **99.9 \(\pm\) 0.0** & **97.0** & 98.7 \(\pm\) 0.4 & **93.1** & **95.4** & 0.1 & 78.9 \(\pm\) 0.2 & 95.2 \(\pm\) 0.2 & 83.6 \(\pm\) 0.6 & **99.8** \(\pm\) 0.1 & 72.9 \(\pm\) 0.2 & 82.6 \(\pm\) 2.
better than H2H-GCN. In particular, on a graph like CiteSeer has a large diameter and fewer triangle motifs, FMGNN performs better on graphs with sparsity.
4. Our model achieves the best overall performance on all the datasets with different characteristics, compared to other baselines through the radar map in Figure 6 (the line in red belongs to FMGNN).

We introduce the hyperbolicity measurements \(\delta\) shown in Table 4, which is originally proposed by Gromov (Gromov, 2016), to measure how much hierarchical structure a graph is inherent with, where the lower \(\delta\) is, the more the graph is like a tree. Empirical results show that the more the graph is like a tree, the better performance the hyperbolic graph embedding can achieve: hyperbolic methods outperform the Euclidean methods in Disease, Airport. Our FMGNN outperforms HGCN and HGNN, meaning that FMGNN can benefit from embedding from the spherical manifolds and the Euclidean manifolds, even in the tree-like graphs.

In the less tree-like graphs with higher \(\delta\), such as PubMed, Cora, and CiteSeer, Euclidean manifold methods outperform the hyperbolic, and our FMGNN still achieves the best performance. Suppose a network diameter is large, such as Citeseer. In that case, models such as H2H-GCN and HGNN that are highly dependent on the ductility of hyperbolic space or even GIL which performs interact learning with Euclidean manifold have relatively low performance due to the fact that in the process of message passing, the distant nodes may be absorbed by accidental connections (weak correlations). Moreover, we also notice that GIL does not perform well on large-scale graph data with many triangles, AmazonPhoto and Coauthor on Physics, which indicates that GIL does not fit well for graphs with densely connected components. Besides, when processing graphs with triangles motifs, before the step of neighborhood aggregation fusing information from other manifolds is of significance. Last but not least, the feature dimensions of FMGNN are always below \(100\), and compared with HGNN, the number of parameters of FMGNN is significantly reduced, reducing the model's complexity.

### Observation on Stability

Our model is more stable than the hybrid-GCN and GIL tested by the randomness. Compared with GIL, FMGNN interacts and aligns embeddings on different manifolds during training, and the mutual relation information on

Figure 6. The comparison of HGCN (color in Mint Tulip), H2H-GCN (color in Madang), GIL (color in Pigeon Post) and FMGNN (color in Mandy) on various benchmarks. In general, FMGNN performs well all-round.
each manifold is preserved in the distances to the geometric coresets, while GIL only executes the interaction learning after the aggregation. The performance shows that such alignment of FMGNN can reduce the random shifts as shown in Section 4 and performance fluctuation, which demonstrates the efficiency of FMGNN.

### Ablation

In order to better understand the role of each component of FMGNN, we conduct a detailed ablation experiment for FMGNN on CORA, PubMed, CiteSeer, AmazonPhoto, and Coauthor-Physics to observe the effects of geometric coreset-based centroid mapping and the necessity of using the three-manifolds. The result is as follows.

**The geometric coreset size.** The FMGNN embedding dimension is decided by the size of the coresets in the three-manifold. We observe the relation between the size of the coreset and the amount of the initial random nodes, and find that the dimension of the coreset is highly relevant to the size of the initially random nodes in each manifold. We set the size of the initially embedding of nodes on each manifold as 100, and we change the coreset size from 10 to 100, and the result is shown in Figure 8.

We can find that our model performs best with 100 dimensions. At the same time, we notice that as the size of geometric coreset increases from 10 to 100, the accuracy does not rise simultaneously. The experiments show that there is a peak in the performance with respective to the size of coresets. What is more, it shows that the coreset reduces the dimension of the vertex embedding. Meanwhile, it also reflects that the larger the coreset, the more redundant information in the absorbed space.

**Every manifold counts.** Given the introduction of GCN in the spherical manifold, we conduct parallel-space ablation experiments on our model. Experiments are performed on only one of the three spaces and discarding one. The experimental results over classic citation networks are shown in Table 5.

As a result, centroid mapping in Euclidean manifold alone can achieve good results in PubMed, and hyperbolic manifold performs better in CiteSeer while spherical manifold suit CORA, which support the conclusion in (Beng et al., 2019). It also shows that Euclidean, hyperbolic and spherical manifolds partially capture the primary information of the nodes on these three graphs. In addition, any combination of manifolds can increase the experimental result since the information each manifold captures is not the same. Finally, we can conclude that the information on the three spaces needs to be collected simultaneously to get better performance.

Figure 7. Fluctuations when training GIL and FMGNN.
## 7. Conclusion

In this paper, we propose FMGNN, a graph neural network that fuses embeddings from Euclidean, hyperbolic, and spherical manifolds with embedding interaction and alignment during training. Such embedding alignment can benefit from the representation capability of each manifold for its best-fit graph structures, while reducing the centroid offset and performance fluctuation during combining the embedding on different manifolds. We also propose the geometric coreset centroid mapping to fuse the mutual relation preserved in each manifold with low dimensions and high efficiency. As a result, we outperform the baseline models in most of the selected benchmarks, especially on node classification tasks. In future, we would like to further exploit and propose the embedding mapping and interacting among different manifolds with conformal or length preserving insights with theoretical guarantees.

\begin{table}
\begin{tabular}{c|c c c} \hline  & CORA & PubMed & CiteSeer \\ \hline \(\mathbb{R}\) & 81.3 & 79.1 & 72.8 \\ \(\mathbb{H}\) & 81.1 & 78.4 & 73.3 \\ \(\mathbb{S}\) & 81.8 & 78.9 & 72.0 \\ \(\mathbb{R}+\mathbb{H}\) & 81.7 & 79.7 & 74.4 \\ \(\mathbb{R}+\mathbb{S}\) & 82.5 & 79.9 & 73.8 \\ \(\mathbb{S}+\mathbb{H}\) & 82.1 & 79.5 & 74.1 \\ \(\mathbb{R}+\mathbb{H}+\mathbb{S}\) & **83.9** & **80.4** & **78.1** \\ \hline \end{tabular}
\end{table}
Table 5. FMGNN manifolds ablation experiments results.

Figure 8. ACC of NC task distribution along with coreset size, where the X-axis denotes the step of the size of the coreset size, and the Y-axis denotes the ACC of FMGNN.