# Learning from Heterogeneity: A Dynamic Learning Framework for Hypergraphs

Tiehua Zhang∗, Yuze Liu∗, Zhishu Shen, Xingjun Ma, Xin Chen, Xiaowei Huang,
Jun Yin, and Jiong Jin Tiehua Zhang, Yuze Liu, Xin Chen, Xiaowei Huang and Jun
Yin are with Ant Group, Shanghai, China (e-mail:{zhangtiehua.zth,
liuyuze.liuyuze, jinming.cx, wei.huangxw, jun.yinj}@antgroup.com).Zhishu Shen
is with the School of Computer Science and Artificial Intelligence, Wuhan
University of Technology, Wuhan, China (e-mail: z_shen@ieee.org).Xingjun Ma is
with the School of Computer Science, Fudan University, Shanghai, China
(e-mail: xingjunma@fudan.edu.cn).Jiong Jin is with the School of Science,
Computing and Engineering Technologies, Swinburne University of Technology,
Melbourne, Australia (e-mail: jiongjin@swin.edu.au).∗ co-first authorship.
(Corresponding author: Tiehua Zhang)

###### Abstract

Graph neural network (GNN) has gained increasing popularity in recent years
owing to its capability and flexibility in modeling complex graph structure
data. Among all graph learning methods, hypergraph learning is a technique for
exploring the implicit higher-order correlations when training the embedding
space of the graph. In this paper, we propose a hypergraph learning framework
named _Learning from Heterogeneity (LFH)_ that is capable of dynamic hyperedge
construction and attentive embedding update utilizing the heterogeneity
attributes of the graph. Specifically, in our framework, the high-quality
features are first generated by the pairwise fusion strategy that utilizes
explicit graph structure information when generating initial node embedding.
Afterwards, a hypergraph is constructed through the dynamic grouping of
implicit hyperedges, followed by the type-specific hypergraph learning
process. To evaluate the effectiveness of our proposed framework, we conduct
comprehensive experiments on several popular datasets with eleven state-of-
the-art models on both node classification and link prediction tasks, which
fall into categories of homogeneous pairwise graph learning, heterogeneous
pairwise graph learning, and hypergraph learning. The experiment results
demonstrate a significant performance gain (average 12.5% in node
classification and 13.3% in link prediction) compared with recent state-of-
the-art methods.

###### Index Terms:

Heterogeneous Hypergraph, Representation Learning, Hypergraph Generation,
Classification.

##  I Introduction

Graph learning has attracted tremendous attention in recent years owing to its
prominent capability when modelling structure-based data. In particular, there
has been witnessed an increasing use of graph models (i.e., GNN: Graph Neural
Network) in many fields of applications, such as social network recommendation
[1], medical diagnosis [2, 3], and text analysis [4]. The goal of graph
learning is to encode the graph structure of different input data into an
embedding space, where the representations can be used for downstream
node/edge/graph tasks.

There are both standard graphs and hypergraphs. Fig. 1 illustrates (a) a
standard graph where one edge connects two vertices regardless of the edge
type and (b) a hypergraph with three hyperedges. Standard graphs can be
modeled using many existing GNN models such as graph convolution network (GCN)
[5], graph attention network (GAT) [6], and heterogeneous graph learning (PC-
HGN) [7]. Most of these models aim to model the pairwise relationships in
well-structured graph data. However, this may not work for hypergraphs, as a
hypergraph can encompass any number of vertices (nodes) at each edge type,
presenting higher-order correlations with rich underlying information.
Pairwise graph learning models, in this case, become intractable.
Specifically, hypergraph presents more complex non-pairwise relationships,
where these implicit relationships are not only dyadic (pairwise) but rather
triadic or even higher [8]. In general, hypergraph learning can be regarded as
a generalized form of traditional graph learning.

Like pairwise graph learning models like GCN, hypergraph learning can also be
seen as a process of uncovering and passing the structure information among
different hyperedges, during which low-dimensional node/graph level embedding
representations can be learned and used for downstream tasks, such as node
classification [9], link prediction [10], and clustering [11].

(a)

(b)

Figure 1: Comparison between (a) a standard graph with 5 nodes and 4 edges,
and (b) a hypergraph with 3 hyperedges (the three colored node groups).

Owing to the great potential of exploring high-order correlations among the
data, hypergraph learning has drawn increasing attention from the academic
community and has been employed in applications such as image classification
[12], video segmentation [13], and hyperspectral image analysis [14]. How to
construct hyperedges lies at the core of hypergraph learning. Existing works
generally take two approaches, i.e., attribute-based construction and
neighborhood-based construction. The early work focused on constructing the
hypergraph based on the features of each vertex [15]. This strategy instructs
the model to use only the feature information which inevitably sacrifices
certain generality from the topological perspective. The neighborhood-based
strategy addresses this weakness by enabling the locality concept.
Specifically, a hyperedge is constructed through clustering a master vertex
and its neighbor vertices [16]. However, the clustering method (e.g.,
k-nearest neighbors (k-NN)) separates hypergraph construction from the graph
learning process, and thus is sensitive to noisy data, especially in visual
classification tasks. Moreover, most neighborhood-based models fail to
incorporate topological heterogeneity into the learning process, which limits
the learning to a single type of hyperedges.

To tackle these challenges, we propose a dynamic learning framework, _Learning
from Heterogeneity (LFH)_ , to improve the quality of representation learning
for hypergraphs. Our LFH framework is composed of three key modules: 1)
initial embedding generation, 2) dynamic hypergraph construction, and 3)
attention-based heterogeneous hypergraph learning. Concretely, initial
embedding generation aims to exploit the pairwise connectivities in the graph,
helping to fuse the explicit topological information into the initial
embedding space. Following that, as an integral part of representation
learning, the heterogeneous hypergraph is constructed and then learned
dynamically.

In summary, the main contributions of this work are:

  * •

We propose a novel hypergraph learning framework LFH with three modules:
initial embedding generation, dynamic hypergraph construction, and attention-
based heterogeneous hypergraph learning. A pairwise fusion strategy is also
considered to fully exploit the explicit pairwise graph information to
generate high-quality initial embedding.

  * •

We design a dynamic learning process to generate different types of implicit
hyperedges to construct the hypergraph and dynamically adapt the hypergraph
construction during the learning process. The embedding is updated iteratively
through a type-specific multi-head attention mechanism, so as to encode the
heterogeneous attributes into the embedding space.

  * •

We conduct extensive experiments on two different downstream tasks, including
node classification and linked prediction, and show that our LFH outperforms
state-of-the-art homogeneous pairwise graph learning models, heterogeneous
pairwise graph learning models, and hypergraph learning models consistently
across all datasets by a large margin.

The remainder of this paper is organized as follows. Section 2 reviews the
related works of graph learning on homogenous, heterogenous, and hypergraphs.
Section 3 provides the preliminary knowledge of hypergraphs, while Section 4
introduces the detail of our proposed LFH framework, along with analyses and
discussions. The experimental results are reported and analyzed in Section 5.
Finally, Section 6 concludes the paper.

##  II Related Work

###  II-A Homogeneous Graph Learning

As a basic graph structure, a homogeneous graph consists of a single node type
with a single relation. Typical GNN models for homogeneous graphs include GCN
[5] and GAT [6]. GCN extends the traditional convolutional neural networks
(CNN) to handle graph-structured data by performing convolutions in the
spectral/spatial domain of the graph nodes to capture the structural
information of the graph. On the other hand, GAT applies the self-attention
mechanism to assign dynamic weights for the neighbors of a node and then take
a weighted sum of their embeddings to obtain the node’s representation. This
enables GAT to learn adaptive neighborhood representations and capture complex
relationships between nodes.

The primary difference between various GNN models lies in the way how messages
are passed between the nodes to learn the representation. Instead of training
individual embeddings for each node, GraphSAGE [17] leverages the node
features in the learning algorithm to train a set of aggregator functions to
generate embeddings for entirely unseen nodes. Aiming for visual question
answering (VQA) services, Li et al. proposed a GAT-based platform that encodes
each image into a graph and models multi-type inter-object relations to learn
relation representation from the graphs [18]. Jiang et al. presented a GCN-
based framework (GLCN) for graph data representation learning. GLCN generates
similarity-based graph structure by simultaneous graph learning and graph
convolution in a unified network architecture [19]. Compared with the
embedding frameworks that can only generate embeddings for a single fixed
graph like transductive learning in GCN, Zeng et al. proposed GraphSAINT, a
graph sampling based inductive representation learning method, to generalize
across different graphs. GraphSAINT generates low-dimensional vector
representations for the nodes and is beneficial for large graphs with rich
node attribute information [20].

###  II-B Heterogeneous Graph Learning

Different from homogeneous graphs, the nodes in a heterogeneous graph are
usually connected with various types of neighbors via different types of
relations. As such, representation learning on heterogeneous graphs is much
more challenging. It needs to not only incorporate heterogeneous structure
(graph) information but also consider the heterogeneous attributes associated
with each node [21].

Zhang et al. proposed a heterogeneous GNN model (HetGNN) for capturing both
structure and content heterogeneity. HetGNN first captures the strongly
correlated heterogeneous neighbors of each node and then aggregates feature
information of these sampled neighbors [22]. Zhao et al. proposed a GNN-based
framework named HGSL that jointly performs heterogeneous graph structure
learning and GNN parameter learning for classification. In HGSL, the feature
similarity graphs, the feature propagation graphs, and the semantic graphs are
generated separately so as to comprehensively learn an optimal heterogeneous
graph [23]. Zhang et al. designed a relation-centered pooling and convolution
(PC-HGN) operation that enables relation-specific sampling and cross-relation
convolutions on heterogeneous graphs, from which the structural heterogeneity
of the graph can be better encoded into the embedding space through the
adaptive training process [7]. HAN (Heterogeneous graph Attention Network)
[24] is a heterogeneous graph learning framework based on node-level and
semantic-level attention mechanisms. Specifically, node-level attention learns
the importance between a node and its meta-path based neighbors, while
semantic-level attention learns the importance of different meta-paths. Hu et
al. presented a heterogeneous graph transformer (HGT) for modeling
heterogeneous graphs [25]. HGT introduces the node-type and edge-type
dependent attention mechanism while different trainable parameters are
assigned to each node and edge type. HGT can incorporate high-order
heterogeneous neighbor information, which automatically learns the importance
of implicit meta path. Considering relation-aware characteristics, Yu et al.
proposed a relation-aware representation learning model for heterogeneous
graphs (R-HGNN). This model derives a fine-grained representation from a group
of relation-specific node representations reflecting the characteristics of
the node associated with a specified relation [26].

###  II-C Hypergraph Learning

Hypergraph learning explores the high-order correlations in the data, which
extends the traditional graph learning models to a high dimensional and more
complete nonlinear space. It offers a promising solution for analyzing complex
structured data with satisfactory performance in practice [16]. The
construction of the hypergraph from the given data is a key step for
hypergraph learning, which significantly affects the final learning
performance.

Zhang et al. proposed dynamic hypergraph structure learning (DHSL) to update
hypergraph structure iteratively during the learning process [27]. It is
essential to make dynamic modifications to the initial hypergraph structures
from adjusted feature embedding. Feng et al. presented a framework named
hypergraph neural network (HGNN) for handling complex and high-order
correlations. In HGNN, the complex data correlation is formulated in a
hypergraph structure, and a hyperedge convolution operation is used to exploit
the high-order data correlation for representation learning [9].

To exploit high-order relations among the features, Jiang et al. proposed a
dynamic hypergraph neural networks (DHGNN) framework that is composed of
dynamic hypergraph construction (DHG) and hypergraph convolution (HGC). DHG
utilizes the k-NN method to generate the basic hyperedge and extends the
adjacent hyperedge set by k-means clustering, with which the local and global
relations can be extracted. HGC is designed to encode high-order data
relations in the hypergraph structure [28]. Cai et al. introduced a hypergraph
structure learning (HSL) framework to optimize the hypergraph structure and
HGNNs simultaneously in an end-to-end way. To efficiently learn the hypergraph
structure, HSL adopts a hyperedge sampling strategy to prune the redundant
hyperedges, which is followed by an incident node sampling for pruning
irrelevant incident nodes and discovering potential implicit connections. The
consistency between the optimized structure and the original structure is
maintained by the intra-hyperedge contrastive learning module [29]. Gao et al.
proposed a tensor-based dynamic hypergraph learning (t-DHL) model to
efficiently learn dynamic hypergraphs. t-DHL utilizes a tensor representation
to characterize the dynamic hypergraph structure more flexibly. During the
optimization of the tensor representation, not only the weights but also the
number and order of hyperedges can be adjusted. As an extended version of
HGNN, a general hypergraph neural network framework named HGNN+ was proposed
for modeling high-order representation among the data, which is achieved by
bridging multi-modal/multi-type data and hyperedge with hyperedge groups [30].

Recent works mostly focus on neighborhood-based construction strategies when
constructing hyperedges in the hypergraph. Specifically, hyperedge is often
constructed through clustering a master vertex and its neighbor vertices.
However, for the commonly adopted clustering methods like k-NN [12, 28, 9],
they treat hypergraph construction and graph learning separately. As a result,
these models are susceptible to noisy data which limits their applications.
Apart from that, most neighborhood-based models fail to incorporate the
heterogeneity of the topology, leading to a single type hyperedge [16]. In
this work, we propose a heterogeneous hypergraph learning framework LFH to
solve the above mentioned issues. LFH constructs the hyperedge based on
different edge types, which can then be integrated into the learning process
to improve the learned representations.

##  III Preliminaries and problem formulation

In this section, we first introduce the relevant definitions and then
formulate the key problems associated with representation learning on
heterogeneous hypergraphs. The notations used in this paper are listed in
Table I.

######  Definition 1.

(Heterogeneous Pairwise Graph). A heterogeneous graph
$\dot{\mathcal{G}}=\\{\dot{\mathcal{V}},\dot{\mathcal{E}},\dot{\mathcal{T}_{e}},\dot{\mathcal{T}_{v}}\\}$
is defined as a graph with multiple node types $\dot{\mathcal{T}_{v}}$ or edge
types $\dot{\mathcal{T}_{e}}$, where
$\dot{\mathcal{V}}=\\{\emph{v}_{1},\emph{v}_{2},...,\emph{v}_{N}\\}$ denotes
the set of nodes and
$\dot{\mathcal{E}}=\left\\{\left(\emph{v}_{i},\emph{v}_{j}\right)|\emph{v}_{i},\emph{v}_{j}\in\dot{\mathcal{V}}\right\\}$
denotes the set of edges. We define the node type mapping function
$\dot{\phi}:\dot{\mathcal{V}}\rightarrow\dot{\mathcal{T}_{v}}$ and the edge
type mapping function
$\dot{\psi}:\dot{\mathcal{E}}\rightarrow\dot{\mathcal{T}_{e}}$, respectively.

######  Definition 2.

Figure 2: An illustrative example of the ACM dataset: (a) Heterogeneous
pairwise graph including four node types (_Author_ , _Paper_ , _Subject_) and
four edge types (_Write_ , _Written by_ , _Belong to_ , _Contain_). (b) A
heterogeneous hypergraph that models two implicit data relations: _preference
for subjects_ and _the relation among the subjects_.

(Heterogeneous Hypergraph). A heterogeneous hypergraph is defined as
$\mathcal{G}=\\{\mathcal{V},\mathcal{E},\mathcal{T}_{e},\mathcal{T}_{v},\mathcal{W}\\}$,
where $\mathcal{V}=\\{\emph{v}_{1},...,\emph{v}_{N}\\}$ is the set of nodes
and $\mathcal{E}=\\{\emph{e}_{1},...,\emph{e}_{M}\\}$ is the set of
hyperedges. N and M are the number of nodes and hyperedges, respectively. For
any hyperedge $\emph{e}\in\mathcal{E}$, it is composed of a subset of nodes
$\mathcal{V}$, which can be defined as
$\emph{e}=\\{\emph{v}_{i},...,\emph{v}_{j}\\}\subseteq\mathcal{V}$. We define
the node type mapping function $\phi:\mathcal{V}\rightarrow\mathcal{T}_{v}$
and the hyperedge type mapping function
$\psi:\mathcal{E}\rightarrow\mathcal{T}_{e}$, respectively. Each heterogeneous
hypergraph presents multiple hyperedge types, i.e., $|\mathcal{T}_{e}|>1$. A
positive diagonal matrix $\mathcal{W}\in\mathcal{R}^{M\times M}$ is used to
denote the weight of each hyperedge, that is,
$diag\left(\mathcal{W}\right)=\left[w\left(\emph{e}_{1}\right),...,w\left(\emph{e}_{M}\right)\right]$.
The relationship between nodes and hyperedges is usually represented by
incidence matrix $\emph{H}\in\mathcal{R}^{N\times M}$, with entries are
continuous numbers within the range of $[0,1]$. If the hyperedge $e_{j}$ is
incidenced with node $v_{i}$, the value of $\emph{H}\left(v_{i},e_{j}\right)$
is positive ; otherwise, $\emph{H}\left(\emph{v}_{i},\emph{e}_{j}\right)=0$.
The element $\emph{H}\left(\emph{v}_{i},\emph{e}_{j}\right)$ can be regarded
as the importance of node $\emph{v}_{i}$ for hyperedge $\emph{e}_{j}$. We can
define the degree of each node $\emph{v}_{i}\in\mathcal{V}$ and the edge
degree of hyperedge $\emph{e}_{j}\in\mathcal{E}$ as:

$d\left(\emph{v}_{i}\right)=\sum_{\emph{e}_{p}\in\mathcal{E}}w\left(\emph{e}_{p}\right)\cdot\emph{H}\left(\emph{v}_{i},\emph{e}_{p}\right)$
(1)
$\delta\left(\emph{e}_{j}\right)=\sum_{\emph{v}_{p}\in\mathcal{V}}\emph{H}\left(\emph{v}_{p},\emph{e}_{j}\right)$
(2)

Figure 2 provides an illustrative comparison between a heterogeneous pairwise
graph and a hypergraph. Figure 2 (a) sketches the ACM dataset composed of
three node types (_Author_ , _Paper_ , _Subject_) and four edge types (_Write_
, _Written by_ , _Belong to_ , _Contain_). By contrast, a heterogeneous
hypergraph is capable of modeling a more complex graph structure by
constructing different types of hyperedges, in which every hyperedge can
connect any number of nodes to model a specific type of data relationship. As
shown in Figure 2 (b), the purple hyperedge models an author’s preference for
subjects, while the yellow one models the relation among different subjects.

Figure 3: An overview of our proposed LFH framework.

######  Problem 1.

(Hyperedge Generation in Heterogeneous Hypergraph). While node set
$\mathcal{V}$ explicitly exists in graph data, the hyperedge set $\mathcal{E}$
is normally implicit. Following Definition 2, any hyperedge
$\emph{e}\in\mathcal{E}$ contains a subset of nodes $\mathcal{V}$. We aim to
learn a hyperedge construction function
$f_{con}:\mathcal{V}\rightarrow\mathcal{E}$, where hyperedges are constructed
based on the given nodes. Moreover, this function can be used for constructing
incidence matrix _H_.

######  Problem 2.

(Representation Learning on Heterogeneous Hypergraph). Given a heterogeneous
hypergraph
$\mathcal{G}=\\{\mathcal{V},\mathcal{E},\mathcal{T}_{e},\mathcal{T}_{v},\mathcal{W}\\}$,
we aim to learn a mapping function
$f_{emb}\left(\mathcal{G}\right)\rightarrow\emph{Z}^{\mathcal{G}}$, where
$\emph{Z}^{\mathcal{V}}\in\mathcal{R}^{d\times N}$ indicates representation
embedding of all nodes in $\mathcal{G}$. This representation embedding can be
used for downstream predictive tasks such as node classification and link
prediction.

TABLE I: Table of key notations

Notation | Description
---|---
$\|$ | Concatenation
$\|\cdot\|_{1}$ |  $l_{1}$ norm
$\|\cdot\|_{2}$ |  $l_{2}$ norm
$|\cdot|$ | The set size
$\mathcal{L}_{recon}$ | Reconstruction loss
$\mathcal{L}_{label}$ | Label loss
$Softmax\left(\cdot\right)$ | Softmax function
$f_{con}$ | Hyperedge construction function
$f_{emb}$ | Representation embedding mapping function
$\Theta_{p}$,$\theta$,$\Theta^{att}$ | Learnable model parameters
$\dot{\mathcal{G}}$ | Heterogeneous pairwise graph
$\dot{\mathcal{V}}$ | The set of nodes in the heterogeneous pairwise graph
$\dot{\mathcal{E}}$ | The set of edgees in the heterogeneous pairwise graph
$\dot{\mathcal{T}_{v}}$ | The set of node types in the heterogeneous pairwise graph
$\dot{\mathcal{T}_{e}}$ | The set of edge types in the heterogeneous pairwise graph
$\dot{\emph{X}}$ | Raw node features in the heterogeneous pairwise graph
$\dot{\emph{x}_{i}}$ | Raw feature of node $v_{i}$ in the heterogeneous pairwise graph
$\dot{\phi}$ | Node type mapping function in the heterogeneous pairwise graph
$\dot{\psi}$ | Edge type mapping function in the heterogeneous pairwise graph
$\mathcal{G}$ | Heterogeneous hypergraph
$\mathcal{V}$ | The set of nodes in the heterogeneous hypergraph
$\mathcal{E}$ | The set of edgees in the heterogeneous hypergraph
$\mathcal{T}_{v}$ | The set of node types in the heterogeneous hypergraph
$\mathcal{T}_{e}$ | The set of edge types in the heterogeneous hypergraph
$\phi$ | Node type mapping function in the heterogeneous hypergraph
$\psi$ | Edge type mapping function in the heterogeneous hypergraph
X | Initial node embedding in the heterogeneous hypergraph
$\emph{x}_{i}$ | Initial node embedding of node $v_{i}$ in the heterogeneous hypergraph
$\mathcal{W}$ | Weight matrix of hyperedges
H | Incidence matrix
$\emph{Z}^{\mathcal{V}}$ | Embedding of all nodes in $\mathcal{V}$
$\emph{Y}^{\mathcal{V}}$ | Labels of nodes
$v$ | A node $v\in\mathcal{V}$
$e$ | A node $e\in\mathcal{E}$
$w\left(e\right)$ | The weight of hyperedge $e$
$d\left(v\right)$ | Degree of node $v$
$\delta\left(e\right)$ | Degree of edge $e$
$v_{i}^{m}$ | The master node of $v_{i}^{m}$
$\emph{S}\left(v_{i}^{m}\right)$ | The slave node set of the master node $v^{m}_{i}$
$\tilde{\emph{S}}\left(v_{i}^{m}\right)$ | The candidate slave node set of the master node $v^{m}_{i}$
$\mathcal{T}\left(\tilde{\emph{S}}\left(v_{i}^{m}\right)\right)$ | The type of the candidate slave node set
$\hat{\emph{X}}\left(v_{i}^{m}\right)$ | The reconstructed vector of the master node $v_{i}^{m}$
$\emph{p}_{v_{i}^{m}}$ | The reconstruction coefficient vector of $v_{i}^{m}$
$c\left(v^{m}_{i}\right)$ | Reconstruction loss of the master node $v^{m}_{i}$
$M$ | The number of hyperedges
$N$ | The number of nodes
$n_{i}$ | The number of candidate slave node set associated with node $v_{i}$
$M_{i}$ | The number of hyperedges associated with the node $v_{i}$
E | Hyperedges embeddding
$K$ | The number of attention heads

##  IV Heterogeneous Hypergraph

In this section, we introduce our proposed _Learning from Heterogeneity_ (LFH)
framework for hypergraphs. Figure 3 provides an overview of LFH. We first
demonstrate the pairwise fusion process from the heterogeneous pairwise graph
to obtain a high-quality initial node embedding. Afterward, we dynamically
generate the heterogeneous hyperedges among the nodes with different node
types through feature reconstruction. After the heterogeneous hypergraph is
constructed, our proposed dynamic hypergraph learning can then be performed to
train the node embedding space, in which node representations can be derived
for downstream node classification and link prediction tasks.

###  IV-A Initial Node Embedding

As demonstrated in Fig. 3, the input of this step is the heterogeneous
pairwise graph. It is of great importance to generate high-quality initial
node embedding to better exploit the implicit data relations in the
hypergraph. Intuitively, fusing pairwise information from a heterogeneous
pairwise graph into the initial node embeddings can help capture high-order
correlations among the nodes and thus affects the construction of the
hypergraph. Given a heterogeneous pairwise graph
$\dot{\mathcal{G}}=\\{\dot{\mathcal{V}},\dot{\mathcal{E}},\dot{\mathcal{T}_{e}},\dot{\mathcal{T}_{v}}\\}$
and its raw node features
$\dot{\emph{X}}=\left[\dot{\emph{x}}_{1},\dot{\emph{x}}_{2},...,\dot{\emph{x}}_{N}\right]\in\mathcal{R}^{D\times
N}$, where the raw feature of node $v_{i}\in\mathcal{V}$ is
$\dot{\emph{x}}_{i}\in\mathcal{R}^{D}$. The pairwise fusion can be implemented
by any GNN model that operates on heterogeneous pairwise graphs. Formally, the
pairwise fusion process can be defined as:

$\emph{X}=\textit{pairwise\\_fusion}\left(\dot{\mathcal{G}},\dot{\emph{X}};\Theta_{p}\right)$
(3)

where $\Theta_{p}$ is the trainable parameters of any GNN model, such as
GAT[6] and PC-HGN[7]. The output
$\emph{X}=\left[\emph{x}_{1},\emph{x}_{2},...,\emph{x}_{N}\right]\in\mathcal{R}^{d\times
N}$ is the generated initial node embeddings that contain pairwise
information.

###  IV-B Heterogeneous Hyperedge Generation

The special structure of hyperedges enables hypergraphs to encode high-order
data relations. By encircling a specific set of nodes with common attributes
or close relationships within one hyperedge, we can effectively represent
local group information among the nodes. Therefore, it is essential to
discover the related nodes for different hyperedges. To facilitate hyperedge
construction, we define two types of nodes: master node and slave node. A
master node $v^{m}_{i}\in\mathcal{V}$ serves as an anchor when constructing a
hyperedge, which dynamically encircles a set of slave nodes $\\{v^{s}_{i}\\}$
to jointly form the hyperedge. The collection of participant nodes is denoted
as $S\left(v^{m}_{i}\right)$, which is united with master node $v_{i}^{m}$ to
represent a complete data relation within a hyperedge. Previous works have
adopted the $k$-NN method to select the k-th nearest nodes to form a hyperedge
[27, 31]. Alternative approaches solve an independent optimization problem to
generate the hyperedges [12, 32], however, this solution fails to incorporate
the heterogeneity of the hyperedges, and thus cannot be integrated into the
representation learning process.

To address the above challenges, here we propose a dynamic learning process
that generates heterogeneous hyperedges and adaptively reconstructs the
hyperedge in different types. Specifically, for each master node
$v^{m}_{i}\in\mathcal{V}$, our framework encloses the related slave nodes of
various types to encode heterogeneous topological structures with implicit
relationships, based on different attributes among the nodes. Suppose the
master node $v^{m}_{i}$ has a set of $n_{i}$ candidate slave nodes
$\tilde{S}^{k}\left(v^{m}_{i}\right)$, where $k=\left[1,...,n_{i}\right]$ with
different types. The hyperedge can be reconstructed based on the node types of
the master node and candidate slave node sets. The type of the candidate slave
node set can be defined as
$\mathcal{T}\left(\tilde{S}^{k}\left(v^{m}_{i}\right)\right)$ and clearly is a
subset of node type $\mathcal{T}_{v}$. Each candidate slave node set
$\tilde{S}^{k}\left(v^{m}_{i}\right)$ contains all nodes with one or several
node types, which is denoted as
$\tilde{S}^{k}\left(v^{m}_{i}\right)=\left\\{v|\phi\left(v\right)\in\mathcal{T}\left(\tilde{S}^{k}\left(v^{m}_{i}\right)\right),v\neq
v_{i}^{m}\right\\}$. Since each hyperedge is generated from the corresponding
candidate slave node set of a certain type, the type number of the hyperedge
is equal to the type number of the candidate slave node set. An exemplary
hyperedge construction is demonstrated in the middle part of Figure 3, where
the master node encompasses $|\mathcal{T}_{v}|$ candidate slave node sets with
distinct types. For each type, a hyperedge (represented as the coloured
triangle) is dynamically constructed. Mathematically, this process of
generating a hyperedge from a master node $v_{i}^{m}$ and one of its candidate
slave node set $\tilde{S}^{k}\left({v_{i}^{m}}\right)$ can be denoted as:

$\hat{\emph{X}}_{k}\left(v^{m}_{i}\right)=\emph{X}\left(\tilde{S}^{k}\left(v^{m}_{i}\right)\right)\cdot\emph{p}_{v^{m}_{i}}^{k}$
(4)

It can be interpreted as the selection of slave nodes for this hypergraph
through a linear combination of candidate slave node embedding
$\emph{X}\left(\tilde{S}^{k}\left(v^{m}_{i}\right)\right)\in\mathcal{R}^{d\times|\tilde{S}^{k}\left(v^{m}_{i}\right)|}$
and trainable reconstruction coefficient vector
$\emph{p}_{v^{m}_{i}}^{k}\in\mathcal{R}^{|\tilde{S}^{k}\left(v^{m}_{i}\right)|}$.
Each element $p_{v^{m}_{i}}^{k}\left(v\right)$ is the learned reconstruction
coefficient associated with the node
$v\in\tilde{S}^{k}\left(v^{m}_{i}\right)$. Based on
$\emph{p}_{v^{m}_{i}}^{k}$, the nodes in the candidate slave node set
$\tilde{S}^{k}\left(v^{m}_{i}\right)$ with reconstruction coefficient larger
than zero are grouped to generate a hyperedge with the master node, which is
denoted as
$S^{k}\left(v^{m}_{i}\right)=\left\\{v|v\in\tilde{S}^{k}\left(v^{m}_{i}\right),\emph{p}_{v^{m}_{i}}^{k}\left(v\right)>0\right\\}$.
The reconstruction error $c^{k}\left(v^{m}_{i}\right)$, which measures the
difference between the master node and the reconstructed master node, is
calculated as follows:

$c^{k}\left(v^{m}_{i}\right)=\|\hat{\emph{X}}_{k}\left(v^{m}_{i}\right)-\theta_{k}\cdot\emph{X}\left(v^{m}_{i}\right)\|_{2}$
(5)

$\theta_{k}\in\mathcal{R}^{d\times d}$ is a type-specific trainable projection
matrix and $\|\cdot\|_{2}$ is the $l_{2}\text{-}norm$ regularization of a
vector. Note that the purpose of hyperedge construction is to dynamically
group the most relevant nodes, forming a hyperedge in which the nodes have
strong dependencies. Practically, we add $l_{1}\text{-}norm$ regularization
and $l_{2}\text{-}norm$ regularization of p simultaneously. The
$l_{1}\text{-}norm$ regularization constrains the reconstruction coefficient
to zero, which tends to group fewer slave nodes to reconstruct the master
node. However, the $l_{1}\text{-}norm$ regularization is sensitive to noise
and outliers, which leads to non-smooth results. For this reason, we consider
the $l_{1}\text{-}norm$ regularization and $l_{2}\text{-}norm$ regularization
simultaneously and use a weight hyperparameter $\gamma$ to trade off the
effect of both. Overall, the loss of this process is defined as:

$\mathcal{L}_{recon}=\sum_{i=\left[1,...,N\right]}\sum_{k=\left[1,...,n_{i}\right]}\lambda
c^{k}\left(v^{m}_{i}\right)+\|\emph{p}_{v^{m}_{i}}^{k}\|_{1}+\gamma\|\emph{p}_{v^{m}_{i}}^{k}\|_{2}~{}$
(6)

where $\|\cdot\|_{1}$ denotes the $l_{1}\text{-}norm$ regularization of a
vector, $\lambda$ is the weight hyperparameter of the reconstruction error,
and $\gamma$ is the norm hyperparameter to trade off the $l_{1}\text{-}norm$
regularization and $l_{2}\text{-}norm$ regularization of the reconstruction
coefficient vector $\emph{p}_{v^{m}_{i}}^{k}$.

From the $n_{i}$ candidate slave node sets of the master node $v_{i}^{m}$, we
thus can generate the $n_{i}$ hyperedges of different types. Iteratively, each
node $v_{j}$ could act as the master node to generate several hyperedges of
different types to lead to the final hypergraph. The corresponding incidence
matrix H of the hypergraph is denoted as:

$\displaystyle\emph{H}\left(v_{j},e^{k}\left(v_{i}^{m}\right)\right)$
$\displaystyle=$ $\displaystyle\begin{cases}1&\text{$v_{j}=v_{i}^{m}$}\\\
\emph{p}_{v^{m}_{i}}^{k}\left(v_{j}\right)&\text{$v_{j}\in
e^{k}\left(v^{m}_{i}\right),v_{j}\neq v_{i}^{m}$}\\\
0&\text{otherwise}\end{cases}$ (7)

Figure 4: Type-specific multi-head attention on node embedding update

###  IV-C Dynamic Hypergraph Learning

As illustrated in Fig 3, dynamic hypergraph learning consists of two key
components: hyperedge embedding updating and multi-head attention node
embedding updating. This section elaborates on these components and loss
function.

####  IV-C1 Hyperedge Embedding Updating

We aggregate node embeddings to the hyperedge containing these nodes. The
hyperedges embedding is denoted as $\emph{E}\in\mathcal{R}^{d\times M}$. The
embedding of each hyperedge $e^{k}\left(v^{m}_{i}\right)$ is obtained as
follows:

$\emph{E}\left(e^{k}\left(v^{m}_{i}\right)\right)=\frac{\emph{X}\cdot\emph{H}\left(e^{k}\left(v^{m}_{i}\right)\right)}{\delta\left(e^{k}\left(v^{m}_{i}\right)\right)}$
(8)

$\emph{H}\left(e^{k}\left(v^{m}_{i}\right)\right)\in\mathcal{R}^{N\times 1}$
is a column of incidence matrix H (See Eq. 7), where each element is a
coefficient that represents the importance of the node to the hyperedge
$e^{k}\left(v^{m}_{i}\right)$. The degree of edge
$\delta\left(e^{k}\left(v^{m}_{i}\right)\right)$ is the normalization factor
in the process of hyperedge embedding updating.

####  IV-C2 Multi-head Attention Node Embedding Updating

Previous works calculate the similarities among hyperedges and then use them
as weighting coefficients when updating respective node embeddings [33, 32].
However, the heterogeneity in the hypergraph is omitted, especially with
multiple types of hyperedge and nodes. To cope with that, we design a
heterogeneous multi-head attention mechanism that can train the importance of
heterogeneous hyperedges with respect to nodes as shown in Figure 4.

We use all hyperedges associated with master node $v^{m}_{i}$ to update the
node embedding. The hyperedges associated with node $v^{m}_{i}$ is represented
as
$\mathcal{E}\left(v^{m}_{i}\right)=\left\\{e^{1}\left(v^{m}_{i}\right),e^{2}\left(v^{m}_{i}\right),...,e^{M_{i}}\left(v^{m}_{i}\right)\right\\}$.

Figure 4 illustrates the calculation process of a master node $v^{m}_{i}$ and
two related hyperedges $e^{1}\left(v^{m}_{i}\right)$ and
$e^{2}\left(v^{m}_{i}\right)$: we first calculate the multi-head attention
between $v_{i}^{m}$ and $e^{1}\left(v^{m}_{i}\right)$, and then use the
normalized attention as the weight of $e^{1}\left(v^{m}_{i}\right)$. The
calculation involves Eq.9 to Eq.13.

$\emph{Q}^{h}\left(v^{m}_{i}\right)=Q\text{-}Proj^{h}_{\phi\left(v^{m}_{i}\right)}\cdot\emph{X}\left(v^{m}_{i}\right)$
(9)

where the dimension of $\emph{X}\left(v^{m}_{i}\right)$ is
$\mathcal{R}^{d\times 1}$, and
$Q\text{-}Proj^{h}_{\phi\left(v^{m}_{i}\right)}\in\mathcal{R}^{\frac{d}{K}\times
d}$ represents the projection matrix on the $h$-th attention head,
$h\in\left[1,K\right]$. $K$ is the number of attention heads. Note the
projection matrices are distinguished on hyperedge types. Similarly, we also
project the hyperedge $e^{1}\left(v^{m}_{i}\right)$ through
$K\text{-}Proj^{h}_{\psi\left(e^{1}\left(v^{m}_{i}\right)\right)}$ into
$\emph{K}^{h}\left(e^{1}\left(v^{m}_{i}\right)\right)$ as:

$\emph{K}^{h}\left(e^{1}\left(v^{m}_{i}\right)\right)=K\text{-}Proj^{h}_{\psi\left(e^{1}\left(v^{m}_{i}\right)\right)}\cdot\emph{E}\left(e^{1}\left(v^{m}_{i}\right)\right)$
(10)

where $\emph{E}\left(e^{1}\left(v^{m}_{i}\right)\right)\in\mathcal{R}^{d\times
1}$ is the embedding of hyperedge $e^{1}\left(v^{m}_{i}\right)$. The dimension
of $K\text{-}Proj^{h}_{\psi\left(e^{1}\left(v^{m}_{i}\right)\right)}$ is
$\mathcal{R}^{\frac{d}{K}\times d}$ while the dimension of the output
$\emph{K}^{h}\left(e^{1}\left(v^{m}_{i}\right)\right)$ is
$\mathcal{R}^{\frac{d}{K}}$.

$\begin{split}att^{h}(v^{m}_{i},e^{1}(v^{m}_{i}))=\big{(}\emph{Q}^{h}&(v^{m}_{i})\cdot\Theta^{att}_{\psi\left(e^{1}\left(v^{m}_{i}\right)\right)}\\\
&\cdot\emph{K}^{h}\left(e^{1}\left(v^{m}_{i}\right)\right)\big{)}\cdot\frac{\mu_{\psi(e^{1}(v^{m}_{i}))}}{\sqrt{d}}\end{split}$
(11)

Eq. 11 derives the attention value of the $h$-th head between $v^{m}_{i}$ and
$e^{1}\left(v^{m}_{i}\right)$. The type-specific learnable matrix
$\Theta^{att}_{\psi\left(e^{1}\left(v^{m}_{i}\right)\right)}\in\mathcal{R}^{\frac{d}{K}\times\frac{d}{K}}$
for edge type $\psi\left(e^{1}\left(v^{m}_{i}\right)\right)$ represents the
learnable semantic information for each edge type. $\mu$ is a scaling factor
for different hyperedge types. Moreover, since the magnitude of K and Q can
increase the attention value significantly and eventually lead to the gradient
explosion problem, we divide the obtained value by $\sqrt{d}$ to stabilize the
training process.

Following that, the calculated attention value
$att^{h}\left(v^{m}_{i},e^{1}\left(v^{m}_{i}\right)\right)$ is able to
quantify the importance of $e^{1}\left(v^{m}_{i}\right)$ to $v^{m}_{i}$ on the
$h$-th adaptively. The weight of hyperedge $e^{1}\left(v^{m}_{i}\right)$ is
obtained as follows:

$\tilde{\emph{w}}\left(v^{m}_{i},e^{1}\left(v^{m}_{i}\right)\right)=\mathop{\operatorname*{\mathchoice{\Big{\|}}{\big{\|}}{\|}{\|}}}\limits_{h\in\left[1,K\right]}att^{h}\left(v^{m}_{i},e^{1}\left(v^{m}_{i}\right)\right)$
(12)

where $\operatorname*{\mathchoice{\Big{\|}}{\big{\|}}{\|}{\|}}$ represents
concatenation. The dimension of weight vector
$\tilde{\emph{w}}\left(v^{m}_{i},e^{1}\left(v^{m}_{i}\right)\right)$ is
$\mathcal{R}^{K}$, stacked by attention value on each head. To yield the final
weight vector for each hyperedge, we gather the weight vectors from all
related hyperedges of $\mathcal{E}\left(v^{m}_{i}\right)$ and conduct
$Softmax$ as:

$\emph{w}\left(v^{m}_{i},e\right)=\mathop{Softmax}\limits_{\forall
e\in\mathcal{E}\left(v^{m}_{i}\right)}\left(\tilde{\emph{w}}\left(v^{m}_{i},e\right)\right)$
(13)

where
$\emph{w}\left(v^{m}_{i},e\right)=\left[\emph{w}\left(v^{m}_{i},e^{1}\left(v^{m}_{i}\right)\right),...,\emph{w}\left(v^{m}_{i},e^{M_{i}}\left(v^{m}_{i}\right)\right)\right]\in\mathcal{R}^{K\times
M_{i}}$, with entries are normalized among all hyperedges per head
(illustrated in the right part of Fig. 4). The attentive aggregation of head
$h$ for node $v^{m}_{i}$ is formulated as:

$\emph{z}_{i}^{h}=\sum_{k\in\left[1,M_{i}\right]}\left(w^{h}\left(v^{m}_{i},e^{k}\left(v^{m}_{i}\right)\right)\times\emph{K}^{h}\left(e^{k}\left(v^{m}_{i}\right)\right)\right)$
(14)

$w^{h}\left(v^{m}_{i},e^{k}\left(v^{m}_{i}\right)\right)$ is the scalar
element of $h$-th head in the weight vector
$\emph{w}\left(v^{m}_{i},e^{k}\left(v^{m}_{i}\right)\right)$ of hyeredge
$e^{k}\left(v^{m}_{i}\right)$, and the dimension of $\emph{z}_{i}^{h}$ is
$\mathcal{R}^{\frac{d}{K}}$. After concatenating all $K$ heads, we obtain the
updated node embedding of $v^{m}_{i}$ through a shallow multi-layer perceptron
(MLP), which is denoted as:

$\emph{Z}^{\mathcal{V}}\left(v^{m}_{i}\right)=\textit{MLP}\left(\mathop{\operatorname*{\mathchoice{\Big{\|}}{\big{\|}}{\|}{\|}}}\limits_{h\in\left[1,K\right]}\emph{z}^{h}_{i}\right)$
(15)

###  IV-D Loss Function

We realize LFH in an end-to-end fashion, including the initialization of node
embedding, the dynamic heterogeneous hyperedges generation and heterogeneous
hyperedges learning. The learnt node representations can then be used in
downstream tasks such as node classification and link prediction. The
integrated training process is conducted in a supervised manner. We propose a
united loss function, which combines the loss of reconstruction in the dynamic
heterogeneous hyperedges generation and the loss of supervised downstream
task. We take node classification as an example of a downstream task. In this
case, the label of the node set $\mathcal{V}$ is $\emph{Y}^{\mathcal{V}}$. We
use an MLP to map node representations to label space, and the loss in node
classification can be calculated as:

$\mathcal{L}_{label}=\textit{CE}\left(\textit{MLP}\left(\emph{Z}^{\mathcal{V}}\right),\emph{Y}^{\mathcal{V}}\right)$
(16)

where CE is the cross-entropy function and used to measure the differences
between estimated labels and the true labels of nodes. Finally, the united
loss is defined as:

$\mathcal{L}_{u}=\left(1-\alpha\right)\times\mathcal{L}_{label}+\alpha\mathcal{L}_{recon}$
(17)

where $\mathcal{L}_{recon}$ is defined in Eq. 6. The process of reconstructing
the master node using slave nodes can uncover implicit relationships between
the master node and others. However, as pointed out above, this process is
sensitive to noisy nodes. Over-reliance on reconstruction loss compromises the
performance of our model. To enhance the generalization ability, we need to
use a weight hyperparameter $\alpha$, to balance the effects of loss of
supervised downstream task and reconstruction loss.

###  IV-E Model Analysis and Discussion

We analyze the computational cost and summarize the advantages of LFH as
below:

####  IV-E1 Computational Cost

In the process of heterogeneous hypergraph generation, the computational cost
mainly lies in the reconstruction of the master node from its candidate slave
node set (Eq. 4). As it is essential to generate hyperedges of different types
for each node, and the computational cost of heterogeneous hypergraph
generation is $\emph{O}\left(d|\mathcal{T}_{v}|N\right)$, where $N$ is the
number of nodes, $d$ is the dimensionality of node embedding, and
$|\mathcal{T}_{v}|$ is the number of node types. Additionally, for the process
of dynamic hypergraph learning, we use the incidence matrix H and nodes
embedding X to update hyperedges embedding, with the computational cost of
$\emph{O}\left(dMN\right)$. Note the number of hyperedges $M$ is proportional
to the product of the $N$ and the number of edge types $|\mathcal{T}_{e}|$,
thus the computational cost of hyperedges embedding updating is
$\emph{O}\left(d|\mathcal{T}_{e}|N^{2}\right)$. Moreover, the computational
cost of multi-head attention node embedding updating is
$\emph{O}\left(d^{2}|\mathcal{T}_{e}|N\right)$, where $\mathcal{T}_{e}$ is the
number of hyperedge types. Hence, the total computational cost of our proposal
is
$\emph{O}\left(d|\mathcal{T}_{e}|N^{2}+\left(d|\mathcal{T}_{v}|+d^{2}|\mathcal{T}_{e}|\right)N\right)$.
The scale of $d$, $\mathcal{T}_{e}$ and $\mathcal{T}_{v}$ are usually far less
than $N$, This computational cost is on par with many existing hypergraph-
based models, such as [12] and [34].

####  IV-E2 Discussion

Compared to pairwise graph learning models such as GAT [6] and PC-HGN [7], LFH
is capable of modeling implicit high-order data relations. Moreover, LFH has
two advantages. First, the process of hyperedges generation is conducted with
hypergraph learning within a united training process while taking into account
the heterogeneity of the graph. This enables an adaptive hypergraph modeling
than HGNN [9], whose hyperedges are generated by clustering method k-NN. Other
works that use static clustering methods for hyperedge generation adopt
different strategies of k-means or a combination of k-NN and k-means to
generate different hyperedges [27, 35]. The main limitation of this line of
work is the static clustering methods are sensitive to noise and outliers.
Another issue is the hyperparameters of clustering method may affect the
hyperedge generation, and adaptively calibrating the hyperparamters of
clustering method during training is non-trivial. LFH, on the other hand, can
incorporate hyperedges information of different types in different heads,
using a heterogeneous multi-head attention mechanism. The mechanism
dynamically quantifies the weights of hyperedges based on the correlation
between the node representation and the related hyperedges representation,
making the weights more descriptive of the importance of the related
hyperedges to the node.

##  V Experiment

In this section, we empirically evaluate the effectiveness of our LFH
framework and analyze the impacts of its key components to the final
performance. We conduct our experiments on three different graph datasets,
including DBLP, IMDB, and ACM. Additionally, we compare the performance of LFH
with eleven baseline models, which include homogenous graph learning,
heterogeneous graph learning, and hypergraph learning models.

###  V-A Experimental Setup

TABLE II: Statistics of the datasets. Dataset | # Nodes | # Node Types | # Edges | # Edge Types | # Node Features
---|---|---|---|---|---
ACM | 8994 | 3 | 12961 | 4 | 1902
DBLP | 18405 | 3 | 33973 | 4 | 334
IMDB | 12772 | 3 | 18644 | 4 | 1256

####  V-A1 Dataset

The experiments are conducted on three datasets, which are popularly used in
classification tasks. The details of these datasets are summarized in Table
II.

  * •

DBLP111https://s3.cn-north-1.amazonaws.com.cn/dgl-
data/dataset/openhgnn/dblp4GTN.zip is an academic network from four research
areas, including database, machine learning, data mining and information
retrieval. It uses Paper (P), Author (A), and Conference (C) as different node
types, while edges are presented as P-A, A-P, P-C and C-P with different edge
types. Four research areas are used as labels for this dataset. The initial
node features are calculated using bag-of-words.

  * •

ACM222 https://s3.cn-north-1.amazonaws.com.cn/dgl-
data/dataset/openhgnn/acm4GTN.zip shares a similar data characterises with
DBLP. It contains Paper (P), Author (A) and Subject (S) as node types, along
with four types of edges (P-A, A-P, P-S and S-P). The papers are labelled into
three classes (Database, Wireless Communication, Data Mining). It also uses
bag-of-words to construct the initial node features.

  * •

IMDB333 https://s3.cn-north-1.amazonaws.com.cn/dgl-
data/dataset/openhgnn/imdb4GTN.zip contains movie (M), actors (A) and
directors (D). Each movie is labelled according to its genre (Action, Comedy,
Drama). Node features are also initialized using bag-of-words.

####  V-A2 Baseline Models and Configurations

We compare with some state-of-the-art baselines, each of which reports
promising results in the classification task. Although these baselines share
the same goal of learning the representation of the graph, they are originally
designed to fulfil graph learning on different graph data, including
homogeneous pairwise graph learning, pairwise heterogeneous graph learning,
and hypergraph learning.

  * •

GCN [5] is a graph convolutional network designed specifically for homogeneous
pairwise graph learning. The depth of the layer in GCN is set to 2.

  * •

GAT [6] is the first work that introduces the attention mechanism in
homogeneous graph learning. It enables the weighted message aggregations from
neighbour nodes. The number of attention heads is set to 3.

  * •

GraphSAGE [17] designs a sampling approach when aggregating messages from
neighbour nodes. It also supports different aggregation functions. The sample
window of GraphSAGE is set to 10.

  * •

GraphSAINT [20] splits nodes and edges from a bigger graph into a number of
subgraphs, on which the GCN is applied for node representation learning. We
adopt the node sampling strategy for GraphSAINT and use 8000 as the node
budget and 25 as the number of subgraphs.

  * •

HAN [24] uses attention techniques on heterogeneous graph learning, in which
the node embeddings are updated through manually designed meta-paths. The
number of attention heads is set to 8.

  * •

HGT [25] designs type-specific attention layers, assigning different trainable
parameters to each node and edge type. For the best performance of HGT, the
number of attention heads is set to 8.

  * •

R-HGNN [26] proposes a relation-aware model to learn the semantic
representation of edges while discerning the node representations with respect
to different relation types. It is designed to learn heterogeneous graph
representations. The depth of the layer in R-HGNN is set to 2.

  * •

PC-HGN [7] employs a sampling-based convolution. It designs an efficient
cross-relation convolution that allows message aggregations of a node from
different connected relation types simultaneously. We set the number of
kernels to 64 and pooling size to 2 as the best performance reported.

  * •

HL [36] uses the hypergraphs to represent complex relationships among the
objects, where where attribute values are regarded as hyperedges. The value of
the regularization factor is set to 0.1.

  * •

HGNN [9] performs convolution with a hypergraph Laplacian, which is further
approximated by truncated Chebyshev polynomials to handle the data correlation
during representation learning. The number of neighbors for generating
hyperedges is set to 10.

  * •

HGNN+ [30] utilizes the multi-modal/multi-type correlation from each
modality/type, and conducts hypergraph convolution in the spatial domain to
learn a general data representation for various task. The value of $k$ for
selecting $k$-hop neighbors to generate hyperedges is set to 1.

The shared parameter setup of the framework can be found in Table III. We
randomly split all datasets into train/validate/test with the ratio of
0.2/0.1/0.7, respectively. It is worth mentioning that the hyperparameters of
all baseline models are set to the optimal values reported in their original
papers. All models are trained with a fixed 100 epochs, using an early
stopping strategy when the performance on the validation set is not improved
for 30 consecutive epochs. All trainable parameters of the neural network are
initialized through Xavier [37] and optimized using Adam [38] with the
learning rate 2e-3. The dropout rate is set to be 0.3 [39]. The optimal
selection of hyperparameters of our model is further analyzed in the following
section.

TABLE III: Shared parameter setup

Name |  Setup
---|---
Optimizer | Adam [38]
Learning rate | 2e-3
Dropout rate | 0.3
Weight initializer | Xavier [37]
Epoch | 100
MLP | 1 linear layer, 1 output layer

TABLE IV: Performance of different models on node classification task.

Dataset |  Data split ratio | Hypergraph-based | Heterogeous-based | Homogeous-based
---|---|---|---|---
|  | LFH | HGNN | HGNN+ | HL | PC-HGN | HAN | R-HGNN | HGT | GCN | GAT | GraphSAGE | GraphSAINT
ACM | 10% | 0.818 | 0.557 | 0.655 | 0.471 | 0.768 | 0.551 | 0.512 | 0.517 | 0.505 | 0.547 | 0.548 | 0.512
30% | 0.871 | 0.801 | 0.851 | 0.544 | 0.845 | 0.842 | 0.847 | 0.814 | 0.762 | 0.767 | 0.763 | 0.771
50% | 0.882 | 0.804 | 0.852 | 0.596 | 0.849 | 0.842 | 0.854 | 0.828 | 0.846 | 0.853 | 0.851 | 0.839
70% | 0.905 | 0.840 | 0.882 | 0.607 | 0.872 | 0.866 | 0.864 | 0.857 | 0.853 | 0.858 | 0.858 | 0.856
IMDB | 10% | 0.491 | 0.410 | 0.432 | 0.415 | 0.407 | 0.372 | 0.406 | 0.341 | 0.322 | 0.384 | 0.351 | 0.361
30% | 0.539 | 0.442 | 0.463 | 0.424 | 0.459 | 0.398 | 0.415 | 0.388 | 0.384 | 0.429 | 0.388 | 0.409
50% | 0.558 | 0.463 | 0.446 | 0.437 | 0.461 | 0.427 | 0.438 | 0.424 | 0.438 | 0.458 | 0.427 | 0.422
70% | 0.571 | 0.494 | 0.492 | 0.459 | 0.488 | 0.464 | 0.487 | 0.435 | 0.445 | 0.488 | 0.454 | 0.460
DBLP | 10% | 0.861 | 0.669 | 0.819 | 0.503 | 0.818 | 0.812 | 0.672 | 0.598 | 0.535 | 0.574 | 0.543 | 0.597
30% | 0.874 | 0.739 | 0.842 | 0.575 | 0.838 | 0.837 | 0.783 | 0.632 | 0.699 | 0.693 | 0.696 | 0.676
50% | 0.888 | 0.767 | 0.866 | 0.634 | 0.861 | 0.855 | 0.853 | 0.724 | 0.755 | 0.775 | 0.771 | 0.776
70% | 0.917 | 0.791 | 0.892 | 0.656 | 0.884 | 0.878 | 0.863 | 0.782 | 0.772 | 0.780 | 0.778 | 0.778

TABLE V: Performance of different models on link prediction.

Model | ACM | DBLP | IMDB
---|---|---|---
Node2Vec [40] | 0.689 | 0.628 | 0.724
GAT [6] | 0.671 | 0.606 | 0.703
PC-HGN [7] | 0.833 | 0.782 | 0.842
HGNN+ [30] | 0.632 | 0.689 | 0.784
LFH | 0.861 | 0.809 | 0.874

###  V-B Performance Analysis

We evaluate the performance of our proposed framework on node classification
and link prediction tasks.

####  V-B1 Node Classification

In the node classification task, we split the original data into training data
and test data. The training data are then split into a range of [10%, 30%,
50%, 70%]. The strategy of generating the training dataset is the same as
[41], where each randomly selected node at least has one pairwise relation
with others. The best F1 results among all datasets are demonstrated in Table
IV, in which the highest scores are marked in bold. We compared three
different categories of graph learning models: homogeneous pairwise graph
learning models, heterogeneous pairwise graph learning models, and hypergraph
learning models. Our proposed LFH achieves performance gain over other
baseline models in all datasets by 2%-35.8%. Our model outperforms all
baseline models in all datasets by an average improvement of 12.5%. Compared
to four homogeneous pairwise GNN baselines GCN, GAT, GraphSAGE and GraphSAINT,
the average F1 score improvement of LFH for all datasets is 14.4%. Regarding
the cherry-picked heterogeneous pairwise graph learning models and hypergraph
learning models, our model achieved an average improvement of 9.7% and 13.7%,
respectively, compared to the best-recorded results in all datasets. It is
worth noting that HGNN+ [30] is the only hypergraph learning baseline that
utilizes pairwise information for hyperedge generation, resulting in better
performance compared to other hypergraph learning models, except for our
model. Furthermore, the training process of our model, which incorporates
type-specific hyperedge generation and attentive embedding updates that
exploits the heterogeneity of the graph, enables LFH to improve HGNN+ by an
average of 5.7% in all datasets. Additionally, compared to the latest
reporting heterogeneous learning model PC-HGN [7], the best results of our
model gains an average increase by 5.2%. For different split ratios of
training data, we observe that LFH still outperforms all baselines. When the
split ratios are 70%, 50%, 30% and 10%, respectively, our model outperforms
other models by 8.6%, 8.7%, 10.9% and 17.7% on average. The obtained result
demonstrates that the robustness and generalization capability of our model in
the informative case.

####  V-B2 Link Prediction

To investigate the effectiveness of our proposed framework, we also apply our
model to the link prediction task. In this task, a graph with a certain
faction of edges removed is given, and thus the objective is to predict these
missing edges. Following the experiment setting adopted in [40], 50% percent
of existing edges are randomly set hidden as positive samples while ensuring
that the residual graph after the edge being removed remains connected. To
generate negative examples, we randomly sample an equal number of node pairs
from the graph, which have no edge connecting them. Similar to the related
works [10, 42], we use a logistic regression classifier as a predictor for
link prediction. To get the representation of each edge, we use Hadamard
operator to compute the element-wise product for the linked node pairs. We
compare the performance of LFH with four baselines. Node2vec [40] is a classic
model for link prediction task. The other three baselines, GAT, PC-HGN, and
HGNN+, are the top-performing baselines in homogenous pairwise graph learning
models, heterogeneous pairwise graph learning models, and hypergraphs learning
models in node classification tasks. Experimental results are summarized in
Table V. LFH obtains performance gains of 16.8%, 18.8%, 2.9% and 14.6%,
compared with Node2vec, GAT, PC-HGN, and HGNN+ and outperforms four baseline
models in three datasets by an average improvement of 13.3%. It shows the
promising capability of our model to capture implicit data relations. It is
worth noting that HGNN+ that performs well in node classification does not
have comparable performance in link prediction. This is because the HGNN+
applied for link prediction task heavily relies on pairwise edges to generate
hyperedges, and thus a sharp decrease in the number of existing edges leads to
a deterioration in performance.

###  V-C Impact of Hyperedge Construction

In our proposed LFH, $\lambda$ and $\gamma$ are used as hyperparameters.
Specifically, $\lambda$ is the weight hyperparameter of the reconstruction
error and $\gamma$ is the norm hyperparameter to trade off $l_{1}\text{-}norm$
regularization norm and $l_{2}\text{-}norm$ regularization of the
reconstruction coefficient vector (See Eq. 6). Figure 5 and 6 illustrate the
overall performance of LFH influenced by $\lambda$ and $\gamma$. Figure 5
illustrates the influence of $\lambda$ for different datasets on F1 score. In
this figure, we fix $\gamma$ as 0.02, 0.2 and 0.2 while $\lambda$ ranges
between 0.0001 and 1. When $\lambda$ is small, the reconstruction loss of
$\lambda c^{k}\left(v^{m}_{i}\right)$ becomes trivial, and the reconstruction
coefficient vector p will not experience large adjustments throughout the
training phase, thus affecting the generation of hyperedges and deteriorating
the performance of LFH. In contrast, when $\lambda$ increases to around 0.2,
reconstruction loss of $\lambda c^{k}\left(v^{m}_{i}\right)$ accounts for the
appropriate proportion of the total loss, thus achieving the best performance
under this setup. When $\lambda$ continually increases to 20, F1 score drops
rapidly, which clearly demonstrates that the dominance of reconstruction loss
compromises the generalization ability of LFH.

On the other hand, Figure 6 describes the influence of $\gamma$. In this
figure, we fix $\lambda$ as 0.001, 0.01 and 0.1, while $\gamma$ ranges between
0.002 and 20. When $\gamma$ is very small, The decisive role of
$l_{1}\text{-}norm$ regularization enables our model to encircle fewer slave
nodes to generate hyperedges, which makes our model more sensitive to noise
and outliers. When $\gamma$ is large, the dominance of $l_{2}\text{-}norm$
regularization leads to the over-smoothing problem. With $\gamma$ ranging in
$\left[0.0001,1\right]$, the F1 score reaches the highest when $\gamma=0.2$
for all fixed value of $\lambda$ in all datasets, and decreases afterwards.
The result demonstrates that considering $l_{1}\text{-}norm$ regularization
norm and $l_{2}\text{-}norm$ regularization simultaneously can improve the
generalization ability of our model.

(a) ACM (b) DBLP (c) IMDB

Figure 5: Performance of LFH as a function of $\lambda$ for several values of
$\gamma$ for different datasets.

(a) ACM (b) DBLP (c) IMDB

Figure 6: Performance of LFH as a function of $\gamma$ for several values of
$\lambda$ for different datasets.

###  V-D Sensitivity Analysis

We study the impact of key parameters in LFH including the node embedding size
and the number of attention head as illustrated in Table VI and VII.

####  V-D1 Impact of the node embedding size

We test the impact of different node embedding sizes for all datasets with
data split ratio of 10%. As shown in Table VI, the performance of the model
increases at first and then starts to drop, reporting the best result with the
embedding size 256. Intuitively, larger dimension sizes of node embedding
bring in extra redundancies in the training process, thus causing unexpected
performance drops.

####  V-D2 Impact of the number of muti-head

We analyze the effect of the number of multi-head $K$. As observed in Table
VII, when the number of attention heads increases, there is typically an
improvement in the performance of our model. However, it is observed that the
performance of LFH improves only marginally afterwards while causing large
computational costs. When $K=4$, it is an optimal point to trade off the
performance and computational cost.

TABLE VI: Performance of LFH with different node embedding sizes.

Initial Node Embedding Size | ACM | DBLP | IMDB
---|---|---|---
128 | 0.800 | 0.818 | 0.456
256 | 0.818 | 0.861 | 0.491
512 | 0.789 | 0.766 | 0.424

TABLE VII: Performance of LFH with different number of attention heads.

Attention Head Number | ACM | DBLP | IMDB
---|---|---|---
2 | 0.801 | 0.850 | 0.472
4 | 0.818 | 0.861 | 0.491
8 | 0.819 | 0.862 | 0.491

Figure 7: Ablation study on pairwise fusion for different models.

(a) ACM (b) DBLP (c) IMDB

Figure 8: The performance change among different datasets with the change of
$\alpha$.

###  V-E Ablation Study

We study the impact of pairwise fusion in this part. We consider different
candidates in the pairwise fusion process, aiming to find the best match that
is capable of deriving high-quality features from the heterogeneous pairwise
graph as the initial node embedding. We analyze the importance of pairwise
fusion and implement different models introduced in Section V-A. Note that we
applied the same hyperparameters of all these models according to the best
performance reported in the respective papers and source codes. As shown in
Figure 7, using PC-HGN in pairwise fusion enlarges the performance gap over
other models in all datasets, with a performance gain of more than 5.4%-14.4%
compared to using GAT, a performance gain of more than 6.6%-18.0% compared to
using HGT, and an overall 11.5% performance gain on average compared to using
other models in pairwise fusion. This result demonstrates the initial node
embedding generated by PC-HGN helps our framework achieve the best results in
all three datasets.

###  V-F Impact of Hyperparameter $\alpha$

As defined in Eq. 6, the proposed united loss is a linear combination of the
reconstruction loss in dynamic heterogeneous hyperedges generation and the
supervised loss for the downstream task. We further study the impact of
$\alpha$ on the performance of LFH. In the experiment, the sensitivity of
$\alpha$ is explored to control the level of impact caused by the
reconstruction of each master when training the model. When the value of
$\alpha$ is close to 1, it means that the model is optimized towards the
reconstruction correctness. Fig. 8 reveals the change of F1 score along with
$\alpha$. With $\alpha$ ranging in $\left[0.001,0.5\right]$, the performance
first reaches to the top and then drops significantly, indicating that over-
reliance on reconstruction loss aggravates the generalization ability of our
model. We observe that F1 score reaches the highest when $\alpha=0.1$ in all
three datasets. The result demonstrates that uniting reconstruction loss in
the training process to some extent can help improve the model performance.

##  VI conclusion

In this paper, we propose a heterogenous hypergraph learning framework for
representation learning. This framework first generates the high-quality
initial node embedding using the designated pairwise fusion function, aiming
to exploit the pairwise graph information at most. Afterwards, the multi-type
hyperedges are constructed dynamically, forming the hypergraph together. The
embedding is then updated iteratively through type-specific attention, aiming
to encode the heterogeneous attributes into the embedding space. We conduct
comprehensive experiments on three widely-used public datasets and comparison
with eleven baseline methods to demonstrate the effectiveness of our proposed
framework. The results and analysis reveal that the proposed framework can
achieve new state-of-the-art performance on both node classification and link
prediction tasks.

## References

  * [1] Q. Guo, F. Zhuang, C. Qin, H. Zhu, X. Xie, H. Xiong, and Q. He, “A survey on knowledge graph-based recommender systems,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 8, pp. 3549–3568, 2022.
  * [2] Y. Rong, Y. Bian, T. Xu, W. Xie, Y. Wei, W. Huang, and J. Huang, “Self-supervised graph transformer on large-scale molecular data,” in Proceedings of Conference on Neural Information Processing Systems, pp. 12559–12571, 2020.
  * [3] T. Zhang, Y. Liu, Z. Shen, R. Xu, X. Chen, X. Huang, and X. Zheng, “FedRel: An adaptive federated relevance framework for spatial temporal graph learning,” arXiv preprint arXiv:2206.03420, 2022.
  * [4] H. Yuan, H. Yu, S. Gui, and S. Ji, “Explainability in graph neural networks: A taxonomic survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 5, pp. 5782–5799, 2023.
  * [5] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in Proceedings of the International Conference on Learning Representations, pp. 1–14, 2017.
  * [6] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in Proceedings of the International Conference on Learning Representations, pp. 1–12, 2018.
  * [7] T. Zhang, Y. Liu, Y. Yao, Y. Xia, X. Chen, X. Huang, and J. Jin, “Towards relation-centered pooling and convolution for heterogeneous graph learning networks,” arXiv preprint arXiv:2210.17142, 2022.
  * [8] C. Berge, Hypergraphs: combinatorics of finite sets, vol. 45.  Elsevier, 1984.
  * [9] Y. Feng, H. You, Z. Zhang, R. J. Ji, and Y. Gao, “Hypergraph neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, p. 3558–3565, 2019.
  * [10] H. Fan, F. Zhang, Y. Wei, Z. Li, C. Zou, Y. Gao, and Q. Dai, “Heterogeneous hypergraph variational autoencoder for link prediction,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 8, pp. 4125–4138, 2022.
  * [11] P. Li and O. Milenkovic, “Inhomogeneous hypergraph clustering with applications,” in Proceedings of Conference on Neural Information Processing Systems, pp. 1–11, 2017.
  * [12] M. Wang, X. Liu, and X. Wu, “Visual classification by $\ell_{1}$-hypergraph modeling,” IEEE Transactions on Knowledge and Data Engineering, vol. 27, no. 9, pp. 2564–2574, 2015.
  * [13] Y. Yan, J. Qin, J. Chen, L. Liu, F. Zhu, Y. Tai, and L. Shao, “Learning Multi-Granular Hypergraphs for Video-Based Person Re-Identification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2899–2908, 2020.
  * [14] F. Luo, L. Zhang, B. Du, and L. Zhang, “Dimensionality reduction with enhanced hybrid-graph discriminant learning for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 8, pp. 5336–5353, 2020.
  * [15] Y. Gao, M. Wang, H. Luan, J. Shen, S. Yan, and D. Tao, “Tag-based social image search with visual-text joint hypergraph learning,” in Proceedings of the ACM international conference on Multimedia, pp. 1517–1520, 2011.
  * [16] Y. Gao, Z. Zhang, H. Lin, X. Zhao, S. Du, and C. Zou, “Hypergraph learning: Methods and practices,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 5, pp. 2548–2566, 2022.
  * [17] W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive Representation Learning on Large Graphs,” in Proceedings of the International Conference on Neural Information Processing Systems, pp. 1025–1035, 2017.
  * [18] L. Li, Z. Gan, Y. Cheng, and J. Liu, “Relation-aware graph attention network for visual question answering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 10313–10322, 2019.
  * [19] B. Jiang, Z. Zhang, D. Lin, J. Tang, and B. Luo, “Semi-supervised learning with graph learning-convolutional networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11305–11312, 2019.
  * [20] H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna, “Graphsaint: Graph sampling based inductive learning method,” arXiv preprint arXiv:1907.04931, 2019.
  * [21] X. Wang, D. Bo, C. Shi, S. Fan, Y. Ye, and P. S. Yu, “A survey on heterogeneous graph embedding: Methods, techniques, applications and sources,” IEEE Transactions on Big Data, vol. 9, no. 2, pp. 415–436, 2023\.
  * [22] C. Zhang, D. Song, C. Huang, A. Swami, and N. V. Chawla, “Heterogeneous graph neural network,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 793–803, 2019.
  * [23] J. Zhao, X. Wang, C. Shi, B. Hu, G. Song, and Y. Ye, “Heterogeneous graph structure learning for graph neural networks,” Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4697–4705, 2021.
  * [24] X. Wang, H. Ji, C. Shi, B. Wang, Y. Ye, P. Cui, and P. S. Yu, “Heterogeneous Graph Attention Network,” in Proceedings of the World Wide Web Conference, p. 2022–2032, 2019.
  * [25] Z. Hu, Y. Dong, K. Wang, and Y. Sun, “Heterogeneous graph transformer,” in Proceedings of the World Wide Web Conference, pp. 2704–2710, 2020.
  * [26] L. Yu, L. Sun, B. Du, C. Liu, W. Lv, and H. Xiong, “Heterogeneous graph representation learning with relation awareness,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 6, pp. 5935–5947, 2023.
  * [27] Z. Zhang, H. Lin, and Y. Gao, “Dynamic Hypergraph Structure Learning,” in Proceedings of the International Joint Conference on Artificial Intelligence, pp. 3162–3169, 2018.
  * [28] J. Jiang, Y. Wei, Y. Feng, J. Cao, and Y. Gao, “Dynamic hypergraph neural networks,” in Proceedings of the International Joint Conference on Artificial Intelligence, pp. 2635–2641, 2019.
  * [29] D. Cai, M. Song, C. Sun, B. Zhang, S. Hong, and H. Li, “Hypergraph structure learning for hypergraph neural networks,” in Proceedings of the International Joint Conference on Artificial Intelligence, pp. 1923–1929, 2022\.
  * [30] Y. Gao, Y. Feng, S. Ji, and R. Ji, “HGNN+: General hypergraph neural networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 3, pp. 3181–3199, 2023.
  * [31] Y. Huang, Q. Liu, F. Lv, Y. Gong, and D. N. Metaxas, “Unsupervised image categorization by hypergraph partition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 6, pp. 1266–1273, 2011.
  * [32] T. Jin, Z. Yu, Y. Gao, S. Gao, X. Sun, and C. Li, “Robust l2- hypergraph and its applications,” Information Sciences, vol. 501, pp. 708–723, 2019.
  * [33] Y. Zhu, Z. Guan, S. Tan, H. Liu, D. Cai, and X. He, “Heterogeneous hypergraph embedding for document recommendation,” Neurocomputing, vol. 216, pp. 150–162, 2016.
  * [34] J. Yu, D. Tao, and M. Wang, “Adaptive hypergraph learning and its application in image classification,” IEEE Transactions on Image Processing, vol. 21, no. 7, pp. 3262–3272, 2012.
  * [35] Y. Gao, M. Wang, D. Tao, R. Ji, and Q. Dai, “3-D object retrieval and recognition with hypergraph analysis,” IEEE Transactions on Image Processing, vol. 21, no. 9, pp. 4290–4303, 2012.
  * [36] D. Zhou, J. Huang, and B. Schölkopf, “Learning with hypergraphs: Clustering, classification, and embedding,” in Proceedings of Conference on Neural Information Processing Systems, pp. 1601–1608, 2006.
  * [37] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 249–256, 2010.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proceedings of the International Conference for Learning Representations, pp. 1269–1272, 2015.
  * [39] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” The Journal of Machine Learning Research, vol. 15, pp. 1929–1958, 2014.
  * [40] A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864, 2016.
  * [41] H. Gui, J. Liu, F. Tao, M. Jiang, B. Norick, L. Kaplan, and J. Han, “Embedding learning with events in heterogeneous information networks,” IEEE Transactions on Knowledge and Data Engineering, vol. 29, no. 11, pp. 2428–2441, 2017.
  * [42] M. Nickel and D. Kiela, “Poincaré embeddings for learning hierarchical representations,” in Proceedings of Conference on Neural Information Processing Systems, pp. 6341–6350, 2017.