Tensorized Hypergraph Neural Networks
Hypergraph neural networks (HGNN) have recently become attractive and
received significant attention due to their excellent performance in various
domains. However, most existing HGNNs rely on first-order approximations of
hypergraph connectivity patterns, which ignores important high-order
information. To address this issue, we propose a novel adjacency-tensor-based
\textbf{T}ensorized \textbf{H}ypergraph \textbf{N}eural \textbf{N}etwork
(THNN). THNN is a faithful hypergraph modeling framework through high-order
outer product feature message passing and is a natural tensor extension of the
adjacency-matrix-based graph neural networks. The proposed THNN is equivalent
to a high-order polynomial regression scheme, which enables THNN with the
ability to efficiently extract high-order information from uniform hypergraphs.
Moreover, in consideration of the exponential complexity of directly processing
high-order outer product features, we propose using a partially symmetric CP
decomposition approach to reduce model complexity to a linear degree.
Additionally, we propose two simple yet effective extensions of our method for
non-uniform hypergraphs commonly found in real-world applications. Results from
experiments on two widely used {hypergraph datasets for 3-D visual object
classification} show the model's promising performance.
Maolin Wang, Yaoming Zhen, Yu Pan, Yao Zhao, Chenyi Zhuang, Zenglin Xu, Ruocheng Guo, Xiangyu Zhao
# Tensorized Hypergraph Neural Networks

###### Abstract

Hypergraph neural networks (HGNN) have recently become attractive and received significant attention due to their excellent performance in various domains. However, most existing HGNNs rely on first-order approximations of hypergraph connectivity patterns, which ignores important high-order information. To address this issue, we propose a novel adjacency-tensor-based Tensorized Hypergraph Neural Network (THNN). THNN is a faithful hypergraph modeling framework through high-order outer product feature message passing and is a natural tensor extension of the adjacency-matrix-based graph neural networks. The proposed THNN is equivalent to an high-order polynomial regression scheme, which enable THNN with the ability to efficiently extract high-order information from uniform hypergraphs. Moreover, in consideration of the exponential complexity of directly processing high-order outer product features, we propose using a partially symmetric CP decomposition approach to reduce model complexity to a linear degree. Additionally, we propose two simple yet effective extensions of our method for non-uniform hypergraphs commonly found in real-world applications. Results from experiments on two widely used hypergraph datasets for 3-D visual object classification show the promising performance of the proposed THNN.

 Hypergraph, graph neural networks, tensor neural networks, tensor decomposition

## I Introduction

The rapid development of graph neural networks (GNNs, [1, 2, 3]) greatly benefits various crucial research areas due to their extraordinary performance. Generally, a conventional GNN only allows objects to have pairwise interaction. However, in many real-world applications, the interactions among objects can go beyond pairwise interactions and involve higher-order relationships. For example, in brain connectivity networks [4, 5], multiple brain regions often work together in a neurological manner to accomplish certain functional tasks. To faithfully characterize such connections, pairwise modeling in graph structure is inadequate, and it is necessary to incorporate high-order interacting information across brain regions. To articulate the correlation among multiple regions, a hypergraph structure [5, 6, 7] can be created with each vertex as a brain region and each hyperedge representing the interactions among several brain regions.

As discussed in [8], there is a clear difference between the pairwise relationship and the high order relationship of multiple objects. The capacity of graph structures is limited as they can only describe pairwise relationships. Compared to graphs, a hypergraph provides significant advantages in modeling the high-order relationships among multiple objects in real-world data [8]. For example, in the case of multi-agent trajectory prediction [9], adopting the multiscale hypergraph can extract the interactions among groups of varying sizes and performs much better than prior graph-based methods which can solely describe pairwise interactions. Hypergraphs have also recently been widely used in a variety of other data mining tasks such as node classification [10], link prediction [11], community detection [12], retrieval [13], multi-label classification [14, 15], 3D object classification [16, 17], tracking [18], point cloud matching [19], and clustering [7].

In these applications, the majority of hypergraph neural network architectures are based on the Chebyshev formula for hypergraph Laplacians proposed by HGNN [17]. These neural hypergraph operators can be seen as constructing a weighted graph and thus can utilize the off-the-shelf graph learning models (e.g., GCN). **However, most of these methods are incapable of learning higher-order information since they only make use of the first-order approximation**, e.g., the clique expansion [7] of a hypergraph. In order to better characterize high-order information in hypergraphs, it is natural to model the high-order interaction information of a hypergraph through high-order representation (like outer product). Some studies [12, 20, 21] have revealed the great success of tensor representation (like adjacency tensor [12] representation) in hypergraph modeling. **However, a general tensor based hypergraph neural network, which conduct a high-order information message passing procedure, has not yet been developed.**

Motivated by the two aforementioned observations, in this

Fig. 1: An example of Graphs and Hypergraphs. A graph is typically represented by an adjacency matrix while a hypergraph is always described by an incidence matrix.
work, we propose a tensor based **T**ensorized **H**ypergraph **N**eural **N**etwork (THNN) to extend the adjacency matrix based graph neural networks into an adjacency tensor based framework. THNN has high expressiveness due to its intrinsic similarity to a high-order outer product feature aggregation scheme [22, 23, 24], which can capture intra-feature and inter-feature dynamics in multilinear interaction information modeling. In other words, the intrinsic multilinear mathematical architecture of THNN is effective and natural in modeling high-order information, resulting in a more accurate extraction of high-order interactions.

Furthermore, because adjacency tensors can only be used to represent uniform hypergraphs, the straightforward THNN is incapable of handling the widely existing non-uniform hypergraphs. Therefore, we propose two novel solutions: (1) adding a global node and (2) multi-uniform processing. To evaluate the performance of the proposed THNN framework, experiments on two 3-D visual object recognition datasets are performed. The experimental results show that the proposed THNN model achieves state-of-the-art performance. In summary, our major contributions are as follows:

* We propose, to the best of our knowledge, the first hypergraph neural network based on adjacency tensor that comes with a message passing mechanism capturing high-order interactions in hypergraphs. Previous hypergraph neural networks are mostly based on the first-order approximation in higher-order learning.
* Given the fact that the naive outer product based model of high-order information suffers from exponential time/space complexity, we propose to utilize partially symmetric CP decomposition to reduce time/space complexity from exponential to linear.
* To handle non-uniform hypergraphs, we propose two simple yet effective solutions, i.e., adding a global node and multi-uniform processing, to overcome the limitation that the straightforward THNN of adjacency tensor methods can only be used to model and process uniform hypergraphs.
* We compare the proposed THNN with the state-of-the-art baselines on two widely used hypergraph datasets for 3-D visual object classification under both uniform and non-uniform settings. Empirical results show that the proposed THNN achieves state-of-the-art performance. We also conducted comprehensive experiments for hyperparameter analysis to provide insights into the behavior of the proposed model.

## II Preliminaries and Background

### _Graph and Hypergraph_

A graph can be denoted by \(G=(V,E)\), where \(V\) is the set of vertices and \(E\) is a set of paired vertices, or edges. A graph can be represented by its adjacency matrix \(\mathbf{A}\in\{0,1\}^{|V|\times|V|}\), where \(|\cdot|\) denotes the set cardinality. The entries of the matrix \(\mathbf{A}\) indicate whether two vertices in the graph are adjacent. More specifically, \(\mathbf{A}_{i,j}=1\) if \(\{v_{i},v_{j}\}\in E\) and 0 otherwise.

A hypergraph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) is a generalization of a graph in which any number of vertices can be joined in one edge. \(\mathcal{V}\) is the set of vertices, and \(\mathcal{E}\) is a set of vertex sets, a.k.a hyperedges. A hypergraph (undirected) is always described by an incidence \(\mathbf{H}\in\{0,1\}^{|\mathcal{V}|\times|\mathcal{E}|}\), where \(|\mathcal{V}|\) is the number of vertices and \(|\mathcal{E}|\) is the number of hyper-edges. Specifically, \(\mathbf{H}_{i,j}=1\), if \(v_{i}\in e_{j}\) and 0 otherwise. The Illustration of graphs, hypergraphs, adjacency matrix and incidence matrix is shown in Figure 1.

Hypergraphs can be approximated by graphs via its clique expansion [7]. As shown in Figure 3, the clique expansion approximates the original hypergraph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) via a graph \(G_{clique}=(\mathcal{V},E_{clique})\), which reduces each hyperedge \(e\in\mathcal{E}\) into a clique in \(G_{clique}\). However, the clique expansion will lead to information loss [25]. The original hypergraph can not be recovered according to the adjacency matrix of clique expansion, as the hyper-dependency and high order relationship collapses into linearity [25].

Another important way to represent hypergraphs is by **Adjacency Tensor**[12]. As shown in Figure 2, adjacency tensor can represent uniform hypergraph where all hyperedges share the same size. The \(m\)-uniform hypergraph means that the sizes of all hyperedges are \(m\). For an \(m\)-uniform hypergraph, the adjacency tensor is defined as the \(m\)-order tensor \(\mathcal{A}\in\{0,1\}^{n\times\ldots\times n}\) with the entry \(\mathcal{A}_{i_{1},\ldots,i_{m}}=1\) if \(\{v_{i_{1}},\ldots,v_{i_{m}}\}\in\mathcal{E}\) and 0 otherwise.

### _Tensor Contraction_

Tensor contraction [26] means that two tensors are contracted into one tensor along their associated pairs of indices. Given two tensors \(\mathcal{A}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}\) and \(\mathcal{B}\in\mathbb{R}^{J_{1}\times J_{2}\times\cdots\times J_{M}}\), with some common modes, \(I_{n_{1}}=J_{m_{1}}\), \(\cdots\)\(I_{n_{B}}=J_{m_{S}}\), the tensor contraction \(\mathcal{A}\times_{(m_{1},m_{2}\cdots,m_{S})}^{(n_{1},n_{2}\cdots,n_{S})} \mathcal{B}\) yields a \((N+M-2S)\)-order tensor \(\mathcal{C}\). Tensor contraction can be formulated as:

\[\mathcal{C}=\mathcal{A}\times_{(m_{1},j_{m_{2}},\ldots,j_{m_{S}})}^{(i_{n_{1} },i_{n_{2}},\ldots i_{N})}\mathcal{B}=\sum_{i_{1},i_{2},\cdots i_{N}}\mathcal{ A}_{i_{1},i_{2},\cdots i_{n_{S}},*}\quad\mathcal{B}_{s,i_{1},i_{2},\cdots i_{n_{S}}} \tag{1}\]

Fig. 2: An example of adjacency tensor of a 3-uniform hypergraph. In this example, the adjacency tensor of hypergraph \(\mathcal{G}\) is defined as the \(3\)-order tensor \(\mathcal{A}\in\{0,1\}^{7\times 7\times 7}\) with the entry \(\mathcal{A}_{v_{i},v_{j},v_{k}}=1\) if \(\{v_{i},v_{j},v_{k}\}\in\mathcal{E}\) and 0 otherwise.
For example, if \(\mathcal{A}\in\mathbb{R}^{I_{1}\times I_{2}\times I_{3}\times I_{4}\times I_{5}}\), \(\mathcal{B}\in\mathbb{R}^{J_{1}\times J_{2}\times J_{3}\times J_{4}}\), \(N=5\), \(M=4\), \(S=2\), \(n_{1}=3\), \(n_{2}=5\), \(m_{1}=3\), and \(m_{2}=2\), then the entries of \(\mathcal{C}=\mathcal{A}\times_{(3,2)}^{(3,5)}\mathcal{B}\) are

\[\mathcal{C}_{i_{1},i_{2},i_{4},j_{1},j_{4}}=\sum_{i_{3},i_{5}} \mathcal{A}_{i_{1},i_{2},i_{3},i_{4},i_{5}}\mathcal{B}_{j_{1},i_{5},i_{3},j_{4 }}. \tag{2}\]

The well known Mode-\(N\) Product is a special case of Tensor Contraction. Given a tensor \(\mathcal{A}\in\mathbb{R}^{I_{1}\times I_{2}\times\cdots\times I_{N}}\) and a matrix \(\mathbf{B}\in\mathbb{R}^{J_{1}\times J_{2}}\). If \(J_{2}=I_{n}\), then

\[\mathcal{C}=\mathcal{A}\times_{(2)}^{(n)}\mathbf{B}=\mathcal{A} \times_{n}\mathbf{B}. \tag{3}\]

### _Graph Convolutional Neural Networks_

Pairwise message passing is the most important component of the widely used Graph Neural Networks (GNNs). It ensures that nodes in a graph are able to repeatedly improve their representations by exchanging information with their neighbors. One of the most classical design of GNNs is the Graph Convolutional Network (GCN) [2]. Given the adjacency matrix \(\mathbf{A}\) of graph \(G\), a GCN layer can be represented as

\[\mathbf{X}^{(l+1)}=\sigma\left(\tilde{\mathbf{D}}^{-\frac{1}{2} }\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-\frac{1}{2}}\mathbf{X}^{(l)}\mathbf{ \Theta}^{(l)}\right), \tag{4}\]

with \(\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}\), where \(\mathbf{I}\) is the identity matrix and \(\tilde{\mathbf{D}}\) is the diagonal node degree matrix of \(\tilde{\mathbf{A}}\) where \(\tilde{\mathbf{D}}_{ii}=\sum_{j}\tilde{\mathbf{A}}_{ij}\). \(\mathbf{\Theta}^{(l)}\) is the learnable parameters in the \(l\)-th layer. \(\mathbf{X}^{(l)}\) is the feature matrix in the \(l\)-th layer. By analyzing the computation of Eq. (4), we can represent the embedding of node \(v_{i}\) in \(l+1\)-th layer \(x_{v_{i}}^{(l+1)}\) with an aggregation function [27] as the following:

\[x_{v_{i}}^{(l+1)}=\sigma\bigg{(}\sum_{v_{j}\in N_{i}}\frac{1}{ \sqrt{d_{i}}\sqrt{d_{j}}}x_{v_{j}}^{(l)}\mathbf{\Theta}^{(l)}\bigg{)}, \tag{5}\]

where \(N_{i}\) is the set of neighbors of node \(v_{i}\) and \(d_{i}=\mathbf{\tilde{D}}_{ii}\). We observe that pairwise message passing in GCN is achieved via a weighted sum (weight is normalized via degree of pairs) of neighbors.

### _CP Decomposition_

CANDECOMP/PARAFAC (CP) Decomposition [28] factorizes a higher-order tensor into a sum of several rank-1 tensor components. For instance, given an order-\(N\) tensor \(\mathcal{X}\in\mathbb{R}^{I_{1}\times I_{2}\ldots I_{N}}\), each of its elements in the CP decomposition form can be decomposed as:

\[x_{i_{1},i_{2},\ldots,i_{N}}\approx\sum_{r=1}^{R}\prod_{n=1}^{N }a_{i_{n},r}^{(n)},\] \[\mathcal{X}\approx\mathcal{I}\times_{1}\mathbf{A}^{(1)}\times_{2} \mathbf{A}^{(2)}\cdots\times_{N}\mathbf{A}^{(N)}, \tag{6}\]

where \(R\) denotes the rank, \(\mathcal{I}\in\mathbb{R}^{R\times R\cdots\times R}\) is identity tensor where all diagnol elements \(\mathcal{I}_{i,i\ldots i}=1\) and \(\mathbf{A}^{(1)},...,\mathbf{A}^{(N)}\in\mathbb{R}^{I_{n}\times R}\) denote a series of factor matrices.

## III Methods

In this section, we first analyze the widely adopted hypergraph neural network - HGNN [17], which uses first-order information for hypergraph representation learning. Next, we propose and analyze tensorized hypergraph neural network based on adjacency tensors. Since the straightforward THNN cannot handle more common non-uniform hypergraphs, we introduce two simple yet effective solutions: global node adding and multi-uniform processing.

### _Analysis of Hypergraph Neural Networks_

Feng et al. [17] develop the classical Hypergraph Neural Networks which use truncated Chebyshev formula as hypergraph Laplacians. Given the incidence matrix \(\mathbf{H}\in\mathbb{R}^{|V|\times|\mathcal{E}|}\) of the hypergraph \(\mathcal{G}\), its operator can be written as

\[\mathbf{X}^{(l+1)}=\sigma\left(\mathbf{D}_{(v)}^{-1/2}\mathbf{H} \mathbf{W}\mathbf{D}_{(e)}^{-1}\mathbf{H}^{\top}\mathbf{D}_{(v)}^{-1/2} \mathbf{X}^{(l)}\mathbf{\Theta}^{(l)}\right), \tag{7}\]

where \(\mathbf{W}\in\mathbb{R}^{|\mathcal{E}|\times|\mathcal{E}|}\) is a diagonal matrix to be learned and \(\mathbf{\Theta}^{(l)}\) is a learnable matrix in layer \(l\). \(\mathbf{D}_{(v)ii}=\sum_{j=1}^{|\mathcal{E}|}W_{jj}H_{ij}\) and \(\mathbf{D}_{(e)jj}=\sum_{i=1}^{|\mathcal{V}|}H_{ij}\) are diagonal degree matrices of vertices and edges, respectively. These methods can be viewed as applying clique expansion graph that all the edges in a same clique sharing the same learnable weight to approximate the hypergraph. Each hyperedge of size \(s\) is approximated by a weighted \(s\)-clique. By analyzing the computation of Eq. (7), as shown in Figure 5, we denote the embedding of node \(v_{i}\) in the \(l+1\)-th layer by \(x_{v_{i}}^{(l+1)}\), which is computed by the following aggregation function

\[x_{v_{k}}^{(l+1)}=\sigma\bigg{(}\frac{1}{\sqrt{d_{v_{k}}}}\sum_{e_{j},v_{k}\in e _{j}}\frac{1}{d_{e_{j}}}\mathbf{W}_{jj}\sum_{v_{i},v_{i}\in e_{j}}\frac{1}{ \sqrt{d_{v_{i}}}}x_{v_{i}}^{l}\mathbf{\Theta}^{(l)}\bigg{)}. \tag{8}\]

Fig. 4: Illustration of concatenating 1 [22]. Every circle corresponds to an element in a vector or tensor and \(\circ\) indicates the outer product. We can concatenate a 1 to each vector, and then the outer product of vectors will help to introduce the lower order dynamics.

Fig. 3: Illustration of Clique Expansion of a hypergraph. Clique expansion is important for the processing and utilization of hypergraphs. However, according to the adjacency matrix of clique expansion, the original hypergraph cannot be fully retrieved, showing that some information is lost during the expansion procedure.
This approach tackles information aggregation by a weighted summation of the linearly processed (via \(\mathbf{\Theta}^{(l)}\)) node embeddings of neighbors in the weighted clique expansion graph. However, it is insufficient for higher-order information extraction as only the first-order linear information is considered in Eq. (8).

### _Tensorized Hypergraph Neural Network_

Eq. (8) has revealed that classical Hypergraph Neural Networks approximate high-order information via the first-order summation. However, higher-order information better characterizes co-occurrence relationship in hypergraph.

In order to characterize the influence of other nodes in the same hyperedge on its high-order interaction information, for a node in a hypergraph, the most intuitive method is to use the outer product pooling [22] of the feature vectors of its neighbors. For example, for a node \(v_{i}\) of a hyperedge \(\{v_{i},v_{j},v_{k}\}\in\mathcal{E}\) in a third-order hypergraph \(\mathcal{G}\), the message of the hyperedge \(\{v_{i},v_{j},v_{k}\}\) to node \(v_{i}\) is \(x_{v_{j}}\circ x_{v_{k}}\in\mathbb{R}^{I_{in}\times I_{in}}\). Similar to other graph neural networks, a trainable weight tensor can be used to process and align features, followed by the aggregation of all hyperedges. Then, the information aggregation of node \(v_{i}\) embedding can be represented as follows:

\[x_{v_{i}}^{(l+1)}=\sum_{(j,k)\in N_{i}}x_{v_{j}}^{l}\circ x_{v_{k}}^{l}\times (2,3)\,\mathcal{W}, \tag{9}\]

where \(\mathcal{W}\in\mathbb{R}^{I_{in}\times I_{in}\times I_{out}}\) is the weight tensor., where \(N_{i}\) is the set of neighbor pairs of the node \(v_{i}\) and \(\circ\) indicates the outer product. Eq.(9) formulate the basic framework of the hypergraph neural network that utilizes high-order polynomial information directly. Such formulation also reminds us of the polynomial regression scheme [29, 30], which are highly recognized successful techniques for high-order interaction information extracting (such as multi-modality analysis [22, 23, 24] and quantum data processing [31]).

Similar to the representation equivalence between the aggregation scheme in Eq.(4) and adjacency matrix formulation in Eq.(5), the adjacency tensor can also be used to describe Eq.(9). The outer product feature aggregation can be reformulated as,

\[\mathbf{X}^{(l+1)}=\sigma\left((\mathcal{A}\times_{2}\mathbf{X}^{(l)}\times_{ 3}\mathbf{X}^{(l)})\times^{(2,3)}_{(1,2)}\mathcal{W}\right), \tag{10}\]

Similar to GCN, simple graph convolution layer without feature normalization can result in numerical instabilities because directly applying convolution layer changes the scale of feature vectors. As a consequence of this, there is a need for an appropriate level of degree-normalization [2]. Similar to the degree-normalized adjacency matrix in Eq. (4), we adopt the well-known normalizing adjacency tensor extension [32],

\[\tilde{\mathcal{A}}_{i_{1}\ldots i_{k}}=\begin{cases}\frac{1}{(k-1)!}\prod_{1 \leqslant j\leqslant k}\frac{1}{\sqrt[d_{ij}]}&\text{if }\{v_{i_{1}},\ldots,v_{i_{k}}\}\in E\\ 0&\text{otherwise}\end{cases}. \tag{11}\]

Thus, Eq.(10) can be then reformulated as,

\[\mathbf{X}^{(l+1)}=\sigma\left((\tilde{\mathcal{A}}\times_{2}\mathbf{X}^{(l) }\times_{3}\mathbf{X}^{(l)})\times^{(2,3)}_{(1,2)}\mathcal{W}\right), \tag{12}\]

Since the size of the parameter tensor will grow exponentially with the order number, such extremely large storage and computational complexity is unacceptable. This phenomenon is known as the curse of dimensions [33], and proper tensor decomposition format can effectively solve this problem. So, we decompose the weight tensor \(\mathcal{W}\in\mathbb{R}^{I_{in}\times I_{in}\times I_{out}}\) into

Fig. 5: Illustration of an example of Hypergraph Neural Networks in Eq. (7) and Eq. (8). Classical hypergraph neural network models can be considered as learning a mapped weighted graph. The message passing procedure is still processed via the weighted sum of neighbors’ embeddings. For instance, the weight of \(x_{v_{3}}^{(l)}\) for generating \(x_{v_{1}}^{(l+1)}\) is \(\left(\frac{1}{d_{e_{1}}}\mathbf{W}_{11}+\frac{1}{d_{e_{2}}}\mathbf{W}_{22} \right)\frac{1}{\sqrt{d_{v_{3}}}}\).

Fig. 6: Illustration of THNN. THNN tries to pass the high-order interactions of neighbors in different hyperedges. The information of interactions is computed via outer product and is processed via tensor contractions.
the following partially symmetric CP decomposition [28, 34] structure with the rank \(R\):

\[\mathcal{W}=\mathcal{I}\times_{1}\mathbf{\Theta}^{(l)}\times_{2}\mathbf{\Theta}^{(l)} \times_{3}\mathbf{Q}^{(l)}.\]

Using partially symmetric constraints is motivated by the assumption that the same combination of nodes in **undirected** hypergraph should result in equal output features after outer product fusion. For example, as for node pair \(v_{j}\) and \(v_{k}\), the weight should hold \(\mathcal{W}\times_{1}x_{v_{j}}\times_{2}x_{v_{k}}=\mathcal{W}\times_{1}x_{v_{k }}\times_{2}x_{v_{j}}\). Therefore, partially symmetry constraints can address this assumption and reduce the number of parameters. The final low-rank aggregation scheme can be represented as follows

\[\mathbf{X}^{(l+1)}=\] \[\sigma\left(\left(\left(\tilde{\mathcal{A}}\times_{2}(\mathbf{X} ^{(l)}\mathbf{\Theta}^{(l)})\times_{3}(\mathbf{X}^{(l)}\mathbf{\Theta}^{(l)})\right) \times^{(2,3)}_{(1,2)}\mathcal{I}\right)\mathbf{Q}^{(l)T}\right), \tag{13}\]

where \(\tilde{\mathcal{A}}\in\mathbb{R}^{|\mathcal{V}|\times|\mathcal{V}|}\), \(\mathbf{X}\in\mathbb{R}^{|\mathcal{V}|\times I_{in}}\), \(\mathcal{I}\in\mathbb{R}^{R\times R\times R}\) is the identity tensor, \(\mathbf{\Theta}^{(l)}\in\mathbb{R}^{I_{in}\times R}\), and \(\mathbf{Q}^{(l)}\in\mathbb{R}^{I_{out}\times R}\). \(\mathbf{\Theta}^{(l)}\) and \(\mathbf{Q}^{(l)}\) are the learnable weights in the \(l\)-th layer. \(I_{in}\) is the input feature dimension number, \(I_{out}\) is the output dimensionality and \(R\) is the number of rank. We define the family of such hypergraph neural networks as **T**ensorized **H**ypergraph **N**eural **N**etworks (**THNN**).

### _Architecture Analysis and Model Details_

Traditional GCN speeds up their computation via sparse matrix operation in Pytorch1 or Tensorflow2. However, sparse tensor operations have not been supported well in common differential programming libraries. The above extension would suffer from high computational space cost of huge adjacency tensor, especially when the order is high. After fully optimizing the order of calculations, we can rewrite the THNN in

Footnote 1: [https://pytorch.org/](https://pytorch.org/)

Footnote 2: [https://www.tensorflow.org/](https://www.tensorflow.org/)

\[{x_{v_{i}}}^{(l+1)}=\] \[\sigma\left(\sum_{(j,k)\in N_{i}}\frac{1}{2}\mathbf{Q}^{(l)}\frac{ 1}{\sqrt[3]{d_{i}}\sqrt[3]{d_{j}}\sqrt[3]{d_{k}}}\left(\mathbf{\Theta}^{(l)\top}x _{v_{j}}^{(l)}\right)\star\left(\mathbf{\Theta}^{(l)\top}x_{v_{k}}^{(l)}\right) \right), \tag{14}\]

where \(N_{i}\) is the set of neighbor pairs of the node \(v_{i}\), and \(\star\) is element-wise dot product. And considering that the low order information can also be very important in some cases, we concatenate a scalar \(1\) in feature vectors to generate lower-order dynamics. Such a strategy could help THNN with the low order information modeling. In detail, Eq. (14) considers more on the 2nd-order interactions and ignores some 1st-order information. As for the 4-uniform situation, the 3rd-order interactions would be considered more. As shown in Figure 4, if we concatenate the original feature vector with a scalar \(1\), such preference can be alleviated.

As dot product of many vectors would lead the numerical insatiability empirically, we add a new activation function \(\sigma^{{}^{\prime}}\) in the original architectures. We evaluated common activation functions and used \(Tanh(\cdot)\) in experiments. A discussion of choosing \(\sigma^{{}^{\prime}}\) is in Section IV-J. The final expression of THNN for uniform Hyper-Graph is represented as follows

\[{x_{v_{i}}}^{(l+1)}=\sigma(\sum_{(j,k)\in N_{i}}(\mathbf{Q}^{(l)} \sigma^{{}^{\prime}}\frac{1}{2}\frac{1}{\sqrt[3]{d_{i}}\sqrt[3]{d_{j}}\sqrt[3] {d_{k}}}\] \[\left(\mathbf{\Theta}^{(l)\top}\left[\begin{array}{c}x_{v_{j}}^{l} \\ 1\end{array}\right]\right)\star\left(\mathbf{\Theta}^{(l)\top}\left[\begin{array}{c }x_{v_{k}}^{l}\\ 1\end{array}\right]\right)))). \tag{15}\]

The whole procedure of THNN is shown in Figure 6. Generalizations of order \(N\) THNN are similarly generated using the same approach, and formulas for the general case are provided in Section III-D and Section III-E in the Appendix.

### _Generalizing Third uniform THNN to the \(N\)-Order_

In Section III-C of the paper, we illustrate the third uniform THNN. Here we derive the more general \(N\)-th order form. Then, the outer product information aggregation of node \(v_{i}\) embedding can be represented as follows:

\[{x_{v_{j_{1}}}}=\sum_{(j_{1},j_{2},\cdots,j_{N-1})\in N_{i}}({x_{v_{j_{1}}}} \circ\cdots\circ{x_{v_{j_{N-1}}}})\times^{(2,3,\cdots,N)}_{(1,2,\cdots,N-1)} \mathcal{W} \tag{16}\]

where \(\mathcal{W}\in\mathbb{R}^{I_{in}\times I_{in}\cdots\times I_{out}}\) is the weight tensor. And we can reformulate Eq. (9) into the adjacency tensor contraction format,

\[\mathbf{X}^{(l+1)}=\sigma\left((\tilde{\mathcal{A}}\times_{2}\mathbf{X}^{(l) }\cdots\times_{N}\mathbf{X}^{(l)})\times^{(2,3,\cdots,N)}_{(1,2,\cdots,N-1)} \mathcal{W}\right), \tag{17}\]

Then, we decompose the weight tensor \(\mathcal{W}\in\mathbb{R}^{I_{in}\times I_{in}\times I_{out}}\) into the following partially symmetric CP decomposition [28, 34] structure with the rank \(R\):

\[\mathcal{W}=\mathcal{I}\times_{1}\mathbf{\Theta}^{(l)}\times_{2}\mathbf{\Theta}^{(l)} \cdots\times_{N-1}\mathbf{\Theta}^{(l)}\times_{N}\mathbf{Q}^{(l)T}.\]

 (18)

We could also rewrite the THNN with the element-wise dot aggregation function form as,

\[{x_{v_{i}}}^{(l+1)}=\sum_{(j_{1},j_{2},\cdots,j_{N-1})\in N_{i}}( \mathbf{Q}^{(l)}(\frac{1}{(N-1)!}\frac{1}{\Pi_{l}\sqrt[3]{d_{j_{1}}}}\] \[\left(\mathbf{\Theta}^{(l)\top}\left[\begin{array}{c}x_{v_{j_{1}}} ^{l}\\ 1\end{array}\right]\right)\star\cdots\left(\mathbf{\Theta}^{(l)\top}\left[ \begin{array}{c}x_{v_{j_{N-1}}}^{l}\\ 1\end{array}\right]\right))). \tag{19}\]

Empirically, dot product of many vectors would lead the numerical insatiability, therefore, we add activation functions in the original structures. The final expression of THNN for \(N\)-th uniform hypergraph is represented as,

\[{x_{v_{i}}}^{(l+1)}=\sigma(\sum_{(j_{1},j_{2},\cdots,j_{N-1})\in N _{i}}(\mathbf{Q}^{(l)}\textit{Tanh}(\frac{1}{(N-1)!}\frac{1}{\Pi_{l}\sqrt[N]{d_ {j_{l}}}}\] \[\left(\mathbf{\Theta}^{(l)\top}\left[\begin{array}{c}x_{v_{j_{1}}}^{l} \\ 1\end{array}\right]\right)\star\cdots\left(\mathbf{\Theta}^{(l)\top}\left[\begin{array} []{c}x_{v_{j_{N-1}}}^{l}\\ 1\end{array}\right]\right)))). \tag{20}\]
### _Detailed Complexity Analysis of General THNN_

In this section, we formalize time and space complexity of general THNN. The outer product based high order message passing scheme in Eq. (16) suffers from a very high space and time complexity. The parameter space complexity is \(\mathcal{O}(I_{in}^{N-1}I_{out}L)\) and the computational time complexity is \(\mathcal{O}(|\mathcal{V}|^{N}(I_{in}^{N-1}+I_{in}^{N-1}I_{out})L)\). \(L\) denoted the number of layers. After exploiting the symmetry property of the feature tensor via a partially CP decomposition format, as stated in Eq. (18), the parameter space complexity and computational time complexity can be reduced into \(\mathcal{O}((I_{in}+I_{out}+1)RL)\) and \(\mathcal{O}(|\mathcal{V}|^{N}R+|\mathcal{V}|R^{N}+|\mathcal{V}|RI_{out}+I_{ in}I_{out}R)\) respectively. If the computation is fully optimized in dot product form in Eq. (20), the time complexity will be reduced to \(\mathcal{O}(|\mathcal{A}|(I_{in}R+R+I_{out}R)N)\). \(|\mathcal{A}|\) denoted the number of non-zeros in adjacency tensor \(\mathcal{A}\). As for non-uniform generalization, adding global node do not change the degree of both space and time complexity. Multi-uniform process will increase the space complexity into \(\mathcal{O}((I_{in}+I_{out}+1)N_{max}RL)\) and will not change degree of the time complexity. Table I shows a summary of complexities of different formulations.

### _Non-Uniform Generalization_

One of the most critical issues of using adjacency tensor in hypergraph analysis is that only uniform hypergraph can be processed. In addition, the proposed THNN in Eq. (15) only considered the uniform hypergraph. But in many real-world situations, non-uniform hypergraphs are needed. Hence, we aim to extend the proposed model for non-uniform hypergraphs. Motivated by [12] and [32], we propose two methods to extend the uniform hypergraph models to handle non-uniform hypergraphs.

**Global Node.** As shown in Figure 7, we can add a global node to the hypergraph. The non-informative global node would be added many times in one hyperedge until the order of the hyperedge is equal to the max-order number. The feature vector of the global point will be a trainable vector with the same size as other node features. Since the non-informative global node has too many neighbors, the representation of the linked neighbors of the informative global node will tend to converge to the same value, which may exacerbate the issue of oversmoothing in the message passing procedure.

**Multi-Uniform Processing.** So in order to mitigate the problem of oversmoothing of global node adding strategies, as shown in Figure 8, we also proposed a multi-uniform processing scheme that decomposes a non-uniform hypergraph into several uniform sub-hypergraphs. One sub-hypergraph contains all the hyperedges with the same number of orders. Then, we could process uniform sub-hypergraphs separately. Finally, we can concatenate feature vectors with different orders and process them via a trainable weight matrix.

In this paper, we mainly focus on node classification tasks. Therefore, when we get the node representation through several layers of THNN, we directly use a fully connected layer to obtain the predicted label and use the **cross-entropy** loss to optimize parameters.

### _Complexity Analysis_

In this section, we formalize the above-mentioned kinds of time and space complexity. Through a series of analyses, we reduce the exponential complexity of Eq. (9) to the current complexity of Eq. (14). The original outer product based high order message passing scheme in Eq. (9) suffers from a very high space and time complexity. The parameter space complexity is \(\mathcal{O}(I_{in}^{2}I_{out}L)\) and the computational time complexity is \(\mathcal{O}(|\mathcal{V}|^{3}(I_{in}^{2}+I_{in}^{2}I_{out}L))\). \(L\) denoted the number of layers. After exploiting the symmetry property of the feature tensor via a partially CP decomposition format, as stated in Eq. (13), the parameter space complexity and computational time complexity can be reduced into \(\mathcal{O}((I_{in}+I_{out}+1)RL)\) and \(\mathcal{O}(|\mathcal{V}|^{3}R+|\mathcal{V}|^{2}R^{2}+|\mathcal{V}|R^{3}+| \mathcal{V}|RI_{out}+I_{in}I_{out}R)\) respectively. If the computation is fully optimized in dot product form in Eq. (14), the time complexity will be reduced to \(\mathcal{O}(|\mathcal{A}|(I_{in}R+R+I_{out}R))\). \(|\mathcal{A}|\) denoted the number of non-zeros in adjacency tensor \(\mathcal{A}\). As for non-uniform generalization, adding global node does not increase both space and time complexity too much. Multi-uniform process will increase the space complexity into \(\mathcal{O}((I_{in}+I_{out}+1)N_{max}RL)\), where \(N_{max}\) is the largest number of order in non-uniform hypergraph, and will not change degree of the time complexity. Table I shows a summary of complexities of different formulations. If parallelism can be fully utilized like the current cuda-based sparse adjacency matrix calculation scheme, the time complexity of the model can be further reduced.

## IV Experiments

In this section, we conduct experiments on two different 3-D visual object classification datasets under both uniform and non-uniform hypergraph construction settings. Experimental results verify the effectiveness of the proposed model. In addition, we also performed ablation analysis and hyperparameter experiments in order to acquire a better knowledge of the model and data.

### _Datasets_

In experiments, two public benchmarks, the Princeton ModelNet40 dataset [35] and the National Taiwan University (NTU) 3D model dataset [36] are used. The ModelNet40 dataset includes 12,311 objects from 40 popular categories, while the NTU dataset has 2,012 3D items from 67 categories. We apply the similar data split settings in [17], [37], and we extract the features of 3D objects through the Multi-view Convolutional Neural Network (MVCNN) [38] and Group-view Convolutional Neural Network (GVCNN) [39]. 12 virtual cameras are used to collect images with a 30 degree interval angle, and MVCNN and GVCNN features are extracted accordingly.

### _Uniform Hypergraph Generation_

After obtaining the vector embedding representation of the 3D object in Euclidean space, we implement a distance-based hypergraph generation [7, 16, 17]. Such a distance-generation approach connects a group of similar vertices to
the same centroid and exploits the correlations among vertices. Distance-based hyperedges can represent node connection in the feature space [7, 16, 17].

More specifically, given the features of all data objects, the affinity matrix \(M\) can be computed as

\[M_{ij}=\exp\left(-\frac{2D_{ij}^{2}}{\Delta}\right),\]

where \(D_{ij}\) indicates the Euclidean distance between vertex \(i\) and \(j\) in the feature embedding space. \(\Delta\) is the average pairwise distance between vertices. Every 3D object in the dataset is chosen as the centroid node, its k-nearest neighbors in the embedding feature space are connected to the centroid by a hyperedge.

By selecting the centroid node \(j\) and k-nearest neighbor nodes \(i\in\textit{KNN}[j]\), we can define \(\hat{H}_{ij}=M_{ij}\). Then, an appropriate probabilistic hypergraph incidence matrix is constructed. Then, we can use \(H_{ij}=\mathds{1}(\hat{H}_{ij}>0)\) to construct a k-uniform hypergraph, where \(\mathds{1}(\cdot)\) denotes the indicator function.

Alternatively, we can sample \(H_{ij}=\textit{Bern}(\hat{H}_{ij})\) to construct a non-uniform hypergraph, where \(\textit{Bern}(\cdot)\) denotes a Bernoulli distribution. Once the non-uniform hypergraph structure is generated, the structure remains consistent in all models under one setting. We choose the \(K=4\) as the default setting in our experiments, and an experiment of it is discussed in

\begin{table}
\begin{tabular}{c|c|c} \hline \hline
**Formulation** & Space Complexity & Time Complexity \\ \hline Outer Product formulation in Eq (0) (order \(3\)) & \(\mathcal{O}(I_{in}^{s}I_{out}L)\) & \(\mathcal{O}(|\mathcal{V}|^{3}(I_{in}^{s}+I_{out}^{s}I_{out}L)L)\) \\ \hline General Outer Product formulation (order \(\kappa=N\)) & \(\mathcal{O}(I_{in}+I_{out}+1)RL)\) & \(\mathcal{O}(|\mathcal{V}|^{3}(I_{in}^{s}+I_{out}^{s}I_{out}^{s}I_{out}^{s}I_{out }+\mathcal{V}|^{3}|\mathcal{V}|RL_{out}+I_{in}I_{out}L)R)\) \\ \hline Tensor Contraction formulation in Eq (13) (order \(\kappa=3\)) & \(\mathcal{O}(I_{in}+I_{out}+1)RL)\) & \(\mathcal{O}(|\mathcal{A}|(I_{in}R+I_{out}L_{out}L))\) \\ \hline General Tensor Contraction formulation (order \(\kappa=N\)) & \(\mathcal{O}(I_{in}^{s}I_{out}L)\) & \(\mathcal{O}(|\mathcal{V}|^{3}(I_{in}^{s}+I_{out}^{s}I_{out}L))\) \\ \hline Our Postnode formulation in Eq (15) (order \(3\)) & \(\mathcal{O}(I_{in}+I_{out}+1)RL)\) & \(\mathcal{O}(|\mathcal{V}|^{3}(I_{in}^{s}+I_{out}^{s}I_{out}L_{out}L_{out}L_{out}R)\) \\ \hline General Den Product formulation (order \(\kappa=N\)) & \(\mathcal{O}(I_{in}+I_{out}+1)RL)\) & \(\mathcal{O}(|\mathcal{A}|(I_{in}R+I_{out}L_{out}L_{out}L_{out}L_{out}L_{out}))\) \\ \hline Adding Global Node Extension of Non-uniform THEN (maxorder \(=N_{max}\)) & \(\mathcal{O}(I_{in}+I_{out}+1)RL+I_{in}\) & \(\mathcal{O}(|\mathcal{A}|(I_{in}R+I_{out}L_{out}L_{out}L_{out}))\) \\ \hline Multi-uniform Processing Extension of Non-uniform THEN (maxorder \(=N_{max}\)) & \(\mathcal{O}(|\mathcal{A}|(I_{in}R+I_{out}L_{out}L_{out}L_{out}L_{out}))\) \\ \hline \hline \end{tabular}
\end{table} TABLE I: Complexity comparison. \(I_{in}\) and \(I_{out}\) are feature input dimensionality number and output dimensionality number. \(L\) is the number of layers and \(R\) is the number of rank. \(|\mathcal{V}|\) is the number of nodes in the hypergraph, and \(|\mathcal{A}|\) is the number of non-zeros in adjacency tensor. \(N\) is the number of orders of the uniform hypergraph, and \(N_{max}\) is the largest number of order in the non-uniform hypergraph.

Fig. 8: We can also process hypergraphs in layers and utilize distinct models to process sub-hypergraphs of different orders. The resultant embedding vectors of distinct layers are therefore concatenated and integrated through a fully-connected layer.

Fig. 7: Inspired by [12], we can add a global node \(v_{g}\) to a non-uniform hypergraph in order to make it uniform. The global node can be added many times in one hyperedge. Following this procedure, the hypergraph can well be represented as an adjacency tensor, allowing it to be directly processed in uniform-hypergraph form models.
## Appendix IV-H

### _Baseline Models_

After constructing hypergraphs, we compare THNN with multiple graph and hypergraph neural network baselines. All results about baseline are reproduced by ourselves. For GNN baselines, we feed the clique-expansion of constructed hypergraphs. The baselines are listed and decribed as follows:

* Graph convolutional network (GCN) [2]. A GCN layer is equivalent to a localized first-order spectral filter on graphs. GCNs can be thought of as convolutional neural networks that are stretched to process graph-structured data;
* Graph attention network (GAT) [1] incorporates an attention mechanism into graph convolution. It can be considered as a GNN with learnable edge weights;
* Graph Isomorphism Network (GIN) [3]. Among the GNNs performing first-order neighborhood aggregation, GIN is proved to be the one with maximal expressiveness;
* HyperGCN [40]. Each hyperedge of the hypergraph is approximated by a collection of edges connecting the pairs of vertices of the hyperedge. Then the hypergraph learning problem becomes a special case of the graph learning.
* Hypergraph Networks with Hyperedge Neurons (HNHN) [41]. HNHN is also a first-order approximation message passing neural network with a node-specific normalization;
* Hypergraph Neural Networks (HGNN) [17]. HGNN has been discussed in Section III-A in detail.

### _Results on the Uniform Setting_

First, we evaluate THNN along with the baselines in a uniform hypergraph setting with \(k=4\). We employ a two-layer THNN model with a rank setting of 128. THNN is trained with the Adam optimizer whose learning rate is set to be 0.001 initially. Similar to the evaluation procedure in [17], we create multiple hypergraph structures for comparison using either the single features or the concatenation multi-features.

Detailed results on the two datasets are reported in Tables II. We have the following observations. First, THNN maintains the best performance in the majority of cases because the proposed models are compelling in extracting high-order information of hypergraph structures. For example, compared with HyperGCN, THNN achieves gains of \(0.92\%\) and \(1.02\%\) on average of 7 settings on the ModelNet40 and the NTU datasets, respectively. Second, the graph-based model performs worse than the hypergraph in most cases because the hypergraph structure can convey complicated high-order correlations among data, whereas the clique expansion introduces some information loss. Furthermore, as a result of information loss, the graph-based models achieve poor results in some settings. For example, GCN achieves \(77.80\%\), and GIN achieves \(87.93\%\) in the **C: MvGv, T: Gv** setting of ModelNet40, while the accuracy of other models under such setting is all above \(90\%\). The representation information under this setting can be better characterized by a model that considers higher-order information. These cases indicate that graph models are unstable in situations where higher-order information dominates.

### _Results on the non-Uniform Setting_

We also create non-uniform hypergraph structures to validate the efficacy of the two proposed extensions of THNN. Because each element of the probability incidence matrix \(\hat{H}\) takes value in \([0,1]\), and the closer the value is to 1, the more similar with centroid node the value is, it is possible to perform Bernoulli sampling on \(\hat{H}\). We employ the two-layer THNN extension models with adding a global node (THNN-AdG) and Multi-Uniform Processing (THNN-Multi) a rank setting of 128. The two extensions are also trained with Adam optimizers whose learning rates are set to be 0.005.

Detailed results are reported in Tables III. In these two tables, the results remain consistent with those in the uniform settings. The two proposed extensions of THNN perform the best in the majority of scenarios. Specifically, THNN-Multi performs better than THNN-AdG. Compared with THNN-AdG, THNN-Multi achieves gains of \(0.59\%\) and \(0.86\%\) on average of the 7 settings on the ModelNet40 and the NTU datasets, respectively. The possible reason is that interactions of different orders are learned via hypergraph separation processing procedure and aligned via merged layer in THNN-Multi, while artificially introduced external global node might disturb the interaction representation of different orders due to the oversmoothing issue in THNN-AdG.

But THNN-Multi requires several THNN models to process the hypergraph message passing procedure of different orders.

As a result, the number of parameters of THNN-Multi increases linearly with the order number, but the parameter number of the THNN-AdG is not influenced by order number. Thus, the well-performing THNN-Multi generally has a larger number of parameters than THNN-AdG when the order number is large. In addition, the graph-based model still performs worse than the hypergraph in most cases and suffers from large variance across hypergraph construction and feature selection settings.

In summary, hypergraph-based models can better model complex correlations among data than graph-based models. Compared with other hypergraph models that adopt first-order approximations,

the proposed adjacency-tensor-based THNN enjoys higher-order information modeling, which will lead to a better performance.

### _Ablation Study_

We also conducted some ablation experiments to verify the effectiveness of the proposed techniques. We choose two uniform hypergraph generation settings. For these two datasets, we evaluate the proposed three techniques. We use \(Tanh\) in \(\sigma^{\prime}\) to make the element-wise product in CP decomposition stable. A discussion of other \(\sigma^{\prime}\) choices is in Section IV-J. Secondly, we use the techniques of the _concatenating ones_ to improve the expressiveness of the model. We also evaluate the normalized adjacency tensor strategy, which is inspired by the normalized Laplacian adjacency matrix to make the training stable. Results show that it is crucial to use \(Tanh\). This is because higher-order tensor operations introduce operations such as \(x^{n}\), which makes training neural networks extremely challenging. Small changes in the \(x\) can cause large disturbances in \(x^{n}\).
### _Hyper-parameter Analysis: Rank and Number of Layers_

As shown in Fig (a)a, we evaluate the impact of rank \(R\) on model performance by generating hypergraphs with varying values of \(K\) (from 2 to 6) in the NUT2012 experiment and by adjusting the rank setting in the THNN. We choose the "C: Mv T: Gv" setting in NUT2012. We found that \(Rank=128\) is a proper setting, as the accuracy does not change much when \(Rank\in[75,150]\). We also examine how the number of layers in THNN affects performance. Under the setting of "C: Mv T: Gv" in NUT2012, we discover that the model's expressiveness cannot be fully explored with a single layer. Still, the model's performance degrades dramatically when there are too many layers. We conject that this may be due to numerical instability (for example, \(10.001^{10}\) is much larger than \(10^{10}\)) of higher order operations or over-smoothing problems in graph learning [42]. The experiment results in Fig (b)b has demonstrated that stacking two layers is optimal.

### _K in Hypergraph Generation_

The numerical properties of the THNN will be unstable when a large number of components are involved in a single hyperedge, which is not a condition our model handles well due to numerical explosion of high-order product operations. Therefore, we did not select \(K=10\) as the HGNN did [17]. We discovered through the experiment that \(K\) does not need to be huge for this task. As demonstrated in Fig. 10, \(K=4\) is an optimal choice.

The numerical properties of the THNN will be unstable when a large number of components are involved in a single hyperedge, which is not a condition our model handles well due to the numerical explosion of high-order product operations. Therefore, we did not select \(K=10\) as the HGNN did [17]. We choose the "C: Mv T: Gv" setting in NUT2012. We discovered through the experiment that \(K\) does not need to be huge for this task. As demonstrated in Fig. 10, \(K=4\) is an optimal choice.
connection to a THNN layer to see whether it helps. We define the new THNN's layers as follows:

\[F^{{}^{\prime}}(x)=F(x)+Wx\]

where \(F(\cdot)\) is the original THNN layer, and \(W\) is trainable weight matrix to align feature dimensions. We adopted the NUT2012 dataset in the uniform case and the result is shown in Fig 11. We found that the residual connection can improve the performance of the NUT2012 dataset (about 0.8% on average) and can help THNN to achieve a deeper layer stacking. However, in order to make a fair comparison in the main text, we did not add residual connections in comparison with the baseline.

### _Other Choice of Activation Function \(\sigma^{{}^{\prime}}\)_

When we first encountered the instability issue during the training procedure, we investigated using tanh in \(\sigma^{{}^{\prime}}\) in Eq (15) would solve this problem. However, it is not certain that other different activation functions would have the same result. We thus attempt to replace tanh with another common activation function while maintaining all other settings the same. We chose two uniform hypergraph generation settings for our evaluation. As shown in Table V, it is apparent that activation functions that restrict the output value to a limited range perform well. Other activation functions that do not restrict value ranges are unstable in some or all Settings. Among the four efficient common activation functions, tanh and sigmoid are the most effective, but Softsign and hardsigmoid do not need time-consuming exp operations. Accordingly, different stable activation functions can be chosen based on specific requirements. We summarize the properties of these activation functions in the TABLE VI.

We summarize the properties of the activation function in the experiment in Section IV-J

### _Hypergraph Learning with Adjacency Tensor_

In hypergraph learning, most existing methods convert the hypergraph into a weighted graph [17, 40, 41], and then existing graph methods can be applied. For example, spectral clustering could be performed on the weighted graph Laplacian [44] for hypergraph community detection. As stated in [21], such a conversion will result in information loss and sub-optimal performance in hypergraph community detection. As tensor is the most natural extension in representing p-adic relationships [32], many adjacency tensor based methods

\begin{table}
\begin{tabular}{c|c c|c c} \hline \hline \multicolumn{1}{c|}{\multirow{2}{*}{**Dataset**}} & \multicolumn{2}{c|}{NUT2012} & \multicolumn{2}{c}{ModelNet40} \\ \hline \multirow{2}{*}{**Function Type**} & **C: Mv** & **C: MvGv** & **C: MvGv** \\  & **T: Gv** & **T: MvGv** & **T: Gv** & **T: MvGv** \\ \hline No Activation & 69.43\% & 68.36\% & 82.74\% & 83.95\% \\ \hline LeakyReLU & 56.03\% & 69.17 & 88.21\% & 46.68\% \\ \hline ReLU & 69.71\% & 65.42\% & 89.34\% & 10.94\% \\ \hline Tanishhrink & 65.41\% & 41.82\% & 87.40\% & 69.69\% \\ \hline ELU & 75.87\% & 83.65\% & 90.03\% & 82.25\% \\ \hline Hardsigmoid & 78.28\% & 83.65\% & 92.34\% & 93.00\% \\ \hline Softsign & 77.75\% & 83.11\% & 91.86\% & 91.15\% \\ \hline Sigmoid & 77.75\% & 83.65\% & 92.02\% & 95.38\% \\ \hline Tnnh & 78.55\% & 83.91\% & 92.38\% & 94.25\% \\ \hline \hline \end{tabular}
\end{table} TABLE V: Results about different activation function

Fig. 10: Influence of K in NUT2012 for THNN under different rank settings.
are proposed to conduct optimal community detection [21, 45] on both uniform hypergraphs and non-uniform ones. To the best of our knowledge, the majority of adjacency tensor based research is focused on spectral properties [46, 47, 48] of hypergraphs and tensor decomposition modelings [12, 20, 49], and there is not any adjacency tensor based hypergraph neural networks.

### _Tensor Fusion Models and Tensorized Neural Networks_

Zadeh et al. [22] propose to use a tensorized outer product as a deep information fusion layer named as Tensor Fusion Layer (TFL), which learns the intra-modality dynamics and inter-modality dynamics as well as aggregates multi-modal interactions. This type of fusion operation can be deemed as a variant of exponential machines [50] or a special case of Higher-order Polynomial Regression [29, 30]. Multi-linear tensor models have been proven to have better expressive power in modeling various types of interactions (all pairs, triplets, and even p-adic). These tensor fusion neural networks are also Tensorized Neural Networks (TNNs) [51, 52]. TNNs are designed to process high order tensor input rather than vectors or matrices via a tensor parameterization procedure. Tensor parameterization procedures have achieved great success in enhancing the expressive power [53, 54, 55] of Neural Networks while compressing the number of parameters [56, 57]. Different from the exiting TNNs in processing multi-modal sentimental data [24], image data [52] or video data [56], the proposed THNN is the first Tensorized Neural Network in modeling high order information interaction in hypergraphs.

## VI Conclusion and Discussion

In this paper, we discover that existing hypergraph neural networks are mostly based on the message passing of first-order approximations of the original hypergraph structure and ignore higher-order interactions encoded in hypergraphs. In order to better model the higher-order information of the hypergraph structure, we propose a novel hypergraph neural network, THNN, which is based on adjacency tensors of hypergraphs and is a high-order extension of traditional Graph Convolution Neural Networks. We also find that the proposed models are well connected with tensor fusion, which is a highly recognized successful technique for high-order interaction modeling. Therefore, the proposed models are compelling in extracting high-order information encoded in hypergraphs. We show that our framework can achieve the promising performance in 3-D visual object classification tasks under both uniform and non-uniform hypergraph settings. In the future, we plan to explore more applications of the proposed tensorized neural network models to verify their advantages.
