Unifying gradient regularization for Heterogeneous Graph Neural Networks
Heterogeneous Graph Neural Networks (HGNNs) are a class of powerful deep
learning methods widely used to learn representations of heterogeneous graphs.
Despite the fast development of HGNNs, they still face some challenges such as
over-smoothing, and non-robustness. Previous studies have shown that these
problems can be reduced by using gradient regularization methods. However, the
existing gradient regularization methods focus on either graph topology or node
features. There is no universal approach to integrate these features, which
severely affects the efficiency of regularization. In addition, the inclusion
of gradient regularization into HGNNs sometimes leads to some problems, such as
an unstable training process, increased complexity and insufficient coverage
regularized information. Furthermore, there is still short of a complete
theoretical analysis of the effects of gradient regularization on HGNNs. In
this paper, we propose a novel gradient regularization method called Grug,
which iteratively applies regularization to the gradients generated by both
propagated messages and the node features during the message-passing process.
Grug provides a unified framework integrating graph topology and node features,
based on which we conduct a detailed theoretical analysis of their
effectiveness. Specifically, the theoretical analyses elaborate the advantages
of Grug: 1) Decreasing sample variance during the training process (Stability);
2) Enhancing the generalization of the model (Universality); 3) Reducing the
complexity of the model (Simplicity); 4) Improving the integrity and diversity
of graph information utilization (Diversity). As a result, Grug has the
potential to surpass the theoretical upper bounds set by DropMessage (AAAI-23
Distinguished Papers). In addition, we evaluate Grug on five public real-world
datasets with two downstream tasks...
Xiao Yang, Xuejiao Zhao, Zhiqi Shen
# Unifying gradient regularization for Heterogeneous Graph Neural Networks

###### Abstract

Heterogeneous Graph Neural Networks (HGNNs) are a class of powerful deep learning methods widely used to learn representations of heterogeneous graphs. Despite the fast development of HGNNs, they still face some challenges such as _over-smoothing_, and _non-robustness_. Previous studies have shown that these problems can be reduced by using gradient regularization methods. However, the existing gradient regularization methods focus on either graph topology or node features. There is no universal approach to integrate these features, which severely affects the efficiency of regularization. In addition, the inclusion of gradient regularization into HGNNs sometimes leads to some problems, such as an unstable training process, increased complexity and insufficient coverage of regularized information. Furthermore, there is still short of a complete theoretical analysis of the effects of gradient regularization on HGNNs. In this paper, we propose a novel gradient regularization method called _Grug_, which iteratively applies regularization to the gradients generated by both propagated messages and the node features during the message-passing process. _Grug_ provides a unified framework integrating graph topology and node features, based on which we conduct a detailed theoretical analysis of their effectiveness. Specifically, the theoretical analyses elaborate the advantages of _Grug_: 1) Decreasing sample variance during the training process **(Stability)**; 2) Enhancing the generalization of the model **(Universality)**; 3) Reducing the complexity of the model **(Simplicity)**; 4) Improving the integrity and diversity of graph information utilization **(Diversity)**. As a result, _Grug_ has the potential to surpass the theoretical upper bounds set by DropMessage. 1. In addition, we evaluate _Grug_ on five public real-world datasets with two downstream tasks. The experimental results show that _Grug_ significantly improves performance and effectiveness, and reduces the challenges mentioned above. Our code is available at: [https://github.com/YX-cloud/Grug](https://github.com/YX-cloud/Grug).

Footnote 1: AAAI-23 Distinguished Papers

## 1 Introduction

As graph neural networks (GNNs) [1; 2] have emerged as powerful architectures for learning and analyzing graph representations. Therefore, in recent years, researchers have actively explored their potential in Heterogeneous Graph Neural Networks (HGNNs) [3; 4; 5]. Heterogeneous graphs consist of multiple types of nodes and edges with different side information. To tackle the challenge of heterogeneity, various HGNNs[6; 7] have been proposed, such as HGT[8], HAN[9], fastGTN[10], SimpleHGNN[11], MAGNN[12], RSHN[13] and HetSANN[14] were developed within recent years.
These HGNNs perform well in a variety of downstream tasks, including node classification, link prediction, and recommendation [15; 16].

Despite the rapid development of the HGNNs, they often face problems such as _over-smoothing_ and _non-robustness_[17], which seriously affects the efficiency of downstream tasks. Heterogeneous graphs (HGs) are more ubiquitous than homogeneous graphs in real-world scenarios, but they are more difficult to represent than homogeneous graphs. Because the diversity of data structures of HGs limits the generalization of HGNNs and reduces their robustness as well. Moreover, during message-passing in HGNNs, each node aggregates messages from its neighbors in each layer, HGNNs tend to blur the features of nodes, which is called over-smoothing. Some studies have proved that HGNNs are a special kind of regularization, and their _over-smoothing_ and _non-robustness_ problems mentioned above can be effectively alleviated by using additional regularization to the HGNNs [17; 18].

Gradient regularization is one of the most popular regularization methods to improve the above problem. It integrates various regularizations into one gradient through different strategies. Many widely used regularization methods have greatly improved the performance of HGNNs, such as random dropping methods [19; 20; 21; 22], Laplacian regularization methods [23; 24] and adversarial training methods [25; 26; 27; 28], etc. In this paper, we prove theoretically that these methods are essentially a special form of gradient regularization.

Although gradient regularization methods have been widely applied to HGNNs, there are still some open questions that need to be addressed. First, the existing gradient regularization methods only perform regularization on one of the following three information dimensions, namely node, edge and propagation message. These methods do not fully explore which dimension is the optimal solution. Second, the inclusion of gradient regularization into HGNNs sometimes leads to some additional problems, such as unstable training process, and parameter convergence difficulty caused by multiple solutions problems. Lastly, there is still short of a complete theoretical analysis of the effects of gradient regularization on HGNNs.

In this paper, we propose a novel gradient regularization method called _Grug_, which can be applied to all message-passing HGNNs. _Grug_ provides a unified framework integrating graph topology and node features by iteratively applying regularization to the gradients generated by both propagated messages and the node features during the message-passing process. We conduct a detailed theoretical analysis of the effectiveness of _Grug_. These theoretical analyses elaborate the advantages of _Grug_ in 4 dimensions: 1) _Grug_ makes the training process more stable by realizing the controllable variance of the input samples **(Stability)**; 2) _Grug_ unify a wider range of regularizations to improve the performance of HGNNs **(Universality)**; 3) _Grug_ ensures the continuity of gradient and the unique solution of the model during gradient regularization, which alleviates parameter convergence difficulty caused by traditional multiple solutions models **(Simplicity)**; 4) _Grug_ can provide more diverse information to the model and improve the integrity and diversity of graph information utilization **(Diversity)**. Thus, _Grug_ has the potential to surpass the DropMessage [22]. In addition, we evaluate _Grug_ on five public real-world datasets with two downstream tasks. Our main contributions are summarized as follows:

* We propose a novel gradient regularization method for all message-passing HGNNs named _Grug_. _Grug_ can unify the existing gradient regularizations on HGNNs into one framework by resetting distribution (dropping or perturbation) in certain rules on the feature matrix and message matrix. In other words, the existing methods can be regarded as one special form of _Grug_.
* We provide a series of comprehensive and detailed theoretical analyses covering the stability, universality, simplicity and diversity of gradient regularization methods. Which sufficiently explains "Why" and "How" _Grug_ has the potential to surpass the theoretical upper bounds set by DropMessage [22].
* We conduct sufficient experiments for different downstream tasks and the results show that _Grug_ not only surpasses the state-of-the-art (SOTA) performance, but also alleviates over-smoothing, and non-robustness obviously compared to the existing gradient regularization methods.
Related Work

In general, random dropping methods and adversarial training methods can be regarded as a form of gradient regularization methods [29; 30; 31]. As a representative of random dropping, Dropout [19; 32] has been proven to be effective in many fields [33] with enough theoretical analysis [34]. Besides, applying random dropping to HGNNs can achieve effective performance, such as DropNode [21] and DropEdge[20], which randomly drop nodes and edges during the training process respectively. Additionally, the best random dropping method is DropMessage [22], which applies random dropping on propagated messages. Adversarial Training is proposed as a method of defense against attacks [35], which generate new adversarial samples according to the gradient and adds them to the training set. On this basis, a stronger adversarial attack model PGD [36] has become the mainstream model. In fact, adversarial training has been shown to be closely related to regularization [30; 31]. Recently adversarial training has also been applied to HGNNs [25; 26; 27; 37]. Specially, FLAG [28] tries to establish node-level adversarial training to improve the representation of the graph and achieve satisfactory results. These existing gradient regularization methods can be used to relieve over-smoothing and over-fitting problems [22; 28; 21]. But all of them only perform regularization on one of the following three information dimensions, namely node, edge and propagation message. They do not fully explore which dimension is the optimal solution [25; 27; 37]. Furthermore, we find that the inclusion of gradient regularization into HGNNs sometimes leads to some additional problems, such as unstable training process, and parameter convergence difficulty caused by multiple solutions problems. All the problems of the existing methods are alleviated by the proposed method in this paper.

## 3 Notations and Preliminaries

**Notations.** For simplicity of analysis, we assume that the heterogeneous graph contains 2 relations, \(r\) and \(r^{\sim}\). **HG** = (**V**,**E**,**R**) represent the heterogeneous graph, where **V** = {\(v_{1}...v_{p}\)} denotes \(p\) heterogeneous nodes, and **E** = {\(e_{1}...e_{q}\)} denotes \(q\) heterogeneous edges. **R** = {\(r\), \(r^{\sim}\)} represents relations in heterogeneous graph. The node features can be denoted as a matrix \(F=\{x_{1}...x_{n},x_{p}\}\in R^{p\times d}\), where \(x_{i}\) is the feature vector of the node \(v_{i}\), and \(d\) is the dimension of node features. The edges and relations describe the relations between nodes and can be denoted as an adjacent matrix \(A=\{a_{1}...a_{q}\}\subseteq R^{q\times q}\), where \(a_{i}\) denotes the \(i\)-th row of the adjacent matrix, and \(A(i,j)\) denotes the relation between node \(v_{i}\) and node \(v_{j}\). When we apply message-passing HGNNs on **HG**, the message matrix can be represented as \(M=\{m_{1}...m_{k}\}\subseteq R^{k\times d^{\prime}}\), where \(m_{i}\) is a message propagated between nodes, \(k\) is the number of messages propagated on the heterogeneous graph and \(d^{\prime}\) is the dimension of the messages.

**Message-passing HGNNs.** In most existing HGNN models, a message-passing framework is adopted, where each node sends and receives messages to/from its neighbors based on their relations. During the propagation process, node representations are updated using both the original node features and the received messages from neighbors. This process can be expressed as follows:

\[h_{i}^{(l+1)}=\sigma^{(l)}(h_{i}^{(l)},AGG_{j\in N(i)}\phi^{(l)}(h_{i}^{(l) },h_{j}^{(l)},e_{j,i})) \tag{1}\]

where \(h_{i}^{(l+1)}\) denotes the representation of node \(v_{i}\) in the \(l\)-th layer, and \(N(i)\) is a set of adjacent nodes to node \(v_{i}\); \(e_{j,i}\) represents the relation from node \(v_{i}\) to node \(v_{j}\); \(AGG\) denotes the aggregation operation; and \(\sigma^{(l)}\) and \(\phi^{(l)}\) are differentiable functions. Besides we can gather all propagated messages into a message matrix \(M\subseteq R^{k\times d^{\prime}}\), which can be expressed as below:

\[M_{i}^{(l+1)}=\phi(h_{i}^{(l)},h_{j}^{(l)},e_{j,i})\]

where \(\phi\) denotes the message mapping information, and the row number \(k\) of the message matrix M is equal to the directed edge number in the graph.
## 4 Our Approach

In this section, we introduce our newly proposed method called _Grug_, which can be applied to all message-passing HGNNs. We first introduce details of our _Grug_, and further prove that the most common gradient regularization methods in HGNNs such as random dropping and adversarial perturbations, can be unified into our framework. Then, we present the theoretical evidence of the effectiveness of these methods. Furthermore, we analyze the superiority of _Grug_, and provide a series of theoretical proofs from the aspects of stability, universality, simplicity and diversity.

### _Grug_

#### Algorithm description.

We apply a single RGCN as the backbone model, which can be formulated as \(f(F,M,W)=FMW\), where \(F\in R^{n\times k}\) denotes the feature matrix, \(M\) indicates the message matrix and \(W\) denotes the transformation matrix. When we use cross-entropy as the loss function, the objective function can be expressed as follows:

\[L(f(F,M,W),y)=\sum_{i=0}^{K-1}y_{i}\log\left(softmax(f(F,M,W))\right) \tag{2}\]

where \(K\) is the class number and \(y_{i}\) is one hot representation of label \(i\).

Different from existing gradient regularization methods, _Grug_ performs gradient regularization both on the feature matrix \(F\) and message matrix \(M\) instead of either one of them and the process of _Grug_ is in Figure 1. For each element \(F_{i,j}\) in \(F\) and \(M_{i,j}\) in \(M\), we generate trained additional elements \(\gamma^{i,j}\) and \(\delta^{i,j}\) to perturb the original distribution, according to uniform distribution \(\gamma^{i,j}\sim Uniform(-\alpha,\alpha)\) and \(\delta^{i,j}\sim Uniform(-\beta,\beta)\). After that, we obtain the perturbed \(F\) and \(M\) by adding each element with its trained additional element. Besides, in the training process, \(\gamma^{i,j}\) and \(\delta^{i,j}\) can be updated according to the gradient of the previous epoch, which can be expressed as follows:

\[\delta_{t}=\delta_{t-1}+\beta\cdot\frac{\partial_{\delta}L(f(F,M,W),y),y)}{|| \partial_{\delta}L(f(F,M,W),y),y)||} \tag{3}\]

#### Unifying gradient regularization methods.

In this section, we show that existing gradient regularization methods can be integrated into our _Grug_. Different from these methods, _Grug_ performs regularization rules on the feature matrix and message

Figure 1: Illustrations of _Grug_. Considering the messages propagated by the center node (i.e., Node 1), _Grug_ allows to perturb the feature matrix and message matrix, which implements the regularization of gradients on the feature matrix and message matrix.
matrix. First, we demonstrate that Dropout [34], DropEdge [20], DropNode [21], DropMessage [22] and FLAG [28] can all be formulated as norm regularization process with specific dimension. More importantly, we find that existing methods are a special case of _Grug_ in terms of norm dimensions. In summary, _Grug_ can be expressed as a unified framework of gradient regularization methods on HGNNs.

**Theorem 1**.: _Dropout, DropEdge, DropNode, DropMessage and FLAG can be formulated in specification norm regularization process, and Grug is the generalization case of these methods._ We show the regularization term of each method below.

_Dropout._ dropping the elements \(F_{i,j}=\eta F_{i,j}\) where \(\eta\sim Bernoulli(\mu)\) is equivalent to perform regularization term \(\frac{1}{2}\cdot||\partial_{F_{\zeta}}L||_{0}\) on message matrix.

_DropEdge._ dropping the elements \(A_{i,j}=\epsilon A_{i,j}\) where \(\epsilon\sim Bernoulli(\mu)\) is equivalent to perform regularization term \(\frac{\partial_{A}M}{2}\cdot||\partial_{M_{t}}L||_{0}\) on message matrix.

_DropNode._ dropping the elements \(F_{i}=\eta F_{i}\) where \(\zeta\sim Bernoulli(\mu)\) is equivalent to perform regularization term \(frac{1}{2}\cdot||\partial_{F_{\zeta}}L||_{0}\) on feature matrix.

_DropMessage._ dropping the elements \(M_{i,j}=\tau M_{i,j}\) where \(\tau\sim Bernoulli(\mu)\) is equivalent to perform regularization term \(\frac{1}{2}\cdot||\partial_{M_{t}}L||_{0}\) on message matrix.

_FLAG._ Perturbing the feature matrix \(F+\delta\), which is equivalent to perform regularization term \(\varepsilon_{t}||\partial_{F}L||_{q_{t}}\) on feature matrix.

_Grug._ Perturbing both the feature matrix \(F+\delta\) and message matrix \(M+\xi\), which is equivalent to perform regularization term \(\varepsilon_{t}\cdot||\partial_{F}L||_{q_{t}}+\xi_{t}\cdot||\partial_{F}L||_{h _{t}}\) on feature matrix and message matrix. Where \(\delta\) and \(\gamma\) are the trained \(F\) perturbation, which are defined as follows:

\[\begin{split}\delta_{t}&=\delta_{t-1}+\beta\cdot \frac{\partial_{\delta}L(f(F+\delta_{t-1},M,W),y)}{||\partial_{\delta}L(f(F+ \delta_{t-1},M,W),y)||}\\ \gamma_{t}&=\gamma_{t-1}+\alpha\cdot\frac{\partial_{ \alpha}L(f(F+\gamma_{t-1},M,W),y)}{||\partial_{\gamma}L(f(F+\gamma_{t-1},M,W),y)||}\end{split} \tag{4}\]

\(||\cdot||_{p_{t}}\) and \(||\cdot||_{l_{t}}\) are the dual norm of \(||\cdot||_{q_{t}}\) and \(||\cdot||_{h_{t}}\), which are defined as follows:

\[\begin{split}||z||_{q}&=sup\{z^{T}x\ ||z||_{p}\leq 1\}\ and\ \frac{1}{p}+\frac{1}{q}=1\\ ||z||_{h}&=sup\{z^{T}x\ ||x||_{l}\leq 1\}\ and\ \frac{1}{h}+\frac{1}{l}=1 \end{split} \tag{5}\]

We find that the range of regularization dimension of _Grug_ is from 0 to \(\infty\). While for _Dropout_, _DropEdge_, _DropNode_ and _DropMessage_ the range of regularization dimension is 0. It is demonstrated that these four methods are special cases of \(Grug\). Compared to \(FLAG\), \(Grug\) can perform different-dimension norm regularization on the feature matrix and message matrix respectively instead of maintaining a consistent-dimension. Thorough proof can be found in Appendix A.1.

### Advantage of _Grug_.

In this section, we present four further theoretical analyses to elaborate the benefits of _Grug_ from the aspects of stability (decreasing sample variance during the training process), universality(), simplicity (reducing complexity of the model), diversity (improving the integrity and diversity of graph information utilization).

**1) Stability.**

Existing gradient regularization methods face the problem of sample stability, which is caused by the data noise introduced by different operations, such as dropping and perturbation. Data noise brings the problem of instability in the training process. Generally, sample variance \(V\) can be used to measure the degree of stability. A small sample variance means that the training process will be more stable.

**Theorem 2**.: _Grug can adjust the sample variance to achieve the stability of the training process._
Intuitively, _Grug_ achieves regularization by adding uniform distribution to the input samples rather than dropping graph data, which means that the sample variance of the input in each epoch is irrelevant to the graph data. In other words, _Grug_ can tune hyperparameters \(\alpha\) and \(\beta\) to limit the range of sample variance. By reducing the sample variance, _Grug_ diminishes the difference between the feature matrix and message matrix, which stabilizes the training process. Specific range inference can be found in Appendix A.1.

**2) Universality.**

**Theorem 3**.: _Grug performs regularization on the feature matrix and message matrix, which is the optimal general solution compared to existing regularization methods._

As we summarized in Section 2, all existing gradient regularization methods only perform regularization on one of the following three information dimensions, namely node, edge and propagation message. In order to explore which dimension is the optimal solution, we first need to know the relations between these three information dimensions. The relations between these three information dimensions can be expressed as follows:

Regularization on message dimension.

\[\partial_{M}L=\partial_{F}L\cdot\partial_{M}F\quad and\quad\partial_{M}L= \partial_{A}L\cdot\partial_{M}A\]

Regularization on edge dimension.

\[\partial_{A}L=\partial_{M}L\cdot\partial_{A}M\quad and\quad\partial_{A}L= \partial_{F}L\cdot\partial_{M}F\cdot\partial_{A}M\]

Regularization on node dimension.

\[\partial_{F}L=\partial_{M}L\cdot\partial_{F}M\quad and\quad\partial_{F}L= \partial_{A}L\cdot\partial_{M}A\cdot\partial_{F}M\]

The above expressions show that the gradient regularization on each information dimension is transferred to the other two information dimensions. Therefore, regularization on any one information dimension is equivalent to regularization on all three information dimensions theoretically. However, we find that these transfers are not in real-time. Take regularization on the node as an example. In epoch \(e_{i}\), the gradient of \(F\) can be calculated by \(L\), which we express as \(\partial_{F}L_{i}\). There is no gradient of \(F\) generated from \(M\), because the model only keeps the gradient from the loss. However, \(\partial_{F}M\) must exist objectively, because \(M\) can be seen as a function involving \(F\) and \(A\). So we can infer that in epoch \(e_{i}\), \(\partial_{F}M\) is correlated to the previous \(e_{i-1}\) epoch. This non-real-time gradient transfer phenomenon is also the same in the edge dimension and message dimension. So we conclude that regularization on only one information dimension cannot apply regularization in real-time to other dimensions. So, gradient regularization on all three information dimensions simultaneously is a better strategy than gradient regularization on one information dimension.

But in practice, we find that compared with node and message dimensions, performing gradient regularization on edge dimension is not only memory consumption, but also reduces the training seed sharply. On the other hand, according to the loss function 2, the node dimension and message dimension, rather than the edge dimension, have the greatest impact on the model loss. Furthermore, performing regularization on the node dimension and message dimension also includes the edge dimension partially. The results of additional experiments show that there are no significant differences between the effects of performing gradient regularization on the edge dimension or not. Therefore, _Grug_ performing regularization on the message and node dimension is the optimal strategy. The Universality is the key for _Grug_ to surpass theoretical upper bounds set by DropMessage and detailed proof are presented in Appendix A.1.

**3) Simplicity.**

**Theorem 4**.: _Message-passing HGNNs using Grug remain one and unique solution, which maintains parameters easily training._

According to Theorem 1 in section 4.1, models using dropping operations on information dimensions can be uniformly expressed as:

\[\begin{split} Drop:\quad\min||\partial_{X}L||_{0}\quad and\quad X =F,M,A\\ \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \
Here \(||\cdot||_{0}\) represents the 0-dimension Paradigm. However, 0-dimension Paradigm is NP-hardness (non-deterministic polynomial-time hardness) [38] and non-convex [39]. Thus, it is hard to find the solution of functions related to dropping, which makes it hard to run at a satisfactory speed.

In formula (6), the \(q_{t}\) and \(h_{t}\) are hyperparameters and we can control them by \(\alpha\) and \(\beta\). So our \(Grug\) uses the 1-dimension Paradigm or 2-dimension Paradigm. In theory, the 1-dimension Paradigm [38] and the 2-dimension [40] Paradigm are all convex and remain one and unique solution. Compared to Dropout, DropNode, DropEdge and DropMessage, _Grug_ will not escalate the difficulty of training from the perspective of convexity. The detailed proof is exhibited in Appendix A.1.

**4) Diversity.**

**Theorem 5**.: _Grug is able to maximize the utilization of diverse information in heterogeneous graph._

Utilization of diverse information contains 2 aspects: 1) Obtaining more diverse information 2) Keeping original information diversity.

Obtaining more diverse information is defined as the total number of sample distributions generated during the training process. We propose that the total number of elements in the message matrix is \(Z\). For dropping methods, the number of new data distributions is generated by DropMessage, which can be expressed as \(C_{Z}^{\frac{Z}{Z}}\) and \(C\) is a combination in mathematics. Our _Grug_ can continue to generate new data distribution until the end of training.

keeping original information diversity is defined as the total number of preserved feature dimensions. In this term, DropMessage has been proven to perform best in dropping operations. However, _Grug_ seems to lose initial information rarely. This is because perturbation \(\delta\) and \(\gamma\) will gradually reduce the value of each element as the training progresses. So _Grug_ may lose information only in the initial stage of training.

From the perspective of these 2 aspects, _Grug_ performs better than existing methods. Although _Grug_ may lose some information as other methods, it generates immense new sample distributions, thus maximizing the utilization of diverse information. More details of the derivation are shown in Appendix A.1 and supplementary experiments can be found in Appendix A.3.

## 5 Experiments

### Experimental Setup

In this section, we present the results of our experiments on five real-world datasets for node classification and link prediction tasks. Our experiments aim to answer the following questions: 1) How does _Grug_ compare to other gradient regularization methods on HGNNs in terms of performance? 2) Can _Grug_ improve the robustness of HGNNs? 3) Does _Grug_ perform better than other methods in addressing over-smoothing problems?

**Datasets** We employ five commonly used heterogeneous graph datasets in our experiments, which are _ACM_, _DBLP_, _IMDB_, _Amazon_ and _LastFM_.

_Node Classification Datasets._ Labels in node classification datasets are split according to 20% for training, 10% for validation and 70% for test in each dataset. Note that node classification datasets are the same version used in the GTN [41].

* **ACM** is a paper citation network, containing 3 types of nodes (papers (P), authors (A), subjects (S)), 4 types of edges (PS, SP, PA, AP).
* **DBLP** is a bibliography website of computer science. It contains 3 types of nodes (papers (P), authors (A), conferences (C)), 4 types of edges (PA, AP, PC, CP).
* **IMDB** is a website about movies and related information. It contains 3 types of nodes (movies (M), actors (A), and directors (D)) and labels are genres of movies.

_Link Prediction Datasets._ Edges in link prediction datasets are split according to 25% for training, 5% for validation and 60% for the test in each dataset. Note that link prediction datasets are subsets pre-processed by HGB dataset [11].
* **Amazon** is an online purchasing platform and it consists of electronics products with co-viewing and co-purchasing links between them.
* **LastFM** is an online music website. It contains 3 types of nodes (artist, user, tag) and 3 types of edges (artist-tag, user-artist, user-user).

**Baseline methods.** We compare our proposed _Grug_ with other existing gradient regularization methods, including Dropout (2012) [19], DropNode (2020) [21], DropMessage (2022) [22], and FLAG (2020) [28]. We adopt these methods on backbone models and compare their performances on different datasets.

**Backbone models.** In order to compare the differences of gradient regularization methods fairly, we choose the basic HGNN models to ensure that the experimental results are not affected by the difference of the models. So we consider two mainstream HGNNs as our backbone models: RGCN [4] and RGAT [5]. We implement the gradient regularization methods mentioned above on backbone models.

### Comparison Results

Table 2 and Table 1 summarize the overall results. For node classification tasks, we employ the Micro F1 score and Macro F1 score as Metrics on 3 datasets (ACM, DBLP, IMDB). For link prediction tasks, the performance is measured by AUC-ROC on Amazon and LastFM. Considering the space limitation, some additional experiments are presented in the Appendix A.3.

Overall, _Grug_ performs well in all scenarios (node classification and link prediction), indicating its stability in diverse situations. For node classification tasks, we have 72 settings, each of which is a combination of different backbone models and datasets in different metrics (e.g., RGCN-ACM-Macro F1). Table 1 shows that _Grug_ achieves the optimal results in all 72 settings. As to 24 settings under the link prediction task, _Grug_ achieves the optimal results in 23 settings, and sub-optimal results in 1 setting. Besides, _Grug_ performs more stable on all datasets and all tasks than other methods clearly. Taking Dropout as the counterexample, it appears strong performance on the DBLP dataset but shows a clear decrease on the ACM dataset and IMDB dataset. In terms of tasks, the performance of Dropout on link prediction is far worse than that on node classification. Dropout on node classification obtains an average accuracy decline of 2.97% compared to _Grug_. This numerical gap has widened to 7.69% on link prediction. A reasonable explanation is that the information dimensions operated by distinct methods vary from each other as shown in section 4.2. With the generalization of its operation on node and message

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & Dataset & ACM & DBLP & IMDB \\ \hline \multirow{8}{*}{**Grug**} & Method & Macro F1 & Macro F1 & Micro F1 & Macro F1 & Micro F1 & Macro F1 \\ \cline{2-8}  & Clean [4] & 90.36\(\pm\)0.60 & 90.44\(\pm\)0.59 & 93.67\(\pm\)0.15 & 92.77\(\pm\)0.16 & 56.05\(\pm\)1.84 & 55.43\(\pm\)1.64 \\  & \multirow{4}{*}{Dropout [19]} & 90.86\(\pm\)0.39 & 90.91\(\pm\)0.40 & 94.35\(\pm\)0.14 & 93.53\(\pm\)0.14 & 57.63\(\pm\)1.01 & 56.04\(\pm\)0.79 \\  & & DropNode [21] & 90.68\(\pm\)0.75 & 90.74\(\pm\)0.77 & 93.58\(\pm\)0.71 & 93.12\(\pm\)0.71 & 57.38\(\pm\)0.37 & 56.20\(\pm\)0.50 \\  & & DropMessage [22] & 91.57\(\pm\)0.19 & 91.57\(\pm\)0.21 & 94.69\(\pm\)0.13 & 93.96\(\pm\)0.13 & 57.60\(\pm\)0.84 & 56.48\(\pm\)0.49 \\ \cline{2-8}  & \multirow{2}{*}{FLAG [28]} & 92.43\(\pm\)0.37 & 92.58\(\pm\)0.34 & 94.22\(\pm\)0.20 & 93.99\(\pm\)0.24 & 56.74\(\pm\)0.33 & 55.63\(\pm\)0.40 \\  & & _Grug_ & **93.27\(\pm\)0.13** & **93.33\(\pm\)0.13** & **95.08\(\pm\)0.08** & **94.17\(\pm\)0.11** & **59.92\(\pm\)0.56** & **58.87\(\pm\)0.71** \\ \hline \multirow{8}{*}{**Grug**} & Clean [4] & 89.32\(\pm\)0.13 & 89.34\(\pm\)0.13 & 92.98\(\pm\)0.61 & 92.15\(\pm\)0.63 & 53.12\(\pm\)0.80 & 27.55\(\pm\)0.78 \\  & & Dropout [19] & 89.38\(\pm\)0.81 & 89.51\(\pm\)0.76 & 93.55\(\pm\)0.48 & 92.69\(\pm\)0.50 & 54.85\(\pm\)0.31 & 53.63\(\pm\)0.26 \\ \cline{1-1}  & & DropNode [21] & 89.86\(\pm\)0.67 & 89.98\(\pm\)0.60 & 93.21\(\pm\)0.67 & 92.42\(\pm\)0.72 & 53.86\(\pm\)1.56 & 52.80\(\pm\)1.60 \\ \cline{1-1}  & & DropMessage [22] & 90.57\(\pm\)0.62 & 90.57\(\pm\)0.62 & 93.89\(\pm\)0.35 & 93.01\(\pm\)0.39 & 56.48\(\pm\)0.76 & 55.29\(\pm\)0.78 \\ \cline{1-1} \cline{2-8}  & \multirow{2}{*}{FLAG [28]} & 91.46\(\pm\)0.59 & 91.53\(\pm\)0.58 & 93.38\(\pm\)0.45 & 92.42\(\pm\)0.35 & 56.09\(\pm\)0.68 & 54.47\(\pm\)0.66 \\ \cline{1-1}  & & **Grug** & **92.84\(\pm\)0.18** & **92.92\(\pm\)0.19** & **94.21\(\pm\)0.21** & **93.50\(\pm\)0.22** & **57.79\(\pm\)0.37** & **56.17\(\pm\)0.18** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison results of different gradient regularization methods for node classification. The best results are in bold.

\begin{table}
\begin{tabular}{c c c c} \hline \hline  & Dataset & Amazon & LastFM \\ \hline \multirow{8}{*}{**Grug**} & Method & AUC-ROC & AUC-ROC \\ \cline{2-4}  & Clean [4] & 70.54\(\pm\)0.00 & 56.66\(\pm\)5.92 \\  & Dropout [19] & 72.83\(\pm\)5.17 & 56.95\(\pm\)2.37 \\  & DropNode [21] & 70.81\(\pm\)3.85 & 56.97\(\pm\)3.86 \\  & DropMessage [22] & **81.80\(\pm\)0.57** & 58.01\(\pm\)4.65 \\ \cline{2-4}  & FLAG [28] & 78.87\(\pm\)0.81 & 56.96\(\pm\)3.42 \\  & _Grug_ & 80.93\(\pm\)0.67 & **60.37\(\pm\)1.24** \\ \hline \multirow{8}{*}{**Grug**} & Clean[4] & 81.98\(\pm\)2.36 & 78.09\(\pm\)3.97 \\  & Dropout [19] & 81.74\(\pm\)6.72 & 78.94\(\pm\)3.43 \\ \cline{1-1}  & DropNode [21] & 83.36\(\pm\)0.77 & 78.10\(\pm\)2.85 \\ \cline{1-1}  & DropMessage [22] & 85.16\(\pm\)0.35 & 78.50\(\pm\)0.52 \\ \cline{1-1} \cline{2-4}  & FLAG [28] & 85.59\(\pm\)0.25 & 78.66\(\pm\)1.86 \\ \cline{1-1}  & **Grug** & **86.84\(\pm\)0.26** & **80.39\(\pm\)2.24** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison results of different gradient regularization methods for link prediction. The best results are in bold.
dimensions, _DropMessage_ obtains a smaller inductive bias. Therefore, _Grug_ is more applicable for most scenarios.

### Additional Results

**Over Smoothing analysis.** In this section, we investigate the issue of over-smoothing in HGNNs, which occurs when node representations become indistinguishable as the depth of the network increases, leading to poor results. To address this problem, we compare the effectiveness of different gradient regularization methods, taking into account the information diversity discussed in Section 4.2. As mentioned in section 4.2, _Grug_ obtaining more diverse information in the training process, it is expected to perform better holdout over-smoothing in theory. To verify this, we conduct experiments on two datasets, ACM and IMDB, using RGCN as the backbone model. As Figure 2 suggests, we can find that all gradient regularization methods can alleviate over-smoothing and _Grug_) outperforming other methods due to their diverse information. Our proposed _Grug_ exhibits consistent superiority in performance across all layers (from layer 1 to layer 7). as shown in Figure 2.

**Robustness analysis.** We evaluate the robustness of different gradient regularization methods by measuring their ability to deal with perturbed heterogeneous graphs. In more detail, we randomly add a certain ratio of edges on ACM and DBLP datasets and address the node classification task. It is found in Figure 3 that _Grug_ has positive effects and results drop the least when the perturbation rate increase from 0.1 to 0.4. Besides, our proposed _Grug_ shows its stability and outperforms other gradient regularization methods in noisy situations.

## 6 Conclusion

In this paper, we propose _Grug_, a general and effective gradient regularization method for HGNNs. We first unify existing gradient regularization methods to our framework via performing regularization

Figure 3: Model Performances Against Attacks.

Figure 2: Over-Smoothing Analysis.
according to the gradient on the feature matrix and message matrix. As _Grug_ operates meticulously on feature and message matrix, it is a generalization of existing methods, which shows greater applicability in general cases. Besides, we provide the theoretical analysis of the superiority (stability, simplicity and diversity) and effectiveness of _Grug_. By conducting sufficient experiments for multiple tasks on five datasets, we show that _Grug_ surpasses the SOTA performance and alleviates over-smoothing, and non-robustness obviously compared to the traditional gradient regularization methods.
