Causal-Based Supervision of Attention in Graph Neural Network: A Better
  and Simpler Choice towards Powerful Attention
Recent years have witnessed the great potential of attention mechanism in
graph representation learning. However, while variants of attention-based GNNs
are setting new benchmarks for numerous real-world datasets, recent works have
pointed out that their induced attentions are less robust and generalizable
against noisy graphs due to lack of direct supervision. In this paper, we
present a new framework which utilizes the tool of causality to provide a
powerful supervision signal for the learning process of attention functions.
Specifically, we estimate the direct causal effect of attention to the final
prediction, and then maximize such effect to guide attention attending to more
meaningful neighbors. Our method can serve as a plug-and-play module for any
canonical attention-based GNNs in an end-to-end fashion. Extensive experiments
on a wide range of benchmark datasets illustrated that, by directly supervising
attention functions, the model is able to converge faster with a clearer
decision boundary, and thus yields better performances.
Hongjun Wang, Jiyuan Chen, Lun Du, Qiang Fu, Shi Han, Xuan Song
# Causal-Based Supervision of Attention in Graph Neural Network:

###### Abstract

Recent years have witnessed the great potential of attention mechanism in graph representation learning. However, while variants of attention-based GNNs are setting new benchmarks for numerous real-world datasets, recent works have pointed out that their induced attentions are less robust and generalizable against noisy graphs due to lack of direct supervision. In this paper, we present a new framework which utilizes the tool of causality to provide a powerful supervision signal for the learning process of attention functions. Specifically, we estimate the direct causal effect of attention to the final prediction, and then maximize such effect to guide attention attending to more meaningful neighbors. Our method can serve as a plug-and-play module for any canonical attention-based GNNs in an end-to-end fashion. Extensive experiments on a wide range of benchmark datasets illustrated that, by directly supervising attention functions, the model is able to converge faster with a clearer decision boundary, and thus yields better performances.

## 1 Introduction

Graph-structured data is widely used in real-world domains, such as social networks [14], recommender systems [23], and biological molecules [16]. The non-euclidean nature of graphs has inspired a new type of machine learning model, Graph Neural Networks (GNNs) [15, 16, 17]. Generally, GNN iteratively updates features of the center node by aggregating those of its neighbors and has achieved remarkable success across various graph analytical tasks. However, the aggregation of features between unrelated nodes has long been an obstacle for GNN, keeping it from further improvement.

Recently, Graph Attention Network (GAT) [21] pioneered the adoption of the attention mechanism, a well-established method with proven effectiveness in deep learning [22], into the neighborhood aggregation process of GNNs to alleviate the issue. The key concept behind GAT is to adaptively assign importance to each neighbor during the aggregation process. Its simplicity and effectiveness have made it the most widely used variant of GNN. Following this line, a myriad of attention-based GNNs have been proposed and have achieved state-of-the-art performance in various tasks [24, 14, 15, 16].

Nevertheless, despite the widespread use and satisfying results, in the past several years, researchers began to rethink if the learned attention functions are truly effective [23, 25, 26, 27, 28, 29]. As we know, most existing attention-based GNNs learn the attention function in a weakly-supervised manner, where the attention modules are simply supervised by the final loss function, without a powerful supervising signal to guide the training process. And the lack of direct supervision on attention might be a potential cause of a less robust and generalizable attention function against real-world noisy graphs [23, 24, 25, 26, 27, 28, 29, 12]. To address this problem, existing work enhances the quality of attention through auxiliary regularization terms (supervision). However, concerns have been raised that these methods often rely heavily on human-specified prior assumptions about a specific task, which limits their generalizability [26, 27]. Additionally, the auxiliary regularization is formulated independently of the primary prediction task, which may disrupt the original optimization target and cause the model to "switch" to a different objective function during training [23, 26].

Recently, causal inference [20] has attracted many researchers in the field of GNNs by utilizing structural causal model (SCM) [28] to handle distribution shift [29] and shortcut learning [14]. In this paper, we argue that the tool of causal inference has also shed light on a promising avenue that could supervise and improve the quality of GNN's attention directly, while in the meantime we will not make any assumptions about specific tasks or models, and the supervision signal for attention implicitly aligns well with the primary task. Before going any deeper, we first provide a general schema for the SCM of attention-based GNNs in Figure 1, which uses nodes to represent variables and edges to indicate causal relations between variables. As we can see, after a high-level abstraction, there are only three key factors in SCM, including the node
features \(X\), attention maps \(A\), and the model's final prediction \(Y\). Note that in causal language, \(X\), \(A\), and \(Y\) also denotes the context, treatment, and outcome respectively. For edges in SCM, the link \(X\to A\) represents that the attention generation relies on the node's features (i.e., context decides treatment). And links \((X,A)\to Y\) indicate that the model's final prediction is based on both the node's features \(X\) and the attention \(A\) (i.e., the final outcome is jointly determined by both context and treatment).

In order to provide a direct supervision signal and further enhance the learning of attention functions, the first step would be finding a way to measure the quality of attention (i.e., quantifying what to improve). Since there are no unified criteria on the way of measurement, researchers usually propose their own solution according to the tasks they are facing, and this is very likely to introduce the unfavorable human-intervened prior assumptions [20]. For example, CGAT [21] believes that better attention should focus more on one-hop neighbors. While this assumption surely works on homophilous graphs, it will suffer from huge performance degradation in heterophilous scenarios. Our method differs a lot from existing work in that we introduce SCM to effectively decouple the direct causal effect of attention on the final prediction (i.e., link \(A\to Y\)), and use such causal effect as a measurement for the quality of attention. In this way, it is the model and data that decide if the attention works well during training instead of human-predefined rules. And this has been shown to be non-trivial in various machine learning fields because what might seem reasonable to a human might not be considered the same way by the model [14, 21]. Another drawback of existing attention regularization methods, as previously mentioned, is the deviation from primary tasks. SuperGAT [14] uses link prediction to improve the attention quality for node classification, but as the author claims in the paper, there is an obvious trade-off between the two tasks. In this paper, we alleviate this problem by directly maximizing the causal effect of attention on the primary task (i.e., strengthening the causal relation \(A\to Y\)). Under mild conditions, we can deem the overall optimization is still towards the primary objective, except that we additionally provide a direct and powerful signal for the learning of attention in a fully-supervised manner.

In summary, this paper presents a **C**ausal **S**upervision for **A**ttention in graph neural networks (abbreviated as **CSA** in the following paragraphs). CSA has strong applicability because no human-intervened assumptions are made on the target models or training tasks. And the supervision of CSA can be easily and smoothly integrated into optimizing the primary task to performing end-to-end training. We list the main contributions in this paper as follows:

* We explore and provide a brand-new perspective to directly boost GNN's attention with the tool of causality. To the best of our knowledge, this is a promising direction that still remains unexplored.
* We propose CSA, a novel causal-based supervision framework for attention in GNNs, which can be formulated as a simple yet effective external plug-in for a wide range of models and tasks to improve their attention quality.
* We perform extensive experiments and analysis on CSA and the universal performance gain on standard benchmark datasets validates the effectiveness of our design.

## 2 Related Work

**Attention-based Graph Neural Networks.** Modeling pairwise importance between elements in graph-structured data dates back to interaction networks [1, 17] and relational networks [16]. Recently GAT [13] rose as one of the representative work of attention-based GNNs using self-attention [16]. The remarkable success of GAT in multiple tasks has motivated many works focusing on integrating attention into GNN [15, 14, 21, 22, 23, 24]. _Lee et al._ have also conducted a comprehensive survey [11] on various types of attention used in GNNs.

**Causal Inference in Graph Neural Network.** Causality [1] provides researchers new methodologies to design robust measurements, discover hidden causal structures and confront data biases. A myriad of studies has shown that incorporating causality is beneficial to graph neural network in various tasks. [23] makes use of counterfactual links to augment data for link prediction improvement. [25] performs interventions on the representations of graph data to identify the causally attended subgraph for graph classification. [10] on the other hand, applies causality to estimate the causal effect of node's local structure to assist node classification.

**Improving Attention in GAT.** There is a great number of work dedicated to improving attention learning in GAT. [14] enhances attention by exploiting two attention forms compatible with a self-supervised task to predict edges. [1] introduces a simple fix by modifying the order of operations in GAT. [21] develops an approach using constraint on the attention weights according to the class boundary and feature aggregation pattern. In addition, causality also plays a role in boosting the attention of GATs recently. [21] estimates the causal effect of edges by intervention and regularizes edges' attention weights according to their causal effects.

## 3 Preliminaries

We start by introducing the notations and formulations of graph neural networks and their attention variant. Let \(\mathcal{G}=\{\mathcal{V},\mathcal{E}\}\) represents a graph where \(\mathcal{V}=\{v_{i}\}_{i=0}^{n}\) is the set of nodes and \(\mathcal{E}\in\mathcal{V}\times\mathcal{V}\) is the set of edges. For each node \(v\in\mathcal{V}\),

Figure 1: Structural causal model of attention-based GNNs
it has its own neighbor set \(N(v)=\{u\in\mathcal{V}\mid(v,u)\in\mathcal{E}\}\) and its initial feature vector \(x_{v}^{0}\in\mathbb{R}^{d^{0}}\), where \(d^{0}\) is the original feature dimension. Generally, GNN follows the message-passing mechanism to perform feature updating, where each node's feature representation is updated by aggregating the representations of its neighbors and then combining the aggregated messages with its ego representation [20]. Let \(m_{v}^{l}\in\mathbb{R}^{d^{l}}\) and \(x_{v}^{l}\in\mathbb{R}^{d}\) be the message vector and representation vector of node \(v\) at layer \(l\), we formally define the updating process of GNN as:

\[m_{v}^{l} =\mathrm{AGGREGATE}\left(\left\{x_{j}^{(l-1)},\forall j\in N(v) \right\}\right)\] \[x_{v}^{l} =\mathrm{COMBINE}\left(x_{v}^{l-1},m_{v}^{l}\right),\]

where \(\mathrm{AGGREGATE}\) and \(\mathrm{COMBINE}\) are aggregation functions (e.g., mean, LSTM) and combination function (e.g., concatenation), respectively. The design of these two functions is what most distinguishes one type of GNN from the other. GAT [21] augments the normal aggregation with the introduction of self-attention. The core idea of self-attention in GAT is to learn a scoring function that computes an attention score for every node in \(N(v)\) to indicate their relational importance to node \(v\). In layer \(l\), such process is defined by the following equation:

\[e\left(x_{v_{i}}^{l},x_{v_{j}}^{l}\right)=\sigma\left((\mathbf{a}^{l})^{\top} \cdot\left[W^{l}\;x_{v_{i}}^{l}\|W^{l}\;x_{v_{j}}^{l}\right]\right),\]

where (\(\mathbf{a}^{l},W^{l}\)), \(\sigma\) are learnable matrices and activation function (e.g., LeakyReLU) respectively, and \(\|\) denotes vector concatenation. The attention scores are then normalized across all neighbors \(v_{j}\in N(v_{i})\) using softmax to ensure consistency:

\[\alpha_{ij}^{l}=\frac{\exp\left(e\left(x_{v_{i}}^{l},x_{v_{j}}^{l}\right) \right)}{\sum_{v_{j}\in N(v_{i})}\exp\left(e\left(x_{v_{i}}^{l},x_{v_{j}}^{l} \right)\right)}\]

Finally, GAT computes a weighted average of the features of the neighboring nodes as the new feature of \(v_{i}\), which is demonstrated as follows:

\[x_{v_{i}}^{l+1}=\sigma\left(\sum\nolimits_{v_{j}\in N(v_{i})}\alpha_{ij}^{l} W^{l}\;x_{v_{j}}^{l}\right).\]

## 4 Casual-based Supervision on GNN's Attention

In this section, we first introduce how the causal effect of attention can be derived from the structural causal model of attention-based GNNs. Specifically, this is done with the help of the widely used counterfactual analysis in causal reasoning. After that, with the obtained causal effects, we elaborate three candidate schemes to incorporate with the training of attention-based GNNs to improve their quality of attention.

### Causal Effects of Attention

As previously mentioned, the first step towards improving attention lies in measuring the quality of existing attention. However, since deep learning models usually exhibit as black boxes, it is generally infeasible to directly assess their attention qualities. Existing works mainly address this issue by introducing human priors to build pre-defined rules for some specific models and tasks. Yet, it has been a long debate on whether human-made rules share consensus with deep learning models during training [21, 19]. Fortunately, the recent rise of causal inference technology has offered effective tools to help us think beyond the black box and analyze causalities between model variables, which leads us to an alternative way to directly utilize the causal effect of attention to measure its quality. Since the obtained causal effects are mainly affected by the model itself, it is a more accurate and unbiased measurement of how well the attention actually learns.

We first give a brief review of the formulation of attention-based graph neural network in causal languages, as shown in Figure 2(a). The generated attention map \(A\) is directly affected by node feature \(X\). And the model prediction \(Y\) is jointly determined by both \(X\) and \(A\). We denote the inferring process of the model as:

\[Y_{x,a}=Y(X=x,A=a), \tag{1}\]

which indicates that model will give value \(Y_{x,a}\) if the value of \(X\) and \(A\) are set to \(x\) and \(a\) respectively. In order to pursue the attention's causal effect, we introduce the widely-used counterfactual analysis [10] in causal reasoning.

The core idea of counterfactual causality lies in asking: given a certain data context (node feature \(X\)), what the outcome (model prediction \(Y\)) would have been if the treatment (attention map \(A\)) had not been the observed value? To answer the imaginary question, we have to manipulate the values of several variables to see the effect, and this is formally termed as _intervention_ in causal inference literature, which can be denoted as \(do(\cdot)\). In \(do(\cdot)\) operation, we compulsively select a counterfactual value to replace the original factual value of the intervened variable. And once a variable is intervened, its all incoming links in the SCM will be cut off and its value is independently given, while other variables that are not affected still maintain the original value. In our case, for example, \(do(A=a^{*})\) means we demand the attention \(A\) to take the non-real value \(a^{*}\) (e.g., reverse/random attention) so that the link \(X\to A\) is cut-off and \(A\) is no longer be affected by its causal parent \(X\). This process is illustrated in Figure 2(b) and the mathematical formulation is given as:

\[Y_{x,a^{*}}=Y(X=x,do(A=a^{*})), \tag{2}\]

which indicates that after \(do(\cdot)\) operation which changes the value of attention to be \(a^{*}\), the output value of the model also changes to \(Y_{x,a^{*}}\). Finally, let us consider a case where we

Figure 2: Deriving causal effects through counterfactual
assign a _dummy_ value \(\tilde{a}\) to the attention map so that for each ego node, all its neighbors share the same attention weights, the feature aggregation of the graph attention model will then degrade to an unweighted average. In this case, according to the theory of causal inferences [20], the Total Direct Effect (TDE) of attention to model prediction can be obtained by computing the differences of model outcome \(Y_{x,a}\) and \(Y_{x,\tilde{a}}\), which is formulated as follows:

\[TDE=Y_{x,a}-Y_{x,\tilde{a}}. \tag{3}\]

It is worth noting that the induction of attention's causal effect does not exert any assumptions and constraints, which lays a solid foundation for our wide applicability on any graph attention models.

### Supervision on Attention with Causal Effects

We have already demonstrated the derivation of attention's causal effect in the previous section. In this part, we will discuss how to utilize the obtained causal effect for attention quality improvement. Previous works that make use of an auxiliary task to regularize attention usually suffered from performance trade-off between the primary task and auxiliary task. In this work, we alleviate this problem by directly maximizing the causal effect of attention on the primary task. The overall schema of this part is shown in Figure 3.

Consider a simple case where we conduct the node classification task with a standard \(L\)-layer GAT. For each layer \(l\), we have the node representations \(X^{l-1}\in\mathbb{R}^{n\times d^{l-1}}\) from the previous layer as input. Then, we perform feature aggregation and updating with factual attention map \(A^{l}\) to obtain the factual output feature \(X^{l}=f(X^{l-1},A^{l})\). Similarly, when we intervene the attention maps of layer \(l\) (e.g., assigning _dummy_ values using \(do(\cdot)\) operation), we can get a counterfactual output feature \(\hat{X}^{l}\). We further employ a learnable matrix \(\mathcal{W}^{l}\in\mathbb{R}^{c\times d^{l}}\) (\(c\) denotes the number of classes) to get the node's factual predicted label \(Y^{l}_{pred}\) and counterfactual predicted label \(\hat{Y}^{l}_{pred}\) using the corresponding features from layer \(l\). Therefore, the causal effect of attention at layer \(l\) is obtained as: \(Y^{l}_{pred}-\hat{Y}^{l}_{pred}\). To this end, we can use the causal effect as a supervision signal to explicitly guide the attention learning process. The new objective of the CSA-assisted GAT model can be formulated as:

\[\mathcal{L}\!\!=\!\!\!\sum_{l}\lambda_{l}\mathcal{L}_{ce}(Y^{l}_{\text{effect }},y)\!+\!\mathcal{L}_{\text{others}}, \tag{4}\]

where \(y\) is the ground-truth label, \(\mathcal{L}_{ce}\) is the cross-entropy loss, \(\lambda_{l}\) is the coefficient to balance training, and \(\mathcal{L}_{\text{others}}\) represents the original objective such as standard classification loss. Note that Equation.(4) is a general form of CSA where we compute additional losses for each GAT layer to supervise attention directly. However in practice it is not necessary, and we found that simply selecting one or two layers is enough for CSA to bring satisfying performance improvement.

Moreover, since our aim is to boost the quality of attention, it is not necessary to estimate the correct causal effect of attention using _dummy_ values. Instead, a strong counterfactual baseline might even be helpful for the attention quality improvement. We hereby further propose three heuristic counterfactual schemes and test them in our experiments. We note that the exact form of how counterfactual is achieved is not limited, and our goal here is just to set the ball rolling.

**Scheme I:** In the first scheme, we utilize the uniform distribution to generate the counterfactual attention map. Specifically, the counterfactual attention is produced by

\[\hat{a}\sim U(e,f), \tag{5}\]

where \(e\) and \(f\) are the lower and upper boundaries. In this case, the generated counterfactual could vary from very bad (i.e., mapping all unrelated neighbors) to very good (i.e., mapping all meaningful neighbors). This is a similar process to the _Randomized Controlled Trial_[17] where all possible treatments are enumerated. We hope that maximizing causal effects computed over all possible treatments can lead to a robust improvement of attention.

**Scheme II:** Scheme I is easy and straightforward to apply. However due to its randomness, a possible concern is that if most of the generated counterfactual attentions are inferior to the factual one, then we will only have very small gradient on attention improvements. Therefore we are actually motivated to find "better" counterfactuals to spur the factual one to evolve. Heuristically, given that MLP is a strong baseline on several datasets (e.g., Texas, Cornell, and Wisconsin), we employ an identical mapping to generate the counterfactual attention, which only attends to the ego-node instead of neighbors. Specifically, the counterfactual attention map is equal to the identity matrix \(I\):

\[\hat{a}\sim I \tag{6}\]

**Scheme III:** Our last schema can be considered as an extension of Schema II. Since the fast development of GAT family has introduced to us some variants that already outperform MLP in many datasets, using counterfactuals derived from

Figure 3: The schematic of CSA is shown above as a plug-in to graph attention methods. The \(a\) and \(\hat{a}\) indicate the factual and counterfactual attention values, respectively. We subtract the counterfactual classification results from the original classification to analyze the causal effects of learned attention (i.e., attention quality) and directly maximize them in the training process towards primary task.
the behavior of MLP does not seem to be a wise choice for these GAT variants to improve their attentions. Inspired by the self-boosting concept [14] widely used in machine learning, we leverage the historical attention map as the counterfactual to urge the factual one keep refining itself. The specific formulation is written as follows:

\[\hat{a}\sim A_{hist}, \tag{7}\]

where \(A_{hist}\) denotes the historical attention map (e.g., the attention map from the last update iteration).

## 5 Experiment

In this section, we conduct extensive node classification experiments to evaluate the performance of CSA. Specifically, we 1) validate the effectiveness of CSA on three popular GAT variants using a wide range of datasets, including both homophily and heterophily scenarios; 2) compare CSA with other attention improvement baselines of GAT to show the superiority of our method; 3) show that CSA can induce better attention that improve the robustness of GAT; 4) test CSA's sensitivity to hyper-parameters; 5) analyze the influences of CSA in feature space; and 6) examine the performances of some special cases of CSA.

### Datasets

For heterophily scenario, we select seven standard benchmark datasets: Wisconsin, Cornell, Texas, Actor, Squirrel, Chameleon, and Crocodile. Wisconsin, Cornell, and Texas collected by Carnegie Mellon University are published in WebKB1. Actor [14] is the actor-only subgraph sampling from a film-director-actor-writer network. Squirrel, Chameleon, and Crocodile are datasets collected from the English Wikipedia [11]. We summarize the basic attributions for each dataset in Table 1. \(\mathcal{H}(\mathcal{G})\) is the homophily ratio [15], where \(\mathcal{H}(\mathcal{G})\to 1\) represents extreme homophily and vice versa.

Footnote 1: [http://www.cs.cmu.edu/](http://www.cs.cmu.edu/) webkb/

For homophily scenario, two large datasets released by Open Graph Benchmark (OGB)2[13]: ogbn-products and ogbn-arxiv, are included in our experiments, together with two small-scale homophily graph datasets: Cora and Citeseer [12]. Similarly, the attribution of the dataset is summarized in Table 2.

Footnote 2: [https://ogb.stanford.edu/](https://ogb.stanford.edu/)

### Experimental Setup

We employ popular node classification models in our experiments as the baselines: GCN [11], GAT [13], SGCN [21], FAGCN [10], GPR-GNN [20], H2GCN [15], WRGAT [23], APPNP [22] and UniMP [24]. We also present the performance of MLPs, serving as a strong non-graph-based baseline. Due to page limit, we only select four models: GAT, FAGCN, WRGAT and UniMP to examine the effectiveness of CSA. These models ranges from the classic ones to the latest ones, and are considered as representatives for state-of-the-art node classification models. One thing to be noted here is that for all these models, we implement CSA only in their first layers to avoid excessive computational cost.

In our experiments, each GNN is run with the best hyperparameters if provided. We set the same random seed for each

\begin{table}
\begin{tabular}{l c c c c c c c}  & **Texas** & **Wisconsin** & **Actor** & **Squirrel** & **Chameleon** & **Cornell** & **Crocodile** \\ \(\mathcal{H}(\mathcal{G})\) & **0.11** & **0.21** & **0.22** & **0.22** & **0.23** & **0.3** & **0.26** \\
**\#Nodes** & 183 & 251 & _7,600_ & _5,201_ & _2,277_ & 183 & 11,631 \\
**\#Edges** & 295 & 466 & 191,506 & 198,493 & 31,421 & 280 & 899,756 \\
**\#Classes** & 5 & 5 & 5 & 5 & 5 & 5 & 6 \\
**\#Features** & 1703 & 1703 & 932 & 2089 & 2325 & 1703 & 500 \\ \hline MLP & 81.32 \(\pm\) 6.22 & 84.38 \(\pm\) 5.34 & 36.09 \(\pm\) 1.35 & 28.98 \(\pm\) 1.32 & 46.21 \(\pm\) 2.89 & 83.92 \(\pm\) 5.88 & 54.35 \(\pm\) 1.90 \\ SGCN & 56.41 \(\pm\) 4.29 & 54.82 \(\pm\) 3.63 & 30.50 \(\pm\) 0.94 & 52.74 \(\pm\) 1.58 & 60.89 \(\pm\) 2.21 & 62.52 \(\pm\) 5.10 & 51.80 \(\pm\) 1.53 \\ GCN & 55.59 \(\pm\) 5.96 & 53.48 \(\pm\) 4.75 & 28.40 \(\pm\) 0.88 & 53.98 \(\pm\) 1.53 & 61.54 \(\pm\) 2.59 & 60.01 \(\pm\) 5.67 & 52.24 \(\pm\) 2.54 \\ H2GCN & 84.81 \(\pm\) 6.94 & 86.64 \(\pm\) 4.63 & 35.83 \(\pm\) 0.96 & 37.95 \(\pm\) 1.89 & 58.27 \(\pm\) 2.63 & 82.08 \(\pm\) 4.71 & 53.10 \(\pm\) 1.23 \\ APPNP & 81.93 \(\pm\) 5.77 & 85.48 \(\pm\) 4.58 & 35.90 \(\pm\) 0.96 & 39.08 \(\pm\) 1.76 & 57.80 \(\pm\) 2.47 & 81.92 \(\pm\) 6.12 & 53.06 \(\pm\) 1.90 \\ GPR-GNN & 79.44 \(\pm\) 5.17 & 84.46 \(\pm\) 6.36 & 35.11 \(\pm\) 0.82 & 32.33 \(\pm\) 2.42 & 46.76 \(\pm\) 2.10 & 79.91 \(\pm\) 6.60 & 52.74 \(\pm\) 1.88 \\ \hline GAT & 55.21 \(\pm\) 5.70 & 52.80 \(\pm\) 6.11 & 29.04 \(\pm\) 0.66 & 40.00 \(\pm\) 0.99 & 59.32 \(\pm\) 1.54 & 61.89 \(\pm\) 6.08 & 51.28 \(\pm\) 1.79 \\ +CSA-I & 56.17 \(\pm\) 5.32 & 53.23 \(\pm\) 6.28 & 29.03 \(\pm\) 0.79 & 40.51 \(\pm\) 0.98 & 60.73 \(\pm\) 1.35 & 62.75 \(\pm\) 6.32 & 51.67 \(\pm\) 1.62 \\ +CSA-II & **58.21 \(\pm\) 4.79** & **54.35 \(\pm\) 6.54** & 29.71 \(\pm\) 0.74 & 41.02 \(\pm\) 1.23 & **61.31 \(\pm\) 1.13** & **64.26 \(\pm\) 5.21** & **52.20 \(\pm\) 1.74** \\ +CSA-III & 58.04 \(\pm\) 5.27 & 53.98 \(\pm\) 6.30 & **29.72 \(\pm\) 0.86** & **41.38 \(\pm\) 1.19** & 61.20 \(\pm\) 1.37 & 63.58 \(\pm\) 6.03 & 52.13 \(\pm\) 1.83 \\ \hline FAGCN & 82.54 \(\pm\) 6.89 & 82.84 \(\pm\) 7.95 & 34.85 \(\pm\) 1.24 & 42.55 \(\pm\) 0.86 & 61.21 \(\pm\) 3.13 & 79.24 \(\pm\) 9.92 & 54.35 \(\pm\) 1.11 \\ +CSA-I & 82.65 \(\pm\) 7.11 & 83.37 \(\pm\) 7.79 & 34.77 \(\pm\) 0.95 & 42.55 \(\pm\) 0.74 & 61.86 \(\pm\) 2.98 & 80.01 \(\pm\) 9.72 & 54.44 \(\pm\) 1.18 \\ +CSA-II & 83.29 \(\pm\) 6.80 & 83.11 \(\pm\) 8.26 & 34.88 \(\pm\) 0.86 & 42.58 \(\pm\) 0.93 & 61.74 \(\pm\) 3.39 & **81.35 \(\pm\) 9.68** & 54.45 \(\pm\) 1.23 \\ +CSA-III & **84.72 \(\pm\) 6.71** & **84.23 \(\pm\) 7.21** & **35.12 \(\pm\) 0.98** & **43.38 \(\pm\) 1.02** & **62.52 \(\pm\) 3.20** & 80.94 \(\pm\) 9.77 & **55.16 \(\pm\) 0.97** \\ \hline WRGAT & 83.62 \(\pm\) 5.50 & 86.98 \(\pm\) 3.78 & 36.53 \(\pm\) 0.77 & 48.85 \(\pm\) 0.78 & 65.24 \(\pm\) 0.87 & 81.62 \(\pm\) 3.90 & 54.76 \(\pm\) 1.12 \\ +CSA-I & 83.69 \(\pm\) 5.63 & 87.23 \(\pm\) 3.94 & 36.55 \(\pm\) 0.93 & **49.46 \(\pm\) 0.74** & 65.36 \(\pm\) 1.05 & 81.88 \(\pm\) 3.93 & 54.86 \(\pm\) 1.31 \\ +CSA-II & 83.76 \(\pm\)
model and dataset for reproducibility. The reported results are in the form of mean and standard deviation, calculated from 10 random node splits (the ratio of train/validation/test is 48%/32%/20% from [20]). Our experiments are conducted on a GPU server with eight NVIDIA DGX A100 graphics cards, and the codes are implemented using Cuda Toolkit 11.5, PyTorch 1.8.1 and torch-geometric 2.0.1.

### Performance Analysis

Table 1 and Table 2 provide the test accuracy of different GNNs in different variants of CSA in the supervised node classification task. A graph's homophily level is the average of its nodes' homophily levels. CSA achieves the best in terms of the vanilla one across all datasets. In particular, the highest improved datasets are Texas, Wisconsin, and Cornell. By observing the performance of MLPs, we can see that the common ground of those three datasets contains distinguishable features and a large part of non-homophilous edges. In the meanwhile, the performance of CSA is proportional to the modeling ability. The mechanism behind CSA is to extend the causal effect between nodes representation and final prediction. Therefore, CSA owns limited performance when the node's representations are chaotic. Our experiments highlight that I) The model, which is already better than MLP, does not improve much in CSA-II, while CSA-III improves it relatively more. This is because in those datasets, the graph structure can provide meaningful information, so that CSA-III have more advantages. II) The dataset, which has distinctive features indicated by the performance of MLPs, is more satisfied CSA-II. Similarly, in this scene, the features can be more informative. III) The random strategy (CSA-I) relatively inferior to others, since the distribution is hard to control and tend to generate worst attention map, whereby the regularization is vanished.

### Comparison with Attention Promotion Baselines

We here apply multiple attention promotion baselines: CAL [14], CAR [23], Super [15], Constraint interventions [23] and the result is shown in Figure 4. Among them, CAL is a method for graph property prediction that relies on an alternative formulation of causal attention with interventions on implicit representations. We adapted CAL for node classification by removing its final pooling layer. Super is well-known as SuperGAT, a method seeking to constrain node feature vectors through a semi-supervised edge prediction task. CAR aligns the attention mechanism with the causal effects of active interventions on graph connectivity in a scalable manner. Constraint method has two auxiliary losses: graph structure-based constraint and class boundary constraint. The results on four datasets are shown in Figure 4. While CAL, CAR, and CSA have related goals of enhancing graph attention using concepts from causal theory, CAL uses abstract perturbations on graph representation to perform causal interventions, and CAR employs an edge intervention strategy that enables causal effects to be computed scalable, while our method does not exert any assumptions and constraints on GATs, compared with CAL and CAR. Therefore, CSA tends to own good generalization ability. In terms of SuperGAT and Constraint method, since there is a trade-off between node classification and regularization. For example, SuperGAT implies that it is hard to learn the relational importance of edges by simply optimizing graph attention for link prediction.

### CSA Provides Robustness

In this section, we systematically study the performance of CSA on robustness against the input perturbations including feature and edge perturbations. Following [21], we conduct node-level feature perturbations by replac

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline Models & **Cora** & **CiteSeer** & **ogbn-products** & **ogbn-arxiv** \\ \(\mathcal{H}(\beta)\) & 0.81 & 0.74 & 0.81 & 0.66 \\
**\#Nodes** & 2.708 & 3,327 & 2,449,029 & 169,343 \\
**\#Edges** & 5,278 & 4,467 & 61,859,140 & 1,166,243 \\
**\#Classes** & 7 & 7 & 7 & 40 \\
**\#Features** & 1433 & 3703 & 100 & 128 \\ \hline GAT & 86.21 \(+\) 0.78 & 75.73 \(+\) 1.23 & 77.02 \(+\) 0.63 & 70.96 \(+\) 0.14 \\ +CSA-I & 86.16 \(+\) 0.95 & 76.81 \(+\) 1.29 & 77.28 \(+\) 0.69 & 71.08 \(+\) 0.14 \\ +CSA-II & 86.89 \(+\) 0.64 & 76.53 \(+\) 1.18 & 77.44 \(+\) 0.63 & 71.05 \(+\) 0.14 \\ +CSA-III & **87.86 \(+\) 0.87** & **77.72 \(+\) 1.25** & **78.36 \(+\) 0.72** & **71.20 \(+\) 0.16** \\ \hline UnifP & 86.89 \(+\) 0.90 & 75.14 \(+\) 0.68 & 81.37 \(+\) 0.47 & 72.92 \(+\) 0.10 \\ +CSA-I & 87.47 \(+\) 0.87 & 75.89 \(+\) 0.73 & 81.55 \(+\) 0.62 & 72.94 \(+\) 0.10 \\ +CSA-II & 85.62 \(+\) 0.73 & 75.87 \(+\) 0.72 & 81.39 \(+\) 0.47 & 72.96 \(+\) 0.10 \\ +CSA-II & **88.64 \(+\) 1.28** & **77.61 \(+\) 0.82** & **82.24 \(+\) 0.63** & **73.08 \(+\) 0.11** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Classification accuracy on homophily datasets.

Figure 4: Comparison with different GATs promotion strategies.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline Models & **Texas** & **Cornell** & **ogbn-arxiv** \\ \hline GAT & \(55.21\pm 5.70\) & \(61.89\pm 6.08\) & \(70.96\pm 0.14\) \\ +CSA (Last) & \(57.83\pm 4.65\) & \(63.52\pm 5.34\) & \(71.08\pm 0.15\) \\ +CSA (Pure) & \(55.79\pm 5.05\) & \(61.97\pm 5.18\) & \(70.96\pm 0.14\) \\ +CSA (Ours) & **58.21 \(+\) 4.79** & **64.26 \(+\) 5.21** & **71.20 \(+\) 0.16** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison with the heuristic causal strategies.
ing them with the noise sampled from the Bernoulli distribution with \(p=0.5\) and edge perturbations by stochastically generating the edges. According to the performance shown in Figure 5, CSA produces robust performance on input perturbations. Figure 5 demonstrate that CSA in higher noise situations achieves more robust results than in lower noise scenes on both node and edge perturbations with perturbation percentages ranging from \(0\%\) to \(40\%\).

### Hyper-parameter Analysis

We analyze the sensitivity of \(\lambda\) and plot node classification performance in Figure 6. For \(\lambda\), there is a specific range that maximizes test performance in all datasets. The performance in Texas is the highest when \(\lambda\) is \(0.4\), but the difference is relatively small compared to Cora. We observe that there is an optimal level of causal supervision for each dataset, and using too large \(\lambda\) degrades node classification performance. Since Cora owns friendly neighbors, its performance is less sensitive than Texas. Based on this, we can also see that Texas relatively needs a larger regularization.

### Pairwise Distance among Classes

To further evaluate whether the good performance of CSA can be contributed to the mitigation of lacking supervision, we visualize the pairwise distance of the node representations among classes learned by CSA and vanilla GAT. Following [10], we calculate the Mean Average Distance (MAD) with cosine distance among node representations for the last layer. The larger the MAD is, the better the node representations are. Results are reported in Figure 7. It can be observed that the node representations learned by CSA keep having a large distance throughout the optimization process, indicating relief of lacking supervision issues. On the contrary, GAT suffers from severely indistinguishable representations of nodes.

### Comparison with Heuristic Causal Strategies.

To validate the effectiveness of CSA, we compare it with two heuristic causal designs (Last and Pure) that 1) directly estimate the total causal effect by subtracting between the model's and the counterfactual's output in the final layer; 2) replace the attention with the static aggregate weights (i.e., each node allocates the same weight). The results are shown in Table 3. We observe that their performance outperforms vanilla one, but is still inferior to ours. In terms of Last, the major difference is whether to explicitly estimate causal effect or not. In our framework, we plug the MLPs into the hidden layers to precisely estimate the causal effect for each layer. Regarding Pure, our strategy can provide more strong baselines, leading to better regularization.

## 6 Conclusion

We introduced CSA, a counterfactual-based regularization scheme that can be applied to graph attention architectures. Unlike other causal approaches, we first built the causal graph of GATs in a general way and do not impose any assumptions and constraints on GATs. Subsequently, we introduce an efficient scheme to directly estimate the causal effect of attention in hidden layers. Applying it to both homophilic and heterophilic node-classification tasks, we found accuracy improvements and robustness in almost all circumstances. We performed three variants of counterfactual attention strategies and found that they can adapt to different situations, respectively.

Figure 5: The robust performance on the node and edge perturbations.

Figure 6: Hyper-parameter analysis on GAT.

Figure 7: Mean Average Distance among node representations of Last GAT layer.
## Contribution Statement

Hongjun Wang and Jiyuan Chen contributed equally to the work. Lun Du and Xuan Song are the corresponding authors. This work is done during Hongjun Wang's internship at Microsoft Research Asia.
