Characterizing Massive Activations of Attention Mechanism in Graph Neural Networks
Lorenzo Bini Marco Sorbi Stephane Marchand-Maillet


: a novel deep learning model to detect single transit events with no prior data filtering in PLATO light curves
    
    H. G. Vivien1 ^https://orcid.org/0000-0001-7239-6700
    < g r a p h i c s > M. Deleuil1 ^https://orcid.org/0000-0001-6036-0225
    < g r a p h i c s > N. Jannsen2 ^https://orcid.org/0000-0003-4670-9616
    < g r a p h i c s > J. De Ridder2 ^https://orcid.org/0000-0001-6726-2863
    < g r a p h i c s > D. Seynaeve2 ^https://orcid.org/0000-0002-0731-8893
    < g r a p h i c s > M.-A. Carpine3 Y. Zerah1 ^https://orcid.org/0000-0003-1786-7367
    < g r a p h i c s >

    Received Month dd, yyyy; accepted Month dd, yyyy
=============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Graph Neural Networks (GNNs) have become increasingly popular for effectively modeling data with graph structures. Recently, attention mechanisms have been integrated into GNNs to improve their ability to capture complex patterns. This paper presents the first comprehensive study revealing a critical, unexplored consequence of this integration: the emergence of Massive Activations (MAs) within attention layers. We introduce a novel method for detecting and analyzing MAs, focusing on edge features in different graph transformer architectures. Our study assesses various GNN models using benchmark datasets, including ZINC, TOX21, and PROTEINS. Key contributions include (1) establishing the direct link between attention mechanisms and MAs generation in GNNs, (2) developing a robust definition and detection method for MAs based on activation ratio distributions, (3) introducing the Explicit Bias Term (EBT) as a potential countermeasure and exploring it as an adversarial framework to assess models robustness based on the presence or absence of MAs. Our findings highlight the prevalence and impact of attention-induced MAs across different architectures, such as GraphTransformer, GraphiT, and SAN. The study reveals the complex interplay between attention mechanisms, model architecture, dataset characteristics, and MAs emergence, providing crucial insights for developing more robust and reliable graph models.




     Codehttps://github.com/msorbi/gnn-ma






§ INTRODUCTION


Graph Neural Networks (GNNs) have emerged as a powerful tool for learning representations of graph-structured data, demonstrating remarkable success across various applications such as social network analysis <cit.>, recommendation systems <cit.> and molecular biology <cit.>. Central to the recent advancements in GNNs is the integration of attention mechanisms, which enable the models to focus on the most relevant parts of the input graph, thereby enhancing their ability to capture intricate patterns and dependencies.

Despite the substantial progress, the phenomenon of Massive Activations (MAs) within attention layers has not been thoroughly explored in the context of GNNs. MAs, characterized by exceedingly large activation values, can significantly impact the stability and performance of neural networks. In particular, understanding and mitigating MAs in GNNs is crucial for ensuring robust and reliable model behavior, especially when dealing with complex and large-scale graphs.

In this paper, we aim to bridge this gap by systematically investigating the occurrence and implications of MAs in attention-based GNNs. We focus on edge features in graph transformers, a state-of-the-art GNN architecture, and analyze how these features contribute to the emergence of MAs. Our study reveals that certain graph structures on edge configurations are more prone to inducing MAs, which in turn affects the overall performance and interpretability of the models.

To address these challenges, we propose a novel methodology for detecting and analyzing MAs in GNNs. Our approach involves a comprehensive evaluation of various GNN architectures, including GraphTransformer <cit.>, GraphiT <cit.>, and SAN <cit.>, across multiple benchmark datasets, like ZINC <cit.>, TOX21 <cit.> and OGBN-PROTEINS <cit.>, which differs from their downstream tasks like graph regression, multi-label graph classification, and multi-label node classification. We introduce specific criteria for identifying MAs and conduct extensive ablation studies to elucidate the role of edge features in this context.

This study represents the first comprehensive investigation of MAs in GNNs, laying the groundwork for future research. Our findings suggest that the scope of MAs analysis can be expanded to include a wider range of architectures and the evaluation of state-of-the-art attack methods, ultimately enhancing our understanding of MAs' influence on GNN performance and robustness. This is crucial for developing more robust and reliable graph transformer models, especially given the increasing popularity and widespread adoption of transformers in various applications today.

Our contributions are threefold:

    
  * We provide the first systematic study on MAs in attention-based GNNs, highlighting their prevalence and impact on model performance.
    
  * We propose a robust detection methodology for MAs, accompanied by detailed experimental protocols and ablation studies.
    
  * We introduce the Explicit Bias Term (EBT) as a potential countermeasure for MAs, and we exploit it in an adversarial framework, called Explicit Bias Attack, to demonstrate the effectiveness of the MAs in compromising GNNs robustness.
 Through this work, we aim to shed light on a critical yet understudied aspect of attention-based GNNs, offering valuable insights for the development of more resilient and interpretable graph-based models. All datasets and code are publicly available on our GitHub repository.



§ RELATED WORKS


GNNs have become effective instruments for studying and extracting insights from graph-structured data, with uses spanning fields like fraud detection <cit.>, traffic prediction <cit.> and recommendation systems <cit.>. The evolution of GNNs has been marked by significant advancements in their architectures and learning mechanisms, with a recent focus on incorporating attention mechanisms to enhance their expressive power and performance. The introduction of attention in GNNs was largely inspired by the success of transformers in natural language processing <cit.>. Graph Attention Networks (GATs) <cit.> were among the first to incorporate self-attention into GNNs, allowing nodes to attend differently to their neighbors based on learned attention weights. This innovation significantly improved the model's ability to capture complex relationships within graph structures.

Building upon the success of GATs, several variants and extensions have been proposed. GraphiT <cit.> introduced a generalization of transformer architectures to graph-structured data, incorporating positional encodings and leveraging the power of multi-head attention mechanisms. Similarly, the Structure-Aware Network (SAN) <cit.> proposed a novel attention mechanism that explicitly considers the structural properties of graphs, leading to improved performance on various graph-based tasks.

Recent studies on Large Language Models (LLMs) and Vision Transformers (ViTs) have revealed the presence of MAs within their internal states, specifically in the attention layer's output <cit.>. This phenomenon prompted investigations into the role of these activations in model behavior, performance, and potential vulnerabilities. Similar observations were made in Vision Transformers (ViTs) <cit.>, suggesting that the presence of MAs might be a common feature in transformer-based architectures across different domains. These findings have led to a growing interest in understanding the implications of MAs for model interpretability, robustness, and potential vulnerabilities to adversarial attacks.

The study of internal representations in deep learning models has been a topic of significant interest in the machine learning community. Works such as <cit.> have explored the interpretability of neural networks by analyzing activation patterns and their relationships to input features and model decisions. However, the specific phenomenon of MAs in GNNs has remained largely unexplored until now, representing a crucial gap in our understanding of these models.

The intersection of adversarial attacks and GNNs is another relevant area of study that relates to the investigation of MAs. Previous work has explored various attack strategies on graph data <cit.>, including topology attacks, feature attacks, adversarial training and hybrid approaches. However, the potential vulnerabilities introduced by MAs represent a novel direction for research in this field. Understanding how MAs might be exploited or manipulated by adversarial inputs could lead to the development of more robust GNN architectures.

However, in the broader context of neural network analysis, techniques for probing and interpreting model internals have been developed. Methods such as feature visualization <cit.> and network dissection <cit.> have provided insights into the functions of individual neurons and layers in convolutional neural networks. Adapting and extending these techniques to analyze MAs in GNNs could provide valuable insights into their role and impact in possible future works.

Finally, the study of attention mechanisms in various neural network architectures has also yielded insights that may be relevant to understanding MAs in GNNs. Work on attention flow <cit.> and attention head importance <cit.> in transformer models has shown that not all attention heads contribute equally to model performance, and some may even be pruned without significant loss of accuracy. These findings raise questions about whether similar patterns might exist in graph transformer models and how they might relate to the presence of MAs.



§ TERMINOLOGY OF MASSIVE ACTIVATIONS IN GNNS


Building upon the work on MAs in LLMs <cit.>, we extend this investigation to GNNs, focusing specifically on graph transformer architectures. Our study encompasses various models, including GraphTransformer (GT) <cit.>, GraphiT <cit.>, and Structure-Aware Network (SAN) <cit.>, applied to diverse task datasets such as ZINC, TOX21, and OGBN-PROTEINS (see Supplementary Material for details on models' configurations and datasets' composition). This comprehensive approach allows us to examine the generality of MAs across different attention-based GNN architectures.



 §.§ Characterization of Massive Activations

MAs in GNNs refer to specific activation values that exhibit unusually high magnitudes compared to the typical activations within a layer. These activations are defined by the following criteria, where an activation value is intended to be its absolute value:

Magnitude Threshold: An activation is classified as massive if its value exceeds a predetermined threshold. This threshold is typically set to a value that is significantly higher than the average activation value within the layer, ensuring that only the most extreme activations are considered.

Relative Threshold: In the paper by <cit.>, MAs were defined as at least 1,000 times larger than the median activation value within the layer. This relative threshold criterion helped differentiate MAs from regular high activations that might occur due to normal variations in the data or model parameters.

The formal definition was represented as:

    MAs = {a | a > 100  and  a > 1000 ×median(𝐀)}

where 𝐀 represents the set of activation values in a given layer.

However, in contrast to previous studies that employed a fixed relative threshold, our approach adopts a more rigorous method. We estimate MAs by comparing the distributions of activation ratios between a base, untrained model with Xavier weight initializations <cit.>, and a fully trained model. This method ensures a more precise identification of MAs based on empirical data rather than an arbitrary fixed threshold. In this way, the untrained model serves as a reference for identifying unusual activations that emerge during training. 



  §.§.§ Detection Methodology


For both the base and trained models, we detected the MAs following a systematic procedure:

Normalization: We normalized the activation values within each layer, dividing them by the edge median on the layer, to account for variations in scale between different layers and models. This normalization step ensures a consistent basis for comparison. The choice of dividing by the edge median comes from the huge amount of MAs being present, since almost every edge in the layers presenting MAs holds at least one MA, as shown from <Ref>. This is probably caused by the fact that attention is computed between pairs of adjacent nodes only, in contrast to LLMs where it is computed among each pair of tokens, therefore the model tends to spread MAs among almost all the edges to make them “available” to the whole graph. Indeed, <Ref> indicates that MAs are a common phenomenon across different models and datasets, that they are not confined to specific layers but are distributed throughout the model architecture, and that MAs are an inherent characteristic of the attention-based mechanism in graph transformers and related architectures, not strictly dependent on the choice of the dataset. 



Batch Analysis: We analyzed the activations on a batch-by-batch basis, minimizing the batch size, to have suitable isolation between the MAs and to ensure that the detection of MAs is not influenced by outliers in specific samples. 
For each activation, we computed the ratio of its magnitude to the edge median:

    ratio(activation) = abs(activation)/median(abs(edge_activations))

and activations whose ratio exceeds the threshold are flagged as massive.
Then, we considered the maximum ratio of each batch to detect those containing MAs.

Layer-wise Aggregation: We performed this analysis across multiple layers of the model to identify patterns and layers that are more prone to exhibiting MAs. This layer-wise aggregation helps in understanding the hierarchical nature of MAs within the model.

<Ref> reports the analysis results. The batch ratios significantly increase in the trained transformers, concerning base ones, often even overcoming the threshold of 1000 defined by previous works <cit.>, showing the presence of MAs in graph transformers, too.




§ METHODOLOGY AND OBSERVATIONS

 
Focusing on edge features, first, we analyzed the ratio defined in <Ref>, taking the maximum for every batch, across the layers of each selected model and dataset, and visually compared the outcomes to value ranges obtained using the same model in a base state (with its parameters randomly initialized, without training) to verify the appearance of MAs. The graphical comparison, reported in <Ref>, shows ratios over the base range in most of the trained models, representing MAs. 



To better characterize MAs, we studied their distribution employing the Kolmogorov-Smirnov statistic <cit.>. We found that a gamma distribution well approximates the negative logarithm of the activations' magnitudes, as well as their ratios. <Ref> shows this approximation for a base model layer.
We point out that, according to the existing definition, items on the left of the -3 are MAs.

We then compared the distributions of the log-values between the base and trained models. <Ref> illustrates this comparison, highlighting a significant shift in the distribution of the trained model compared to the base model. Moreover, this shift underscores the emergence of MAs during the training process, affirming that the threshold around -log(ratio) = -3 (e.g., a ratio of 1000 or higher) effectively captures these significant activations, though sometimes it appears to be slightly shifted to the right as in <Ref>.

When MAs appear, we have found two possible phenomenons:

    
  * A lot of massive activation values are added on the left-hand side of the distribution, preventing a good approximation (<Ref>).
    
  * A few values appear on the left-hand side of the distribution, as spikes or humps or out-of-distribution values, which may or may not deteriorate the approximation, as shown in <Ref>.



For example, histogram in <Ref> represents the base model with untrained weights (only Xavier initialization). The gamma approximation fits the sample histogram well, with a low Kolmogorov-Smirnov (KS) statistic of 0.020, indicating a very nice fit.

<Ref> shows that the distribution of the trained model exhibits a significant shift due to a big hump appearing on the left side, representing extreme activation ratios (MAs). Indeed, the gamma approximation does not fit well, with a higher KS statistic of 0.168, indicating a poor match caused by the presence of MAs.

Moreover, in the histogram of <Ref> the trained model's distribution exhibits a clear spike on the left side at -log(ratio) = -3, corresponding to a ratio of 1000. This separation indicates the distinction between basic and massive activation regimes. The gamma distribution doesn't fit well this time, because of this spike preventing a good approximation, with a KS statistic of 0.027 highlighting the model's shift due to training. 

<Ref> also shows the trained model's distribution, with a noticeable hump on the left side indicating MAs. The gamma approximation fits better than in <Ref>, with a KS statistic of 0.019, but still indicates the presence of MAs in the trained model, meaning that MAs have been added on the left-hand side of the distribution.



 §.§ Insights and Implications

From <Ref> and <Ref> we can highlight the following points.

    
  * Dataset Influence:
    
        
  * The ZINC and OGBN-PROTEINS datasets consistently show higher activation values across all models compared to TOX21, suggesting that the nature of these datasets significantly influences the emergence of MAs. Even though many MAs are emerging form GT on TOX21.
    

    
  * Model Architecture:
    
        
  * Different GNN models exhibit varying levels of MAs. For instance, GraphTransformer and GraphiT tend to show more pronounced MAs than SAN, indicating that model architecture plays a crucial role.
    


    
  * Impact of Attention Bias:
    
        
  * Previous works suspect that MAs have the function of learned bias, showing that they disappear introducing bias at the attention layer. This holds for LLMs and ViTs, and for our GNNs as well, as shown in <Ref> where the presence of MAs is affected by the introduction of the Explicit Bias Term on the attention. <Ref> and text below suggest that MAs are intrinsic to the models' functioning, being anti-correlated with the learned bias.
    

The consistent observation of MAs in edge features, across various GNN models and datasets, points to a fundamental characteristic of how these models process relational information.  

Inspired by recent advancements in addressing bias instability in LLMs <cit.>, we introduced an Explicit Bias Term (EBT) into our graph transformer models. This bias term is discovered to counteract the emergence of MAs by stabilizing the activation magnitudes during the attention computation. The EBT is computed as follows:

    b_e    = Q k e' 
    
        b_v    = softmax(A_e) v' ,
 where k, e, v ∈ℝ^d are the key, edge, and node value bias terms (one per each attention head), and A_e is the edge attention output. b_e and b_v represent the edge and node bias terms and are added to the edge and node attention outputs, respectively. By incorporating EBT into the edge and node attention computations, and adding bias in the linear projections of the attention inputs, we regulated the distribution of activation values, thus mitigating the occurrence of MAs. 

As shown in <Ref>, the introduction of these bias terms significantly reduces the frequency and magnitude of MAs, bringing the activation ratios closer to those observed in the base models. The effect of EBT is evident across all the different datasets. Whether it's ZINC, TOX21, or OGBN-PROTEINS, the activation ratios are brought closer to the baseline levels observed in the untrained models. This consistency underscores the general applicability of EBT in various contexts and downstream tasks. Moreover, <Ref> shows that EBT mitigates MAs across different layers of the models. This is crucial as it indicates that EBT's effect is not limited to specific parts of the network but is extended throughout the entire architecture. For example, GraphTransformer on ZINC without EBT shows MAs frequently exceed 10^4, while when EBT has been applied these ratios are significantly reduced, aligning more closely with the base model's range.


<Ref> shows that EBT does not systematically influence the test loss equally across different models and datasets. We have considered the test loss metric to keep the approach general, making it extendable to different downstream tasks. This ensures that the proposed method can be applied broadly across various applications of graph transformers. 



Although the test loss remains relatively unchanged with the introduction of EBT, its presence helps in mitigating the occurrence of MAs, as evidenced by the reduction in extreme activation values observed in earlier figures. By analyzing these results, it becomes evident that while EBT does not drastically alter the test performance, it plays a crucial role in controlling activation anomalies, thereby contributing to the robustness and reliability of graph transformer models. 

In the next section, we will demonstrate how attacking the model with and without MAs can directly impact the robustness of the architectures. This will provide deeper insights into the robustness of graph transformers in the presence of MAs, suggesting their potential pitfalls.




§ EXPLICIT BIAS ATTACK

The study of adversarial attacks on GNNs has become increasingly important as these models are deployed in critical applications. While various attack strategies have been explored <cit.>, the vulnerability introduced by MAs remains largely unexplored. Understanding how MAs can be exploited by adversaries is crucial for developing more robust GNN architectures and their downstream tasks. In this section, we propose the Explicit Bias Attack, a gradient-based method designed to exploit MAs and assess model robustness. Our approach is inspired by gradient ascent attacks previously applied to image classifiers <cit.> and adapted for graph data <cit.>. By analyzing the effectiveness of gradient ascent attack with and without the presence of EBT and MAs, we aim to provide insights into the role of these activations in model fragility.

Therefore, inspired by previous section, we exploited EBT as computed in <Ref> and <Ref> to analyze the importance of MAs for a gradient ascent attack at test time, where noise (added to the input feature embedding) is learned to directly maximize the loss function.
The effectiveness of an attack is evaluated by comparing the average test loss before and after the attack (i.e., with random and optimized noise, with the same standard deviations, respectively), using a gain defined as

    attack gain   = optimized noise loss - random noise lossrandom noise loss
 thus a higher gain means a more dangerous attack.
We focus on GraphTransformer (GT) with TOX21 because the presence of MAs in each layer - as shown by <Ref> - highlights the MAs effect for the attack, 
and compare the power of this method with and without the use of the EBT, which calls off the model's MAs.

 

<Ref> shows a stable increase of gain when dealing with MAs, using noise with standard deviation values of 0.01, 0.03, and 0.1 (the input feature embedding has a standard deviation of about 0.9) optimized for 1000 epochs on the test set. <Ref> highlights that MAs can be dangerous for the robustness of a model, and potentially exploited by attacks.






    








These results indicate that a gradient ascent attack is effective in degrading model performance, especially in the presence of MAs. However, the introduction of explicit bias, consistent with the reduction of MAs, can significantly mitigate the impact of the attack, leading to more robust models. This highlights the importance of considering bias in designing defenses against these types of adversarial attacks, to prevent them from exploiting the presence of MAs. 

In future work, to enable us to comprehensively assess the correlation between model robustness/fragility and the presence of MAs, we intend to delve deeper into different graph attack configurations while targeting MAs. This will offer a richer understanding of how these vulnerabilities can be mitigated, in favor of more reliable models.



§ CONCLUSION AND FUTURE WORK

This paper presents the first comprehensive study of MAs in attention-based GNNs. We have introduced a novel methodology for detecting and analyzing MAs, focusing on edge features in various graph transformer architectures across multiple benchmark datasets. Our findings reveal that MAs are prevalent across different models and datasets, and demonstrate that they could be effectively leveraged by adversaries to degrade the performance of GNNs.

We showed that the introduction of Explicit Bias Terms (EBT) can effectively mitigate the occurrence of MAs, leading to more stable activation distributions. However, our results also showed that this mitigation does not always translate to improved test performance, highlighting the complex role of MAs in GNNs' behavior.

Furthermore, we introduced the Explicit Bias Attack, a gradient-ascent adversarial framework, that demonstrates how MAs, if not mitigated by EBT, can expose models to vulnerabilities in their tasks. This further points out the importance of considering these activations in the context of model robustness.

Future research will expand this analysis to a wider range of architectures and advanced attack methods, further clarifying the influence of MAs on GNN performance and robustness, and potentially leading to more interpretable and stable graph-based models. Specifically, future research could explore:


    
  * Customized Adversarial MAs: Developing more adversarial techniques to regulate and attack these activations to enhance model stability and performance, like injecting fake MAs or exploiting state-of-the-art graph attack methods.
    
  * Downstream-driven MAs: Leveraging MAs for specific downstream task, investigating how to harness these significant activations to improve models and their interpretability on specific assignments such as link prediction or drug design.
    
  * Comparative Analysis: Extending the study to additional models and datasets to generalize the findings further and uncover broader patterns.


These insights provide a deeper understanding of the internal mechanisms of attention-based GNNs and highlight the way for improvements in graph learning models. By addressing the challenges and opportunities presented by MAs, we can work towards developing more robust, interpretable, and effective GNN architectures for a wide range of applications.





§ REPRODUCIBILITY CHECKLIST

This paper:

    
  * Includes a conceptual outline and/or pseudocode description of AI methods introduced: partial.
    
  * Clearly delineates statements that are opinions, hypothesis, and speculation from objective facts and results: yes.
    
  * Provides well marked pedagogical references for less-familiare readers to gain background necessary to replicate the paper: yes.


Does this paper make theoretical contributions? no.



Does this paper rely on one or more datasets? yes.

    
  * A motivation is given for why the experiments are conducted on the selected datasets: yes.
    
  * All novel datasets introduced in this paper are included in a data appendix: NA.
    
  * All novel datasets introduced in this paper will be made publicly available upon publication of the paper with a license that allows free usage for research purposes: NA.
    
  * All datasets drawn from the existing literature (potentially including authors’ own previously published work) are accompanied by appropriate citations: yes.
    
  * All datasets drawn from the existing literature (potentially including authors’ own previously published work) are publicly available: yes.
    
  * All datasets that are not publicly available are described in detail, with explanation why publicly available alternatives are not scientifically satisficing: NA.


Does this paper include computational experiments? yes.

    
  * Any code required for pre-processing data is included in the appendix: yes.
    
  * All source code required for conducting and analyzing the experiments is included in a code appendix: yes.
    
  * All source code required for conducting and analyzing the experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes: yes.
    
  * All source code implementing new methods have comments detailing the implementation, with references to the paper where each step comes from: partial.
    
  * If an algorithm depends on randomness, then the method used for setting seeds is described in a way sufficient to allow replication of results: yes.
    
  * This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount of memory; operating system; names and versions of relevant software libraries and frameworks: no.
    
  * This paper formally describes evaluation metrics used and explains the motivation for choosing these metrics: yes.
    
  * This paper states the number of algorithm runs used to compute each reported result: no.
    
  * Analysis of experiments goes beyond single-dimensional summaries of performance (e.g., average; median) to include measures of variation, confidence, or other distributional information: no.
    
  * The significance of any improvement or decrease in performance is judged using appropriate statistical tests (e.g., Wilcoxon signed-rank): partial.
    
  * This paper lists all final (hyper-)parameters used for each model/algorithm in the paper’s experiments: yes.
    
  * This paper states the number and range of values tried per (hyper-) parameter during development of the paper, along with the criterion used for selecting the final parameter setting: NA.

