Low-bit Quantization for Deep Graph Neural Networks with
  Smoothness-aware Message Propagation
Graph Neural Network (GNN) training and inference involve significant
challenges of scalability with respect to both model sizes and number of
layers, resulting in degradation of efficiency and accuracy for large and deep
GNNs. We present an end-to-end solution that aims to address these challenges
for efficient GNNs in resource constrained environments while avoiding the
oversmoothing problem in deep GNNs. We introduce a quantization based approach
for all stages of GNNs, from message passing in training to node
classification, compressing the model and enabling efficient processing. The
proposed GNN quantizer learns quantization ranges and reduces the model size
with comparable accuracy even under low-bit quantization. To scale with the
number of layers, we devise a message propagation mechanism in training that
controls layer-wise changes of similarities between neighboring nodes. This
objective is incorporated into a Lagrangian function with constraints and a
differential multiplier method is utilized to iteratively find optimal
embeddings. This mitigates oversmoothing and suppresses the quantization error
to a bound. Significant improvements are demonstrated over state-of-the-art
quantization methods and deep GNN approaches in both full-precision and
quantized models. The proposed quantizer demonstrates superior performance in
INT2 configurations across all stages of GNN, achieving a notable level of
accuracy. In contrast, existing quantization approaches fail to generate
satisfactory accuracy levels. Finally, the inference with INT2 and INT4
representations exhibits a speedup of 5.11 $\times$ and 4.70 $\times$ compared
to full precision counterparts, respectively.
Shuang Wang, Bahaeddin Eravci, Rustam Guliyev, Hakan Ferhatosmanoglu
# Low-bit Quantization for Deep Graph Neural Networks with Smoothness-aware Message Propagation

###### Abstract.

Graph Neural Network (GNN) training and inference involve significant challenges of scalability with respect to both model sizes and number of layers, resulting in degradation of efficiency and accuracy for large and deep GNNs. We present an end-to-end solution that aims to address these challenges for efficient GNNs in resource constrained environments while avoiding the oversmoothing problem in deep GNNs. We introduce a quantization based approach for all stages of GNNs, from message passing in training to node classification, compressing the model and enabling efficient processing. The proposed GNN quantizer learns quantization ranges and reduces the model size with comparable accuracy even under low-bit quantization. To scale with the number of layers, we devise a message propagation mechanism in training that controls layer-wise changes of similarities between neighboring nodes. This objective is incorporated into a Lagrangian function with constraints and a differential multiplier method is utilized to iteratively find optimal embeddings. This mitigates oversmoothing and suppresses the quantization error to a bound. Significant improvements are demonstrated over state-of-the-art quantization methods and deep GNN approaches in both full-precision and quantized models. The proposed quantizer demonstrates superior performance in INT2 configurations across all stages of GNN, achieving a notable level of accuracy. In contrast, existing quantization approaches fail to generate satisfactory accuracy levels. Finally, the inference with INT2 and INT4 representations exhibits a speedup of \(5.11\times\) and \(4.70\times\) compared to full precision counterparts, respectively.

graph neural networks; quantization; oversmoothing in GNNs; large-scale graph management; scalable machine learning +
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

[

+
Footnote †: Footnote †: thanks: [

[

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †: Footnote †: thanks: [

+
Footnote †:
recommender systems which commonly employ graph neural networks.

Unlike conventional applications of quantization, GNNs present unique challenges due to their intrinsic characteristics, which are not, effectively, addressed by the current methods. (i) The process of neighborhood aggregation in GNNs can lead to significant variance in high in-degree node embeddings, thereby exacerbating the quantization error, especially in low-bit cases (Shi et al., 2017). (ii) As GNNs deepen, they tend to experience the "oversmoothing" issue where each embedding loses its discriminative information due to the repeated, unregulated message passing (Kirshick et al., 2017). It is important to understand if this problem remains or is aggravated with the introduction of model quantization. Thus, while reducing GNN size and enabling compressed processing are pivotal for performance efficiency, addressing oversmoothing is crucial to ensure accuracy, especially in deeper models.

While recent studies (Wang et al., 2019; Wang et al., 2019; Wang et al., 2019) have delved into GNN quantization, the problem is far from being solved and there is no effective solution for low-bit quantization that scales for deeper GNNs. Our paper underscores this challenge, revealing that state-of-the-art GNN quantization methods undergo significant degradation at low bit counts (INT4 and INT2). This is more pronounced in deeper GNNs, due to accumulated layer-by-layer quantization errors. We aim to address these intricacies and develop an end-to-end solution.

Our solution involves a quantizer that learns the quantization ranges (QLR) along with a skewness-aware bitwise truncation (BT*) mechanism. Additionally, we introduce a smoothness-aware message propagation scheme (SMP) to counter the oversmoothing issue in quantized models. This quantizer determines an optimized, data-aware learnable range grounded in the input data distribution, thereby minimizing model redundancy. It is shown to retain its effectiveness with low-bit representations, which makes it apt for large deep GNNs. The skewness-aware truncation embedded within the quantizer improves the accuracy particularly in low-bit (INT2) scenarios. Our message propagation scheme aims to mitigate oversmoothing in deep GNNs by constraining the layer-wise shifts in similarities among neighboring nodes. Furthermore, we prove that by using SMP, the quantization error can be suppressed to a bound. Finally, we demonstrate the efficiency and accuracy of our solution through node classification accuracy on quantized GNN models.

Experimental results demonstrate improvements over the state-of-the-art approaches across various performance measures and workloads. Specifically, our quantizer (QLR) demonstrates remarkable advancements in low-bit quantization, outperforming existing quantization methods while resulting in reduced model sizes. For deeper GNNs, our SMP method delivers more accurate classification compared to other deep GNN approaches both in full-precision and quantized versions. The low-bit quantized SMP, using QLR, achieves greater improvement over alternative deep quantized GNN approaches with the help of the quantization error bound with SMP. BT* improves node classification accuracy on large datasets with INT2 representation, making it comparable to INT8 accuracy. We also show that the INT2 quantization model can yield an inference speedup of 5.11 \(\times\) compared to the full-precision model.

## 2. Related Work

Quantization has been commonly employed for neural network (NN) models (Wang et al., 2019). NN training is bottlenecked by high memory requirements to handle large data involving intermediate results and feature maps (Beng et al., 2019). NNs can be trained with low precision using dynamic fixed point arithmetic (Chen et al., 2019).

Quantization for neural networks can be performed during or after training. The post-training approaches quantize weights or activation of neural networks on a pre-trained full-precision model (Chen et al., 2019; Wang et al., 2019; Wang et al., 2019). Their low-bit quantization performance incurs significant accuracy degradation. The quantization-aware training aims to avoid this performance degradation (Chen et al., 2019; Wang et al., 2019). A useful technique is to expose errors from the quantization operation to the forward pass at model training and use straight-through estimator(STE) to compute the gradients (Chen et al., 2019). Banner et al. (Banner et al., 2019) provide a theoretical analysis showing considerable room for quantization under Gaussian weight assumption leading to 8-bit DNNs with comparable accuracy. The success of quantization has led to binary NNs (BNN) drastically reducing computation and memory requirements using hardware-supported bitwise operations with strong precision performances (Wang et al., 2019). The efficacy of high-order bit representations, involving bitwise truncation applied to 32-bit word embeddings, has been demonstrated in previous studies (Chen et al., 2019).

Pioneering work on learning node representations (Wang et al., 2019) has been followed by variants of architectures, utilizing convolutions (Wang et al., 2019; Wang et al., 2019; Wang et al., 2019) and autoencoder structures (Chen et al., 2019; Wang et al., 2019). GNN based representations have been used for various analytics tasks, including node similarity search (Chen et al., 2019), link prediction (Wang et al., 2019), and entity disambiguation (Wang et al., 2019). Solutions for scaling GNNs mostly focus on distributed processing (Wang et al., 2019; Wang et al., 2019; Wang et al., 2019). Scalability challenges on large graphs have also been studied in the context of memory optimizations (Wang et al., 2019) and scalable processing (Wang et al., 2019; Wang et al., 2019; Wang et al., 2019).

GNN quantization have started to receive attention in recent years. Tailor et al. (Tailor et al., 2019) propose quantization-aware training for GNNs, where high in-degree nodes are selected for full-precision training while all other nodes are converted to INT8/INT4. This can achieve reasonable accuracy especially on INT8 models. Huang et al. (Huang et al., 2019) employ product quantization to compress input data but do not address the more challenging task of quantization of parameters. A recent GNN quantization approach (Wang et al., 2019) addresses low-bit representation of the weights and input features by learning the parameters that are equal with the weight dimension and the number of input nodes, respectively, while leaving the core message propagation unquantized. However, this approach necessitates the learning of parameters that scale proportionally with the number of input nodes, resulting in considerable storage and space overheads. Neural Architecture Search (NAS) is used to span possible quantization levels suggesting an INT4 weight and INT8 activation as an effective strategy for GNNs (Wang et al., 2019). Recent studies adapt binary NN methods for GNNs (Chen et al., 2019; Wang et al., 2019) offering a trade-off between time/space efficiency and classification accuracy. These methods typically either need an additional teacher model for knowledge distillation or learn binary weights for each layer's input message, which require higher storage and computational load than a typical quantization based approach.
Towards addressing oversmoothing in deep GNNs, Liu et al. (Liu et al., 2018) propose Elastic Graph Neural Network with long-range information propagation using \(\ell_{1}\) and \(\ell_{2}\)-based graph smoothing. APPNP (Liu et al., 2018) addresses the oversmoothing with a propagation scheme based on an approximation of personalized PageRank. Zhu et al. (Zhu et al., 2019) proposed low-pass and high-pass filtering kernels which have empirically reduced the effect of oversmoothing. DropEdge (Zhu et al., 2019) aims to address the oversmoothing by dropping a number of edges, which can be interpreted as both a data augmentation method generating random deformed graphs and message passing reducer by sparsifying edge connections. PairNorm (Zhu et al., 2019) quantifies the oversmoothing and proposes a two-step center-and-scale normalization layer to prevent nodes converging to similar representations. Compared to enforcing local smoothness, our method, constrains the layer-wise message propagation to counteract oversmoothing, which achieves performance improvements over the prior approaches as also demonstrated in our experiments.

## 3. Preliminaries and Analysis

We first provide the technical background, covering quantization for GNNs, analysis of quantization errors, and the oversmoothing problem in GNNs.

### GNN Basics

A graph \(\mathcal{G}\) is represented as \(\mathcal{G}=(V,E,\mathbf{X})\), where \(V=\{v_{1},\cdots,v_{n}\}\) is the set of \(n\) nodes \(|V|=n\), \(E\) is the set of all edges, \(\mathbf{H}^{1}=[\mathbf{h}^{1}_{1},\cdots,\mathbf{h}^{1}_{n}]^{\top}\) is the node feature (embedding) matrix for layer \(l\in L\) where \(L\) represents the number of layers in \(\mathcal{G}\), and \(\mathbf{h}_{i}\in\mathbb{R}^{d_{l}}\) is the feature vector for \(v_{i}\in V\) node with initial \(\mathbf{H^{0}}=\mathbf{X}\). The adjacency matrix of \(\mathcal{G}\) is a binary matrix \(\mathbf{A}\in\mathbb{R}^{n\times n}\), where \(\mathbf{A}(i,j)=1\) if the edge between nodes \(v_{i}\) and \(v_{j}\) exists (\(e_{ij}\in E\)), and 0 otherwise.

GNNs comprise a sequence of layers with three main functions for each layer: message, aggregate and update. This framework is generally called Message Passing NNs (MPNN) (Kip
value in Equation 6 is represented as \(\hat{U}\), the mean squared error (MSE) between \(U\) and \(\hat{U}\) is given by

(7) \[\begin{split} E[(U-\hat{U})^{2}]=\int_{-\infty}^{\alpha}\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
### Skewness-aware Bitwise Truncation

A basic approach for INT2 can be simply keeping the most (two) significant bits of the higher precision output (e.g., INT8, INT4) as

\[Q_{b_{2}\gets b_{1}}(U_{q},s_{0})=\lfloor\frac{U_{q}}{s_{0}}\rceil s_{0} \tag{12}\]

where \(b_{1}\) and \(b_{2}\) (\(b_{1}\!\geq\!b_{2}\)) are the number of bits used for quantization, \(b_{2}\!\leftarrow\!b_{1}\) means the \(b_{1}\)-bit quantized representation being truncated into \(b_{2}\) bits; \(U_{q}\) denotes the \(b_{1}\)-bit representation obtained with Equation 8, and \(s_{0}\) is the scale for truncating the low-significant bit representation depending on \(b_{1}\) and \(b_{2}\). \(s_{0}\) can be obtained as

\[s_{0}=\frac{\alpha_{q_{1}}-\beta_{q_{1}}}{\alpha_{q_{2}}-\beta_{q_{2}}} \tag{13}\]

where [\(\alpha_{q_{1}}\), \(\beta_{q_{1}}\)] and [\(\alpha_{q_{2}}\), \(\beta_{q_{2}}\)] are the quantization levels for \(b_{1}\)-bit and \(b_{2}\)-bit quantization, respectively.

While such formulation implicitly assumes the uniformity of \(U\), for GNNs can significantly vary depending on the graph topology. Measures such as kurtosis (\(\kappa\)) and skewness (\(sk\)) (Kal
**Definition 5.1**: (Layer-wise Smoothness) Given a graph \(\mathcal{G}\)=\((V,E,\mathbf{X})\), the \(l\)-th layer-wise smoothness is the change of connected nodes \(\nabla(u,v_{j})\in E\) with a degree normalization from layer \(l\)-1 to layer \(l\).

The layer-wise smoothness can be formulated as

\[\begin{split}\mathbf{S}_{l}=\sum_{(u,v_{j})\in E}\|(\frac{ \mathbf{h}_{i}^{l}}{\sqrt{d_{i}}}-\frac{\mathbf{h}_{j}^{l}}{\sqrt{d_{j}}})-( \frac{\mathbf{h}_{i}^{l-1}}{\sqrt{d_{i}}}-\frac{\mathbf{h}_{j}^{l-1}}{\sqrt{d_ {j}}})\|_{2}^{2}\end{split} \tag{15}\]

Specifically, \(\mathbf{S}_{l}\) can also be represented as

\[\begin{split}\mathbf{S}_{l}&=\sum_{(\alpha,v_{j}) \in E}\|\frac{(\mathbf{h}_{i}^{l}-\mathbf{h}_{i}^{l-1})}{\sqrt{d_{i}}}-\frac{ (\mathbf{h}_{j}^{l}-\mathbf{h}_{j}^{l-1})}{\sqrt{d_{j}}}\|_{2}^{2}\\ &=\mathbf{tr}((\mathbf{H}^{\mathrm{I}}-\mathbf{H}^{\mathrm{I}- \mathbf{I}})^{\top}\tilde{\mathbf{L}}(\mathbf{H}^{\mathrm{I}}-\mathbf{H}^{ \mathrm{I}-\mathbf{I}}))\end{split} \tag{16}\]

where \(\tilde{\mathbf{L}}\) represents the normalized Laplacian matrix, \(\tilde{\mathbf{L}}=\mathbf{I}-\tilde{\mathbf{A}},\tilde{\mathbf{A}}=\mathbf{ D}^{-\frac{1}{2}}\tilde{\mathbf{A}}\mathbf{D}^{-\frac{1}{2}}\), \(\tilde{\mathbf{A}}=\mathbf{I}+\mathbf{A}\), and \(\mathbf{D}_{\tilde{\mathbf{H}}}=d_{i}=\sum_{j}\hat{\mathbf{A}}_{ij}\). The \(\mathbf{tr}(\mathbf{H}^{\top}\tilde{\mathbf{L}}\mathbf{H})\) is the Laplacian regularization to make \(\mathbf{H}\) smooth over graph \(\mathcal{G}\), similarly, the \(l\)-th layer-wise smoothness \(\mathbf{tr}((\mathbf{H}^{\mathrm{I}}-\mathbf{H}^{\mathrm{I}-\mathbf{I}})^{ \top}\tilde{\mathbf{L}}(\mathbf{H}^{\mathrm{I}}-\mathbf{H}^{\mathrm{I}-1}))\) can be explained as smoothing the changes from layer \(l\)-1 to \(l\) over \(\mathcal{G}\).
**SMP Contribution to Quantization.** The quantization error for the \(l\)-th layer representation can be expressed as \(f_{e}^{L}=\|\mathbf{H}^{l}-\mathbf{H}^{l,q}\|_{2}^{2}\), where \(\mathbf{H}^{l,q}\) is the quantized representation of \(\mathbf{H}^{l}\). Accordingly, for the quantized SMP, the smoothness constraint can be written as \(S_{l,q}=\text{tr}(\mathbf{H}^{l,q}-\mathbf{H}^{l-1,q})^{\top}\tilde{\mathbf{L} }(\mathbf{H}^{l,q}-\mathbf{H}^{l-1,q}))\). We can prove \(f_{e}^{L}\) is smaller than a bound as provided in LEMMA 1, which underlines the superiority of SMP in terms of quantization.

Lemma 1 ().: _For the \(l\)-th layer representation \(\mathbf{H}^{l}\), the quantization error is \(f_{e}^{L}\leq l\delta\text{tr}(\mathbf{\Lambda}^{-1})+\|\mathbf{H}^{l}- \mathbf{X}^{q}\|_{2}^{2}\)_

Proof.: The Laplacian matrix \(\tilde{L}\) is eigendecomposable, i.e., \(\tilde{L}=\text{U}\text{A}\text{U}^{\top}\), where \(\mathbf{U}\) is orthogonal matrix (UU\({}^{\top}\)= 1 ). \(S_{l,q}\) can be represented as \(S_{l,q}=\left\lvert\mathbf{\Lambda}^{\frac{1}{2}}\mathbf{U}^{\top}(\mathbf{H} ^{l,q}-\mathbf{H}^{l-1,q})\right\rvert_{2}^{2}\leq\delta\). The derivation process is summarized as follows

(1) \[\|\mathbf{H}^{l,q}-\mathbf{H}^{l}\|_{2}^{2}-\|\mathbf{H}^{l-1,q}- \mathbf{H}^{l}\|_{2}^{2}\leq\|\mathbf{H}^{l,q}-\mathbf{H}^{l-1,q}\|_{2}^{2}\leq\] \[\|\mathbf{H}^{l,q}-\mathbf{H}^{l}\|_{2}^{2}-\|\mathbf{H}^{l-1,q}- \mathbf{H}^{l}\|_{2}^{2}\leq\|\mathbf{H}^{l,q}-\mathbf{H}^{l-1,q}\|_{2}^{2}\leq\] \[\|(\mathbf{H}^{l,q}-\mathbf{H}^{l-1,q})\mathbf{\Lambda}^{\frac{1}{2 }}\|_{2}^{2}\|\mathbf{\Lambda}^{-\frac{1}{2}}\|_{2}^{2}\leq\delta\text{tr}( \mathbf{\Lambda}^{-1})\] (2) \[f_{e}^{L}\leq\delta\text{tr}(\mathbf{\Lambda}^{-1})+\|\mathbf{H} ^{l-1,q}-\mathbf{H}^{l}\|_{2}^{2}\] (3) \[\|\mathbf{H}^{l-1,q}-\mathbf{H}^{l}\|_{2}^{2}\leq\|\mathbf{H}^{l -1,q}-\mathbf{H}^{l-2,q}\|_{2}^{2}+\|\mathbf{H}^{l-2,q}-\mathbf{H}^{l}\|_{2}^{2}\] \[=\delta\text{tr}(\mathbf{\Lambda}^{-1})+\|\mathbf{H}^{l-2q}- \mathbf{H}^{l}\|_{2}^{2},\text{ where }1\leq i\leq l.\] (3) \[f_{e}^{L}\leq l\delta\text{tr}(\mathbf{\Lambda}^{-1})+\| \mathbf{H}^{l}-\mathbf{X}^{q}\|_{2}^{2}\] (5)

## 6. Experiments

This section presents our experiments on benchmark datasets that illustrate the effectiveness of QLR and QLR with BT (BT\({}^{*}\)) quantizers under low-bit settings. We also compare our SMP with comparable deep GNN baselines, highlighting the capability of SMP in addressing the oversmoothing issue.

### Experimental Setup

**Datasets and Baselines.** Our experiments are performed on five datasets, Cora, PubMed, CiteSeer (Zhu et al., 2017), CS (Zhu et al., 2017) and Reddit (Zhu et al., 2017) in a semi-supervised node classification setting. The statistics of the datasets are summarized in Table 1.

We start by comparing QLR against two state-of-the-art GNN quantizers, Degree-Quant (Zhu et al., 2017) and Aggregate-Quant (Zhu et al., 2017), on GCN (Zhu et al., 2017). These experiments are complemented with their respective model sizes. To showcase the effectiveness of SMP for quantization and oversmoothing we compare with 4 comparable deep GNN methods, which are, APPNP (Zhu et al., 2017), DropEdge (Zhu et al., 2017), PairNorm (Zhu et al., 2017) and EMP (Zhu et al., 2017). They are evaluated on 10-layer GNNs with 64 hidden units. Subsequently, we apply BT (BT\({}^{*}\)) within SMP and EMP pipeline to verify its effectiveness on extreme low-bit representations and its scalability with respect to the numbers of layers. Finally, we present the runtime efficiency by measuring the throughput under the representations of various quantization levels.

**Parameter settings.** For APPNP, DropEdge, PairNorm and EMP, we used the optimal parameters provided within their public repositories. For SMP, we set the parameters from the following search space: (1) learning rate (lr) \(\in\) {0.005, 0.008, 0.01, 0.015}; (2) weight decay (wd) \(\in\) {5e\({}^{-4}\), 1e\({}^{-4}\), 5e\({}^{-5}\)}; (3) drop rate \(\in\) {0.8}; (4) \(\mu\) {3, 6, 9}; (5) the initial value of \(\nu\) {1.0} \(\in\) {6} \(\eta_{\lambda}\), 7\(\approx\) {1e\({}^{-5}\),1e\({}^{-6}\)}; (7) \(\delta_{0}\)\(\in\) {0.01, 0.1, 0.5, 1, 2}. Due to a significant difference in magnitude between the gradients of the scaling parameters (y) and other GNN parameters, we have established a distinct search space for the former: learning rate (lr\({}_{Y}\)\(\in\) {0.001, 0.002}) and weight decay (wd\({}_{Y}\)\(\in\) {1e\({}^{-4}\), 5e\({}^{-5}\)}).

We present the average accuracy and standard deviation over 10 random data splits for Cora and CiteSeer, and 5 for PubMed, CS and Reddit. For Reddit dataset, owing to its size, we have employed mini-batch training with a batch_size of 20000. All of the experiments are based on Pytorch (Paszasz et al., 2017) and PyTorch Geometric (Kipip et al., 2018). The experiments are ran on Ubuntu 20.04 with 64GB RAM.

### Comparison with different quantizers

We compare QLR with the state-of-the-art GNN quantization solutions, Degree-Quant and Aggregate-Quant. Results are summarized in Table 2.

We notice that Aggregate-Quant in default maintains a fixed quantization level of INT4 for weights, while having smaller bits for input features. Moreover, it does not quantize the message-passing blocks of GCN, whereas QLR and Degree-Quant quantize all the elements equally. Hence, for fairness, we also add quantizers for its message-passing blocks and removed the INT4 constraint on its model weights. Also important to note that Aggregate-Quant maintains a step size parameter for each node, which can be viewed as an extension of learned step size quantization (LSQ) (Kip et al., 2018), which makes it highly inflexible for inductive tasks. Due to that, it does not support mini-batch training, as the topology of the input graph changes with each mini-batch training iteration.

We observe that performance of QLR significantly outperforms those of its competitors irrespective of the quantization level. The approach of optimizing the quantization range in the backward pass makes QLR more robust and effective, especially in low-bit cases. Aggregate-Quant demonstrates superior performance for CiteSeer when applied to INT8 quantization, which can be courtesy of its significantly larger parameter size. However, the accuracy of low-bit cases degrades significantly when message passing blocks are also quantized fairly. As for Degree-Quant, while it can achieve comparable performance on INT8, it cannot generate expected performance with INT4 and INT2 quantization on Reddit. Due to its mask sampling strategy and low quantization level, one node

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Dataset & Cora & CiteSeer & PubMed & CS & Reddit \\ \hline Nodes & 2708 & 3327 & 19717 & 18333 & 232965 \\ Edges & 5278 & 4552 & 44324 & 81894 & 114848857 \\ Features & 1433 & 3703 & 500 & 6805 & 602 \\ Labels & 7 & 6 & 3 & 15 & 41 \\ \hline \hline \end{tabular}
\end{table}
Table 1. Statistics of benchmark datasets

Figure 4. Avg layer-wise smoothness against different epochs (\(\overline{5}\)) for GNNs
sampled to different mini-batches will generate different representations at different batch training, which curbs the overall accurate representation. However, QLR can directly optimize the learnable quantization range based on the observations of subgraphs, hence, it can reduce the comprehensive quantization errors. These further confirm that optimizing the quantization range in QLR enables better preservation of accuracy in low-bit representations.

It is noteworthy that QLR preserves its accuracy results even for INT2 quantization across all datasets, while the alternatives fail to get comparable accuracy. Additionally, QLR even outperforms the full precision (FP) model in INT8 quantization in many cases, showcasing its effectiveness as a noise filter for GNNs.

In Table 3, we report the model sizes of different quantization approaches with varying quantization levels and hidden units (\(d\)). Due to space limit, we only present the model sizes on CS dataset. As there is a native 8-bit support, under constant \(d\), the sizes of INT8 with QLR and Degree-Quant are consistently reduced to approximately one-fourth of the FP counterpart. For smaller bits, however, we pack INT2 and INT4 similar to the process described in (Wang et al., 2018). The size of QLR is slightly larger to that of Degree-quant due to storage of \(s_{Y}\), \(z_{Y}\) and \(\gamma\) parameters. Given the superior accuracy performance of QLR in low-bit settings, its slight increase in model size becomes negligible in comparison. Overall, with QLR and Degree-quant, the INT2 and INT4 model sizes are significantly smaller than their FP counterparts, with reductions of 16x and 8x, respectively. However, the size of Aggregate-Quant is 2\(-\)6 times that of its counterparts, largely due to the dimension and per-node nature of the parameters.

### Comparisons with existing deep GNNs

#### 6.3.1. Node classification with existing deep GNNs

We compare SMP with the existing deep GNNs in terms of both full-precision (FP) and quantized models using QLR. Table 4 presents the classification accuracy results using 10-layer GNN. Notably, SMP consistently outperforms the alternative methods on Cora, CiteSeer and CS with FP models, and slightly lower than that of APPNP on PubMed. SMP improves over EMP by enforcing smoothness at layer-wise message propagation during training and inference.

For quantized models, although INT8 achieves high accuracy close to FP for all methods, INT4 performance of DropEdge and PairNorm drops significantly, rendering it incomparable in some cases, and throws OOM errors on larger datasets. This is primarily due to the complexity of the "backbone" models (Wang et al., 2018). For example, PairNorm, employing a GCN backbone, trains a weight matrix for each layer. Likewise, DropEdge utilizes more intricate backbones such as GCN, ResGCN (Wang et al., 2018), IncepGCN (Wang et al., 2018), and introduces connection perturbations at every layer. Consequently, these factors contribute to higher computational and storage requirements, which further escalate with the depth of the model. On the contrary, EMP, APPNP, and SMP employ much simpler architectures that involve only two weight matrices prior to GNN propagation. As a result, these models have more moderate and scalable requirements, making them more amenable for deep GNN quantization.
the accuracy of INT2-8\({}^{*}\) is, continuously, around 5%-7% higher than that of INT2-8 on CS dataset with respect to varying number of layers. This shows the effectiveness of BT\({}^{*}\) in reaching relatively high accuracy with low-bit representations.

Figure 6 presents the full training process of SMP and EMP with variations of our quantization approach on CS dataset. While SMP and EMP show unstable performance with INT2 representation generated by the basic quantizer, applying skewness-aware BT (BT\({}^{*}\)) contributes improvements that are close to that of INTs quantization. Furthermore, SMP is more robust than EMP, due to the existence of quantization error bound in SMP.

### Inference Speedup

In Table 5, we elucidate the inference times associated with varying quantization levels for the 2-layer GCN and SMP architectures, respectively. These model inferences are conducted on the Reddit dataset. The \(\uparrow\) signifies the inference speedup comparing with FP model. To realize quantized GNN speed improvements across different quantization levels, we leverage the recent Tensor Core-based approach, QGCT (Wang et al., 2017), applied to both GCN and SMP. We observe a notable speedup of 5.11 \(\times\) and 6.44 \(\times\) with SMP and GCN, respectively, in the context of low-bit representation (INT2), in comparison to the FP counterparts. Notably, the speedup for SMP exhibits a slight reduction compared to GCN, attributed to the additional computational overhead of SMP. Remarkably, with the same number of layer (\(L\)), SMP showcases superior accuracy performance relative to GCN. This is exemplified in Table 2 and Figure 5 in the CS dataset with \(L\)=2. Specifically, for SMP, the INT8 and INT4 accuracy outperforms GCN by approximately 2%, while SMP in INT2 mode demonstrates a performance advantage over GCN by up to 13.5%.

## 7. Conclusion

We have introduced an end-to-end solution towards achieving scalable deep GNNs, involving an efficient quantization with learnable ranges, with skewness-aware bitwise truncation, and a smoothness-aware message propagation (SMP) mechanism for efficient training and managing large deep GNNs. The solution reduces the model size and maintains its accuracy for classification even in low-bit representations. The message passing block in training is enforced to have layer-wise smoothness and constrains the changes between neighbor nodes. We formulate it as an additional constraint to a graph denoising optimization function and solved by Lagrange functions with an iterative BDMM algorithm. It aims to mitigate the oversmoothing problem in GNNs and to avoid the performance degradation encountered in low-bit quantization-aware training. We provide an upper bound on the error for the quantized SMP algorithm. Experiments show how the proposed solution achieves significant improvements over the-state-of-the-art approaches, providing a significant reduction in model sizes, an order of magnitude smaller than the full precision (FP) model with comparable accuracy results, and mitigating the oversmoothing problem on benchmark datasets.

###### Acknowledgements.

 This work is supported in part by the UK Engineering and Physical Sciences Research Council under Grant No. EP/T51794X/1.

\begin{table}
\begin{tabular}{c|c c c c c c c c c c c c c c c c} \hline \hline  & & \multicolumn{3}{c|}{SMP} & \multicolumn{3}{c|}{SMP-INT8} & \multicolumn{3}{c|}{SMP-INT8} & \multicolumn{3}{c|}{EMP-INT8} & \multicolumn{3}{c}{SMP-INT8} \\ \hline Dataset & FP & INTs & INT4 & INT2 & FP & INTs & INT4 & INT2 & FP & INTs & INT4 & INT2 & FP & INTs & INT4 & FP & INTs & INT4 \\ \hline Cora & 82.91 & 82.92 & 82.69 & 72.69 & 82.59 & 81.58 & 77.93 & 66.20 & 80.78 & 81.82 & 79.68 & 71.28 & 71.20 & 70.21 & – & 76.08 & 79.10 & 78.31 \\ \hline \(\pm\)0.64 & \(\pm\)0.51 & \(\pm\)0.68 & \(\pm\)2.48 & \(\pm\)0.67 & \(\pm\)0.79 & \(\pm\)2.62 & 66.75 & \(\pm\)1.42 & \(\pm\)1.44 & \(\pm\)1.81 & \(\pm\)2.34 & \(\pm\)2.14 & \(\pm\)1.50 & – & \(\pm\)2.86 & \(\pm\)1.00 & \(\pm\)1.01 \\ \hline Citeseer & 71.60 & 71.40 & 69.76 & 65.01 & 70.84 & 71.02 & 67.53 & 61.53 & 70.30 & 68.90 & 68.77 & 63.87 & 51.39 & – & – & 60.07 & 61.15 & 59.60 \\  & \(\pm\)1.26 & \(\pm\)1.58 & \(\pm\)1.63 & \(\pm\)1.26 & \(\pm\)1.20 & \(\pm\)1.49 & \(\pm\)2.68 & \(\pm\)3.65 & \(\pm\)1.53 & \(\pm\)2.19 & \(\pm\)2.01 & \(\pm\)2.24 & \(\pm\)4.16 & – & – & \(\pm\)5.76 & \(\pm\)2.88 & \(\pm\)3.73 \\ \hline PubMed & 79.36 & 79.90 & 87.44 & 75.97 & 78.64 & 78.68 & 76.18 & 74.22 & 79.59 & 79.31 & 78.49 & 73.56 & – & – & – & 75.69 & \(\pm\)2.88 & \(\pm\)3.73 \\ \hline \multirow{2}{*}{CS} & 79.49 & 79.24 & 79.24 & 83.74 & 92.17 & 91.16 & 79.18 & 82.02 & 91.58 & 92.37 & 91.94 & 81.25 & 75.12 & – & – & – & 81.81 & \multirow{2}{*}{OOM} & \multirow{2}{*}{OOM} & \multirow{2}{*}{OOM} & \multirow{2}{*}{OOM} \\  & \(\pm\)0.64 & \(\pm\)0.52 & \(\pm\)0.38 & \(\pm\)1.80 & \(\pm\)0.46 & \(\pm\)0.42 & \(\pm\)0.54 & \(\pm\)1.09 & \(\pm\)0.66 & \(\pm\)0.57 & \(\pm\)0.57 & \(\pm\)0.88 & \(\pm\)4.24 & OOM & OOM & \(\pm\)1.64 & OOM & OOM \\ \hline \multicolumn{10}{l}{- denotes accuracy \(\leq\) 40.00\%,} & \multicolumn{3}{c|}{OOM means ’out-of-memory’} \\ \end{tabular}
\end{table}
Table 4. Classification accuracy of Deep GNN methods (%) on benchmark datasets

Figure 5. Results of SMP and EMP with varying layers

Figure 6. Quantization performance of SMP and EMP on CS

\begin{table}
\begin{tabular}{c|c|c c c c c} \hline  & FP & INT8 & INT4 & INT2 & \(\uparrow\) \\ \hline SMP & 178.3 & 46.46 & 3.84 \(\times\) & 37.96 & 4.70 \(\times\) & 34.89 & 5.11 \(\times\) \\ \hline GCN & 156.97 & 34.96 & 4.49 \(\times\) & 27.29 & 5.75 \(\times\) & 24.37 & 6.44 \(\times\) \\ \hline \end{tabular}
\end{table}
Table 5. Inference time (ms) of SMP and GCN in different quantization levels on Reddit dataset