Graph Neural Networks Provably Benefit from Structural Information: A
  Feature Learning Perspective
Graph neural networks (GNNs) have pioneered advancements in graph
representation learning, exhibiting superior feature learning and performance
over multilayer perceptrons (MLPs) when handling graph inputs. However,
understanding the feature learning aspect of GNNs is still in its initial
stage. This study aims to bridge this gap by investigating the role of graph
convolution within the context of feature learning theory in neural networks
using gradient descent training. We provide a distinct characterization of
signal learning and noise memorization in two-layer graph convolutional
networks (GCNs), contrasting them with two-layer convolutional neural networks
(CNNs). Our findings reveal that graph convolution significantly augments the
benign overfitting regime over the counterpart CNNs, where signal learning
surpasses noise memorization, by approximately factor $\sqrt{D}^{q-2}$, with
$D$ denoting a node's expected degree and $q$ being the power of the ReLU
activation function where $q > 2$. These findings highlight a substantial
discrepancy between GNNs and MLPs in terms of feature learning and
generalization capacity after gradient descent training, a conclusion further
substantiated by our empirical simulations.
Wei Huang, Yuan Cao, Haonan Wang, Xin Cao, Taiji Suzuki
# Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective

###### Abstract

Graph neural networks (GNNs) have pioneered advancements in graph representation learning, exhibiting superior feature learning and performance over multilayer perceptrons (MLPs) when handling graph inputs. However, understanding the feature learning aspect of GNNs is still in its initial stage. This study aims to bridge this gap by investigating the role of graph convolution within the context of feature learning theory in neural networks using gradient descent training. We provide a distinct characterization of signal learning and noise memorization in two-layer graph convolutional networks (GCNs), contrasting them with two-layer convolutional neural networks (CNNs). Our findings reveal that graph convolution significantly augments the benign overfitting regime over the counterpart CNNs, where signal learning surpasses noise memorization, by approximately factor \(\sqrt{D}^{q-2}\), with \(D\) denoting a node's expected degree and \(q\) being the power of the ReLU activation function where \(q>2\). These findings highlight a substantial discrepancy between GNNs and MLPs in terms of feature learning and generalization capacity after gradient descent training, a conclusion further substantiated by our empirical simulations.

## 1 Introduction

Graph neural networks (GNNs) have recently demonstrated remarkable capability in learning node or graph representations, yielding superior results across various downstream tasks, such as node classifications [1; 2; 3], graph classifications [4; 5; 6; 7] and link predictions [8; 9; 10], etc. However, the theoretical understanding of why GNNs can achieve such success is still in its infancy. Compared to multilayer perceptron (MLPs), GNNs enhance representation learning with an added message passing operation [11]. Take graph convoluational network (GCN) [1] as an example, it aggregates a node's attributes with those of its neighbors through a _graph convolution_ operation. This operation, which leverages the structural information (adjacency matrix) of graph data, forms the core distinction between GNNs and MLPs. Empirical evidence from three node classification tasks, as shown in
Figure 1, suggests GCNs outperform MLPs. Motivated by the superior performance of GNNs, we pose a critical question about graph convolution:

* _What role does graph convolution play during gradient descent training, and what mechanism enables a GCN to exhibit better generalization after training?_

Several recent studies have embarked on a theoretical exploration of graph convolution's role in GNNs. For instance, Baranwal et al. (2021) [12] considered a setting of linear classification of data generated from a contextual stochastic block model [13]. Their findings indicate that graph convolution extends the regime where data is linearly separable by a factor of approximately \(1/\sqrt{D}\) compared to MLPs, with \(D\) denoting a node's expected degree. Baranwal et al. (2023) [14] further investigated the impact of graph convolutions in multi-layer networks, showcasing improved non-linear separability. However, these studies these examples, while insightful, assume the Bayes optimal classifier of GNNs, thereby losing a comprehensive characterization of the GNNs' optimization process. Consequently, there exists a notable gap in characterization of learning process and corresponding generalization ability of GNNs between existing theoretical explorations and the detailed examination of GNNs.

To respond to the growing demand for a comprehensive theoretical understanding of graph convolution, we delve into the feature learning process [15; 16; 17] of graph neural networks. In our study, we introduce a data generation model--termed SNM-SBM--that combines a signal-noise model [15; 18] for feature creation and a stochastic block model [19] for graph construction. Our analysis is centered on the convergence and generalization attributes of two-layer graph convolution networks (GCNs) when trained via gradient descent, compared with the established outcomes for two-layer convolutional neural networks (CNNs) as presented by Cao et al. (2022) [15]. While both GCNs and CNNs demonstrate the potential to achieve near-zero training error, our study effectively sheds light on the discrepancies in their generalization abilities. We emphasize the crucial contribution of graph convolution to the enhanced performance of GNNs. Our study's key contributions are as follows:

* We establish global convergence guarantees for graph neural networks training on data drawn from SNM-SBM model. We demonstrate that, despite the nonconvex optimization landscape, GCNs can achieve zero training error after a polynomial number of iterations.
* We further establish population loss bounds of overfitted GNN models trained by gradient descent. We show that under certain conditions on the signal-to-noise ratio, GNNs trained by gradient descent will prioritize learning the signal over memorizing the noise, and thus achieves small test losses.
* We delineate a marked contrast in the generalization capabilities of GCNs and CNNs following gradient descent training. We identify a specific regime where GCNs can attain nearly zero test error, whereas the performance of the model discovered by CNNs does not exceed random guessing. This conclusion is further substantiated by empirical verification.

## 2 Related Work

Role of Graph Convolution in GNNs.Enormous empirical studies of various GNNs models with graph convolution [20; 21; 22; 23; 24] have been demonstrating that graph convolutions can enhance the performance of traditional classification methods, such as a multi-layer perceptron (MLP). Towards theoretically understanding the role of graph convolution, Xu et al. (2020) [25] identify conditions under which MLPs and GNNs extrapolate, thereby highlighting the superiority of GNNs for extrapolation problems. Their theoretical analysis leveraged the concept of the over-parameterized networks and the neural tangent kernel [26]. Huang et al. (2021) [27] employed a similar approach to examine the role of graph convolution in deep GNNs within a node classification setting. They discovered that excessive graph convolution layers can hamper the optimization and generalization of GNNs, corroborating the well-known over-smoothing issue in deep GNNs [28]. Another pertinent work by Hou et al. (2022) [29] proposed two smoothness metrics to measure the quantity and quality of information derived from graph data, along with a novel attention-based framework. Some

Figure 1: Performance (test accuracy) comparison between GCN and MLP on node classification tasks.
rent works [12; 14; 21] have demonstrated that graph convolution broadens the regime in which a multi-layer network can classify nodes, compared to methods that do not utilize the graph structure, especially when the graph is dense and exhibits homophily. Yang et al. (2022) [30] attributed the major performance gains of GNNs to their inherent generalization capability through graph neural tangent kernel (GNTK) and extrapolation analysis. As for neural network theory, these works either gleaned insights from GNTK [31; 27; 32] or studied the role of graph convolution within a linear neural network setting. Unlike them, our work extends beyond NTK and investigates a more realistic setting concerning the convergence and generalization of neural networks in terms of feature learning.

Feature learning in neural networks.This work builds upon a growing body of research on how neural networks learn features. Allen-Zhu et al. (2020) [18] formulated a theory illustrating that when data possess a "multi-view" feature, ensembles of independently trained neural networks can demonstrably improve test accuracy. Further, Allen-Zhu et al. (2022) [16] demonstrated that adversarial training can purge certain small dense mixtures from the hidden weights during the training process of a neural network, thus refining the hidden weights. Ba et al. (2022) [33] established that the initial gradient update contains a rank-1'spike', which leads to an alignment between the first-layer weights and the linear component feature of the teacher model. Cao et al. (2022) [15] investigated the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN), illustrating that under certain conditions related to the signal-to-noise ratio, a two-layer CNN trained by gradient descent can achieve exceedingly low test loss through feature learning. Alongside related works [34; 35; 17; 36; 37; 38; 39; 40], all these studies have underscored the existence of feature learning in neural networks during gradient descent training, forming a critical line of inquiry that this work continues to explore. However, the neural tangent kernel (NTK) theory [41; 42; 43; 44], also known as "lazy training" [45], where the neural network function is approximately linear in its parameters, cannot demonstrate feature learning. Thus, the optimization and generalization analysis in our study extends beyond the NTK regime.

## 3 Problem Setup and Preliminary

### Notations

We use lower bold-faced letters for vectors, upper bold-faced letters for matrices, and non-bold-faced letters for scalars. For a vector \(\mathbf{v}=(v_{1},v_{2},\cdots,v_{d})^{\top}\), its \(\ell_{2}\)-norm is denoted as \(\|\mathbf{v}\|_{2}\triangleq\sqrt{\sum_{i=1}^{d}v_{i}^{2}}\). For a matrix \(\mathbf{A}=(a_{ij})\in\mathbb{R}^{m\times n}\), we use \(\|\mathbf{A}\|_{2}\) to denote its spectral norm and \(\|\mathbf{A}\|_{F}\) for its Frobenius norm. When comparing two sequences \(\{a_{n}\}\) and \(\{b_{n}\}\), we employ standard asymptotic notations such as \(O(\cdot)\), \(o(\cdot)\), \(\Omega(\cdot)\), and \(\Theta(\cdot)\) to describe their limiting behavior. Specifically, we write \(a_{n}=O(b_{n})\) if there exists a positive real number \(C_{1}\) and a positive integer \(N\) such that \(|a_{n}|\leq C_{1}|b_{n}|\) for all \(n\geq N\). Similarly, we write \(a_{n}=\Omega(b_{n})\) if there exists \(C_{2}>0\) and \(N>0\) such that \(|a_{n}|>C_{2}|b_{n}|\) for all \(n\geq N\). We say \(a_{n}=\Theta(b_{n})\) if \(a_{n}=O(b_{n})\) and \(a_{n}=\Omega(b_{n})\). Besides, if \(\lim_{n\to\infty}|a_{n}/b_{n}|=0\), we express this as \(a_{n}=o(b_{n})\). We use \(\widetilde{O}(\cdot)\), \(\widetilde{\Omega}(\cdot)\), and \(\widetilde{\Theta}(\cdot)\) to hide logarithmic factors in these notations respectively. Moreover, we denote \(a_{n}=\text{poly}(b_{n})\) if \(a_{n}=O((b_{n})^{p})\) for some positive constant \(p\) and \(a_{n}=\text{polylog}(b_{n})\) if \(a_{n}=\text{poly}(\log(b_{n}))\). Lastly, sequences of integers are denoted as \([m]=\{1,2,\dots,m\}\).

### Data model

In our approach, we utilize a signal-noise model for feature generation, combined with a stochastic block model for graph structure generation. Specifically, we define the feature matrix as \(\mathbf{X}\in\mathbb{R}^{n\times 2d}\), with \(n\) representing the number of samples and \(2d\) being the feature dimensionality. Each feature associated with a data point is generated from a _signal-noise model_ (SNM), conditional on the Rademacher random variable \(y\in\{-1,1\}\), and a latent vector \(\boldsymbol{\mu}\in\mathbb{R}^{d}\):

\[\mathbf{x}=[\mathbf{x}^{(1)},\mathbf{x}^{(2)}]=[y\boldsymbol{\mu}, \boldsymbol{\xi}], \tag{1}\]

where \(\mathbf{x}^{(1)},\mathbf{x}^{(2)}\in\mathbb{R}^{d}\), and \(\boldsymbol{\xi}\sim\mathcal{N}(\mathbf{0},\sigma_{p}^{2}\cdot(\mathbf{I}-\| \boldsymbol{\mu}\|_{2}^{-2}\cdot\boldsymbol{\mu}\boldsymbol{\mu}^{\top}))\) consists of independent standard normal entries with \(\sigma_{p}^{2}\) as the variance. The term \(\mathbf{I}-\|\boldsymbol{\mu}\|_{2}^{-2}\cdot\boldsymbol{\mu}\boldsymbol{\mu} ^{\top}\) is employed to guarantee that the noise vector is orthogonal to the signal vector \(\boldsymbol{\mu}\). The signal-noise model we have adopted is inspired by the structure of an image composed of multiple patches, where we consider a two-patch
model for simplicity. The first patch \(\mathbf{x}^{(1)}\), represented by the signal vector, corresponds to the target or meaningful information in an image, such as an object or feature of interest. The second patch \(\mathbf{x}^{(2)}\), represented by the noise vector, corresponds to the background or other irrelevant details in the image, which can be considered as nuisance variables or features. It's worth mentioning that a series of recent works [18; 15; 35; 46] have explored similar signal-noise models to illustrate the feature learning process and benign overfitting of neural networks.

Following this, we implement a stochastic block model with inter-class edge probability \(p\) and intra-class edge probability \(s\), devoid of self-loops. Specifically, the adjacency matrix \(\mathbf{A}=(a_{ij})_{n\times n}\) is Bernoulli distributed, with \(a_{ij}\sim\text{Ber}(p)\) when \(y_{i}=y_{j}\), and \(a_{ij}\sim\text{Ber}(s)\) when \(y_{i}=-y_{j}\). The combination of a stochastic block model with the signal-noise model (1) is represented as \(\mathrm{SNM}-\mathrm{SBM}(n,p,s,\boldsymbol{\mu},\sigma_{p},d)\). Consequently, the raw feature and graph structure are generated as \((\mathbf{A},\mathbf{X},\mathbf{y})\sim\mathrm{SNM}-\mathrm{SBM}(n,p,s, \boldsymbol{\mu},\sigma_{p},d)\), allowing the data model (1) used in MLP to be considered as a special case where \(p=s=0\). Note that the primary distinction between the SNM-SBM and Contextual stochastic block model (CSBM) [47] lies in the representation of the contextual part. In our SNM-SBM model, the contextual part, or input feature, is given by a two-patch model. Conversely, in CSBM, the input feature is given by a Gaussian mixture model.

### Neural network model

In this section, we present two distinct types of neural network models: a two-layer convolutional neural network (CNN), which falls under the category of a multilayer perceptron (MLP), and a Graph Convolutional Neural Network (GCN) [1].

Cnn.We introduce a two-layer CNN model, denoted as \(f\), which utilizes a non-linear activation function, \(\sigma(\cdot)\). Specifically, we employ a polynomial ReLU activation function defined as \(\sigma(z)=\max\{0,z\}^{q}\), where \(q>2\) is a hyperparameter. Note that the use of a polynomial ReLU activation function aligns with related studies [18; 16; 15; 35; 48] that investigate neural network feature learning. Mathematically, given the input data \(\mathbf{x}\), the CNN's output is represented as \(f(\mathbf{W},\mathbf{x})=F_{+1}(\mathbf{W}_{+1},\mathbf{x})-F_{-1}(\mathbf{W}_ {-1},\mathbf{x})\), where \(F_{+1}(\mathbf{W}_{+1},\mathbf{x})\) and \(F_{-1}(\mathbf{W}_{+1},\mathbf{x})\) are defined as follows:

\[F_{j}(\mathbf{W}_{j},\mathbf{x})=\frac{1}{m}\sum_{r=1}^{m}\left[ \sigma(\mathbf{w}_{j,r}^{\top}\mathbf{x}^{(1)})+\sigma(\mathbf{w}_{j,r}^{\top }\mathbf{x}^{(2)})\right], \tag{2}\]

where \(m\) is the width of hidden layer, the second layer parameters are fixed as either \(+1\) or \(-1\). We assume a poly-logarithmic network width in relation to the training sample size, i.e., \(m=\mathrm{polylog}(n)\), and \(\mathbf{w}_{j,r}\in\mathbb{R}^{d}\) refers to the weight of the first layer's \(r\)-th neuron connected to the second layer's \(j\) class. The symbol \(\mathbf{W}\) collectively represents the model's weights. Moreover, each weight in the first layer is initialized from a random draw of a Gaussian random variable, \(\mathbf{w}_{j,r}\sim\mathcal{N}(\mathbf{0},\sigma_{0}^{2}\cdot\mathbf{I}_{d \times d})\) for all \(r\in[m]\) and \(j\in\{-1,1\}\), with \(\sigma_{0}\) regulating the initialization magnitude for the first layer's weight.

Upon receiving training data \(\mathcal{S}\triangleq\{\mathbf{x}_{i},y_{i}\}_{i=1}^{n}\) drawn from \(\mathrm{SNM}-\mathrm{SBM}(n,p=0,s=0,\boldsymbol{\mu},\sigma_{p},d)\), we aim to learn the network's parameter \(\mathbf{W}\) by by minimizing the empirical cross-entropy loss function:

\[L_{\mathcal{S}}^{\mathrm{CNN}}(\mathbf{W})=\frac{1}{n}\sum_{i=1}^ {n}\ell(y_{i}\cdot f(\mathbf{W},\mathbf{x}_{i})), \tag{3}\]

where \(\ell(y\cdot f(\mathbf{W},\mathbf{x}))=\log(1+\exp(-f(\mathbf{W},\mathbf{x}) \cdot y))\). The update rule for the gradient descent used in the CNN is then given as:

\[\mathbf{w}_{j,r}^{(t+1)} =\mathbf{w}_{j,r}^{(t)}-\eta\cdot\nabla_{\mathbf{w}_{j,r}}L_{ \mathcal{S}}^{\mathrm{CNN}}(\mathbf{W}^{(t)})\] \[=\mathbf{w}_{j,r}^{(t)}-\frac{\eta}{nm}\sum_{i=1}^{n}\ell_{i}^{ \prime(t)}\cdot\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t)},\boldsymbol{\xi}_ {i}\rangle)\cdot jy_{i}\boldsymbol{\xi}_{i}-\frac{\eta}{nm}\sum_{i=1}^{n}\ell_ {i}^{\prime(t)}\cdot\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t)},y_{i} \boldsymbol{\mu}\rangle)\cdot j\boldsymbol{\mu}, \tag{4}\]

where we define the loss derivative as \(\ell_{i}^{\prime}\triangleq\ell^{\prime}(y_{i}\cdot f_{i})=-\frac{\exp(-y_{i} \cdot f_{i})}{1+\exp(-y_{i}\cdot f_{i})}\). It's important to clarify that the model we use for the MLP part is a two-layer CNN network. We categorize it as an MLP for comparison purposes with the graph neural network.
Gcn.Graph neural network (GNNs) fuse graph structure information and node features to learn representation of nodes. Consider a two-layer GCN \(f\) with graph convolution operation on the first layer. The output of the GCN is given by \(f(\mathbf{W},\tilde{\mathbf{x}})=F_{+1}(\mathbf{W}_{+1},\tilde{\mathbf{x}})-F_{- 1}(\mathbf{W}_{-1},\tilde{\mathbf{x}})\), where \(F_{+1}(\mathbf{W}_{+1},\tilde{\mathbf{x}})\) and \(F_{-1}(\mathbf{W}_{+1},\tilde{\mathbf{x}})\) are defined as follows:

\[F_{j}(\mathbf{W}_{j},\tilde{\mathbf{x}})=\frac{1}{m}\sum_{r=1}^{m}\left[\sigma( \mathbf{w}_{j,r}^{\top}\tilde{\mathbf{x}}^{(1)})+\sigma(\mathbf{w}_{j,r}^{ \top}\tilde{\mathbf{x}}^{(2)})\right]. \tag{5}\]

Here, \(\tilde{\mathbf{X}}\triangleq[\tilde{\mathbf{x}}_{1},\tilde{\mathbf{x}}_{2}, \cdots,\tilde{\mathbf{x}}_{n}]^{\top}=\tilde{\mathbf{D}}^{-1}\tilde{\mathbf{ A}}\mathbf{X}\in\mathbb{R}^{n\times 2d}\) with \(\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}_{n}\) representing the adjacency matrix with self-loop, and \(\tilde{\mathbf{D}}\) is a diagonal matrix that records the degree of each node, namely, \(\tilde{D}_{ii}=\sum_{j}\tilde{A}_{ij}\). For simplicity we denote \(D_{i}\triangleq\tilde{D}_{ii}\). Therefore, in contrast to the CNN model (2), the GCNs (5) incorporate the normalized adjacency matrix \(\tilde{\mathbf{D}}^{-1}\tilde{\mathbf{A}}\), also termed as graph convolution, which serves as a pivotal component. The structure of the networks has been carefully chosen for the following reasons: 1) The decision to fix the second layer in our network structure is a choice to facilitate theoretical derivation. This approach is standard in literature (as seen in references [15; 44]). Even with the second layer fixed, the optimization problem remains non-convex, allowing us to study complex learning dynamics. 2) By observing signal learning and noise memorization (defined in Equation 10), which depend only on the first layer's weight, we can gain valuable insights into how a neural network learns signal and noise from data. This approach enables us to analyze the concrete learning process, contributing to a deeper understanding of GNNs.

### Objective function and test loss

With the training data \(\mathcal{S}\triangleq\{\mathbf{x}_{i},y_{i}\}_{i=1}^{n}\) and \(\mathbf{A}\in\mathbb{R}^{n\times n}\) drawn from \(\mathrm{SNM}-\mathrm{SBM}(n,p,s,\boldsymbol{\mu},\sigma_{p},d)\), we consider to learn the network's parameter \(\mathbf{W}\) by optimizing the empirical cross-entropy loss function:

\[L_{\mathcal{S}}^{\mathrm{GCN}}(\mathbf{W})=\frac{1}{n}\sum_{i=1}^{n}\ell(y_{i} \cdot f(\mathbf{W},\tilde{\mathbf{x}}_{i})). \tag{6}\]

The gradient descent update for the first layer weight \(\mathbf{W}\) in GCN can be expressed as:

\[\mathbf{w}_{j,r}^{(t+1)} =\mathbf{w}_{j,r}^{(t)}-\eta\cdot\nabla_{\mathbf{w}_{j,r}}L_{ \mathcal{S}}^{\mathrm{GCN}}(\mathbf{W}^{(t)})\] \[=\mathbf{w}_{j,r}^{(t)}-\frac{\eta}{nm}\sum_{i=1}^{n}\ell_{i}^{ \prime(t)}\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t)},\tilde{y}_{i} \boldsymbol{\mu}\rangle)\cdot j\tilde{y}_{i}\boldsymbol{\mu}-\frac{\eta}{nm} \sum_{i=1}^{n}\ell_{i}^{\prime(t)}\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t) },\tilde{\boldsymbol{\xi}}_{i}\rangle)\cdot jy_{i}\tilde{\boldsymbol{\xi}}_{i}, \tag{7}\]

where we define "aggregated label" \(\tilde{y}_{i}=D_{i}^{-1}\sum_{k\in\mathcal{N}(i)}y_{k}\) and "aggregated noise vector" \(\tilde{\boldsymbol{\xi}}_{i}=D_{i}^{-1}\sum_{k\in\mathcal{N}(i)}\boldsymbol{\xi}_ {k}\), with \(\mathcal{N}(i)\) being a set that contains all the neighbor of node \(i\).

In this study, our primary objective is to demonstrate the enhanced feature learning capabilities of GNNs in comparison to CNNs. This is achieved by examining the generalization ability of the GNN model through the lens of test loss (population loss). Our test loss is defined based on unseen test data. We build a stochastic block model to model how the data can be generated. What we study is following: given \(n\) training data points and the corresponding graph with \(n\) nodes are generated following the data model. We train a GNN model based on the \(n\) training data points. We suppose that the new test data are generated following the same distribution, and its connection in the graph to the training data points are still following the stochastic block model. We study the loss at the new test data achieved by GNN trained on the \(n\) training points. We specifically study the population loss by taking the expectation over the randomness of the test data, which is formulated as follows:

\[L_{\mathcal{D}}^{\mathrm{GCN}}(\mathbf{W})=\mathbb{E}_{\tilde{\mathbf{x}},y \sim\mathcal{D}=\mathrm{SNM}-\mathrm{SBM}}\ell(y\cdot f(\mathbf{W},\tilde{ \mathbf{x}})). \tag{8}\]

## 4 Thereotical Results

In this section, we introduce our key theoretical findings that elucidate the optimization and generalization processes of feature learning in GCNs. Through the application of the gradient descent rule outlined in Equation (7), we observe that the gradient descent iterate \(\mathbf{w}_{j,r}^{(t)}\) is a linear combination of
its random initialization \(\mathbf{w}_{j,r}^{(0)}\), the signal vector \(\boldsymbol{\mu}\) and the noise vectors in the training data \(\boldsymbol{\xi}_{i}\)1 for \(i\in[n]\). Consequently, for \(r\in[m]\), the decomposition of weight vector iteration can be expressed:

Footnote 1: By referring to Equation (7), we assert that the gradient descent update moves in the direction of \(\tilde{\boldsymbol{\xi}}_{i}\) for each \(i\in[n]\). Then we can apply the definition of \(\tilde{\boldsymbol{\xi}}_{i}=D_{i}^{-1}\sum_{k\in\mathcal{N}(i)}\boldsymbol{ \xi}_{k}\).

\[\mathbf{w}_{j,r}^{(t)}=\mathbf{w}_{j,r}^{(0)}+\gamma_{j,r}^{(t)}\cdot\| \boldsymbol{\mu}\|_{2}^{-2}\cdot\boldsymbol{\mu}+\sum_{i=1}^{n}\rho_{j,r,i}^{(t )}\cdot\|\boldsymbol{\xi}_{i}\|_{2}^{-2}\cdot\boldsymbol{\xi}_{i}. \tag{9}\]

where \(\gamma_{j,r}^{(t)}\) and \(\rho_{j,r,i}^{(t)}\) serve as coefficients. We refer to Equation (9) as the signal-noise decomposition of \(\mathbf{w}_{j,r}^{(t)}\). The normalization factors \(\|\boldsymbol{\mu}\|_{2}^{-2}\) and \(\|\boldsymbol{\xi}_{i}\|_{2}^{-2}\) are introduced to ensure that \(\gamma_{j,r}^{(t)}\approx\langle\mathbf{w}_{j,r}^{(t)},\boldsymbol{\mu}\rangle\), and \(\rho_{j,r,i}^{(t)}\approx\langle\mathbf{w}_{j,r}^{(t)},\boldsymbol{\xi}_{i}\rangle\). We employ \(\gamma_{j,r}^{(t)}\) to characterize the process of signal learning and \(\rho_{j,r,i}^{(t)}\) to characterize the noisy represent. From an intuitive standpoint, if, for some iteration certain \(\gamma_{j,r}^{(t)}\) values are sufficiently large while all \(|\rho_{j,r,i}^{(t)}|\) are relatively small, this indicates that the neural network is primarily learning the label through feature learning. This scenario can lead to _benign overfitting_, characterized by both minimal training and test errors. Conversely, if some \(|\rho_{j,r,i}^{(t)}|\) values are relatively large while all \(\gamma_{j,r}^{(t)}\) are small, the neural network will achieve a low training loss but a high test loss. This occurs as the neural network attempts to generalize by memorizing noise, resulting in a _harmful overfitting_ regime.

To facilitate a fine-grained analysis for the evolution of coefficients, we introduce the notations \(\overline{p}_{j,r,i}^{(t)}\triangleq\rho_{j,r,i}^{(t)}\mathds{1}(\rho_{j,r,i }^{(t)}\geq 0)\), \(\underline{\rho}_{j,r,i}^{(t)}\triangleq\rho_{j,r,i}^{(t)}\mathds{1}(\rho_{j, r,i}^{(t)}\leq 0)\). Consequently, we further express the vector weight decomposition (9) as:

\[\mathbf{w}_{j,r}^{(t)}=\mathbf{w}_{j,r}^{(0)}+j\cdot\gamma_{j,r}^{(t)}\cdot \|\boldsymbol{\mu}\|_{2}^{-2}\cdot\boldsymbol{\mu}+\sum_{i=1}^{n}\overline{p} _{j,r,i}^{(t)}\cdot\|\boldsymbol{\xi}_{i}\|_{2}^{-2}\cdot\boldsymbol{\xi}_{i} +\sum_{i=1}^{n}\underline{\rho}_{j,r,i}^{(t)}\cdot\|\boldsymbol{\xi}_{i}\|_{2} ^{-2}\cdot\boldsymbol{\xi}_{i}. \tag{10}\]

Our analysis will be made under the following set of assumptions:

**Assumption 4.1**.: Suppose that

1. The dimension \(d\) is sufficiently large: \(d=\tilde{\Omega}(m^{2\nu[4/(q-2)]}n^{4\nu[(2q-2)/(q-2)]})\).
2. The size of training sample \(n\) and width of GCNs \(m\) adhere to \(n,m=\Omega(\operatorname{polylog}(d))\).
3. The learning rate \(\eta\) satisfies \(\eta\leq\tilde{O}(\min\{\|\boldsymbol{\mu}\|_{2}^{-2},\sigma_{p}^{-2}d^{-1} \})\).
4. The edge probability \(p,s=\Omega(\sqrt{\log(n)/n})\) and \(\Xi\triangleq\frac{p-s}{p+s}\) is a positive constant.
5. The standard deviation of Gaussian initialization \(\sigma_{0}\) is chosen such that \(\sigma_{0}\leq\tilde{O}(m^{-2/(q-2)}n^{-[1/(q-2)]\lor 1}\cdot\min\{(\sigma_{p} \sqrt{d/(n(p+s))})^{-1},\Xi^{-1}\|\boldsymbol{\mu}\|_{2}^{-1}\}\).

_Remark 4.2_.: (1) The requirement for a high dimension in our assumptions is specifically aimed at ensuring that the learning occurs in a sufficiently over-parameterized setting [49, 15] when the second layer remains fixed. We are interested in exploring the over-parameterization scenario where both GNNs and CNNs can be trained to achieve arbitrarily small training loss, and then comparing the test losses after the training loss has been minimized. (2) It's necessary for the sample size and neural network width to be at least polylogarithmic in the dimension \(d\). This condition ensures certain statistical properties of the training data and weight initialization hold with a probability of at least \(1-d^{-1}\). (3) The condition on \(\eta\) is to ensure that gradient descent can effectively minimize the training loss. (4) The assumption regarding edge probability guarantees a sufficient level of concentration in the degree and an adequate display of homophily of graph data. (5) Lastly, the conditions imposed on initialization strength \(\sigma_{0}\) are intended to guarantee that the training loss can effectively converge to a sufficiently small value and to discern the differential learning speed between signal and noise.

Given the above assumptions, we present our main result on feature learning of GCNs in the following theorem.

**Theorem 4.3**.: _Suppose \(\epsilon>0\), and let \(T=\tilde{\Theta}(\eta^{-1}m\sigma_{0}^{-(q-2)}\Xi^{-q}\|\boldsymbol{\mu}\|_{2} ^{-q}+\eta^{-1}\epsilon^{-1}m^{3}\|\boldsymbol{\mu}\|_{2}^{-2})\). Under Assumption 4.1, if \(n\cdot\operatorname{SNR}^{q}\cdot\sqrt{n(p+s)}^{q-2}=\tilde{\Omega}(1)\), where \(\operatorname{SNR}\triangleq\|\boldsymbol{\mu}\|_{2}/(\sigma_{p}\sqrt{d})\) is the signal-to-noise ratio, then with probability at least \(1-d^{-1}\), there exists a \(0\leq t\leq T\) such that:_
* _The GCN learns the signal:_ \(\max_{r}\gamma_{j,r}^{(t)}=\Omega(1)\) _for_ \(j\in\{\pm 1\}\)_._
* _The GCN does not memorize the noises in the training data:_ \(\max_{j,r,i}|\rho_{j,r,i}^{(T)}|=\tilde{O}(\sigma_{0}\sigma_{p}\sqrt{d/n(p+s)})\)_._
* _The training loss converges to_ \(\epsilon\)_, i.e.,_ \(L_{\mathcal{S}}^{\mathrm{GCN}}(\mathbf{W}^{(t)})\leq\epsilon\)_._
* _The trained GCN achieves a small test loss:_ \(L_{\mathcal{D}}^{\mathrm{GCN}}(\mathbf{W}^{(t)})\leq c_{1}\epsilon+\exp(-c_{2} n^{2})\)_._

_where \(c_{1}\) and \(c_{2}\) are positive constants._

Theorem 4.3 outlines the scenario of _benign overfitting_ for GCNs. It reveals that, provided \(n\cdot\mathrm{SNR}^{q}\cdot\sqrt{n(p+s)}^{q-2}=\tilde{\Omega}(1)\), the GCN can learn the signal by achieving \(\max_{r}\gamma_{j,r}^{(t)}=\Omega(1)\) for \(j\in\{\pm 1\}\), and on the other hand, the noise memorization during gradient descent training is suppressed by \(\max_{j,r,i}|\rho_{j,r,i}^{(T)}|=\tilde{O}(\sigma_{0}\sigma_{p}\sqrt{d/n(p+s)})\), given that \(\sigma_{0}\sigma_{p}\sqrt{d/n(p+s)}\ll 1\) according to assumption 4.1. Because the signal learned by the network is large enough and much stronger than the noise memory, it can perfectly predict the label in the test sample according to the learned signal when it generalizes. Consequently, the learned neural network can attain small training and test losses.

To illustrate the pronounced divergence between GNN and CNN in terms of generalization capability post-gradient descent training, we show that, under identical conditions, a GCN engages in signal learning while a CNN emphasizes noise memorization, and thus diverges in the ability of generalization:

**Corollary 4.4** (Informal).: _Under assumption 4.1, if \(n\cdot\mathrm{SNR}^{q}\cdot\sqrt{n(p+s)}^{q-2}=\tilde{\Omega}(1)\) and \(n^{-1}\cdot\mathrm{SNR}^{-q}=\tilde{\Omega}(1)\), then with probability at least \(1-d^{-1}\), then there exists a \(t\) such that:_

* _The trained GNN achieves a small test loss:_ \(L_{\mathcal{D}}^{\mathrm{GCN}}(\mathbf{W}^{(t)})\leq c_{1}\epsilon+\exp(-c_{2} n^{2})\)_._
* _The trained CNN has a constant order test loss:_ \(L_{\mathcal{D}}^{\mathrm{CNN}}(\mathbf{W}^{(t)})=\Theta(1)\)_._

Corollary 4.4 clearly provides a condition that GNNs learn the signal and achieves a small test loss while the CNNs can only memorize noises and will have a \(\Theta(1)\) test loss. Whether a CNN learns the signal or noise depends on the signal-to-noise ratio (\(\mathrm{SNR}\)), and the number of samples \(n\). According to Cao et al. [15], CNN can learn the signal and filter out the noise when \(n\cdot\mathrm{SNR}^{q}>1\). On the other hand, when \(n\cdot\mathrm{SNR}^{q}<1\), as focused in this work, the strength of signal and number sample are not large enough for a CNN to focus on the signal learning and generalize well on the unseen data. We have illustrated this demontration in Figure 2. The improvement in _benign overfitting_ regime is facilitated by graph convolution, a process that will be elaborated on in the subsequent section. Through the precise characterization of neural network feature learning from optimization to generalization, we have successfully demonstrated that the graph neural network can gain superiority with the help of graph convolution.

## 5 Proof Sketches

In this section, we outline the proof sketches, drawing inspiration from the study of feature learning in CNNs. Our approach follows and builds upon the methodologies described in [15], allowing us to extend and apply these concepts in a new context. We discuss the primary challenges encountered during the study of GNN training, and illustrate the key techniques we employed in our proofs to overcome these challenges:

* Graph convolution aggregates information from neighboring nodes to the central node, which often leads to the loss of statistical stability for the aggregated noise vectors and labels. To overcome this challenge, we utilize a dense graph input, achieved by setting the edge probability as per Assumption 4.1.
* Graph convolution can potentially cause erratic iterative dynamics of coefficients during the feature learning process. To mitigate this issue, we introduce the concept of homophily into the graph input, which helps in stabilizing the coefficient iterations.
* Lastly, for the generalization analysis, depicting the generalization ability of graph neural networks poses a significant challenge. To address this issue, we introduce an expectation over the distribution for a single data point and develop an algorithm-dependent test error analysis.

These main techniques are further elaborated upon in the following sections, and detailed proofs for all the results can be found in the appendix.

### Iterative analysis of the signal-noise decomposition under graph convolution

To analyze the feature learning process of graph neural networks during gradient descent training, we introduce an iterative methodology, based on the signal-noise decomposition in decomposition (10) and gradient descent update (7). The following lemma offers us a means to monitor the iteration of the signal learning and noise memorization under graph convolution:

**Lemma 5.1**.: _The coefficients \(\gamma_{j,r}^{(t)},\overline{\rho}_{j,r,i}^{(t)},\underline{\rho}_{j,r,i}^{(t)}\) in decomposition (10) adhere to the following equations:_

\[\gamma_{j,r}^{(0)},\overline{\rho}_{j,r,i}^{(0)},\underline{\rho }_{j,r,i}^{(0)}=0, \tag{11}\] \[\gamma_{j,r}^{(t+1)}=\gamma_{j,r}^{(t)}-\frac{\eta}{nm}\cdot \sum_{i=1}^{n}\ell_{i}^{(t)}\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t)}, \tilde{y}_{i}\boldsymbol{\mu}_{i}\rangle)y_{i}\tilde{y}_{i}\|\boldsymbol{ \mu}\|_{2}^{2},\] (12) \[\overline{\rho}_{j,r,i}^{(t+1)}=\overline{\rho}_{j,r,i}^{(t)}- \frac{\eta}{nm}\cdot\sum_{k\in\mathcal{N}(i)}D_{k}^{-1}\cdot\ell_{k}^{\prime (t)}\cdot\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t)},\tilde{\boldsymbol{ \xi}}_{k}\rangle)\cdot\|\boldsymbol{\xi}_{i}\|_{2}^{2}\cdot\mathds{1}(y_{k}=j),\] (13) \[\underline{\rho}_{j,r,i}^{(t+1)}=\underline{\rho}_{j,r,i}^{(t)}+ \frac{\eta}{nm}\cdot\sum_{k\in\mathcal{N}(i)}D_{k}^{-1}\cdot\ell_{k}^{\prime (t)}\cdot\sigma^{\prime}(\langle\mathbf{w}_{j,r}^{(t)},\tilde{\boldsymbol{ \xi}}_{k}\rangle)\cdot\|\boldsymbol{\xi}_{i}\|_{2}^{2}\cdot\mathds{1}(y_{k}=-j). \tag{14}\]

Lemma 5.1 simplifies the analysis of the feature learning in GCNs by reducing it to the examination of the discrete dynamical system expressed by equations (11)-(14). Our proof strategy emphasizes an in-depth evaluation of the coefficient values \(\gamma_{j,r}^{(t)},\overline{\rho}_{j,r,i}^{(t)},\underline{\rho}_{j,r,i}^{( t)}\) throughout the training.

Figure 2: Illustration of performance comparison between GNN and CNN. The benign overfitting represents the setting under which the overfitted NN trained by gradient descent is guaranteed to have small test loss, and the harmful overfitting region represents the setting under which the test loss is of constant order (large). The red curve represents the separation condition between two phases for CNN, namely \(n\cdot\mathrm{SNR}^{q}=1\)[15]. Above the red line, CNN generalizes well; below the red line, CNN generalizes poorly. On the other hand, the blue curve represents the separation condition for GNN, namely \(n\cdot\mathrm{SNR}^{q}\sqrt{n(p+s)}^{q-2}=1\) (Theorem 4.3). Above the blue curve, GNN generalizes well. The orange band region above the blue curve but below the red curve highlights where GNN can outperform CNN.
### A two-phase dynamics analysis

We then propose a two-stage dynamics analysis to elucidate the behavior of these coefficients inspired by [15]. Subsequently, we can depict the generalization ability of GCN with the learned weight.

Stage 1.Intuitively, the initial neural network weights are small enough so that the neural network at initialization has constant level cross-entropy loss derivatives on all the training data: \(\ell_{i}^{\prime(0)}=\ell^{\prime}[y_{i}\cdot f(\mathbf{W}^{(0)},\tilde{\mathbf{ x}}_{i})]=\Theta(1)\) for all \(i\in[n]\). This is guaranteed under Condition 4.1 on \(\sigma_{0}\). Motivated by this, the dynamics of the coefficients in (12) - (14) can be greatly simplified by replacing the \(\ell_{i}^{\prime(t)}\) factors by their constant upper and lower bounds. The following lemma summarizes our main conclusion at stage 1 for signal learning:

**Lemma 5.2**.: _Under the same conditions as Theorem 4.3, there exists \(T_{1}=\tilde{O}(\eta^{-1}m\sigma_{0}^{2-q}\Xi^{-q}\|\boldsymbol{\mu}\|_{2}^{-q})\) such that_

* \(\max_{r}\gamma_{j,r}^{(T_{1})}=\Omega(1)\) _for_ \(j\in\{\pm 1\}\)_._
* \(|\rho_{j,r,i}^{(t)}|=O\left(\sigma_{0}\sigma_{p}\sqrt{d}/\sqrt{n(p+s)}\right)\) _for all_ \(j\in\{\pm 1\}\)_,_ \(r\in[m]\)_,_ \(i\in[n]\) _and_ \(0\leq t\leq T_{1}\)_._

The proof can be found in Appendix C.1. Lemmas 5.2 leverages the period of training when the derivatives of the loss function are of a constant order. It's important to note that graph convolution plays a significant role in diverging the learning speed between signal learning and noise memorization in this first stage. Originally, the learning speeds are roughly determined by \(\|\boldsymbol{\mu}\|_{2}\) and \(\|\boldsymbol{\xi}\|_{2}\) respectively. However, after applying graph convolution, the learning speeds are approximately determined by \(|\tilde{y}|\cdot\|\boldsymbol{\mu}\|_{2}\) and \(\|\boldsymbol{\xi}\|_{2}\) respectively. Here, \(|\tilde{y}|\cdot\|\boldsymbol{\mu}\|_{2}\) is close to \(\|\boldsymbol{\mu}\|_{2}\), but \(\|\tilde{\boldsymbol{\xi}}\|_{2}\) is much smaller than \(\|\boldsymbol{\xi}\|_{2}\) (see Figure 3 for an illustration). This means that graph convolution can slow down the learning speed of noise memorization, thus enabling GNNs to focus more on signal learning in the initial training stage.

Stage 2.Building on the results from the first stage, we then move to the second stage of the training process. In this stage, the loss derivatives are no longer constant, and we demonstrate that the training loss can be minimized to an arbitrarily small amount. Importantly, the scale differences established during the first stage of learning continue to be maintained throughout the training process:

**Lemma 5.3**.: _Let \(T,T_{1}\) be defined in Theorem 4.3 and Lemma 5.2 respectively and \(\mathbf{W}^{*}\) be the collection of GCN parameters \(\mathbf{w}_{j,r}^{*}=\mathbf{w}_{j,r}^{(0)}+2qm\log(2q/\epsilon)\cdot j\cdot \|\boldsymbol{\mu}\|_{2}^{-2}\cdot\boldsymbol{\mu}\). Then under the same conditions as Theorem 4.3, for any \(t\in[T_{1},T]\), it holds that:_

Figure 3: An illustrative example of noise vector before and after graph aggregation. In this example, we consider \(d=3\) and all degree are 1. The black vectors stand for noise vectors \(\boldsymbol{\xi}\) before graph convolution. Each of them are orthogonal to each other. The red vectors represent noise vectors after graph convolution \(\tilde{\boldsymbol{\xi}}\). They are graph convoluted noise vectors of two original noise vectors. Note that the \(\ell_{2}\) norm between two kinds of vector follows \(\|\tilde{\boldsymbol{\xi}}\|_{2}=\frac{\sqrt{2}}{2}\|\boldsymbol{\xi}\|_{2}\). This plot demonstrates how graph convolution shrinks the \(\ell_{2}\) norm of noise vectors.
* \(\max_{r}\gamma_{j,r}^{(T_{1})}\geq 2,\forall j\in\{\pm 1\}\) and \(|\rho_{j,r,i}^{(t)}|\leq\sigma_{0}\sigma_{p}\sqrt{d/(n(p+s))}\) for all \(j\in\{\pm 1\}\), \(r\in[m]\) and \(i\in[n]\).
* \(\frac{1}{t-T_{1}+1}\sum_{s=T_{1}}^{t}L_{\mathcal{S}}^{\mathrm{SCN}}(\mathbf{W}^ {(s)})\leq\frac{\|\mathbf{W}^{(T_{1})}-\mathbf{W}^{\star}\|_{F}^{2}}{(2q-1) \eta(t-T_{1}+1)}+\frac{\epsilon}{(2q-1)}\).

_Here we denote \(\|\mathbf{W}\|_{F}\triangleq\sqrt{\|\mathbf{W}_{+1}\|_{F}^{2}+\|\mathbf{W}_{- 1}\|_{F}^{2}}\)._

Lemma 5.3 presents two primary outcomes related to feature learning. Firstly, throughout this training phase, it ensures that the coefficients of noise vectors, denoted as \(\rho_{j,r,i}^{(t)}\), retain a significantly small value while coefficients of feature vector, denoted as \(\gamma_{j,r}^{(t)}\) can achieve large value. Furthermore, it offers an optimization-oriented outcome, indicating that the optimal iterate within the interval \([T_{1},T]\). In this process, graph convolution and gradient descent will continue to maintain the speed gap between signal learning and noise memory, and when the time is large enough, the training loss will tend to receive an arbitrarily small value.

### Test error analysis

Finally, we consider a new data point \((\mathbf{x},y)\) drawn from the distribution SNM-SBM. The lemma below further gives an upper bound on the test loss of GNNs post-training:

**Lemma 5.4**.: _Let \(T\) be defined in Theorem 4.3. Under the same conditions as Theorem 4.3, for any \(t\leq T\) with \(L_{\mathcal{S}}^{\mathrm{SCN}}(\mathbf{W}^{(t)})\leq 1\), it holds that \(L_{\mathcal{D}}^{\mathrm{SCN}}(\mathbf{W}^{(t)})\leq c_{1}\cdot L_{\mathcal{S }}^{\mathrm{SCN}}(\mathbf{W}^{(t)})+\exp(-c_{2}n^{2})\)._

The proof is presented in the appendix. Lemma 5.4 demonstrates that GNNs achieve _benign overfitting_ and completes the last step of feature learning theory.

## 6 Experiments

In this section, we validate our theoretical findings through numerical simulations using synthetic data, specifically generated according to the SNM-SBM model. We set the signal vector, \(\mathbf{\mu}\), to drawn from a standard normal distribution \(\mathcal{N}(\mathbf{0},\mathbf{I})\). The noise vector, \(\mathbf{\xi}\), is drawn from a Gaussian distribution \(\mathcal{N}(\mathbf{0},\sigma_{p}^{2}\mathbf{I})\). We train a two-layer CNN defined as per equation (2) and a two-layer GNN as per equation (5) with polynomial ReLU \(q=3\). We used the gradient descent method with a learning rate of \(\eta=0.03\). The primary task we focused on was node classification, where the goal was to predict the class labels of nodes in a graph.

Feature learning dynamics.Firstly, we display the training loss, test loss, training accuracy, and test accuracy for both the CNN and GNN in Figure 4. In this case, we further set the training data size to \(n=250\), input dimension to \(d=500\), noise strength to \(\sigma_{p}=20\), and edge probability to \(p=0.5\), \(s=0.08\). We observe that both the GNN and CNN can achieve zero training error. However, while the GNN obtains nearly zero test error, the CNN fails to generalize effectively to the test set. This simulation result serves to validate our theoretical results in Theorem 4.3 and Corollary 4.4.

Verification via real-world data.We conducted an experiment using real data, specifically by replacing the synthetic feature with MNIST input features. We select numbers 1 and 2 from the ten digital numbers, and applied both CNN and GNN models as described in our paper. Detailed results and visualizations can be found in the Figure 5. The results were consistent with our theoretical

Figure 4: Training loss, testing loss, training accuracy, and testing accuracy for both CNN and GNN over a span of 100 training epochs.
conclusions, reinforcing the insights derived from our analysis. We believe that this experiment adds a valuable dimension to our work, bridging the gap between theory and practice.

Phase diagram.We then explore a range of Signal-to-Noise Ratios (SNRs) from 0.045 to 0.98, and a variety of sample sizes, \(n\), ranging from 200 to 7200. Based on our results, we train the neural network for 200 steps for each combination of SNR and sample size \(n\). After training, we calculate the test accuracy for each run. The results are presented as a heatmap in Figure 6. Compared to CNNs, GCNs demonstrate a perfect accuracy score of 1 across a more extensive range in the SNR and \(n\) plane, indicating that GNNs have a broader _benign overfitting_ regime. This further validates our theoretical findings.

## 7 Conclusion and Limitations

This paper utilizes a signal-noise decomposition to study the signal learning and noise memorization process in training a two-layer GCN. We provide specific conditions under which a GNN will primarily concentrate on signal learning, thereby achieving low training and testing errors. Our results theoretically demonstrate that GCNs, by leveraging structural information, outperform CNNs in terms of generalization ability across a broader benign overfitting regime. As a pioneering work that studies feature learning of GNNs, our theoretical framework is constrained to examining the role of graph convolution within a specific two-layer GCN and a certain data generalization model. In fact, the feature learning of a neural network can be influenced by a myriad of other factors, such as activation function, optimization algorithm, and data model [48, 35, 37]. Future work can extend our framework to consider the influence of a wider array of factors on feature learning within GCNs.
