Wide Neural Networks as Gaussian Processes: Lessons from Deep
  Equilibrium Models
Neural networks with wide layers have attracted significant attention due to
their equivalence to Gaussian processes, enabling perfect fitting of training
data while maintaining generalization performance, known as benign overfitting.
However, existing results mainly focus on shallow or finite-depth networks,
necessitating a comprehensive analysis of wide neural networks with
infinite-depth layers, such as neural ordinary differential equations (ODEs)
and deep equilibrium models (DEQs). In this paper, we specifically investigate
the deep equilibrium model (DEQ), an infinite-depth neural network with shared
weight matrices across layers. Our analysis reveals that as the width of DEQ
layers approaches infinity, it converges to a Gaussian process, establishing
what is known as the Neural Network and Gaussian Process (NNGP) correspondence.
Remarkably, this convergence holds even when the limits of depth and width are
interchanged, which is not observed in typical infinite-depth Multilayer
Perceptron (MLP) networks. Furthermore, we demonstrate that the associated
Gaussian vector remains non-degenerate for any pairwise distinct input data,
ensuring a strictly positive smallest eigenvalue of the corresponding kernel
matrix using the NNGP kernel. These findings serve as fundamental elements for
studying the training and generalization of DEQs, laying the groundwork for
future research in this area.
Tianxiang Gao, Xiaokai Huo, Hailiang Liu, Hongyang Gao
# Wide Neural Networks as Gaussian Processes:

###### Abstract

Neural networks with wide layers have attracted significant attention due to their equivalence to Gaussian processes, enabling perfect fitting of training data while maintaining generalization performance, known as benign overfitting. However, existing results mainly focus on shallow or finite-depth networks, necessitating a comprehensive analysis of wide neural networks with infinite-depth layers, such as neural ordinary differential equations (ODEs) and deep equilibrium models (DEQs). In this paper, we specifically investigate the deep equilibrium model (DEQ), an infinite-depth neural network with shared weight matrices across layers. Our analysis reveals that as the width of DEQ layers approaches infinity, it converges to a Gaussian process, establishing what is known as the Neural Network and Gaussian Process (NNGP) correspondence. Remarkably, this convergence holds even when the limits of depth and width are interchanged, which is not observed in typical infinite-depth Multilayer Perceptron (MLP) networks. Furthermore, we demonstrate that the associated Gaussian vector remains non-degenerate for any pairwise distinct input data, ensuring a strictly positive smallest eigenvalue of the corresponding kernel matrix using the NNGP kernel. These findings serve as fundamental elements for studying the training and generalization of DEQs, laying the groundwork for future research in this area.

## 1 Introduction

Neural networks with wide layers have recently received significant attention due to their intriguing equivalence to Gaussian processes, known as the Neural Network and Gaussian Process (NNGP) correspondence. It has been established that two-layer fully-connected networks tend towards Gaussian processes as the width of the layers approaches infinity [29; 25]. This equivalence has also been theoretically demonstrated in various neural network architectures, including deep feed-forward networks [28], convolutional neural networks [33; 16], recurrent networks [39], and residual neural networks [35]. This equivalence not only sheds light on the training dynamics of these networks but also highlights their generalization performance, especially when the corresponding covariance matrix is strictly positive definite. These discoveries have paved the way for overparameterized neural networks to achieve perfect fit on training data [11; 31; 3], while maintaining low generalization error on unseen data [4; 30; 2], a phenomenon known as benign overfitting [7; 8; 27].
In recent years, the emergence of infinite-depth neural network architectures, such as neural ordinary differential equations (ODEs) [9] and deep equilibrium models (DEQs) [6], has demonstrated their potential to capture complex dynamic behaviors and achieve superior modeling capabilities [24; 34; 22; 6]. However, the analysis of these architectures in the context of wide neural networks with infinite-depth layers remains largely unexplored. Understanding the convergence properties and relationship to Gaussian processes of these networks is crucial to unravel their underlying mechanisms and unlock their full potential. While some limited studies have investigated the convergence properties of neural ODEs or ResNet architectures [21], such as demonstrating the convergence to a diffusion process in the infinite-depth limit for a specific ResNet architecture [35] and introducing scaling to allow the interchange of the two limits [20], to the best of our knowledge, there is no existing work studying the commutative limits of DEQs.

In this paper, we focus on analyzing the deep equilibrium model (DEQ), an infinite-depth neural network architecture with shared weight matrices across layers. Our objective is to comprehensively analyze the properties of DEQs in the context of wide neural networks and investigate their convergence behavior as the width of the layers tends to infinity. We establish that as the width approaches infinity, DEQ tends to a Gaussian process. Furthermore, under appropriate scaling of the weight matrices, the limits of depth and width commute, enhancing our understanding of the convergence properties of DEQs. Additionally, we demonstrate that the resulting covariance function is strictly positive definite for any distinct input data, provided that the activation function is non-polynomial.

## 2 Related Works

Implicit neural networks [12], such as deep equilibrium models (DEQs), have gained significant attention in the research community over the past decade. Recent studies have shown that implicit neural network architecture encompasses a broader class of models, making it a versatile framework that includes feed-forward neural networks, convolutional neural networks, residual networks, and recurrent neural networks [12; 6]. Moreover, DEQs have been recognized for their competitive performance compared to standard deep neural networks, offering the advantage of achieving comparable results while demanding much fewer computational and memory resources, especially due to the utilization of shared weights [6]. Despite the practical success of DEQs in various real-world applications, our theoretical understanding of DEQs remains limited.

On the other hand, numerous studies [29; 25; 28; 33; 16; 39; 35] have made observations that finite-depth neural networks with random initialization tend to exhibit behavior similar to Gaussian processes as the width approaches infinity, known as NNGP correspondence. This correspondence has led to investigations into the global convergence properties of gradient-based optimization methods. The work of [23] established that the trajectory of the gradient-based method can be characterized by the spectral property of a kernel matrix that is computed by the so-called neural tangent kernel (NTK). Consequently, if the limiting covariance function or NNGP kernel \(\Sigma^{L}\) can be shown to be strictly positive definite under mild conditions, simple first-order methods such as stochastic gradient descent can be proven to converge to a global minimum at a linear rate, provided the neural networks are sufficiently overparameterized [11; 3; 40; 32; 5; 31; 15; 13]. Furthermore, this equivalence offers valuable insights into the generalization performance of neural networks on unseen data. It suggests that wide neural networks can be viewed as kernel methods, and the Rademacher complexity of these networks can be easily computed if the parameter values remain bounded during training. As a result, a line of current research [4; 30; 2; 7; 8; 27; 14] has demonstrated that gradient-based methods can train neural networks of various architectures to achieve arbitrarily small generalization error, given that the neural networks are sufficiently overparameterized.

Unfortunately, the existing results in the literature primarily focus on finite-depth neural networks, and there has been relatively limited research investigating the training and generalization properties of infinite-depth neural networks. One key challenge arises when the limits of depth and width do not commute, leading to distinct behaviors depending on whether the depth or width is relatively larger. For instance, [35] demonstrates that a ResNet with bounded width tends to exhibit a diffusion process, while [20] observes heavy-tail distributions in standard MLPs when the depth is relatively larger than the width. These unstable behaviors give rise to a loss of expressivity in large-depth neural networks, specifically in terms of perfect correlation among network outputs for different inputs, as highlighted by studies such as [36; 37; 19]. This raises significant issue, as it indicates the networks loss the covariance structure from the inputs as growth of depth. In the case of ResNets,
a proposed solution to mitigate this problem involves employing a carefully chosen scaling on the residual branches [20; 18], resulting in commutative limits of depth and width. However, to the best of our knowledge, there is currently no research exploring the interplay between the limits of depth and width for DEQs, which represent another class of infinite-depth neural networks with shared weights.

## 3 Preliminary and Overview of Results

In this paper, we consider a simple deep equilibrium model \(f_{\theta}(x)\) defined as follows:

\[f_{\theta}(x)= V^{T}h^{*}(x). \tag{1}\]

Here, \(h^{0}(x)=0\), and \(h^{*}(x)\) represents the limit of the transition defined by:

\[h^{\ell}(x) =\phi(W^{T}h^{\ell-1}(x)+U^{T}x), \tag{2}\] \[h^{*}(x) =\lim_{\ell\to\infty}h^{\ell}(x), \tag{3}\]

where \(\phi(\cdot)\) is an activation function. The parameters are defined as \(U\in\mathbb{R}^{n_{in}\times n}\), \(W\in\mathbb{R}^{n\times n}\), and \(V\in\mathbb{R}^{n\times n_{out}}\). The following equilibrium equality arises from the fact that \(h^{*}(x)\) is a fixed point of the equation (2):

\[h^{*}(x)=\phi(W^{T}h^{*}(x)+U^{T}x). \tag{4}\]

To initialize the parameters \(\theta=:\text{vec}\left(U,W,V\right)\), we use random initialization as follows:

\[U_{ij}\stackrel{{\text{iid}}}{{\asymp}}\mathcal{N}\left(0, \frac{\sigma_{u}^{2}}{n_{in}}\right),\quad W_{ij}\stackrel{{\text {iid}}}{{\asymp}}\mathcal{N}\left(0,\frac{\sigma_{w}^{2}}{n}\right),\quad V_{ ij}\stackrel{{\text{iid}}}{{\asymp}}\mathcal{N}\left(0,\frac{\sigma_{v}^{2}}{n} \right), \tag{5}\]

where \(\sigma_{u},\sigma_{w},\sigma_{v}>0\) are fixed variance parameters.

To ensure the well-definedness of the neural network parameterized by (1), we establish sufficient conditions for the existence of the unique limit \(h^{*}\) by leveraging fundamental results from random matrix theory applied to random square matrices \(A\in\mathbb{R}^{n\times n}\):

\[\lim_{n\to\infty}\frac{\|A\|_{op}}{\sqrt{n}}=\sqrt{2}\quad\text{ almost surely (a.s.)},\]

where \(\|A\|_{op}\) denotes the operator norm of \(A\).

**Proposition 3.1** (Informal Version of Lemma F.1).: _There exists an absolute small constant \(\sigma_{w}>0\) such that \(h^{*}(x)\) is uniquely determined almost surely for all \(x\)._

While similar results have been obtained in [15] using non-asymptotic analysis, our contribution lies in the asymptotic analysis, which is essential for studying the behaviors of DEQ under the limit of width approaching infinity. This asymptotic perspective allows us to investigate the convergence properties and relationship between the limits of depth and width. We refer readers to Section 4 and Theorem 4.4 where we leverage this result to demonstrate the limits of depth and width commutes.

After ensuring the well-definedness of \(h^{*}(x)\), the next aspect of interest is understanding the behavior of the neural network \(f_{\theta}\) as a random function at the initialization. Previous studies [28; 33; 16; 39; 35] have demonstrated that finite-depth neural networks behave as Gaussian processes when their width \(n\) is sufficiently large. This raises the following question:

_Q1: Do wide neural networks still exhibit Gaussian process behavior when they have infinite-depth, particularly with shared weights?_

Unfortunately, the answer is generally _No_. The challenge arises when the limits of depth and width do not commute. Several studies have observed that switching the convergence sequence of depth and width leads to different limiting behaviors. While wide neural networks behave as Gaussian processes, [26; 20] have observed heavy-tail distributions when the depth becomes relatively larger than the width. However, in the case of DEQs, we demonstrate that such deviations from Gaussian process behavior do not occur, since the infinite width limit and infinite depth limit do commute for DEQs given by (2). This crucial property is established through a meticulous analysis, focusing on fine-grained analysis to accurately determine the convergence rates of the two limits. Our findings affirm the stability and consistent Gaussian process behavior exhibited by DEQs, reinforcing their unique characteristics in comparison to other wide neural networks with infinite-depth layers.
**Theorem 3.1** (Informal Version of Theorem 4.4).: _Under the limit of width \(n\to\infty\), the neural network \(f_{\theta}\) defined on (1) tends to a centered Gaussian process with a covariance function \(\Sigma^{*}\)._

Once we have confirm width DEQs acts as Gaussian process, given a set of inputs, the corresponding multidimensional Gaussian random vectors are of interest, especially the nondegeneracy of the covariance matrix. This raises the following question:

_Q2: Is the covariance function \(\Sigma^{*}\) strictly positive definite?_

If the covariance function \(\Sigma^{*}\) of the Gaussian process associated with the DEQ is strictly positive definite, it implies that the corresponding covariance matrix is nondegenerate and has a strictly positive least eigenvalue. This property is crucial in various classical statistical analyses, including inference, prediction, and parameter estimation. Furthermore, the strict positive definiteness of \(\Sigma^{*}\) has implications for the global convergence of gradient-based optimization methods used in training neural networks. In the context of wide neural networks, these networks can be viewed as kernel methods under gradient descent, utilizing the NTK [23]. By making appropriate assumptions on the activation function \(\phi\), we establish that the covariance function \(\Sigma^{*}\) of DEQs is indeed strictly positive definite, meaning that the corresponding covariance matrix \(K^{*}\) has strictly positive least eigenvalue when the inputs are distinct.

**Theorem 3.2** (Informal Version of Theorem 4.5).: _If the activation function \(\phi\) is nonlinear but non-polynomial, then the covariance function \(\Sigma^{*}\) is strictly positive definite._

These findings expand the existing literature on the convergence properties of infinite-depth neural networks and pave the way for further investigations into the training and generalization of DEQs.

## 4 Main Results

To study the DEQ, we introduce the concept of finite-depth neural networks, denoted as \(f^{L}_{\theta}(x)=V^{T}h^{L-1}(x)\), where \(h^{\ell}(x)\) represents the post-activation values. The definition of \(h^{\ell}(x)\) for \(\ell\in[L-1]\) is as follows:

\[\begin{split} g^{1}(x)&=U^{T}x,\quad h^{1}(x)=\phi (g^{1}(x)),\\ g^{\ell}(x)&=W^{T}h^{\ell-1},\quad h^{\ell}(x)= \phi(g^{\ell}(x)+g^{1}(x)),\;\text{for}\;\ell=2,3,\ldots,L-1.\end{split} \tag{6}\]

**Remark 4.1**.: _We assume \(U\), \(W\), and \(V\) are randomly initialized according to (5). The post-activation values \(h^{\ell}\) differ slightly from those in classical Multilayer Perceptron (MLP) models due to the inclusion of input injection. It is worth mentioning that \(f^{L}_{\theta}\) is equivalent to the DEQ \(f_{\theta}\) when we let \(L\to\infty\), provided that the limit exists._

### \(f^{L}_{\theta}\) as a Gaussian Process

The finite-depth neural network \(f^{L}_{\theta}\) can be expressed as a Tensor Program, which is a computational algorithm introduced in [39] for implementing neural networks. In their work, [39] provides examples of various neural network architectures represented as tensor programs. They also establish that all G-var (or pre-activation vectors in our case) in a tensor program tend to Gaussian random variables as the width \(n\) approaches infinity [39, Theorem 5.4]. Building upon this result, we can employ a similar argument to demonstrate that the neural network \(f^{L}_{\theta}\) defined by (6) converges to a Gaussian process, with the covariance function computed recursively, under the assumption of a controllable activation function.

**Definition 4.1**.: _A real-valued function \(\phi:\mathbb{R}^{k}\to\mathbb{R}\) is called **controllable** if there exists some absolute constants \(C,c>0\) such that \(|\phi(x)|\leq Ce^{c\sum_{i=1}^{k}|x_{i}|}\)._

It is important to note that controllable functions are not necessarily smooth, although smooth functions can be easily shown to be controllable. Moreover, controllable functions, as defined in [39, Definition 5.3], can grow faster than exponential but remain \(L^{1}\) and \(L^{2}\)-integrable with respect to the Gaussian measure. However, the simplified definition presented here encompasses almost most functions encountered in practice.

Considering the activation function \(\phi\) as controllable and conditioned on previous layers, we observe that the pre-activation \(g^{\ell}_{k}(x)\) behaves like independent and identically distributed (i.i.d.) Gaussian
random variables. Through induction, both the conditioned and unconditioned distributions of \(g_{k}^{\ell}(x)\) converge to the same Gaussian random variable \(z^{\ell}(x)\) as the limit approaches infinity. This result is proven in Appendix B.

**Theorem 4.1**.: _For a finite-depth neural network \(f_{\theta}^{L}\) defined in (6), as the width \(n\to\infty\), the output functions \(f_{\theta,k}^{L}\) for \(k\in[1,n_{out}]\) tends to centered Gaussian processes in distribution with covariance function \(\Sigma^{L}\) defined recursively as follows: for all \(\ell\in[2,L-1]\)_

\[\Sigma^{1}(x,x^{\prime}) =\sigma_{u}^{2}\left\langle x,x^{\prime}\right\rangle \tag{7}\] \[\Sigma^{2}(x,x^{\prime}) =\sigma_{w}^{2}\mathbb{E}\phi(z^{1}(x))\phi(z^{1}(x^{\prime}))\] (8) \[\Sigma^{\ell+1}(x,x^{\prime}) =\sigma_{w}^{2}\mathbb{E}\phi(z^{\ell}(x)+z^{1}(x))\phi(z^{\ell}( x^{\prime})+z^{1}(x^{\prime})), \tag{9}\]

_where_

\[\begin{bmatrix}z^{1}(x)\\ z^{\ell}(x)\\ z^{1}(x^{\prime})\\ z^{\ell}(x^{\prime})\end{bmatrix}\sim\mathcal{N}\left(0,\left[\begin{array}{ c|c|c}\Sigma^{1}(x,x)&0&\Sigma^{1}(x,x^{\prime})&0\\ 0&\Sigma^{\ell}(x,x)&0&\Sigma^{\ell}(x,x^{\prime})\\ \hline\Sigma^{1}(x^{\prime},x)&0&\Sigma^{1}(x^{\prime},x^{\prime})&0\\ 0&\Sigma^{\ell}(x^{\prime},x)&0&\Sigma^{\ell}(x^{\prime},x^{\prime})\end{array} \right]\right) \tag{10}\]

Furthermore, we derive a compact form of the covariance function \(\Sigma^{L}\) in Corollary 4.2 by using the fact \(z^{1}\) and \(z^{\ell}\) are independent, which is proven in Appendix C.

**Corollary 4.2**.: _The covariance function \(\Sigma^{L}\) in Theorem 4.1 is rewritten as follows: \(\forall\ell\in[1,L-1]\)_

\[\Sigma^{1}(x,x^{\prime}) =\sigma_{u}^{2}\left\langle x,x^{\prime}\right\rangle/n_{in}, \tag{11}\] \[\Sigma^{\ell+1}(x,x^{\prime}) =\sigma_{w}^{2}\mathbb{E}\phi(u^{\ell}(x))\phi(u^{\ell}(x^{\prime })), \tag{12}\]

_where \((u^{\ell}(x),u^{\ell}(x^{\prime}))\) follows a centered bivariate Gaussian distribution with covariance_

\[\mathrm{Cov}(u^{\ell}(x),u^{\ell}(x^{\prime}))=\begin{cases}\Sigma^{1}(x,x^{ \prime}),&\ell=1\\ \Sigma^{\ell}(x,x^{\prime})+\Sigma^{1}(x,x^{\prime}),&\ell\in[2,L-1]\end{cases} \tag{13}\]

**Remark 4.2**.: _It is worth noting that the same Gaussian process or covariance function \(\Sigma^{L}\) is obtained regardless of whether the same weight matrix \(W\) is shared among layers. Additionally, there is no dependence across layers in the limit if different weight matrices are used. That is, if \(W^{\ell}\neq W^{k}\), then \(\mathrm{Cov}(z^{\ell}(x),z^{k}(x^{\prime}))=0\). These observation align with studies of recurrent neural networks [1, 39], where the same weight matrix \(W\) is applied in each layer._

### On the Strictly Positive Definiteness of \(\Sigma^{L}\)

To clarify the mathematical context, we provide a precise definition of the strict positive definiteness of a kernel function:

**Definition 4.2**.: _A kernel function \(k:X\times X\to\mathbb{R}\) is said to be strictly positive definite if, for any finite set of pairwise distinct points \(x_{1},x_{2},\ldots,x_{n}\in X\), the matrix \(K=[k(x_{i},x_{j})]_{i,j=1}^{n}\) is strictly positive definite. In other words, for any non-zero vector \(c\in\mathbb{R}^{n}\), we have \(c^{T}Kc>0\)._

Recent works [11, 38, 3, 3] have studied the convergence of (stochastic) gradient descent to global minima when training neural networks. It has been shown that the covariance function or NNGP kernel \(\Sigma^{L}\) being strictly positive definite guarantees convergence. In the case where the data set is supported on a sphere, we can establish the strict positive definiteness of \(\Sigma^{L}\) using Gaussian integration techniques and the existence of strictly positive definiteness of priors. The following theorem (Theorem 4.3) is proven in Appendix D.

**Theorem 4.3**.: _For a non-polynomial Lipschitz nonlinear \(\phi\), for any input dimension \(n_{0}\), the restriction of the limiting covariance function \(\Sigma^{L}\) to the unit sphere \(\mathbb{S}^{n_{0}-1}=\{x:\|x\|=1\}\), is strictly positive definite for \(2\leq L<\infty\)._

This theorem establishes that the limiting covariance function \(\Sigma^{L}\) of finite-depth neural network \(f_{\theta}^{L}\) is strictly positive definite when restricted to the unit sphere \(\mathbb{S}^{n_{0}-1}\), provided that a non-polynomial activation function is used.
### \(f_{\theta}\) as a Gaussian Process

In this subsection, we explore the convergence behavior of the infinite-depth neural network \(f_{\theta}\) to a Gaussian process as the width \(n\) tends to infinity. Since we have two limits involved, namely the depth and the width, it can be considered as a double sequence. Therefore, it is essential to review the definitions of convergence in double sequences.

**Definition 4.3**.: _Let \(\left\{a_{m,n}\right\}\) be a double sequence, then it has two types of **iterated limits**_

\[\lim_{m\to\infty}\lim_{n\to\infty}a_{m,n}=\lim_{m\to\infty}\left(\lim_{n\to \infty}a_{m,n}\right), \tag{14}\]

\[\lim_{n\to\infty}\lim_{m\to\infty}a_{m,n}=\lim_{n\to\infty}\left(\lim_{m\to \infty}a_{m,n}\right). \tag{15}\]

_The **double limit** of \(\left\{a_{m,n}\right\}\) is denoted by_

\[L:=\lim_{m,n\to\infty}a_{m,n}, \tag{16}\]

_which means that for all \(\varepsilon>0\), there exists \(N(\varepsilon)\in\mathbb{N}\) s.t. \(m,n\geq N(\epsilon)\) implies \(|a_{m,n}-L|\leq\epsilon\)._

In Subsection 4.1, we have previously shown that \(f_{\theta}^{L}\) converges to a centered Gaussian process with a covariance function \(\Sigma^{L}\), which is recursively defined. However, it is important to note that this convergence does not necessarily imply that the infinite-depth neural network \(f_{\theta}\) also converges to a Gaussian process, as the order in which the limits are taken can affect the result. Recent studies, such as [28, 20], have demonstrated that the convergence behavior of neural networks depends on the order in which the width and depth limits are taken. Specifically, when the width tends to infinity first, followed by the depth, a standard multi-layer perceptron (MLP) converges weakly to a Gaussian process. However, if the depth tends to infinity first, followed by the width, a heavy-tail distribution emerges. Additionally, when both the depth and width tend to infinity at a fixed ratio, a log-normal distribution is observed for Residual neural networks. Hence, the two limits are not necessarily equivalent unless they commute.

When studying the convergence behaviors of DEQs, it is more crucial to focus on the infinite-depth-then-width limit, as DEQs are defined as infinite-depth neural networks. Therefore, to establish the Gaussian process nature of DEQs, it is important to show that the infinite-depth-then-width limit is equal to the infinite-width-then-depth limit. Fortunately, we can demonstrate that these two limits commute and are equal to the double limit, as the convergence of the depth is much faster than the width, as proven in Appendix F.

**Lemma 4.1**.: _Choose \(\sigma_{w}>0\) small such that \(\gamma:=2\sqrt{2}\sigma_{w}<1\). Then for every \(x,x^{\prime}\in\mathbb{S}^{n_{in}-1}\), \(\lim_{\ell\to\infty}\lim_{n\to\infty}\frac{1}{n}\left\langle h^{\ell}(x),h^{ \ell}(x^{\prime})\right\rangle\) and \(\lim_{n\to\infty}\lim_{\ell\to\infty}\frac{1}{n}\left\langle h^{\ell}(x),h^{ \ell}(x^{\prime})\right\rangle\) exist and equal to \(\Sigma^{*}(x,x^{\prime})\) a.s., i.e.,_

\[\Sigma^{*}(x,x^{\prime}):=\lim_{\ell\to\infty}\lim_{n\to\infty}A_{n,\ell}= \lim_{n\to\infty}\lim_{\ell\to\infty}A_{n,\ell}=\lim_{\ell,n\to\infty}A_{n, \ell}, \tag{17}\]

_where \(A_{n,\ell}:=\frac{1}{n}\left\langle h^{\ell}(x),h^{\ell}(x^{\prime})\right\rangle\)._

Proof.: Proof is provided in Appendix F

Lemma 4.1 confirms the two iterated limits of the empirical covariance \(\frac{1}{n}\left\langle h_{n}^{\ell}(x),h_{n}^{\ell}(x^{\prime})\right\rangle\) of the pre-activation \(g_{k}^{\ell}\) exist and equal to the double limit \(\Sigma^{*}(x,x^{\prime})\) for any \(x,x^{\prime}\in\mathbb{S}^{n_{in}-1}\). Consequently, it establishes the commutation of the two limits, implying that the limit-depth-then-width and limit-width-then-depth have the same limit. Building upon this result, we can state Theorem 4.4, which asserts that as the width \(n\) of the infinite-depth neural network \(f_{\theta}\) tends to infinity, the output functions \(f_{\theta,k}\) converge to independent and identically distributed (i.i.d.) centered Gaussian processes with the covariance function \(\Sigma^{*}(x,x^{\prime})\). The detailed proofs can be found in Appendix G.

**Theorem 4.4**.: _Choose \(\sigma_{w}>0\) small such that \(\gamma:=2\sqrt{2}\sigma_{w}<1\). For infinite-depth neural network \(f_{\theta}\) defined on (1), as width \(n\to\infty\), the output functions \(f_{\theta,k}\) tend to i.i.d. centered Gaussian processes with covariance function \(\Sigma^{*}\) defined by_

\[\Sigma^{*}(x,x^{\prime})=\lim_{\ell\to\infty}\Sigma^{\ell}(x,x^{\prime}), \tag{18}\]

_where \(\Sigma^{\ell}\) are defined in Theorem 4.1._
### The Strict Positive Definiteness of \(\Sigma^{*}\)

We conclude this section by establishing the strictly positive definiteness of the limiting covariance function \(\Sigma^{*}\). Notably, the proof techniques used in Theorem 4.3 are not applicable here, as the strict positive definiteness of \(\Sigma^{L}\) may diminish as \(L\) approaches infinity.

Instead, we leverage the inherent properties of \(\Sigma^{*}\) itself and Hermitian expansion of the dual activation \(\hat{\phi}\) of \(\phi\)[10]. To explore the essential properties of \(\Sigma^{*}\), we perform a fine analysis on the pointwise convergence of the covariance function \(\Sigma^{L}\) for each pair of inputs \((x,x^{\prime})\).

**Lemma 4.2**.: _Choose \(\sigma_{w}>0\) small for which \(\beta:=\frac{\sigma_{w}^{2}}{2}\mathbb{E}|z|^{2}|z^{2}-1|<1\), where \(z\) is standard Gaussian random variable. Then for all \(x,x^{\prime}\in\mathbb{S}^{n_{in}-1}\), the function \(\Sigma^{\ell}\) satisfies_

1. \(\Sigma^{\ell}(x,x)=\Sigma^{\ell}(x^{\prime},x^{\prime})\)_,_
2. \(\Sigma^{\ell}(x,x)\leq(1+1/\beta)\Sigma^{2}(x,x)\)_._

_Consequently, \(\Sigma^{*}(x,x^{\prime})=\lim_{\ell\to\infty}\Sigma^{\ell}(x,x^{\prime})\) is well-defined and satisfies for all \(x,x^{\prime}\in\mathbb{S}^{n_{in}-1}\)_

\[0<\Sigma^{*}(x,x)=\Sigma^{*}(x^{\prime},x^{\prime})<\infty.\]

Lemma 4.2, proven in Appendix E, ensures the well-definedness of the limiting covariance function \(\Sigma^{*}\) for all \(x,x^{\prime}\in\mathbb{S}^{n_{in}-1}\) by choosing a small \(\sigma_{w}>0\). The lemma also guarantees that \(\Sigma^{*}(x,x)\) and \(\Sigma^{*}(x^{\prime},x^{\prime})\) are strictly positive, equal, and finite for all \(x,x^{\prime}\in\mathbb{S}^{n_{in}-1}\). These findings are crucial for demonstrating the strict positive definiteness of \(\Sigma^{*}\). Specifically, by leveraging these properties of \(\Sigma^{*}\), we can derive its Hermitian expansion of the limiting kernel \(\Sigma^{*}\).

By utilizing [23, Theorem 3], we establish in Theorem 4.5, as proven in Appendix H, that \(\Sigma^{*}\) is strictly positive definite if \(\phi\) is non-polynomial. It is important to note that our analysis can be extended to analyze the covariance or kernel functions induced by neural networks, particularly those that are defined as limits or induced by infinite-depth neural networks. This is because the analysis does not rely on the existence of the positive definiteness of priors. Instead, we examine the intrinsic properties of \(\Sigma^{*}\), which remain independent of the properties of the activation function \(\phi\).

**Theorem 4.5**.: _For a non-polynomial Lipschitz nonlinear \(\phi\), for any input dimension \(n_{0}\), the restriction of the limiting covariance \(\Sigma^{*}\) to the unit sphere \(\mathbb{S}^{n_{0}-1}=\{x:\|x\|=1\}\), is strictly positive definite._

## 5 Experimental Results

In this section, we present a series of numerical experiments to validate the theoretical results we have established. Our experiments aim to verify the well-posedness of the fixed point of the transition equation (2). We also investigate whether the DEQ behaves as a Gaussian process when the width is sufficiently large, as stated in our main result, Theorem 4.4. Additionally, we examine the strictly positive definiteness of the limiting covariance function \(\Sigma^{*}\), as established in Theorem 4.5, by computing the smallest eigenvalue of the associated covariance matrix \(K^{*}\). These experiments serve to empirically support our theoretical findings.

### Convergence to the fixed point

Proposition 3.1 guarantees the existence of a unique fixed point for the DEQ. To verify this, we conducted simulations using neural networks with the transition equation (2). We plotted the relative error between \(h^{\ell}\) and \(h^{\ell+1}\) in Figure 1. As shown in the figure, we observed that the iterations required for convergence to the fixed point were approximately 25 for various widths. This observation aligns with our theoretical findings in Lemma F.1, where the random initialization (5) scales the weight matrix \(W\) such that \(\|W\|_{\text{op}}=\mathcal{O}(\sigma_{w})\).

### The Gaussian behavior

Theorem 4.4 predicts that the outputs of DEQ tends to follow a Gaussian process as the width approaches infinity. To demonstrate this, we consider a specific DEQ with \(n_{in}=10\) and \(n_{out}=10\)
activated by tanh. We analyze the output distributions of 10,000 neural networks. An important implication of Theorem 4.4 is that the output forms an independent identical Gaussian distribution. To visualize this, we plot a pairplot in Figure 1 illustrating the randomly selected three outputs, confirming the validity of this implication.

Next, we generate histograms of the 10,000 neural networks to approximate the distribution of the first neuron in the output layer. In the third plot of Figure 1, we present the histogram for a width of 1000. Remarkably, the output distribution exhibits a strong adherence to the Gaussian model, as evidenced by a Kolmogorov-Smirnov (KS) statistic of 0.0056 and a corresponding p-value of 0.9136. Furthermore, in Figure 1(a) in the supplementary material, we provide histograms for widths of 10, 50, 100, 500, and 1000. As the width increases, the output distribution progressively converges towards a Gaussian distribution. This is evident from the decreasing KS statistics and the increasing p-values as the width extends from 10 to 1000.

Based on Theorem 4.4, the outputs of the neural network exhibit a behavior reminiscent of a joint Gaussian distribution for different inputs \(x\) and \(x^{\prime}\). To illustrate this, we plot the first output of the 10,000 neural networks for two distinct inputs as the first plot in Figure 2. Notably, the predicted limiting Gaussian level curves, derived from the limiting kernel function stated in Lemma 4.2, perfectly match the results of the simulations when the width is set to 1000.

### Convergence of the kernel

According to Theorem 4.4, the DEQs tends to a Gaussian process with a covariance function \(\Sigma^{*}=\lim_{\ell\to\infty}\bar{\Sigma}^{\ell}\). Given \(N\) distinct inputs \(\{x_{i}\}_{i=1}^{N}\), as stated in Theorem 4.1, the limiting covariance matrix \(K^{*}\) can be computed recursively, _i.e._, \(K^{\ell}_{ij}=\Sigma^{\ell}(x_{i},x_{j})\). By Lemma 4.1, each element \(K^{\ell}_{ij}\) can be approximated by \(\frac{1}{n}\langle h^{\ell}(x_{i}),h^{\ell}(x_{j})\rangle\). We conduct a series of numerical experiments to visually assess this convergence

First of all, we examine the convergence in width. We fix a large depth \(\ell\) and vary the widths by \(2^{2-13}\). We draw the errors between limiting covariance matrix \(K^{*}\) and finite-width empirical estimate \(K^{\ell}_{n}\) in the first two plots of Figure 3. The relative errors \(\|K^{\ell}_{n}-K^{*}\|_{F}/\|K^{*}\|_{F}\) consistently decreases as the growth of the width, and a convergence rate of order \(n^{-1}\) is observed.

Figure 1: Convergence to the fixed point (left); distribution of the first neuron of the output for \(10,000\) neural networks, KS statistics, p-value (middle); joint distributions for the first neuron for three outputs for \(10,000\) neuron networks, with orange curve denotes the Gaussian distribution (right)

Figure 2: Joint distributions for the first neuron for two different inputs over \(10,000\) neural networks (left); Covariance matrix obtained by a neural network (middle); Covariance matrix obtained by Gaussian process (right)
Next, we examine the convergence in depth by fixing a large width. The results are shown in the third and fourth plots of Figure 3. From these plots, we can observe that the error converges rapidly as the depth of the network increases, illustrating an exponential convergence rate.

### The positive definiteness of the kernel

Theorem 4.5 establishes that the NNGP kernel is strictly positive. As discussed earlier, the kernel matrix \(K^{L}\) can be computed recursively, as stated in Theorem 4.1 or Corollary 4.2. We refer to this computation as the _theoretical_ approach. Alternatively, it can be calculated as the covariance through simulation, which we denote as _simulation_ approach. We employ both methods to compute the smallest eigenvalue of the kernel matrix \(K^{L}\). The results are summarized in Figure 4. It is evident from the figure that the smallest eigenvalues increase with increasing depths and become stable once the kernel is well approximated. Furthermore, the smallest eigenvalue increases with higher values of \(\sigma_{w}\).

### Test Performance

To complement the theoretical analysis, we conducted numerical experiments demonstrating the NNGP correspondence for DEQs on real datasets with varying widths. A visual representation of these findings is available in Figure 4. Intriguingly, our observations consistently reveal that the NNGP continually outperforms trained finite-width DEQs. Moreover, a compelling trend emerges: as network width increases, the performance of DEQs converges more closely to NNGP performance. Notably, this phenomenon mirrors observations made in the context of standard feedforward neural networks [25; 28]. These experiments stand as practical evidence, effectively shedding light on the behavior of DEQs across different network sizes. The insights gleaned from these experiments have been thoughtfully integrated into our paper to enhance its comprehensiveness and practical relevance.

## 6 Conclusion and Future Work

This paper establishes that DEQs (Deep Equilibrium Models) can be characterized as Gaussian processes with a strict positive definite covariance function \(\Sigma^{*}\) in the limit of the width of the network approaching infinity. This finding contributes to the understanding of the convergence properties of infinite-depth neural networks, demonstrating that when and how the depth and width limits commute. An important direction for future research is to leverage the results presented in this paper to investigate the training and generalization performance of DEQs. While the results obtained in

Figure 4: From left to right: \(\lambda_{\min}(K^{\ell})\) across varying depths \(\ell\); \(\lambda_{\min}(K^{*})\) for different \(\sigma_{w}\) (blue curve: theory; orange curve: simulation); Test accuracy of the MNIST dataset using NNGP and DEQs with various widths; MSE of the MNIST dataset using NNGP and DEQs with various widths.

Figure 3: Covariance behaviors with varying width and depth
this paper hold for commonly used activation functions, it would be interesting to explore more complex transition functions in future work.

## 7 Acknowledgements

We would like to acknowledge the generous support of the National Science Foundation (NSF) under grant DMS-1812666 and III-2104797.
