Commutative Width and Depth Scaling in Deep Neural Networks
This paper is the second in the series Commutative Scaling of Width and Depth
(WD) about commutativity of infinite width and depth limits in deep neural
networks. Our aim is to understand the behaviour of neural functions (functions
that depend on a neural network model) as width and depth go to infinity (in
some sense), and eventually identify settings under which commutativity holds,
i.e. the neural function tends to the same limit no matter how width and depth
limits are taken. In this paper, we formally introduce and define the
commutativity framework, and discuss its implications on neural network design
and scaling. We study commutativity for the neural covariance kernel which
reflects how network layers separate data. Our findings extend previous results
established in [55] by showing that taking the width and depth to infinity in a
deep neural network with skip connections, when branches are suitably scaled to
avoid exploding behaviour, result in the same covariance structure no matter
how that limit is taken. This has a number of theoretical and practical
implications that we discuss in the paper. The proof techniques in this paper
are novel and rely on tools that are more accessible to readers who are not
familiar with stochastic calculus (used in the proofs of WD(I))).
Soufiane Hayou
# Commutative Width and Depth Scaling

###### Abstract

This paper is the second in the series _Commutative Scaling of Width and Depth_ (\(\overleftrightarrow{\mathbf{WD}}\)) about commutativity of infinite width and depth limits in deep neural networks. Our aim is to understand the behaviour of neural functions (functions that depend on a neural network model) as width and depth go to infinity (in some sense), and eventually identify settings under which commutativity holds, i.e. the neural function tends to the same limit no matter how width and depth limits are taken. In this paper, we formally introduce and define the commutativity framework, and discuss its implications on neural network design and scaling. We study commutativity for the neural covariance kernel which reflects how network layers separate data. Our findings extend previous results established in [55] by showing that taking the width and depth to infinity in a deep neural network with skip connections, when branches are suitably scaled to avoid exploding behaviour, result in the same covariance structure no matter how that limit is taken. This has a number of theoretical and practical implications that we discuss in the paper. The proof techniques in this paper are novel and rely on tools that are more accessible to readers who are not familiar with stochastic calculus (used in the proofs of \(\overleftrightarrow{\mathbf{WD}(I)}\))).

## 1 Introduction

The success of large language and vision models has recently amplified an existing trend of research on large size neural network. There are generally two ways to increase the size of a neural network model: increasing the width, for instance the number of neurons in hidden layers in a fully-connected network, the number of channels in a convolutional network, or the number of attention heads in a transformer architecture; and increasing the depth of the network, i.e. the number of layers. A suitable appraoch to understand the behaviour of large neural networks is by analyzing some pre-defined quantity as the width and/or depth tend to infinity. While the width limit by itself is now relatively well understood in different contexts [2, 6, 9, 15, 37], the depth limit and the interaction between the two have not been studied as much. In particular, given some pre-defined quantity of interest that depends on the network model, a basic question is: _do these two limits commute?_ (in the sense that the behaviour of the quantity of interest as width and depth go to infinity does not change depending on the order of which these limits are taken). One statistical quantity of interest is the _neural covariance_ kernel which reflects how layers in a neural network model separate input data. In this context, recent literature suggests that, at initialization,
in certain kinds of multi-layer perceptrons (MLPs) or residual neural networks (ResNets) with scaled main branch, the depth and width limits generally do _not_ commute [45, 57]; this would imply that in practice, such networks would behave quite differently depending on whether width is much larger than depth or the other way around. However, in the case of ResNets with suitably scaled residual blocks, recent work [55] showed that, to the contrary, at initialization, for a ResNet with blocks scaled the natural way so as to avoid blowing up the output, the width and depth limits _do commute_. An interesting practical implication of this result is that it justifies prior calculations that take the width limit first, then depth, to understand the behavior of deep residual networks, such as prior works in the signal propagation literature [6, 7, 33].

In this work, we introduce and formalize the framework of commutativity of the width and depth limits and generalize (and improve) existing results on the covariance from [55] for arbitrary sequences of scaling factors; these sequences are used to scale the residual blocks so as to avoid exploding behaviour as depth grows. Table 1 shows the difference between this work and the previous work in the \(\overleftarrow{\mathbf{WD}}\) series. We discuss the theoretical and practical implications of commutativity by addressing the natural question; _why should we care about commutativity at all?_ (see Section 3).

In addition to the significance of the results and the new framework, the mathematical novelty of this paper lies in the proof techniques: in contrast to [55] where the depth limit is taken first (fixing the width), followed by the width limit, we first take the width to infinity this time, which is a more conventional approach in the theory of signal propagation in deep networks. As such, the proof techniques in this paper can be seen as 'orthogonal' to the machinery developed in [55], and are more accessible to readers who are not familiar with stochastic calculus. Our results provide new insights into the behavior of deep neural networks with general depth scaling factors with implications on the design and analysis of these networks.

All the proofs are deferred to the appendix and referenced after each result. Empirical evaluations are provided to illustrate the theoretical results.

## 2 Related Work

The theoretical analysis of randomly initialized neural networks with an infinite number of parameters has yielded a stream of interesting results, both theoretical and practical. A majority of this research has concentrated on examining the scenario in which the width

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Paper & Block scaling & Neural Functions & Proof Techniques \\ \hline \(\overleftarrow{\mathbf{WD}}\)(I) ([55]) & \(1/\sqrt{depth}\) & Neural Covariance & Tools from \\ Neural Distribution & Stochastic Calculus \\ \hline \(\overleftarrow{\mathbf{WD}}\)(II) (This work) & General Block Scaling & Neural Covariance & Standard \\  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & Concentration results \\ \hline \end{tabular}
\end{table}
Table 1: Commutative Width and Depth Scaling Series. Block scaling refers to a scaling factor in front of the residual block. Neural functions are formally defined in Section 3.
of the network is taken to infinity while the depth is considered fixed. However, in recent years, there has been a growing interest in exploring the large depth limit of these networks. In this overview, we present a summary of existing results on this topic, though it is not exhaustive. A more comprehensive literature review is provided in Appendix A.

\(\bullet\)_Infinite-Width Limit:_ The study of the infinite-width limit of neural network architectures has been a topic of significant research interest, yielding various theoretical and algorithmic innovations. These include initialization methods, such as the Edge of Chaos [5, 6, 7, 15], and the selection of activation functions [15, 35, 50, 52], which have been shown to have practical benefits. In the context of Bayesian analysis, the infinite-width limit presents an intriguing framework for Bayesian deep learning, as it is characterized by a Gaussian process prior. Several studies (e.g. [2, 9, 10, 25, 29]) have investigated the weak limit of neural networks as the width goes to infinity, and have demonstrated that the network's output converges to a distribution modeled by a Gaussian process. Bayesian inference utilizing this "neural" Gaussian process has been explored in [9, 33]. 1

Footnote 1: It is worth mentioning that kernel methods such as NNGP and NTK significantly underperform properly tuned finite-width network trained using SGD, see [51].

\(\bullet\)_Infinite-Depth Limit:_ The infinite-depth limit of randomly initialized neural networks is a less explored topic compared to the infinite-width limit. Existing results can be categorized depending on how the two limits are taken. For instance, in the case of sequential limits, the width of the neural network is taken to infinity first, followed by the depth. This limit has been extensively utilized to explore various aspects of neural networks, such as examining the neural covariance, deriving the Edge of Chaos initialization scheme ([5, 6, 7, 15]), evaluating the impact of the activation function [15, 35, 52], studying the behavior of the Neural Tangent Kernel (NTK) [23, 28], and deriving the distribution of the limiting networks at initialization [33, 54]. Another interesting limit is the proportional limit where the ratio of depth to width is fixed, and both are jointly taken to infinity. In [34], the authors showed that for a particular type of residual neural networks (ResNets), the network output exhibits a (scaled) log-normal behavior in this limit, which differs from the sequential limit in which the width is first taken to infinity followed by depth, in which case the distribution of the network output is asymptotically normal ([6, 15]). Additionally, in [45], the authors examined the neural covariance of a multi-layer perceptron (MLP) in the joint limit and proved that it weakly converges to the solution of a Stochastic Differential Equation (SDE). Other works have investigated this limit and found similar results [14, 36, 38, 40, 57]. A third interesting approach is the general limit \(\min\{n,L\}\rightarrow\infty\), where width and depth can to infinity in any order. To the best of our knowledge, this limit was only studied in [55] (\(\overleftarrow{\mathbf{WD}}(\mathbf{I})\)) where convergence of the neural covariance in this limit was established for suitably scaled ResNet, implying that the infinite width and depth limits _commute_.

\(\bullet\)_Commutativity of the limits:_ given some neural function (a function that depends on the network parameters, to be defined later), we can think about whether taking the width and depth limits result in different behaviour depending on how this limit is taken. In this context, we distinguish between two notions of commutativity: _weak commutativity_ which implies that the sequential limits "width \(\rightarrow\infty\), then depth\(\rightarrow\infty\)" and "depth \(\rightarrow\infty\), then width\(\rightarrow\infty\)" yield the same limit, and _strong commutativity_ which implies that the limit "\(\min\{\text{width, depth}\}\rightarrow\infty\)" exists and is unique. Moreover, limits are always defined in some
sense (e.g. \(L_{2}\), weak limit etc.). In this context, strong commutativity was shown in [55] for neural distribution (distribution of a neuron in the network) with Wasserstein distance and for neural covariance kernel with \(L_{2}\) distance. In a recent work by [54], the authors showed weak commutativity of the neural distribution for _controlled_ ResNets, a form of ResNets with scaling factors given by the increments from some reference function.2 In this work, we will focus on the neural covariance instead of neural distribution and show strong commutativity under general depth scaling.

Footnote 2: In [54], while the results are with weak commutativity, the proofs can be in-principle extended to show strong commutativity for the weak convergence of the neural distribution.

## 3 Setup and Definitions: Commutativity and Neural Functions

When analyzing the asymptotic behavior of randomly initialized neural networks, various notions of probabilistic convergence can be employed, depending on the context. In this work, we particularly focus on strong convergence, defined to be the \(L_{2}\) convergence as described in the following definition.

**Definition 1** (Strong Convergence).: _Let \(d\geq 1\). We say that a sequence of \(\mathbb{R}^{d}\)-valued random variables \((X_{k})_{k\geq 1}\) converges in \(L_{2}\) (or strongly) to a continuous random variable \(Z\) if \(\lim_{k\to\infty}\|X_{k}-Z\|_{L_{2}}=0\), where the \(L_{2}\) is defined by \(\|X\|_{L_{2}}=\left(\mathbb{E}[\|X\|^{2}]\right)^{1/2}\)._

With this notion of strong convergence, we are now ready to introduce the commutativity framework for general neural network models.

Notation.Throughout the paper, the width and depth of a neural network model are denoted by \(n\) and \(L\), respectively, and the input dimension is denoted by \(d\). We write \([N]:=\{1,2,\ldots,N\}\) for any \(N\geq 1\).

Let us now consider a general neural network model of width \(n\geq 1\) and depth \(L\geq 1\), given by

\[\begin{cases}Y_{0}(a)=W_{in}a,\quad a\in\mathbb{R}^{d}\\ Y_{l}(a)=\mathcal{F}_{l}(W_{l},Y_{l-1}(a)),\;l\in[L],Y_{l}(a)\in\mathbb{R}^{n}, \end{cases} \tag{1}\]

where \(\mathcal{F}_{l}\) is a mapping that defines the nature of the \(l^{th}\) layer and \(W_{in}\in\mathbb{R}^{n\times d},W_{l}\in\mathbb{R}^{n\times n}\) are model weights. For the sake of simplification, we omit the dependence of \(Y_{l}\) on \(n\) and \(L\) in the notation. We refer to the vectors \(\{Y_{l},l=0,\ldots,L\}\) as _pre-activations_. Let \(\theta_{n,L}=(W_{in},W_{1},\ldots,W_{l})\) be the model weights and assume that \(\theta_{n,L}^{0}\sim\mu_{n,L}^{0}\), where \(\theta_{n,L}^{0}\) are the weights at initialization and \(\mu^{0}\) is a distribution that (naturally) depends on network width \(n\) and depth \(L\). Let us now define the notion of neural functions.

**Definition 2** (Neural Function).: _Given a general neural network model (Eq. (1)) of width \(n\) and depth \(L\), a set of network inputs \(\mathbf{a}=(a_{1},a_{2},\ldots,a_{k})\in(\mathbb{R}^{d})^{k}\), a neural function \(T\) is any function of the form \(T(n,L,\mathbf{a})=\mathcal{G}(\theta_{n,L}^{0},\mathbf{a})\), where \(\mathcal{G}\) is a general mapping with output in \(\mathbb{R}\).3_

Footnote 3: This definition of neural functions can be extended to general mappings \(\mathcal{G}\) with outputs in \(\mathbb{R}^{p}\) for some \(p\geq 1\). This is not required in this paper since we will be focusing on neural covariance kernel which has output in \(\mathbb{R}\).
Note that (almost) any quantity of interest in the training process of neural networks can be represented as a neural function. This remark was first observed in the series of Tensor Programs [37] where the result of any neural computation can be seen as a random quantity where the randomness is inherited from the initialization weights. The training dataset is considered deterministic in this case and consists of a sequence of inputs \((a_{1},a_{2},\ldots,a_{k})\). In this paper, we particularly think of neural functions as _proxy_ functions that track some behaviour of the network as we scale width and depth with the goal of providing insights on scaling strategies (see below for a specific choice of the neural function). With this definition of neural functions, we now formalize the notion of commutativity of the width and depth limits.

**Definition 3** (Commutativity).: _Given a neural function \(T\),4 we say that \(T\) satisfies universality for the width and depth limits if for any set of inputs \(\mathbf{a}=(a_{1},a_{2},\ldots,a_{k})\), \(T(n,L,\mathbf{a})\) converges in \(L_{2}\) in the limit \(\min\left\{n,L\right\}\to\infty\)._

Footnote 4: Note that by definition, a neural function is associated with a network model. When we consider a neural function \(T\), the underlying model is assumed to be fixed.

We can define a weak notion of commutativity where only sequential limits are considered, i.e. \(n\) or \(L\) limits are taken in a sequential order.

**Definition 4** (Weak Commutativity).: _Given a sequence of neural functions \(T=(T_{n,L})_{n,L\geq 1}\), we say that \(T_{n,L}\) satisfies commutativity for the width and depth limits if for any set of inputs \((a_{1},a_{2},\ldots,a_{k})\), both \(\lim\limits_{L\to\infty}\lim\limits_{n\to\infty}T_{n,L}(a_{1},a_{2},\ldots,a_{ k})\) and \(\lim\limits_{n\to\infty}\lim\limits_{L\to\infty}T_{n,L}(a_{1},a_{2},\ldots,a_{ k})\) exist in \(L_{2}\) and are equal._

Weak commutativity is trivially implied by commutativity. Intuitively, weak commutativity only deals with the 'extreme' scenarios \(L\gg n\gg 1\) and \(n\gg L\gg 1\) and does not consider the cases where for instance \(L\approx n\gg 1\).

Implications of Commutativity.Naturally, one might ask why we should care about commutativity at first. Commutativity of width and depth limits in neural networks holds significant importance for several compelling reasons:

1. _Unification of Width and Depth Scaling:_ when we aim to scale a neural network for improved performance, we often encounter scenarios where we must decide whether to increase the network's width or depth. Each of these choices generally lead to different design considerations, including variations in initialization schemes, activation functions, and learning rates. However, commutativity of the width and depth limits for some neural function \(T\) ensures that regardless of how we scale the network--whether by increasing width before depth, growing both width and depth proportionally, or taking width to infinity before depth--the resulting limiting behavior remains consistent. This means that once an effective scaling strategy is identified for a specific scenario with large width and depth, it remains a viable choice as long as both width and depth are large, simplifying the scaling process.
2. _Robust Scaling:_ as a result of commutativity, scaling the width and depth becomes robust to extreme changes in neural functions. This allows some flexibility in the
scaling procedure; in practice, one might want to increase width significantly while fixing depth, or the opposite, while preserving desirable properties captured by the neural function.
3. _Transfer of Insights:_ commutativity facilitates the transfer of insights from simplified theoretical settings to practical applications. When dealing with neural networks of large width and depth, it can be challenging to analyze their behavior directly. However, commutativity allows us to explore different limits, such as taking width to infinity first and then depth or vice versa, to gain a better understanding of the network's behaviour. Because the limit is universal (no matter how width and depth go to infinity), the insights we get in the simplified setting (e.g. sequential limit) transfers to all settings (e.g. when depth is of the same order as width).
4. _Commutativity is Feasible in Practice:_ we show that by introducing a simple scaling factor in front of the residual block in ResNets, commutativity holds for the neural covariance function at initialization (defined below). This neural function is used as a measure of how network layers separate input data, and led to many interesting practical methods (initialization schemes, neural network Gaussian process, choice of the activation function etc.) [6, 9, 15]. An in-depth discussion on this topic is provided below.

Neural Covariance.In this paper, we focus on neural functions given by the covariance/correlation functions _at initialization_. Given two inputs \(a,b\in\mathbb{R}^{d}\backslash\{0\}\),5 the neural covariance and correlation kernels at layer \(l\) are given by

Footnote 5: Here, we assume that the inputs are non-zero, other all the pre-activations \(Y_{l}\) are zero, and the correlation is undefined in this case. All the results in this paper are trivial if \(a=0\) or \(b=0\). We will therefore always assume that \(a,b\neq 0\).

\[\begin{cases}q_{l,n}(a,b)=\frac{\langle Y_{l}(a),Y_{l}(b)\rangle}{n}\\ c_{l,n}(a,b)=\frac{\langle Y_{l}(a),Y_{l}(b)\rangle}{\|Y_{l}(a)\|\|Y_{l}(b)\|}, \end{cases}\]

where the correlation is only defined when \(\|Y_{l}(a)\|,\|Y_{l}(b)\|\neq 0\).

Note that in general, if commutativity holds for the covariance kernel, then it holds for the neural correlation kernel, and vice-versa. This is true as long as pre-activations norms \(\|Y_{l}(a)\|\) are non-zero with high probability, which is generally satisfied, see Lemma 5 for a rigorous proof of this result. Hereafter, we will interchangeably discuss commutativity for neural covariance and correlation, while stating the theoretical results only for neural covariance. The results on the convergence of neural covariance are stated for two inputs \(a,b\), but they can be readily generalized to the case of multiple inputs \(a_{1},a_{2},\ldots,a_{k}\in\mathbb{R}^{d}\), where we can define the neural covariance matrix at layer \(l\) by

\[\mathbf{q}_{l,n}(a_{1},a_{2},\ldots,a_{k})=\begin{pmatrix}q_{l,n}(a_{1},a_{1}) &\ldots&q_{l,n}(a_{1},a_{k})\\ \vdots&\ddots&\vdots\\ q_{l,n}(a_{k},a_{1})&\ldots&q_{l,n}(a_{k},a_{k})\end{pmatrix}.\]
Why neural Covariance/Correlation?In the literature on signal propagation, there is a significant interest in understanding the covariance/correlation between the pre-activation vectors \(Y_{[tL]}(a)\) and \(Y_{[tL]}(b)\) for two different inputs \(a,b\in\mathbb{R}^{d}\). A natural question in this context is: _Why should we care about this covariance function?_

It is well-established that even with properly initialized multi-layer perceptrons (MLPs), the network outputs \(Y_{L}(a)\) and \(Y_{L}(b)\) become perfectly correlated (correlation=1) in the limit of "\(n\to\infty\), _then_\(L\to\infty\)" [5, 6, 15, 21]. This can lead to unstable behavior of the gradients and make the model untrainable as the depth increases and also results in the inputs being non-separable by the network6. To address this issue, several techniques involving targeted modifications of the activation function have been proposed [35, 52]. In the case of ResNets, the correlation still converges to 1, but at a polynomial rate [7]. A solution to this problem has been proposed by introducing well-chosen scaling factors in the residual branches, preventing the correlation kernel from converging to 1. This analysis was carried in the limit "\(n\to\infty\), then, \(L\to\infty\)" in [33], and recently extended in [55] to the case where "\(\min(n,L)\to\infty\)", showing that commutativity holds in this case. Some of these works have provided empirical evidence showing an association between favorable characteristics of the neural covariance/correlation and good trainability properties of deep networks.7

Footnote 6: To see this, assume that the inputs are normalized. In this case, the correlation between the pre-activations of the last layer for two different inputs converges to 1. This implies that as the depth grows, the network output becomes similar for all inputs, and the network no longer separates the data. This is problematic for the first step of gradient descent as it implies that the information from the data is (almost) unused in the first gradient update.

Footnote 7: By favorable characteristics of the neural covariance, we refer for instance to non-degeneracy as \(L\to\infty\) as reported in [33].

## 4 Overview of Existing Results

In this section, we present corollaries of existing results showing different scenarios where commutativity is satisfied or not for the neural covariance. The aim of this section is show that commutativity depends on the neural architecture.

### Non-Commutativity in MLPs

Let \(d,n,L\geq 1\), and consider a simple MLP architecture given by the following:

\[\begin{cases}Y_{0}(a)=W_{in}a,\quad a\in\mathbb{R}^{d}\\ Y_{l}(a)=W_{l}\phi(Y_{l-1}(a)),\;l\in[L],\end{cases} \tag{2}\]

where \(\phi:\mathbb{R}\to\mathbb{R}\) is the ReLU activation function, \(W_{in}\in\mathbb{R}^{n\times d}\), and \(W_{l}\in\mathbb{R}^{n\times n}\) is the weight matrix in the \(l^{th}\) layer. We assume that the weights are randomly initialized with _iid_ Gaussian variables \(W_{l}^{ij}\sim\mathcal{N}(0,\frac{2}{n})\),8\(W_{in}^{ij}\sim\mathcal{N}(0,\frac{1}{d})\). While the activation function is only defined for real numbers (1-dimensional), we abuse the notation and write \(\phi(z)=(\phi(z^{1}),\ldots,\phi(z^{k}))\) for any \(k\)-dimensional vector \(z=(z^{1},\ldots,z^{k})\in\mathbb{R}^{k}\) for any \(k\geq 1\). We refer to the vectors \(\{\phi(Y_{l}),l=0,\ldots,L\}\) as _post-activations_.
In the case of the joint limit \(n,L\to\infty\) with \(n/L\) fixed, it has been shown that the covariance/correlation between \(Y_{[tL]}(a)\) and \(Y_{[tL]}(b)\) becomes similar to that of a Markov chain that incorporates random terms. However, the correlation still converges to 1 in this limit.

**Proposition 1** (Correlation, [15, 45]).: _Consider the MLP architecture given by Eq. (2) and let \(a,b\in\mathbb{R}^{d}\) such that \(a,b\neq 0\). Then, in the limit "\(\eta\to\infty\), then \(L\to\infty\)" or the the joint limit "\(n,L\to\infty\), \(L/n\) fixed", the correlation \(\frac{\langle Y_{L}(a),Y_{L}(b)\rangle}{\|Y_{L}(a)\|\|Y_{L}(b)\|}\) converges9 weakly to 1._

Footnote 9: Note that weak convergence to a constant implies also convergence in probability.

The convergence of the correlation to 1 in the infinite depth limit of a neural network poses a significant issue, as it indicates that the network loses all of the covariance structure from the inputs as the depth increases. This results in degenerate gradients (see e.g. [6]), rendering the network untrainable. To address this problem in MLPs, various studies have proposed the use of depth-dependent shaped ReLU activations, which prevent the correlation from converging to 1 and exhibit stochastic differential equation (SDE) behavior. As a result, the correlation of the last layer does not converge to a deterministic value in this case.

**Proposition 2** (Correlation SDE, Corollary of Thm 3.2 in [45]).: _Consider the MLP architecture given by Eq. (2) with the following activation function \(\phi_{L}(z)=z+\frac{1}{\sqrt{L}}\phi(z)\) (a modified ReLU). Let \(a,b\in\mathbb{R}^{d}\) such that \(a,b\neq 0\). Then, in the joint limit "\(n,L\to\infty\), \(L/n\) fixed", the correlation \(\frac{\langle Y_{L}(a),Y_{L}(b)\rangle}{\|Y_{L}(a)\|\|Y_{L}(b)\|}\) converges weakly to a nondeterministic random variable.10_

Footnote 10: In [45], the authors show that the correlation of \(\frac{(\phi_{L}(Y_{L}(a)),\phi_{L}(Y_{L}(b)))}{\sqrt{\|\phi_{L}(Y_{L}(a))\|} \sqrt{\|\phi_{L}(Y_{L}(b))\|}}\) converges to a random variable in the joint limit. Since \(\phi_{L}\) converges to the identity function in this limit, simple calculations show that the correlation between the pre-activations \(\frac{\langle Y_{L}(a),Y_{L}(b)\rangle}{\|Y_{L}(a)\|Y_{L}(b)\|}\) is also random in this limit.

The joint limit, therefore, yields non-deterministic behaviour of the covariance structure. It is easy to check that even with shaped ReLU as in Proposition 2, taking the width to infinity first, then depth, the result is a deterministic covariance structure. The main takeaway from this section is the following:

**Corollary 1**.: _With MLPs (Eq. (2)), the width and depth limits do not commute for the neural covariance/correlation._

### Commutativity with Scaled Residual Networks

Using the same notation as in the MLP case, consider the following ResNet architecture of width \(n\) and depth \(L\)

\[\begin{split} Y_{0}(a)&=W_{in}a,\quad a\in\mathbb{ R}^{d}\\ Y_{l}(a)&=Y_{l-1}(a)+\frac{1}{\sqrt{L}}W_{l}\phi(Y_{ l-1}(a)),\,l\in[1:L],\end{split} \tag{3}\]

where \(\phi:\mathbb{R}\to\mathbb{R}\) is the ReLU activation function. Assume that the weights are randomly initialized with _iid_ Gaussian variables \(W_{l}^{ij}\sim\mathcal{N}(0,\frac{1}{n})\), \(W_{in}^{ij}\sim\mathcal{N}(0,\frac{1}{d})\). If we consider the
set of scaling factors of the form \(L^{-\gamma}\) for \(\gamma>0\), then the choice of \(\gamma=1/2\) is the smallest value of \(\gamma\) such that the network output do not explode in the infinite-depth limit (see Lemma 1). Therefore, in some sense, this scaling is the 'optimal' amongst uniform scalings (meaning all residual branches are scaled with the same factor) for two reasons: it stabilizes the network as depth increases, and it does not result in trivial behaviour (see discussion after Proposition 3).

With the ResNet architecture Eq. (3), we have the following result for the covariance kernel, which establishes commutativity in this case.

**Proposition 3** (Thm 2 in [55]).: _Let \(a,b\in\mathbb{R}^{d}\) such that \(a,b\neq 0\) and \(a\neq b\). Then, we have the following_

\[\sup_{t\in[0,1]}\left\|q_{[tL],n}(a,b)-q_{t}(a,b)\right\|_{L_{2}}\leq C\left( \frac{1}{\sqrt{n}}+\frac{1}{\sqrt{L}}\right)\]

_where \(C\) is a constant that depends only on \(\|a\|\), \(\|b\|\), and \(d\), and \(q_{t}(a,b)\) is the solution of the following differential flow_

\[\begin{cases}\frac{dq_{t}(a,b)}{dt}&=\frac{1}{2}\frac{f(c_{t}(a,b))}{c_{t}(a, b)}q_{t}(a,b),\\ c_{t}(a,b)&=\frac{q_{t}(a,b)}{\sqrt{q_{t}(a,a)}\sqrt{q_{t}(b,b)}},\\ q_{0}(a,b)&=\frac{\langle a,b\rangle}{d},\end{cases} \tag{4}\]

_where the function \(f:[-1,1]\to[-1,1]\) is given by_

\[f(z)=\frac{1}{\pi}(z\arcsin(z)+\sqrt{1-z^{2}})+\frac{1}{2}z.\]

This result suggests that commutativity for the neural covariance depends on the architecture, and holds in this particular case. More importantly, with this residual architecture, taking the width and depth limits to infinity yield a non-trivial limit of the neural covariance given by the function \(q_{t}\). In [33], it was shown that \(q_{t}\) is a universal kernel, meaning that, it is not only non-trivial, but one can approximate any sufficiently smooth function on some compact set with features from the kernel \(q_{t}\). This has a number of implications, especially in the context of neural network Gaussian processes. We invite the reader to check [33] for a more in-depth discussion. Another recent result showed that trivial behaviour can be avoided by scaling the main branch of the ResNet. The neural covariance converges weakly to a random variable in the proportional limit, which implies that such scaling breaks commutativity.

**Proposition 4** (Corrollary of Thm in [57]).: _Conider a ResNet where the hidden layers are of the form \(Y_{l}(a)=\beta Y_{l-1}(a)+\sqrt{1-\beta^{2}}W_{l}\phi_{L}(Y_{l-1}(a))\), where \(\beta\in(0,1)\) is a constant, and \(\phi_{L}\) is the shaped ReLU (defined in Proposition 2). Then, the width and depth limits for the covariance kernel do not commute in this case._

Scaling the main branch of the residual network results in a similar behaviour to the MLP case. Intuitively, with the factor \(\beta\), the direct contribution of any layer to the main branch decreases exponentially with depth, hence simulating the'multiplicative' nature of MLPs. Note that the use of shaped ReLU is essential with this scaling in order to avoid
degeneracy problems; with ReLU, the correlation converges to 1 in the proportional limit. In the same paper, the authors show a similar result for Transformers which is a more modern residual architecture.

With the background information provided above, we are now able to present our findings. In the next section, we demonstrates commutativity of the width and depth limits for a general class of ResNet architectures, extending the results of [55].

## 5 Main Results: Commutativity under General Scaling

In this section, we present our main results regarding commutativity of the width and depth limits under general scaling rules. All the proofs are deferred to the Appendix. We first define the _sequence of scaling factors_, a notion that will be frequently used in the paper.

**Definition 5** (Sequence of Scaling Factors).: _A sequence of scaling factors is an infinite triangular array of non-negative real numbers. It has the form \(\alpha=(\alpha_{l,L})_{l\in\{1,\dots,L\},L\geq 1}\)._

Visually, one can think of \(\alpha\) as an infinite object of the form

\[\alpha=\begin{cases}\alpha_{1,1}&\\ \alpha_{1,2}&\alpha_{2,2}\\ \vdots&\vdots\ddots\\ \alpha_{1,L}&\dots&\alpha_{L,L}\\ \vdots&\dots&\dots\end{cases}\]

The use of such notation will come handy when we scale up the depth of a neural network. Such sequences will be used to define a scaling strategy as network depth grows.

Setup.Recall the previously introduced notation, the width and depth of the network are denoted by \(n\) and \(L\), respectively, and the input dimension is denoted by \(d\). Let \(n,L,d\geq 1\), and consider the following neural network model with skip connections

\[\begin{cases}Y_{0}(a)=W_{in}a,\quad a\in\mathbb{R}^{d},\\ Y_{l}(a)=Y_{l-1}(a)+\alpha_{l,L}W_{l}\,\phi(Y_{l-1}(a)),\quad l\in[L],\end{cases} \tag{5}\]

where \(\phi\) is the ReLU activation function,11\(W_{in}\in\mathbb{R}^{n\times d}\) is the input layer weight matrix, and \(W_{l}\in\mathbb{R}^{n\times n}\) is the weight matrix in the \(l^{th}\) layer. We assume that the weights are randomly initialized as \(W_{d}^{ij}\sim\mathcal{N}(0,1/d)\), and \(W_{l}^{ij}\sim\mathcal{N}(0,1/n)\) for \(l\in[L]\), \(a\neq 0\) is an arbitrary input in \(\mathbb{R}^{d}\), \(\alpha=(\alpha_{l,L})_{L\geq 1,l\in[L]}\) is a sequence of scaling factors. For the sake of simplification, we only consider networks with no bias, and we omit the dependence of \(Y_{l}\) on \(n\) and \(L\) in the notation. For a vector \(Z\in\mathbb{R}^{k}\), we write \(Z=(Z^{1},Z^{2},\dots,Z^{k})\in\mathbb{R}^{k}\) to denote its entries. Hereafter, we consider two inputs \(a,b\in\mathbb{R}^{d}\) satisfying \(a,b\neq 0\) and \(\langle a,b\rangle\neq 0\).12
As depth increases, the pre-activations might grow arbitrarily large, depending on the choice of the sequence \(\alpha\). The next result fully characterizes sequences that guarantee stability in terms of the \(L_{2}\) norm.

**Lemma 1**.: _For all \(L\geq 1,l\in[L],i\in[n]\)_

\[\mathbb{E}\left[Y_{l}^{i}(a)^{2}\right]=\frac{\|a\|^{2}}{d}\prod_{k=1}^{l} \left(1+\frac{\alpha_{k,L}^{2}}{2}\right).\]

_As a result, \(\sup_{l\in[L],L\geq 1,i\in[n]}\mathbb{E}\left[Y_{l}^{i}(a)^{2}\right]\) is bounded iff \(\sup_{L\geq 1}\sum_{l=1}^{L}\alpha_{l,L}^{2}<\infty\). 13_

Footnote 13: This stability condition was introduced in [33]. It is worth mentioning that in [54], in the context of “controlled” ResNets, the authors use an \(L_{2}\) condition on the control process to show weak commutativity for the output distribution where convergence is considered in the weak sense. This condition is similar in flavour to our stability condition.

Proof.: Simple calculations yield

\[\mathbb{E}\left[Y_{l}^{i}(a)^{2}\right]=\mathbb{E}\left[Y_{l-1}^{i}(a)^{2} \right]+\alpha_{l,L}^{2}\mathbb{E}\left[\phi(Y_{l}^{i}(a))^{2}\right].\]

To conclude, it suffices to see that \(Y_{l}^{i}(a)^{2}\) is a symmetric random variable, and therefore \(\mathbb{E}\left[\phi(Y_{l}^{i}(a))^{2}\right]=\frac{1}{2}\mathbb{E}\left[Y_{l }^{i}(a)^{2}\right]\). 

The result of Lemma 1 is independent from the width \(n\). Hence, a necessary and sufficient condition so that the pre-activations do not blow up with depth (in \(L_{2}\) norm), for any width \(n\), is to have \(\sup_{L\geq 1}\sum_{l=1}^{L}\alpha_{l,L}^{2}<\infty.\) We say that such sequences of scaling factors are stable.

**Definition 6** (Stable Sequence of Scaling Factors).: _Let \(\alpha\) be a sequence of scaling factors. We say that \(\alpha\) is stable if it satisfies \(\sup_{L\geq 1}\sum_{l=1}^{L}\alpha_{l,L}^{2}<\infty\). We denote the space of stable sequences of scaling factors by \(\mathcal{S}\). For \(\alpha\in\mathcal{S}\), we define the \(\mathcal{S}\)-norm of \(\alpha\) by \(\|\alpha\|_{S}=\sqrt{\sup_{L\geq 1}\sum_{l=1}^{L}\alpha_{l,L}^{2}}\).14_

Footnote 14: If we allow negative values for \(\alpha_{l,L}\), then we can show that the space \(\mathcal{S}\), endowed with the inner product \(\langle\alpha,\beta\rangle_{S}=\sup_{L\geq 1}\sum_{l=1}^{L}\alpha_{l,L}\beta_{ l,L}\), is a complete space (Banach space). We omit these technicalities in this paper.

Stable Sequences of Scaling Factors have first appeared in [33]. In that work, the sequential limit 'infinite-width, then infinite-depth' was considered, and such sequences were proven to stabilize the gradients as well, and yield other favorable network properties regarding the neural covariance kernel and the neural tangent kernel.

In the next two (sub)sections, we show that unlike in MLPs or residual networks with scaled main branch where the neural covariance/correlation exhibits different limiting behaviors depending on how the width and depth limits are taken, under general conditions on the sequence \(\alpha\), for the ResNet architecture given by Eq. (5), the neural covariance converges strongly to a deterministic kernel, which depends on the choice of the sequence \(\alpha\), in the limit \(\min(n,L)\rightarrow\infty\) regardless of the relative rate at which \(n\) and \(L\) tend to infinity. We show different examples and recover and strengthen previous results as special cases.
### Sequence of Scaling Factors as Convergent Series

In this section, we consider sequences \(\alpha\) that "converge" to a series in a specific way. We show that in this case, the neural covariance kernel converges to the same limiting kernel with a specific convergence rate in the limit \(\min(n,L)\to\infty\), hence inducing commutativity.

**Theorem 1** (Commutativity with Quasi-Convergent Series).: _Let \(\alpha\in\mathcal{S}\). Assume that there exists a sequence \(\zeta=(\zeta_{i})_{i\geq 1}\in\ell_{2}(\mathbb{N})\) such that \(\sum_{l=1}^{L}|\alpha_{l,L}^{2}-\zeta_{l}^{2}|\to 0\) as \(L\to\infty\). Then, we have that for all \(t\in[0,1]\)_

\[\sup_{t\in(0,1]}\|q_{\lfloor tL\rfloor,n}(a,b)-q_{\infty}^{\zeta}(a,b)\|_{L_{2 }}\leq C\left(n^{-1/2}+\sum_{l=1}^{L}|\alpha_{l,L}^{2}-\zeta_{l}^{2}|+\sum_{l \geq L}\zeta_{l}^{2}\right),\]

_where \(C\) is a constant that depends only on \(\|a\|,\|b\|,d,\|\zeta\|_{S}\), and \(q_{\infty}^{\zeta}(a,b)=\lim_{L\to\infty}q_{L}^{\zeta}(a,b)\) and \(q_{L}^{\zeta}\) is given by the recursive formula_

\[\begin{cases}q_{L}^{\zeta}(a,b)=q_{L-1}^{\zeta}(a,b)+\frac{1}{2}\zeta_{L}^{2 }\frac{f(c_{L-1}(a,b))}{c_{L-1}(a,b)}q_{L-1}(a,b),\quad L\geq 1\\ c_{L}(a,b)=\frac{q_{L}(a,b)}{\sqrt{q_{L}(a,a)q_{L}(b,b)}},\\ q_{0}^{\zeta}(a,b)=\frac{\langle a,b\rangle}{d},\end{cases}\]

_where \(f:[-1,1]\to[-1,1]\) is given by_

\[f(z)=\frac{1}{\pi}(z\arcsin z+\sqrt{1-z^{2}})+\frac{1}{2}z.\]

Theorem 1 shows that the neural covariance kernel converges to the same limiting kernel no matter how the width and depth limits are taken. In the proof, provided in Appendix C, we first show the existence of the limit of \(q_{L}\), then proceed to bound the difference with the neural covariance kernel. The convergence rate depends on he properties of the series \(\zeta\) that approximates \(\alpha\) as depth grows. Notice that the limiting kernel \(q_{\infty}^{\zeta}\) does not depend on \(t\in(0,1]\). This is because the entries of \(\zeta\) do not depend on depth \(L\).

_Examples._ The conditions of Theorem 1 are satisfied by many sequences \(\alpha\). Examples include:

* "Decreasing" scaling: assume that \(\alpha_{l,L}=\zeta_{l}\) for all \(L\geq 1,l\in[L]\), where \(\zeta\in\ell_{2}(\mathbb{N})\). We call this scaling decreasing because \(\lim_{l\to\infty}\zeta_{l}=0\). This choice of scaling factors trivially satisfies the conditions of Theorem 1 and the convergence rate is given by \(\mathcal{O}(n^{-1}+\sum_{l\geq L}\zeta_{l}^{2})\). An examples of such scaling was studied in [33] and empirical results (performance of trained networks) were reported with \(\zeta=\left((l\log(l+1)^{2})^{-1/2}\right)_{l\geq 1}\).
* "Aggressive" Uniform scaling: assume that \(\alpha_{l,L}=L^{-\gamma}\) for some constant \(\gamma>1/2\). This scaling is called uniform because all the residual branches have the same scaling factor. This sequence of scaling factors satisfies the conditions of Theorem 1 with \(\zeta=0_{\ell_{2}(\mathbb{N})}\). The convergence rate is given by \(\mathcal{O}(n^{-1}+L^{-(2\gamma-1)})\), and the limiting kernel is trivial and given by \(q_{\infty}^{\zeta}=q_{0}^{\zeta}\), hence the nomenclature 'aggressive' since this scaling removes all contributions of the hidden layers in the limiting kernel. Note that this case covers the Neural ODE limit with scaling factors \(\alpha_{l,L}=L^{-1}\). In the next section, we will see that another kind of uniform scaling(non-aggressive) that yield non-trivial limits.
### Normalized Sequences of Scaling Factors

In this section, we discuss another type of sequences of scaling factors. We know from [55] that with \(\alpha_{l,L}=L^{-1/2}\), the limiting kernel is given by the solution of an ODE. In this section, we generalize this result by considering all sequences \(\alpha\) that satisfy the condition \(\sum_{l=1}^{L}\alpha_{l,L}^{2}=1\) for all \(L\geq 1\). Let us first give a formal definition of such sequences.

**Definition 7** (Normalized Sequence of Scaling Factors).: _Let \(\alpha\) be a sequence of scaling factors. We say that \(\alpha\) is normalized if it satisfies \(\sum_{l=1}^{L}\alpha_{l,L}^{2}=1\) for all \(L\geq 1\). The space of normalized sequences of scaling factors is denoted by \(\mathcal{S}_{1}\).15_

Footnote 15: Note that the constant \(1\) in this definition can be replaced by any constant \(M>0\) and all the subsequent results remain valid.

It is trivial that \(\mathcal{S}_{1}\subset\mathcal{S}\), and for all \(\alpha\in\mathcal{S}_{1},\|\alpha\|_{\mathcal{S}}=1\) (hence the subscript in \(\mathcal{S}_{1}\)). The next result establishes commutativity of the infinite width and depth limit for normalized sequences.

**Theorem 2** (Commutativity with Normalized Sequences).: _Consider a sequence of scaling factors \(\alpha\in\mathcal{S}_{1}\). Let \(h_{L}=\max_{1\leq l\leq L}\alpha_{l,L}^{2}\) and assume that \(Lh_{L}^{2}=o_{L}(1).\) Then, we have_

\[\sup_{t\in(0,1]}\|q_{\lfloor tL\rfloor,n}(a,b)-q_{t_{L}}(a,b)\|_{L_{2}}\leq C \left(n^{-1/2}+h_{L}+Lh_{L}^{2}\right),\]

_where \(C\) depends only on \(\|a\|,\|b\|,\) and \(d\), and \(t_{L}\) is given by \(t_{L}=\sum_{k=1}^{\lfloor tL\rfloor}\alpha_{k,L}^{2}\), and \(q_{t}\) is given by the solution of the following differential flow_

\[\begin{cases}\frac{dq_{t}(a,b)}{dt}&=\frac{1}{2}\frac{f(c_{t}(a,b))}{c_{t}(a,b )}q_{t}(a,b),\\ c_{t}(a,b)&=\frac{q_{t}(a,b)}{\sqrt{q_{t}(a,a)}\sqrt{q_{t}(b,b)}},\\ q_{0}(a,b)&=\frac{(a,b)}{d},\end{cases} \tag{6}\]

_where the function \(f:[-1,1]\to[-1,1]\) is given by_

\[f(z)=\frac{1}{\pi}(z\arcsin(z)+\sqrt{1-z^{2}})+\frac{1}{2}z.\]

_Moreover, assume that there exists a function \(\lambda:[0,1]\to[0,1]\) such that the sequence \(\alpha\) satisfies \(\sup_{t\in[0,1]}\left|\sum_{k=1}^{\lfloor tL\rfloor}\alpha_{k,L}^{2}-\lambda (t)\right|\leq r_{L}\) and \(\lim_{L\to\infty}r_{L}=0\). Then, we have_

\[\sup_{t\in(0,1]}\|q_{\lfloor tL\rfloor,n}(a,b)-q_{\lambda(t)}(a,b)\|_{L_{2}} \leq C^{\prime}\left(n^{-1/2}+h_{L}+Lh_{L}^{2}+r_{L}\right),\]

_where \(C^{\prime}\) depends only on \(\|a\|,\|b\|,d\)._

The result of Theorem 2 generalizes previous results from [55] to arbitrary normalized sequences. Using this theorem, we recover those results by choosing \(\alpha_{l,L}=L^{-1/2}\) and verifying the conditions in the theorem. In particular, with the new proof techniques developed in this paper, we obtain a stronger convergence rate for depth.
**Corollary 2** (Normalized Uniform Scaling).: _Assume that \(\alpha_{l,L}=L^{-1/2}\) for all \(L\geq 1\) and \(l\in[L]\). Then, the results of Theorem 2 are satisfied with \(\lambda(t)=t\), \(r_{L}=L^{-1}\), and \(h_{L}=L^{-1}\). As a result, we have that_

\[\sup_{t\in(0,1]}\|q_{\lfloor tL\rfloor,n}(a,b)-q_{t}(a,b)\|_{L_{2}}\leq C\left( n^{-1/2}+L^{-1}\right),\]

_where \(C\) depends only on \(\|a\|,\|b\|,d\), and \(q_{t}\) is defined in Theorem 2._

Proof.: With \(\alpha_{l,L}=L^{-1/2}\), we trivially have \(h_{L}=L^{-1}\) and \(Lh_{L}^{2}=L^{-1}\). Moreover, given \(t\in(0,1]\) we have that \(\sum_{k=1}^{\lfloor tL\rfloor}\alpha_{l,L}^{2}=\frac{\lfloor tL\rfloor}{L}\), and therefore \(\left|\sum_{k=1}^{\lfloor tL\rfloor}\alpha_{l,L}^{2}-t\right|\leq L^{-1}\). 

Better depth rate.In this paper, we obtain a depth rate of order \(L^{-1}\) in contrast to the \(L^{-1/2}\) convergence rate reported in [55]. The reason lies in the differences of the proof techniques used to derive the results. The proof techniques in both results are essentially 'orthogonal' in the following sense: in [55], the proofs rely on taking the depth to infinity first, while controlling the effect of width at the same time. With this approach, the best depth rate one can obtain is \(L^{-1/2}\) which is induced by the Euler disctretization error (note that with \(\alpha_{l,L}=L^{-1/2}\), the ResNet behaves as the solution of a Stochastic Differential Equation (SDE) in the infinite depth limit when the width is fixed). However, in this work, we first take the width to infinity while controlling the depth. By doing this, all the randomness in the covariance is removed as \(n\to\infty\), regardless of the depth \(L\). As a result, by taking depth to infinity, we deal with deterministic dynamical systems instead of stochastic ones (the SDE case), in which case the Euler disctretization error is of order \(L^{-1}\). We refer the reader to Section 7 for more details about the proof techniques.

_Remark_.: The normalized uniform scaling is optimal in terms of the depth-related error in Theorem 2. More precisely, the depth-related error is given by the term \(\mathcal{R}_{L}(\alpha)=h_{L}+Lh_{L}^{2}+r_{L}\) up to constant \(C\). Therefore, a natural question one might ask is: what properties should the sequence of scaling factors satisfy in order to minimize this error? Given a fixed depth \(L\), this problem can be formulated as a constrained minimization problem

\[\min_{\alpha\in\mathcal{S}_{1}}\mathcal{R}_{L}(\alpha)=h_{L}+Lh_{L}^{2}+r_{L}, \tag{7}\]

where the constraint is given by the fact that \(\alpha\in\mathcal{S}_{1}\).

**Lemma 2**.: _The normalized uniform scaling given by \(\alpha_{l,L}=L^{-1/2}\) is a solution to problem (7)._

To explain the intuition behind the result of Lemma 2, we first need to understand what each term in \(\mathcal{R}_{L}\) represents. The first term \(h_{L}\) is well-known in numerical methods and represents the Euler discretization (global) error. The second term \(Lh_{L}^{2}\) is a bound on the error between the Euler scheme of the ODE satisfied by \(q_{t}\) and the actual neural covariance kernel from the finite depth network. The last term \(r_{L}\) is induced by the behaviour of the scaling sequence as \(L\) grows. If we consider just the sum of the first two terms, uniform scaling balances the two terms which should intuitively minimize that sum. It also happens that for this choice of scaling \(r_{L}\) is of the same order as \(h_{L}+Lh_{L}^{2}\).
## 6 Experiments and Practical Implications

In this section, we validate our theoretical results with simulations on large width and depth residual neural networks of the form Eq. (5) with different choices of the sequence \(\alpha\).

### Convergence of the neural covariance

Theorem 2 and Theorem 1 predict that the covariance \(q_{l,n}(a,b)\) for two inputs \(a,b\) converges in \(L_{2}\) norm in the limit \(\min(n,L)\to\infty\).

Uniform scaling \(\alpha_{l,L}=L^{-1/2}\).In Fig. 1, we compare the empirical covariance \(q_{l,n}\) with the theoretical prediction \(q_{t}\) from Theorem 2 for \(n\in\{2^{3},2^{8},2^{14}\}\) and \(L\in\{2^{1},2^{3},2^{8}\}\). We chose maximum depth to be much smaller than maximum width to take into account the difference in the width and depth convergence rates: \(n^{-1/2}\) versus \(L^{-1}\) in this case.

The empirical \(L_{2}\) error between \(q_{L,n}\) and \(q_{1}\) (from Theorem 2) is also reported. As the width increases, we observe an excellent match with the theory. The role of the depth is less

Figure 1: The blue curve represents the average covariance \(q_{l,n}(a,b)\) for ResNet Eq. (5) with \(n\in\{2^{3},2^{8},2^{14}\}\), \(L\in\{2^{1},2^{3},2^{8}\}\), \(d=30\), and \(a\) and \(b\) are sampled randomly from \(\mathcal{N}(0,I_{d})\) and normalized to have \(\|a\|=\|b\|=1\). The average is calculated based on \(N=100\) simulations. The shaded blue area represents 1 standard deviation of the observations. The red dashed line represents the theoretical covariance \(q_{t}(a,b)\) predicted in Theorem 2. The empirical \(L_{2}\) error for \(t=1\) is reported.
noticeable, but for instance, with width \(n=2^{1}4\), we can see that the \(L_{2}\) error is smaller with depth \(L=256\) as compared to depth \(L=2\). The theoretical prediction \(q_{t}\) is approximated with a PDE solver (RK45 method, [1]) for \(t\in[0,1]\) with a discretization step \(\Delta t=\)1e-6.

Convergent Scaling \(\alpha_{l,L}=l^{-1}\).In Fig. 2, we run the same experiment for decreasing scaling \(\alpha_{l,L}=l^{-1}\). Note that in this case, the limiting neural covariance does not depend on \(t\). The red dashed line represents this limiting value in the figure (estimated with \(q_{L,n}\) with \(L=\)1e5 and \(n=\)1e5). Similar to the results with uniform scaling, we observe a convergence pattern to the red line as \(L\) and \(n\) increase. The role of depth in this case is more pronounced.

### Comparison with other architectures

In Fig. 3, we show the evolution of the distribution of \(q_{L,n}\) for three different architectures with \(L=n\). With our choice of scaling factors \(\alpha_{l,L}\), the distribution concentrates around the deterministic limit given by the solution of the ODE described in Theorem 2. For MLP with shaped ReLU, and the Shaped ResNet (Proposition 4, the main branch is scaled with \(\beta=1/2\)), we observe that the neural covariance remains random as width (and depth) grows. The sequential infinite-width-then-depth is illustrated in blue, and shows that with

Figure 2: Same setup of Fig. 1, with \(\alpha_{l,L}=l^{-1}\). The red dashed line represents the theoretical covariance \(q_{\infty}(a,b)\) predicted in Theorem 1. The empirical \(L_{2}\) error for \(t=1\) is reported.
our choice of scaling factors, the covariance concentrates around this sequential limit even when \(n=L\to\infty\). In contrast, with shaped MLP/ResNet, the two limits (sequential vs proportional) exhibit different behaviours, confirming that commutativity does not hold in these two cases.

### Improved Depth rate

In [55], commutativity was established with the choice of scaling factors \(\alpha_{l,L}=L^{-1/2}\). The reported convergence rate (as \(n\) and \(L\) go to infinity) of the neural covariance is of the form \(\mathcal{O}(n^{-1/2}+L^{-1/2})\). In this paper, we established an improved convergence rate of order \(\mathcal{O}(n^{-1/2}+L^{-1})\), which suggests that convergence is more sensitive to the width than to depth. To validate this result, we conduct the following experiment: we take \(n\) to infinity while fixing \(L\) and obtain the infinite-width neural covariance \(q_{L,\infty}\) (infinite-width covariance for the last layer). We then measure \(\Delta_{L}=|q_{L,\infty}(a,b)-q_{t=1}(a,b)|\) where \(q_{t}\) is given in Theorem 2 and \((a,b)\) are sampled randomly following the procedure in Fig. 1. We observe a perfect match of the \(L^{-1}\) convergence rate (where the intercept was adjusted so that all the lines start from the same initial value).

## 7 Outline of Proof Techniques

In [55], it was shown that the neural covariance satisfies commutativity with the specific scaling \(\alpha_{l,L}=L^{-1/2}\). The main technical novelty in that work is taking the depth \(L\) to infinity first, while controlling the dependence of the constants on the width \(n\). Given a fixed width \(n\), taking depth \(L\) to infinity results in an SDE behaviour (Stochastic Differential

Figure 3: The distribution of the \(q_{L,n}\) with \(L=n\) for varying \(n\in[5,4000]\). **(Left)** ResNet described in Eq. (5) with \(\alpha_{l,L}=L^{-1/2}\). **(Center)** MLP described in Eq. (2) with shaped ReLU \(\phi_{L}\)[45]. **(Right)** Shaped ResNet with \(\beta=1/2\) (Proposition 4). The vertical blue line represents the limit sequential limit \(\lim_{L\to\infty}\lim_{n\to\infty}q_{L,n}\). The inputs \(a,b\) are sampled following the same procedure in Fig. 1.
Equation), and the main tools to study such convergence are numerical methods for SDEs (Euler discretization scheme). In this case, it is possible to obtain infinite-depth strong convergence where the constants do not depend on width \(n\). Commutativity then follows by studying the infinite-width limit of these SDEs. This involves the use of tools from mean-field stochastic calculus (namely McKean-Vlasov processes).

In this work, we take an orthogonal approach where the width is taken to infinity first, and the constants are well chosen so that they do not depend on depth, followed by infinite-depth which concludes the proof. The main innovation in the proofs is related to the introduction of _the auxiliary process_\(\tilde{Y}\): given a residual network of the form Eq. (5), we introduce an auxiliary process \(\tilde{Y}_{l}\) that shares some properties with the original neural process \(Y_{l}\). We bound the difference between \(Y_{l}\) and \(\tilde{Y}_{l}\) using Gronwall's type of techniques, and show that the constants in this bound can be chosen to be independent of depth, hence providing a depth-uniform bound for the infinite-width limit. More importantly, the auxiliary process has iid Gaussian entries, which facilitates the study of the covariance kernel related to \(\tilde{Y}_{l}\), and allow us to conclude on commutativity. Some technical results involve the use of concentration inequalities to deal with low probability events such as \(\phi(Y_{l})=0_{\mathbb{R}^{n}}\) when \(n\) is large.

## 8 Conclusion and Limitations

In this paper, we have shown that, at initialization, under general assumptions on the sequence of scaling factors, the large-depth and large-width limits of a residual neural network (resnet) commute for the neural covariance. We used novel proof techniques. Our results generalize and strengthen previous works on commutativity.

However, our results are restricted to the neural covariance function and cannot imply anything about commutativity for other neural functions. More importantly, it is unclear what happens during training, and potentially, different behaviors can occur depending on how the learning rate is chosen as a function of width and depth. While we can heuristically conjecture that commutativity holds during training under suitable scaling strategies, commutativity is a precise mathematical statement that requires rigorous proofs.

One might also ask whether commutativity is needed in the current context of Large Language Models, where most architectures are in the regime \(n\gg L\gg 1\) (e.g. \(n\sim 1000,L\sim 50\)) and that this regime can be fairly described by the sequential limit '\(n\to\infty\), then \(L\to\infty\)'. While this might be true to some extent, note that convergence of neural functions can happen at different width and depth rates (e.g. \(n^{-1/2}\) and \(L^{-1}\) in the case of neural covariance), which implies that small changes in depth (or width) could significantly change the behaviour of the neural function. We leave this question for future work.