Phase Diagram of Initial Condensation for Two-layer Neural Networks
The phenomenon of distinct behaviors exhibited by neural networks under
varying scales of initialization remains an enigma in deep learning research.
In this paper, based on the earlier work by Luo et al.~\cite{luo2021phase}, we
present a phase diagram of initial condensation for two-layer neural networks.
Condensation is a phenomenon wherein the weight vectors of neural networks
concentrate on isolated orientations during the training process, and it is a
feature in non-linear learning process that enables neural networks to possess
better generalization abilities. Our phase diagram serves to provide a
comprehensive understanding of the dynamical regimes of neural networks and
their dependence on the choice of hyperparameters related to initialization.
Furthermore, we demonstrate in detail the underlying mechanisms by which small
initialization leads to condensation at the initial training stage.
Zhengan Chen, Yuqing Li, Tao Luo, Zhangchen Zhou, Zhi-Qin John Xu
# Phase Diagram of Initial Condensation for Two-layer Neural Networks

###### Abstract

The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the earlier work by Luo et al. [16], we present a phase diagram of initial condensation for two-layer neural networks. Condensation is a phenomenon wherein the weight vectors of neural networks concentrate on isolated orientations
during the training process, and it is a feature in non-linear learning process that enables neural networks to possess better generalization abilities. Our phase diagram serves to provide a comprehensive understanding of the dynamical regimes of neural networks and their dependence on the choice of hyperparameters related to initialization. Furthermore, we demonstrate in detail the underlying mechanisms by which small initialization leads to condensation at the initial training stage.

_Keywords--_ two-layer neural network, phase diagram, dynamical regime, condensation

## 1 Introduction

In deep learning, one intriguing observation is the distinct behaviors exhibited by Neural Networks (NNs) depending on the scale of initialization. Specifically, in a particular regime, NNs trained with gradient descent can be viewed as a kernel regression predictor known as the Neural Tangent Kernel (NTK) [11, 5, 10, 15], and Chizat et al. [4] identify it as the lazy training regime in which the parameters of overparameterized NNs trained with gradient based methods hardly varies. However, under a different scaling, the Gradient Flow (GF) of NN shows highly nonlinear features and a mean-field analysis [19, 23, 3, 24] has been established for infinitely wide two-layer networks to analyze its behavior. Additionally, small initialization is proven to give rise to condensation [18, 16, 31, 32], a phenomenon where the weight vectors of NNs concentrate on isolated orientations during the training process. This is significant as NNs with condensed weight vectors are equivalent to "smaller" NNs with fewer parameters, as revealed by the embedding principle (the loss landscape of a DNN "contains" all the critical points of all the narrower DNNs [30, 29]), thus reducing the complexity of the output functions of NNs. As the generalization error can be bounded in terms of the complexity [1], NNs with condensed parameters tend to possess better generalization abilities. In addition, the study of the embedding principle also found the number of the descent directions in a condensed large network is no less than that of the equivalent small effective network, which may lead to easier training of a large network [30, 29].

Taken together, identifying the regime of condensation and understanding the mechanism of condensation are important to understand the non-linear training of neural networks. Our contributions can be categorized into two aspects.

Firstly, we established the phase diagram of initial condensation for two-layer neural networks (NNs) with a wide class of smooth activation functions, as illustrated Figure 1. Note that the phase diagram drawn in [16] is only for two-layer
wide ReLU networks and the phase diagram in [32] is empirical for three-layer wide ReLU networks. The phase diagram of a two-layer neural network refers to a graphical representation of the dynamical behavior of the network as a function of its initialization scales. In this diagram, different regions correspond to different types of behaviors exhibited by NNs, such as the linear regime, where the network behaves like a linear model, and the condensed regime, where the network exhibits the initial condensation phenomenon.

Secondly, we reveal the mechanism of initial condensation for two-layer NNs and identify the directions towards which the weight parameters condense. There has been a flurry of recent papers endeavor to analyze the mechanism underlying the condensation of NNs at the initial training stage under small initialization [18, 21, 16, 17, 32]. For instance, Maennel et al. [18] uncovered that for two-layer ReLU NNs, the GF limits the weight vectors to a certain number of directions depending sorely on the input data. Zhou et al. [32] showed empirically that condensation is a common feature in non-linear training regime for three-layer ReLU NNs. Theoretically, Maennel et al. [18] argued that GF prefers "simple" func

Figure 1: Phase diagram of two-layer NNs.
tions over "complex" ones, and Zhou et. al. [31] demonstrated that the maximal number of condensed orientations at initial training stage is twice the multiplicity (Definition 1) of the activation function. However, these proofs are heuristic as they do not account for the dynamics of parameters. Pellegrini and Biroli [21] derived a mean-field model demonstrating that two-layer ReLU NNs, when trained with hinge loss and infinite data, lead to a linear classifier. Nonetheless, their analysis does not illustrate how the initial condensation depends on the scale of initialization and does not specify which directions NNs condense on.

The organization of the paper is listed as follows. In Section 2, we discuss some related works. In Section 3, we give some preliminary introduction to our problems. In Section 4, we state our main results and show some empirical evidence. In Section 5, we give out the outline of proofs for our main results, and conclusions are drawn in Section 6. All the details of the proof are deferred to the Appendix.

## 2 Related Works

There has been a rich literature on the choice of initialization schemes in order to facilitate neural network training [7, 9, 19, 24], and most of the work identified the width \(m\) as a hyperparameter, where the kernel regime is reached when the width grows towards infinity [11, 27, 6]. However, with the introduction of lazy training by Chizat et al. [4], instead of the width \(m\), one shall take the initialization scale as the relevant hyperparameter. The lazy training refers to the phenomenon in which a heavily over-parameterized NN trained with gradient-based methods could converge exponentially fast to zero training loss with its parameters hardly varying, and such phenomenon can be observed in any non-convex model accompanied by the choice of an appropriate scaling factor of the initialization. Follow-up work by Woodworth et al. [26] focus on how the scale of initialization acts as a controlling quantity for the transition between two very different regimes, namely the kernel regime and the rich regime, for the matrix factorization problems. As for two-layer ReLU NNs, the phase digram in Luo et al. [16] identified three regimes, namely the linear regime, the critical regime and the condensed regime, based on the relative change of input weights as the width \(m\) approaches infinity. In summary, the selection of appropriate initialization scales plays a crucial role in the training of NNs.

Several theoretical works studying the dynamical behavior of NNs with small initialization can be connected to implicit regularization effect provided by the weight initialization schemes, and the condensation phenomenon has also been studied under different names. Ji and Telgarsky [12] analyzed the implicit regularization of GF on deep linear networks and observed the matrix alignment
phenomena, i.e., weight matrices belonging to different layers share the same direction. The weight quantization effect [18] in training two-layer ReLU NNs with small initialization is really the condensation phenomenon in disguise, and so is the case for the weight cluster effect [2] in learning the MNIST task for a three-layer CNN. Luo et al. [16] focused on how the condensation phenomenon can be clearly detected by the choice of initialization schemes, but they did not show the reason behind it. Zhang et al. [28, 30] proposed a general Embedding Principle of loss landscape of DNNs, showing that a larger DNN can experience critical points with condensed parameter, and its output is the same as that of a much smaller DNN, but their analysis did not involve its dynamical behavior. Zhou et al. [31] presented a theory for the initial direction towards which the weight vector condenses, yet it is far from satisfactory.

## 3 Preliminaries

### Notations

We begin this section by introducing some notations that will be used in the rest of this paper. We set \(n\) for the number of input samples and \(m\) for the width of the neural network. We set \(\mathcal{N}(\boldsymbol{\mu},\Sigma)\) as the normal distribution with mean \(\boldsymbol{\mu}\) and covariance \(\Sigma\). We let \([n]=\{1,2,\ldots,n\}\). We denote vector \(L^{2}\) norm as \(\left\|\cdot\right\|_{2}\), vector or function \(L_{\infty}\) norm as \(\left\|\cdot\right\|_{\infty}\), matrix spectral (operator) norm as \(\left\|\cdot\right\|_{2\to 2}\), matrix infinity norm as \(\left\|\cdot\right\|_{\infty\to\infty}\), and matrix Frobenius norm as \(\left\|\cdot\right\|_{\mathrm{F}}.\) For a matrix \(\mathbf{A}\), we use \(\mathbf{A}_{i,j}\) to denote its \((i,j)\)-th entry. We will also use \(\mathbf{A}_{i,\;\text{ }\text{ }}\) to denote the \(i\)-th row vector of \(\mathbf{A}\) and define \(\mathbf{A}_{i,j:k}=[\mathbf{A}_{i,j},\mathbf{A}_{i,j+1},\cdots,\mathbf{A}_{i, k}]^{\intercal}\) as part of the vector. Similarly, \(\mathbf{A}_{:,i}\) is the \(i\)-th column vector and \(\mathbf{A}_{j:k,i}\) is a part of the \(i\)-th column vector. For a semi-positive-definite matrix \(\boldsymbol{A}\), we denote its smallest eigenvalue by \(\lambda_{\min}(\boldsymbol{A})\), and correspondingly, its largest eigenvalue by \(\lambda_{\max}(\boldsymbol{A})\). We use \(\mathcal{O}(\cdot)\) and \(\Omega(\cdot)\) for the standard Big-O and Big-Omega notations. We finally denote the set of continuous functions \(f(\cdot):\mathbb{R}\to\mathbb{R}\) possessing continuous derivatives of order up to and including \(r\) by \(\mathcal{C}^{r}(\mathbb{R})\), the set of analytic functions \(f(\cdot):\mathbb{R}\to\mathbb{R}\) by \(\mathcal{C}^{\omega}(\mathbb{R})\), and \(\langle\cdot,\cdot\rangle\) for standard inner product between two vectors.

### Problem Setup

We use almost the same settings in Luo et al. [16] by starting with the original model

\[f_{\boldsymbol{\theta}}(\boldsymbol{x})=\sum_{k=1}^{m}a_{k}\sigma(\boldsymbol {w}_{k}^{\intercal}\boldsymbol{x}), \tag{3.1}\]
whose parameters \(\mathbf{\theta}^{0}:=\text{vec}(\mathbf{\theta}^{0}_{a},\mathbf{\theta}^{0}_{\mathbf{w}})\) are initialized by

\[a^{0}_{k}\sim\mathcal{N}(0,\nu^{2}),\quad\mathbf{w}^{0}_{k}\sim\mathcal{N}(\mathbf{0}, \varepsilon^{2}\mathbf{I}_{d}), \tag{3.2}\]

and the empirical risk is

\[R_{S}(\mathbf{\theta})=\frac{1}{2n}\sum_{i=1}^{n}{\left(f_{\mathbf{\theta}}(\mathbf{x}_{i} )-y_{i}\right)^{2}}. \tag{3.3}\]

Then the training dynamics based on gradient descent (GD) at the continuous limit obeys the following gradient flow precisely reads: For \(k\in[m]\),

\[\frac{\mathrm{d}a_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}a_{k^{ \prime}}\sigma(\mathbf{w}^{\intercal}_{k^{\prime}}\mathbf{x}_{i})-y_{i}\right)\sigma( \mathbf{w}^{\intercal}_{k}\mathbf{x}_{i}), \tag{3.4}\] \[\frac{\mathrm{d}\mathbf{w}_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}a_{k^{ \prime}}\sigma(\mathbf{w}^{\intercal}_{k^{\prime}}\mathbf{x}_{i})-y_{i}\right)a_{k} \sigma^{(1)}(\mathbf{w}^{\intercal}_{k}\mathbf{x}_{i})\mathbf{x}_{i}.\]

We identify the parameters \(\mathbf{\theta}_{a}:=\text{vec}(\{a_{k}\}_{k=1}^{m})\) and \(\mathbf{\theta}_{\mathbf{w}}:=\text{vec}(\{\mathbf{w}_{k}\}_{k=1}^{m})\) as variables of order one by setting

\[a_{k}=\nu\bar{a}_{k},\quad\mathbf{w}_{k}=\varepsilon\bar{\mathbf{w}}_{k},\]

then the rescaled dynamics can be written as

\[\nu\frac{\mathrm{d}\bar{a}_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}\nu \varepsilon\bar{a}_{k^{\prime}}\frac{\sigma(\varepsilon\bar{\mathbf{w}}^{\intercal }_{k^{\prime}}\mathbf{x}_{i})}{\varepsilon}-y_{i}\right)\varepsilon\frac{\sigma( \varepsilon\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})}{\varepsilon}, \tag{3.5}\] \[\varepsilon\frac{\mathrm{d}\bar{\mathbf{w}}_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}\nu \varepsilon\bar{a}_{k^{\prime}}\frac{\sigma(\varepsilon\bar{\mathbf{w}}^{\intercal }_{k^{\prime}}\mathbf{x}_{i})}{\varepsilon}-y_{i}\right)\nu\bar{a}_{k}\sigma^{(1)} (\varepsilon\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})\mathbf{x}_{i}.\]

For the case where \(\varepsilon\ll 1\) and \(\varepsilon\gg 1\), the expressions \(\frac{\sigma(\varepsilon\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})}{\varepsilon}\) and \(\sigma^{(1)}(\varepsilon\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})\) are hard to handle at first glance. However, in the case where \(\varepsilon\ll 1\), under the condition (Assumption 1) that \(\sigma(0)=0\) and \(\sigma^{(1)}(0)=1\), we obtain that

\[\frac{\sigma(\varepsilon\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})}{\varepsilon} \approx\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i},\quad\sigma^{(1)}(\varepsilon \bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})\approx 1,\]

hence \(\frac{\sigma(\varepsilon\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})}{\varepsilon}\) and \(\sigma^{(1)}(\varepsilon\bar{\mathbf{w}}^{\intercal}_{k}\mathbf{x}_{i})\) are of order one.

In the case where \(\varepsilon\gg 1\), under the condition (Assumption 2) that

\[\lim_{x\to-\infty}\sigma^{(1)}(x)=a,\quad\lim_{x\to+\infty}\sigma^{(1)}(x)=b,\]
we obtain that

\[\frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal}\mathbf{x}_{i})}{\varepsilon} \approx\sigma^{(1)}(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal}\mathbf{x}_{i}),\]

hence \(\frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal}\mathbf{x}_{i})}{\varepsilon}\) and \(\sigma^{(1)}(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal}\mathbf{x}_{i})\) are also of order one. Under these two aforementioned conditions, \(\sigma(\cdot)\) acts like a linear activation in the case where \(\varepsilon\ll 1\), and acts like a leaky-ReLU activation in the case where \(\varepsilon\gg 1\), both of which are homogeneous functions. Hence the above dynamics can be simplified into

\[\frac{\mathrm{d}\bar{a}_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}\nu \varepsilon\bar{a}_{k^{\prime}}\frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k^{ \prime}}^{\intercal}\mathbf{x}_{i})}{\varepsilon}-y_{i}\right)\frac{\varepsilon} {\nu}\frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal}\mathbf{x}_{i})}{ \varepsilon}, \tag{3.6}\] \[\frac{\mathrm{d}\bar{\mathbf{w}}_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}\nu \varepsilon\bar{a}_{k^{\prime}}\frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k^{ \prime}}^{\intercal}\mathbf{x}_{i})}{\varepsilon}-y_{i}\right)\frac{\nu}{ \varepsilon}\bar{a}_{k}\sigma^{(1)}(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal} \mathbf{x}_{i})\mathbf{x}_{i}.\]

We hereby introduce two scaling parameters

\[\kappa:=\nu\varepsilon,\quad\kappa^{\prime}:=\frac{\nu}{\varepsilon}, \tag{3.7}\]

then the dynamics (3.6) can be written as a _normalized_ flow

\[\frac{\mathrm{d}\bar{a}_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}\kappa \bar{a}_{k^{\prime}}\frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k^{\prime}}^{ \intercal}\mathbf{x}_{i})}{\varepsilon}-y_{i}\right)\frac{1}{\kappa^{\prime}} \frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal}\mathbf{x}_{i})}{\varepsilon}, \tag{3.8}\] \[\frac{\mathrm{d}\bar{\mathbf{w}}_{k}}{\mathrm{d}t} =-\frac{1}{n}\sum_{i=1}^{n}\left(\sum_{k^{\prime}=1}^{m}\kappa \bar{a}_{k^{\prime}}\frac{\sigma(\varepsilon\bar{\mathbf{w}}_{k^{\prime}}^{ \intercal}\mathbf{x}_{i})}{\varepsilon}-y_{i}\right)\kappa^{\prime}\bar{a}_{k} \sigma^{(1)}(\varepsilon\bar{\mathbf{w}}_{k}^{\intercal}\mathbf{x}_{i})\mathbf{x}_{i}.\]

with the following initialization

\[\bar{a}_{k}^{0}\sim\mathcal{N}(0,1),\quad\bar{\mathbf{w}}_{k}^{0}\sim\mathcal{N}( \mathbf{0},\mathbf{I}_{d}). \tag{3.9}\]

In the following discussion throughout this paper, we always refer to this rescaled model (3.8) and drop all the "bar"s of \(\{a_{k}\}_{k=1}^{m}\) and \(\{\mathbf{w}_{k}\}_{k=1}^{m}\) for notational simplicity.

As \(\kappa\) and \(\kappa^{\prime}\) are always in specific power-law relations to the width \(m\), we introduce two independent coordinates

\[\gamma:=\lim_{m\to\infty}-\frac{\log\kappa}{\log m},\quad\gamma^{\prime}:= \lim_{m\to\infty}-\frac{\log\kappa^{\prime}}{\log m}, \tag{3.10}\]

which meet all the guiding principles [16] for finding the coordinates of a phase diagram.

Before we end this section, we list out some commonly-used initialization methods with their scaling parameters as shown in Table 1.
## 4 Main Results

### Activation function and input

In this part, we shall impose some technical conditions on the activation function and input samples. We start with a technical condition [31, Definition 1] on the activation function \(\sigma(\cdot)\)

**Definition 1** (Multiplicity \(p\)).: \(\sigma(\cdot):\mathbb{R}\to\mathbb{R}\) _has multiplicity \(p\) if there exists an integer \(p\geq 1\), such that for all \(0\leq s\leq p-1\), the \(s\)-th order derivative satisfies \(\sigma^{(s)}(0)=0\), and \(\sigma^{(p)}(0)\neq 0\)._

We list out some examples of activation functions with different multiplicity.

**Remark 1**.:
* \(tanh(x):=\frac{\exp(x)-\exp(-x)}{\exp(x)+\exp(-x)}\) _is with multiplicity_ \(p=1\)_;_
* \(SiLU(x):=\frac{x}{1+\exp(-x)}\) _is with multiplicity_ \(p=1\)_;_
* \(xtanh(x):=\frac{x\exp(x)-x\exp(-x)}{\exp(x)+\exp(-x)}\) _is with multiplicity_ \(p=2\)_._

**Assumption 1** (Multiplicity 1).: _The activation function \(\sigma\in\mathcal{C}^{2}(\mathbb{R})\), and there exists a universal constant \(C_{L}>0\), such that its first and second derivatives satisfy_

\[\left\|\sigma^{(1)}(\cdot)\right\|_{\infty}\leq C_{L},\quad\left\|\sigma^{(2)} (\cdot)\right\|_{\infty}\leq C_{L}. \tag{4.1}\]

_Moreover,_

\[\sigma(0)=0,\quad\sigma^{(1)}(0)=1. \tag{4.2}\]

\begin{table}
\begin{tabular}{|l|c c c c c|} \hline Name & \(\nu\) & \(\varepsilon\) & \(\kappa\) (\(\nu\varepsilon\)) & \(\kappa^{\prime}\) (\(\nu/\varepsilon\)) & \(\gamma\) & \(\gamma^{\prime}\) \\ \hline LeCun [14] & \(\sqrt{\frac{1}{m}}\) & \(\sqrt{\frac{1}{d}}\) & \(\sqrt{\frac{1}{md}}\) & \(\sqrt{\frac{d}{m}}\) & \(\frac{1}{2}\) & \(\frac{1}{2}\) \\ He [8] & \(\sqrt{\frac{2}{m}}\) & \(\sqrt{\frac{2}{d}}\) & \(\sqrt{\frac{4}{md}}\) & \(\sqrt{\frac{d}{m}}\) & \(\frac{1}{2}\) & \(\frac{1}{2}\) \\ Xavier [7] & \(\sqrt{\frac{2}{m+1}}\) & \(\sqrt{\frac{2}{m+d}}\) & \(\sqrt{\frac{4}{(m+1)(d+1)}}\) & \(\sqrt{\frac{m+d}{m+1}}\) & \(1\) & \(0\) \\ Huang [10] & \(1\) & \(\sqrt{\frac{1}{m}}\) & \(\sqrt{\frac{1}{m}}\) & \(\sqrt{m}\) & \(\frac{1}{2}\) & \(-\frac{1}{2}\) \\ \hline \end{tabular}
\end{table}
Table 1: Initialization methods with their scaling parameters
**Remark 2**.: _We remark that \(\sigma\) has multiplicity \(1\). \(\sigma^{(1)}(0)=1\) can be replaced by \(\sigma^{(1)}(0)\neq 0\), and we set \(\sigma^{(1)}(0)=1\) for simplicity, and it can be easily satisfied by replacing the original activation \(\sigma(\cdot)\) with \(\frac{\sigma(\cdot)}{\sigma^{(1)}(0)}\)._

_We note that Assumption 1 can be satisfied by using the tanh activation:_

\[\sigma(x)=\frac{\exp(x)-\exp(-x)}{\exp(x)+\exp(-x)},\]

_and the scaled SiLU activation_

\[\sigma(x)=\frac{2x}{1+\exp(-x)}.\]

**Assumption 2**.: _The activation function \(\sigma\in\mathcal{C}^{\omega}(\mathbb{R})\) and is not a polynomial function, also its function value at \(0\) satisfy_

\[\sigma(0)=0, \tag{4.3}\]

_also there exists a universal constant \(C_{L}>0\), such that its first and second derivatives satisfy_

\[\sigma^{(1)}(0)=1,\quad\left\|\sigma^{(1)}(\cdot)\right\|_{\infty}\leq C_{L}, \quad\left\|\sigma^{(2)}(\cdot)\right\|_{\infty}\leq C_{L}. \tag{4.4}\]

_Moreover,_

\[\lim_{x\to-\infty}\sigma^{(1)}(x)=a,\quad\lim_{x\to+\infty}\sigma^{(1)}(x)=b, \tag{4.5}\]

_and \(a\neq b\)._

**Remark 3**.: _We note that Assumption 2 can be satisfied by by using the scaled SiLU activation:_

\[\sigma(x)=\frac{2x}{1+\exp(-x)},\]

_where \(a=0\) and \(b=2\)._

_Some other functions also satisfy this assumption, for instance, the modified scaled softplus activation:_

\[\sigma(x)=2\left(\log(1+\exp(x))-\log 2\right),\]

_where \(a=0\) and \(b=2\)._

**Assumption 3** (Non-degenerate data).: _The training inputs and labels \(\mathcal{S}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n}\) satisfy that there exists a universal constant \(c>0\), such that for all \(i\in[n]\),_

\[\frac{1}{c}\leq\left\|\mathbf{x}_{i}\right\|_{2},\quad|y_{i}|\leq c,\]
_and_

\[\sum_{i=1}^{n}y_{i}\mathbf{x}_{i}\neq\mathbf{0}. \tag{4.6}\]

_We denote by_

\[\mathbf{z}:=\frac{1}{n}\sum_{i=1}^{n}y_{i}\mathbf{x}_{i}, \tag{4.7}\]

_and assume further that for some universal constant \(c>0\), the following holds_

\[\frac{1}{c}\leq\left\|\mathbf{z}\right\|_{2}\leq c, \tag{4.8}\]

_and its unit vector_

\[\hat{\mathbf{z}}:=\frac{\sum_{i=1}^{n}y_{i}\mathbf{x}_{i}}{\left\|\sum_{i=1}^{n}y_{i} \mathbf{x}_{i}\right\|_{2}}. \tag{4.9}\]

**Assumption 4**.: _The training inputs and labels \(\mathcal{S}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n}\) satisfy that there exists a universal constant \(c>0\), such that for all \(i\in[n]\),_

\[\frac{1}{c}\leq\left\|\mathbf{x}_{i}\right\|_{2},\quad|y_{i}|\leq c,\]

_and all training inputs are non-parallel with each other, i.e., for any \(i\neq j\) and \(i,j\in[n]\),_

\[\mathbf{x}_{i}\nparallel\mathbf{x}_{j}.\]

We remark that the requirements in Assumption 3 are easier to meet compared with Assumption 4, and both assumptions impose the input sample \(\mathcal{S}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n}\) to be of order one.

**Assumption 5**.: _The following limit exists_

\[\gamma_{1}:=\lim_{m\to\infty}-\frac{\log\nu}{\log m},\quad\gamma_{2}:=\lim_{m \to\infty}-\frac{\log\varepsilon}{\log m}, \tag{4.10}\]

_then by definition_

\[\gamma=\lim_{m\to\infty}-\frac{\log\nu\varepsilon}{\log m}=\gamma_{1}+\gamma_ {2},\quad\gamma^{\prime}=\lim_{m\to\infty}-\frac{\log\frac{\nu}{\varepsilon} }{\log m}=\gamma_{1}-\gamma_{2}.\]
### Regime Characterization at Initial Stage

Before presenting our theory that establishes a consistent boundary to separate the diagram into two distinct areas, namely the linear regime area and the condensed regime area, we introduce a quantity that has proven to be valuable in the analysis of NNs.

It is known that the output of a two-layer NN is linear with respect to \(\mathbf{\theta}_{a}\), hence if the set of parameter \(\mathbf{\theta}_{\mathbf{w}}\) remain stuck to its initialization throughout the whole training process, then the training dynamics of a two-layer NN can be linearized around the initialization. In the phase diagram, the linear regime area precisely corresponds to the region where the output function of a two-layer NN can be well approximated by its linearized model, i.e., in the linear regime area, the following holds

\[\begin{split} f_{\mathbf{\theta}}(\mathbf{x})\approx f\left(\mathbf{x},\mathbf{ \theta}(0)\right)&+\left\langle\nabla_{\mathbf{\theta}_{a}}f\left(\mathbf{ x},\mathbf{\theta}(0)\right),\mathbf{\theta}_{a}(t)-\mathbf{\theta}_{a}(0)\right\rangle\\ &+\left\langle\nabla_{\mathbf{\theta}_{\mathbf{w}}}f\left(\mathbf{x},\mathbf{ \theta}(0)\right),\mathbf{\theta}_{\mathbf{w}}(t)-\mathbf{\theta}_{\mathbf{w}}(0)\right\rangle. \end{split} \tag{4.11}\]

In general, this linear approximation holds only when \(\mathbf{\theta}_{\mathbf{w}}(t)\) remains within a small neighbourhood of \(\mathbf{\theta}_{\mathbf{w}}(0)\). Since the size of this neighbourhood scales with \(\|\mathbf{\theta}_{\mathbf{w}}(0)\|_{2}\), therefore we use the following quantity as an indicator of how far \(\mathbf{\theta}_{\mathbf{w}}(t)\) deviates away from \(\mathbf{\theta}_{\mathbf{w}}(0)\) throughout the training process

\[\sup_{t\in[0,+\infty)}\mathrm{RD}(\mathbf{\theta}_{\mathbf{w}}(t))=\frac{\left\|\mathbf{ \theta}_{\mathbf{w}}(t)-\mathbf{\theta}_{\mathbf{w}}(0)\right\|_{2}}{\|\mathbf{\theta}_{\mathbf{w }}(0)\|_{2}}. \tag{4.12}\]

We demonstrate that as \(m\to\infty\), under suitable choice of the initialization scales (the blue area in Figure 1), the NN training dynamics fall into the linear regime (Theorem 1), and for large enough \(m\),

\[\sup_{t\in[0,+\infty)}\mathrm{RD}(\mathbf{\theta}_{\mathbf{w}}(t))\to 0.\]

We also demonstrate that under some other choices of the initialization scales (the green area in Figure 1), the NN training dynamics fall into the condensed regime (Theorem 2), where

\[\sup_{t\in[0,+\infty)}\mathrm{RD}(\mathbf{\theta}_{\mathbf{w}}(t))\to+\infty,\]

and the phenomenon of condensation can be observed, and \(\mathbf{\theta}_{\mathbf{w}}\) condense toward the direction of \(\mathbf{z}\). We observe that in both cases, as \(\mathbf{\theta}_{\mathbf{w}}\) deviates far away from initialization, the approximation (4.11) fails, and NN training dynamics is essentially nonlinear with respect to \(\mathbf{\theta}_{\mathbf{w}}\).

Moreover, under the remaining choice of the initialization scales (the solid blue line in Figure 1), the NN training dynamics fall into the critical regime area, and
we conjecture that

\[\sup_{t\in[0,+\infty)}\operatorname{RD}(\boldsymbol{\theta}_{\boldsymbol{w}}(t)) \rightarrow\mathcal{O}(1),\]

whose study is beyond the scope of this paper.

**Theorem 1** (Linear regime).: _Given any \(\delta\in(0,1)\), under Assumption 2, Assumption 4 and Assumption 5, if \(\gamma<1\) or \(\gamma^{\prime}>\gamma-1\), then with probability at least \(1-\delta\) over the choice of \(\boldsymbol{\theta}^{0}\),_

\[\lim_{m\rightarrow\infty}\sup_{t\in[0,+\infty)}\frac{\|\boldsymbol{\theta}_{ \boldsymbol{w}}(t)-\boldsymbol{\theta}_{\boldsymbol{w}}(0)\|_{2}}{\| \boldsymbol{\theta}_{\boldsymbol{w}}(0)\|_{2}}=0. \tag{4.13}\]

**Remark 4**.: _The linear regime area is split into two parts, one is termed the \(\boldsymbol{\theta}\)-lazy area (blue area in Figure 2), where \(\gamma<1\), the other is termed the \(\boldsymbol{w}\)-lazy area (pink area in Figure 2), where \(\gamma\geq 1\) and \(\gamma^{\prime}>\gamma-1>0\)._

_In the \(\boldsymbol{\theta}\)-lazy area, the following relation holds_

\[\lim_{m\rightarrow\infty}\sup_{t\in[0,+\infty)}\frac{\|\boldsymbol{\theta}(t) -\boldsymbol{\theta}(0)\|_{2}}{\|\boldsymbol{\theta}(0)\|_{2}}=0, \tag{4.14}\]

_whose detailed reasoning can be found in Appendix B.2, and in the \(\boldsymbol{w}\)-lazy area, relation (4.14) does not hold._

Figure 2: Different training regimes in the phase diagram.
**Theorem 2** (Condensed regime).: _Given any \(\delta\in(0,1)\), under Assumption 1, Assumption 3 and Assumption 5, if \(\gamma>1\) and \(\gamma^{\prime}<\gamma-1\), then with probability at least \(1-\delta\) over the choice of \(\mathbf{\theta}^{0}\), there exists \(T>0\), such that_

\[\lim_{m\to\infty}\sup_{t\in[0,T]}\frac{\left\|\mathbf{\theta}_{\mathbf{w}}(t)-\mathbf{ \theta}_{\mathbf{w}}(0)\right\|_{2}}{\left\|\mathbf{\theta}_{\mathbf{w}}(0)\right\|_{2}}=+\infty, \tag{4.15}\]

_and_

\[\lim_{m\to\infty}\sup_{t\in[0,T]}\frac{\left\|\mathbf{\theta}_{\mathbf{w},\mathbf{z}}(t) \right\|_{2}}{\left\|\mathbf{\theta}_{\mathbf{w}}(t)\right\|_{2}}=1, \tag{4.16}\]

_where \(\mathbf{\theta}_{\mathbf{w},\mathbf{z}}(t):=[\left\langle\mathbf{w}_{1},\hat{\mathbf{z}}\right\rangle,\left\langle\mathbf{w}_{2},\hat{\mathbf{z}}\right\rangle,\cdots,\left\langle\mathbf{w}_{ m},\hat{\mathbf{z}}\right\rangle]^{\intercal}\)._

**Remark 5**.: _The condensed regime area is split into two parts, one is termed the \(\mathbf{w}\)-lag area (orange area in Figure 2), where \(\gamma>1\) and \(0\leq\gamma^{\prime}<\gamma-1\), the other is termed the \(a\)-lag area (yellow area in Figure 2), where \(\gamma>1\) and \(\gamma^{\prime}<0\)._

_In the \(\mathbf{w}\)-lag regime area, as illustrated in (5.14), \(\mathbf{\theta}_{\mathbf{w}}\) waits for a period of time of order one until \(\mathbf{\theta}_{a}\) attains a magnitude that is commensurate with that of \(\mathbf{\theta}_{\mathbf{w}}\), and the time \(T\) in Theorem 2 satisfies that_

\[T\geq\log\left(\frac{1}{4}\right)+\frac{\gamma-\gamma^{\prime}-1}{8}\log(m), \tag{4.17}\]

_and as \(m\to\infty\), \(T\to\infty\)._

_In the \(a\)-lag regime area, as illustrated in (5.14), \(\mathbf{\theta}_{a}\) waits for a period of time of order one until \(\mathbf{\theta}_{\mathbf{w}}\) attains a magnitude that is commensurate with that of \(\mathbf{\theta}_{a}\), and it is exactly during this interval of time that the phenomenon of initial condensation can be observed. Hence for some \(\alpha>0\), the time \(T\) in Theorem 2 can be chosen as_

\[m^{-\alpha}\leq T\leq 2m^{-\alpha}, \tag{4.18}\]

_which is obviously of order one, see Appendix C.4 for more details._

### Experimental Demonstration

In order to distinguish between the \(\mathbf{w}\)-lag regime and \(a\)-lag regime, it is necessary to estimate the time \(T\) in Remark 5, which is also a reasonable way to empirically validate our theoretical analysis. An empirical approximation of \(T\) can be obtained by determining the time interval \(\hat{T}\), starting from the initial stage at \(t=0\), up to the point at which the quantity \(\frac{\left\|\mathbf{\theta}_{\mathbf{w},\mathbf{z}}(t)\right\|_{2}}{\left\|\mathbf{\theta}_{ \mathbf{w}}(t)\right\|_{2}}\) reaches its climax for sufficiently large values of \(m\) (\(m=50000,100000,200000,400000,800000,1600000\)), as we are unable to run experiments at \(m\to\infty\).
#### 4.3.1 \(\mathbf{w}\)-lag regime

We validate the effectiveness of our estimates by performing a simple linear regression to visualize the relation (4.17), where \(\hat{T}\) is set as the response variable and \(\log m\) as the single independent variable. Figure 3 shows that NNs with different values of \(\gamma\) but fixed \(\gamma^{\prime}\) satisfy the relation (4.17), thereby demonstrating the accuracy and reliability of our estimates.

Figure 4: \(\log\hat{T}\) (ordinate) vs \(\log m\) (abscissa) with different values of \(\gamma\) but fixed \(\gamma^{\prime}=-0.4\) for two-layer NNs with tanh activation indicated by blue dots. The black line is a linear fit, and \(R^{2}\) is the coefficient of determination.

Figure 3: \(\hat{T}\) (ordinate) vs \(\log m\) (abscissa) with different values of \(\gamma\) but fixed \(\gamma^{\prime}=0\) for two-layer NNs with tanh activation indicated by blue dots. The black line is a linear fit, and \(R^{2}\) is _the coefficient of determination_ that provides information about the goodness of fit of a linear regression. The closer \(R^{2}\) is to 1, the better the model fits the data.
#### 4.3.2 \(a\)-lag regime

We repeat the strategy in Section 4.3.1 except that in one hand we are hereby to visualize the relation (4.18) in Figure 4, and in the other, \(\log\hat{T}\) is set as the response variable and \(\gamma^{\prime}\) is no longer \(0\). We can still see a good agreement between the experimental data and its linear fitting, thus, validating the relation (4.18).

## 5 Technique Overview

In this part, we describe some technical tools and present the sketch of proofs for the above two theorems. Before we proceed, a rigorous description of the updated notations and definitions is required.

We start by a two-layer normalized NN model

\[f_{\mathbf{\theta}}(\mathbf{x})=\sum_{k=1}^{m}\nu\varepsilon a_{k}\frac{\sigma( \varepsilon\mathbf{w}_{k}^{\intercal}\mathbf{x})}{\varepsilon}, \tag{5.1}\]

with the normalized parameters \(\mathbf{\theta}^{0}:=\text{vec}(\mathbf{\theta}_{a}^{0},\mathbf{\theta}_{\mathbf{w}}^{0})\) initialized by

\[a_{k}^{0} :=a_{k}(0)\sim\mathcal{N}(0,1),\] \[\mathbf{w}_{k}^{0} :=\mathbf{w}_{k}(0)\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{d}).\]

For all \(i\in[n]\), we denote hereafter that

\[e_{i}:=e_{i}(\mathbf{\theta}):=f_{\mathbf{\theta}}(\mathbf{x}_{i})-y_{i},\]

and

\[\mathbf{e}:=\mathbf{e}(\mathbf{\theta}):=[e_{1}(\mathbf{\theta}),e_{2}(\mathbf{\theta}),\ldots,e_ {n}(\mathbf{\theta})]^{\intercal}.\]

Then the normalized flow reads: For all \(k\in[m]\),

\[\begin{split}\frac{\mathrm{d}a_{k}}{\mathrm{d}t}&=- \frac{\varepsilon}{\nu}\frac{1}{n}\sum_{i=1}^{n}e_{i}\frac{\sigma(\varepsilon \mathbf{w}_{k}^{\intercal}\mathbf{x}_{i})}{\varepsilon},\\ \frac{\mathrm{d}\mathbf{w}_{k}}{\mathrm{d}t}&=-\frac{ \nu}{\varepsilon}\frac{1}{n}\sum_{i=1}^{n}e_{i}a_{k}\sigma^{(1)}(\varepsilon \mathbf{w}_{k}^{\intercal}\mathbf{x}_{i})\mathbf{x}_{i}.\end{split} \tag{5.2}\]

### Linear Regime

We define the normalized kernels as follows

\[\begin{split} k^{[a]}(\mathbf{x},\mathbf{x}^{\prime})&:=\frac {1}{\varepsilon^{2}}\mathbb{E}_{\mathbf{w}}\sigma(\varepsilon\mathbf{w}^{\intercal} \mathbf{x})\sigma(\varepsilon\mathbf{w}^{\intercal}\mathbf{x}^{\prime}),\\ k^{[\mathbf{w}]}(\mathbf{x},\mathbf{x}^{\prime})&:=\mathbb{E} _{(a,\mathbf{w})}a^{2}\sigma^{(1)}(\varepsilon\mathbf{w}^{\intercal}\mathbf{x})\sigma^{( 1)}(\varepsilon\mathbf{w}^{\intercal}\mathbf{x}^{\prime})\left<\mathbf{x},\mathbf{x}^{\prime} \right>,\end{split} \tag{5.3}\]
thus, the components of the Gram matrices \(\mathbf{K}^{[a]}\) and \(\mathbf{K}^{[\mathbf{w}]}\) of at infinite width respectively reads: For any \(i,j\in[n]\),

\[\mathbf{K}^{[a]} :=\left[K^{[a]}_{ij}\right]_{n\times n}, \tag{5.4}\] \[K^{[a]}_{ij} :=k^{[a]}(\mathbf{x}_{i},\mathbf{x}_{j}),\] \[\mathbf{K}^{[\mathbf{w}]} :=\left[K^{[\mathbf{w}]}_{ij}\right]_{n\times n},\] \[K^{[\mathbf{w}]}_{ij} :=k^{[\mathbf{w}]}(\mathbf{x}_{i},\mathbf{x}_{j}),\]

we conclude that under Assumption 2 and Assumption 4, \(\mathbf{K}^{[a]}\) and \(\mathbf{K}^{[\mathbf{w}]}\) are strictly positive definite, and both of their least eigenvalues are of order one (Theorem 4).

We define the normalized Gram matrices \(\mathbf{G}^{[a]}(\mathbf{\theta})\), \(\mathbf{G}^{[\mathbf{w}]}(\mathbf{\theta})\), and \(\mathbf{G}(\mathbf{\theta})\) for a finite width two-layer network as follows: For any \(i,j\in[n]\),

\[\mathbf{G}^{[a]}(\mathbf{\theta}) :=\left[G^{[a]}_{ij}(\mathbf{\theta})\right]_{n\times n},\] \[G^{[a]}_{ij}(\mathbf{\theta}) :=\frac{1}{m}\sum_{k=1}^{m}\left\langle\nabla_{a_{k}}f_{\mathbf{ \theta}}(\mathbf{x}_{i}),\frac{\varepsilon}{\nu}\nabla_{a_{k}}f_{\mathbf{\theta}}(\bm {x}_{j})\right\rangle\] \[=\frac{\nu^{2}}{m}\frac{\varepsilon}{\nu}\sum_{k=1}^{m}\sigma( \varepsilon\mathbf{w}_{k}^{\intercal}\mathbf{x}_{i})\sigma(\varepsilon\mathbf{w}_{k}^{ \intercal}\mathbf{x}_{j})\] \[=\frac{\nu\varepsilon^{3}}{m}\sum_{k=1}^{m}\frac{1}{\varepsilon^ {2}}\sigma(\varepsilon\mathbf{w}_{k}^{\intercal}\mathbf{x}_{i})\sigma(\varepsilon\bm {w}_{k}^{\intercal}\mathbf{x}_{j}), \tag{5.5}\] \[\mathbf{G}^{[\mathbf{w}]}(\mathbf{\theta}) :=\left[G^{[\mathbf{w}]}_{ij}(\mathbf{\theta})\right]_{n\times n},\] \[G^{[\mathbf{w}]}_{ij}(\mathbf{\theta}) :=\frac{1}{m}\sum_{k=1}^{m}\left\langle\nabla_{\mathbf{w}_{k}}f_{\mathbf{ \theta}}(\mathbf{x}_{i}),\frac{\nu}{\varepsilon}\nabla_{\mathbf{w}_{k}}f_{\mathbf{\theta}} (\mathbf{x}_{j})\right\rangle\] \[=\frac{\nu^{3}\varepsilon}{m}\sum_{k=1}^{m}a_{k}^{2}\sigma^{(1)} (\varepsilon\mathbf{w}_{k}^{\intercal}\mathbf{x}_{i})\sigma^{(1)}(\varepsilon\mathbf{w}_ {k}^{\intercal}\mathbf{x}_{j})\left\langle\mathbf{x}_{i},\mathbf{x}_{j}\right\rangle,\]

and

\[\mathbf{G}(\mathbf{\theta}):=\mathbf{G}^{[a]}(\mathbf{\theta})+\mathbf{G}^{[\mathbf{w}]}(\mathbf{\theta}). \tag{5.6}\]

**Remark 6**.: _We conclude that_

\[\lambda_{\min}\left(\mathbf{G}^{[a]}\left(\mathbf{\theta}^{0}\right)\right)\sim\Omega (\nu\varepsilon^{3}),\quad\lambda_{\min}\left(\mathbf{G}^{[\mathbf{w}]}\left(\mathbf{ \theta}^{0}\right)\right)\sim\Omega(\nu^{3}\varepsilon), \tag{5.7}\]

_and it has been rigorously achieved by Proposition 2 located in Appendix B.1._
Finally, we obtain that

\[\frac{\mathrm{d}}{\mathrm{d}t}R_{S}(\mathbf{\theta}) =-\left(\sum_{k=1}^{m}\frac{\varepsilon}{\nu}\left\langle\nabla_{a_{ k}}R_{S}(\mathbf{\theta}),\nabla_{a_{k}}R_{S}(\mathbf{\theta})\right\rangle+\sum_{k=1}^{m} \frac{\nu}{\varepsilon}\left\langle\nabla_{\mathbf{w}_{k}}R_{S}(\mathbf{\theta}), \nabla_{\mathbf{w}_{k}}R_{S}(\mathbf{\theta})\right\rangle\right)\] \[=-\frac{m}{n^{2}}\mathbf{e}^{\intercal}\left(\mathbf{G}^{[a]}(\mathbf{ \theta})+\mathbf{G}^{[\mathbf{w}]}(\mathbf{\theta})\right)\mathbf{e}.\]

In the case where \(\gamma<1\) (\(\mathbf{\theta}\)-lazy regime), the following holds for all \(t>0\):

\[\lambda_{\min}\left(\mathbf{G}^{[a]}(\mathbf{\theta}(t))\right)\geq\frac{1}{2}\nu \varepsilon^{3}\lambda,\quad\lambda_{\min}\left(\mathbf{G}^{[\mathbf{w}]}(\mathbf{\theta} (t))\right)\geq\frac{1}{2}\nu^{3}\varepsilon\lambda,\]

for some universal constant \(\lambda>0\). Hence, we obtain that

\[\frac{\mathrm{d}}{\mathrm{d}t}R_{S}(\mathbf{\theta}(t)) =-\frac{m}{n^{2}}\mathbf{e}^{\intercal}\left(\mathbf{G}^{[a]}(\mathbf{\theta }(t))+\mathbf{G}^{[\mathbf{w}]}(\mathbf{\theta}(t))\right)\mathbf{e}\] \[\leq-\frac{2m}{n}\lambda_{\min}\left(\mathbf{G}(\mathbf{\theta}(t)) \right)R_{S}(\mathbf{\theta}(t))\] \[\leq-\frac{m}{n}\nu^{2}\varepsilon^{2}\left(\frac{\varepsilon}{ \nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)R_{S}(\mathbf{\theta}(t)),\]

then

\[R_{S}(\mathbf{\theta}(t))\leq\exp\left(-\frac{m}{n}\nu^{2}\varepsilon^{2}\left( \frac{\varepsilon}{\nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)t\right) R_{S}(\mathbf{\theta}(0)). \tag{5.8}\]

The following relation

\[\lim_{m\to\infty}\sup_{t\in[0,+\infty)}\frac{\|\mathbf{\theta}(t)-\mathbf{\theta}(0) \|_{2}}{\|\mathbf{\theta}(0)\|_{2}}=0, \tag{5.9}\]

is illustrated through an intuitive scaling analysis. Since

\[\frac{\mathrm{d}}{\mathrm{d}t}R_{S}(\mathbf{\theta}(t))=-\frac{\varepsilon}{\nu} \left\|\nabla_{\mathbf{\theta}_{a}}R_{S}(\mathbf{\theta}(t))\right\|_{2}^{2}-\frac{ \nu}{\varepsilon}\left\|\nabla_{\mathbf{\theta}_{\mathbf{w}}}R_{S}(\mathbf{\theta}(t)) \right\|_{2}^{2}\sim-\frac{m}{n}\nu^{2}\varepsilon^{2}\left(\frac{\varepsilon }{\nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)R_{S}(\mathbf{\theta}(t)),\]

then we have that

\[R_{S}(\mathbf{\theta}(t))\sim\exp\left(-\frac{m}{n}\nu^{2}\varepsilon^{2}\left( \frac{\varepsilon}{\nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)t\right) R_{S}(\mathbf{\theta}(0)),\]

and

\[\left\|\nabla_{\mathbf{\theta}_{a}}R_{S}(\mathbf{\theta}(t))\right\|_{2} \sim\sqrt{\frac{m}{n}\nu^{3}\varepsilon\left(\frac{\varepsilon}{ \nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)}\sqrt{R_{S}(\mathbf{\theta}(t))}\] \[\sim\sqrt{\frac{m}{n}\nu^{3}\varepsilon\left(\frac{\varepsilon}{ \nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)}\exp\left(-\frac{m}{2n}\nu^ {2}\varepsilon^{2}\left(\frac{\varepsilon}{\nu}\lambda+\frac{\nu}{\varepsilon }\lambda\right)t\right)\sqrt{R_{S}(\mathbf{\theta}(0))},\] \[\left\|\nabla_{\mathbf{\theta}_{\mathbf{w}}}R_{S}(\mathbf{\theta}(t))\right\|_{2} \sim\sqrt{\frac{m}{n}\nu\varepsilon^{3}\left(\frac{\varepsilon}{ \nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)}\sqrt{R_{S}(\mathbf{\theta}(t))}\]
\[\sim\sqrt{\frac{m}{n}}\nu\varepsilon^{3}\left(\frac{\varepsilon}{\nu} \lambda+\frac{\nu}{\varepsilon}\lambda\right)\exp\left(-\frac{m}{2n}\nu^{2} \varepsilon^{2}\left(\frac{\varepsilon}{\nu}\lambda+\frac{\nu}{\varepsilon} \lambda\right)t\right)\sqrt{R_{S}(\boldsymbol{\theta}(0))},\]

both holds, hence

\[\|\boldsymbol{\theta}(t)-\boldsymbol{\theta}(0)\|_{2} \leq\|\boldsymbol{\theta}_{a}(t)-\boldsymbol{\theta}_{a}(0)\|_{2 }+\|\boldsymbol{\theta}_{\boldsymbol{w}}(t)-\boldsymbol{\theta}_{\boldsymbol{ w}}(0)\|_{2}\] \[\leq\frac{\varepsilon}{\nu}\int_{0}^{t}\left\|\nabla_{ \boldsymbol{\theta}_{a}}R_{S}(\boldsymbol{\theta}(s))\right\|_{2}\mathrm{d}s +\frac{\nu}{\varepsilon}\int_{0}^{t}\left\|\nabla_{\boldsymbol{\theta}_{w}}R_{S }(\boldsymbol{\theta}(s))\right\|_{2}\mathrm{d}s\] \[\leq\frac{\varepsilon}{\nu}\int_{0}^{\infty}\left\|\nabla_{ \boldsymbol{\theta}_{a}}R_{S}(\boldsymbol{\theta}(s))\right\|_{2}\mathrm{d}s +\frac{\nu}{\varepsilon}\int_{0}^{\infty}\left\|\nabla_{\boldsymbol{\theta}_{ w}}R_{S}(\boldsymbol{\theta}(s))\right\|_{2}\mathrm{d}s\] \[\lesssim\left(\sqrt{\frac{\varepsilon}{\nu}}+\sqrt{\frac{\nu}{ \varepsilon}}\right)\sqrt{\frac{n}{m\nu^{2}\varepsilon^{2}\left(\frac{ \varepsilon}{\nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)}}\sqrt{R_{S}( \boldsymbol{\theta}(0))},\]

and

\[\left\|\boldsymbol{\theta}(0)\right\|_{2}\sim\sqrt{m},\]

hence

\[\frac{\|\boldsymbol{\theta}(t)-\boldsymbol{\theta}(0)\|_{2}}{ \left\|\boldsymbol{\theta}(0)\right\|_{2}} \lesssim\left(\sqrt{\frac{\varepsilon}{\nu}}+\sqrt{\frac{\nu}{ \varepsilon}}\right)\sqrt{\frac{n}{m^{2}\nu^{2}\varepsilon^{2}\left(\frac{ \varepsilon}{\nu}\lambda+\frac{\nu}{\varepsilon}\lambda\right)}}\sqrt{R_{S}( \boldsymbol{\theta}(0))} \tag{5.10}\] \[\lesssim\sqrt{\frac{n}{m^{2}\nu^{2}\varepsilon^{2}}}\sqrt{R_{S}( \boldsymbol{\theta}(0))}.\]

The rigorous statements of relations (5.8) and (5.9) are given in Theorem 5.

In the case where \(\gamma\geq 1\) and \(\gamma^{\prime}>\gamma-1\) (\(\boldsymbol{w}\)-lazy regime), the following holds for all \(t>0\):

\[\lambda_{\min}\left(\boldsymbol{G}^{[a]}(\boldsymbol{\theta}(t))\right)\geq \frac{1}{2}\nu\varepsilon^{3}\lambda,\]

for some universal constant \(\lambda>0\). Hence, we have

\[\frac{\mathrm{d}}{\mathrm{d}t}R_{S}(\boldsymbol{\theta}(t)) =-\frac{m}{n^{2}}\boldsymbol{e}^{\intercal}\left(\boldsymbol{G}^{[ a]}(\boldsymbol{\theta}(t))+\boldsymbol{G}^{[\boldsymbol{w}]}(\boldsymbol{\theta}(t)) \right)\boldsymbol{e}\] \[\leq-\frac{2m}{n}\lambda_{\min}\left(\boldsymbol{G}^{[a]}( \boldsymbol{\theta}(t))\right)R_{S}(\boldsymbol{\theta}(t))\] \[\leq-\frac{m}{n}\nu^{2}\varepsilon^{2}\frac{\varepsilon}{\nu} \lambda R_{S}(\boldsymbol{\theta}(t))\] \[=-\frac{m}{n}\nu\varepsilon^{3}\lambda R_{S}(\boldsymbol{\theta}( t)),\]

thus the following holds

\[R_{S}(\boldsymbol{\theta}(t))\leq\exp\left(-\frac{m\nu\varepsilon^{3}\lambda t }{n}\right)R_{S}(\boldsymbol{\theta}(0)),\]
and (5.9) does not hold anymore. However, we still have

\[\lim_{m\to\infty}\sup_{t\in[0,+\infty)}\frac{\|\boldsymbol{\theta}_{\boldsymbol{w}} (t)-\boldsymbol{\theta}_{\boldsymbol{w}}(0)\|_{2}}{\|\boldsymbol{\theta}_{ \boldsymbol{w}}(0)\|_{2}}=0, \tag{5.11}\]

and it can also be illustrated through an intuitive scaling analysis. Since

\[\frac{\mathrm{d}}{\mathrm{d}t}R_{S}(\boldsymbol{\theta}(t))=-\frac{\varepsilon }{\nu}\left\|\nabla_{\boldsymbol{\theta}_{a}}R_{S}(\boldsymbol{\theta}(t)) \right\|_{2}^{2}-\frac{\nu}{\varepsilon}\left\|\nabla_{\boldsymbol{\theta}_{ \boldsymbol{w}}}R_{S}(\boldsymbol{\theta}(t))\right\|_{2}^{2}\sim-\frac{m}{n} \nu\varepsilon^{3}\lambda R_{S}(\boldsymbol{\theta}(t)),\]

then

\[\left\|\nabla_{\boldsymbol{\theta}_{\boldsymbol{w}}}R_{S}( \boldsymbol{\theta}(t))\right\|_{2} \sim\sqrt{\frac{m}{n}\varepsilon^{4}\lambda}\sqrt{R_{S}( \boldsymbol{\theta}(t))}\] \[\sim\sqrt{\frac{m}{n}\varepsilon^{4}\lambda}\exp\left(-\frac{m \nu\varepsilon^{3}\lambda t}{2n}\right)\sqrt{R_{S}(\boldsymbol{\theta}(0))},\]

hence

\[\|\boldsymbol{\theta}_{\boldsymbol{w}}(t)-\boldsymbol{\theta}_{ \boldsymbol{w}}(0)\|_{2} \leq\frac{\nu}{\varepsilon}\int_{0}^{t}\left\|\nabla_{\boldsymbol{ \theta}_{\boldsymbol{w}}}R_{S}(\boldsymbol{\theta}(s))\right\|_{2}\mathrm{d}s\] \[\leq\frac{\nu}{\varepsilon}\int_{0}^{\infty}\left\|\nabla_{ \boldsymbol{\theta}_{\boldsymbol{w}}}R_{S}(\boldsymbol{\theta}(s))\right\|_{2} \mathrm{d}s\] \[\lesssim\sqrt{\frac{n}{m\varepsilon^{4}\lambda}}\sqrt{R_{S}( \boldsymbol{\theta}(0))},\]

and as

\[\left\|\boldsymbol{\theta}_{\boldsymbol{w}}(0)\right\|_{2}\sim\sqrt{m},\]

then

\[\frac{\|\boldsymbol{\theta}_{\boldsymbol{w}}(t)-\boldsymbol{\theta}_{ \boldsymbol{w}}(0)\|_{2}}{\left\|\boldsymbol{\theta}_{\boldsymbol{w}}(0) \right\|_{2}} \lesssim\sqrt{\frac{n}{m^{2}\varepsilon^{4}\lambda}}\sqrt{R_{S}( \boldsymbol{\theta}(0))} \tag{5.12}\] \[\lesssim\sqrt{\frac{n}{m^{2}\varepsilon^{4}}}\sqrt{R_{S}( \boldsymbol{\theta}(0))}.\]

The rigorous statements of relation (5.11) are given in Theorem 6. To end this part, we provide a sketch of the proofs for Theorem 1, see Figure 5.

### Condensed Regime

We remark that the \(\{a_{k},\boldsymbol{w}_{k}\}_{k=1}^{m}\) dynamics

\[\frac{\mathrm{d}a_{k}}{\mathrm{d}t} =-\frac{\varepsilon}{\nu}\frac{1}{n}\sum_{i=1}^{n}e_{i}\frac{ \sigma(\varepsilon\boldsymbol{w}_{k}^{\intercal}\boldsymbol{x}_{i})}{ \varepsilon}, \tag{5.13}\] \[\frac{\mathrm{d}\boldsymbol{w}_{k}}{\mathrm{d}t} =-\frac{\nu}{\varepsilon}\frac{1}{n}\sum_{i=1}^{n}e_{i}a_{k} \sigma^{(1)}(\varepsilon\boldsymbol{w}_{k}^{\intercal}\boldsymbol{x}_{i}) \boldsymbol{x}_{i},\]
is coupled in the sense that the solution of at least one of the equations in the system depends on knowing one of the other solutions in the system, and a coupled system is usually hard to solve.

However, in the condense regime, as \(\varepsilon\ll 1\) and \(\varepsilon\nu\ll\frac{1}{m}\), the evolution of \(\{e_{i}\}_{i=1}^{n}\) is slow enough so that it remains close to \(\{-y_{i}\}_{i=1}^{n}\) over a period of time \(T>0\) at the initial stage, hence (5.13) approximately reads

\[\frac{\mathrm{d}a_{k}}{\mathrm{d}t} \approx\frac{\varepsilon}{\nu}\frac{1}{n}\sum_{i=1}^{n}y_{i} \boldsymbol{w}_{k}^{\intercal}\boldsymbol{x}_{i}=\frac{\varepsilon}{\nu} \boldsymbol{w}_{k}^{\intercal}\boldsymbol{z}, \tag{5.14}\] \[\frac{\mathrm{d}\boldsymbol{w}_{k}}{\mathrm{d}t} \approx\frac{\nu}{\varepsilon}\frac{1}{n}\sum_{i=1}^{n}y_{i}a_{k} \sigma^{(1)}(0)\boldsymbol{x}_{i}=\frac{\nu}{\varepsilon}a_{k}\boldsymbol{z},\]

Figure 5: Sketch of proof for Theorem 1.
and the coupled dynamics is reduced to linear dynamics.

We are able to solve out the linear dynamics (5.14) (Proposition 7), whose solutions read: For each \(k\in[m]\), under the initial condition \(\left[\nu a_{k}(0),\varepsilon\mathbf{w}_{k}^{\intercal}(0)\right]^{\intercal}= \left[\nu a_{k}^{0},(\varepsilon\mathbf{w}_{k}^{0})^{\intercal}\right]^{\intercal}\), we obtain that

\[\nu a_{k}(t) =\nu\left(\frac{1}{2}\exp(\|\mathbf{z}\|_{2}t)+\frac{1}{2}\exp(-\|\bm {z}\|_{2}t)\right)a_{k}^{0} \tag{5.15}\] \[\quad+\varepsilon\left(\frac{1}{2}\exp(\|\mathbf{z}\|_{2}t)-\frac{1}{ 2}\exp(-\|\mathbf{z}\|_{2}t)\right)\left\langle\mathbf{w}_{k}^{0},\hat{\mathbf{z}}\right\rangle,\] \[\varepsilon\mathbf{w}_{k}(t) =\nu\left(\frac{1}{2}\exp(\|\mathbf{z}\|_{2}t)-\frac{1}{2}\exp(-\| \mathbf{z}\|_{2}t)\right)a_{k}^{0}\hat{\mathbf{z}}\] \[\quad+\varepsilon\left(\frac{1}{2}\exp(\|\mathbf{z}\|_{2}t)+\frac{1}{ 2}\exp(-\|\mathbf{z}\|_{2}t)\right)\left\langle\mathbf{w}_{k}^{0},\hat{\mathbf{z}}\right\rangle \hat{\mathbf{z}}\] \[\quad-\varepsilon\left\langle\mathbf{w}_{k}^{0},\hat{\mathbf{z}}\right\rangle \hat{\mathbf{z}}+\varepsilon\mathbf{w}_{k}^{0}.\]

We remark that \(\{a_{k},\mathbf{w}_{k}\}_{k=1}^{m}\) are the normalized parameters, then \(\{\nu a_{k},\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\) corresponds to the original parameters in (3.4).

In the case where \(\gamma>1\) and \(0\leq\gamma^{\prime}<\gamma-1\) (\(\mathbf{w}\)-lag regime), as \(\varepsilon\gg\nu\), then the magnitude of \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\) is much larger than that of \(\{\nu a_{k}\}_{k=1}^{m}\) at \(t=0\). Based on (5.15), it can be observed that \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\) remains dormant until \(\{\nu a_{k}\}_{k=1}^{m}\) attain a magnitude that is commensurate with that of \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\), and only then do the magnitudes of \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\) and \(\{\nu a_{k}\}_{k=1}^{m}\) experience exponential growth simultaneously. In order for the initial condensation of \(\mathbf{\theta}_{\mathbf{w}}\) to be observed, one has to wait for some growth in the magnitude of \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\), hence \(T\sim\Omega(\log(m))\). More importantly, compared with the \(\mathbf{w}\)-lazy regime, the condition \(\gamma^{\prime}<\gamma-1\) enforces \(\varepsilon\ll\frac{1}{\sqrt{m}}\), thus providing enough room for \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\) to grow in the \(\mathbf{z}\)-direction before \(\{e_{i}\}_{i=1}^{n}\) deviates away from \(\{-y_{i}\}_{i=1}^{m}\).

In the case where \(\gamma>1\) and \(\gamma^{\prime}<0\) (\(a\)-lag regime), as \(\varepsilon\ll\nu\), then the initial magnitudes of \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\) is much smaller than that of \(\{\nu a_{k}\}_{k=1}^{m}\). Based on (5.15), it takes only \(T\sim\Omega(1)\) for \(\{\varepsilon\mathbf{w}_{k}\}_{k=1}^{m}\) to attain a magnitude comparable to \(\{\nu a_{k}\}_{k=1}^{m}\), and this rapid growth leads to the observation of initial condensation towards the \(\mathbf{z}\)-direction, where \(\gamma>1\) impose \(\{e_{i}\}_{i=1}^{n}\) to a small neighbourhood of \(\{-y_{i}\}_{i=1}^{m}\) for a period of time which is at least of order one. To end this part, we provide a sketch of the proofs for Theorem 2, see Figure 6.

## 6 Conclusions

In this paper, we present the phase diagram of initial condensation for two-layer NNs with a wide class of smooth activation functions. We demonstrate the distinct features exhibited by NNs in the linear regime area and condensed regime
area, and we provide a complete and detailed analysis for the transition across the boundary (critical regime) in the phase diagram. Moreover, in comparison with the work of Luo et al. [16], we identify the direction towards which the weight parameters condense, and we draw estimates on the time required for initial condensation to occur in contrast to the work of Zhou et al. [31]. The phase diagram at initial stage is crucial in that it is a valuable tool for understanding the implicit regularization effect provided by weight initialization schemes, and it serves as a cornerstone upon which future works can be done to provide thorough characterization of the dynamical behavior of general NNs at each of the identified regime.

In future, we endeavor to establish a framework for the analysis of initial condensation by a series of papers. In our upcoming publication, we plan to extend this formalism to Convolutional Neural Network (CNN) [20], and apply it to investigate the phenomenon of condensation for a wide range of NN architectures, including fully-connected deep network (DNN) and Residual Network (ResNet) [9].

## Acknowledgments

This work is sponsored by the National Key R&D Program of China Grant No. 2022YFA1008200 (Z. X., T. L.), the Shanghai Sailing Program, the Natural Science Foundation of Shanghai Grant No. 20ZR1429000 (Z. X.), the National Natural

Figure 6: Sketch of proof for Theorem 2.
Science Foundation of China Grant No. 62002221 (Z. X.), the National Natural Science Foundation of China Grant No. 12101401 (T. L.), Shanghai Municipal Science and Technology Key Project No. 22JC1401500 (T. L.), Shanghai Municipal of Science and Technology Major Project No. 2021SHZDZX0102, and the HPC of School of Mathematical Sciences and the Student Innovation Center, and the Siyuan-1 cluster supported by the Center for High Performance Computing at Shanghai Jiao Tong University.
