Understanding the Initial Condensation of Convolutional Neural Networks
Previous research has shown that fully-connected networks with small
initialization and gradient-based training methods exhibit a phenomenon known
as condensation during training. This phenomenon refers to the input weights of
hidden neurons condensing into isolated orientations during training, revealing
an implicit bias towards simple solutions in the parameter space. However, the
impact of neural network structure on condensation has not been investigated
yet. In this study, we focus on the investigation of convolutional neural
networks (CNNs). Our experiments suggest that when subjected to small
initialization and gradient-based training methods, kernel weights within the
same CNN layer also cluster together during training, demonstrating a
significant degree of condensation. Theoretically, we demonstrate that in a
finite training period, kernels of a two-layer CNN with small initialization
will converge to one or a few directions. This work represents a step towards a
better understanding of the non-linear training behavior exhibited by neural
networks with specialized structures.
Zhangchen Zhou, Hanxu Zhou, Yuqing Li, Zhi-Qin John Xu
# Understanding the Initial Condensation of Convolutional Neural Networks

###### Abstract

Previous research has shown that fully-connected networks with small initialization and gradient-based training methods exhibit a phenomenon known as condensation during training. This phenomenon refers to the input weights of hidden neurons condensing into isolated orientations during training, revealing an implicit bias towards simple solutions in the parameter space. However, the impact of neural network structure on condensation has not been investigated yet. In this study, we focus on the investigation of convolutional neural networks (CNNs). Our experiments suggest that when subjected to small initialization and gradient-based training methods, kernel weights within the same CNN layer also cluster together during training, demonstrating a significant degree of condensation. Theoretically, we demonstrate that in a finite training period, kernels of a two-layer CNN with small initialization will converge to one or a few directions. This work represents a step towards a better understanding of the non-linear training behavior exhibited by neural networks with specialized structures.

## 1 Introduction

As large neural networks continue to demonstrate impressive performance in numerous practical tasks, a key challenge has come to understand the reasons behind the strong generalization capabilities often exhibited by over-parameterized networks (Breiman, 1995; Zhang et al., 2021). A commonly employed approach to understanding neural networks is to examine their implicit biases during the training process. Several studies have shown that neural networks tend to favor simple solutions. For instance, from a Fourier perspective, neural networks have a bias toward low-frequency functions, which is known as the frequency principle (Xu et al., 2019, 2020) or spectral bias (Rahaman et al., 2019). In the parameter space, Luo et al. (2021) observed a condensation phenomenon, i.e., the input weights of hidden neurons in two-layer \(\mathrm{ReLU}\) neural networks condense into isolated orientations during training in the non-linear regime, particularly with small initialization. Fig. 1 presents an illustrative example in which a large condensed network can be reduced to an effective smaller network with only two neurons. Based on complexity theory (Bartlett and Mendelson, 2002), as the condensation phenomenon reduces the network complexity, it might provide insights into how over-parameterized neural networks achieve good generalization performance in practice. Zhang and Xu (2022) drew inspiration from this phenomenon and found that dropout (Hinton et al., 2012; Srivastava et al., 2014), a commonly used optimization technique for improving generalization,
exhibits an implicit bias towards condensation through experiments and theory. Prior literature has predominantly centered on the study of fully-connected neural networks, thereby leaving the emergence and properties of the condensation phenomenon in neural networks with different structural characteristics inadequately understood. Consequently, this paper aims to investigate the occurrence of condensation in convolutional neural networks (CNNs).

The success of deep learning relies heavily on the structures used, such as convolution and attention. Convolution is an ideal starting point for investigating the impact of structure on learning outcomes, as it is widely used and has simple structure. To achieve a clear condensation phenomenon in CNNs, we adopt a strategy of initializing weights with small values. Small weight initialization can result in rich non-linearity of neural network (NN) training behavior (Mei et al., 2019; Rotskoff and Vanden-Eijnden, 2018; Chizat and Bach, 2018; Sirignano and Spiliopoulos, 2020). Over-parameterized NNs with small initialization can, for instance, achieve low generalization error (Advani et al., 2020) and converge to a solution with maximum margin (Phuong and Lampert, 2020). In contrast to the condensation in fully-connected networks, each kernel in CNNs is considered as a unit, and condensation is referred to the behavior, that a set of kernels in the same layer evolves towards the same direction.

Understanding the initial condensation can benefit understanding subsequent training stages (Fort et al., 2020; Hu et al., 2020; Luo et al., 2021; Jiang et al., 2019; Li et al., 2018). Previous research has shown how neural networks with small initialization can condense during the initial training stage in fully-connected networks (Maennel et al., 2018; Pellegrini and Biroli, 2020; Zhou et al., 2022; Chen et al., 2023). This work aims to demonstrate the initial condensation in CNNs during training. A major advantage of studying the initial training stage of neural networks with small initialization is that the network can be approximated accurately by the leading-order Taylor expansion at zero weights. Further, the structure of CNNs may cause kernels in the same layer to exhibit similar dynamics. Through theoretical proof, we show that CNNs can condense into one or a few directions within a finite training period with small initialization. This initial condensation serves an important role in resetting the neural network of different initializations to a similar and simple state, thus reducing the sensitivity of initialization and facilitating the tuning of hyper-parameters of initialization.

## 2 Related works

For fully-connected neural networks, it has been generally studied that different initializations can lead to very different training behavior regimes (Luo et al., 2021), including linear regime (similar to the lazy regime) (Jacot et al., 2018; Arora et al., 2019; Zhang et al., 2020; E et al., 2020; Chizat and Bach, 2019), critical regime (Mei et al., 2019; Rotskoff and Vanden-Eijnden, 2018; Chizat and Bach, 2018; Sirignano and Spiliopoulos, 2020) and condensed regime (non-linear regime). The relative change of

Figure 1: Illustration of condensation. The color and its intensity of a line indicate the strength of the weight. Initially, weights are random. Soon after training, the weights from an input node to all hidden neurons are clustered into two groups, i.e., condensation. Multiple hidden neurons can be replaced by an effective neuron with low complexity, which has the same input weight as original hidden neurons and the same output weight as the summation of all output weights of original hidden neurons.
input weights as the width approaches infinity is a critical parameter that distinguishes the different regimes, namely \(0\), \(O(1)\), and \(+\infty\). Zhou et al. (2022) demonstrated that these regimes also exist for three-layer \(\mathrm{ReLU}\) neural networks with infinite width. Experiments suggest that condensation is a frequent occurrence in the non-linear regime.

Zhang et al. (2021, 2021)discovered an embedding principle in loss landscapes between narrow and wide neural networks, based on condensation. This principle suggests that the loss landscape of a deep neural network (DNN) includes all critical points of narrower DNNs, which is also studied in Fukumizu and Amari (2000); Fukumizu et al. (2019); Simsek et al. (2021). The embedding structure indicates the existence of global minima where condensation can occur. Zhang et al. (2022) has demonstrated that NNs exhibiting condensation can achieve the desired function with a substantially lower number of samples compared to the number of parameters. However, these studies fail to demonstrate the training process's role in causing condensation.

CNN is one of the fundamental structures in deep learning (Gu et al., 2018). He et al. (2016) introduced the use of residual connections for training deep CNNs, which has greatly enhanced the performance of CNNs on complex practical tasks. Recently, there are also many theoretical advances. Zhou (2020) shows that CNN can be used to approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough. Arora et al. (2019) exactly compute the neural tangent kernel of CNN. Provided that the signal-to-noise ratio satisfies certain conditions, Cao et al. (2022) have demonstrated that a two-layer CNN, trained through gradient descent, can obtain negligible training and test losses. In this work, we focus on the training process of CNNs.

## 3 Preliminaries

### Some Notations

For a matrix \(\mathbf{A}\), we use \(\mathbf{A}_{i,j}\) to denote its \((i,j)\)-th entry. We also use \(\mathbf{A}_{i,:}\) to denote the \(i\)-th row vector of \(\mathbf{A}\) and define \(\mathbf{A}_{i,j:k}:=(\mathbf{A}_{i,j},\mathbf{A}_{i,j+1},\cdots,\mathbf{A}_{i,k})\) as part of the vector. Similarly \(\mathbf{A}_{:,k}\) is the \(k\)-th column vector and \(\mathbf{A}_{i:j,k}:=(\mathbf{A}_{i,k},\mathbf{A}_{i+1,k},\cdots,\mathbf{A}_{j,k})^{\mathsf{T}}\) is part of the \(k\)-th column vector.

We let \([n]=\{1,2,\ldots,n\}\). We set \(\mathcal{N}(\boldsymbol{\mu},\Sigma)\) as the normal distribution with mean \(\boldsymbol{\mu}\) and covariance \(\Sigma\). We set a special vector \(\mathbb{1}:=(1,1,1,\ldots,1)^{\mathsf{T}}\), whose dimension varies. For a vector \(\mathbf{v}\), we use \(\|\mathbf{v}\|_{2}\) to denote its Euclidean norm, and we use \(\langle\cdot,\cdot\rangle\) to denote the standard inner product between two vectors. Finally, for a matrix \(\mathbf{A}\), we use \(\|\mathbf{A}\|_{2\to 2}\) to denote its operator norm.

### Problem Setup

We focus on the empirical risk minimization problem given by the quadratic loss:

\[\min_{\boldsymbol{\theta}}R_{S}(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=1}^{ n}\left(f(\boldsymbol{x}_{i},\boldsymbol{\theta})-y_{i}\right)^{2}. \tag{1}\]

In the above, \(n\) is the total number of training samples, \(\{\boldsymbol{x}_{i}\}_{i=1}^{n}\) are the training inputs, \(\{y_{i}\}_{i=1}^{n}\) are the labels, \(f(\boldsymbol{x}_{i},\boldsymbol{\theta})\) is the prediction function, and \(\boldsymbol{\theta}\) are the parameters to be optimized, which is modeled by a \((L+1)\)-layer CNN with filter size \(m\times m\). We denote \(\boldsymbol{x}^{[l]}(i)\) as the output of the \(l\)-th layer with respect to the \(i\)-th sample for \(l\geq 1\), and \(\boldsymbol{x}^{[0]}(i):=\boldsymbol{x}_{i}\) is the \(i\)-th training input. For any \(l\in[0:L]\), we denote the size of width, height, channel of \(\boldsymbol{x}^{[l]}\) as \(W_{l},\ H_{l}\), and \(C_{l}\), respectively, i.e., \(\{\boldsymbol{x}^{[l]}(i)\}_{i=1}^{n}\subset\mathbb{R}^{W_{l}\times H_{l} \times C_{l}}\). We introduce a filter operator \(\chi(\cdot,\cdot)\), which maps the width and height indices of the output of all layers to a binary variable, i.e., for a filter of size \(m\times m\), the filter operator reads

\[\chi(p,q)=\left\{\begin{array}{ll}1,&\text{for}\ \ 0\leqslant p,q\leqslant m-1\\ 0,&\text{otherwise,}\end{array}\right. \tag{2}\]
then the \((L+1)\)-layer CNN with filter size \(m\times m\) is recursively defined for \(l\in[2:L]\),

\[\mathbf{x}_{u,v,\beta}^{[1]} :=\left[\sum_{\alpha=1}^{C_{0}}\left(\sum_{p=-\infty}^{+\infty} \sum_{q=-\infty}^{+\infty}\mathbf{x}_{u+p,v+q,\alpha}^{[0]}\cdot\mathbf{W}_{p,q,\alpha, \beta}^{[1]}\cdot\chi(p,q)\right)\right]+\mathbf{b}_{\beta}^{[1]},\] \[\mathbf{x}_{u,v,\beta}^{[l]} :=\left[\sum_{\alpha=1}^{C_{L-1}}\left(\sum_{p=-\infty}^{+\infty} \sum_{q=-\infty}^{\infty}\sigma\left(\mathbf{x}_{u+p,v+q,\alpha}^{[l-1]}\cdot\mathbf{W }_{p,q,\alpha,\beta}^{[l]}\cdot\chi(p,q)\right)\right)+\mathbf{b}_{\beta}^{[l]},\] \[f(\mathbf{x},\mathbf{\theta}) :=f_{\mathrm{CNN}}(\mathbf{x},\mathbf{\theta}):=\left\langle\mathbf{a}, \sigma\left(\mathbf{x}^{[L]}\right)\right\rangle=\sum_{\beta=1}^{C_{L}}\sum_{u=1} ^{W_{L}}\sum_{v=1}^{H_{L}}\mathbf{a}_{u,v,\beta}\cdot\sigma\left(\mathbf{x}_{u,v, \beta}^{[L]}\right),\]

where \(\sigma(\cdot)\) is the activation function applied coordinate-wisely to its input, and for each layer \(l\in[L]\), all parameters belonging to this layer are initialized by: For \(p,q\in[m-1]\), \(\alpha\in[C_{l-1}]\) and \(\beta\in[C_{l}]\),

\[\mathbf{W}_{p,q,\alpha,\beta}^{[l]}\sim\mathcal{N}(0,\beta_{1}^{2}),\quad\mathbf{b}_{ \beta}^{[l]}\sim\mathcal{N}(0,\beta_{1}^{2}). \tag{3}\]

Note that for a pair of \(\alpha\) and \(\beta\), \(\mathbf{W}_{\cdot,\cdot,\alpha,\beta}^{[l]}\) is called a kernel. Moreover, for \(u\in[W_{L}]\) and \(v\in[H_{L}]\),

\[\mathbf{a}_{u,v,\beta}\sim\mathcal{N}(0,\beta_{2}^{2}), \tag{4}\]

and for convenience in theory, we set \(\beta_{1}=\beta_{2}=\varepsilon,\) where \(\varepsilon>0\) is the scaling parameter.

**Cosine similarity:** The cosine similarity between two vectors \(\mathbf{u}_{1}\) and \(\mathbf{u}_{2}\) is defined as

\[D(\mathbf{u}_{1},\mathbf{u}_{2})=\frac{\mathbf{u}_{1}^{\intercal}\mathbf{u}_{2}}{(\mathbf{u}_{1}^ {\intercal}\mathbf{u}_{1})^{1/2}(\mathbf{u}_{2}^{\intercal}\mathbf{u}_{2})^{1/2}}. \tag{5}\]

We remark that in order to compute the cosine similarity between two kernels, each kernel \(\mathbf{W}_{\cdot,\cdot,\alpha,\beta}^{[l]}\) shall be vectorized.

In the theoretical part, as we consider two-layer CNNs (\(L=1\)), thus the upper case \([l]\) can be omitted since the number of weight vectors is equal to \(1\), i.e., \(\mathbf{W}_{\cdot,\cdot,\alpha,\beta}:=\mathbf{W}_{\cdot,\cdot,\alpha,\beta}^{[1]}\). We denote \(M:=C_{1}\), the number of channels in \(\mathbf{x}^{[1]}(i)\), which can be heuristically understood as the 'width' of the hidden layer in the case of two-layer neural networks (NNs).

In experiments, the parameters are trained by either Adam or gradient descent (GD), while in theory, only GD is used. In the following, we demonstrate that kernels in multi-layer CNNs exhibit clear condensation in experiments during the initial training stage, with small initialization. We then provide theoretical evidence for two-layer CNNs.

## 4 Condensation of the convolution kernels in experiments

In this section, we will show condensation of convolution kernels in image datasets using different activation functions.

### Experimental setup

For the CIFAR10 dataset: \(500\) samples are randomly selected from CIFAR10 dataset for training. We use the CNN with the structure: \(n\times 32C\)-\((1024)\)-\(d\). The output dimension \(d=10\) or \(1\) is used to classify the pictures or for regression respectively. The parameters of the convolution layer is initialized by the \(\mathcal{N}(0,\sigma_{1}^{2})\), and the parameters of the linear layer is \(\mathcal{N}(0,\sigma_{2}^{2})\). \(\sigma_{1}\) is given by \((\frac{(c_{in}+c_{out})*m^{2}}{2})^{-\gamma}\) where \(c_{in}\) and \(c_{out}\) are the number of in channels and out channels respectively, \(\sigma_{2}\) is given empirically by \(0.0001\). The training method is GD or Adam with full batch. The training loss of each experiment is shown in Fig.15 in Appendix.

### CIFAR10 examples

We first show that when initialized with small weights, the convolutional kernels of a \(\tanh(x)\) CNN undergo condensation during the training process. As shown in Fig. 2, we train a CNN with three
layers by cross-entropy loss until the training accuracy reaches \(100\%\) (the accuracy during training is shown in Fig. 8 in appendix). In each layer, we compute the cosine similarity between each pair of kernels. This reveals a clear condensation phenomenon after training.

Understanding the mechanism of condensation phenomenon is challenging. To this end, we start by studying the initial training stage. We then study the initial condensation in CNNs.

The initial stage (accuracy less than \(20\%\)) of Fig. 2 is shown in Fig. 3. For each layer, nearly all kernels are condensed into two opposite directions in Fig. 3. In layer one, there are actually two pairs of opposite directions. This subtle phenomenon can be explained by our theory.

We further examine the different activation functions. For illustration, we consider two-layer CNNs with activations \(\mathrm{ReLU}(x)\), \(\mathrm{Sigmoid}(x)\), or \(\tanh(x)\). As shown in Fig. 4, we can still see a very clear condensation phenomenon. Note that, as the learning rate is rather small to see the detailed training process, the epoch selected for studying the initial stage may appear large.

In our theoretical study, we consider the MSE loss. Therefore, we also show an example with MSE loss (softmax is also attached with the output layer) for activation \(\mathrm{tanh(x)}\) in Fig. 5. Similarly, condensation is clearly observed. An MSE example without softmax is shown in 9 in appendix.

These examples demonstrate that condensation occurs when using the Adam method to train networks. We can also observe the condensation with the gradient descent (GD) method. Fig. 6 illustrates that with GD, the kernel weights of a two-layer CNN still undergo condensation at two opposite directions during training. Moreover, we observe that the direction of the condensation is \(\boldsymbol{v}=1\). Fig. 6(b) shows that the cosine similarity between each kernel weight and \(1\) is almost equal to 1 or -1.

Similar results of MNIST dataset are also shown in Figs. 10 for CNNs with different activations, Fig. 11 for multi-layer \(\mathrm{tanh}\) CNN in appendix. Also, two-layer CNNs with \(32\) and \(320\) kernels trained by GD are shown in Fig.12 and Fig.13, respectively, in appendix.

Figure 3: Initial stage of Fig. 2, selected at epoch \(=300\) with accuracy less than \(20\%\).

Figure 2: Final condensation of a CNN with three convolution layers. The kernel size is \(m=5\). The colors in the figure show the cosine similarity of weight vectors of each convolution kernel. The activation for all convolution layers are \(\mathrm{tanh(x)}\). The number of the steps is all at the end of the training. The convolution kernels are initialized by \(\gamma=2\). The learning rate is \(2\times 10^{-6}\). We use cross-entropy (with softmax) as the criterion. The optimizer is full batch Adam on CIFAR10 dataset.
Figure 4: Condensation of CNNs with different activations (indicated by sub-captions) for convolutional layers. The network has 32 kernels in the convolution layer, followed by a ReLU fully-connected layer with 1024 neurons and 10-dimensional output with softmax. The kernel size is \(m=5\). The learning rate for all three experiments is \(5\times 10^{-7}\). epoch\(=1000\), epoch\(=5000\) and epoch\(=300\). The convolution layer is initialized by \(\gamma=2\). We use full batch Adam with cross-entropy on CIFAR10 dataset.

Figure 5: Condensation of \(\tanh(\mathrm{x})\) CNN with three convolution layer by MSE loss. The kernel size is \(m=5\). The color indicates the cosine similarity between kernels. The number of the steps are at epoch \(=200\), epoch \(=300\) and epoch \(=300\). The convolution kernels are initialized by \(\gamma=1.2\). The learning rate is \(1\times 10^{-6}\). We use one hot vector as label and use softmax. The optimizer is full-batch Adam on CIFAR10 dataset.

Figure 6: Condensation of two-layer CNN by GD and MSE training on CIFAR10 dataset with data size \(n=500\). (a) cosine similarity. (b) left ordinate (red): the amplitude of each kernel; (b) right ordinate (blue): cosine similarity between kernel weight and \(\mathbb{1}\). The activation function of the convolution part is \(\tanh(\mathrm{x})\). The kernel size is \(m=3\). The learning rate is \(5\times 10^{-6}\). The number of the selected steps is epoch \(=4500\). The convolution layer is initialized by \(\gamma=4\).
Theory

In the following section, we aim to provide theoretical evidence that kernels in the same layer tend to converge to one or a few directions when initialized with small values. In this part, we shall impose some technical conditions on the activation function and input samples. We start with a technical condition (Zhou et al., 2022, Definition 1) on the activation function \(\sigma(\cdot)\).

**Definition 1** (Multiplicity \(r\)).: \(\sigma(\cdot):\mathbb{R}\to\mathbb{R}\) _has multiplicity \(r\) if there exists an integer \(r\geq 1\), such that for all \(0\leq s\leq r-1\), the \(s\)-th order derivative satisfies \(\sigma^{(s)}(0)=0\), and \(\sigma^{(r)}(0)\neq 0\)._

We list out some examples of activation functions with different multiplicities.

**Remark 1**.:
* \(\tanh(\mathrm{x}):=\frac{\exp(\mathrm{x})-\exp(-\mathrm{x})}{\exp(\mathrm{x}) +\exp(-\mathrm{x})}\) _is with multiplicity_ \(r=1\)_;_
* \(\mathrm{SiLU(\mathrm{x})}:=\frac{\mathrm{x}}{1+\exp(-\mathrm{x})}\) _is with multiplicity_ \(r=1\)_;_
* \(\mathrm{xtanh(\mathrm{x})}:=\frac{\mathrm{x}\exp(\mathrm{x})-\mathrm{x}\exp (-\mathrm{x})}{\exp(\mathrm{x})+\exp(-\mathrm{x})}\) _is with multiplicity_ \(r=2\)_._

**Assumption 1** (Multiplicity \(1\)).: _The activation function \(\sigma\in\mathcal{C}^{2}(\mathbb{R})\), and there exists a universal constant \(C_{L}>0\), such that its first and second derivatives satisfy_

\[\left\|\sigma^{(1)}(\cdot)\right\|_{\infty}\leq C_{L},\quad\left\|\sigma^{(2) }(\cdot)\right\|_{\infty}\leq C_{L}. \tag{6}\]

_Moreover,_

\[\sigma(0)=0,\quad\sigma^{(1)}(0)=1. \tag{7}\]

**Remark 2**.: _We remark that \(\sigma\) has multiplicity \(1\). \(\sigma^{(1)}(0)=1\) can be replaced by \(\sigma^{(1)}(0)\neq 0\), and we set \(\sigma^{(1)}(0)=1\) for simplicity, which can be easily satisfied by replacing the original activation \(\sigma(\cdot)\) with \(\frac{\sigma(\cdot)}{\sigma^{(1)}(0)}\)._

**Assumption 2**.: _The training inputs \(\{\mathbf{x}_{i}\}_{i=1}^{n}\) and labels \(\{y_{i}\}_{i=1}^{n}\) satisfy that there exists a universal constant \(c>0\), such that given any \(i\in[n]\), then for each \(u\in[W_{0}]\), \(v\in[H_{0}]\) and \(\alpha\in[C_{0}]\), the following holds_

\[\frac{1}{c}\leq\left|\mathbf{x}_{u,v,\alpha}(i)\right|,\quad|y_{i}|\leq c.\]

We assume further that

**Assumption 3**.: _The following limit exists_

\[\gamma:=\lim_{M\to\infty}-\frac{\log\varepsilon^{2}}{\log M}. \tag{8}\]

As \(M\to\infty\), then \(\epsilon\to 0\) and since multiplicity \(r=1\), we can use the first-order Taylor expansion to approximate the network output and obtain a linear dynamics

\[\frac{\mathrm{d}\mathbf{\theta}_{\beta}}{\mathrm{d}t}=\mathbf{A}\mathbf{\theta}_{\beta}, \tag{9}\]

where

\[\mathbf{\theta}_{\beta}:=\Big{(} \mathbf{W}_{0,0,1,\beta},\mathbf{W}_{0,1,1,\beta},\cdots,\mathbf{W}_{0,m-1,1, \beta};\mathbf{W}_{1,0,1,\beta},\cdots,\mathbf{W}_{1,m-1,1,\beta};\cdots\cdots\mathbf{W}_ {m-1,m-1,1,\beta};\] \[\mathbf{W}_{0,0,2,\beta},\mathbf{W}_{0,1,2,\beta},\cdots,\mathbf{W}_{0,m-1,2, \beta};\mathbf{W}_{1,0,2,\beta},\cdots,\mathbf{W}_{1,m-1,2,\beta};\cdots\cdots\mathbf{W}_ {m-1,m-1,2,\beta};\] \[\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots \cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\cdots\] \[\mathbf{W}_{0,0,C_{0},\beta},\mathbf{W}_{0,1,C_{0},\beta},\cdots,\mathbf{W}_ {0,m-1,C_{0},\beta};\cdots,\mathbf{W}_{1,m-1,C_{0},\beta};\cdots\cdots\mathbf{W}_{m-1, m-1,C_{0},\beta};\mathbf{b}_{\beta};\] \[\mathbf{a}_{1,1,\beta},\mathbf{a}_{1,2,\beta},\cdots,\mathbf{a}_{1,H_{1},\beta };\mathbf{a}_{2,1,\beta},\cdots,\mathbf{a}_{2,H_{1},\beta};\cdots\cdots\mathbf{a}_{W_{1},H_ {1},\beta})^{\mathsf{T}},\]
\[\mathbf{A}:=\left[\begin{array}{cc}\mathbf{0}_{(C_{0}m^{2}+1)\times(C_{0}m^{2}+1)}&\mathbf{Z }^{\intercal}\\ \mathbf{Z}&\mathbf{0}_{W_{1}H_{1}\times W_{1}H_{1}}\end{array}\right], \tag{10}\]

where \(\mathbf{Z}\) is detailed described by multi channel (70) and single channel (35) in appendix. We perform singular value decomposition (SVD) on \(\mathbf{Z}\), i.e.,

\[\mathbf{Z}=\mathbf{U}\Lambda\mathbf{V}^{\intercal}, \tag{11}\]

where

\[\mathbf{U}=\left[\mathbf{u}_{1},\mathbf{u}_{2},\cdots,\mathbf{u}_{W_{1}H_{1}}\right],\quad\mathbf{ V}=\left[\mathbf{v}_{1},\mathbf{v}_{2},\cdots,\mathbf{v}_{m^{2}+1}\right],\]

and as we denote \(r:=\mathrm{rank}(\mathbf{Z})\), naturally, \(r\leq\min\{W_{1}H_{1},m^{2}+1\}\), we have \(r\) singular values,

\[\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{r}>0,\]

and WLOG, we assume that

**Assumption 4** (Spectral Gap of \(\mathbf{Z}\)).: _The singular values \(\{\lambda_{k}\}_{k=1}^{r}\) of \(\mathbf{Z}\) satisfy that_

\[\lambda_{1}>\lambda_{2}\geq\cdots\geq\lambda_{r}>0, \tag{12}\]

_and we denote the spectral gap between \(\lambda_{1}\) and \(\lambda_{2}\) by_

\[\Delta\lambda:=\lambda_{1}-\lambda_{2}.\]

We denote that

\[\mathbf{\theta}_{\mathbf{W},\beta} :=\left(\mathbf{W}_{0,0,\beta},\mathbf{W}_{0,1,\beta},\cdots,\mathbf{W}_{0,m- 1,\beta};\mathbf{W}_{1,0,\beta},\cdots,\mathbf{W}_{1,m-1,\beta};\cdots\cdots\mathbf{W}_{m -1,m-1,\beta};\mathbf{b}_{\beta}\right)^{\intercal},\] \[\mathbf{\theta}_{\mathbf{a},\beta} :=\left(\mathbf{a}_{1,1,\beta},\mathbf{a}_{1,2,\beta},\cdots,\mathbf{a}_{1,H_ {1},\beta};\mathbf{a}_{2,1,\beta},\cdots,\mathbf{a}_{2,H_{1},\beta};\cdots\cdots\mathbf{a }_{W_{1},H_{1},\beta}\right)^{\intercal},\]

hence

\[\mathbf{\theta}_{\beta}=\left(\mathbf{\theta}_{\mathbf{W},\beta}^{\intercal},\mathbf{\theta} _{\mathbf{a},\beta}^{\intercal}\right)^{\intercal}.\]

In order to study the phenomenon of condensation, we concatenate the vectors \(\left\{\mathbf{\theta}_{\mathbf{W},\beta}\right\}_{\beta=1}^{M}\) into

\[\mathbf{\theta}_{\mathbf{W}}:=\mathrm{vec}\left(\left\{\mathbf{\theta}_{\mathbf{W},\beta} \right\}_{\beta=1}^{M}\right),\]

and we denote further that

\[\mathbf{\theta}_{\mathbf{W},\mathbf{v}_{1}}:=\mathcal{P}_{1}\mathbf{\theta}_{\mathbf{W}}:=\left( \left\langle\mathbf{\theta}_{\mathbf{W},1},\mathbf{v}_{1}\right\rangle,\left\langle\mathbf{ \theta}_{\mathbf{W},2},\mathbf{v}_{1}\right\rangle,\cdots\left\langle\mathbf{\theta}_{ \mathbf{W},M},\mathbf{v}_{1}\right\rangle\right)^{\intercal},\]

where \(\mathbf{v}_{1}\) is the eigenvector of the largest eigenvalue of \(\mathbf{Z}^{\intercal}\mathbf{Z}\), or the first column vector of \(\mathbf{V}\) in (36). In appendix, we prove that for any \(\eta_{0}>\frac{\gamma-1}{100}>0\), there exists

\[T_{\mathrm{eff}}>\frac{1}{\lambda_{1}}\left[\log\left(\frac{1}{4}\right)+ \left(\frac{\gamma-1}{4}-\eta_{0}\right)\log(M)\right], \tag{13}\]

leading to the following theorem.

**Theorem 1**.: _Given any \(\delta\in(0,1)\), under Assumption 1, Assumption 2, Assumption 3 and Assumption 4, if \(\gamma>1\), then with probability at least \(1-\delta\) over the choice of \(\mathbf{\theta}^{0}\), we have_

\[\lim_{M\rightarrow+\infty}\sup_{t\in[0,T_{\mathrm{eff}}]}\frac{\left\|\mathbf{ \theta}_{\mathbf{W}}(t)-\mathbf{\theta}_{\mathbf{W}}(0)\right\|_{2}}{\left\|\mathbf{\theta}_{ \mathbf{W}}(0)\right\|_{2}}=+\infty, \tag{14}\]

_and_

\[\lim_{M\rightarrow+\infty}\sup_{t\in[0,T_{\mathrm{eff}}]}\frac{\left\|\mathbf{ \theta}_{\mathbf{W},\mathbf{v}_{1}}(t)\right\|_{2}}{\left\|\mathbf{\theta}_{\mathbf{W}}(t) \right\|_{2}}=1. \tag{15}\]

The theorem shows that as the number of channels increases, two implications arise within a finite training time in the small initialization scheme. Firstly, the relative change of kernel weight becomes considerably large, demonstrating significant non-linear behavior. Secondly, the kernel weight concentrates on one direction.

The mechanism of the concentration effect, or the phenomenon of initial condensation, arises from the spectral gap in Assumption 4. The original definition of spectral gap is a fundamental concept
in the analysis of Markov chains, which refers to the minimum difference between the two largest eigenvalues of the transition matrix of a Markov chain, and is closely related to the convergence rate of the chain to its stationary distribution Komorowski et al. (2012). A large spectral gap indicates that the Markov chain converges quickly to its stationary distribution, while a small spectral gap suggests a slower convergence rate. Similar to that, one may observe from relation (64) that the larger the spectral gap is, the faster the kernel weight concentrates onto the direction of \(\mathbf{v}_{1}\). Fig. 7 shows an example of eigenvalues of \(\mathbf{Z}^{\intercal}\mathbf{Z}\), in which a huge spectral gap can be observed. We then study the eigenvector of the maximum eigenvalue. In this example, since the input has 3 channels, we decompose the eigenvector into corresponding three parts and find the inner product of each part with \(1\) is close to 1, i.e., \(0.9891\pm 0.0349\), \(0.9982\pm 0.0009\), and \(0.9992\pm 0.0003\), respectively, computed from 50 independent trials, where in each trial we randomly selected 500 images. The spectral gap for MNIST is also displayed in Fig. 14 and the inner product between the eigenvector of the maximum eigenvalue and \(1\) is shown in the appendix. Also note that the second eigen direction can sometimes be observed, such as the case in Fig. 3(a).

## 6 Conclusion

In this work, our experiments, supported by theoretical analysis, demonstrate the phenomenon of condensation in CNNs during training with small initialization. Specifically, we observe that the kernels in the same layer evolve towards one or few directions. This initial condensation of kernels resets the neural network to a simpler state with few distinct kernels. This process plays a crucial role in the subsequent learning process of the neural network.

## 7 Acknowledgements

This work is sponsored by the National Key R&D Program of China Grant No. 2022YFA1008200, the National Natural Science Foundation of China Grant No. 62002221, Shanghai Municipal of Science and Technology Major Project No. 2021SHZDZX0102, and the HPC of School of Mathematical Sciences and the Student Innovation Center, and the Siyuan-1 cluster supported by the Center for High Performance Computing at Shanghai Jiao Tong University.
