Local Kernel Renormalization as a mechanism for feature learning in
  overparametrized Convolutional Neural Networks
Feature learning, or the ability of deep neural networks to automatically
learn relevant features from raw data, underlies their exceptional capability
to solve complex tasks. However, feature learning seems to be realized in
different ways in fully-connected (FC) or convolutional architectures (CNNs).
Empirical evidence shows that FC neural networks in the infinite-width limit
eventually outperform their finite-width counterparts. Since the kernel that
describes infinite-width networks does not evolve during training, whatever
form of feature learning occurs in deep FC architectures is not very helpful in
improving generalization. On the other hand, state-of-the-art architectures
with convolutional layers achieve optimal performances in the finite-width
regime, suggesting that an effective form of feature learning emerges in this
case. In this work, we present a simple theoretical framework that provides a
rationale for these differences, in one hidden layer networks. First, we show
that the generalization performance of a finite-width FC network can be
obtained by an infinite-width network, with a suitable choice of the Gaussian
priors. Second, we derive a finite-width effective action for an architecture
with one convolutional hidden layer and compare it with the result available
for FC networks. Remarkably, we identify a completely different form of kernel
renormalization: whereas the kernel of the FC architecture is just globally
renormalized by a single scalar parameter, the CNN kernel undergoes a local
renormalization, meaning that the network can select the local components that
will contribute to the final prediction in a data-dependent way. This finding
highlights a simple mechanism for feature learning that can take place in
overparametrized shallow CNNs, but not in shallow FC architectures or in
locally connected neural networks without weight sharing.
R. Aiudi, R. Pacelli, A. Vezzani, R. Burioni, P. Rotondo
Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks

###### Abstract

Feature learning, or the ability of deep neural networks to automatically learn relevant features from raw data, underlies their exceptional capability to solve complex tasks. However, feature learning seems to be realized in different ways in fully-connected (FC) or convolutional architectures (CNNs). Empirical evidence shows that FC neural networks in the infinite-width limit eventually outperform their finite-width counterparts. Since the kernel that describes infinite-width networks does not evolve during training, whatever form of feature learning occurs in deep FC architectures is not very helpful in improving generalization. On the other hand, state-of-the-art architectures with convolutional layers achieve optimal performances in the finite-width regime, suggesting that an effective form of feature learning emerges in this case. In this work, we present a simple theoretical framework that provides a rationale for these differences, in one hidden layer networks. First, we show that the generalization performance of a finite-width FC network can be obtained by an infinite-width network, with a suitable choice of the Gaussian priors (technically, this result holds in the asymptotic limit where the size of the training set \(P\) and the size of the hidden layer \(N_{1}\) are taken to infinity keeping their ratio \(\alpha_{1}=P/N_{1}\) fixed). Second, we derive a finite-width effective action for an architecture with one convolutional hidden layer and compare it with the result available for FC networks. Remarkably, we identify a completely different form of kernel renormalization: whereas the kernel of the FC architecture is just globally renormalized by a single scalar parameter, the CNN kernel undergoes a local renormalization, meaning that the network can select the local components that will contribute to the final prediction in a data-dependent way. This finding highlights a simple mechanism for feature learning that can take place in overparametrized shallow CNNs, but not in shallow FC architectures or in locally connected neural networks without weight sharing.

## I Introduction

Deep learning achieves state-of-the-art performances on a variety of pattern recognition tasks, ranging from computer vision to natural language processing [1]. A minimal theory of deep learning should be able (at least) to predict practically relevant scores, such as the training and test accuracy, from knowledge of the training data. Another even more fundamental aspect that such a theory should quantitatively address is the so-called _feature learning regime_[2; 3], i.e. the capability of deep networks to automatically detect useful representations from raw data.

Concerning the first aspect, fundamental theoretical progress has been achieved in the so-called _infinite-width limit_[4; 5; 6; 7; 8; 9; 10; 11; 12], which is informally defined as the regime where the size of each hidden layer \(N_{\ell}\) (\(\ell=1,\ldots,L\), \(L\) being the (finite) depth of the network) is much larger than the size of the training set \(P\). Here, one shows that the stochastic process that describes information flow in the deep neural network is a familiar Gaussian process, which is completely determined by a non-linear kernel.

A fundamental consequence of this finding is that learning in the infinite-width limit is equivalent to kernel learning [13; 14; 15; 16] with a static kernel, completely fixed by the statistics of the weights at initialization, that does not evolve during training. This last observation suggests that feature learning is essentially absent in infinite-width networks [17]. Notably, whereas original work on the infinite-width limit considered deep architectures with FC hidden layers only, the generalization to architectures with convolutional layers is straightforward [9], if one replaces neurons in the FC hidden layers with the channels/convolutional filters of the CNN.

Recently, the authors of Ref. [18] conducted a large-scale empirical study comparing finite-width FC networks, CNNs with finite number of channels and their infinite-width limit kernel counterparts, where they explore a plenitude of possible settings to improve generalization. This analysis prompts a few striking empirical observations: (i) infinite-width kernels systematically outperform their finite-width counterpart in the case of FC deep neural networks; (ii) CNNs with finite number of channels often outperform their corresponding infinite-width kernel performance.

Several similar observations have been reported in the literature: the authors of Ref. [19] very recently pointed out that infinite-width deep FC neural networks eventually outperform their finite-width counterpart as the size of the training set grows. In one of the
-seminal papers dealing with the infinite-width limit [7], the authors observe that increasing the hidden layers size leads to optimal test accuracy on deep FC architectures trained on MNIST and on CIFAR10, two of the benchmark datasets for computer vision learning problems. In Ref. [9] this observation is extended to _locally connected networks_ without weight sharing (LCNs). The interplay between the lazy training regime and the mean field limit [20] has been the subject of a thorough investigation in [21, 22].

These observations suggest that CNNs leverage a better feature-learning mechanism at finite width than FCNs and LCNs, and prompt at least two conceptual questions: (i) why is it ultimately convenient to employ large-width architectures when only FC layers are available? (ii) Why is this not the case when convolutional layers are employed? And how does a CNN operatively exploit the finite-width regime for efficient feature learning? Preliminary theoretical work in the direction of understanding the feature learning regime of deep nets was carried out in Refs. [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34]. The authors of Refs. [35, 36] analytically investigated the advantages of employing convolutional neural tangent kernels in the infinite-width limit and understood why infinite-width FCNs perform worse in the mean field regime than in the lazy-training one [37].

In this work, we try to rationalize the aforementioned empirical observations through the lens of _kernel renormalization_. This represents a physical consequence of a recently-derived effective action for Bayesian learning in finite-width deep neural networks [38] (formally speaking, the finite-width regime is defined as the thermodynamic limit where the size of the FC hidden layers and of the training set are taken to infinity, \(N_{\ell},P\rightarrow\infty\) and their ratio \(\alpha_{\ell}=P/N_{\ell}\) is kept finite). First, we show that the theoretical framework of Ref. [38] provides a simple, yet very instructive answer to question (i) for one hidden layer networks, by showing that the performance of a finite-width shallow network can be obtained with the corresponding infinite-width kernel and a suitable choice of the Gaussian priors over the weights of each layer. Second, we derive an effective action for a simple architecture with one convolutional hidden layer, and we compare the result with the one available for FCNs. We find a striking difference in the way the kernel of the two architectures renormalize at finite width: whereas the FC kernel is just globally renormalized by a scalar parameter, the CNN kernel undergoes a local renormalization, meaning that many more free parameters are allowed to be fine-tuned during training (see also Fig. 1). We employ this finding to provide preliminary insight to question (ii): we highlight a simple mechanism for feature learning that can take place in finite-width shallow CNNs, but neither in shallow FC architectures nor in LCNs without weight sharing.

Figure 1: **Global and local kernels in FC and convolutional one hidden layer networks**. Our calculations in the proportional limit \(P,N_{1}\rightarrow\infty\) at fixed \(\alpha_{1}=P/N_{1}\) reveal a striking difference between FCNs and CNNs. In the first case, the NNGP kernel \(K_{\mu\nu}\) that enters into the Bayesian effective action is a global one, which is only capable of tracking global correlations between different training patterns \((\mu,\nu)\) (left). A FCN network in this regime will use the information contained in the dataset to fine-tune a single parameter, \(Q\), that will globally rescale the kernel matrix \(K_{\mu\nu}\). On the other hand, in the CNN case, we identify a local kernel with two additional (spatial) indices. If we consider \(2d\) images, the effect of the local components is to track local correlations between different patches \(i\) and \(j\) of (possibly) different pairs of images \((\mu,\nu)\) (right). The spatial information contained in these correlations allows the CNN to optimize a matrix of parameters \(\bar{Q}_{ij}\), which will renormalize the kernel matrix \(K_{\mu\nu}^{ij}\) in a non-trivial way. In the figure, we display the four-index local kernel \(K_{\mu\nu}^{ij}\) as a matrix using the multi-index notation \(K_{(\mu,i),(\nu,j)}\) and choosing an ordering for the multi-index \((\mu,i)\). Global and local kernels are computed for a subset of \(P=10\) gray-scaled CIFAR10 images down-sized to \(N_{0}=784\). The local kernels are empirically evaluated for \(2d\) convolutions with linear size of the filter mask \(M=14\) and stride \(S=14\) (non-overlapping filters).
II Finite-width one hidden layer FC NNS cannot outperform infinite-width kernels in the overparametrized regime

We consider a supervised regression problem with training set \(\mathcal{T}_{P}=\{x^{\mu},y^{\mu}\}_{\mu=1}^{P}\), where each \(x^{\mu}\in\mathbb{R}^{N_{0}}\) and the corresponding labels \(y^{\mu}\in\mathbb{R}\). We also restrict our analysis to quadratic loss function \(\mathcal{L}\). For a generic one hidden layer network that implements the function \(f_{\rm 1HL}(x)\), we define the loss function as:

\[\mathcal{L}=\frac{1}{2}\sum_{\mu}\left[y^{\mu}-f_{\rm 1HL}(x^{\mu})\right]^{2}+ \frac{T\,\lambda_{0}}{2}\|W\|^{2}+\frac{T\,\lambda_{1}}{2}\|v\|^{2} \tag{1}\]

where \(W\) and \(v\) are respectively the first and last layer weights, and \(T\) is the temperature. The parameters \(\lambda_{0},\lambda_{1}\) can be equivalently thought as Gaussian priors over the weights of each layer, or as \(L^{2}\) regularization terms rescaled by the temperature \(T\). A neural network with one FC hidden layer is formally defined starting from the pre-activations at the first layer:

\[h_{i_{1}}(x)=\frac{1}{\sqrt{N_{0}}}\sum_{i_{0}=1}^{N_{0}}W_{i_{1}i_{0}}x_{i_{0 }}+b_{i_{1}}\,, \tag{2}\]

where \(b\) are the first-layer biases. Since we are interested in regression problems, we add one last readout layer, and we define the function implemented by the one hidden layer FCN as:

\[f_{\rm FCN}(x)=\frac{1}{\sqrt{N_{1}}}\sum_{i_{1}=1}^{N_{1}}v_{i_{1}}\sigma[h_{ i_{1}}(x)]\, \tag{3}\]

where \(\sigma\) is a non-linear activation function. An effective action arises in this setting when one considers the canonical partition function at temperature \(T=1/\beta\) associated to the train loss function \(\mathcal{L}\) and the thermodynamic limit \(N_{1},P\to\infty\) at fixed ratios \(\alpha_{1}=P/N_{1}\). The calculation amounts to reduce the integral over the weights of the network to a form suitable to saddle-point integration over two order parameters \(\{Q,\bar{Q}\}\), \(Z=\int dQd\bar{Q}e^{-N_{1}/2S_{\rm FCN}(\{Q,\bar{Q}\})}\). In the case of odd-activation function, the action is given by [38]:

\[S_{\rm FCN} =-Q\bar{Q}+\log(1+Q)\] \[+\frac{\alpha_{1}}{P}\text{Tr}\log\beta\left(\frac{1}{\beta}+K^{ \rm(R)}(\bar{Q})\right)\] \[+\frac{\alpha_{1}}{P}y^{T}\left(\frac{1}{\beta}+K^{\rm(R)}(\bar{ Q})\right)^{-1}y\,, \tag{4}\]

where \(y\) is the vector of labels \(y=(y^{1},\ldots,y^{P})\). The renormalized kernel \(K^{\rm(R)}(\bar{Q})\) is a \(P\times P\) matrix that processes pairs of input training data \((x^{\mu},x^{\nu})\) and it is given by:

\[K^{\rm(R)}(\bar{Q})=\frac{\bar{Q}}{\lambda_{1}}K(C)\,,\ \ \ \ \ C_{\mu\nu}=\frac{1}{\lambda_{0}}\frac{x^{\mu}\cdot x^{\nu}}{N_{0}}\,. \tag{5}\]

The non-linear operator \(K\) takes as input any (symmetric) \(P\times P\) matrix \(M\) and computes a new matrix in the following way:

\[K_{\mu\nu}(M) =\int d^{2}t\,\mathcal{N}_{t}\left(0,\tilde{M}_{\mu\nu}\right) \sigma(t_{1})\sigma(t_{2})\,, \tag{6}\] \[\tilde{M}_{\mu\nu} =\begin{pmatrix}M_{\mu\mu}&M_{\mu\nu}\\ M_{\mu\nu}&M_{\nu\nu}\end{pmatrix}\,. \tag{7}\]

Here and in the following we are denoting normalized Gaussians as \(\mathcal{N}_{x}\left(0,\Sigma\right)\equiv\exp\left(-x^{\top}\Sigma^{-1}x/2 \right)/\sqrt{\det\!2\pi\Sigma}\).

It is worth noticing that the minimization of the effective action in Eq. (4) is straightforward if \(\alpha_{1}\to 0\), since one easily finds that \(Q^{*}=0\), \(\bar{Q}=1\) and recovers the well-known infinite-width limit Neural Network Gaussian Process (NNGP) kernel. At finite \(\alpha_{1}\), one finds data-dependent solutions for the order parameters that produce a renormalization of the infinite-width kernel, as expressed by Eq. (5).

The effective action at finite width presented in Eq. (4) has been firstly derived for deep linear networks in Ref. [39]. The non-asymptotic evaluation of the partition function in the linear case is given in Ref. [40] in terms of Meyer-G functions. An effective action for globally gated deep linear networks has also been derived [41]. In Ref. [38] we obtained the effective action for one FC hidden layer networks with generic activation function leveraging on a Gaussian equivalence [42; 43; 44; 45] informally justified via a generalized central limit theorem due to Breuer and Major [46]. A result for \(L\) layers can be somewhat bootstrapped from the observation that the stochastic process that describes information flow in shallow non-linear and deep linear networks in the proportional limit is related to a Student's \(t\)-process [47] (see also [38] for a more detailed discussion). Recent work considers the proportional setting in a teacher-student scenario, averaging over the data distribution using the replica method [48; 49] and a first preliminary rigorous result in an almost-proportional limit [50] appeared very recently, leveraging the interpolation method [51; 52].

We now highlight a straightforward consequence of the theoretical framework presented above. First, we notice that the renormalized kernel \(K^{\rm(R)}\) enters into the predictor's statistics for a new unseen test element exactly in the same way the NNGP kernel does in the infinite-width limit, and thus it completely determines the generalization performance at finite width, once the saddle-point equations for the parameter \(\bar{Q}\) are solved [38]. Second, we observe that the scalar parameter \(\bar{Q}\) always appears in combination with the corresponding Gaussian prior \(\lambda_{1}\) as \(\bar{Q}/\lambda_{1}\), meaning that, once evaluated on the saddle-point, \(\bar{Q}\) is just a data-dependent scalar renormalization of the Gaussian prior. This implies
that, once the size of the training set and the activation function \(\sigma\) are fixed, it is always possible to re-obtain the performance of any finite-width network just by carefully fine-tuning the Gaussian prior \(\lambda_{1}\) in the corresponding infinite-width kernel. Therefore, the generalization performance of any finite-width one hidden layer FC network is bounded by the one of a suitable infinite-width kernel with optimal choice of the Gaussian prior.

The empirical observation that infinite-width one hidden layer FCNs seem to systematically outperform their finite-width counterpart -which now finds a possible explanation in this framework- points to the somewhat disappointing consequence that feature learning is not particularly effective in networks with FC layers alone at finite width.

We stress here that the aforementioned observation is not ruling out at all other possible forms of feature learning in FC architectures: (i) Renormalization of the infinite-width kernel is not the only source of feature learning possible. Higher order kernels, irrelevant in the infinite-width limit, may play a role in feature learning, especially as long as one considers the case where the size of the dataset \(P\) roughly scales as the number of parameters \(\sim L\times N^{2}\) (\(N_{\ell}=N\)\(\forall\ell\)) of the FC deep neural network. The recent work [53] is a notable example of how a FC network can learn a convolutional structure in special settings; (ii) Our result holds for Bayesian learning, i.e. when the weights of the networks are sampled from the canonical Gibbs ensemble. This is only obtained if training is governed by a Markov chain Monte Carlo. Practical learning algorithms with state-of-the-art optimizers may behave differently and possibly activate alternative forms of feature learning; (iii) we are limiting our analysis to the standard parametrization setting where the last layer is normalized as \(1/\sqrt{N_{1}}\), and we cannot describe the mechanism for feature learning that may occur in the mean field regime [20] where the last layer is normalized as \(1/N_{1}\) (we note that the framework of [23] seems more suitable to deal with this case). Nonetheless, it must be also said that the empirical evidence provided in Refs. [7; 9; 18; 19; 22] is quite against the fact that these forms of feature learning can play a significant role in improving the generalization performance of FC networks.

We now move to the second question raised in the Introduction of this manuscript. How -contrarily to what occurs in deep nets with FC layers only- do deep architectures with convolutional layers exploit the finite-width regime for efficient feature learning? In view of the previous discussion, one possibility is that CNNs at finite width could be able, in some way, to break the global (trivial) renormalization of the infinite-width kernel that occurs in FC deep networks. In order to test this hypothesis, in the next section we investigate a simple model of CNN with one convolutional hidden layer.

## III Finite-width effective action for a shallow convolutional network

The two fundamental ingredients of a convolutional layer are _local connectivity_ and _weight sharing_: whereas a given neuron in a FC hidden layer receives input from all the neurons in the previous layer, neurons in a CNN are arranged in a \(d\)-dimensional array that reflects the corresponding \(d\)-dimensional arrangement of the input data (e.g. \(d=2\) for images). Each neuron in a given layer here interacts only with a local neighborhood of neurons in the previous layer, in a translational invariant way implemented via a shared (usually small) \(d\)-dimensional mask of learnable weights. These operations define a single convolutional channel that takes as input a \(d\)-dimensional array and outputs another \(d\)-dimensional array, whose dimensions are determined by technical details (such as the stride, padding and dimension of the filter mask). Usually, many of these channels are piled up in a convolutional layer to form, overall, a \((d+1)\)-dimensional array.

Let us now define a simple architecture with one hidden convolutional layer. For simplicity, we will restrict our analysis to one-dimensional convolutions, but the model can be easily generalized to \(d\)-dimensional convolutions. Pre-activations in the hidden layer are given by:

\[h_{i}^{a}(x)=\frac{1}{\sqrt{M}}\sum_{m=-\lfloor M/2\rfloor}^{\lfloor M/2 \rfloor}W_{m}^{a}x_{Si+m}\,. \tag{8}\]

Here \(M\) is the dimension of the channel mask, \(S\) is the stride, the index \(i=1,\ldots,\lfloor N_{0}/S\rfloor\) runs over the input coordinates, the index \(a=1,\ldots,N_{c}\) runs over the channels and the index \(m\) moves through the spatial mask of the convolutional filter. For simplicity, we define periodic boundary conditions (PBCs) over the input coordinates, and we consider odd values of \(M\). As for the one hidden layer FC network, we add one readout layer. In conclusion, the function implemented by the CNN is given by:

\[f_{\text{CNN}}(x)=\frac{1}{\sqrt{N_{c}\lfloor N_{0}/S\rfloor}}\sum_{i=1}^{ \lfloor N_{0}/S\rfloor}\sum_{a=1}^{N_{c}}v_{i}^{a}\sigma\left[h_{i}^{a}(x) \right]\,, \tag{9}\]

where \(\sigma\) is a odd activation function and the \(v\)'s are the learnable weights of the readout layer. Our goal is to derive an effective action in the same setting of Refs. [38], in the thermodynamic limit where \((N_{c},P)\rightarrow\infty\) and their ratio \(\alpha_{c}=P/N_{c}\) is finite.

The partition function for the learning problem with the aforementioned simple CNN can be approximated as an integral over an ensemble of \(\lfloor N_{0}/S\rfloor\times\lfloor N_{0}/S\rfloor\) matrices \(Q\) and \(\tilde{Q}\): \(Z_{\text{CNN}}=\int\mathcal{D}\mathcal{Q}\tilde{Q}e^{-N_{c}/2S_{\text{CNN}}( Q,\tilde{Q})}\) (see Appendices for more details on the derivation),
where the effective action is given by:

\[S_{\rm CNN}(Q,\bar{Q})\equiv -{\rm Tr}\,Q\bar{Q}+{\rm Tr}\log(\mathbb{1}+Q)+\] \[+\frac{\alpha_{c}}{P}{\rm Tr}\log\beta\Big{(}\frac{\mathbb{1}}{ \beta}+K^{(\rm R)}_{\rm CNN}\Big{)}+\] \[+\frac{\alpha_{c}}{P}y^{\top}\Big{(}\frac{\mathbb{1}}{\beta}+K^{ (\rm R)}_{\rm CNN}\Big{)}^{-1}y\,. \tag{10}\]

The trace in the first two terms is over \(\lfloor N_{0}/S\rfloor\times\lfloor N_{0}/S\rfloor\) operators, whereas the renormalized kernel \(K^{(\rm R)}_{\rm CNN}\) is a \(P\times P\) matrix and therefore the trace in the third term and the scalar products with the output labels lie in a \(P\)-dimensional vectorial space. The matrix elements of the renormalized kernel are given by:

\[\Big{[}K^{(\rm R)}_{\rm CNN}(\bar{Q})\Big{]}_{\mu\nu}=\frac{1}{\lambda_{1} \lfloor N_{0}/S\rfloor}\sum_{ij=1}^{\lfloor N_{0}/S\rfloor}\bar{Q}_{ij}K^{ij}_ {\mu\nu}\,, \tag{11}\]

where the kernel \(K^{ij}_{\mu\nu}\) is given in terms of the elements of the following local covariance matrix

\[C^{ij}_{\mu\nu}=\frac{1}{\lambda_{0}M}\sum_{m=-\lfloor M/2\rfloor}^{\lfloor M /2\rfloor}x^{\mu}_{Si+m}x^{\nu}_{Sj+m}, \tag{12}\]

with \(\lambda_{0}\) being the hidden layer Gaussian prior, via the same functional relations of the FC case:

\[K^{ij}_{\mu\nu}=\int d^{2}t\,{\cal N}_{t}\left(0,\bar{C}^{ij}_{ \mu\nu}\right)\sigma(t_{1})\sigma(t_{2})\,,\] \[\bar{C}^{ij}_{\mu\nu}=\begin{pmatrix}C^{ii}_{\mu\mu}&C^{ij}_{\mu \nu}\\ C^{ii}_{\mu\nu}&C^{ij}_{\mu\nu}\end{pmatrix}\,. \tag{13}\]

It is worth noticing that such a local kernel has been already found in the seminal work on the Gaussian process limit of infinite-width CNNs [9]. However, one finds that in the limit of infinitely many channels \(N_{c}\gg P\), the CNN evolves according to the following averaged kernel:

\[\bar{K}_{\mu\nu}=\frac{1}{\lambda_{1}\lfloor N_{0}/S\rfloor}\sum_{i=1}^{ \lfloor N_{0}/S\rfloor}K^{ii}_{\mu\nu}\,. \tag{14}\]

In our effective action framework we recover this result by solving the saddle-point equations for the matrix \(\bar{Q}\) in the limit of \(\alpha_{c}\to 0\), where one finds \(\bar{Q}_{ij}=\delta_{ij}\) and only the diagonal components of the local kernel contribute to the prediction.

Note that the derivation of this result is based on a Gaussian equivalence that is informally justified using a general class of central limit theorems (such as the Breuer-Major theorem in [38]). This suggests that the dimension of the input \(N_{0}\) should scale to infinity at the same rate as the size of the dataset \(P\). If this is the case, as it was in [38], the stride \(S\) should be chosen to scale extensively in \(N_{0}\), so that the final effective action will depend on a finite number of order parameters.

## IV A simple mechanism for feature learning in finite-width CNNs: local kernel renormalization

Let us now analyze the saddle-point equations deriving from Eq. (II). We can gain analytical insight in the zero temperature limit, where we find the following matrix equations that determine \(Q\) and \(\bar{Q}\):

\[\bar{Q}_{ij}=(\mathbb{1}+Q)^{-1}_{ij} \tag{15}\] \[Q_{ij}=\frac{\alpha_{c}}{P}{\rm Tr}\Big{(}K^{ij}\left[K^{(\rm R) }_{\rm CNN}(\bar{Q})\right]^{-1}\Big{)}+\] \[-\frac{\alpha_{c}}{P}y^{T}\Big{(}\left[K^{(\rm R)}_{\rm CNN}(\bar {Q})\right]^{-1}K^{ij}\left[K^{(\rm R)}_{\rm CNN}(\bar{Q})\right]^{-1}\Big{)}y \tag{16}\]

where the shortcut \(K^{ij}\) has been introduced to indicate the partial trace over indices \((\mu,\nu)\) in the local kernel defined in Eq. (11). More explicitly, given a matrix \(O\) with elements \(O_{\mu\nu}\), the partial trace is defined as \({\rm Tr}(K^{ij}O):=\sum_{\mu\nu=1}^{P}K^{ij}_{\mu\nu}O_{\mu\nu}\). The exact solution of these matrix equations cannot be obtained in closed form. However, it is possible to compute it perturbatively around the infinite-width limit \(\alpha_{c}\to 0\). Parametrizing the solution of the first equation as \(\bar{Q}_{ij}=\delta_{ij}+\alpha_{c}\delta\bar{Q}_{ij}+o(\alpha_{c})\), we find the simple result:

\[\delta\bar{Q}_{ij}=\frac{1}{P}\tilde{y}^{T}K^{ij}\tilde{y}-\frac{1}{P}{\rm Tr }\left(K^{ij}\bar{K}^{-1}\right)\,, \tag{17}\]

where \(\tilde{y}^{\mu}=\sum_{\mu=1}^{P}\bar{K}^{-1}_{\mu\nu}y^{\mu}\) and \(\bar{K}\) is the infinite-width CNN averaged kernel defined in Eq. (14). We now provide a physical interpretation of this finding.

Our claim is that the matrix \(\bar{Q}\) provides a compact description of feature learning in the CNN model under consideration: the feature matrix elements \(\bar{Q}_{ij}\) that

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline Architecture & Kernel type & IW Kernel & Renormalized Kernel \\ \hline \hline FC & \(K_{\mu\nu}\) & \(K_{\mu\nu}\) & \(\bar{Q}K_{\mu\nu}\) \\ \hline LCN & \(K^{ii}_{\mu\nu}\) & \(\sum_{i}K^{ii}_{\mu\nu}\) & \(\sum_{i}\bar{Q}_{i}K^{ii}_{\mu\nu}\) \\ \hline CNN & \(K^{ij}_{\mu\nu}\) & \(\sum_{i}K^{ii}_{\mu\nu}\) & \(\sum_{ij}\bar{Q}_{ij}K^{ij}_{\mu\nu}\) \\ \hline \end{tabular}
\end{table}
Table 1: **Summary of the analytical results for one hidden layer networks**. We identify three different types of kernels in the Bayesian effective action for FC, LC and convolutional networks: a global kernel in the FC case, a local diagonal kernel in LCNs and a local (with additional non-diagonal components) in CNNs. The infinite-width kernel that describes LCNs and CNNs is the same, as shown in [9]. In the proportional limit, the renormalization of the FC kernel is scalar, i.e. the NNGP kernel is only globally rescaled by a data-dependent number \(\bar{Q}\). In LCNs, we can adjust a vector of components \(\bar{Q}_{i}\), which determines how much the self-correlations of the \(i\)-th patch of the diagonal local kernel do contribute to the final prediction. In CNNs, due to locality and weight-sharing, we can fine-tune a full matrix \(\bar{Q}_{ij}\), which controls how much the local correlations of patches \(i\) and \(j\) will enter the effective kernel learned by the network after training. This is what we call _local kernel renormalization_.
are optimized during training are in fact in one-to-one correspondence with the combinations of pairs of the \(\lfloor N_{0}/S\rfloor\) patches of the local covariance matrix, which is defined from the trainset elements \(x^{\mu}\). From the first term of the r.h.s. of Eq. (17) we can notice that whenever the local kernel of the CNN at the spatial locations \((i,j)\) will have a significant overlap with the effective label vector \(\tilde{y}\), the element \(\bar{Q}_{ij}\) of the feature matrix \(\bar{Q}\) will differ from one. In other words, this is a measure of the relative importance of the pairwise spatial correlations in the input dataset wrt the labels. From Eq. (11), we can now interpret the matrix \(\bar{Q}\) as a feature, data-dependent matrix that tells us how much a given component of the local kernel contribute to the renormalized kernel \(K_{\rm CNN}^{\rm(R)}\).

The local kernel renormalization that takes place in this CNN model in the proportional limit is completely different from the trivial global one that occurs in the corresponding FC one hidden layer network. At this point one can wonder whether both the ingredients in CNNs design, local connectivity and weight sharing, are needed to observe the phenomenon of local kernel renormalization. The answer is affirmative: local kernel renormalization does not occur in a shallow network with local connectivity only, as we explicitly check in the appendix. In that case, only the diagonal components of the local kernel enter the effective action, and the network is not sensible to spatial correlations between different patches (see also Table 1 for a summary of the analytical results).

## V Empirical evidence for global and local kernel renormalization

Extracting the renormalized kernel from the measure of a physical observable is not straightforward, and we have no way, at the moment, to directly access it in numerical experiments. One indirect experimental blueprint of the form of kernel renormalization at finite width in different architectures is given by the _similarity matrix_ of the internal representations before and after training. We define this observable for both FCNs and CNNs as:

\[O_{\mu\nu}\equiv\frac{1}{N_{1}}\sum_{i=1}^{N_{1}}\sigma\left(h_{i}^{\mu} \right)\sigma\left(h_{i}^{\nu}\right)\,, \tag{18}\]

where \(N_{1}\) denotes the number of neurons in the last layer (for CNNs we have \(N_{1}=N_{c}\lfloor N_{0}/S\rfloor\)). We can track the effect of training by taking the difference between the similarity matrix at initialization, which is by definition the NNGP kernel (up to the Gaussian prior \(\lambda_{1}\)), and the same quantity after training: \(\Delta K_{\mu\nu}\equiv\left(O_{\mu\nu}\right)-K_{\mu\nu}\), where the average is done over the Gibbs ensemble of the weights. We analytically compute this observable for FCNs and CNNs, as shown in appendix C. The final result in the two cases respectively reads (at zero temperature):

\[\Delta K_{\mu\nu}^{\rm FCN} =-\frac{1}{N_{1}}\left[K_{\mu\nu}-\frac{\lambda_{1}}{\bar{Q}}y^{ \mu}y^{\nu}\right]; \tag{19}\] \[\Delta K_{\mu\nu}^{\rm CNN} =-\frac{1}{\lambda_{1}N_{1}}\sum_{ij,\lambda\rho=1}^{\lfloor N_{ 0}/S\rfloor,P}\bar{Q}_{ij}P_{\mu\lambda\nu\rho}^{ij}\] \[\times\left[\left(K_{\rm CNN}^{\rm(R)}\right)_{\lambda\rho}^{-1} -\sum_{\omega=1}^{P}\left(K_{\rm CNN}^{\rm(R)}\right)_{\lambda\epsilon}^{-1} \left(K_{\rm CNN}^{\rm(R)}\right)_{\rho\omega}^{-1}y^{\epsilon}y^{\omega} \right], \tag{20}\]

where \(P_{\mu\lambda\nu\rho}^{ij}\equiv\frac{1}{2}\sum_{k}\left[K_{\mu\lambda}^{ik}K_ {\nu\rho}^{kj}+K_{\mu\lambda}^{ki}K_{\nu\rho}^{jk}\right]\). Let us now highlight a few interesting physical implications that directly follow from these formulas and can be checked with numerical experiments:

1. the difference between the trained and untrained similarity matrix \(\Delta K_{\mu\nu}^{\rm FCN}\) converges to zero in the thermodynamic limit \(P,N_{1}\rightarrow\infty\) at fixed \(\alpha_{1}=P/N_{1}\), as long as the terms in the square brackets are of order \(o(N_{1})\). This means that in the proportional regime under consideration the effect of training on the internal representations is just a finite-size correction \(O(1/N_{1})\) and the trained similarity matrix stays very close to its value at initialization. In particular, we expect that the empirical distribution of the elements of the matrix \(\Delta K_{\mu\nu}^{\rm FCN}\) will be increasingly peaked around zero in finite-size scaling experiments where we consider increasing values of \(P\) and \(N_{1}\) keeping their ratio \(\alpha_{1}=P/N_{1}\) fixed;
2. by choosing a learning task with labels \(y^{\mu}=0,1\) and input patterns \(x^{\mu}\) such that the matrix elements of the kernel \(K_{\mu\nu}\) are on average centered in zero, the distribution of the matrix elements of the block \(\Delta K_{\mu\nu}^{\rm FCN}|_{11}\) (corresponding to the subset of training patterns with label one) should drift towards zero at a rate \(1/N_{1}\), if one performs experiments at constant \(\alpha_{1}\), while increasing \(P\) and \(N_{1}\). Interestingly, the remaining blocks of the matrix \(\Delta K_{\mu\nu}^{\rm FCN}\) should not be affected by this drift;
3. the difference between the trained and untrained convolutional similarity matrix \(\Delta K_{\mu\nu}^{\rm CNN}\) displays a different behavior, due to local kernel renormalization. For instance, even if we restrict our analysis to the learning setting outlined in bullet (ii), the elements of the block \(\Delta K_{\mu\nu}^{\rm CNN}|_{01}\) should not identically converge to zero in the thermodynamic proportional limit. A straightforward scaling analysis in fact shows that the contribution due to the term in the square bracket does not vanish in the proportional limit.

In Fig. 2 we present the results of a finite-size scaling analysis to check the aforementioned predictions, where we trained one hidden layer FCNs and CNNs (
convolutions) on a synthetic dataset composed of random Gaussian patterns whose labels are given by a linear teacher function, i.e. \(y=\frac{1}{2}\left[1+\text{sign}(t\cdot x)\right]\), where \(t\) is an \(N_{0}\)-dimensional vector with unitary entries. Overall, the fact that the change in the internal representations of FCNs is a finite-size effect is confirmed by the experiments. On the other hand, the same numerical simulations with CNNs clearly identify a genuine change in the internal representations that occurs also in the thermodynamic proportional limit. We note that such a clean signature of the differences between FCNs and CNNs derive also from our choice of working close to zero temperature (where the architectures can precisely fit the training data). Equations at finite \(T\) are more involved, and measuring the indirect effects of global and local kernel renormalization in this case is more difficult. Interestingly, even though our theory should be valid only in the Bayesian setting, it turns out to be also predictive for a learning algorithm that does not explicitly sample from the Gibbs posterior distribution: the aforementioned experiments were carried out using gradient descent and the so-called ADAM optimizer [54] (see Appendix D for more details).

Another empirical evidence for the local renormalization in CNNs can be found analyzing the bias/variance decomposition of the generalization error over a new example \((\mathbf{x}^{0},y^{0})\):

\[\langle\epsilon_{\text{g}}(\mathbf{x}^{0},y^{0})\rangle=(y^{0}-\Gamma)^{2}+ \sigma^{2} \tag{21}\]

with:

\[\Gamma=\sum_{\mu\nu=1}^{P}\kappa_{\mu}^{(\text{R})}\left(\frac{ \mathds{1}}{\beta}+K^{(\text{R})}(\{\bar{Q}\})\right)_{\mu\nu}^{-1}y_{\nu}, \tag{22}\] \[\sigma^{2}=\kappa_{0}^{(\text{R})}-\sum_{\mu\nu=1}^{P}\kappa_{\mu }^{(\text{R})}\left(\frac{\mathds{1}}{\beta}+K^{(\text{R})}(\{\bar{Q}\}) \right)_{\mu\nu}^{-1}\kappa_{\nu}^{(\text{R})} \tag{23}\]

Figure 2: **Predicting the effect of global and local kernel renormalization on the internal representations of FCNs and CNNs. A finite-size scaling analysis.** Here we show the results of numerical experiments performed in order to check the predictions of Eqs. (19), (20) for the difference \(\Delta K_{\mu\nu}\) between the similarity matrix after and before training. In particular, we train architectures with a single hidden FC\(/1d\)-convolutional layer with tanh activation (see details in Appendix D) on a learning task with random Gaussian inputs and zero/one labels given by a linear teacher. We perform a finite-size scaling analysis by keeping \(\alpha_{1}=1\) and choosing increasing values of \(P,N_{1}\). In panel (a) we observe that the variance of the empirical distribution of the matrix elements of the zero-one block \(\Delta K^{\text{PC}}_{\mu\nu}|_{01}\) rapidly converges to zero as \(N_{1}\) grows and \(\alpha_{1}\) is kept fixed to one, in agreement with the prediction of Eq. (19) (top). In particular, the variance goes to zero as \(1/N_{1}^{2}\) (middle). The fact that the similarity matrices after and before training are more and more similar in the thermodynamic limit is also clearly visible in the explicit array plot of a subsample of the zero-one block \(\Delta K^{\text{PCN}}_{\mu\nu}|_{01}\) (bottom). In panel (b) we display the histogram of the distribution of the matrix elements of the one-one block \(\Delta K^{\text{PCN}}_{\mu\nu}|_{11}\) (top). As predicted by Eq. (19), the distribution shrinks while drifting towards zero at a rate \(\sim 1/N_{1}\) (middle). Overall, (a) and (b) represent a clear indication that the change in the after-training internal representations in one hidden layer FCNs is a finite-size effect. In (c) and (d) we repeat the same experiment for an architecture with a \(1d\)-convolutional hidden layer. The striking difference wrt the FC case is that the variance of the empirical distribution of the matrix elements \(\Delta K^{\text{CNN}}_{\mu\nu}\) does not shrink to zero as \(N_{c}\) grows and \(\alpha_{c}\) is kept fixed to one, as one can also appreciate by directly looking at the array plot of a subsample of the block matrices \(\Delta K^{\text{CNN}}_{\mu\nu}|_{01}\) and \(\Delta K^{\text{CNN}}_{\mu\nu}|_{11}\).
These expressions are valid for FCNs, and can be easily extended to CNNs with the substitution:

\[K^{(\mathrm{R})}\to K^{(\mathrm{R})}_{\mathrm{CNN}}\quad\kappa^{(\mathrm{R})}\to \kappa^{(\mathrm{R})}_{\mathrm{CNN}}\,, \tag{24}\]

where the renormalized kernel matrices \(K^{(\mathrm{R})}\) and \(K^{(\mathrm{R})}_{\mathrm{CNN}}\) are given respectively in equations (5) and (11). We recall the definition of the additional kernel integrals that depend on the new example \(x^{0}\), that is \(\kappa^{(\mathrm{R})}_{\mu}\), where the index \(\mu=0,\ldots,P\) covers both the dataset and the new test element \(x^{0}\):

\[\kappa_{\mu}=\int\mathrm{d}t\mathcal{N}_{t}\left(0,\tilde{C}_{\mu}\right) \sigma(t_{1})\sigma(t_{2})\,, \tag{25}\]

where:

\[\tilde{C}_{\mu}=\begin{pmatrix}C_{\mu\mu}&C_{\mu 0}\\ C_{\mu 0}&C_{00}\end{pmatrix}\,,\ C_{\mu 0}=\frac{x^{\mu}\cdot x^{0}}{\lambda_{0}N_ {0}}\,, \tag{26}\]

In the CNN case, these quantities are computed starting from the local covariance matrix \(C^{ij}_{\mu\nu}\) defined in Eq. (12). These terms undergo the same type of renormalization as the correspondent kernel matrices (see table 1), that is:

\[\kappa^{(\mathrm{R})}_{\mu}=\frac{\bar{Q}}{\lambda_{1}}\kappa_{ \mu}\,, \tag{27}\] \[\left[\kappa^{(\mathrm{R})}_{\mathrm{CNN}}\right]_{\mu}=\frac{1} {\lambda_{1}\lfloor N_{0}/S\rfloor}\sum_{ij=1}^{\lfloor N_{0}/S\rfloor} \bar{Q}_{ijk}\kappa_{\mu}^{ij}\,, \tag{28}\]

The \(\beta\to\infty\) limit of Eq. (22) yields different behaviors for the bias in the two cases:

\[\Gamma^{\mathrm{FC}}=\sum_{\mu\nu=1}^{P}\kappa_{\mu}K^{-1}_{\mu \nu}\ y_{\nu}\,, \tag{29}\] \[\Gamma^{\mathrm{CNN}}=\sum_{\mu\nu=1}^{P}\left[\kappa^{(\mathrm{ R})}_{\mathrm{CNN}}\right]_{\mu}\left(K^{(\mathrm{R})}_{\mathrm{CNN}}(\{ \bar{Q}\})\right)^{-1}_{\mu\nu}\,y_{\nu}, \tag{30}\]

From these expressions one can make two observations: (i) as already discussed in [38], \(\Gamma^{\mathrm{FC}}\) at finite width is independent of \(\bar{Q}\), and therefore the bias is identical to the infinite-width case for any \(N_{1}\). (ii) In CNNs, on the other hand, the dependence of the bias from the renormalized Kernel does not disappear, and neither does the dependence on the width \(N_{1}\).

In the large Gaussian prior regime \(\lambda_{1}\gg 1\), the previous observations can be made into straightforward predictions that can be tested with numerical experiments. It is easy to check that the variance is vanishing in this regime, since it is always proportional to a term \(1/\lambda_{1}\) both in FCNs and CNNs. Moreover, when \(\lambda_{1}\gg 1\), the analytical criterion found in [38] to predict whether a finite-width FCN will outperform its infinite-width counterpart, is never satisfied. This is a special case of the general considerations already presented in section II. To summarize, in this regime we expect that (i) the finite FC network can not outperform its infinite-width counterpart. (ii) On the contrary, we expect CNNs to possibly beat their infinite-width performance, since the bias can be reduced through the optimization of the feature matrix \(Q^{ij}\).

To test these predictions, we performed numerical simulations with \(2d\) CNNs trained on CIFAR10 examples, close to the zero-temperature limit and in the bias-dominated setting where we let the Gaussian prior \(\lambda_{1}\) be large. As shown in Fig. 3, the test loss of the FCN is a monotonically decreasing function of the hidden layer size \(N_{1}\) and it does reach its asymptotic (best) performance already for \(\alpha\sim 1\). Compatibly to local kernel renormalization, CNNs with finite number of channels best their infinite-width counterpart, already

Figure 3: **Test loss vs number of \(2d\) convolutional filters for one hidden layer CNNs with different filter sizes \(M\).** All the networks are trained on the same binary regression task with \(P=500\) examples from two classes of the CIFAR10 dataset, ”cars” and ”planes”, respectively labeled \(0\) and \(1\). Error bars represent one standard deviation. The images are gray-scaled and coarse-grained to \(N_{0}=28\times 28\). We choose the filter size \(M\) to be an integer divisor of \(28\), and the filters to be non overlapping (taking stride \(S=M\)). A sketch of how the images are parsed by filters is reported for each network. With this choice, the FC network is retrieved setting \(M=28\). The nets are trained with a zero temperature Langevin dynamics (see appendix REF), and with large Gaussian priors \(\lambda_{1}\). In this bias-dominated regime, we observe that: (i) in FCNs, the best possible performance is the infinite-width one. Here the variance is monotonically decreasing with the width \(N_{c}\), while the bias is a constant. Therefore, the test loss approaches its minimum monotonically from above, as predicted by our theory. (ii) finite width CNNs can outperform their infinite-width counterpart. The local kernel renormalization mechanism allows the CNN to optimize both bias and variance at finite width, making it even convenient to have a small number of filters (the minimum is reached for \(N_{c}\sim 50\)). Eventually, the CNNs reach their asymptotic performance from below.
for large filter sizes (\(M=14,7\)) and ultimately approach the infinite-width limit performance from below.

## VI Discussion

In this work we have derived an effective action in the proportional limit for CNNs and LCNs with one hidden layer, and we have extensively compared these results with the FC case found in [38].

We have highlighted a striking difference in the kernel renormalization mechanism that the three different architectures undergo at finite width, summarized in Table 1, and we have suggested a mechanism for feature learning (local kernel renormalization) that can take place in CNNs, but not in architectures with LC or FC layers alone.

We have made a preliminary study on the effect of global and local kernel renormalization at the level of the internal representations of trained networks, which is in agreement with our predictions. We have also investigated the behavior of the generalization loss of one hidden layer networks in a controlled Bayesian setting where the performance is dominated by the bias, which confirms the observation that, in this regime, (i) finite-width one hidden layer FC networks cannot outperform their infinite-width counterpart; (ii) CNNs with finite number of channels can improve the infinite-width performance, which is in qualitative agreement with our local kernel renormalization hypothesis for the bias in Eq. (30).

We conclude by pointing out a number of possible follow-ups of this work: (i) it would be interesting to extend our framework to \(L\) convolutional layers, following the same strategy proposed for FC layers in [38]; (ii) our current approach cannot cover the mean-field regime, where the feature learning mechanism for FCNs and CNNs may quantitatively change. We think that the formalism presented in [23] might be more suited to investigate finite-width effects in this case; (iii) we are writing a detailed open-source numerical routine to extensively check the theoretical predictions of our effective theory for FCNs, LCNs and CNNs, which will also serve to point out the limitations of the present framework.

_Acknowledgements_.- R.B. and P.R. are supported by #NEXTGENERATIONEU (NGEU) and funded by the Ministry of University and Research (MUR), National Recovery and Resilience Plan (NRRP), project MNESYS (PE0000006) "A Multiscale integrated approach to the study of the nervous system in health and disease" (DN. 1553 11.10.2022). The authors thank F. Cagnetta for giving feedback on a preliminary version of this manuscript and for pointing out interesting related works, G. Naveh, Z. Ringel and I. Seroussi for discussions and clarifications on their results in Refs. [23; 24] and P. Baglioni, M. Gherardi and M. Pastore for discussions.
