Structures of Neural Network Effective Theories
We develop a diagrammatic approach to effective field theories (EFTs)
corresponding to deep neural networks at initialization, which dramatically
simplifies computations of finite-width corrections to neuron statistics. The
structures of EFT calculations make it transparent that a single condition
governs criticality of all connected correlators of neuron preactivations.
Understanding of such EFTs may facilitate progress in both deep learning and
field theory simulations.
Ian Banta, Tianji Cai, Nathaniel Craig, Zhengkang Zhang
# Structures of Neural Network Effective Theories

###### Abstract

We develop a diagrammatic approach to effective field theories (EFTs) corresponding to deep neural networks at initialization, which dramatically simplifies computations of finite-width corrections to neuron statistics. The structures of EFT calculations make it transparent that a single condition governs criticality of all connected correlators of neuron preactivations. Understanding of such EFTs may facilitate progress in both deep learning and field theory simulations.

_Introduction --_ Machine learning (ML) has undergone a revolution in recent years, with applications ranging from image recognition and natural language processing, to self-driving cars and playing Go. Central to all these developments is the engineering of deep neural networks, a class of ML architectures consisting of multiple layers of artificial neurons. Such networks are apparently rather complex, with a deterring number of trainable parameters, which means practical applications have often been guided by expensive trial and error. Nevertheless, extensive research is underway toward opening the black box.

That a theoretical understanding of such complex systems is possible has to do with the observation that a wide range of neural network architectures actually admit a simple limit: they reduce to Gaussian processes when the network width (number of neurons per layer) goes to infinity [1; 2; 3; 4; 5; 6], and evolve under gradient-based training as linear models governed by the neural tangent kernel [7; 8; 9]. However, an infinitely-wide network neither exists in practice, nor provides an accurate model for deep learning. It is therefore crucial to understand finite-width effects, which have recently been studied by a variety of methods [10; 11; 12; 13; 14; 15; 16; 17; 18; 19; 20; 21; 22; 23].

This line of research in ML theory has an intriguing synergy with theoretical physics [24]. In particular, it has been realized that neural networks have a natural correspondence with (statistical or quantum) field theories [25; 26; 27; 28; 29; 30; 31; 32; 33; 34]. Infinite-width networks--which are Gaussian processes--correspond to free theories, while finite-width corrections in wide networks can be calculated perturbatively as in weakly-interacting theories. This allows for a systematically-improvable characterization of neural networks beyond the (very few) exactly-solvable special cases [35; 36; 37]. Meanwhile, from an effective theory perspective [21], information propagation through a deep neural network can be understood as a renormalization group (RG) flow. Examining scaling behaviors near RG fixed points reveals strategies to tune the network to criticality [38; 39; 40], which is crucial for mitigating the notorious exploding and vanishing gradient problems in practical applications. In the reverse direction, this synergy also points to new opportunities to study field theories with neural networks [33].

Inspired by recent progress, in this letter we further explore the structures of effective field theories (EFTs) corresponding to archetypical deep neural networks. To this end, we develop a novel diagrammatic formalism.1 Our approach largely builds on the frameworks of Refs. [21; 22], which enable systematic calculations of finite-width corrections. The diagrammatic formalism dramatically simplifies these calculations, as we demonstrate by concisely reproducing known results in the main text and presenting further examples with new results in the Supplemental Material. Interestingly, the structures of diagrams in the RG analysis suggest that neural network EFTs are of a quite special type, where a single condition governs the critical tuning of all neuron correlators. The study of these EFTs may lend new insights into both neural network properties and novel field-theoretic phenomena.

Footnote 1: See also Refs. [13; 17; 18; 26; 27; 31; 32; 41] for Feynman diagram-inspired approaches to ML.

_EFT of deep neural networks --_ The archetype of deep neural networks, the multilayer perceptron, can be defined by a collection of neurons whose values \(\phi_{i}^{(0)}\) (called preactivations) are determined by the following operations given an input \(\vec{x}\in\mathbb{R}^{n_{0}}\):

\[\phi_{i}^{(1)}(\vec{x}) =\sum_{j=1}^{n_{0}}W_{ij}^{(1)}x_{j}+b_{i}^{(1)}\,,\] \[\phi_{i}^{(\ell)}(\vec{x}) =\sum_{j=1}^{n_{\ell-1}}W_{ij}^{(\ell)}\sigma\big{(}\phi_{j}^{( \ell-1)}(\vec{x})\big{)}+b_{i}^{(\ell)}\quad(\ell\geq 2)\,. \tag{1}\]

Here superscripts in parentheses label layers, subscripts \(i,j\) label neurons within a layer (of which there are \(n_{\ell}\) at the \(\ell\) th layer), and \(\sigma(\phi)\) is the activation function (common choices include \(\tanh(\phi)\) or \(\text{ReLU}(\phi)\equiv\max(0,\phi)\)). The weights \(W_{ij}^{(\ell)}\) and biases \(b_{i}^{(\ell)}\) (\(\ell=1,\ldots,L\)) are the network parameters which are adjusted to minimize a loss function during training, such that the trained network can approximate the desired function.

The basic idea of an EFT of deep neural networks is to consider an ensemble of networks, where at initialization, \(W_{ij}^{(\ell)}\) and \(b_{i}^{(\ell)}\) are drawn independently from zero-mean
Gaussian distributions with variances \(C_{W}^{(\ell)}/n_{\ell-1}\) and \(C_{b}^{(\ell)}\), respectively. The statistics of this ensemble encode both the typical behavior of neural networks initialized in this manner and how a particular network may fluctuate away from typicality. In the field theory language, these are captured by a Euclidean action, \(\mathcal{S}[\phi]=-\log P(\phi)\), for all neuron preactivation fields \(\phi_{i}^{(\ell)}(\vec{x})\), where \(P(\phi)\) is the joint probability distribution. As we review in the Supplemental Material, at initialization the conditional probability distribution at each layer is Gaussian:

\[P\big{(}\phi^{(\ell)}\big{|}\phi^{(\ell-1)}\big{)} =\big{[}\text{det}\big{(}2\pi\mathcal{G}^{(\ell)}\big{)}\big{]}^{ \frac{-\pi}{2}}\;e^{-\mathcal{S}_{0}^{(\ell)}}\,, \tag{2}\] \[\mathcal{S}_{0}^{(\ell)} =\!\int\!d\vec{x}_{1}d\vec{x}_{2}\,\frac{1}{2}\sum_{i=1}^{n_{ \ell}}\phi_{i}^{(\ell)}\!(\vec{x}_{1})\big{(}\mathcal{G}^{(\ell)}\big{)}^{\!- \!1}\!(\vec{x}_{1},\vec{x}_{2})\,\phi_{i}^{(\ell)}\!(\vec{x}_{2}), \tag{3}\]

where \(\mathcal{G}^{(\ell)}(\vec{x}_{1},\vec{x}_{2})=\frac{1}{n_{\ell-1}}\sum_{j=1}^ {n_{\ell-1}}\mathcal{G}_{j}^{(\ell)}(\vec{x}_{1},\vec{x}_{2})\), with

\[\mathcal{G}_{j}^{(\ell)}(\vec{x}_{1},\vec{x}_{2}) =C_{b}^{(\ell)}+C_{W}^{(\ell)}\,\sigma\big{(}\phi_{j}^{(\ell-1)}( \vec{x}_{1})\big{)}\,\sigma\big{(}\phi_{j}^{(\ell-1)}(\vec{x}_{2})\big{)}\] \[\equiv C_{b}^{(\ell)}+C_{W}^{(\ell)}\,\sigma_{j,\vec{x}_{1}}^{(\ell-1)} \,\sigma_{j,\vec{x}_{2}}^{(\ell-1)} \tag{4}\]

for \(\ell\geq 2\), and \(\mathcal{G}_{j}^{(1)}(\vec{x}_{1},\vec{x}_{2})=C_{b}^{(1)}+C_{W}^{(1)}\,x_{1j }\,x_{2j}\). We have taken the continuum limit in input space to better parallel field theory analyses. \(\big{(}\mathcal{G}^{(\ell)}\big{)}^{\!-\!1}\) is understood as the pseudoinverse when \(\mathcal{G}^{(\ell)}\) is not invertible. We see that for \(\ell\geq 2\), \(\mathcal{G}^{(\ell)}(\vec{x}_{1},\vec{x}_{2})\) is an operator of the \((\ell-1)\)th-layer neurons, so Eq. (3) is actually an interacting theory with interlayer couplings. This also means the determinant in Eq. (2) is not a constant prefactor. To account for its effect, we introduce auxiliary anticommuting fields \(\psi\), \(\vec{\psi}\) which are analogs of ghosts and antighosts in the Faddeev-Popov procedure. Including all layers, we have

\[e^{-\mathcal{S}[\phi]}=\int\!\mathcal{D}\psi\mathcal{D}\bar{\psi}\;e^{-\sum \limits_{i=1}^{L}\big{(}\mathcal{S}_{0}^{(\ell)}[\phi]+\mathcal{S}_{\psi}^{( \ell)}[\phi,\psi,\vec{\psi}]\big{)}}\,, \tag{5}\]

where \(\mathcal{S}_{0}^{(\ell)}\) is given by Eq. (3) above and

\[\mathcal{S}_{\psi}^{(\ell)}=-\!\int\!d\vec{x}_{1}d\vec{x}_{2}\sum_{i^{\prime}= 1}^{n_{\ell}/2}\bar{\psi}_{i^{\prime}}^{(\ell)}\!(\vec{x}_{1})\big{(}\mathcal{ G}^{(\ell)}\big{)}^{\!-\!1}\!(\vec{x}_{1},\vec{x}_{2})\,\psi_{i^{\prime}}^{(\ell)} \!(\vec{x}_{2})\,. \tag{6}\]

The \(\ell\)th-layer neurons interact with the \((\ell-1)\)th-layer and \((\ell+1)\)th-layer neurons via \(\mathcal{S}_{0}^{(\ell)}\) and \(\mathcal{S}_{0}^{(\ell+1)}\), respectively, while their associated ghosts have opposite-sign couplings to the \((\ell-1)\)th-layer neurons but do not couple to \((\ell+1)\)th-layer neurons. This means \(\phi^{(\ell)}\) and \(\psi^{(\ell)}\) loops cancel as far as their couplings to \(\phi^{(\ell-1)}\) are concerned, which must be the case since the network has directionality--neurons at a given layer cannot be affected by what happens at deeper layers.

_Neuron statistics from Feynman diagrams_ -- We are interested in calculating neuron statistics, _i.e._ connected correlators of neuron preactivation fields \(\phi_{i}^{(\ell)}(\vec{x})\) in the EFT above. More precisely, we would like to track the evolution of neuron correlators as a function of network layer \(\ell\), which encodes how information is processed through a deep neural network and has an analogous form to RG flows in field theory. To this end, we develop an efficient diagrammatic framework to recursively determine \(\ell\)th-layer neuron correlators in terms of \((\ell-1)\)th-layer neuron correlators.

Starting from the action Eq. (3), we can derive the following Feynman rule (see Supplemental Material for details):

\[\begin{array}{c}\frac{1}{n_{\ell-1}}\,\mathcal{G}_{j}^{(\ell)} \!(\vec{x}_{1},\vec{x}_{2})\\ \\ \end{array} = \tag{7}\]

As indicated above, a blob means taking the expectation value of the operator (or product of operators) attached to it. Eq. (7) contains both what we normally call propagators and vertices: the first term on the right-hand side, when summed over \(j\), is the full propagator (or two-point correlator) for \(\phi_{i}^{(\ell)}\),

\[\big{\langle}\phi_{i_{1}}^{(\ell)}\!(\vec{x}_{1})\,\phi_{i_{2}}^{(\ell)}\!( \vec{x}_{2})\big{\rangle}=\delta_{i_{1}i_{2}}\big{\langle}\mathcal{G}^{(\ell) }\!(\vec{x}_{1},\vec{x}_{2})\big{\rangle}\,, \tag{8}\]

while the second term is an interaction vertex between \(\phi_{i}^{(\ell)}\) bilinears and operators built from \(\phi_{j}^{(\ell-1)}\):

\[\Delta_{j}^{(\ell-1)}\!(\vec{x}_{1},\vec{x}_{2})\equiv\sigma_{j,\vec{x}_{1}}^{( \ell-1)}\,\sigma_{j,\vec{x}_{2}}^{(\ell-1)}-\big{\langle}\sigma_{j,\vec{x}_{1}}^{( \ell-1)}\,\sigma_{j,\vec{x}_{2}}^{(\ell-1)}\big{\rangle} \tag{9}\]

for \(\ell\geq 2\), and \(\Delta_{j}^{(0)}\!(\vec{x}_{1},\vec{x}_{2})=0\). From Eq. (7) it is clear that each \(\phi^{2}\Delta\) vertex comes with a factor of \(\frac{1}{n}\) (where \(n\) collectively denotes \(n_{1},\cdots,n_{L-1}\)). In the infinite-width limit, \(n\to\infty\), the EFT is a free theory, whereas for large but finite \(n\), we have a weakly-interacting theory where higher-point connected correlators can be perturbatively calculated as a \(\frac{1}{n}\) expansion.

To see how this works, let us first take a closer look at the two-point correlator Eq. (8) (which is automatically connected since we have normalized \(\int\mathcal{D}\phi\,e^{-\mathcal{S}}=1\), meaning the sum of vacuum bubbles vanishes). We can write it as an expansion in \(\frac{1}{n}\):

\[\big{\langle}\mathcal{G}^{(\ell)}\!(\vec{x}_{1},\vec{x}_{2})\big{\rangle}=\sum_ {p=0}^{\infty}\frac{1}{n_{\ell-1}^{p}}\,\mathcal{K}_{p}^{(\ell)}\!(\vec{x}_{1}, \vec{x}_{2})\,. \tag{10}\]

The leading-order (LO) term \(\mathcal{K}_{0}^{(\ell)}\) is known as the kernel; it is the propagator for \(\phi_{i}^{(\ell)}\) in the free-theory limit \(n\to\infty\). Evaluating \(\big{\langle}\mathcal{G}^{(\ell)}\!(\vec{x}_{1},\vec{x}_{2})\big{\rangle}\) in this limit amounts to using free-theory propagators \(\mathcal{K}_{0}^{(\ell-1)}\) for the previous
layer neurons \(\phi_{j}^{(\ell-1)}\) in the blob in Eq. (7):

\[\mathcal{K}_{0}^{(\ell)}(\vec{x}_{1},\vec{x}_{2}) =\sum_{j}\frac{1}{n_{\ell-1}}\big{\langle}\mathcal{G}_{j}^{(\ell)} (\vec{x}_{1},\vec{x}_{2})\big{\rangle}_{\mathcal{K}_{0}^{(\ell-1)}}\] \[=C_{b}^{(\ell)}+C_{W}^{(\ell)}\big{\langle}\sigma_{\vec{x}_{1}} \sigma_{\vec{x}_{2}}\big{\rangle}_{\mathcal{K}_{0}^{(\ell-1)}}\,. \tag{11}\]

Here subscript \(\mathcal{K}_{0}^{(\ell-1)}\) means the expectation value is computed with the free-theory propagator \(\mathcal{K}_{0}^{(\ell-1)}\) (_cf._ Eq. (15) in the Supplemental Material). We have dropped both neuron and layer indices on \(\sigma\) because the \(\mathcal{K}_{0}^{(\ell-1)}\) subscript already indicates the layer, and the expectation value is identical for all neurons in that layer. One can further evaluate \(\big{\langle}\sigma_{\vec{x}_{1}}\sigma_{\vec{x}_{2}}\big{\rangle}_{\mathcal{ K}_{0}^{(\ell-1)}}\big{\rangle}_{\mathcal{K}_{0}^{(\ell-1)}}\) for specific choices of activation functions \(\sigma\), but we stay activation-agnostic for the present analysis.

Eq. (11) allows us to recursively determine \(\mathcal{K}_{0}^{(\ell)}\) from \(\mathcal{K}_{0}^{(\ell-1)}\), and has been well-known from studies of infinite-width networks. It may also be viewed as the RG flow of \(\mathcal{K}_{0}\), with ultraviolet boundary condition \(\mathcal{K}_{0}^{(1)}(\vec{x}_{1},\vec{x}_{2})=C_{b}^{(1)}+\frac{C_{W}^{(1)} }{n_{0}}\,\vec{x}_{1}\!\cdot\!\vec{x}_{2}\). It is straightforward to extend the diagrammatic calculation to \(\mathcal{K}_{p\geq 1}\). We present a simple derivation of the RG flow of \(\mathcal{K}_{1}\) in the Supplemental Material.

Next, consider the connected four-point correlator:

\[\big{\langle}\phi_{i_{1}}^{(\ell)}(\vec{x}_{1})\,\phi_{i_{2}}^{( \ell)}(\vec{x}_{2})\,\phi_{i_{3}}^{(\ell)}(\vec{x}_{3})\,\phi_{i_{4}}^{(\ell)} (\vec{x}_{4})\big{\rangle}_{c}\] \[=\delta_{i_{1}i_{2}}\delta_{i_{3}i_{4}}\,\frac{1}{n_{\ell-1}}\,V_ {4}^{(\ell)}(\vec{x}_{1},\vec{x}_{2};\vec{x}_{3},\vec{x}_{4})+\text{perms.}, \tag{12}\]

where

\[\frac{1}{n_{\ell-1}}\,V_{4}^{(\ell)}(\vec{x}_{1},\vec{x}_{2};\vec{x}_{3},\vec {x}_{4})=\sum_{j_{1},j_{2}}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\, \
diagrams, each internal propagator connecting two correlators (blobs) is counted twice. To avoid double-counting we thus insert an _inverse_ propagator for each internal line in the diagram. This explains the factors of \(\left(\mathcal{K}_{0}^{(\ell-1)}\right)^{-1}\) in the first line of Eq. (15), which effectively amputate the connected four-point correlator (or equivalently the larger blobs in the diagram). The final expression in Eq. (15) is obtained by Wick contraction, which yields factors of \(\mathcal{K}_{0}^{(\ell-1)}\) that cancel \(\left(\mathcal{K}_{0}^{(\ell-1)}\right)^{-1}\).

Adding up Eqs. (14) and (15) gives the final result for \(V_{4}^{(\ell)}\) in terms \(V_{4}^{(\ell-1)}\) and \(\mathcal{K}_{0}^{(\ell-1)}\), _i.e._ the RG flow of \(V_{4}\), which agrees with Refs. [14; 21]. Both equations are \(\mathcal{O}(\frac{1}{n})\), so \(V_{4}^{(\ell)}\) defined by Eq. (12) is \(\mathcal{O}(1)\).

The diagrammatic calculation extends straightforwardly to higher-point connected correlators, and provides a concise framework to systematically analyze finite-width effects in deep neural networks. In the Supplemental Material we present new results for the connected six-point and eight-point correlators as further examples.

The RG flow can also be formulated at the level of the EFT action. The idea is to consider a tower of EFTs, \(\mathcal{S}_{\text{eff}}^{(\ell)}\) (\(\ell=1,\ldots,L\)), obtained by integrating out the neurons and ghosts in all but the \(\ell\) th layer. They take the form:

\[\mathcal{S}_{\text{eff}}^{(\ell)}=\int\!\!d\vec{x}_{1}d\vec{x}_{2 }\left(\mathcal{K}_{0}^{(\ell)}\!+\!\mu^{(\ell)}\right)^{\!-\!1}\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
sufficient to require the networks are well-behaved on average, but the scaling behavior of each network must be close to the average. At first sight, criticality seems to impose more constraints than the number of tunable hyperparameters, if we require power-law scaling of all higher-point correlators at arbitrary input points. However, as we will show, the structures of RG flow, manifest in the diagrammatic formulation, are such that tuning \(\mathcal{K}_{0}\) to criticality actually ensures power-law scaling of all higher-point connected correlators near the fixed point.

Let us start with the two-point correlator. The asymptotic scaling behavior (exponential vs. power-law) can be inferred from the following question: upon an infinitesimal variation at the \((\ell-1)\) th layer,

\[\big{\langle}\mathcal{G}^{(\ell\cdot 1)}(\vec{x}_{1},\vec{x}_{2})\big{\rangle} \to\big{\langle}\mathcal{G}^{(\ell\cdot 1)}(\vec{x}_{1},\vec{x}_{2})\big{\rangle}+ \delta\big{\langle}\mathcal{G}^{(\ell\cdot 1)}(\vec{x}_{1},\vec{x}_{2})\big{\rangle}\,, \tag{20}\]

how does \(\big{\langle}\mathcal{G}^{(\ell)}(\vec{x}_{1},\vec{x}_{2})\big{\rangle}\) change? Diagrammatically, this can be calculated as follows:

\[= \tag{21}\]

where a blob labeled "\(\delta\)" denotes the variation of the (two-point) correlator. The result is

\[\delta\big{\langle}\mathcal{G}^{(\ell)}(\vec{x}_{1},\vec{x}_{2}) \big{\rangle}=\int\!d\vec{y}_{1}d\vec{y}_{2}\,\chi^{(\ell)}(\vec{x}_{1},\vec{x }_{2};\vec{y}_{1},\vec{y}_{2})\] \[\delta\big{\langle}\mathcal{G}^{(\ell\cdot 1)}(\vec{y}_{1},\vec{y}_{2} )\big{\rangle}\,, \tag{22}\]

or, equivalently:

\[\frac{\delta\big{\langle}\mathcal{G}^{(\ell)}(\vec{x}_{1},\vec{x}_{2})\big{\rangle} }{\delta\big{\langle}\mathcal{G}^{(\ell\cdot 1)}(\vec{y}_{1},\vec{y}_{2}) \big{\rangle}}\,=\,\chi^{(\ell)}(\vec{x}_{1},\vec{x}_{2};\vec{y}_{1},\vec{y}_ {2})\,, \tag{23}\]

where

\[\chi^{(\ell)}(\vec{x}_{1},\vec{x}_{2};\vec{y}_{1},\vec{y}_{2})\] \[=\] \[= \frac{C_{W}^{(\ell)}}{2}\bigg{\langle}\frac{\delta^{2}\Delta( \vec{x}_{1},\vec{x}_{2})}{\delta\phi(\vec{y}_{1})\delta\phi(\vec{y}_{2})} \bigg{\rangle}_{\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!
where \(\chi_{\parallel}=\chi_{\perp}=1\). In the Supplemental Material, we show that, at least for the scale-invariant and \(\mathcal{K}^{\star}=0\) universality classes, this tuning actually implies a stronger condition is satisfied (at LO in \(\frac{1}{n}\)):

\[\chi^{(\ell)}(\vec{x}_{1},\vec{x}_{2};\vec{y}_{1},\vec{y}_{2}) \Big{|}_{\mathcal{K}_{0}^{(\ell\cdot 1)}=\mathcal{K}^{\star}}\] \[=\frac{1}{2}\Big{[}\delta(\vec{x}_{1}-\vec{y}_{1})\,\delta(\vec{ x}_{2}-\vec{y}_{2})+\delta(\vec{x}_{1}-\vec{y}_{2})\,\delta(\vec{x}_{2}-\vec{y}_{1 })\Big{]}. \tag{31}\]

Eq. (31) ensures perturbations around the fixed point stay constant through the layers, not just for the two-point correlator, \(\delta\big{\langle}\mathcal{G}^{(\ell)}(\vec{x}_{1},\vec{x}_{2})\big{\rangle}= \delta\big{\langle}\mathcal{G}^{(\ell\cdot 1)}(\vec{x}_{1},\vec{x}_{2})\big{\rangle}\), but for the entire tower of higher-point connected correlators, \(\frac{1}{n_{\ell-1}^{\ell}}\,\delta V_{2k}^{(\ell)}(\vec{x}_{1},\ldots,\vec{x} _{2k})=\frac{1}{n_{\ell-2}^{\ell}}\,\delta V_{2k}^{(\ell\cdot 1)}\big{(}\vec{x}_{1}, \ldots,\vec{x}_{2k}\big{)}\). This in turn implies that for all of them, RG flow toward the fixed point is power-law instead of exponential, once the single condition Eq. (31) is satisfied. The discussion above makes it transparent that power-law scaling of higher-point connected correlators at criticality (previously observed in Refs. [21; 22] up to eight-point level in the degenerate-input limit) has its roots in the structures of EFT interactions, as manifested by the common structure shared by the diagrams in Eqs. (21), (26) and (29).

Summary and outlook--In this letter, we introduced a diagrammatic formalism that significantly simplifies perturbative calculations of finite-width effects in EFTs corresponding to archetypical deep neural networks. The concise reproduction of known results and derivation of new results highlights the efficiency of the diagrammatic approach, while the incorporation of ghosts vastly simplifies \(\frac{1}{n}\) counting in the EFT action. Our analysis also made transparent the structures of such EFTs which underlie the success of critical tuning in deep neural networks. In fact, a universal diagrammatic structure emerges in the RG analysis of all higher-point connected correlators of neuron preactivations, which means criticality (_i.e._ power-law as opposed to exponential scaling) of all the neuron statistics at initialization is governed by a single condition, Eq. (31).

From the deep learning point of view, an obvious next step is to extend the diagrammatic formalism to incorporate gradient-based training and simplify perturbative calculations involving the neural tangent kernel [7; 8] and its differentials [11; 12; 13; 21]. From the fundamental physics point of view, we are hopeful that much more can be learned from the intimate connection between neural networks and field theories. Understanding the structures of EFTs corresponding to other neural network architectures (_e.g._ recurrent neural networks [32; 42] and transformers [43]) will allow us to gain further insights into this connection and potentially point to novel ML architecture designs for simulating field theories.

Acknowledgments--We are particularly grateful to Sho Yaida for helpful conversations throughout the course of this work. We thank Hannah Day, Marat Freyttsis, Boris Hanin, Yonatan Kahn, and Anindita Maiti for useful discussions and comments on a preliminary draft, and Guy Gur-Ari for related discussions. Feynman diagrams in this work were drawn using tikz-feynman [44]. This work was supported in part by the U.S. Department of Energy under the grant DE-SC0011702. This work was performed in part at the Aspen Center for Physics, supported by the National Science Foundation under Grant No. NSF PHY-2210452, and the Kavli Institute for Theoretical Physics, supported by the National Science Foundation under Grant No. NSF PHY-1748958.
