On the Disconnect Between Theory and Practice of Neural Networks: Limits
  of the NTK Perspective
The neural tangent kernel (NTK) has garnered significant attention as a
theoretical framework for describing the behavior of large-scale neural
networks. Kernel methods are theoretically well-understood and as a result
enjoy algorithmic benefits, which can be demonstrated to hold in wide synthetic
neural network architectures. These advantages include faster optimization,
reliable uncertainty quantification and improved continual learning. However,
current results quantifying the rate of convergence to the kernel regime
suggest that exploiting these benefits requires architectures that are orders
of magnitude wider than they are deep. This assumption raises concerns that
architectures used in practice do not exhibit behaviors as predicted by the
NTK. Here, we supplement previous work on the NTK by empirically investigating
whether the limiting regime predicts practically relevant behavior of
large-width architectures. Our results demonstrate that this is not the case
across multiple domains. This observed disconnect between theory and practice
further calls into question to what degree NTK theory should inform
architectural and algorithmic choices.
Jonathan Wenger, Felix Dangel, Agustinus Kristiadi
# On the Disconnect Between Theory and Practice of Overparametrized Neural Networks

###### Abstract

The infinite-width limit of neural networks (NNs) has garnered significant attention as a theoretical framework for analyzing the behavior of large-scale, overparametrized networks. By approaching infinite width, NNs effectively converge to a linear model with features characterized by the neural tangent kernel (NTK). This establishes a connection between NNs and kernel methods, the latter of which are well understood. Based on this link, theoretical benefits and algorithmic improvements have been hypothesized and empirically demonstrated in synthetic architectures. These advantages include faster optimization, reliable uncertainty quantification and improved continual learning. However, current results quantifying the rate of convergence to the kernel regime suggest that exploiting these benefits requires architectures that are orders of magnitude wider than they are deep. This assumption raises concerns that practically relevant architectures do not exhibit behavior as predicted via the NTK. In this work, we empirically investigate whether the limiting regime either describes the behavior of large-width architectures used in practice or is informative for algorithmic improvements. Our empirical results demonstrate that this is _not_ the case in optimization, uncertainty quantification or continual learning. This observed disconnect between theory and practice calls into question the practical relevance of the infinite-width limit.

## 1 Introduction

The behavior of large-scale, overparametrized neural networks (NNs) has for a long time been poorly understood theoretically. This is in stark contrast to their state-of-the-art performance on tasks like image classification (He et al., 2016; Zagoruyko and Komodakis, 2016), natural language processing (Devlin et al., 2019; Sun et al., 2019), as well as generative and sequence modeling (Brown et al., 2020; Touvron et al., 2023). The seminal work of Jacot et al. (2018) established a link between the evolution of NNs during training and kernel methods by considering networks with infinite width. In this limit, NNs effectively converge to linear models with fixed features such that their predictions are equivalent to those made by a Gaussian process (GP) model using the neural tangent kernel (NTK). Kernel methods and GPs are theoretically well-understood (Rasmussen and Williams, 2005). Consequently, this finding has led to a flurry of research interest in the NTK with the hope of an improved understanding of the behavior of NNs (e.g. Du et al., 2019; Zhou et al., 2020; Bowman and Montufar, 2022b; Mirzadeh et al., 2022).

Kernel methods enjoy several theoretical benefits which if applicable to NNs would be desirable. First, training a linear model or kernel regressor requires solving a quadratic optimization problem, which reduces to solving a linear system with the kernel matrix evaluated pairwise at the training data (Rasmussen and Williams, 2005). Conceptually this simplifies training significantly as the well-studied machinery of convex optimization and numerical linear algebra can be exploited. This is in contrast to the challenges of large-scale stochastic optimization, which compared to the convex setting suffers from slow convergence, requires manual tuning, and choosing an optimizer from a long list of available methods (Schmidt et al., 2021). Second, via the connection to GPs in the case of regression, uncertainty can be quantified via the posterior covariance defined through the NTK. As for prediction, uncertainty quantification then reduces to well-studied numerical methods (Rasmussen and Williams, 2005), unlike Bayesian NNs which generally suffer from similar issues as optimization (Zhang et al., 2020; Kristiadi et al., 2022). Third, data often becomes available continually and we
want to incorporate it into the model rather than retrain from scratch. This _continual learning_ setting in practice often leads to a drop in performance on previous tasks, known as _catastrophic forgetting_McCloskey and Cohen (1989); Goodfellow et al. (2013). It has been observed that large-scale over-parametrized networks forget less catastrophically Ramasesh et al. (2022); Mirzadeh et al. (2022) and this has been hypothesized to be a consequence of these NNs behaving according to the NTK regime. If that was the case, the amount of worst-case forgetting could be predicted theoretically Evron et al. (2022); Le et al. (2023) and algorithmically mitigated Bennani et al. (2020); Doan et al. (2021).

Given the advantageous network properties and algorithmic implications in terms of training, uncertainty quantification, and continual learning close to the kernel regime, the question becomes _when_ a network architecture satisfies the necessary assumptions. Loosely speaking, most theoretical results on NN convergence to a kernel regressor give rates of the form \(O(1/\sqrt{m})\) in the (minimum) width of the hidden layers \(m\)Du et al. (2019); Lee et al. (2019); Bowman and Montufar (2022). However, this asymptotic notation suppresses a dependence on the _network depth_\(L\), which is generally at least _polynomial_Arora et al. (2019) or even _exponential_Bowman and Montufar (2022). Even for quite shallow networks, this requires layer widths that are orders of magnitude larger than any of the common architectures (such as WideResNets). Figure 1 illustrates, that even shallow networks, if not sufficiently wide, can behave very differently from their infinite-width Gaussian process limit. This prompts the important question of whether assumptions based on the kernel regime, and methods derived thereof, apply to deep architectures that are used in practice.

**Contribution** In this work, we consider three areas for which the neural tangent kernel perspective on neural networks promises to be informative: optimization, uncertainty quantification and continual learning. We empirically evaluate whether the infinite-width regime either describes the behavior of large-width architectures used in practice or is useful for algorithm design. We find that in all three domains, assumptions based on NTK theory do _not_ translate to predictable phenomena or improved performance. This disconnect between theory and practice challenges the significance of overparametrization theory when applied to common architectures. We hope our negative findings can serve as a call to action for theory development and as a cautionary tale for algorithm design.

**Limitations** Our work studies architectures that are _currently_ being used in practice. This does _not_ mean that future architectures with large widths are not described well via the kernel regime. However, achieving competitive performance with wide architectures is a challenge, likely due to reduced representation learning Pleiss and Cunningham (2021); Zavatone-Veth et al. (2021); Noci et al. (2021); Coker et al. (2022). Further, we also do _not_ claim that methods we consider fail to be competitive in _any_ setting, rather just that their motivation via the kernel regime is unsuitable for practical architecture choices and problems. They may work well on specific choices of models and datasets.

## 2 Overparametrization Theory: An Overview

Let \(f:\mathcal{X}\times\Theta\rightarrow\mathcal{Y}\) be a neural network (NN) with input space \(\mathcal{X}\subseteq\mathbb{R}^{D}\), output space \(\mathcal{Y}\subseteq\mathbb{R}^{C}\) and parameter space \(\Theta\subseteq\mathbb{R}^{P}\). Assume we linearize \(f\) around a parameter vector \(\mathbf{\theta}_{0}\in\Theta\), i.e.

\[f(\mathbf{x};\mathbf{\theta})\approx f_{\text{lin}}(\mathbf{x};\mathbf{\theta})\coloneqq f( \mathbf{x};\mathbf{\theta}_{0})+\mathbf{J}(\mathbf{x};\mathbf{\theta}_{0})(\mathbf{\theta}-\mathbf{\theta }_{0}) \tag{1}\]

Figure 1: _Infinitely-wide NN in theory and its finite-width approximation in practice.1 The two models make very different predictions about the data-generating latent function, suggesting that the finite-width NN with a commonly selected architecture for real-world regression (\(L=3,m=128,P=21\)M, Li et al., 2023) is not well-described by the kernel regime. Increasing the width by an order of magnitude does not significantly improve the approximation via the empirical NTK._
where \(\mathbf{J}(\mathbf{x};\mathbf{\theta}_{0})\coloneqq(\nicefrac{{\partial f_{\mathbf{\theta}}(\mathbf{x})}}{{ \partial\mathbf{\theta}}})_{\mathbf{\theta}=\mathbf{\theta}_{0}}\in\mathbb{R}^{C\times P}\). When \(\mathbf{\theta}\) is close to \(\mathbf{\theta}_{0}\), the linear model \(f_{\mathrm{lin}}(\mathbf{x};\mathbf{\theta})\) with features defined by the network's Jacobian \(\mathbf{J}(\mathbf{x};\mathbf{\theta}_{0})\) is a good approximation of \(f(\mathbf{x};\mathbf{\theta})\). Consider a fully connected neural network \(f_{\mathrm{MLP}}(\mathbf{x};\mathbf{\theta})=\mathbf{h}^{L}(\mathbf{x}^{L-1})\) defined recursively as

\[\mathbf{h}^{\ell}(\mathbf{x}^{\ell-1})=\mathbf{W}^{\ell}\mathbf{x}^{\ell-1}+\mathbf{b}^{\ell}, \qquad\mathbf{x}^{\ell-1}(\mathbf{h}^{\ell-1})=\varphi(\mathbf{h}^{\ell-1}),\qquad\mathbf{x}^ {0}=\mathbf{x} \tag{2}\]

for \(\ell=L,\ldots,1\) with parameters \(\mathbf{\theta}=\{\mathbf{W}^{\ell}=\nicefrac{{1}}{{\sqrt{m_{\ell-1}}}}\mathbf{V}^{\ell} \}_{\ell=1}^{L}\cup\{\mathbf{b}^{\ell}\}_{\ell=1}^{L}\), s.t. \(\mathbf{V}^{\ell}_{ij},\mathbf{b}^{\ell}_{i}\sim\mathcal{N}(0,1)\), layer widths \(m_{\ell}\) and activation function \(\varphi\).2 Remarkably, Jacot et al. (2018) showed that for infinitely wide fully connected NNs, the parameters remain sufficiently close to their initialization \(\mathbf{\theta}_{0}\) during training via gradient descent. This means we can (approximately) understand the properties of a wide NN \(f_{\mathrm{MLP}}(\mathbf{x};\mathbf{\theta})\) by considering a much simpler-to-understand _linear_ model with features defined by the Jacobian \(\mathbf{J}(\mathbf{x};\mathbf{\theta}_{0})\) instead. Or more precisely, from a function space perspective, in the infinite width limit the behavior of a fully connected NN is described by the (deterministic) _neural tangent kernel_\(K_{\text{NTK}}\), defined as the limit in probability of the _finite-width_ or _empirical NTK_

Footnote 2: This normalized form of the weight matrices is known as the _NTK parametrization_ (NTP).

\[K^{\mathbf{\theta}}_{\text{eNTK}}(\mathbf{x},\mathbf{x}^{\prime})\coloneqq\mathbf{J}(\mathbf{x}; \mathbf{\theta})\mathbf{J}(\mathbf{x}^{\prime};\mathbf{\theta})^{\top}\stackrel{{ P}}{{\longrightarrow}}K_{\text{NTK}}(\mathbf{x},\mathbf{x}^{\prime}) \quad\text{as }m_{1},\ldots,m_{L}\to\infty.\]

This is known as the linear or _kernel regime_. In this regime, at initialization, the implicit prior over functions defined by the network is given by a Gaussian process \(\mathcal{GP}(0,K_{\text{NTK}})\) with zero-mean and covariance function defined by the NTK. Further, the (continuous-time) training dynamics of the network can be described via the differential equation \(\partial_{t}f(\mathbf{x};\mathbf{\theta}_{t})=-K_{\text{NTK}}(\mathbf{x},\mathbf{X})\nabla_{f} \mathcal{L}(f(\mathbf{X};\mathbf{\theta}_{t}))\) i.e. the optimization trajectory of \(f(\mathbf{x};\mathbf{\theta})\) is given by kernel gradient descent with respect to the loss function \(\mathcal{L}\). In the case of square loss regression on a training dataset \((\mathbf{X},\mathbf{y})\), this is a linear ODE, which in the limit of infinite training \(t\to\infty\) admits a closed-form solution. The network prediction is equivalent to the mean function

\[\lim_{t\to\infty}f(\mathbf{x};\mathbf{\theta}_{t})=\mu_{*}(\mathbf{x})\coloneqq f(\mathbf{x}; \mathbf{\theta}_{0})+K_{\text{NTK}}(\mathbf{x},\mathbf{X})K_{\text{NTK}}(\mathbf{X},\mathbf{X})^{ -1}(\mathbf{y}-f(\mathbf{X};\mathbf{\theta}_{0})) \tag{3}\]

of a GP posterior \(\mathcal{GP}(\mu_{*},K_{*})\), resulting from conditioning the prior \(\mathcal{GP}(0,K_{\text{NTK}})\) on observations \(\mathbf{y}=f_{*}(\mathbf{X})\) from the latent function \(f_{*}\) generating the data. These results for fully connected NNs have since been extended to nearly all architectures currently used in practice, such as CNNs, RNNs, and GNNs (Yang and Littwin, 2021).

Implications for Training, Uncertainty Quantification and Continual LearningThe connection to GP regression with the NTK demonstrates why the kernel regime is powerful as a theoretical framework. First, training a neural network simplifies to solving a linear system or equivalently a convex optimization problem (assuming \(K_{\text{NTK}}\) is positive (semi-)definite) since

\[K_{\text{NTK}}(\mathbf{X},\mathbf{X})^{-1}(\mathbf{y}-f(\mathbf{X};\mathbf{\theta}_{0}))=\arg\min_{ \mathbf{v}}\tfrac{1}{2}\mathbf{v}^{\top}K_{\text{NTK}}(\mathbf{X},\mathbf{X})\mathbf{v}-(\mathbf{y}-f( \mathbf{X};\mathbf{\theta}_{0}))^{\top}\mathbf{v} \tag{4}\]

which can be solved using well-understood, fast converging algorithms from numerical analysis (Nocedal and Wright, 2021). This is in contrast to the challenges of stochastic, non-convex optimization (Schmidt et al., 2021). Second, one often cited limitation of NNs is their lack of uncertainty quantification (Hein et al., 2019; Kristiadi et al., 2022; 2020). The connection to the posterior mean of a GP in the kernel regime when training to convergence (3) provides a strategy for Bayesian deep learning (Lee et al., 2018; Khan et al., 2019), by using the posterior covariance function

\[K_{*}(\mathbf{x},\mathbf{x}^{\prime})\coloneqq K_{\text{NTK}}(\mathbf{x},\mathbf{x}^{\prime})-K_ {\text{NTK}}(\mathbf{x},\mathbf{X})K_{\text{NTK}}(\mathbf{X},\mathbf{X})^{-1}K_{\text{NTK}}(\mathbf{ X},\mathbf{x}^{\prime}) \tag{5}\]

to quantify uncertainty. Finally, in a continual learning problem, the similarity between tasks in the kernel regime is measured via the NTK, which in turn describes the amount of catastrophic forgetting when training on new tasks (Doan et al., 2021).

Convergence to the Kernel RegimeWhen should we expect a neural network's behavior to be well-described by the NTK? We can characterize how quickly a network approaches the kernel regime as a function of its (minimum) width \(m=\min_{\ell\in\{1,\ldots,L-1\}}m_{\ell}\). The typical rate of convergence of the finite-width NTK at initialization to the NTK is

\[|K^{\mathbf{\theta}_{0}}_{\text{eNTK}}(\mathbf{x},\mathbf{x}^{\prime})-K_{\text{NTK}}(\mathbf{x}, \mathbf{x}^{\prime})|=\tilde{O}(\nicefrac{{1}}{{\sqrt{m}}}) \tag{6}\]
either pointwise (Du et al., 2019; Arora et al., 2019; Huang and Yau, 2020) or uniform (Buchanan et al., 2021; Bowman and Montufar, 2022; Bowman and Montufar, 2022; 2) with high probability. These results assume an _overparametrized_ NN with width \(m=\Omega(\mathrm{poly}(N))\) significantly exceeding the number of training datapoints \(N\).3 Note that the asymptotic notation in (6) suppresses a dependence on the (constant) depth \(L\) of the NN. This dependence of the width on the depth is _polynomial_ (e.g. \(m=\Omega(L^{6})\) in Arora et al., 2019) or even _exponential_(Bowman and Montufar, 2022), which suggests that to approach the kernel regime, a large network width is required already at moderate depth.

Footnote 3: Note, that Bowman and Montufar (2022) relax this requirement to \(m\approx N\) via the use of a stopping time.

For most architectures, an analytical/efficient-to-evaluate expression for the NTK is not known.4 Therefore in practice, the finite-width NTK \(K^{\mathbf{\theta}}_{\mathrm{NTK}}\approx K_{\text{NTK}}\) is used as an approximation. However as Fig. 1 illustrates, the prediction of a finite-width NN and the associated uncertainty given by the empirical NTK can be very different from the network's theoretical infinite-width limit. Therefore, making assumptions based on the kernel regime can potentially be misleading.

Footnote 4: A fully connected neural network with ReLU activations being a notable exception (Lee et al., 2018).

## 3 Connecting Theory and Practice

To empirically study whether the predictions from the kernel regime about the behavior of overparametrized NNs are reproducible in architectures used in practice, we take a closer look at training, uncertainty quantification and continual learning. For each of these, the kernel regime either makes predictions about the behavior of the network, motivates certain algorithms, or both.

### Training: Second-order Optimization

The empirical risk is typically a convex function of the network output, but generally non-convex in the network's parameters. In addition, stochastic approximations are often necessary due to memory constraints. However, for a NN close to the kernel regime, informally, the loss landscape becomes more convex, since the network approaches a linear model. In fact, for square loss the problem becomes quadratic, see (4). Using this intuition about the kernel regime, Du et al. (2019) have shown that gradient descent, a first-order method, can achieve zero training loss in spite of non-convexity. First-order methods, such as SGD and ADAM, are state-of-the-art for deep learning optimization (Schmidt et al., 2021). This is in contrast to "classical" convex optimization, in which second-order methods are favored due to their fast convergence (e.g. Nesterov, 2008, 2021). If NNs in practice are described well theoretically via the kernel regime, this may seem like a missed opportunity, since the near-convexity of the problem would suggest second-order optimizers to be an attractive choice.

There are multiple challenges for making second-order methods practical. They are less stable under noise--predominant in deep learning where data is fed in mini-batches--have higher per-iteration costs, and are often more complex to implement efficiently. Here, we investigate whether the theoretical argument in favor of second-order methods applies to real-world networks in that they are sufficiently close to the kernel regime. We exclude the aforementioned additional challenges, which amount to an entire research field (e.g. Martens and Grosse, 2015; Gupta et al., 2018; Ren and Goldfarb, 2021), since in the overparametrization regime approximate second-order methods that overcome these challenges can be shown to exhibit similar behavior than their deterministic counterparts (Karakida and Osawa, 2020).

**Fast Convergence of NGD in the Kernel Regime** To empirically test whether (practical) NNs can be efficiently optimized via second-order optimization as predicted by theory in the infinite-width limit, we consider natural gradient descent (NGD). Zhang et al. (2019) studied the convergence behavior of NGD theoretically. They give conditions for fast convergence of finite-step NGD, which extend to approximate NGD methods such as KFAC (Martens and Grosse, 2015). For simplicity, we focus on the special case of NNs with scalar-valued output and mean-squared error. Consider a network \(f(\mathbf{\theta},\mathbf{x}):=f(\mathbf{x};\mathbf{\theta})\) with flattened parameters \(\mathbf{\theta}\in\mathbb{R}^{P}\). For a dataset \(\{(\mathbf{x}_{n},y_{n})\}_{n=1}^{N}\), we minimize the empirical risk \(\mathcal{L}(\mathbf{\theta})=\nicefrac{{1}}{{2N}}\sum_{n=1,\ldots,N}\|\mathbf{f}-\mathbf{y }\|_{2}^{2}\) where \(\mathbf{f}(t):=(f(\mathbf{\theta}(t),\mathbf{x}_{1})\)\(\ldots\)\(f(\mathbf{\theta}(t),\mathbf{x}_{N}))^{\top}\in\mathbb{R}^{N}\) and \(\mathbf{y}:=(y_{1}\)\(\ldots\)\(y_{N})^{\top}\). Zhang et al. (2019) describe two conditions to ensure fast convergence of finite-step NGD to a global minimum which apply to _arbitrary_ architectures:
1. _Full row-rank of the network Jacobian_ \(\mathbf{J}(\mathbf{X};\mathbf{\theta}(0))\in\mathbb{R}^{N\times P}\) _at initialization_, implying \[\lambda_{\text{min}}(\mathbf{G}(0))>0,\] (7) where \(\mathbf{G}(0)=K_{\text{enTK}}^{\mathbf{\theta}(0)}(\mathbf{X},\mathbf{X})\) and restricting the trajectory to be close to initialization.5 Footnote 5: Note, that this implicitly assumes \(P\geq N\), i.e. overparametrization.
2. _Stable Jacobian_, \(\exists 0\leq C\leq\nicefrac{{1}}{{2}}\) such that \(\forall\mathbf{\theta}:\left\|\mathbf{\theta}-\mathbf{\theta}(0)\right\|_{2}\leq\rho \coloneqq\nicefrac{{3\|\mathbf{y}-\mathbf{f}(0)\|_{2}}}{{\sqrt{\lambda_{\text{min}}( \mathbf{G}(0))}}}\)

\[\left\|\mathbf{J}(\mathbf{\theta})-\mathbf{J}(0)\right\|_{2}\leq\tfrac{C}{3}\sqrt{\lambda_ {\text{min}}(\mathbf{G}(0))}\,. \tag{8}\]

The smaller \(C\), the 'closer' to linear the network is to initialization, with equality for \(C=0\).

We can evaluate both conditions in a scalable, matrix-free fashion using standard functions of automatic differentiation frameworks (see Section A.1). As a proxy for (8), we compute \(C^{\prime}(\mathbf{\theta})\coloneqq\nicefrac{{3\|\mathbf{J}(\mathbf{\theta})-\mathbf{J}(0)\|_ {2}}}{{\sqrt{\lambda_{\text{min}}(\mathbf{G}(0))}}}\) with \(\mathbf{\theta}\) drawn uniformly from a sphere with radius \(\rho\). If \(C^{\prime}(\mathbf{\theta})>\nicefrac{{1}}{{2}}\) for any \(\mathbf{\theta}\), then \(C\not\in[0;\nicefrac{{1}}{{2}}]\) and the network violates the stable Jacobian condition. Figures 2 and 3 summarize our findings which we now discuss in more detail.

**Shallow ReLU Net + Synthetic Data** We start with a synthetic problem for which Zhang et al. (2019) give theoretical guarantees. We generate a regression problem with \(N=16\) by i.i.d. drawing \(\mathbf{x}_{n}\in\mathbb{R}^{2}\sim\mathcal{U}([0;1^{2}]),\epsilon_{n}\in\mathbb{ R}\sim\mathcal{N}(0,1)\), and setting \(y_{n}=\sin(2\pi([\mathbf{x}_{n}]_{1}+[\mathbf{x}_{n}]_{2}))+0.1\epsilon_{n}\). Our model is a shallow two-layer ReLU net \(f(\mathbf{x},\mathbf{\theta})=\nicefrac{{1}}{{\sqrt{n}}}\mathbf{W}^{(2)}\mathrm{ReLU}( \mathbf{W}^{(1)}\mathbf{x})\) where \(\mathbf{W}^{(1)}\in\mathbb{R}^{m\times 2}\sim\mathcal{N}(\mathbf{0},\nu^{2}\mathbf{I})\) with \(\nu=1\), \(\mathbf{W}^{(2)}\in\mathbb{R}^{1\times m}\sim\mathcal{U}(\{-1,1\}^{m})\). Only \(\mathbf{W}^{(1)}\) is trainable and each input is normalized in the pre-processing stage, \(\mathbf{x}_{n}\leftarrow\nicefrac{{x_{n}}}{{\|\mathbf{x}_{n}\|_{2}}}\), to satisfy the theoretical assumptions. In this setting, Zhang et al. (2019) show that \(m=\Omega(N^{\star}\nicefrac{{\nu^{2}}}{{\lambda_{\text{min}}^{\star}}}\delta^ {s})\) is required for fast convergence of NGD with probability at least \(1-\delta\) and achieves an improvement of \(\mathcal{O}(\nicefrac{{\lambda_{0}}}{{N}})\) over GD.6 The Jacobian has full row-rank with high probability for \(m=\Omega(\nicefrac{{N\log(N/\delta)}}{{\lambda_{\text{0}}}})\) and we empirically observe a sharp increase in \(\lambda_{\text{min}}(\mathbf{G}(0))\) at relatively low widths (around \(m=500\)) in Fig. 2. However, the Jacobian stabilizes with \(\left\|\mathbf{J}(\mathbf{\theta})-\mathbf{J}(0)\right\|_{2}=\mathcal{O}(m^{-\nicefrac{{1} }{{\theta}}})\), and even for extreme widths (up to \(10^{7}\)) we observe that \(C^{\prime}>\nicefrac{{1}}{{2}}\), and therefore \(C>\nicefrac{{1}}{{2}}\).

Footnote 6: Here \(\lambda_{0}=\lambda_{\text{min}}(K_{\text{NTK}}(\mathbf{X},\mathbf{X}))\) is the minimum eigenvalue of the NTK from Du et al. (2019).

**Deep ReLU Net + Synthetic Data** Next, we move away from the kernel regime by adding depth to the architecture while keeping the same synthetic data and pre-processing. We use two fully connected NNs, as defined in (2), with \(L\in\{3,5\}\) layers of equal width and ReLU activations. For these models, scaling to large

Figure 3: More data _decreases_ the Jacobian stability for WideResNet on a subset of CIFAR-10.

Figure 2: _Conditions for fast convergence of NGD for different NN widths and problems_ (dots represent medians and shaded areas represent the 25/75-quantiles over three independent initializations and five parameter perturbations). The problems range from shallow (_theory_) ReLU MLP and depth-3 and -5 ReLU MLPs on a synthetic regression task to WideResNets on a sub-set of CIFAR-10 with \(N=400\) (_practice_). _Left:_ Relatively small widths are sufficient to satisfy the Gram matrix condition (7). _Right:_ None of the NNs achieve the required Jacobian stability (horizontal dashed line at \(\nicefrac{{1}}{{2}}\)) from (8) for any width both for synthetic and benchmark data.
widths is more expensive than for the shallow case, as the NN's parameters grow quadratically in \(m\). For both depths, we observe a sharp transition in \(\lambda_{\text{min}}(\mathbf{G}(0))\) at relatively small widths (around \(m=500\)) that are computationally accessible. In the close-to-linearity measure, we can see that depth increases non-linearity. While we observe a similar sharp transition in \(C^{\prime}\) to smaller values around \(m=500\) for both depths, its values remain well above \(\nicefrac{{1}}{{2}}\).

**CNN + Benchmark Data** Finally, we investigate a practical architecture (WideResNet, depth 10) on CIFAR-10. We convert the classification task into a regression problem for class indices and use a subset of the data. We rely on the implementation of Kuen (2017) and use its built-in widening factor, which scales the channels of the intermediate features within a block, as a measure for the network's width \(m\). In contrast to the previous cases, this net's Jacobian has a full row rank even for small widths. However, for larger widths attainable within our compute budget, \(C^{\prime}\) remains many orders of magnitude above \(\nicefrac{{1}}{{2}}\). And the stability further deteriorates when using more data (Fig. 3).

**Summary:** In the kernel regime, NGD has favorable convergence over GD in theory. However, empirically we find that the necessary conditions consistently do _not_ hold throughout problem scales--even for a shallow network with theoretical guarantees.

### Uncertainty Quantification: Neural Bandits

In sequential decision-making problems, not only predictive accuracy of a model is important, but crucially also accurate uncertainty quantification (Lattimore and Szepesvari, 2020; Garnett, 2023). Recently, the connection between infinitely wide NNs and GPs has been exploited to design neural bandit algorithms, whose guarantees rely on the assumption that the surrogate model \(f_{\mathbf{\theta}_{t}}\) is sufficiently close to the kernel regime (Zhou et al., 2020; Zhang et al., 2021; Kassaie and Krause, 2022; Nguyen-Tang et al., 2022). We empirically test whether this assumption holds in practice.

**Neural Contextual Bandits via the Kernel Regime** Our goal is to sequentially take optimal actions with regard to an unknown time-varying reward function \(r_{t}(a_{t},\mathbf{x}_{t})\in\mathbb{R}\) which depends on an action-context pair \((a_{t},\mathbf{x}_{t})\in\{1,\ldots,K\}\times\mathbb{R}^{n}\) where \(t=1,\ldots,T\). To do so, we learn a surrogate \(f_{\mathbf{\theta}_{t}}(a_{t},\mathbf{x}_{t})\approx r_{t}(a_{t},\mathbf{x}_{t})\) approximating the reward from past data \(\mathcal{D}_{t}=\{(a_{t^{\prime}},\mathbf{x}_{t^{\prime}}),r_{t^{\prime}}(a_{t^{ \prime}},\mathbf{x}_{t^{\prime}})\}_{t^{\prime}=1}^{t-1}\). An action \(\widetilde{a}_{t}=\arg\max_{a_{t}}u(a_{t},\mathbf{x}_{t})\) is then selected based on a _utility function_\(u\) which generally depends on both the prediction and uncertainty of the neural surrogate for the reward. Overall we want to minimize the _cumulative regret_\(R(T)=\sum_{t=1}^{T}\!r_{t}(\widetilde{a}_{t},\mathbf{x}_{t})-r_{t}(a_{t}^{*},\mathbf{x}_ {t})\), where \(a_{t}^{*}\) are the optimal actions, and the reward \(r_{t}(\widetilde{a}_{t},\mathbf{x}_{t})\) is only observable once an action \(\widetilde{a}_{t}\) is taken. Here we use the popular UCB (Auer, 2002) utility function

\[u(a_{t},\mathbf{x}_{t})=f_{\mathbf{\theta}_{t}}(a_{t},\mathbf{x}_{t})+\gamma_{t}\sqrt{ \operatorname{var}f_{\mathbf{\theta}_{t}}(a_{t},\mathbf{x}_{t})}, \tag{9}\]

where \(\gamma_{t}>0\) controls the exploration-exploitation tradeoff. Due to the importance of uncertainty quantification in the selection of an action based on \(u(a_{t},\mathbf{x}_{t})\), GPs have been used extensively as surrogates (Krause and Ong, 2011). Here, we consider neural surrogates instead, which quantify uncertainty via the empirical NTK at a MAP estimate \(\mathbf{\theta}_{t}\). This can be thought of as a finite-width approximation to the limiting GP in the kernel regime, or equivalently from a weight space view as a linearized Laplace approximation (LLA, MacKay, 1992; Khan et al., 2019; Immer et al., 2021b; Daxberger et al., 2021; Kristiadi et al., 2023b).

**Exploration-Exploitation Trade-off** The parameter \(\gamma_{t}\) in (9) controlling the exploration-exploitation trade-off strongly impacts the cumulative regret, making its choice an important problem in practice. Recent works prove (near-)optimal regret bounds for the neural bandit setting by choosing \(\gamma_{t}\) based on the kernel regime (Zhou et al., 2020; Kassaie and Krause, 2022). To approach the kernel regime, the convergence results discussed in Section 2 require the width of the network \(m\) to be polynomial in the depth \(L\) and number of training data \(N=t-1\). This poses the question whether the proposed choice of \(\gamma_{t}\) is useful in practice. Here, we consider the NeuralUCB algorithm proposed by Zhou et al. (2020), where the exploration parameter \(\gamma_{t}=\tilde{O}(\mathrm{poly}(\nicefrac{{1}}{{\sqrt{m}}},L,t))\). We find that even for shallow NNs (\(L=3\)), \(\gamma_{t}\) rapidly grows very large (see Fig. 4), which by (9) results in essentially no exploitation, only exploration. This suggests that for \(\gamma_{t}\) to achieve a non-vacuous value, \(m\) must be potentially unfeasibly large.

Figure 4: Setting \(\gamma_{t}\) via NTK theory results in overexploration in practice.
**Experiment Setup** We empirically test whether the assumptions based on the kernel regime in the neural bandit setting result in good performance in practice for realistic architectural choices. We use standard contextual bandit benchmark problems, based on UCI classification datasets (Zhou et al., 2020; Zhang et al., 2021; Gupta et al., 2022) (see Section A.2). We compare (i) a _random_ baseline policy and various neural UCB baselines, (ii) the UCB policy with _constant_ exploration parameter \(\gamma_{t}\equiv\gamma\in\{0.01,0.1,1,10\}\) as for simplicity often used in practice, (iii) the UCB policy where \(\gamma_{t}\) is set via the NTK theory with widths \(m\in\{100,1024\}\)(Zhou et al., 2020), and (iv) setting \(\gamma_{t}\equiv 1\), but leveraging the connection between the (empirical) NTK and the LLA in Bayesian deep learning (Immer et al., 2021) to learn a prior precision hyperparameter via marginal likelihood both _post-hoc_ and _online_(Immer et al., 2021).7

Footnote 7: Computing the evidence in the LLA setting incurs no overhead since the LA itself is an approximation of _both_ the posterior \(p(\mathbf{\theta}\mid\mathcal{D}_{t})\)_and_ the marginal likelihood \(p(\mathcal{D}_{t})=\int p(\mathcal{D}_{t}\mid\mathbf{\theta})\,p(\mathbf{\theta})\,d \mathbf{\theta}\)(MacKay, 1992).

**Experiment Results** The results of our experiment are given in Fig. 5, which shows the cumulative regret \(R(t)\) over time. Perhaps frustratingly, the NTK-based policy performs poorly, oftentimes no better than the random baseline, with an order of magnitude larger width having no discernable effect. This is likely explained by the overexploration problem discussed previously. Therefore, in this setting, relying on assumptions based on the kernel regime results in a poorly performing algorithm in practice. In fact, Zhou et al. (2020) set \(\gamma_{t}\) to be constant in their experiments instead of according to the proposed (near-)optimal value based on NTK theory. This disconnect between NTK theory and practice can also be observed for other utility functions such as expected improvement (Gupta et al., 2022) and Thompson sampling (Zhang et al., 2021). We find that setting \(\gamma_{t}\equiv\gamma\) to a value with a well-chosen \(\gamma\) performs best in our experiments (see also Fig. 6 top) However, the optimal value of \(\gamma\) is _unknown a-priori_ and can only be obtained via grid search. This can be problematic in a real-world setting, where a sufficiently large, representative, validation set may not be available, and multiple experiments may not be possible prior to running the "real" experiment--it defeats the spirit of _online_ learning. With that in mind, the marginal-likelihood-based choice of \(\gamma_{t}\) both _post-hoc_ and online perform well in terms of their cumulative regret. While using grid search provides better results in terms of rank (Fig. 6 top), the difference in terms of the cumulative regret \(R(t)\) is small for all \(t\)--see Fig. 6 bottom. The minimal difference in cumulative regret between the marginal-likelihood-based strategies and the best strategy suggests that learning a good exploration-exploitation trade-off is possible, but likely _not_ via an algorithm motivated via the kernel regime.

**Summary:** Avoid setting the exploration parameter in eNTK-based neural bandits via the NTK theory. Instead, use the toolbox of the Laplace approximation to optimize the scale of the posterior variance via evidence maximization.

Figure 5: _Cumulative regret of neural bandits with different degrees of exploration \((\gamma_{t})_{t}\)_ on benchmark data. Setting the exploration parameter \(\gamma_{t}\) via NTK theory yields second-worst performance after random search. Constant exploration achieves the best results but the optimal \(\gamma_{t}\equiv\gamma=10^{-2}\) is a-priori unknown. Online marginal-likelihood (ML) calibration performs near-optimally.
### Continual Learning: Catastrophic Forgetting

In many applications, such as robotics, NNs need to be trained continually on new _tasks_, given by a sequence of training datasets. The primary challenge in _continual learning_(Thrun and Mitchell, 1995; Parisi et al., 2019) is to train on new tasks without a significant loss of performance on previous tasks, known as _catastrophic forgetting_(McCloskey and Cohen, 1989; Goodfellow et al., 2013).

Catastrophic Forgetting in the Kernel RegimeAssuming the NN is sufficiently wide to be in the linear regime, worst-case forgetting can be described theoretically, as well as the convergence to an offline solution--i.e. training on data from all tasks at once (Evron et al., 2022; Evron et al., 2023). One way to algorithmically avoid forgetting is _orthogonal gradient descent_(OGD, Farajtabar et al., 2020), which projects gradients on new tasks such that model predictions on previous tasks change minimally. Bennani et al. (2020) show that, in the kernel regime, OGD provably avoids catastrophic forgetting on an arbitrary number of tasks (assuming infinite memory). Additionally, for SGD and OGD generalization bounds have been given, which are based on the task similarity with respect to the NTK (Bennani et al., 2020; Doan et al., 2021). Ramasesh et al. (2022) investigated catastrophic forgetting empirically in the pretraining paradigm and found that forgetting systematically decreases with scale of both model and pretraining dataset size. Mirzadeh et al. (2022) report that increasing the width of a neural network reduces catastrophic forgetting significantly as opposed to increasing the depth. The hypothesis for this is that as the model becomes wider, gradients across tasks become increasingly orthogonal, and training becomes "lazy", meaning the initial parameters change very little during training, consistent with the convergence of the empirical NTK at initialization to the NTK (6). This naturally leads to the question of whether increasing the width of networks that are used in practice is in fact a simple way to mitigate catastrophic forgetting.

Experiment SetupTo test whether the predictions about continual learning in the kernel regime apply in practice, we train increasingly wide NNs on a sequence of tasks. We train toy two-layer NNs with ReLU activations on the RotatedMNIST dataset, as well as WideResNets (Zagoruyko and Komodakis, 2016) on the SplitCIFAR10, SplitCIFAR100 and SplitTinyImageNet datasets. See Section A.3 for details. Our main goal is to study the effect of width on forgetting. Let \(a_{t,i}\) denote test accuracy on task \(i\) after training on task \(t\). We compute the _average forgetting_\(\bar{\phi}_{T}=\nicefrac{{1}}{{T}}-\nicefrac{{1}}{{\sum_{i=1}^{T}}}\max_{t \in\{1,\ldots,T-1\}}(a_{t,i}-a_{T,i})\), i.e. the average maximal accuracy difference during task-incremental training; the _average accuracy_\(\bar{\alpha}_{T}=\nicefrac{{1}}{{T}}\sum_{i=1}^{T}a_{T,i}\), i.e. the average accuracy across tasks after training on all tasks; and the _learning accuracy_\(\bar{\alpha}_{\max}=\nicefrac{{1}}{{T}}\sum_{i=1}^{T}a_{i,i}\), i.e. the average accuracy across tasks immediately after training on the current task.8 To ascertain whether a network operates in the lazy training/kernel regime, we also track the relative distance in parameter space \(d(\mathbf{w}_{T},\mathbf{w}_{0})=\nicefrac{{\|\mathbf{w}_{T}-\mathbf{w}_{0}\|_{2}}}{{\|\mathbf{ w}_{0}\|_{2}}}\) between the initial parameters \(\mathbf{w}_{0}\) and the parameters \(\mathbf{w}_{T}\) after training on all tasks.

Footnote 8: In practice, \(\bar{\alpha}_{\max}\) almost always equals the average maximum accuracy per task, justifying the notation.

Experiment ResultsThe results of our experiment are shown in Fig. 7. We find that for the shallow NN, average forgetting decreases monotonically with the network width. Further, the relative change in parameters \(d(\mathbf{w}_{T},\mathbf{w}_{0})\) approaches zero, consistent with the lazy training hypothesis in the kernel regime. This seems to confirm observations by Mirzadeh et al. (2022) that wide neural networks forget less. However, for WideResNets we find that a crucial confounding factor is whether the network has been fully trained. NNs that are trained insufficiently show a decrease in forgetting as they become wider. But, this decrease is primarily due to lower learning accuracy, and thus a smaller gap between peak accuracy and minimum accuracy across tasks (see Section B.2). In contrast, training networks to high accuracy increases average forgetting since the peak performance across tasks increases. This can be explained by the fact that they are not actually operating in the kernel regime. The relative change of the weights during training remains large even as width increases beyond what is used in practice, meaning the networks are still adapting to unseen tasks.

Summary: Increasing the width of a practical NN architecture only avoids catastrophic forgetting if not trained to high accuracy per task. Since a smaller change in the weights of the network throughout training correlates with reduced forgetting, strategies that constrain parameter updates when training on new tasks (e.g. Kirkpatrick et al., 2017) or along directions which minimally change performance on previous tasks (e.g. OGD) promise to be more useful strategies in practice than increasing network width.
## 4 Conclusion

In this work, we empirically evaluated whether predictions about the behavior of overparametrized neural networks through the theoretical framework of the neural tangent kernel hold in architectures used in practice. We considered three different areas in which the kernel regime either makes predictions about the behavior of a neural network or informs algorithmic choices. We find that across optimization, uncertainty quantification, and continual learning, theoretical statements in the infinite-width limit do not translate to observable phenomena or improvements in practical architectures with realistic widths. For optimization, we found that such architectures are not sufficiently close to linear to enjoy fast convergence from a second-order method as predicted by existing theory. For uncertainty quantification, we found that controlling the exploration-exploitation trade-off in a sequential decision-making problem via assumptions based on the kernel regime led to performance only marginally better than a random baseline. Finally, in continual learning, we found that wide neural networks as used in practice, if fully trained, do not actually forget less catastrophically.

This observed disconnect between theory and practice leads to two important conclusions. First, our theoretical understanding of the behavior of large-scale overparametrized neural networks is still limited and in particular restricted to architectures that do not resemble those used in practice. This paper is empirical evidence to that effect and thus calls for an effort to improve our understanding by developing a theory under more practically relevant assumptions. Second, algorithms motivated by the neural tangent kernel theory should be scrutinized closely in terms of their practical performance, and researchers should be careful in basing their ideas too strongly on the infinite-width limit. We hope in this way our negative results can serve as a cautionary tale and will ultimately benefit _both_ the theory and practice of deep neural networks.

Figure 7: _Catastrophic forgetting of wide NNs in theory and practice._ As the width of the shallow MLP approaches the kernel regime, average forgetting decreases, while average accuracy increases. Similar observations hold for WideResNets if trained for a few of epochs only—consistent with experiments by Mirzadeh et al. (2022). However, if they are trained to convergence on each task, resulting in increased learning accuracy, _forgetting does not decrease with width_. This suggests that architectures in practice are not actually wide enough to reduce forgetting via the kernel regime.
#### Acknowledgments

JW was supported by the Gatsby Charitable Foundation (GAT3708), the Simons Foundation (542963) and the Kavli Foundation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
