Mind the spikes: Benign overfitting of kernels and neural networks in
  fixed dimension
The success of over-parameterized neural networks trained to near-zero
training error has caused great interest in the phenomenon of benign
overfitting, where estimators are statistically consistent even though they
interpolate noisy training data. While benign overfitting in fixed dimension
has been established for some learning methods, current literature suggests
that for regression with typical kernel methods and wide neural networks,
benign overfitting requires a high-dimensional setting where the dimension
grows with the sample size. In this paper, we show that the smoothness of the
estimators, and not the dimension, is the key: benign overfitting is possible
if and only if the estimator's derivatives are large enough. We generalize
existing inconsistency results to non-interpolating models and more kernels to
show that benign overfitting with moderate derivatives is impossible in fixed
dimension. Conversely, we show that rate-optimal benign overfitting is possible
for regression with a sequence of spiky-smooth kernels with large derivatives.
Using neural tangent kernels, we translate our results to wide neural networks.
We prove that while infinite-width networks do not overfit benignly with the
ReLU activation, this can be fixed by adding small high-frequency fluctuations
to the activation function. Our experiments verify that such neural networks,
while overfitting, can indeed generalize well even on low-dimensional data
sets.
Moritz Haas, David Holzmüller, Ulrike von Luxburg, Ingo Steinwart
# Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension

###### Abstract

The success of over-parameterized neural networks trained to near-zero training error has caused great interest in the phenomenon of benign overfitting, where estimators are statistically consistent even though they interpolate noisy training data. While benign overfitting in fixed dimension has been established for some learning methods, current literature suggests that for regression with typical kernel methods and wide neural networks, benign overfitting requires a high-dimensional setting where the dimension grows with the sample size. In this paper, we show that the smoothness of the estimators, and not the dimension, is the key: benign overfitting is possible if and only if the estimator's derivatives are large enough. We generalize existing inconsistency results to non-interpolating models and more kernels to show that benign overfitting with moderate derivatives is impossible in fixed dimension. Conversely, we show that benign overfitting is possible for regression with a sequence of spiky-smooth kernels with large derivatives. Using neural tangent kernels, we translate our results to wide neural networks. We prove that while infinite-width networks do not overfit benignly with the ReLU activation, this can be fixed by adding small high-frequency fluctuations to the activation function. Our experiments verify that such neural networks, while overfitting, can indeed generalize well even on low-dimensional data sets.

## 1 Introduction

While neural networks have shown great practical success, our theoretical understanding of their generalization properties is still limited. A promising line of work considers the phenomenon of benign overfitting, where researchers try to understand when and how models that interpolate noisy training data can generalize (Zhang et al., 2021; Belkin et al., 2018, 2019). In the high-dimensional regime, where the dimension grows with the number of sample points, consistency of minimum-norm interpolants has been established for linear models and kernel regression (Hastie et al., 2022; Bartlett et al., 2020; Liang and Rakhlin, 2020; Bartlett et al., 2021). In fixed dimension, minimum-norm interpolation with standard kernels is inconsistent (Rakhlin and Zhai, 2019; Buchholz, 2022).

In this paper, we shed a differentiated light on benign overfitting with kernels and neural networks. We argue that the dimension-dependent perspective does not capture the full picture of benign overfitting. In particular, we show that harmless interpolation with kernel methods and neural networks is possible, even in small fixed dimension, with adequately designed kernels and activation functions. The key is to properly design estimators of the form'signal+spike'. While minimum-norm criteria have widely been considered a useful inductive bias, we demonstrate that designing unusual norms can resolve the shortcomings of standard norms. For wide neural networks, harmless interpolation can be realized by adding tiny fluctuations to the activation function. Such networks do not require regularization and can simply be trained to overfit (Figure 1).
On a technical level, we additionally prove that overfitting in kernel regression can only be consistent if the estimators have large derivatives. Using neural tangent kernels or neural network Gaussian process kernels, we can translate our results from kernel regression to the world of neural networks (Neal, 1996; Jacot et al., 2018). In particular, our results enable the design of activation functions that induce benign overfitting in fixed dimension: the spikes in kernels can be translated into infinitesimal fluctuations that can be added to an activation function to achieve harmless interpolation with neural networks. Such small high frequency oscillations can fit noisy observations without affecting the smooth component too much. Training finite neural networks with gradient descent shows that spiky-smooth activation functions can indeed achieve good generalization even when interpolating small, low-dimensional data sets (Figure 1 b,c).

Thanks to new technical contributions, our inconsistency results significantly extend existing ones. We use a novel noise concentration argument (Lemma D.6) to generalize existing inconsistency results on minimum-norm interpolants to the much more realistic regime of overfitting estimators with comparable Sobolev norm scaling, which includes training via gradient flow and gradient descent with "late stopping" as well as low levels of ridge regression. Moreover, a novel connection to eigenvalue concentration results for kernel matrices (Proposition 4) allows us to relax the smoothness assumption and to treat heteroscedastic noise in Theorem 5. Lastly, our Lemma E.1 translates inconsistency results from bounded open subsets of \(\mathbb{R}^{d}\) to the sphere \(\mathbb{S}^{d}\), which leads to results for the neural tangent kernel and neural network Gaussian processes.

## 2 Setup and prerequisites

**General approach.** We consider a general regression problem on \(\mathbb{R}^{d}\) with an arbitrary, fixed dimension \(d\) and analyze kernel-based approaches to solve this problem: kernel ridge regression, kernel gradient flow and gradient descent, minimum-norm interpolation, and more generally, overfitting norm-bounded estimators. We then translate our results to neural networks via the neural network Gaussian process and the neural tangent kernel. Let us now introduce the formal framework.

**Notation.** We denote scalars by lowercase letters \(x\), vectors by bold lowercase letters \(x\) and matrices by bold uppercase letters \(X\). We denote the eigenvalues of \(A\) as \(\lambda_{1}(\mathbf{A})\geq\ldots\geq\lambda_{n}(\mathbf{A})\) and the Moore-Penrose pseudo-inverse by \(\mathbf{A}^{+}\). We say that a probability distribution \(P\) has lower and upper bounded density if its density \(p\) satisfies \(0<c<p(\mathbf{x})<C\) for suitable constants \(c,C\) and all \(x\) on a given domain.

**Regression setup.** We consider a data set \(D=((\mathbf{x}_{1},y_{1}),\ldots,(\mathbf{x}_{n},y_{n}))\in( \mathbb{R}^{d}\times\mathbb{R})^{n}\) with i.i.d. samples \((\mathbf{x}_{i},y_{i})\sim P\), written as \(D\sim P^{n}\), where \(P\) is a probability distribution on \(\mathbb{R}^{d}\times\mathbb{R}\). We define \(\mathbf{X}\coloneqq(\mathbf{x}_{1},\ldots,\mathbf{ x}_{n})\) and \(\mathbf{y}\coloneqq(y_{1},\ldots,y_{n})^{\top}\in\mathbb{R}^{n}\). Random variables \((\mathbf{x},y)\sim P\) denote test points

Figure 1: **Spiky-smooth overfitting in 2 dimensions.****a.** We plot the predicted function for ridgeless kernel regression with the Laplace kernel (blue) versus our spiky-smooth kernel (4) with Laplace components (orange) on \(\mathbb{S}^{1}\). The dashed black line shows the true regression function, black ’x’ denote noisy training points. Further details can be found in Section 6.2. **b.** The predicted function of a trained 2-layer neural network with ReLU activation (blue) versus ReLU plus shifted high-frequency sin-function (8) (orange). Using the weights learned with the spiky-smooth activation function in a ReLU network (green) disentangles the spike component from the signal component. **c.** Training error (solid lines) and test error (dashed lines) over the course of training for b. evaluated on \(10^{4}\) test points. The dotted black line shows the optimal test error. The spiky-smooth activation function does not require regularization and can simply be trained to overfit.
independent of \(D\), and \(P_{X}\) denotes the probability distribution of \(\mathbf{x}\). The (least squares) _empirical risk_\(R_{D}\) and _population risk_\(R_{P}\) of a function \(f:\mathbb{R}^{d}\to\mathbb{R}\) are defined as

\[R_{D}(f)\coloneqq\frac{1}{n}\sum_{i=1}^{n}(y_{i}-f(x_{i}))^{2},\qquad R_{P}(f) \coloneqq\mathbb{E}_{\mathbf{x},y}[(y-f(\mathbf{x}))^{2}]\.\]

We assume \(\operatorname{Var}(y|\mathbf{x})<\infty\) for all \(\mathbf{x}\). Then, \(R_{P}\) is minimized by the target function \(f_{P}^{*}(\mathbf{x})=\mathbb{E}[y|\mathbf{x}]\), and the _excess risk_ of a function \(f\) is given by

\[R_{P}(f)-R_{P}(f_{P}^{*})=\mathbb{E}_{\mathbf{x}}(f_{P}^{*}(\mathbf{x})-f(\mathbf{x}))^{2}\.\]

We call a data-dependent estimator \(f_{D}\)_consistent for \(P\)_ if its excess risk converges to \(0\) in probability, that is, for all \(\varepsilon>0\), \(\lim_{n\to\infty}P^{n}\left(D\in(\mathbb{R}^{d}\times\mathbb{R})^{n}\ |\ \ R_{P}(f_{D})-R_{P}(f_{P}^{*})\geq \varepsilon\right)=0.\) We call \(f_{D}\)_consistent in expectation for \(P\)_ if \(\lim_{n\to\infty}\mathbb{E}_{D}R_{P}(f_{D})-R_{P}(f_{P}^{*})=0\). We call \(f_{D}\)_universally consistent_ if is it consistent for all Borel probability measures \(P\) on \(\mathbb{R}^{d}\times\mathbb{R}\).

**Solutions by kernel regression.** Recall that a kernel \(k\) induces a reproducing kernel Hilbert space \(\mathcal{H}_{k}\), abbreviated RKHS (more details in Appendix B). For \(f\in\mathcal{H}_{k}\), we consider the objective

\[\mathcal{L}_{\rho}(f)\coloneqq\frac{1}{n}\sum_{i=1}^{n}(y_{i}-f(\mathbf{x}_{i}))^ {2}+\rho\|f\|_{\mathcal{H}_{k}}^{2}\]

with regularization parameter \(\rho\geq 0\). Denote by \(f_{t,\rho}\) the solution to this problem that is obtained by optimizing on \(\mathcal{L}_{\rho}\) in \(\mathcal{H}_{k}\) with gradient flow until time \(t\in[0,\infty]\), using fixed a regularization constant \(\rho>0\), and initializing at \(f=0\in\mathcal{H}_{k}\). We show in Appendix C.1 that it is given as

\[f_{t,\rho}(\mathbf{x})\coloneqq k(\mathbf{x},\mathbf{X})\left(\mathbf{I}_{n}-e^{-\frac{2}{n}t (k(\mathbf{X},\mathbf{X})+\rho n\mathbf{I}_{n})}\right)\left(k(\mathbf{X},\mathbf{X})+\rho n\mathbf{I} _{n}\right)^{-1}\mathbf{y}\, \tag{1}\]

where \(k(\mathbf{x},\mathbf{X})\) denotes the row vector \((k(\mathbf{x},\mathbf{x}_{i}))_{i\in[n]}\) and \(k(\mathbf{X},\mathbf{X})=(k(\mathbf{x}_{i},\mathbf{x}_{j}))_{i,j\in[n]}\) the kernel matrix. \(f_{t,\rho}\) elegantly subsumes several popular kernel regression estimators as special cases: (i) classical kernel ridge regression for \(t\to\infty\), (ii) gradient flow on the unregularized objective for \(\rho\searrow 0\), and (iii) kernel "ridgeless" regression \(f_{\infty,0}(\mathbf{x})=k(\mathbf{x},\mathbf{X})k(\mathbf{X},\mathbf{X})^{+}\mathbf{y}\) in the joint limit of \(\rho\to 0\) and \(t\to\infty\). If \(k(\mathbf{X},\mathbf{X})\) is invertible, \(f_{\infty,0}\) is the interpolating function \(f\in\mathcal{H}_{k}\) with the smallest \(\mathcal{H}_{k}\)-norm.

**From kernels to neural networks: the neural tangent kernel (NTK) and the neural network Gaussian process (NNGP).** Denote the output of a NN with parameters \(\mathbf{\theta}\) on input \(\mathbf{x}\) by \(f_{\mathbf{\theta}}(\mathbf{x})\). It is known that for suitable random initializations \(\mathbf{\theta}_{0}\), in the infinite-width limit the random initial function \(f_{\mathbf{\theta}_{0}}\) converges in distribution to a Gaussian Process with the so-called Neural Network Gaussian Process (NNGP) kernel (Neal, 1996; Lee et al., 2018; Matthews et al., 2018). In Bayesian inference, the posterior mean function is then of the form \(f_{\infty,\rho}\). With minor modifications (Arora et al., 2019; Zhang et al., 2020), training infinitely wide NNs with gradient flow corresponds to learning the function \(f_{t,0}\) with the neural tangent kernel (NTK) (Jacot et al., 2018; Lee et al., 2019). If only the last layer is trained, the NNGP kernel should be used instead (Daniely et al., 2016). For ReLU activation functions, the RKHS of the infinite-width NNGP and NTK on the sphere \(\mathbb{S}^{d}\) is typically a Sobolev space (Bietti and Bach, 2021; Chen and Xu, 2021), see Appendix B.4.

## 3 Related work

We here provide a short summary of related work. A more detailed account is provided in Appendix A.

**Kernel regression.** With appropriate regularization, kernel ridge regularization with typical universal kernels like the Gauss, Matern, and Laplace kernels is universally consistent (Steinwart and Christmann, 2008, Chapter 9). Optimal rates in Sobolev RKHS can also be achieved using cross-validation of the regularization \(\rho\)(Steinwart et al., 2009) or early stopping rules (Yao et al., 2007; Raskutti et al., 2014; Wei et al., 2017). In the high-dimensional regime, the class of functions that is learnable with rotation-invariant kernels is quite limited (Donhauser et al., 2021; Ghorbani et al., 2021; Liang et al., 2020).
**Inconsistency results.** Besides Rakhlin and Zhai (2019) and Buchholz (2022), Beaglehole et al. (2022) derive inconsistency results for ridgeless kernel regression given assumptions on the spectral tail in the Fourier basis, and Li et al. (2023) show that polynomial convergence is impossible for common kernels including ReLU NTKs. Mallinar et al. (2022) conjecture inconsistency for interpolation with ReLU NTKs based on their semi-rigorous result, which essentially assumes that the eigenfunctions can be replaced by structureless Gaussian random variables. Lai et al. (2023) show an inconsistency-type result for overfitting two-layer ReLU NNs with \(d=1\), but for fixed inputs \(\mathbf{X}\). They also note that an earlier inconsistency result by Hu et al. (2021) relies on an unproven result. Mucke and Steinwart (2019) show that global minima of NNs can overfit both benignly and harmfully, but their result does not apply to gradient descent training. Overfitting with typical linear models around the interpolation peak is inconsistent (Ghosh and Belkin, 2022; Holzmuller, 2021).

**Classification.** For binary classification, benign overfitting is a more generic phenomenon than for regression (Muthukumar et al., 2021; Shamir, 2022), and consistency has been shown under linear separability assumptions (Montanari et al., 2019; Chatterji and Long, 2021; Frei et al., 2022), through complexity bounds for reference classes (Cao and Gu, 2019; Chen et al., 2019) or as long as the total variation distance of the class conditionals is sufficiently large and \(f^{*}(\mathbf{x})=\mathbb{E}[y|\mathbf{x}]\) lies in the RKHS with bounded norm (Liang and Recht, 2023). Chapter 8 of Steinwart and Christmann (2008) discusses how the overlap of the two classes may influence learning rates under positive regularization.

## 4 Inconsistency of overfitting with common kernel estimators

We consider a regression problem on \(\mathbb{R}^{d}\) in arbitrary, fixed dimension \(d\) that is solved by kernel regression. In this section, we derive several new results, stating that overfitting estimators with moderate Sobolev norm are inconsistent, in a variety of settings. In the next section, we establish the other direction: overfitting estimators can be consistent when we adapt the norm that is minimized.

### Beyond minimum-norm interpolants: general overfitting estimators with bounded norm

Existing generalization bounds often consider the perfect minimum norm interpolant. This is a rather theoretical construction; estimators obtained by training with gradient descent algorithms merely overfit and, in the best case, approximate interpolants with small norm. In this section, we extend existing bounds to arbitrary overfitting estimators whose norm does not grow faster than the minimum norm that would be required to interpolate the training data. Before we can state the theorem, we need to establish some technical assumptions.

Assumptions on the data generating process.The following assumptions (as in Buchholz (2022)) allow for quite general domains and distributions. They are standard in nonparametric statistics.

1. Let \(P_{X}\) be a distribution on a bounded open Lipschitz domain \(\Omega\subseteq\mathbb{R}^{d}\) with lower and upper bounded Lebesgue density. Consider data sets \(D=\{(\mathbf{x}_{1},y_{1}),\ldots,(\mathbf{x}_{n},y_{n})\}\), where \(\mathbf{x}_{i}\sim P_{X}\) i.i.d. and \(y_{i}=f^{*}(\mathbf{x}_{i})+\varepsilon_{i}\), where \(\varepsilon_{i}\) is i.i.d. Gaussian noise with positive variance \(\sigma^{2}>0\) and \(f^{*}\in C_{c}^{\infty}(\Omega)\backslash\{0\}\) denotes a smooth function with compact support.

Assumptions on the kernel.Our assumption on the kernel is that its RKHS is equivalent to a Sobolev space. For integers \(s\in\mathbb{N}\), the norm of a Sobolev space \(H^{s}(\Omega)\) can be defined as

\[\|f\|_{H^{s}(\Omega)}^{2}:=\sum_{0\leq|u|\leq s}\|D^{\alpha}f\|_{L_{2}(\Omega)} ^{2},\]

where \(D^{\alpha}\) denotes partial derivatives in multi-index notation for \(\alpha\). It measures the magnitude of derivatives up to some order \(s\). For general \(s>0\), \(H^{s}(\Omega)\) is (equivalent to) an RKHS if and only if \(s>d/2\). For example, Laplace and Matern kernels (Kanagawa et al., 2018, Example 2.6) have
Sobolev RKHSs. The RKHS of the Gaussian kernel \(\mathcal{H}^{\mathrm{Gauss}}\) is contained in every Sobolev space, \(\mathcal{H}^{\mathrm{Gauss}}\subsetneq H^{s}\) for all \(s\geq 0\)(Steinwart and Christmann, 2008, Corollary 4.36). Due to its smoothness, the Gaussian kernel is potentially even more prone to harmful overfitting than Sobolev kernels (Mallinar et al., 2022). We make the following assumption on the kernel:

1. Let \(k\) be a positive definite kernel function whose RKHS \(\mathcal{H}_{k}\) is equivalent to the Sobolev space \(H^{s}\) for \(s\in(\frac{d}{2},\frac{3d}{4}]\).

Now we are ready to state the main result of this section:

**Theorem 1** (**Overfitting estimators with small norms are inconsistent**).: _Let assumptions (D1) and (K) hold. Let \(c_{\mathrm{fit}}\in(0,1]\) and \(C_{\mathrm{norm}}>0\). Then, there exist \(c>0\) and \(n_{0}\in\mathbb{N}\) such that the following holds for all \(n\geq n_{0}\) with probability \(1-O(1/n)\) over the draw of the data set \(D\) with \(n\) samples: Every function \(f\in\mathcal{H}_{k}\) that satisfies the following two conditions_

1. \(\frac{1}{n}\sum_{i=1}^{n}(f(x_{i})-y_{i})^{2}\leq(1-c_{\mathrm{fit}})\cdot \sigma^{2}\) _(training error of_ \(f\) _is below Bayes risk)_
2. \(\|f\|_{\mathcal{H}_{k}}\leq C_{\mathrm{norm}}\|f_{\infty,0}\|_{\mathcal{H}_{k}}\) _(norm comparable to minimum-norm interpolant (_1_)),_

_has an excess risk that satisfies_

\[R_{P}(f)-R_{P}(f^{*})\geq c>0. \tag{2}\]

In words: In fixed dimension \(d\), every differentiable function \(f\) that overfits the training data and is not much "spikier" than the minimum RKHS-norm interpolant is inconsistent!

**Proof idea.** Our proof follows a similar approach as Rakhlin and Zhai (2019); Buchholz (2022), and also holds for kernels with adaptive bandwidths. For small bandwidths, \(\|f_{\infty,0}\|_{L_{2}(P_{X})}\) is too small, because \(f_{\infty,0}\) decays to \(0\) between the training points, which shows that purely "spiky" estimators are inconsistent. For all other bandwidths, interpolating \(\Theta(n)\) many noisy labels \(y_{i}\) incurs \(\Theta(1)\) error in an area of volume \(\Omega(1/n)\) around \(\Theta(n)\) data points with high probability, which accumulates to a total error \(\Omega(1)\). Our observation is that the same logic holds when overfitting by a constant fraction. Formally, we show that \(f^{*}\) and \(f\) must then be separated by a constant on a constant fraction of training points, with high probability, by using the fact that a constant fraction of the total noise cannot concentrate on less than \(\Theta(n)\) noise variables, with high probability (Lemma D.6). The full proof can be found in Appendix D. 

Assumption (O) is necessary in Theorem 1, because optimally regularized kernel ridge regression fulfills all other assumptions of Theorem 1 while achieving consistency with minimax optimal convergence rates (see Section 3). The necessity of Assumption (N) is demonstrated by Section 5.

The following proposition establishes that Theorem 1 covers the entire overfitting regime of the popular (regularized) gradient flow estimators \(f_{t,\rho}\) for all times \(t\in[0,\infty]\) and any regularization \(\rho\geq 0\). The proof in Appendix C.2 also covers gradient descent.

**Proposition 2** (**Popular estimators fulfill the norm bound (N)**).: _Let \(t\in[0,\infty]\) and let \(\rho\geq 0\) arbitrary. Then \(f_{t,\rho}\) as defined in (1) fulfills Assumption (N) with \(C_{\mathrm{norm}}=1\)._

### Inconsistency of overfitting with neural kernels

We would now like to apply the above results to neural kernels, which would allow us to translate our inconsistency results from the kernel domain to neural networks. However, to achieve this, we need to take one more technical hurdle: the equivalence results for NTKs and NNGPs only hold for probability distributions on the sphere \(\mathbb{S}^{d}\) (detailed summary in Appendix B.4). Lemma E.1 provides the missing technical link: It establishes a smooth correspondence between the respective kernels, Sobolev spaces, and probability distributions. The inconsistency of overfitting with (deep) ReLU NTKs and NNGP kernels then immediately follows from adapting Theorem 1 via Lemma E.1.

**Theorem 3** (**Overfitting with neural network kernels in fixed dimension is inconsistent**).: _Let \(c\in(0,1)\), and let \(P\) be a probability distribution with lower and upper bounded Lebesgue density on an arbitrary spherical cap \(T\coloneqq\{\mathbf{x}\in\mathbb{S}^{d}\mid x_{d+1}<v\}\subseteq\mathbb{S}^{d}\), \(v\in(-1,1)\). Let \(k\) either be_

1. _the fully-connected ReLU NTK with_ \(0\)_-initialized biases of any fixed depth_ \(L\geq 2\)_, and_ \(d\geq 2\)_, or_
_(ii) the fully-connected ReLU NNGP kernel without biases of any fixed depth \(L\geq 3\), and \(d\geq 6\)._

_Then, if \(f_{t,\rho}\) fulfills Assumption \((O)\) with probability at least \(c\) over the draw of the data set \(D\), \(f_{t,\rho}\) is inconsistent for \(P\)._

Theorem 3 also holds for more general estimators as in Theorem 1, cf. the proof in Appendix E.

Mallinar et al. (2022) already observed empirically that overfitting common network architectures yields suboptimal generalization performance on large data sets in fixed dimension. Theorem 3 now provides a rigorous proof for this phenomenon since sufficiently wide trained neural networks and the corresponding NTKs have a similar generalization behavior (e.g. (Arora et al., 2019, Theorem 3.2)).

### Relaxing smoothness and noise assumptions via spectral concentration bounds

In this section, we consider a different approach to derive lower bounds for the generalization error of overfitting kernel regression: through concentration results for the eigenvalues of kernel matrices. On a high level, we obtain similar results as in the last section. The novelty of this section is on the technical side, and we suggest that non-technical readers skip this section in their first reading.

We define the convolution kernel of a given kernel \(k\) as \(k_{*}(\mathbf{x},\mathbf{x}^{\prime})\coloneqq\int k(\mathbf{x},\mathbf{x}^{\prime\prime})k( \mathbf{x}^{\prime\prime},\mathbf{x}^{\prime})\,\mathrm{d}P_{X}(\mathbf{x}^{\prime\prime})\), which is possible whenever \(k(\mathbf{x},\cdot)\in L_{2}(P_{X})\) for all \(\mathbf{x}\). The latter condition is satisfied for bounded kernels. Our starting point is the following new lower bound:

**Proposition 4** (**Spectral lower bound**).: _Assume that the kernel matrix \(k(\mathbf{X},\mathbf{X})\) is almost surely positive definite, and that \(\mathrm{Var}(y|\mathbf{x})\geq\sigma^{2}\) for \(P_{X}\)-almost all \(\mathbf{x}\). Then, the expected excess risk satisfies_

\[\mathbb{E}_{D}R_{P}(f_{t,\rho})-R_{P}^{*}\geq\frac{\sigma^{2}}{n}\sum_{i=1}^{ n}\mathbb{E}_{\mathbf{X}}\frac{\lambda_{i}(k_{*}(\mathbf{X},\mathbf{X})/n)\left(1-e^{-2t( \lambda_{i}(k(\mathbf{X},\mathbf{X})/n)+\rho)}\right)^{2}}{(\lambda_{i}(k(\mathbf{X},\mathbf{ X})/n)+\rho)^{2}}. \tag{3}\]

Using concentration inequalities for kernel matrices and the relation between the integral operators of \(k\) and \(k_{*}\), it can be seen that for \(t=\infty\) and \(\rho=0\), every term in the sum in Eq. (3) should converge to \(1\) as \(n\to\infty\). However, since the number of terms in the sum increases with \(n\) and the convergence may not be uniform, this is not sufficient to show inconsistency in expectation. Instead, relative concentration bounds that are even stronger than the ones by Valdivia (2018) would be required to show inconsistency in expectation. However, by combining multiple weaker bounds and further arguments on kernel equivalences, we can still show inconsistency in expectation for a class of dot-product kernels on the sphere, including certain NTK and NNGP kernels (Appendix B.4):

**Theorem 5** (**Inconsistency for Sobolev dot-product kernels on the sphere**).: _Let \(k\) be a dot-product kernel on \(\mathbb{S}^{d}\), i.e., a kernel of the form \(k(\mathbf{x},\mathbf{x}^{\prime})=\kappa(\langle\mathbf{x},\mathbf{x}^{\prime}\rangle)\), such that its RKHS \(\mathcal{H}_{k}\) is equivalent to a Sobolev space \(H^{s}(\mathbb{S}^{d})\), \(s>d/2\). Moreover, let \(P\) be a distribution on \(\mathbb{S}^{d}\times\mathbb{R}\) such that \(P_{X}\) has a lower and upper bounded density w.r.t. the uniform distribution \(\mathcal{U}(\mathbb{S}^{d})\), and such that \(\mathrm{Var}(y|\mathbf{x})\geq\sigma^{2}>0\) for \(P_{X}\)-almost all \(\mathbf{x}\in\mathbb{S}^{d}\). Then, for every \(C>0\), there exists \(c>0\) independent of \(\sigma^{2}\) such that for all \(n\geq 1\), \(t\in(C^{-1}n^{2s/d},\infty]\), and \(\rho\in[0,Cn^{-2s/d})\), the expected excess risk satisfies_

\[\mathbb{E}_{D}R_{P}(f_{t,\rho})-R_{P}^{*}\geq c\sigma^{2}>0\.\]

The assumptions of Theorem 5 and Theorem 3 differ in several ways. Theorem 5 applies to arbitrarily high smoothness \(s\) and therefore to ReLU NTKs and NNGPs in arbitrary dimension \(d\). Moreover, it applies to distributions on the whole sphere and allows more general noise distributions. On the flip side, it only shows inconsistency in expectation, which we believe could be extended to inconsistency for Gaussian noise. Moreover, it only applies to functions of the form \(f_{t,\rho}\) but provides an explicit bound on \(t\) and \(\rho\) to get inconsistency. For \(t=\infty\), the bound \(\rho=O(n^{-2s/d})\) appears to be tight, as larger \(\rho\) yield consistency for comparable Sobolev kernels on \(\mathbb{R}^{d}\)(Steinwart et al., 2009, Corollary 3).

The spectral lower bounds in Theorem F.2 show that our approach can directly benefit from developing better kernel matrix concentration inequalities. Conversely, the investigation of consistent kernel interpolation might provide information about where such concentration inequalities do not hold.
Consistency via spiky-smooth estimators - even in fixed dimension

In Section 4, we have seen that when common kernel estimators overfit, they are inconsistent for many kernels and a wide variety of distributions. We now design consistent interpolating kernel estimators. The key is to violate Assumption (N) and allow for quickly exploding derivatives.

### Almost universal consistency of spiky-smooth ridgeless kernel regression

In high dimensional regimes (where the dimension \(d\) is supposed to grow with the number of data points), benign overfitting of linear and kernel regression has been understood by an additive decomposition of the minimum-norm interpolant into a smooth regularized component that is responsible for good generalization, and a spiky component that interpolates the noisy data points while not harming generalization (Bartlett et al., 2021). This inspires us to enforce such a decomposition in arbitrary fixed dimension by adding a sharp kernel spike \(\rho\tilde{k}_{\gamma_{n}}\) to a common kernel \(\tilde{k}\). In this way, we can still generate any Sobolev RKHS (see Appendix G.2).

**Definition 6** (Spiky-smooth kernel).: Let \(\tilde{k}\) denote any universal kernel function on \(\mathbb{R}^{d}\). We call it the smooth component. Consider a second, translation invariant kernel \(\tilde{k}_{\gamma}\) of the form \(k_{\gamma}(\mathbf{x},\mathbf{y})=q(\frac{\mathbf{x}-\mathbf{y}}{\gamma})\), for some function \(q:\mathbb{R}^{d}\to\mathbb{R}\). We call it the spiky component. Then we define the \(\rho\)_-regularized spiky-smooth kernel with spike bandwidth \(\gamma\)_ as

\[k_{\rho,\gamma}(\mathbf{x},\mathbf{y})=\tilde{k}(\mathbf{x},\mathbf{y})+\rho\cdot\tilde{k}_{ \gamma}(\mathbf{x},\mathbf{y}),\qquad\mathbf{x},\mathbf{y}\in\mathbb{R}^{d}. \tag{4}\]

We now show that the minimum-norm interpolant of the spiky-smooth kernel sequence with properly chosen \(\rho_{n},\gamma_{n}\to 0\) is consistent for a large class of distributions, on a space with fixed (possibly small) dimension \(d\). We establish our result under the following assumption (as in Mucke and Steinwart (2019)), which is weaker than our previous Assumption (D1).

* There exists a constant \(\beta_{X}>0\) and a continuous function \(\phi:[0,\infty)\to[0,1]\) with \(\phi(0)=0\) such that the data generating probability distribution satisfies \(P_{X}(B_{t}(\mathbf{x}))\leq\phi(t)=O(t^{\beta_{X}})\) for all \(\mathbf{x}\in\Omega\) and all \(t\geq 0\) (here \(B_{t}(\mathbf{x})\) denotes the Euclidean ball of radius \(t\) around \(\mathbf{x}\)).

**Theorem 7** (Consistency of spiky-smooth ridgeless kernel regression).: _Assume that the training set \(D\) consists of \(n\) i.i.d. pairs \((\mathbf{x},y)\sim P\) such that the marginal \(P_{X}\) fulfills (D2) and \(\mathbb{E}y^{2}<\infty\). Let the kernel components satisfy:_

* \(\tilde{k}\) _is a universal kernel, and_ \(\rho_{n}\to 0\) _and_ \(n\rho_{n}^{4}\to\infty\)_._
* \(\tilde{k}_{\gamma_{n}}\) _denotes the Laplace kernel with a sequence of positive bandwidths_ \((\gamma_{n})\) _fulfilling_ \(\gamma_{n}=O\left(n^{-(2+\alpha)/\beta_{X}}/\log(n)\right)\)_, where_ \(\alpha>0\) _arbitrary._

_Then the minimum-norm interpolant of the \(\rho_{n}\)-regularized spiky-smooth kernel sequence \(k_{n}\coloneqq k_{\rho_{n},\gamma_{n}}\) is consistent for \(P\)._

Proof idea.With sharp spikes \(\gamma\to 0\), it holds that \(\tilde{k}_{\gamma}(\mathbf{X},\mathbf{X})\approx\mathbf{I}_{n}\), with high probability. Hence, ridgeless kernel regression with the spiky-smooth kernel interpolates the training set while approximating kernel ridge regression with the smooth component \(\tilde{k}\) and regularization \(\rho\). 

The theorem even holds under much weaker assumptions on the decay behavior of the spike component \(\tilde{k}_{\gamma_{n}}\), including Gaussian and Matern kernels. The full version of the theorem and its proof can be found in Appendix G. It also applies to kernels and distributions on the sphere \(\mathbb{S}^{d}\).

Figure 2: The spiky-smooth kernel with Laplace components (orange) consists of a Laplace kernel (blue) plus a Laplace kernel of height \(\rho\) and small bandwidth \(\gamma\).
### From spiky-smooth kernels to spiky-smooth activation functions

So far, our discussion revolved around the properties of kernels and whether they lead to estimators that are consistent. We now turn our attention to the neural network side. The big question is whether it is possible to specifically design activation functions that enable benign overfitting in fixed, possibly small dimension. We will see that the answer is yes: similarly to adding sharp spikes to a kernel, we add tiny fluctuations to the activation function. Concretely, we exploit (Simon et al., 2022, Theorem 3.1). It states that any dot-product kernel on the sphere that is a dot-product kernel in every dimension \(d\) can be written as an NNGP kernel or an NTK of two-layer fully-connected networks with a specifically chosen activation function. Further details can be found in Appendix H.

**Theorem 8** (**Connecting kernels and activation functions**(Simon et al., 2022)).: _Let \(\kappa:[-1,1]\to\mathbb{R}\) be a function such that \(k_{d}:\mathbb{S}^{d}\times\mathbb{S}^{d}\to\mathbb{R},k_{d}(\mathbf{x},\mathbf{x}^{ \prime})=\kappa(\langle\mathbf{x},\mathbf{x}^{\prime}\rangle)\) is a kernel for every \(d\geq 1\). Then, there exist \(b_{i}\geq 0\) with \(\sum_{i=0}^{\infty}b_{i}<\infty\) such that \(\kappa(t)=\sum_{i=0}^{\infty}b_{i}t^{i}\), and for any choice of signs \((s_{i})_{i\in\mathbb{N}}\subseteq\{-1,+1\}\), the kernel \(k_{d}\) can be realized as the NNGP or NTK of a two-layer fully-connected network with activation function_

\[\phi_{NNGP}^{k_{d}}(x)=\sum_{i=0}^{\infty}s_{i}(b_{i})^{1/2}h_{i}(x),\qquad \phi_{NTK}^{k_{d}}(x)=\sum_{i=0}^{\infty}s_{i}\left(\frac{b_{i}}{i+1}\right)^{ 1/2}h_{i}(x). \tag{5}\]

_Here, \(h_{i}\) denotes the \(i\)-th Probabilist's Hermite polynomial normalized such that \(\|h_{i}\|_{L_{2}(\mathcal{N}(0,1))}=1\)._

The following proposition justifies the approach of adding spikes \(\rho^{1/2}\phi^{\mathds{k}_{\gamma}}\) to an activation function to enable harmless interpolation with wide neural networks. Here we state the result for the case of the NTK; an analogous result holds for induced NNGP activation functions.

**Proposition 9** (**Additive decomposition of spiky-smooth activation functions**).: _Fix \(\tilde{\gamma},\rho>0\) arbitrary. Let \(k=\tilde{k}+\rho\tilde{k}_{\gamma}\) denote the spiky-smooth kernel where \(\tilde{k}\) and \(\tilde{k}_{\gamma}\) are Gaussian kernels of bandwidth \(\tilde{\gamma}\) and \(\gamma\), respectively. Assume that we choose signs \(\{s_{i}\}_{i\in\mathbb{N}}\) and then the activation functions \(\phi_{NTK}^{k}\), \(\phi_{NTK}^{\tilde{k}}\) and \(\phi_{NTK}^{\tilde{k}}\) as in Theorem 8. Then, for \(\gamma>0\) small enough, it holds that_

\[\|\phi_{NTK}^{k}-(\phi_{NTK}^{\tilde{k}}+\sqrt{\rho}\cdot\phi_{NTK}^{\tilde{k} _{\gamma}})\|_{L_{2}(\mathcal{N}(0,1))}^{2}\leq 2^{1/2}\rho\gamma^{3/2}\exp \left(-\frac{1}{\gamma}\right)+\frac{4\pi(1+\tilde{\gamma})\gamma}{\tilde{ \gamma}}.\]

**Proof idea.** When the spikes are sharp enough (\(\gamma\) small enough), the smooth and the spiky component of the activation function are approximately orthogonal in \(L_{2}(\mathcal{N}(0,1))\) (Figure 3c), so that the spiky-smooth activation function can be approximately additively decomposed into the smooth activation component \(\phi^{\tilde{k}}\) and the spike component \(\phi^{\tilde{k}}\) responsible for interpolation. 

To motivate why the added spike functions \(\rho^{1/2}\phi^{\tilde{k}_{\gamma}}\) should have small amplitudes, observe that Gaussian activation components \(\phi^{\tilde{k}_{\gamma}}\) satisfy

\[\|\phi_{NNGP}^{\tilde{k}_{\gamma}}\|_{L_{2}(\mathcal{N}(0,1))}^{2}=1,\qquad\| \phi_{NTK}^{\tilde{k}_{\gamma}}\|_{L_{2}(\mathcal{N}(0,1))}^{2}=\frac{\gamma}{ 2}\left(1-\exp\left(-\frac{2}{\gamma}\right)\right). \tag{6}\]

Hence, the average amplitude of NNGP spike activation components \(\rho^{1/2}\phi^{\tilde{k}_{\gamma}}\) does not depend on \(\gamma\), while the average amplitude of NTK spike components decays to \(0\) with \(\gamma\to 0\). Since consistency requires the quasi-regularization \(\rho\to 0\), the spiky component of induced NTK as well as NNGP activation functions should vanish for large data sets \(n\to\infty\) to achieve consistency.

## 6 Experiments

Now we explore how appropriate spiky-smooth activation functions might look like and whether they indeed enable harmless interpolation for trained networks of finite width on finite data sets. Further experimental results are reported in Appendix I.1
### What do common activation functions lack in order to achieve harmless interpolation?

To understand which properties we have to introduce into activation functions to enable harmless interpolation, we plot NTK spike components \(\phi^{k_{\gamma}}\) induced by the Gaussian kernel (Figure 3a,b) as well as their Hermite series coefficients (Figure 3c). Remarkably, the spike components \(\phi^{k_{\gamma}}\) approximately correspond to a shifted, high-frequency \(\sin\)-curve, when choosing the signs \(s_{i}\) in (5) to alternate every second \(i\), that is \(s_{i}=+1\) iff \(\lfloor i/2\rfloor\) even (Figure 3a). We empirically determine (Appendix I.6) that the NNGP activation functions are well approximated by the fluctuation function

\[\omega_{\text{NNGP}}(x;\gamma)\coloneqq\sqrt{2}\cdot\sin\left(\sqrt{2/\gamma }\cdot x+\pi/4\right)=\sin\left(\sqrt{2/\gamma}\cdot x\right)+\cos\left(\sqrt{ 2/\gamma}\cdot x\right), \tag{7}\]

where the last equation follows from the trigonometric addition theorem. For small bandwidths \(\gamma\), the NTK activation functions are increasingly well approximated by

\[\omega_{\text{NTK}}(x;\gamma)\coloneqq\sqrt{\gamma}\cdot\sin\left(\sqrt{2/ \gamma}\cdot x+\pi/4\right)=\sqrt{\gamma/2}\left(\sin\left(\sqrt{2/\gamma} \cdot x\right)+\cos\left(\sqrt{2/\gamma}\cdot x\right)\right). \tag{8}\]

With decreasing bandwidth \(\gamma\to 0\) the frequency increases, while the amplitude decreases for the NTK and remains constant for the NNGP (see Eq. (6)). Plotting equivalent spike components \(\phi^{k_{\gamma}}\) with different choices of the signs \(s_{i}\) (Figure 3b and Appendix I.5) suggests that harmless interpolation requires activation functions that contain **small high-frequency oscillations** or that **explode at large**\(|x|\), which only affects few neurons. The Hermite series expansion of suitable activation functions should contain **non-negligible weight spread across high-order coefficients** (Figure 3c). While Simon et al. (2022) already truncate the Hermite series of induced activation functions at order 5, Figure 3c shows that an accurate approximation of spiky-smooth activation functions requires the truncation index to be larger than \(2/\gamma\). Only a careful implementation allows us to capture the high-order fluctuations in the Hermite series of the spiky activation functions.

### Training neural networks to achieve harmless interpolation in low dimension

In Figure 1, we plot the results of (a) ridgeless kernel regression and (b) trained 2-layer neural networks with standard choices of kernels and activation functions (blue) as well as our spiky-smooth alternatives (orange). We trained on 15 points sampled i.i.d. from \(x=(x_{1},x_{2})\sim\mathcal{U}(\mathbb{S}^{1})\) and \(y=x_{1}+\varepsilon\) with \(\varepsilon\sim\mathcal{N}(0,0.25)\). The figure shows that both the Laplace kernel and standard ReLU networks interpolate the training data too smoothly in low dimension, and do not generalize well. However, our spiky-smooth kernel and neural networks with spiky-smooth activation functions achieve close to optimal generalization while interpolating the training data with sharp spikes. We achieve this by using the adjusted activation function with high-frequency oscillations \(x\mapsto\text{ReLU}(x)+\omega_{\text{NTK}}(x;\frac{1}{5000})\) as defined in Eq. (8). With this choice, we avoid activation functions with exploding behavior,

Figure 3: **a., b. Gaussian NTK activation components \(\phi^{k_{\gamma}}_{NTK}\) defined via (5) induced by the Gaussian kernel with varying bandwidth \(\gamma\in[0.2,0.1,0.05]\) (the darker, the smaller \(\gamma\)) for **a.** bi-alternating signs \(s_{i}=+1\) iff \(\lfloor i/2\rfloor\) even, and **b.** randomly iid chosen signs \(s_{i}\sim\mathcal{U}(\{-1,+1\})\). **c.** Coefficients of the Hermite series of a Gaussian NTK activation component with varying bandwidth \(\gamma\). Observe peaks at \(2/\gamma\). For reliable approximations of activation functions use a truncation \(\geq 4/\gamma\). The sum of squares of the coefficients follows Eq. (6). Figure 1.8 visualizes NNGP activation components.
which would induce exploding gradients. Other choices of amplitude and frequency in Eq. (8) perform worse. Over the course of training (Figure 0(c)), the standard ReLU network exhibits harmful overfitting, whereas the NN with a spiky-smooth activation function quickly interpolates the training set with nearly optimal generalization. Training details and hyperparameter choices can be found in Appendix I.1. Although the high-frequency oscillations perturb the gradients, the NN with spiky smooth activation has a stable training trajectory using gradient descent with a large learning rate of 0.4 or stochastic gradient descent with a learning rate of 0.04. Since our activation function is the sum of two terms, we can additively decompose the network into its ReLU-component and its \(\omega_{\text{NTK}}\)-component. Figure 0(b) and Appendix I.2 demonstrate that our interpretation of the \(\omega_{\text{NTK}}\)-component as'spiky' is accurate: The oscillations in the hidden neurons induced by \(\omega_{\text{NTK}}\) interfere constructively to interpolate the noise in the training points and regress to 0 between training points. This entails immediate access to the signal component of the trained neural network in form of its ReLU-component.

## 7 Conclusion

Conceptually, our work shows that inconsistency of overfitting is quite a generic phenomenon for regression in fixed dimension. However, particular spiky-smooth estimators enable benign overfitting, even in fixed dimension. We translate the spikes that lead to benign overfitting in kernel regression into infinitesimal fluctuations that can be added to activation functions to consistently interpolate with wide neural networks. Our experiments verify that neural networks with spiky-smooth activation functions can exhibit benign overfitting even on small, low-dimensional data sets.

Technically, our inconsistency results cover many distributions, Sobolev spaces of arbitrary order, and arbitrary RKHS-norm-bounded overfitting estimators. Lemma E.1 serves as a generic tool to extend generalization bounds to the sphere \(\mathbb{S}^{d}\), allowing us to cover (deep) ReLU NTKs and ReLU NNGPs.

Future work.While our experiments serve as a promising proof of concept, it remains unclear how to design activation functions that enable harmless interpolation of more complex neural network architectures and data sets. As another interesting insight, our consistent kernel sequence shows that although kernels may have equivalent RKHS (see Appendix G.2), their generalization error can differ arbitrarily much; the constants of the equivalence matter and the narrative that depth does not matter in the NTK regime as in Bietti and Bach (2021) is too simplified. More promisingly, analyses that extend our analysis in the infinite-width limit to a joint scaling of width and depth could help us to understand the influence of depth (Fort et al., 2020; Li et al., 2021; Seleznova and Kutyniok, 2022).

## Acknowledgements

Funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2075 - 390740016 and EXC 2064/1 - Project 390727645. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Moritz Haas and David Holzmuller. We want to thank Tizian Wenzel for interesting discussions. We also thank Nadine Grosse, Jens Wirth, and Daniel Winkle for helpful comments on Sobolev spaces.
