Gradient Descent in Neural Networks as Sequential Learning in RKBS
The study of Neural Tangent Kernels (NTKs) has provided much needed insight
into convergence and generalization properties of neural networks in the
over-parametrized (wide) limit by approximating the network using a first-order
Taylor expansion with respect to its weights in the neighborhood of their
initialization values. This allows neural network training to be analyzed from
the perspective of reproducing kernel Hilbert spaces (RKHS), which is
informative in the over-parametrized regime, but a poor approximation for
narrower networks as the weights change more during training. Our goal is to
extend beyond the limits of NTK toward a more general theory. We construct an
exact power-series representation of the neural network in a finite
neighborhood of the initial weights as an inner product of two feature maps,
respectively from data and weight-step space, to feature space, allowing neural
network training to be analyzed from the perspective of reproducing kernel {\em
Banach} space (RKBS). We prove that, regardless of width, the training sequence
produced by gradient descent can be exactly replicated by regularized
sequential learning in RKBS. Using this, we present novel bound on uniform
convergence where the iterations count and learning rate play a central role,
giving new theoretical insight into neural network training.
Alistair Shilton, Sunil Gupta, Santu Rana, Svetha Venkatesh
# Gradient Descent in Neural Networks as Sequential Learning in RKBS

###### Abstract

The study of Neural Tangent Kernels (NTKs) has provided much needed insight into convergence and generalization properties of neural networks in the over-parametrized (wide) limit by approximating the network using a first-order Taylor expansion with respect to its weights in the neighborhood of their initialization values. This allows neural network training to be analyzed from the perspective of reproducing kernel Hilbert spaces (RKHS), which is informative in the over-parametrized regime, but a poor approximation for narrower networks as the weights change more during training. Our goal is to extend beyond the limits of NTK toward a more general theory. We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights as an inner product of two feature maps, respectively from data and weight-step space, to feature space, allowing neural network training to be analyzed from the perspective of reproducing kernel _Banach_ space (RKBS). We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning in RKBS. Using this, we present novel bound on uniform convergence where the iterations count and learning rate play a central role, giving new theoretical insight into neural network training.

## 1 Introduction

The remarkable progress made in neural networks in recent decades has led to an explosion in their adoption in a wide swathe of applications. However this widespread success has also left unanswered questions, the most obvious of which is why non-convex, massively over-parameterised networks are able to perform so much better than predicted by traditional machine learning theory.

Neural tangent kernels represent an attempt to answer this question. As per (Jacot et al., 2018; Arora et al., 2019b), during training, the evolution of an over-parameterised neural network follows the kernel gradient of the functional cost with respect to a neural tangent kernel (NTK). It was shown that, for a sufficiently wide network with random weight initialisation, the NTK is effectively fixed, and results from machine learning in reproducing kernel Hilbert space (RKHS) can thus be brought to bear on the problem. This has led to a plethora of results analysing the convergence (Du et al., 2019; Allen-Zhu et al., 2019; Du et al., 2019; Zou et al., 2020; Zou and Gu, 2019) and generalisation (Arora et al., 2019b, a; Cao and Gu, 2019) properties of neural networks.

Despite their successes, NTK models are not without problems. As noted in (Bai and Lee, 2019), the expressive power of the linear approximation used by NTK is limited to that of the corresponding, randomised feature space or RKHS, as evidenced by the observed gap between NTK predictions and actual performance. To break out of this regime, (Bai and Lee, 2019) proposed using a second or higher-order approximation of the network. Moreover it is natural to ask how well a linear approximation of the behaviour, constructed on the assumption of small weight-steps, will scale to larger weight steps in narrower networks.

To overcome these difficulties we replace the Taylor approximation used in NTK with an an exact power series representation of the neural network in a finite neighbourhood around the initial weights. We demonstrate that this leads to a representation as an inner product between two feature maps, from data and weight-step space respectively. This structure underlies the construction of reproducing kernel Banach spaces (RKBS, (Lin et al., 2022)), allowing us to go on to show an equivalence between back-propagation and sequential learning in RKBS, which is similar to NTK but without the constraints of linearity, allowing us to derive new bounds on uniform convergence for networks of arbitrary width.

## 2 Related Work

There has been a significant amount of work looking at uniform convergence behaviour of networks of different types using variety of assumptions during training (Neyshabur et al., 2015, 2018, 2019, 2017; Harvey et al., 2017; Bartlett et al., 2017; Golowich et al., 2018; Arora et al., 2018; Allen-Zhu et al., 2018; Draxler et al., 2018; Li and Liang, 2018; Nagarajan and Kolter, 2019a, b; Zhou et al., 2019).

The study of the connection between kernel methods and neural networks has a long history. (Neal, 1996) demonstrated that, in the infinite-width limit, iid randomly initialised single-layer networks converge to draws from a Gaussian process. This was extended to multi-layered neural networks in (Lee et al., 2018; Matthews et al., 2018) by assuming random weights up to (but not including) the output layer. Other works deriving approximate kernels by assuming random weights include (Rahimi and Benjamin, 2009; Bach, 2014, 2017; Daniely et al., 2016; Daniely, 2017).
Neural tangent kernels (Jacot et al., 2018; Arora et al., 2019b) are a more recent development. The basis of NTK is to approximate the behaviour of neural network (for a given input \(\mathbf{x}\)) as the weights and biases vary about some initial values using a first-order Taylor approximation. This approximation is linear in the change in weights, and the coefficients of this approximation are functions of \(\mathbf{x}\) and may therefore be treated as a feature map, making the model amenable to the kernel trick and subsequent analysis in terms of RKHS theory. This approach may be generalised to higher order approximations (Bai and Lee, 2019), but the size of change in the weights that can be approximated remains limited except in the over-parametrised limit, where the variation of the weights becomes small.

Arc-cosine kernels (Cho and Saul, 2009) work on a similar premise. For activation functions of the form \(\tau(\xi)=(\xi)_{+}^{p}\), \(p\in\mathbb{N}\), in the infinite-width limit, arc-cosine kernels capture the feature map of the network. Depth is achieved by composition of kernels. However once again this approach is restricted to networks of infinite width, whereas our approach works for arbitrary networks.

Finally there has been some very recent work (Bartolucci et al., 2021; Sanders, 2020; Parhi and Nowak, 2021; Unser, 2021) in a similar vein to the current work, seeking to connect neural networks to RKBS theory. However these works consider only 1 and 2 layer networks (we consider networks of arbitrary depth), and no equivalence is established between the weight-steps found by back-propagation and those found by regularised learning in RKBS.

## 3 Notations

Let \(\mathbb{N}=\{0,1,2,\ldots\}\), \(\mathbb{N}_{n}=\{0,1,\ldots,n-1\}\). Vectors and matrices are denoted \(\mathbf{a}\) and \(\mathbf{A}\), respectively, with elements \(a_{i}\), \(A_{i,\nu}\), and columns \(\mathbf{A}_{:i}\) indexed by \(i,i^{\prime}\in\mathbb{N}\). We define:

\[\begin{array}{l}\|\mathbf{x}\|_{p}=(\sum_{i}|x_{i}|^{p})^{1/p},\|\mathbf{A}\| _{p,q}=\|[\|\mathbf{A}_{:i}\|_{p}]\|_{q}\\ \|\mathbf{x}\|_{\infty}=\max_{i}\{|x_{i}|\},\|\mathbf{x}\|_{-\infty}=\min_{i}\{ |x_{i}|\}\end{array}\]

\(\forall p,q\in[-\infty,0)\cup(0,\infty]\), which are norms if \(p,q\in[1,\infty]\). The Frobenius norm and inner product are \(\|\cdot\|_{F}=\|\cdot\|_{2,2}\). \(\langle\mathbf{A},\mathbf{B}\rangle_{F}=\mathrm{Tr}(\mathbf{A}^{T}\mathbf{B})\). The Kronecker and Hadamard product are \(\mathbf{a}\otimes\mathbf{b}\), \(\mathbf{a}\odot\mathbf{b}\). The Kronecker and Hadamard powers are \(\mathbf{a}^{\otimes e}=\mathbf{a}\odot\mathbf{a}\odot\mathbf{a}\odot\mathbf{a}\), \(\mathbf{a}^{\otimes e}=\mathbf{a}\odot\mathbf{a}\odot\mathbf{a}\). The elementwise absolute and sign are \(|\mathbf{a}|\), \(\mathrm{sgn}(\mathbf{a})\). Finally, for vectors \(\mathbf{a}\), \(\mathbf{b}\) we let \(\mathbf{a}(\mathbf{a},\mathbf{b})=[\,a_{0}(\mathbf{b}^{\otimes 1})^{T},a_{1}( \mathbf{b}^{\otimes 2})^{T},\ldots\,]^{T}\). \(\mathrm{diag}(\mathbf{A})\) is a vector containing the diagonal elements of \(\mathbf{A}\), and conversely \(\mathrm{diag}(\mathbf{a})\) is a diagonal matrix with diagonal elements from \(\mathbf{a}\).

We study fully connected \(D\)-layer neural networks \(\mathbf{f}:(\mathbb{X}\subset\mathbb{R}^{n})\rightarrow(\mathbb{Y}\subset \mathbb{R}^{m})\) with layer widths \(H^{[j]}\) (and \(H^{[-1]}=n\)) trained on a training set \(\{(\mathbf{x}^{(k)},\mathbf{y}^{(k)})\in\mathbb{X}\times\mathbb{Y}:k\in \mathbb{N}_{N}\}\). We use index range conventions \(k\in\mathbb{N}_{N}\), \(j\in\mathbb{N}_{D}\) and \(i_{j+1}\in\mathbb{N}_{H^{[j]}}\), and for clarity we write:

\[\begin{array}{c}\text{Relating to:}\quad\frac{\text{Training vector $k$ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ Layer \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ $j$}}{\sum_{\begin{subarray}{c}\mathbf{x}^{ \{\{k}}[j](r)}\\ \mathbf{x}^{\{\{k}}[j](r)}\\ \mathbf{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{ \mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{ \mathcal{        \mathcal{           }}}}}}}}} \end{\mathbf{\mathbf{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{ \mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal{\mathcal                                          \
exists a kernel \(K\) such that:

\[f\left(\mathbf{x}\right)=\left\langle f\left(\cdot\right),K\left(\mathbf{x}, \cdot\right)\right\rangle_{\mathcal{H}}\quad\forall f\in\mathcal{H}\]

Subsequently \(K(\mathbf{x},\mathbf{x}^{\prime})=\left\langle K(\mathbf{x},\cdot),K(\mathbf{x }^{\prime},\cdot)\right\rangle\) and, by the Moore-Aronszajn theorem, \(K\) is uniquely defined by \(\mathcal{H}\) and vice-versa. \(K\) is called the reproducing kernel, and the corresponding RKHS is denoted \(\mathcal{H}_{K}\). RKHS based approaches have gained popularity as they are well suited to many aspects of machine learning (Steinwart and Christman, 2008; Shawe-Taylor and Cristianini, 2004). The inner product structure enables the kernel trick, and the kernel is readily understood as a similarity measure. Furthermore the structure of RKHSs has led to a rich framework of complexity analysis and generalisation bounds (Steinwart and Christman, 2008; Shawe-Taylor and Cristianini, 2004). More recently neural tangent kernels were introduced (Jacot et al., 2018), allowing RKHS theory to be applied to the neural network training in the over-parametrised regime.

In an effort to introduce a richer set of geometrical structures into RKHS theory, reproducing kernel Banach spaces (RKBSs) generalise RKHSs by starting with a Banach space of functions (Der and Lee, 2007; Zhang et al., 2009; Song et al., 2013; Xu and Ye, 2014; Lin et al., 2022) etc. Precisely:

**Definition 1** (Reproducing kernel Banach space (RKBS - (Lin et al., 2022))).: A reproducing kernel Banach space \(\mathcal{B}\) on a set \(\mathbb{X}\) is a Banach space of functions \(f:\mathbb{X}\rightarrow\mathbb{Y}\) such that every point evaluation \(\delta_{\mathbf{x}}:\mathcal{B}\rightarrow\mathbb{Y}\), \(\mathbf{x}\in\mathbb{X}\), on \(\mathcal{B}\) is continuous (so \(\forall\mathbf{x}\in\mathbb{X}\)\(2C_{\mathbf{x}}\in\mathbb{R}_{+}\) such that \(|\delta_{\mathbf{x}}(f)|=|f(\mathbf{x})|\leq C_{\mathbf{x}}\|f\|_{\mathcal{B}} \)\(\forall f\in\mathcal{B}\)).

There are several distinct approaches to RKBS construction. In the present context however we find the approach of (Lin et al., 2022, Theorem 2.1) most convenient. Given the components outlined in Figure 1, and assuming that \(\mathbf{\Phi}_{\mathcal{O}}(\mathbb{X})\) is dense in \(\mathcal{X}_{\mathcal{O}}\) and that \(\mathbf{\Psi}_{\mathcal{O}}(\mathbb{W}_{\mathcal{O}})\) is dense in \(\mathcal{W}_{\mathcal{O}}\), we define the reproducing kernel Banach space \(\mathcal{B}_{\mathcal{O}}\) on \(\mathbb{X}\) as:

\[\begin{array}{l}\mathcal{B}_{\mathcal{O}}=\left\{\left.\left\langle\mathbf{ \Phi}_{\mathcal{O}}\left(\cdot\right),\mathbf{\Omega}\right\rangle_{\mathcal{ X}_{\mathcal{O}}\times\mathcal{W}_{\mathcal{O}}}\right|\mathbf{\Omega}\in \mathcal{W}_{\mathcal{O}}\right\}\\ \text{where: }\left\|\left\langle\mathbf{\Phi}_{\mathcal{O}}\left(\cdot\right), \mathbf{\Omega}\right\rangle_{\mathcal{X}_{\mathcal{O}}\times\mathcal{W}_{ \mathcal{O}}}\right\|_{\mathcal{B}_{\mathcal{O}}}=\left\|\mathbf{\Omega} \right\|_{\mathcal{W}_{\mathcal{O}}}\end{array} \tag{1}\]

with reproducing Banach kernel:

\[K_{\mathcal{O}}\left(\mathbf{x},\mathbf{W}_{\Delta}\right)=\left\langle \mathbf{\Phi}_{\mathcal{O}}\left(\mathbf{x}\right),\mathbf{\Psi}_{\mathcal{O }}\left(\mathbf{W}_{\Delta}\right)\right\rangle_{\mathcal{X}_{\mathcal{O}} \times\mathcal{W}_{\mathcal{O}}} \tag{2}\]

## 5 Setup and Assumptions

We assume a fully-connected, \(D\)-layer feedforward neural network \(\mathbf{f}:(\mathbb{X}\subseteq\mathbb{R}^{n})\rightarrow(\mathbb{Y}\subseteq \mathbb{R}^{m})\) with layers of widths \(H^{[0]},H^{[1]},\ldots,H^{[D-1]}\), where \(H^{[D-1]}=m\) and we define \(H^{[-1]}=n\). We assume layer \(j\in\mathbb{N}_{D}\) (\(j\in\mathbb{N}_{D}\) throughout) contains only neurons with activation function \(\tau^{[j]}:\mathbb{R}\rightarrow\mathbb{R}\). The network is defined recursively \(\forall j\in\mathbb{N}_{D}\):

\[\begin{array}{l}\mathbf{f}\left(\mathbf{x}\right)=\mathbf{x}^{[D]}\in \mathbb{R}^{H^{[D-1]}}\\ \mathbf{x}^{[j+1]}=\tau^{[j]}(\tilde{\mathbf{x}}^{[j]})\in\mathbb{R}^{H^{[j]}} \\ \tilde{\mathbf{x}}^{[j]}=\frac{1}{\sqrt{H^{[D]}}}\mathbf{W}^{[j]}\mathbf{T} \mathbf{x}^{[j]}+\alpha^{[j]}\mathbf{b}^{[j]}\in\mathbb{R}^{H^{[j]}}\\ \mathbf{x}^{[0]}=\mathbf{x}\in\mathbb{X}\subset\mathbb{R}^{H^{[-1]}}\ (H^{[-1]}=n) \end{array} \tag{3}\]

where \(\mathbf{W}^{[j]}\in\mathbb{R}^{H^{[j-1]}\times H^{[j]}}\) and \(\mathbf{b}^{[j]}\in\mathbb{R}^{H^{[j]}}\) are weights and biases, which we summarise as \(\mathbf{W}\in\mathbb{W}\), and \(\alpha^{[j]}\in\mathbb{R}_{+}\) are fixed. The set of functions of this form is denoted \(\mathcal{F}\).

We assume the goal of training is to take a training set and find weights and biases to minimise the empirical risk:

\[\begin{array}{l}\mathbf{W}^{*}=\operatorname*{argmin}_{\begin{subarray}{c} \mathbf{W}\in\mathbb{W}\\ R_{E}\left(\mathbf{W},\mathbb{D}\right)=\sum_{k}E\left(\mathbf{x}^{\{k\}}, \mathbf{y}^{\{k\}},\mathbf{f}_{\mathbf{W}}\left(\mathbf{x}^{\{k\}}\right) \right)\end{subarray}}\end{array} \tag{4}\]

where \(\mathbf{f}_{\mathbf{W}}\) is a network of the form (3) with weights and biases \(\mathbf{W}\), \(\mathbb{D}=\{(\mathbf{x}^{\{k\}},\mathbf{y}^{\{k\}})\in\mathbb{X}\times \mathbb{Y}:k\in\mathbb{N}_{N}\}\) is a training set (\(k\in\mathbb{N}_{N}\) throughout), and \(E:\mathbb{X}\times\mathbb{Y}\times\mathbb{R}^{m}\rightarrow\mathbb{R}\) is an error function defining the purpose of the network.

We make the following technical assumptions:

1. Input space: \(\mathbb{X}=[-1,1]^{n}\).
2. Error function: \(E:\mathbb{X}\times\mathbb{Y}\times\mathbb{R}^{m}\rightarrow\mathbb{R}\) is \(\mathcal{C}^{1}\) and \(L_{E}\)-Lipschitz in its third argument.
3. Activation functions: for all \(j\in\mathbb{N}_{D}\), \(\tau^{[j]}:\mathbb{R}\rightarrow[-1,1]^{H^{[j]}}\) is bounded, \(\mathcal{C}^{\infty}\), and has a power-series representations with region of convergence (ROC) at least \(\rho^{[j]}\in\mathbb{R}_{+}\) around all \(z\in\mathbb{R}\).
4. Weight non-triviality: for all \(j\in\mathbb{N}_{D}\), \(\mathbf{W}^{[j]}\neq\mathbf{0}\) at all times during training.1 Footnote 1: Networks that do not meet this requirement have a constant output independent of input \(\mathbf{x}\). We do not consider this a restrictive assumption as it is highly unlikely that a randomly initialised network trained with a typical training set will ever reach this state.
5. Weight initialisation: we assume LeCun initialisation, so for all \(j\in\mathbb{N}_{D}\), \(W^{[j]}_{i_{j},i_{j+1}},b^{[j]}_{i_{j+1}}\sim\mathcal{N}(0,1)\).
6. Training: we assume training is gradient descent (back-propagation) with learning rate \(\eta\in\mathbb{R}_{+}\).
### Back-Propogation Training

As stated above, we assume the network is trained using back-propagation (gradient descent) [11]. This is an iterative approach. An iteration starts with initial weights and biases \(\mathbf{W}_{\mathcal{O}}\in\mathbb{W}\). A weight-step:

\[\mathbf{W}_{\Delta}^{\boxplus}=-\eta\;\frac{\partial}{\partial\mathbf{W}} \sum_{k}E\left(\mathbf{x}^{\{k\}},\mathbf{y}^{\{k\}},\mathbf{f}_{\mathbf{W}} \left(\mathbf{x}^{\{k\}}\right)\right)\big{|}_{\mathbf{W}=\mathbf{W}_{ \mathcal{O}}}\]

is calculated, and weights and biases are updated as \(\mathbf{W}=\mathbf{W}_{\mathcal{O}}+\mathbf{W}_{\Delta}^{\boxplus}\). Our notational convention for activations before an iteration, and the subsequent change due to a weight step, are given in figure 2. The weight-step is [11] (see appendix B for a derivation):

\[\begin{array}{l}\mathbf{W}_{\Delta_{ij+1}}^{[j]\boxplus}=-\frac{\eta}{ \sqrt{H^{[-1]}H^{[-2]}\dots H^{[j+1]}}}\sum_{k}\gamma_{\mathcal{O}_{ij+1}}^{ \{k\}[j]}\frac{\mathbf{x}_{\mathcal{O}}^{[j]\boxplus}}{\sqrt{H^{[j]}}}\\ b_{\Delta_{ij+1}}^{[j]\boxplus}=-\frac{\eta}{\sqrt{H^{[-1]}H^{[-2]}\dots H^{[j+ 1]}}}\sum_{k}\gamma_{\mathcal{O}_{ij+1}}^{[k]\boxplus}\alpha[j]\end{array} \tag{5}\]

for all \(j\in\mathbb{N}_{D},i_{j+1}\) where, recursively \(\forall j\in\mathbb{N}_{D-1}\):

\[\begin{array}{l}\gamma_{\mathcal{O}_{ij}}^{\{k\}[j-1]}=\sum_{i_{j+1}}\gamma _{\mathcal{O}_{ij+1}}^{\{k\}[j]}W_{\mathcal{O}_{ij},i_{j+1}^{\prime}}^{[j]} \tau^{[j-1](1)}\left(\tilde{x}_{\mathcal{O}_{ij}}^{\{k\}[j-1]}\right)\\ \gamma_{\mathcal{O}_{ij}}^{\{k\}[j-1]}=\nabla_{i_{D}}E\left(\dots^{\{k\}}\right) \tau^{[D-1](1)}\left(\tilde{x}_{\mathcal{O}_{ij}}^{\{k\}[j-1]}\right)\end{array}\]

Note that the change in bias \(\mathbf{b}_{\Delta}^{[j]}\) is proportional to \(\alpha^{[j]}\).

## 6 Analysis of a Single Iteration

In the first phase of our analysis we consider the change in neural network behaviour resulting from a small in weights and biases (a weight-step). The overall training sequence is readily extrapolated from this as per section 7. Our first goal is to rewrite the neural network after a training iteration as:

\[\mathbf{f}\left(\mathbf{x}\right)=\mathbf{f}_{\mathcal{O}}\left(\mathbf{x} \right)+\mathbf{f}_{\Delta}\left(\mathbf{x}\right) \tag{6}\]

where \(\mathbf{f}_{\mathcal{O}}=\mathbf{f}_{\mathbf{W}_{\mathcal{O}}}:(\mathbb{X} \subset\mathbb{R}^{n})\rightarrow(\mathbb{Y}\subset\mathbb{R}^{m})\) is the neural network before the iteration and \(\mathbf{f}_{\Delta}:(\mathbb{X}\subset\mathbb{R}^{n})\rightarrow\mathbb{R}^{m}\) is the change in network behaviour due to the change \(\mathbf{W}=\mathbf{W}_{\mathcal{O}}\rightarrow\mathbf{W}=\mathbf{W}_{ \mathcal{O}}+\mathbf{W}_{\Delta}\) in weights and biases for this iteration, as detailed in Figure 2, so that:

\[\begin{array}{l}\mathbf{f}_{\Delta}\left(\mathbf{x}\right)=\left\langle \mathbf{\Phi}_{\mathcal{O}}\left(\mathbf{x}\right),\mathbf{\Psi}_{\mathcal{O}} \left(\mathbf{W}_{\Delta}\right)\right\rangle_{\mathcal{X}\mathbb{O}\times \mathcal{W}_{\mathcal{O}}}\\ \text{where:}\quad\mathbf{\Phi}_{\mathcal{O}}:\mathbb{X}\rightarrow\mathcal{X}_ {\mathcal{O}}=\mathrm{span}\left(\mathbf{\Phi}_{\mathcal{O}}\left(\mathbb{X} \right)\right)\subset\mathbb{R}^{\infty\times m}\\ \mathbf{\Psi}_{\mathcal{O}}:\mathbb{W}_{\mathcal{O}}\rightarrow\mathcal{W}_{ \mathcal{O}}=\mathrm{span}\left(\mathbf{\Psi}_{\mathcal{O}}\left(\mathbb{W}_ {\mathcal{O}}\right)\right)\subset\mathbb{R}^{\infty\times m}\end{array} \tag{7}\]

are feature maps determined entirely by the structure of the (number and width of layers, activation functions) and the initial weights and biases \(\mathbf{W}_{\mathcal{O}}\); and \(\left\langle\mathbf{\Xi},\mathbf{\Omega}\right\rangle_{\mathcal{X}_{ \mathcal{O}}\times\mathcal{W}_{\mathcal{O}}}=\mathrm{diag}\left(\mathbf{\Xi}^ {\mathrm{T}}\mathbf{\Omega}\right)\) is a bilinear form. Subsequently our second goal is to derive kernels and norms from these feature maps to allow us to study their convergence properties, and we finish by proving an equivalence between the weight-step \(\mathbf{W}_{\Delta}^{\boxplus}\) due to a single step of back-propagation and the analogous weight-step \(\mathbf{W}_{\Delta}^{\boxplus}\) that minimises the (RKBS) regularised risk.

### Contribution 1: Feature-Map Expansion

In this section we derive appropriate feature maps to express the change in neural network behaviour for a finite weight-step. Our approach is simple in principle but technical, so details are reserved for appendix B. Roughly speaking however, we begin by noting that, for a smooth activation function \(\tau^{[j]}:\mathbb{R}\rightarrow\mathbb{R}\), \(z\in\mathbb{R}\) and finite-dimensional vectors \(\mathbf{c},\mathbf{c}^{\prime}\) whose inner product lies in the radius of convergence \(\rho^{[j]}\) (so that \(|\left\langle\mathbf{c},\mathbf{c}\right\rangle|<\rho^{[j]}\)), the power-series representation of \(\tau^{[j]}\) about \(z\) can be written \(\tau^{[j]}(z+\left\langle\mathbf{c},\mathbf{c}\right\rangle)=\tau^{[j]}(z)+ \left\langle\boldsymbol{\rho}(\mathbf{g}^{[j]}(z),\mathbf{c}),\boldsymbol{ \rho}(\mathbf{1}_{\infty},\mathbf{c}^{\prime})\right\rangle\), where:

\[\begin{array}{l}\boldsymbol{\varrho}\left(\mathbf{a},\mathbf{d}\right)=\left[ \begin{array}{c}a_{0}\mathbf{d}^{\otimes 1\mathrm{T}}\;a_{1}\mathbf{d}^{\otimes 2\mathrm{T}}\;a_{2} \mathbf{d}^{\otimes 3\mathrm{T}}\;\dots\end{array}\right]^{\mathrm{T}}\\ \mathbf{g}^{[j]}\left(z\right)=\left[\begin{array}{c}\frac{1}{1!}\tau^{[j](1 )}\left(z\right)\;\frac{1}{2!}\tau^{[j](2)}\left(z\right)\;\frac{1}{3!}\tau^{[ j](3)}\left(z\right)\;\dots\end{array}\right]^{\mathrm{T}}\end{array}\]

Figure 2: Definition of terms for neural network before and after an iteration.
Given an input \(\mathbf{x}\), starting at layer \(0\) and working forward, and with reference to Figure 2, we can write the change \(\mathbf{x}_{\Delta}^{[1]}\) in the output of layer \(0\) due to the weight-step \(\mathbf{W}_{\Delta}\) as:

\[\begin{split}\mathbf{x}_{\Delta}^{[1]}&=\left[\left\langle \boldsymbol{\Phi}_{\mathcal{O};i_{1}}^{[0]}\left(\mathbf{x}\right),\boldsymbol{ \Psi}_{\mathcal{O};i_{1}}^{[0]}\left(\mathbf{W}_{\Delta}\right)\right\rangle \right]_{i_{1}}\\ \text{where: }\boldsymbol{\Phi}_{\mathcal{O};i_{1}}^{[0]}\left( \mathbf{x}\right)&=\boldsymbol{\varrho}\left(\mathbf{g}^{[0]} \left(\tilde{\boldsymbol{x}}_{\mathcal{O}i_{1}}^{[0]}\right),\mu_{i_{1}}^{[0] }\right]\frac{1}{\sqrt{2}\tilde{\boldsymbol{\boldsymbol{\boldsymbol{ \boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{ \boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{\boldsymbol{ \boldsymbol{    \boldsymbol{   }}}}}}}}}}}}} \lambda_{\mathcal{O}}^{[0]}\left]\right)\\ \boldsymbol{\Psi}_{\mathcal{O};i_{1}}^{[0]}\left(\mathbf{W}_{ \Delta}\right)&=\boldsymbol{\varrho}\left(\mathbf{1}_{\infty}, \frac{1}{\mu_{i_{1}}^{[0]}}\left[\begin{array}{c}\sqrt{2}b_{\Delta i_{1}}^{ [0]}\\ \sqrt{2}\mathbf{W}_{\Delta i_{1}}^{[0]}\end{array}\right]\right)\end{split} \tag{8}\]

where we note that both feature maps have a finite radius of convergence. The feature maps are parameterised by the scale factors \(\mu_{i_{1}}^{[0]}\in\mathbb{R}_{+}\) whose role is mainly technical, insofar as they will allow us to show equivalence between RKBS regularised risk minimisation and back-propogation.2 Their exact value (beyond existence) is unimportant here.

Footnote 2: See appendix for more discussion.

The process is repeated for subsequent layers (see appendix B for details). After working through all layers:

\[\mathbf{f}_{\Delta}\left(\mathbf{x}\right)=\left\langle\boldsymbol{\Phi}_{ \mathcal{O}}\left(\mathbf{x}\right),\boldsymbol{\Psi}_{\mathcal{O}}\left( \mathbf{W}_{\Delta}\right)\right\rangle_{\mathcal{X}_{0}\times\mathcal{W}_{ \mathcal{O}}} \tag{9}\]

where \(\boldsymbol{\Phi}_{\mathcal{O}}=\boldsymbol{\Phi}_{\mathcal{O}}^{[D-1]}\), \(\boldsymbol{\Psi}_{\mathcal{O}}=\boldsymbol{\Psi}_{\mathcal{O}}^{[D-1]}\), and \(\forall j\in\mathbb{N}_{D}\backslash\{0\}\):

(10)

recursively \(\forall j\in\mathbb{N}_{D}\backslash\{0\}\) which are parameterised by scale factors \(\mu_{i_{j+1}}^{[j]}\in\mathbb{R}_{+}\) and shadow weights \(\omega_{i_{j}}^{[j]}\), \(\widetilde{\omega}_{i_{j+1}}^{[j]}\in\mathbb{R}_{+}\), which play a role in the equivalence between RKBS regularised risk minimisation and back-propagation (otherwise their exact values are unimportant).

### Contribution 2: Induced Kernels and Norms

In the previous section we established that, as a result of a single weight-step \(\mathbf{W}_{\Delta}\), we can write:

\[\begin{split}\mathbf{f}\left(\mathbf{x}\right)&= \mathbf{f}_{\mathcal{O}}\left(\mathbf{x}\right)+\mathbf{f}_{\Delta}\left( \mathbf{x}\right)\\ \mathbf{f}_{\Delta}\left(\mathbf{x}\right)&=\left\langle \boldsymbol{\Phi}_{\mathcal{O}}\left(\mathbf{x}\right),\boldsymbol{\Psi}_{ \mathcal{O}}\left(\mathbf{W}_{\Delta}\right)\right\rangle_{\mathcal{X}_{0} \times\mathcal{W}_{\mathcal{O}}}\end{split}\]

where \(\mathbf{f}_{\mathcal{O}}:\mathbb{X}\rightarrow\mathbb{Y}\) is the neural network pre-iteration and the feature maps \(\boldsymbol{\Phi}_{\mathcal{O}}\), \(\boldsymbol{\Psi}_{\mathcal{O}}\) are feature maps (8-10). Using these, we induce kernels on \(\mathbb{X}\), \(\mathbb{W}_{\mathcal{O}}\) using the kernel trick:

\[\begin{split}\mathbf{K}_{\mathcal{X}_{0}}\left(\mathbf{x}, \mathbf{x}^{\prime}\right)&=\left\langle\boldsymbol{\Phi}_{ \mathcal{O}}\left(\mathbf{x}\right),\boldsymbol{\Phi}_{\mathcal{O}}\left( \mathbf{x}^{\prime}\right)\right\rangle_{\mathcal{X}_{0}\times\mathcal{X}_{0} }\\ &=\boldsymbol{\Phi}_{\mathcal{O}}^{\mathbb{X}}\left(\mathbf{x} \right)\boldsymbol{\Phi}_{\mathcal{O}}\left(\mathbf{x}^{\prime}\right)\\ \mathbf{K}_{\mathcal{W}_{\mathcal{O}}}\left(\mathbf{W}_{\Delta}, \mathbf{W}_{\Delta}^{\prime}\right)&=\left\langle\boldsymbol{\Psi}_{ \mathcal{O}}\left(\mathbf{W}_{\Delta}\right),\boldsymbol{\Psi}_{\mathcal{O}} \left(\mathbf{W}_{\Delta}^{\prime}\right)\right\rangle_{\mathcal{W}_{ \mathcal{O}}\times\mathcal{W}_{\mathcal{O}}}\\ &=\boldsymbol{\Psi}_{\mathcal{O}}^{\mathbb{Y}}\left(\mathbf{W}_{ \Delta}\right)\boldsymbol{\Psi}_{\mathcal{O}}\left(\mathbf{W}_{\Delta}^{ \prime}\right)\end{split}\]

We call these kernels _neural neighbourhood kernels_ (NNK) as they describe the similarity structure in the finite neighbourhood of the \(\mathbf{W}_{\mathcal{O}}\) (c/f NTK, which is the behaviour tangent to, or in the infinitessimal neighbourhood of, \(\mathbf{W}_{\mathcal{O}}\)). These matrix-valued kernels are symmetric and positive definite by construction, and could potentially be used (transfered) in support vector machines (SVMs) or similar kernel-based methods, measuring similarity on \(\mathbb{X}\) and \(\mathbb{W}_{\mathcal{O}}\), respectively. Similarly, we induce a Banach kernel:

\[\begin{split}\mathbf{K}_{\mathcal{O}}\left(\mathbf{x},\mathbf{W}_{ \Delta}^{\prime}\right)&=\left\langle\boldsymbol{\Phi}_{\mathcal{O}} \left(\mathbf{x}\right),\boldsymbol{\Psi}_{\mathcal{O}}\left(\mathbf{W}_{ \Delta}^{\prime}\right)\right\rangle_{\mathcal{X}_{0}\times\mathcal{W}_{ \mathcal{O}}}\\ &=\operatorname{diag}\left(\mathbf{f}_{\Delta}\left(\mathbf{x} \right)\right)\end{split} \tag{11}\]

which is trivially the change in the network output \(\mathbf{f}_{\Delta}(\mathbf{x})\) under weight-step \(\mathbf{W}_{\Delta}\) for input vector \(\mathbf{x}\), diagonalised.
The precise form of the neural neighbourhood kernels is rather complicated (the derivation is straightforward but resulting recursive equation is very long). The NNK is the (un-approximated) analog of the NTK. However while neural networks evolve - to first order - in the RKHS defined by the NTK, the same is not true of the NNK. Indeed, it is not difficult to see that _RKHS regularisation using the NNK will always result in a weight vector that is not the image of a weight-step under \(\boldsymbol{\Psi}_{\mathcal{O}}\) (i.e. RKHS theory is insufficient, and we need RKBS theory to proceed)_. Thus, as the NNK is not the main focus of our paper will not reproduce them in the body of the paper - the interested reader can find them in appendix C. Using these induced kernels we may obtain expressions for the norms of the images of \(\mathbb{X}\) and \(\mathbb{W}_{\mathcal{O}}\) in feature space:

\[\left\|\boldsymbol{\Phi}_{\mathcal{O}}\left(\mathbf{x}\right) \right\|_{\mathcal{X}_{\mathcal{O}}}^{2} =\left\|\boldsymbol{\Phi}_{\mathcal{O}}^{[D-1]}\left(\mathbf{x} \right)\right\|_{F}^{2} \tag{12}\] \[=\mathrm{Tr}\left(\mathbf{K}_{\mathcal{X}_{\mathcal{O}}}\left( \mathbf{x},\mathbf{x}\right)\right)\] \[\left\|\boldsymbol{\Psi}_{\mathcal{O}}\left(\mathbf{W}_{\Delta} \right)\right\|_{\mathcal{W}_{\mathcal{O}}}^{2} =\left\|\boldsymbol{\Psi}_{\mathcal{O}}^{[D-1]}\left(\mathbf{W}_{ \Delta}\right)\right\|_{F}^{2}\] \[=\mathrm{Tr}\left(\mathbf{K}_{\mathcal{W}_{\mathcal{O}}}\left( \mathbf{W}_{\Delta},\mathbf{W}_{\Delta}\right)\right)\]

Once again, the precise form of these expressions is complicated (the derivation is straightforward but the answer is very long - details can be found in appendix C). The importance of these norms lies in deriving conditions on convergence of the feature maps. Defining the helper function:

\[\overline{\sigma}^{[j]}\left(\zeta\right)=\max_{z\in\mathbb{R}_{+}\cup\{0\}} \left\{\sum_{l=1}^{\infty}\left(\frac{1}{l!}\right)^{2}\tau^{[j]\left(l\right) }\left(z\right)\tau^{[j]\left(l\right)}\left(z\right)\zeta^{l}\right\} \tag{13}\]

and the constants:

\[s^{[j]2}=\left\{\begin{array}{ll}\frac{1}{2}\alpha^{[j]2}+\frac{1}{2}\frac{ H^{[j-1]}}{H^{[j-1]}}&\text{if }j=0\\ \frac{1}{2}\alpha^{[j]2}+\frac{H^{[j-1]}}{H^{[j-1]}}&\text{otherwise}\end{array}\right. \tag{14}\]

which are, loosely speaking, surrogates for, respectively, the expansion/contraction (fanout) in width from layer \(j\) to layer \(j-1\) and the size of the weight-step at layer \(j\); in appendix C.6 we derive the following Lemmas (see corresponding proofs of Theorems 5 and 6 in the appendix):

**Lemma 1**.: _Let \(\epsilon_{\phi}^{[j]}\in(0,1)\)\(\forall j\in\mathbb{N}_{D}\). For a given neural network and initial weights \(\mathbf{W}_{\mathcal{O}}\) define \(\frac{\phi^{[j]}}{H^{[j]}}=\overline{\sigma}^{[j]}((1-\epsilon_{\phi}^{[j]}) \sqrt{\rho^{[j]}})\)\(\forall j\in\mathbb{N}_{D}\). If the scale factors satisfy:_

\[\mu_{ij+1}^{[j]2} \leq\]

\(\forall j\in\mathbb{N}_{D},i_{j+1}\) _then \(\left\|\boldsymbol{\Phi}_{\mathcal{O}}^{[j]}(\mathbf{x})\right\|_{F}^{2}\leq \phi^{[j]}\)\(\forall\mathbf{x}\in\mathbb{X}\)._

**Lemma 2**.: _Let \(\epsilon_{\psi}^{[j]}\in(0,1)\)\(\forall j\in\mathbb{N}_{D}\). For a given neural network and initial weights \(\mathbf{W}_{\mathcal{O}}\) and weight-step \(\mathbf{W}_{\Delta}\), if:_

\[t_{\Delta i_{1}}^{[0]2}\leq\left(1-\epsilon_{\psi}^{[0]}\right) \mu_{i_{1}}^{[0]2}\] \[t_{\Delta i_{j+1}}^{[j]2}+\sum_{i_{j}}\frac{\left\|\boldsymbol{ \Phi}_{\mathcal{O}}^{[j-1]}\left(\mathbf{W}_{\Delta}\right)\right\|_{\mathbb{ Z}}^{2}}{\omega_{i_{j}}^{[j]2}}\left(\widetilde{\omega}_{i_{j},i_{j+1}}^{[j]2}+W_{ \Delta i_{j},i_{j+1}}^{[j]2}\right)\leq\left(1-\epsilon_{\psi}^{[j]}\right) \mu_{i_{j+1}}^{[j]2}\]

\(\forall j\in\mathbb{N}_{D},i_{j^{\prime}+1}\) _then \(\left\|\boldsymbol{\Psi}_{\mathcal{O}}^{[j]}(\mathbf{W}_{\Delta})\right\|_{F}^{2} \leq H^{[j]}\frac{1-\epsilon_{\psi}^{[j]}}{\epsilon_{\psi}^{[0]}}\)._

Lemma 1 allows us to place bounds on the scale factors to ensure that the feature map \(\boldsymbol{\Phi}_{\mathcal{O}}^{[j]}:\mathbb{X}\rightarrow\mathcal{X}_{ \mathcal{O}}\) is finite (well defined) for all \(\mathbf{x}\in\mathbb{X}\). Lemma 2 is similar, but rather than bounding the scale factors it takes these as given (by Lemma 1) and places bounds on the size of the weight-step \(\mathbf{W}_{\Delta}\) for which the feature map is \(\boldsymbol{\Psi}_{\mathcal{O}}^{[j]}:\mathbb{W}\rightarrow\mathcal{W}_{ \mathcal{O}}\) is finite (well defined). Taken together, therefore, they give some bound on the size of weight-step that can be modelled for a give neural network structure and initial weights \(\mathbf{W}_{\mathcal{O}}\). However they say nothing directly about the shadow weights. In the next section we use these Lemmas to establish a link between the weight-step generated by gradient descent and learning in RKBS, as well as clarifying the size of weight-step which we can model using our construct.
### Contribution 3: Equivalence of Gradient Descent and regularised Risk Minimisation in RKBS

In our previous contributions we showed that the change \(\mathbf{f}_{\Delta}\) in neural network behaviour can be represented as by the form (9) with feature maps (8,10), derived kernels from these feature maps and gave bounds on the size of the weight-step for which this representation is valid. In this section we use these results to establish a link between gradient descent learning in neural networks and regularised risk minimisation in reproducing kernel Banach space (RKBS).

To begin, it is not difficult to see that the feature maps \(\mathbf{\Phi}_{\mathcal{O}}:\mathbb{X}\rightarrow\mathcal{X}_{\mathcal{O}}\) and \(\mathbf{\Psi}_{\mathcal{O}}:\mathbb{W}_{\mathcal{O}}\rightarrow\mathcal{W}_{ \mathcal{O}}\) define a RKBS \(\mathcal{B}_{\mathcal{O}}\) imbued with \(\|\cdot\|_{\mathcal{B}_{\mathcal{O}}}=\|\cdot\|_{F}\) using (1) with the reproducing Banach kernel (11) deriving from (2):3

Footnote 3: In the appendix we prove that these maps satisfy the relevant density requirements.

\[\mathbf{K}_{\mathcal{O}}\left(\mathbf{x},\mathbf{W}_{\Delta}^{ \prime}\right) =\left\langle\mathbf{\Phi}_{\mathcal{O}}\left(\mathbf{x}\right), \mathbf{\Psi}_{\mathcal{O}}\left(\mathbf{W}_{\Delta}^{\prime}\right)\right\rangle _{\mathcal{X}_{\mathcal{O}}\times\mathcal{W}_{\mathcal{O}}}\] \[=\operatorname{diag}\left(\mathbf{f}_{\Delta}\left(\mathbf{x} \right)\right)\]

in terms of which the change \(\mathbf{f}_{\Delta}^{\oplus}\) in the network's behaviour due to a back-progation iteration may be written:

\[\mathbf{f}_{\Delta}^{\oplus}\left(\cdot\right)=\mathbf{K}_{\mathcal{O}}\left( \cdot,\mathbf{W}_{\Delta}^{\oplus}\right)\mathbf{1}\]

For a given neural network with initial weights \(\mathbf{W}_{\mathcal{O}}\), we assume that the weight-step is chosen using gradient descent.4 An alternative approach might be to select a weight-step to minimise the regularised risk in RKBS, specifically:

Footnote 4: Note that the training set \(\mathbb{D}\) here is for this iteration only and may be a random subset of a larger training set.

\[\mathbf{W}_{\Delta}^{\bullet}=\underset{\mathbf{W}_{\mathcal{O}}\in\mathcal{ W}_{\mathcal{O}}}{\operatorname{argmin}}\,R_{\lambda}\left(\mathbf{W}_{ \Delta}\right) \tag{15}\] \[R_{\lambda}\left(\mathbf{W}_{\Delta}\right)=\lambda\|\mathbf{ \Psi}_{\mathcal{O}}\left(\mathbf{W}_{\Delta}\right)\|_{\mathcal{W}_{\mathcal{O }}}+R_{E}\left(\mathbf{W}_{\Delta}+\mathbf{W}_{\mathcal{O}},\mathbb{D}\right)\]

where we call \(\lambda\) the trade-off coefficient. Larger trade-off coefficients favour smaller weight-steps, and vice-versa. The advantage of this form over the back-propagation derived weight-step is that we can directly apply complexity bounds etc. from RKBS theory, and then extend to the complete training process. This motivates us to ask:

_For a given neural network with initial weights and biases \(\mathbf{W}_{\mathcal{O}}\), let \(\mathbf{W}_{\Delta}^{\oplus}\) be the back-propagation weight-step (gradient descent with learning rate \(\eta\)) defined by (5), and let \(\mathbf{W}_{\Delta}^{\bullet}\) be a weight-step solving the regularised risk minimisation problem (15). Given the gradient-descent derived weight-step \(\mathbf{W}_{\Delta}^{\#}\), can we select scale factors, shadow weights and trade-off parameter \(\lambda\) (as a function of \(\mathbf{W}_{\Delta}^{\#}\)) that would guarantee that \(\mathbf{W}_{\Delta}^{\#}=\mathbf{W}_{\Delta}^{\#}\)?_

If the answer is yes (which we demonstrate) then we can gain understanding of back-propagation by analysing (15). Now, the solution to (15) must satisfy first-order optimality conditions (assuming differentiability for simplicity), so:

\[\tfrac{\partial}{\partial\mathbf{W}_{\Delta}}\left\|\mathbf{\Psi}_{\mathcal{O }}\left(\mathbf{W}_{\Delta}^{\bullet}\right)\right\|_{\mathcal{W}_{\mathcal{O }}}=\tfrac{-1}{\lambda}\tfrac{\partial}{\partial\mathbf{W}_{\Delta}}R_{E} \left(\mathbf{W}_{\Delta}^{\bullet}+\mathbf{W}_{\mathcal{O}},\mathbb{D}\right)\]

Note that _if_ the gradient of the regularisation term satisfies:

\[\tfrac{\partial}{\partial\mathbf{W}_{\Delta}}\left\|\mathbf{\Psi}_{\mathcal{O }}\left(\mathbf{W}_{\Delta}^{\#}\right)\right\|_{\mathcal{W}_{\mathcal{O}}}= \nu\mathbf{W}_{\Delta}^{\#}\]

for some \(\nu\in\mathbb{R}_{+}\), and \(\frac{1}{\lambda}=\eta\nu\), then \(\mathbf{W}_{\Delta}^{\bullet}=\mathbf{W}_{\Delta}^{\#}\). Thus the question of whether there exists scaling factors, shadow weights and \(\lambda\) such that the regularised risk minimisation weight-step corresponds to the gradient-descent weight-step for a specified learning rate \(\eta\) can be answered in the affirmative by proving the existence of canonical scalings, which we define as follows:

**Definition 2** (Canonical Scaling).: For a given neural network, initial weights \(\mathbf{W}_{\mathcal{O}}\) and weight step \(\mathbf{W}_{\Delta}^{\#}\) generated by back-propagation, we define a _canonical scaling_ to be a set of scaling factors and shadow weights for which \(\frac{\partial}{\partial\mathbf{W}_{\Delta}}\left\|\mathbf{\Psi}_{\mathcal{O}} \left(\mathbf{W}_{\Delta}^{\#}\right)\right\|_{\mathcal{W}_{\mathcal{O}}}=\nu \mathbf{W}_{\Delta}^{\#}\) for \(\nu\in\mathbb{R}_{+}\), and \(\left\|\mathbf{\Psi}_{\mathcal{O}}(\mathbf{W}_{\Delta}^{\#})\right\|_{\mathcal{W} _{\mathcal{O}}},\left\|\mathbf{\Phi}_{\mathcal{O}}(\mathbf{x})\right\|_{ \mathcal{X}_{\mathcal{O}}}<\infty\)\(\forall\mathbf{x}\in\mathbb{X}\).

To prove the existence of canonical scaling and thus the key connection between gradient descent (back-propogation) and learning in RKBS, using (14), defining:

\[t_{\Delta i_{j+1}}^{[j]\#2}=\left\{\begin{array}{ll}2b_{\Delta i_{1}}^{[0] \#2}+2\left\|\mathbf{W}_{\Delta i_{1}}^{[0]\#1}\right\|_{2}^{2}&\text{ if }j=0\\ 2b_{\Delta i_{j+1}}^{[j]\#2}+2\left\|\mathbf{W}_{\Delta i_{j+1}}^{[j]\#2}\right\| _{2}^{2}&\text{ otherwise}\end{array}\right.\]

\(\forall j\in\mathbb{N}_{D},i_{j+1}\), in the appendix we prove the following key result based on Lemmas 1 and 2:

**Theorem 1**.: _Let \(\epsilon,\chi\in(0,1)\). For a given neural network with initial weights \(\mathbf{W}_{\mathcal{O}}\), let \(\mathbf{W}_{\Delta}^{\#}\) be the weight-step for this derived from back-propagation, assuming wlog that \(\alpha^{[j]}\in\mathbb{R}_{+}\) is chosen such that \(\forall j\in\mathbb{N}_{D-1}\):5_

Footnote 5: Note that \(b_{\Delta i_{j+1}}^{[j]\#}\) is proportional to \(\alpha^{[j]}\), so we can always increase \(t_{\Delta i_{j+1}}^{[j]\#2}\) to ensure the condition holds by adjusting \(\alpha^{[j]}\).

\[\left\|\mathbf{W}_{\Delta}^{[j+1]\#}\right\|_{F}^{2}=\chi\left\|\mathbf{t}_{ \Delta}^{[j]\#2}\right\|_{\infty}^{2}\]
_Let \(\epsilon_{\psi}=1-\frac{1}{1-\chi}\frac{s^{[D-1]\mathbbm{n}}}{(1-\epsilon)\sqrt{ \rho^{[D-1]}}}\|\mathbf{t}_{\Delta}^{[D-1]\mathbbm{n}}\|_{\infty}^{2}\) and:6_

Footnote 6: The positivity of \(\epsilon_{\psi}\) is due to the constraints on \(\|\mathbf{t}_{\Delta}^{[D-1]\mathbbm{n}}\|_{\infty}^{2}\).

\[\epsilon_{\phi}^{[j]}=1-\frac{1}{\sqrt{\rho^{[j]}}}\overline{\sigma}^{[j]-1} \left(\frac{\frac{1}{1-\chi}\frac{\chi^{D-1-j_{\phi}}\left(1-\frac{[j+1]}{D} \sqrt{\rho^{[j+1]}}\right)}{\left\|\chi^{D+1}\|\mathbf{t}_{\Delta}^{[D-1] \mathbbm{n}}\right\|_{\infty}^{2}}-\frac{\chi^{D-1-j_{\phi}}}{1-\chi^{D-1-j_{ \phi}}}\left|\mathcal{F}\right|^{j+12}}{\frac{\max\left\{\frac{1}{\rho^{[j+1] }}\left[\left|\mathbf{W}_{\Delta}^{[j+1]}\right|_{\infty}^{2}\right]^{2}}{ \left\|\mathbf{W}_{\Delta}^{[j+1]}\right\|_{\infty}^{2}\right\|_{\infty}^{2}} \left\|\mathbf{W}_{\Delta}^{[j+1]}\right\|_{\infty}^{2}+\frac{H^{[j]}}{H^{[j+1 ]}}}\right)\]

\(\forall j\in\mathbb{N}_{D-1}\)_, where \(\epsilon_{\phi}^{[D-1]}=\epsilon\). If the weight-step satisfies:_

\[\left\|\mathbf{t}_{\Delta}^{[j]\mathbbm{n}}\right\|_{\infty}^{2} <B^{[j]}=\left\{\begin{array}{ll}\frac{(1-\chi)^{2}\left(1- \chi^{D-j-1}\epsilon_{\psi}\right)}{\left(\frac{\epsilon^{[j]\mathbbm{n}}}{(1- \epsilon)\sqrt{\rho^{[j]}}}\right)}&\mbox{if }j<D-1\\ \frac{(1-\chi)^{2}}{\left(\frac{\epsilon^{[D-1]}}{(1-\epsilon)\sqrt{\rho^{[D -1]}}}\right)}&\mbox{otherwise}\end{array}\right.\]

\(\forall j\in\mathbb{N}_{D}\) _then there exists of a canonical scaling:_

\[\begin{array}{ll}\frac{\partial}{\partial\mathbf{W}_{\Delta}}\left\| \mathbf{\Psi}_{\mathcal{O}}\left(\mathbf{W}_{\Delta}\right)\right\|_{\mathbf{ W}_{\Delta}=\mathbf{W}_{\Delta}^{\mathbbm{n}}}=\nu\mathbf{W}_{\Delta}^{ \mathbbm{n}}\\ \nu=\frac{4}{\left\|\mathbf{t}_{\Delta}^{[D-1]\mathbbm{n}}\right\|_{\infty}^{ 2}}\frac{(1-\epsilon_{\phi})(1-\chi)}{(\epsilon_{\phi}-\chi)^{2}}<\infty \end{array}\]

_where \(\|\mathbf{\Phi}_{\mathcal{O}}(\mathbf{x})\|_{F}^{2}\leq H^{[D-1]}\overline{ \sigma}^{[D-1]}((1-\epsilon)\sqrt{\rho^{[D-1]}})\)\(\forall\mathbf{x}\in\mathbb{X}\) and \(\|\mathbf{\Psi}_{\mathcal{O}}(\mathbf{W}_{\Delta})\|_{F}^{2}\leq H^{[D-1]} \frac{1-\epsilon_{\psi}}{\epsilon_{\psi}}\)._

This is proven as corollary 10 in the appendix. This theorem tells us that, for any set of initial weights \(\,\mathbf{W}_{\mathcal{O}}\), for a sufficiently small weight-step \(\mathbf{W}_{\Delta}^{\mathbbm{n}}\) generated by back proposition, there exists a canonical scaling - i.e. a set of scaling factors and shadow weights such that the gradient-descent weight-step is exactly equivalent to the step generated by regularised RKBS learning using an appropriate trade-off parameter \(\lambda\). Note that:

* The maximum step-size \(B^{[j]}\) in layer \(j\) (up to near-identity scaling terms) is determined by the inverse fanout \(1/s^{[j]}\) (which scales roughly as \(H^{[j]}/H^{[j-1]}\)) and the scaled radius of convergence \((1-\epsilon_{\phi}^{[j]})\sqrt{\rho^{[j]}}\) (which scales roughly inverse the the weight-step in subsequent layers). This bound will tend to get smaller as we move from the output layer back toward the input, but so too will the weight-steps in many cases due to the problem of vanishing gradients.
* The trade-off coefficient (degree of regularisation) required by this canonical scaling is: \[\lambda=\frac{1}{\eta\nu}=\frac{B^{[D-1]}}{4\eta}\left(1-\left(\frac{\frac{ \mathbbm{n}}{B^{[j]}}\mathbbm{n}}{B^{[D-1]}}\right)^{2}\right)^{2}\] as shown in figure 3. Note that (a) the degree of regularisation required to generate an equivalent RKBS weight-step is inversely proportional to the learning rate used to generate the original back-propogation weight-step and (b) larger gradient-descent weight-steps are equivalent to less regularised RKBS weight-steps, as might be expected.

## 7 Application - Rademacher Complexity

Having established an exact representation of neural networks in finite neighbourhoods of weights and biases, established the link with RKBS theory and demonstrated that a gradient descent step is equivalent to a regularised step in RKBS by appropriate, a-posterior selection of scale factors and shadow weights (canonical scaling), we now consider an application of this framework to uniform convergence analysis using Rademacher complexity. The Rademacher complexity of a set \(\mathcal{G}\) of real-valued functions is a measure of its capacity. Assuming training vectors \(\mathbf{x}_{i}\sim\nu\) and Rademacher random variables \(\epsilon_{i}\in\{-1,1\}\), the Rademacher complexity of \(\mathcal{G}\) is [10]:

\[\mathcal{R}_{N}\left(\mathcal{G}\right)=\mathbb{E}_{\omega,\epsilon}\left[ \sup_{f\in\mathcal{G}}\left|\frac{1}{N}{\sum_{i}}\epsilon_{i}f\left(\mathbf{x}_ {i}\right)\right|\right]\]

This may be used in uniform convergence analysis to bound how quickly the empirical risk converges to the expected risk, typically of the form:

\[\left|\overline{R}\left(g\right)-R_{E}\left(g\right)\right|\leq c\mathcal{R} _{N}\left(\mathcal{G}\right)+\mbox{excess risk}\]

The following theorem demonstrates how our framework may be used to bound Rademacher complexity for a scalar-output neural network:
**Theorem 2**.: _Let \(\epsilon,\chi\in(0,1)\) and for a given neural network with initial weights \(\mathbf{W}_{\mathcal{O}}\), and let \(\mathbf{W}_{\Delta}^{\boxplus}\) be the weight-step for this derived from back-propagation satisfying the conditions set out in corollary 10. Then \(\widehat{f}_{\Delta}^{\boxplus}\in\mathcal{F}^{\bullet}\), where the Rademacher complexity of \(\mathcal{F}^{\bullet}\) is bounded as:_

\[\frac{\mathcal{R}_{N}(\mathcal{F}^{\bullet})}{H^{[D-1]}}\leq\sqrt{\frac{ \sigma^{[D-1]}\left(\left(1-\epsilon\right)\sqrt{\rho^{[D-1]}}\right)}{N} \frac{\left\|\mathbf{t}_{\Delta}^{[D-1]}\right\|_{\infty}^{2}}{\frac{1}{1- \lambda}\mathbf{t}_{\Delta}^{[D-1]}\mathbf{t}_{\Delta}^{[D-1]}\mathbf{t}_{ \Delta}^{2}}}\]

The proof of this theorem is a straightforward application of theorem 1 (see proof of theorem 11 in the appendix for details). Apart from the usual \(\frac{1}{N}\) scaling this theorem is very different from typical bounds on Rademacher complexity, as it directly bounds the complexity using the size of the weight-step for a single iteration of gradient descent. Note:

* By the properties of Rademacher complexity (Bartlett and Mendelson, 2002, Theorem 12) the complexity of the trained neural network after \(T\) iterations may be bounded by a simple summation of the bounds on each step. This supports the practice of using early-stopping to prevent overfitting in neural networks.7 Footnote 7: We suspect that a more careful analysis accounting for the overlap between the RKBSs for each step may give sublinear dependence on \(T\), but this is beyond the scope of the present work.
* The size of the back-propogation weight-step scales propotially with the learning rate, so for sufficiently small learning rates we find: \[\frac{\mathcal{R}_{N}(\mathcal{F}^{\bullet})}{H^{[D-1]}}\lessapprox\sqrt{ \frac{\sigma^{[D-1]}\left(\left(1-\epsilon\right)\sqrt{\rho^{[D-1]}}\right)}{ N}}\left\|\mathbf{t}_{\Delta}^{[D-1]\boxplus}\right\|_{\infty}\] However as the learning rate increases the bound will increase at an accelerating rate.8 This supports the lower learning rates act as a form of regularisation in neural network training. Footnote 8: Eventually the bound will become meaningless as the step-size exceeds the size we can model using an RKBS.

To finish we consider a concrete example. Assume a 2-layer network with a scalar output (\(D=2\), \(n=H^{[1]}=1\)) with activation functions \(\tau^{[0]}=\tau^{[1]}=\tanh\) (so \(\rho^{[0]}=\rho^{[1]}=\frac{\pi}{2}\) and \(\tau^{(1)[0]}(\zeta),\tau^{(1)[1]}(\zeta)\in[-1,1]\). The form of \(\sigma_{z,z^{\prime}}^{[j]}\) is non-trivial (see appendix F for details), but it is not difficult to see that \(\overline{\sigma}^{[j]}=\sigma_{0,0}^{[j]}\), so, using the power-series about 0:

\[\overline{\sigma}^{[0]}\left(\zeta\right)=\overline{\sigma}^{[1]}\left(\zeta \right)=\sum_{m=1}^{\infty}\left(\frac{2^{2m}\left(2^{2m}-1\right)B_{2m}}{(2m )!}\right)^{2}\zeta^{2m-1}\]

and moreover \(s^{[0]2}=\alpha^{[0]2}/2+m/2H^{[0]}\) and \(s^{[1]2}=\alpha^{[1]2}/2+H^{[0]}\). Using our assumptions and the back-propagation equations, it is not difficult to see that:

\[\left\|\mathbf{t}_{\Delta}^{[0]\boxplus}\right\|_{\infty}^{2} \leq 2\eta^{2}N^{2}L_{k}^{2}\left\|\mathbf{W}_{\mathcal{O}}^{[1]} \right\|_{F}^{2}\left(1+\alpha^{[0]2}\right)\] \[\left\|\mathbf{t}_{\Delta}^{[1]\boxplus}\right\|_{\infty}^{2} \leq 2\eta^{2}N^{2}L_{k}^{2}\left(1+\alpha^{[1]2}\right)\]
and \(B^{[1]}=\frac{(1-\chi)^{2}(1-\epsilon)\sqrt{\frac{2}{2}}}{2\alpha^{[12}+H]^{0}}\). Assuming a learning rate:

\[\eta=s\frac{(1-\chi)}{NLE_{N}\sqrt{2(1+\alpha\nicefrac{{1}}{{2}})}}\sqrt{\frac{(1 -\epsilon)\sqrt{\frac{2}{2}}}{2\alpha^{[12}+H]^{0}}}\]

where \(0<s\ll 1\) - which follows the practice of scaling the learning rate inversely with the batch size and the network width - we obtain the following Rademacher complexity bound for the neural network trained for \(T\) iterations:

\[\frac{\mathcal{R}_{N}(\mathcal{F}^{\star})}{H^{[D-1]}}\lessapprox sT\sqrt{2 \frac{\sigma^{[D-1]}(\left(1-\epsilon\right)\sqrt{\frac{2}{2}})}{N}\frac{(1- \chi)^{2}(1-\epsilon)\sqrt{\frac{2}{2}}}{\frac{1}{2}\alpha^{[12}+H]^{0}}}\]

which we note scales inversely with the both the number of training vectors \(N\)_and_ the width of the network \(H^{[0]}\).

## 8 Conclusions and Future Directions

In this paper we have established a connection between neural network training using gradient descent and regularised learning in reproducing kernel Banach space. We have introduced an exact representation of the behaviour of neural networks as the weights and biases are varied in a finite neighbourhood of some initial weights and biases in terms of an inner product of two feature maps, one from data space to feature space, the other from weight-step space to feature space. Using this, we showed that the change in neural network behaviour due to a single iteration of back-propagation lies in a reproducing kernel Banach space, and moreover that the weight-step found by back-propagation can be exactly replicated through regularised risk minimisation in RKBS. Subsequently we presented an upper bound on the Rademacher complexity of neural networks applicable to both the over- and under-parametrised regimes, and discussed how this bound depends on learning rate, dataset size, network width and the number of training iterations used.

With regard to future work we foresee a number of useful directions. First, the analysis should be extended to non-smooth activation functions such as ReLU, presumably by modifying the feature map using a representation other than a power series expansion. Second, the precise influence of the learning rate needs to be further explicated, along with other details of Theorem 1. With regard to the neural neighbourhood kernels themselves, it would be helpful if these could be reduced to closed form to allow them to be used in practice.9 Finally, more work is needed to understand the impact of the depth of the network on this theory.

Footnote 9: A Taylor approximation the NNKs is readily obtained, but it is difficult to say how accurate this may be without further analysis.
