Function Approximation with Randomly Initialized Neural Networks for
  Approximate Model Reference Adaptive Control
Classical results in neural network approximation theory show how arbitrary
continuous functions can be approximated by networks with a single hidden
layer, under mild assumptions on the activation function. However, the
classical theory does not give a constructive means to generate the network
parameters that achieve a desired accuracy. Recent results have demonstrated
that for specialized activation functions, such as ReLUs and some classes of
analytic functions, high accuracy can be achieved via linear combinations of
randomly initialized activations. These recent works utilize specialized
integral representations of target functions that depend on the specific
activation functions used. This paper defines mollified integral
representations, which provide a means to form integral representations of
target functions using activations for which no direct integral representation
is currently known. The new construction enables approximation guarantees for
randomly initialized networks for a variety of widely used activation
functions.
Tyler Lekang, Andrew Lamperski
# Function Approximation with Randomly Initialized Neural Networks

###### Abstract

Classical results in neural network approximation theory show how arbitrary continuous functions can be approximated by networks with a single hidden layer, under mild assumptions on the activation function. However, the classical theory does not give a constructive means to generate the network parameters that achieve a desired accuracy. Recent results have demonstrated that for specialized activation functions, such as ReLUs, high accuracy can be achieved via linear combinations of _randomly initialized_ activations. These recent works utilize specialized integral representations of target functions that depend on the specific activation functions used. This paper defines _modified integral representations_, which provide a means to form integral representations of target functions using activations for which no direct integral representation is currently known. The new construction enables approximation guarantees for randomly initialized networks using any activation for which there exists an established base approximation which may not be constructive. We extend the results to the supremum norm and show how this enables application to an extended, approximate version of (linear) model reference adaptive control.

## I Introduction

Adaptive control lies at the intersection of machine learning methods and the control of dynamics systems [1], and has considerable development in the controls literature [2, 3, 4]. Deep neural network based machine learning approaches have achieved unprecedented levels of performance and utility in areas such as language processing [5], medical computer vision [6], and reinforcement learning [7]. In this paper, we aim to contribute novel ideas to function approximation techniques and apply them to an approximate extension of Model Reference Adaptive Control (MRAC) established in [8].

A key challenge in function approximation work is that many theorems describe the existence of approximations that are linear combinations of some activation function (such as a sigmoid), but are not constructive. In [9], many of these results are reviewed, including classic results on the universal approximation properties of continuous sigmoidal activations [10, 11, 12], as well as another classic result by [13] for arbitrary sigmoidal activations, which was expanded to arbitrary hinge functions by [14].

Random approximation provides a key way to avoid the non-constructiveness by randomly sampling the internal parameters of the activations from a known set, if the target function of interest has an integral representation. However, determining these representations for general classes of target functions and activations is a nontrivial task. [15] gives a constructive method, but only for target and activation functions in \(L_{1}\). In [16] and [17], they propose constructive methods for a class of target functions with unit step and ReLU activations respectively. In [18], functions are approximated using trigonometric polynomial ridge functions, which can then be shown in expectation to be equivalent to randomly initialized ReLU activations.

There are several interesting and important results in the literature having to do with neural networks with random (typically Gaussian) parameter initialization, sometimes as a consequence of using randomly initialized gradient descent for training the network. A classic result by [19] shows that the output of a single hidden-layer network with Gaussian randomly initialized parameters goes to a Gaussian Process as the width goes to infinity. Similar results are then achieved in [20] for deep fully-connected networks as all hidden layer widths go to infinity. Also for deep fully-connected networks, in [21] the authors define the Neural Tangent Kernel and propose that its limit, as the hidden layer widths go to infinity, can be used to study the timestep evolution and dynamics of the parameters, and the corresponding network output function, in gradient descent. In [22], the authors show that single hidden-layer networks cannot achieve the same rates of increase in a measure of curvature produced by the network output, as deep networks can, with parameters Gaussian randomly initialized and bounded activation functions. In [23], the authors show that single hidden-layer networks of a sufficient width can use Gaussian randomly initialized gradient descent on values of a target function, and achieve guaranteed generalization to the entire function. And in [24], the authors show that deep fully-connected networks, where each hidden layer width meets a sufficient size, can be trained with Gaussian randomly initialized gradient descent and be guaranteed to reach the global minimum at a linear rate.

Our primary contributions in this paper are: developing a novel method for bridging convex combinations of activation functions (eg. step and ReLU) with unknown parameters to approximations using randomly initialized parameters by using a mollified integral representation, obtaining a main overall result bounding the approximation error for the random approximations in the \(L_{2}\) norm, extending these results to the \(L_{\infty}\) norm under certain conditions, and then applying those results to an approximate extension of MRAC adaptive control.

The organization of the remaining parts of the paper are as follows. Section II presents preliminary notation. Section III presents background on function approximation with neural
networks. Section IV presents theoretical results on mollified approximations, while Section V gives an overall approximation error bound in \(L_{2}\) norm for randomly initialized approximations. Section VI presents an extension to \(L_{\infty}\) norm bounds and an application on approximate MRAC, and we provide closing remarks in Section VII.

## II Notation

We use \(\mathbb{R},\mathbb{N}\) to denote the real and natural numbers. \(\mathbb{I}\) denotes the indicator function, while \(\mathbb{E}\) denotes the expected value. We use \(\mathcal{B},\mathcal{K}\) to denote bounded and compact convex subsets of \(\mathbb{R}^{n}\). The integer set \(\{1,\ldots,k\}\) is denoted by \([k]\). \(B_{x}(r)\subset\mathbb{R}^{n}\) denotes the radius \(r\) Euclidean ball centered at \(x\in\mathbb{R}^{n}\). We interpret \(w,x\in\mathbb{R}^{n}\) as column vectors and denote their inner product as \(w^{\top}x\). Similarly, we denote the product of matrix \(W\in\mathbb{R}^{n\times N}\) with vector \(x\) as \(W^{\top}x\), which is a length \(N\) vector where the \(i\)th element is the inner product \(W^{\top}_{i}x\). Subscripts on vectors and matrices denote the row index, for example \(W^{\top}_{i}\) is the \(i\)th row of \(W^{\top}\). The standard Euclidean norm is denoted \(\|w\|\). We use subscripts on time-dependent variables to reduce parentheses, for example \(\phi(x(t))\) is instead denoted \(\phi(x_{t})\). The \(i\)th row of a time-varying vector or matrix is then \(x_{t,i}\).

## III Background on Function Approximation with Neural Networks

In this paper, we consider the approximation of target functions \(f:\mathbb{R}^{n}\to\mathbb{R}\) that are continuous on a bounded set \(\mathcal{B}\subset\mathbb{R}^{n}\) containing the origin, which we denote as \(f\in C(\mathcal{B})\).

Formally, for a given target function \(f\in C(\mathcal{B})\), positive integer \(N\in\mathbb{N}\), and positive scalar \(\varepsilon_{N}>0\) (which may depend on \(N\)) there exists some vector \(\theta\in\mathbb{R}^{N}\) and nonlinear, continuous almost everywhere basis functions \(\mathcal{B}_{1},\ldots,\mathcal{B}_{N}:\mathbb{R}^{n}\to\mathbb{R}\) such that the approximation

\[\widehat{f}(x)\ :=\ \sum_{i=1}^{N}\theta_{i}\,\mathcal{B}_{i}(x) \tag{1}\]

satisfies

\[\left\|\widetilde{f}(x)-\widehat{f}(x)\right\|_{\mathcal{B}}\ \leq\ \varepsilon_{N}\ . \tag{2}\]

Here, we denote the the \(L_{2}(\mathcal{B},\bar{\mu}_{\mathcal{B}})\) norm

\[\left\|\widetilde{f}(x)\right\|_{\mathcal{B}}\ :=\ \left(\int_{B}|\widetilde{f}(x)|^{2}\, \bar{\mu}_{\mathcal{B}}(\text{d}x)\right)^{\frac{1}{2}}\]

with \(\bar{\mu}_{\mathcal{B}}\) as the uniform probability measure over \(\mathcal{B}\) such that \(\int_{\mathcal{B}}\bar{\mu}_{\mathcal{B}}(\text{d}x)=1\), and some _affine shift_\(\widetilde{f}(x)=f(x)-a\,^{\top}\!x-\!\ell\), for some \(a\in\mathbb{R}^{n}\) and \(\ell\in\mathbb{R}\).

We will assume that the basis function is parameterized \(\mathcal{B}(x\,;\gamma)\) and is a composition of some nonlinear, continuous almost everywhere function \(\sigma:\mathbb{R}\to\mathbb{R}\) together with an affine transformation of \(\mathbb{R}^{n}\) into \(\mathbb{R}\) according to the _parameter_\(\gamma\in\Upsilon\). We define each point \(\gamma:=[w^{\top}\ b]^{\top}=[w_{1}\ \ldots\ w_{n}\ b]^{\top}\) with \(w\in\mathbb{R}^{n}\backslash\{0_{n}\}\) as the _weight vector_ and \(b\in\mathbb{R}\) as the _bias term_. We assume the _parameter set_\(\Upsilon\) is a bounded subset of \(\mathbb{R}^{n}\backslash\{0_{n}\}\times\mathbb{R}\). And so, we set \(\mathcal{B}(x\,;\gamma_{i})=\sigma(w^{\top}_{i}x+b_{i})=\sigma(\gamma^{\top}_ {i}X)\), where \(X:=[x_{1}\ \cdots\ x_{n}\ 1]^{\top}\) and we denote \(\gamma_{i}=[w^{\top}_{i}\ b_{i}]^{\top}=[w_{i,1}\ \ldots\ w_{i,n}\ b_{i}]^{\top}\), for all \(i\in[N]\).

We will use the term _base approximation_ to define, for some vector \(\theta\in\mathbb{R}^{N}\) and _fixed_ parameters \(\gamma_{1},\ldots,\gamma_{N}\in\Upsilon\), for any integer \(N\in\mathbb{N}\), the sum

\[\widehat{f}_{N}(x)=\sum_{i=1}^{N}\theta_{i}\,\sigma(\gamma^{\top}_{i}X)\ \, \tag{3}\]

and _random approximation_ to define, for some vector \(\vartheta\in\mathbb{R}^{R}\) and _randomly sampled_ parameters \(\gamma_{1},\ldots,\gamma_{R}\in\Upsilon\), for any integer \(R\in\mathbb{N}\), the sum

\[\widehat{\mathcal{H}}_{R}(x\,;\gamma_{1},\ldots,\gamma_{R})=\sum_{j=1}^{R} \vartheta_{j}\,\sigma(\boldsymbol{\gamma}^{\top}_{j}X)\ . \tag{4}\]

Note that these linear combinations are equivalent to a feedforward neural network with a single hidden layer, as depicted in Figure 1, where \(x_{0}=1\). Hence why we refer to \(\sigma\) as an _activation function_, as the output of each neuron is the activation of the weighted inputs (plus bias term) to the neuron. We will assume that the activation function \(\sigma\) is bounded (growth) as follows.

**Assumption 1**: The nonlinear, continuous almost everywhere activation function \(\sigma:\mathbb{R}\to\mathbb{R}\) is bounded (growth) over all feasible inputs, such that either of

1. \(|\sigma(u)-\sigma(v)|\ \leq\ \mathcal{O}_{\sigma}\)__
2. \(|\sigma(u)-\sigma(v)|\ \leq\ \mathcal{L}_{\sigma}\,|u-v|\)__

_hold for all \(u,v\in\mathcal{I}:=\{\gamma^{\top}\!X\ |\ (\gamma,X)\in\Upsilon\times \mathcal{B}\times\{1\}\}\), with positive scalar \(\mathcal{O}_{\sigma}\) or \(\mathcal{L}_{\sigma}\)._

If we shift the activation to the origin as \(\breve{\sigma}(y):=\sigma(y)-\sigma(0)\), this maintains the assumption with the same constant in either case. The (scaled) step function \(\sigma_{c\mathcal{S}}\) and the ReLU function \(\sigma_{\mathcal{R}}\), defined as

\[\sigma_{c\mathcal{S}}(y) =\begin{cases}0&\text{if }y\leq 0\\ c&\text{if }y>0\end{cases}\qquad\text{(scaled step)} \tag{5}\] \[\sigma_{\mathcal{R}}(y) =\begin{cases}0&\text{if }y\leq 0\\ y&\text{if }y>0\end{cases}\qquad\text{(ReLU)}\ \, \tag{6}\]

are examples meeting Assumption 1, with \(\mathcal{O}_{\sigma}=c\) and \(\mathcal{L}_{\sigma}=1\) respectively, where \(c>0\) is an arbitrary positive scalar.

Fig. 1: Single hidden-layer neural network with \(\sigma\) activation function.
## IV Modified Approximations

In this section, we define the _mollified approximation_\(\widehat{f}_{\lambda,N}\) using a novel method of forming a _mollified integral representation_ for a base approximation \(\widehat{f}_{N}\) to a target function \(f\in C(\mathcal{B})\), and bound their difference in the \(L_{2}(\mathcal{B},\bar{\mu}_{\mathcal{B}})\) norm.

### _Functions with Integral Representation_

If a target function \(f:\mathbb{R}^{n}\to\mathbb{R}\) has an _integral representation_ over the parameters set \(\Upsilon\) with some activation function \(\sigma:\mathbb{R}\to\mathbb{R}\), then

\[f(x)\ =\ \int_{\Upsilon}\,g(\gamma)\,\sigma(\gamma^{\top}X)\,\mathrm{d}\gamma \tag{7}\]

holds with some bounded, continuous _coefficient function_\(g:\Upsilon\to\mathbb{R}\). This is a version of (3) that is continuous in the parameters \(\gamma_{i}\) over \(\Upsilon\). Indeed, if we define \(\gamma_{1},\ldots,\gamma_{N}\) to be a sequence of points that uniformly grids \(\Upsilon\) with volume \(\mathrm{d}_{N}\) between any neighboring points, then as \(N\to\infty\) (and \(\mathrm{d}_{N}\to 0\)) we have

\[\lim_{N\to\infty}\sum_{i=1}^{N}g(\gamma_{i})\,\sigma(\gamma_{i}^{\top}X)\, \mathrm{d}_{N}=\int_{\Upsilon}\,g(\gamma)\,\sigma(\gamma^{\top}X)\,\mathrm{d} \gamma=f(x)\.\]

This implies that the (infinite) sum of the absolute values of the coefficients is finite, with

\[\lim_{N\to\infty}\sum_{i=1}^{N}|g(\gamma_{i})|\,\mathrm{d}_{N}\ =\ \int_{\Upsilon}|g(\gamma)|\,\mathrm{d}\gamma\ <\ \infty\]

holding since \(g\) is bounded and the integral is over the bounded set \(\Upsilon\).

However, finding such integral representations (7) for general classes of functions \(f\) and activation functions \(\sigma\) is a nontrivial task. If _both_ are \(L_{1}\) (absolutely integrable over \(\mathbb{R}^{n}\) and \(\mathbb{R}\) respectively), then Theorem 1 in [15] shows that the integral representation can always be constructed with \(g\) related to the Fourier transforms of \(f\) and \(\sigma\). But this is a limiting restriction, since none of the typical activation functions like sigmoids, steps, ReLU's, and their variants meet this requirement. Even so, in [12] this was used to prove the universal approximation properties of sigmoid activations.

### _Mollified Integral Representation_

Now we will propose a novel method of forming an integral representation for _any_ function \(f\in C(\mathcal{B})\), so long as a base approximation \(\widehat{f}_{N}\) of the form (3) exists and the bounded parameter set \(\Upsilon\subset\mathbb{R}^{n+1}\) and a bound \(\mathcal{S}_{N}\) on the coefficient size \(\sum_{i=1}^{N}|\theta_{i}|\) are known.

For any fixed parameters \(\gamma_{1},\ldots,\gamma_{N}\in\Upsilon\), this approximation \(\widehat{f}_{N}\) can be defined as

\[\widehat{f}_{N}(x) =\ \sum_{i=1}^{N}\theta_{i}\,\sigma(\gamma_{i}^{\top}X)\] \[=\ \int_{\Upsilon}\,\sum_{i=1}^{N}\theta_{i}\,\delta^{n+1}( \gamma-\gamma_{i})\ \sigma(\gamma^{\top}X)\,\mathrm{d}\gamma\ \, \tag{8}\]

where \(\delta^{n+1}\) is the \((n+1)\)-dimensional Dirac delta which satisfies for any \(\gamma_{i}\in\Upsilon\) that \(\int_{\Upsilon}\,\delta^{n+1}(\gamma-\gamma_{i})\,\sigma(\gamma^{\top}X)\, \mathrm{d}\gamma=\sigma(\gamma_{i}^{\top}X)\.\)

Thus, the RHS of (8) is like an integral representation (7) of \(\widehat{f}_{N}\) with a coefficient mapping \(g(\gamma)=\sum_{i=1}^{N}\theta_{i}\,\delta^{n+1}(\gamma-\gamma_{i})\). However, this mapping is not continuous and bounded as required, and the Dirac delta is not even a function in the regular sense. Indeed, if the overall coefficients of a random approximation \(\widehat{\mathcal{G}}_{R}\) were set as \(\vartheta_{j}=g(\boldsymbol{\gamma}_{j})\) for each randomly sampled \(\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R}\in\Upsilon\), then in this case they almost surely would all be zero.

We propose the _Mollified integral representation_ to solve this issue, where the Dirac delta in (8) is replaced with the _mollified_ delta function \(\hat{\delta}_{\lambda}^{n+1}\). Recalling that \(\gamma=[w_{1}\ \cdots\ w_{n}\ b\ ]\), this is defined as the \((n+1)\)-dimensional bump function

\[\hat{\delta}_{\lambda}^{n+1}(\gamma)\ :=\] (9) \[\left\{\begin{aligned} &(\eta\lambda)^{n+1}\prod_{i=1}^{n} \exp\left(\frac{-1}{1-\lambda^{2}w_{i}^{2}}\right)\exp\left(\frac{-1}{1- \lambda^{2}b^{2}}\right)\\ &\gamma\in(-\frac{1}{\lambda},\frac{1}{\lambda})^{n+1}\\ & 0\
that \(\mathop{\mathchoice{\vbox{\hbox{$\int$}}\hbox{\vbox{\hbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \int$}}\hbox{\vbox{$\int$}}\hbox{\vbox{$ \
For the corresponding \(\Upsilon_{\lambda}\), we define the diameter \(D_{\Upsilon_{\lambda}}:=\sup\{\|\boldsymbol{\gamma}-\phi\|_{2}\mid\boldsymbol{ \gamma},\phi\in\Upsilon_{\lambda}\}\) and largest point \(\gamma_{\lambda\max}:=\sup\{\|\boldsymbol{\gamma}\|_{2}\mid\boldsymbol{\gamma} \in\Upsilon_{\lambda}\}\), and for the selected \(P_{\lambda}\) we define the minimum \(P_{\lambda\min}:=\inf\{P_{\lambda}(\boldsymbol{\gamma})\mid\boldsymbol{\gamma} \in\Upsilon_{\lambda}\}\).

Then, note that the RHS of (11) is equivalently

\[\begin{split}&\widehat{f}_{\lambda,N}(x)\ =\\ &\int_{\Upsilon_{\lambda}}\frac{g_{\lambda}(\boldsymbol{\gamma})}{P _{\lambda}(\boldsymbol{\gamma})}\,\sigma(\boldsymbol{\gamma}^{\top}z)\,P_{ \lambda}(\boldsymbol{\gamma})\text{d}\boldsymbol{\gamma}=\mathop{\mathbb{E}} _{\boldsymbol{\gamma}\sim P_{\lambda}}\left[\frac{g_{\lambda}(\boldsymbol{ \gamma})}{P_{\lambda}(\boldsymbol{\gamma})}\,\sigma(\boldsymbol{\gamma}^{\top} z)\right]\.\end{split}\]

Thus, a random approximation \(\widehat{\mathcal{H}}_{R}\) exists which satisfies (16) by setting its coefficients, for all \(j\in[R]\), as

\[\vartheta_{j}=\frac{g_{\lambda}(\boldsymbol{\gamma}_{j})}{P_{\lambda}( \boldsymbol{\gamma}_{j})\,R}\ =\ \frac{\sum_{i=1}^{N}\theta_{i}\,\hat{\delta}_{\lambda}^{n+1}(\boldsymbol{\gamma }_{j}-\boldsymbol{\gamma}_{i})}{P_{\lambda}(\boldsymbol{\gamma}_{j})\,R}\ \.\]

### _Random Approximations to Base Approximations_

We now state and prove an overall bound on the approximation error in \(L_{2}(\mathcal{B},\bar{\mu}_{\mathcal{B}})\) norm, between a base approximation \(\widehat{f}_{N}\) of the form (3) to a target function \(f\in\mathcal{C}(\mathcal{B})\) and a random approximation \(\widehat{\mathcal{H}}_{R}(x\,;\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{ \gamma}_{R})\) of the form (4).

**Theorem 2**: _Let there exist a base approximation \(\widehat{f}_{N}\) of the form (3), for some integer \(N\in\mathbb{N}\), to a target function \(f\in\mathcal{C}(\mathcal{B})\), with unknown parameters \(\gamma_{1},\ldots,\gamma_{N}\) in a known bounded parameter set \(\Upsilon\subset\mathbb{R}^{n+1}\), unknown coefficients \(\theta\in\mathbb{R}^{N}\) with a known bound on the coefficient size \(\sum_{i=1}^{N}|\theta_{i}|\leq\mathcal{S}_{N}\), and using an activation function \(\sigma\) satisfying Assumption 1. Assume that \(\left\|\widetilde{f}(x)-\widehat{f}_{N}(x)\right\|_{\mathcal{B}}\leq\varepsilon _{N}\) holds for some \(\varepsilon_{N}>0\) which may depend on \(N\), for an affine shift \(\widetilde{f}(x)=f-a^{\top}x-\xi f(0)\) with some \(a\in\mathbb{R}^{n}\) and \(\xi\in\mathbb{R}\). The mollified approximation \(\widehat{f}_{\lambda,N}\) as in (11) satisfies the results of Theorem 1 for any mollification factor \(\lambda>0\). Let the random parameters \(\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R}\) be sampled iid according to any nonzero density \(P_{\lambda}\) over the \(\lambda\)-expanded set \(\Upsilon_{\lambda}\), which has diameter \(D_{\Upsilon_{\lambda}}\) and farthest point \(\gamma_{\lambda\max}\), and let \(E_{\mathcal{B}}\) be as in (13). Then for any \(\nu\in(0,1)\), with probability greater than \(1-\nu\) there exists a random approximation \(\widehat{\mathcal{H}}_{R}(x\,;\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{ \gamma}_{R})\) as in (4) with \(|\vartheta_{j}|\leq\frac{g_{\lambda\max}}{P_{\lambda\min}R}\) for all \(j\in[R]\) such that either of the following hold:_

\[\begin{split}&\left\|\,\widetilde{f}(x)-\widehat{\mathcal{H}}_{R}(x \,;\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\right\|_{ \mathcal{B}}\ \leq\\ i)&\varepsilon_{N}+\mathcal{S}_{N}\mathcal{D}_{ \sigma}\left(1\ +\ \frac{\lambda^{n+1}}{\sqrt{R}}\,K_{\partial_{\sigma}}(\nu)\right)\\ ii)&\varepsilon_{N}+\mathcal{S}_{N}\mathcal{L}_{ \sigma}\left(\frac{\sqrt{n+1}}{\lambda}\,E_{\mathcal{B}}+\frac{\lambda^{n+1} }{\sqrt{R}}\,K_{\mathcal{L}_{\sigma}}(\nu)\right)\ \ \.\end{split}\]

_The coefficients \(K_{\partial_{\sigma}}\) and \(K_{\mathcal{L}_{\sigma}}\) are defined in terms of the selected \(\nu\) as:_

\[\begin{split}& K_{\partial_{\sigma}}(\nu)\ :=\ \frac{\eta^{n+1}}{e^{n+1}P_{\lambda\min}}\left(1+\frac{|\sigma(0)|}{ \mathcal{D}_{\sigma}}\right.\\ &\left.+\bigg{(}3+2\frac{|\sigma(0)|}{\mathcal{D}_{\sigma}} \bigg{)}\sqrt{\frac{1}{2}\log\left(\frac{1}{\nu}\right)}\right)\\ & K_{\mathcal{L}_{\sigma}}(\nu)\ :=\ \frac{\eta^{n+1}}{e^{n+1}P_{ \lambda\min}}\left(\gamma_{\lambda\max}E_{\mathcal{B}}+\frac{|\sigma(0)|}{ \mathcal{L}_{\sigma}}\right.\\ &\left.+\bigg{(}(D_{\Upsilon_{\lambda}}+2\gamma_{\lambda\max})E_{ \mathcal{B}}+2\frac{|\sigma(0)|}{\mathcal{L}_{\sigma}}\bigg{)}\sqrt{\frac{1} {2}\log\left(\frac{1}{\nu}\right)}\right).\end{split}\]

Given in Appendix IV.

Proof SketchWe closely follow the strategy in Lemma 4 of [25], which uses McDiarmid's inequality to obtain the final result.

We start by defining function \(h_{\mathcal{B}}:\Upsilon_{\lambda}^{R}\to\mathbb{R}\), which captures the norm difference between the random approximation and its expected value over the iid draws of \((\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\in\Upsilon_{\lambda}^ {R}\), as

\[h_{\mathcal{B}}(\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\ :=\\ \left\|\,\widehat{\mathcal{H}}_{R}(x\,;\boldsymbol{\gamma}_{1}, \ldots,\boldsymbol{\gamma}_{R})-\mathop{\mathbb{E}}_{\boldsymbol{\gamma}_{j} \sim P_{\lambda}}\left[\widehat{\mathcal{H}}_{R}(x\,;\boldsymbol{\gamma}_{1}, \ldots,\boldsymbol{\gamma}_{R})\right]\right\|_{\mathcal{B}}\ \.\]

Thus, we can use Lemma (1) to show that there exists \(\widehat{\mathcal{H}}_{R}\) such that (18) is equivalently

\[h_{\mathcal{B}}(\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\ =\ \left\|\,\widehat{f}_{\lambda,N}(x)-\widehat{ \mathcal{H}}_{R}(x\,;\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R}) \right\|_{\mathcal{B}}\.\]

We then can bound the expected value

\[\mathop{\mathbb{E}}_{\boldsymbol{\gamma}_{j}\sim P_{\lambda}}\Big{[}h_{ \mathcal{B}}(\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\Big{]}\]

and the absolute value of the difference between any

\[\left|h_{\mathcal{B}}(\boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{j}, \ldots,\boldsymbol{\gamma}_{R})-h_{\mathcal{B}}(\boldsymbol{\gamma}_{1},\ldots, \widehat{\boldsymbol{\gamma}}_{j},\ldots,\boldsymbol{\gamma}_{R})\right|\]

that differ only at one coordinate, over each \(j\in[R]\). These two bounds then allow us to apply the results of McDiarmid's inequality, thus bounding (in high probability)

\[\left\|\,\widehat{f}_{\lambda,N}(x)-\widehat{\mathcal{H}}_{R}(x\,;\boldsymbol{ \gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\right\|_{\mathcal{B}}\.\]

The proof is then completed by using the triangle inequality, with

\[\begin{split}&\left\|\,\widetilde{f}(x)-\widehat{\mathcal{H}}_{R}(x\,; \boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\right\|_{\mathcal{B}}\ \leq\ \left\|\widetilde{f}(x)-\widehat{f}_{N}(x)\right\|_{\mathcal{B}}+\\ &\left\|\widehat{f}_{N}(x)-\widehat{f}_{\lambda,N}(x)\right\|_{ \mathcal{B}}+\left\|\widehat{f}_{\lambda,N}(x)-\widehat{\mathcal{H}}_{R}(x\,; \boldsymbol{\gamma}_{1},\ldots,\boldsymbol{\gamma}_{R})\right\|_{\mathcal{B}}\.\end{split}\]

where the first term is assumed bounded by \(\varepsilon_{N}\) for the base approximation \(\widehat{f}_{N}\) and the second term is bounded by Theorem 1.

**Remark 2**: _Classic results from [13] and [14] prove the existence of convex combinations of step (5) and ReLU (6) activations that approximate (affine shifts of) functions in specific classes arbitrarily well, with an approximation error proportional to \(\varepsilon_{N}=O\left(\frac{1}{\sqrt{N}}\right)\). Importantly, the convexity property allows the length (number of terms) \(N\) to
### _Extending the Error Bound to \(L_{\infty}\). for Lipschitz Functions on Convex Compact Sets_

We assume that \(\mathcal{K}\) is full dimension, with nonzero Lebesgue measure (volume) \(\mathcal{V}=\mu(\mathcal{K})>0\) and diameter \(\mathcal{D}>0\). It also must hold that \(\mathcal{K}\subseteq B_{0}(\rho)\) for a sufficiently large radius \(0<\rho\leqslant\mathcal{D}\).

Then, for any approximation \(\widehat{f}_{N}\) of the form (3) using an activation function \(\sigma\) satisfying Assumption 1, we have that

\[\left|\sum_{i=1}^{N}\theta_{i}\,\sigma(\gamma_{i}^{\top}X)-\sum_{i= 1}^{N}\theta_{i}\,\sigma(\gamma_{i}^{\top}Y)\right|\] \[\leqslant\sum_{i=1}^{N}\left|\theta_{i}\,\sigma(\gamma_{i}^{\top} X)-\theta_{i}\,\sigma(\gamma_{i}^{\top}Y)\right|\] \[=\sum_{i=1}^{N}\left|\theta_{i}\right||\sigma(\gamma_{i}^{\top}X)- \sigma(\gamma_{i}^{\top}Y)\big{|}\leqslant\sum_{i=1}^{N}\left|\theta_{i} \right||\mathcal{L}_{\sigma}\left|\gamma_{i}^{\top}X-\gamma_{i}^{\top}Y\right|\] \[=\sum_{i=1}^{N}\left|\theta_{i}\right||\mathcal{L}_{\sigma} \left|w_{i}^{\top}(x-y)\right|\ \leqslant\ \sum_{i=1}^{N}\left|\theta_{i}\right||w_{i}\big{|}_{2}\ \left|x-y\right|_{2}\]

holds for all \(x,y\in\mathbb{R}^{n}\), recalling that \(X:=[x_{1}\ \cdots\ x_{n}\ 1]^{\top}\) and \(Y:=[y_{1}\ \cdots\ y_{n}\ 1]^{\top}\), since \(\left|\sigma(u)-\sigma(v)\right|\leqslant\mathcal{L}_{\sigma}\left|u-v\right|\) for any \(u,v\in\mathbb{R}\) by the Lipschitz condition of Assumption 1. Thus, such approximations are \(\widehat{\mathcal{L}}\)-Lipschitz with constant

\[\widehat{\mathcal{L}}\ =\ \mathcal{L}_{\sigma}\sum_{i=1}^{N}\left|\theta_{i} \right||w_{i}\big{|}_{2}\ \ . \tag{19}\]

A similar calculation gives that such approximations are \(\widehat{\mathcal{D}}\)-bounded with constant

\[\widehat{\mathcal{D}}\ =\ \mathcal{D}_{\sigma}\sum_{i=1}^{N}\left|\theta_{i}\right| \tag{20}\]

if \(\left|\sigma(u)-\sigma(v)\right|\leqslant\mathcal{D}_{\sigma}\) for any \(u,v\in\mathbb{R}\) by the bounded condition of Assumption 1.

Let an affine shift to function \(f\) be defined as

\[\widetilde{f}(x)\ :=\ f(x)-a^{\top}x-\xi \tag{21}\]

for any \(a\in\mathbb{R}^{n}\) and \(\xi\in\mathbb{R}\). Thus, if \(f\) is \(\mathcal{L}\)-Lipschitz, then

\[\left|\widetilde{f}(x)-\widetilde{f}(y)\right|\ =\ \left|f(x)-f(y)-a^{\top}(x-y)\right|\] \[\leqslant\ \left|f(x)-f(y)\right|+\left|a^{\top}(x-y)\right|\] \[\leqslant\ \mathcal{L}\left\|x-y\right\|_{2}+\left|a\right\|_{2} \left\|x-y\right\|_{2}\]

holds for all \(x,y\in\mathbb{R}^{n}\). And so, \(\widetilde{f}\) is also Lipschitz, with constant \(\widetilde{\mathcal{L}}=\mathcal{L}+\left\|a\right\|_{2}\).

**Lemma 2**: _Let \(f:\mathbb{R}^{n}\to\mathbb{R}\) be \(\mathcal{L}\)-Lipschitz on a convex compact set \(\mathcal{K}\subset\mathbb{R}^{n}\) containing the origin and differentiable at the origin. Let there also be an approximation \(\widetilde{f}:\mathbb{R}^{n}\to\mathbb{R}\) which is \(\widehat{\mathcal{D}}\)-bounded or \(\widehat{\mathcal{L}}\)-Lipschitz on \(\mathcal{K}\) and such that the approximation error \(\widetilde{f}(x)=\widetilde{f}(x)-\widehat{f}(x)\), with \(\widetilde{f}\) as in (21), satisfies the \(L_{2}(\mathcal{K},\bar{\mu}_{\mathcal{K}})\) norm bound_

\[\left\|\widetilde{f}\right\|_{\mathcal{K}}=\sqrt{\int_{\mathcal{K}}\left| \widetilde{f}(x)\right|^{2}\bar{\mu}_{\mathcal{K}}(\mathsf{d}x)}\ \leqslant\ \varepsilon_{N}\]

_where the bound \(\varepsilon_{N}>0\) is a finite constant and \(\bar{\mu}_{\mathcal{K}}(\mathsf{d}x)\) is the uniform probability measure on \(\mathcal{K}\) with \(\int_{\mathcal{K}}\bar{\mu}_{\mathcal{K}}(\mathsf{d}x)=1\)._

_Assume \(\mathcal{K}\) is full dimension with diameter \(\mathcal{D}>0\), and it holds that \(\mathcal{K}\subseteq B_{0}(\rho)\) for a ball of sufficiently large radius \(0<\rho\leqslant\mathcal{D}\). Then, the approximation error also satisfies the \(L_{\infty}(\mathcal{K})\) norm bound_

\[\sup_{x\in\mathcal{K}}\left|\widetilde{f}(x)\right|\ \leqslant\ 2\ \left(\frac{r}{\mathcal{D}(r+\sqrt{r^{2}+ \mathcal{D}^{2}})}\right)^{\frac{-n}{n+2}}\ K(\varepsilon_{N}) \tag{22}\]

_where_

\[K(\varepsilon_{N}) =\] \[i) \left(\left(\mathcal{L}+\left\|a\right\|_{2}\right)^{n}\varepsilon _{N}^{2}\right)^{\frac{1}{n+2}}+\widehat{\mathcal{D}}\] \[ii) \left(\left(\mathcal{L}+\left\|a\right\|_{2}+\widehat{\mathcal{L }}\right)^{n}\varepsilon_{N}^{2}\right)^{\frac{1}{n+2}}\]

_and \(r\) is the largest radius ball within \(\mathcal{K}\) centered at its centroid._

Given in Appendix V-B.

### _Application to MRAC_

In Chapter 12 of [8], an approximate extension to the general linear MRAC setup (see Chapter 9) is introduced which allows for nonlinearities \(f:\mathbb{R}^{n}\to\mathbb{R}^{\ell}\) as \(f(x)\ =\ \Theta^{\top}\Psi(x)+\epsilon_{f}(x)\) such that the plant is given by

\[\dot{x}_{t} =\ A\,x_{t}+B\big{(}u_{t}+f(x)\big{)} \tag{23}\] \[=\]

where \(A\) is a known \(n\times n\) state matrix for the plant state \(x_{t}\in\mathbb{R}^{n}\), \(B\) is a known \(n\times\ell\) input matrix for the input \(u_{t}\in\mathbb{R}^{\ell}\), and \(\Theta\) is an unknown \(N\times\ell\) matrix which linearly parameterizes the known vector function \(\Psi:\mathbb{R}^{n}\to\mathbb{R}^{N}\). We assume \((A,B)\) is controllable. The general setup also includes an unknown diagonal scaling matrix \(\Lambda\), such that the overall input matrix is \(B\Lambda\), and assumes that \(A\) is unknown. For simplicity, we assume \(A\) is known and omit \(\Lambda\).

It is assumed that there exists an \(n\times\ell\) matrix of feedback gains \(K_{x}\) and an \(\ell\times\ell\) matrix of feedforward gains \(K_{r}\) satisfying the _matching conditions_

\[A+BK_{x}^{\top} =A_{r}\] \[BK_{r}^{\top} =B_{r}\]

to a controllable, linear reference model

\[\dot{x}_{t}^{r}=A_{r}\,x_{t}^{r}+B_{r}\,r_{t}\ \,\]

where \(A_{r}\) is a known Hurwitz \(n\times n\) reference state matrix for the reference state \(x_{t}^{r}\in\mathbb{R}^{n}\), \(B_{r}\) is a known \(n\times\ell\) reference input matrix, and \(r_{t}\in\mathbb{R}^{\ell}\) is a bounded reference input. Here, we assume that \(K_{x}\) and \(K_{r}\) can be directly calculated from known \(A\) and \(B\), and used directly in the control law.

It is required that the nonlinearity satisfy the bound

\[\left\|\epsilon_{f}(x)\right\|_{2}\ \leqslant\ \bar{\varepsilon} \tag{24}\]

for some constant \(\bar{\varepsilon}>0\) and for all \(x\in B_{0}(r)\) with some radius \(r>0\). Then, the adaptive control law

\[u_{t} = K_{x}^{\top}x_{t}-\widehat{\Theta}_{t}^{\top}\Psi(x_{t})+\left(1- \mu(x)\right)K_{r}^{\top}r_{t}+\mu(x)\widetilde{u}(x_{t})\]
is shown to stabilize the state tracking error \(e_{t}=x_{t}-x_{t}^{r}\) down to a compact set about the origin, where the \(N\times\ell\) matrix of parameter estimates \(\hat{\Theta}_{t}\) is dynamically updated with the update rule

\[\hat{\hat{\Theta}}_{t}=\Gamma\,\Psi(x_{t})\,e_{t}^{\top}P_{x}B\ \.\]

The scalar function \(\mu:\mathbb{R}^{n}\rightarrow[0,1]\) transitions the control law from tracking the reference input to simply returning the plant state \(x_{t}\) to the bounded region \(B_{0}(r)\) within which (24) is valid, using an appropriately defined \(\widetilde{u}(x_{t})\) based on assumptions about the growth of \(\epsilon_{f}(x)\) outside of \(B_{0}(r)\). (See Chapter 12 of [8] for details.)

And so, if the vector function \(\Psi(x)\) is constructed as

\[\Psi(x)=\begin{bmatrix}\sigma(\gamma_{1}^{\top}X)\\ \vdots\\ \sigma(\gamma_{N}^{\top}X)\end{bmatrix}\, \tag{25}\]

with an activation function \(\sigma\) satisfying the bounded (growth) conditions of Assumption 1 with constant \(\mathcal{D}_{\sigma}\) or \(\mathcal{L}_{\sigma}\), then the above setup is valid for any nonlinearity \(f=[f_{1}\ \cdots\ f_{\ell}]\) where we can show that

\[\sup_{x\in\mathcal{K}}\left|f_{i}(x)-\Theta_{i}^{\top}\Psi(x)\right|\ \leqslant\ \bar{ \varepsilon}_{i}\]

holds for some constants \(\bar{\varepsilon}_{1},\ldots,\bar{\varepsilon}_{\ell}>0\). Here, \(f_{1},\ldots,f_{\ell}:\mathbb{R}^{n}\rightarrow\mathbb{R}\) and each \(\Theta_{i}^{\top}\in\mathbb{R}^{N}\) is the corresponding row of the true parameters \(\Theta^{\top}\).

## VII Conclusion and Future Directions

In this paper, we developed a novel method for bridging approximations using activation functions with unknown parameters to approximations using randomly initialized parameters, with the mollified integral representation. We showed that this could be extended to the supremum norm for use with an extended, approximate version of the linear MRAC setup.

There are two immediate paths that we can see along the lines of extending this work and application. 1) Is it possible to do "approximate persistency of excitation" on the extended, approximate version of the linear MRAC setup, similar to what was achieved in [26] but with the parameter estimates only converging to a compact set about the origin, instead of being asymptotically stable. 2) Extending the work of [15] towards the determination of integral representations for more general classes of functions. In particular, using linear combinations of step functions, which cannot be trained using backpropagation in the modern paradigm of training deep neural networks. Yet in the 1D case, we can easily show that the step function is superior at uniformly approximating functions than the ReLU function. In fact, the step can handle discontinuous (regulated) functions, while the ReLU can only deal with continuous functions.
