# Approximation of RKHS Functionals by Neural Networks

Tian-Yi Zhou1 Namjoon Suh2 Guang Cheng2 Xiaoming Huo1

(1H. Milton Stewart School of Industrial and Systems Engineering, Georgia
Institute of Technology
2Department of Statistics & Data Science, UCLA
)

###### Abstract

Motivated by the abundance of functional data such as time series and images,
there has been a growing interest in integrating such data into neural
networks and learning maps from function spaces to $\mathbb{R}$ (i.e.,
functionals). In this paper, we study the approximation of functionals on
reproducing kernel Hilbert spaces (RKHS’s) using neural networks. We establish
the universality of the approximation of functionals on the RKHS’s.
Specifically, we derive explicit error bounds for those induced by inverse
multiquadric, Gaussian, and Sobolev kernels. Moreover, we apply our findings
to functional regression, proving that neural networks can accurately
approximate the regression maps in generalized functional linear models.
Existing works on functional learning require integration-type basis function
expansions with a set of pre-specified basis functions. By leveraging the
interpolating orthogonal projections in RKHS’s, our proposed network is much
simpler in that we use point evaluations to replace basis function expansions.

Keywords: neural networks, reproducing kernel Hilbert space, approximation
theory, functional learning, functional data analysis

##  1 Introduction

This paper studies the approximation of smooth functionals defined over a
reproducing kernel Hilbert space (RKHS) using tanh neural networks. A
functional maps from a space of functions that has infinite dimensions to
$\mathbb{R}$. In recent years, neural networks have been widely employed in
operator learning tasks. We are interested in investigating their capability
to approximate nonlinear functionals, a special type of operator.

Neural networks have been known as universal approximators since [Cybenko,
1989], i.e., to approximate any continuous function, mapping a finite-
dimensional input space into another finite-dimensional output space, to
arbitrary accuracy. These days, many interesting tasks entail learning
operators, i.e., mappings between an infinite-dimensional input Banach space
and (possibly) an infinite-dimensional output space. A prototypical example in
scientific computing is to map the initial datum into the (time series of)
solution of a nonlinear time-dependent partial differential equation (PDE). A
priori, it is unclear if neural networks can be successfully employed to learn
such operators from data, given that their universality only pertains to
finite-dimensional functions.

One of the first successful uses of neural networks in the context of operator
learning was provided by [Chen and Chen, 1995]. A shallow neural network-based
architecture, termed as neural operator, is proposed and is proved to possess
an universal approximation property for nonlinear operators from an infinite-
dimensional space to a finite-dimensional Euclidean space. Recently, there has
been an active line of research developing new methods for learning operators
by deep neural networks, such as the DeepONet [Lu et al., 2021], physics-
informed neural networks (PINNs) [Raissi et al., 2019, Wang et al., 2021],
Fourier Neural Operator (FNO) [Li et al., 2021], and DeepGreen [Gin et al.,
2021]. Although these methods have shown success through numerical experiments
in approximating operators in many interesting examples, theoretical
guarantees in this context are relatively scant. Some recent quantitative
approximation results are obtained in [Kovachki et al., 2021, Lanthaler et
al., 2022, Liu et al., 2024].

This work focuses on the approximation of functionals by neural networks.
Approximating functionals by neural networks is a crucial step in
approximating operators by neural networks in the DeepONet and similar methods
[Lanthaler et al., 2022], as used in [Chen and Chen, 1995]. In particular, we
consider the approximation of functionals on reproducing kernel Hilbert spaces
(RKHS’s), where a major advantage of RKHS is that it enables us to replace
integration-type basis function expansions by simple point evaluations on a
discrete subset of the domain.

We adopt neural networks equipped with tanh activation functions. Most of the
recent results on neural network approximation focus on ReLU networks.
Although ReLU activation functions are very common in practical applications
of deep learning, there are many areas where other activation functions are
employed. Among the most popular activations are the tanh activation function
and the related sigmoid or logistic function (a scaled and shifted tanh). In
the area of operator learning, smooth activation functions such as tanh are
preferred over ReLU, especially in PINNs, for solving forward and inverse
problems for PDEs (see [Raissi et al., 2019] and references therein). Also,
tanh activation functions are the basis of heavily used recurrent neural
networks (RNN), such as Long Short-term Memory (LSTM) [Hochreiter and
Schmidhuber, 1997] and Gated Recurrent Unit (GRU) [Cho et al., 2014].

To establish a theoretical guarantee that neural networks can learn
functionals well, we need to show the convergence of generalization error,
which measures the distance between the ground-truth functional and the
estimated functional given by a certain class of neural networks. In this
paper, we show that a tanh neural network with two hidden layers suffices to
approximate a functional on RKHS to any desired accuracy. Deriving
approximation error is the first step of generalization analysis. We also
apply our findings to functional data analysis (FDA) and prove that neural
networks can approximate regression maps in generalized functional linear
models well without any assumptions on the model parameters. Our approximation
result will help theoretically justify the use of deep neural networks in
functional data analysis. In fact, by following [Shang and Cheng, 2015], we
are now able to derive generation error and even statistical inferences for
generalized functional linear regression models, which will be a future topic.
More discussions regarding generalization studies of neural networks can be
found in the recent survey paper [Suh and Cheng, 2024].

###  1.1 Related Works

In an earlier work [Mhaskar and Hahm, 1997], a shallow neural network with
infinitely differentiable activation function is used to approximate
nonlinear, continuous functionals on $L^{p}$ spaces, for $1\leq p\leq\infty$.
Though the approximation rate given there does not seem satisfactory — when
measuring the approximation error on the unit ball of a Hölder space, the
number of network parameters grows exponentially with a large exponent as the
accuracy increases — it is proven that such a rate is, in fact, optimal. Later
in [Stinchcombe, 1999], a generalized neural network is introduced to
approximate functionals whose domain is a locally convex topological vector
space. Based on the aforementioned work, a novel functional multi-layer
perception (MLP) is introduced by [Rossi and Conan-Guez, 2005], which can be
directly applied to functional data. Recently in [Thind et al., 2023], an
optimization algorithm is proposed to implement the functional MLP in
practice. The functional MLP is based on, but different from, a classical
fully-connected neural network.

To illustrate this special network’s design, we first formulate a standard
fully-connected network. For
$\mathbf{b}=(b_{1},\ldots,b_{r})\in\mathbb{R}^{r}$, let
$\sigma_{\mathbf{b}}:\mathbb{R}^{r}\rightarrow\mathbb{R}^{r}$ be the shifted
activation function as

$\sigma_{\mathbf{b}}\begin{pmatrix}x_{1}\\\ \vdots\\\
x_{r}\end{pmatrix}:=\begin{pmatrix}\sigma(x_{1}-b_{1})\\\ \vdots\\\
\sigma(x_{r}-b_{r})\end{pmatrix},$

where $\sigma(x)$ is an activation function such as tanh:
$\sigma(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$ and ReLU:
$\sigma(x)=\max\\{x,0\\}$. We use the vector
$\mathbf{d}=(d_{1},\ldots,d_{L})\in\mathbb{N}^{L}$ to indicate the width in
each hidden layer.

######  Definition 1.1 (Standard Fully-connected Network).

A fully-connected neural network
$\widehat{G}:\mathbb{R}^{N}\rightarrow\mathbb{R}$ with $L$ hidden layers is a
function that takes the form

$\widehat{G}(x)=a^{T}\sigma_{\mathbf{b}_{L}}(W_{L}\sigma_{\mathbf{b}_{L-1}}(W_{L-1}\cdots\sigma_{\mathbf{b}_{1}}(W_{1}x))),$

where $x\in\mathbb{R}^{N}$ is the input, $a\in\mathbb{R}^{d_{L}}$ is the outer
weight, $d_{0}=N$, $W_{\ell}\in\mathbb{R}^{d_{\ell}\times d_{\ell-1}}$ denote
the weight matrices, $b_{\ell}\in\mathbb{R}^{d_{\ell}}$ denote the bias
vectors over the layers $\ell=1,\ldots,L,$ in the network.

The functional MLP introduced in [Rossi and Conan-Guez, 2005] is designed to
approximate functionals defined on $L^{p}[-1,1]^{s}$ for $p\geq 1$ and
$s\in\mathbb{N}\geq 1$, where the network input is a function covariate
$f(x):L^{p}[-1,1]^{s}\rightarrow\mathbb{R}$, $x\in[-1,1]^{s}$. We give the
formulation of functional MLP below.

######  Definition 1.2 (Functional MLP).

A functional MLP $\Theta:L^{p}[-1,1]^{s}\rightarrow\mathbb{R}$ with $L$ hidden
layers is any functional that takes the form

$\Theta(f)=a^{T}\sigma_{\mathbf{b}_{L}}W_{L}\sigma_{\mathbf{b}_{L-1}}W_{L-1}\cdots\sigma_{\mathbf{b}_{2}}W_{2}T(f),$

where $f\in L^{p}[-1,1]^{s}$, $T:L^{p}[-1,1]^{s}\rightarrow\mathbb{R}^{d_{1}}$
is a bounded linear operator with

$T(f)=\left(\int_{[-1,1]^{s}}f(x)g_{1}(x)dx,\cdots,\int_{[-1,1]^{s}}f(x)g_{d_{1}}(x)dx\right)^{T},$
(1.1)

where $\\{g_{k}\\}_{k=1}^{d_{1}}\in L^{q}[-1,1]^{s}$ is a set of pre-selected
basis functions and $\frac{1}{p}+\frac{1}{q}=1$.

As we can see in (1.1), the initial layer of this network converts the
function $f$ into a vector form that can be fed directly into a standard
neural network via basis function expansion. This is referred to as the
discretization step. One drawback of such a design is that a prior selection
of basis functions is required, often without information about the task at
hand. A universal approximation theorem of the functional MLP is established
in [Rossi and Conan-Guez, 2005], where a scheme to learn
$\\{g_{k}\\}_{k=1}^{d_{1}}$ is introduced by assuming them in a parametric
function space.

More recently, two variants of functional MLP have been introduced in [Yao et
al., 2021] and [Song et al., 2023]. In [Yao et al., 2021], each basis function
is parameterized with a micro neural network embedded in the MLP; while in
[Song et al., 2023], the basis functions are specifically chosen to be some
families of orthogonal polynomials. For approximating functionals on $L^{p}$
spaces ($p\geq 1$), the network proposed in [Song et al., 2023] achieves a
similar approximation rate as in [Mhaskar and Hahm, 1997]. At this point, a
question arises: can we adopt a simpler network to approximate nonlinear
functionals without integration in the first layer?

All the above-mentioned functional learning methods are based on a set of
basis functions defined on a domain in an Euclidean space. In this paper, we
study functionals on RKHS’s, and the discretization of functions can be simply
done by point evaluations. In other words, we adopt the standard fully-
connected neural networks with the input vector given by the discretely
observed functional values. Our design eliminates the need for a preprocessing
step in the form of integration that typically requires a manually
prespecified choice of basis function. Our network speaks directly to the
state-of-the-art operator learning methods, as DeepONet, PINNs, DeepGreen all
do not involve the basis expansion of network input.

###  1.2 Problem Formulation

We first define RKHS. Let $K$ be a Mercer kernel on a compact metric space
$(\mathcal{X},d_{\mathcal{X}})$, which induces an unique RKHS [Aronszajn,
1950] defined as:

######  Definition 1.3 (RKHS).

The RKHS $\mathcal{H}_{K}$ is the closure of the set
$\left\\{\sum_{j=1}^{p}\alpha_{j}K(\cdot,x_{j}):p\in\mathbb{N},\alpha_{j}\in\mathbb{R},x_{j}\in\mathcal{X}\right\\}$
under the metric induced by the inner product
$\langle\cdot,\cdot\rangle_{\mathcal{H}_{K}}$ given by

$\left\langle\sum_{i=1}^{q}\beta_{i}K(\cdot,y_{i}),\sum_{j=1}^{p}\alpha_{j}K(\cdot,x_{j})\right\rangle_{\mathcal{H}_{K}}=\sum_{i=1}^{q}\sum_{j=1}^{p}\beta_{i}\alpha_{j}K(y_{i},x_{j}).$

The RKHS $\mathcal{H}_{K}$ is characterized by the property that the point
evaluation functional $L_{x}$ on $\mathcal{H}_{K}$ given by $f\mapsto f(x)$ is
continuous for every $x\in\mathcal{X}$.

Denote by $\|\cdot\|_{\mathcal{H}_{K}}$ the RKHS norm induced by the inner
product $\langle\cdot,\cdot\rangle_{\mathcal{H}_{K}}$. We define the unit ball
of $\mathcal{H}_{K}$ given by
$\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq 1\\}.$ We are
interested in approximating the functional
$F:\mathcal{K}\rightarrow\mathbb{R}$, where $\mathcal{K}$ is regarded as a
compact subset of $C(\mathcal{X})$, the space of all continuous functions on
$\mathcal{X}$ with norm $\|f\|_{\infty}=\sup_{x\in\mathcal{X}}|f(x)|$.
Throughout this paper, we assume that $F$ is $s$-Hölder continuous for
$0<s\leq 1$ with constant $C_{F}\geq 0$:

$|F(f)-F(\tilde{f})|\leq C_{F}\|f-\tilde{f}\|^{s}_{\infty},\qquad\forall
f,\tilde{f}\in\mathcal{K}.$ (1.2)

In this paper, we adopt the classic fully-connected neural network defined in
Definition 1.1 equipped with the tanh activation function
$\sigma(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$. We aim to construct tanh neural
networks $\widehat{G}:\mathbb{R}^{N}\to\mathbb{R}$ and obtain an upper bound
on the following uniform approximation error

$\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|,$ (1.3)

where the network inputs
$f(\bar{t})=(f(t_{1}),\ldots,f(t_{N}))\in\mathbb{R}^{N}$ are function values
at sufficient but finitely many discrete locations
$\bar{t}=\\{t_{1},\ldots,t_{N}\\}\subset\mathcal{X}$. Uniform spacing of
locations is not necessary. Our method does not require basis function
expansion.

###  1.3 Main Contributions

We consider fully-connected tanh neural networks with two hidden layers. The
novelty of our proposed networks is that we evaluate the input
$f\in\mathcal{H}_{K}$ by simply taking its function values at some discrete
points in the domain $\mathcal{X}$, that is,
$f(\bar{t})=(f(t_{1}),\ldots,f(t_{N}))\in\mathbb{R}^{N}$ with
$\bar{t}=\\{t_{1},\ldots,t_{N}\\}\subset\mathcal{X}$. We do not require
uniform partition of these discrete points. Our design is motivated by the
practice of representing continuous signals or images by discrete objects in
various fields of science and engineering. To represent continuous inputs,
such as continuous images and signals defined on a compact domain, it is
common to take measurements of values over a discrete subset of the domain as
discrete images or signals. Conversely, the integration-type function
expansion with a set of pre-specified basis functions employed in previous
works is not a typical approach in practice. We adopt a much simpler network
by leveraging the interpolating orthogonal projections in RKHS’s.

Our results are summarized below. We establish novel error bounds for the
approximation of functionals on the RKHS induced by a general Mercer kernel
using tanh neural networks (Theorem 4.1). Prior works on functional
approximation study only those defined on $L^{p}$ spaces, making our results
unprecedented in the literature. The approximation error bound exhibits a
trade-off associated with $N\in\mathbb{N}$ (i.e., the number of discrete
points used to evaluate our input $f$): when $N$ is small, the power function
(which measures the distance between a function and its interpolant)
increases; while when $N$ is too large, the network size becomes too big and
can results in overfitting. By determining the optimal choices of $N$, we
derive explicit rates of approximating functionals on Sobolev space (Theorem
3.4), and on RKHS’s induced by inverse multiquadric kernel (Theorem 3.6) and
Gaussian kernel (Theorem 3.9), respectively. These kernels are commonly
considered in kernel methods, and their translation-invariance properties
facilitate our theoretical analysis. Our results show that neural networks can
achieve any desired approximation accuracy with sufficiently many network
parameters. Particularly, for the inverse multiquadric and Gaussian kernels,
we achieve approximation rates faster than logarithmic rates, w.r.t. the
number of network parameters. Our results have shown improvement compared to
the previous results on $L^{p}$ functionals in [Mhaskar and Hahm, 1997] and
[Song et al., 2023].

This is a theoretical work on the approximation theory of RKHS functionals. We
leave generalization analysis and numerical studies for future studies.

###  1.4 Organization of this work

The rest of the paper is organized as follows: In Section 2, we present
applications that require learning nonlinear functionals. We then give our
main results in Section 3 and 4. We broadly divide our main results into two
parts; in Section 3, we give explicit rates of approximating RKHS functionals
induced by specific kernels by tanh neural networks, which include Sobolev
kernel (Theorem 3.4), inverse multiquadric kernel (Theorem 3.6), and Gaussian
kernel (Theorem 3.9); in Section 4, we give approximation results for general
RKHS functionals using tanh networks (Theorem 4.1). In Section 5, we apply our
theoretical findings to approximate regression maps in generalized functional
regression models (Corollary 5.1). In Section 6, we discuss the general
approaches and methods for establishing our main results. Concluding remarks
are given in Section 7. Finally, Appendix A contains the proof of some
remarks, Appendix B contains the proof of some supporting lemmas, and Appendix
C contains the proof of our main results.

##  2 Motivations

Many tasks in statistics and machine learning require learning nonlinear
functionals, including functional regression, ODE/PDE learning, and
distribution regression. Our work is motivated by the broad applications
functionals offer, as well as the abundance of functional data. In the
following, we present some related applications.

Example 1: Functional Regression

With the progress of technology, we observe more and more data of a functional
nature, such as curves, trajectories, time series, and images. The term
functional data traditionally refers to repeated measurements of an individual
over an interval, although it is broader in meaning; for example, it can refer
to functions on higher dimensional domains (e.g., images over domains in
$\mathbb{R}^{2}$ or $\mathbb{R}^{3}$) [Greven and Scheipl, 2017].

As functional data become more common, researchers are increasingly interested
in regression models for functional data (i.e., relating functional variables
to other variables of interest). A classical functional regression model is
the generalized functional linear model (FLM) introduced by [Muller and
Stadtmüller, 2005]. In this model, we aim to learn a regression map from a
function covariate to a scaler response.

Suppose $X$ is a square-integrable random function defined over a compact
interval $\mathcal{X}$ and $Y$ is a scaler response. With the regressor
$F:L^{2}(\mathcal{X})\rightarrow\mathbb{R}$, the generalized FLM can be
formulated as

$Y=g\left(\int_{\mathcal{X}}\beta(t)X(t)dt\right)+\varepsilon,\qquad
t\in\mathcal{X},$ (2.1)

where $\beta\in L^{2}(\mathcal{X})$ is the unknown functional parameter, $g$
is a link function that may possess high-order smoothness (e.g., a logit
link), and $\varepsilon$ is a centered noise random variable. A fundamental
problem in statistics is to determine the regression map $F$ and to
subsequently retrieve the regression mean
$\mathbb{E}[Y|X]:=g\left(\int_{\mathcal{X}}\beta(t)X(t)\right)$.

To solve the functional regression problem, it is crucial to learn the
regression map $F:L^{2}(\mathcal{T})\rightarrow\mathbb{R}$. While the powerful
capabilities of neural networks have been clearly demonstrated for
vector/text/graph data, how to adapt these models to functional data remains
under-explored. In this paper, we will show that for any given function
covariate $X$ residing in a RKHS inside the $L^{2}$ space, we can approximate
the regression map of generalized FLM using fully-connected neural networks.
We assume $g$ is Lipschitz continuous. This is a novel result that has not
been proven before in statistics literature. Detailed discussions can be found
in Section 5.

Example 2: ODE/PDE Learning

Systems of coupled ODEs and PDEs are undoubtedly the most widely used models
in science and engineering, as they often capture the dynamical behavior of
physical systems. Even simple ODE functions can describe complex dynamical
behaviors. Recently, an active line of research [Raissi et al., 2019, Wang et
al., 2021, Li et al., 2021] has been developing deep learning tools to infer
solutions of ODEs/PDEs with lower cost compared to traditional tools (e.g.,
finite element methods).

In solving ODEs and PDEs, we aim to learn a map from the parametric function
space to the ODE/PDE solution space. Consider a very simple initial value
problem of an ODE:

$\begin{cases}\frac{dh(x)}{dx}=g(x,f(x),h(x)),\qquad x\in[a,b],\\\
h(a)=h_{0}.\end{cases}$ (2.2)

We aim to learn $h(b)$ for any function $f$, that is, a map $F$ from an input
$f$ to an output $h(b)\in\mathbb{R}$. If we denote the solution to (2.2) as
$G(f)$, then

$F(f)=G(f)(b),$ (2.3)

where $G$ is a nonlinear operator being the unique solution to the integral
equation

$G(f)(x)=h_{0}+\int_{a}^{x}g(t,f(t),G(f)(t))dt.$ (2.4)

Our work contributes to the area of ODE/PDE learning by proving that a two-
layer standard neural network is sufficient to approximate (to any accuracy) a
functional that is a solution map to a problem like (2.2).

Example 3: Distribution Regression

We find unique applications that require functionals on RKHS’s. One of the
motivations for learning RKHS functionals is distribution regression:
regressing from probability measures to real-valued responses or, more
generally, to vector-valued outputs.

Let $(\mathcal{X},\tau)$ be a topological space and let
$\mathcal{B}(\mathcal{X})$ be the Borel $\sigma$-algebra induced by the
topology $\tau$. Let $\mathcal{M}^{+}(\mathcal{X})$ denote the set of Borel
probability measures on $(\mathcal{X},\tau)$, and $\mathcal{Y}$ be a separable
Hilbert space. Consider $z=\\{(X_{i},y_{i})\\}_{i=1}^{\ell}$, where
$X_{i}\in\mathcal{M}^{+}(\mathcal{X})$, $y_{i}\in\mathcal{Y}$ is its
observable label, each $(X_{i},y_{i})$ pair is drawn i.i.d. from a joint meta
distribution $\mathcal{M}$. We do not observe $X$ directly; rather, we observe
a sample $x_{i,1},\ldots,x_{i,N_{i}}\overset{i.i.d.}{\sim}X_{i}$. Thus the
observed data are
$\tilde{z}=\\{(\\{x_{i,n}\\}_{n=1}^{N_{i}},y_{i})\\}_{i=1}^{\ell}$. The goal
of distribution regression is to learn the map between the random distribution
$X$ and its label $y$ based on the observed data $\tilde{z}$.

To solve the distribution regression problem, an approach via kernel ridge
regression is proposed [Szabó et al., 2016] and a fast convergence is proven.
The approach involves kernel mean embedding as an intermediate step. Denote by
$\mu(\mathcal{M}^{+}(\mathcal{X}))=\\{\mu_{X}:X\in\mathcal{M}^{+}\\}\subseteq\mathcal{H}_{K}\text{
with }\mu_{X}=\int_{\mathcal{X}}\textbf{Ker}(\cdot,u)dX(u)\in\mathcal{H}_{K}$
the set of mean embeddings of $X$ to a RKHS $\mathcal{H}_{K}$. We also let
$H_{K}$ be the $Y$-valued RKHS of
$\mu(\mathcal{M}^{+}(\mathcal{X}))\rightarrow\mathcal{Y}$ functions with
reproducing kernels
$\textbf{Ker}:\mu(\mathcal{M}^{+}(\mathcal{X}))\times\mu(\mathcal{M}^{+}(\mathcal{X}))\rightarrow\mathcal{Y}$.
The distribution $X\in\mathcal{M}^{+}$ is first mapped to
$\mu(\mathcal{M}^{+}(\mathcal{X}))\subseteq\mathcal{H}_{K}$ by the mean
embedding $\mu$, then the result is composed with function $f\in H_{K}$:

$\mathcal{M}^{+}\overset{\mu}{\rightarrow}\mu(\mathcal{M}^{+}(\mathcal{X}))\overset{f\in
H_{K}}{\longrightarrow}\mathcal{Y}.$

An important part of the method is to learn the functionals that map
$\mu(\mathcal{M}^{+}(\mathcal{X}))$ (a subset of RKHS $\mathcal{H}_{K}$) to
$\mathcal{Y}$.

In this paper, we give a theoretical guarantee that neural networks can
approximate functionals on RKHS well. Our findings suggest that neural
networks have the potential to tackle distribution regression problems, as
they can be used to find the map from $\mu(\mathcal{M}^{+}(\mathcal{X}))$ to
$\mathcal{Y}$.

##  3 Main Results I: Approximation of RKHS Functionals induced by specific
kernels

We consider the domain $\mathcal{X}=[0,1]^{d}$. We present the first part of
our main results here. The second part of our main results is in Section 4,
where we give approximation results for functionals on general RKHS’s. In this
section, we give explicit error rates of approximating functionals on RKHS’s—
induced by the following three specific reproducing kernels — to demonstrate
our methods and approaches:

  * •

Sobolev space $W^{r}_{2}(\mathcal{X})$. Although this is not a kernel, it is
known that the Sobolev space $W^{r}_{2}(\mathbb{R}^{d})$ of order $r$ is a
RKHS if $r>d/2$ (see, e.g., [Brezis and Brézis, 2011, Wainwright, 2019]).

  * •

Inverse multiquadric kernel: $K(u,v)=(\sigma^{2}+|u-v|^{2})^{-\beta}$, where
$u,v\in\mathcal{X}$ and $\sigma,\beta>0$. The parameter $\sigma$ is often
called the shape parameter.

  * •

Gaussian kernel: $K(u,v)=e^{-\frac{|u-v|^{2}}{2\sigma^{2}}}$, where
$u,v\in\mathcal{X}$ and $\sigma>0$.

We would like to point out that both the inverse multiquadratic kernel and the
Gaussian kernel are important examples of translation-invariant kernels (also
known as convolution-type kernels). The definition of a translation-invariant
kernel is given below.

######  Definition 3.1 (Definition of translation-invariant kernel).

A kernel $K:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}$ is called
translation-invariant (or convolution-type) if it only depends on the
difference between its arguments, that is,

$K(u,v)=\phi(u-v),\qquad u,v\in\mathbb{R}^{d}$ (3.1)

for some function $\phi:\mathbb{R}^{d}\rightarrow\mathbb{R}$. Such a function
$\phi$ is considered positive definite if the corresponding kernel $K$ is
positive definite.

Note that we can represent a translation-invariant kernel both as
$K:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}$ with two arguments
and as $\phi:\mathbb{R}^{d}\rightarrow\mathbb{R}$ with only one argument.

The Fourier transform of a function $\phi$ in $\mathbb{R}^{d}$ is the function
$\widehat{\phi}:\mathbb{R}^{d}\rightarrow\mathbb{C}$ defined by

$\widehat{\phi}(\xi)=\int_{\mathbb{R}^{d}}\phi(x)e^{-2\pi i\xi\cdot
x}dx,\qquad\xi\in\mathbb{R}^{d}.$ (3.2)

From the inverse form of the Fourier transform, we know

$\phi(x)=\int_{\mathbb{R}^{d}}\widehat{\phi}(\xi)e^{2\pi i\xi\cdot
x}d\xi,\qquad x\in\mathbb{R}^{d}.$ (3.3)

It is sometimes difficult to check if a kernel is positive definite. But if a
kernel is translation-invariant, we can do so by easily checking its Fourier
transform.

######  Lemma 3.2.

A translation-invariant kernel $K$ induced by
$\phi:\mathbb{R}^{d}\to\mathbb{R}$ is a reproducing kernel (i.e., positive
semi-definite) if its Fourier transform $\widehat{\phi}$ is real-valued and
non-negative.

In fact, the positivity of the Fourier transform almost everywhere is
sufficient for the positive definiteness of a kernel [Schaback and Wendland,
2006]. The proof of Lemma 3.2 is relegated to Appendix B.

In the following, we present explicit error bounds for approximating
functionals on Sobolev space and RKHSs induced by inverse multiquadric and
Gaussian Kernels. We adopt standard neural networks with two hidden layers.
The network input is a function $f$ evaluated at some discrete points in
$\mathcal{X}$, that is, $\\{f(t_{1}),\ldots,f(t_{N})\\}$. We want to highlight
that the points $\\{t_{1},\ldots,t_{N}\\}$ need not be uniformly partitioned
in $\mathcal{X}$. Since we only consider RKHS’s induced by translation-
invariant kernels in this section, it seems natural to choose
$\\{t_{1},\ldots,t_{N}\\}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}$ for
$m\in\mathbb{N}$ and $N=(m+1)^{d}$.

###  3.1 Approximation of Functionals on Sobolev Space
$W^{r}_{2}(\mathcal{X})$

We consider the Sobolev Space $W^{r}_{2}(\mathcal{X})$ of order $r$ for
$r\in\mathbb{R}_{+}$ and $d\in\mathbb{N}$. We allow $r$ to be any arbitrary
real number (i.e., $r$ need not be an integer), and the corresponding Sobolev
space $W^{r}_{2}(\mathcal{X})$ is often referred to as a fractional Sobolev
space. The following well-known result suggests that when the embedding
condition $r>d/2$ holds, $W^{r}_{2}(\mathcal{X})$ is a RKHS.

######  Proposition 3.3.

The standard Sobolev space $W^{r}_{2}(\mathbb{R}^{d})$, with arbitrary
positive $r$ and $d$ for which $r>d/2$, is a RKHS induced by a translation-
invariant kernel $K(u,v)=\phi(u-v)$, where its Fourier transform is the
function $\widehat{\phi}(\xi)$ given by

$\widehat{\phi}(\xi)=(1+|\xi|^{2})^{-r},\qquad\xi\in\mathbb{R}^{d}.$ (3.4)

From Lemma 3.2, since $\widehat{\phi}(\xi)=(1+|\xi|^{2})^{-r}>0$ for all
$\xi\in\mathbb{R}^{d}$, $K$ (or $\phi$) is a reproducing kernel. It is somehow
surprising that it is difficult to find explicit formulas for reproducing
kernels of Sobolev spaces, except for the univariate case $d=1$. More
discussions can be found in [Novak et al., 2018] and references therein.

The classic Sobolev Embedding Theorem tells us that, for $r>d/2$ and
$r-d/2\notin\mathbb{N}$, we have $W^{r}_{2}(\mathcal{X})\subset
W_{\infty}^{r-d/2}(\mathcal{X})$, where $W_{\infty}^{r-d/2}(\mathcal{X})$
represents the Hölder space of order $r-d/2$. Moreover, we have

$\|f\|_{W_{\infty}^{r-d/2}}\leq C_{r,d}\|f\|_{W^{r}_{2}(\mathbb{R}^{d})},$

where $C_{r,d}$ is a constant depending on only $r$ and $d$. When $r>d/2$ and
$r-d/2\in\mathbb{N}$, we can easily extend the above results to get
$W^{r}_{2}(\mathcal{X})\subset W_{\infty}^{r-d/2-\varepsilon}(\mathcal{X})$
and $\|f\|_{W_{\infty}^{r-d/2-\varepsilon}}\leq
C_{r,d}\|f\|_{W^{r}_{2}(\mathbb{R}^{d})}$, with any $0<\varepsilon<r-d/2$.
Hereafter we assume $r-d/2\notin\mathbb{N}$.

The following result establishes explicit error bound in approximating
functionals on Sobolev Space $W^{r}_{2}(\mathcal{X})$ using two-layer tanh
neural networks. Its proof is relegated to Appendix C.

######  Theorem 3.4 (Approximation of Functionals on Sobolev Space).

Let $r\in\mathbb{R}_{+},d\in\mathbb{N}$, $\mathcal{X}=[0,1]^{d}$, and
$r-d/2>1$. We assume that $r-d/2\notin\mathbb{N}$. Consider
$\mathcal{K}:=\\{f\in
W^{r}_{2}(\mathcal{X}):\|f\|_{W^{r}_{2}(\mathcal{X})}\leq 1\\}$. Suppose $F$
is $s$-Hölder continuous with $s\in(0,1]$. There exists some
$M_{0}\in\mathbb{N}$ such that for every $M\in\mathbb{N}$ with $M>M_{0}$, by
taking $m=\left\lceil M^{\frac{1}{2s(2r-1)}}\right\rceil$,
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}\subset\mathcal{X}$, and
$N=(m+1)^{d}$, we have a tanh neural network
$\widehat{G}:\mathbb{R}^{N}\to\mathbb{R}$ with two hidden layers of widths at
most $N(M-1)$ and $3\left(\frac{N+1}{2}\right)(5M)^{N}$ satisfying

$\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|\leq
C_{r,s,d,F}\
d^{s(r+\frac{1}{2})}\left(\frac{1}{M}\right)^{\frac{2r-d}{2(2r-1)}},$

where $C_{r,s,d,F}$ is a constant depending on $r,s,d$ and the Lipschitz
constant of $F$.

######  Remark 3.5.

The maximum total number of parameters in this network is
$\mathcal{N}=\mathcal{O}(N(5M)^{N})$, where $N=(m+1)^{d}=\left(\left\lceil
M^{\frac{1}{2s(2r-1)}}\right\rceil+1\right)^{d}$. As $M$ increases (i.e., the
network widths increase), the approximation error converges to $0$. The
convergence rate given in the above theorem — w.r.t. the number of parameters
$\mathcal{N}$ — achieves a logarithmic rate. We defer the technical proof of
this statement to Appendix A for conciseness.

###  3.2 Approximation of RKHS Functionals Induced by Inverse Multiquadric
Kernel

Here, we present the result on the approximation of functionals defined over
$\mathcal{K}$, an RKHS induced by the inverse multiquadric kernel

$K(u,v)=(\sigma^{2}+|u-v|^{2})^{-\beta},\qquad u,v\in\mathcal{X},\
\sigma,\beta>0$ (3.5)

using two-layer tanh neural networks. The inverse multiquadric kernel is a
reproducing kernel as long as $\sigma,\beta>0$ (see, [Bozzini et al., 2015,
Tolstikhin et al., 2017]). The proof of the following theorem is given in
Appendix C.

######  Theorem 3.6 (Approximation of RKHS Functionals induced by inverse
multiquadric Kernel).

Let $d\in\mathbb{N}$, $\sigma,\beta>0$, $\mathcal{X}=[0,1]^{d}$, and
$M_{d}=12\left(\frac{\pi\Gamma^{2}(\frac{d+2}{2})}{9}\right)\leq 6.38d$.
Consider $\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq
1\\}$ with $\mathcal{H}_{K}$ induced by an inverse multiquadric kernel given
in (3.5). Suppose $F$ is $s$-Hölder continuous with $s\in(0,1]$. There exists
some $M_{0}\in\mathbb{N}$ such that for every $M\in\mathbb{N}$ with $M>M_{0}$,
by taking $m=\left\lceil\frac{\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}\right\rceil$,
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}\subset\mathcal{X}$, and
$N=(m+1)^{d}$, we have a tanh neural network
$\widehat{G}:\mathbb{R}^{N}\to\mathbb{R}$ with two hidden layers of widths at
most $N(M-1)$ and $3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$ satisfying

$\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|\leq
C_{\sigma,\beta,s,d,F}(\log(M))^{\max\\{0,2d-s\beta\\}}\left(\frac{1}{M}\right)^{\frac{c}{4M_{d}\sqrt{d}\sigma+c}},$

where $C_{\sigma,\beta,s,d,F}$ is a constant depending on $\sigma,\beta,s,d$
and the Lipschitz constant of $F$, $c>0$ is a positive constant given for the
multiquadric kernel by (6.9).

######  Remark 3.7.

The maximum number of network parameters is
$\mathcal{N}=\mathcal{O}\left(N(5M)^{N}\right)$, where
$N=(m+1)^{d}=\left(\left\lceil\frac{\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}\right\rceil+1\right)^{d}$. As $M$ increases, the
approximation error converges to $0$. The convergence rate of the
approximation error given in the above theorem — w.r.t. the number of network
parameters $\mathcal{N}$ — is always faster than the logarithmic rate but
slower than the polynomial rate. We defer the technical proof of this
statement to Appendix A for conciseness.

######  Remark 3.8.

It is indeed quite disappointing that the tanh neural network cannot achieve a
polynomial rate for approximating RKHS functionals. The only lower bound of
approximation of continuous functionals is established in [Mhaskar and Hahm,
1997, Theorem 2.2]. They studied the approximation of
$F:L^{p}(\mathcal{X})\rightarrow\mathbb{R}$ for $1\leq p\leq\infty$ on some
certain compact domain $K\subset W^{\beta}_{\infty}(\mathcal{X})$, the set of
functions satisfying a Hölder condition of order $\beta>0$. They obtained a
lower bound, with $0<\lambda\leq 1$,

$\sup_{f\in K}\left|F(f)-\widehat{G}(f)\right|\geq
C(\log(\mathcal{N}))^{-\frac{\beta\lambda}{d}},$

where $\widehat{G}$ represents neural networks equipped with sigmoid-type
infinitely differentiable activation functions (e.g., tanh function), and
$\mathcal{N}$ is the total number of network parameters. This is a logarithmic
rate of approximation error. We cannot directly compare our results to their
lower bound because we study RKHS functionals instead of $L^{p}$ functionals.
However, such a lower bound suggested that the approximation of continuous
functionals may be generally slow.

###  3.3 Approximation of RKHS Functionals Induced by Gaussian Kernel

Gaussian kernel is an important translation-invariant kernel given by

$K(u,v)=e^{-\frac{|u-v|^{2}}{2\sigma^{2}}},\qquad u,v\in\mathcal{X},\sigma>0,$
(3.6)

or $\phi(x)=e^{-\frac{x^{2}}{2\sigma^{2}}}$ for $x\in\mathcal{X}$. From Lemma
3.2, the Gaussian kernel is a reproducing kernel because its Fourier transform
is known to be
$\widehat{\phi}(\xi)=(2\sigma^{2}\pi)^{\frac{d}{2}}e^{-2\sigma^{2}\pi^{2}|\xi|^{2}}>0$
for all $\xi\in\mathbb{R}^{d}$. The following result gives an explicit error
bound of approximation of functional $F$ over $\mathcal{K}$, an RKHS induced
by a Gaussian kernel, using two-layer tanh neural networks. Its proof is given
in Appendix C.

######  Theorem 3.9 (Approximation of RKHS Functionals induced by Gaussian
Kernel).

Let $d\in\mathbb{N}$, $\sigma>0$, $\mathcal{X}=[0,1]^{d}$. We also let $c>0$
be some positive constant. Consider
$\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq 1\\}$ with
$\mathcal{H}_{K}$ induced by a Gaussian kernel given in (3.6). Suppose $F$ is
$s$-Hölder continuous with $s\in(0,1]$. There exists some $M_{0}\in\mathbb{N}$
such that for every $M\in\mathbb{N}$ with $M>M_{0}$, by taking

$m=\left\lceil\frac{2\log(M)}{\frac{cs}{\sqrt{d}}+\sqrt{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)}}\right\rceil>\sqrt{d},$
(3.7)

$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}\subset\mathcal{X}$, and
$N=(m+1)^{d}$, we have a tanh neural network
$\widehat{G}:\mathbb{R}^{N}\to\mathbb{R}$ with two hidden layers of widths at
most $N(M-1)$ and $3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$ satisfying

$\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|\leq
C_{\sigma,\pi,s,d,F}(\log(M))^{d}\left(\frac{1}{M}\right)^{\frac{1}{2(1+\sigma\pi
d)}\left(\frac{1}{2}\log\log(M)-\log(cs+\sigma\pi d\sqrt{s})\right)},$ (3.8)

where $C_{\sigma,\pi,s,d,F}$ is a constant depending on $\sigma,\pi,s,d$ and
the Lipschitz constant of $F$.

######  Remark 3.10.

The maximum total number of parameters of the network is
$\mathcal{N}=\mathcal{O}\left(N(5M)^{N}\right)$, where $N=(m+1)^{d}$ with $m$
given in (3.7). As $M$ increases, the error bound given in (3.8) converges to
$0$. Similar to Remark 3.7 for Theorem 3.6, we can check that the convergence
rate (w.r.t. $\mathcal{N}$) is faster than a logarithmic rate but slower than
a polynomial rate.

In the next section, we give our result for approximating functionals on
general RKHS’s.

##  4 Main Results II: Approximation of general RKHS Functionals

In this section, we give our results on approximating functionals on RKHS’s
induced by a general Mercer kernel using tanh neural networks. We consider an
RKHS $\mathcal{H}_{K}$ induced by some Mercer kernel
$K:\mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$. We assume $K$ is
$\alpha$-Hölder continuous for $0<\alpha\leq 1$ with constant $C_{K}\geq 0$,
that is,

$|K(u,v)-K(u,\tilde{v})|\leq
C_{K}(d_{\mathcal{X}}(v,\tilde{v}))^{\alpha},\qquad
u,v,\tilde{v}\in\mathcal{X}.$ (4.1)

In $\mathcal{X}$, we choose a set of finitely many locations
$\bar{t}=\\{t_{i}\\}_{i=1}^{N}\subset\mathcal{X}$ for $N\in\mathbb{N}$ such
that the corresponding Gram matrix

$K[\bar{t}]=(K(t_{i},t_{j}))_{i,j=1}^{N}=\begin{bmatrix}K(t_{1},t_{1})&K(t_{1},t_{2})&\cdots&K(t_{1},t_{N})\\\
K(t_{2},t_{1})&K(t_{2},t_{2})&\cdots&K(t_{2},t_{N})\\\
\vdots&\vdots&\ddots&\vdots\\\
K(t_{N},t_{1})&K(t_{N},t_{2})&\cdots&K(t_{N},t_{N})\end{bmatrix}$ (4.2)

is invertible. We do not need these locations to be uniformly spaced in
$\mathcal{X}$. In general, we select $\bar{t}$ such that

$\max_{x\in\mathcal{X}}\min_{t\in\bar{t}}\ d_{\mathcal{X}}(x,t)\rightarrow
0,\qquad\text{as }N\rightarrow\infty.$

Later, we will show that the norm of the inverse of $K[\bar{t}]$ (i.e.,
$\|(K[\bar{t}])^{-1}\|_{op}$) has a significant influence on the approximation
rate of RKHS functionals.

Before presenting our last theorem, we introduce two geometric quantities
employed in our analysis: the fill distance and the separation radius. For any
$x\in\mathcal{X}$ and $\bar{t}\subset\mathcal{X}$, let $t_{j_{x}}$ be the
closest point in $\bar{t}$ to $x$, with $j_{x}\in\\{1,\ldots,N\\}$. The fill
distance of $\bar{t}$ is given by

$\displaystyle
h_{\bar{t}}:=h_{\bar{t},\mathcal{X}}:=\sup_{x\in\mathcal{X}}\min_{1\leq i\leq
N}d_{\mathcal{X}}(x,t_{i})=\sup_{x\in\mathcal{X}}d_{\mathcal{X}}(x,t_{j_{x}}),$
(4.3)

which indicates how well the points in $\bar{t}$ fill out the domain
$\mathcal{X}$. It measures the radius of the largest possible empty ball that
can be placed among the points in $\bar{t}$. The second geometric quantity is
the separation radius of $\bar{t}$, which is half the distance between the two
closest points in $\bar{t}$. It is given by

$q_{\bar{t}}:=\frac{1}{2}\min_{u\neq v\in\bar{t}}|u-v|.$ (4.4)

We can see that $h_{\bar{t}}$ and $q_{\bar{t}}$ decreases as $N\in\mathbb{N}$
increases.

In order to quantify the worst-case (i.e., for any $f$ in ${\mathcal{H}_{K}}$)
uniform error between a function and its interpolant, the so-called power
function is widely adopted in kernel interpolation literature (see, e.g.,
[Fasshauer, 2005, Pazouki and Schaback, 2011, Kanagawa et al., 2018]). In our
analysis, we use the power function to, in turn, measure the regularity of a
Mercer kernel $K$. For a kernel $K$ and
$\bar{t}=\\{t_{i}\\}_{i=1}^{N}\subset\mathcal{X}$, the power function is
defined as

$\epsilon_{K}(\bar{t}):=\max_{x\in\mathcal{X}}\min_{c\in\mathbb{R}^{N}}\left\|K_{x}-\sum_{i=1}^{N}c_{i}K_{t_{i}}\right\|_{\mathcal{H}_{K}}=\max_{x\in\mathcal{X}}\min_{c\in\mathbb{R}^{N}}\left\\{K(x,x)-2\sum_{i=1}^{N}c_{i}K(x,t_{i})+\sum_{i=1}^{N}\sum_{j=1}^{N}c_{i}c_{j}K(t_{i},t_{j})\right\\}^{\frac{1}{2}}.$
(4.5)

For any $\bar{t}=\\{t_{i}\\}_{i=1}^{N}\subset\mathcal{X}$, by the Gram matrix
given in (4.2), the fill distance given in (4.3), and the power function
defined in (4.5), we are able to derive an error bound for the approximation
of functional on RKHS induced by a general Mercer kernel.

######  Theorem 4.1 (Approximation of general RKHS Functionals).

Let $d\in\mathbb{N}$, $\mathcal{X}=[0,1]^{d}$. Consider
$\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq 1\\}$ with
$\mathcal{H}_{K}$ induced by some Mercer kernel $K$ which is $\alpha$-Hölder
continuous for $\alpha\in(0,1]$ with constant $C_{K}\geq 0$. Suppose $F$ is
$s$-Hölder continuous for $s\in(0,1]$ with constant $C_{F}\geq 0$. There
exists some $M_{0}\in\mathbb{N}$ such that for every $M\in\mathbb{N}$ with
$M>M_{0}$, by taking some $\bar{t}=\\{t_{i}\\}_{i=1}^{N}\subset\mathcal{X}$
with $N\in\mathbb{N}$, we have a tanh neural network $\widehat{G}$ with two
hidden layers of widths at most $N(M-1)$ and
$3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$ satisfying

$\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|\leq
C_{F}(\epsilon_{K}(\bar{t}))^{s}+\frac{7N^{2}C_{G}}{M},$ (4.6)

with

$C_{G}=C_{F}\left(1+\|(K[\bar{t}])^{-1}\|_{op}\sqrt{N}C_{K}\left(h_{\bar{t}}\right)^{\alpha}\right)^{s},$

where the Gram matrix $K[\bar{t}]$ is given in (4.2), the fill distance
$h_{\bar{t}}$ is given in (4.3), and the power function
$\epsilon_{K}(\bar{t})$ is defined in (4.5).

Let us look closely at the approximation error bound we obtained in (5.5),
particularly how the number of locations $N\in\mathbb{N}$ in $\bar{t}$ affects
the bound. The error bound consists of two components: the former term
corresponds to the power function $\epsilon_{K}(\bar{t})$ which typically
decreases as $N$ increases; the latter term is of
$\mathcal{O}(N^{2+\frac{s}{2}})$ which obviously increases with $N$. We can
see a trade-off associated with $N$ between these two terms. We shall strike
the right balance and find the “sweet spot” of $N$ so neither of these terms
blows up. Later, in the proofs of our main results in Appendix C, we give
detailed derivations of the optimal choices of $N$.

##  5 Approximate Regression Map in Functional Regression Using Neural Network

In this section, we apply our findings to approximate regression maps in
generalized functional linear regression models.

Despite their widespread success, the application of neural networks to
functional data remains scarce today. Extensively studied in the statistics
literature, functional data appear frequently in scientific studies, such as
in datasets of air pollution, fMRI scans, and growth curves [Yao et al., 2005,
Leng and Müller, 2006, Chiou, 2012]. Functional regression with observed
random functions as predictors coupled with scalar responses is one of the
core tools of functional data analysis [Morris, 2015, Wang et al., 2016]. The
classical model of functional regression is the functional linear model (FLM),
where the response $Y\in\mathbb{R}$ is related to a square-integrable random
function $X\in L^{2}([0,1])$ through

$Y=\int_{0}^{1}X(t)\beta(t)dt+\varepsilon=\langle
X,\beta\rangle_{L^{2}([0,1])}+\varepsilon.$ (5.1)

Here the unknown functional parameter $\beta\in L^{2}([0,1])$ is referred to
as slope function, and $\varepsilon$ is a centered noise random variable. An
extension of the above model is a generalized functional linear model [Muller
and Stadtmüller, 2005] given by

$Y=g\left(\int_{0}^{1}X(t)\beta(t)dt\right)+\varepsilon=g\left(\langle
X,\beta\rangle_{L^{2}([0,1])}\right)+\varepsilon,$ (5.2)

where $g:\mathbb{R}\to\mathbb{R}$ is a link function. We shall assume $g$ is
Lipschitz continuous in $\mathbb{R}$. A fundamental problem in statistics is
estimating the slope function $\beta$ in the FLM based on a training sample
consisting of $n$ independent copies of $(X,Y)$ (see, e.g., [Muller and
Stadtmüller, 2005, Chen et al., 2011, Yuan and Cai, 2010]). An efficient
estimator of $\beta$ is crucial for subsequently retrieving the regression
mean $\mathbb{E}[Y|X]:=g\left(\int_{0}^{1}X(t)\beta(t)dt\right)$. This paper
studies the approximation using neural networks; estimation based on training
samples is left for future study. Here, we aim to approximate the regression
map of generalized FLM by neural networks and derive the rate of convergence
of approximation error.

For a function covariate $X\in L^{2}([0,1])$, let
$F:L^{2}([0,1])\to\mathbb{R}$ be the regression map of generalized FLM given
by

$F:X\mapsto g\left(\int_{0}^{1}X(t)\beta(t)dt\right),\qquad X\in
L^{2}([0,1]).$ (5.3)

We shall restrict the domain of $F$ onto a subset of a RKHS
$\mathcal{H}_{K_{0}}$ induced by a Mercer kernel $K_{0}$ on
$[0,1]\times[0,1]$. Consider the unit ball
$\mathcal{K}_{0}:=\\{f\in\mathcal{H}_{K_{0}}\subset
L^{2}([0,1]):\|f\|_{\mathcal{H}_{K_{0}}}\leq 1\\}$. We assume
$\mathcal{H}_{K_{0}}$ is dense in $L^{2}([0,1])$. It is known that the RKHS
induced by an inverse multiquadric kernel or a Gaussian kernel is dense in
$L^{2}$ space, as well as the Sobolev space in $\mathbb{R}$ with order
$r>1/2$. We define a RKHS functional
$F|_{\mathcal{H}_{K_{0}}}:\mathcal{K}_{0}\to\mathbb{R}$ by

$F|_{\mathcal{H}_{K_{0}}}:X\mapsto
g\left(\int_{0}^{1}X(t)\beta(t)dt\right),\qquad X\in\mathcal{K}_{0}\subset
L^{2}([0,1]).$ (5.4)

Applying our previous theoretical findings, we prove that two-layer tanh
neural networks can approximate the generalized functional regression map
well. Concretely, for any function covariate $X\in\mathcal{K}_{0}$ and some
discrete points $\bar{t}=\\{t_{1},\cdots,t_{N}\\}\subset[0,1]$ (even spacing
is not necessary), there exists a network
$\widehat{G}:\mathbb{R}^{N}\to\mathbb{R}$ that takes
$X(\bar{t})=(X(t_{1}),\cdots,X(t_{N}))\in\mathbb{R}^{N}$ as input such that
$\sup_{X\in\mathcal{K}_{0}}\left|F|_{\mathcal{H}_{K_{0}}}(X)-\widehat{G}(X(\bar{t}))\right|\to
0$ as network widths increases.

For a function $f:[-T,T]\to\mathbb{R}$, we define its Lipschitz seminorm as

$\|f\|_{Lip[-T,T]}:=\sup_{x,y\in[-T,T],x\neq y}\frac{|f(x)-f(y)|}{|x-y|}.$

With a slight abuse of notation, we use
$\|\cdot\|_{L^{2}}:=\|\cdot\|_{L^{2}([0,1])}$ denote the $L^{2}$ norm over
$[0,1]$. The following result follows from Theorem 4.1 and Theorem 3.9. Its
proof is relegated to Appendix C.

######  Corollary 5.1 (Approximation of Regression Map in Generalized FLM).

Let $d=1$ and $\mathcal{X}=[0,1]$. Assume $\mathcal{H}_{K_{0}}$ is dense in
$L^{2}([0,1])$, where $\mathcal{H}_{K_{0}}$ is a RKHS induced by some Mercer
kernel $K_{0}$ which is $\alpha$-Hölder continuous for some $\alpha\in(0,1]$
with constant $C_{K_{0}}\geq 0$. Let
$\kappa=\sup_{t\in[0,1]}\sqrt{K_{0}(t,t)}$. Consider the regression map
$F|_{\mathcal{H}_{K_{0}}}$ defined in (5.4). There exists some
$M_{0}\in\mathbb{N}$ such that for every $M\in\mathbb{N}$ with $M>M_{0}$, by
taking some $\bar{t}=\\{t_{i}\\}_{i=1}^{N}\subset\mathcal{X}$ with
$N\in\mathbb{N}$, we have a tanh neural network
$\widehat{G}:\mathbb{R}^{N}\to\mathbb{R}$ with two hidden layers of widths at
most $N(M-1)$ and $3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$ satisfying

$\sup_{X\in\mathcal{K}_{0}}\left|F|_{\mathcal{H}_{K_{0}}}(X)-\widehat{G}(X(\bar{t}))\right|\leq
C_{F}(\epsilon_{K_{0}}(\bar{t}))+\frac{7N^{2}C_{G}}{M},$ (5.5)

with

$C_{F}=\|g\|_{Lip([-\kappa\|\beta\|_{L^{2}},\kappa\|\beta\|_{L^{2}}])}\|\beta\|_{L^{2}}\kappa,C_{G}=C_{F}\left(1+\|(K_{0}[\bar{t}])^{-1}\|_{op}\sqrt{N}C_{K}\left(h_{\bar{t}}\right)^{\alpha}\right),$

where the Gram matrix $K_{0}[\bar{t}]$ is defined in (4.2), the fill distance
$h_{\bar{t}}$ is given in (4.3), and the power function
$\epsilon_{K_{0}}(\bar{t})$ is defined in (4.5).

In particular, if we take $K_{0}$ to be a Gaussian kernel with $\sigma>0$,
$m=\left\lceil\frac{2\log(M)}{c+\sqrt{c^{2}+4\sigma^{2}\pi^{2}\log(M)}}\right\rceil>1$,
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}$, and $N=m+1$, we have

$\sup_{X\in\mathcal{K}}\left|F|_{\mathcal{H}_{K}}(X)-\widehat{G}(X(\bar{t}))\right|\leq
C_{\sigma,\pi,F}\log(M)\left(\frac{1}{M}\right)^{\frac{1}{2(1+\sigma\pi)}\left(\frac{1}{2}\log\log(M)-\log(c+\sigma\pi)\right)},$
(5.6)

where $C_{\sigma,\pi,F}$ is a constant depending on $\sigma,\pi$ and $C_{F}$,
$c$ is a positive constant given by $\eqref{f-Pf_Gaussian}$.

Note that we can also take $K_{0}$ to be an inverse multiquadric kernel or a
Sobolev kernel in $\mathbb{R}$. Then, by applying Theorem 3.6 or Theorem 3.4,
we can obtain the corresponding approximation error bound.

By the assumption that $\mathcal{H}_{K_{0}}$ is a dense subset, for any $X\in
cl(\mathcal{K})$ (i.e., closure of $\mathcal{K}$), there exists a sequence of
functions $\\{X_{n}:\mathcal{K}\to\mathbb{R},n\in\mathbb{N}\\}$ uniformly
convergent to $X$ (i.e., $\sup_{t\in[0,1]}|X_{n}(t)-X(t)|\to 0$ as
$n\to\infty$). We can see that the functional $F|_{\mathcal{H}_{K}}$ can be
extended to $F$ on $cl(\mathcal{K})$ as

$F(X)=\lim_{n\to\infty}F|_{\mathcal{H}_{K}}(X_{n})=\lim_{n\to\infty}g\left(\int_{0}^{1}X_{n}(t)\beta(t)dt\right),\qquad
X\in cl(\mathcal{K})\subset L^{2}([0,1])$

where $\\{X_{n}:\mathcal{K}\to\mathbb{R},n\in\mathbb{N}\\}$ is a sequence of
functions uniformly convergent to $X$.

In this paper, we only consider approximating functional regression maps using
neural networks. Based on our results, we find that the choice of $\bar{t}$
and the smoothness of the kernel that induces a RKHS play important roles in
improving the approximation abilities. To prove that neural networks can learn
functional regression maps well, we shall extend our work to a generalization
analysis in the future. Mathematically, the generalization error consists of
the approximation error and the estimation error (based on training samples of
independent copies of $(X,Y)$). We may assume that the slope function $\beta$
resides in RKHS in $L^{2}$ space to determine the estimation error. This is,
in fact, a common assumption imposed in the statistics literature on the
estimation and inference of FLM (see, [Yuan and Cai, 2010, Shang and Cheng,
2015, Balasubramanian et al., 2022]).

The next section discusses the general approaches and methods for establishing
our theorems.

##  6 Proof Sketch of Main Results

This section presents the general methods for approximating RKHS functionals
by tanh neural networks.

Our proofs are constructive and rely on three key ideas: (i) the construction
of a projection operator on RKHS using a set of nodal functions, (ii) the
estimation of power functions, which is widely adopted in the kernel
interpolation literature to quantify the worst-case uniform error between a
function and its interpolant, and (iii) the approximation of Lipschitz
continuous functions by tanh neural networks.

We begin our analysis with a decomposition of the uniform approximation error
$\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$. Consider
the following error decomposition:

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq$
$\displaystyle\underbrace{\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|}_{\text{error
term
(I)}}+\underbrace{\sup_{f\in\mathcal{K}}\left|F(Pf)-\widehat{G}(f(\bar{t}))\right|}_{\text{error
term (II)}},$ (6.1)

where $Pf:\mathcal{X}\to\mathbb{R}$ is a function carefully chosen.

We will first introduce our choice of $Pf$ in the Subsection 6.1 and show that
$\|f-Pf\|_{\infty}:=\max_{x\in\mathcal{X}}|f(x)-Pf(x)|$ can be estimated by
the power function given in (4.5). Then, we will estimate the power function
for different reproducing kernels of our interest in Subsection 6.2. This is
how we bound the error term (I). Next, to bound the second error term (II), we
bound the operator norm $\|(K[\bar{t}])^{-1}\|_{op}$ (in Subsection 6.4) and
then apply some existing results in neural network approximation (in
Subsection 6.5).

###  6.1 Approximation of Functions via Nodal functions

Recall that we choose a set of data locations
$\bar{t}=\\{t_{i}\\}_{i=1}^{N}\subset\mathcal{X}$ such that the Gram matrix
$K[\bar{t}]=(K(t_{i},t_{j}))_{i,j=1}^{N}$ given in (4.2) is invertible. We
construct a set of nodal functions $\\{\psi_{i}\\}_{i=1}^{N}$ on $\mathcal{X}$
associated with $\bar{t}$ as

$\psi_{i}(x):=\sum_{j=1}^{N}c_{j}K_{t_{j}}(x),\qquad
c_{j}\in\mathbb{R},i=1,\ldots,N,$ (6.2)

such that

$\psi_{i}(t_{j})=\begin{cases}1,&\text{if }j=i,\\\ 0,&\text{if }j\neq
i.\end{cases}$

We can easily check that the set of nodal functions $\\{\psi_{i}\\}_{i=1}^{N}$
are linearly independent.

Next, with the chosen $\bar{t}=\\{t_{i}\\}_{i=1}^{N}$, we define a projection
operator $P:\mathcal{H}_{K}\rightarrow\mathcal{H}_{K}$ as

$Pf=\sum_{i=1}^{N}f(t_{i})\psi_{i},\qquad f\in\mathcal{H}_{K}.$ (6.3)

$Pf$ is the orthogonal projection onto the finite-dimensional space
$\text{span}\\{K_{t_{i}}\\}_{i=1}^{N}$, as shown in the following lemma about
the interpolation and projection properties of $P$.

######  Lemma 6.1.

For any $f\in\mathcal{H}_{K}$ and $\bar{t}\subset\mathcal{X}$, we have

  1. 1.

$f(t_{\ell})=Pf(t_{\ell}),\qquad\ell=1,\ldots,N,$

  2. 2.

$\langle f-Pf,K_{t_{\ell}}\rangle_{\mathcal{H}_{K}}=0,\qquad\ell=1,\ldots,N.$

###### Proof of Lemma 6.1.

The nodal functions $\\{\psi_{i}\\}_{i=1}^{N}$ can be expressed explicitly as

$\begin{bmatrix}\psi_{1}\\\ \vdots\\\
\psi_{N}\end{bmatrix}=(K[\bar{t}])^{-1}\begin{bmatrix}K_{t_{1}}\\\ \vdots\\\
K_{t_{N}}\end{bmatrix}.$ (6.4)

In this way, we have

$\displaystyle\begin{bmatrix}\psi_{1}\\\ \vdots\\\
\psi_{N}\end{bmatrix}(t_{\ell})$ $\displaystyle=$
$\displaystyle(K[\bar{t}])^{-1}\begin{bmatrix}K(t_{1},t_{\ell})\\\ \vdots\\\
K(t_{N},t_{\ell})\end{bmatrix}$ $\displaystyle=$
$\displaystyle\left((K[\bar{t}])^{-1}K[\bar{t}]\right)_{\cdot,\ell}\qquad(\text{the
$\ell$-th column of }(K[\bar{t}])^{-1}K[\bar{t}])$ $\displaystyle=$
$\displaystyle I_{\cdot,\ell}.\qquad(\text{the $\ell$-th column of identity
matrix }I)$

We can see that

$\psi_{i}(t_{\ell})=\begin{cases}1,&\text{if }i=\ell,\\\ 0,&\text{if
}i\neq\ell.\end{cases}$

Then we have

$\displaystyle
Pf(t_{\ell})=\sum_{i=1}^{N}f(t_{i})\psi_{i}(t_{\ell})=f(t_{\ell})\psi_{\ell}(t_{\ell})=f(t_{\ell}).$

The proof of statement 1. is complete.

Recall the reproducing property of RKHS: $f(x)=\langle
f,K_{x}\rangle_{\mathcal{H}_{K}}$. Since $f(t_{\ell})-Pf(t_{\ell})=0$, we know

$\langle f,K_{t_{\ell}}\rangle_{\mathcal{H}_{K}}-\langle
Pf,K_{t_{\ell}}\rangle_{\mathcal{H}_{K}}=f(t_{\ell})-Pf(t_{\ell})=0\iff\langle
f-Pf,K_{t_{\ell}}\rangle_{\mathcal{H}_{K}}=0.$

The proof of statement 2. is complete. ∎

The next lemma presents a uniform bound of $|f(x)-Pf(x)|$ for any
$x\in\mathcal{X}$, implying that $Pf$ gives a good approximation of $f$ on
$\mathcal{X}$. We use the power function given in (4.5) for the measurement:

$\epsilon_{K}(\bar{t}):=\max_{x\in\mathcal{X}}\min_{c\in\mathbb{R}^{N}}\left\|K_{x}-\sum_{i=1}^{N}c_{i}K_{t_{i}}\right\|_{\mathcal{H}_{K}}=\max_{x\in\mathcal{X}}\min_{c\in\mathbb{R}^{N}}\left\\{K(x,x)-2\sum_{i=1}^{N}c_{i}K(x,t_{i})+\sum_{i=1}^{N}\sum_{j=1}^{N}c_{i}c_{j}K(t_{i},t_{j})\right\\}^{\frac{1}{2}}.$

Lemma 6.2 is a well-known result in the kernel literature. We provide its
proof for completeness in Appendix B.

######  Lemma 6.2.

For any $\bar{t}\subset\mathcal{X}$ and $\\{\psi_{i}\\}_{i=1}^{N}$ defined
explicitly in (6.4), we have

$\|f-Pf\|_{\infty}\leq\|f\|_{\mathcal{H}_{K}}\epsilon_{K}(\bar{t}),\qquad
f\in\mathcal{H}_{K}.$ (6.5)

Recall that we assume, throughout the paper, $F$ is $s$-Hölder continuous for
$0<s\leq 1$ with constant $C_{F}\geq 0$. As a consequence of Lemma 6.2, we
know

$|F(f)-F(Pf)|\leq C_{F}\|f-Pf\|^{s}_{\infty}\leq
C_{F}\|f\|^{s}_{\mathcal{H}_{K}}(\epsilon_{K}(\bar{t}))^{s},\qquad\forall
f\in\mathcal{H}_{K}.$

We consider $\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq
1\\}$ with $\mathcal{H}_{K}$ induced by some Mercer kernel $K$. We then have,

$\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|\leq C_{F}(\epsilon_{K}(\bar{t}))^{s}.$
(6.6)

This is an upper bound for the error term (I) in (6.1) for a general Mercer
kernel $K$. Notice that this is the first term in the error bound presented in
(5.5) in Theorem 4.1.

Now the important questions are: how should we choose
$\bar{t}=\\{t_{i}\\}_{i=1}^{N}$, and how do we estimate the power function
$\epsilon_{K}(\bar{t})$ of a Mercer kernel of interest?

###  6.2 Estimation of the Power Function

From now on, we take a set of $N=(m+1)^{d}$ data locations
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}$ on
$\mathcal{X}=[0,1]^{d}$, for some $m\in\mathbb{N}$. This is our choice of
$\bar{t}$ in Theorem 3.6, 3.9, and 3.4. We can see that

$\max_{x\in\mathcal{X}}\min_{t\in\bar{t}}\ d_{\mathcal{X}}(x,t)\rightarrow
0\quad\text{ as }N\rightarrow\infty.$

For any $x\in\mathcal{X}$, let $t_{j_{x}}$ be the closest point in $\bar{t}$
to $x$, with $j_{x}\in\\{1,\ldots,N\\}$. Then,

$|x_{k}-(t_{j_{x}})_{k}|\leq\frac{1}{m},\qquad k=1,\ldots,d$

and

$\displaystyle
d_{\mathcal{X}}(x,t_{j_{x}})\leq\sqrt{d\left(\frac{1}{m}\right)^{2}}=\frac{\sqrt{d}}{m},$

and thus the fill distance of $\bar{t}$ is

$h_{\bar{t}}=\sup_{x\in\mathcal{X}}d_{\mathcal{X}}(x,t_{j_{x}})\leq\frac{\sqrt{d}}{m}.$
(6.7)

Recall that we assume the Mercer kernel $K$ is $\alpha$-Hölder continuous for
$0<\alpha\leq 1$ with constant $C_{K}\geq 0$. The following lemma presents an
estimate of the power function $\epsilon_{K}(\bar{t})$ of a general
$\alpha$-Hölder continuous Mercer kernel $K$, and a uniform bound of $|f-Pf|$
for $f\in\mathcal{H}_{K}$ induced by such a kernel $K$.

######  Lemma 6.3.

For $\mathcal{X}=[0,1]^{d}$, choose
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}$ for some
$m\in\mathbb{N}$. We have

$\epsilon_{K}(\bar{t})\leq
2C_{K}\left(\frac{\sqrt{d}}{m}\right)^{\alpha}\quad\text{ and thus
}\quad\|f-Pf\|_{\infty}\leq
2C_{K}\left(\frac{\sqrt{d}}{m}\right)^{\alpha}\|f\|_{\mathcal{H}_{K}},\qquad
f\in\mathcal{H}_{K}.$ (6.8)

The proof of Lemma 6.3 is given in Appendix B. In the following, we give
explicit bounds of $\|f-Pf\|_{\infty}$ for specific kernels of interest.

For an inverse multiquadric kernel given in (3.5), an upper bound of
$\epsilon_{K}(\bar{t})$ and $\|f-Pf\|_{\infty}$ are established in [Fasshauer,
2005, Equation (15)] for some positive constant $c$:

$\epsilon_{K}(\bar{t})\leq e^{\frac{-cm}{\sqrt{d}}}\quad\text{and thus
}\quad\|f-Pf\|_{\infty}\leq
e^{\frac{-cm}{\sqrt{d}}}\|f\|_{\mathcal{H}_{K}},\qquad f\in\mathcal{H}_{K}.$
(6.9)

For Gaussian kernel given in (3.6), an upper bound of $\epsilon_{K}(\bar{t})$
and $\|f-Pf\|_{\infty}$ are established in [Fasshauer, 2005, Equation (14)]
for some positive constant $c$:

$\epsilon_{K}(\bar{t})\leq
e^{\frac{-cm\left|\log\left(\frac{\sqrt{d}}{m}\right)\right|}{\sqrt{d}}}\quad\text{and
thus }\quad\|f-Pf\|_{\infty}\leq
e^{\frac{-cm\left|\log\left(\frac{\sqrt{d}}{m}\right)\right|}{\sqrt{d}}}\|f\|_{\mathcal{H}_{K}},\qquad
f\in\mathcal{H}_{K}.$ (6.10)

While $d\in\mathbb{N}$ represents the data dimension, $m\in\mathbb{N}$
measures the number of locations in $\bar{t}$ and is a variable of our choice.
If we choose $m$ such that $m>\sqrt{d}$, we
have$\left|\log\left(\frac{\sqrt{d}}{m}\right)\right|=\log\left(\frac{m}{\sqrt{d}}\right)$
and thus

$\|f-Pf\|_{\infty}\leq
e^{\frac{-cm\log\left(\frac{m}{\sqrt{d}}\right)}{\sqrt{d}}}\|f\|_{\mathcal{H}_{K}},\qquad
f\in\mathcal{H}_{K}.$ (6.11)

We now turn our attention to the Sobolev Space $W^{r}_{2}(\mathcal{X})$ as an
RKHS $\mathcal{H}_{K}$. According to [Schaback, 1997, Theorem 6.5.3], when
$r>d/2$ and $r-d/2\notin\mathbb{N}$, an upper bound of the power function is
derived from the well-known Sobolev Embedding Theorem as

$\epsilon_{K}(\bar{t})\leq C^{\prime}_{r,d}d^{r-\frac{d}{2}}m^{d-2r},$

where $C^{\prime}_{r,d}$ is some positive constant depending on $r$ and $d$.
Thus, we have

$\|f-Pf\|_{\infty}\leq
C^{\prime}_{r,d}d^{r-\frac{d}{2}}m^{d-2r}\|f\|_{\mathcal{H}_{K}}.$ (6.12)

Later, we will use the above uniform bounds of $\|f-Pf\|_{\infty}$ to prove
our main theorems.

###  6.3 A New Function $G$ and its Regularity

Now, we move on to bound the error term (II) in (6.1):
$\sup_{f\in\mathcal{K}}\left|F(Pf)-\widehat{G}(f(\bar{t}))\right|$.

In order to approximate $F(Pf)=F\left(\sum_{i=1}^{N}f(t_{i})\psi_{i}\right)$,
we define another function $G:[0,1]^{N}\rightarrow\mathbb{R}$ by

$G(c)=F\left(\sum_{i=1}^{N}c_{i}\psi_{i}\right),\qquad c\in[0,1]^{N}.$ (6.13)

Then, we have

$G(f(\bar{t}))=F\left(\sum_{i=1}^{N}f(t_{i})\psi_{i}\right)=F(Pf).$

The idea of approximating the functional $F$ is by approximating the function
$G$ using a fully-connected neural network. For this purpose, we need to
figure out the regularity of $G$, presented in the following lemma. Its proof
is relegated to Appendix B.

######  Lemma 6.4 (Regularity of the function $G$).

For any $x\in\mathcal{X}$ and
$\bar{t}=\\{t_{i}\\}_{i=1}^{N}\subset\mathcal{X}$, if the functional
$F:\mathcal{K}\rightarrow\mathbb{R}$ is $s$-Hölder continuous for $0<s\leq 1$
with constant $C_{F}\geq 0$, then the associated function $G$ is $s$-Hölder
continuous with constant

$C_{G}=C_{F}\left(1+\|(K[\bar{t}])^{-1}\|_{op}\sqrt{N}C_{K}\left(h_{\bar{t}}\right)^{\alpha}\right)^{s},$

where $\|T\|_{op}=\underset{\|x\|=1}{\sup}\|T(x)\|$ denotes the operator norm
of $T$, $h_{\bar{t}}$ denotes the fill distance of $\bar{t}$ given in (4.3).

To know the Hölder constant $C_{G}$ explicitly, we proceed to compute the
operator norm $\|(K[\bar{t}])^{-1}\|_{op}$. Since $K[\bar{t}]$ is a real,
symmetric matrix, it is diagonalizable, and so is $(K[\bar{t}])^{-1}$. Let
$\\{\lambda_{1},\ldots,\lambda_{N}\\}$ denote the non-increasing sequence of
eigenvalues of $K[\bar{t}]$. We know

$K[\bar{t}]=V\Lambda V^{-1}\qquad\text{and }\qquad
K[\bar{t}]^{-1}=V\Lambda^{-1}V^{-1},$

where $\Lambda$ is a $N\times N$ diagonal matrix whose diagonal entries are
the eigenvalues $\\{\lambda_{1},\ldots,\lambda_{N}\\}$, and the columns of $V$
are eigenvectors of $K[\bar{t}]$. We then have

$\|(K[\bar{t}])^{-1}\|_{op}=\frac{1}{\min_{1\leq j\leq
N}\lambda_{j}}=:\frac{1}{\lambda_{N}},$ (6.14)

where $\lambda_{N}$ is the smallest eigenvalue of $K[\bar{t}]$.

Since $K[\bar{t}]v=\lambda_{N}v$ for some eignevector
$v=(v_{1},\cdots,v_{N})\in\mathbb{R}^{N}$ of norm $1$, we have

$\displaystyle\lambda_{N}=\lambda_{N}v^{T}v=v^{T}K[\bar{t}]v=\sum_{i=1}^{N}\sum_{\ell=1}^{N}v_{i}v_{\ell}K(t_{i},t_{\ell}).$
(6.15)

In other words, to compute $\|(K[\bar{t}])^{-1}\|_{op}$, it suffices to
compute
$\lambda_{N}=\sum_{i=1}^{N}\sum_{\ell=1}^{N}v_{i}v_{\ell}K(t_{i},t_{\ell})$.

###  6.4 Bounding the Operator Norm $\|(K[\bar{t}])^{-1}\|_{op}$

In general, computing the operator norm $\|(K[\bar{t}])^{-1}\|_{op}$ can be
challenging, especially when we do not have information about the eigenvalues
or eigenvectors of the Gram matrix $K[\bar{t}]$. The lemma below tells us that
if kernel $K$ is translation-invariant, we can bound
$\|(K[\bar{t}])^{-1}\|_{op}$ by the Fourier transform of $K$:

######  Lemma 6.5.

Let $\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}$ on $\mathcal{X}$ for
some $m\in\mathbb{N}$. For a translation-invariant and reproducing kernel $K$
(or $\phi)$ defined over $\mathbb{R}^{d}$, we have

$\|(K[\bar{t}])^{-1}\|_{op}=\frac{1}{\lambda_{N}}\leq\frac{1}{m\Gamma_{m}}\quad\text{with
}\Gamma_{m}:=\min_{\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}}\widehat{\phi}(\xi)>0,$
(6.16)

where $\lambda_{N}$ is the smallest eigenvalue of $K[\bar{t}]$,
$\widehat{\phi}$ is the Fourier transform of $\phi$.

The proof of Lemma 6.5 is relegated to Appendix B.

Next, we apply Lemma 6.4 and Lemma 6.5 to bound $\|(K[\bar{t}])^{-1}\|_{op}$
and subsequently estimate the regularity of $G$ corresponds to functional $F$
on Sobolev space, and on RKHS’s induced by the inverse multiquadric kernel and
Gaussian kernel, respectively.

####  6.4.1 Sobolev Space $W^{r}_{2}(\mathcal{X})$

From Subsection 3.1, we know that $W^{r}_{2}(\mathcal{X})$ with order $r>d/2$
is a RKHS induced by a translation-invariant kernel, whose Fourier transform
is given by $\widehat{\phi}(\xi)=(1+|\xi|^{2})^{-r},\xi\in\mathbb{R}^{d}$. For
$\widehat{\phi}$ and any $m\in\mathbb{N}$,

$\Gamma_{m}=\min_{\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}}\widehat{\phi}(\xi)\geq\left(1+\frac{dm^{2}}{4}\right)^{-r}$
(6.17)

because $|\xi|^{2}\leq\frac{dm^{2}}{4}$ for
$\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}$. Using this fact, we establish the
regularity of the function $G$ corresponding to a functional $F$ on
$W^{r}_{2}(\mathcal{X})$, presented in the lemma below. Its proof is relegated
to Appendix B.

######  Lemma 6.6.

Let $d\in\mathbb{N},r\in\mathbb{R}_{+},\mathcal{X}=[0,1]^{d}$, and $r-d/2>1$.
Take $\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}$ on $\mathcal{X}$ for
some $m\in\mathbb{N}$, and $N=(m+1)^{d}$. Consider $\mathcal{K}:=\\{f\in
W^{r}_{2}(\mathcal{X}):\|f\|_{W^{r}_{2}(\mathcal{X})}\leq 1\\}$. If
$F:\mathcal{K}\to\mathbb{R}$ is $s$-Hölder continuous with $s\in(0,1]$ and
constant $C_{F}\geq 0$, the corresponding function $G:[0,1]^{N}\to\mathbb{R}$
given by (6.13) is $s$-Hölder continuous with constant

$C_{G}=C_{F}C_{r,d,s}m^{2rs+\frac{ds}{2}-2s}d^{rs+\frac{s}{2}},$

where $C_{r,d,s}$ is a positive constant depends only on $r,d$, and $s$.

####  6.4.2 Inverse Multiquadric Kernel

The inverse multiquadric kernel is a translation-invariant and reproducing
kernel. We know $|u-v|\leq\underbrace{\sqrt{1+\cdots+1}}_{d\text{
times}}=\sqrt{d}$ for $u,v\in\mathcal{X}$. By the Mean Value Theorem, we have
for $u,v,\tilde{v}\in\mathcal{X}$,

$\displaystyle\ |K(u,v)-K(u,\tilde{v})|$ $\displaystyle=$
$\displaystyle\left|(\sigma^{2}+|u-v|^{2})^{-\beta}-(\sigma^{2}+|u-\tilde{v}|^{2})^{-\beta}\right|$
$\displaystyle\leq$
$\displaystyle\beta\sigma^{-2\beta-2}\left|\sigma^{2}+|u-v|^{2}-\sigma^{2}-|u-\tilde{v}|^{2}\right|$
$\displaystyle\leq$
$\displaystyle\beta\sigma^{-2\beta-2}(|u-v|+|u-\tilde{v}|)(|u-v|-|u-\tilde{v}|)$
$\displaystyle\leq$ $\displaystyle
2\sqrt{d}\beta\sigma^{-2\beta-2}|v-\tilde{v}|.$

We see that this kernel is $\alpha$-Hölder continuous for $\alpha=1$ with
constant $C_{K}=2\sqrt{d}\beta\sigma^{-2\beta-2}$. Based on this information,
we can deduce the regularity of function $G$ (given by (6.13)) corresponds to
the functional $F$ on inverse multiquadric kernel-induced RKHS’s. The proof of
the following lemma is given in Appendix B.

######  Lemma 6.7.

Let $d\in\mathbb{N},\sigma,\beta>0,\mathcal{X}=[0,1]^{d}$, and
$M_{d}=12\left(\frac{\pi\Gamma^{2}(\frac{d+2}{2})}{9}\right)\leq 6.38d$. Take
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}\in\mathcal{X}$ for some
$m\in\mathbb{N}$, and $N=(m+1)^{d}$. Consider
$\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq 1\\}$ with
$\mathcal{H}_{K}$ induced by a inverse multiquadric kernel. If
$F:\mathcal{K}\to\mathbb{R}$ is $s$-Hölder continuous with $s\in(0,1]$ and
constant $C_{F}\geq 0$, the function $G:[0,1]^{N}\rightarrow\mathbb{R}$ given
by (6.13) is $s$-Hölder continuous with constant

$C_{G}=C_{F}\left(1+\frac{4d\beta\sigma^{-2\beta-2}}{C_{\sigma,\beta,d}}(2m)^{-\beta-\frac{1}{2}}e^{4\sigma
M_{d}m}\right)^{s},$ (6.18)

where $C_{\sigma,\beta,d}$ is a constant depending only on $\sigma,\beta,d$.

####  6.4.3 Gaussian Kernel

The Gaussian kernel $K(u,v)=e^{-\frac{|u-v|^{2}}{2\sigma^{2}}}$ for
$u,v\in\mathcal{X}$ is translation-invariant and reproducing, corresponding to
$\phi(x)=e^{-\frac{x^{2}}{2\sigma^{2}}}$. Then, by the Mean Value Theorem, we
have for $u,v,\tilde{v}\in\mathcal{X}$,

$\displaystyle\ |K(u,v)-K(u,\tilde{v})|$ $\displaystyle=$
$\displaystyle\left|e^{-\frac{|u-v|^{2}}{2\sigma^{2}}}-e^{-\frac{|u-\tilde{v}|^{2}}{2\sigma^{2}}}\right|$
$\displaystyle\leq$ $\displaystyle
1\cdot\left|\frac{|u-v|^{2}}{2\sigma^{2}}-\frac{|u-\tilde{v}|^{2}}{2\sigma^{2}}\right|$
$\displaystyle\leq$
$\displaystyle\frac{|u-v|+|u-\tilde{v}|}{2\sigma^{2}}(|u-v|-|u-\tilde{v}|)$
$\displaystyle\leq$ $\displaystyle\frac{\sqrt{d}}{\sigma^{2}}|v-\tilde{v}|.$

We see that the Gaussian kernel is $\alpha$-Hölder continuous for $\alpha=1$
with constant $C_{K}=\frac{\sqrt{d}}{\sigma^{2}}$. The following lemma
presents the regularity of function $G$ corresponding to functional $F$ on
Gaussian kernel-induced RKHS’s. Its proof is relegated to Appendix B.

######  Lemma 6.8.

Let $d\in\mathbb{N},\sigma>0,$ and $\mathcal{X}=[0,1]^{d}$. Take
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}\in\mathcal{X}$ for some
$m\in\mathbb{N}$, and $N=(m+1)^{d}$. Consider
$\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq 1\\}$ with
$\mathcal{H}_{K}$ induced by a Gaussian kernel. If
$F:\mathcal{K}\to\mathbb{R}$ is $s$-Hölder continuous with $s\in(0,1]$ and
constant $C_{F}\geq 0$, the corresponding function
$G:[0,1]^{N}\rightarrow\mathbb{R}$ given by (6.13) is $s$-Hölder continuous
with constant

$C_{G}=C_{F}C_{\sigma,d,s}e^{\sigma^{2}\pi^{2}dsm^{2}},$ (6.19)

where $C_{\sigma,d,s}$ is a constant depending only on $\sigma,d,$ and $s$.

Now that we have established the regularity of function $G$, we proceed to
approximate $G$ using tanh neural networks.

###  6.5 Approximation of Function using Tanh Neural Networks

Here, we would like to apply a recent result in approximating Lipschitz
continuous functions by tanh neural network from [De Ryck et al., 2021]. We
will use this existing result to approximate the function $G$ using two-layer
tanh neural networks.

######  Lemma 6.9 (Corollary 5.4 in [De Ryck et al., 2021]).

Let $N\in\mathbb{N}$ and let $f:[0,1]^{N}\rightarrow\mathbb{R}$ be a Lipschitz
continuous function with Lipschitz constant $L>0$. For every $M\in\mathbb{N}$
with $M>5N^{2}$, there exists a tanh neural network $\widehat{f}$ with two
hidden layers of widths at most $N(M-1)$ and
$3\left\lceil\frac{N+1}{2}\right\rceil\begin{pmatrix}2N-1\\\
N\end{pmatrix}M^{N}$ (or $M-1$ and $6M$ for $N=1$), such that

$\displaystyle\left\|\widehat{f}-f\right\|_{L^{\infty}[0,1]^{N}}\leq\frac{7N^{2}L}{M}.$

######  Remark 6.10.

It is shown in [De Ryck et al., 2021, Lemma 2.1] that $\begin{pmatrix}2N-1\\\
N\end{pmatrix}<5^{N}$. The total number of parameters in the network is at
most $\mathcal{O}(N(5M)^{N})$ with $M>5N^{2}$.

We first present an approximation result regarding a general translation-
invariant Mercer kernel $K$ and $\mathcal{X}=[0,1]^{d}$.

######  Theorem 6.11.

Let $m\in\mathbb{N},d\in\mathbb{N},\mathcal{X}=[0,1]^{d}$, and $K$ is a
translation-invariant kernel. Also let
$\Gamma_{m}=\min_{\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}}\widehat{\phi}(\xi)>0$.
We take a set of $N=(m+1)^{d}$ points
$\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}$ on $\mathcal{X}$.
Consider the function $G:[-1,1]^{N}\rightarrow\mathbb{R}$ defined in (6.13).
It follows from Lemma 6.4 and Lemma 6.5 that $G$ is $s$-Hölder continuous for
$0\leq s\leq 1$ with constant

$C_{G}=C_{F}\left(1+\frac{1}{m\Gamma_{m}}\sqrt{N}C_{K}\left(\frac{\sqrt{d}}{m}\right)^{\alpha}\right)^{s}.$

From Lemma 6.9, for every $M\in\mathbb{N}$ with $M>5N^{2}$, there exists a
tanh neural network $\widehat{G}:[-1,1]^{N}\rightarrow\mathbb{R}$ with two
hidden layers of widths at most $N(M-1)$ and
$3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$, such that

$\displaystyle\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{N}}\leq\frac{7N^{2}C_{G}}{M}.$

As consequences of the above Theorem 6.11, we can derive explicit error bounds
on the approximation of $G$ functions induced from Reproducing kernel in
Sobolev space $W^{r}_{2}(\mathcal{X})$ (Corollary 6.12), inverse multiquadric
kernel (Corollary 6.13), and Gaussian Kernel (Corollary 6.14), respectively.

######  Corollary 6.12 (Reproducing kernel in Sobolev space
$W^{r}_{2}(\mathcal{X})$).

Suppose $K$ is a reproducing kernel in $W^{r}_{2}(\mathcal{X})$. We take
$\mathcal{X},N,$ and $\bar{t}$ as in Theorem 6.11. Consider the function
$G:[-1,1]^{N}\rightarrow\mathbb{R}$ defined in (6.13). We know that the
corresponding $G$ function is $s$-Holder with constant

$C_{G}=C_{F}C_{r,d,s}m^{2rs+\frac{ds}{2}-2s}d^{rs+\frac{s}{2}}.$

For every $M\in\mathbb{N}$ with $M>5N^{2}$, there exists a tanh neural network
$\widehat{G}:[-1,1]^{N}\rightarrow\mathbb{R}$ with two hidden layers of widths
at most $N(M-1)$ and $3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$ (or $M-1$
and $6M$ for $N=1$), such that

$\displaystyle\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{N}}=\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{(m+1)^{d}}}\leq\frac{7(m+1)^{2d}}{M}C_{F}C_{r,d,s}m^{2rs+\frac{ds}{2}-2s}d^{rs+\frac{s}{2}}.$

######  Corollary 6.13 (Inverse multiquadric kernel).

Suppose $K$ is an inverse multiquadric kernel given by (3.5). We take
$\mathcal{X},N,$ and $\bar{t}$ as in Theorem 6.11. Consider the function
$G:[-1,1]^{N}\rightarrow\mathbb{R}$ defined in (6.13). From Lemma 6.4 and
(6.18), the function $G$ is $s$-Hölder for $0\leq s\leq 1$ with constant

$C_{G}=C_{F}C_{\sigma,\beta,d,s}(2m)^{-s\beta-\frac{s}{2}}e^{4\sigma
M_{d}sm},$

where $M_{d}=12\left(\frac{\pi\Gamma^{2}(\frac{d+2}{2})}{9}\right)\leq 6.38d$.
For every $M\in\mathbb{N}$ with $M>5N^{2}$, there exists a tanh neural network
$\widehat{G}:[-1,1]^{N}\rightarrow\mathbb{R}$ with two hidden layers of widths
at most $N(M-1)$ and $3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$, such
that

$\displaystyle\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{N}}=\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{(m+1)^{d}}}\leq\frac{7(m+1)^{2d}(2m)^{-s\beta-s/2}C_{F}C_{\sigma,\beta,d,s}e^{4\sigma
M_{d}sm}}{M}.$

######  Corollary 6.14 (Gaussian kernel).

Suppose $K$ is a Gaussian kernel given in (3.6). We take $\mathcal{X},N,$ and
$\bar{t}$ as in Theorem 6.11. Consider the function
$G:[-1,1]^{N}\rightarrow\mathbb{R}$ defined in (6.13). It follows from Lemma
6.4 and (6.19) that $G$ is $s$-Hölder continuous for $0\leq s\leq 1$ with
constant

$C_{G}=C_{F}C_{\sigma,d,s}e^{\sigma^{2}\pi^{2}dsm^{2}}.$

For every $M\in\mathbb{N}$ with $M>5N^{2}$, there exists a tanh neural network
$\widehat{G}:[-1,1]^{N}\rightarrow\mathbb{R}$ with two hidden layers of widths
at most $N(M-1)$ and $3\left\lceil\frac{N+1}{2}\right\rceil(5M)^{N}$, such
that

$\displaystyle\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{N}}=\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{(m+1)^{d}}}\leq\frac{7(m+1)^{2d}}{M}C_{F}C_{\sigma,d,s}e^{\sigma^{2}\pi^{2}dsm^{2}}.$

##  7 Remarks and Conclusions

While it is widely acknowledged that neural networks are universal function
approximators, whether such universality can be extended to infinite-
dimensional spaces remains unknown. This paper investigates the approximation
theory of functionals, which are mappings from a space of functions to
$\mathbb{R}$.

In summary, we establish explicit error bounds, with respect to the number of
network parameters, on the approximations of functionals on Sobolev space and
RKHS’s induced by the inverse multiquadric kernel and Gaussian kernel,
respectively. We adopt standard neural networks equipped with tanh activation
functions, commonly used in operator learning these days [Lu et al., 2021,
Wang et al., 2021]. Our main theorems suggest that such networks can achieve
any arbitrary approximation accuracy if there are sufficiently many network
parameters. We achieve near polynomial approximation rates for approximating
RKHS functionals induced by the inverse multiquadric and Gaussian kernels. Our
results have shown improvement compared to the previous rates given in
[Mhaskar and Hahm, 1997] and [Song et al., 2023].

Most existing works [Rossi and Conan-Guez, 2005, Song et al., 2023] require a
set of handcrafted basis functions to convert functional inputs to vectors via
basis function expansion; our design evaluates the input function at some
discrete points in the domain where uniform spacing is not necessary. Our
method is much simpler than those in the literature.

Moreover, we present a general error bound on approximating functionals on the
RKHS induced by some general Mercer kernel. This error bound exhibits a trade-
off associated with $N$, which denotes the number of points we use to evaluate
our input function. When $N$ is too small, we observe an increase in the power
function (measures the distance between a function and its interpolant), while
with a very large $N$, the network size becomes too big and results in
overfitting. We shall choose an optimal $N$ to strike a balance. In the
previously mentioned theorems, we present our choice of $N$; its derivation is
given in the proofs. Furthermore, we apply our findings to functional
regression, proving that tanh neural networks can accurately approximate the
regression maps in generalized functional linear models. This novel result
provides an insight that deep learning may be applied successfully in
functional data analysis.

To our knowledge, this work is the first to study the approximation of
functionals on RKHS’s. All existing works in this regard focus on functionals
on $L^{p}$ spaces. We also want to highlight that no method other than neural
networks, such as spline or sieve, has been studied for approximating
functionals. By considering RKHS, our methods allow extensions to low-
dimensional manifolds embedded in Euclidean spaces. This is beyond the scope
of this paper with $\mathcal{X}=[0,1]^{d}$ and will be discussed in our
further study.

##  8 Acknowledgments and Disclosure of Funding

This material is based upon work supported by the National Science Foundation
under grant no. 2229876 is partly supported by funds provided by the National
Science Foundation, the Department of Homeland Security, and IBM. Any
opinions, findings, conclusions, or recommendations expressed in this material
are those of the authors and do not necessarily reflect the views of the
National Science Foundation or its federal agency and industry partners. Tian-
Yi Zhou and Xiaoming Huo are also partially sponsored by NSF grants DMS
2015363 and the A. Russell Chandler III Professorship at Georgia Institute of
Technology. Guang Cheng and Namjoon Suh are partially supported by the Office
of Naval Research under ONR N00014-22-1-2680 and the National Science
Foundation under SCALE MoDL 2134209. Namjoon Suh is also partially supported
by the Institute of Digital Research & Education (IDRE) Postdoctoral
Fellowship at UCLA.

##  Appendix A Proof of Statements in Remarks

Here, we present the technical proofs of the statements given in Remark 3.5
and Remark 3.7, respectively.

###### Proof of Remark 3.5.

Recall that in Theorem 3.4, we established an approximation error bound of
$\mathcal{O}\left(M^{-\frac{2r-d}{2(2r-1)}}\right)$, where
$r\in\mathbb{R}_{+},d\in\mathbb{N}$, and $r-d/2>1$. The maximum total number
of network parameters is $\mathcal{N}=\mathcal{O}\left(N(5M)^{N}\right)$,
where $N=(m+1)^{d}=\left(\left\lceil
M^{\frac{1}{2s(2r-1)}}\right\rceil+1\right)^{d}$. We will prove that the
convergence rate of the approximation error achieves a logarithmic rate (in
terms of $\mathcal{N}$).

Notice that

$\displaystyle\log(\mathcal{N})\leq
C^{\prime}N\log(M)=C^{\prime}(m+1)^{d}\log(M)=C^{\prime}\left(\left\lceil
M^{\frac{1}{2s(2r-1)}}\right\rceil+1\right)^{d}\log(M),$

where $C^{\prime}\leq 4\log(C)\log(5)$. We thus have

$\displaystyle\log(\mathcal{N})\leq
C^{\prime}\left(M^{\frac{1}{2s(2r-1)}}+2\right)^{d}\log(M)\leq
C^{\prime}3^{d}M^{\frac{d}{2s(2r-1)}}\log(M).$

We also know that $\mathcal{N}$ is at least $m$ and thus

$\displaystyle\log(\mathcal{N})\geq C^{\prime\prime}M^{\frac{d}{2s(2r-1)}}$

and

$\displaystyle\log(\log(\mathcal{N}))\geq\log(C^{\prime\prime})+\frac{d}{2s(2r-1)}\log(M),$

where $C^{\prime\prime}$ is a positive constant. Then, we get

$\displaystyle\frac{\log(\mathcal{N})}{\log(\log(\mathcal{N}))}\leq\frac{C^{\prime}3^{d}M^{\frac{d}{2s(2r-1)}}\log(M)}{\log(C^{\prime\prime})+\frac{d}{2s(2r-1)}\log(M)}\leq\frac{C^{\prime}3^{d}M^{\frac{d}{2s(2r-1)}}}{\frac{d}{2s(2r-1)}}=C^{\prime}3^{d}\frac{2s(2r-1)}{d}M^{\frac{d}{2s(2r-1)}}$

and thus

$\displaystyle
M\geq\frac{d}{C^{\prime}3^{d}2s(2r-1)}\left(\frac{\log(\mathcal{N})}{\log(\log(\mathcal{N}))}\right)^{\frac{2s(2r-1)}{d}}.$

Finally, we obtain

$\displaystyle M^{-\frac{2r-d}{2(2r-1)}}\leq
C_{d,r,s}\left(\frac{\log(\mathcal{N})}{\log(\log(\mathcal{N}))}\right)^{-\frac{s(2r-d)}{d}},$

where
$C_{d,r,s}=\left(\frac{d}{C^{\prime}3^{d}2s(2r-1)}\right)^{-\frac{2r-d}{2(2r-1)}}$.
The proof is complete. ∎

Next, we present the proof of the statement in Remark 3.7.

###### Proof of Remark 3.7.

Recall in Theorem 3.6, we establish an approximation error bound of order

$\mathcal{O}\left((\log(M))^{\max\\{0,2d-s\beta\\}}\left(\frac{1}{M}\right)^{\frac{c}{4M_{d}\sqrt{d}\sigma+c}}\right),$

where $d\in\mathbb{N},\sigma,\beta>0,s\in(0,1],M_{d}\leq 6.38d$, and $c>0$ is
a positive constant given in (6.9). The maximum number of network parameters
is $\mathcal{N}=\mathcal{O}\left(N(5M)^{N}\right)$, where
$N=(m+1)^{d}=\left(\left\lceil\frac{\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}\right\rceil+1\right)^{d}$.

We first prove that the convergence rate of the approximation error is faster
than the logarithmic rate w.r.t. $\mathcal{N}$. We know $\mathcal{N}\leq
CN(5M)^{N}$ for some positive constant $C$. We then have

$\displaystyle\log(\mathcal{N})\leq C^{\prime}N\log(M)$ $\displaystyle=$
$\displaystyle C^{\prime}(m+1)^{d}\log(M)$ (A.1) $\displaystyle=$
$\displaystyle C^{\prime}\left(\left\lceil\frac{\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}\right\rceil+1\right)^{d}\log(M)=C_{\sigma,s,d}(\log(M))^{d+1},$

where $C^{\prime}\leq 4\log(C)\log(5)$, $C_{\sigma,s,d}\leq
C^{\prime}\left(\frac{1}{4M_{d}\sigma s+\frac{cs}{\sqrt{d}}}+2\right)^{d}$.

We thus have

$\log(M)\geq\left(\frac{\log(\mathcal{N})}{C_{\sigma,s,d}}\right)^{\frac{1}{d+1}}.$
(A.2)

We can express $\left(\frac{1}{M}\right)^{\frac{c}{4M_{d}\sqrt{d}\sigma+c}}$
in terms of $\mathcal{N}$:

$\displaystyle
M^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}}=\exp\left(-\frac{c}{4M_{d}\sqrt{d}\sigma+c}\log(M)\right)$
$\displaystyle\leq$
$\displaystyle\exp\left(-\frac{c}{4M_{d}\sqrt{d}\sigma+c}\left(\frac{\log(\mathcal{N})}{C_{\sigma,s,d}}\right)^{\frac{1}{d+1}}\right)$
$\displaystyle=$ $\displaystyle B^{(\log(\mathcal{N}))^{\frac{1}{d+1}}},$

where
$B=\exp\left(-\frac{c}{4M_{d}\sqrt{d}\sigma+c}\left(\frac{1}{C_{\sigma,s,d}}\right)^{\frac{1}{d+1}}\right)\in(0,1)$.

Note that $\mathcal{N}$ must be at least $M$, implying that
$\log(\mathcal{N})\geq\log(M)$. Using this fact, we get

$\displaystyle(\log(M))^{\max\\{0,2d-s\beta\\}}M^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}}$
$\displaystyle\leq$
$\displaystyle(\log(\mathcal{N}))^{\max\\{0,2d-s\beta\\}}B^{(\log(\mathcal{N}))^{\frac{1}{d+1}}}$
$\displaystyle=$
$\displaystyle(\log(\mathcal{N}))^{\frac{1}{d+1}\max\\{0,2d-s\beta\\}(d+1)}B^{(\log(\mathcal{N}))^{\frac{1}{d+1}}}$
$\displaystyle\stackrel{{\scriptstyle\text{let
$u=(\log(\mathcal{N}))^{\frac{1}{d+1}}$}}}{{=}}$ $\displaystyle
u^{\max\\{0,2d-s\beta\\}(d+1)}B^{u}.$

As $u=(\log(\mathcal{N}))^{\frac{1}{d+1}}$ increases,
$u^{\max\\{0,2d-s\beta\\}(d+1)}$ converges to infinity polynomially w.r.t.
$u$, while $B^{u}$ converges to $0$ exponentially w.r.t. $u$. Thus, the term
$(\log(M))^{\max\\{0,2d-s\beta\\}}M^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}}$
converges to $0$.

Since $u^{\max\\{0,2d-s\beta\\}(d+1)}B^{u}=o(u^{-\ell})$ for any
$\ell\in\mathbb{N}$, we have

$\displaystyle(\log(M))^{\max\\{0,2d-s\beta\\}}M^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}}=o\left((\log(\mathcal{N}))^{-\frac{\ell}{d+1}}\right)=o\left((\log(\mathcal{N}))^{-k}\right),\qquad\forall
k>0,$

because for any $k>0$, there always exists some $\ell\in\mathbb{N}$ such that
$\frac{\ell}{d+1}\geq k$.

Since
$(\log(M))^{\max\\{0,2d-s\beta\\}}\left(\frac{1}{M}\right)^{\frac{c}{4M_{d}\sqrt{d}\sigma+c}}=o\left((\log(\mathcal{N}))^{-k}\right)$
for any $k>0$, we can see that the above theorem gives a convergence rate
w.r.t. $\mathcal{N}$ faster than the logarithmic rate.

At the same time, we can show that the convergence rate we obtained is slower
than the polynomial rate (i.e., a rate of the form
$\mathcal{O}(\mathcal{N}^{-\varepsilon})$ for $\varepsilon>0$). From (A.1) and
(A.2), we can give
$(\log(M))^{\max\\{0,2d-s\beta\\}}\left(\frac{1}{M}\right)^{\frac{c}{4M_{d}\sqrt{d}\sigma+c}}$
a lower bound as follows:

$\displaystyle(\log(M))^{\max\\{0,2d-s\beta\\}}M^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}}$
$\displaystyle\overset{(\ref{logM})}{\geq}$
$\displaystyle\left(\frac{\log(\mathcal{N})}{C_{\sigma,s,d}}\right)^{\frac{1}{d+1}{\max\\{0,2d-s\beta\\}}}M^{(m+1)^{d}\left(-\frac{c}{4M_{d}\sqrt{d}\sigma+c}\cdot\frac{1}{(m+1)^{d}}\right)}$
$\displaystyle\geq$
$\displaystyle\left(\frac{\log(\mathcal{N})}{C_{\sigma,s,d}}\right)^{\frac{1}{d+1}{\max\\{0,2d-s\beta\\}}}\mathcal{N}^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}\cdot\frac{1}{(m+1)^{d}}}$
$\displaystyle\overset{(\ref{logN})}{\geq}$
$\displaystyle\left(\frac{\log(\mathcal{N})}{C_{\sigma,s,d}}\right)^{\frac{1}{d+1}{\max\\{0,2d-s\beta\\}}}\mathcal{N}^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}\cdot\frac{\log(M)}{\log(\mathcal{N})}}$
$\displaystyle\overset{(\ref{logM})}{\geq}$
$\displaystyle\left(\frac{\log(\mathcal{N})}{C_{\sigma,s,d}}\right)^{\frac{1}{d+1}{\max\\{0,2d-s\beta\\}}}\mathcal{N}^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}\cdot(\log(\mathcal{N}))^{-d/(d+1)}\cdot(C_{\sigma,s,d})^{-1/(d+1)}},$

where $C_{\sigma,s,d}\leq\left(\frac{1}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}+2\right)^{d}$. Thus, as $\mathcal{N}$ grows,
$(\log(M))^{\max\\{0,2d-s\beta\\}}\left(\frac{1}{M}\right)^{\frac{c}{4M_{d}\sqrt{d}\sigma+c}}$
always decays slower than $\mathcal{N}^{-\varepsilon}$ for some
$\varepsilon>0$. The proof is complete. ∎

##  Appendix B Proof of Supporting Lemmas

Here, we present the proofs of supporting lemmas. We begin with the proof of
Lemma 3.2.

###### Proof of Lemma 3.2.

The proof is relatively straightforward. Suppose $K$ is a translation-
invariant kernel. For any $c,u\in\mathbb{R}^{d}$,

$\displaystyle\sum_{k,j=1}^{d}c_{k}c_{j}K(u_{k},u_{j})=\sum_{k,j=1}^{d}c_{k}c_{j}\phi(u_{k}-u_{j})$
$\displaystyle\overset{(\ref{eq:inversefourier})}{=}$
$\displaystyle\sum_{k,j=1}^{d}c_{k}c_{j}\int_{\mathbb{R}^{d}}\widehat{\phi}(\xi)e^{2\pi
i\xi\cdot(u_{k}-u_{j})}d\xi$ $\displaystyle=$
$\displaystyle\int_{\mathbb{R}^{d}}\widehat{\phi}(\xi)\sum_{k=1}^{d}c_{k}e^{2\pi
i\xi\cdot u_{k}}\sum_{j=1}^{d}c_{j}e^{-2\pi i\xi\cdot u_{j}}d\xi$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}^{d}}\widehat{\phi}(\xi)\left|\sum_{k=1}^{d}c_{k}e^{2\pi
i\xi\cdot u_{k}}\right|^{2}d\xi.$

Thus, if $\widehat{\phi}$ is non-negative, then
$\sum_{k,j=1}^{d}c_{k}c_{j}K(u_{k},u_{j})\geq 0$ and thus $K$ is positive
semi-definite. ∎

Next, let us present the proof of Lemma 6.2.

###### Proof of Lemma 6.2.

With $\\{\psi_{i}\\}_{i=1}^{N}$ defined in (6.4), for $x\in\mathcal{X}$, by
the reproducing property and the Cauchy–Schwarz inequality, we have

$\displaystyle|f(x)-Pf(x)|=\left|f(x)-\sum_{i=1}^{N}f(t_{i})\psi_{i}(x)\right|$
$\displaystyle=$ $\displaystyle\left|\langle
f,K_{x}\rangle_{\mathcal{H}_{K}}-\sum_{i=1}^{N}\langle
f,K_{t_{i}}\rangle_{\mathcal{H}_{K}}\psi_{i}(x)\right|$ $\displaystyle=$
$\displaystyle\left|\langle
f,K_{x}-\sum_{i=1}^{N}\psi_{i}(x)K_{t_{i}}\rangle_{\mathcal{H}_{K}}\right|$
$\displaystyle\leq$
$\displaystyle\|f\|_{\mathcal{H}_{K}}\left\|K_{x}-\sum_{i=1}^{N}\psi_{i}(x)K_{t_{i}}\right\|_{\mathcal{H}_{K}}.$

For $c\in\mathbb{R}^{N}$, let the function $H$ be

$H(c):=\left\|K_{x}-\sum_{i=1}^{N}c_{i}K_{t_{i}}\right\|^{2}_{\mathcal{H}_{K}}=K(x,x)-2\sum_{i=1}^{N}c_{i}K(t_{i},x)+\sum_{i,j=1}^{N}c_{i}c_{j}K(t_{i},t_{j}).$

Then

$\displaystyle\frac{\partial H}{\partial c}=0\iff
2K[\bar{t}]c=2\begin{bmatrix}K(x,t_{1})\\\ \vdots\\\
K(x,t_{N})\end{bmatrix}\iff c=(K[\bar{t}])^{-1}\begin{bmatrix}K(x,t_{1})\\\
\vdots\\\ K(x,t_{N})\end{bmatrix}=\begin{bmatrix}\psi_{1}(x)\\\ \vdots\\\
\psi_{N}(x)\end{bmatrix}.$

Thus,
$\|K_{x}-\sum_{i=1}^{N}\psi_{i}(x)K_{t_{i}}\|_{\mathcal{H}_{K}}=\min_{c\in\mathbb{R}^{N}}\sqrt{H(c)}\leq\max_{x\in\mathcal{X}}\min_{c\in\mathbb{R}^{N}}\sqrt{H(c)}=\epsilon_{K}(\bar{t})$.
This is valid for every $x\in\mathcal{X}$. The proof is complete. ∎

Next, we give the proof of Lemma 6.3.

###### Proof of Lemma 6.3.

Recall from Lemma 6.2:

$|f(x)-Pf(x)|\leq\|f\|_{\mathcal{H}_{K}}\cdot\min_{c\in\mathbb{R}^{N}}\left\|K_{x}-\sum_{i=1}^{N}c_{i}K_{t_{i}}\right\|_{\mathcal{H}_{K}},\qquad
f\in\mathcal{H}_{K},x\in\mathcal{X},\bar{t}\subset\mathcal{X}.$

From (6.7),
$h_{\bar{t}}=\sup_{x\in\mathcal{X}}d_{\mathcal{X}}(x,t_{j_{x}})\leq\frac{\sqrt{d}}{m}$,
where $t_{j_{x}}$ is the closest point in $\bar{t}$ to any $x\in\mathcal{X}$.
Choose $c$ such that

$c_{i}=\begin{cases}1,&\text{if $i=j_{x}$ },\\\
0,&\text{otherwise}.\end{cases}$ (B.1)

Then, by the $\alpha$-Hölder regularity of $K$ and symmetry, we get

$\displaystyle
K(x,x)-2\sum_{i=1}^{N}c_{i}K(x,t_{i})+\sum_{i=1}^{N}\sum_{j=1}^{N}c_{i}c_{j}K(t_{i},t_{j})$
$\displaystyle=$ $\displaystyle K(x,x)-2K(x,t_{j_{x}})+K(t_{j_{x}},t_{j_{x}})$
$\displaystyle\leq$ $\displaystyle
2C_{K}(d_{\mathcal{X}}(x,t_{j_{x}}))^{\alpha}\leq
2C_{K}\left(\frac{\sqrt{d}}{m}\right)^{\alpha},$

and thus $\epsilon_{K}(\bar{t})\leq
2C_{K}\left(\frac{\sqrt{d}}{m}\right)^{\alpha}$. The Lemma follows after Lemma
6.2. ∎

Next, we present the proof of Lemma 6.4.

###### Proof of Lemma 6.4.

For any $c,\tilde{c}\in\mathbb{R}^{N}$, we have

$|G(c)-G(\tilde{c})|=\left|F\left(\sum_{i=1}^{N}c_{i}\psi_{i}\right)-F\left(\sum_{i=1}^{N}\tilde{c_{i}}\psi_{i}\right)\right|\leq
C_{F}\left\|\sum_{i=1}^{N}(c_{i}-\tilde{c_{i}})\psi_{i}\right\|^{s}_{\infty}.$
(B.2)

We will proceed to bound $\|\sum_{i=1}^{N}a_{i}\psi_{i}\|_{\infty}$ for any
$a\in\mathbb{R}^{N}$. For $x\in\mathcal{X}$, let $t_{j_{x}}$ be the closest
point in $\bar{t}$ to $x$. Observe that
$\left(\sum_{i=1}^{N}a_{i}\psi_{i}\right)(t_{j_{x}})=a_{j_{x}}$ and thus

$\left|\sum_{i=1}^{N}a_{i}\psi_{i}(x)\right|=\left|\sum_{i=1}^{N}a_{i}\psi_{i}(t_{j_{x}})+\sum_{i=1}^{N}a_{i}(\psi_{i}(x)-\psi_{i}(t_{j_{x}}))\right|=\left|a_{j_{x}}+\sum_{i=1}^{N}a_{i}(\psi_{i}(x)-\psi_{i}(t_{j_{x}}))\right|.$

Next, we have

$\displaystyle\left|\sum_{i=1}^{N}a_{i}(\psi_{i}(x)-\psi_{i}(t_{j_{x}}))\right|$
$\displaystyle=$
$\displaystyle\left|a^{T}(K[\bar{t}])^{-1}\begin{bmatrix}K(t_{1},x)-K(t_{1},t_{j_{x}})\\\
\vdots\\\ K(t_{N},x)-K(t_{N},t_{j_{x}})\end{bmatrix}\right|$ $\displaystyle=$
$\displaystyle\left|((K[\bar{t}])^{-1}a)^{T}\begin{bmatrix}K(t_{1},x)-K(t_{1},t_{j_{x}})\\\
\vdots\\\ K(t_{N},x)-K(t_{N},t_{j_{x}})\end{bmatrix}\right|$
$\displaystyle\leq$
$\displaystyle\|(K[\bar{t}])^{-1}\|_{op}\|a\|\sqrt{\sum_{i=1}^{N}|K(t_{i},x)-K(t_{i},t_{j_{x}})|^{2}}.$

Since $K$ is $\alpha$-Hölder continuous, we know that
$|K(t_{i},x)-K(t_{i},t_{j_{x}})|\leq
C_{K}\left(d_{\mathcal{X}}(x,t_{j_{x}})\right)^{\alpha}.$ Hence, we know

$\displaystyle\left|\sum_{i=1}^{N}a_{i}(\psi_{i}(x)-\psi_{i}(t_{j_{x}}))\right|$
$\displaystyle\leq$
$\displaystyle\|(K[\bar{t}])^{-1}\|_{op}\|a\|\sqrt{N}C_{K}\left(\sup_{x\in\mathcal{X}}d_{\mathcal{X}}(x,t_{j_{x}})\right)^{\alpha}.$

Thus, we have

$\displaystyle|G(c)-G(\tilde{c})|$ $\displaystyle\leq$ $\displaystyle
C_{F}\left\\{\left|c_{j_{x}}-\tilde{c}_{j_{x}}\right|+\|(K[\bar{t}])^{-1}\|_{op}\|c-\tilde{c}\|\sqrt{N}C_{K}\left(\sup_{x\in\mathcal{X}}d_{\mathcal{X}}(x,t_{j_{x}})\right)^{\alpha}\right\\}^{s}$
$\displaystyle\leq$ $\displaystyle
C_{F}\|c-\tilde{c}\|^{s}\left\\{1+\|(K[\bar{t}])^{-1}\|_{op}\sqrt{N}C_{K}\left(\sup_{x\in\mathcal{X}}d_{\mathcal{X}}(x,t_{j_{x}})\right)^{\alpha}\right\\}^{s}.$

The proof is complete. ∎

Next, we give the proof of Lemma 6.5.

###### Proof of Lemma 6.5.

From Lemma 3.2, we know that the Fourier transform of a translation-invariant
and reproducing kernel $K$ is real-valued and non-negative. We also know that,
from (6.14) and (6.15), to estimate $\|(K[\bar{t}])^{-1}\|_{op}$ for a
translation-invariant $K$ over $\mathbb{R}^{d}$, it suffices to estimate the
smallest eigenvalue of $K[\bar{t}]$:
$\lambda_{N}=\sum_{k=1}^{N}\sum_{\ell=1}^{N}v_{k}v_{\ell}K(t_{k},t_{\ell})$,
where $v\in\mathbb{R}^{N}$ is an eigenvector of $K[\bar{t}]$ of norm $1$.

For any $\xi\in\mathbb{R}^{d}$, we have

$\displaystyle\lambda_{N}=\sum_{k=1}^{N}\sum_{\ell=1}^{N}v_{k}v_{\ell}K(t_{k},t_{\ell})$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}^{d}}\widehat{\phi}(\xi)\left|\sum_{k=1}^{N}v_{k}e^{2\pi
i\xi\cdot t_{k}}\right|^{2}d\xi\ \geq\
\int_{[-\frac{m}{2},\frac{m}{2}]^{d}}\widehat{\phi}(\xi)\left|\sum_{k=1}^{N}v_{k}e^{2\pi
i\xi\cdot t_{k}}\right|^{2}d\xi.$

Let
$\Gamma_{m}=\min_{\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}}\widehat{\phi}(\xi)$.
Then, we have

$\displaystyle\lambda_{N}\geq\Gamma_{m}\int_{[-\frac{m}{2},\frac{m}{2}]^{d}}\left|\sum_{k=1}^{N}v_{k}e^{2\pi
i\xi\cdot t_{k}}\right|^{2}d\xi$ $\displaystyle\geq$
$\displaystyle\Gamma_{m}\int_{[-\frac{1}{2},\frac{1}{2}]^{d}}\left|\sum_{k\in\\{0,1,\ldots,m\\}^{d}}v_{k}e^{2\pi
i\xi\cdot k}\right|^{2}md\xi$ (B.3) $\displaystyle=$ $\displaystyle
m\Gamma_{m}\|v\|^{2}$ $\displaystyle=$ $\displaystyle m\Gamma_{m}.$

Here, we have used the fact that $\\{e^{2\pi i\xi\cdot
k}\\}_{k\in\mathbb{Z}^{d}}$ is an orthonormal basis of
$L_{2}[(-1/2,1/2)^{d}]$. ∎

Next, we present the proof of Lemma 6.6.

###### Proof of Lemma 6.6.

We know that the Sobolev space $W^{r}_{2}(\mathcal{X})$ with $r>d/2$ is an
RKHS induced by a translation-invariant kernel, whose Fourier transform is
$\widehat{\phi}(\xi)=(1+|\xi|^{2})^{-r}$, for $\xi\in\mathbb{R}^{d}$. From
(6.17), we know
$\Gamma_{m}=\min_{\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}}\widehat{\phi}(\xi)\geq\left(1+\frac{dm^{2}}{4}\right)^{-r}$
for this function $\widehat{\phi}$. According to Lemma 6.5,

$\|(K[\bar{t}])^{-1}\|_{op}\leq\frac{1}{m\Gamma_{m}}\leq\frac{1}{m}\left(1+\frac{dm^{2}}{4}\right)^{r}.$

For the Sobolev Space $W^{r}_{2}(\mathcal{X})$ and $\mathcal{X}=[0,1]^{d}$, it
follows from Lemma 6.4 the associated $G$ function
$G(c)=F\left(\sum_{i=1}^{N}c_{i}\psi_{i}\right)$ is $s$-Hölder continuous
$s\in(0,1]$ with constant

$\displaystyle
C_{F}\left(1+\|(K[\bar{t}])^{-1}\|_{op}\sqrt{N}C_{K}\left(h_{\bar{t}}\right)^{\alpha}\right)^{s}$
$\displaystyle\leq$ $\displaystyle
C_{F}\left(1+\frac{1}{m}\left(1+\frac{dm^{2}}{4}\right)^{r}(m+1)^{\frac{d}{2}}C_{r,d}\left(\frac{\sqrt{d}}{m}\right)^{\min\\{1,r-d/2\\}}\right)^{s}$
$\displaystyle\leq$ $\displaystyle
C_{F}\left(1+\frac{1}{m}\left(\frac{d^{r}m^{2r}}{2^{r}}\right)(m+1)^{\frac{d}{2}}C_{r,d}\left(\frac{\sqrt{d}}{m}\right)\right)^{s}$
$\displaystyle\leq$ $\displaystyle
C_{F}\left(1+C_{r,d}\frac{2^{\frac{d}{2}}}{2^{r}}m^{2r+\frac{d}{2}-2}d^{r+\frac{1}{2}}\right)^{s}.$

Thus, $G$ is $s$-Hölder continuous with constant

$C_{G}=C_{F}C_{r,d,s}m^{2rs+\frac{ds}{2}-2s}d^{rs+\frac{s}{2}},$

where $C_{r,d,s}$ is a positive constant depends only on $r,d$, and $s$. ∎

Next, we give the proof of Lemma 6.7.

###### Proof of Lemma 6.7.

For an inverse multiquadric kernel $K$, a lower bound of $\lambda_{N}$ (i.e.,
the smallest eigenvalue of the Gram matrix $K[\bar{t}]$) is given in
[Fasshauer, 2005, Lemma 8.2] in terms of the separation distance
$q_{\bar{t}}=\frac{1}{2}\min_{u\neq v\in\bar{t}}|u-v|$ as

$\lambda_{N}\geq
C_{\sigma,\beta,d}q_{\bar{t}}^{-\beta+\frac{1-d}{2}}e^{-\frac{2\sigma
M_{d}}{q_{\bar{t}}}},$ (B.4)

where $M_{d}=12\left(\frac{\pi\Gamma^{2}(\frac{d+2}{2})}{9}\right)\leq 6.38d$,
and $C_{\sigma,\beta,d}$ is another explicitly known constant depending on
$\sigma,\beta,$ and $d$.

With our choice of $\bar{t}=\\{0,\frac{1}{m},\ldots,\frac{m}{m}\\}^{d}$, we
have $q_{\bar{t}}=\frac{1}{2m}.$ Then, we know

$\lambda_{N}\geq C_{\sigma,\beta,d}(2m)^{\beta+\frac{d-1}{2}}e^{-4\sigma
M_{d}m},$

and it follows from Lemma 6.5 that

$\|(K[\bar{t}])^{-1}\|_{op}=\frac{1}{\lambda_{N}}\leq\frac{1}{mC_{\sigma,\beta,d}}(2m)^{-\beta-\frac{d-1}{2}}e^{4\sigma
M_{d}m}.$ (B.5)

For the inverse multiquadric kernel $K$, it follows from Lemma 6.4 that the
associated $G$ function $G(c)=F\left(\sum_{i=1}^{N}c_{i}\psi_{i}\right)$ is
$s$-Hölder continuous with constant

$C_{F}\left(1+2d\beta\sigma^{-2\beta-2}\frac{1}{C_{\sigma,\beta,d}}(2m)^{-\beta-\frac{d-1}{2}}e^{4\sigma
M_{d}m}\frac{(m+1)^{\frac{d}{2}}}{m}\right)^{s}.$

Thus, $G$ is $s$-Hölder continuous with constant
$C_{G}=C_{F}\left(1+\frac{4d\beta\sigma^{-2\beta-2}}{C_{\sigma,\beta,d}}(2m)^{-\beta-\frac{1}{2}}e^{4\sigma
M_{d}m}\right)^{s}$. The proof is complete. ∎

Next, we give the proof of Lemma 6.8.

###### Proof of Lemma 6.8.

The Fourier transform of Gaussian kernel is
$\widehat{\phi}(\xi)=(2\sigma^{2}\pi)^{\frac{d}{2}}e^{-2\sigma^{2}\pi^{2}|\xi|^{2}}>0$
for all $\xi\in\mathbb{R}^{d}$. For $\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}$, we
have $|\xi|^{2}\leq d\left(\frac{m}{2}\right)^{2}$. Then, we have

$\Gamma_{m}=\min_{\xi\in[-\frac{m}{2},\frac{m}{2}]^{d}}\widehat{\phi}(\xi)\geq(2\sigma^{2}\pi)^{\frac{d}{2}}e^{-2\sigma^{2}\pi^{2}d\left(\frac{m}{2}\right)^{2}}=(2\sigma^{2}\pi)^{\frac{d}{2}}e^{-\sigma^{2}\pi^{2}dm^{2}/2}.$

It follows from Lemma 6.5 that

$\|(K[\bar{t}])^{-1}\|_{op}\leq\frac{1}{m}(2\sigma^{2}\pi)^{-\frac{d}{2}}e^{\sigma^{2}\pi^{2}dm^{2}/2}.$

For Gaussian kernel $K$, it follows from Lemma 6.4 that the associated $G$
function $G(c)=F\left(\sum_{i=1}^{N}c_{i}\psi_{i}\right)$ is $s$-Hölder
continuous for $0<s\leq 1$ with constant

$C_{F}\left(1+(2\sigma^{2}\pi)^{-\frac{d}{2}}e^{\sigma^{2}\pi^{2}dm^{2}/2}\frac{d(m+1)^{\frac{d}{2}}}{m\sigma^{2}}\right)^{s}.$

Because
$\frac{(m+1)^{\frac{d}{2}}}{m}=\mathcal{O}(e^{\sigma^{2}\pi^{2}dm^{2}/2})$, we
know $G$ is $s$-Hölder continuous with constant

$C_{G}=C_{F}C_{\sigma,d,s}e^{\sigma^{2}\pi^{2}dsm^{2}},$ (B.6)

where $C_{\sigma,d,s}$ is a constant depending only on $\sigma,d,$ and $s$. ∎

##  Appendix C Proof of Main Results

We are in a position to prove our main results. We first present the proof of
Theorem 3.4 below.

###### Proof of Theorem 3.4.

Recall that from the Sobolev Embedding Theorem, we know that the reproducing
kernel $K$ in $W^{r}_{2}(\mathcal{X})$ is $\alpha$-Hölder continuous with
$\alpha=r-d/2$ and the constant $C_{K}=C_{r,d}>0$ depends only on $r$ and $d$.
In our analysis, we require $\alpha\leq 1$, which is why we assume $\alpha\leq
1$ throughout the paper. Thus, we assume that $\alpha=\min\\{1,r-d/2\\}$.

We consider the error decomposition in (6.1), that is,

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|F(Pf)-\widehat{G}(f(\bar{t}))|$
$\displaystyle=\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|G(f(\bar{t}))-\widehat{G}(f(\bar{t}))|.$

Here, we have used
$G(f(\bar{t}))=F\left(\sum_{i=1}^{N}f(t_{i})\psi_{i}\right)=F(Pf)$, where the
function $G$ is defined previously in equation (6.13).

With Corollary 6.12, we get

$\displaystyle\sup_{f\in\mathcal{K}}\left|G(f(\bar{t}))-\widehat{G}(f(\bar{t}))\right|\leq\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{(m+1)^{d}}}$
$\displaystyle\leq$
$\displaystyle\frac{C_{F}C_{r,d,s}}{M}m^{2rs+\frac{ds}{2}+2d-2s}d^{rs+\frac{s}{2}}$
$\displaystyle\leq$
$\displaystyle\frac{C_{F}C_{r,d,s}}{M}m^{s(2r+d-2)}d^{s(r+\frac{1}{2})}.$

According to [Schaback, 1997], when $r>d/2$ and $r-d/2\notin\mathbb{N}$, an
upper bound of the power function is derived by Sobolev Embedding Theorem as

$\epsilon_{K}(\bar{t})\leq C^{\prime}_{r,d}d^{r-\frac{d}{2}}m^{d-2r},$

where $C^{\prime}_{r,d}$ is some positive constant depends on $r$ and $d$.
Thus, we have

$\|f-Pf\|_{\infty}\leq\epsilon_{K}(\bar{t})\|f\|_{\mathcal{H}_{K}}\leq
C^{\prime}_{r,d}d^{r-\frac{d}{2}}m^{d-2r}$

and

$\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|\leq C_{F}\|f-Pf\|^{s}_{\infty}\leq
C_{F}(C^{\prime}_{r,d})^{s}d^{s(r-\frac{d}{2})}m^{s(d-2r)}.$

Now combining the two error bounds together, we get

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq$ $\displaystyle
C_{F}(C^{\prime}_{r,d})^{s}d^{s(r-\frac{d}{2})}m^{s(d-2r)}+\frac{C_{F}C_{r,d,s}}{M}m^{s(2r+d-2)}d^{s(r+\frac{1}{2})}$
$\displaystyle\leq$ $\displaystyle
C_{F}\max\\{(C^{\prime}_{r,d})^{s},C_{r,d,s}\\}\left(d^{s(r-\frac{d}{2})}m^{s(d-2r)}+\frac{1}{M}m^{s(2r+d-2)}d^{s(r+\frac{1}{2})}\right).$

We take $m=\left\lceil M^{\frac{1}{s(4r-2)}}\right\rceil\in\mathbb{N}$ such
that $m^{s(d-2r)}\approx\frac{1}{M}m^{s(2r+d-2)}$. Then,

$m^{s(d-2r)}\leq M^{\frac{s(d-2r)}{s(4r-2)}}=M^{-\frac{2r-d}{2(2r-1)}}.$

We know

$\frac{1}{M}m^{s(2r+d-2)}\leq 2^{s(2r+d-2)}M^{-\frac{2r-d}{2(2r-1)}}.$

We also know that $d^{s(r-\frac{d}{2})}\leq d^{s(r+\frac{1}{2})}$. The proof
is complete. ∎

Next, we present the proof of Theorem 3.6.

###### Proof of Theorem 3.6.

We consider the error decomposition in (6.1), that is,

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|F(Pf)-\widehat{G}(f(\bar{t}))|$
$\displaystyle=\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|G(f(\bar{t}))-\widehat{G}(f(\bar{t}))|.$

Here, we have used
$G(f(\bar{t}))=F\left(\sum_{i=1}^{N}f(t_{i})\psi_{i}\right)=F(Pf)$, where the
function $G$ is defined previously in equation (6.13).

With the assumption that $F$ is $s$-Hölder continuous with constant $C_{F}\geq
0$ and the upper bound of $\|f-Pf\|_{\infty}$ given in (6.9), we have

$\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|\leq C_{F}\|f-Pf\|^{s}_{\infty}\leq
C_{F}e^{\frac{-csm}{\sqrt{d}}}\|f\|^{s}_{\mathcal{H}_{K}}\leq
C_{F}e^{\frac{-csm}{\sqrt{d}}}.$

From Corollary 6.13, we get

$\displaystyle\sup_{f\in\mathcal{K}}\left|G(f(\bar{t}))-\widehat{G}(f(\bar{t}))\right|\leq\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{(m+1)^{d}}}\leq\frac{(2m)^{2d-s\beta}}{M}C_{F}C_{\sigma,\beta,d,s}e^{4M_{d}\sigma
sm}.$

Then, combining the above two error bounds together, we get

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq$ $\displaystyle
C_{F}e^{\frac{-csm}{\sqrt{d}}}+\frac{(2m)^{2d-s\beta}}{M}C_{F}C_{\sigma,\beta,d,s}e^{4M_{d}\sigma
sm}$ $\displaystyle\leq$ $\displaystyle
C_{F}C_{\sigma,\beta,d,s}\left(e^{\frac{-csm}{\sqrt{d}}}+\frac{(2m)^{2d-s\beta}}{M}e^{4M_{d}\sigma
sm}\right).$

We can see that there is a trade-off of $m$ between
$e^{\frac{-csm}{\sqrt{d}}}$ and $\frac{(2m)^{2d-s\beta}}{M}e^{4M_{d}\sigma
sm}$. To balance the trade-off, we choose $m\in\mathbb{N}$ such that
$e^{\frac{-csm}{\sqrt{d}}}\approx\frac{1}{M}e^{4M_{d}\sigma sm}$. We let

$\displaystyle e^{\frac{-csm}{\sqrt{d}}}=\frac{1}{M}e^{4M_{d}\sigma sm}$
$\displaystyle\iff$ $\displaystyle\log(M)=4M_{d}\sigma
sm+\frac{csm}{\sqrt{d}}=m\left(4M_{d}\sigma s+\frac{cs}{\sqrt{d}}\right)$
$\displaystyle\iff$ $\displaystyle m=\frac{\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}.$

We take $m=\left\lceil\frac{\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}\right\rceil\in\mathbb{N}$. Then,

$\displaystyle
e^{\frac{-csm}{\sqrt{d}}}=\exp\left(-\frac{cs}{\sqrt{d}}\left\lceil\frac{\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}\right\rceil\right)\leq\exp\left(-\frac{c\log(M)}{4M_{d}\sqrt{d}\sigma+c}\right)=M^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}}.$

We know $\frac{1}{M}e^{4M_{d}\sigma sm}=e^{\frac{-csm}{\sqrt{d}}}\leq
M^{-\frac{c}{4M_{d}\sqrt{d}\sigma+c}}$. Note that $2d-s\beta$ may be negative.
We have

$\displaystyle(2m)^{2d-s\beta}\leq(2m)^{\max\\{0,2d-s\beta\\}}\leq\left(\frac{2\log(M)}{4M_{d}\sigma
s+\frac{cs}{\sqrt{d}}}+2\right)^{\max\\{0,2d-s\beta\\}}\leq
C_{\sigma,\beta,d,s}(\log(M))^{\max\\{0,2d-s\beta\\}}.$

The proof is complete. ∎

Next, We combine the error bound in (6.11) and Corollary 6.14 together to
prove Theorem 3.9.

###### Proof of Theorem 3.9.

Again, we use the error decomposition:

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|F(Pf)-\widehat{G}(f(\bar{t}))|$
$\displaystyle=\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|G(f(\bar{t}))-\widehat{G}(f(\bar{t}))|.$

Recall that we assume $F$ is $s$-Hölder continuous with constant $C_{F}\geq
0$. Suppose $m>\sqrt{d}$, it follows from (6.11) that

$\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|\leq C_{F}\|f-Pf\|^{s}_{\infty}\leq
C_{F}e^{\frac{-csm\log\left(\frac{m}{\sqrt{d}}\right)}{\sqrt{d}}}\|f\|^{s}_{\mathcal{H}_{K}}\leq
C_{F}e^{\frac{-csm\log\left(\frac{m}{\sqrt{d}}\right)}{\sqrt{d}}}.$

With Corollary 6.14, we have

$\displaystyle\sup_{f\in\mathcal{K}}\left|G(f(\bar{t}))-\widehat{G}(f(\bar{t}))\right|\leq\left\|\widehat{G}-G\right\|_{L^{\infty}[-1,1]^{(m+1)^{d}}}$
$\displaystyle\leq$
$\displaystyle\frac{7(m+1)^{2d}}{M}C_{F}C_{\sigma,d,s}e^{\sigma^{2}\pi^{2}dsm^{2}}$
$\displaystyle\leq$
$\displaystyle\frac{7(2m)^{2d}}{M}C_{F}C_{\sigma,d,s}e^{\sigma^{2}\pi^{2}dsm^{2}}.$

Combining the above two error bounds, we get

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq$ $\displaystyle
C_{F}e^{\frac{-csm\log\left(\frac{m}{\sqrt{d}}\right)}{\sqrt{d}}}+\frac{7(2m)^{2d}}{M}C_{F}C_{\sigma,d,s}e^{\sigma^{2}\pi^{2}dsm^{2}}$
$\displaystyle\leq$ $\displaystyle
C_{F}C_{\sigma,d,s}\left(e^{\frac{-csm\log\left(\frac{m}{\sqrt{d}}\right)}{\sqrt{d}}}+\frac{7(2m)^{2d}}{M}e^{\sigma^{2}\pi^{2}dsm^{2}}\right).$

We let

$\displaystyle
e^{\frac{-csm\log\left(\frac{m}{\sqrt{d}}\right)}{\sqrt{d}}}=\frac{1}{M}e^{\sigma^{2}\pi^{2}dsm^{2}}$
$\displaystyle\iff$
$\displaystyle\sigma^{2}\pi^{2}dsm^{2}+\frac{csm}{\sqrt{d}}\log\left(\frac{m}{\sqrt{d}}\right)=\log(M).$

We solve $m$ for the following quadratic equation:

$\displaystyle\sigma^{2}\pi^{2}dsm^{2}+\frac{cs}{\sqrt{d}}m-\log(M)=0$

and we take the positive solution of $m$ given by

$\displaystyle m$ $\displaystyle=$
$\displaystyle\frac{-\frac{cs}{\sqrt{d}}+\sqrt{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)}}{2\sigma^{2}\pi^{2}ds}$
$\displaystyle=$
$\displaystyle\frac{-{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)+\frac{c^{2}s^{2}}{d}}}{2\sigma^{2}\pi^{2}ds\left(\frac{cs}{\sqrt{d}}+\sqrt{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)}\right)}$
$\displaystyle=$
$\displaystyle\frac{2\log(M)}{\frac{cs}{\sqrt{d}}+\sqrt{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)}}.$

We choose

$m=\left\lceil\frac{2\log(M)}{\frac{cs}{\sqrt{d}}+\sqrt{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)}}\right\rceil\in\mathbb{N}.$
(C.1)

We can easily check that the chosen $m$ is greater than $\sqrt{d}$. With our
choice of $m$ in (C.1), we know that

$\displaystyle m$ $\displaystyle\geq$
$\displaystyle\frac{2\log(M)}{\frac{cs}{\sqrt{d}}+\sqrt{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)}}$
(C.2) $\displaystyle\geq$
$\displaystyle\frac{2\log(M)}{\frac{cs}{\sqrt{d}}+\frac{cs}{\sqrt{d}}+2\sigma\pi\sqrt{ds\log(M)}}$
$\displaystyle\geq$
$\displaystyle\frac{\log(M)}{\frac{cs}{\sqrt{d}}+\sigma\pi\sqrt{ds\log(M)}}$

and

$\displaystyle m$ $\displaystyle\leq$
$\displaystyle\frac{-\frac{cs}{\sqrt{d}}+\sqrt{\frac{c^{2}s^{2}}{d}+4\sigma^{2}\pi^{2}ds\log(M)}}{2\sigma^{2}\pi^{2}ds}$
(C.3) $\displaystyle\leq$
$\displaystyle\frac{2\sigma\pi\sqrt{ds\log(M)}}{2\sigma^{2}\pi^{2}ds}$
$\displaystyle\leq$
$\displaystyle\frac{1}{\sigma\pi}\sqrt{\frac{\log(M)}{ds}}.$

Then, from (C.2), we have

$\displaystyle e^{\frac{-csm\log\left(\frac{\sqrt{d}}{m}\right)}{\sqrt{d}}}$
$\displaystyle\leq$
$\displaystyle\exp\left(-\frac{cs}{\sqrt{d}}\frac{\log(M)}{\frac{cs}{\sqrt{d}}+\sigma\pi\sqrt{ds\log(M)}}\log\left(\frac{\log(M)}{cs+\sigma\pi
d\sqrt{s\log(M)}}\right)\right)$ $\displaystyle\leq$
$\displaystyle\exp\left(-\frac{cs\log(M)}{cs\sqrt{\log(M)}(1+\sigma\pi
d)}\log\left(\frac{\log(M)}{\sqrt{\log(M)}(cs+\sigma\pi
d\sqrt{s})}\right)\right)$ $\displaystyle\leq$
$\displaystyle\exp\left(-\frac{\sqrt{\log(M)}}{1+\sigma\pi
d}\log\left(\frac{\sqrt{\log(M)}}{cs+\sigma\pi d\sqrt{s}}\right)\right)$
$\displaystyle\leq$ $\displaystyle\exp\left(-\frac{\log(\sqrt{M})}{1+\sigma\pi
d}\log\left(\frac{\sqrt{\log(M)}}{cs+\sigma\pi d\sqrt{s}}\right)\right)$
$\displaystyle=$ $\displaystyle\sqrt{M}^{-\frac{1}{1+\sigma\pi
d}\log\left(\frac{\sqrt{\log(M)}}{cs+\sigma\pi d\sqrt{s}}\right)}$
$\displaystyle=$ $\displaystyle M^{-\frac{1}{2(1+\sigma\pi
d)}\log\left(\frac{\sqrt{\log(M)}}{cs+\sigma\pi d\sqrt{s}}\right)}.$

We know that

$\frac{1}{M}e^{\sigma^{2}\pi^{2}dsm^{2}}=e^{\frac{-csm\log\left(\frac{\sqrt{d}}{m}\right)}{\sqrt{d}}}\leq
M^{-\frac{1}{2(1+\sigma\pi d)}\log\left(\frac{\sqrt{\log(M)}}{cs+\sigma\pi
d\sqrt{s}}\right)}.$

It follows from (C.3) that

$\displaystyle(2m)^{2d}\leq\left(\frac{2}{\sigma\pi}\sqrt{\frac{\log(M)}{ds}}\right)^{2d}\leq\left(\frac{2}{\sigma\pi\sqrt{ds}}\right)^{2d}(\log(M))^{d}.$

The proof is complete. ∎

Next, we give the proof of Theorem 4.1.

###### Proof of Theorem 4.1.

The proof is rather straightforward. Here, we consider
$\mathcal{K}:=\\{f\in\mathcal{H}_{K}:\|f\|_{\mathcal{H}_{K}}\leq 1\\}$ with
$\mathcal{H}_{K}$ induced by some Mercer kernel $K$ which is $\alpha$-Hölder
continuous for $\alpha\in(0,1]$ with constant $C_{K}\geq 0$. Note that we do
not require $K$ to be translation-invariant. Again, we consider the error
decomposition

$\displaystyle\sup_{f\in\mathcal{K}}\left|F(f)-\widehat{G}(f(\bar{t}))\right|$
$\displaystyle\leq\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|F(Pf)-\widehat{G}(f(\bar{t}))|$
$\displaystyle=\sup_{f\in\mathcal{K}}|F(f)-F(Pf)|+\sup_{f\in\mathcal{K}}|G(f(\bar{t}))-\widehat{G}(f(\bar{t}))|.$

The upper bound of the first term is derived earlier in (6.6), which is a
consequence of Lemma 6.2. The theorem follows directly after Lemma 6.4 and
Theorem 6.9. ∎

Lastly, we give the proof of Corollary 5.1.

###### Proof of Corollary 5.1.

We first prove $F|_{\mathcal{H}_{K}}$ is Hölder continuous. By the
Cauchy–Schwarz inequality,

$|\langle
X,\beta\rangle_{L^{2}([0,1])}|\leq\|\beta\|_{L^{2}}\|X\|_{L^{2}}\leq\|\beta\|_{L^{2}}\kappa\|X\|_{\mathcal{H}_{K_{0}}}\leq\kappa\|\beta\|_{L^{2}}.$

For $X,U\in\mathcal{K}$, we have

$\displaystyle\left|F|_{\mathcal{H}_{K}}(X)-F|_{\mathcal{H}_{K}}(U)\right|=\left|g\left(\langle
X,\beta\rangle_{L^{2}([0,1])}\right)-g\left(\langle
U,\beta\rangle_{L^{2}([0,1])}\right)\right|$ $\displaystyle\leq$
$\displaystyle\|g\|_{Lip([-\kappa\|\beta\|_{L^{2}},\kappa\|\beta\|_{L^{2}}])}\|\beta\|_{L^{2}}\|X-U\|_{L^{2}}$
$\displaystyle\leq$
$\displaystyle\|g\|_{Lip([-\kappa\|\beta\|_{L^{2}},\kappa\|\beta\|_{L^{2}}])}\|\beta\|_{L^{2}}\kappa\|X-U\|_{\mathcal{H}_{K}}.$

We see that $F|_{\mathcal{H}_{K}}$ is $s$-Hölder continuous with $s=1$ and
constant
$C_{F}=\|g\|_{Lip([-\kappa\|\beta\|_{L^{2}},\kappa\|\beta\|_{L^{2}}])}\|\beta\|_{L^{2}}\kappa$.
Substitute $d=1$, $s=1$, and
$C_{F}=\|g\|_{Lip([-\kappa\|\beta\|_{L^{2}},\kappa\|\beta\|_{L^{2}}])}\|\beta\|_{L^{2}}\kappa$
into Theorem 4.1 and Theorem 3.9, the proof is complete. ∎

## References

  * Aronszajn [1950] N. Aronszajn.  Theory of Reproducing Kernels.  _Transactions of the American Mathematical Society_ , 68(3):337–404, 1950.
  * Balasubramanian et al. [2022] K. Balasubramanian, H.-G. Müller, and B. K. Sriperumbudur.  Unified RKHS methodology and analysis for functional linear and single-index models.  _arXiv preprint arXiv:2206.03975_ , 2022.
  * Bozzini et al. [2015] M. Bozzini, M. Rossini, R. Schaback, and E. Volontè.  Radial kernels via scale derivatives.  _Advances in Computational Mathematics_ , 41:277–291, 2015\.
  * Brezis and Brézis [2011] H. Brezis and H. Brézis.  _Functional Analysis, Sobolev Spaces and Partial Differential Equations_ , volume 2.  Springer, 2011.
  * Chen et al. [2011] D. Chen, P. Hall, and H.-G. Müller.  Single and multiple index functional regression models with nonparametric link.  _The Annals of Statistics_ , 39(3):1720–1747, 2011.
  * Chen and Chen [1995] T. Chen and H. Chen.  Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems.  _IEEE Transactions on Neural Networks_ , 6(4):911–917, 1995.
  * Chiou [2012] J.-M. Chiou.  Dynamical functional prediction and classification, with application to traffic flow prediction.  _The Annals of Applied Statistics_ , pages 1588–1614, 2012.
  * Cho et al. [2014] K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014.
  * Cybenko [1989] G. Cybenko.  Approximation by superpositions of a sigmoidal function.  _Mathematics of control, signals and systems_ , 2(4):303–314, 1989.
  * De Ryck et al. [2021] T. De Ryck, S. Lanthaler, and S. Mishra.  On the approximation of functions by tanh neural networks.  _Neural Networks_ , 143:732–750, 2021.
  * Fasshauer [2005] G. Fasshauer.  Meshfree Methods.  _Handbook of theoretical and computational nanotechnology_ , 27:33–97, 2005.
  * Gin et al. [2021] C. R. Gin, D. E. Shea, S. L. Brunton, and J. N. Kutz.  DeepGreen: deep learning of Green’s functions for nonlinear boundary value problems.  _Scientific reports_ , 11(1):21614, 2021.
  * Greven and Scheipl [2017] S. Greven and F. Scheipl.  A general framework for functional regression modelling.  _Statistical Modelling_ , 17(1-2):1–35, 2017\.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  _Neural Computation_ , 9(8):1735–1780, 1997.
  * Kanagawa et al. [2018] M. Kanagawa, P. Hennig, D. Sejdinovic, and B. K. Sriperumbudur.  Gaussian processes and kernel methods: A review on connections and equivalences.  _arXiv preprint arXiv:1807.02582_ , 2018.
  * Kovachki et al. [2021] N. Kovachki, S. Lanthaler, and S. Mishra.  On universal approximation and error bounds for Fourier Neural Operators.  _The Journal of Machine Learning Research_ , 22(1):13237–13312, 2021.
  * Lanthaler et al. [2022] S. Lanthaler, S. Mishra, and G. E. Karniadakis.  Error estimates for DeepONets: A deep learning framework in infinite dimensions.  _Transactions of Mathematics and Its Applications_ , 6(1):tnac001, 2022.
  * Leng and Müller [2006] X. Leng and H.-G. Müller.  Classification using functional data analysis for temporal gene expression data.  _Bioinformatics_ , 22(1):68–76, 2006.
  * Li et al. [2021] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar.  Fourier Neural Operator for parametric partial differential equations.  _The International Conference on Learning Representations (ICLR)_ , 2021.
  * Liu et al. [2024] H. Liu, H. Yang, M. Chen, T. Zhao, and W. Liao.  Deep nonparametric estimation of operators between infinite dimensional spaces.  _The Journal of Machine Learning Research_ , 25:1–67, 2024\.
  * Lu et al. [2021] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis.  Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators.  _Nature Machine Intelligence_ , 3(3):218–229, 2021.
  * Mhaskar and Hahm [1997] H. N. Mhaskar and N. Hahm.  Neural networks for functional approximation and system identification.  _Neural Computation_ , 9(1):143–159, 1997.
  * Morris [2015] J. S. Morris.  Functional regression.  _Annual Review of Statistics and its Application_ , 2:321–359, 2015.
  * Muller and Stadtmüller [2005] H. Muller and U. Stadtmüller.  Generalized functional linear models.  _The Annals of Statistics_ , 33(2):774–805, 2005\.
  * Novak et al. [2018] E. Novak, M. Ullrich, H. Woźniakowski, and S. Zhang.  Reproducing kernels of Sobolev Spaces on $\mathbb{R}^{d}$ and applications to embedding constants and tractability.  _Analysis and Applications_ , 16(05):693–715, 2018.
  * Pazouki and Schaback [2011] M. Pazouki and R. Schaback.  Bases for kernel-based spaces.  _Journal of Computational and Applied Mathematics_ , 236(4):575–588, 2011.
  * Raissi et al. [2019] M. Raissi, P. Perdikaris, and G. E. Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  _Journal of Computational physics_ , 378:686–707, 2019\.
  * Rossi and Conan-Guez [2005] F. Rossi and B. Conan-Guez.  Functional Multi-Layer Perceptron: a non-linear tool for functional data analysis.  _Neural networks_ , 18(1):45–60, 2005.
  * Schaback [1997] R. Schaback.  Reconstruction of multivariate functions from scattered data.  _Unpublished manuscript_ , 1997.
  * Schaback and Wendland [2006] R. Schaback and H. Wendland.  Kernel techniques: from machine learning to meshless methods.  _Acta numerica_ , 15:543–639, 2006.
  * Shang and Cheng [2015] Z. Shang and G. Cheng.  Nonparametric inference in generalized functional linear models.  _The Annals of Statistics_ , 43(4):1742–1773, 2015.
  * Song et al. [2023] L. Song, J. Fan, D.-R. Chen, and D.-X. Zhou.  Approximation of nonlinear functionals using deep ReLU networks.  _Journal of Fourier Analysis and Applications_ , 29(50), 2023.
  * Stinchcombe [1999] M. B. Stinchcombe.  Neural network approximation of continuous functionals and continuous functions on compactifications.  _Neural Networks_ , 12(3):467–477, 1999.
  * Suh and Cheng [2024] N. Suh and G. Cheng.  A survey on statistical theory of deep learning: Approximation, training dynamics, and generative models.  _Invited for review in Annual Review of Statistics and its Application_ , 2024.
  * Szabó et al. [2016] Z. Szabó, B. K. Sriperumbudur, B. Póczos, and A. Gretton.  Learning theory for distribution regression.  _The Journal of Machine Learning Research_ , 17(1):5272–5311, 2016.
  * Thind et al. [2023] B. Thind, K. Multani, and J. Cao.  Deep learning with functional inputs.  _Journal of Computational and Graphical Statistics_ , 32(1):171–180, 2023.
  * Tolstikhin et al. [2017] I. Tolstikhin, B. K. Sriperumbudur, and K. Muandet.  Minimax estimation of kernel mean embeddings.  _The Journal of Machine Learning Research_ , 18(1):3002–3048, 2017.
  * Wainwright [2019] M. J. Wainwright.  _High-Dimensional Statistics: A Non-Asymptotic Viewpoint_ , volume 48.  Cambridge university press, 2019.
  * Wang et al. [2016] J.-L. Wang, J.-M. Chiou, and H.-G. Müller.  Functional data analysis.  _Annual Review of Statistics and its application_ , 3:257–295, 2016.
  * Wang et al. [2021] S. Wang, H. Wang, and P. Perdikaris.  Learning the solution operator of parametric partial differential equations with physics-informed DeepONets.  _Science advances_ , 7(40):eabi8605, 2021.
  * Yao et al. [2005] F. Yao, H.-G. Müller, and J.-L. Wang.  Functional data analysis for sparse longitudinal data.  _Journal of the American Statistical Association_ , 100(470):577–590, 2005.
  * Yao et al. [2021] J. Yao, J. Mueller, and J.-L. Wang.  Deep learning for functional data analysis with adaptive basis layers.  In _International Conference on Machine Learning (ICML)_ , pages 11898–11908. PMLR, 2021.
  * Yuan and Cai [2010] M. Yuan and T. T. Cai.  A reproducing kernel Hilbert space approach to functional linear regression.  _The Annals of Statistics_ , 38(6):3412–3444, 2010.