Sparse-penalized deep neural networks estimator under weak dependence
We consider the nonparametric regression and the classification problems for
$\psi$-weakly dependent processes. This weak dependence structure is more
general than conditions such as, mixing, association, $\ldots$. A penalized
estimation method for sparse deep neural networks is performed. In both
nonparametric regression and binary classification problems, we establish
oracle inequalities for the excess risk of the sparse-penalized deep neural
networks estimators. Convergence rates of the excess risk of these estimators
are also derived. The simulation results displayed show that, the proposed
estimators overall work well than the non penalized estimators.
William Kengne, Modou Wade
# Sparse-penalized deep neural networks estimator under weak dependence

###### Abstract

We consider the nonparametric regression and the classification problems for \(\psi\)-weakly dependent processes. This weak dependence structure is more general than conditions such as, mixing, association, \(\ldots\). A penalized estimation method for sparse deep neural networks is performed. In both nonparametric regression and binary classification problems, we establish oracle inequalities for the excess risk of the sparse-penalized deep neural networks estimators. Convergence rates of the excess risk of these estimators are also derived. The simulation results displayed show that, the proposed estimators overall work well than the non penalized estimators.

_Keywords:_ Deep neural network, \(\psi\)-weakly dependence, sparsity, penalization, convergence rate.

William Kengne 1 and Modou Wade 2

Footnote 1: Developed within the ANR BREAKRISK: ANR-17-CE26-0001-01 and the CY Initiative of Excellence (grant ”Investissements d’Avenir” ANR-16-IDEX-0008), Project ”EcoDep” PSI-AAP2020-0000000013

Footnote 2: Supported by the MME-DII center of excellence (ANR-11-LABEX-0023-01)

_THEMA, CY Cergy Paris Universite, 33 Boulevard du Port, 95011 Cergy-Pontoise Cedex, France_

_E-mail: william.kengne@cyu.fr ; modou.wade@cyu.fr_

## 1 Introduction

Deep learning has received a considerable attention in the literature and has shown great success in several applications in artificial intelligence, for example, image processing (see ([15])), speech recognition (see([7])) \(\cdots\). In the recent decades, many researchers have contributed to understand the theoretical properties of deep neural networks (DNNs) predictors with the sparse regularization. See, for instance, [18], [20], [1], [21], [22], [12][9] (and the references therein) for some results with independent and identical distribution (i.i.d.) observations, and [2], [14], [16], [17], [11], [10] for some results with dependent or non-i.i.d. observations. Sparse-penalized DNNs estimators have been studied, among others, by [19], [16]. These authors establish some oracle-type inequalities for i.i.d. data and dependent observations under a mixing condition.

Let us consider the set of observations \(D_{n}\coloneqq\{(X_{1},Y_{1}),\cdots,(X_{n},Y_{n})\}\) (the training sample) from a stationary and ergodic process \(\{Z_{t}=(X_{t},Y_{t}),t\in\mathbb{Z}\}\), which takes values in \(\mathcal{Z}=(\mathcal{X}\times\mathcal{Y})\), where \(\mathcal{X}\) is the input space and \(\mathcal{Y}\) the output space. Consider for the sequel, a loss function \(\ell:\mathbb{R}\times\mathcal{Y}\to[0,\infty)\).
We perform sparse DNNs predictors for nonparametric regression and classification tasks, based on a penalized empirical risk minimization procedure. The estimator \(\widehat{h}_{n}\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\) is also called, the sparse-penalized DNN (SPDNN) predictor, and is given by

\[\widehat{h}_{n}=\operatorname*{argmin}_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_ {n},F)}\left[\frac{1}{n}\sum_{i=1}^{n}\ell(h(X_{i}),Y_{i})+J_{n}(h)\right], \tag{1.1}\]

where \(\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\) is a class of DNNs (see (3.3)) with an activation function \(\sigma\) and with suitably chosen architecture parameters \(L_{n},N_{n},B_{n},F\); and \(J_{n}(h)\) is the sparse penalty given by

\[J_{n}(h)\coloneqq J_{\lambda_{n},\tau_{n}}(h)\coloneqq\lambda_{n}\|\theta(h) \|_{\operatorname{clip},\tau_{n}},\]

for tuning parameters \(\lambda_{n}>0,\;\tau_{n}>0\) and \(\theta(h)\) is the vector of parameters of \(h\). \(\|\cdot\|_{\operatorname{clip},\tau}\) denotes the clipped \(L_{1}\) norm with a clipping threshold \(\tau>0\) see [24] defined as

\[\|\theta\|_{\operatorname{clip},\tau}=\sum_{j=1}^{p}\left(\frac{|\theta_{j}|}{ \tau}\wedge 1\right),\]

where \(\theta=(\theta_{1},\cdots,\theta_{p})^{\prime}\) is a \(p\)-dimensional vector and \({}^{\prime}\) denotes the transpose.

For both nonparametric regression and classification under weak dependence, we focus on the calibration of the parameters \(L_{n},N_{n},B_{n},F\) of the networks class, and \(\lambda_{n}\), \(\tau_{n}\) of the penalty term, that allow the SPDNN predictor \(\widehat{h}_{n}\) to enjoy an oracle property and to derive its convergence rate. These issues have been addressed by [19] in the i.i.d. case. They have established an oracle inequality of the excess risk and have proved that, the SPDNN can adaptively attain minimax optimality. [16] carried out SPDNN estimator for nonparametric time series regression under \(\beta\)-mixing condition. They provided a generalization error bound of the SPDNN estimator and proved that, this estimator attains the minimax optimal rate up to a poly-logarithmic factor. Besides addressing the time series classification issue, the \(\psi\)-weak dependence considered here is more general than mixing conditions (see [3]).

In this new contribution, we consider SPDNNs estimators for learning a \(\psi\)-weakly dependent process \(\{Z_{t}=(X_{t},Y_{t}),\;t\in\mathbb{Z}\}\) with value in \(Z=\mathcal{X}\times\mathcal{Y}\subseteq\mathbb{R}^{d}\times\mathbb{R}\), based on the training sample \(D_{n}=(X_{1},Y_{1}),\cdots,(X_{n},Y_{n})\) and we address the following issues.

* **Oracle inequality for the nonparametric time series regression**. We provide conditions on \(L_{n},N_{n},B_{n},F,\lambda_{n}\), \(\tau_{n}\), and establish an oracle inequality of the \(L_{2}\) error of the SPDNN estimator.
* **Oracle inequality for the binary time series classification**. Conditions on \(L_{n},N_{n},B_{n},F,\lambda_{n}\), \(\tau_{n}\) are provided, and an oracle inequality of the excess risk of the SPDNN estimator is established.
* **Convergence rates of the excess risk**. For both nonparametric regression and time series classification, the convergence rate (which depends on \(L_{n},N_{n},B_{n},F,\lambda_{n},\tau_{n}\)) of the excess risk is derived. When the true regression function (in the regression problem) and the target function (in the classification task) are sufficiently smooth, these rates are close to \(\mathcal{O}(n^{-1/2})\).

The rest of the paper is organized as follows. In Section 2, we set some notations and assumptions. Section 3 defines the class of DNNs considered. Section 4 is devoted to the nonparametric regression whereas Section 5 focuses on the binary time series classification. Some simulation results are provided in Section 6 and Section 7 is devoted to the proof of the main results.
## 2 Notations and assumptions

For two separable Banach spaces \(E_{1},E_{2}\) equipped with norms \(\|\cdot\|_{E_{1}}\) and \(\|\cdot\|_{E_{2}}\) respectively, denote by \(\mathcal{F}(E_{1},E_{2})\), the set of measurable functions from \(E_{1}\) to \(E_{2}\). For any \(h\in\mathcal{F}(E_{1},E_{2})\) and \(\epsilon>0\), \(B(h,\epsilon)\) denotes the ball of radius \(\epsilon\) of \(\mathcal{F}(E_{1},E_{2})\) centered at \(h\), that is,

\[B(h,\epsilon)=\big{\{}f\in\mathcal{F}(E_{1},E_{2}),\ \|f-h\|_{\infty}\leq \epsilon\big{\}},\]

where \(\|\cdot\|_{\infty}\) stands for the sup-norm defined below. Let \(\mathcal{H}\subset\mathcal{F}(E_{1},E_{2})\), the \(\epsilon\)-covering number \(\mathcal{N}(\mathcal{H},\epsilon)\) of \(\mathcal{H}\) is the minimal number of balls of radius \(\epsilon\) needed to cover \(\mathcal{H}\); that is,

\[\mathcal{N}(\mathcal{H},\epsilon)=\inf\Big{\{}m\geq 1\ :\exists h_{1},\cdots,h_{ m}\in\mathcal{H}\ \text{such that}\ \mathcal{H}\subset\bigcup_{i=1}^{m}B(h_{i},\epsilon)\Big{\}}.\]

For a function \(h:E_{1}\to E_{2}\) and \(U\subseteq E_{1}\), define,

\[\|h\|_{\infty}=\sup_{x\in E_{1}}\|h(x)\|_{E_{2}},\ \|h\|_{\infty,U}=\sup_{x\in U} \|h(x)\|_{E_{2}}\ \text{and}\]

\[\text{Lip}_{\alpha}(h)\coloneqq\sup_{x_{1},x_{2}\in E_{1},\ x_{1}\neq x_{2}} \frac{\|h(x_{1})-h(x_{2})\|_{E_{2}}}{\|x_{1}-x_{2}\|_{E_{1}}^{\alpha}}\ \text{for any}\ \alpha\in[0,1].\]

For any \(\mathcal{K}_{\ell}>0\) and \(\alpha\in[0,1]\), \(\Lambda_{\alpha,\mathcal{K}_{\ell}}(E_{1},E_{2})\) (simply \(\Lambda_{\alpha,\mathcal{K}_{\ell}}(E_{1})\) when \(E_{2}\subseteq\mathbb{R}\)) is the set of functions \(h:E_{1}^{u}\to E_{2}\) for some \(u\in\mathbb{N}\), satisfies \(\|h\|_{\infty}<\infty\) and \(\text{Lip}_{\alpha}(h)\leq\mathcal{K}_{\ell}\). When \(\alpha=1\), we set \(\text{Lip}_{1}(h)=\text{Lip}(h)\) and \(\Lambda_{1}(E_{1})=\Lambda_{1,1}(E_{1},\mathbb{R})\).

We define the weak dependence structure, see [6] and [3]. Let \(E\) be a separable Banach space.

**Definition 2.1**.: _An \(E-valued\)\(process\ (Z_{t})_{t\in\mathbb{Z}}\) is said to be \((\Lambda_{1}(E),\psi,\epsilon)-\)weakly dependent \(\ if\ there\ exists\ a\ function\ \ \psi:[0,\infty)^{2}\times\mathbb{N}^{2}\to[0,\infty)\ and\ a\ sequence\epsilon=(\epsilon(r))_{r\in\mathbb{N}}\) decreasing to zero at \(infinity\ such\ that,\ for\ any\ g_{1},\ g_{2}\in\Lambda_{1}(E)\ with\ g_{1}:E^{u}\to\mathbb{R},\ g_{2}:E^{v}\to\mathbb{R}\,\ (u,v\in\mathbb{N})\ and\ for\ any\ u-tuple\ (s_{1},\cdots,s_{u})\) and any \(v-tuple\ (t_{1},\cdots,t_{v})\ with\ s_{1}\leq\cdots\leq s_{u}\leq s_{u}+r\leq t_{1}\leq \cdots\leq t_{v},\ the\ following\ inequality\ is\ fulfilled\):_

\[|\text{Cov}(g_{1}(Z_{s_{1}},\cdots,Z_{s_{u}}),g_{2}(Z_{t_{1}},\cdots,Z_{t_{v}} ))|\leq\psi(\text{Lip}(g_{1}),\text{Lip}(g_{2}),u,v)\epsilon(r).\]

For example, following choices of \(\psi\) leads to some well-known weak dependence conditions.

* \(\psi\left(\text{Lip}(g_{1}),\text{Lip}(g_{2}),u,v\right)=v\text{Lip}(g_{2})\): the \(\theta\)-weak dependence, then denote \(\epsilon(r)=\theta(r)\);
* \(\psi\left(\text{Lip}(g_{1}),\text{Lip}(g_{2}),u,v\right)=u\text{Lip}(g_{1})+ v\text{Lip}(g_{2})\): the \(\eta\)-weak dependence, then denote \(\epsilon(r)=\eta(r)\);
* \(\psi\left(\text{Lip}(g_{1}),\text{Lip}(g_{2}),u,v\right)=uv\text{Lip}(g_{1}) \cdot\text{Lip}(g_{2})\): the \(\kappa\)- weak dependence, then denote \(\epsilon(r)=\kappa(r)\);
* \(\psi\left(\text{Lip}(g_{1}),\text{Lip}(g_{2}),u,v\right)=u\text{Lip}(g_{1}) +v\text{Lip}(g_{2})+uv\text{Lip}(g_{1})\cdot\text{Lip}(g_{2})\): the \(\lambda\)-weak dependence, then denote \(\epsilon(r)=\lambda(r)\).

We consider the process \(\{Z_{t}=(X_{t},Y_{t}),t\in\mathbb{Z}\}\) with values in \(\mathcal{Z}=\mathcal{X}\times\mathcal{Y}\subset\mathbb{R}^{d}\times\mathbb{R}\), the loss function \(\ell:\mathbb{R}\times\mathcal{Y}\to[0,\infty)\), the class of DNNs \(\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\) with the activation function \(\sigma:\mathbb{R}\to\mathbb{R}\), and set of following assumptions.

**(A1)**: : There exists a constant \(C_{\sigma}>0\) such that the activation function \(\sigma\in\Lambda_{1,C_{\sigma}}(\mathbb{R})\).
* : There exists \(\mathcal{K}_{\ell}>0\) such that, the loss function \(\ell\in\Lambda_{1,\mathcal{K}_{\ell}}(\mathbb{R}\times\mathcal{Y})\) and \(M=\sup_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}\sup_{z\in\mathcal{Z}}| \ell(h,z)|<\infty\). In the case of margin-based loss, these conditions are assumed to the function \((u,y)\mapsto\ell(uy)\).

Let us set the weak dependence assumption.

* : The process \(\{Z_{t}=(X_{t},Y_{t}),t\in\mathbb{Z}\}\) is stationary ergodic and \((\Lambda_{1}(\mathcal{Z}),\psi,\epsilon)\)-weakly dependent with \(\epsilon_{r}=\mathcal{O}(r^{-\gamma})\) for some \(\gamma>3\).

## 3 Deep Neural Networks

A DNN, with \((L,\mathbf{p})\) network architecture, where \(L\in\mathbb{N}\) is the number of hidden-layers and \(\mathbf{p}=(p_{0},\cdots,p_{L+1})\in\mathbb{N}^{L+2}\) the width vector, is any function \(h\) of the form,

\[h:\mathbb{R}^{p_{0}}\rightarrow\mathbb{R}^{p_{L+1}},\ x\mapsto h (x)=A_{L+1}\circ\sigma_{L}\circ A_{L}\circ\sigma_{L-1}\circ\cdots\circ\sigma _{1}\circ A_{1}(x), \tag{3.1}\]

where \(A_{j}:\mathbb{R}^{p_{j-1}}\rightarrow\mathbb{R}^{p_{j}}\) is a linear affine map, defined by \(A_{j}(x):=W_{j}x+\mathbf{b}_{j}\), for given \(p_{j-1}\times p_{j}\) weight matrix \(W_{j}\) and a shift vector \(\mathbf{b}_{j}\in\mathbb{R}^{p_{j}}\) and \(\sigma_{j}:\mathbb{R}^{p_{j}}\rightarrow\mathbb{R}^{p_{j}}\) is a nonlinear element-wise activation map, defined by \(\sigma_{j}(z)=(\sigma(z_{1}),\cdots,\sigma(z_{p_{j}}))^{{}^{\prime}}\). For a DNN of the form (3.1), denote by,

\[\theta(h)\coloneqq\left(vec(W_{1})^{{}^{\prime}},\mathbf{b}_{1} ^{{}^{\prime}},\cdots,vec(W_{L+1})^{{}^{\prime}},\mathbf{b}_{L+1}^{{}^{\prime} }\right)^{{}^{\prime}}, \tag{3.2}\]

the vector of its parameters, where \(vec(W)\) transforms the matrix \(W\) into the corresponding vector by concatenating the column vectors. Let \(\mathcal{H}_{\sigma,p_{0},p_{L+1}}\) be the class of DNNs predictors that take \(p_{0}\)-dimensional input to produce \(p_{L+1}\)-dimensional output and use the activation function \(\sigma:\mathbb{R}\rightarrow\mathbb{R}\). In our setting here, \(p_{0}=d\) and \(p_{L+1}=1\). For a DNN \(h\) as in (3.1), let \(\mathrm{depth}(h)\) and \(\mathrm{width}(h)\) be respectively the depth and width of \(h\); that is, \(\mathrm{depth}(h)\)\(=L\) and \(\mathrm{width}(h)=\max\limits_{1\leq j\leq L}p_{j}\). For any positive constants \(L,N,B\) and \(F\), we set

\[\mathcal{H}_{\sigma}(L,N,B)\coloneqq\big{\{}h\in\mathcal{H}_{ \sigma,q,1}:\mathrm{depth}(h)\leq L,\mathrm{width}(h)\leq N,\|\theta(h)\|_{ \infty}\leq B\big{\}},\]

and

\[\mathcal{H}_{\sigma}(L,N,B,F)\coloneqq\big{\{}h:h\in H_{\sigma}(L,N,B),\|h\|_{\infty,\mathcal{X}}\leq F\big{\}}. \tag{3.3}\]

A class of sparsity constrained DNNs with sparsity level \(S>0\) is defined by

\[\mathcal{H}_{\sigma}(L,N,B,F,S)\coloneqq\left\{h\in\mathcal{H} _{\sigma}(L,N,B,F)\ :\ \|\theta(h)\|_{0}\leq S\right\}, \tag{3.4}\]

where \(\|x\|_{0}=\sum_{i=1}^{p}\mathds{1}(x_{i}\neq 0)\) for all \(x=(x_{1},\ldots,x_{p})^{\prime}\in\mathbb{R}^{p}\) (\(p\in\mathbb{N}\)). In the sequel, we will establish some theoretical results for the SPDNN estimators (1.1) in both regression and classification problems under the \(\psi\)-weak dependence.

## 4 Nonparametric regression

In this section we aim to study the nonparametric time series regression, with the output \(Y_{t}\in\mathbb{R}\) and the input \(X_{t}\in\mathbb{R}^{d}\) are generated from the model,

\[Y_{t}=h^{*}(X_{t})+\epsilon_{t},\ X_{0}\sim\mathrm{P}_{X_{0}}, \tag{4.1}\]
where \(h^{*}:\mathbb{R}^{d}\rightarrow\mathbb{R}\) is the unknown regression function and \(\epsilon_{t}\) is an error variable independent of the input variable \(X_{t}\). Let us consider the sub-Gaussian assumption to the error,

\[\mathbb{E}[e^{t\epsilon_{t}}]\leq e^{t^{2}\rho^{2}/2}, \tag{4.2}\]

for any \(t\in\mathbb{R}\) for some \(\rho>0\). So, denote by \(\mathcal{H}_{\rho,H^{*}}\) the set of distributions \((X_{0},Y_{0})\) satisfying the model (4.1)

\[\mathcal{H}_{\rho,H^{*}}\coloneqq\left\{\text{Model}(\ref{eq:eq:eq:eq:eq:eq:eq: eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eq:eq:eq:eqeq:eq:eq:eq:eqeq:eq:eq:eq:eq:eq:eqeq:eq:eqeq:eq:eq:eqeq:eqeq:eq:eqeq:eq:eq:eqeq:eq:eqeq:eq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eq:eq:eq:eqeq:eqeq:eqeq:eq:eq:eq:eqeq:eq:eq:eqeq:eq:eq:eqeq:eq:eqeq:eqeq:eqeq:eq:eqeq:eqeq:eqeq:eq:eqeq:eqeq:eq:eqeq:eqeq:eq:eqeq:eqeq:eqeq:eqeq:eqeq:eq:eqeq:eqeq:eq:eqeq:eq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eq:eq:eqeqeq:eq:eqeq:eq:eqeqeq:eqeq:eqeq:eqeqeq:eqeq:eq:eqeq:eq:eqeqeq:eq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eq:eqeq:eq:eqeq:eqeq:eqeq:eqeq:eqeq:eq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeq:eqeqeq:eqeqeq:eqeq:eqeq:eqeq:eqeq:eqeqeq:eqeq:eqeq:eqeq:eqeqeq:eqeq:eqeqeq:eqeq:eqeqeq:eqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeq:eqeqeqeq:eqeqeq:eqeqeq:eqeqeqeq:eq:eqeqeq:eqeq:eqeqeqeq:eqeqeqeq:eqeq:eqeqeq:eqeqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeq:eqeqeq:eqeqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeq:eqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq:eqeqeqeqeq:eqeqeqeqeqeq
_with \(S_{n,\epsilon}\coloneqq C\epsilon^{-\kappa}(\log n)^{r}\) for any \(\epsilon\in(0,\epsilon_{0})\) and \(n\in\mathbb{N}\). Then, the SPDNN estimator defined in (4.4) satisfies_

\[\sup_{H\in\mathcal{H}_{\rho,H^{*}}:h^{*}\in\mathcal{H}^{*}}\mathbb{E}\Big{[}\| \widehat{h}_{n}-h^{*}\|_{2,P_{X_{0}}}^{2}\Big{]}\lesssim\frac{(\log n)^{r+ \nu_{3}}}{n^{\frac{2\nu_{4}}{r+2}}}\vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}, \tag{4.7}\]

_for some constant \(\nu_{3}>0,\ 0<\nu_{4}<1,\ \nu_{5}>0,\ \nu_{6}>0\ \Big{(}\nu_{6}<1/2,\nu_{4}+\nu_{6}<1 \Big{)}\) or \(\Big{(}\nu_{6}<1/2,\nu_{4}+\nu_{6}=1,\nu_{5}>1-\nu_{3}\Big{)}\)._

**Remark 4.3**.: _Consider the Holder space of smoothness \(s>0\), with radius \(\mathcal{K}>0\) given by,_

\[\mathcal{C}^{s,\mathcal{K}}(\mathcal{X})=\big{\{}h:\mathcal{X}\to\mathbb{R},\ \|h\|_{\mathcal{C}^{s}(\mathcal{X})}\leq\mathcal{K}\big{\}},\]

_where \(\|\cdot\|_{\mathcal{C}^{s}(\mathcal{X})}\) denotes the Holder norm defined by,_

\[\|h\|_{\mathcal{C}^{s}(\mathcal{X})}=\sum_{\beta\in\mathbb{N}_{0}^{d},\ |\beta|\leq[s]}\|\partial^{\beta}h\|_{\infty}+\sum_{\beta\in\mathbb{N}_{0}^{d}, \ |\beta|=[s]}Lip_{s-[s]}(\partial^{\beta}h),\]

_with, \(\mathbb{N}_{0}=\mathbb{N}\cup\{0\}\), \(|\beta|=\sum_{i=1}^{d}\beta_{i}\) for all \(\beta=(\beta_{1},\ldots,\beta_{d})^{\prime}\in\mathbb{N}_{0}^{d}\), \(\partial^{\beta}\) denotes the partial derivative of order \(\beta\) and \([x]\) denotes the integer part of \(x\). So, if the true regression function satisfies for instance \(h^{*}\in\mathcal{C}^{s,\mathcal{K}}(\mathcal{X})\) for some \(\mathcal{K}>0\), then, condition (4.6) holds for a various classes of activation function, including ReLU with \(\kappa=d/s\) and \(r=1\) (see [11]). In this case, if \(s\gg d\), then, the convergence rate of the SPDNN estimator is close to \(\mathcal{O}(n^{-1/2})\)._

## 5 Binary time series classification

We consider the binary classification of the process \(\{Z_{t}=(X_{t},Y_{t}),t\in\mathbb{Z}\}\), which values in \(\mathbb{R}^{d}\times\{-1,1\}\). The goal is to construct a function \(h:\mathbb{R}^{d}\to\mathbb{R}\) such that, \(h(X_{t})\) is used to predict the label \(Y_{t}\in\{-1,1\}\). We will focus on a margin-based loss function to evaluate the performance of the prediction by \(h\); ; that is, a loss function of the form: \((u,y)\mapsto\ell(uy)\). For instance, the hinge loss with \(\ell(uy)=\max(1-uy,0)\). We assume in the sequel that, the input \(X_{t}\in\mathbb{R}^{d}\) and the output \(Y_{t}\in\{-1,1\}\) are generated from the model

\[Y_{t}|X_{t}=\mathrm{x}\sim 2\mathcal{B}(\eta(\mathrm{x}))-1,\qquad X_{0}\sim \mathrm{P}_{X_{0}}, \tag{5.1}\]

where \(\eta(\mathrm{x})=P(Y_{t}=1|X_{t}=\mathrm{x})\) and \(\mathcal{B}(\eta(\mathrm{x}))\) is the Bernoulli distribution with parameter \(\eta(\mathrm{x})\). The aim is to build a classifier \(h\) so that, the excess risk of \(h\) defined by,

\[\mathcal{E}_{Z_{0}}(h)\coloneqq\mathbb{E}[\ell(Y_{0}h(X_{0}))]-\mathbb{E}[\ell (Y_{0}h_{\ell}^{*}(X_{0}))],\ \text{with}\ Z_{0}\coloneqq(X_{0},Y_{0}) \tag{5.2}\]

is "close" to zero, where \(\ell\) is a given margin-based loss function, \(h_{\ell}^{*}=\underset{h\in\mathcal{F}}{\mathrm{argmin}}\mathbb{E}\left[\ell(Y _{0}h(X_{0}))\right]\) is a target function, suppose to be bound, that is \(\|h_{\ell}^{*}\|_{\infty}\leq H^{*}\) for some \(H^{*}>0\) and \(\mathcal{F}\) is the set of measurable functions from \(\mathbb{R}^{d}\) to \(\mathbb{R}\). For the sequel, define the set of distributions satisfying this assumption by,

\[\mathcal{Q}_{H^{*}}\coloneqq\{Model(\ref{eq:1}):\|h_{\ell}^{*}\|_{\infty}\leq H ^{*}\}. \tag{5.3}\]

The following theorem provides an oracle inequality for the excess risk of the SPDNN estimator based on a margin-based loss function \(\ell\); whose, the strict convexity is not required as in [19].
**Theorem 5.1**.: _Assume that (**A1**)-(**A3**) hold and that the true generative model \(H\) is in \(\mathcal{Q}_{H^{*}}\). Let \(F>0\), \(L_{n}\lesssim\log n,N_{n}\lesssim n^{\nu_{1}},1\leq B_{n}\lesssim n^{\nu_{2}}\) for some \(\nu_{1}>0,\ \nu_{2}>0\). Then, the SPDNN estimator defined by,_

\[\widehat{h}_{n}=\operatorname*{argmin}_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B _{n},F)}\left[\frac{1}{n}\sum_{i=1}^{n}\ell(Y_{i}h(X_{i}))+\lambda_{n}\|\theta (h)\|_{\text{clip},\tau_{n}}\right], \tag{5.4}\]

_with \(\lambda_{n}\asymp(\log n)^{\nu_{3}}/n,\ \tau_{n}\leq\frac{\beta_{n}}{4 \mathcal{K}_{\ell}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\) for \(\beta_{n}\coloneqq(\log n)^{\nu_{5}}/n^{\nu_{6}},\ \mathcal{K}_{\ell}>0\), satisfies_

\[\mathbb{E}\left[\mathcal{E}_{Z_{0}}(\widehat{h}_{n})\right]\leq 2\inf_{h\in \mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}\left\{\mathcal{E}_{Z_{0}}(h)+ \lambda_{n}\|\theta(h)\|_{\text{clip},\tau_{n}}\right\}\vee\frac{C(\log n)^{ \nu_{5}}}{n^{\nu_{6}}}, \tag{5.5}\]

_for some universal constant \(C>0,\ \nu_{5}>0,\ \nu_{6}>0,\ \left(\nu_{6}<1/2,\nu_{4}+\nu_{6}<1\right)\) or \(\left(\nu_{6}<1/2,\nu_{4}+\nu_{6}=1,\nu_{5}>1-\nu_{3}\right)\), where the expectation is taken over the training data._

As Theorem 4.2 in the nonparametric regression, the following theorem provides a useful tool to derive a convergence rate of the SPDNN estimator.

**Theorem 5.2**.: _Assume the conditions of Theorem 5.1, with similar choices of \(F,L_{n},N_{n},B_{n},\lambda_{n},\tau_{n}\). Let \(H^{*}>0\) and let \(\mathcal{H}^{*}\) be a set of real-valued functions on \(\mathbb{R}^{d}\) and assume there are constants \(\kappa>0,\ r>0,\epsilon_{0}>0\), and \(C>0\) such that,_

\[\sup_{h^{\circ}\in\mathcal{H}^{*}:\|h^{\circ}\|_{\infty}\leq H^{*}\ \ h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F,S_{n,\epsilon})}\|h-h^{\circ} \|_{1,P_{X_{0}}}\leq\epsilon, \tag{5.6}\]

_with \(S_{n,\epsilon}\coloneqq C\epsilon^{-\kappa}(\log n)^{r}\) for any \(\epsilon\in(0,\epsilon_{0})\) and \(n\in\mathbb{N}\). Then, SPDNN estimator defined in (5.4) satisfies,_

\[\sup_{H\in\mathcal{Q}_{H^{*}}:\widehat{h}_{\ell}^{*}\in\mathcal{H}^{*}} \mathbb{E}\left[\mathcal{E}_{Z_{0}}(\widehat{h}_{n})\right]\lesssim\frac{( \log n)^{r+\nu_{3}}}{n^{\frac{\nu_{4}}{\epsilon+1}}}\vee\frac{(\log n)^{\nu_{5 }}}{n^{\nu_{6}}}. \tag{5.7}\]

As stressed in Remark 4.3, such result can be used to derive a convergence rate of the excess risk of the SPDNN estimator in several situations.

## 6 Some numerical results

In this section, we carry out the prediction of autoregressive models and binary time series by SPDNN.

### Prediction of autoregressive models

Let us consider a nonlinear autoregressive process with exogenous covariate \((Y_{t},\mathcal{X}_{t})_{t\in\mathbb{Z}}\) with values in \(\mathbb{R}\times\mathbb{R}\) satisfying:

\[Y_{t}=f(Y_{t-1},\ldots,Y_{t-p};\mathcal{X}_{t-1})+\epsilon_{t}, \tag{6.1}\]

for some measurable function \(f:\mathbb{R}^{p+1}\to\mathbb{R}\) (\(p\in\mathbb{N}\)), where \((\epsilon_{t})_{t\in\mathbb{Z}}\) is i.i.d. generated from a standardized uniform distribution \(\mathcal{U}[-2;2]\) and \((\mathcal{X}_{t})_{t\in\mathbb{Z}}\) is an AR (1) process generated with a standardized \(\mathcal{U}[-2;2]\) innovations. The process \((Y_{t},\mathcal{X}_{t})_{t\in\mathbb{Z}}\) in (6.1) is a specific example of the affine causal models with exogenous covariates studied in [4]. In the sequel, we set \(X_{t}=(Y_{t-1},\cdots,Y_{t-p},\mathcal{X}_{t-1})\). So, one can see that, this nonlinear autoregressive process is a particular case of the model (4.1). If \(f\in\Lambda_{1}(\mathbb{R}^{p+1})\) (that is, \(f\) is Lipschitz) and
under some classical conditions on the Lipschitz-type coefficients of \(f\) (for example, \(Lip(f)<1/p\)), then, there exists a solution \((Y_{t},X_{t})_{t\in\mathbb{Z}}\) from (6.1), that fulfills the assumption **(A3)** above; see details in [5].

Let \((Y_{1},\mathcal{X}_{1}),\cdots,(Y_{n},\mathcal{X}_{n})\) be a trajectory of the process \((Y_{t},\mathcal{X}_{t})_{t\in\mathbb{Z}}\). We aim to predict \(Y_{n+1}\) from this training sample. We perform the learning theory with SPDNN predictors developed above with, the input variable \(X_{t}=(Y_{t-1},\mathcal{X}_{t-1})\), the input space \(\mathcal{X}\subset\mathbb{R}^{p}\times\mathbb{R}\) and, the output space \(\mathcal{Y}\subset\mathbb{R}\). We consider the following cases in (6.1):

\[\text{DGP1}:\;\;Y_{t}=1-0.2Y_{t-1}+0.3Y_{t-2}+0.25Y_{t-3}-0.6\frac{ 1}{1+\mathcal{X}_{t-1}^{2}}+\epsilon_{t};\] \[\text{DGP2}:\;\;Y_{t}=0.5+\big{(}-0.4+0.25e^{-2Y_{t-1}^{2}}\big{)} Y_{t-1}+1.5\mathcal{X}_{t-1}+\epsilon_{t}.\]

DGP1 is a classical autoregressive model with nonlinear covariate, whereas DGP2 is an exponential autoregression with covariate.

For each of these DGPs, we perform a network architecture of 2 hidden layers with 100 hidden nodes for each layer. The ReLU and linear activation functions are used respectively in the hidden layers and the output layer. The network weights are trained in the R software with the package Keras, by using the algorithm Adam ([13]) with learning rate \(10^{-3}\) and the minibatch size of 32. The training is stopped when the mean square error (MSE) is not improved within 30 epochs.

For \(n=250,500\) and 1000, a trajectory \(((Y_{1},\mathcal{X}_{1}),(Y_{2},\mathcal{X}_{2}),\ldots,(Y_{n},\mathcal{X}_{ n}))\) is generated from the true DGP. The predictor \(\widehat{h}_{n}\) is obtained from (4.4), with the tuning parameters of the form \(\lambda_{n}=10^{-i}\log(n)/n\) and \(\tau_{n}=10^{-j}/\log(n)\), where \(i,j=0,1,\ldots,10\) are calibrated by minimizing the MSE on a validation data set \(((Y_{1}^{\prime},\mathcal{X}_{1}^{\prime}),(Y_{2}^{\prime},\mathcal{X}_{2}^{ \prime}),\ldots,(Y_{n}^{\prime},\mathcal{X}_{n}^{\prime}))\). The empirical \(L_{2}\) error of \(\widehat{h}_{n}\) is then computed based on a new trajectory \(((Y_{1}^{\prime\prime},\mathcal{X}_{1}^{\prime\prime}),(Y_{2}^{\prime\prime}, \mathcal{X}_{2}^{\prime\prime}),\ldots,(Y_{m}^{\prime\prime},\mathcal{X}_{m}^ {\prime\prime}))\) with \(m=10^{4}\).

Figure 1 displays the boxplots of this empirical \(L_{2}\) error of the SPDNN and non penalized DNN (NPDNN, obtained from (4.4) with \(\lambda_{n}=0\)) predictors over 100 replications. We can see that, the SPDNN estimator outperforms the NPDNN estimator in DGP1. In DGP2, the performance of the SPDNN estimator is slightly better than that of the NPDNN estimator. These findings show that, the SPDNN estimator can improve the \(L_{2}\) error, compared to the NPDNN estimator.

### Prediction of binary time series

Let us consider a binary autoregressive process with exogenous covariates \((Y_{t},\mathcal{X}_{t})_{t\in\mathbb{Z}}\) with values in \(\{-1,1\}\times\mathbb{R}\) satisfying,

\[Y_{t}|\mathcal{F}_{t-1}\sim 2\mathcal{B}(p_{t})-1\text{ with }2p_{t}-1=\mathbb{E}[Y_{t}| \mathcal{F}_{t-1}]=f(Y_{t-1},\cdots,Y_{t-p},;\mathcal{X}_{t-1}), \tag{6.2}\]

where \(\mathcal{F}_{t-1}=\sigma\{Y_{t-1},\cdots;\mathcal{X}_{t-1},\cdots\}\) for some measurable function \(f:\mathbb{R}^{p+1}\rightarrow[-1,1]\) (\(p\in\mathbb{N}\)), \((\mathcal{X}_{t})_{t\in\mathbb{Z}}\) is an AR(1) process and \(\mathcal{B}(p_{t})\) denotes the Bernoulli distribution with parameter \(p_{t}\). Set \(X_{t}=(Y_{t-1},\cdots,Y_{t-p},\mathcal{X}_{t-1})\). Therefore, one can see that, the binary model (6.2) is a specific case of (5.1). Under some classical Lipschitz-type condition on \(f\), the process \((Y_{t},\mathcal{X}_{t})_{t\in\mathbb{Z}}\) fulfills the weak dependence assumption **(A3)**, see [11].

Let \((Y_{1},\mathcal{X}_{1}),\cdots,(Y_{n},\mathcal{X}_{n})\) be a trajectory of the process \((Y_{t},\mathcal{X}_{t})_{t\in\mathbb{Z}}\), and the aim is to predict \(Y_{n+1}\) from this training sample. We perform the learning theory with SPDNN predictor proposed here with
\((Y_{t-1},\mathcal{X}_{t-1})\) in the DGP3, and \(p=2,\ X_{t}=(Y_{t-1},Y_{t-2},\mathcal{X}_{t-1})\) in the DGP4. We study the following cases of (6.2):

\[\text{DGP3}: f(Y_{t-1};\mathcal{X}_{t-1})=-0.15+\left(0.1-0.2e^{-0.5Y_{t-1}^{2} }\right)Y_{t-1}+0.25\frac{1}{1+\mathcal{X}_{t-1}^{2}}\] \[\text{DGP4}: f(Y_{t-1},Y_{t-2};\mathcal{X}_{t-1})=0.1+0.15Y_{t-1}-0.25Y_{t-2 }-0.2e^{-\mathcal{X}_{t-1}^{2}}\]

Figure 1: _Boxplots of empirical \(L_{2}\) errors of the SPDNN and NPDNN predictors with \(n=250,500\) and 1000 in DGP1 (a) and DGP2 (b)._
In the following, we consider the hinge loss function (\(\ell(z)=\max(1-z,0)\)) and defined

\[h_{\ell}^{*}(X_{t})=2\mathbbm{1}_{\{f(X_{t})\geq 0\}}-1\text{ for all }t\in\mathbb{Z}. \tag{6.3}\]

One can see that, \(h_{\ell}^{*}\) is the Bayes classifier with respect to this loss; that is

\[\mathcal{E}_{Z_{0}}(h_{\ell}^{*})=\inf_{h\in\mathcal{F}(\mathcal{X},\mathcal{ Y})}\mathcal{E}_{Z_{0}}(h),\]

where \(\mathcal{E}_{Z_{0}}(\cdot)\) is defined in (5.2) and \(\mathcal{F}(\mathcal{X},\mathcal{Y})\) is the class of measurable functions from \(\mathcal{X}\) to \(\mathcal{Y}\). Also, for each of these DGPs, we perform a network architecture as in Subsection (6.1).

For \(n=250,500\) and \(1000\), a trajectory \(((Y_{1},\mathcal{X}_{1}),\ldots,(Y_{n},\mathcal{X}_{n}))\) is generated from the true DGP. The predictor \(\widehat{h}_{n}\) is obtained from (5.4), where the tuning parameters \(\lambda_{n},\ \tau_{n}\) are chosen as in Subsection (6.1), based on a validation data set \(((Y_{1}^{\prime},\mathcal{X}_{1}^{\prime}),(Y_{2}^{\prime},\mathcal{X}_{2}^{ \prime}),\ldots,(Y_{n}^{\prime},\mathcal{X}_{n}^{\prime}))\). The empirical excess risk of \(\widehat{h}_{n}\) is computed from a new trajectory \(((Y_{1}^{\prime\prime},\mathcal{X}_{1}^{\prime\prime}),(Y_{2}^{\prime\prime},\mathcal{X}_{2}^{\prime\prime}),\ldots,(Y_{m}^{\prime\prime},\mathcal{X}_{m }^{\prime\prime}))\) with \(m=10^{4}\).

Figure 2 displays the boxplots of the empirical excess risk of the SPDNN predictor and that of the NPDNN predictor overs \(100\) replications. One can observe that, the performance of the SPDNN overall better than that of the NPDNN. Which shows once again that, the SPDNN estimator can improve the prediction accuracy, compared to the NPDNN estimator.

## 7 Proofs of the main results

### Proof of Theorem 4.1

For the proof, we write \(a_{n}\lesssim_{\rho,H^{*}}b_{n}\) if there is a constant \(C_{\rho,H^{*}}>0\) depending only on \(\rho\) and \(H^{*}\) such that \(a_{n}\leq C_{\rho,H^{*}}b_{n}\) for any \(n\in\mathbb{N}\). Let \(K_{n}\coloneqq(\sqrt{32\rho^{2}}(\log n)^{1/2})\lor H^{*}\). Let \(Y_{0}^{\perp}\coloneqq\operatorname{sign}(Y_{0})(|Y_{0}|\wedge K_{n})\), which is a truncated version of \(Y_{0}\) and \(h^{\perp}\) be the regression function of \(Y_{0}^{\perp}\), that is,

\[h^{\perp}(x)\coloneqq\mathbb{E}(Y_{0}^{\perp}|X_{0}=x).\]

For national convenience, we suppress the dependency on \(n\) in the notation \(Y_{0}^{\perp}\) and \(h^{\perp}\). As in [19], consider the following decomposition

\[\|\widehat{h}_{n}-h^{*}\|_{2,P_{X_{0}}}^{2}=\mathbb{E}\Big{[}\left(Y_{0}- \widehat{h}_{n}(X_{0})\right)^{2}\Big{]}-\mathbb{E}\Big{[}\left(Y_{0}-h^{*}(X _{0})\right)^{2}\Big{]}=\sum_{i=1}^{4}A_{i,n}, \tag{7.1}\]

where,

\[A_{1,n} \coloneqq\Big{[}\mathbb{E}\left(Y_{0}-\widehat{h}_{n}(X_{0}) \right)^{2}-\mathbb{E}\left(Y_{0}-h^{*}(X_{0})\right)^{2}\Big{]}-\Big{[} \mathbb{E}\left(Y_{0}^{\perp}-\widehat{h}_{n}(X_{0})\right)^{2}-\mathbb{E} \left(Y_{0}^{\perp}-h^{\perp}(X_{0})\right)^{2}\Big{]};\] \[A_{2,n} \coloneqq\Big{[}\mathbb{E}\left(Y_{0}^{\perp}-\widehat{h}_{n}(X_{0 })\right)^{2}-\mathbb{E}\left(Y_{0}^{\perp}-h^{\perp}(X_{0})\right)^{2}\Big{]} -2\Big{[}\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-\widehat{h}_{n}(X_{i}) \right)^{2}-\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-h^{\perp}(X_{i}) \right)^{2}\Big{]}-2J_{\lambda_{n},\tau_{n}}(\widehat{h}_{n});\] \[A_{3,n} \coloneqq 2\Big{[}\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-\widehat{h} _{n}(X_{i})\right)^{2}-\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-h^{\perp}(X _{i})\right)^{2}\Big{]}-2\Big{[}\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-\widehat{ h}_{n}(X_{i})\right)^{2}-\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-h^{*}(X_{i}) \right)^{2}\Big{]};\] \[A_{4,n} \coloneqq 2\left[\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-\widehat{h}_{n}(X_{ i})\right)^{2}-\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}-h^{*}(X_{i})\right)^{2}\right]+2J_{ \lambda_{n},\tau_{n}}(\widehat{h}_{n}).\]
The first equality in (7.1) holds by using the independence of \(X_{0}\) and \(\epsilon_{0}\). To bound \(A_{1,n}\), let us recall the properties of sub-Gaussian variables \(\mathbb{E}(\epsilon_{0})=0\) and \(\mathbb{E}(e^{\epsilon_{0}^{2}/4\rho^{2}})\leq\sqrt{2}\), see, for instance, Theorem 2.6 in [23]. Let

\[A_{1,1,n} \coloneqq\mathbb{E}\left((Y_{0}^{\perp}-Y_{0})(2\widehat{h}_{n}(X _{0})-Y_{0}-Y_{0}^{\perp})\right);\] \[A_{1,2,n} \coloneqq\mathbb{E}\left[(Y_{0}^{\perp}-h^{\perp}(X_{0})-Y_{0}+h^ {*}(X_{0}))(Y_{0}^{\perp}-h^{\perp}(X_{0})+Y_{0}-h^{*}(X_{0}))\right].\]

Figure 2: _Boxplots of the empirical excess risk of the SPDNN and NPDNN predictors with \(n=250,500\) and 1000 in DGP3 (a) and DGP4 (b)._
One can see that \(A_{1,n}=A_{1,1,n}+A_{1,2,n}\). We use the Cauchy-Schwarz inequality to obtain

\[|A_{1,1,n}|\leq\sqrt{E(Y_{0}^{\perp}-Y_{0})^{2}}\sqrt{\mathbb{E}(2\widehat{h}_{n} (X_{0})-Y_{0}-Y_{0}^{\perp})^{2}}.\]

We have,

\[(Y_{0}^{\perp}-Y_{0})^{2} =(|Y_{0}|\wedge K_{n})^{2}-2|Y_{0}|(|Y_{0}|\wedge K_{n})+|Y_{0}|^{2}\] \[=(|Y_{0}|-K_{n})^{2}\mathds{1}(|Y_{0}|>K_{n}).\]

Hence,

\[\mathbb{E}(Y_{0}^{\perp}-Y_{0})^{2} =\mathbb{E}\left[(|Y_{0}|-K_{n})^{2}\mathds{1}(|Y_{0}|>K_{n})\right]\] \[\leq\mathbb{E}\left[|Y_{0}|^{2}\mathds{1}(|Y_{0}|>K_{n})\right]. \tag{7.2}\]

From the assumption (4.3) and the independence of \(X_{0}\) and \(\epsilon_{0}\), we get, \(\mathbb{E}(e^{Y_{0}^{2}/(8\rho^{2})})\leq e^{(H^{*})^{2}/(4\rho^{2})}\mathbb{ E}e^{\epsilon_{0}^{2}/(4\rho)}\leq\sqrt{2}e^{(H^{*})^{2}/(4\rho^{2})}\). Also, one can easily see that, \(Y_{0}^{2}\leq 16\rho^{2}e^{Y_{0}^{2}/16\rho^{2}}\) and \(\mathds{1}(|Y_{0}|>K_{n})\leq e^{(Y_{0}^{2}-K_{n})/16\rho^{2}}\). Therefore,

\[\mathbb{E}(Y_{0}^{\perp}-Y_{0})^{2} \leq\mathbb{E}\left[16\rho^{2}e^{Y_{0}^{2}/(16\rho^{2})}e^{Y_{0}^ {2}/(16\rho^{2})-K_{n}^{2}/(16\rho^{2})}\right]\] \[\leq 16\sqrt{2}\rho^{2}e^{(H^{*})^{2}/(4\rho^{2})}e^{-2\log n} \leq 16\sqrt{2}\rho^{2}e^{(H^{*})^{2}/(4\rho^{2})}n^{-2}. \tag{7.3}\]

We also have,

\[\mathbb{E}(2\widehat{h}_{n}(X_{0})-Y_{0}-Y_{0}^{\perp})^{2}\leq 2\mathbb{E}(Y_ {0}^{2})+2\mathbb{E}(2\widehat{h}_{n}(X_{0})-Y_{0}^{\perp})^{2},\]

and,

\[|(2\widehat{h}_{n}(X_{0})-Y_{0}^{\perp})^{2}| \leq 4|\widehat{h}_{n}(X_{0})|^{2}+4|\widehat{h}_{n}(X_{0})||Y_{0} ^{\perp}|+|Y_{0}^{\perp}|^{2}\] \[\leq 4K_{n}^{2}+4K_{n}^{2}+K_{n}^{2}.\]

Since, there exists a constant \(C\) such that \(K_{n}^{2}\leq C\log n\), we have

\[\mathbb{E}(2\widehat{h}_{n}(X_{0})-Y_{0}-Y_{0}^{\perp})^{2} \leq 16\rho^{2}\mathbb{E}(e^{Y^{2}/(8\rho^{2})})+18K_{n}^{2}\] \[\lesssim C_{\rho,H^{*}}\log n. \tag{7.4}\]

Hence, \(|A_{1,1,n}|\lesssim C_{\rho,H^{*}}\log n/n\). Let us deal now with \(A_{1,2,n}\). By using the Cauchy-Schwarz inequality, we get,

\[|A_{1,2,n}|\leq\sqrt{2\mathbb{E}(Y_{0}^{\perp}-Y_{0})^{2}+2\mathbb{E}(h^{ \perp}(X_{0})-h^{*}(X_{0}))^{2}}\times\sqrt{\mathbb{E}(Y_{0}+Y_{0}^{\perp}-h ^{\perp}(X_{0})-h^{*}(X_{0}))^{2}}.\]

In a similar way as in (7.4) we get, \(\mathbb{E}(Y_{0}+Y_{0}^{\perp}-h^{\perp}(X_{0})-h^{*}(X_{0}))^{2}\lesssim_{C_ {\rho},H^{*}}\log n\). Since \(\mathbb{E}(\epsilon_{0})=0\), from the Jensen's inequality, one can easily get,

\[\mathbb{E}\big{[}(h^{\perp}(X_{0})-h^{*}(X_{0}))^{2}\big{]}=\mathbb{E}\left( \mathbb{E}(Y_{0}^{\perp}|X_{0})-\mathbb{E}(Y_{0}|X_{0})\right)^{2}\leq\mathbb{ E}(Y_{0}^{\perp}-Y_{0})^{2}.\]

By using (7.2), one can obtain \(|A_{1,2,n}|\lesssim C_{\rho,H^{*}}\log n/n\). Now, we have,

\[\mathbb{E}[A_{3,n}]=-\frac{2}{n}\sum_{i=1}^{n}\left\{\left[\mathbb{E}[(Y_{i} -\widehat{h}_{n}(X_{i}))^{2}]-\mathbb{E}[(Y_{i}-h^{*}(X_{i}))^{2}]\right]- \left[\mathbb{E}[(Y_{i}^{\perp}-\widehat{h}_{n}(X_{i}))^{2}]-\mathbb{E}[(Y_{i} ^{\perp}-h^{\perp}(X_{i}))^{2}]\right]\right\}.\]
For \(i=1,\cdots,n\), set,

\[A_{3,n,i} =\left[\mathbb{E}[(Y_{i}-\widehat{h}_{n}(X_{i}))^{2}]-\mathbb{E}[(Y_ {i}-h^{*}(X_{i}))^{2}]\right]-\left[\mathbb{E}[(Y_{i}^{\perp}-\widehat{h}_{n}(X _{i}))^{2}]-\mathbb{E}[(Y_{i}^{\perp}-h^{\perp}(X_{i}))^{2}]\right]\] \[A_{3,1,n,i} =\mathbb{E}\left[(Y_{i}^{\perp}-Y_{i})(2\widehat{h}_{n}(X_{i})-Y _{i}-Y_{i}^{\perp})\right]\] \[A_{3,2,n,i} =\mathbb{E}\left[(Y_{i}^{\perp}-h^{\perp}(X_{i})-Y_{i}+h^{*}(X_{i }))(Y_{i}^{\perp}-h^{\perp}(X_{i})+Y_{i}-h^{*}(X_{i}))\right].\]

We have, for \(i=1,\cdots,n\),

\[|A_{3,1,n,i}|\leq\sqrt{\mathbb{E}\left[(Y_{i}^{\perp}-Y_{i})^{2}\right]}\sqrt{ \mathbb{E}\left[(2\widehat{h}_{n}(X_{i})-Y_{i}-Y_{i}^{\perp})^{2}\right]}.\]

By using similar arguments as in \(A_{1,1,n}\), we get for \(i=1,\cdots,n\),

\[|A_{3,1,n,i}|\lesssim C_{\rho,H^{*}}\log n/n.\]

Also, by going as in \(A_{1,2,n}\), it holds for \(i=1,\cdots,n\), that,

\[|A_{3,2,n,i}|\leq\sqrt{\mathbb{E}\left[(Y_{i}^{\perp}-h^{\perp}(X_{i})-Y_{i}+ h^{*}(X_{i}))^{2}\right]}\sqrt{\mathbb{E}\left[(Y_{i}^{\perp}-h^{\perp}(X_{i})+Y _{i}-h^{*}(X_{i}))^{2}\right]},\]

and we can also obtain \(|A_{3,2,n,i}|\lesssim C_{\rho,H^{*}}\log n/n\). Thus, for \(i=1,\cdots,n\),

\[|A_{3,n,i}|\lesssim C_{\rho,H^{*}}\log n/n.\]

Hence,

\[|\mathbb{E}(A_{3,n})|=|-\frac{2}{n}\sum_{i=1}^{n}A_{3,n,i}|\leq\frac{2}{n} \sum_{i=1}^{n}|A_{3,n,i}|\lesssim C_{\rho,H^{*}}\log n/n.\]

For \(A_{2,n}\), define \(\Delta h(Z_{0})\coloneqq(Y_{0}^{\perp}-h(X_{0}))^{2}-(Y_{0}^{\perp}-h^{\perp }(X_{0}))^{2}\) with \(Z_{0}\coloneqq(X_{0},Y_{0})\) for \(h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\). Let \(\alpha>0\), we can write

\[P(A_{2,n}>\alpha) \leq P\left(\sup_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)} \frac{\mathbb{E}(\Delta h(Z_{0}))-\frac{1}{n}\sum_{i=1}^{n}\Delta h(Z_{i})}{ \alpha+2J_{\lambda_{n},\tau_{n}}+\mathbb{E}(\Delta h(Z_{0}))}\geq\frac{1}{2}\right)\] \[\leq\sum_{j=0}^{\infty}P\left(\sup_{h\in\mathcal{H}_{n,j,\alpha} }\frac{\mathbb{E}(\Delta h(Z_{0}))-\frac{1}{n}\sum_{1=1}^{n}\Delta h(Z_{i})}{2 ^{j}\alpha+\mathbb{E}(\Delta h(Z_{0}))}\geq\frac{1}{2}\right),\]

where

\[\mathcal{H}_{n,j,\alpha}\coloneqq\Big{\{}h\in\mathcal{H}_{\sigma}(L_{n},N_{n}, B_{n},F):2^{j-1}\mathds{1}(j\neq 0)\alpha\leq J_{\lambda_{n},\tau_{n}}(h)\leq 2^{j} \alpha\Big{\}}. \tag{7.5}\]
Indeed,

\[P(A_{2,n}>\alpha)\] \[=P\Bigg{(}\left[\mathbb{E}\left(Y_{0}^{\perp}-\widehat{h}_{n}(X_{0}) \right)^{2}-\mathbb{E}\left(Y_{0}^{\perp}-h^{\perp}(X_{0})\right)^{2}\right]-2 \left[\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-\widehat{h}_{n}(X_{i}) \right)^{2}-\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-h^{\perp}(X_{i}) \right)^{2}\right]\] \[-2J_{\lambda_{n},\tau_{n}}(\widehat{h}_{n})>\alpha\Bigg{)}\] \[\leq P\Bigg{(}2\left[\mathbb{E}\left(Y_{0}^{\perp}-\widehat{h}_{n }(X_{0})\right)^{2}-\mathbb{E}\left(Y_{0}^{\perp}-h^{\perp}(X_{0})\right)^{2} \right]-\left[\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-\widehat{h}_{n}(X_{i })\right)^{2}-\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-h^{\perp}(X_{i}) \right)^{2}\right]\] \[>\alpha+2J_{\lambda_{n},\tau_{n}}(\widehat{h}_{n})+\mathbb{E} \left(Y_{0}^{\perp}-\widehat{h}_{n}(X_{0})\right)^{2}-\mathbb{E}\left(Y_{0}^ {\perp}-h^{\perp}(X_{0})\right)^{2}\Bigg{)}\] \[\leq P\Bigg{(}\exists h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F ):\left[\mathbb{E}\left(Y_{0}^{\perp}-h(X_{0})\right)^{2}-\mathbb{E}\left(Y_ {0}^{\perp}-h^{\perp}(X_{0})\right)^{2}\right]\] \[-\left[\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-h(X_{i}) \right)^{2}-\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i}^{\perp}-h^{\perp}(X_{i}) \right)^{2}\right]\] \[>\frac{1}{2}\left(\alpha+2J_{\lambda_{n},\tau_{n}}(h(X_{0}))+ \mathbb{E}\left(Y_{0}^{\perp}-h(X_{0})\right)^{2}-\mathbb{E}\left(Y_{0}^{ \perp}-h^{\perp}(X_{0})\right)^{2}\right)\Bigg{)}\]

\[\leq P\Bigg{(}\exists h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F):\] \[\leq\sum_{j=0}^{\infty}P\Bigg{(}\sup_{h\in\mathcal{H}_{\sigma}(L_ {n},N_{n},B_{n},F)}\] \[\leq\sum_{j=0}^{\infty}P\Bigg{(}\sup_{h\in\mathcal{H}_{\sigma}(L_ {n},N_{n},B_{n},F)}\] \[\leq\sum_{j=0}^{\infty}P\Bigg{(}\sup_{h\in\mathcal{H}_{n,j,\alpha} }\frac{\mathbb{E}\Delta h(X_{0})-\frac{1}{n}\sum_{i=1}^{n}\Delta h(X_{i})}{ \left(2^{j}\alpha+2J_{\lambda_{n},\tau_{n}}(h(X_{0}))+\mathbb{E}\Delta h(X_{0 })\right)}\geq\frac{1}{2}\Bigg{)}.\]

For \(h\in\mathcal{H}_{n,j,\alpha}\), we have

\[P\Bigg{(}\sup_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}\frac{\mathbb{E }\Delta h(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}\Delta h(Z_{i})}{\left(\alpha+2J_{ \lambda_{n},\tau_{n}}(h(X_{0}))+\mathbb{E}\Delta h(Z_{0})\right)}\geq\frac{1} {2}\Bigg{)}\leq\sum_{j=0}^{\infty}P\Bigg{(}\sup_{h\in\mathcal{H}_{n,j,\alpha} }\frac{\mathbb{E}\Delta h(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}\Delta h(Z_{i})}{ \left(2^{j}\alpha+\mathbb{E}\Delta h(Z_{0})\right)}\geq\frac{1}{2}\Bigg{)}.\]

One can easily show that,

\[\mathbb{E}(\Delta h(Z_{0})] =\mathbb{E}[(h(X_{0})-h^{\perp}(X_{0}))^{2}]-\mathbb{E}[(h^{ \perp}(X_{0}))^{2}]+2\mathbb{E}[(h^{\perp}(X_{0}))^{2}]-\mathbb{E}[(h^{\perp} )^{2}(X_{0})]\] \[=\mathbb{E}[(h(X_{0})-h^{\perp}(X_{0}))^{2}]\geq 0.\]
Hence,

\[\sum_{j=1}^{\infty}P\Bigg{(}\sup_{h\in\mathcal{H}_{n,j,\alpha}}\frac{ \mathbb{E}\Delta h(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}\Delta h(Z_{i})}{(2^{j} \alpha+\mathbb{E}\Delta h(Z_{0}))}\geq\frac{1}{2}\Bigg{)}\leq\sum_{j=1}^{ \infty}P\Bigg{(}\sup_{h\in\mathcal{H}_{n,j,\alpha}}\frac{\mathbb{E}\Delta h(Z_{ 0})-\frac{1}{n}\sum_{i=1}^{n}\Delta h(Z_{i})}{2^{j}\alpha}\geq\frac{1}{2} \Bigg{)}\] \[\leq\sum_{j=1}^{\infty}P\Bigg{(}\sup_{h\in\mathcal{H}_{n,j,\alpha }}\Bigg{\{}\mathbb{E}\Delta h(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}\Delta h(Z_{i}) \Bigg{\}}\geq\frac{2^{j}\alpha}{2}\Bigg{)}\leq\sum_{j=1}^{\infty}P\Bigg{(}\sup _{g\in\mathcal{G}_{n,j,\alpha}}\Bigg{\{}\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_ {i=1}^{n}g\Bigg{\}}\geq\frac{2^{j}\alpha}{2}\Bigg{)},\]

where,

\[\mathcal{G}_{n,j,\alpha}\coloneqq\Big{\{}\Delta(h):\mathbb{R}^{d}\times \mathbb{R}\rightarrow\mathbb{R},h\in\mathcal{H}_{n,j,\alpha}\Big{\}}. \tag{7.6}\]

Let \(h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\). Consider the function \(g(x,y)=\Delta h(x,y)\). One can easily prove that, \(g\) is Lipschitz with Lipschitz coefficient \(\max\big{(}2FLip(h)+2K_{n}Lip(h),6K_{n}+2F\big{)}\). Therefore, one can get that, the process \((g(Z_{t}))_{t\in\mathbb{Z}}\) is also \(\psi\)-weakly dependent. Thus, we have from [8] (see also [5]),

\[P\left\{\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})> \varepsilon\right\} =P\left\{\sum_{i=1}^{n}\left(\mathbb{E}[g(X_{0},Y_{0})]-g(X_{i},Y_ {i})\right)\geq n\varepsilon\right\}\] \[\leq P\left\{\left|\sum_{i=1}^{n}\left(\mathbb{E}[g(X_{0},Y_{0})] -g(X_{i},Y_{i})\right)\right|\geq n\varepsilon\right\}\] \[\leq C_{3}\log n\exp\left(-\frac{n^{2}\varepsilon^{2}}{A_{n}^{ \prime}+B_{n}^{\prime}(n\varepsilon)^{\nu}}\right)\] \[\leq C_{3}\exp\left(\log\log n-\frac{n^{2}\varepsilon^{2}}{A_{n} ^{\prime}+B_{n}^{\prime}(n\varepsilon)^{\nu}}\right). \tag{7.7}\]

For a some constant \(C_{3}>0\), any sequence \((A_{n}^{\prime})_{n\in\mathbb{N}}\), satisfying \(A_{n}^{\prime}\geq\mathbb{E}\left[\left(\sum_{i=1}^{n}\left(g(X_{i},Y_{i})- \mathbb{E}[g(X_{0},Y_{0})]\right)\right)^{2}\right]\) and \(B_{n}=\frac{n^{3/4}\log n}{A_{n}^{\prime}}\). Let \(l=\mathcal{N}(\varepsilon,\mathcal{G}_{n,j,\alpha},\|\cdot\|_{\infty})\). For n large enough, we have

\[P\left\{\sup_{g\in\mathcal{G}_{n,j,\alpha}}\Bigg{[}\mathbb{E}[g (Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\Bigg{]}>\varepsilon\right\} \leq C_{3}\sum_{i=1}^{l}\exp\left(\log\log n-\frac{n^{2} \varepsilon^{2}/4}{A_{n}^{\prime}+B_{n}^{\prime}(n\varepsilon/2)^{\nu}}\right)\] \[\leq C_{3}\cdot l\exp\left(\log\log n-\frac{n^{2}\varepsilon^{2}/ 4}{A_{n}^{\prime}+B_{n}^{\prime}(n\varepsilon/2)^{\nu}}\right)\] \[\leq C_{3}\mathcal{N}(\varepsilon,\mathcal{G}_{n,j,\alpha})\exp \left(\log\log n-\frac{n^{2}\varepsilon^{2}/4}{A_{n}^{\prime}+B_{n}^{\prime}( n\varepsilon/2)^{\nu}}\right).\]

In [19], we have

\[\mathcal{N}(\varepsilon,\mathcal{G}_{n,j,\alpha},\|\cdot\|_{\infty})\leq \mathcal{N}(\frac{\varepsilon}{4K_{n}},\mathcal{H}_{n,j,\alpha},\|\cdot\|_{ \infty}). \tag{7.8}\]

One can easily see that,

\[\mathcal{H}_{n,j,\alpha}\subset\left\{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{ n},F,\frac{2^{j}\alpha}{\lambda_{n}}):\|\theta(h)\|_{\text{clip},\tau_{n}}\leq\frac{2^{j} \alpha}{\lambda_{n}}\right\}. \tag{7.9}\]

Thus, in [19], we have the following inequality

\[\mathcal{N}(\varepsilon,\mathcal{G}_{n,j,\alpha},\|\cdot\|_{\infty}) \leq\mathcal{N}(\frac{\varepsilon}{4K_{n}},\mathcal{H}_{n,j,\alpha },\|\cdot\|_{\infty})\] \[\leq\mathcal{N}(\frac{\varepsilon}{4K_{n}},\mathcal{H}_{\sigma}(L_ {n},N_{n},B_{n},F,\frac{2^{j}\alpha}{\lambda_{n}}),\|\cdot\|_{\infty})\] \[\leq\exp\left(\,2\frac{2^{j}\alpha}{\lambda_{n}}(L_{n}+1)\log \left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{\varepsilon}{4K_{n}}-\tau_{n}(L_{n}+1) ((N_{n}+1)B_{n})^{L_{n}+1}}\right)\right). \tag{7.10}\]
We have,

\[P\Big{\{}\sup_{g\in\mathcal{G}_{n,j,\alpha}}\left[\mathbb{E}[g(Z_{0}) ]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\varepsilon\Big{\}}\] \[\leq C_{3}\exp\left(2\frac{2^{j}\alpha}{\lambda_{n}}(L_{n}+1)\log \left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{\varepsilon}{4K_{n}}-\tau_{n}(L_{n} +1)((N_{n}+1)B_{n})^{L_{n}+1}}\right)+\log\log n-\frac{n^{2}\varepsilon^{2}/4}{ A_{n}^{\prime}+B_{n}^{\prime}(n\varepsilon/2)^{\nu}}\right). \tag{7.11}\]

Hence,

\[\sum_{j=1}^{\infty}P\left\{\sup_{g\in\mathcal{G}_{n,j,\alpha}} \left[\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\varepsilon\right\}\] \[\leq C_{3}\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{ \lambda_{n}}(L_{n}+1)\log\left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{ \varepsilon}{4K_{n}}-\tau_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\right)+\log \log n-\frac{n^{2}\varepsilon^{2}/4}{A_{n}^{\prime}+B_{n}^{\prime}(n\varepsilon /2)^{\nu}}\right). \tag{7.12}\]

Let \(\varepsilon=\frac{2^{j}\alpha}{2}\), we have

\[\sum_{j=1}^{\infty}P\Big{\{}\sup_{g\in\mathcal{G}_{n,j,\alpha}} \left[\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\frac{2^{ j}\alpha}{2}\Big{\}}\] \[\leq C_{3}\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{ \lambda_{n}}(L_{n}+1)\log\left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{2^{j} \alpha}{8K_{n}}-\tau_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\right)+\log\log n -\frac{(n2^{j}\alpha)^{2}/16}{A_{n}^{\prime}+B_{n}^{\prime}(n2^{j}\alpha/4)^ {\nu}}\right)\] \[\leq C_{3}\exp(\log\log n)\sum_{j=1}^{\infty}\exp\left(2\frac{2^{ j}\alpha}{\lambda_{n}}(L_{n}+1)\log\left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{ \frac{2^{j}\alpha}{8K_{n}}-\tau_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}} \right)-\frac{(n2^{j}\alpha)^{2}/16}{A_{n}^{\prime}+B_{n}^{\prime}(n2^{j} \alpha/4)^{\nu}}\right). \tag{7.13}\]

By with choice \(A_{n}^{\prime}=nC\) and \(B_{n}^{\prime}=\log(\frac{n}{n^{1/4C}})\), we have

\[\sum_{j=1}^{\infty}P\Big{\{}\sup_{g\in\mathcal{G}_{n,j,\alpha}} \left[\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\frac{2^{ j}\alpha}{2}\Big{\}}\] \[\leq C_{3}\exp(\log\log n)\] \[\leq C_{3}\exp(\log\log n)\] \[\times\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{\lambda_{n} }(L_{n}+1)\log\left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{2^{j} \alpha}{8K_{n}}-\tau_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\right)-\frac{(n2 ^{j}\alpha)^{2}/16}{nC+\log(\frac{n}{n^{1/4C}})(n2^{j}\alpha/4)^{\nu}}\right). \tag{7.14}\]

Let,

\[\log(\frac{n}{n^{1/4}C})(n2^{j}\alpha/8)^{\nu}>nC\Longrightarrow\alpha>\frac{ 8}{n}\left(\frac{nC}{\log(\frac{n}{n^{1/4C}})}\right)^{1/\nu}\coloneqq\alpha_ {n}. \tag{7.15}\]

We can easily see that from (7.15)

\[-\frac{(n2^{j}\alpha)^{2}/16}{nC+\log(\frac{n}{n^{1/4C}})(n2^{j}\alpha/8)^{ \nu}}\leq-\frac{(n2^{j}\alpha)^{2}/16}{2\log(\frac{n}{n^{1/4C}})(n2^{j}\alpha/4 )^{\nu}}.\]
We can see also asymptotically \(\alpha_{n}>1\).

**Step \(\mathbf{1}:\alpha>\alpha_{n}\)** Under the assumption

\[\tau_{n}\leq\frac{1}{16K_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}},\]

And for \(2^{j}\alpha>1\), we have

\[\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{\lambda_{n}}(L_{ n}+1)\log\left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{2^{j}\alpha-\tau_{n}(L_{n}+1)((N_{n}+1 )B_{n})^{L_{n}+1}}\right)-\frac{(n2^{j}\alpha)^{2}/16}{nC+\log(\frac{n}{n^{1/4 }C})(n2^{j}\alpha/4)^{\nu}}\right)\] \[\leq\exp\left(2\frac{2^{j}\alpha}{\lambda_{n}}(L_{n}+1)\log\left( 16K_{n}(L_{n}+1)(N_{n}+1)B_{n}\right)-\frac{(n2^{j}\alpha)^{2-\nu}/16}{2\log( \frac{n}{n^{1/4}C})/4^{\nu}}\right). \tag{7.16}\]

For \(\alpha>\alpha_{n}\) we have \(2^{j}\alpha<((2\alpha)^{2-\nu})^{j}\) Thus,

\[\sum_{j=1}^{\infty}P\Big{\{}\sup_{g\in\mathcal{G}_{n,j,\alpha}} \left[\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\frac{2^{j }\alpha}{2}\Big{\}}\] \[\leq\sum_{j=1}^{\infty}\exp\left(2^{j}\alpha\left(\frac{2}{ \lambda_{n}}(L_{n}+1)\log\left(16K_{n}(L_{n}+1)(N_{n}+1)B_{n}\right)-\frac{n^{ 2-\nu}/32}{\log(\frac{n}{n^{1/4}C})/4^{\nu}}\right)\right). \tag{7.17}\]

We can see that for n large enough,

\[\delta_{n,1}\coloneqq\frac{2}{\lambda_{n}}(L_{n}+1)\log\left(16K_{n}(L_{n}+1) (N_{n}+1)B_{n}\right)\leq\frac{n^{2-\nu}/32}{2\log(\frac{n}{n^{1/4}C})/4^{\nu}}.\]

Thus,

\[\sum_{j=1}^{\infty}P\Big{\{}\sup_{g\in\mathcal{G}_{n,j,\alpha}}\left[\mathbb{ E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\frac{2^{j}\alpha}{2} \Big{\}}\leq C_{3}\log n\sum_{j=1}^{\infty}\exp\left(-2^{j}\frac{n^{2-\nu}}{64 \log(\frac{n}{n^{1/4}C})/4^{\nu}}\alpha\right).\]

Let

\[\beta_{n}\coloneqq\frac{n^{2-\nu}}{64\log\left(\frac{n}{n^{1/4}C}\right)/4^{ \nu}}\alpha,\]

we have

\[\sum_{j=0}^{\infty}\exp\left(-\beta_{n}(2^{j}\right)\leq\exp\left(-\beta_{n} \right)\sum_{j=0}^{\infty}\left(2^{-\frac{\beta_{n}}{\log 2}}\right)^{j}\leq\exp \left(-\beta_{n}\right)\left(\frac{1}{\begin{array}{c}\\ \\ \\ 1-2\end{array}\frac{\beta_{n}}{\log 2}}\right)\leq\exp\left(-\beta_{n}\right) \tag{7.18}\]

By applying (7.18), we have

\[\sum_{j=1}^{\infty}\exp\left(-2^{j}\frac{n^{2-\nu}}{64\log(\frac{n}{n^{1/4}C} )/4^{\nu}}\alpha\right)\leq\exp\left(-\frac{n^{2-\nu}}{64\log(\frac{n}{n^{1/4} C})/4^{\nu}}\alpha\right). \tag{7.19}\]

Hence

\[P(A_{2,n}>\alpha)\lesssim C_{3}\log n\sum_{j=1}^{\infty}\exp\left(-2^{j}\frac{ n^{2-\nu}}{64\log(\frac{n}{n^{1/4}C})/4^{\nu}}\alpha\right)\lesssim C_{3}\log n \exp\left(-\frac{n^{2-\nu}}{64\log(\frac{n}{n^{1/4}C})/4^{\nu}}\alpha\right), \tag{7.20}\]
for \(\alpha\geq\alpha_{n}\).

\[\int_{\alpha_{n}}^{\infty}P(A_{2,n}>\alpha)d\alpha \leq\int_{\alpha_{n}}^{\infty}C_{3}\log n\exp\left(-\frac{n^{2-\nu} }{64\log(\frac{n}{n^{1/4}C})/4^{\nu}}\alpha\right)d\alpha\] \[\leq 64C_{3}\log n\frac{\log(\frac{n}{n^{1/4}C})}{n^{2-\nu}}\exp \left(-\frac{n^{2-\nu}}{64\log(\frac{n}{n^{1/4}C})/4^{\nu}}\times\frac{8}{n} \left(\frac{nC}{\log\frac{n}{n^{1/4}C}}\right)\right).\]

Let

\[nC<\log(\frac{n}{n^{1/4}C})(n2^{j}\alpha/4)^{\nu}\Longrightarrow\alpha<\frac{ 8}{n}\left(\frac{nC}{\log(\frac{n}{n^{1/4}C})^{\nu}}\right)^{1/\nu}\coloneqq \alpha_{n}, \tag{7.21}\]

Thus,

\[-\frac{(n2^{j}\alpha)^{2}/16}{nC+\log(\frac{n}{n^{1/4}C})(n2^{j}\alpha/4)^{ \nu}}\leq-\frac{(n2^{j}\alpha)^{2}/16}{2nC}.\]

We can see that asymptotically \(\alpha_{n}>1,\ 2^{j}>1\). Thus,

**Step2**: \(1\leq\alpha\leq\alpha_{n}\)

Thus we have

\[\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{\lambda_{n}}(L_{ n}+1)\log\left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{2^{j}\alpha}{8K_{n}}-\tau_{n} (L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\right)-\frac{(n2^{j}\alpha)^{2}/16}{nC+ \log(\frac{n}{n^{1/4}C})(n2^{j}\alpha/4)^{\nu}}\right)\] \[\leq\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{\lambda_{n}} (L_{n}+1)\log\left(16K_{n}(L_{n}+1)(N_{n}+1)B_{n}\right)-\frac{(n2^{j}\alpha) ^{2}/16}{2nC}\right). \tag{7.22}\]

For \(1\leq\alpha\leq\alpha_{n}\), we have \(2^{j}\alpha\leq(2^{j}\alpha)^{2}\)

\[\sum_{j=1}^{\infty}P\left\{\sup_{g\in\mathcal{G}_{n,j,\alpha}} \left[\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\frac{2^{j }\alpha}{2}\right\}\] \[\leq C_{3}\log n\sum_{j=1}^{\infty}\exp\left(2^{j}\alpha\left( \frac{2}{\lambda_{n}}(L_{n}+1)\log\left(16K_{n}(L_{n}+1)(N_{n}+1)B_{n}\right)- \frac{n^{2}}{32nC}\right)\right). \tag{7.23}\]

We can see that for n large enough

\[\delta_{n,2}\coloneqq\frac{2}{\lambda_{n}}(L_{n}+1)\log\left(16K_{n}(L_{n}+1) (N_{n}+1)B_{n}\right)\leq\frac{n^{2}/16}{4nC}.\]

Thus,

\[\sum_{j=1}^{\infty}P\Big{\{}\sup_{g\in\mathcal{G}_{n,j,\alpha}}\left[\mathbb{ E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\frac{2^{j}\alpha}{2} \Big{\}}\leq C_{3}\log n\sum_{j=1}^{\infty}\exp\left(-2^{j}\frac{n}{64C} \alpha\right). \tag{7.24}\]

Set \(\beta_{n}^{{}^{\prime}}=\frac{n}{64C}\) using similar arguments as (7.18) we have

\[P(A_{2,n}>\alpha)\lesssim C_{3}\log n\sum_{j=1}^{\infty}\exp\left(-2^{j}\frac {n}{64C}\alpha\right)\lesssim C_{3}\log n\exp\left(-\frac{n}{64C}\alpha\right),\]
for \(1\leq\alpha\leq\alpha_{n}\). Which implies

\[\int_{1}^{\alpha_{n}}P(A_{2,n}>\alpha) \leq C_{3}\log n\int_{1}^{\alpha_{n}}\exp\left(-\frac{n}{64C}\alpha \right)d\alpha\] \[\leq\frac{64CC_{3}\log n}{n}\exp\left(-\frac{n}{64C}\times\frac{8 }{n}\left(\frac{nC}{\log(\frac{n}{n^{1/4}C})}\right)^{1/\nu}\right)-\frac{64CC_ {3}\log n}{n}\exp\left(-\frac{n}{64C}\right).\]

**Step3**: \(0<\alpha\leq 1<\alpha_{n}\).

Recall the the assumption

\[\tau_{n}\leq\frac{\beta_{n}}{16K_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}},\]

with the conditions \(\beta_{n}:=(\log n)^{\nu_{5}}/n^{\nu_{6}}\) for some \(\nu_{5},\nu_{6}>0\) and

\[\Big{(}\nu_{6}<1/2,\nu_{4}+\nu_{6}<1\Big{)}\text{ or }\Big{(}\nu_{6}<1/2,\nu_{4} +\nu_{6}=1,\nu_{5}>1-\nu_{3}\Big{)}. \tag{7.25}\]

We have for all \(\alpha\in(\beta_{n},1)\),

\[\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{\lambda_{n}}(L_ {n}+1)\log\left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{2^{j}\alpha}{ \lambda_{n}}-\tau_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\right)-\frac{(n2^{j} \alpha)^{2}/16}{nC+\log(\frac{n}{n^{1/4}C})(n2^{j}\alpha/8)^{\nu}}\right)\] \[\leq\sum_{j=1}^{\infty}\exp\left(\frac{2(L_{n}+1)2^{j}\alpha}{ \lambda_{n}}\log\left(\frac{16K_{n}(L_{n}+1)(N_{n}+1)B_{n}}{\beta_{n}}\right)- \frac{(n2^{j}\alpha)^{2}}{32nC}\right)\] \[\leq\sum_{j=1}^{\infty}\exp\bigg{[}-(2^{j})^{2}\bigg{(}\frac{n \alpha^{2}}{32C}-\frac{2(L_{n}+1)\alpha}{\lambda_{n}}\log\left(\frac{16K_{n}( L_{n}+1)(N_{n}+1)B_{n}}{\beta_{n}}\right)\bigg{)}\bigg{]}\] \[\leq\sum_{j=1}^{\infty}\exp\big{(}-4^{j}\phi_{n}(\alpha)\big{)} \tag{7.26}\]

with

\[\phi_{n}(\alpha)\coloneqq\frac{n\alpha^{2}}{32C}-\frac{2(L_{n}+1)\alpha}{ \lambda_{n}}\log\left(\frac{16K_{n}(L_{n}+1)(N_{n}+1)B_{n}}{\beta_{n}}\right) \text{ for all }\beta_{n}<\alpha\leq 1<\alpha_{n}. \tag{7.27}\]

Since \(K_{n},L_{n},N_{n},B_{n}\geq 1\), we have \(16K_{n}(L_{n}+1)(N_{n}+1)B_{n}/\beta_{n}>1\) and therefore,

\[\phi_{n}(\alpha)\geq\frac{n\beta_{n}^{2}}{32C}-\frac{2(L_{n}+1)\beta_{n}}{ \lambda_{n}}\log\left(\frac{16K_{n}(L_{n}+1)(N_{n}+1)B_{n}}{\beta_{n}}\right) =\phi_{n}(\beta_{n}),\text{ \ for all }\beta_{n}<\alpha\leq 1<\alpha_{n}. \tag{7.28}\]

With the assumptions \(\lambda_{n}\asymp(\log n)^{\nu_{3}}/n^{\nu_{4}}\) for some \(0<\nu_{4}<1\), \(\nu_{6}<1/2\) and (7.25), we get,

\[\varphi_{n}\coloneqq\phi_{n}(\beta_{n})\underset{n\to\infty}{\longrightarrow}\infty. \tag{7.29}\]

One can see that,

\[\exp(-\beta 4^{j})\leq\exp(-\beta)\times\exp(-\beta j),\text{for all }\beta\geq 0,\ j \in\mathbb{N}.\]
Hence, it follows from (7.26), (7.28) and (7.29) that, for all \(\alpha\in(\beta_{n},1)\), and for \(n\) large enough,

\[\sum_{j=1}^{\infty}\exp\left(2\frac{2^{j}\alpha}{\lambda_{n}}(L_{n} +1)\log\left(\frac{16K_{n}(L_{n}+1)(N_{n}+1)B_{n}}{2^{j}\alpha}\right)-\frac{(n 2^{j}\alpha)^{2}/16}{nC+\log(\frac{n}{n^{1/4}C})(n2^{j}\alpha/8)^{\nu}}\right)\] \[\leq\sum_{j=1}^{\infty}\exp\big{(}-4^{j}\varphi_{n}\big{)}\leq \sum_{j=1}^{\infty}\exp(-\varphi_{n})\exp(-\varphi_{n}j)\leq\frac{\exp(-2 \varphi_{n})}{1-\exp(-\varphi_{n})}\leq 2\exp(-2\varphi_{n}).\]

Therefore, for sufficiently large \(n\),

\[\int_{0}^{1}P(A_{2,n}>\alpha)d\alpha\leq\beta_{n}+\int_{\beta_{n}}^{1}P(A_{2,n }>\alpha)d\alpha\leq\beta_{n}+2\exp(-2\varphi_{n})\leq 2\beta_{n},\]

where \(\beta_{n}\) is defined above, and satisfies (7.25). Thus, for \(\nu_{5}>0,\ \nu_{6}<1/2\), we have

\[\mathbb{E}[A_{2,n}] =\int_{0}^{\infty}P(A_{2,n}>\alpha)d\alpha\] \[\leq\int_{0}^{\beta_{n}}P(A_{2,n}>\alpha)d\alpha+\int_{\beta_{n}} ^{1}P(A_{2,n}>\alpha)d\alpha+\int_{\alpha_{n}}^{\alpha_{n}}P(A_{2,n}>\alpha)d\alpha +\int_{\alpha_{n}}^{\infty}P(A_{2,n}>\alpha)d\alpha\] \[\leq 2\beta_{n}+\frac{64CC_{3}\log n}{n}\exp\left(-\frac{n}{64C} \times\frac{8}{n}\left(\frac{nC}{\log(\frac{n}{n^{1/4}C})}\right)^{1/\nu} \right)-\frac{64CC_{3}\log n}{n}\exp\left(-\frac{n}{64C}\right)\] \[+32C_{3}\log n\frac{\log(\frac{n}{n^{1/4}C})}{n^{2-\nu}}\exp \left(-\frac{n^{2-\nu}}{32\log(\frac{n}{n^{1/4}C})/8^{\nu}}\times\frac{8}{n} \left(\frac{nC}{\log\frac{n}{n^{1/4}C}}\right)\right)\] \[\lesssim 2\beta_{n}. \tag{7.30}\]

For \(A_{4,n}\) we choose a neural network function \(h_{n}^{0}\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\) such that

\[\|h_{n}^{0}-h^{*}\|_{2,P_{X_{0}}}^{2}+J_{\lambda_{n},\tau_{n}}(h_{n}^{0})\leq \inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}\Big{[}\|h-h^{*}\|_{2,P_{X _{0}}}^{2}+J_{\lambda_{n},\tau_{n}}(h)\Big{]}+n^{-1}.\]

Then by the basic inequality

\[\frac{1}{n}\sum_{i=1}^{n}\big{(}Y_{i}-\widehat{h}_{n}(X_{i})\big{)}^{2}+J_{ \lambda_{n},\tau_{n}}(\widehat{h}_{n})\leq\frac{1}{n}\sum_{i=1}^{n}\left(Y_{i }-h(X_{i})\right)^{2}+J_{\lambda_{n},\tau_{n}}(h),\]

for any \(h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\), we have

\[A_{4,n} =2\Big{[}\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-\widehat{h}_{n}(X_{i}))^ {2}-\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-h_{n}^{0}(X_{i}))^{2}\Big{]}+2J_{\lambda_{n },\tau_{n}}(\widehat{h}_{n})\] \[+2\Big{[}\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-h_{n}^{0}(X_{i}))^{2}- \frac{1}{n}\sum_{i=1}^{n}(Y_{i}-h^{*}(X_{i}))^{2}\Big{]} \tag{7.31}\] \[\leq 2J_{\lambda_{n},\tau_{n}}(h_{n}^{0})+2\Big{[}\frac{1}{n}\sum_ {i=1}^{n}(Y_{i}-h_{n}^{0}(X_{i}))^{2}-\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-h^{*}(X_ {i}))^{2}\Big{]}\]

and we have

\[\mathbb{E}[A_{4,n}]\leq 2J_{\lambda_{n},\tau_{n}}(h_{n}^{0})+\frac{2}{n}\sum_ {i=1}^{n}\Big{[}\mathbb{E}(Y_{i}-h_{n}^{0}(X_{i}))^{2}-\mathbb{E}(Y_{i}-h^{*}(X _{i}))^{2}\Big{]},\]

One can see that for \(i=1,\ldots,n\),

\[\mathbb{E}[(Y_{i}-h_{n}^{0}(X_{i}))^{2}]-\mathbb{E}[(Y_{i}-h^{*}(X_{i}))^{2}] \leq\|h_{n}^{0}-h^{*}\|_{2,P_{X_{0}}}^{2}.\]
Thus,

\[\mathbb{E}[A_{4,n}]\leq 2J_{\lambda_{n},\tau_{n}}(h_{n}^{0})+2\|h_{n}^{0}-h^{*} \|_{2,P_{X_{0}}}^{2}\quad\leq 2\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)} \Bigl{[}\|h-h^{*}\|_{2,P_{X_{0}}}^{2}+J_{\lambda_{n},\tau_{n}}(h)\Bigr{]}+2n^{-1}.\]

Thus,

\[\mathbb{E}\Bigl{[}\|\widehat{h}_{n}-h^{*}\|_{2,P_{X_{0}}}^{2} \Bigr{]}=\mathbb{E}\left[\sum_{i=1}^{4}A_{i,n}\right]\lesssim 3\frac{\log n}{n}+2 \frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}+2\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N _{n},B_{n},F)}\Bigl{[}\|h-h^{*}\|_{2,P_{X_{0}}}^{2}\Bigr{]}+2n^{-1}\] \[\lesssim 2\Biggl{(}\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}\{\|h-h^{*}\|_{2,P_{X_{0}}}^{2}+\lambda_{n}\|\theta(h)\|_{\mathrm{clip}, \tau_{n}}\}\vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}\Biggr{)}. \tag{7.32}\]

This completes the proof of the theorem.

### Proof of Theorem 4.2

Let \(\epsilon_{n}=n^{-\nu_{4}/(\kappa+2)}\). From the Theorem (4.1), the assumption (4.6) and the fact that \(\|\theta(h)\|_{\mathrm{clip},\tau}\leq\|\theta(h)\|_{0}\) for any \(\tau>0\), we have that for any \(h^{*}\in\mathcal{H}^{*}\),

\[\sup_{h\in\mathcal{H}_{\rho,H^{*}}:h^{*}\in\mathcal{H}^{*}} \mathbb{E}\Bigl{[}\|\widehat{h}_{n}-h^{*}\|_{2,P_{X_{0}}}^{2}\Bigr{]}\] \[\lesssim\sup_{h\in\mathcal{H}_{\rho,H^{*}}:h^{*}\in\mathcal{H}^{*} }\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F,C\epsilon_{n}^{-\kappa}( \log n)^{\tau})}\Bigl{\{}\|h-h^{*}\|_{2,P_{X_{0}}}^{2}+\lambda_{n}\|\theta(h) \|_{\mathrm{clip},\tau_{n}}\Bigr{\}}\vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}\] \[\lesssim\sup_{h\in\mathcal{H}_{\rho,H^{*}}:h^{*}\in\mathcal{H}^{*} }\Bigl{[}\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F,C\epsilon_{n}^{- \kappa}(\log n)^{\tau})}\|h-h^{*}\|_{2,P_{X_{0}}}^{2}+\lambda_{n}\epsilon_{n}^ {-\kappa}(\log n)^{\tau}\Bigr{]}\vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}\] \[\lesssim\sup_{h\in\mathcal{H}_{\rho,H^{*}}:h^{*}\in\mathcal{H}^{* }}\Bigl{[}\epsilon_{n}^{2}+\frac{(\log n)^{\nu_{3}}}{n^{\nu_{4}}}\epsilon_{n}^ {-\kappa}(\log n)^{\tau}\Bigr{]}\vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}\] \[\lesssim\frac{(\log n)^{\tau+\nu_{3}}}{n^{2\nu_{4}/(\kappa+2)}} \vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}. \tag{7.33}\]

Thus, the theorem follows.

### Proof of Theorem 5.1

Let us decompose \(\mathcal{E}_{Z_{0}}(\widehat{h}_{n})\) as as follows.

\[\mathcal{E}_{Z_{0}}(\widehat{h}_{n})=\mathbb{E}[\ell(Y_{0}\widehat{h}_{n}(X_{0 }))]-\mathbb{E}[\ell(Y_{0}h_{\ell}^{*}(X_{0}))]\coloneqq B_{1,n}+B_{2,n}, \tag{7.34}\]

where

\[B_{1,n} =\Bigl{[}\mathbb{E}[\ell(Y_{0}\widehat{h}_{n}(X_{0}))]-\mathbb{E} [\ell(Y_{0}h_{\ell}^{*}(X_{0}))]\Bigr{]}-\frac{2}{n}\sum_{i=1}^{n}\Big{[}\ell(Y _{i}\widehat{h}_{n}(X_{i}))-\ell(Y_{i}h_{\ell}^{*}(X_{i}))\Bigr{]}-2J_{\lambda _{n},\tau_{n}}(\widehat{h}_{n});\] \[B_{2,n} =\frac{2}{n}\sum_{i=1}^{n}\Big{[}\ell(Y_{i}\widehat{h}_{n}(X_{i}) )-\ell(Y_{i}h_{\ell}^{*}(X_{i}))\Bigr{]}+2J_{\lambda_{n},\tau_{n}}(\widehat{h}_ {n}).\]

One can bound \(B_{1,n}\) in the same way as for \(A_{2,n}\) in the proof of Theorem 4.1. Let \(\Delta(h)(Z_{0})\coloneqq\ell(Y_{0}h(X_{0}))-\ell(Y_{0}h_{\ell}^{*}(X_{0}))\), with \(Z_{0}\coloneqq(X_{0},Y_{0})\). Set

\[\mathcal{H}_{n,j,\alpha}\coloneqq\{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F) :2^{j-1}\mathds{1}(j\neq 0)\alpha\leq J_{\lambda_{n},\tau_{n}}(h)\leq 2^{j}\alpha\},\]
and

\[\mathcal{G}_{n,j,\alpha}\coloneqq\Big{\{}\Delta(h):\mathbb{R}^{d}\times\{-1,1\} \mapsto\mathbb{R}:h\in\mathcal{H}_{n,j,\alpha}\Big{\}}. \tag{7.35}\]

It holds from the definition of \(h_{\ell}^{*}\) that \(\mathbb{E}\Delta(h)(Z_{0})\geq 0\). We have for all \(\alpha>0\),

\[P(B_{1,n}>\alpha) \leq P\Big{(}\sup_{h\in\mathcal{H}_{\varkappa}(L_{n},N_{n},B_{n},F )}\frac{\mathbb{E}\Delta(h)(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}\Delta(h)(Z_{i})}{ \alpha+2J_{\lambda_{n},\tau_{n}}(h)+\mathbb{E}\Delta(h)(Z_{0})}\geq\frac{1}{2} \Big{)}\] \[\leq\sum_{j=0}^{\infty}P\Big{(}\sup_{h\in\mathcal{H}_{n,j,\alpha} }\frac{\mathbb{E}\Delta(h)(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}\Delta(h)(Z_{i})}{ 2^{j}\alpha+\mathbb{E}\Delta(h)(Z_{0})}\geq\frac{1}{2}\Big{)}\] \[\leq\sum_{j=0}^{\infty}P\Big{(}\sup_{g\in\mathcal{G}_{n,j,\alpha }}\frac{\mathbb{E}g(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})}{2^{j}\alpha} \geq\frac{1}{2}\Big{)}\leq\sum_{j=0}^{\infty}P\Big{(}\sup_{g\in\mathcal{G}_{n,j,\alpha}}\mathbb{E}g(Z_{0})-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\geq\frac{2^{j} \alpha}{2}\Big{)}\] \[\leq\sum_{j=0}^{\infty}P\Big{(}\sup_{g\in\mathcal{G}_{n,j,\alpha }}\sum_{i=1}^{n}\Big{(}\mathbb{E}g(Z_{0})-g(Z_{i})\Big{)}\geq\frac{2^{j} \alpha n}{2}\Big{)}.\]

Let \(\Delta(h_{1}),\ \Delta(h_{2})\in\mathcal{G}_{n,j,\alpha}\) and \((x,y)\in\mathbb{R}^{d}\times\{-1,1\}\), we have

\[\|\Delta(h_{1})(x,y)-\Delta(h_{2})(x,y)\|_{\infty}\leq\|\ell(yh_{1}(x))-\ell( yh_{2}(x))\|_{\infty}\leq\mathcal{K}_{\ell}|h_{1}(x)-h_{2}(x)|.\]

Thus,

\[\mathcal{N}\Big{(}\varepsilon,\mathcal{G}_{n,j,\alpha},\|\cdot\|_{\infty} \Big{)}\leq\mathcal{N}\Big{(}\frac{\varepsilon}{\mathcal{K}_{\ell}},\mathcal{ H}_{n,j,\alpha},\|\cdot\|_{\infty}\Big{)}.\]

As stressed in the proof of Theorem 4.1, the process \(\Big{(}g(Z_{t})\coloneqq\ell(h(X_{t}),Y_{t})\Big{)}_{t\in\mathbb{Z}}\) is also \(\psi\)-weakly dependent. From (7.9) and (7.10) we have,

\[\mathcal{N}(\varepsilon,\mathcal{G}_{n,j,\alpha},\|\cdot\|_{\infty}) \leq\mathcal{N}\Big{(}\frac{\varepsilon}{\mathcal{K}_{\ell}}, \mathcal{H}_{n,j,\alpha},\|\cdot\|_{\infty}\Big{)}\] \[\leq\mathcal{N}\Big{(}\frac{\varepsilon}{\mathcal{K}_{\ell}}, \mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F,\frac{2^{j}\alpha}{\lambda_{n}}),\| \cdot\|_{\infty}\Big{)}\] \[\leq\exp\Big{(}2\frac{2^{j}\alpha}{\lambda_{n}}(L_{n}+1)\log \Big{(}\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\varepsilon/\mathcal{K}_{\ell}-\tau_{n}( L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\Big{)}\Big{)}. \tag{7.36}\]

Let \(\varepsilon\coloneqq\frac{2^{j}\alpha}{2}\). By using similar arguments as in (7.14). We have

\[\sum_{j=1}^{\infty}P\Big{\{}\sup_{g\in\mathcal{G}_{n,j,\alpha}} \left[\mathbb{E}[g(Z_{0})]-\frac{1}{n}\sum_{i=1}^{n}g(Z_{i})\right]>\frac{2^{j }\alpha}{2}\Big{\}}\] \[\leq C_{3}\exp(\log\log n)\] \[\times\sum_{j=1}^{\infty}\exp\left(\ 2\frac{2^{j}\alpha}{\lambda_{n}}(L_{n}+1)\log \left(\frac{(L_{n}+1)(N_{n}+1)B_{n}}{\frac{2^{j}\alpha}{2\mathcal{K}_{\ell}}- \tau_{n}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+1}}\right)-\frac{(n2^{j}\alpha)^{2}/ 16}{nC+\log(\frac{n}{n^{1/4}\mathcal{C}})(n2^{j}\alpha/8)^{\nu}}\right).\]

With the assumptions \(L_{n}\lesssim\log n,\ N_{n}\lesssim n^{\nu_{1}},\ B_{n}\lesssim n^{\nu_{2}}, \ \tau_{n}\leq\frac{\beta_{n}}{4\mathcal{K}_{\ell}(L_{n}+1)((N_{n}+1)B_{n})^{L_{n}+ 1}}\); under the conditions on \(\nu_{1}>0,\ \nu_{2}>0,\ \beta_{n},\nu_{5},\ \nu_{6}\), by going as in \(A_{2,n}\) (see the proof of Theorem 4.1), we get,

\[\mathbb{E}[B_{1,n}]\lesssim 2\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}.\]
Let us deal now with \(B_{2,n}\). We choose a neural network function \(h_{n}^{\circ}\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)\) such that

\[\mathcal{E}_{Z_{0}}(h_{n}^{\circ})+J_{\lambda_{n},\tau_{n}}(h_{n}^{\circ})\leq \inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}\Bigl{[}\mathcal{E}_{Z_{0} }(h)+J_{\lambda_{n},\tau_{n}}(h)\Bigr{]}+\frac{1}{n}. \tag{7.37}\]

Then, from the basic inequality,

\[\frac{1}{n}\sum_{i=1}^{n}\ell(Y_{i}\widehat{h}_{n}(X_{i}))+J_{\lambda_{n}, \tau_{n}}(\widehat{h}_{n})\leq\frac{1}{n}\sum_{i=1}^{n}\ell(Y_{i}h(X_{i}))+J_{ \lambda_{n},\tau_{n}}(h),\]

we have,

\[B_{2,n} =\frac{2}{n}\sum_{i=1}^{n}\left[\ell(Y_{i}\widehat{h}_{n}(X_{i}) )-\ell(Y_{i}h_{n}^{\circ}(X_{i}))\right]+2J_{\lambda_{n},\tau_{n}}(\widehat{h }_{n}))+\frac{2}{n}\Bigl{[}\ell(Y_{i}h_{n}^{\circ}(X_{i}))-\ell(Y_{i}h_{\ell}^ {*}(X_{i}))\Bigr{]}\] \[\leq 2J_{\lambda_{n},\tau_{n}}(h_{n}^{\circ})+\frac{2}{n}\Bigl{[} \ell(Y_{i}h_{n}^{\circ}(X_{i}))-\ell(Y_{i}h_{\ell}^{*}(X_{i}))\Bigr{]}\] \[\leq 2\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)} \Bigl{[}\mathcal{E}_{Z_{0}}(h)+J_{\lambda_{n},\tau_{n}}(h)\Bigr{]}+\frac{1}{n}.\]

Hence

\[\mathbb{E}\Bigl{[}\mathcal{E}_{Z_{0}}(\widehat{h}_{n})\Bigr{]}\leq 2\inf_{h\in \mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F)}\Bigl{\{}\mathcal{E}_{Z_{0}}(h)+ \lambda_{n}\|\theta(h)\|_{\mathrm{clip},\tau_{n}}\Bigr{\}}\vee\frac{(\log n)^ {\nu_{5}}}{n^{\nu_{6}}}.\]

This establishes the theorem. 

### Proof of Theorem 5.2

Under the assumption \(\ell\) is \(\mathcal{K}_{\ell}-\mathrm{Lipschitz}\), we have

\[|\mathcal{E}_{z_{0}}(h)| =|\mathbb{E}[\ell(Y_{0}h(X_{0}))]-\mathbb{E}[\ell(Y_{0}h_{\ell}^{ *}(X_{0}))]|\] \[\leq\mathbb{E}[|\ell(h(X_{0}))-\ell(h_{\ell}^{*}(X_{0}))|]\] \[\leq K_{\ell}\mathbb{E}[|h(X_{0})-h_{\ell}^{*}(X_{0})|]\]

Thus,

\[\mathcal{E}_{Z_{0}}(h)\leq\mathcal{K}_{\ell}\|h-h_{\ell}^{*}\|_{1,P_{X_{0}}}.\]

Let \(\epsilon_{n}=n^{-\dfrac{\nu_{4}}{\kappa+1}}\). By Theorem 5.1, the condition (5.6), and the fact that \(\|\theta(h)\|_{\mathrm{clip},\tau}\leq\|\theta\|_{0}\) for any \(\tau>0\), we have that for any \(h_{\ell}^{*}\in\mathcal{H}^{*}\),

\[\mathbb{E}\Bigl{[}\mathcal{E}_{Z_{0}}(\widehat{h}_{n})\Bigr{]} \lesssim\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F,C\epsilon _{n}^{-\kappa}(\log n)^{r})}\Bigl{\{}\mathcal{E}_{Z_{0}}(h)+\lambda_{n}\| \theta(h)\|_{\mathrm{clip},\tau_{n}}\Bigr{\}}\vee\frac{(\log n)^{\nu_{5}}}{n^{ \nu_{6}}}\] \[\lesssim\Bigl{[}\inf_{h\in\mathcal{H}_{\sigma}(L_{n},N_{n},B_{n},F,C\epsilon_{n}^{-\kappa}(\log n)^{r})}\|h-h_{\ell}^{*}\|_{1,P_{X_{0}}}+\lambda _{n}\epsilon_{n}^{-\kappa}(\log n)^{r}\Bigr{]}\vee\frac{(\log n)^{\nu_{5}}}{n^{ \nu_{6}}}\] \[\lesssim\frac{(\log n)^{r+\nu_{3}}}{n\dfrac{\nu_{4}}{\kappa+1}} \vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}.\]

Thus,

\[\sup_{H\in\mathcal{O}_{n}^{*}:h_{\ell}^{*}\in\mathcal{H}^{*}}\mathbb{E}\Bigl{[} \mathcal{E}_{Z_{0}}(\widehat{h}_{n})\Bigr{]}\lesssim\frac{(\log n)^{r+\nu_{3}} }{n\dfrac{\nu_{4}}{\kappa+1}}\vee\frac{(\log n)^{\nu_{5}}}{n^{\nu_{6}}}. \tag{7.38}\]