Towards Optimal Neural Networks: the Role of Sample Splitting in
  Hyperparameter Selection
When artificial neural networks have demonstrated exceptional practical
success in a variety of domains, investigations into their theoretical
characteristics, such as their approximation power, statistical properties, and
generalization performance, have concurrently made significant strides. In this
paper, we construct a novel theory for understanding the effectiveness of
neural networks, which offers a perspective distinct from prior research.
Specifically, we explore the rationale underlying a common practice during the
construction of neural network models: sample splitting. Our findings indicate
that the optimal hyperparameters derived from sample splitting can enable a
neural network model that asymptotically minimizes the prediction risk. We
conduct extensive experiments across different application scenarios and
network architectures, and the results manifest our theory's effectiveness.
Shijin Gong, Xinyu Zhang
# Towards Optimal Neural Networks: the Role of Sample Splitting in Hyperparameter Selection

###### Abstract

When artificial neural networks have demonstrated exceptional practical success in a variety of domains, investigations into their theoretical characteristics, such as their approximation power, statistical properties, and generalization performance, have concurrently made significant strides. In this paper, we construct a novel theory for understanding the effectiveness of neural networks, which offers a perspective distinct from prior research. Specifically, we explore the rationale underlying a common practice during the construction of neural network models: sample splitting. Our findings indicate that the optimal hyperparameters derived from sample splitting can enable a neural network model that asymptotically minimizes the prediction risk. We conduct extensive experiments across different application scenarios and network architectures, and the results manifest our theory's effectiveness.

Role of Sample Splitting towards Optimal NNs

## 1 Introduction

Artificial neural networks (NNs) have garnered remarkable success across various industries, leading to a constant stream of research efforts aimed at exploring novel neural network models to enhance the scope of application and improve performance. Over the past few decades, researchers have developed convolutional neural networks (CNNs) (Fukushima, 1980; Krizhevsky et al., 2012), recurrent neural networks (RNNs) (Rumelhart et al., 1986; Hochreiter and Schmidhuber, 1997), and further models based on more complicated structures or components (e.g., Scarselli et al., 2008; Vaswani et al., 2017), which have demonstrated efficacy in solving a board range of practical problems. In parallel, efforts by scientists to uncover the theoretical characteristics underpinning neural networks, such as approximation power, statistical properties, and generalization ability, have resulted in a tremendous amount of studies and discussions. In what follows, we provide a concise overview of these major strides in neural network research that are currently garnering interest. Within this context, our framework emerges as a novel perspective aiming to expand the ongoing effort.
Hornik et al. (1989) presented an universal approximation theorem for feed-forward neural networks (FNNs), leading to numerous subsequent research on approximation capabilities of NNs (e.g., Hornik, 1991; Leshno et al., 1993; Pinkus, 1999). These studies concluded that neural networks with sufficiently many hidden neurons possess the capacity to approximate any function within a broad class of functions. More literature has emerged in recent years concerning the approximation capabilities of neural networks, particularly deep networks that employ ReLU activation functions (e.g., Lu et al., 2017; Lin and Jegelka, 2018).

White (1990) examined NNs from a nonparametric perspective, showing that the approximation mentioned in Hornik et al. (1989) are learnable by showcasing the consistency property of NNs, provided the complexity of the network increases with the sample size at specific growth rates. Work on consistency of neural networks continued (Mielniczuk and Tyrcha, 1993; Lugosi and Zeger, 1995). Further, Barron (1994); McCaffrey and Gallant (1994) developed error bounds for neural networks that demonstrate the rate of convergence. These studies laid the foundation for subsequent work on convergence rates, such as Chen and White (1999); Kohler and Krzyzak (2005), and Shen et al. (2023) established the asymptotic normality of neural network estimators along with their convergence rates. In recent years, discussions on learnability of NNs has broadened to encompass more than just a statistical perspective, increasingly incorporating an optimization viewpoint. This shift in focus has also seen a transition from exploring shallow neural networks to deep neural networks (DNNs)(e.g., Brutzkus and Globerson, 2017; Du et al., 2018). The objective of these investigations is to elucidate the reasons why the networks trained using algorithms, such as stochastic gradient descent (SGD), achieve successful convergence. Furthermore, the prevalent applications of DNNs have prompted research into the convergence under the over-parameterized setting (e.g., Li and Liang, 2018; Allen-Zhu et al., 2019).

Another vital facet of theoretical work on neural networks has been centered around understanding their ability to extrapolate to new data. This ability, widely known as generalization, is evaluated by the generalization error, which quantifies a network's performance on unseen data. To establish generalization error bounds for NNs, abundant theories have focused on sample complexity, utilizing measures such as VC-dimension and Rademacher complexity (e.g., Baum and Haussler, 1988; Blumer et al., 1989; Haussler, 1992; Bartlett, 1996). These proposed theoretical generalization bounds quantify the amount of data required to ensure accurate performance of an NN. Bousquet and Elisseeff (2002) introduced the notions of stability for learning algorithms, providing an alternative pathway for deriving generalization bounds, which was followed by numerous subsequent studies such as Hardt et al. (2016). The work of Zhang et al. (2017), which implemented a randomization test challenging various prior studies, stimulated continued research on improving generalization bounds (e.g., Neyshabur et al., 2017; Golowich et al., 2018). Simultaneously, research developed under the over-parameterized setting has also gained traction in recent years (e.g., Arora et al., 2019; Allen-Zhu et al., 2019).

In this paper, we incorporate the framework studied in traditional statistic theory (White, 1989) and offer a novel perspective on the question of why neural networks perform well, which goes beyond the characteristics of an individual NN. Commonly, the process of building a neural network necessarily involves the determination about two types of parameters. The first, known as model parameters, are gradually learned through the training process, while the second, referred to as hyperparameters, must be specified prior to train
ing. Hyperparameters play a pivotal role in shaping the structure of neural networks and monitoring the learning process, contributing to substantial variability that allows neural networks to adapt to a diverse range of problems. Consequently, optimization of hyperparameters is a critical task in real-world applications of neural networks.

A variety of algorithms (Yang and Shami, 2020) can achieve hyperparameter optimization. A striking majority of these algorithms are reliant on sample splitting. Typically, after hyperparameters are set, it is necessary to continually evaluate the model's performance during training. This process can then direct subsequent hyperparameter tuning to improve the model. To do this evaluation, a validation set comprising samples that have not been used for training is often required, thus necessitating a split from the original dataset prior to training. In this work, our theoretical and empirical explorations underscore the importance of sample splitting, since this procedure enables us to attain a nearly optimal neural network from those with all alternative hyperparameters. Specifically, hyperparameters tuned by optimizing the performance on validation set are asymptotically optimal in the sense of minimizing prediction risk. Therefore, the practical success of neural networks can be attributed to sample splitting, a common practice in machine learning.

In fact, the efficacy of sample splitting can be recognized in several traditional statistical theories. For example, Li (1987) and Shao (1997) explored multiple procedures in the context of model selection. They derived the asymptotic optimality of these procedures, such as cross-validation, AIC, \(C_{p}\), and other criteria, within the realm of linear models. The lack of explicit reference to sample splitting in these studies probably stems from the fact that the majority of these procedures do not necessarily require it. It is notable, however, that cross-validation, one of the procedures studied, is based on sample splitting, since it partitions the data into separate subsets for model training and evaluation purposes. As such, sample splitting has the potential to account for the effectiveness of models from a model selection standpoint. This perspective coincides with the central thesis of this paper, which investigates the success of neural network models through the lens of sample splitting. It is worth mentioning that our theory is established based on a basic splitting procedure that segregates samples into one training set and one validation set, yet it remains flexible enough to accommodate more sophisticated methods such as cross-validation.

The remainder of our article is structured as follows. Section 2 introduces the mathematical modeling of NNs and their hyperparameters, along with the problem statement and the objective. In Section 3, we present the main theoretical results, their conditions, and an interpretation from a perspective of error decomposition. Section 4 presents experiments designed to validate our findings. Conclusion and future work are drawn in Section 5. Detailed proofs of our main results and additional discussions can be found in the Appendices.

## 2 Preliminaries

Suppose that we have \((n_{1}+n_{2})\) independent and identically distributed (i.i.d.) observations \(S=\{(X_{i},Y_{i})\}_{i=1}^{n_{1}+n_{2}}\), where \(X_{i}\) is a vector of predictors and \(Y_{i}\) is a scalar dependent variable. These samples can be partitioned into training set \(S_{train}=\{(X_{i}^{t},Y_{i}^{t})\}_{i=1}^{n_{1}}\) and validation set \(S_{val}=\{(X_{i}^{v},Y_{i}^{v})\}_{i=1}^{n_{2}}\). The available set of hyperparameters of NNs, including continuous, discrete and categorical variables, is denoted by \(\Lambda\). In this paper, we assume that \(\Lambda\) is a finite set, i.e. \(|\Lambda|<\infty\). The choices of categorical hyperparameters, e.g., the
activation function and optimizer, are intrinsically finite. The domains of continuous and discrete hyperparameters are generally bounded in practical applications (Bergstra et al., 2011), which results in finite choices of discrete ones. Regarding the rationality of assuming continuous hyperparameters, for instance, the learning rate, as belonging to a finite set, several frequently used hyperparameter optimization techniques consider a discretized continuous hyperparameter space and perform optimization on it, including grid search and random search (Bergstra and Bengio, 2012). Consequently, it is reasonable to develop our theory under the assumption that \(|\Lambda|<\infty\).

Given any \(\lambda\in\Lambda\), a hypothesis space of neural networks is defined, and we denote a neural network in this space by \(f_{\lambda}(\cdot,w):\mathbb{R}^{p}\rightarrow\mathbb{R}\), which represents a neural network with the hyperparameter \(\lambda\) and the model parameter \(w\in\mathbb{R}^{d(\lambda)}\). Here, the model parameter \(w\) can interpreted as originating from weights connecting layers, vectors of biases for nodes, and other similar elements, which are then reshaped into column vectors and stacked on top of one another. The term \(d(\lambda)\) represents the dimension of \(w\), that is, the number of trainable variables in the model, and is certainly finite for any given \(\lambda\). Therefore, the maximum dimension of model parameters under \(\lambda\in\Lambda\), which is equal to \(\sup_{\lambda\in\Lambda}d(\lambda)\), is finite in this paper as \(|\lambda|<\infty\), which is also independent of sample size.

For any hyperparameter \(\lambda\), we can train an NN model by \(S_{train}\), achieving the parameter \(\hat{w}_{n_{1}}(\lambda)\) and the trained neural network \(f_{\lambda}(\cdot,\hat{w}_{n_{1}}(\lambda)):\mathbb{R}^{p}\rightarrow\mathbb{R}\). We define the validation loss as the mean squared loss on validation set.

\[L_{n_{2}}(\lambda)=\frac{1}{n_{2}}\sum_{i=1}^{n_{2}}\{f_{\lambda}(X_{i}^{v}, \hat{w}_{n_{1}}(\lambda))-Y_{i}^{v}\}^{2}. \tag{1}\]

To optimize the hyperparameter, the ideal method is to select \(\lambda\) that achieves the best performance on a validation set. Therefore, we define the selected optimal hyperparameter as

\[\hat{\lambda}=\underset{\lambda\in\Lambda}{\text{argmin}}\ L_{n_{2}}(\lambda). \tag{2}\]

Note that the existence of \(\hat{\lambda}\) can be guaranteed by \(|\Lambda|<\infty\).

In real-world applications, the training set size \(n_{1}\) and the validation set size \(n_{2}\) are often set according to a proportion, such as a 90%-10% split of the total sample size. This means that \(n_{1}\) and \(n_{2}\) are generally of the same order with respect to the total sample size. Thus, without loss of generality and for notational simplicity in the ensuing text, we assume \(n_{1}=n_{2}=n\). By adopting \(\hat{\lambda}\) as the hyperparameter and completing the training process, we obtain a network \(f_{\hat{\lambda}}(\cdot,w_{n}(\hat{\lambda}))\). Our objective now is to uncover properties of \(\hat{\lambda}\) or \(f_{\hat{\lambda}}(\cdot,w_{n}(\hat{\lambda}))\). Suppose we have a set of new i.i.d. observations, denoted as \(S_{0}=\{(X_{j}^{0},Y_{j}^{0})\}_{j=1}^{n_{0}}\), but we do not have the knowledge of dependent variables. As We will demonstrate in the next section, \(\hat{\lambda}\) is, in the asymptotic sense, optimal for making prediction for this new observation set \(S_{0}\).

In the subsequent sections, we use \(\|\cdot\|\) to denote the \(\ell_{2}\)-norm of a vector and \(\top\) to represent matrix transpose. Absolute constants, which may vary from line to line are denoted by \(c\) and \(C\). For notational simplicity, we abbreviate the validation sample \((X_{i}^{v},Y_{i}^{v})\)
as \((X_{i},Y_{i})\) for \(i=1,...,n\), when it does not cause ambiguity. Since the samples are i.i.d., we sometimes use \((X_{0},Y_{0})\) to represent any sample from the set \(S_{0}\) also for notational convenience.

## 3 Theoretical Results

In this section, we present the asymptotic properties of the selected hyperparameter \(\hat{\lambda}\). We define the following notations

\[L(\lambda;X,Y)=\{f_{\lambda}(X,\hat{w}_{n}(\lambda))-E(Y|X)\}^{2}, \tag{3}\] \[L_{0}(\lambda)=\frac{1}{|S_{0}|}\sum_{(X,Y)\in S_{0}}L(\lambda;X,Y)=\frac{1}{n_{0}}\sum_{j=1}^{n_{0}}L(\lambda;X_{j}^{0},Y_{j}^{0}),\] (4) \[\text{and}\qquad R_{0}(\lambda)=E(L_{0}(\lambda)), \tag{5}\]

where \(|S_{0}|\) denotes the number of elements in set \(S_{0}\). For hyperparameter \(\lambda\in\Lambda\), \(L(\lambda;X,Y)\) measures the squared loss between the conditional expectation \(E(Y|X)\) and the network output \(f_{\lambda}(\cdot,\hat{w}_{n}(\lambda))\) with input \(X\); \(L_{0}(\lambda)\) is the mean prediction loss on set \(S_{0}\); and \(R_{0}(\lambda)\) is the prediction risk, the expectation of \(L_{0}(\lambda)\). Since observations in \(S_{0}\) are i.i.d., we can reformulate \(R_{0}(\lambda)=E\{f_{\lambda}(X,\hat{w}_{n}(\lambda))-E(Y|X)\}^{2}\), where \((X,Y)\in S_{val}\bigcup S_{0}\) can represent any sample independent of \(S_{train}\).

### Optimality of the Selected Hyperparameter

We present the assumptions that are required for our theoretical results. Unless otherwise specified, the following limiting processes are studied with respect to \(n\to\infty\).

**Assumption 1**: _For any \(\lambda\in\Lambda\), there exist a parameter \(w^{*}(\lambda)\), such that \(\|\hat{w}_{n}(\lambda)-w^{*}(\lambda)\|=O_{p}(n^{-1/2})\)._

Assumption 1 imports a high-level condition ensuring that for any given hyperparameter \(\lambda\), the trained parameters of the neural networks \(\hat{w}_{n}(\lambda)\) will have a limiting value \(w^{*}(\lambda)\). This assumption holds under some reasonable conditions; see conditions in Theorem 1 of White (1989).

With the importation of \(w^{*}(\lambda)\), we can further introduce these notations

\[L^{*}(\lambda;X,Y)=\{f_{\lambda}(X,w^{*}(\lambda))-E(Y|X)\}^{2},\] \[L^{*}_{0}(\lambda)=\frac{1}{|S_{0}|}\sum_{(X,Y)\in S_{0}}L^{*}( \lambda;X,Y)=\frac{1}{n_{0}}\sum_{j=1}^{n_{0}}L^{*}(\lambda;X_{j}^{0},Y_{j}^ {0}),\] \[\text{and}\qquad R^{*}_{0}(\lambda)=E(L^{*}_{0}(\lambda)),\]

which are similar to the definitions of \(L(\lambda;X,Y)\), \(L_{0}(\lambda)\), and \(R_{0}(\lambda)\) in (3), (4), and (5) except for substituting \(w^{*}(\lambda)\) for \(\hat{w}_{n}(\lambda)\).
**Assumption 2**: _For any \((X,Y)\in S_{train}\bigcup S_{val}\bigcup S_{0}\) and for any \(\lambda\in\Lambda\), (i) \(E|f_{\lambda}(X,w^{*}(\lambda))-Y|\) and \(E|f_{\lambda}(X,w^{*}(\lambda))-E(Y|X)|\) are both \(O(1)\); (ii) \(f_{\lambda}(X,w)\) is differentiable with respect to \(w\), and there exists a constant \(\rho\), such that_

\[E\sup_{w^{0}\in\mathcal{O}(w^{*}(\lambda),\rho)}\bigg{\|}\frac{\partial f_{ \lambda}(X,w)}{\partial w}\Big{|}_{w=w^{0}}\bigg{\|}=O(1), \tag{6}\]

_where \(\mathcal{O}(w^{*}(\lambda),\rho)\) denotes an open ball with center \(w^{*}(\lambda)\) and radius \(\rho\)._

Assumption 2 makes some restrictions on boundedness and differentiability. Assumption 2(i) holds when \(f_{\lambda}(X,w^{*}(\lambda))\), \(Y\) and \(E(Y|X)\) are bounded. Examples to illustrate the rationality of Assumption 2(ii) under some primitive conditions are given in Appendix A.

Let \(\xi_{n}=\inf_{\lambda\in\Lambda}R_{0}^{*}(\lambda)\) denote the minimum risk among all \(\lambda\in\Lambda\) with neural networks using limiting value of parameters \(w^{*}(\lambda)\), and we make these three assumptions:

**Assumption 3**: \(\Delta_{n}\) _is integrable, where_

\[\Delta_{n}=\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\big{|}\{f_{\lambda}(Y_{0}, \hat{w}_{n}(\lambda))-Y_{0}\}^{2}-\{f_{\lambda}(Y_{0},w_{n}^{*}(\lambda))-Y_{ 0}\}^{2}\big{|}.\]

**Assumption 4**: _For any \(\lambda\in\Lambda\), \(\xi_{n}^{-1}n^{-1/2}=o(1)\)._

**Assumption 5**: _For any sample \((X,Y)\in S_{val}\) and any \(\lambda\in\Lambda\) that (i) \(\mathrm{var}[f_{\lambda}(X,w^{*}(\lambda))\{Y-E(Y|X)\}]=O(1)\); (ii) \(\mathrm{var}[\{f_{\lambda}(X,w^{*}(\lambda))-E(Y|X)\}^{2}]=O(1)\)._

As shown in (36) in the proof of Theorem 1 that \(\Delta_{n}\) is \(o_{p}(1)\), Assumption 3 is only imposed to ensure the expectation of \(\Delta_{n}\) is o(1). Assumption 4 is similar to Condition (11) in Ando and Li (2014), Condition (C.3) of Zhang et al. (2016), and Assumption 2.3 in Liu and Okui (2013), which makes a restriction on the minimum risk \(\xi_{n}\), requiring the speed of its converging to zero to be not too fast--at least slower than \(n^{-1/2}\). As a concrete example, Assumption 4 is satisfied if \(c\leq\xi_{n}\) for any \(n\) for some constant \(c\). Such cases arise when the hypothesis space, defined by any hyperparameter \(\lambda\in\Lambda\), is insufficiently capable of approximating the objective function. Conversely, Assumption 4 is violated if \(\xi_{n}\equiv 0\) for any \(n\geq 0\). See interpretation from the perspective of error decomposition in Section 3.2.Assumption 5 has regularity conditions ensuring certain terms are not heavy-tailed so that their variance can be bounded. Now we are ready to present our first theorem.

**Theorem 1**: _Under Assumptions 1-5, we have_

\[\frac{R_{0}(\hat{\lambda})}{\inf_{\lambda\in\Lambda}R_{0}(\lambda)}=1+O_{p}( \xi_{n}^{-1}n^{-1/2})=1+o_{p}(1). \tag{7}\]

Theorem 1 indicates that the selected hyperparameter \(\hat{\lambda}\) is optimal in the sense that the prediction risk of the neural network using hyperparameter \(\hat{\lambda}\) is asymptotically identical to the risk of the infeasible best neural network. Note that in (7), since \(\hat{\lambda}\) is directly plugged in the expressions \(R_{0}(\lambda)\), the randomness of \(\hat{\lambda}\) is not considered. This plug-in is done after taking the expectation in \(R_{0}(\hat{\lambda})\), which makes (7) independent of test size \(n_{0}\), and hence Theorem 1 holds for arbitrary \(n_{0}\). In what follows, our second result shows that we can build other forms of optimalities based on \(L_{0}(\lambda)\) with some additional assumptions.
**Assumption 6**: _For any \(\lambda\in\Lambda\), \(\xi_{n}^{-1}n_{0}^{-1/2}=o(1)\)._

Assumption 6 is similar to Assumption 4 except that the large sample condition is also with respect to \(n_{0}\). If we assume that \(n_{0}\) and \(n\) are of same order, Assumption 4 and Assumption 6 are actually equivalent. As \(L_{0}(\lambda)\) denotes the mean prediction loss on the test set \(S_{0}\), this measure can be affected by the random fluctuations of samples. Therefore, it is required that the size of the test set, \(n_{0}\), to be divergent, as assumed in 6.

**Assumption 7**: _(i) There exists a constant \(C\) such that \(L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}(\lambda)<C\); (ii) \(\{L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}/\inf_{\lambda \in\Lambda}L_{0}(\lambda)\) is uniformly integrable._

Assumption 7(i) holds when \(f_{\lambda}(X,w^{*}(\lambda))\) and \(E(Y|X)\) are bounded. Assumption 7(ii) is imposed to obtain the \(L_{1}\) convergence of the term on the left hand side of (8) in Theorem 2.

**Theorem 2**: _For a diverging \(n_{0}\), under Assumptions 1-6, we have_

\[\frac{L_{0}(\hat{\lambda})}{\inf_{\lambda\in\Lambda}L_{0}(\lambda)}=1+O_{p}( \xi_{n}^{-1}n^{-1/2}+\xi_{n}^{-1}n_{0}^{-1/2})=1+o_{p}(1). \tag{8}\]

_Furthermore, if Assumption 7 holds, then we have_

\[E\Big{\{}\frac{L_{0}(\hat{\lambda})}{\inf_{\lambda\in\Lambda}L_{0}(\lambda)} \Big{\}}=1+o(1) \tag{9}\]

_and_

\[\frac{E\{L_{0}(\hat{\lambda})\}}{E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}} =1+o(1). \tag{10}\]

Theorem 2 contains three types of optimalities. Equation (8) concerns the optimality in terms of asymptotically minimizing \(L_{0}\), the mean prediction loss on test set \(S_{0}\), which is established based on the result in Theorem 1. Equation (9) and (10) concern optimalities involving different forms of expectation. It is important to note that \(E\{L_{0}(\hat{\lambda})\}\) is distinct from \(R_{0}(\hat{\lambda})\) presented in Theorem 1, since the randomness of \(\hat{\lambda}\) is taken into consideration in (9) and (10) but not in \(R_{0}(\hat{\lambda})\). Consequently, the optimalities outlined in (9) and (10) is more reflective of real-world scenarios compared to Theorem 1, thereby offering theoretical evidence in practical application. To verify the justifications in Theorem 2, we will carry out a series of experimental studies in Section 4.

### Error Decomposition Interpretation

For a more precise comprehension of theory, we revisit the objective in this paper from the perspective of error decomposition in machine learning theory. The optimalities of \(\hat{\lambda}\) in Theorem 1 and Theorem 2 are obtained in the sense of minimizing \(L_{0}(\lambda)\) or \(R_{0}(\lambda)\), which both involve the difference between the target \(E(Y|X)\) and the trained model \(f_{\lambda}(X,\hat{w}_{n}(\lambda))\). This discrepancy can be written as

\[E(Y|X)-f_{\lambda}(X,\hat{w}_{n}(\lambda))\] \[= \{E(Y|X)-f_{\lambda}(X,w^{*}(\lambda))\}+\{f_{\lambda}(X,w^{*}( \lambda))-f_{\lambda}(X,\hat{w}_{n}(\lambda))\}.\]
Now we preceive \(f_{\lambda}(X,w^{*}(\lambda))\) as a quasi-optimal estimate of \(E(Y|X)\) within models defined by hyperparamter \(\lambda\), and then we can interpret the two terms respectively as 1) the model error \(\{E(Y|X)-f_{\lambda}(X,w^{*}(\lambda))\}\) (also referred to as approximation error)--the error caused by the inadequate hypothesis space of the model determined by the choice of \(\lambda\); and 2) the estimation error \(\{f_{\lambda}(X,w^{*}(\lambda))-f_{\lambda}(X,\hat{w}_{n}(\lambda))\}\)--the error resulting from the gap, given \(\lambda\), between the quasi-optimal parameter \(w^{*}(\lambda)\) and the parameter \(\hat{w}_{n}(\lambda)\) derived from only \(n\) training samples. If we revisit the definition of \(L_{0}^{*}(\lambda)\), the model error \(\{E(Y|X)-f_{\lambda}(X,w^{*}(\lambda))\}\) corresponds to the term within \(L_{0}^{*}(\lambda)\)'s square. Therefore, any assumptions imposed on \(L_{0}^{*}(\lambda)\) or \(R_{0}^{*}(\lambda)\) (such as Assumption 4, 6, and 7) inherently carry assumptions about the model error. Assumption 1 imports a condition ensuring the convergence of \(\hat{w}_{n}(\lambda)\) to \(w^{*}(\lambda)\) at a certain rate. We leverage this condition to manage the estimation error \(\{f_{\lambda}(X,w^{*}(\lambda))-f_{\lambda}(X,\hat{w}_{n}(\lambda))\}\). This indicates that our work contemplates both model error and estimation error from the perspective of error decomposition. The optimalities established in our theorems can be viewed as accounting for the effectiveness of sample splitting with the modeling of both errors.

Note that the optimization error--the error incurred during training process--is not our main focus, which is consistent with previous work that study neural networks from a statistical perspective (e.g., White, 1989; Barron, 1994; Bauer and Kohler, 2019). By utilizing Assumption 1, we have addressed the estimation error, as mentioned in the previous paragraph, while concurrently avoided the explicit modeling of the optimization error. By invoking the convergence property of \(\hat{w}_{n}(\lambda)\) under Assumption 1, we directly perceive \(\hat{w}_{n}(\lambda)\) as a set of satisfactory network parameters obtained under \(n\) training samples, thereby dispensing with the need to model the specific training process of obtaining them. The specific modeling of these elements--the training process and optimization error--might well serve as a avenue for future research.

## 4 Experimental Verification

In this section, we consider several common neural network structures and use them to model a range of diverse problems, including linear/nonlinear regression, binary-classification, and time series forecasting, to verify our theoretical results in Theorem 2.

### Multilayer Perceptrons

We conduct experiments in three scenarios based on a basic NN structure multilayer perceptrons (MLPs), which follows a fully connected structure. The first and second scenarios are respectively based on linear and nonlinear regression problems. The third scenario simulates a classification problem. The training and validation set sizes are set to \(n_{1}=n_{2}=n\)

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline  & Scenario & LR & HS & Depth & BS \\ \hline \multirow{3}{*}{MLP} & Linear & 0.1, 0.01, 0.001 & 5, 10, 20 & 1, 2 & 8, 16, 32 \\  & Nonlinear & 0.01, 0.001 & 50, 100 & 1, 2, 3 & 16, 32, 64 \\  & Classification & 0.001, 0.0001 & 50, 100 & 1, 3, 5 & 16, 32, 128 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Design of \(\Lambda\) for experiments with MLPs
with \(n\) gradually increasing from 50 to 50000, while the test set size \(n_{0}=n/5\). The set of hyperparameters, denoted by \(\Lambda\), consists of four types varying hyperparameters: LR (learning rate), HS (number of hidden nodes in each layer), depth (number of hidden layers), and BS (batch size), with the alternative ranges of these hyperparameters in different scenarios shown in Table 1. All other hyperparameters are fixed as follows. The maximum epoch was set to 50, the activation function was the Relu function, and the optimizer used was Adam. The Mean Squared Error (MSE) loss function is used for training regression models, while the commonly used Cross Entropy (CE) loss function is used for training classification models. Now we will present the specific designs and results for each scenario.

#### 4.1.1 Linear Regression

For the first scenario in our simulations, we consider a linear relationship between \(X\) and \(Y\).

\[Y_{i}=X_{i}^{\top}\beta+\epsilon_{i}=\sum_{l=1}^{5}\beta_{l}X_{il}+\epsilon_{i },\quad i=1,...,n,\]

where \(\epsilon_{i}\sim N(0,\sigma^{2})\), \(X_{i}=(X_{i1},...,X_{i5})^{\top}\sim N(\mathbf{0},\Omega)\), with the diagonal elements of \(\Omega\) set to 1 and off-diagonal elements set to 0.5. We set the coefficient vector \(\beta=(\beta_{1},...,\beta_{5})^{\top}\) as \(\beta_{l}=1\) for \(1\leq l\leq 3\) and \(\beta_{l}=-1\) for \(4\leq l\leq 5\). The variance of the noise, denoted by \(\sigma^{2}\), is set to vary between \(\sigma^{2}=1/3,1,3\), corresponding to \(R^{2}=0.9,0.75,0.5\), respectively. The hyperparameters were set according to the 'Linear' row in Table 1.

For each value of \(n=50,...,50000\), we repeat the same simulation for 50 times. In each simulation replication, we train a number of neural networks with every hyperparameter choice \(\lambda\in\Lambda\). After the training process is completed, we select the best hyperparameters by \(\hat{\lambda}=\operatorname*{argmin}_{\lambda\in\Lambda}\ L_{n}(\lambda)\), record \(L_{0}(\lambda)\) for every \(\lambda\in\Lambda\), and then calculate \(\inf_{\lambda\in\Lambda}L_{0}(\lambda)\). Note that \(E(Y|X)=X^{\top}\beta\) when calculating \(L_{0}(\lambda)\) here. To verify (9), we calculate the ratio \(L_{0}(\hat{\lambda})/\inf_{\lambda\in\Lambda}L_{0}(\lambda)\) in all replications. Figure 1(a) presents the average value of this ratio across all replications to represent \(E\{L_{0}(\hat{\lambda})/\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\). To verify (10), we separately calculate the average values of \(L_{0}(\hat{\lambda})\) and \(\inf_{\lambda\in\Lambda}L_{0}(\lambda)\) separately across all replications and then divide these two values to represent the ratio \(E\{L_{0}(\hat{\lambda})\}/E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\). Figure 1(b) presents this ratio under different settings. The results presented in Figure 1 demonstrate that both \(E\{L_{0}(\hat{\lambda})/\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\) and \(E\{L_{0}(\hat{\lambda})\}/E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\) converge to 1 regardless of the choice of \(\sigma^{2}\), manifesting the validity of Theorem 2 in this scenario.

#### 4.1.2 Nonlinear Regression

For the second simulation design, we consider a non linear regression problem of the form:

\[Y_{i}=\frac{10\sin\left(\|X_{i}\|_{2}\right)}{\|X_{i}\|_{2}}+\epsilon_{i}, \quad i=1,...,n,\]

where \(\epsilon_{i}\sim N(0,\sigma^{2})\) and \(X_{i}=(X_{i1},...,X_{i5})^{\top}\). For \(l=1,...,5\), \(X_{il}\) is independent of each other and follows a uniform distribution between \(-10\) and \(10\). An additional simulation was conducted to approximate the variance of \(\sin\left(|X_{i}|_{2}\right)/|X_{i}|_{2}\). Accordingly, the choice of \(\sigma\) are set to ensure \(R^{2}=0.9,0.75,0.5\), respectively. The varying hyperparameters are
set according to the 'Nonlinear' row in Table 1, and the training and analysis process is conducted in the same manner as that in our linear regression scenario. Note that \(E(Y|X)=10\sin{(\|X\|_{2})}/\|X\|_{2}\) when calculating \(L_{0}(\lambda)\) here.

Figure 2(a) and 2(b) show that \(E\{L_{0}(\hat{\lambda})/\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\) and \(E\{L_{0}(\hat{\lambda})\}/E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\) converge to 1 in nonlinear settings as well.

#### 4.1.3 Classification

For the third simulation design, we address a binary classification problem. Specifically, we assume that \(Y_{i}\) takes values of 0 or 1, and the conditional probability of \(Y_{i}\) being 1 given \(X_{i}\), denoted as \(\mathrm{P}(Y_{i}=1|X_{i})\), is given by \(g(X_{i})\):

\[Y_{i}=0,1,\quad\mathrm{P}(Y_{i}=1|X_{i})=g(X_{i}),\quad i=1,...,n.\]

Here \(g(\mathbf{x})=1/[1+\exp\{-(h(\mathbf{x})-\mu)^{2}\}]\), \(\mathbf{x}=(x_{1},...,x_{10})^{\top}\in R^{10}\), where \(h(\mathbf{x})\) is a nonlinear function of \(\mathbf{x}\) defined as

\[h(\mathbf{x}) =12x_{1}(x_{2}-0.5)^{2}-16\{x_{3}(x_{5}-0.2)\}^{4}\] \[+2\log\{3-4\left(x_{4}-0.3\right)^{2}+x_{5}+\exp(-x_{6}x_{7}+x_{5 })\}\] \[+2\tan[4\{(x_{1}(x_{8}-0.5)\}^{2}+0.1]\]

with \(x_{9}\) and \(x_{10}\) being redundant. We generate the an sample input as \(X_{i}=(X_{i1},...,X_{i10})^{\top}\), where \(X_{il}\) are independent and uniformly distributed between 0 and 1 for \(l=1,...,10\). We set \(\mu\) to ensure that \(E(h(X_{i}))=\mu\), and \(\Lambda\)'s setting is shown in 'Classification' row in Table
1. The activation function is Relu, and we use a sigmoid function before the output layer to limit the range of output to interval \((0,1)\).

The loss function for training is cross-entropy (CE). However, for consistency with our theory, the determination of \(\hat{\lambda}\) through the validation set and subsequent verification of the theorem are based on mean squared error (MSE) instead of CE. Specifically, \(\hat{\lambda}\) is still chosen according to (2), i.e., by minimizing the MSE between real-valued \(f_{\lambda}(X,\hat{w}_{n}(\lambda))\) and one-hot value \(Y\). In terms of computing \(L_{0}(\lambda)\), since we have \(E(Y|X)=P(Y=1|X)=g(X)\), we calculate the square error of real-valued \(f_{\lambda}(X,\hat{w}_{n}(\lambda))\) and real-valued \(g(X)\). Although this procedure leads to inconsistency between the loss function for training and the performance measurement, we state that the theoretical results are not affected as the properties that form during the training process are related to Assumption 1 and will not affect subsequent reasoning. Therefore, the optimality of \(\hat{\lambda}\) still holds, and the simulation results also demonstrate this statement. We record the data in replicate simulations to approximate \(E\{L_{0}(\hat{\lambda})/\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\) and \(E\{L_{0}(\hat{\lambda})\}/E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\). Figure 3 shows these ratios also converge to 1 in this scenario, which is consistent with Theorem 2.

Note that our experimental analysis is focused exclusively on binary classification problems since our theoretical framework can not model the expectation of labels in a multi-class classification problem. As such, the experiments we conduct in the following sections regarding classification will be restricted to binary classification problems.

### Convolutional Neural Networks

In this section, we carry out experiments involving convolutional neural networks (CNNs), a specialized form of feedforward neural networks that exhibit distinct connection structures
compared to MLPs. Since CNNs are often employed when processing image-based inputs, the data generating process in previous simulations is not applicable to this context. Thus, we decide to utilize widely-used public datasets for image recognition tasks. However, we are faced with a problem that a mathematical formulation representing the conditional expectation \(E(Y|X)\) is unavailable in real-world datasets. This limitation prevents us from calculating the value of any \(L_{0}(\lambda)\) in our theorem. To address this issue, we design a special method to generate new samples from the MNIST/Fashion-MNIST datasets as follows, ensuring that the theoretical value of \(E(Y|X)\) remains attainable:

* Define a binary-class classification problem for the MNIST/Fashion-MNIST dataset.
* Change the labels of samples in the dataset to 1 and 0 to represent positive and negative samples of the binary-class classification problem, respectively.
* Train a well-performed classification model \(F:\mathbb{R}^{28\times 28}\rightarrow[0,1]\) with these samples.
* Relabel the samples by using the trained model \(F\). Specifically, for any \(X_{i}\) from a sample in the dataset, we generate its new label \(Y_{i}\) by \[\mathrm{P}(Y_{i}=1|X_{i})=F(X_{i}).\]

This method treats \(F(X)\) as the true probability of input \(X\) belonging to the positive class. For example, if an input \(X\) satisfies \(F(X)=0.75\), we generate a new sample \((X,Y)\) with \(\mathrm{P}(Y=1|X)=0.75\), and consequently, it is evident that \(E(Y|X)=0.75\). Through this procedure, we can transform the common classification task on a real-world dataset into a binary-classification problem, and possess true knowledge of \(E(Y|X)\) to verify our theorem.
#### 4.2.1 Binary-Classification on Fashion-MNIST

We first describe the design of the binary-classification problem on Fashion-MNIST. Specifically, clothing that covering the upper human body is set to be Class 0, which involves label sets [0,2,3,4,6], and the rest of labels are Class 1, including clothing such as thousands, shoes, and bags. Adopting the data generating process and training procedure introduced in Section 4.2, we create a new dataset derived from fashion-MNIST with labels converted to either 0 or 1 and train a well-performed model \(F_{1}\) on this task. See Appendix C for details of \(F_{1}\).

We consider training a set of CNNs with a standard architecture consisting of two convolutional layers and two max pooling layers, followed by two fully connected layers that produce the final output. The choices of varying hyperparamters in \(\Lambda\) are shown in 'Fashion-MNIST' row in Table 2, where Conv KS refers to kernel size of the squared convolutional layer, Chan Out refers to number of output channels by the convolution (number of convolution filters), Pool KS is kernel size of the squared pooling layer, and Pool Str is stride of pooling. Besides, learning rate, batch size, activation function and loss function are set to 0.0005, 16, Relu and CE respectively. The hidden size of the fully connected layer is set to 128. With \(n\) taking values from 50 to 10000, we perform the same analysis as in the MLP classification problem described in Section 4.1.3. Results given in Figure 4 show Theorem 2 also works in this scenario.

#### 4.2.2 Binary-Classification on MNIST

We also consider a binary-Classification problem on the MNIST dataset, which aims at identifying whether a handwritten digit represents a number '1'. Following the procedure and the data generating process introduced in Section 4.2, we create a new dataset derived from MNIST for binary-classification and train a well-performed neural network model \(F_{2}\) on it. See Appendix C for details of \(F_{2}\).

We consider training a set of CNNs that have the same standard architecture in Section 4.2.1. The fixed hyperparameters are also set as same as those in Section 4.2.1, and the varying hyperparamters in \(\Lambda\) can be viewed in 'MNIST' Table 2. The results are shown in Figure 5. It is clear that the ratio converge to 1 in Figure 5(a). The convergence speed of the ratio shown in 5(b) is shower than in other scenarios, which might be attributed to simplicity of this problem. Specifically, we observed the performance of models under different hyperparamters and discovered that almost every network achieved satisfactory prediction results, which means a larger sample size is required to effectively distinguish the optimal hyperparameter.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Scenario & (Conv KS, Chan Out) & Pool KS & Pool Str \\ \hline Fashion-MNIST & (3,4),(3,16),(3,64),(4,4),(4,16),(4,64),(5,4),(5,16) & 2,3 & 1,2 \\ MNIST & (3,4),(3,16),(3,64),(4,16),(4,64) & 2,3 & 1,2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Design of \(\Lambda\) for experiments with CNNs
Figure 4: Results of CNNs for binary classification on Fashion-MNIST. (a) The estimated value of \(E\{L_{0}(\hat{\lambda})/\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\). (b) The estimated value of \(E\{L_{0}(\hat{\lambda})\}/E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\).
### Recurrent Neural Networks

In this section, we present an experiment to show that the optimality based on another neural network architecture, recurrent neural network (RNN), holds as well. Since RNN is often used to deal with temporal dynamic behavior of inputs, we designed a time series forecasting problem here. Two types of time series we consider are:

1. \(y_{t}=0.6y_{t-1}+0.3y_{t-2}-0.1y_{t-3}+\epsilon_{t}\),
2. \(y_{t}=0.3y_{t-1}+0.6y_{t-2}+(0.1-0.9y_{t-1}+0.8y_{t-2})\left[1+\exp\left(-10y_{t -1}\right)\right]^{-1}+\epsilon_{t}\),

which are linear and nonlinear respectively, where \(\epsilon_{t}\) is independent for \(t\geq 0\) and follows \(N(0,\sigma^{2})\) and the initial values of these time series follow uniform distribution between \(-6\) to \(6\). Our task here is to make one-step forward prediction for \(y_{t}\) given the history values of \(\{y_{i}\}_{i=1}^{t-1}\). To construct the samples for training from the origin time series, we apply the sliding window technique. A time series with length \(T\) can generate \(T-p\) training samples, where \(p\) is the input length.

Apart from the hyperparameters considered in the previous simulation, the input length is considered to be another hyperparameter, which is commonly required to be tuned in practice. See Table 3 for details of \(\Lambda\), where \(L\) represents the input length. Again we proceed with our simulations with different choice of sample size \(n\), and deal with the results similarly, and we show them in Figure 6. Although the speed of convergence is slower than in those cases in feedforward neural networks, the two concerning ratios are indeed approaching \(1\).

Since the samples obtained by (11) and the subsequent sliding window method are not independent, in violation of our theory's i.i.d. assumption at the beginning of Section 2, this scenario is distinct from previous ones. Nonetheless, we still regard our theorem valid in the case that samples are correlated according to results in Figure 6.

## 5 Conclusion

Our study offers a new theoretical explanation for the success of neural networks in practical applications. In order to construct an effective neural network model for any given task, optimization should be performed to obtain the most appropriate hyperparameters for the neural network, as they have a decisive impact on the performance of the final network. According to our proposed theories, it is through the use of sample splitting that we can acquire the optimal hyperparameters for the network, thus enabling the attainment of the optimal neural network from a family of neural network models. Through a series of experiments across various problem domains and data distributions, we verify our theoretical results on several prevalent neural network architectures.

Our framework has contributed new theoretical insights into neural networks, but further research is needed to address the limitations in this paper and explore more widely applicable

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & LR & HS & Depth & L \\ \hline RNN & 0.01, 0.001 & 50, 100 & 1, 2 & 3, 4, 5, 6 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Design of \(\Lambda\) for experiments with RNNs
theories. In the process of establishing Theorem 2, our intuition suggested that the condition of the divergence of \(n_{0}\) could be relaxed while proving (9) and (10), since both equations involve taking expectations. Nonetheless, we have encountered challenges with a relaxed condition and thus, leave it as a topic for future research. Inspired by our experiment on recurrent neural networks, extending our theory beyond an i.i.d. assumption will also be part of our future work. Finally, while the largest parameter space remained fixed in this article, another future extension would be to consider the size of the available set of hyperparameters diverging with the number of samples, which would also involve the diverging dimension of model parameter \(w\).
## Appendix A Examples for Verifying Assumption 2(ii)

In this section, we provide several examples to illustrate that Assumption 2(ii) holds under some primitive conditions. Specifically, we only assume that \(E\|X\|^{2}=O(1)\). Before proceeding, we claim that

\[\sup_{\lambda\in\Lambda}\sup_{w^{0}\in\mathcal{O}(w^{*}(\lambda), \rho)}\|w^{0}\|\leq C_{w}, \tag{12}\]

where \(C_{w}\) is a fixed constant. In fact, since \(|\Lambda|<\infty\), this is straightforward by letting

\[C_{w}=\sup_{\lambda\in\Lambda}\|w^{*}(\lambda)\|+\rho.\]

Then, we present two examples as follows, whose conclusions can also be extended to more complicated architectures.

### Multilayer Perceptrons

Consider a two-layer full-connected neural network corresponding to hyperparameter \(\lambda\) with \(k\) hidden nodes and input \(x=(x_{1},...,x_{p})^{\top}\in\mathbb{R}^{p}\)

\[f_{\lambda}(x,w)=\sum_{i=1}^{k(\lambda)}\alpha_{i}h_{i}(x)+ \alpha_{0},\] \[h_{i}(x)=\sigma(\sum_{j=1}^{p}\beta_{ij}x_{j}+b_{i}),\quad i=1,...,k,\]

where \(\sigma(\cdot)\) is the sigmoid activation function. Different choices of \(\lambda\) correspond to varying values of \(k\). Here, \(w=(\alpha_{0},...,\alpha_{k},b_{1},...,b_{k},\beta_{11},...,\beta_{kp})^{\top}\). It can be easily shown that \(\partial f/\partial\alpha_{0}=1\), \(\partial f/\partial\alpha_{i}=h_{i}(x)\), \(\partial f/\partial b_{i}=\alpha_{i}h_{i}(x)(1-h_{i}(x))\), and that \(\partial f/\partial\beta_{ij}=\alpha_{i}x_{j}h_{i}(x)(1-h_{i}(x))\). This implies

\[\left\|\frac{\partial f_{\lambda}(X,w)}{\partial w}\right\|^{2} \leq c_{1}+c_{2}(1+\|X\|^{2})\cdot\|w\|^{2}.\]

where \(c_{1}\) and \(c_{2}\) are constants that can be chosen to be independent of \(k\) due to \(|\Lambda|\leq\infty\). Then, by (12) and \(E\|X\|^{2}=O(1)\), we obtain

\[E\sup_{w^{0}\in\mathcal{O}(w^{*}(\lambda),\rho)}\left\|\frac{ \partial f_{\lambda}(X,w)}{\partial w}\Big{|}_{w=w^{0}}\right\|\leq E\big{\{} \sqrt{c_{1}+c_{2}(1+\|X\|^{2})C_{w}^{2}}\big{\}}=O(1)\]

This demonstrates that (6) in Assumption 2 holds for this example.

### Recurrent Neural Networks

We consider a simple recurrent neural network (RNN) structure

\[h^{(1)}=\sigma(Ux^{(1)}+b),\]
\[h^{(t)} =\sigma(Wh^{(t-1)}+Ux^{(t)}+b),\quad t\geq 2\] \[o^{(t)} =\tanh(V\cdot h^{(t)}+b^{\prime}),\quad t\geq 2,\]

where \(x^{(t)}\in\mathbb{R}^{p}\) denote the input, with \(p\) representing the size of the input, and \(o^{(t)}\in\mathbb{R}\) denote output at time step \(t\). The sigmoid and hyperbolic tangent activation functions are denoted by \(\sigma(\cdot)\) and \(\tanh(\cdot)\), respectively. Let \(w=\operatorname{vec}(U,V,W,b,b^{\prime})\) denote the network parameters, where \(U\in\mathbb{R}^{K\times p}\), \(W\in\mathbb{R}^{K\times K}\), \(V\in\mathbb{R}^{K}\), \(b\in\mathbb{R}^{K}\), and \(b^{\prime}\in\mathbb{R}\).

Now consider a sequential input \(x=(x^{(1)},...,x^{(T)})\in\mathbb{R}^{p\times T}\), and assume that the output of this network is given by \(f_{\lambda}(w,x)=o^{(T)}\). The different choices of \(\lambda\) can correspond to varying values of hyperparameters \(T\), \(p\) and \(K\). To simplify the notation, we define the following terms:

\[c^{(t)}=Wh^{(t-1)}+Ux^{(t)}+b,\quad z^{(t)}=Vh^{(t)}+b^{\prime},\]

and we use subscripts to represent the entries of vector or matrix as follows: \(U=(U_{ij})_{K\times p}\), \(W=(W_{ij})_{K\times K}\), \(V=(V_{i})_{K}\), \(c^{(t)}=(c^{(t)}_{i})_{K}\), and \(h^{(t)}=(h^{(t)}_{i})_{K}\). To verify Assumption 2(ii), we derive some upper bounds for the derivatives of \(o^{(T)}\) with respect to \(w\).

By performing straightforward differentiation, we obtain

\[\frac{\partial o^{(T)}}{\partial b^{\prime}} =\frac{\partial o^{(T)}}{\partial z^{(T)}},\] \[\frac{\partial o^{(T)}}{\partial V_{i}} =h^{(T)}_{i}\frac{\partial o^{(T)}}{\partial z^{(T)}},\quad 1\leq i \leq K.\]

According to properties of sigmoid and tanh function, for any \(t\geq 1\) and any \(1\leq j,l\leq K\), we have

\[\left|h^{(t)}_{j}\right|<1,\quad\left|z^{(t)}\right|<1,\] \[\left|\frac{\partial h^{(t)}_{l}}{\partial c^{(t)}_{l}}\right| =\left|\sigma(c^{(t)}_{l})(1-c^{(t)}_{l})\right|<1,\] \[\left|\frac{\partial o^{(t)}}{\partial z^{(t)}}\right| =\left|1-\tanh^{2}(z^{(t)})\right|<1, \tag{13}\]

which implies that the derivatives of \(o^{(T)}\) with respect to \(b^{\prime}\) and \(V\) are bounded by constants. For the remaining parameters, we claim that for any \(t=2,...,T\), there exists constants \(c^{(t)}\) and polynomials \(C^{(t)}=C^{(t)}(\|w\|)\) respect to \(\|w\|\) with at most degree \(t\) and non-negative coefficients, such that

\[\left|\frac{\partial h^{(t)}_{l}}{\partial b_{i}}\right|\leq C^{ (t)},\quad 1\leq i,l\leq K\] \[\left|\frac{\partial h^{(t)}_{l}}{\partial W_{ij}}\right|\leq C^{ (t)},\quad 1\leq i,j,l\leq K\] \[\left|\frac{\partial h^{(t)}_{l}}{\partial U_{ij}}\right|\leq\|x \|\cdot C^{(t)},\quad 1\leq i,l\leq K,\ 1\leq j\leq p \tag{14}\]
To illustrate the claim (14) by mathematical induction, we first consider the case when \(t=2\). By applying the chain rule, we have

\[\frac{\partial h_{l}^{(2)}}{\partial b_{i}} =\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\frac{ \partial c_{l}^{(2)}}{\partial b_{i}}\] \[=\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\left\{ \delta_{il}+W_{li}\frac{\partial h_{i}^{(1)}}{\partial c_{i}^{(1)}}\right\} \tag{15}\] \[\frac{\partial h_{l}^{(2)}}{\partial W_{ij}} =\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\frac{ \partial c_{l}^{(2)}}{\partial W_{ij}}\] \[=\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\frac{ \partial(\sum_{m=1}^{K}W_{lm}h_{m}^{(1)})}{\partial W_{ij}}=h_{j}^{(1)}\frac{ \partial h_{l}^{(2)}}{\partial c_{l}^{(2)}},\] (16) \[\frac{\partial h_{l}^{(2)}}{\partial U_{ij}} =\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\frac{ \partial c_{l}^{(2)}}{\partial U_{ij}}\] \[=\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\left\{ \frac{\partial\sum_{m=1}^{K}W_{lm}\frac{\partial h_{m}^{(1)}}{\partial U_{ij} }+\frac{\partial\sum_{m=1}^{K}U_{lm}x_{m}^{(2)}}{\partial U_{ij}}\right\}\] \[=\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\bigg{\{} \sum_{m=1}^{K}W_{lm}\frac{\partial h_{m}^{(1)}}{\partial U_{ij}}+\delta_{il}x_ {j}^{(2)}\bigg{\}}\] \[=\frac{\partial h_{l}^{(2)}}{\partial c_{l}^{(2)}}\cdot\bigg{\{} W_{li}x_{j}^{(1)}\frac{\partial h_{i}^{(1)}}{\partial c_{i}^{(1)}}+\delta_{il}x_ {j}^{(2)}\bigg{\}}, \tag{17}\]

Then, by (13), (15), (16) and (17), we can obtain

\[\left|\frac{\partial h_{l}^{(2)}}{\partial b_{i}}\right| \leq 1+\|W\|,\] \[\left|\frac{\partial h_{l}^{(2)}}{\partial W_{ij}}\right| <1,\] \[\left|\frac{\partial h_{l}^{(2)}}{\partial U_{ij}}\right| \leq(1+\|W\|)\|x\|, \tag{18}\]

Therefore, by (18) and \(\|V\|,\|W\|\leq\|w\|\), our claim (14) holds for the case \(t=2\). Now, we assume that (14) holds for case \(t-1\), and consider the case at \(t\). By the chain rule, we have

\[\frac{\partial h_{l}^{(t)}}{\partial b_{i}} =\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\cdot\frac{ \partial c_{l}^{(t)}}{\partial b_{i}}\] \[=\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\cdot\bigg{\{} \delta_{il}+\sum_{m=1}^{K}W_{lm}\frac{\partial h_{m}^{(t-1)}}{\partial b_{i}} \bigg{\}},\] \[\frac{\partial h_{l}^{(t)}}{\partial W_{ij}} =\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\cdot\frac{ \partial c_{l}^{(t)}}{\partial W_{ij}}\]
\[=\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\cdot\frac{ \partial(\sum_{m=1}^{K}W_{lm}h_{m}^{(t-1)})}{\partial W_{ij}}\] \[=\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\bigg{\{}\sum_{m =1}^{K}W_{lm}\frac{\partial h^{(t-1)}}{\partial W_{ij}}+\delta_{il}h_{j}^{(t-1 )}\bigg{\}},\] \[\frac{\partial h_{l}^{(t)}}{\partial U_{ij}} =\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\cdot\frac{ \partial c_{l}^{(t)}}{\partial U_{ij}}\] \[=\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\cdot\bigg{\{} \frac{\partial\sum_{m=1}^{K}W_{lm}h_{m}^{(t-1)}}{\partial U_{ij}}+\frac{ \partial\sum_{m=1}^{K}U_{lm}x_{m}^{(t)}}{\partial U_{ij}}\bigg{\}}\] \[=\frac{\partial h_{l}^{(t)}}{\partial c_{l}^{(t)}}\cdot\bigg{\{} \sum_{m=1}^{K}W_{lm}\frac{\partial h_{m}^{(t-1)}}{\partial U_{ij}}+\delta_{il }x_{j}^{(t)}\bigg{\}}\]

which implies

\[\bigg{|}\frac{\partial h_{l}^{(t)}}{\partial b_{i}}\bigg{|}\leq 1 +K\|W\|\max_{m\leq K}\bigg{|}\frac{\partial h_{l}^{(t-1)}}{\partial b_{i}} \bigg{|}\] \[\bigg{|}\frac{\partial h_{l}^{(t)}}{\partial W_{ij}}\bigg{|}\leq 1 +K\|W\|\max_{m\leq K}\bigg{|}\frac{\partial h_{l}^{(t-1)}}{\partial W_{ij}} \bigg{|},\] \[\bigg{|}\frac{\partial h_{l}^{(t)}}{\partial U_{ij}}\bigg{|}\leq \|x\|+K\|W\|\max_{m\leq K}\bigg{|}\frac{\partial h_{l}^{(t-1)}}{\partial U_{ ij}}\bigg{|}. \tag{19}\]

By using (19) and the induction hypothesis, we can construct an appropriate \(C^{(t)}\) for (14) for the case \(t\), thereby completing the induction.

Now, we proceed to upper bound our objectives. By applying the chain rule, we have

\[\frac{\partial o^{(t)}}{\partial b_{i}} =\frac{\partial o^{(t)}}{\partial z^{(t)}}\sum_{l=1}^{K}V_{l} \cdot\frac{\partial h_{l}^{(t)}}{\partial b_{i}},\] \[\frac{\partial o^{(t)}}{\partial U_{ij}} =\frac{\partial o^{(t)}}{\partial z^{(t)}}\sum_{l=1}^{K}V_{l} \cdot\frac{\partial h_{l}^{(t)}}{\partial U_{ij}},\] \[\frac{\partial o^{(t)}}{\partial W_{ij}} =\frac{\partial o^{(t)}}{\partial z^{(t)}}\sum_{l=1}^{K}V_{l} \cdot\frac{\partial h_{l}^{(t)}}{\partial W_{ij}}. \tag{20}\]

Then, it follows from (13), (19), (20) and Cauchy-Schwarz inequality that

\[\bigg{|}\frac{\partial o^{(T)}}{\partial b_{i}}\bigg{|} \leq\sqrt{K}\|V\|\bigg{|}\frac{\partial h_{l}^{(t)}}{\partial b_{ i}}\bigg{|},\] \[\bigg{|}\frac{\partial o^{(T)}}{\partial W_{ij}}\bigg{|} \leq\sqrt{K}\|V\|\bigg{|}\frac{\partial h_{l}^{(t)}}{\partial W_{ ij}}\bigg{|},\] \[\bigg{|}\frac{\partial o^{(T)}}{\partial U_{ij}}\bigg{|} \leq\sqrt{K}\|V\|\bigg{|}\frac{\partial h_{l}^{(t)}}{\partial U_{ ij}}\bigg{|}. \tag{21}\]
Incorporating (14) and (21), we obtain that

\[\left\|\frac{\partial f_{\lambda}(x,w)}{\partial w}\right\|=\left\| \frac{\partial o^{(T)}}{\partial w}\right\| \leq\sqrt{c_{1}+c_{2}\{C^{(T)}(\|w\|)\}^{2}+c_{3}\{C^{(T)}(\|w\|) \}^{2}\|x\|^{2}}\] \[\leq c_{1}+c_{2}\{C^{(T)}(\|w\|)\}^{2}+c_{3}\{C^{(T)}(\|w\|)\}^{2} \|x\|^{2} \tag{22}\]

where \(c_{1}\) (\(c_{1}>1\)), \(c_{2}\) and \(c_{3}\) are constants that, according to \(|\Lambda|\leq\infty\), can be selected independently of any \(K\), \(p\), and \(T\). Finally, since \(C^{(T)}\) is monotonically increasing with respect to \(\|w\|\), it follows from (12) and (22)that

\[E\sup_{w^{0}\in\mathcal{O}(w^{\star}(\lambda),\rho)}\left\| \frac{\partial f_{\lambda}(X,w)}{\partial w}\right\|_{w=w^{0}}\right\| \leq E\Big{\{}c_{1}+c_{2}\{C^{(T)}(C_{w})\}^{2}+c_{3}\{C^{(T)}(C_ {w})\}^{2}\|X\|^{2}\Big{\}}\] \[=O(1),\]

where the equality follows from our assumption \(E\|X\|^{2}=O(1)\). This verify Assumption 2(ii) in this example.

## Appendix B Proof of Theoretical Results

### Supporting Lemma(s)

We introduce the following lemma that will be used in our proof.

**Lemma 3**: _Let \(\Lambda\) be the available set of hyperparameters, which can have finite or diverging elements. For any \(\lambda\in\Lambda\), \(A_{n}(\lambda)\), \(a_{n}(\lambda)\), and \(B_{n}(\lambda)\) are sequences related to \(\lambda\). Let \(\eta_{n}\) be a real-valued positive sequence that satisfies \(\eta_{n}=o(1)\). Define \(\hat{\lambda}=\operatorname*{argmin}_{\lambda\in\Lambda}\{A_{n}(\lambda)+a_{ n}(\lambda)\}\). If_

\[\sup_{\lambda\in\Lambda}\left|\frac{a_{n}(\lambda)}{B_{n}(\lambda)}\right|=O_ {p}(\eta_{n}), \tag{23}\]

\[\sup_{\lambda\in\Lambda}\left|\frac{A_{n}(\lambda)-B_{n}(\lambda)}{B_{n}( \lambda)}\right|=O_{p}(\eta_{n}), \tag{24}\]

_and there exists a interger \(N\) and a constant \(\kappa\) such that when \(n\geq N\),_

\[\inf_{\lambda\in\Lambda}B_{n}(\lambda)\geq\kappa>0, \tag{25}\]

_then_

\[\frac{A_{n}(\hat{\lambda})}{\inf_{\lambda\in\Lambda}A_{n}(\lambda)}=1+O_{p}( \eta_{n})=1+o_{p}(1). \tag{26}\]

**Proof** By the definition of infimum, there exist a sequence \(\lambda(n)\in\Lambda\) and a non-negative sequence \(\vartheta_{n}(\lambda(n))=o_{p}(1)\) such that

\[\inf_{\lambda\in\Lambda}A_{n}(\lambda)=A_{n}(\lambda(n))-\vartheta_{n}( \lambda(n)). \tag{27}\]
In addition, it follows from (24) that

\[\inf_{\lambda\in\Lambda}\frac{A_{n}(\lambda)}{B_{n}(\lambda)} =\inf_{\lambda\in\Lambda}\left(\frac{A_{n}(\lambda)}{B_{n}(\lambda) }-1\right)+1\] \[\geq-\sup_{\lambda\in\Lambda}\left|\frac{A_{n}(\lambda)}{B_{n}( \lambda)}-1\right|+1=1-O_{p}(\eta_{n}). \tag{28}\]

By (25) and (28), we have

\[\inf_{\lambda\in\Lambda}\frac{|A_{n}(\lambda)-\vartheta_{n}( \lambda(n))|}{B_{n}(\lambda)} \geq\inf_{\lambda\in\Lambda}\frac{A_{n}(\lambda)-\vartheta_{n}( \lambda(n))}{B_{n}(\lambda)}\geq\inf_{\lambda\in\Lambda}\frac{A_{n}(\lambda)}{ B_{n}(\lambda)}-\frac{\vartheta_{n}(\lambda(n))}{\inf_{\lambda\in\Lambda}B_{n}( \lambda)}\] \[\geq-\sup_{\lambda\in\Lambda}\left|\frac{A_{n}(\lambda)}{B_{n}( \lambda)}-1\right|+1-\frac{\vartheta_{n}(\lambda(n))}{\inf_{\lambda\in \Lambda}B_{n}(\lambda)}\] \[\geq 1-O_{p}(\eta_{n})-\frac{\vartheta_{n}(\lambda(n))}{\kappa}. \tag{29}\]

Then, we have that

\[\left|\frac{\inf_{\lambda\in\Lambda}A_{n}(\lambda)}{A_{n}(\hat{ \lambda})}-1\right| =\frac{A_{n}(\hat{\lambda})-\inf_{\lambda\in\Lambda}A_{n}(\lambda) }{A_{n}(\hat{\lambda})}\] \[=\frac{\inf_{\lambda\in\Lambda}(A_{n}(\lambda)+a_{n}(\lambda))-a_ {n}(\hat{\lambda})-\inf_{\lambda\in\Lambda}A_{n}(\lambda)}{A_{n}(\hat{\lambda })}\] \[\leq\frac{A_{n}(\lambda(n))+a_{n}(\lambda(n))-a_{n}(\hat{\lambda}) -A_{n}(\lambda(n))+\vartheta_{n}}{A_{n}(\hat{\lambda})}\] \[\leq\frac{|a_{n}(\lambda(n))|}{A_{n}(\hat{\lambda})}+\frac{|a_{n}( \hat{\lambda})|}{A_{n}(\hat{\lambda})}+\frac{\vartheta_{n}}{A_{n}(\hat{\lambda })}\] \[\leq\frac{|a_{n}(\lambda(n))|}{\inf_{\lambda\in\Lambda}A_{n}( \lambda)}+\frac{|a_{n}(\hat{\lambda})|}{A_{n}(\hat{\lambda})}+\frac{\vartheta _{n}}{A_{n}(\hat{\lambda})}\] \[\leq\frac{|a_{n}(\lambda(n))|}{A_{n}(\lambda(n))-\vartheta_{n}}+ \frac{|a_{n}(\hat{\lambda})|}{A_{n}(\hat{\lambda})}+\frac{\vartheta_{n}}{A_{n} (\hat{\lambda})}\] \[\leq\sup_{\lambda\in\Lambda}\frac{|a_{n}(\lambda)|}{B_{n}( \lambda)}\sup_{\lambda\in\Lambda}\frac{B_{n}(\lambda)}{|A_{n}(\lambda)- \vartheta_{n}|}+\sup_{\lambda\in\Lambda}\frac{|a_{n}(\lambda)|}{B_{n}(\lambda )}\sup_{\lambda\in\Lambda}\frac{B_{n}(\lambda)}{A_{n}(\lambda)}\] \[\qquad\qquad+\sup_{\lambda\in\Lambda}\frac{\vartheta_{n}}{B_{n}( \lambda)}\sup_{\lambda\in\Lambda}\frac{B_{n}(\lambda)}{A_{n}(\lambda)}\] \[\leq\sup_{\lambda\in\Lambda}\frac{|a_{n}(\lambda)|}{B_{n}( \lambda)}\bigg{[}\inf_{\lambda\in\Lambda}\frac{|A_{n}(\lambda)-\vartheta_{n}|} {B_{n}(\lambda)}\bigg{]}^{-1}+\sup_{\lambda\in\Lambda}\frac{|a_{n}(\lambda)|}{ B_{n}(\lambda)}\bigg{[}\inf_{\lambda\in\Lambda}\frac{A_{n}(\lambda)}{B_{n}( \lambda)}\bigg{]}^{-1}\] \[\qquad\qquad+\frac{\vartheta_{n}}{\inf_{\lambda\in\Lambda}B_{n}( \lambda)}\bigg{[}\inf_{\lambda\in\Lambda}\frac{A_{n}(\lambda)}{B_{n}(\lambda)} \bigg{]}^{-1}, \tag{30}\]

where the first equality follows from the definition of \(\hat{\lambda}\), the first and third inequality follows from (27). By (23), (28), (29), and \(\vartheta_{n}(\lambda(n))=o_{p}(1)\), we obtain that

\[\frac{\inf_{\lambda\in\Lambda}A_{n}(\lambda)}{A_{n}(\hat{\lambda})}\leq\frac{O _{p}(\eta_{n})}{1-O_{p}(\eta_{n})}+\frac{O_{p}(\eta_{n})}{1-O_{p}(\eta_{n})-o_{p }(1)/\kappa}+\frac{o_{p}(1)/\kappa}{1-O_{p}(\eta_{n})}. \tag{31}\]
Since \(\eta_{n}=o(1)\), we obtain \(\inf_{\lambda\in\Lambda}A_{n}(\lambda)/A_{n}(\hat{\lambda})=1+O_{p}(\eta_{n})\) based on straightforward derivation and the definition of \(O_{p}\), which also implies \(A_{n}(\hat{\lambda})/\inf_{\lambda\in\Lambda}A_{n}(\lambda)=1+O_{p}(\eta_{n})\). Therefore, we complete the proof of Lemma 3.

### Proof of Theorem 1

**Proof** Define \(L_{n}^{*}(\lambda)=L_{n}(\lambda)-\frac{1}{n}\sum_{i=1}^{n}\{Y_{i}-E(Y_{i}|X_{ i})\}\{Y_{i}+E(Y_{i}|X_{i})\}\). We have

\[\hat{\lambda} =\underset{\lambda\in\Lambda}{\operatorname{argmin}}L_{n}( \lambda)=\underset{\lambda\in\Lambda}{\operatorname{argmin}}L_{n}^{*}(\lambda)\] \[=\underset{\lambda\in\Lambda}{\operatorname{argmin}}\{R_{0}( \lambda)+L_{n}^{*}(\lambda)-R_{0}(\lambda)\}.\]

By applying Lemma 3 with \(A_{n}(\lambda)\), \(B_{n}(\lambda)\), \(a_{n}(\lambda)\), and \(\eta_{n}\) being \(R_{0}(\lambda)\), \(R_{0}^{*}(\lambda)\), \(L_{n}^{*}(\lambda)-R_{0}(\lambda)\), and \(\xi_{n}^{-1}n^{-1/2}\), respectively, it suffices to prove

\[\sup_{\lambda\in\Lambda}\frac{|R_{0}(\lambda)-R_{0}^{*}(\lambda)|}{|R_{0}^{*}( \lambda)|}=O(\xi_{n}^{-1}n^{-1/2})=o(1) \tag{32}\]

and

\[\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-R_{0}(\lambda)|}{|R_{0}^{*}( \lambda)|}=O_{p}(\xi_{n}^{-1}n^{-1/2})=o_{p}(1), \tag{33}\]

to establish Theorem 1. Observe that if (32) holds, the following equation

\[\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-R_{0}^{*}(\lambda)|}{|R_{0}^ {*}(\lambda)|}=O_{p}(\xi_{n}^{-1}n^{-1/2})=o_{p}(1). \tag{34}\]

is sufficient to prove (33). Therefore, we aim to prove (32) and (34).

By Assumption 1 and Assumption 2(ii), we obtain

\[\big{|}f_{\lambda}(X_{0},\hat{w}_{n}(\lambda))-f_{\lambda}(X_{0}, w^{*}(\lambda))\big{|} =\bigg{|}(\hat{w}_{n}(\lambda)-w^{*}(\lambda))^{\top}\frac{\partial f _{\lambda}(X_{0},w)}{\partial w}\Big{|}_{w=w^{0}}\bigg{|}\] \[\leq\|\hat{w}_{n}(\lambda)-w^{*}(\lambda)\|\times\bigg{\|}\frac{ \partial f_{\lambda}(X_{0},w)}{\partial w}|_{w=w^{0}}\bigg{\|}\] \[=O_{p}(n^{-1/2})\times O_{p}(1)=O_{p}(n^{-1/2}). \tag{35}\]

Therefore,

\[\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\big{|}\{f_{\lambda}(X_{0}, \hat{w}_{n}(\lambda))-E(Y_{0}|X_{0})\}^{2}-\{f_{\lambda}(X_{0},w^{*}(\lambda)) -E(Y_{0}|X_{0})\}^{2}\big{|}\] \[=\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\bigg{|}\{f_{\lambda}(X_{0}, \hat{w}_{n}(\lambda))-f_{\lambda}(X_{0},w^{*}(\lambda))\}\Big{|}\{f_{\lambda} (X_{0},\hat{w}_{n}(\lambda))-f_{\lambda}(X_{0},w^{*}(\lambda))\}\] \[\qquad\qquad+2\{f_{\lambda}(X_{0},w^{*}(\lambda))-E(Y_{0}|X_{0}) \}\Big{|}\bigg{|}\] \[\leq\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\Big{|}\|\hat{w}_{n}( \lambda)-w^{*}(\lambda)\|\times\big{\|}\frac{\partial f_{\lambda}(X_{0},w)}{ \partial w}|_{w=w^{0}}\big{\|}\]
\[\times\Big{[}\|\hat{w}_{n}(\lambda)-w^{*}(\lambda)\|\times\big{\|} \frac{\partial f_{\lambda}(X_{0},w)}{\partial w}|_{w=w^{0}}\big{\|}+2\{f_{ \lambda}(X_{0},w^{*}(\lambda))-E(Y_{0}|X_{0})\}\Big{]}\bigg{|}\] \[=\xi_{n}^{-1}\times[O_{p}(n^{-1/2})\times\{O_{p}(n^{-1/2})+O_{p}( 1)\}]\] \[=\xi_{n}^{-1}O_{p}(n^{-1/2})=o_{p}(1), \tag{36}\]

where the second inequality follows from Mean value theorem and the third equality follows from Assumption 2(i) and (35). Consequently, we have

\[\sup_{\lambda\in\Lambda}\frac{|R_{0}(\lambda)-R_{0}^{*}(\lambda)|} {|R_{0}^{*}(\lambda)|}\] \[\leq\xi_{n}^{-1}\sup_{\lambda\in\Lambda}|R_{0}(\lambda)-R_{0}^{* }(\lambda)|\] \[=\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\Big{|}E\big{[}\{f_{ \lambda}(X_{0},\hat{w}_{n}(\lambda))-E(Y_{0}|X_{0})\}^{2}-\{f_{\lambda}(X_{0 },w^{*}(\lambda))-E(Y_{0}|X_{0})\}^{2}\big{]}\Big{|}\] \[\leq E\Big{[}\xi_{n}^{-1}\cdot\sup_{\lambda\in\Lambda}\big{|}\{f_{ \lambda}(X_{0},\hat{w}_{n}(\lambda))-E(Y_{0}|X_{0})\}^{2}-\{f_{\lambda}(X_{0 },w^{*}(\lambda))-E(Y_{0}|X_{0})\}^{2}\big{|}\Big{]}\] \[= O(\xi_{n}^{-1}n^{-1/2})=o(1),\]

where the third inequality follows from Assumption 3 and Jensen inequality and the forth equality follows from (36). This complete the proof of (32).

Now we present the proof of (34). Firstly, we have

\[\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-R_{0}^{*}(\lambda)|}{|R_{0} ^{*}(\lambda)|}\leq\xi_{n}^{-1}\sup_{\lambda\in\Lambda}|L_{n}^{*}(\lambda)-R_{ 0}^{*}(\lambda)|. \tag{37}\]

Then, we examine the supremum on the right hand side of (37):

\[|L_{n}^{*}(\lambda)-R_{0}^{*}(\lambda)|\] \[=\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\big{[}\{f_{\lambda}(X_{i}, \hat{w}_{n}(\lambda))-Y_{i}\}^{2}-\{Y_{i}-E(Y_{i}|X_{i})\}\{Y_{i}+E(Y_{i}|X_{i })\}\big{]}-R_{0}^{*}(\lambda)\bigg{|}\] \[\leq\underbrace{\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\big{[}\{f_{ \lambda}(X_{i},w^{*}(\lambda))-Y_{i}\}^{2}-\{Y_{i}-E(Y_{i}|X_{i})\}\{Y_{i}+E(Y _{i}|X_{i})\}\big{]}-R_{0}^{*}(\lambda)\bigg{|}}_{(I)}\] \[\qquad\qquad\qquad+\underbrace{\bigg{|}\frac{1}{n}\sum_{i=1}^{n} \big{[}\{f_{\lambda}(X_{i},\hat{w}_{n}(\lambda))-Y_{i}\}^{2}-\{f_{\lambda}(X_ {i},w^{*}(\lambda))-Y_{i}\}^{2}\big{]}\bigg{|}}_{(II)}, \tag{38}\]

and we address \((I)\) and \((II)\) in (38) separately.

Regarding \((I)\), we have

\[(I)=\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\big{[}\{f_{\lambda}(X_{i}, w^{*}(\lambda))-E(Y_{i}|X_{i})+E(Y_{i}|X_{i})-Y_{i}\}^{2}\] \[\qquad\qquad\qquad-\{Y_{i}-E(Y_{i}|X_{i})\}\{Y_{i}+E(Y_{i}|X_{i}) \}\big{]}-R_{0}^{*}(\lambda)\bigg{|}\]
\[=\bigg{|}\frac{1}{n}\sum_{i=1}^{n}\big{[}\{f_{\lambda}(X_{i},w^{*}( \lambda))-E(Y_{i}|X_{i})\}^{2}+2f_{\lambda}(X_{i},w^{*}(\lambda))\{E(Y_{i}|X_{i} )-Y_{i}\}\big{]}-R_{0}^{*}(\lambda)\bigg{|}\] \[\leq\frac{1}{n}\Big{|}\sum_{i=1}^{n}\{f_{\lambda}(X_{i},w^{*}( \lambda))-E(Y_{i}|X_{i})\}^{2}-nR_{0}^{*}(\lambda)\Big{|}+\frac{2}{n}\Big{|} \sum_{i=1}^{n}f_{\lambda}(X_{i},w^{*}(\lambda))\{Y_{i}-E(Y_{i}|X_{i})\}\Big{|}. \tag{39}\]

Define

\[S_{n}^{\prime}(\lambda)=\sum_{i=1}^{n}\{f_{\lambda}(X_{i},w^{*}(\lambda))-E(Y _{i}|X_{i})\}^{2}\]

and

\[S_{n}^{\prime\prime}(\lambda)=\sum_{i=1}^{n}f_{\lambda}(X_{i},w^{*}(\lambda)) \{Y_{i}-E(Y_{i}|X_{i})\}.\]

Since the samples are i.i.d., it holds that

\[E(S_{n}^{\prime}(\lambda))=\sum_{i=1}^{n}E(f_{\lambda}(X_{i},w^{*}(\lambda))- E(Y_{i}|X_{i}))^{2}=nR_{0}^{*}(\lambda). \tag{40}\]

It follows from (39) and (40) that

\[(I)\leq\frac{1}{n}\big{|}S_{n}^{\prime}(\lambda)-E(S_{n}^{\prime}(\lambda)) \big{|}+\frac{2}{n}|S_{n}^{\prime\prime}(\lambda)|. \tag{41}\]

Regarding first term on the right hand side of (41), for any \(n\), \(\lambda^{\prime}\in\Lambda\), and \(\delta>0\), we have

\[\Pr\Big{\{}\Big{|}n^{-1}\{S_{n}^{\prime}(\lambda^{\prime})-E(S_{ n}^{\prime}(\lambda^{\prime}))\}\Big{|}>\delta\Big{\}} =\Pr\big{\{}|S_{n}^{\prime}(\lambda^{\prime})-E(S_{n}^{\prime}( \lambda^{\prime}))|>n\delta\big{\}}\] \[\leq\ n^{-2}\delta^{-2}\operatorname{var}(S_{n}^{\prime}(\lambda^ {\prime})) \tag{42}\]

Then, for any \(\delta^{*}>0\) and any \(n\), we set \(\delta=\delta_{n}=n^{-1}(\delta^{*})^{-1}\{\operatorname{var}(S_{n}^{\prime}( \lambda^{\prime}))\}^{1/2}\), plug it into (42), and find it holds for any \(n\) that

\[\Pr\Big{\{}\big{|}n^{-1}\{S_{n}^{\prime}(\lambda^{\prime})-E(S_{ n}^{\prime}(\lambda^{\prime}))\}\big{|}>n^{-1}(\delta^{*})^{-1/2}\{ \operatorname{var}(S_{n}^{\prime}(\lambda^{\prime}))\}^{1/2}\Big{\}}\leq\delta ^{*}, \tag{43}\]

which implies that

\[n^{-1}|S_{n}^{\prime}(\lambda^{\prime})-E(S_{n}^{\prime}(\lambda^ {\prime}))| =O_{p}(n^{-1}\{\operatorname{var}(S_{n}^{\prime}(\lambda^{\prime}) )\}^{1/2})\] \[=O_{p}\Big{(}n^{-1/2}\big{\{}\operatorname{var}[\{f_{\lambda^{ \prime}}(X_{1},w^{*}(\lambda^{\prime}))-E(Y_{1}|X_{1})\}^{2}]\big{\}}^{1/2} \Big{)}\] \[=O_{p}(n^{-1/2})\]

for any \(\lambda^{\prime}\in\Lambda\). Given that \(\Lambda\) is a finite set, we obtain

\[\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\Big{|}\frac{1}{n}\{S_{n}^{\prime}(\lambda )-E(S_{n}^{\prime}(\lambda))\}\Big{|}=O_{p}(\xi_{n}^{-1}n^{-1/2}). \tag{44}\]
For second term on the right hand side of (41), due to the property of conditional expectation, we have

\[E(S_{n}^{\prime\prime}(\lambda)) =\sum_{i=1}^{n}E\Big{\{}E\big{[}f_{\lambda}(X_{i},w^{*}(\lambda)) \{Y_{i}-E(Y_{i}|X_{i})\}|X_{i}|\Big{\}}\] \[=\sum_{i=1}^{n}E\Big{\{}f_{\lambda}(X_{i},w^{*}(\lambda))E\{Y_{i}- E(Y_{i}|X_{i})|X_{i}\}\Big{\}}=0.\]

For any \(\lambda^{\prime\prime}\in\Lambda\) and \(\delta>0\), employing Chebyshev inequality, we have

\[\Pr\Big{\{}\big{|}2n^{-1}S_{n}^{\prime\prime}(\lambda^{\prime \prime})\big{|}>\delta\Big{\}} =\Pr\big{\{}|S_{n}^{\prime\prime}(\lambda^{\prime\prime})|>n \delta/2\big{\}}\] \[\leq 4n^{-2}\delta^{-2}\operatorname{var}(S_{n}^{\prime\prime}( \lambda^{\prime\prime})). \tag{45}\]

Similar to (43), this implies

\[n^{-1}S_{n}^{\prime\prime}(\lambda^{\prime\prime}) =O_{p}(n^{-1}\{\operatorname{var}(S_{n}^{\prime\prime}(\lambda^{ \prime\prime}))\}^{1/2})\] \[=O_{p}(n^{-1/2}\{\operatorname{var}\big{(}f_{\lambda}(X_{1},w^{* }(\lambda^{\prime\prime}))\{Y_{1}-E(Y_{1}|X_{1})\}\big{)}\}^{1/2})\] \[=O_{p}(n^{-1/2})\]

and

\[\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\Big{|}n^{-1}S_{n}^{\prime \prime}(\lambda)\Big{|}=O_{p}(\xi^{-1}n^{-1/2}) \tag{46}\]

By combining (41), (44), and (46), we achieve that

\[\xi_{n}^{-1}\sup_{\lambda\in\Lambda}(I)\leq\xi_{n}^{-1}\sup_{ \lambda\in\Lambda}\Big{|}n^{-1}\{S_{n}^{\prime}(\lambda)-E(S_{n}^{\prime}( \lambda))\}\Big{|}+\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\Big{|}2n^{-1}S_{n}^{ \prime\prime}(\lambda)\Big{|}=O_{p}(\xi^{-1}n^{-1/2}). \tag{47}\]

Now we proceed to handle \((II)\). Apply a technique similar to that in (36), we have

\[\sup_{\lambda\in\Lambda}(II) =\sup_{\lambda\in\Lambda}\frac{1}{n}\Big{|}\sum_{i=1}^{n}\big{[} \{f_{\lambda}(X_{i},\hat{w}_{n}(\lambda))-Y_{i}\}^{2}-\{f_{\lambda}(X_{i},w^{* }(\lambda))-Y_{i}\}^{2}\big{]}\Big{|}\] \[=\sup_{\lambda\in\Lambda}\frac{1}{n}\Big{|}\sum_{i=1}^{n}\big{\{} f_{\lambda}(X_{i},\hat{w}_{n}(\lambda))-f_{\lambda}(X_{i},w^{*}(\lambda))\big{\}}\] \[\qquad\qquad\qquad\times\Big{[}\big{\{}f_{\lambda}(X_{i},\hat{w }_{n}(\lambda))-f_{\lambda}(X_{i},w^{*}(\lambda))\big{\}}+2\{f_{\lambda}(X_{i },w^{*}(\lambda))-Y_{i}\}\Big{]}\Big{|}\] \[=O_{p}(n^{-1})+O_{p}(n^{-1/2})=O_{p}(n^{-1/2}),\]

where the second equality follows from (35) and the last equality follows from Assumption 2(i). Then, by Asssumption 4, we obtain that

\[\xi_{n}^{-1}\sup_{\lambda\in\Lambda}(II)=O_{p}(\xi_{n}^{-1}n^{-1/2})=o_{p}(1). \tag{48}\]
Finally, by combining (37), (38), (47), and (48), we have

\[\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-R_{0}^{*}(\lambda) |}{|R_{0}^{*}(\lambda)|} \leq\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\{(I)+(II)\}\] \[\leq\xi_{n}^{-1}\sup_{\lambda\in\Lambda}(I)+\xi_{n}^{-1}\sup_{ \lambda\in\Lambda}(II)\] \[=O_{p}(\xi_{n}^{-1}n^{-1/2})=o_{p}(1),\]

which demonstrates that (34) holds. Therefore, we complete the proof of Theorem 1.

### Proof of Theorem 2

Proof.: We consider (8), the first justification in Theorem 2. Rewrite the definition of \(\hat{\lambda}\) as

\[\hat{\lambda}=\operatorname*{argmin}_{\lambda\in\Lambda}\left\{L_{0}( \lambda)+(L_{n}^{*}(\lambda)-L_{0}(\lambda))\right\}.\]

Similar to the proof of Theorem 1, applying Lemma 3 with \(A_{n}(\lambda)\), \(B_{n}(\lambda)\), \(a_{n}(\lambda)\), and \(\eta_{n}\) being \(L_{0}(\lambda)\), \(R_{0}^{*}(\lambda)\), \(L_{n}^{*}(\lambda)-L_{0}(\lambda)\) and \(\xi_{n}^{-1}n^{-1/2}+\xi_{n}^{-1}n_{0}^{-1/2}\), it suffices to prove

\[\sup_{\lambda\in\Lambda}\frac{|R_{0}^{*}(\lambda)-L_{0}(\lambda)|}{|R_{0}^{*}( \lambda)|}=O_{p}(\xi_{n}^{-1}n^{-1/2}+\xi_{n}^{-1}n_{0}^{-1/2})=o_{p}(1) \tag{49}\]

and

\[\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-L_{0}(\lambda)|}{|R_{0}^{*}( \lambda)|}=O_{p}(\xi_{n}^{-1}n^{-1/2}+\xi_{n}^{-1}n_{0}^{-1/2})=o_{p}(1). \tag{50}\]

Notice that \(L_{n}^{*}(\lambda)-L_{0}(\lambda)\) can be rewritten to \((L_{n}^{*}(\lambda)-R_{0}^{*}(\lambda))+(R_{0}^{*}(\lambda)-L_{0}(\lambda))\). Besides, according to the conclusion of Theorem 1, we have

\[\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-R_{0}^{*}(\lambda)|}{|R_{0}^ {*}(\lambda)|}=O_{p}(\xi_{n}^{-1}n^{-1/2})=o_{p}(1), \tag{51}\]

We claim that (49) is sufficient to prove (50). In fact, the sufficiency is easily obtained by

\[\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-L_{0}(\lambda)|}{|R_{0}^{*} (\lambda)|}\leq\sup_{\lambda\in\Lambda}\frac{|L_{n}^{*}(\lambda)-R_{0}^{*}( \lambda)|}{|R_{0}^{*}(\lambda)|}+\sup_{\lambda\in\Lambda}\frac{|R_{0}^{*}( \lambda)-L_{0}(\lambda)|}{|R_{0}^{*}(\lambda)|}.\]

Therefore, we come to prove (49).

Notice that

\[\sup_{\lambda\in\Lambda}\frac{|R_{0}^{*}(\lambda)-L_{0}(\lambda)|} {|R_{0}^{*}(\lambda)|} \leq\xi_{n}^{-1}\sup_{\lambda\in\Lambda}|L_{0}(\lambda)-R_{0}^{*} (\lambda)|\] \[\leq\xi_{n}^{-1}\sup_{\lambda\in\Lambda}\left\{|L_{0}(\lambda)-L_ {0}^{*}(\lambda)|+|L_{0}^{*}(\lambda)-R_{0}^{*}(\lambda)|\right\}. \tag{52}\]

Similar to (36) in the proof of Theorem 1, we have

\[\xi_{n}^{-1}\sup_{\lambda\in\Lambda}|L_{0}(\lambda)-L_{0}^{*}(\lambda)|\]
\[= \xi_{n}^{-1}\sup_{\lambda\in\Lambda}\left|n_{0}^{-1}\sum_{j=1}^{n_{0}} \left[\{f_{\lambda}(X_{j}^{0},\hat{w}_{n}(\lambda))-E(Y_{j}^{0}|X_{j}^{0})\}^{2}- \{f_{\lambda}(X_{j}^{0},w^{*}(\lambda))-E(Y_{j}^{0}|X_{j}^{0})\}^{2}\right]\right|\] \[= \ O_{p}(\xi_{n}^{-1}n_{0}^{-1/2})=o_{p}(1), \tag{53}\]

where the second equality follows from (35) and the third equality follows from Assumption 4. Then, by the definition that \(R_{0}^{*}(\lambda)=E(L_{0}^{*}(\lambda))\), it holds for \(\lambda\in\Lambda\), \(\delta>0\) that

\[\Pr\left(|L_{0}^{*}(\lambda)-R_{0}^{*}(\lambda)|>\delta\right) \leq\delta^{-2}\operatorname{var}(L_{0}^{*}(\lambda))\] \[=n_{0}^{-1}\delta^{-2}\operatorname{var}\left[\{f_{\lambda}(X_{0 },\hat{w}_{n}(\lambda))-E(Y_{0}|X_{0})\}^{2}\right] \tag{54}\]

By Assumption 6, we obtain

\[\xi_{n}^{-1}\sup_{\lambda\in\Lambda}|L_{0}^{*}(\lambda)-E\{L_{0}^{*}(\lambda) \}|=O_{p}(\xi_{n}^{-1}n_{0}^{-1/2})=o_{p}(1). \tag{55}\]

Combining (52), (53), and (55), we obtain (49). Thus we complete the proof of (8).

Now we are ready to prove (9) and (10). According to (8), we have

\[\frac{L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}(\lambda)}{\inf_{ \lambda\in\Lambda}L_{0}(\lambda)}=\frac{L_{0}(\hat{\lambda})}{\inf_{\lambda \in\Lambda}L_{0}(\lambda)}-1=o_{p}(1) \tag{56}\]

By the uniformly integrablity in Assumption 7(ii), we obtain

\[\frac{L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}(\lambda)}{\inf_{ \lambda\in\Lambda}L_{0}(\lambda)}\stackrel{{ L_{1}}}{{\longrightarrow}}0,\]

which implies

\[E\Big{\{}\frac{L_{0}(\hat{\lambda})}{\inf_{\lambda\in\Lambda}L_{0}(\lambda)} \Big{\}}\longrightarrow 1. \tag{57}\]

This proves (9). As for (10), we have

\[E\big{\{}L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}( \lambda)\big{\}}\] \[\leq E\big{\{}\frac{L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0 }(\lambda)}{\inf_{\lambda\in\Lambda}L_{0}(\lambda)}\big{\}}\times E\big{[}\inf _{\lambda\in\Lambda}L_{0}(\lambda)\cdot\{L_{0}(\hat{\lambda})-\inf_{\lambda \in\Lambda}L_{0}(\lambda)\}\big{]}\] \[< C\times E\big{\{}\frac{L_{0}(\hat{\lambda})-\inf_{\lambda\in \Lambda}L_{0}(\lambda)}{\inf_{\lambda\in\Lambda}L_{0}(\lambda)}\big{\}}\times E \{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}\]

where the first inequality follows from Cauchy-Schwarz inequality and the second inequality follows from Assumption 7(i). This implies

\[0\leq\frac{E\big{\{}L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}(\lambda )\big{\}}}{E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}}\leq C\times E\big{\{} \frac{L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}(\lambda)}{\inf_{\lambda \in\Lambda}L_{0}(\lambda)}\big{\}}. \tag{58}\]

By (57), we know that the right hand side of (58) converges to \(0\), which also implies

\[\frac{E\big{\{}L_{0}(\hat{\lambda})-\inf_{\lambda\in\Lambda}L_{0}(\lambda) \big{\}}}{E\{\inf_{\lambda\in\Lambda}L_{0}(\lambda)\}}\longrightarrow 0.\]

Therefore, we completes the proof of Theorem 2.
## Appendix C Design of models in Image classification problems

We employ the same structure for model \(F_{1}\) and \(F_{2}\), which follows the standard AlexNet architecture (Krizhevsky et al., 2012), with necessary modifications tailored to our tasks. The model consists of two parts: feature extraction and classification. Specifically, the feature extraction part is composed of the following layers in sequence: convolutional layer 1 (32 filters of \(5\times 5\)), convolutional layer 2 (64 filters of \(3\times 3\)), max-pooling layer 1 (kernel size \(2\times 2\) and stride 2), convolutional layer 3 (96 filters of \(3\times 3\), convolutional layer 4 (64 filters of \(3\times 3\), convolutional layer 5 (32 filters of \(3\times 3\), max-pooling layer 1 (kernel size \(2\times 2\) and stride 1)). The classification part comprises three fully connected layers with two ReLu activation functions, having output sizes of 512, 128, and 1. Dropout layers are placed between the fully connected layers. An input image will be first normalized to have values in \([-1,1]\) and then processed through the feature extraction and classification components, and a sigmoid activation function is applied at the end of the network to generate the final output probabilities.

We train models \(F_{1}\) and \(F_{2}\) following the procedure outlined in Section 4.2, using 60000 relabeled samples from the training set of the Fashion-MNIST or MNIST datasets, respectively. The achieved accuracies of \(F_{1}\) and \(F_{2}\) on their repective 10000 test samples exceeds 99.5%.
